Opened 5 years ago

Closed 2 years ago

#10040 closed enhancement (fixed)

we don't use all available SD card space

Reported by: wad Owned by: dsd
Priority: blocker Milestone: 12.1.0
Component: build-system Version: Development build as of this date
Keywords: Cc: cjb, pgf, sridhar
Blocked By: Blocking:
Deployments affected: Action Needed: no action
Verified: no

Description

Using Q3A25 on an XO-1.5, I'm having trouble using OFW to fs-update an Adata 4GB class 2 SD card. To my knowledge, this is the first time we've tried this particular model in an XO-1.5.

I am installing the os110.zd image (successfully installed from these USB keys on other SD cards). The error message from OFW is: "Image size is larger than output device", and then it aborts the fs-update.

fdisk reports that the factory installed partition on these cards starts at 8192, and ends at 7698431 (units of 512 bytes), with 3845120 blocks in the partition.

/sys reports the OEM ID as 0x534d, name as 0x0, fwid as 0x0, and hwid as 0x1.

This has been reproduced on multiple Adata SD cards of this model.

Attachments (2)

dmesg.txt.gz (7.4 KB) - added by mavrothal 2 years ago.
dmesg after failed fs expansion
auckland-dmesg-120602.tar.bz2 (14.6 KB) - added by carrott 2 years ago.
dmesg logs from auckland testing, first boot of 12.1.0 os12 after installing new kernel, all failed to expand, at least two different failure modes

Download all attachments as: .zip

Change History (49)

comment:1 Changed 5 years ago by wad

  • Cc cjb added

As I suspect that this might be due to the image size we are generating, and not a flaw in OFW, I'm cc'ing cjb.

comment:2 Changed 5 years ago by wmb@…

  • Component changed from ofw - open firmware to build-system
  • Owner changed from wmb@… to cjb

Your suspicion is correct, so I am changing the component to build-system and the owner to cjb.

comment:3 Changed 5 years ago by cjb

  • Cc pgf added

Paul's looking at a solution involving shipping one image (smaller than 2G), which is resize2fs'd up to whatever the card's blocksize is. If that works robustly, it would be a better solution to this problem than just making the 4G size a little smaller for everyone.

comment:4 follow-up: Changed 5 years ago by pgf

i just timed the expansion of our 2g (201) image to fit my 4g card. it took 38.4 seconds, real-time. the filesystem was unmounted, and i was running from a rootfs on a USB stick. although the resize2fs man page claims to be able to resize mounted filesystems, it got EBUSY when trying to open /dev/mmcblk0p2 when that was the root fs. (we could get around this by running resize2fs in the initramfs before the mount.)

i think 38 seconds is too long to delay the first boot. however, this technique might still be useful if we continued building two images: one of them 1.6G, the other 3.6G (for example).

this would be a good way to get full use of bigger cards. the Adata card has a device capacity of 7698432 512-byte blocks. by comparison, the card in my C1 has 7954432. that's over 3% smaller.

but we need to think about whether the risk of a power cycle, if resize2fs is unsafe in that regard (likely), makes it worth it.

comment:5 in reply to: ↑ 4 Changed 5 years ago by cjb

Replying to pgf:

but we need to think about whether the risk of a power cycle, if resize2fs is unsafe in that regard (likely), makes it worth it.

Let's test that. (i.e. start the resize2fs, wait twenty seconds, pull the battery.)

comment:6 Changed 5 years ago by Quozl

I urge caution; using resize2fs this way means we're now also using Linux as an installer, rather than just an image builder and image runner.

Previous Linux distributions have solved the initial layout variability by running an installer than does mke2fs. We haven't had to; there's enough functionality in OpenFirmware to let us unpack an image block by block. Apart from adding mke2fs and tar support to OpenFirmware, or writing an installer, I've no suggestion. ;-)

I'm also not sure that testing by pulling the power during resize2fs will generate useful conclusions. A code review might be more appropriate. The comment in resize2fs.c immediately prior to move_itables() says:

/* Resize processing, phase 5.
 *
 * In this phase we actually move the inode table around, and then
 * update the summary statistics.  This is scary, since aborting here
 * will potentially scramble the filesystem.  (We are moving the
 * inode tables around in place, and so the potential for lost data,
 * or at the very least scrambling the mapping between filenames and
 * inode numbers is very high in case of a power failure here.)
 */

Lastly, the man page for resize2fs has this gem: "If the filesystem is mounted, it can be used to expand the size of the
mounted filesystem, assuming the kernel supports on-line resizing.
" ... and fs/ext3/resize.c points out in the comment to ext3_group_extend() that a resize can be done with mount options "remount,resize=<size>". This looks more fun than resize2fs. Does it work for us?

comment:7 Changed 5 years ago by pgf

thank you. in fact, my last words to chris as i left the office were "i think we need to look at the code", and i was just sitting down to do that. i was hoping it might be safe because after pulling the plug in the middle, the filesystem came up clean, and i believe has a size somewhere between it's old size and the new requested size. but moving the inode table in place doesn't sound good.

as for "-o remount,resize", the mount man page only documents that option for jfs and reiserfs. i doubt it would be easier for the kernel to do it for ext2/3 than for resize2fs.

comment:8 Changed 5 years ago by pgf

the plan of record now is to place a flag file in /, indicating that it's okay for resize2fs to be run when the system boots. since the resize can be done online, it will be set up to occur while the laptop is being used. on success, the flag file will be removed, so that the resize only happens automatically just once. it can be retriggered by touching the file, or suppressed entirely by a deployment by removing the file during build.

issue: an olpc-update will cause the flag file to come back, i believe.

in order for this to work, the partition must be sized to fill the disk, otherwise the filesystem will have no room to grow.

comment:9 Changed 5 years ago by cjb

Do you think we should worry about a case where the resize consistently fails, yet is retried every boot? (I don't know what'd cause it to fail..)

comment:10 Changed 5 years ago by pgf

good point. i can certainly make the removal of the flag file be unconditional.

probably better to be cautious at this stage. a full log of the process is kept, so we can see what happens/happened.

comment:11 Changed 4 years ago by Quozl

Any interesting progress on this one? ;-)

comment:12 Changed 4 years ago by pgf

online automatic resizing proved to take too long and be too expensive in terms of machine responsiveness to be done a) at boot, and b) in the background. the temptation to power-cycle the machine because it appears "hung" during the operation is just too great. (and power-cycling during an online resize is a bad thing.)

i think we should fall back to a control panel button to "Allow full use of my storage", or some such wording. a) we should have a place for reporting disk usage anyway, b) we could provide feedback of the operation in progress, and c) users are less likely to interrupt operations that they started explicitly.

a requirerement we still have in this case, however, is that OFW set the partition size o the last partition to be the full remainder of the disk. i think it's because of the complicated mount setup, or maybe just because it's the root fs, but linux won't honor partition size changes on this fs until after a reboot.

comment:13 Changed 4 years ago by edmcnierney

  • Priority changed from high to normal

User effect is small, so setting priority to normal per review with cjb.

comment:14 Changed 4 years ago by Quozl

  • Milestone changed from 10.1.1 to 10.1.2

comment:15 follow-up: Changed 4 years ago by Quozl

Impact will be greater if internal SD is raised to 8GB.

comment:16 in reply to: ↑ 15 Changed 4 years ago by mikus

Replying to Quozl:

Impact will be greater if internal SD is raised to 8GB.

I've been using an 8GB internal micro-SD for a while now. 'fs-update' works fine with the non-qualified .zd file. [It creates an approximately 4GB partition on the micro-SD card. I don't currently care about the lost space.]

When I first started experimenting with the 8GB micro-SD, I wiped it clean an used 'fdisk' (and 'mkfs') to put an approximately 8GB empty partition on it. I then did 'fs-update' (with the build that was then current) to an external non-micro SD card, and used 'rsync' to copy that external card content onto the micro-SD card. The XO-1.5 booted fine (and ran fine) from the build sitting in the 8GB partition on the micro-SD card.

comment:17 Changed 4 years ago by Quozl

Such a procedure, while technically correct, is unsuited to deployment teams that lack the skills you have, and it would not easily scale up to a school laptop population.

If a deployment is ordering a million laptops with 8GB internal SD and we deliver them preinstalled with a 4GB image, we wouldn't be honestly satisfying the requirements.

comment:18 Changed 4 years ago by dsd

  • Milestone changed from 10.1.2 to 10.1.3
  • Summary changed from OFW unable to burn 4GB image onto 4GB SD card to we don't use all available SD card space

Not a regression (always existed for XO-1.5), moving to next milestone

comment:19 Changed 4 years ago by sridhar

  • Cc sridhar added

comment:20 Changed 4 years ago by dsd

  • Cc dsd added

I suspect the time needed and impact on system performance during resize is mostly because the process has to zero out a load of inode tables.

John Gilmore notes that new kernels have an option to defer the inode table zeroing until much later:

There's now a mkfs option that doesn't zero all the inodes while
building the file system.  It instead sets an extent for the inode
tables, in the superblock, and the kernel zeroes more inodes as it
needs to, in parallel with ordinary system operation.  It also has a
similar option for the block groups.  This is the lazy_itable_init
option set with mke2fs -E and the uninit_bg option set with mke2fs -O.

uninit_bg works only with ext4.  I think lazy_itable_init works with
ext3.

This may make resize a much quicker and cheaper operation.

comment:21 Changed 3 years ago by dsd

  • Milestone changed from 11.2.0-M3 to Future Release
  • Type changed from defect to enhancement

comment:22 Changed 3 years ago by pgf

  • Action Needed changed from diagnose to design
  • Milestone changed from Future Release to 12.1.0

it's time to look at this issue seriously again. on a 1.75, with 8G EMMC, with an initial installed os27 (i.e., kernel 3.0.19, plus olpc patches) image size of 4G:

bash-4.1# time olpc-nosleep resize2fs /dev/mmcblk0p2
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mmcblk0p2 to 1924096 (4k) blocks.
The filesystem on /dev/mmcblk0p2 is now 1924096 blocks long.


real    0m13.052s
user    0m0.010s
sys     0m0.030s

bash-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mmcblk0p2        7.3G  1.7G  5.6G  23% /
...

comment:23 Changed 3 years ago by martin.langhoff

Agreed, definitely a target for 12.1.0.

comment:24 Changed 3 years ago by pgf

  • Blocked By 11690 added

comment:25 Changed 3 years ago by Quozl

performance data point, XO-1.75, os29, 4gb resize to 8gb external SD class 4 15MB/s*, 1min 39sec.

comment:26 Changed 3 years ago by dsd

  • Cc dsd removed
  • Owner changed from cjb to dsd

Emiliano posted one way of doing this:
http://lists.laptop.org/pipermail/devel/2012-March/034516.html

I'll try and adopt this project this development cycle. Some initial thoughts:

  • Needs rewrite in sh because python is not included in our boot initramfs
  • Need to confirm if the firmware will do the partition resize or if the initramfs will do it (OFW doing it would great)
  • Check the logic used for deciding whether to resize the partition or not - is there a more future-proof way than looking at XO version? Is there a trivial way of determining if there is free space beyond the last partition on disk?
  • Use the new ext4 online resize which is supposedly much quicker (use the opportunity to ask the author how safe this is in terms of unexpected interruption/power loss), and do it from olpc-configure rather than initramfs, to avoid needing to put resize2fs in the initramfs

comment:27 Changed 3 years ago by Quozl

further performance data points, XO-1.75, os29, using eMMC instead of external SD,

unitavailable 1k blocksused %resize timeavailable 1k blocksused %
pouzl-c218363640%14.836s607506820%
pouzl-d218449240%14.822s606781220%
pouzl-e218460040%0.148s227666439%
pouzl-a217601640%0.137s226808039%
pouzl-x218465240%0.140s227671639%
pouzl-v218457240%13.372s607600420%

all tests done via ssh while Sugar home view was displayed and system was idle.

the results are comparable with pgf's from a few weeks ago, and suggest we should continue for XO-1.75.

comment:28 Changed 3 years ago by Quozl

  • Blocked By 11690 removed

(In #11690) Previous plan abandoned based on Daniel and Martin's advice that initramfs will need to do the resize anyway for units in the field that are upgraded using fs-update.

A reduced patch has been committed, which has no impact on fs-update, and requires no changes to olpc-os-builder.

comment:29 Changed 3 years ago by dsd

dracut-modules-olpc 345dbd3b35 adjusts the partition table. fs resize part still pending.

comment:30 Changed 3 years ago by dsd

x86-3.3 b8ba15396 has the required fixes for ext4 online resize, and is being built with dracut-modules-0.6.5 (for the initramfs part).

comment:31 Changed 3 years ago by dsd

  • Action Needed changed from design to add to build

olpc-os-builder 2ac0e0db602e creates output disk images based on the size of the data contained (plus a little bit of free space)

olpc-utils-2.0.6 grows the root filesystem on first boot. boot is blocked while the resize happens, but it's really quick with the new online resize in Linux 3.3. Resizing XO-1.5 partition from 1.6GB to 3.7GB takes 0.7 seconds.

All done!

comment:32 Changed 3 years ago by pbrobinson

  • Action Needed changed from add to build to test in build

comment:33 Changed 3 years ago by greenfeld

With 12.1.0 os8 I don't think the resize is guaranteed to happen.

I have a mix of XO 1.5's and 1.75s, most of which where the resize happened. But one XO-1.75 reports having a 2 GB main partition with only ~110 MB free, and fdisk shows /dev/mmcblk0 has not allocated ~1.8 GB of space.

Rebooting this XO does not result in the resize occurring.

I would need more information on how to research this to diagnose why.

comment:34 Changed 3 years ago by greenfeld

  • Action Needed changed from test in build to diagnose

Talked with dsd on IRC - This does not seem to be reliably done.

Output from fdisk -l /dev/mmcblk0 on an XO-1.75 where the repartioning did not occur:

Disk /dev/mmcblk0: 3959 MB, 3959422976 bytes                                    
32 heads, 32 sectors/track, 7552 cylinders, total 7733248 sectors               
Units = sectors of 1 * 512 = 512 bytes                                          
Sector size (logical/physical): 512 bytes / 512 bytes                           
I/O size (minimum/optimal): 512 bytes / 512 bytes                               
Disk identifier: 0x00000000                                                     
                                                                                
        Device Boot      Start         End      Blocks   Id  System             
/dev/mmcblk0p1   *        8192      139263       65536   83  Linux              
/dev/mmcblk0p2          139264     4153503     2007120   83  Linux  

comment:35 Changed 2 years ago by mavrothal

Happened to me also on SHC2060006C (4GB card).
Tried 2 times with 21009o2.zd same result.
Image/stick works fine on another XO-1.75
dmesg after boot is attached

Changed 2 years ago by mavrothal

dmesg after failed fs expansion

comment:36 Changed 2 years ago by Quozl

Worked perfectly for me first time on XO-1.75 C2 SKU204 which has an 8GB eMMC.

comment:37 Changed 2 years ago by mavrothal

Worked fine with 21010o2 on the same machine where 21009o2 failed twice.
Kernel log on first boot shows that the card size was detected and resized correctly.
Just in case here is the card data:

cid: 45010053454d3034479022d3f01a3e00
csd: d00f00320f5903fffffffdff92404000
date: 03/2011
enhanced_area_offset: 18446744073709551594
enhanced_area_size: 4294967274
erase_size: 262144
fwrev: 0x0
hwrev: 0x0
manfid: 0x000045
name: SEM04G
oemid: 0x0100
preferred_erase_size: 2097152
serial: 0x22d3f01a
type: MMC

comment:38 Changed 2 years ago by dsd

Yes, this has always been a bug that only occurred sometimes - i.e. reflash (even the same version) and you will probably find different results.

I caught one on build 10.

[    5.470179] Dsize 7733248 Psize 4039040 Pstart 139264 Pend 4178304
[    5.470321] Disk should be resized.
[    5.470427] Will attempt resize
[    6.062281] hub 1-0:1.0: hub_suspend
[    6.062317] usb usb1: bus auto-suspend
[    6.062330] pxau2o-ehci pxau2o-ehci.0: suspend root hub
[    6.292950] Try resize: sfdisk -N2 -uS -S 32 -H 32 /dev/disk/mmc/mmc2
[    6.426806] sfdisk returned 1

Unconclusive unfortunately. I've modified future initramfs's to capture stderr from sfdisk which will hopefully tell us more.

comment:39 Changed 2 years ago by dsd

This can be tested in 12.1.0 build 12 (and maybe earlier) - we need to catch this happening again and look at dmesg output of first boot.

comment:40 Changed 2 years ago by Quozl

Test in os11 or defer to os13, because of wrong kernel in os12.

comment:41 Changed 2 years ago by dsd

This can still be tested on XO-1.5 in build 12, if anyone feels like it.

comment:42 Changed 2 years ago by Quozl

Oh, sorry, I thought you wanted first boot testing only.

Changed 2 years ago by carrott

dmesg logs from auckland testing, first boot of 12.1.0 os12 after installing new kernel, all failed to expand, at least two different failure modes

comment:43 Changed 2 years ago by carrott

In auckland-dmesg-120602.tar.bz2 (attached) we find two different failure modes after installing 12.1.0 os12 on 4 XO-1.75s. Most of them repartitioned the drive correctly but did not expand the filesystem. Moa failed to resize the partition table, complaining the filesystem was in use.

Stripping off the timestamps with cut -c 16- makes the logs very comparable.

comment:44 Changed 2 years ago by dsd

  • Priority changed from normal to blocker

Selected as blocker in 12.1.0 release meeting.

comment:45 Changed 2 years ago by dsd

  • Action Needed changed from diagnose to add to build

udevd's block device probing was sometimes causing the disk to be "in use" when sfdisk kicked in. Fixed in dracut-modules-olpc-0.6.8 by waiting for udev to settle first.

comment:46 Changed 2 years ago by dsd

  • Action Needed changed from add to build to test in build

This can be tested in 12.1.0 build 13 and onwards.

comment:47 Changed 2 years ago by greenfeld

  • Action Needed changed from test in build to no action
  • Resolution set to fixed
  • Status changed from new to closed

I have not seen installs not resize yet with 12.1.0 os13, 14, & 15.

If someone does see this happen then they should reopen the ticket.

Note: See TracTickets for help on using tickets.