"Resuming" a large file from USB copies it into NAND (filling NAND)
|Reported by:||gnu||Owned by:||tomeu|
|Keywords:||relnote||Cc:||jg, kimquirk, legutierr, mstone, krstic, walter|
|Deployments affected:||Action Needed:|
(From rt# 2968) User was browsing media from a USB drive, using Watch & Listen. Their internal flash filled up, and it made their system unbootable.
I tried to reproduce it tonight. Easy.
I installed Watch&Listen from the wiki Activities page, put a 300+MB mp3 file on a USB memory stick, inserted it into the XO, went to the journal, clicked on the USB logo, found the file (despite the Journal tearing off half of its file name), clicked it to get details, hovered over the upper right icon, and resumed it with Watch & Listen. I had less than 300MB free in my internal NAND flash.
Something started copying the file into internal flash. From previous experience (#5719) I looked at the datastore process. It had 21 file descriptors open (in /proc/1618/fd), and 20 and 21 were:
20 -> /media/COMPUSA/raiders_of_old_california.mp3
21 -> /home/olpc/.sugar/default/data/raiders_of_old_california.mp3
The file was copied until the NAND filesystem filled up. (9 MB free). A truncated copy of the file was left in .sugar/default/data.
After I closed the "detail" page from the Journal, a second copy of the file started to get made in the data/ directory!!! It didn't get very far, since there was minimal space left.
The Journal was unable to unmount the USB stick, because "umount" failed with a file system full error.
Earlier, I tried the same thing with a 15MB mp3 file. The file also got copied to the datastore. It was there while the file was played by the media player. It was removed after the media player terminated.
Then I rebooted and tried the same thing with a 125MB Ogg Theora file from the Ubuntu Month of Screencasts (http://screencasts.ubuntu.com/videos/20070911_updating_and_upgrading_theora_400k_vorbis_1280x720.ogg). The Watch&Listen activity started up, even while the USB light was blinking as the file was copied -- but the activity could not access the file. The Play button was useless; the timeline didn't show the length of the video file. Not until the entire file was copied -- more than a minute. Ultimately it never was able to play the file at all -- I don't know why. But TWO copies of it were made in default/data -- not two links to the same file, but two separate copies. I don't know why. I terminated the program, yet the file copies remained! When I tried to resume the same file with EToys, it made a THIRD copy, which filled the file system, causing all kinds of mischief including making X crash and restart every 20 seconds or so.
After deleting the files to make the X crashes stop, I then went back and tried resuming this file with eToys. eToys started up, said "Opening journal entry 20070911_updating_and_upgrading_theora_400k_vorbis_1280x720", then shortly afterward, said "Error: dbus send timed out". Apparently it was copying the file yet again, and etoys was too impatient to wait. Yep, even after it timed out, the USB access light is flashing, and now another copy is appearing.
There's something deeply broken about how Sugar, the Journal, and the Datastore deal with files -- particularly on removable or networked media.
The current design precludes an activity from working with any file larger than the available NAND space (1GB maximum). No activity is ever going to play a DVD off a USB DVD drive. No activity is going to be able to record more than 1GB of video from the onboard video camera, no matter if 300GB of hard-drive space is available (either plugged in on USB, or over the network in a school server).
Even if the current policy of copy-everything was defensible, it's incredibly inefficient. The activity can't use the file until it has been fully copied (and automatically compressed!). Then the copy has to be deleted. Both operations are expensive and useless, when all that is really needed is an open().
Instead, like everything else on Linux, activities should be reading the file(s) they operate on directly from whatever filesystem they are on -- whether that's local or remote, removable or not.
Change History (21)
comment:6 Changed 8 years ago by jg
- Keywords relnote added; security? removed
- Milestone changed from Update.1 to Update.2