orphaned files

[walter]

Details of this bug were posted to the wiki..

http://wiki.laptop.org/go/Ask_OLPC_a_Question/New#Orphaned_files_are_created_-_and_cannot_be_easily_deleted_or_seen_from_the_Journal.2C_when_they_fail_to_open.

[jg]

This seems pretty serious to me. How hard is it to fix?

[tomeu]

If I correctly understand the issue, this is not a problem at all after ship.2. We stopped copying files to there and are hard linking them instead.

Could someone try to reproduce the issue in one of the update.1 builds?

[tomeu]

But I guess we should anyway clean up that dir after booting, right?

[bernie]

What would be a reliable way to identify the orphaned files? We'd better be extra cautious when deleting user data.

[tomeu]

We don't need to delete any file. We wish to remove the hard links from that dir, because I guess every hard link takes up some space.

[jg]

The links take almost no space.

If there is a hard link to a file, even if the other link is removed, the space will not be freed...

So long as the link count is one or higher, files will not actually be deallocated.

[tomeu]

I cannot see this happening any more in update.1 builds, although I guess some hard links could be leaked in /activities/uid_to_instance_dir/* by bad-behaved activities. I'm not sure if those directories are cleaned up when booting.

Perhaps Ivan has some plans for the new datastore that won't use hard links?

Anyway, I don't see this as a blocker for update1.

[jg]

tomeu, mstone: if this has been tested as fixed, then please close the bug...

[marco]

jg, not fully fixed yet.

Michael is going to see if we can have rainbow rm -rf the instance dir, as a solution for Update.1. (on the longer term we will have to revisit the exact activity instance lifecycle)

[chihyu]

I am creating test cases for this issue, but the original bug description has disappeared from the wiki. It will be great if someone could suggest how to test this issue, and whether it should be tested in Update.1.

Thanks a lot!

Test cases: http://wiki.laptop.org/go/Update.1#Security.2FUpgrades

[gnu]

The original problem, I think, is that the Datastore made copies of files that were deleted or opened from USB storage devices or SD cards. This was popular, as G1G1 people tried to play movies or music from USB disks or flash sticks. Copying in a 300MB movie, however, tended to fill the NAND filesystem. And then nothing deleted those copies of the files. These are the bugs: #5719, #5744. Here is another bug that mentions this issue, in the context of ship.1 to update.1 upgrades: #5955.

Recipe to reproduce:

• Install Ship.1 build 650.
• Plug in a USB stick containing a large file (e.g. the 128MB Ogg Theora from #5744)
• Attempt to open that file (resulting in a copy in NAND).
• Let the USB light stop blinking (Sugar is done copying).
• Check "df" to see how much space is occupied, and check "du /home/olpc/.sugar/default/data/"
• Eject (in journal) and physically remove the USB stick
• Bring up a terminal, and upgrade to update.1.
• Reboot into update.1
• Check "df" to see how much space is occupied, and check "du /home/olpc/.sugar/default/data/"

to make sure that all the files are gone from there.

[mstone]

The rainbow side of this bug has been complete since rainbow-0.7.7.

[nobled]

chihyu, I tracked down the original bug description; Walter moved it here:

http://wiki.laptop.org/go/Ask_OLPC_a_Question_about_the_XO#Orphaned_files_are_created_-_and_cannot_be_easily_deleted_or_seen_from_the_Journal.2C_when_they_fail_to_open.