nand test crashes 760

[pgf]

wad has been running nand tests on a set of laptops. after 12 hours, 5 of his laptops have crashed with jffs2-related backtraces.

screen photos attached.

[pgf]

screenshot 1

[pgf]

another screenshot

[pgf]

from wad: Steps which triggered the crash:

http://www.laptop.org/teamwiki/index.php/Tech:NAND_Testing#Test_Plan

[dilinger]

I notice that there are prior oopses. It's pretty important to have the first oops backtrace. Can you log this via serial console?

[pgf]

test scripts:

first the script to fill nand to within 32MB:

http://dev.laptop.org/git?p=users/wad/NANDtest;a=blob;f=fill_jffs2.sh;hb=HEAD

and then the actual test script:

http://dev.laptop.org/git?p=users/wad/NANDtest;a=blob;f=test.sh;hb=HEAD

[pgf]

yes, we'll set up the test with serial connected.

[wad]

This is the serial console log from a laptop that recreated this bug

[wad]

The crash starts with: BUG: unable to handle kernel paging request at 00100100

After running the same test overnight on three laptops, only one recreated the problem. There was something special about starting from a fresh image install that encouraged more laptops to show this bug sooner. I will reinstall a fresh image and try to recreate again.

[wad]

Another console log from a machine showing this bug

[wad]

I just wasn't patient enough. Another of the machine running test overnight with a serial console crashed. The console log is attached.

[dsaxena]

Please install http://dev.laptop.org/~dsaxena/debuginfo-8467/kernel-2.6.25-20080925.1.olpc.f10b654367d7065.i586.rpm and rerun the test. We have debuginfo packages for this kernel, meaning that I can look at the crash dump and an objdump from the vmlinux file to start digging at this.

[wad]

An update on the status of testing. I installed the new kernel RPM on three machines running with serial console logging, and have started them running. Since that first night when five machines crashed, there hasn't been a time when all five crashed overnight. Reinstalling the OS from scratch on two of these didn't trigger the crash again, so I don't know what triggered that first night.

I also had a machine w. build 656 running the same test, and it never crashed with the kernel oops, even after writing/erasing 100 GB to the disk. It did have other issues (JFFS2 mysteriously lost space, and the test failed for that reason).

[wad]

Logs from a crash running dsaxena's debug kernel

[wad]

Finally got a crash with the debug kernel.

And I have seen the same crash on the stock kernel in build 656, so this problem is definitely already out in the field.

[wad]

Another kernel crash using dsaxena's debug kernel

[wad]

Both the laptop running 656 (w. stock kernel) and the one running 760 (w. dsaxena's debug kernel) crashed again overnight. Their JFFS2 filesystems seems to have gotten into a state which triggers this problem.

[wad]

[emailed from dsaxena]

What would be good is you could boot from an SD card and do a dump of the flash device since you think the FS in a bad state.

Do you mean a save-nand of the built-in Flash ? (That doesn't require me to build a bootable SD card) Or did you have something else in mind ? (dd | tar doesn't seem right for JFFS2 filesystems...)

[wad]

Crash log from laptop running build 656 stock kernel

[dsaxena]

Replying to wad:

[emailed from dsaxena] What would be good is you could boot from an SD card and do a dump of the flash device since you think the FS in a bad state. Do you mean a save-nand of the built-in Flash ? (That doesn't require me to build a bootable SD card) Or did you have something else in mind ? (dd | tar doesn't seem right for JFFS2 filesystems...)

Yes, it looks like save-nand will give me what I want (a JFFS2 image).

[wad]

You can find a save-nand image from laptop JFFS1 (the one that has crashed repeatedly) at: http://dev.laptop.org/~wad/nand/crash/

The write throughput of a laptop running JFFS2 degrades greatly over time. The laptop gets to a state where data which could be previously written can no longer be written. Even after freeing up another 32 MB (only 20MB was being written)so the test can proceed, writes remain very slow. For timings showing this happening to two different laptops, see: http://dev.laptop.org/~wad/nand/alt/JFFS2/

[dwmw2]

Hm, we are calling the thread_should_wake() function from the GC thread without locking. And it goes wandering through the block lists.... no wonder it crashes :)

[dwmw2]

Potential fix.

[wad]

The description of the tests required to trigger this have been moved to a publicly accessible page:

http://wiki.laptop.org/go/NAND_Testing

[dsaxena]

Wad, I've uploaded an RPM to http://dev.laptop.org/~dsaxena/kernels/test/kernel-2.6.25-20081025.1.olpc.fix_gc_race.i586.rpm

[wad]

The preliminary report is that the patch seems to fix the problem. I've had five laptops with full Flash running the new kernel for over 48 hours now, and no kernel panics. I'll close out the ticket if another 48 hours pass without a problem.

[mstone-xmlrpc]

Pushing out to 9.1.0, per edmcnierney's request.

[dsaxena]

I am going through all kernel bugs marked as 9.1 or future release and updating their status, next action, etc in preparation of 9.1 bug scrubbing and future release planning.

The attached patch has not been committed upstream from what I can tell but I'll go ahead and merge into our 2.6.27 kernel. David, can you merge this upstream?