OS28: Crashes in S/R on a busy network - related to WOL

[martin.langhoff]

Dilinger dixit:

I can run a machine for a week w/ aggressive s/r on a quiet network if I start pinging the machine from another machine, or pinging broadcast, it crashes within 30 mins

Stock os28. no changes other than to enable wol w/ broadcast (I don't recall if I backed out that change; I can test again to verify, though)

[pgf]

epitaph dump. serial disabled. wake-on-wlan loop.

[pgf]

wol_hang-1.gz was obtained with a repetitive wake-on-wlan loop. another machine was set up to ping the laptop every 4 seconds. the laptop was running:

#!/bin/sh

stop powerd

ethtool -s eth0 wol umb

zcat /runin/sdkit-arm/forth.gz > /tmp/forth
chmod +x /tmp/forth

sed -i 's/^wdt-long /wdt-short /' /runin/sdkit-arm/watchdog.fth

trap "/runin/sdkit-arm/watchdog stop" 0

/runin/sdkit-arm/watchdog start

while sleep 3
do
	/runin/sdkit-arm/watchdog ping
	rtcwake -s11 -mmem
done

[pgf]

forgot to say:

 kernel commandline was: initcall_debug debug no_console_suspend log_buf_len=4M

i'll reduce the buffer length.

[pgf]

same as wol_hang-1, smaller 1MB log buffer

[pgf]

yet another

[pgf]

as above, pm_async disabled

[pgf]

one more note about all these hangs: the machine has siv120d blacklisted.

[pgf]

wol_hang-8.noesc is from a kernel now patched with 1220526, for more output. in addition, there's now a flush_cache_all() at the end of printk, to eliminate the junk we're getting at the log rollover point:

diff --git a/kernel/printk.c b/kernel/printk.c
index a032d5e..0eab4e1 100644
--- a/kernel/printk.c 
+++ b/kernel/printk.c 
@@ -16,6 +16,8 @@ 
  *     01Mar01 Andrew Morton
  */

+#include <asm/cacheflush.h> 
+ 
 #include <linux/kernel.h>
 #include <linux/mm.h>
 #include <linux/tty.h>
@@ -832,6 +834,8 @@ static inline void printk_delay(void) 
        }
 }

+extern bool olpc_xo_1_75_is_suspending; 
+ 
 asmlinkage int vprintk(const char *fmt, va_list args)
 {
        int printed_len = 0;
@@ -969,6 +973,9 @@ asmlinkage int vprintk(const char *fmt, va_list args) 
 out_restore_irqs:
        raw_local_irq_restore(flags);

+       if (unlikely(olpc_xo_1_75_is_suspending)) 
+           flush_cache_all(); 
+ 
        preempt_enable();
        return printed_len;
 }

[pgf]

cache flushed during printk, and more output logging

[pgf]

same config as wol_hang-8

[pgf]

i'm not going to attach -10 and -11, but what's interesing is all three of -9, -10, and -11 end right at

<3>[  788.931408] process_one_work: worker thread bf04e940 start

and bf04e940 is the address of if_sdio_host_to_card_worker().

[pgf]

%p --> %pF for workqueue handler prints, and dilinger's techteam object patch

[pgf]

this hang is very clearly related to the wireless and/or sdio drivers. the logs we've collected point to the hang occurring somewere in sdhci when we try and send a command to the card after having suspended sdhci. as a test, if we comment out the disable of the clock in sdhci_pxa_suspend(), we get through the previous hang point, only to fall over with

mmc1: Timeout waiting for hardware interrupt.

[martin.langhoff]

Test in OS30

[greenfeld]

The major WOL hang issues seem to have been resolved in 11.3.1 os884/os31 on an XO-1.75.

However there still is at least one less-common case where the libertas driver hangs (#11711).