need multicast group wakeups using the 8686 for wake-on-wlan vs collaboration

[cjb]

We need to implement multicast group wakeups using the wireless device, in order that collaboration doesn't break as soon as the system suspends.

---

Previous description: Wake-on-WLAN is needed so that collaboration doesn't break as soon as the system suspends. Need support for the hostsleep libertas command in order to wake on WLAN activity. Ronak at Marvell is managing this.

[cjb]

pgf's been working on this, and has discovered which registers need to be set. At the moment, Linux is unsetting one of them during SD init.

[pgf]

unsurprisingly, the bit goes away when sdhci_reset() is called, after the completion of the SDHCI_SOFTWARE_RESET command.

for the record, this routine will access the right register:

void sdhci_dump_wakeup_reg(char *id, struct sdhci_host *host)
{
        printk(KERN_DEBUG DRIVER_NAME "%s: Wake-up:  0x%02x\n", id,
                sdhci_readb(host, SDHCI_WAKE_UP_CONTROL));
}
EXPORT_SYMBOL_GPL(sdhci_dump_wakeup_reg);

[pgf]

(copying from email to trac, so this doesn't get lost)

there are two bits that need to be set, as described in the programming manual:

bit 8 of 12F0 0x84-0x97 (Power Management Control and Status)

Enables PME wakeup (for the controller)

bit 0 of SDIO-MMIO 0x2b (Wakeup Control) for the wlan slot

"Wakeup Event Enable on Card Interrupt" (for the slot)

to set these bits with OFW:

    ok 100 6084 config-l!
    ok select /sd
    ok 0 2 set-address
    ok 1 2b cb!
    ok resume

if you set these bits before booting, they're lost by the time linux is running.

[pgf]

i've hacked (literally) support for the wakeup bits into the sdhc driver, and the system now very happily wakes-on-wlan.

too happily -- we need a way to disable this, otherwise we can't even stay asleep when we close the lid or push the button.

[cjb]

we need a way to disable this

Hopefully, that's what Bing's patches and calling ethtool does.

[cjb]

Pushed a patch for this (disabling wake-on-wlan), based on Nico's suggestion:

http://dev.laptop.org/git/olpc-2.6/commit/?h=olpc-2.6.31&id=08ffa130743995e04d44dd332d833ceee14fe6ce

It turns off power to the card, too.

[dsd]

Wake-on-WLAN works in XO-1.5 build os57, can this bug be closed?

[Quozl]

If it works on os57, yes, I think this bug can be closed. I'd like to test, but I've no XO-1.5 with appropriate ECO.

[dsd]

I've added Marvell's pending patches, let's retest this in the first 10.2 development build

[Quozl]

Is specific hardware required for testing?

(Research: there are two Marvell originated patches 1dd7d4076a3cdcdef94aff95e2a9911c1471a376 and 14f2fe76fe0b16685b409ab1d2471b9f4c1dc54d in git olpc-2.6, and there is a commit 2029a80e7c00d0ae6d4cb38994b216cf98d8cb5a after them, and os100 and os101 both have kernels with this commit. So the build number to test in is os100 or later.)

[pgf]

fix the current code that enables sd slot wakeups for wlan -- support B2 boards

[pgf]

i've had the attached wlan_wake.patch floating around for a couple of weeks. can someone confirm that supporting wake-on-wlan is correct on a B2? this can only work on a B2 that has the "always-powered" ECO, but it also ceases enabling wakeups for the external SD slot (which might be an interesting thing to do, but isn't currently our intent).

[dsaxena]

Trying to wrap my head around what's happening with this one. We have patches from Marvell in our kernel for WOL support, do we still need pgf's patch (it is currently in our kernel and enabled)?

[pgf]

the marvell patches (as i understand) allow us to configure the card so that it will interrupt us when we should be woken up. the patch i put in allows us to wake up when we get that interrupt. so both are necessary.

however, there's a possibility that we could move my patch into the acpi DSDT code, which i guess would be preferable. i can look at this.

[Quozl]

Tested on os200 on XO-1.5 C1 using a one second repeating ping from a desktop system. This effectively keeps the laptop awake, which is no doubt the design intent.

Stopping the ping allows the laptop to idle suspend.

Resuming the ping using just one packet (ping -c 1 -w 10) wakes the laptop; the first packet latency is 266 to 267 ms (over five samples).

Given that this works, I'm lowering the priority. What else needs to be done on this ticket?

[Quozl]

Summary: wake on ICMP echo request works, wake on ARP does not, wake on multicast UDP (as used by Chat) does not.

Details: tested with Chat activity. Set up a Chat shared across two XO-1.5 via an access point. Confirmed text transmission both ways. Allowed one XO-1.5 to idle suspend.

Tried to wake it by sending text from the other XO-1.5. The receiving XO-1.5 did not wake.

Manually waking it caused the sent text to appear; it had been buffered.

Tried to wake it with a ping from a host that had no ARP cache entry. There was no response. The pinging host reported Destination Host Unreachable.

Tried to wake it with a ping from a host that had an ARP cache entry. This woke the laptop, and caused the sent text to appear. This implies the packets are stored by the wireless device but that the kernel is not woken. (I can't see how we can wake on these packets and yet not wake on chatter amongst other laptops that we should not be concerned about).

Verified each test above by running tcpdump in Terminal and examining the output.

Chat on the non-suspended laptop correctly identified the other user as "left the chat" when the other laptop had been suspended. On resume, "joined the chat" appeared.

[pgf]

currently i think ohmd sets "ethtool -u" only, i.e., wake-on-unicast. if "ethtool -a" does the right thing, i.e. wakes only on arps that are searching for us (as opposed to just any arp), then i think we should add it, since otherwise the presence wake-on-unicast becomes moot after the senders arp cache times out.

[pgf]

unfortunately, the code in net/wireless/libertas/ethtool.c explicitly excludes support for the WAKE_ARP bit:

        if (wol->wolopts & ~(WAKE_UCAST|WAKE_MCAST|WAKE_BCAST|WAKE_PHY))
                return -EOPNOTSUPP;

this means "ethtool -s eth0 wol a" won't work. we should ask marvell whether this is of necessity or not.

[Quozl]

Tested on os116 with Chat via an access point.

Established bidirectional chatter without external Jabber server. Allowed one laptop to idle suspend while continuing to send messages with the other. The laptop that idle suspended ceased displaying the messages from the point it had suspended. When the laptop was woken using keyboard or touchpad, the messages that were missed were displayed.

There appears to be no change to Chat behaviour since seven weeks ago, but I do see that wake on ping or other unicast traffic now works.

[Quozl]

Tested on os116 with Chat via ad-hoc Create new wireless network.

Same symptoms as above happened.

It was possible also to wake the receiving laptop from the sending laptop by sending a ping. (Hmm, Chat could send a ping. Hack.)

[pgf]

Replying to Quozl:

Tested on os116 with Chat via an access point. Established bidirectional chatter without external Jabber server. Allowed one laptop to idle suspend while continuing to send messages with the other. The laptop that idle suspended ceased displaying the messages from the point it had suspended....

how is a laptop informed that new chat messages are available? (i.e., who owns the protocol transaction?)

[Quozl]

Replying to pgf:

how is a laptop informed that new chat messages are available? (i.e., who owns the protocol transaction?)

It seems either laptop may own this duty, it is not clear from what I see.

Chat activity on the wire consists of Clique packets. I'm not sure what Clique means, but it is text in every packet exchanged. They are UDP multicast packets. Presumably because in the absence of a Telepathy Gabble we are instead using Telepathy Salut.

When the host is suspended, the Clique packets from the peer do not wake it, and are lost. They are not queued. They don't appear on the network interface on resume.

When the host wakes, within three seconds it sends a Clique packet. Or it receives a Clique packet from the other participants. It isn't clear which happens first.

The packets probably contain some sequence numbering or something, I'm not sure.

In response to a Clique packet from the newly woken host, the peer sends the line of chat again.

It means both the peer and the host have to be not suspended for enough time for the chat to catch up. If the host is woken to send text, and suspends before the peer is woken, then the message is not seen by the peer.

The line of chat itself is XML. 306 bytes for the word "ok". And as a bonus you get it in HTML as well.

<?xml version="1.0" encoding="UTF-8"?>
<message from="b697ac9f@xo-a7-2a-5a" to="5c329a272168bdba1b45969005f787a30f7e81c9" type="groupchat">
<body>ok</body>
<html xmlns="http://jabber.org/protocol/xhtml-im">
<body xmlns="http://www.w3.org/1999/xhtml">ok</body>
</html>
</message>

Method of discovery: ran tcpdump on both laptops in a Terminal and analysed the output after the event of interest. The tcpdump command was:

tcpdump -i eth0 -n -s 0 -p -X

[Quozl]

For a three member Chat, where two are allowed to suspend in a way that would normally prevent them communicating, the third member holds and retransmits chat lines for them.

Therefore it only takes two link-local Chat instances to be awake for each chat line to survive into the transcript of all participants.

[cjb]

This wouldn't happen if we woke on multicast selectively as well as unicast, but we don't yet know exactly how to program the packet filter to wake on just the multicast packets that are for us, so if we simply went from wake-on-unicast to wake-on-unicast-and-multicast, we wouldn't spend very long asleep -- all multicast would wake us up.

[Quozl]

It does not seem to be practical to packet filter in the Chat instance ... there might be other Chat groups on the local network, so we'd have to filter by stateful inspection of the group identifier in the XML. Then there's all those other activities.

Even if we wake on multicast, we don't have a way to know whether we should immediately sleep or not. If we wake for 15 seconds, then on any network with more than one XO we will stay awake, since an XO advertises presence frequently.

It's a zero sum game; either we have XO to XO collaboration that works perfectly, or we have suspend.

Can we decide that what we have now is adequate?

[cjb]

Replying to Quozl:

It does not seem to be practical to packet filter in the Chat instance ... there might be other Chat groups on the local network, so we'd have to filter by stateful inspection of the group identifier in the XML. Then there's all those other activities.

No, my understanding is that each activity instance uses (or should use) a different multicast group. We would packet filter by telling the 8686 which multicast groups we are members of, which would be equivalent to giving it a list of the activities we are currently members of. No data inspection needed.

Can we decide that what we have now is adequate?

I'd put it in terms of "we need to implement multicast group wakeups using the 8686, but it's not necessary to do before production starts". What we have now is acceptable for now, even though it isn't the designed behavior.

[martin.langhoff]

Waking this up in light of feedback from the field.

I read the "zero sum game" comment to mean that collaboration is broken when aggressive suspend is on

• ad-hoc + salut
• infra + salut
• infra + gabble

Is this correct?

I assume that something in the Sugar / Tubes / Telepathy stack *knows* that we are in a collaboration session. Can some component of that stack block suspend when we are in that mode?

That means that we're in aggressive power-saving mode when no collab sessions are open.

[Quozl]

I disagree with raising priority to blocker since the current state is at least usable if idle suspend is turned off. I'd like to see it fixed, but raising priority isn't going to get it fixed any sooner.

[gnu]

We have had the same problem on XO-1 for three years; see #4616. Everybody has always wimped out with "well, we can just turn off suspend, or not collaborate", instead of making the rather simple fixes required.

[martin.langhoff]

Wake on multicast if salut is in use...

[martin.langhoff]

As a workaround, we could use something like powerd-multicast-if-salut.patch . It is cheap to check for Salut (Telepathy Salut listens on port 5298).

This means we burn battery when in a chatty Salut environment (but only in that case). In other cases we're ok.

Paul -- what do you think?

[pgf]

is there a way to make this check cheaper, for example by having salut somehow leave an indication that it's in use? or, can we detect in-use some other way? does it create any sort of temp file, or unix-domain socket? if not, then i guess this is okay.

also, is this the correct solution on both xo-1 and 1.5?

[erikos]

Replying to pgf:

is there a way to make this check cheaper, for example by having salut somehow leave an indication that it's in use? or, can we detect in-use some other way? does it create any sort of temp file, or unix-domain socket? if not, then i guess this is okay.

There is no temp file or unix-domain socket afaik. I could write/remove a file from the sugar-presence-service. I guess putting it under /var/run would be fine. Name could be 'telepathy-running-salut'.

also, is this the correct solution on both xo-1 and 1.5?

I did test XO-1 and XO-1.5. Both worked fine.

[martin.langhoff]

Discussed with pgf yesterday...

• Yes, this is super-efficient as is. I strace'd netstat and it reads a file in proc. Actually, straced & timed it again right now and it's all of 65ms . On XO we use LANG=C and I assume powerd runs under that env -- otherwise we pay for attempting to read locale stuff.
• No, have not tested on XO-1. Aggressive suspend seems to be broken on XO-1 -- at least #10232 (do we have an "aggressive suspend issues on XO-1" metabug?). Would be good to check quickly that we don't break things further.

For those tests on XO-1, it'd be important to know how to enable logging in powerd, and where the logging goes. As of os852, I've followed the (very nice) documentation about using powerd-config but could not get any logging. I stopped the service, restarted it, ran it straight from the console, even ran it under bash -x. It'd at most give me a trace of its startup, but seemed to close its stderr on me so the interesting traces / debug logs I've never seen...

[pgf]

powerd tracing output goes to /var/log/powerd.trace.

i couldn't even get "time netstat ... | grep..." to show 65ms -- more like under 20ms.

the compulsive part of me still wants to use:

grep -q ": 00000000:14B2" /proc/net/tcp

instead of:

netstat -l --inet -n | grep -q ':5298 '

since it takes about half the time. (i promise to include the netstat|grep version in a comment, along with explanation. :-)

BUT. here's what i don't understand: on my freshly booted XO-1.5 (running, admittedly, os850), port 5298 is already bound, though i have no activities running at all. i've added various rpms to this laptop, so it could be something special, but the socket seems to be bound by avahi-daemon, which is part of the base system. what am i missing?

[erikos]

Can't we just write a file like I proposed in #comment:36 ? That way we are sure that the PS is telling us the right thing.

[erikos]

Replying to pgf:

powerd tracing output goes to /var/log/powerd.trace. i couldn't even get "time netstat ... | grep..." to show 65ms -- more like under 20ms. the compulsive part of me still wants to use: {{{ grep -q ": 00000000:14B2" /proc/net/tcp }}} instead of: {{{ netstat -l --inet -n | grep -q ':5298 ' }}} since it takes about half the time. (i promise to include the netstat|grep version in a comment, along with explanation. :-) BUT. here's what i don't understand: on my freshly booted XO-1.5 (running, admittedly, os850), port 5298 is already bound, though i have no activities running at all. i've added various rpms to this laptop, so it could be something special, but the socket seems to be bound by avahi-daemon, which is part of the base system. what am i missing?

Hmm, I get here the right thing (telepathy-salut) when I do 'lsof -i :5298'.

[pgf]

re: a file -- that would of course be fine by me, but i don't pretend to know exactly how mdns/sharing/collaboration works, and whether the file solution is practical.

re: the port -- after a reboot, port 5298 is no longer bound, by avahi-daemon or anyone else. i have no idea what's changed, except that the machine was up longer before, and had been carried to and used in various locations. (see my comment above about how much i know about this part of the laptop stack.) but it really did go away as soon as i killed avahi-daemon, last time.

[pgf]

freshly installed os852 on an xo-1, freshly booted, no activities have ever been shared on this machine:

[root@xo-0d-27-d0 dev]# lsof -i :5298
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
telepathy 2029 olpc    8u  IPv4   7724      0t0  TCP *:presence (LISTEN)
telepathy 2029 olpc    9u  IPv6   7725      0t0  TCP *:presence (LISTEN)

i think the theory behind this patch is suspect.

[erikos]

This looks like the way we want to go to fix "collab vs suspend".

[pgf]

Replying to erikos:

This looks like the way we want to go to fix "collab vs suspend".

what does? i'm confused.

[martin.langhoff]

pgf -- grepping straight /proc/net/tcp is just fine with me. Or lsof for that matter.

The theory behind this patch is solid (as long as the driver/fw holds its end of the deal), and what you're seeing is as expected. The machine is using Salut. Perhaps it's not in a collaborative session right now, but it is using Salut and as such it should wake up on mcast frames.

I strongly prefer just reading /proc/net/tcp or a moral equivalent such as netstat or lsof.

(Yes, it'd be ideal to wake up on only some mcast frames, such as presence announcements and collab sessions we are on. And ponies. In the meantime, the patch to powerd I am proposing is a fairly simple and effective workaround.)

About Erikos comment -- we had been looking at other workarounds higher up the stack. Given that this powerd patch works better and is less intrusive than the others, we want to go with it. And drop the others.

[pgf]

Replying to martin.langhoff:

... and what you're seeing is as expected. The machine is using Salut. Perhaps it's not in a collaborative session right now, but it is using Salut and as such it should wake up on mcast frames.

then how is this solution different from simply disabling power management? i must be missing something.

[martin.langhoff]

"how is this solution different from simply disabling power management?"

Simple!

If we just disable power mgmt (when Salut is in use), we are awake all the time. Given that a laptop "alone under a tree" (ie: no AP in sight...) auto-tunes to adhoc 1 (mimicking auto-tuning to mesh 1 in earlier releases) and to Salut (as it won't find an XMPP server), then the most common situation for a laptop has it fully awake and burning power, for no good reason.

With the patch I propose, power mgmt stays on, and we save battery. In a perfectly quiet RF environment, we save lots of battery. Same on a mostly quiet network.

If we see mcast or ucast frames we wake up briefly -- this means that presence announcements are received ok, and that if we are in a collab session we stay in sync with other nodes. So the Salut-is-broken-with-power-mgmt-on situation is avoided completely.

If the laptop is on a very busy infra or ad-hoc network, where other XOs are sending mcast frames, we'll stay awake longer. If it's busy enough with mcast frames, we'll stay awake all the time.

This gradual effect is a significant win: we save power in many/most cases, and Salut doesn't break.

The only not-nice scenario is that a busy Salut collab session between A and B will keep C, D and E awake. So multicast group wakeups _are_ still the right fix. But my patch (with improvements as you've suggested) is the best workaround so far, by a large margin.

[pgf]

doh. of course. pardon my brain-fade. i was thinking that the open port would prevent suspend. what actually happens is a change to the ethtool line. i knew that. i guess i should have put it into powerd-30. i guess we'll need a -31.

[erikos]

I adapted the patch from Martin and changed it to use /proc/net/tcp

diff --git a/powerd b/powerd
index d20bf5e..e3f61de 100755
--- a/powerd
+++ b/powerd
@@ -977,7 +977,13 @@ set_wake_on_wlan()
     # a resume/suspend cycle
 
     case $1 in
-    yes) ethtool -s $WLANIFACE wol u ;;
+    yes)
+           if grep -q ": 00000000:14B2" /proc/net/tcp; then
+               ethtool -s $WLANIFACE wol um;
+           else
+               ethtool -s $WLANIFACE wol u;
+           fi
+           ;;
     no)  ethtool -s $WLANIFACE wol d ;;
     esac
 }

This works fine, though there are some occasions where I am not sure why a machine wakes up. The powerd logs indicate only due to a wlan package. No activity was shared yet, only two machines (XO-1.5) around. Maybe a presence update. Will try to get some more info here.

Another issue is that the XO-1 do not seem to idel-suspend anymore. I made sure that the file '/etc/powerd/flags/inhibit-suspend' is not there. I tried as well with powerd-26, to see if there was a regression, but on 353 the XO-1 just don't want to idle suspend. I am quite sure it worked for me before. If someone has an idea what change that could have caused it...

[erikos]

Replying to erikos:

Another issue is that the XO-1 do not seem to idel-suspend anymore. I made sure that the file '/etc/powerd/flags/inhibit-suspend' is not there. I tried as well with powerd-26, to see if there was a regression, but on 353 the XO-1 just don't want to idle suspend. I am quite sure it worked for me before. If someone has an idea what change that could have caused it...

Ahh, I had an optical mouse plugged over usb, that was the issue! Same behavior for XO-1 and XO-1.5, powerd says "usb busy: HID".

[pgf]

okay. please let me know if/when this patch is fully approved, and i'll build a new powerd package.

[erikos]

Replying to pgf:

okay. please let me know if/when this patch is fully approved, and i'll build a new powerd package.

About the wakeups on certain packages: I checked with tcpdump, the packages are MDNS ones. I had one with the note (Cache flush) or ((QM)?). I presume those are avahi syncup packages. I have not seen them that often and the machines kept on sleeping again directly after waking up for that packet. So I think our approach is still ok.

[pgf]

patch applied to powerd, released powerd-31

[erikos]

Is in 354.

[greenfeld]

We wake up on multicast events in 10.1.3 os354 for both XO-1 {with aggressive power management enabled; otherwise we are always on} and XO-1.5 systems from the idle sleep state, even if the multicast event is due to a non-XO machine doing something. Watched the network with Wireshark for several minutes to verify that multicast traffic was what the XOs were waking up on.

[martin.langhoff]

Re-opening -- we track the workadound in #10363 . Let's leave this bug open, set to future release, to remind us of applying the 'root fix': multicast group wakeups.

[martin.langhoff]

Have things moved forward upstream (on our F14 kernels) on multicast group wakeups?

If they have, we can do something smarter than wake up on any mcast frame.

[sascha_silbe]

Replying to martin.langhoff:

Have things moved forward upstream (on our F14 kernels) on multicast group wakeups? If they have, we can do something smarter than wake up on any mcast frame.

It's not clear to me what could be changed on the kernel side to improve behaviour. It's up to other parts to behave better:

1. libertas firmware: Don't wake the host if a multicast packet doesn't match any of the configured multicast addresses (might already behave this way, but I believe not).
2. Configure APs that behave as multicast routers to do group membership polling much less often. We must reply to these queries or we'll get thrown off the multicast group.
3. Add support for operating as a Sleep Proxy to APs and/or XOs (the latter would each take a turn at providing the service, similar to how some mesh protocols work), and add client support for using such a proxy to avahi on XOs. This should take care of regular wake-ups due to mDNS maintenance packets.

[dsd]

I think this bug should be closed (the original task has been done), and we should open a new Future Release enhancement ticket under the wireless component for the appropriate firmware enhancement to make sure we only wake up on interesting multicast events.

[dsd]

We'll use this ticket to track the required wireless firmware improvements.