olpc-update's irsync_pristine step always fails with "Contents manifest failure at line 383" on localtime

[skierpage]

As reported in bug 5805 comment #5 and in bug 5954, in olpc-update, irsync_pristine fails always/a lot, on the file "localtime":

Trying irsync_pristine update from /tmp/tmpdKozIo
- Cleaning tree.
- Fetching contents.
- Performing incremental rsync.
... [many minutes go by]
Verifying update
Contents manifest failure at line 383
Last file examined: localtime

I've seen this on three out of 4 olpc-updates, latest is joyride-2301 to 8.2-757. I have /etc/localtime and /etc/avahi/etc/localtime that are dated 2007-11-02.

If the step is always going to fail, why bother doing it? Seems like this might be worth investigating.

[mavrothal]

Same here with a unsecure G1G1 machine running joyride 2298/q2e14
and then it attempts

irsync_dirty

and in the middle of it (after ~5 minuts) crashes gives a white screen then a black screen with running text (that I can not read) and restarts Sugar (it does not reboot) with my old build

[thomaswamm]

Similar here, on olpc-update to jr2301 and before that to jr2263.

olpc-update takes so much time that I became a fan of clean-installs from USB stick. Then I can quickly reflash anytime I want.

[skierpage]

Replying to mavrothal:

and in the middle of it (after ~5 minuts) crashes gives a white screen then a black screen with running text (that I can not read) and restarts Sugar (it does not reboot) with my old build

That sounds like bug #5954. As I understand it, olpc-update uses a lot of memory and the kernel's out-of-memory handler kills the X Window System which restarts. In a terminal enter dmesg | less and you may see references to memory and oom.

As for this bug, m_stone commented on IRC "irsync_pristine is always going to fail until we actually manage to make the root filesystem readonly. (we were close to that for a while, then were pulled farther away, and now are getting close again)" Maybe that option should be removed from "hints" in /usr/bin/olpc-update.

[mavrothal]

Replying to skierpage:

That sounds like bug #5954. As I understand it, olpc-update uses a lot of memory and the kernel's out-of-memory handler kills the X Window System which restarts. In a terminal enter dmesg | less and you may see references to memory and oom.

So what's the remidy? The #5954 ticket has at the end a link to an addition to the "post-olpc-update" script http://dev.laptop.org/ticket/28#comment:19 on ticket #28, but is totaly unlear (to me) "which, where and when" we are takling about.

[cscott]

Replying to thomaswamm:

Similar here, on olpc-update to jr2301 and before that to jr2263. olpc-update takes so much time that I became a fan of clean-installs from USB stick. Then I can quickly reflash anytime I want.

You probably want to use the '-f' option to olpc-update, then. This is not enabled by default because it is hard on our poor rsync server.

[cscott]

Replying to skierpage:

As for this bug, m_stone commented on IRC "irsync_pristine is always going to fail until we actually manage to make the root filesystem readonly. (we were close to that for a while, then were pulled farther away, and now are getting close again)" Maybe that option should be removed from "hints" in /usr/bin/olpc-update.

That's not exactly true; irsync_pristine has a number of hacks hard-coded to make it expected to succeed on 'normal' systems. But it's not perfect (it will be in the 9.1 timeframe, as mstone says), and you can still dirty your pristine copy in some situtations.

This seems to be one of them: how on earth are you writing to /etc/localtime? Sugar's timezone setting feature explicitly does *not* set this. Elsewhere /etc/avahi/etc/localtime was fingered as a co-culprit -- is there some network environment in which avahi can be induced into writing to /etc/localtime? That certainly doesn't happen at 1cc. I'd be interested in more information. Perhaps an Apple machine is on the network, broadcasting "interesting" mDNS services?

[mikus]

Seen the "manifest failure, Last file examined: localtime" situation many many times.

I notmally use 'olpc-update --usb', but nowadays, because of this exact problem, I include the --full parameter when I use olpc-update..

[mavrothal]

Replying to cscott:

Perhaps an Apple machine is on the network, broadcasting "interesting" mDNS services?

Good call!. An apple machine is on the same WiFi network as a client. However, I have no idea what is broadcasting, if anything. How do I find out?

[ridderman]

I see this on all updates, and I don't have any Apple devices on my network. I'm also on a G1G1 laptop. We'll see if the fix that went into joyride works for me.

[akonstam]

Replying to skierpage:

As reported in bug 5805 comment #5 and in bug 5954, in olpc-update, irsync_pristine fails always/a lot, on the file "localtime": {{{ Trying irsync_pristine update from /tmp/tmpdKozIo - Cleaning tree. - Fetching contents. - Performing incremental rsync. ... [many minutes go by] Verifying update Contents manifest failure at line 383 Last file examined: localtime }}} I've seen this on three out of 4 olpc-updates, latest is joyride-2301 to 8.2-757. I have /etc/localtime and /etc/avahi/etc/localtime that are dated 2007-11-02. If the step is always going to fail, why bother doing it? Seems like this might be worth investigating.

I had exzactly the same error in installing v 711 and v 757 today. The installation seems to suceed so I an unclear what exactly the implication of the error.

[thomaswamm]

Terminal capture of $ sudo olpc-update 8.2-757 (from joyride-2301). Looks not too smooth. But took only 20 minutes after midnight PDT.

[cscott]

There were fixes to the MemoryError and dirty /etc/localtime problems landed in joyride-2369; please help test.

[cscott]

olpc-update 2.17-1

Added to stable stream; should be in 758 and following.

[ridderman]

I updated from joyride-2373 to joyride-2386 and did not see this bug. The irsync_pristine worked for me for the first time in ages. Nice work!

[thomaswamm]

Terminal transcript of $ sudo olpc-update 8.2-759 (from 8.2-757). Took under 30 minutes near midnight. Still has localtime bug, but went smoother than previous update to 8.2-757.

[cscott]

irsync_pristine worked for me from joyride-2414 to 2420 or so.

I finally found the likely culprit overwriting /etc/localtime. When we make our DHCP request, we explicitly ask for "Subnet-Mask, BR, Time-Zone, Default-Gateway, Domain-Name, Domain-Name-Server, Hostname, YD, YS, NTP" using DHCP option 55 (section 9.8 at http://tools.ietf.org/html/rfc2132). I'm guessing dh_client is obliging by overwriting our /etc/localtime for us based on the Time-Zone.

Is this a bug or a feature, I wonder?

[thomaswamm]

From 8.2-759 I did $ sudo olpc-update 8.2-760 and had my smoothest-ever update experience, pristine, with no localtime nuisance. It took 17 minutes to "update succeeded".

I will attach a Terminal transcript just for the record, in case anyone has a bad experience and wants to compare notes.

[thomaswamm]

golden Terminal transcript for $ sudo olpc-update 8.2-760 (it went very well).

[mstone-xmlrpc]

Pushing out to 9.1.0, per edmcnierney's request.