Wikipedia should find a way to work on both F11 and F14 builds

[martin.langhoff]

wp.so links to python libraries. The Python 2.6 to 2.7 version bump makes it problematic.

Ideas -

• Include various wp.so in different directories, and fiddle the library path accordingly early in activity startup.
• Improve seek-bzip2 to read from a file (rather than stdin). Code at hg clone http://bitbucket.org/james_taylor/seek-bzip2 -- it can do the "random access to bzip2" for us.
• Switch to pure python implementation. Downside: python-only bzlib decompression is likely to be very slow, and it's unlikely we'll find a python-to-C library that allows us to do the odd thing we do (uncompressing random blocks).
• Push wp.so into Fedora somehow. Long road.

[martin.langhoff]

Gonzalo -- happy to work with you on this one.

[dsd]

Not sure of the details so this might not be realistic... but another option could be ctypes? i.e. make wp.so independent of libpython

[mikus]

Perhaps you can have _wp.so make use of an indirect pointer, which has been set up by the activity's initialization routine.

That's how I handle references to python in Mikus-written batch scripts (which exist to customize certain Sugar functions): For me, a script run at build install time determines which version of python (2.5 or 2.6 or 2.7) is installed, and places a pointer to the available python's directory tree (/usr/lib/python...) in a "well known location". Then the individual "customization scripts" pick up the contents of that "well known location" and put that value in a variable (from where it can be used to construct paths for accessing python's site-packages subdirectories).

[martin.langhoff]

We probably need a 'C-type' programmer to tackle this :-)

Yeah, wp.so could be rewritten in Python using ctypes. A reorg of the code would be needed so that wp.so doesn't link into python.

[dsd]

Thats an option too, but my suggestion was intended to be a bit different (less invasive, leaves more in C): change wp.so so that the interface is a regular C API, rather than a Python object/module, and then call into it via ctypes.

[martin.langhoff]

From a quick look, it seems that Daniel's plan is doable. We do get a wp.so built that doesn't link into python -- the _wp.so we ship has additional glue, auto-generated from http://dev.laptop.org/git/projects/wikiserver/tree/woip/py

So that glue needs to be reworked as ctypes declarations. IIRC, it's easier and shorter in ctypes.

[martin.langhoff]

This is trickier than I thought. We have 3 SOs that link against libpython:

• _wp.so, from woip/c/wp.c . I have an initial version, compiles, loads, needs to be tested.
• mwlib/_expander.so: conversion is tricky (wide unicode chars, returns array of strings), is deprecated in mwlib upstream
• mwlib/_uscan.so: conversion is tricky, in use upstream

The mwlib side is nasty

• High costs to sync: Our code is based on an early development version of mwlib, and it looks like we hacked it and patched it plenty. Upgrading to latest is a major enterprise in integration.
• Performance: mwlib upstream seems to have switched expander from performance-optimized C to maintainership-optimized Python; and the expander routines are a bottleneck on XOs.

[martin.langhoff]

WIP patch attached. To generage libwp.so do

ld -shared -soname libwp.so.1 -o libwp.so.1.0 -lc libwp.o safe.o lsearcher.o bzipreader.o blocks.o -lbz2

Though probably g++ (as used for expander.so) is better.

[martin.langhoff]

Update

• Explored mwlib sync, impractical at the moment -- notes at #10526
• Worked around the situation using versioned binary libs - updated in git

Now need to respin the EN and ES builds with the changed files.

[martin.langhoff]

These wikipedias now work on F11 and F14:

• http://dev.laptop.org/~martin/wikipedia_en/WikipediaEN-24.xo
• http://dev.laptop.org/~martin/wikipedia_es/Wikipedia-24.xo

dsd - add to build? cjb - can I get access to the activity in ASLO?

[cjb]

Martin, I don't think there's an ASLO entry for it -- it predates ASLO. Feel free to (confirm and) create a new one.

[RafaelOrtiz]

Martin there is not a bundle for Wikibrowse in ASLO, please upload freely.

thanks.

[martin.langhoff]

Ok then, will coordinate ASLO creation and mgmt w Gonzalo.

In the meantime, it looks like it's hardcoded in f14-xo1.5.ini but it's pointing to a rather old version.

[RafaelOrtiz]

Further information.

Wikipedia-24.xo is not working on xo-1.5 failling in searchtoolbar-py line 32 object has no attribute 'browser'

[RafaelOrtiz]

This error goes adding import browser on activity.py but then xulrunner-1.9.1/python/ xpcom/components.py fails at line 153

log attached.

[RafaelOrtiz]

wikipedia log xo.15

[martin.langhoff]

Argh. Problems:

• Upgrades do *not* remove files, and this upgrade needs to (re)move .so files -- #10655
• Browse since 109 has tabs and the API changed -- attaching patch
• New toolbars are incompat with old toolbars -- this needs work

[martin.langhoff]

Attached the patch. To work on this...

• install f14/os6
• upgrade to wikipedia-24 from http://dev.laptop.org/~martin/wikipedia_es/
• remove the .so files that are not in binarylibraries: _wp.so mwlib/*.so
• apply 0001-Add-support-for-tabbed-Browse-108.patch

[dsd]

moving out of "add to build" based on IRC comment (not ready yet, based on new problems)

[godiard]

Added patch enable work with the old toolbars in sugar 0.84 and new toolbars in 0.90/F14.

[godiard]

The search icon does not honor stroke and fill colors. We can add a icon in our activity or push it to sugar-artwork (better, but will display badly in old sugar releases).

Attached is a fixed icon.

[martin.langhoff]

After a test-run with search.svg and discussion in IRC, the SVG icon needs to be updated in sugar-artwork. The work here is done.

Please add to build

• http://dev.laptop.org/~martin/wikipedia_en/WikipediaEN-25.xo
• http://dev.laptop.org/~martin/wikipedia_es/Wikipedia-25.xo

[RafaelOrtiz]

Tested and running as expected on xo-1.5 sugar 0.88/0.90 should i upload this to aslo ?

[RafaelOrtiz]

Btw where is this activity's repo ? .

[godiard]

Replying to RafaelOrtiz:

Btw where is this activity's repo ? .

http://dev.laptop.org/git/projects/wikiserver/

[godiard]

Now I tested with a Browse 120 with the home button visible and found a stupid error.

Attached is a small patch to fix it

[godiard]

The fix is for wikiserver, not in browse

[bernie]

Could someone upload the bundle to aslo, please?

[martin.langhoff]

Bernie -- once this is ready for a real release yes. For now, we have a build that starts properly in F14, that's all. Meets our milestone 2 criteria (all activities start and basically work). Has other warts. Not suitable for ASLO.

[RafaelOrtiz]

I've tested this bundles with sugar 0.88/0.90 and they seems ok, besided last patch from gonzalo.

It can be uploaded to aslo setting compatibility flags, afaik, there are not further problems. that avoid uploading to aslo.

[RafaelOrtiz]

This activity has some peru specifics, like DB and locales, anyways i see no harm in uploading to aslo.

uploading could generate interest in other deployments. that can ask for specifics changes to this activity.

I've uploaded wikipedia 26 with gonzalo's patch

http://people.sugarlabs.org/rafael/Wikipedia-26.xo

Also uploaded to aslo now in sandbox zone

http://activities.sugarlabs.org/es-ES/sugar/downloads/file/27202/wikipedia-26.xo

[dsd]

new versions on ASLO queued for next build

[dsd]

There's some confusion as to whether this version works on both F11 and F14. Gonzalo told me yesterday that it does, but Simon said he was told that this new version doesn't work on 10.1.3. Gonzalo, can you clarify?

[martin.langhoff]

We worked with Gonzalo last week on cleaning up the build infra (which was horrid), and he spun a new set of EN and ES wikipedias.

Not sure where he uploaded these -- but this is not the stuff Rafael uploaded to ASLO. Make sure you're getting the right files :-)

We did everything so that it'd work on F11 and F14 -- but TBH I think we only tested F14. I don't have a URL for the files so cannot test to confirm...

[martin.langhoff]

Ah, ok, so versions 25 on ASLO --

http://activities.sugarlabs.org/es-ES/sugar/addon/4401

http://activities.sugarlabs.org/es-ES/sugar/addon/4411

[godiard]

The activities uploaded in ASLO, are the new ones we cooked.

I found a new problem with F11, not related to binaries, but to old toolbars in Browse. I have a fix in git, will update new versions in this days.

[godiard]

New versions (26) in aslo with the old toolbar problem resolved.

[dsd]

test in 11.2.0-14

[rmo]

seek-bzip2-based seeking proof-of-concept

[rmo]

From the bug report:

Improve seek-bzip2 to read from a file (rather than stdin). Code at hg clone http://bitbucket.org/james_taylor/seek-bzip2 -- it can do the "random access to bzip2" for us.

It may no longer be relevant, but I have attached a short python snippet -- attachment:"blockindex.py" -- that uses seek-bzip2 in place of libwp to generate indices and interactively seek to articles from an xml.bz2 file, as generated by dumps.wikimedia.org. (This uses seek-bzip2 unmodified, as a pair of executables within the PATH.)

On a slightly related note, does anyone know why we generate an .xml.bz2.processed file instead of using the .xml.bz2 directly?

[greenfeld]

Wikipedia works as an upgrade or installed directly into 10.1.3 (F11) as well as in 11.2.0-18 (F14). Tested both English and Spanish versions.

The seek-bzip2 proposal probably should be moved to its own enhancement request.

[bernie]

Where is the bundle? I can't find it on ASLO.

[bernie]

We also have two Wikipedia entries on ASLO, which is confusing:

* http://activities.sugarlabs.org/en-US/sugar/addon/4401 * http://activities.sugarlabs.org/en-US/sugar/addon/4411

Perhaps we should modify the descriptions to make it clear that one of them is in English and the other one is in Spanish.

[godiard]

You can read the description in ASLO: Wikipedia "WikiBrowse is a wiki server and compressed set of wiki pages that together act as a self-contained browsable offline wikireader (in spanish)" and WikipediaEN "This is a offline version of the english Wikipedia."

We also changed the icons, adding a "ES" and "EN".

[rmo]

@greenfeld: I set up ticket:10875 to focus a seek-bzip2 discussion (and after some frustration with @bernie the other night trying to build _wp.so for a 64-bit system I pushed a seek-bzip2 branch to an almost-working state).