Ticket #10510 (closed defect: fixed)

Opened 4 years ago

Last modified 3 years ago

Wikipedia should find a way to work on both F11 and F14 builds

Reported by: martin.langhoff Owned by: martin.langhoff
Priority: normal Milestone: 11.2.0-M2
Component: wikibrowse-activity Version: not specified
Keywords: Cc: godiard, bernie, RafaelOrtiz, cjb, rmo
Action Needed: no action Verified: no
Deployments affected: Blocked By: #10655
Blocking:

Description

wp.so links to python libraries. The Python 2.6 to 2.7 version bump makes it problematic.

Ideas -

  • Include various wp.so in different directories, and fiddle the library path accordingly early in activity startup.
  • Improve seek-bzip2 to read from a file (rather than stdin). Code at hg clone http://bitbucket.org/james_taylor/seek-bzip2 -- it can do the "random access to bzip2" for us.
  • Switch to pure python implementation. Downside: python-only bzlib decompression is likely to be very slow, and it's unlikely we'll find a python-to-C library that allows us to do the odd thing we do (uncompressing random blocks).
  • Push wp.so into Fedora somehow. Long road.

Attachments

ctypes_wip.patch (4.9 kB) - added by martin.langhoff 4 years ago.
org.laptop.WikipediaActivity-21.log (1.6 kB) - added by RafaelOrtiz 4 years ago.
wikipedia log xo.15
0001-Add-support-for-tabbed-Browse-108.patch (1.9 kB) - added by martin.langhoff 4 years ago.
0001-Enable-work-with-old-and-new-toolbars.patch (2.8 kB) - added by godiard 4 years ago.
search.svg (0.9 kB) - added by godiard 4 years ago.
0001-fix-variable-name-in-loaf_homepage.patch (0.8 kB) - added by godiard 4 years ago.
blockindex.py (3.7 kB) - added by rmo 3 years ago.
seek-bzip2-based seeking proof-of-concept

Change History

  Changed 4 years ago by martin.langhoff

  • owner set to cjb
  • type changed from defect to enhancement
  • next_action changed from never set to design
  • component changed from not assigned to wikiserver
  • milestone changed from Not Triaged to Future Release

  Changed 4 years ago by martin.langhoff

  • component changed from wikiserver to wikibrowse-activity

  Changed 4 years ago by martin.langhoff

  • cc godiard added
  • milestone changed from Future Release to 11.2.0-M2

Gonzalo -- happy to work with you on this one.

  Changed 4 years ago by dsd

Not sure of the details so this might not be realistic... but another option could be ctypes? i.e. make wp.so independent of libpython

  Changed 4 years ago by mikus

Perhaps you can have _wp.so make use of an indirect pointer, which has been set up by the activity's initialization routine.

That's how I handle references to python in Mikus-written batch scripts (which exist to customize certain Sugar functions): For me, a script run at build install time determines which version of python (2.5 or 2.6 or 2.7) is installed, and places a pointer to the available python's directory tree (/usr/lib/python...) in a "well known location". Then the individual "customization scripts" pick up the contents of that "well known location" and put that value in a variable (from where it can be used to construct paths for accessing python's site-packages subdirectories).

  Changed 4 years ago by martin.langhoff

We probably need a 'C-type' programmer to tackle this :-)

Yeah, wp.so could be rewritten in Python using ctypes. A reorg of the code would be needed so that wp.so doesn't link into python.

  Changed 4 years ago by bernie

  • cc bernie added

  Changed 4 years ago by RafaelOrtiz

  • cc RafaelOrtiz added

  Changed 4 years ago by dsd

Thats an option too, but my suggestion was intended to be a bit different (less invasive, leaves more in C): change wp.so so that the interface is a regular C API, rather than a Python object/module, and then call into it via ctypes.

  Changed 4 years ago by martin.langhoff

From a quick look, it seems that Daniel's plan is doable. We do get a wp.so built that doesn't link into python -- the _wp.so we ship has additional glue, auto-generated from http://dev.laptop.org/git/projects/wikiserver/tree/woip/py

So that glue needs to be reworked as ctypes declarations. IIRC, it's easier and shorter in ctypes.

  Changed 4 years ago by martin.langhoff

  • type changed from enhancement to defect

This is trickier than I thought. We have 3 SOs that link against libpython:

  • _wp.so, from woip/c/wp.c . I have an initial version, compiles, loads, needs to be tested.
  • mwlib/_expander.so: conversion is tricky (wide unicode chars, returns array of strings), is deprecated in mwlib upstream
  • mwlib/_uscan.so: conversion is tricky, in use upstream

The mwlib side is nasty

  • High costs to sync: Our code is based on an early development version of mwlib, and it looks like we hacked it and patched it plenty. Upgrading to latest is a major enterprise in integration.
  • Performance: mwlib upstream seems to have switched expander from performance-optimized C to maintainership-optimized Python; and the expander routines are a bottleneck on XOs.

Changed 4 years ago by martin.langhoff

  Changed 4 years ago by martin.langhoff

WIP patch attached. To generage libwp.so do

ld -shared -soname libwp.so.1 -o libwp.so.1.0 -lc libwp.o safe.o lsearcher.o bzipreader.o blocks.o -lbz2

Though probably g++ (as used for expander.so) is better.

  Changed 4 years ago by martin.langhoff

Update

  • Explored mwlib sync, impractical at the moment -- notes at #10526
  • Worked around the situation using versioned binary libs - updated in git

Now need to respin the EN and ES builds with the changed files.

  Changed 4 years ago by martin.langhoff

  • cc cjb added
  • next_action changed from design to add to build

These wikipedias now work on F11 and F14:

dsd - add to build? cjb - can I get access to the activity in ASLO?

  Changed 4 years ago by cjb

Martin, I don't think there's an ASLO entry for it -- it predates ASLO. Feel free to (confirm and) create a new one.

  Changed 4 years ago by RafaelOrtiz

Martin there is not a bundle for Wikibrowse in ASLO, please upload freely.

thanks.

  Changed 4 years ago by martin.langhoff

Ok then, will coordinate ASLO creation and mgmt w Gonzalo.

In the meantime, it looks like it's hardcoded in f14-xo1.5.ini but it's pointing to a rather old version.

  Changed 4 years ago by RafaelOrtiz

Further information.

Wikipedia-24.xo is not working on xo-1.5 failling in searchtoolbar-py line 32 object has no attribute 'browser'

  Changed 4 years ago by RafaelOrtiz

This error goes adding import browser on activity.py but then xulrunner-1.9.1/python/ xpcom/components.py fails at line 153

log attached.

Changed 4 years ago by RafaelOrtiz

wikipedia log xo.15

  Changed 4 years ago by martin.langhoff

  • blockedby 10655 added

Argh. Problems:

  • Upgrades do *not* remove files, and this upgrade needs to (re)move .so files -- #10655
  • Browse since 109 has tabs and the API changed -- attaching patch
  • New toolbars are incompat with old toolbars -- this needs work

Changed 4 years ago by martin.langhoff

  Changed 4 years ago by martin.langhoff

Attached the patch. To work on this...

  Changed 4 years ago by dsd

  • next_action changed from add to build to code

moving out of "add to build" based on IRC comment (not ready yet, based on new problems)

Changed 4 years ago by godiard

  Changed 4 years ago by godiard

Added patch enable work with the old toolbars in sugar 0.84 and new toolbars in 0.90/F14.

  Changed 4 years ago by godiard

The search icon does not honor stroke and fill colors. We can add a icon in our activity or push it to sugar-artwork (better, but will display badly in old sugar releases).

Attached is a fixed icon.

Changed 4 years ago by godiard

  Changed 4 years ago by martin.langhoff

  • owner changed from cjb to martin.langhoff
  • status changed from new to assigned
  • next_action changed from code to add to build

After a test-run with search.svg and discussion in IRC, the SVG icon needs to be updated in sugar-artwork. The work here is done.

Please add to build

  Changed 4 years ago by RafaelOrtiz

Tested and running as expected on xo-1.5 sugar 0.88/0.90 should i upload this to aslo ?

follow-up: ↓ 28   Changed 4 years ago by RafaelOrtiz

Btw where is this activity's repo ? .

in reply to: ↑ 27   Changed 4 years ago by godiard

Replying to RafaelOrtiz:

Btw where is this activity's repo ? .

http://dev.laptop.org/git/projects/wikiserver/

  Changed 4 years ago by godiard

Now I tested with a Browse 120 with the home button visible and found a stupid error.

Attached is a small patch to fix it

Changed 4 years ago by godiard

  Changed 4 years ago by godiard

The fix is for wikiserver, not in browse

  Changed 4 years ago by godiard

  • next_action changed from add to build to review

  Changed 4 years ago by bernie

Could someone upload the bundle to aslo, please?

  Changed 4 years ago by martin.langhoff

Bernie -- once this is ready for a real release yes. For now, we have a build that starts properly in F14, that's all. Meets our milestone 2 criteria (all activities start and basically work). Has other warts. Not suitable for ASLO.

  Changed 4 years ago by RafaelOrtiz

I've tested this bundles with sugar 0.88/0.90 and they seems ok, besided last patch from gonzalo.

It can be uploaded to aslo setting compatibility flags, afaik, there are not further problems. that avoid uploading to aslo.

  Changed 4 years ago by RafaelOrtiz

This activity has some peru specifics, like DB and locales, anyways i see no harm in uploading to aslo.

uploading could generate interest in other deployments. that can ask for specifics changes to this activity.

I've uploaded wikipedia 26 with gonzalo's patch

http://people.sugarlabs.org/rafael/Wikipedia-26.xo

Also uploaded to aslo now in sandbox zone

http://activities.sugarlabs.org/es-ES/sugar/downloads/file/27202/wikipedia-26.xo

  Changed 4 years ago by dsd

  • next_action changed from review to add to build

new versions on ASLO queued for next build

  Changed 4 years ago by dsd

There's some confusion as to whether this version works on both F11 and F14. Gonzalo told me yesterday that it does, but Simon said he was told that this new version doesn't work on 10.1.3. Gonzalo, can you clarify?

  Changed 4 years ago by martin.langhoff

We worked with Gonzalo last week on cleaning up the build infra (which was horrid), and he spun a new set of EN and ES wikipedias.

Not sure where he uploaded these -- but this is not the stuff Rafael uploaded to ASLO. Make sure you're getting the right files :-)

We did everything so that it'd work on F11 and F14 -- but TBH I think we only tested F14. I don't have a URL for the files so cannot test to confirm...

  Changed 4 years ago by godiard

The activities uploaded in ASLO, are the new ones we cooked.

I found a new problem with F11, not related to binaries, but to old toolbars in Browse. I have a fix in git, will update new versions in this days.

  Changed 4 years ago by godiard

  • next_action changed from add to build to package

  Changed 4 years ago by godiard

  • next_action changed from package to add to build

New versions (26) in aslo with the old toolbar problem resolved.

  Changed 3 years ago by dsd

  • next_action changed from add to build to test in build

test in 11.2.0-14

Changed 3 years ago by rmo

seek-bzip2-based seeking proof-of-concept

  Changed 3 years ago by rmo

From the bug report:

Improve seek-bzip2 to read from a file (rather than stdin). Code at hg clone http://bitbucket.org/james_taylor/seek-bzip2 -- it can do the "random access to bzip2" for us.

It may no longer be relevant, but I have attached a short python snippet -- attachment:"blockindex.py" -- that uses seek-bzip2 in place of libwp to generate indices and interactively seek to articles from an xml.bz2 file, as generated by dumps.wikimedia.org. (This uses seek-bzip2 unmodified, as a pair of executables within the PATH.)

On a slightly related note, does anyone know why we generate an .xml.bz2.processed file instead of using the .xml.bz2 directly?

  Changed 3 years ago by rmo

  • cc rmo added

  Changed 3 years ago by greenfeld

  • status changed from assigned to closed
  • next_action changed from test in build to no action
  • resolution set to fixed

Wikipedia works as an upgrade or installed directly into 10.1.3 (F11) as well as in 11.2.0-18 (F14). Tested both English and Spanish versions.

The seek-bzip2 proposal probably should be moved to its own enhancement request.

  Changed 3 years ago by bernie

Where is the bundle? I can't find it on ASLO.

  Changed 3 years ago by bernie

We also have two Wikipedia entries on ASLO, which is confusing:

* http://activities.sugarlabs.org/en-US/sugar/addon/4401 * http://activities.sugarlabs.org/en-US/sugar/addon/4411

Perhaps we should modify the descriptions to make it clear that one of them is in English and the other one is in Spanish.

  Changed 3 years ago by godiard

You can read the description in ASLO: Wikipedia "WikiBrowse is a wiki server and compressed set of wiki pages that together act as a self-contained browsable offline wikireader (in spanish)" and WikipediaEN "This is a offline version of the english Wikipedia."

We also changed the icons, adding a "ES" and "EN".

  Changed 3 years ago by rmo

@greenfeld: I set up ticket:10875 to focus a seek-bzip2 discussion (and after some frustration with @bernie the other night trying to build _wp.so for a 64-bit system I pushed a seek-bzip2 branch to an almost-working state).

Note: See TracTickets for help on using tickets.