School server component needed to provide human readable index of journal backup

[cscott]

Trac #3469 was somehow closed without adding the trac bug for the solution we decided on, which was a school server component which read the journal backup format and created a simple index of the child's files, ideally with the same visual layout used in the Journal.

[tomeu]

By which process are the files transferred to the school server?

In which structure are the files stored? And the metadata?

Which searching, sorting and filtering functionality we would need in that remote journal component?

If we need a complex web module for this, perhaps would be easier to implement support in the datastore for remote entries.

In the journal we would need no other modification other than adapt to the difference of speed in the queries.

In the datastore, we would need a new backend that would connect to a remote xapian in the school server, and when files are requested, retrieve and make them accessible locally. See OllyBetts' comment in #2711.

So we need to decide what to do about this in the future milestones.

[cscott]

We discussed this at length in a meeting some weeks ago.

The file format on the school server is *exactly* the one on the XO. We are doing a simple rsync of the files from the XO to the school server.

As a start, we don't need *any* searching or sorting. Just extracting the human-readable names of the files from the datastore metadata and displaying them in a unadorned list would be sufficient to allow children a simple file-sharing ability.

It's true that we might want more a more elaborate implementation in the future, but let's restrict discussion to what is absolutely required by FRS.

[tomeu]

Scott, in what environment should that component run?

A cgi script in python would be sufficient?

How that script would know which child's files are being requested?

How will children point their browsers to that script? Through a link in the Library home page?

How we map them to the directory containing the backup?

Can I ssh into an existing school server?

[tomeu]

Kim, what do we need for MP or FRS (perhaps we should have two separate tickets for each milestone)?

Should be decided soon, given the impact this task could have.

[jg]

Kim, either we need to resource this somehow, or delay to 1.1...

[cscott]

If I can get a simple standalone script which will run on an ordinary Fedora Core 7 system and print out the filenames, human-readable names, and timestamps of the files, I can work from there to integrate it into the schoolserver and webify it. But I'd like help getting the basic datastore pieces separated out.

[kimquirk]

changing ownership to Ben.

[krstic]

I hashed out a spec for this that made tomeu declare he loves me:

http://wiki.laptop.org/go/XS_backup_restore

As per Marco, reassigning this to Tomeu to write the XO part that deals with the datastore and writes the indices (this isn't Ben's usual domain, since it's more a client use of the DS).

[marco]

I reviewed Ivan spec and it looks pretty good. It address all the concerns I had about the plain rsync approach we discussed yesterday. I'd say it's the way to go for 1.0.

[tomeu]

Ok, so just to clarify and check if I understood correctly the spec, my part is to write a func like this:

path_to_backup_idx, path_to_backup_state_idx, path_to_backup_files_idx = \
    write_backup_index_since(timestamp, max_items=None)

That filters the items to backup according to timestamp and max_items and creates the following files:

• backup.idx: file that contains the metadata of the entries to backup,

• backup-state.idx: file with the ids of all the entries currently present in the DS (not filtered nor limited by max_items),

• backup-files.idx: file with the absolute paths to the data files and thumbnails of the entries to backup.

[tomeu]

From some mail:

On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

- XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual file plus metadata in a .xoc file. After Browse has downloaded it and put into the Journal like with all downloads, the Journal could then expand it and put the metadata into xapian.

This would also serve for sharing journal entries, one journal would send through tubes a bundled entry, and the other side would expand it and recreate in its own datastore.

[tomeu]

Replying to tomeu:

From some mail: On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

- XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual file plus metadata in a .xoc file. After Browse has downloaded it and put into the Journal like with all downloads, the Journal could then expand it and put the metadata into xapian. This would also serve for sharing journal entries, one journal would send through tubes a bundled entry, and the other side would expand it and recreate in its own datastore.

This is in #4380.

[tomeu]

Attached is the script that extracts the metadata from the datastore and the files that need to be rsync'ed to the school server.

Will also attach some sample output files.

[krstic]

Tomeu did his part, now it's my turn.

[marco]

Ivan, can you provide an update on this? If the client/server code is in a reasonable state I think we should propose this ticket for killjoy, it's a really important feature.

[marco]

Ivan, ping? :)

[kimquirk]

moving to update2

[marco]

I think Martin took this over.

[tomeu]

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON.

A patch has been sent to the code-review list and a release including it will be made ASAP.

[tomeu]

Replying to tomeu:

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON. A patch has been sent to the code-review list and a release including it will be made ASAP.

Got into this rpm:

http://koji.fedoraproject.org/koji/buildinfo?buildID=53410

[martinlanghoff]

This has been fixed on Joyride for a while... :-)

[martin.langhoff]

The XS has a "tester usable" index of the Journal as of OLPC_XS_164 (xs-0.3). I am putting for xs-0.4 an end-user usable version of the Journal.

[martin.langhoff]

The end-user version of this depends on magic auth between XO and XS, so defer...