Opened 7 years ago

Closed 6 years ago

#4100 closed task (fixed)

School server component needed to provide human readable index of journal backup

Reported by: cscott Owned by: martin.langhoff
Priority: blocker Milestone: xs-0.6
Component: school server Version:
Keywords: 8.2.0:? Cc: walter, marcopg, bcsaller, tomeu, kimquirk, cscott, krstic, wad, mstone
Blocked By: #7604 Blocking: #7392
Deployments affected: Action Needed: never set
Verified: no

Description

Trac #3469 was somehow closed without adding the trac bug for the solution we decided on, which was a school server component which read the journal backup format and created a simple index of the child's files, ideally with the same visual layout used in the Journal.

Attachments (4)

export_datastore_backup.py (3.5 KB) - added by tomeu 7 years ago.
backup-stateJkP6Kt.idx (8.6 KB) - added by tomeu 7 years ago.
backup9-nTOj.idx (54.9 KB) - added by tomeu 7 years ago.
backup-filesnbF982.idx (15.9 KB) - added by tomeu 7 years ago.

Download all attachments as: .zip

Change History (37)

comment:1 Changed 7 years ago by tomeu

  • Cc walter marcopg bcsaller tomeu added
  • Owner changed from tomeu to kimquirk

By which process are the files transferred to the school server?

In which structure are the files stored? And the metadata?

Which searching, sorting and filtering functionality we would need in that remote journal component?

If we need a complex web module for this, perhaps would be easier to implement support in the datastore for remote entries.

In the journal we would need no other modification other than adapt to the difference of speed in the queries.

In the datastore, we would need a new backend that would connect to a remote xapian in the school server, and when files are requested, retrieve and make them accessible locally. See OllyBetts' comment in #2711.

So we need to decide what to do about this in the future milestones.

comment:2 follow-up: Changed 7 years ago by cscott

  • Cc kimquirk added

We discussed this at length in a meeting some weeks ago.

The file format on the school server is *exactly* the one on the XO. We are doing a simple rsync of the files from the XO to the school server.

As a start, we don't need *any* searching or sorting. Just extracting the human-readable names of the files from the datastore metadata and displaying them in a unadorned list would be sufficient to allow children a simple file-sharing ability.

It's true that we might want more a more elaborate implementation in the future, but let's restrict discussion to what is absolutely required by FRS.

comment:3 Changed 7 years ago by jg

  • Milestone changed from Never Assigned to First Deployment, V1.0

comment:4 in reply to: ↑ 2 Changed 7 years ago by tomeu

Scott, in what environment should that component run?

A cgi script in python would be sufficient?

How that script would know which child's files are being requested?

How will children point their browsers to that script? Through a link in the Library home page?

How we map them to the directory containing the backup?

Can I ssh into an existing school server?

comment:5 Changed 7 years ago by tomeu

  • Owner changed from kimquirk to tomeu

comment:6 Changed 7 years ago by tomeu

  • Milestone changed from First Deployment, V1.0 to Untriaged
  • Owner changed from tomeu to kimquirk

Kim, what do we need for MP or FRS (perhaps we should have two separate tickets for each milestone)?

Should be decided soon, given the impact this task could have.

comment:7 Changed 7 years ago by jg

  • Milestone changed from Untriaged to First Deployment, V1.0

Kim, either we need to resource this somehow, or delay to 1.1...

comment:8 Changed 7 years ago by cscott

  • Cc cscott added

If I can get a simple standalone script which will run on an ordinary Fedora Core 7 system and print out the filenames, human-readable names, and timestamps of the files, I can work from there to integrate it into the schoolserver and webify it. But I'd like help getting the basic datastore pieces separated out.

comment:9 Changed 7 years ago by kimquirk

  • Owner changed from kimquirk to bcsaller

changing ownership to Ben.

comment:10 Changed 7 years ago by krstic

  • Owner changed from bcsaller to tomeu

I hashed out a spec for this that made tomeu declare he loves me:

http://wiki.laptop.org/go/XS_backup_restore

As per Marco, reassigning this to Tomeu to write the XO part that deals with the datastore and writes the indices (this isn't Ben's usual domain, since it's more a client use of the DS).

comment:11 Changed 7 years ago by krstic

  • Cc krstic added

comment:12 Changed 7 years ago by marco

I reviewed Ivan spec and it looks pretty good. It address all the concerns I had about the plain rsync approach we discussed yesterday. I'd say it's the way to go for 1.0.

comment:13 Changed 7 years ago by tomeu

Ok, so just to clarify and check if I understood correctly the spec, my part is to write a func like this:

path_to_backup_idx, path_to_backup_state_idx, path_to_backup_files_idx = \
    write_backup_index_since(timestamp, max_items=None)

That filters the items to backup according to timestamp and max_items and creates the following files:

  • backup.idx: file that contains the metadata of the entries to backup,
  • backup-state.idx: file with the ids of all the entries currently present in the DS (not filtered nor limited by max_items),
  • backup-files.idx: file with the absolute paths to the data files and thumbnails of the entries to backup.

comment:14 follow-up: Changed 7 years ago by tomeu

From some mail:

On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

  • XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual
file plus metadata in a .xoc file. After Browse has downloaded it and
put into the Journal like with all downloads, the Journal could then
expand it and put the metadata into xapian.

This would also serve for sharing journal entries, one journal would
send through tubes a bundled entry, and the other side would expand it
and recreate in its own datastore.

comment:15 in reply to: ↑ 14 Changed 7 years ago by tomeu

Replying to tomeu:

From some mail:

On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

  • XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual
file plus metadata in a .xoc file. After Browse has downloaded it and
put into the Journal like with all downloads, the Journal could then
expand it and put the metadata into xapian.

This would also serve for sharing journal entries, one journal would
send through tubes a bundled entry, and the other side would expand it
and recreate in its own datastore.

This is in #4380.

comment:16 Changed 7 years ago by tomeu

Attached is the script that extracts the metadata from the datastore and the files that need to be rsync'ed to the school server.

Will also attach some sample output files.

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

comment:17 Changed 7 years ago by krstic

  • Owner changed from tomeu to krstic

Tomeu did his part, now it's my turn.

comment:18 Changed 7 years ago by marco

Ivan, can you provide an update on this? If the client/server code is in a reasonable state I think we should propose this ticket for killjoy, it's a really important feature.

comment:19 Changed 7 years ago by kimquirk

  • Milestone changed from Update.2 to Update.1

comment:20 Changed 7 years ago by marco

Ivan, ping? :)

comment:21 Changed 7 years ago by marco

  • Cc wad added

comment:22 Changed 7 years ago by kimquirk

  • Milestone changed from Update.1 to Update.2

moving to update2

comment:23 Changed 7 years ago by marco

  • Owner changed from krstic to martin.langhoff

I think Martin took this over.

comment:24 Changed 7 years ago by mstone

  • Cc mstone added

comment:25 follow-up: Changed 7 years ago by tomeu

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON.

A patch has been sent to the code-review list and a release including it will be made ASAP.

comment:26 Changed 6 years ago by marco

  • Component changed from journal-activity to school server
  • Keywords 8.2.0:? added

comment:27 in reply to: ↑ 25 Changed 6 years ago by tomeu

Replying to tomeu:

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON.

A patch has been sent to the code-review list and a release including it will be made ASAP.

Got into this rpm:

http://koji.fedoraproject.org/koji/buildinfo?buildID=53410

comment:28 Changed 6 years ago by gregorio

  • Blocking 7392 added

comment:29 Changed 6 years ago by martinlanghoff

  • Action Needed set to never set

This has been fixed on Joyride for a while... :-)

comment:30 Changed 6 years ago by martin.langhoff

  • Milestone changed from 8.2.0 (was Update.2) to xs-0.4

The XS has a "tester usable" index of the Journal as of OLPC_XS_164 (xs-0.3). I am putting for xs-0.4 an end-user usable version of the Journal.

comment:31 Changed 6 years ago by martin.langhoff

  • Milestone changed from xs-0.4 to Future Release

The end-user version of this depends on magic auth between XO and XS, so defer...

comment:32 Changed 6 years ago by martin.langhoff

  • Blocked By 7604 added

comment:33 Changed 6 years ago by martin.langhoff

  • Milestone changed from Future Release to xs-0.6
  • Resolution set to fixed
  • Status changed from new to closed
  • Type changed from defect to task
Note: See TracTickets for help on using tickets.