Ticket #4100 (closed task: fixed)

Opened 7 years ago

Last modified 5 years ago

School server component needed to provide human readable index of journal backup

Reported by: cscott Owned by: martin.langhoff
Priority: blocker Milestone: xs-0.6
Component: school server Version:
Keywords: 8.2.0:? Cc: walter, marcopg, bcsaller, tomeu, kimquirk, cscott, krstic, wad, mstone
Action Needed: never set Verified: no
Deployments affected: Blocked By: #7604
Blocking: #7392

Description

Trac #3469 was somehow closed without adding the trac bug for the solution we decided on, which was a school server component which read the journal backup format and created a simple index of the child's files, ideally with the same visual layout used in the Journal.

Attachments

export_datastore_backup.py (3.5 kB) - added by tomeu 7 years ago.
backup-stateJkP6Kt.idx (8.6 kB) - added by tomeu 7 years ago.
backup9-nTOj.idx (54.9 kB) - added by tomeu 7 years ago.
backup-filesnbF982.idx (15.9 kB) - added by tomeu 7 years ago.

Change History

  Changed 7 years ago by tomeu

  • cc walter, marcopg, bcsaller, tomeu added
  • owner changed from tomeu to kimquirk

By which process are the files transferred to the school server?

In which structure are the files stored? And the metadata?

Which searching, sorting and filtering functionality we would need in that remote journal component?

If we need a complex web module for this, perhaps would be easier to implement support in the datastore for remote entries.

In the journal we would need no other modification other than adapt to the difference of speed in the queries.

In the datastore, we would need a new backend that would connect to a remote xapian in the school server, and when files are requested, retrieve and make them accessible locally. See OllyBetts' comment in #2711.

So we need to decide what to do about this in the future milestones.

follow-up: ↓ 4   Changed 7 years ago by cscott

  • cc kimquirk added

We discussed this at length in a meeting some weeks ago.

The file format on the school server is *exactly* the one on the XO. We are doing a simple rsync of the files from the XO to the school server.

As a start, we don't need *any* searching or sorting. Just extracting the human-readable names of the files from the datastore metadata and displaying them in a unadorned list would be sufficient to allow children a simple file-sharing ability.

It's true that we might want more a more elaborate implementation in the future, but let's restrict discussion to what is absolutely required by FRS.

  Changed 7 years ago by jg

  • milestone changed from Never Assigned to First Deployment, V1.0

in reply to: ↑ 2   Changed 7 years ago by tomeu

Scott, in what environment should that component run?

A cgi script in python would be sufficient?

How that script would know which child's files are being requested?

How will children point their browsers to that script? Through a link in the Library home page?

How we map them to the directory containing the backup?

Can I ssh into an existing school server?

  Changed 7 years ago by tomeu

  • owner changed from kimquirk to tomeu

  Changed 7 years ago by tomeu

  • owner changed from tomeu to kimquirk
  • milestone changed from First Deployment, V1.0 to Untriaged

Kim, what do we need for MP or FRS (perhaps we should have two separate tickets for each milestone)?

Should be decided soon, given the impact this task could have.

  Changed 7 years ago by jg

  • milestone changed from Untriaged to First Deployment, V1.0

Kim, either we need to resource this somehow, or delay to 1.1...

  Changed 7 years ago by cscott

  • cc cscott added

If I can get a simple standalone script which will run on an ordinary Fedora Core 7 system and print out the filenames, human-readable names, and timestamps of the files, I can work from there to integrate it into the schoolserver and webify it. But I'd like help getting the basic datastore pieces separated out.

  Changed 7 years ago by kimquirk

  • owner changed from kimquirk to bcsaller

changing ownership to Ben.

  Changed 7 years ago by krstic

  • owner changed from bcsaller to tomeu

I hashed out a spec for this that made tomeu declare he loves me:

http://wiki.laptop.org/go/XS_backup_restore

As per Marco, reassigning this to Tomeu to write the XO part that deals with the datastore and writes the indices (this isn't Ben's usual domain, since it's more a client use of the DS).

  Changed 7 years ago by krstic

  • cc krstic added

  Changed 7 years ago by marco

I reviewed Ivan spec and it looks pretty good. It address all the concerns I had about the plain rsync approach we discussed yesterday. I'd say it's the way to go for 1.0.

  Changed 7 years ago by tomeu

Ok, so just to clarify and check if I understood correctly the spec, my part is to write a func like this:

path_to_backup_idx, path_to_backup_state_idx, path_to_backup_files_idx = \
    write_backup_index_since(timestamp, max_items=None)

That filters the items to backup according to timestamp and max_items and creates the following files:

  • backup.idx: file that contains the metadata of the entries to backup,
  • backup-state.idx: file with the ids of all the entries currently present in the DS (not filtered nor limited by max_items),
  • backup-files.idx: file with the absolute paths to the data files and thumbnails of the entries to backup.

follow-up: ↓ 15   Changed 7 years ago by tomeu

From some mail:

On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

- XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual file plus metadata in a .xoc file. After Browse has downloaded it and put into the Journal like with all downloads, the Journal could then expand it and put the metadata into xapian.

This would also serve for sharing journal entries, one journal would send through tubes a bundled entry, and the other side would expand it and recreate in its own datastore.

in reply to: ↑ 14   Changed 7 years ago by tomeu

Replying to tomeu:

From some mail: On Tue, 2007-10-16 at 14:41 -0400, Kim Quirk wrote:

- XO can retreive one file at a time into the journal from the school server webpage.

Could be nice if the server component could bundle using zip the actual file plus metadata in a .xoc file. After Browse has downloaded it and put into the Journal like with all downloads, the Journal could then expand it and put the metadata into xapian. This would also serve for sharing journal entries, one journal would send through tubes a bundled entry, and the other side would expand it and recreate in its own datastore.

This is in #4380.

  Changed 7 years ago by tomeu

Attached is the script that extracts the metadata from the datastore and the files that need to be rsync'ed to the school server.

Will also attach some sample output files.

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

Changed 7 years ago by tomeu

  Changed 7 years ago by krstic

  • owner changed from tomeu to krstic

Tomeu did his part, now it's my turn.

  Changed 7 years ago by marco

Ivan, can you provide an update on this? If the client/server code is in a reasonable state I think we should propose this ticket for killjoy, it's a really important feature.

  Changed 7 years ago by kimquirk

  • milestone changed from Update.2 to Update.1

  Changed 7 years ago by marco

Ivan, ping? :)

  Changed 7 years ago by marco

  • cc wad added

  Changed 7 years ago by kimquirk

  • milestone changed from Update.1 to Update.2

moving to update2

  Changed 6 years ago by marco

  • owner changed from krstic to martin.langhoff

I think Martin took this over.

  Changed 6 years ago by mstone

  • cc mstone added

follow-up: ↓ 27   Changed 6 years ago by tomeu

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON.

A patch has been sent to the code-review list and a release including it will be made ASAP.

  Changed 6 years ago by marco

  • keywords 8.2.0:? added
  • component changed from journal-activity to school server

in reply to: ↑ 25   Changed 6 years ago by tomeu

Replying to tomeu:

Has been decided that instead of being the client backup script to export the metadata from the xapian index, the DS will create a file for every entry containing its metadata in JSON. A patch has been sent to the code-review list and a release including it will be made ASAP.

Got into this rpm:

http://koji.fedoraproject.org/koji/buildinfo?buildID=53410

  Changed 6 years ago by gregorio

  • blocking 7392 added

  Changed 6 years ago by martinlanghoff

  • next_action set to never set

This has been fixed on Joyride for a while... :-)

  Changed 6 years ago by martin.langhoff

  • milestone changed from 8.2.0 (was Update.2) to xs-0.4

The XS has a "tester usable" index of the Journal as of OLPC_XS_164 (xs-0.3). I am putting for xs-0.4 an end-user usable version of the Journal.

  Changed 6 years ago by martin.langhoff

  • milestone changed from xs-0.4 to Future Release

The end-user version of this depends on magic auth between XO and XS, so defer...

  Changed 6 years ago by martin.langhoff

  • blockedby 7604 added

  Changed 5 years ago by martin.langhoff

  • status changed from new to closed
  • type changed from defect to task
  • resolution set to fixed
  • milestone changed from Future Release to xs-0.6
Note: See TracTickets for help on using tickets.