Need ability to copy things to and from Journal/datastore via the command line

[Phil]

Right now, the only way to add things to the Journal involves doing something with an activity.

It would be nice to have a way to add things to the Journal at the command-line. This would facilitate preloading, say, books/music/whatever on an XO by sending out a series of commands that add local files into the Journal for later access.

[rwh]

I agree. Here's a script to do so. Currently doesn't run from the console because gtk is dragged in by the datastore import. However, does run in the terminal activity. Maybe we could set a bit more metadata.

[Phil]

That's nice, rwh. I'll bang on it a bit more tonight and add functionality that allows you to specify metadata on the command-line as well.

It would be very nice if it could run directly from the console, though, so it could be remotely run on the XOs in a preload session.

[Phil]

New version of copy-to-journal that supports metadata.

[Phil]

I've created a new version of the script that supports adding metadata.

However, both versions of this script fail for me when trying to add an Ogg Vorbis file with the following error:

WARNING:root:DSObject was deleted without cleaning up first. Please call DSObject.destroy() before disposing it.
Error: org.freedesktop.DBus.Python.AttributeError: Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/dbus/service.py", line 692, in _message_cb
    retval = candidate_method(self, *args, **keywords)
  File "/usr/lib/python2.5/site-packages/olpc/datastore/datastore.py", line 215, in create
    mp.create_async(props, filelike, can_move=transfer_ownership,
AttributeError: 'NoneType' object has no attribute 'create_async'

Not sure what's going on here.

[mstone]

cjb hypothesized that we might need to set a mime-type on the DS entry we're assembling.

[Phil]

It turns out that the theory cjb hypothesized was a fact. I now require a MIME-type when adding a file to the datastore ... and it works. I can now have Daft Punk on my XO without having to break the Journal with USB keys doing it.

I attached the updated version of copy-to-journal (v3) that supports (and requires) these MIME-type shenanigans.

Thanks for the insight! And thanks again, rwh, for the initial script.

[Phil]

Third revision of copy-to-journal (plus bugfix)

[Phil]

For those who downloaded -3 before I changed it just now, note that it set 'mimetype' and not 'mime_type'. Oops. This is fixed now.

[tomeu]

The DataStore is already scriptable by using python-dbus. Do we need to do anything in this regard?

Anybody thinks we should ship a script in the builds to make this easier? If so, which are the most important use cases so we can define the CLI UI?

Otherwise, perhaps is enough to have this script or others in the wiki.

[Phil]

tomeu, m_stone helped me roll new RPMs and patches and everything for the sugar-datastore package with this script added. Alas, the eMails that involve it are still waiting for approval to the devel@ mailing list, as they were sent before I was a member.

You can find the RPMs here:

http://teach.laptop.org/~phil/

and I can send you the git patches separately if they don't clear through devel@ soon.

m_stone and I believe that this would be useful to have as part of the official datastore build, but it is of course your call. (It really needs to not have to be run from the XO directly, though, which means reengineering it to use the DBus API.)

[Phil]

Use cases I envision:

- An adult preloading a particular XO with music, pictures, and so on, without having to fight with the currently crash-prone USB-to-Journal interface, not to mention the slow click-and-drag nature of said interface;

- Preloading XOs being used in a particular class with all materials needed for that class: ebooks, sound clips, and so on.

This would be made easier if it could be run over an SSH session, which (currently, until #5594 is fixed) requires changing the script to use the DBus APIs directly. But as-is, kicking a simple script off that would rsync or scp files to the laptop and shove them all into the Journal is still much simpler than one-by-one drag-and-dropping files.

[Phil]

Thanks again, rwh. I'll be attaching a lightly spruced-up version of the script in a minute. (Also, yay for it not being very complex!)

This one has more immediate use cases, for those of you out there counting; there have been /many/ people in-channel who use Browse to download some random .tar.gz and then want to retrieve it from the datastore onto the filesystem.

[Phil]

Update my eMail address to my canonical one in this script.

[Phil]

Second release of copy-from-journal, with my minor tweaks.

[31d1]

a little script to explore the datastore, sort of a journal-ls, submitted for your thoughts. (Plus tiny change)

[ffm]

Just a note from someone who is doing user support:

This is pretty important when doing installs. A lot of people do not like to type in long URLS, and cannot paste into terminal, so they end up downloading files with browse. Thus, it is locked into the datastore and it is difficult to extract the file out of it.

It would be a great help if we could get this into update.1.

[bernie]

Replying to ffm:

This is pretty important when doing installs. A lot of people do not like to type in long URLS, and cannot paste into terminal, so they end up downloading files with browse. Thus, it is locked into the datastore and it is difficult to extract the file out of it.

+1

[tomeu]

Could any of the people who find that these use cases are important propose a CLI for this utility or set of utilities?

Would ds-query, ds-get and ds-put be enough?

[Phil]

As a long-time AIX admin, I really like the fact that 'ls<tab>' shows you every "ls-y" command on the system, as does 'ch<tab>' and other similar UNIX-y prefixes.

My personal preference would be 'lsds', 'cptods', and 'cpfromds,' but I'm not married to any particular set of names.

I'd like to add --verbose/-v and --pretend/-p options to the various scripts, as appropriate, but on the whole I think their use of parameters right now are fairly reasonable. I went through the two 'copy' scripts and made their behaviours match as much as was reasonable; another pass could be done (by me or someone else) to make the three look as similar as possible.

[zimm]

Replying to ffm:

Just a note from someone who is doing user support: This is pretty important when doing installs. A lot of people do not like to type in long URLS, and cannot paste into terminal, so they end up downloading files with browse. Thus, it is locked into the datastore and it is difficult to extract the file out of it.

+1

[tomeu]

So, has the community agreed in a set of scripts to ship on the XOs?

[homunq]

just do it, tomeu.

I vote for calling them lsds.py , cpfromds.py and cptods.py .

[tomeu]

Added copy-from-journal, still need to add it to the rpm. Once that's is done, this ticket will be moved to 9.1.0.

[tomeu]

Also added copy-to-journal, any feature requests and bugs about these utilities should be reported in new tickets.

[AlbertCahalan]

These do not work when most needed: when the GUI is unusable

They need to work from the Linux console, as "olpc" or "root".

[AlbertCahalan]

On joyride 2241, from August 2008, the jls script just restarts sugar. I think it may have killed the X server.

None of this is really practical for cleaning out the spam anyway. (don't even think about the hopeless GUI) I have 1150 entries. About 15 of those are activities that I may need to keep.

What we really need is a tool that deletes all files except some specific classes, and can do so without needing to rely on sugar/dbus/X/etc. The specific classes should be the user's choice of: photos, video recordings, audio recordings, *.xo files, and anything else that people might value. Like this:

unspam --keep=photo,xo

[gregorio]

this looks cool! However, I can't tell whether it still works. Can you confirm and if so I'll put it in the release notes.

Thanks,

Greg S