Beat XP in activity startup time.

[cscott]

Measured activity startup time in 650, for Pippy, is 10 seconds. We need to greatly improve this!

First step is to implement a forking python process.

Apparently 'inspect' (used by dbus and telepathy for better traceback) is another bottleneck; use of this package should be changed to lazily load it when an exception is thrown.

A wrapper for 'import foo' which creates a stub object 'foo' which will lazily load the module when foo is first dereferenced should also help.

[marco]

See #5228

[mtd]

I volunteer for "A wrapper for 'import foo' which creates a stub object 'foo' which will lazily load the module when foo is first dereferenced should also help."

The rest I think are covered by #5228, at a quick read.

[cscott]

mtd: great!

This probably won't work for *all* modules, but it should work for a good number of them.

I'm hoping for a result that let's us replace (say): :import foo with :lazy_import('foo') or some such (how close to the standard python syntax can we make it?) so that the sugar folks can keep their 'all imports must be at the top of the file' coding guidelines without killing our performance.

I look forward to seeing a patch! This always seemed like a fun grungy python project to me, I just haven't had time to do it in the past six months... (sigh)

[mtd]

Replying to cscott:

mtd: great! This probably won't work for *all* modules, but it should work for a good number of them.

It's tempting to play with autoimp but that does seem like overkill

I'm hoping for a result that let's us replace (say): :import foo with :lazy_import('foo') or some such (how close to the standard python syntax can we make it?) so that the sugar folks can keep their 'all imports must be at the top of the file' coding guidelines without killing our performance.

Indeed, that's what I was thinking too. Importing does exactly that (import foo --> foo = lazyModule("foo")), and a py.std type of syntactic sugar could then be added if absolutely necessary (but IMHO that's overkill - lazyModule("foo") seems fine to me).

I look forward to seeing a patch! This always seemed like a fun grungy python project to me, I just haven't had time to do it in the past six months... (sigh)

:)

I have added some sugar guys to the cc: so they might not be totally surprised when I ask them about this on irc.

[tomeu]

Replying to cscott:

mtd: great! This probably won't work for *all* modules, but it should work for a good number of them. I'm hoping for a result that let's us replace (say): :import foo with :lazy_import('foo')

I'm afraid that this won't have as much impact as you think, as most of the modules will be imported anyway during startup:

• because they are used anyway before the window is brought up,
• because some other module not in our codebase imports it eagerly.

But I would love to be _proved_ wrong ;)

or some such (how close to the standard python syntax can we make it?) so that the sugar folks can keep their 'all imports must be at the top of the file' coding guidelines without killing our performance.

As I said somewhere else, if someone can provide a patch that improves user perceivable performance measurably by just shuffling imports around, I don't think that any style guideline will get in the way.

[mtd]

I'm getting closer to tomeu's perspective on this. Attaching a file that will do the lazy importing...now if we could only figure out where to put it :).

I've had a play with sugar-toolkit/activity/activity.py and Terminal-activity/terminal.py and I don't think we're going to get more than a second or so out of lazy importing.

To use, here's an example of some changes to terminal.py that I tried:

import time
t0 = time.time()

from lazyimport import lazyModule

os = lazyModule('os')  # import os                                                                                                                                

logging = lazyModule('logging')  # import logging                                                                                                                 
from gettext import gettext as _

gtk = lazyModule('gtk')  # import gtk                                                                                                                             
dbus = lazyModule('dbus')  # import dbus                                                                                                                          

from sugar.activity import activity
from sugar import env
from sugar.graphics.toolbutton import ToolButton
from sugar.graphics.palette import Palette
import ConfigParser
import os.path

vte = lazyModule('vte')  # import vte                                                                                                                             
pango = lazyModule('pango')  # import pango                                                                                                                       

t1 = time.time()
logging.debug("terminal.py: imports took %s secs" % (t1 - t0))

[tomeu]

import time
t0 = time.time()

from lazyimport import lazyModule

os = lazyModule('os')  # import os

At this point, 'os' quite probably has already been imported by time or lazyimport.

logging = lazyModule('logging')  # import logging
from gettext import gettext as _

gtk = lazyModule('gtk')  # import gtk
dbus = lazyModule('dbus')  # import dbus

from sugar.activity import activity

This last import imported gtk and dbus.

from sugar import env
from sugar.graphics.toolbutton import ToolButton
from sugar.graphics.palette import Palette
import ConfigParser
import os.path

vte = lazyModule('vte')  # import vte
pango = lazyModule('pango')  # import pango

pango had already been imported above by gtk, etc.

And vte is needed to display the activity, so it will be imported anyway before we can show anything to the user. Same for gettext.

t1 = time.time()
logging.debug("terminal.py: imports took %s secs" % (t1 - t0))

So I would be surprised if using lazy imports in this way would bring any improvements, I would just expect a small amount of overhead.

If lazy imports are to be useful, it should be applied to all modules, without having to change code. We should move the module initialization from import time to first-use time.

Perhaps we should patch python? I think that for python 3k there will be hooks that will allow to do this kind of things without patching.

[tomeu]

See PyImport_ExecCodeModuleEx in:

http://svn.python.org/view/python/tags/r251/Python/import.c?view=markup

The patch linked below will help analysing how we import modules and the time it takes to do so. If anyone wants to make python rpms with this patch, ask me in irc if you need some pointers.

http://dev.laptop.org/~tomeu/python-2.5.1-time-import.patch

[mtd]

Replying to tomeu:

So I would be surprised if using lazy imports in this way would bring any improvements, I would just expect a small amount of overhead.

I agree -- the code was just so others could see how the naive approach really won't get much.

If lazy imports are to be useful, it should be applied to all modules, without having to change code. We should move the module initialization from import time to first-use time.

autoimp would do this. We (in theory) would be able to replace all imports with "import autoimp" and then things would be imported only as necessary. Perhaps this would be useful at least just to establish an upper-bound to the benefits of lazy importing, as security / stability considerations might preclude its use in real life.

To be fair cscott mentioned Pippy, and not (or only by implication) all sugar apps are slow by inherent sugar design.

Perhaps we should patch python? I think that for python 3k there will be hooks that will allow to do this kind of things without patching.

Maybe someone is interested in enabling this lazy-import behavior a crazy, runtime-option for py3k? So we don't have to do it...

[tomeu]

Replying to mtd:

Replying to tomeu:

Perhaps we should patch python? I think that for python 3k there will be hooks that will allow to do this kind of things without patching.

Maybe someone is interested in enabling this lazy-import behavior a crazy, runtime-option for py3k? So we don't have to do it...

This is what I was thinking about: http://www.python.org/dev/peps/pep-0302

[mtd]

Replying to tomeu:

This is what I was thinking about: http://www.python.org/dev/peps/pep-0302

I think what you want is autoimp, by default, plus a no-op default import hook, right? autoimp uses those hooks from PEP 302 (IIRC).

[tomeu]

Replying to mtd:

Replying to tomeu:

This is what I was thinking about: http://www.python.org/dev/peps/pep-0302

I think what you want is autoimp, by default, plus a no-op default import hook, right? autoimp uses those hooks from PEP 302 (IIRC).

Looks like, sounds good to you? Do you think you could find time to give it a try and report back?

[mtd]

Replying to tomeu:

Replying to mtd:

I think what you want is autoimp, by default, plus a no-op default import hook, right? autoimp uses those hooks from PEP 302 (IIRC).

Looks like, sounds good to you? Do you think you could find time to give it a try and report back?

Yes. Will do.