OSK opened by keyboard frame device does not send key events

[dsd]

Opening the OSK using the keyboard frame device and typing does not actually send key events anywhere (useful at least). Typing on the OSK raised by the device icon does nothing.

Sugar shell, steps to reproduce:

1. in ebook mode
2. on the home favourites view
3. raise the frame
4. tap the frame keyboard device
5. start typing on the OSK

Result:

• Nothing happens (other than the OSK happily accepting your typing)

Expectations:

• The typed text should trigger focus to the 'Search In Home' widget, filtering the view

Calculate, steps to reproduce:

1. in ebook mode
2. start calculate
3. note that tapping the Label field raises the OSK
4. tap the main calculate input field (OSK is hidden so you can use the canvas buttons)
5. raise the frame
6. tap the frame keyboard device
7. start typing on the OSK

Result:

• Nothing happens (other than the OSK happily accepting your typing)

Expectations:

• The typed text should be going into the main calculate input field, just like you were typing on the physical keyboard

Turtle Art, steps to reproduce:

1. in ebook mode
2. start turtle art
3. switch to the Palette of variable blocks (open box icon)
4. drag the green text block onto the canvas
5. tap it to give it focus (a cursor will appear)
6. raise the frame
7. tap the frame keyboard device
8. start typing on the OSK

Result:

• Nothing happens (other than the OSK happily accepting your typing)

Expectations:

• The typed text should be going into the text block, just like you were typing on the physical keyboard

[dsd]

Originally filed at http://bugs.sugarlabs.org/ticket/4054

This is harder than it might sound.

The OSK normally works as follows: certain GTK+ input elements (e.g. GtkInput) have special code that calls into the input method when the focus happens. The maliit IM then contacts the maliit server over dbus saying "I'm an active input field!".

Maliit server responds by recording that the client is active, then when the user clicks on the OSK keys, it sends dbus messages back to the active client. The maliit IM in receives those messages, and since it is an IM for an active text field, it has the ability to ask GTK+ to insert characters.

The case when the OSK is manually triggered is very different. In this case, the maliit server has no idea which app has the active text input field, so it doesn't know where to send the messages. The app is probably not even listening for those messages - if the OSK has to be pulled up manually we can assume that we don't have a GTK IM context (because we're not working with a GtkEntry or something).

This situation must be handled with some different approach to simulating keyboard events. One option would be for maliit-server to detect the situation where it doesn't have an active IM client, and in such case, use XSendEvent or XTest to inject a regular keypress to the active window at the X level.

Maliit upstream acked the idea: http://lists.maliit.org/pipermail/maliit-discuss-maliit.org/2012-November/000256.html

But, coming down to implementation, this seems like a dead end. XTest and XSendKey are limited to keys that are on the physical keyboard, and even being able to access all the keys seems difficult.

I posted to the xorg list to see if there are other alternatives. http://lists.x.org/archives/xorg/2012-November/055033.html

Also, perhaps I could have 5 minutes of Carlos G's time to see if he has ideas?

[dsd]

We have examined the use cases and their associated challenges and have decided to push out the manually invoked OSK feature for this release.

We will renew focus on item #2 below to minimize the impact where realistically possible.

Here is the full breakdown:

Manual (frame-triggered) OSK use cases

1. OSK use in laptop mode. You want to type in a language that you don't have on your physical keyboard. You have (or recently have had) a GTK IM context to work with.

This might be fixable but it seems like it would need a good few hours of work, and it might be painful. There are 4 things to address.

a) The first critical problem in this case, preventing further investigation of this item, is that you currently cannot close the OSK while retaining focus to the input widget you were working with. As soon as you click the button to close the OSK, the focus goes somewhere else, and I'm not sure where or why.

b) Once that issue is fixed, the real test can be run: what happens when you activate a text field (causing the OSK to automatically appear), then hide the OSK (the text field should still be focused), then activate the OSK via the frame? For this to work we need the text field to retain input focus throughout the whole sequence, and for its IM context of the input widget not to be deactivated without then being immediately activated again. My prediction is that work will be needed to make this happen.

c) Once that is working, we will need a solution for #12281. This will need another few hours work, since a maliit API change is needed, and I think a change to the internals too. At a glance, Maliit seems to destroy all the OSK widget stuff when the laptop is in tablet mode, rather than just "soft disabling" it because the switch is closed, so this will not be as easy as just force-activating the OSK.

d) A potential issue that will affect (b) and (c) is that the way that the Sugar frame code brings up the OSK involves creating and activating a new IM context. That will deactivate the IM context of the active widget. I'm not 100% sure of this point, but if true, this will need to be changed, potentially requiring new maliit API and changes to the internals.

2. GTK+ activities where there is no IM context because a custom input field is used (e.g. Paint, Labyrinth, FotoToon).

This can be fixed on an activity-by-activity basis. Activities can either overlay a standard GTK+ input widget on the right part of the screen, or the custom input widgets can be made to interact with the GtkIMContext API which will cause the OSK to come up as appropriate.

If no activity-level fix is made, these cases fall into the following item #3 "Non-GTK+ activities."

Note that the "fix" here is not really a fix - the manual OSK will still be broken for these cases unless #1 is fixed too. However, things will be "fixed" from the respect that affected activities can be used in tablet mode since the OSK will come up automatically, removing the requirement to invoke it manually.

3. Non-GTK+ activities (etoys, Scratch, pygame-based activities)

These are out of the question for 13.1.0 (and perhaps indefinitely), since we just don't have an appropriate way into the X event stream (http://lists.x.org/archives/xorg/2012-November/055033.html)

---

If we do go with solving #1, and given that #3 is impossible, we will be faced with a challenge of offering a good manual OSK user experience. This is because the manual OSK will not work in the #3 situation, even though it will pop up on-demand and give the normal interactive keyboard experience, just being unable to actually send keypresses to the active app.

The maliit server could be extended to know when there is no context available, and make that state available over dbus. If the frame were to send a dbus message every time it were opened (needs new maliit API), it could discover this state, and modify the frame icon behaviour accordingly (hide it, or make it greyed out). However the idea of sending a dbus message at this point sounds bad, and also, would this really provide a better user experience? For the #3 case, instead of presenting a keyboard that doesn't work, having a hidden or greyed out icon would be just as confusing to the user ("where's the friggin OSK gone???").