Handheld mode: rotation, activity switching, and key-map

[gnu]

When the laptop is in ebook mode, at any of the standard Sugar screens, every accessible button is inoperative. You can't start an application, can't stop one, can't pick a wireless hub, can't share an activity. You can rotate the screen (using the backlight button), and you can power off the device (usually inadvertently). But both sets of cursor keys are ignored! You are forced to open up the laptop to e.g. get into the book reader.

It's unfortunate but excusable for an individual application to be deaf-and-dumb when in ebook mode. In a transformer-hinged laptop, having the window manager & main user interface in this state is crippling to the usability of ebook mode.

It's either a UI design problem, or a UI implementation problem. Is there a UI defined yet for the main sugar screen using just the cursor pad plus the four symbol keys?

[jg]

Yes, we know. Haven't had time to implement this in the book reading utilities yet, or other button capable activities.

[AlbertCahalan]

I think his point was about Sugar itself, not the "book reading utilities" or "other button capable activities". He's somewhat excusing the individual apps. Without support in Sugar, you have to lift up the screen to control Sugar itself.

Fixing this might mean reserving a button for Sugar. I suggest having the backlight/rotate button bring up a menu rather than acting immediately. Normally I hate pie menues with a passion, but they might be appropriate here because of the limited controls.

[Eben]

Early thoughts for the mode specifically considered that Sugar would not be supported; Many of the main functions of Sugar aren't suitable for the controls available. Recent discussion has brought up the possibility of allowing switching of activities via handheld mode as the sole Sugar supported operation, and I think this is a reasonable thing to shoot for.

Sugar should automatically jump to the activity view when entering handheld mode, since the other views won't be supported. We'll need to formalize a means by which to switch activities. I'm beginning to like the idea of commandeering the rotate button as the "handheld menu" button. We've already decided that we want to require a second action when rotating the screen, and therefore want some on screen visual when this button is pressed. I also feel that its a terrible idea to steal any of the other 8 buttons from the activities, since they are already limited, and I think it's useful to maintain the natural diametrically opposing pairs of buttons in many cases.

I've been wanting an on screen key mapping for handheld mode which illustrates both press and hold actions for the buttons visually. I think it makes sense to tie this key mapping to the rotate key, since the mapping may change when the screen rotates. It also makes sense to tie this key mapping to the switcher interface, so that you can review the mapping for the selected activity when you switch to it. I really liked the idea of pressing the direction that you want up to be, but perhaps we could settle for using left/right to switch activity, up/down to rotate. We could also map activity switching to one set of buttons, and rotation to the other. I'm not sure what the right solution is, but does it sound reasonable to design such that this single button is the "handheld management" button?

Fortunately for us, the icon on the button could be interpreted both as a rotation button and also as a cycle through activities button. Perhaps we could find a better icon in the future, but it actually seems ok as is for now, even if we do overload its purpose.

[Eben]

Some history on the topic of the rotate button can be found in ticket #1049, for reference.

[bemasc]

I have several independent points to make.

Virtual Cursor: Ideally, each activity will have optimized tablet-mode key bindings. However, I feel we should still provide a virtual cursor, controlled by the D-pad. This virtual cursor would be considered an advanced feature, for those not satisfied with the limitations of keybindings, so its presence need not be extremely discoverable. To switch into virtual cursor mode, there might be a virtual cursor item in the rotate key's menu. For a few activities such as Paint, this may even be a sensible default.

Once in virtual cursor mode, two of the game buttons should be bound to the mouse buttons (perhaps the X and O buttons, for consistency with the mouse-button labels). Screen rotation should automatically alter the directions of the D-pad for consistency.*

Virtual cursor mode would also make the rest of the Sugar "available", because the frame can be brought up and navigated by pushing the pointer into one corner. For the randomized Mesh and Neighborhood views, there is really no better way.

*: this should probably happen regardless of virtual cursor mode.

Home Screen Bindings: I see no reason why the home screen should not be accessible in Ebook mode. The interface suggests very obvious keybindings: up/down to switch between the activity list, the ring, the clipboard, and the friendslist. Left/right to choose within each of these. Check to "click". If necessary, North/South may be used to navigate within dropdown menus. This would enable starting activities after moving into Ebook mode, which seems like a perfectly reasonable thing to support. Switching to and from the home screen should be an option in the rotate button's menu.

Activity Rotation preferences: Not all activities make sense to rotate. Many activities are designed around a fixed aspect ratio (e.g. Connect), and the user does not benefit from forcing these activities to support rotation. Each activity should be able to list which rotations it supports. Also, there is no need for Sugar to support rotation, if future UI designs make rotation difficult.

[AlbertCahalan]

Replying to bemasc:

Activity Rotation preferences: Not all activities make sense to rotate. Many activities are designed around a fixed aspect ratio (e.g. Connect), and the user does not benefit from forcing these activities to support rotation. Each activity should be able to list which rotations it supports. Also, there is no need for Sugar to support rotation, if future UI designs make rotation difficult.

Some programs will be able to support all 4 rotations, but will not be able to switch without restarting. This will probably be the case for Tux Paint in the near future.

Being upside-down seems unlikely to break anything. The booleans are thus: portrait, landscape, switchable.

[homunq]

Re virtual cursor: I think that, given the form packing structure, it should be possible to use the arrow keys jump directly from widget to widget, rather than having a mouse cursor to move around. I have a proposal involving this idea that I've discussed on devel list and privately with Eben; I'll try to summarize it here when I get the time. Suffice it to say that I think this is worth it.

Separately: another obviously necessary function is the brightness control, and, for any activity with sound, volume. So, based on Eben's idea above, the rotate button would let you: rotate using an arrow key for which way is up; switch activities (alt-tab) with W and E; adjust brightness with N and S; or adjust volume with N hold and S hold. (That last is less-than-obvious, but it's not such a core function for most activities, and activities for which it is should provide a quicker way.)

[homunq]

Basic scheme

This is a proposal for

arrows

Note that arrows change mapping depending on screen orientation.

static pane (Read, Paint, etc.)

press: page up, down, left, or right (or, if entire pane fits on screen and no scrolling is possible, 'out') hold: continuous scroll hold when, as of initial press, already scrolled to the maximum in that direction: 'out'

text pane (Write, etc.)

press when selection is empty: arrow key press when selection is full: arrow key when button is released (buttonUp) hold: repeating shift-arrow-key (extend selection), starting back from cursor point when button pressed. hold when, as of initial press, already scrolled to the maximum in that direction: 'out'

controls (Calculator, etc.)

press: select next control in given direction hold: jump 'out' (note that the sugar frame would have the center as a 'selectable control' for dismissing it and for easier navigation from bottom to side.)

Pane with mixed graphics and controls (Browse, etc.)

Two modes, 'inside' and 'outside'. In 'inside' mode, as for 'controls' above. In 'outside' mode, as 'static pane', above. To switch modes out, as above or W. To switch modes in, E.

Popup list or similar one-dimensional compound control

Up and down choose items L, R, U hold and D hold: choose next control in that direction.

shape buttons

W: dismiss menu/dismiss selection (leaving cursor at 'most recent' end)/out/get frame. Note that the square shape is a good mapping for the frame button W hold: app-defined. E: select/in. Note check mark maps to enter. E hold: app-defined context menu (when text is selected, has 'copy' option). Like right-click. N,S,N hold,S hold: app-defined

==two kinds of menus== context menus summoned by a hold (such as N hold or S hold) are sticky menus. Arrows to choose option (defaults to most recent selection) then E to select, or W to get out.

context menus summoned by a press (if any) are called 'rockers'. They remap 7 of the 8 buttons (excepting W, which dismisses them). Depending on the function assigned to each button, it may leave the menu active or dismiss it. The function on E should be one which dismisses the menu.

A special case of the rocker menu is the one summoned by the rotate key. An arrow key would set that direction as up, E and W would alt-tab through activities, N and S would be the brightness controls (very necessary), N and S hold would be volume control, and E or W hold would toggle the bulletin board (as this will become a fundamental part of classroom use IMO). Clicking rotate again would go back to normal and briefly show a graphic indicator of what all the buttons do (the indicator also flashes as you push the buttons, except for simple arrow presses).

Again, I understand that for any given activity, it is probably possible with a lot of thought to make a better interface by statically mapping W and E to common functions. However, I think that the advantages of increased consistency for the user, of generalized navigability, and of taking that task as much as possible off the shoulders of the programmer, outweigh this. Also note that these buttons are available for mouse-free navigation even with the laptop open (if the mouse or part of the screen is broken, or for running a laptop with some kind of remote control, or for generalized accessibility, or whatever).

[homunq]

oops I meant to preview that first. Well, good enough. Please discuss the above, it's only a proposal.

[rharrison]

I too would like to see the dpad control the cursor with the laptop in ebook mode. The browse and read activities would be much more useful than they are now. I continually find myself having to open the laptop to get to the touch pad so I can click on a link in a web page or pdf.

[mtd]

Am I correct in summarizing a subset of the latest, uncontested ideas as:

1a) switch to activity view when in sugar view (not in an activity) and ebook/handheld mode is invoked [eben]

1b) in activity view, a rotate button press should not rotate but flick through open (running) activities [eben, maybe maybe]

1c) in activity view, a dpad Left/Right (West/East?) cycles around the (launchable) activities in the activity ring and Check launches [bemasc, maybe]

I think the above would be fairly easy to implement (building on my ebook-switch-recognition patch from #2954).

Alternately, should these be:

1a) same

1b) rotate button press should bring up an "on screen key mapping" overlay [eben] showing / enabling some active-activity / brightness / volume manipulation functions [homunq, #comment:12 ]

1c) same

...? The "overlay" seems harder.

[Eben]

I think we need some more feedback on the use cases, personally. My initial inclination was that handheld mode would be entered within the context of a specific activity which took advantage of its benefits, and not that it would provide a parallel full-featured means of operating the entire Sugar UI. I think that the questions:

1. Can I switch among running activities?
2. Can I open new activities?
3. Can I interact with Home/Group/Neighborhood views?
4. Can I interact with the Frame?

all need to be answered independently, and will have an effect on how we choose to use the buttons available. I initially answered no to all of these.

I think (referring to the previous comment now) (1a) is a good idea assuming that (B) and (C) are answered "no". I think that (1c) is reasonable if (B) and (C) are answered "yes". I think that The alternate (1b) (the overlay) is the better way to go, assuming that (B) and (C) are answered "no", but that this also complicates the simple action of rotating the screen as well, which I'm dissatisfied with.

But in the end, I still might prefer keeping things simple and answering "no" to (B), (C), and (D), and perhaps "yes" to (A) assuming we can find a reasonable way to make the activity switch happen. We have an intern doing research on potential use cases and input mappings for handheld mode, and I think we should wait until that comes further along before finalizing any of these decisions.

[mtd]

Replying to Eben:

I think we need some more feedback on the use cases, personally.

...

We have an intern doing research on potential use cases and input mappings for handheld mode, and I think we should wait until that comes further along before finalizing any of these decisions.

Thanks for the feedback. Look forward to the results.

[mstone]

See also #5549.

[gregorio]

Hi Guys,

This came up in my scan of bugs with relnote in the keyword.

I'm going to close this assuming you that it works as designed now.

Re-open as a bug if you think its not doing what the manual says it should do.

Thanks,

Greg S

[Eben]

This effort hasn't even started yet! Assuming something works won't get us too far. We've put a lot of effort into designs for Handheld mode, in conjunction with our season of usability student, but we haven't even begun to implement anything here.

[mikus]

What I'm the most concerned about is establishing "common standards" among Activities when they are run in 'ebook configuration' -- for instance, should the topmost (depending upon the current screen rotation mode) game button be used for 'page up' ?

This can be accomplished by either publishing a specification, and asking all Activities to adhere to it. Or by actually coding a "front panel button support service" which acts whenever the screen 'rotate' button is pressed, and with which Activities need to communicate (instead of setting the front panel buttons themselves).