Keyboard issues w/Orca screen reader

[Carlos Garnacho]

Hi Mike!,

On Wed, Feb 24, 2021 at 11:03 PM Mike Gorse <mgorse at suse.com> wrote:
>
> As part of the GNOME project, we have Orca. It is a screen reader that reads
> what an application is displaying out loud, or sends it to a Braille display.
> It allows a blind user to interact with GNOME. There are currently several
> issues with Orca on Wayland, since it (and AT-SPI, the accessibility API that
> it uses) were originally designed for X11, without regard to the kind of
> security constraints that Wayland has.
>
> Traditionally, Orca has used what amounts to a key filter implemented by
> AT-SPI. The toolkit (gtk, qt, etc) would make a dbus method call to AT-SPI when
> a key is pressed, passing on the keycode, keysym, and text associated with the
> key (Orca uses the latter to announce the key that was just pressed). The
> AT-SPI registry daemon would make a method call to Orca, Orca would return a
> boolean to indicate whether it consumed the key or not, and AT-SPI would return
> the result to the toolkit. This is no longer supported by gtk 4, so I am trying
> to solve things another way. There is a good amount of discussion at
> https://gitlab.gnome.org/GNOME/gtk/-/issues/1739
>
> Orca uses this key filter for a few things. It wants to announce the key that
> the user just pressed. If the user presses the control key, then Orca will stop
> reading whatever it is reading. It also implements commands in this way (the
> user can press a key to have it read the current line, to give one example).

I may need to be reminded again... are there other commands that are
contextual to the application? I mainly wonder about whether there's a
reason to keep applications in the event emission chain. Otherwise,
seems like part of this could be covered by a "description query"
protocol.

> For X11, I can have AT-SPI/Orca watch for XI_KeyPress events and call
> XIGrabKeycode when Orca wants to grab a key, but I'm struggling to figure out
> what to do on Wayland. I think that I would need to propose a new protocol or
> two, but I foresee security being a challenge/concern, so I'm wondering if
> anyone has any advice in particular. Xwayland-keyboard-grab looks like it would
> do part of what I need, if Orca ran as an xwayland client, but it wouldn't help
> in terms of passing key press notifications on to Orca for the purposes of key
> echoing and interrupting speech, unless a grab was active for the particular
> key. If I enable my X11 code for Wayland, then I don't see XI_KeyPress events
> for keys going to other applications, which I think is expected given Wayland's
> security model.

IIUC (and I'm sure I miss/forget things), there's 2 needs expressed here:
1. To make Orca part of the chain of command
  - Getting notified of all key events
  - Handle global keyboard shortcuts
    - And make others miss them (i.e. in a grabbing manner)
2. To make clients work seamlessly with Orca

To fix 1, I tend to think this is something best left to an
integration library (as IIUC the goal is also to maximize
reutilization). The API sounds like it could be reasonably agnostic of
compositor details.

Fixing 2 sounds most definitely protocol land, which I wonder if it
could be more semantic and less attached to input specifics.

Cheers,
  Carlos