Keyboard issues w/Orca screen reader

[Carsten Haitzler]

On Wed, 24 Feb 2021 15:59:59 -0600 (CST) Mike Gorse <mgorse at suse.com> said:

> As part of the GNOME project, we have Orca. It is a screen reader that reads 
> what an application is displaying out loud, or sends it to a Braille display. 
> It allows a blind user to interact with GNOME. There are currently several 
> issues with Orca on Wayland, since it (and AT-SPI, the accessibility API that 
> it uses) were originally designed for X11, without regard to the kind of 
> security constraints that Wayland has.
> 
> Traditionally, Orca has used what amounts to a key filter implemented by 
> AT-SPI. The toolkit (gtk, qt, etc) would make a dbus method call to AT-SPI
> when a key is pressed, passing on the keycode, keysym, and text associated
> with the key (Orca uses the latter to announce the key that was just
> pressed). The AT-SPI registry daemon would make a method call to Orca, Orca
> would return a boolean to indicate whether it consumed the key or not, and
> AT-SPI would return the result to the toolkit. This is no longer supported by
> gtk 4, so I am trying to solve things another way. There is a good amount of
> discussion at https://gitlab.gnome.org/GNOME/gtk/-/issues/1739

In this case, your issue would be with gtk. It should still support this. In a
wayland world, security-wise this would hand the decision to talk to orca via
at-spi and if that app wants to leak input somewhere else - it is that app's
decision. It can leak only a subset of keys too and selectively pass or don't
pass depending on the situation (e.g. if entering text in a password field.
the client can choose to not pass the key input to at-spi and instead maybe
just pass some dummy key much like password entries display *** instead of the
letters). The compositor does not need to be involved. This would IMHO be
certainly the best way to do this.

> Orca uses this key filter for a few things. It wants to announce the key that 
> the user just pressed. If the user presses the control key, then Orca will
> stop reading whatever it is reading. It also implements commands in this way
> (the user can press a key to have it read the current line, to give one
> example). For X11, I can have AT-SPI/Orca watch for XI_KeyPress events and
> call XIGrabKeycode when Orca wants to grab a key, but I'm struggling to
> figure out what to do on Wayland. I think that I would need to propose a new
> protocol or two, but I foresee security being a challenge/concern, so I'm
> wondering if anyone has any advice in particular. Xwayland-keyboard-grab
> looks like it would do part of what I need, if Orca ran as an xwayland
> client, but it wouldn't help in terms of passing key press notifications on
> to Orca for the purposes of key echoing and interrupting speech, unless a
> grab was active for the particular key. If I enable my X11 code for Wayland,
> then I don't see XI_KeyPress events for keys going to other applications,
> which I think is expected given Wayland's security model.

As above - the client should do this. Keep in mind that I also do not exclude
the compositor itself from being an AT-SPI client. If the compositor wishes to
pass its own keybindings/shortcuts selectively etc.. It may be a compositor has
it's own "blind" mode where it may actually directly output some information to
both speakers and a braille keyboard to send feedback/information to the user
and thus not need orca or AT-SPI.

> Thanks,
> -Mike
> 
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/wayland-devel
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - raster at rasterman.com