Questions and thoughts about input method protocol

[Pekka Vuorela]

On 31.01.2013 02:54, Yichao Yu wrote:
> Hi,
>
> On Wed, Jan 30, 2013 at 4:09 PM, Pekka Vuorela <pvuorela at iki.fi> wrote:
>> On 29.01.2013 04:49, Yichao Yu wrote:
>>>
>>> Hi,
>>>
>>> With the comment on the recent patches for the input method protocol,
>>> it seems that we are finally on the way to support cursor following.
>>> However I still have some questions about the protocol (and it's
>>> limitation).
>>>
>>> 1, Is there any plan to support xwayland?

>>
>> X based input methods are using X windowing, so wouldn't they just continue
>> working for X apps? (Provided that xwayland has the ability to put windows
>> into absolute positions). Translation between wayland text <-> XIM doesn't
>> seem worthwhile to me.
>>
>
> I c. So it should still be possible for a X window based input method
> to work on xwayland. It's good to know that xwayland is able to put a
> window (which means x window right?) at a absolute position (and will
> also be able to know the size of screens?). However, in order to use
> such functionality, I guess the input method (that support wayland)
> would still have to draw the ui for x clients using X instead of
> (native) wayland making it harder to provide a uniform look for all
> clients. This applies not only to the input overlay window but also to
> the on screen keyboard since the compositor may have zero knowledge of
> how to keep the virtual keyboard away from the text cursor. Moreover,
> it will require lots of additional effort if the ui is provided by a
> different process or a completely different program [1][2].
>
> [1] http://kde-apps.org/content/show.php?content=140967
> [2] http://userbase.kde.org/Tutorials/Kimpanel

I haven't been following xwayland discussion too much, but I recall 
there was some talk about allowing xwayland to use absolute positions. 
Normally under Wayland that is not possible.

What I had in mind was wayland text input serving wayland apps, and X 
apps using what X input method they have configured. Yes, uniformity is 
not promised, but desktop setup can make such available hopefully. For 
virtual keyboard I don't think it should be worried too much, normally 
such is used in more integrated solution being fully X or fully Wayland.


>>> 2, Is it possible for the input method to know anything about the client?
>>>
>> Not wanting to ridicule chinese input needs, but fetching window title,
>> executable name and/or application parameters to alter behavior is a _huge_
>> hack. Going deeper into knowing what url is visible in a browser is even
>> worse. Does not take much effort to break that kind of functionality.
>
> Well, technically I don't really think that it is more hack than a
> input method based on language model making use of surrounding text,
> previous input results as well as results from the internet when
> necessary. In fact, I fully agree that each good Chinese input method
> is, by all mean, a __HUUUGE__ hack. It's just the nature of the
> language and taking the window title into account is nothing more
> special than any existing hacks. It may seem hacky if you think the
> window title as a text string far away from the text cursor. However,
> the window title is supposed to be and in most cases (including the
> most important case: browsers) really is a short and perfect summary
> of what's going on inside the window (as well as the text field where
> the input is happening). As long as the program still set the window
> title to a reasonable value, this functionality will not be broken.
>
> And whether or not you think using window titles or any kinds of free
> text context is a hack, providing per-program input state using pid of
> the client shouldn't be considered a hack at all.
>
> By mentioning amazon and arxiv, I don't really mean knowing what url
> is visible (although it would also be helpful if it is indeed
> available) but still determine the context from the title of the
> window (in this case, the title of the webpage).

No, these are in completely different categories. I don't consider 
surrounding text or composing chinese with pinyin really hacks to begin 
with.

Problem with window titles and all is that they are information not 
meant to input method. Those are meant for the user and can change any 
time and to anything. And using those will require application specific 
logic in the input method, like specific rules for specific application.
Don't see that feasible except for very special cases.

If input methods are able to get such pieces of information somewhere, 
fine, their problem, but I'm quite opposed on explicitly adding that 
kind of stuff to input method protocols. More on this below.


>>
>> I would try to find better ways to achieve targeted goals. One feature I've
>> myself been thinking is virtual keyboard used with messaging app knowing
>> what language, or type of language, the recipient is speaking and adjusting
>> prediction/layout based on that. That could be done by having locale in text
>> editor's state, but alternatively could also be editor setting a globally
>> unique identifier (integer, string?) as state for text fields, after which
>> text input side could learn what language got used. Similarly for chinese
>> input such mechanism could be used to remember preferences. Out-of-the box
>> amazon and arxiv wouldn't make difference, but after some use they would. To
>> support this, the toolkits and applications would also need some
>> enhancements, though.
>
> First of all, I think there is already perfect solution to your
> problem. Agree or not, keyboard layout (in this case) is just a set of
> specialized and simplified input methods. As you can see, the virtual
> keyboard in walyand is done in input methods (in fact, is there
> another way in wayland for a program to send key events to another??),
> fcitx supports keyboard layout as input methods around one year ago[3]
> and is built-in in the release arround May. 2012 and ibus 1.5 also
> ships with a broken keyboard layout setting functionality. Therefore,
> the information you want to know is already known by the virtual
> keyboard provider (or from the current keyboard layout setting) and
> there is no need to query anything from the client.
>
> [3] https://www.csslayer.info/wordpress/fcitx-dev/fcitx-keyboard/

No. I wasn't talking about user's ability to change keyboard layouts. 
Doing that is trivial and requires nothing from the application or input 
method protocol.

I was talking about input method _automatically_ changing to one layout 
or another based on what was used in a specific text editor previously.

In more detail: I have an application MyChat and I activate there a 
conversation text field with you. Wayland input method gets activation 
and a call like set_context("MyChat-Yichao_Yu"), it remembers that I use 
to speak Chinese with you so it enables pinyin automatically.

Or with a browser, set_context("MyBrowser-www.example.com"), input 
method changes prediction to prioritize words I normally use at 
www.example.com. That's the same information as in window title, but not 
using concepts from the wrong level. Maybe even the toolkit integration 
could try to help and set context as default to executable name or 
something if application doesn't override it.



> One major difference between the language code and input context is
> that language codes are finite and standardized, but input context is
> not. A reasonable use case of the context (window title) is relate it
> with some areas (e.g. shopping and science for the example I
> mentioned), and load/use/set different weight on different
> dictionaries accordingly. This is really just some dirty works that
> needs to be frequently updated, impossible to standardized, and is
> definitely not sth you would want to put into every programs/toolkits,
> therefore, the client would generally have no idea (especially for a
> web browser which the context varies a lot) how to process the context
> (title, url etc.) except passing them directly to the input method.
>
> In order to support such functionality, inspired by csslayer's
> suggestion[4], I think it will be helpful to add sth like a
> input_context::context_info(key, value) event which will notify the
> input method about some information (and their changes) of the client.
> Events with key="window_title", "pid", "program_name" can be
> automatically sent by the compositor so clients that don't want to
> provide additional information has nothing to worry about and those
> that want to support input method can just send out additional
> key-value pairs on the corresponding client side interface. If the
> compositor ever have some more knowledge about the client that the
> input method may be interested in, it can also be easily added here.

Extendability is definitely something this protocol needs and is 
possible by separate protocols or newer methods later. Key, value pair 
context information might make sense as an extension mechanism too.


> [4] http://blog.jpetersen.org/2012/06/20/text-input-method-support-in-wayland/#comment-1327
>
> P.S. As for requiring non-out-of-box input method support, it is just
> not practical in the Linux world. Most people out of the CJK[5]
> community never ever care about input methods and most of them even
> don't know what a input method is (Oh, yes sure, you guys are perfect
> exceptions). If a feature that the developer don't care about (or even
> worse, have no idea what that is) require additional effort to
> implement, it will never be done. You can have a look at how XIM is
> broken in many applications[6], how it haven't been improved over
> almost 20 years[7], how bugs cannot get fixed or have any developers
> ever looking at it for more than 10 years (REMAIN NEW!! WITH WORKING
> PATCH!!)[8]. We really hope the input method protocol in wayland can
> finally provide us a working input method, not having to worry about
> application freezing, missing key events or other weird problems[9],
> not having to maintain input method modules for 4 toolkits[10][11] and
> without any regression or missing functionality.

Yeah I feel you. Luckily there's now a chance to do input method 
protocol that could be used with wayland for all toolkits and even 
avoiding problems built into XIM before the protocol is set in stone. 
I'm hoping some CJK input method could be ported to use the weston text 
input protocol.

> P.P.S
> Any ideas for my questions 3,4,5? =)

For windowing, not much at the moment.