Questions and thoughts about input method protocol

[Yichao Yu]

On Thu, Jan 31, 2013 at 5:19 PM, Pekka Vuorela <pvuorela at iki.fi> wrote:
> On 31.01.2013 02:54, Yichao Yu wrote:
>>
>> Hi,
>>
>> On Wed, Jan 30, 2013 at 4:09 PM, Pekka Vuorela <pvuorela at iki.fi> wrote:
>>>
>>> On 29.01.2013 04:49, Yichao Yu wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> With the comment on the recent patches for the input method protocol,
>>>> it seems that we are finally on the way to support cursor following.
>>>> However I still have some questions about the protocol (and it's
>>>> limitation).
>>>>
>>>> 1, Is there any plan to support xwayland?
>
>
>>>
>>> X based input methods are using X windowing, so wouldn't they just
>>> continue
>>> working for X apps? (Provided that xwayland has the ability to put
>>> windows
>>> into absolute positions). Translation between wayland text <-> XIM
>>> doesn't
>>> seem worthwhile to me.
>>>
>>
>> I c. So it should still be possible for a X window based input method
>> to work on xwayland. It's good to know that xwayland is able to put a
>> window (which means x window right?) at a absolute position (and will
>> also be able to know the size of screens?). However, in order to use
>> such functionality, I guess the input method (that support wayland)
>> would still have to draw the ui for x clients using X instead of
>> (native) wayland making it harder to provide a uniform look for all
>> clients. This applies not only to the input overlay window but also to
>> the on screen keyboard since the compositor may have zero knowledge of
>> how to keep the virtual keyboard away from the text cursor. Moreover,
>> it will require lots of additional effort if the ui is provided by a
>> different process or a completely different program [1][2].
>>
>> [1] http://kde-apps.org/content/show.php?content=140967
>> [2] http://userbase.kde.org/Tutorials/Kimpanel
>
>
> I haven't been following xwayland discussion too much, but I recall there
> was some talk about allowing xwayland to use absolute positions. Normally
> under Wayland that is not possible.
>
> What I had in mind was wayland text input serving wayland apps, and X apps
> using what X input method they have configured. Yes, uniformity is not
> promised, but desktop setup can make such available hopefully. For virtual
> keyboard I don't think it should be worried too much, normally such is used
> in more integrated solution being fully X or fully Wayland.
>

That will absolutely make it harder to have a input method working for
both wayland and X at the same time....

>
>>>> 2, Is it possible for the input method to know anything about the
>>>> client?
>>>>
>>> Not wanting to ridicule chinese input needs, but fetching window title,
>>> executable name and/or application parameters to alter behavior is a
>>> _huge_
>>> hack. Going deeper into knowing what url is visible in a browser is even
>>> worse. Does not take much effort to break that kind of functionality.
>>
>>
>> Well, technically I don't really think that it is more hack than a
>> input method based on language model making use of surrounding text,
>> previous input results as well as results from the internet when
>> necessary. In fact, I fully agree that each good Chinese input method
>> is, by all mean, a __HUUUGE__ hack. It's just the nature of the
>> language and taking the window title into account is nothing more
>> special than any existing hacks. It may seem hacky if you think the
>> window title as a text string far away from the text cursor. However,
>> the window title is supposed to be and in most cases (including the
>> most important case: browsers) really is a short and perfect summary
>> of what's going on inside the window (as well as the text field where
>> the input is happening). As long as the program still set the window
>> title to a reasonable value, this functionality will not be broken.
>>
>> And whether or not you think using window titles or any kinds of free
>> text context is a hack, providing per-program input state using pid of
>> the client shouldn't be considered a hack at all.
>>
>> By mentioning amazon and arxiv, I don't really mean knowing what url
>> is visible (although it would also be helpful if it is indeed
>> available) but still determine the context from the title of the
>> window (in this case, the title of the webpage).
>
>
> No, these are in completely different categories. I don't consider
> surrounding text or composing chinese with pinyin really hacks to begin
> with.
>
> Problem with window titles and all is that they are information not meant to
> input method. Those are meant for the user and can change any time and to
> anything. And using those will require application specific logic in the
> input method, like specific rules for specific application.
> Don't see that feasible except for very special cases.

They will never change to random things and the only necessary thing
to do is keyword matching. And it will able to work for most cases
unless you have a window title like "window 1".

>
> If input methods are able to get such pieces of information somewhere, fine,
> their problem, but I'm quite opposed on explicitly adding that kind of stuff
> to input method protocols. More on this below.
>
>
>
>>>
>>> I would try to find better ways to achieve targeted goals. One feature
>>> I've
>>> myself been thinking is virtual keyboard used with messaging app knowing
>>> what language, or type of language, the recipient is speaking and
>>> adjusting
>>> prediction/layout based on that. That could be done by having locale in
>>> text
>>> editor's state, but alternatively could also be editor setting a globally
>>> unique identifier (integer, string?) as state for text fields, after
>>> which
>>> text input side could learn what language got used. Similarly for chinese
>>> input such mechanism could be used to remember preferences. Out-of-the
>>> box
>>> amazon and arxiv wouldn't make difference, but after some use they would.
>>> To
>>> support this, the toolkits and applications would also need some
>>> enhancements, though.
>>
>>
>> First of all, I think there is already perfect solution to your
>> problem. Agree or not, keyboard layout (in this case) is just a set of
>> specialized and simplified input methods. As you can see, the virtual
>> keyboard in walyand is done in input methods (in fact, is there
>> another way in wayland for a program to send key events to another??),
>> fcitx supports keyboard layout as input methods around one year ago[3]
>> and is built-in in the release arround May. 2012 and ibus 1.5 also
>> ships with a broken keyboard layout setting functionality. Therefore,
>> the information you want to know is already known by the virtual
>> keyboard provider (or from the current keyboard layout setting) and
>> there is no need to query anything from the client.
>>
>> [3] https://www.csslayer.info/wordpress/fcitx-dev/fcitx-keyboard/
>
>
> No. I wasn't talking about user's ability to change keyboard layouts. Doing
> that is trivial and requires nothing from the application or input method
> protocol.
>
> I was talking about input method _automatically_ changing to one layout or
> another based on what was used in a specific text editor previously.
>
> In more detail: I have an application MyChat and I activate there a
> conversation text field with you. Wayland input method gets activation and a
> call like set_context("MyChat-Yichao_Yu"), it remembers that I use to speak
> Chinese with you so it enables pinyin automatically.
>
> Or with a browser, set_context("MyBrowser-www.example.com"), input method
> changes prediction to prioritize words I normally use at www.example.com.
> That's the same information as in window title, but not using concepts from
> the wrong level. Maybe even the toolkit integration could try to help and
> set context as default to executable name or something if application
> doesn't override it.

That is not the solution and that is far from what a working Chinese
input method need in another way. The right way is to let the input
method track every input_method_context and store any state
internally, as how it is done in any existing input method system.

>
>
>
>
>> One major difference between the language code and input context is
>> that language codes are finite and standardized, but input context is
>> not. A reasonable use case of the context (window title) is relate it
>> with some areas (e.g. shopping and science for the example I
>> mentioned), and load/use/set different weight on different
>> dictionaries accordingly. This is really just some dirty works that
>> needs to be frequently updated, impossible to standardized, and is
>> definitely not sth you would want to put into every programs/toolkits,
>> therefore, the client would generally have no idea (especially for a
>> web browser which the context varies a lot) how to process the context
>> (title, url etc.) except passing them directly to the input method.
>>
>> In order to support such functionality, inspired by csslayer's
>> suggestion[4], I think it will be helpful to add sth like a
>> input_context::context_info(key, value) event which will notify the
>> input method about some information (and their changes) of the client.
>> Events with key="window_title", "pid", "program_name" can be
>> automatically sent by the compositor so clients that don't want to
>> provide additional information has nothing to worry about and those
>> that want to support input method can just send out additional
>> key-value pairs on the corresponding client side interface. If the
>> compositor ever have some more knowledge about the client that the
>> input method may be interested in, it can also be easily added here.
>
>
> Extendability is definitely something this protocol needs and is possible by
> separate protocols or newer methods later. Key, value pair context
> information might make sense as an extension mechanism too.
>

And the window title as well as pid are just the most helpful thing
the application can help the input method even without any support
from the application itself.

>
>
>> [4]
>> http://blog.jpetersen.org/2012/06/20/text-input-method-support-in-wayland/#comment-1327
>>
>> P.S. As for requiring non-out-of-box input method support, it is just
>> not practical in the Linux world. Most people out of the CJK[5]
>> community never ever care about input methods and most of them even
>> don't know what a input method is (Oh, yes sure, you guys are perfect
>> exceptions). If a feature that the developer don't care about (or even
>> worse, have no idea what that is) require additional effort to
>> implement, it will never be done. You can have a look at how XIM is
>> broken in many applications[6], how it haven't been improved over
>> almost 20 years[7], how bugs cannot get fixed or have any developers
>> ever looking at it for more than 10 years (REMAIN NEW!! WITH WORKING
>> PATCH!!)[8]. We really hope the input method protocol in wayland can
>> finally provide us a working input method, not having to worry about
>> application freezing, missing key events or other weird problems[9],
>> not having to maintain input method modules for 4 toolkits[10][11] and
>> without any regression or missing functionality.
>
>
> Yeah I feel you. Luckily there's now a chance to do input method protocol
> that could be used with wayland for all toolkits and even avoiding problems
> built into XIM before the protocol is set in stone. I'm hoping some CJK
> input method could be ported to use the weston text input protocol.

And that's why we CJK developers want to have no regression and being
able to have at least the same functionality from the weston im
protocol.

>
>
>> P.P.S
>> Any ideas for my questions 3,4,5? =)
>
>
> For windowing, not much at the moment.

:P

>
>