[PATCH weston 0/6] ivi-shell proposal

[Michael.Schuldt at bmw.de]

Hi all,

we will come up with a new solution, at clarify what needs to be standardized
and why. 

Regards

Michael.
>> Hi Michael-san,
>> 
>> As Pekka and Jason suggests in feedback, shall we clearly explain two
>> topics before stepping in details. - what ivi-shell tries to
>> standardizes - why ivi-controller tries to conrol surfaces and layers
>> in separated process from a compositor.
>> 
>> @Pekka and Jason,
>> Thank you for many feedbacks. does it make sense about two topics to
> be clarified at first?
> 
> Yes, I think that sounds like a good idea.
> --Jason Ekstrand
> 
>> 
>> Best regards,
>> Tanibata
>> 
>> 
>> 2013-09-12 20:13 に Pekka Paalanen さんは書きました:
>> 
>>> On Mon, 9 Sep 2013 08:01:51 +0000
>>> <Michael.Schuldt at bmw.de> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> let me give some technical background on the extension, and why we
>>>> have decided to design the first approach of the extensions like it
>>>> is published from Tanibata-san. In general, we would like to get the
>>>> feedback on the extensions, to get it fitting in the FOSS approach,
>>>> fulfilling the ivi requirements. See my comments in line.
>>> 
>>> 
>>> Jason Ekstrand had an excellent point asking which parts you are
>>> actually looking for to standardize.
>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I would like to get feedback about that if somebody has a similar
>>>>>> motivation to support ivi as well as desktop and tablet. So it is
>>>>>> not a stone, just a proposal. If somebody has good idea, I would
>>>>>> like to use it or collaborate it.
>>>>> 
>>>>> Ok, I just had the understanding that the layer manager simply has
>>>>> to be a separate process and not built into the compositor.
>>>> 
>>>> Not really. Currently in the compliance from genivi, we have the
>>>> separation of Graphic Backend Server and LayerManagement. But this
>>>> should now change we want to skip that approach and want define a
>>>> protocol + ivi extensions, including a reference compositor. We have
>>>> decided to use wayland + ivi extension as compositor protocol and
>>>> weston as the reference implementation for that.
>>> 
>>> 
>>> Very cool!
>>> 
>>>>> If that is not the case, then that is very good news indeed.
>>>>> Everything that manages surfaces, layers, windows, or whatever
>>>>> belongs into the compositor process, where they are much easier to
>>>>> implement and you don't need to introduce interfaces and IPC which
>>>>> are later hard to develop further and cause latencies.
>>>> 
>>>> Sure that is the technical best fitting solution for that, we
>>>> do not want to use separate IPC for the composition approach.
>>>>> So I'm roughly on the same track as Andreas Pokorny.
>>>> 
>>>> We too, but unfortunately it is not possible all the time to get
>>>> the complete user interface included in the compositor itself. We
>>>> have to guarantee fast startup time to get early video running in 2
>>>> seconds. On different systems we can not achieve that with complete
>>>> HMI-UI is up.
>>> 
>>> 
>>> Oh, no, I am not suggesting to put user interfaces in the compositor,
>>> not at all. Only the window management, which means window
>>> positioning, z-order, etc. For instance, in Weston virtual desktops
>>> are implemented in the compositor, and not by an external process
>>> juggling window lists via IPC and setting visibilities and positions.
>>> 
>>> Proposing just like with weston and desktop-shell, there is the weston
>>> plugin called desktop-shell.so which does window management, and then
>>> there is the helper program weston-desktop-shell, that does all the UI
>>> drawing in a separate process.
>>> 
>>> Like Jason Ekstrand explained, desktop-shell.so and
>>> weston-desktop-shell communicate with each other not only using the
>>> core Wayland protocol, but also using a private (or proprietary)
>>> protocol extension.
>>> 
>>> While the basic IPC mechanism is all the same (Wayland), there are
>>> different parts: extensions. The desktop-shell extension is actually a
>>> protected one, only weston-desktop-shell as started by desktop-
>>> shell.so can access it.
>>> 
>>>>> Have you tried to map your IVI concepts of surface/layer/display to
>>>>> Wayland wl_surface, wl_subsurface, and wl_output? I don't really see
>>>>> what kind of interfaces your applications (Wayland clients?) expect
>>>>> to use.
>>>>> 
>>>>> When I look at the protocol in ivi-shell.xml, I get the feeling
>>>>> that:
>>>>> - Interfaces ivi_layer, ivi_controller_surface,
>>>>>   ivi_controller_layer, ivi_controller_screen, and ivi_controller
>>>>> should be internal implementation details inside the weston process,
>>>>> not protocol. Having these as interfaces looks like the X
>>>>> architecture, where the X server process and window manager process
>>>>> continuously struggle to keep each other up-to-date, and carefully
>>>>> try to keep state in sync (and fail), which also makes races and
>>>>> glitches practically unavoidable. - Interface ivi_client is just a
>>>>> reinvention of wl_compositor and wl_subcompositor. - Interface
>>>>> ivi_surface is a reinvention of wl_surface.
>>>> 
>>>> Let me explain, why this separation occurs. From the ivi perspective
>>>> we have separated the different applications in ivi-client,
>>>> ivi-control.
>>>> 
>>>> ivi-client: Is just a client, like browser, navigation which is
>>>> embedded in the browser. That belongs to the ivi-surface. These
>>>> process have really limited access the can just create the surface
>>>> and that's all. The only important point which the want to know is if
>>>> the visibility has changed and the size to adapt the rendered
>>>> content.  A navigation should not render the content if it is not
>>>> visible or assigned to a layer.
>>> 
>>> 
>>> That is actually very well fitting to what we already have in Wayland.
>>> Just replace ivi-client with a wl_client (that's not a protocol
>>> object, it represents a display connection from a client process), and
>>> ivi-surface with wl_surface+wl_shell_surface.
>>> 
>>> The limited access is already there: in Weston, clients are free to
>>> create any number of wl_surfaces. However, no surface is shown
>>> anywhere on screen, unless it is assigned a suitable role. Examples of
>>> roles are: "pointer cursor", "top-level window", "drag icon", and
>>> "popup (menu)". Without a role, the compositor cannot know what to do
>>> with a surface.
>>> 
>>> "top-level window" and "popup menu" are shell concepts. If you replace
>>> wl_shell with your own ivi_shell, you could have roles like: "TV
>>> output", "rear camera", "speedometer". I'm totally pulling these out
>>> of my hat, I don't know what actually makes sense for you.
>>> 
>>> Then, your shell module in weston could handle such surfaces as you
>>> want. For example, if you get a surface with the role "TV output", the
>>> shell plugin could show that surface cloned on all entertainment
>>> monitors. Or whatever.
>>> 
>>> Or if something signals that now would be a /really/ good time to show
>>> the rear view camera, then your shell plugin can make it so, if there
>>> is a surface with such role assigned.
>>> 
>>> Now, something like a "rear view camera" sounds like a pretty
>>> important role for a surface, so you may want to make it privileged.
>>> You don't want any random application to claim to be a rear view
>>> camera. In that case, you can simply add whatever authentication you
>>> need to do in the shell plugin. For example, only this one process
>>> started directly by the shell plugin is allowed to claim the rear view
>>> camera role. Anyone else trying that will get killed. On the protocol
>>> design, such protection is pretty irrelevant. All you need is for the
>>> shell plugin to know who is authorized for what, and it can then
>>> enforce the restriction, whether the privileged role is requested by a
>>> method in a otherwise public protocol extension or in its own
>>> extension.
>>> 
>>>> ivi-controller : This is the main application which is controlling
>>>> the compositor like the UI. That could be the hmi, but it can also
>>>> some debug applications and application which is triggering a screen
>>>> shot of surfaces, layer, displays. If you driving a car during the
>>>> development you want to get some screenshots which are normally
>>>> triggered by an external application like the log and trace
> framework.
>>> 
>>> 
>>> Ok. I am proposing to have window management in the compositor plugin.
>>> But, you can offer additional debugging interfaces in the protocol.
>>> 
>>> For example, Weston does not allow clients to control *any* input
>>> devices. Clients simply cannot feed fake input events to other
>>> clients. But for testing and debugging, we need a way to fake input
>>> from a client. The solution: a test plugin.
>>> 
>>> The test pluging, which is loaded by the compositor only for testing
>>> (in your case debugging) purposes registers a new protocol extension:
>>> the test interface. That test interface then allows to do all kinds of
>>> ugly things from clients.
>>> 
>>> I'm not sure what you mean by "UI", but we separate controlling
>>> windows from drawing content, like I explained of the desktop shell.
>>> 
>>> As a recap, the shell plugin is what enforces and executes policies.
>>> Clients only indicate intent or purpose, like "This surface is a menu,
>>> and I want it to pop up right there on my window." A menu is an
>>> interesting example, since it is more than just a surface role: it
>>> comes with an input device grab. While the menu is up, input is
>>> directed to it. But, it is the shell plugin that can decide, that ok,
>>> you're done now, grab is broken, the menu surface hidden, and the
>>> client is signalled that something dismissed the menu. Also, clients
>>> cannot put up a menu at random times, it has to be done as a response
>>> to an input event. All this prevents many kinds of client misbehaviour.
>>> 
>>>>> Yes, I see there are some details to may want to control like
>>>>> surface opacity, that the current Wayland protocols do not support,
>>>>> but I don't think that replacing everything is a good way to start.
>>>>> It is also very hard to see how objects from all these interfaces
>>>>> are created, and how (if?) they associate to any other protocol
>>>>> objects.
>>>> 
>>>> On goal is to get setup a complete scenery during the startup of the
>>>> compositor. Therefore the compositor is able to just initialize the
>>>> main visible attribute during startup. That implies too that it
>>>> should be possible, without the final application is running (like
>>>> navigation). Therefore we need some logical objects like ivi-
>>>> surface, ivi-layer. Which can be already controlled, whithout a
>>>> application which holds the content is already running.
>>> 
>>> 
>>> But if all layer and window management is internal to the compositor
>>> process, then you don't need any of this control protocol.
>>> 
>>> Just create some internal data structures, and when a client comes up
>>> with a surface and requests a particular role, you can plug that
>>> surface into the slot you prepared in advance.
>>> 
>>> So, yes, this does imply that the compositor or the shell plugin knows
>>> the scenegraph, but... isn't that exactly the compositor's purpose, to
>>> gather the scenegraph and get it on screen? Clients "only" provide
>>> content.
>>> 
>>> For example, the desktop-shell protocol extension has special surface
>>> roles "background" and "panel". When weston-desktop-shell client
>>> assigns these roles to its surfaces, the compositor (shell plugin)
>>> will immediately plug them into the correct places on screen. If the
>>> shell plugin wanted to reserve screen space for them in advance, it
>>> could. After all, the shell plugin is responsible for arranging
>>> everything on (and off) screen.
>>> 
>>>>> Btw. if you need support for surface scaling and cropping, there
>>>>> have been discussion on the Wayland mailing list to bring a crop &
>>>>> scale protocol extension to Wayland. It is actually necessary for
>>>>> efficient video playback etc., so pushing that forward would be
>>>>> nice.
>>>> 
>>>> Yeah I see that, and we will consider it. I think we can redesign the
>>>> approach like source and destination region to the crop and scale
>>>> instead.
>>>> 
>>>>> After looking through the two links you gave, the ivi-shell.xml, and
>>>>> what you have wrote in the emails, I still have no clue what is the
>>>>> big picture here.
>>>>> 
>>>>> - What processes are going to use which interfaces? It looks to me
>>>>>   like some interfaces are not meant for all Wayland clients, but
>>>>>   how is it supposed to work?
>>>>> - What components are in a whole IVI system, from the point of
> view
>>>>>   of Wayland protocol? What are the responsibilities of each
>>>>>   component and how are these distributed into processes?
>>>> 
>>>> ivi-control : Is the only process which is allowed to control the
>>>> compositor itself. But they need some informations about other
>>>> applications which are running. In the ivi-domain, each graphical
>>>> application has a unique ID which is defined by the OEM or platform
>>>> provider. ivi-client :  are just other applications like browser or
>>>> navigation.
>>> 
>>> 
>>> Ok, so you probably need a protocol interface to tell the compositor
>>> the unique ID, and perhaps an authentication/authorization interface,
>>> so that a shell plugin has all the needed information to do the
>>> controlling.
>>> 
>>>>> - What does a typical IVI application do in terms of Wayland
>>>>>   protocol? Are you using wl_compositor at all? Or any other
>>>>>   Wayland core interfaces?
>>>> 
>>>> We were focused on just using the wayland core interface. But we are
>>>> open to extend this approach.
>>> 
>>> 
>>> For the record, "Wayland core" is everything in
>>> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
>>> except wl_shell and wl_shell_surface, which are specific to desktop
>>> applications. But you are free to pick and choose individual global
>>> interfaces.
>>> 
>>>>> The protocol you propose seems to have many references to
>>>>> "id_native" and "native content", what is all this "native" stuff
>>>>> about? Or all the integer id's you seem to be sending back and
>>>>> forth, why can't you use real protocol objects to refer to those?
>>>> 
>>>> This is a little bit confusing, the initial set of these objects like
>>>> it is defined here :
>>>> 
>>>> 
> http://git.projects.genivi.org/?p=layer_management.git;a=blob;f=wayland-
> ivi-extension/protocol/ivi-
> shell.xml;h=6092b13ef31e55f3db338be2e471d0e956be5b9e;hb=ivi-extension
>>>> 
>>>> Is focused on using the ivi-surface instead of the native id. The
>>>> rational behind that is to define unique ids over the complete
>>>> process space. Therefore the HMI is able to control the different
>>>> ivi-surface properties.
>>> 
>>> 
>>> Ok, you are inventing global names for these protocol objects, so you
>>> could do "window management" in a separate process, right? If you made
>>> the HMI controller a compositor plugin, these global names would not
>>> be needed, perhaps?
>>> 
>>> Perhaps I still do not fully grasp what the HMI controller is. And I
>>> have more of your emails to read still.
>>> 
>>> You may have a very good reason to keep the HMI controller separate,
>>> but the thing I am questioning is, is the interface towards the
>>> compositor the right one. Should the controller really be responsible
>>> for window management, or should it just send simple messages like
>>> "rear view camera on/off" or even "gear: reverse".
>>> 
>>> Even if the HMI controller was one big block in your architecture
>>> diagrams, you could split it into a compositor plugin and an external
>>> process. Then the private Wayland protocol extension they use to
>>> communicate with each other will be just an implementation detail of
>>> the whole block, which you might not need to standardize.
>>> 
>>> There actually exists examples of such a design: most of the
>>> Wayland-enabled EGL implementations use a private protocol extension
>>> to communicate between the server and the client. The trick is, both
>>> the server and the clients load the same libEGL, so the protocol
>>> becomes a completely hidden implementation detail.
>>> 
>>>> In the end we are very lucky about any input to get a final best
>>>> fitting solution for both the ivi aspects and the final
>>>> implementation for that in the FOSS domain.
>>> 
>>> 
>>> I am very glad you came forward. :-)
>>> 
>>> 
>>> Thanks,
>>> pq
>> 
>> _______________________________________________
>> wayland-devel mailing list
>> wayland-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel
>
-------------------------------------------------------
BMW Group
Michael Schuldt
Plattformtechnol. Infotainment-Kompon.
Max-Diamand-Straße 13
80788 München

Mailto: michael.schuldt at bmw.de
URL: http://www.bmwgroup.com 
-------------------------------------------------------
Bayerische Motoren Werke Aktiengesellschaft
Vorstand: Norbert Reithofer, Vorsitzender,
Milagros Caiña Carreiro-Andree, Herbert Diess,
Klaus Draeger, Friedrich Eichiner, Harald Krüger,
Ian Robertson, Peter Schwarzenbauer.
Vorsitzender des Aufsichtsrats: Joachim Milberg
Sitz und Registergericht: München HRB 42243
--------------------------------------------------------