[PATCH weston 0/6] ivi-shell proposal

Fri Sep 13 06:33:56 PDT 2013

Hi Michael-san,

As Pekka and Jason suggests in feedback, shall we clearly explain two 
topics before stepping in details.
- what ivi-shell tries to standardizes
- why ivi-controller tries to conrol surfaces and layers in separated 
process from a compositor.

@Pekka and Jason,
Thank you for many feedbacks. does it make sense about two topics to be 
clarified at first?

Best regards,
Tanibata

2013-09-12 20:13 に Pekka Paalanen さんは書きました:
> On Mon, 9 Sep 2013 08:01:51 +0000
> <Michael.Schuldt at bmw.de> wrote:
> 
>> Hi all,
>> 
>> let me give some technical background on the extension, and why we
>> have decided to design the first approach of the extensions like it
>> is published from Tanibata-san. In general, we would like to get the
>> feedback on the extensions, to get it fitting in the FOSS approach,
>> fulfilling the ivi requirements. See my comments in line.
> 
> Jason Ekstrand had an excellent point asking which parts you are
> actually looking for to standardize.
> 
>> >> Hi,
>> >>
>> >> I would like to get feedback about that if somebody has a similar
>> >> motivation to support ivi as well as desktop and tablet. So it is
>> >> not a stone, just a proposal. If somebody has good idea, I would
>> >> like to use it or collaborate it.
>> >
>> > Ok, I just had the understanding that the layer manager simply has
>> > to be a separate process and not built into the compositor.
>> 
>> Not really. Currently in the compliance from genivi, we have the
>> separation of Graphic Backend Server and LayerManagement. But this
>> should now change we want to skip that approach and want define a
>> protocol + ivi extensions, including a reference compositor. We have
>> decided to use wayland + ivi extension as compositor protocol and
>> weston as the reference implementation for that.
> 
> Very cool!
> 
>> > If that is
>> > not the case, then that is very good news indeed. Everything that
>> > manages surfaces, layers, windows, or whatever belongs into the
>> > compositor process, where they are much easier to implement and you
>> > don't need to introduce interfaces and IPC which are later hard to
>> > develop further and cause latencies.
>> 
>> Sure that is the technical best fitting solution for that, we
>> do not want to use separate IPC for the composition approach.
>> > So I'm roughly on the same track as Andreas Pokorny.
>> 
>> We too, but unfortunately it is not possible all the time to get
>> the complete user interface included in the compositor itself. We
>> have to guarantee fast startup time to get early video running in 2
>> seconds. On different systems we can not achieve that with complete
>> HMI-UI is up.
> 
> Oh, no, I am not suggesting to put user interfaces in the compositor,
> not at all. Only the window management, which means window positioning,
> z-order, etc. For instance, in Weston virtual desktops are implemented
> in the compositor, and not by an external process juggling window lists
> via IPC and setting visibilities and positions.
> 
> Proposing just like with weston and desktop-shell, there is the weston
> plugin called desktop-shell.so which does window management, and then
> there is the helper program weston-desktop-shell, that does all the UI
> drawing in a separate process.
> 
> Like Jason Ekstrand explained, desktop-shell.so and
> weston-desktop-shell communicate with each other not only using the
> core Wayland protocol, but also using a private (or proprietary)
> protocol extension.
> 
> While the basic IPC mechanism is all the same (Wayland), there are
> different parts: extensions. The desktop-shell extension is actually a
> protected one, only weston-desktop-shell as started by desktop-shell.so
> can access it.
> 
>> > Have you tried to map your IVI concepts of surface/layer/display to
>> > Wayland wl_surface, wl_subsurface, and wl_output? I don't really
>> > see what kind of interfaces your applications (Wayland clients?)
>> > expect to use.
>> >
>> > When I look at the protocol in ivi-shell.xml, I get the feeling
>> > that:
>> > - Interfaces ivi_layer, ivi_controller_surface,
>> >   ivi_controller_layer, ivi_controller_screen, and ivi_controller
>> > should be internal implementation details inside the weston
>> > process, not protocol. Having these as interfaces looks like the X
>> > architecture, where the X server process and window manager process
>> > continuously struggle to keep each other up-to-date, and carefully
>> > try to keep state in sync (and fail), which also makes races and
>> > glitches practically unavoidable. - Interface ivi_client is just a
>> > reinvention of wl_compositor and wl_subcompositor.
>> > - Interface ivi_surface is a reinvention of wl_surface.
>> 
>> Let me explain, why this separation occurs.
>> From the ivi perspective we have separated the different applications
>> in ivi-client, ivi-control.
>> 
>> ivi-client: Is just a client, like browser, navigation which is
>> embedded in the browser. That belongs to the ivi-surface. These
>> process have really limited access the can just create the surface
>> and that's all. The only important point which the want to know is if
>> the visibility has changed and the size to adapt the rendered
>> content.  A navigation should not render the content if it is not
>> visible or assigned to a layer.
> 
> That is actually very well fitting to what we already have in Wayland.
> Just replace ivi-client with a wl_client (that's not a protocol
> object, it represents a display connection from a client process), and
> ivi-surface with wl_surface+wl_shell_surface.
> 
> The limited access is already there: in Weston, clients are free to
> create any number of wl_surfaces. However, no surface is shown
> anywhere on screen, unless it is assigned a suitable role. Examples of
> roles are: "pointer cursor", "top-level window", "drag icon", and
> "popup (menu)". Without a role, the compositor cannot know what to do
> with a surface.
> 
> "top-level window" and "popup menu" are shell concepts. If you replace
> wl_shell with your own ivi_shell, you could have roles like: "TV
> output", "rear camera", "speedometer". I'm totally pulling these out of
> my hat, I don't know what actually makes sense for you.
> 
> Then, your shell module in weston could handle such surfaces as you
> want. For example, if you get a surface with the role "TV output", the
> shell plugin could show that surface cloned on all entertainment
> monitors. Or whatever.
> 
> Or if something signals that now would be a /really/ good time to show
> the rear view camera, then your shell plugin can make it so, if there
> is a surface with such role assigned.
> 
> Now, something like a "rear view camera" sounds like a pretty important
> role for a surface, so you may want to make it privileged. You don't
> want any random application to claim to be a rear view camera. In that
> case, you can simply add whatever authentication you need to do in the
> shell plugin. For example, only this one process started directly by
> the shell plugin is allowed to claim the rear view camera role. Anyone
> else trying that will get killed. On the protocol design, such
> protection is pretty irrelevant. All you need is for the shell plugin
> to know who is authorized for what, and it can then enforce the
> restriction, whether the privileged role is requested by a method in a
> otherwise public protocol extension or in its own extension.
> 
>> ivi-controller : This is the main application which is controlling
>> the compositor like the UI. That could be the hmi, but it can also
>> some debug applications and application which is triggering a screen
>> shot of surfaces, layer, displays. If you driving a car during the
>> development you want to get some screenshots which are normally
>> triggered by an external application like the log and trace framework.
> 
> Ok. I am proposing to have window management in the compositor plugin.
> But, you can offer additional debugging interfaces in the protocol.
> 
> For example, Weston does not allow clients to control *any* input
> devices. Clients simply cannot feed fake input events to other clients.
> But for testing and debugging, we need a way to fake input from a
> client. The solution: a test plugin.
> 
> The test pluging, which is loaded by the compositor only for testing
> (in your case debugging) purposes registers a new protocol extension:
> the test interface. That test interface then allows to do all kinds of
> ugly things from clients.
> 
> I'm not sure what you mean by "UI", but we separate controlling windows
> from drawing content, like I explained of the desktop shell.
> 
> As a recap, the shell plugin is what enforces and executes policies.
> Clients only indicate intent or purpose, like "This surface is a menu,
> and I want it to pop up right there on my window." A menu is an
> interesting example, since it is more than just a surface role: it
> comes with an input device grab. While the menu is up, input is
> directed to it. But, it is the shell plugin that can decide, that ok,
> you're done now, grab is broken, the menu surface hidden, and the
> client is signalled that something dismissed the menu. Also, clients
> cannot put up a menu at random times, it has to be done as a response
> to an input event. All this prevents many kinds of client misbehaviour.
> 
>> > Yes, I see there are some details to may want to control like
>> > surface opacity, that the current Wayland protocols do not support,
>> > but I don't think that replacing everything is a good way to start.
>> > It is also very hard to see how objects from all these interfaces
>> > are created, and how (if?) they associate to any other protocol
>> > objects.
>> 
>> On goal is to get setup a complete scenery during the startup of the
>> compositor. Therefore the compositor is able to just initialize the
>> main visible attribute during startup. That implies too that it
>> should be possible, without the final application is running (like
>> navigation). Therefore we need some logical objects like ivi-surface,
>> ivi-layer. Which can be already controlled, whithout a application
>> which holds the content is already running.
> 
> But if all layer and window management is internal to the compositor
> process, then you don't need any of this control protocol.
> 
> Just create some internal data structures, and when a client comes up
> with a surface and requests a particular role, you can plug that
> surface into the slot you prepared in advance.
> 
> So, yes, this does imply that the compositor or the shell plugin knows
> the scenegraph, but... isn't that exactly the compositor's purpose, to
> gather the scenegraph and get it on screen? Clients "only" provide
> content.
> 
> For example, the desktop-shell protocol extension has special surface
> roles "background" and "panel". When weston-desktop-shell client
> assigns these roles to its surfaces, the compositor (shell plugin) will
> immediately plug them into the correct places on screen. If the shell
> plugin wanted to reserve screen space for them in advance, it could.
> After all, the shell plugin is responsible for arranging everything on
> (and off) screen.
> 
>> > Btw. if you need support for surface scaling and cropping, there
>> > have been discussion on the Wayland mailing list to bring a crop &
>> > scale protocol extension to Wayland. It is actually necessary for
>> > efficient video playback etc., so pushing that forward would be
>> > nice.
>> 
>> Yeah I see that, and we will consider it. I think we can redesign the
>> approach like source and destination region to the crop and scale
>> instead.
>> 
>> > After looking through the two links you gave, the ivi-shell.xml,
>> > and what you have wrote in the emails, I still have no clue what is
>> > the big picture here.
>> >
>> > - What processes are going to use which interfaces? It looks to me
>> >   like some interfaces are not meant for all Wayland clients, but
>> >   how is it supposed to work?
>> > - What components are in a whole IVI system, from the point of view
>> >   of Wayland protocol? What are the responsibilities of each
>> >   component and how are these distributed into processes?
>> 
>> ivi-control : Is the only process which is allowed to control the
>> compositor itself. But they need some informations about other
>> applications which are running. In the ivi-domain, each graphical
>> application has a unique ID which is defined by the OEM or platform
>> provider. ivi-client :  are just other applications like browser or
>> navigation.
> 
> Ok, so you probably need a protocol interface to tell the compositor
> the unique ID, and perhaps an authentication/authorization interface,
> so that a shell plugin has all the needed information to do the
> controlling.
> 
>> > - What does a typical IVI application do in terms of Wayland
>> >   protocol? Are you using wl_compositor at all? Or any other
>> >   Wayland core interfaces?
>> 
>> We were focused on just using the wayland core interface. But we are
>> open to extend this approach.
> 
> For the record, "Wayland core" is everything in
> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
> except wl_shell and wl_shell_surface, which are specific to desktop
> applications. But you are free to pick and choose individual global
> interfaces.
> 
>> > The protocol you propose seems to have many references to
>> > "id_native" and "native content", what is all this "native" stuff
>> > about? Or all the integer id's you seem to be sending back and
>> > forth, why can't you use real protocol objects to refer to those?
>> 
>> This is a little bit confusing, the initial set of these objects like
>> it is defined here :
>> 
>> http://git.projects.genivi.org/?p=layer_management.git;a=blob;f=wayland-ivi-extension/protocol/ivi-shell.xml;h=6092b13ef31e55f3db338be2e471d0e956be5b9e;hb=ivi-extension
>> 
>> Is focused on using the ivi-surface instead of the native id. The
>> rational behind that is to define unique ids over the complete
>> process space. Therefore the HMI is able to control the different
>> ivi-surface properties.
> 
> Ok, you are inventing global names for these protocol objects, so you
> could do "window management" in a separate process, right? If you made
> the HMI controller a compositor plugin, these global names would not be
> needed, perhaps?
> 
> Perhaps I still do not fully grasp what the HMI controller is. And I
> have more of your emails to read still.
> 
> You may have a very good reason to keep the HMI controller separate,
> but the thing I am questioning is, is the interface towards the
> compositor the right one. Should the controller really be responsible
> for window management, or should it just send simple messages like
> "rear view camera on/off" or even "gear: reverse".
> 
> Even if the HMI controller was one big block in your architecture
> diagrams, you could split it into a compositor plugin and an external
> process. Then the private Wayland protocol extension they use to
> communicate with each other will be just an implementation detail of
> the whole block, which you might not need to standardize.
> 
> There actually exists examples of such a design: most of the
> Wayland-enabled EGL implementations use a private protocol extension to
> communicate between the server and the client. The trick is, both the
> server and the clients load the same libEGL, so the protocol becomes a
> completely hidden implementation detail.
> 
>> In the end we are very lucky about any input to get a final best
>> fitting solution for both the ivi aspects and the final
>> implementation for that in the FOSS domain.
> 
> I am very glad you came forward. :-)
> 
> 
> Thanks,
> pq