[PATCH weston 0/6] ivi-shell proposal

[Pekka Paalanen]

On Mon, 9 Sep 2013 08:01:51 +0000
<Michael.Schuldt at bmw.de> wrote:

> Hi all,
> 
> let me give some technical background on the extension, and why we
> have decided to design the first approach of the extensions like it
> is published from Tanibata-san. In general, we would like to get the
> feedback on the extensions, to get it fitting in the FOSS approach,
> fulfilling the ivi requirements. See my comments in line. 

Jason Ekstrand had an excellent point asking which parts you are
actually looking for to standardize.

> >> Hi,
> >> 
> >> I would like to get feedback about that if somebody has a similar
> >> motivation to support ivi as well as desktop and tablet. So it is
> >> not a stone, just a proposal. If somebody has good idea, I would
> >> like to use it or collaborate it.
> > 
> > Ok, I just had the understanding that the layer manager simply has
> > to be a separate process and not built into the compositor. 
> 
> Not really. Currently in the compliance from genivi, we have the
> separation of Graphic Backend Server and LayerManagement. But this
> should now change we want to skip that approach and want define a
> protocol + ivi extensions, including a reference compositor. We have
> decided to use wayland + ivi extension as compositor protocol and
> weston as the reference implementation for that.

Very cool!

> > If that is
> > not the case, then that is very good news indeed. Everything that
> > manages surfaces, layers, windows, or whatever belongs into the
> > compositor process, where they are much easier to implement and you
> > don't need to introduce interfaces and IPC which are later hard to
> > develop further and cause latencies. 
> 
> Sure that is the technical best fitting solution for that, we
> do not want to use separate IPC for the composition approach. 
> > So I'm roughly on the same track as Andreas Pokorny.
> 
> We too, but unfortunately it is not possible all the time to get
> the complete user interface included in the compositor itself. We
> have to guarantee fast startup time to get early video running in 2
> seconds. On different systems we can not achieve that with complete
> HMI-UI is up. 

Oh, no, I am not suggesting to put user interfaces in the compositor,
not at all. Only the window management, which means window positioning,
z-order, etc. For instance, in Weston virtual desktops are implemented
in the compositor, and not by an external process juggling window lists
via IPC and setting visibilities and positions.

Proposing just like with weston and desktop-shell, there is the weston
plugin called desktop-shell.so which does window management, and then
there is the helper program weston-desktop-shell, that does all the UI
drawing in a separate process.

Like Jason Ekstrand explained, desktop-shell.so and
weston-desktop-shell communicate with each other not only using the
core Wayland protocol, but also using a private (or proprietary)
protocol extension.

While the basic IPC mechanism is all the same (Wayland), there are
different parts: extensions. The desktop-shell extension is actually a
protected one, only weston-desktop-shell as started by desktop-shell.so
can access it.

> > Have you tried to map your IVI concepts of surface/layer/display to
> > Wayland wl_surface, wl_subsurface, and wl_output? I don't really
> > see what kind of interfaces your applications (Wayland clients?)
> > expect to use.
> > 
> > When I look at the protocol in ivi-shell.xml, I get the feeling
> > that:
> > - Interfaces ivi_layer, ivi_controller_surface,
> >   ivi_controller_layer, ivi_controller_screen, and ivi_controller
> > should be internal implementation details inside the weston
> > process, not protocol. Having these as interfaces looks like the X
> > architecture, where the X server process and window manager process
> > continuously struggle to keep each other up-to-date, and carefully
> > try to keep state in sync (and fail), which also makes races and
> > glitches practically unavoidable. - Interface ivi_client is just a
> > reinvention of wl_compositor and wl_subcompositor.
> > - Interface ivi_surface is a reinvention of wl_surface.
> 
> Let me explain, why this separation occurs. 
> From the ivi perspective we have separated the different applications
> in ivi-client, ivi-control. 
> 
> ivi-client: Is just a client, like browser, navigation which is
> embedded in the browser. That belongs to the ivi-surface. These
> process have really limited access the can just create the surface
> and that's all. The only important point which the want to know is if
> the visibility has changed and the size to adapt the rendered
> content.  A navigation should not render the content if it is not
> visible or assigned to a layer. 

That is actually very well fitting to what we already have in Wayland.
Just replace ivi-client with a wl_client (that's not a protocol
object, it represents a display connection from a client process), and
ivi-surface with wl_surface+wl_shell_surface.

The limited access is already there: in Weston, clients are free to
create any number of wl_surfaces. However, no surface is shown
anywhere on screen, unless it is assigned a suitable role. Examples of
roles are: "pointer cursor", "top-level window", "drag icon", and
"popup (menu)". Without a role, the compositor cannot know what to do
with a surface.

"top-level window" and "popup menu" are shell concepts. If you replace
wl_shell with your own ivi_shell, you could have roles like: "TV
output", "rear camera", "speedometer". I'm totally pulling these out of
my hat, I don't know what actually makes sense for you.

Then, your shell module in weston could handle such surfaces as you
want. For example, if you get a surface with the role "TV output", the
shell plugin could show that surface cloned on all entertainment
monitors. Or whatever.

Or if something signals that now would be a /really/ good time to show
the rear view camera, then your shell plugin can make it so, if there
is a surface with such role assigned.

Now, something like a "rear view camera" sounds like a pretty important
role for a surface, so you may want to make it privileged. You don't
want any random application to claim to be a rear view camera. In that
case, you can simply add whatever authentication you need to do in the
shell plugin. For example, only this one process started directly by
the shell plugin is allowed to claim the rear view camera role. Anyone
else trying that will get killed. On the protocol design, such
protection is pretty irrelevant. All you need is for the shell plugin
to know who is authorized for what, and it can then enforce the
restriction, whether the privileged role is requested by a method in a
otherwise public protocol extension or in its own extension.

> ivi-controller : This is the main application which is controlling
> the compositor like the UI. That could be the hmi, but it can also
> some debug applications and application which is triggering a screen
> shot of surfaces, layer, displays. If you driving a car during the
> development you want to get some screenshots which are normally
> triggered by an external application like the log and trace framework.

Ok. I am proposing to have window management in the compositor plugin.
But, you can offer additional debugging interfaces in the protocol.

For example, Weston does not allow clients to control *any* input
devices. Clients simply cannot feed fake input events to other clients.
But for testing and debugging, we need a way to fake input from a
client. The solution: a test plugin.

The test pluging, which is loaded by the compositor only for testing
(in your case debugging) purposes registers a new protocol extension:
the test interface. That test interface then allows to do all kinds of
ugly things from clients.

I'm not sure what you mean by "UI", but we separate controlling windows
from drawing content, like I explained of the desktop shell.

As a recap, the shell plugin is what enforces and executes policies.
Clients only indicate intent or purpose, like "This surface is a menu,
and I want it to pop up right there on my window." A menu is an
interesting example, since it is more than just a surface role: it
comes with an input device grab. While the menu is up, input is
directed to it. But, it is the shell plugin that can decide, that ok,
you're done now, grab is broken, the menu surface hidden, and the
client is signalled that something dismissed the menu. Also, clients
cannot put up a menu at random times, it has to be done as a response
to an input event. All this prevents many kinds of client misbehaviour.

> > Yes, I see there are some details to may want to control like
> > surface opacity, that the current Wayland protocols do not support,
> > but I don't think that replacing everything is a good way to start.
> > It is also very hard to see how objects from all these interfaces
> > are created, and how (if?) they associate to any other protocol
> > objects.
> 
> On goal is to get setup a complete scenery during the startup of the
> compositor. Therefore the compositor is able to just initialize the
> main visible attribute during startup. That implies too that it
> should be possible, without the final application is running (like
> navigation). Therefore we need some logical objects like ivi-surface,
> ivi-layer. Which can be already controlled, whithout a application
> which holds the content is already running.

But if all layer and window management is internal to the compositor
process, then you don't need any of this control protocol.

Just create some internal data structures, and when a client comes up
with a surface and requests a particular role, you can plug that
surface into the slot you prepared in advance.

So, yes, this does imply that the compositor or the shell plugin knows
the scenegraph, but... isn't that exactly the compositor's purpose, to
gather the scenegraph and get it on screen? Clients "only" provide
content.

For example, the desktop-shell protocol extension has special surface
roles "background" and "panel". When weston-desktop-shell client
assigns these roles to its surfaces, the compositor (shell plugin) will
immediately plug them into the correct places on screen. If the shell
plugin wanted to reserve screen space for them in advance, it could.
After all, the shell plugin is responsible for arranging everything on
(and off) screen.

> > Btw. if you need support for surface scaling and cropping, there
> > have been discussion on the Wayland mailing list to bring a crop &
> > scale protocol extension to Wayland. It is actually necessary for
> > efficient video playback etc., so pushing that forward would be
> > nice.
> 
> Yeah I see that, and we will consider it. I think we can redesign the
> approach like source and destination region to the crop and scale
> instead.
> 
> > After looking through the two links you gave, the ivi-shell.xml,
> > and what you have wrote in the emails, I still have no clue what is
> > the big picture here.
> > 
> > - What processes are going to use which interfaces? It looks to me
> >   like some interfaces are not meant for all Wayland clients, but
> >   how is it supposed to work?
> > - What components are in a whole IVI system, from the point of view
> >   of Wayland protocol? What are the responsibilities of each
> >   component and how are these distributed into processes?
> 
> ivi-control : Is the only process which is allowed to control the
> compositor itself. But they need some informations about other
> applications which are running. In the ivi-domain, each graphical
> application has a unique ID which is defined by the OEM or platform
> provider. ivi-client :  are just other applications like browser or
> navigation.

Ok, so you probably need a protocol interface to tell the compositor
the unique ID, and perhaps an authentication/authorization interface,
so that a shell plugin has all the needed information to do the
controlling.

> > - What does a typical IVI application do in terms of Wayland
> >   protocol? Are you using wl_compositor at all? Or any other
> >   Wayland core interfaces?
> 
> We were focused on just using the wayland core interface. But we are
> open to extend this approach.

For the record, "Wayland core" is everything in
http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
except wl_shell and wl_shell_surface, which are specific to desktop
applications. But you are free to pick and choose individual global
interfaces.

> > The protocol you propose seems to have many references to
> > "id_native" and "native content", what is all this "native" stuff
> > about? Or all the integer id's you seem to be sending back and
> > forth, why can't you use real protocol objects to refer to those?
> 
> This is a little bit confusing, the initial set of these objects like
> it is defined here :
> 
> http://git.projects.genivi.org/?p=layer_management.git;a=blob;f=wayland-ivi-extension/protocol/ivi-shell.xml;h=6092b13ef31e55f3db338be2e471d0e956be5b9e;hb=ivi-extension
> 
> Is focused on using the ivi-surface instead of the native id. The
> rational behind that is to define unique ids over the complete
> process space. Therefore the HMI is able to control the different
> ivi-surface properties.

Ok, you are inventing global names for these protocol objects, so you
could do "window management" in a separate process, right? If you made
the HMI controller a compositor plugin, these global names would not be
needed, perhaps?

Perhaps I still do not fully grasp what the HMI controller is. And I
have more of your emails to read still.

You may have a very good reason to keep the HMI controller separate,
but the thing I am questioning is, is the interface towards the
compositor the right one. Should the controller really be responsible
for window management, or should it just send simple messages like
"rear view camera on/off" or even "gear: reverse".

Even if the HMI controller was one big block in your architecture
diagrams, you could split it into a compositor plugin and an external
process. Then the private Wayland protocol extension they use to
communicate with each other will be just an implementation detail of
the whole block, which you might not need to standardize.

There actually exists examples of such a design: most of the
Wayland-enabled EGL implementations use a private protocol extension to
communicate between the server and the client. The trick is, both the
server and the clients load the same libEGL, so the protocol becomes a
completely hidden implementation detail.

> In the end we are very lucky about any input to get a final best
> fitting solution for both the ivi aspects and the final
> implementation for that in the FOSS domain.

I am very glad you came forward. :-)


Thanks,
pq