[PATCH weston 0/6] ivi-shell proposal

Mon Sep 9 19:33:20 PDT 2013

Michale & Nobuhiko,
First of all, thank you for the clarification and thank you for sending
this to the list and being willing to work with the FOSS community to try
and make a standard.  I'm sorry that this reply is not inline.  I think
that would get disorganized and more confusing but I'll try to hit
everything I saw.

The first distinction that needs to be made (and I think Pekka was trying
to hint at this) is what should be standardized.  If you look at the
current [wayland core protocol][1] there is only one shell protocol called
wl_shell.  I have proposed another which will probably get called
[wl_fullscreen_shell][2].  Both of these have something in common: they are
purely client-side.  There is nothing whatsoever in the standard about
managing surfaces.  I think that we should focus on what you have
designated ivi_client.

What about the controller?  If you look in the [weston protocol folder][3]
you will find a number of different protocol files.  Some of these are for
experimental extensions such as subsurfaces which have not yet made it into
wayland core.  However, a number of them such as desktop-shell,
screenshooter, etc. will *never* be standardized in the wayland core.
These protocols are completely internal to weston and are considered
implementation details.  The primary example is desktop-shell.  This
protocol exists for the purpose of allowing the out-of-process shell
controller manage surfaces similar to what you propose with ivi_shell.
There are other shell plugins for weston (hawaii & orbital) that each have
their own shell plugin and can have their own protocol for talking to an
out-of-process controller.

How does this impact your proposed protocol?  Unless you are convinced that
every single IVI system manufacturer will want to manage surfaces the same
way, the controller should be left as a private implementation detail.  You
are free to do it out-of-process and talk the wayland protocol to do so
(desktop-shell does) but there is no need to expose it as part of a
standard protocol.  By only standardizing the client interface you leave
app developers (GPS, Media players, etc.) free to design their apps however
they want and you leave IVI system manufacturers free to handle those
clients and surfaces in whatever way they want.

Ok, now on to actual suggestions.  From this point forward, I am going to
completely ignore the controller side of things.

First, I would propose to follow the pattern of wl_shell and make two
interfaces for clients to talk to the compositor. For now, I am going to
call them wl_ivi_shell and wl_ivi_surface.  We can come up with different
names if you'd like, but those seem reasonable.  If we follow the pattern
of wl_shell, wl_ivi_shell will probably exist for the sole purpose of
creating wl_ivi_surface objects.  This pattern is common in the protocol
(wl_shell, wl_subcompositor, wl_compositor, etc.).

The main question, then, becomes what to put in wl_ivi_surface.  I'm not
100% sure what you intend with some of this surface and layer stuff, so I'm
afraid I don't have a whole lot of specific suggestions on that for now.  I
do, however have some general thoughts and questions:

First, I agree with Pekka that you can probably avoid the layers thing by
simply using subsurfaces.

Second, Why are you specifying pixel formats in ivi_surface?  Is the
compositor supposed to tell the client what format to render in?

Third, concerning the "visibility" flag.  The wayland protocol as it
currently stands tries to avoid telling clients specifically whether or not
they are visible and where they are on screen.  This is because, when
clients abuse this information, compositors lose the freedom to throw
surfaces around how they want.  Instead of a visibility flag, the
wl_surface interface provides a "frame" callback that the clients can use
to know when was the last time they were drawn to the screen.  A client
should throttle rendering based on these frame events.  If the surface is
offscreen and the compositor wants the client to stop rendering it simply
stop sending it frame events and the client will stop drawing.

Once again, thank you for mailing the list.  I hope my thoughts above are
helpful and can clear a few things up.
Thanks,
--Jason Ekstrand

[1]: http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
[2]:
http://lists.freedesktop.org/archives/wayland-devel/2013-August/010720.html
[3]: http://cgit.freedesktop.org/wayland/weston/tree/protocol

On Mon, Sep 9, 2013 at 7:59 PM, Nobuhiko Tanibata <
NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:

>
> ----- Original Message ----- From: "Pekka Paalanen" <ppaalanen at gmail.com>
> To: "nobuhiko_tanibata" <nobuhiko_tanibata at localhost.**xddp.denso.co.jp<nobuhiko_tanibata at localhost.xddp.denso.co.jp>
> >
> Cc: "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.**co.jp<NOBUHIKO_TANIBATA at xddp.denso.co.jp>>;
> <securitycheck at denso.co.jp>; <wayland-devel at lists.**freedesktop.org<wayland-devel at lists.freedesktop.org>
> >
> Sent: Sunday, September 08, 2013 6:31 PM
>
> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>
>
>  On Sun, 08 Sep 2013 00:13:55 +0900
>> nobuhiko_tanibata <nobuhiko_tanibata at localhost.**xddp.denso.co.jp<nobuhiko_tanibata at localhost.xddp.denso.co.jp>>
>> wrote:
>>
>>  2013-09-06 19:16 に Pekka Paalanen さんは書きました:
>>> > On Fri, 6 Sep 2013 08:39:24 +0900
>>> > "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.**co.jp<NOBUHIKO_TANIBATA at xddp.denso.co.jp>>
>>> wrote:
>>> >
>>> >> ----- Original Message -----
>>> >> From: "Pekka Paalanen" <ppaalanen at gmail.com>
>>> >> To: "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.**co.jp<NOBUHIKO_TANIBATA at xddp.denso.co.jp>
>>> >
>>> >> Cc: <wayland-devel at lists.**freedesktop.org<wayland-devel at lists.freedesktop.org>>;
>>> <securitycheck at denso.co.jp>
>>> >> Sent: Thursday, September 05, 2013 10:02 PM
>>> >> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>>> >>
>>> >>
>>> >> > On Wed, 4 Sep 2013 09:08:26 +0900
>>> >> > "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.**co.jp<NOBUHIKO_TANIBATA at xddp.denso.co.jp>>
>>> wrote:
>>> >> >
>>> >> >> Hi,
>>> >> >>
>>> >> >> This series implements ivi-shell to fulfill use cases of In-Vehicle
>>> >> >> Infotainment, IVI. Such use cases are well overviewed in a project;
>>> >> >> Genivi IVI layer management.
>>> >> >> http://projects.genivi.org/**ivi-layer-management/node/13<http://projects.genivi.org/ivi-layer-management/node/13>
>>> >> >>
>>> >> >> A motivation of this series and basis idea are introduced by Ossama
>>> >> >> at Automotive Linux Summit 2012 spring. The series implements
>>> >> >> ivi-shell part. Additionally, GENIVI LM Client Library at slide 20
>>> >> >> is
>>> >> >> contributed to ivi-layer-management project to support compatible
>>> >> >> interfaces for Genivi Layer management users.
>>> >> >> http://events.linuxfoundation.**org/images/stories/pdf/**
>>> als2012_othman.pdf<http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf>
>>> >> >>
>>> >> >> Before I start implementation of ivi-shell, Core members of Genivi
>>> >> >> IVI layer management defined draft of ivi-shell.xml to fulfill
>>> >> >> requirements of IVI layer management, inviting Kristian. The series
>>> >> >> also includes the ivi-shell.xml with updates I faced in actual
>>> >> >> implementation.
>>> >> >>
>>> >> >> Please give me any suggestions.
>>> >> >
>>> >> > Hi,
>>> >> >
>>> >> > I have understood that the whole design has been cut deep into
>>> stone >> > a
>>> >> > long time ago. What are you able to change now? I do not think it is
>>> >> > worth commenting on things you can no longer change, so what
>>> aspects >> > are
>>> >> > you looking suggestions for?
>>> >> >
>>> >>
>>> >> Hi,
>>> >>
>>> >> I would like to get feedback about that if somebody has a similar
>>> >> motivation
>>> >> to support ivi as well as desktop and tablet.
>>> >> So it is not a stone, just a proposal. If somebody has good idea, I
>>> >> would
>>> >> like to use it or collaborate it.
>>> >
>>> > Ok, I just had the understanding that the layer manager simply has to
>>> > be a separate process and not built into the compositor. If that is
>>> > not the case, then that is very good news indeed. Everything that
>>> > manages surfaces, layers, windows, or whatever belongs into the
>>> > compositor process, where they are much easier to implement and you
>>> > don't need to introduce interfaces and IPC which are later hard to
>>> > develop further and cause latencies. Roundtrips and synchronous
>>> > calls between processes can become a difficult bottle-neck, as X11
>>> > has taught us. Also having too many processes becomes a real mess
>>> > when trying to avoid deadlocks but still keep things coherent and
>>> > glitch-free (see X11 server vs. window manager vs. ...).
>>> > So I'm roughly on the same track as Andreas Pokorny.
>>> >
>>> > Weston should become the window manager and layer manager, while
>>> > weston backends deal with the hardware details of compositing. For
>>> > example, the Raspberry Pi backend of Weston forwards all
>>> > compositing to the VideoCore firmware, unlike any other backend
>>> > who actually render a composite themselves. However, the rpi
>>> > backend does not forward all surfaces to VideoCore all the time,
>>> > but only the visible ones as needed. And Weston is the only
>>> > component that *can* know what is visible at any time, since Weston
>>> > contains the complete scene graph. Weston's internal architecture
>>> > is also well-suited for *automatically* deciding how to use the
>>> > limited hardware resources like overlays efficiently, and fall back
>>> > as necessary, per each output frame. You cannot do that, if you
>>> > tell applications about specific hardware resources.
>>>
>>> Yes, my understind is the same as yours.
>>>
>>
>> Great. :-)
>>
>>  And I expects Weston backend
>>> you mentioned next paragraph is ideally released as reference from SoC
>>> vender.
>>> For exmple, some SoC vendor supports 2D blit engine which can effciently
>>> composite surfaces and relese weston backend as reference.
>>>
>>
>> Tbh, I doubt vendor's abilities in producing a proper Weston
>> backend, and maintaining it as Weston changes. But yeah, something
>> like that.
>>
>>  > I suspect there might be some terminology differences here.
>>> > Something like what IVI calls a "compositor" is a "weston backend"
>>> > when talking about Weston and Wayland, and IVI layer manager is
>>> > actually a window manager which is just a shell plugin to Weston.
>>> > Or am I completely off?
>>> >
>>>
>>> Yes, you are correct. And I focus on shell part to realize ivi
>>> requirements.
>>>
>>
>> Shell is usually the part which is primarily used by applications, not
>> supporting components of a graphical environment (see wl_shell vs.
>> desktop_shell protocol interfaces). I mean the public part of
>> shell, like the wl_shell interface for desktop applications.
>>
>> Which brings me to another question: how likely is it that you
>> actually want to support PC desktop applications unmodified
>> directly on an IVI system? And I mean on the main compositor of an
>> IVI system, which I would imagine to be quite a critical component.
>>
>> I think native IVI applications could be different enough to PC desktop
>> applications, that creating a new shell interface to *replace*
>> wl_shell (or whatever shell interface we will be using in the
>> future for PC desktop) would be a right choice.
>>
>> If you actually do want to support PC desktop applications, I could
>> see you having another, nested compositor just for supporting PC
>> applications, which could run on a more restricted environment and
>> maybe access to a more complex GPU which would be too risky for
>> the main compositor in IVI. A sort of "untrusted" domain.
>>
>> Well, Genivi probably has already designed all that, so I'm just
>> reinventing the wheel badly here.
>>
>>  > Have you tried to map your IVI concepts of surface/layer/display to
>>> > Wayland wl_surface, wl_subsurface, and wl_output? I don't really
>>> > see what kind of interfaces your applications (Wayland clients?)
>>> > expect to use.
>>>
>>> Yes. As you comment, some use case; visibility and crop/scaling is not
>>> supported now.
>>> So I thought starting new set of protocal to cover ivi requiremnts would
>>> be better.
>>> But I will re-consider them and mail it back.
>>>
>>
>> Right. I replied from purely FOSS point of view. You probably have
>> time and money deadlines which you must meet while creating a
>> self-standing product, and in that case, I do understand going with
>> a big re-invention. I understand it, but I'm not happy about it,
>> although if you can publish your ad hoc approach like you do here,
>> you are contributing valuable experience to the community.
>> Especially, if you say something about the shortcomings of the
>> design and use experiences.
>>
>> I'm very happy to see this proposal on the mailing list, even when
>> I do not agree with it.
>>
>>  > When I look at the protocol in ivi-shell.xml, I get the feeling that:
>>> > - Interfaces ivi_layer, ivi_controller_surface,
>>> >   ivi_controller_layer, ivi_controller_screen, and ivi_controller
>>> >   should be internal implementation details inside the weston
>>> >   process, not protocol. Having these as interfaces looks like the
>>> >   X architecture, where the X server process and window manager
>>> >   process continuously struggle to keep each other up-to-date, and
>>> >   carefully try to keep state in sync (and fail), which also makes
>>> >   races and glitches practically unavoidable.
>>>
>>> I have basic question to the above; strugling. wl_subsurface supports
>>> set_poistion now.
>>>
>>
>> set_position for sub-surfaces is *always* relative to the parent
>> surface. The sub-surface position is given as a point on the parent
>> surface. You still have no control where on the output any surface
>> will appear, because you cannot control where the root surface of a
>> tree of sub-surfaces will appear.
>>
>> If a parent surface is moved on screen, all its sub-surfaces stay
>> glued to it automatically without any client interaction.
>>
>>  I thought it implies positioning from a windowmanger is allowed on
>>> weston basic concept.
>>>
>>
>> From a window manager, yes, but this assumes that the window manager
>> is in-process with weston; the window manager must be a compositor
>> plugin. Making it an external process will be a huge amount of
>> trouble and performance loss.
>>
>> And the protocol you propose seems to be for an external window
>> manager process, from my understanding.
>>
>>  Each other up-to-date may occurs from window manager and weston internl
>>> decision.
>>> Or positioning of sub surfaces is out of scope of weston and it just
>>> composite them according to attributes.
>>> I may be wrong. Please let me know history.
>>>
>>
>> The first thing is that in Wayland core protocol, there is no
>> global positioning system. There are no global coordinates in the
>> protocol. All coordinates that clients deal with, are relative to
>> some wl_surface. (This also allows the compositor to do arbitrary
>> transformations on surfaces, because there is no need for clients
>> to know about them, and so no need to express transformations in
>> the protocol.)
>>
>> Sub-surfaces do not change that design.
>>
>> Another thing is that with sub-surface positioning, the information
>> flows strictly into one direction, and we use wl_surface.commit
>> request on the parent surface to synchronize everything. That means
>> that a client can manipulate a whole tree of sub-surfaces, and
>> *guarantee* an atomic, flicker-free, glitch-free update on screen.
>>
>> If one needed IPC between a compositor and a window manager, you
>> would either risk visual glitches as compositor first draws one
>> thing before the window manager says otherwise, or jerky compositor
>> performance as it needs to wait for the window manager to respond
>> before it can draw anything. I don't see any way around that.
>>
>>  > - Interface ivi_client is just a reinvention of wl_compositor and
>>> >   wl_subcompositor.
>>> > - Interface ivi_surface is a reinvention of wl_surface.
>>> >
>>> > Yes, I see there are some details to may want to control like
>>> > surface opacity, that the current Wayland protocols do not support,
>>> > but I don't think that replacing everything is a good way to start.
>>> >
>>> > It is also very hard to see how objects from all these interfaces
>>> > are created, and how (if?) they associate to any other protocol
>>> > objects.
>>> >
>>> > Btw. if you need support for surface scaling and cropping, there
>>> > have been discussion on the Wayland mailing list to bring a crop &
>>> > scale protocol extension to Wayland. It is actually necessary for
>>> > efficient video playback etc., so pushing that forward would be
>>> > nice.
>>> >
>>>
>>> Thank you. I will check.
>>>
>>
>> Search for "wl_scaler", that was the working name of a proposal.
>>
>>  > After looking through the two links you gave, the ivi-shell.xml,
>>> > and what you have wrote in the emails, I still have no clue what is
>>> > the big picture here.
>>> >
>>>
>>> I will draw a pciture to explain them. I will mail it back later.
>>>
>>
>>
> Hi,
>
> I am enclosing a pdf to show relationship compositor, surface, layer, and
> controller.
>
> As Michael Schuldt says in reply, only a process will control attribute of
> surfaces and layers like position, visivility, order, and so on.
> The controller process is kind of central one to manage policy of layout
> and status of surfaces and layers by taking account into infomation in
> vehicle. For example, when gear position is rear, rear view camera would be
> set to the top of order and visible. When I see your comment,
> I think wl_subsurface can be applied to layer and parent can be mapped to
> layer. Now I am considering it. However, I have one use case for ivi to
> control attribute of wl_surface from outside, e.g set invisible to TV
> screen when speed is e.g 10km/s for safty. it is not welcome for users but
> I have to consider it. Each application can do it but on the safty point of
> view, it shall be handle in cetranl controller.
>
> Addtionally, As Michael Schuldt says in reply as well, for debugging
> purpose, controlling wl_surface would be required.
>
> When I see my picture by myself, ivi-shell will be window manager in side
> of weston. So it might not be conflict be concept of shell.
>
> Please give me your feedback.
>
> Best regards,
> Tanibata
>
>  Cool, thanks.
>>
>> I saw the terms surface, layer, etc. in the IVI docs but I didn't
>> really get what they are used for.
>>
>>  > - What processes are going to use which interfaces? It looks to me
>>> >   like some interfaces are not meant for all Wayland clients, but
>>> >   how is it supposed to work?
>>> >
>>> > - What components are in a whole IVI system, from the point of view
>>> >   of Wayland protocol? What are the responsibilities of each
>>> >   component and how are these distributed into processes?
>>> >
>>> > - What does a typical IVI application do in terms of Wayland
>>> >   protocol? Are you using wl_compositor at all? Or any other
>>> >   Wayland core interfaces?
>>> >
>>> > These are just few questions to get you oriented on what kind of
>>> > things puzzle me here. Obviously, I have never been in touch with
>>> > Genivi stuff before, and I would assume most here have not either.
>>> >
>>> > The protocol you propose seems to have many references to
>>> > "id_native" and "native content", what is all this "native" stuff
>>> > about? Or all the integer id's you seem to be sending back and
>>> > forth, why can't you use real protocol objects to refer to those?
>>> >
>>> >
>>> > The above is the first impression from someone, who does not know
>>> > anything about the IVI architecture, but is fairly familiar with
>>> > Wayland. Sorry if it came out harsh, but I feel like I totally
>>> > missed the whole background of this proposal: why design it like
>>> > that?
>>> >
>>>
>>> Thank you again for good feedback. They are very helpfull for me.
>>>
>>
>>
>> Thank you,
>> pq
>> ______________________________**_________________
>> wayland-devel mailing list
>> wayland-devel at lists.**freedesktop.org<wayland-devel at lists.freedesktop.org>
>> http://lists.freedesktop.org/**mailman/listinfo/wayland-devel<http://lists.freedesktop.org/mailman/listinfo/wayland-devel>
>>
>>
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/wayland-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20130909/3e8364b8/attachment-0001.html>