[PATCH weston 0/6] ivi-shell proposal

Michael.Schuldt at bmw.de Michael.Schuldt at bmw.de
Tue Sep 10 04:13:06 PDT 2013 

Hi Jason, Tanibata-san

> Hi Jason,
> 
> Thank you very much for feedback.
> 
>> Michale & Nobuhiko,
>> 
>> First of all, thank you for the clarification and thank you for
>> sending this to the list and being willing to work with the FOSS
>> community to try and make a standard. I'm sorry that this reply is not
>> inline. I think that would get disorganized and more confusing but
>> I'll try to hit everything I saw.
>> 
>> The first distinction that needs to be made (and I think Pekka was
>> trying to hint at this) is what should be standardized. If you look at
>> the current [wayland core protocol][1] there is only one shell
>> protocol called wl_shell. I have proposed another which will probably
>> get called [wl_fullscreen_shell][2]. Both of these have something in
>> common: they are purely client-side. There is nothing whatsoever in
>> the standard about managing surfaces. I think that we should focus on
>> what you have designated ivi_client.

Ok, now I got it. No we do not want to push it to the core protocol. We want
to define a IVI-Shell like desktop shell and want to use weston for the reference
implementation. The ivi-shell extension should be the minimum subset which shall provided
by each ivi-compositor. But in terms of the genivi compliance we are able to define
a shell protocol and say this is the minimal subset which is supported, and therefore
we need a reference implementation too. Maybe we have mixed something here from our side.
We will clarify that

>> What about the controller? If you look in the [weston protocol
>> folder][3] you will find a number of different protocol files. Some of
>> these are for experimental extensions such as subsurfaces which have
>> not yet made it into wayland core. However, a number of them such as
>> desktop-shell, screenshooter, etc. will *never* be standardized in the
>> wayland core. These protocols are completely internal to weston and
>> are considered implementation details. The primary example is
>> desktop-shell. This protocol exists for the purpose of allowing the
>> out-of-process shell controller manage surfaces similar to what you
>> propose with ivi_shell. There are other shell plugins for weston
>> (hawaii & orbital) that each have their own shell plugin and can have
>> their own protocol for talking to an out-of-process controller.

Yes this is our focus too, like I have described above.

>> How does this impact your proposed protocol? Unless you are convinced
>> that every single IVI system manufacturer will want to manage surfaces
>> the same way, the controller should be left as a private
>> implementation detail. You are free to do it out-of-process and talk
>> the wayland protocol to do so (desktop-shell does) but there is no
>> need to expose it as part of a standard protocol. By only
>> standardizing the client interface you leave app developers (GPS,
>> Media players, etc.) free to design their apps however they want and
>> you leave IVI system manufacturers free to handle those clients and
>> surfaces in whatever way they want.
>> 
>> Ok, now on to actual suggestions. >From this point forward, I am going
>> to completely ignore the controller side of things.
>> 
>> First, I would propose to follow the pattern of wl_shell and make two
>> interfaces for clients to talk to the compositor. For now, I am going
>> to call them wl_ivi_shell and wl_ivi_surface. We can come up with
>> different names if you'd like, but those seem reasonable. If we follow
>> the pattern of wl_shell, wl_ivi_shell will probably exist for the sole
>> purpose of creating wl_ivi_surface objects. This pattern is common in
>> the protocol (wl_shell, wl_subcompositor, wl_compositor, etc.).
>> 
>> The main question, then, becomes what to put in wl_ivi_surface. I'm
>> not 100% sure what you intend with some of this surface and layer
>> stuff, so I'm afraid I don't have a whole lot of specific suggestions
>> on that for now. I do, however have some general thoughts and
>> questions:
>> 
>>  First, I agree with Pekka that you can probably avoid the layers
>> thing by simply using subsurfaces.
>> 
> I see. However, we have a use case that several application, different
> process share a layer. E.g. Navigation map and Route guidance are
> separated into other application. It may kind of grouping parent
> surfaces.
> 
>> Second, Why are you specifying pixel formats in ivi_surface? Is the
>> compositor supposed to tell the client what format to render in?
>> 
>> Third, concerning the "visibility" flag. The wayland protocol as it
>> currently stands tries to avoid telling clients specifically whether
>> or not they are visible and where they are on screen. This is because,
>> when clients abuse this information, compositors lose the freedom to
>> throw surfaces around how they want. Instead of a visibility flag, the
>> wl_surface interface provides a "frame" callback that the clients can
>> use to know when was the last time they were drawn to the screen. A
>> client should throttle rendering based on these frame events. If the
>> surface is offscreen and the compositor wants the client to stop
>> rendering it simply stop sending it frame events and the client will
>> stop drawing.
>> 
> 
> I have two concerning to use "frame" for realizing invisible.
> - To set invisible, application needs to call clear color by itself. I
> think it might be overhead for GPU. If we can realize it in shell, it
> simply skips it to be composite. Of course, application shall stop
> drawing as well.
> - Invisible shall be conrolled by cetral controller due to safy
> reason.It shall be done in lower part as much as possible. Ideally, if
> we can allocate it to another physical plane, it may be best. If Display
> contoller doesn't support it, next is compositor.

Here I am full inline with the post from Tanibata-san

Regards

Michael.
>> Once again, thank you for mailing the list. I hope my thoughts above
>> are helpful and can clear a few things up.
>>  Thanks,
>> --Jason Ekstrand
>> 
>>  [1]:
>> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
>> [2]
>>  [2]:
>> http://lists.freedesktop.org/archives/wayland-devel/2013-
>> August/010720.html [3]
>>  [3]: http://cgit.freedesktop.org/wayland/weston/tree/protocol [4]
>> On Mon, Sep 9, 2013 at 7:59 PM, Nobuhiko Tanibata
>> <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>> 
>  ppaalanen at gmail.com> wrote on o: "nobuhiko_tanibata" <nobuhiko_tanibata at localhost.xddp.denso.co.jp>:
>>  Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>> On Sun, 08 Sep 2013 00:13:55 +0900
>>  nobuhiko_tanibata <nobuhiko_tanibata at localhost.xddp.denso.co.jp>
>> wrote:
>> 
>> 2013-09-06 19:16 に Pekka Paalanen さんは書きました:
>>> On Fri, 6 Sep 2013 08:39:24 +0900
>>> "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>>> 
>>> "Pekka Paalanen" <ppaalanen at gmail.com> wrote on o: "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp>:
>>>> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>>>> 
>>>> 
>>>>> On Wed, 4 Sep 2013 09:08:26 +0900
>>>>> "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> This series implements ivi-shell to fulfill use cases of In-Vehicle
>>>>>> Infotainment, IVI. Such use cases are well overviewed in a project;
>>>>>> Genivi IVI layer management.
>>>>>> http://projects.genivi.org/ivi-layer-management/node/13 [5]
>>>>>> 
>>>>>> A motivation of this series and basis idea are introduced by Ossama
>>>>>> at Automotive Linux Summit 2012 spring. The series implements
>>>>>> ivi-shell part. Additionally, GENIVI LM Client Library at slide 20
>>>>>> >> >> is contributed to ivi-layer-management project to support
>>>>>> compatible interfaces for Genivi Layer management users.
>>>>>> 
>> 
>> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
>> [6]
>>>>>> 
>>>>>> Before I start implementation of ivi-shell, Core members of Genivi
>>>>>> IVI layer management defined draft of ivi-shell.xml to fulfill
>>>>>> requirements of IVI layer management, inviting Kristian. The series
>>>>>> also includes the ivi-shell.xml with updates I faced in actual
>>>>>> implementation.
>>>>>> 
>>>>>> Please give me any suggestions.
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I have understood that the whole design has been cut deep into stone
>>>>> >> > a long time ago. What are you able to change now? I do not
>>>>> think it is worth commenting on things you can no longer change, so
>>>>> what aspects >> > are you looking suggestions for?
>>>>> 
>>>> 
>>>> Hi,
>>>> 
>>>> I would like to get feedback about that if somebody has a similar
>>>> motivation to support ivi as well as desktop and tablet. So it is not
>>>> a stone, just a proposal. If somebody has good idea, I would like to
>>>> use it or collaborate it.
>>> 
>>> Ok, I just had the understanding that the layer manager simply has to
>>> be a separate process and not built into the compositor. If that is
>>> not the case, then that is very good news indeed. Everything that
>>> manages surfaces, layers, windows, or whatever belongs into the
>>> compositor process, where they are much easier to implement and you
>>> don't need to introduce interfaces and IPC which are later hard to
>>> develop further and cause latencies. Roundtrips and synchronous calls
>>> between processes can become a difficult bottle-neck, as X11 has
>>> taught us. Also having too many processes becomes a real mess when
>>> trying to avoid deadlocks but still keep things coherent and
>>> glitch-free (see X11 server vs. window manager vs. ...). So I'm
>>> roughly on the same track as Andreas Pokorny.
>>> 
>>> Weston should become the window manager and layer manager, while
>>> weston backends deal with the hardware details of compositing. For
>>> example, the Raspberry Pi backend of Weston forwards all
>>> compositing to the VideoCore firmware, unlike any other backend
>>> who actually render a composite themselves. However, the rpi
>>> backend does not forward all surfaces to VideoCore all the time,
>>> but only the visible ones as needed. And Weston is the only
>>> component that *can* know what is visible at any time, since Weston
>>> contains the complete scene graph. Weston's internal architecture
>>> is also well-suited for *automatically* deciding how to use the
>>> limited hardware resources like overlays efficiently, and fall back
>>> as necessary, per each output frame. You cannot do that, if you
>>> tell applications about specific hardware resources.
>>> 
>>  Yes, my understind is the same as yours.
>>  
>>  Great. :-)
>> And I expects Weston backend
>>  you mentioned next paragraph is ideally released as reference from SoC
>>  vender. For exmple, some SoC vendor supports 2D blit engine which can
>>  effciently composite surfaces and relese weston backend as reference.
>>  
>>  Tbh, I doubt vendor's abilities in producing a proper Weston
>>  backend, and maintaining it as Weston changes. But yeah, something
>>  like that.
>>  
>>> I suspect there might be some terminology differences here.
>>> Something like what IVI calls a "compositor" is a "weston backend"
>>> when talking about Weston and Wayland, and IVI layer manager is
>>> actually a window manager which is just a shell plugin to Weston.
>>> Or am I completely off?
>>> 
>>> 
>>  Yes, you are correct. And I focus on shell part to realize ivi
>>  requirements.
>>  
>>  Shell is usually the part which is primarily used by applications, not
>>  supporting components of a graphical environment (see wl_shell vs.
>>  desktop_shell protocol interfaces). I mean the public part of shell,
>>  like the wl_shell interface for desktop applications.
>>  
>>  Which brings me to another question: how likely is it that you
>>  actually want to support PC desktop applications unmodified
>>  directly on an IVI system? And I mean on the main compositor of an
>>  IVI system, which I would imagine to be quite a critical component.
>>  
>>  I think native IVI applications could be different enough to PC
>>  desktop applications, that creating a new shell interface to *replace*
>>  wl_shell (or whatever shell interface we will be using in the future
>>  for PC desktop) would be a right choice.
>>  
>>  If you actually do want to support PC desktop applications, I could
>>  see you having another, nested compositor just for supporting PC
>>  applications, which could run on a more restricted environment and
>>  maybe access to a more complex GPU which would be too risky for
>>  the main compositor in IVI. A sort of "untrusted" domain.
>>  
>>  Well, Genivi probably has already designed all that, so I'm just
>>  reinventing the wheel badly here.
>>  
>>> Have you tried to map your IVI concepts of surface/layer/display to
>>> Wayland wl_surface, wl_subsurface, and wl_output? I don't really
>>> see what kind of interfaces your applications (Wayland clients?)
>>> expect to use.
>>> 
>>  Yes. As you comment, some use case; visibility and crop/scaling is not
>>  supported now. So I thought starting new set of protocal to cover ivi
>>  requiremnts would be better. But I will re-consider them and mail it
>>  back.
>>  
>>  Right. I replied from purely FOSS point of view. You probably have
>>  time and money deadlines which you must meet while creating a
>>  self-standing product, and in that case, I do understand going with
>>  a big re-invention. I understand it, but I'm not happy about it,
>>  although if you can publish your ad hoc approach like you do here,
>>  you are contributing valuable experience to the community.
>>  Especially, if you say something about the shortcomings of the
>>  design and use experiences.
>>  
>>  I'm very happy to see this proposal on the mailing list, even when
>>  I do not agree with it.
>>  
>>> When I look at the protocol in ivi-shell.xml, I get the feeling that:
>>> - Interfaces ivi_layer, ivi_controller_surface, ivi_controller_layer,
>>> ivi_controller_screen, and ivi_controller should be internal
>>> implementation details inside the weston process, not protocol. Having
>>> these as interfaces looks like the X architecture, where the X server
>>> process and window manager process continuously struggle to keep each
>>> other up-to-date, and carefully try to keep state in sync (and fail),
>>> which also makes races and glitches practically unavoidable.
>>> 
>>  I have basic question to the above; strugling. wl_subsurface supports
>>  set_poistion now.
>>  
>>  set_position for sub-surfaces is *always* relative to the parent
>>  surface. The sub-surface position is given as a point on the parent
>>  surface. You still have no control where on the output any surface
>>  will appear, because you cannot control where the root surface of a
>>  tree of sub-surfaces will appear.
>>  
>>  If a parent surface is moved on screen, all its sub-surfaces stay
>>  glued to it automatically without any client interaction.
>> I thought it implies positioning from a windowmanger is allowed on
>>  weston basic concept.
>>  
>>> From a window manager, yes, but this assumes that the window manager
>>> is in-process with weston; the window manager must be a compositor
>>> plugin. Making it an external process will be a huge amount of
>>> trouble and performance loss.
>>> 
>>  And the protocol you propose seems to be for an external window
>>  manager process, from my understanding.
>> Each other up-to-date may occurs from window manager and weston
>> internl
>>  decision.
>>  Or positioning of sub surfaces is out of scope of weston and it just
>>  composite them according to attributes.
>>  I may be wrong. Please let me know history.
>>  
>>  The first thing is that in Wayland core protocol, there is no
>>  global positioning system. There are no global coordinates in the
>>  protocol. All coordinates that clients deal with, are relative to
>>  some wl_surface. (This also allows the compositor to do arbitrary
>>  transformations on surfaces, because there is no need for clients
>>  to know about them, and so no need to express transformations in
>>  the protocol.)
>>  
>>  Sub-surfaces do not change that design.
>>  
>>  Another thing is that with sub-surface positioning, the information
>>  flows strictly into one direction, and we use wl_surface.commit
>>  request on the parent surface to synchronize everything. That means
>>  that a client can manipulate a whole tree of sub-surfaces, and
>>  *guarantee* an atomic, flicker-free, glitch-free update on screen.
>>  
>>  If one needed IPC between a compositor and a window manager, you
>>  would either risk visual glitches as compositor first draws one
>>  thing before the window manager says otherwise, or jerky compositor
>>  performance as it needs to wait for the window manager to respond
>>  before it can draw anything. I don't see any way around that.
>>  
>>> - Interface ivi_client is just a reinvention of wl_compositor and
>>> wl_subcompositor.
>>> - Interface ivi_surface is a reinvention of wl_surface.
>>> 
>>> Yes, I see there are some details to may want to control like
>>> surface opacity, that the current Wayland protocols do not support,
>>> but I don't think that replacing everything is a good way to start.
>>> 
>>> It is also very hard to see how objects from all these interfaces
>>> are created, and how (if?) they associate to any other protocol
>>> objects.
>>> 
>>> Btw. if you need support for surface scaling and cropping, there
>>> have been discussion on the Wayland mailing list to bring a crop &
>>> scale protocol extension to Wayland. It is actually necessary for
>>> efficient video playback etc., so pushing that forward would be
>>> nice.
>>> 
>>> 
>>  Thank you. I will check.
>>  
>>  Search for "wl_scaler", that was the working name of a proposal.
>>  
>>> After looking through the two links you gave, the ivi-shell.xml,
>>> and what you have wrote in the emails, I still have no clue what is
>>> the big picture here.
>>> 
>>> 
>>  I will draw a pciture to explain them. I will mail it back later.
>> Hi,
>> 
>>  I am enclosing a pdf to show relationship compositor, surface, layer,
>> and controller.
>> 
>>  As Michael Schuldt says in reply, only a process will control
>> attribute of surfaces and layers like position, visivility, order, and
>> so on.
>>  The controller process is kind of central one to manage policy of
>> layout and status of surfaces and layers by taking account into
>> infomation in vehicle. For example, when gear position is rear, rear
>> view camera would be set to the top of order and visible. When I see
>> your comment,
>>  I think wl_subsurface can be applied to layer and parent can be
>> mapped to layer. Now I am considering it. However, I have one use case
>> for ivi to control attribute of wl_surface from outside, e.g set
>> invisible to TV screen when speed is e.g 10km/s for safty. it is not
>> welcome for users but I have to consider it. Each application can do
>> it but on the safty point of view, it shall be handle in cetranl
>> controller.
>> 
>>  Addtionally, As Michael Schuldt says in reply as well, for debugging
>> purpose, controlling wl_surface would be required.
>> 
>>  When I see my picture by myself, ivi-shell will be window manager in
>> side of weston. So it might not be conflict be concept of shell.
>> 
>>  Please give me your feedback.
>>  
>>  Best regards,
>>  Tanibata
>>  
>>> Cool, thanks.
>>> 
>>> I saw the terms surface, layer, etc. in the IVI docs but I didn't
>>> really get what they are used for.
>>> 
>>>> - What processes are going to use which interfaces? It looks to me
>>>> like some interfaces are not meant for all Wayland clients, but
>>>> how is it supposed to work?
>>>> 
>>>> - What components are in a whole IVI system, from the point of view
>>>> of Wayland protocol? What are the responsibilities of each component
>>>> and how are these distributed into processes?
>>>> 
>>>> - What does a typical IVI application do in terms of Wayland
>>>> protocol? Are you using wl_compositor at all? Or any other
>>>> Wayland core interfaces?
>>>> 
>>>> These are just few questions to get you oriented on what kind of
>>>> things puzzle me here. Obviously, I have never been in touch with
>>>> Genivi stuff before, and I would assume most here have not either.
>>>> 
>>>> The protocol you propose seems to have many references to
>>>> "id_native" and "native content", what is all this "native" stuff
>>>> about? Or all the integer id's you seem to be sending back and
>>>> forth, why can't you use real protocol objects to refer to those?
>>>> 
>>>> 
>>>> The above is the first impression from someone, who does not know
>>>> anything about the IVI architecture, but is fairly familiar with
>>>> Wayland. Sorry if it came out harsh, but I feel like I totally
>>>> missed the whole background of this proposal: why design it like
>>>> that?
>>>> 
>>> 
>>> Thank you again for good feedback. They are very helpfull for me.
>>> 
>>> Thank you,
>>> pq
>>> _______________________________________________
>>> wayland-devel mailing list
>>> wayland-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>>> 
>>  _______________________________________________
>>  wayland-devel mailing list
>>  wayland-devel at lists.freedesktop.org
>>  http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>> 
>> 
>> Links: ------ [1]
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [2]
>> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
>> [3] http://lists.freedesktop.org/archives/wayland-devel/2013-
>> August/010720.html [4]
>> http://cgit.freedesktop.org/wayland/weston/tree/protocol [5]
>> http://projects.genivi.org/ivi-layer-management/node/13 [6]
>> 
> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
> _______________________________________________ wayland-devel mailing
> list wayland-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/wayland-devel
-------------------------------------------------------
BMW Group
Michael Schuldt
Plattformtechnol. Infotainment-Kompon.
Max-Diamand-Straße 13
80788 München

Mailto: michael.schuldt at bmw.de
URL: http://www.bmwgroup.com 
-------------------------------------------------------
Bayerische Motoren Werke Aktiengesellschaft
Vorstand: Norbert Reithofer, Vorsitzender,
Milagros Caiña Carreiro-Andree, Herbert Diess,
Klaus Draeger, Friedrich Eichiner, Harald Krüger,
Ian Robertson, Peter Schwarzenbauer.
Vorsitzender des Aufsichtsrats: Joachim Milberg
Sitz und Registergericht: München HRB 42243
--------------------------------------------------------

More information about the wayland-devel mailing list