[PATCH weston 0/6] ivi-shell proposal

nobuhiko_tanibata nobuhiko_tanibata at xddp.denso.co.jp
Tue Sep 10 08:49:42 PDT 2013 

2013-09-10 20:13 に Michael.Schuldt at bmw.de さんは書きました:
> Hi Jason, Tanibata-san
> 
>> Hi Jason,
>> 
>> Thank you very much for feedback.
>> 
>>> Michale & Nobuhiko,
>>> 
>>> First of all, thank you for the clarification and thank you for
>>> sending this to the list and being willing to work with the FOSS
>>> community to try and make a standard. I'm sorry that this reply is 
>>> not
>>> inline. I think that would get disorganized and more confusing but
>>> I'll try to hit everything I saw.
>>> 
>>> The first distinction that needs to be made (and I think Pekka was
>>> trying to hint at this) is what should be standardized. If you look 
>>> at
>>> the current [wayland core protocol][1] there is only one shell
>>> protocol called wl_shell. I have proposed another which will probably
>>> get called [wl_fullscreen_shell][2]. Both of these have something in
>>> common: they are purely client-side. There is nothing whatsoever in
>>> the standard about managing surfaces. I think that we should focus on
>>> what you have designated ivi_client.
> 
> Ok, now I got it. No we do not want to push it to the core protocol. We 
> want
> to define a IVI-Shell like desktop shell and want to use weston for
> the reference
> implementation. The ivi-shell extension should be the minimum subset
> which shall provided
> by each ivi-compositor. But in terms of the genivi compliance we are
> able to define
> a shell protocol and say this is the minimal subset which is
> supported, and therefore
> we need a reference implementation too. Maybe we have mixed something
> here from our side.
> We will clarify that
> 

Hi Jason,

The motivation from me is the same as the above as MichaelSchuldt says. 
It would not be pushed to core protocol.

Best regards,
Tanibata

>>> What about the controller? If you look in the [weston protocol
>>> folder][3] you will find a number of different protocol files. Some 
>>> of
>>> these are for experimental extensions such as subsurfaces which have
>>> not yet made it into wayland core. However, a number of them such as
>>> desktop-shell, screenshooter, etc. will *never* be standardized in 
>>> the
>>> wayland core. These protocols are completely internal to weston and
>>> are considered implementation details. The primary example is
>>> desktop-shell. This protocol exists for the purpose of allowing the
>>> out-of-process shell controller manage surfaces similar to what you
>>> propose with ivi_shell. There are other shell plugins for weston
>>> (hawaii & orbital) that each have their own shell plugin and can have
>>> their own protocol for talking to an out-of-process controller.
> 
> Yes this is our focus too, like I have described above.
> 
>>> How does this impact your proposed protocol? Unless you are convinced
>>> that every single IVI system manufacturer will want to manage 
>>> surfaces
>>> the same way, the controller should be left as a private
>>> implementation detail. You are free to do it out-of-process and talk
>>> the wayland protocol to do so (desktop-shell does) but there is no
>>> need to expose it as part of a standard protocol. By only
>>> standardizing the client interface you leave app developers (GPS,
>>> Media players, etc.) free to design their apps however they want and
>>> you leave IVI system manufacturers free to handle those clients and
>>> surfaces in whatever way they want.
>>> 
>>> Ok, now on to actual suggestions. >From this point forward, I am 
>>> going
>>> to completely ignore the controller side of things.
>>> 
>>> First, I would propose to follow the pattern of wl_shell and make two
>>> interfaces for clients to talk to the compositor. For now, I am going
>>> to call them wl_ivi_shell and wl_ivi_surface. We can come up with
>>> different names if you'd like, but those seem reasonable. If we 
>>> follow
>>> the pattern of wl_shell, wl_ivi_shell will probably exist for the 
>>> sole
>>> purpose of creating wl_ivi_surface objects. This pattern is common in
>>> the protocol (wl_shell, wl_subcompositor, wl_compositor, etc.).
>>> 
>>> The main question, then, becomes what to put in wl_ivi_surface. I'm
>>> not 100% sure what you intend with some of this surface and layer
>>> stuff, so I'm afraid I don't have a whole lot of specific suggestions
>>> on that for now. I do, however have some general thoughts and
>>> questions:
>>> 
>>>  First, I agree with Pekka that you can probably avoid the layers
>>> thing by simply using subsurfaces.
>>> 
>> I see. However, we have a use case that several application, different
>> process share a layer. E.g. Navigation map and Route guidance are
>> separated into other application. It may kind of grouping parent
>> surfaces.
>> 
>>> Second, Why are you specifying pixel formats in ivi_surface? Is the
>>> compositor supposed to tell the client what format to render in?
>>> 
>>> Third, concerning the "visibility" flag. The wayland protocol as it
>>> currently stands tries to avoid telling clients specifically whether
>>> or not they are visible and where they are on screen. This is 
>>> because,
>>> when clients abuse this information, compositors lose the freedom to
>>> throw surfaces around how they want. Instead of a visibility flag, 
>>> the
>>> wl_surface interface provides a "frame" callback that the clients can
>>> use to know when was the last time they were drawn to the screen. A
>>> client should throttle rendering based on these frame events. If the
>>> surface is offscreen and the compositor wants the client to stop
>>> rendering it simply stop sending it frame events and the client will
>>> stop drawing.
>>> 
>> 
>> I have two concerning to use "frame" for realizing invisible.
>> - To set invisible, application needs to call clear color by itself. I
>> think it might be overhead for GPU. If we can realize it in shell, it
>> simply skips it to be composite. Of course, application shall stop
>> drawing as well.
>> - Invisible shall be conrolled by cetral controller due to safy
>> reason.It shall be done in lower part as much as possible. Ideally, if
>> we can allocate it to another physical plane, it may be best. If 
>> Display
>> contoller doesn't support it, next is compositor.
> 
> Here I am full inline with the post from Tanibata-san
> 
> Regards
> 
> Michael.
>>> Once again, thank you for mailing the list. I hope my thoughts above
>>> are helpful and can clear a few things up.
>>>  Thanks,
>>> --Jason Ekstrand
>>> 
>>>  [1]:
>>> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
>>> [2]
>>>  [2]:
>>> http://lists.freedesktop.org/archives/wayland-devel/2013-
>>> August/010720.html [3]
>>>  [3]: http://cgit.freedesktop.org/wayland/weston/tree/protocol [4]
>>> On Mon, Sep 9, 2013 at 7:59 PM, Nobuhiko Tanibata
>>> <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>>> 
>>  ppaalanen at gmail.com> wrote on o: "nobuhiko_tanibata" 
>> <nobuhiko_tanibata at localhost.xddp.denso.co.jp>:
>>>  Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>>> On Sun, 08 Sep 2013 00:13:55 +0900
>>>  nobuhiko_tanibata <nobuhiko_tanibata at localhost.xddp.denso.co.jp>
>>> wrote:
>>> 
>>> 2013-09-06 19:16 に Pekka Paalanen さんは書きました:
>>>> On Fri, 6 Sep 2013 08:39:24 +0900
>>>> "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>>>> 
>>>> "Pekka Paalanen" <ppaalanen at gmail.com> wrote on o: "Nobuhiko 
>>>> Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp>:
>>>>> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>>>>> 
>>>>> 
>>>>>> On Wed, 4 Sep 2013 09:08:26 +0900
>>>>>> "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> This series implements ivi-shell to fulfill use cases of 
>>>>>>> In-Vehicle
>>>>>>> Infotainment, IVI. Such use cases are well overviewed in a 
>>>>>>> project;
>>>>>>> Genivi IVI layer management.
>>>>>>> http://projects.genivi.org/ivi-layer-management/node/13 [5]
>>>>>>> 
>>>>>>> A motivation of this series and basis idea are introduced by 
>>>>>>> Ossama
>>>>>>> at Automotive Linux Summit 2012 spring. The series implements
>>>>>>> ivi-shell part. Additionally, GENIVI LM Client Library at slide 
>>>>>>> 20
>>>>>>> >> >> is contributed to ivi-layer-management project to support
>>>>>>> compatible interfaces for Genivi Layer management users.
>>>>>>> 
>>> 
>>> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
>>> [6]
>>>>>>> 
>>>>>>> Before I start implementation of ivi-shell, Core members of 
>>>>>>> Genivi
>>>>>>> IVI layer management defined draft of ivi-shell.xml to fulfill
>>>>>>> requirements of IVI layer management, inviting Kristian. The 
>>>>>>> series
>>>>>>> also includes the ivi-shell.xml with updates I faced in actual
>>>>>>> implementation.
>>>>>>> 
>>>>>>> Please give me any suggestions.
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I have understood that the whole design has been cut deep into 
>>>>>> stone
>>>>>> >> > a long time ago. What are you able to change now? I do not
>>>>>> think it is worth commenting on things you can no longer change, 
>>>>>> so
>>>>>> what aspects >> > are you looking suggestions for?
>>>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I would like to get feedback about that if somebody has a similar
>>>>> motivation to support ivi as well as desktop and tablet. So it is 
>>>>> not
>>>>> a stone, just a proposal. If somebody has good idea, I would like 
>>>>> to
>>>>> use it or collaborate it.
>>>> 
>>>> Ok, I just had the understanding that the layer manager simply has 
>>>> to
>>>> be a separate process and not built into the compositor. If that is
>>>> not the case, then that is very good news indeed. Everything that
>>>> manages surfaces, layers, windows, or whatever belongs into the
>>>> compositor process, where they are much easier to implement and you
>>>> don't need to introduce interfaces and IPC which are later hard to
>>>> develop further and cause latencies. Roundtrips and synchronous 
>>>> calls
>>>> between processes can become a difficult bottle-neck, as X11 has
>>>> taught us. Also having too many processes becomes a real mess when
>>>> trying to avoid deadlocks but still keep things coherent and
>>>> glitch-free (see X11 server vs. window manager vs. ...). So I'm
>>>> roughly on the same track as Andreas Pokorny.
>>>> 
>>>> Weston should become the window manager and layer manager, while
>>>> weston backends deal with the hardware details of compositing. For
>>>> example, the Raspberry Pi backend of Weston forwards all
>>>> compositing to the VideoCore firmware, unlike any other backend
>>>> who actually render a composite themselves. However, the rpi
>>>> backend does not forward all surfaces to VideoCore all the time,
>>>> but only the visible ones as needed. And Weston is the only
>>>> component that *can* know what is visible at any time, since Weston
>>>> contains the complete scene graph. Weston's internal architecture
>>>> is also well-suited for *automatically* deciding how to use the
>>>> limited hardware resources like overlays efficiently, and fall back
>>>> as necessary, per each output frame. You cannot do that, if you
>>>> tell applications about specific hardware resources.
>>>> 
>>>  Yes, my understind is the same as yours.
>>> 
>>>  Great. :-)
>>> And I expects Weston backend
>>>  you mentioned next paragraph is ideally released as reference from 
>>> SoC
>>>  vender. For exmple, some SoC vendor supports 2D blit engine which 
>>> can
>>>  effciently composite surfaces and relese weston backend as 
>>> reference.
>>> 
>>>  Tbh, I doubt vendor's abilities in producing a proper Weston
>>>  backend, and maintaining it as Weston changes. But yeah, something
>>>  like that.
>>> 
>>>> I suspect there might be some terminology differences here.
>>>> Something like what IVI calls a "compositor" is a "weston backend"
>>>> when talking about Weston and Wayland, and IVI layer manager is
>>>> actually a window manager which is just a shell plugin to Weston.
>>>> Or am I completely off?
>>>> 
>>>> 
>>>  Yes, you are correct. And I focus on shell part to realize ivi
>>>  requirements.
>>> 
>>>  Shell is usually the part which is primarily used by applications, 
>>> not
>>>  supporting components of a graphical environment (see wl_shell vs.
>>>  desktop_shell protocol interfaces). I mean the public part of shell,
>>>  like the wl_shell interface for desktop applications.
>>> 
>>>  Which brings me to another question: how likely is it that you
>>>  actually want to support PC desktop applications unmodified
>>>  directly on an IVI system? And I mean on the main compositor of an
>>>  IVI system, which I would imagine to be quite a critical component.
>>> 
>>>  I think native IVI applications could be different enough to PC
>>>  desktop applications, that creating a new shell interface to 
>>> *replace*
>>>  wl_shell (or whatever shell interface we will be using in the future
>>>  for PC desktop) would be a right choice.
>>> 
>>>  If you actually do want to support PC desktop applications, I could
>>>  see you having another, nested compositor just for supporting PC
>>>  applications, which could run on a more restricted environment and
>>>  maybe access to a more complex GPU which would be too risky for
>>>  the main compositor in IVI. A sort of "untrusted" domain.
>>> 
>>>  Well, Genivi probably has already designed all that, so I'm just
>>>  reinventing the wheel badly here.
>>> 
>>>> Have you tried to map your IVI concepts of surface/layer/display to
>>>> Wayland wl_surface, wl_subsurface, and wl_output? I don't really
>>>> see what kind of interfaces your applications (Wayland clients?)
>>>> expect to use.
>>>> 
>>>  Yes. As you comment, some use case; visibility and crop/scaling is 
>>> not
>>>  supported now. So I thought starting new set of protocal to cover 
>>> ivi
>>>  requiremnts would be better. But I will re-consider them and mail it
>>>  back.
>>> 
>>>  Right. I replied from purely FOSS point of view. You probably have
>>>  time and money deadlines which you must meet while creating a
>>>  self-standing product, and in that case, I do understand going with
>>>  a big re-invention. I understand it, but I'm not happy about it,
>>>  although if you can publish your ad hoc approach like you do here,
>>>  you are contributing valuable experience to the community.
>>>  Especially, if you say something about the shortcomings of the
>>>  design and use experiences.
>>> 
>>>  I'm very happy to see this proposal on the mailing list, even when
>>>  I do not agree with it.
>>> 
>>>> When I look at the protocol in ivi-shell.xml, I get the feeling 
>>>> that:
>>>> - Interfaces ivi_layer, ivi_controller_surface, 
>>>> ivi_controller_layer,
>>>> ivi_controller_screen, and ivi_controller should be internal
>>>> implementation details inside the weston process, not protocol. 
>>>> Having
>>>> these as interfaces looks like the X architecture, where the X 
>>>> server
>>>> process and window manager process continuously struggle to keep 
>>>> each
>>>> other up-to-date, and carefully try to keep state in sync (and 
>>>> fail),
>>>> which also makes races and glitches practically unavoidable.
>>>> 
>>>  I have basic question to the above; strugling. wl_subsurface 
>>> supports
>>>  set_poistion now.
>>> 
>>>  set_position for sub-surfaces is *always* relative to the parent
>>>  surface. The sub-surface position is given as a point on the parent
>>>  surface. You still have no control where on the output any surface
>>>  will appear, because you cannot control where the root surface of a
>>>  tree of sub-surfaces will appear.
>>> 
>>>  If a parent surface is moved on screen, all its sub-surfaces stay
>>>  glued to it automatically without any client interaction.
>>> I thought it implies positioning from a windowmanger is allowed on
>>>  weston basic concept.
>>> 
>>>> From a window manager, yes, but this assumes that the window manager
>>>> is in-process with weston; the window manager must be a compositor
>>>> plugin. Making it an external process will be a huge amount of
>>>> trouble and performance loss.
>>>> 
>>>  And the protocol you propose seems to be for an external window
>>>  manager process, from my understanding.
>>> Each other up-to-date may occurs from window manager and weston
>>> internl
>>>  decision.
>>>  Or positioning of sub surfaces is out of scope of weston and it just
>>>  composite them according to attributes.
>>>  I may be wrong. Please let me know history.
>>> 
>>>  The first thing is that in Wayland core protocol, there is no
>>>  global positioning system. There are no global coordinates in the
>>>  protocol. All coordinates that clients deal with, are relative to
>>>  some wl_surface. (This also allows the compositor to do arbitrary
>>>  transformations on surfaces, because there is no need for clients
>>>  to know about them, and so no need to express transformations in
>>>  the protocol.)
>>> 
>>>  Sub-surfaces do not change that design.
>>> 
>>>  Another thing is that with sub-surface positioning, the information
>>>  flows strictly into one direction, and we use wl_surface.commit
>>>  request on the parent surface to synchronize everything. That means
>>>  that a client can manipulate a whole tree of sub-surfaces, and
>>>  *guarantee* an atomic, flicker-free, glitch-free update on screen.
>>> 
>>>  If one needed IPC between a compositor and a window manager, you
>>>  would either risk visual glitches as compositor first draws one
>>>  thing before the window manager says otherwise, or jerky compositor
>>>  performance as it needs to wait for the window manager to respond
>>>  before it can draw anything. I don't see any way around that.
>>> 
>>>> - Interface ivi_client is just a reinvention of wl_compositor and
>>>> wl_subcompositor.
>>>> - Interface ivi_surface is a reinvention of wl_surface.
>>>> 
>>>> Yes, I see there are some details to may want to control like
>>>> surface opacity, that the current Wayland protocols do not support,
>>>> but I don't think that replacing everything is a good way to start.
>>>> 
>>>> It is also very hard to see how objects from all these interfaces
>>>> are created, and how (if?) they associate to any other protocol
>>>> objects.
>>>> 
>>>> Btw. if you need support for surface scaling and cropping, there
>>>> have been discussion on the Wayland mailing list to bring a crop &
>>>> scale protocol extension to Wayland. It is actually necessary for
>>>> efficient video playback etc., so pushing that forward would be
>>>> nice.
>>>> 
>>>> 
>>>  Thank you. I will check.
>>> 
>>>  Search for "wl_scaler", that was the working name of a proposal.
>>> 
>>>> After looking through the two links you gave, the ivi-shell.xml,
>>>> and what you have wrote in the emails, I still have no clue what is
>>>> the big picture here.
>>>> 
>>>> 
>>>  I will draw a pciture to explain them. I will mail it back later.
>>> Hi,
>>> 
>>>  I am enclosing a pdf to show relationship compositor, surface, 
>>> layer,
>>> and controller.
>>> 
>>>  As Michael Schuldt says in reply, only a process will control
>>> attribute of surfaces and layers like position, visivility, order, 
>>> and
>>> so on.
>>>  The controller process is kind of central one to manage policy of
>>> layout and status of surfaces and layers by taking account into
>>> infomation in vehicle. For example, when gear position is rear, rear
>>> view camera would be set to the top of order and visible. When I see
>>> your comment,
>>>  I think wl_subsurface can be applied to layer and parent can be
>>> mapped to layer. Now I am considering it. However, I have one use 
>>> case
>>> for ivi to control attribute of wl_surface from outside, e.g set
>>> invisible to TV screen when speed is e.g 10km/s for safty. it is not
>>> welcome for users but I have to consider it. Each application can do
>>> it but on the safty point of view, it shall be handle in cetranl
>>> controller.
>>> 
>>>  Addtionally, As Michael Schuldt says in reply as well, for debugging
>>> purpose, controlling wl_surface would be required.
>>> 
>>>  When I see my picture by myself, ivi-shell will be window manager in
>>> side of weston. So it might not be conflict be concept of shell.
>>> 
>>>  Please give me your feedback.
>>> 
>>>  Best regards,
>>>  Tanibata
>>> 
>>>> Cool, thanks.
>>>> 
>>>> I saw the terms surface, layer, etc. in the IVI docs but I didn't
>>>> really get what they are used for.
>>>> 
>>>>> - What processes are going to use which interfaces? It looks to me
>>>>> like some interfaces are not meant for all Wayland clients, but
>>>>> how is it supposed to work?
>>>>> 
>>>>> - What components are in a whole IVI system, from the point of view
>>>>> of Wayland protocol? What are the responsibilities of each 
>>>>> component
>>>>> and how are these distributed into processes?
>>>>> 
>>>>> - What does a typical IVI application do in terms of Wayland
>>>>> protocol? Are you using wl_compositor at all? Or any other
>>>>> Wayland core interfaces?
>>>>> 
>>>>> These are just few questions to get you oriented on what kind of
>>>>> things puzzle me here. Obviously, I have never been in touch with
>>>>> Genivi stuff before, and I would assume most here have not either.
>>>>> 
>>>>> The protocol you propose seems to have many references to
>>>>> "id_native" and "native content", what is all this "native" stuff
>>>>> about? Or all the integer id's you seem to be sending back and
>>>>> forth, why can't you use real protocol objects to refer to those?
>>>>> 
>>>>> 
>>>>> The above is the first impression from someone, who does not know
>>>>> anything about the IVI architecture, but is fairly familiar with
>>>>> Wayland. Sorry if it came out harsh, but I feel like I totally
>>>>> missed the whole background of this proposal: why design it like
>>>>> that?
>>>>> 
>>>> 
>>>> Thank you again for good feedback. They are very helpfull for me.
>>>> 
>>>> Thank you,
>>>> pq
>>>> _______________________________________________
>>>> wayland-devel mailing list
>>>> wayland-devel at lists.freedesktop.org
>>>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>>>> 
>>>  _______________________________________________
>>>  wayland-devel mailing list
>>>  wayland-devel at lists.freedesktop.org
>>>  http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>>> 
>>> 
>>> Links: ------ [1]
>>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [2]
>>> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
>>> [3] http://lists.freedesktop.org/archives/wayland-devel/2013-
>>> August/010720.html [4]
>>> http://cgit.freedesktop.org/wayland/weston/tree/protocol [5]
>>> http://projects.genivi.org/ivi-layer-management/node/13 [6]
>>> 
>> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
>> _______________________________________________ wayland-devel mailing
>> list wayland-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> -------------------------------------------------------
> BMW Group
> Michael Schuldt
> Plattformtechnol. Infotainment-Kompon.
> Max-Diamand-Straße 13
> 80788 München
> 
> Mailto: michael.schuldt at bmw.de
> URL: http://www.bmwgroup.com 
> -------------------------------------------------------
> Bayerische Motoren Werke Aktiengesellschaft
> Vorstand: Norbert Reithofer, Vorsitzender,
> Milagros Caiña Carreiro-Andree, Herbert Diess,
> Klaus Draeger, Friedrich Eichiner, Harald Krüger,
> Ian Robertson, Peter Schwarzenbauer.
> Vorsitzender des Aufsichtsrats: Joachim Milberg
> Sitz und Registergericht: München HRB 42243
> --------------------------------------------------------
> 
> 
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/wayland-devel

More information about the wayland-devel mailing list