[PATCH weston 0/6] ivi-shell proposal
nobuhiko_tanibata
nobuhiko_tanibata at xddp.denso.co.jp
Tue Sep 10 03:24:21 PDT 2013
Hi Jason,
Thank you very much for feedback.
> Michale & Nobuhiko,
>
> First of all, thank you for the clarification and thank you for
> sending this to the list and being willing to work with the FOSS
> community to try and make a standard. I'm sorry that this reply is not
> inline. I think that would get disorganized and more confusing but
> I'll try to hit everything I saw.
>
> The first distinction that needs to be made (and I think Pekka was
> trying to hint at this) is what should be standardized. If you look at
> the current [wayland core protocol][1] there is only one shell
> protocol called wl_shell. I have proposed another which will probably
> get called [wl_fullscreen_shell][2]. Both of these have something in
> common: they are purely client-side. There is nothing whatsoever in
> the standard about managing surfaces. I think that we should focus on
> what you have designated ivi_client.
>
> What about the controller? If you look in the [weston protocol
> folder][3] you will find a number of different protocol files. Some of
> these are for experimental extensions such as subsurfaces which have
> not yet made it into wayland core. However, a number of them such as
> desktop-shell, screenshooter, etc. will *never* be standardized in the
> wayland core. These protocols are completely internal to weston and
> are considered implementation details. The primary example is
> desktop-shell. This protocol exists for the purpose of allowing the
> out-of-process shell controller manage surfaces similar to what you
> propose with ivi_shell. There are other shell plugins for weston
> (hawaii & orbital) that each have their own shell plugin and can have
> their own protocol for talking to an out-of-process controller.
>
> How does this impact your proposed protocol? Unless you are convinced
> that every single IVI system manufacturer will want to manage surfaces
> the same way, the controller should be left as a private
> implementation detail. You are free to do it out-of-process and talk
> the wayland protocol to do so (desktop-shell does) but there is no
> need to expose it as part of a standard protocol. By only
> standardizing the client interface you leave app developers (GPS,
> Media players, etc.) free to design their apps however they want and
> you leave IVI system manufacturers free to handle those clients and
> surfaces in whatever way they want.
>
> Ok, now on to actual suggestions. >From this point forward, I am going
> to completely ignore the controller side of things.
>
> First, I would propose to follow the pattern of wl_shell and make two
> interfaces for clients to talk to the compositor. For now, I am going
> to call them wl_ivi_shell and wl_ivi_surface. We can come up with
> different names if you'd like, but those seem reasonable. If we follow
> the pattern of wl_shell, wl_ivi_shell will probably exist for the sole
> purpose of creating wl_ivi_surface objects. This pattern is common in
> the protocol (wl_shell, wl_subcompositor, wl_compositor, etc.).
>
> The main question, then, becomes what to put in wl_ivi_surface. I'm
> not 100% sure what you intend with some of this surface and layer
> stuff, so I'm afraid I don't have a whole lot of specific suggestions
> on that for now. I do, however have some general thoughts and
> questions:
>
> First, I agree with Pekka that you can probably avoid the layers
> thing by simply using subsurfaces.
>
I see. However, we have a use case that several application, different
process share a layer. E.g. Navigation map and Route guidance are
separated into other application. It may kind of grouping parent
surfaces.
> Second, Why are you specifying pixel formats in ivi_surface? Is the
> compositor supposed to tell the client what format to render in?
>
> Third, concerning the "visibility" flag. The wayland protocol as it
> currently stands tries to avoid telling clients specifically whether
> or not they are visible and where they are on screen. This is because,
> when clients abuse this information, compositors lose the freedom to
> throw surfaces around how they want. Instead of a visibility flag, the
> wl_surface interface provides a "frame" callback that the clients can
> use to know when was the last time they were drawn to the screen. A
> client should throttle rendering based on these frame events. If the
> surface is offscreen and the compositor wants the client to stop
> rendering it simply stop sending it frame events and the client will
> stop drawing.
>
I have two concerning to use "frame" for realizing invisible.
- To set invisible, application needs to call clear color by itself. I
think it might be overhead for GPU. If we can realize it in shell, it
simply skips it to be composite. Of course, application shall stop
drawing as well.
- Invisible shall be conrolled by cetral controller due to safy
reason.It shall be done in lower part as much as possible. Ideally, if
we can allocate it to another physical plane, it may be best. If Display
contoller doesn't support it, next is compositor.
> Once again, thank you for mailing the list. I hope my thoughts above
> are helpful and can clear a few things up.
> Thanks,
>
> --Jason Ekstrand
>
> [1]:
> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
> [2]
> [2]:
> http://lists.freedesktop.org/archives/wayland-devel/2013-August/010720.html
> [3]
> [3]: http://cgit.freedesktop.org/wayland/weston/tree/protocol [4]
>
> On Mon, Sep 9, 2013 at 7:59 PM, Nobuhiko Tanibata
> <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
>
> ----- Original Message ----- From: "Pekka Paalanen"
> <ppaalanen at gmail.com>
>
> To: "nobuhiko_tanibata" <nobuhiko_tanibata at localhost.xddp.denso.co.jp>
> Cc: "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp>;
> <securitycheck at denso.co.jp>; <wayland-devel at lists.freedesktop.org>
> Sent: Sunday, September 08, 2013 6:31 PM
>
> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
>
> On Sun, 08 Sep 2013 00:13:55 +0900
> nobuhiko_tanibata <nobuhiko_tanibata at localhost.xddp.denso.co.jp>
> wrote:
>
> 2013-09-06 19:16 に Pekka Paalanen さんは書きました:
> > On Fri, 6 Sep 2013 08:39:24 +0900
> > "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
> >
> >> ----- Original Message -----
> >> From: "Pekka Paalanen" <ppaalanen at gmail.com>
> >> To: "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp>
> >> Cc: <wayland-devel at lists.freedesktop.org>;
> <securitycheck at denso.co.jp>
> >> Sent: Thursday, September 05, 2013 10:02 PM
> >> Subject: Re: [PATCH weston 0/6] ivi-shell proposal
> >>
> >>
> >> > On Wed, 4 Sep 2013 09:08:26 +0900
> >> > "Nobuhiko Tanibata" <NOBUHIKO_TANIBATA at xddp.denso.co.jp> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> This series implements ivi-shell to fulfill use cases of
> In-Vehicle
> >> >> Infotainment, IVI. Such use cases are well overviewed in a
> project;
> >> >> Genivi IVI layer management.
> >> >> http://projects.genivi.org/ivi-layer-management/node/13 [5]
> >> >>
> >> >> A motivation of this series and basis idea are introduced by
> Ossama
> >> >> at Automotive Linux Summit 2012 spring. The series implements
> >> >> ivi-shell part. Additionally, GENIVI LM Client Library at slide
> 20 >> >> is
> >> >> contributed to ivi-layer-management project to support
> compatible
> >> >> interfaces for Genivi Layer management users.
> >> >>
> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
> [6]
> >> >>
> >> >> Before I start implementation of ivi-shell, Core members of
> Genivi
> >> >> IVI layer management defined draft of ivi-shell.xml to fulfill
> >> >> requirements of IVI layer management, inviting Kristian. The
> series
> >> >> also includes the ivi-shell.xml with updates I faced in actual
> >> >> implementation.
> >> >>
> >> >> Please give me any suggestions.
> >> >
> >> > Hi,
> >> >
> >> > I have understood that the whole design has been cut deep into
> stone >> > a
> >> > long time ago. What are you able to change now? I do not think
> it is
> >> > worth commenting on things you can no longer change, so what
> aspects >> > are
> >> > you looking suggestions for?
> >> >
> >>
> >> Hi,
> >>
> >> I would like to get feedback about that if somebody has a similar
> >> motivation
> >> to support ivi as well as desktop and tablet.
> >> So it is not a stone, just a proposal. If somebody has good idea,
> I
> >> would
> >> like to use it or collaborate it.
> >
> > Ok, I just had the understanding that the layer manager simply has
> to
> > be a separate process and not built into the compositor. If that is
> > not the case, then that is very good news indeed. Everything that
> > manages surfaces, layers, windows, or whatever belongs into the
> > compositor process, where they are much easier to implement and you
> > don't need to introduce interfaces and IPC which are later hard to
> > develop further and cause latencies. Roundtrips and synchronous
> > calls between processes can become a difficult bottle-neck, as X11
> > has taught us. Also having too many processes becomes a real mess
> > when trying to avoid deadlocks but still keep things coherent and
> > glitch-free (see X11 server vs. window manager vs. ...).
> > So I'm roughly on the same track as Andreas Pokorny.
> >
> > Weston should become the window manager and layer manager, while
> > weston backends deal with the hardware details of compositing. For
> > example, the Raspberry Pi backend of Weston forwards all
> > compositing to the VideoCore firmware, unlike any other backend
> > who actually render a composite themselves. However, the rpi
> > backend does not forward all surfaces to VideoCore all the time,
> > but only the visible ones as needed. And Weston is the only
> > component that *can* know what is visible at any time, since Weston
> > contains the complete scene graph. Weston's internal architecture
> > is also well-suited for *automatically* deciding how to use the
> > limited hardware resources like overlays efficiently, and fall back
> > as necessary, per each output frame. You cannot do that, if you
> > tell applications about specific hardware resources.
>
> Yes, my understind is the same as yours.
>
> Great. :-)
>
> And I expects Weston backend
> you mentioned next paragraph is ideally released as reference from
> SoC
> vender.
> For exmple, some SoC vendor supports 2D blit engine which can
> effciently
> composite surfaces and relese weston backend as reference.
>
> Tbh, I doubt vendor's abilities in producing a proper Weston
> backend, and maintaining it as Weston changes. But yeah, something
> like that.
>
>> I suspect there might be some terminology differences here.
> > Something like what IVI calls a "compositor" is a "weston backend"
> > when talking about Weston and Wayland, and IVI layer manager is
> > actually a window manager which is just a shell plugin to Weston.
> > Or am I completely off?
> >
>
> Yes, you are correct. And I focus on shell part to realize ivi
> requirements.
>
> Shell is usually the part which is primarily used by applications,
> not
> supporting components of a graphical environment (see wl_shell vs.
> desktop_shell protocol interfaces). I mean the public part of
> shell, like the wl_shell interface for desktop applications.
>
> Which brings me to another question: how likely is it that you
> actually want to support PC desktop applications unmodified
> directly on an IVI system? And I mean on the main compositor of an
> IVI system, which I would imagine to be quite a critical component.
>
> I think native IVI applications could be different enough to PC
> desktop
> applications, that creating a new shell interface to *replace*
> wl_shell (or whatever shell interface we will be using in the
> future for PC desktop) would be a right choice.
>
> If you actually do want to support PC desktop applications, I could
> see you having another, nested compositor just for supporting PC
> applications, which could run on a more restricted environment and
> maybe access to a more complex GPU which would be too risky for
> the main compositor in IVI. A sort of "untrusted" domain.
>
> Well, Genivi probably has already designed all that, so I'm just
> reinventing the wheel badly here.
>
>> Have you tried to map your IVI concepts of surface/layer/display to
> > Wayland wl_surface, wl_subsurface, and wl_output? I don't really
> > see what kind of interfaces your applications (Wayland clients?)
> > expect to use.
>
> Yes. As you comment, some use case; visibility and crop/scaling is
> not
> supported now.
> So I thought starting new set of protocal to cover ivi requiremnts
> would
> be better.
> But I will re-consider them and mail it back.
>
> Right. I replied from purely FOSS point of view. You probably have
> time and money deadlines which you must meet while creating a
> self-standing product, and in that case, I do understand going with
> a big re-invention. I understand it, but I'm not happy about it,
> although if you can publish your ad hoc approach like you do here,
> you are contributing valuable experience to the community.
> Especially, if you say something about the shortcomings of the
> design and use experiences.
>
> I'm very happy to see this proposal on the mailing list, even when
> I do not agree with it.
>
>> When I look at the protocol in ivi-shell.xml, I get the feeling
> that:
> > - Interfaces ivi_layer, ivi_controller_surface,
> > ivi_controller_layer, ivi_controller_screen, and ivi_controller
> > should be internal implementation details inside the weston
> > process, not protocol. Having these as interfaces looks like the
> > X architecture, where the X server process and window manager
> > process continuously struggle to keep each other up-to-date, and
> > carefully try to keep state in sync (and fail), which also makes
> > races and glitches practically unavoidable.
>
> I have basic question to the above; strugling. wl_subsurface supports
> set_poistion now.
>
> set_position for sub-surfaces is *always* relative to the parent
> surface. The sub-surface position is given as a point on the parent
> surface. You still have no control where on the output any surface
> will appear, because you cannot control where the root surface of a
> tree of sub-surfaces will appear.
>
> If a parent surface is moved on screen, all its sub-surfaces stay
> glued to it automatically without any client interaction.
>
> I thought it implies positioning from a windowmanger is allowed on
> weston basic concept.
>
> >From a window manager, yes, but this assumes that the window manager
> is in-process with weston; the window manager must be a compositor
> plugin. Making it an external process will be a huge amount of
> trouble and performance loss.
>
> And the protocol you propose seems to be for an external window
> manager process, from my understanding.
>
> Each other up-to-date may occurs from window manager and weston
> internl
> decision.
> Or positioning of sub surfaces is out of scope of weston and it just
> composite them according to attributes.
> I may be wrong. Please let me know history.
>
> The first thing is that in Wayland core protocol, there is no
> global positioning system. There are no global coordinates in the
> protocol. All coordinates that clients deal with, are relative to
> some wl_surface. (This also allows the compositor to do arbitrary
> transformations on surfaces, because there is no need for clients
> to know about them, and so no need to express transformations in
> the protocol.)
>
> Sub-surfaces do not change that design.
>
> Another thing is that with sub-surface positioning, the information
> flows strictly into one direction, and we use wl_surface.commit
> request on the parent surface to synchronize everything. That means
> that a client can manipulate a whole tree of sub-surfaces, and
> *guarantee* an atomic, flicker-free, glitch-free update on screen.
>
> If one needed IPC between a compositor and a window manager, you
> would either risk visual glitches as compositor first draws one
> thing before the window manager says otherwise, or jerky compositor
> performance as it needs to wait for the window manager to respond
> before it can draw anything. I don't see any way around that.
>
>> - Interface ivi_client is just a reinvention of wl_compositor and
> > wl_subcompositor.
> > - Interface ivi_surface is a reinvention of wl_surface.
> >
> > Yes, I see there are some details to may want to control like
> > surface opacity, that the current Wayland protocols do not support,
> > but I don't think that replacing everything is a good way to start.
> >
> > It is also very hard to see how objects from all these interfaces
> > are created, and how (if?) they associate to any other protocol
> > objects.
> >
> > Btw. if you need support for surface scaling and cropping, there
> > have been discussion on the Wayland mailing list to bring a crop &
> > scale protocol extension to Wayland. It is actually necessary for
> > efficient video playback etc., so pushing that forward would be
> > nice.
> >
>
> Thank you. I will check.
>
> Search for "wl_scaler", that was the working name of a proposal.
>
>> After looking through the two links you gave, the ivi-shell.xml,
> > and what you have wrote in the emails, I still have no clue what is
> > the big picture here.
> >
>
> I will draw a pciture to explain them. I will mail it back later.
>
> Hi,
>
> I am enclosing a pdf to show relationship compositor, surface, layer,
> and controller.
>
> As Michael Schuldt says in reply, only a process will control
> attribute of surfaces and layers like position, visivility, order, and
> so on.
> The controller process is kind of central one to manage policy of
> layout and status of surfaces and layers by taking account into
> infomation in vehicle. For example, when gear position is rear, rear
> view camera would be set to the top of order and visible. When I see
> your comment,
> I think wl_subsurface can be applied to layer and parent can be
> mapped to layer. Now I am considering it. However, I have one use case
> for ivi to control attribute of wl_surface from outside, e.g set
> invisible to TV screen when speed is e.g 10km/s for safty. it is not
> welcome for users but I have to consider it. Each application can do
> it but on the safty point of view, it shall be handle in cetranl
> controller.
>
> Addtionally, As Michael Schuldt says in reply as well, for debugging
> purpose, controlling wl_surface would be required.
>
> When I see my picture by myself, ivi-shell will be window manager in
> side of weston. So it might not be conflict be concept of shell.
>
> Please give me your feedback.
>
> Best regards,
> Tanibata
>
>> Cool, thanks.
>>
>> I saw the terms surface, layer, etc. in the IVI docs but I didn't
>> really get what they are used for.
>>
>>> - What processes are going to use which interfaces? It looks to me
>>> like some interfaces are not meant for all Wayland clients, but
>>> how is it supposed to work?
>>>
>>> - What components are in a whole IVI system, from the point of
>> view
>>> of Wayland protocol? What are the responsibilities of each
>>> component and how are these distributed into processes?
>>>
>>> - What does a typical IVI application do in terms of Wayland
>>> protocol? Are you using wl_compositor at all? Or any other
>>> Wayland core interfaces?
>>>
>>> These are just few questions to get you oriented on what kind of
>>> things puzzle me here. Obviously, I have never been in touch with
>>> Genivi stuff before, and I would assume most here have not
>> either.
>>>
>>> The protocol you propose seems to have many references to
>>> "id_native" and "native content", what is all this "native" stuff
>>> about? Or all the integer id's you seem to be sending back and
>>> forth, why can't you use real protocol objects to refer to those?
>>>
>>>
>>> The above is the first impression from someone, who does not know
>>> anything about the IVI architecture, but is fairly familiar with
>>> Wayland. Sorry if it came out harsh, but I feel like I totally
>>> missed the whole background of this proposal: why design it like
>>> that?
>>>
>>
>> Thank you again for good feedback. They are very helpfull for me.
>>
>> Thank you,
>> pq
>> _______________________________________________
>> wayland-devel mailing list
>> wayland-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/wayland-devel [1]
>
>
>
> Links:
> ------
> [1] http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> [2]
> http://cgit.freedesktop.org/wayland/wayland/tree/protocol/wayland.xml
> [3]
> http://lists.freedesktop.org/archives/wayland-devel/2013-August/010720.html
> [4] http://cgit.freedesktop.org/wayland/weston/tree/protocol
> [5] http://projects.genivi.org/ivi-layer-management/node/13
> [6]
> http://events.linuxfoundation.org/images/stories/pdf/als2012_othman.pdf
More information about the wayland-devel
mailing list