[PATCH RFC wayland-protocols] unstable/linux-dmabuf: add wp_linux_dmabuf_device_hint

[Simon Ser]

On Monday, November 12, 2018 10:18 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> On Sat, 10 Nov 2018 13:34:31 +0000
> Simon Ser <contact at emersion.fr> wrote:
>
> > On Monday, November 5, 2018 9:57 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> > > > > Yeah. Another option is to send a wl_array of modifiers per format and
> > > > > tranch.
> > > >
> > > > True. Any reason why this hasn't been done in the global?
> > >
> > > For formats? Well, it is simpler without a wl_array, and there might be
> > > a lot of formats.
> > >
> > > Could there be a lot of modifiers per format? Would a wl_array make
> > > anything easier? Just a thought.
> >
> > It's true that for this list of formats sorted by preference, we'll probably
> > need to split modifiers anyway so I don't think we'd benefit a lot from this
> > approach.
>
> Hi Simon,
>
> just to be clear, I was thinking of something like:
>
> event(uint format, wl_array(modifiers))
>
> But I definitely do not insist on it if you don't see any obvious
> benefits with it.

Yeah. I think the benefits would not be substantial as we need to "split" these
to order them by preference. So it would look like so:

  event(format1, wl_array(modifiers))
  barrier()
  event(format1, wl_array(modifiers))
  event(format2, wl_array(modifiers))
  barrier()
  event(format1, wl_array(modifiers))
  barrier()

Also this is not consistent with the rest of the protocol. Maybe we can discuss
this again for linux-dmabuf-unstable-v2.

> It seems you and I made very different assumptions on how the hints
> would be sent, I only realized it just now. More about that below.
>
> > > > > I suppose it will be enough to send tranches for just the currently
> > > > > used format? Otherwise it could be "a lot" of data.
> > > >
> > > > What do you mean by "the currently used format"?
> > >
> > > This interface is used to send clients hints after they are already
> > > presenting, which means they already have a format chosen and probably
> > > want to stick with it, just changing the modifiers to be more optimal.
> >
> > If we only send the modifiers for the current format, how do clients tell the
> > difference between the initial hints (which don't have a "currently used
> > format") and the subsequent hints?
>
> I'm not sure I understand why they would need to see the difference.
> But yes, I was short-sighted here and didn't consider the
> initialization when a surface is not mapped yet. I didn't expect that
> hints can be calculated if the surface is not mapped, but of course a
> compositor can provide some defaults. I suppose the initial default
> hints would boil down to what is most efficient to composite.
>
> > > > I expect clients to bind to this interface and create a surface hints object
> > > > before the surface is mapped. In this case there's no "currently used format".
> > >
> > > Right, that's another use case.
> > >
> > > > It will be a fair amount of data, yes. However it's just a list of integers.
> > > > When we send strings over the protocol (e.g. toplevel title in xdg-shell) it's
> > > > about the same amount of data I guess.
> > >
> > > If the EGLConfig or GLXFBConfig or GLX visual lists are of any
> > > indication... yes, they account for depth, stencil, aux, etc. but then
> > > we will have modifiers.
> > >
> > > We already advertise the list of everything supported of format+modifer
> > > in the linux_dmabuf extension. Could we somehow minimize the number of
> > > recommended format+modifiers in hints? Or maybe that's not a concern
> > > for the protocol spec?
> >
> > I'm not sure.
> >
> > After this patch, I'm not even sure how the formats+modifiers advertised by the
> > global work. Are these formats+modifiers supported on the GPU the compositor
> > uses for rendering? Intersection or union of formats+modifiers supported on all
> > GPUs?
>
> The format+modifier advertised by the global before this patch are the
> ones that can work at all, or the compositor is willing to make them
> work at least in the worst fallback case. This patch must not change
> that meaning. These formats also must always work regardless of which
> GPU a client decides to use, but that is already implied by the
> compositor being able to import a dmabuf. The compositor does not need
> to try to factor in what other GPUs on the system might be able to
> render or not, that is for the client to figure out when it knows the
> formats the compositor can accept and is choosing a GPU to render with.
> It is theoretically possible that a client tries to use a GPU that
> cannot render any formats the compositor can use, but that is the
> client's responsibility to figure out.

Okay, that makes sense. And if a GPU doesn't support direct scan-out for some
format+modifier, then it can always fallback to good ol' compositing.

> So clearly the formats from the global can be used by a client at any
> time. The hint formats OTOH has no reason to list absolutely
> everything the compositor supports, but a compositor can choose on its
> own judgement to send only a sub-set it would prefer.

Yes, this makes sense.

> However, after a client has picked a format and used it, then there
> should be hints with that format, at least if they can make any
> difference.

Okay, I get it now. :)

> I'm not sure. Not listing everything always was my intuitive
> assumption, and I believe you perhaps assumed the opposite so that a
> client has absolutely all the information to e.g. optimize the modifier
> of a format that the compositor would not prefer at all even though it
> does work.
>
> It would be simpler to always send everything, but that will be much
> more protocol traffic. Would it be too much? I don't know, could you
> calculate some examples of how many bytes a typical hints update would
> be if sending everything always?

If I'm understanding the protocol marshalling right, each event would have:

* A 2*32-bit header for sender, size and opcode
* One 32-bit field for the format
* A 2*32-bit field for the modifier

Total 5*32 = 160 bits, ie. 20 bytes per event.

My current setup lists ~100 format+modifier combinations. So that means each
time a client binds to wp_linux_dmabuf, ~2KB of data is sent. That's a lot, your
concerns are correct.

(In a future version of the protocol, maybe we could use shared memory, just
like the wl_keyboard keymap.)

In the meantime, we could decide to do as you suggest. So the compositor would
always advertise a subset of the supported modifiers. When the hints object is
created, the compositor would send its preferred format+modifier pairs. When the
client submits a buffer with a new format, the compositor can decide to send the
preferred modifiers for this format. I wonder how we should phrase this in the
protocol (can/should/must?). Thoughts?

In the case where a client is scanned out and the GPU used for scan-out
supports more formats than the GPU used for compositing, as discussed before the
compositor won't be able to advertise these additional formats, because the
client could keep using them when not scanned out anymore.