[VDPAU] [PATCH 0/1] Add new frame and field mode chroma types. Add VdpDecoderQueryProfileCapability API

Thu Nov 8 06:05:32 UTC 2018

Stephen, Manoj,

I get where you guys are coming from, but I'm just looking at this
pragmatically, with respect to the media player ecosystem that I'm
aware of. You may have other more private consumers that can drive
requirements, and in that case, by all means, do what you need to do.

But as it stands, I see the following:

* mpv: As discussed, it does reinterleaving as a separate step, so
  whether the copy is done in the driver or the app makes little
  difference
* vlc: Appears to only use vdpau via the mixer, and so only interops
  with output surfaces
* kodi: Dropping vdpau support in the next release
* mplayer: Doesn't do GL interop (and is generally all kinds of
  obsolete)
* gstreamer ecosystem: Doesn't meaningfully support vdpau

So, if you were approaching this from the perspective of what this
ecosystem needs, I think you're signing up to do extra work that won't
really get used. And with that context, I say that because the enum
additions are a bit of an ugly hack, you're better off not making them.

But is it the end of the world, no not really. As I said - if you're
aware of real clients that can benefit from this, then obviously the
trade-offs are very different from what I'm describing.

Thanks,

--phil

On Tue, 6 Nov 2018 05:55:55 +0000
Manoj Bonda <mbonda at nvidia.com> wrote:

> Hi Philip, 
> 
> As stated in the interop2 spec
> (https://github.com/KhronosGroup/OpenGL-Registry/pull/223/files )
> 
> I would like to restate below points
> 
> 1) Existing chroma types, existing interop API: Always fields for
> interop, driver does field splitting if necessary.
> 
> 2) Existing chroma types, new interop API. Application can choose
> between fields/frames for interop, driver does field combining /
> field splitting if necessary.
> 
> 3) New chroma types, existing or new interop API. Driver never
> performs internal field/frame conversion, allowing app to guarantee
> it uses the most efficient path. This requires knowing the interop
> format before performing a decode so that HW that can decode to both
> formats will decode to the format that will eventually be used for
> interop. 
> So clients which are fine with driver internal field combining /
> field splitting can continue to use existing chroma types with new
> interop API to get frames for interop.
> 
> new VDP_CHROMA_TYPES only provide optimized path. Applications which
> are fine with re-interleaving step in driver can use the existing
> VDP_CHROMA_TYPES and the VDPAU implementation will perform this
> re-interleaving itself as a separate step.
> 
> Thanks,
> ManojGupta.
> 
> > -----Original Message-----
> > From: VDPAU <vdpau-bounces at lists.freedesktop.org> On Behalf Of
> > Stephen Warren
> > Sent: Tuesday, November 6, 2018 4:34 AM
> > To: Philip Langdale <philipl at overt.org>; vdpau at lists.freedesktop.org
> > Subject: Re: [VDPAU] [PATCH 0/1] Add new frame and field mode
> > chroma types. Add VdpDecoderQueryProfileCapability API
> > 
> > Philip Langdale wrote at Friday, November 2, 2018 2:43 PM:
> > > On Fri, 2 Nov 2018 19:59:10 +0000
> > > Stephen Warren <swarren at nvidia.com> wrote:
> > >
> > > > I would not expect any client to do an explicit separate
> > > > re-interleave step. That would certainly introduce the
> > > > inefficiency we're trying to avoid. Rather, I assume that any
> > > > application performing interop has some specific processing it
> > > > wants to perform on the surface, and hence uses a shader to
> > > > read the input surface, process the data, and then write the
> > > > results to a new surface. The process of re-interleaving the
> > > > input data can be completely hidden in the existing surface
> > > > fetching part of the shader, without introducing any separate
> > > > temporary surface or any extra copy operation.
> > > >
> > > > If the VDPAU implementation is forced to provide interop
> > > > surfaces in a particular format that the video decoder HW
> > > > itself doesn't support, then the VDPAU implementation must
> > > > perform this re-interleaving itself as a separate step, rather
> > > > than combining it with the desired application processing. This
> > > > is what we're trying to avoid.
> > >
> > > Ok, so this is the disconnect. mpv most definitely doesn't try and
> > > integrate the reinterleave into the other post processing it
> > > does. it just has a work-around in place when vdpau is used:
> > > reinterleave as a copy and then pass on to the regular
> > > post-processing pipeline. I cannot speak for other media players,
> > > but I would imagine they do the same thing - they have to support
> > > many different backends and trying to elide this extra copy means
> > > having additional versions of various post processing stages.
> > >
> > > So, assuming you did what I suggested, and said that interop2 was
> > > the frame output version, we'd just add a conditional based on
> > > extension availability and skip the reinterleave step. We
> > > wouldn't care if an extra copy was being done in the driver or
> > > not - that's up to the internal representation.
> > >
> > > > > In other words, no matter what, we've already established
> > > > > that the existing interop API returns separate fields and a
> > > > > client must expect that. This reality caused grief for other
> > > > > vendors trying to implement interop when their hardware
> > > > > returned whole frames.
> > > >
> > > > At least the extra work is the easy way around; providing two
> > > > separate sub-surfaces to interop with doubled "stride" is much
> > > > easier than trying to transparently combine two separate
> > > > surfaces into one for interop without being forced to just copy
> > > > the surface and be
> > > > inefficient:-)
> > >
> > > True, and this speaks to another point - if we assume that media
> > > players will actively want to use interop2, then you should just
> > > default to whole-frame layout in the driver whenever possible and
> > > if you have to split fields for an interop1 client, it's fine in
> > > practice, and if you have to reinterleave, that's work that was
> > > going to happen anyway.
> > 
> > The approach you suggest does indeed make life simple for simple
> > clients. However, it also prevents any more advanced clients from
> > operating in a way that completely avoids these driver-internal
> > surface copies in all situations. I'd like the API to expose enough
> > information and/or features to allow those copies to be avoided
> > where possible. That requires deciding at surface creation time, or
> > at least some time before any decode operation uses the surface,
> > how interop will expose those surfaces. That's what drove the
> > current API design.
> > 
> > This approach doesn't prevent the later introduction of an interop
> > v2 API that always exposes frame surfaces too; it's just that's not
> > what this change is about.
> > 
> > --
> > nvpublic