[VDPAU] [PATCH 0/1] Add new frame and field mode chroma types. Add VdpDecoderQueryProfileCapability API

Thu Nov 15 03:46:35 UTC 2018

Hi Philip,

Agreed that media players currently might not use the new enums.
But the API changes are done to cater , for all potential use cases vs known use cases.
we believe that the API need to address all the use cases hence it is designed as such.

> * mpv: As discussed, it does reinterleaving as a separate step, so
>   whether the copy is done in the driver or the app makes little
>   difference

I would just like to clarify that for  Existing chroma types and new interop API, The implementation is free to decode to any surface format internally whichever is the best suited for the underlying h/w and codec.
field combining / field splitting is done only if necessary.

Thanks,
ManojGupta.

> -----Original Message-----
> From: VDPAU <vdpau-bounces at lists.freedesktop.org> On Behalf Of Philip
> Langdale
> Sent: Thursday, November 8, 2018 11:36 AM
> To: vdpau at lists.freedesktop.org
> Subject: Re: [VDPAU] [PATCH 0/1] Add new frame and field mode chroma
> types. Add VdpDecoderQueryProfileCapability API
> 
> Stephen, Manoj,
> 
> I get where you guys are coming from, but I'm just looking at this
> pragmatically, with respect to the media player ecosystem that I'm aware of.
> You may have other more private consumers that can drive requirements, and
> in that case, by all means, do what you need to do.
> 
> But as it stands, I see the following:
> 
> * mpv: As discussed, it does reinterleaving as a separate step, so
>   whether the copy is done in the driver or the app makes little
>   difference
> * vlc: Appears to only use vdpau via the mixer, and so only interops
>   with output surfaces
> * kodi: Dropping vdpau support in the next release
> * mplayer: Doesn't do GL interop (and is generally all kinds of
>   obsolete)
> * gstreamer ecosystem: Doesn't meaningfully support vdpau
> 
> So, if you were approaching this from the perspective of what this ecosystem
> needs, I think you're signing up to do extra work that won't really get used.
> And with that context, I say that because the enum additions are a bit of an
> ugly hack, you're better off not making them.
> 
> But is it the end of the world, no not really. As I said - if you're aware of real
> clients that can benefit from this, then obviously the trade-offs are very
> different from what I'm describing.
> 
> Thanks,
> 
> --phil
> 
> On Tue, 6 Nov 2018 05:55:55 +0000
> Manoj Bonda <mbonda at nvidia.com> wrote:
> 
> > Hi Philip,
> >
> > As stated in the interop2 spec
> > (https://github.com/KhronosGroup/OpenGL-Registry/pull/223/files )
> >
> > I would like to restate below points
> >
> > 1) Existing chroma types, existing interop API: Always fields for
> > interop, driver does field splitting if necessary.
> >
> > 2) Existing chroma types, new interop API. Application can choose
> > between fields/frames for interop, driver does field combining / field
> > splitting if necessary.
> >
> > 3) New chroma types, existing or new interop API. Driver never
> > performs internal field/frame conversion, allowing app to guarantee it
> > uses the most efficient path. This requires knowing the interop format
> > before performing a decode so that HW that can decode to both formats
> > will decode to the format that will eventually be used for interop.
> > So clients which are fine with driver internal field combining / field
> > splitting can continue to use existing chroma types with new interop
> > API to get frames for interop.
> >
> > new VDP_CHROMA_TYPES only provide optimized path. Applications which
> > are fine with re-interleaving step in driver can use the existing
> > VDP_CHROMA_TYPES and the VDPAU implementation will perform this
> > re-interleaving itself as a separate step.
> >
> > Thanks,
> > ManojGupta.
> >
> > > -----Original Message-----
> > > From: VDPAU <vdpau-bounces at lists.freedesktop.org> On Behalf Of
> > > Stephen Warren
> > > Sent: Tuesday, November 6, 2018 4:34 AM
> > > To: Philip Langdale <philipl at overt.org>; vdpau at lists.freedesktop.org
> > > Subject: Re: [VDPAU] [PATCH 0/1] Add new frame and field mode chroma
> > > types. Add VdpDecoderQueryProfileCapability API
> > >
> > > Philip Langdale wrote at Friday, November 2, 2018 2:43 PM:
> > > > On Fri, 2 Nov 2018 19:59:10 +0000
> > > > Stephen Warren <swarren at nvidia.com> wrote:
> > > >
> > > > > I would not expect any client to do an explicit separate
> > > > > re-interleave step. That would certainly introduce the
> > > > > inefficiency we're trying to avoid. Rather, I assume that any
> > > > > application performing interop has some specific processing it
> > > > > wants to perform on the surface, and hence uses a shader to read
> > > > > the input surface, process the data, and then write the results
> > > > > to a new surface. The process of re-interleaving the input data
> > > > > can be completely hidden in the existing surface fetching part
> > > > > of the shader, without introducing any separate temporary
> > > > > surface or any extra copy operation.
> > > > >
> > > > > If the VDPAU implementation is forced to provide interop
> > > > > surfaces in a particular format that the video decoder HW itself
> > > > > doesn't support, then the VDPAU implementation must perform this
> > > > > re-interleaving itself as a separate step, rather than combining
> > > > > it with the desired application processing. This is what we're
> > > > > trying to avoid.
> > > >
> > > > Ok, so this is the disconnect. mpv most definitely doesn't try and
> > > > integrate the reinterleave into the other post processing it does.
> > > > it just has a work-around in place when vdpau is used:
> > > > reinterleave as a copy and then pass on to the regular
> > > > post-processing pipeline. I cannot speak for other media players,
> > > > but I would imagine they do the same thing - they have to support
> > > > many different backends and trying to elide this extra copy means
> > > > having additional versions of various post processing stages.
> > > >
> > > > So, assuming you did what I suggested, and said that interop2 was
> > > > the frame output version, we'd just add a conditional based on
> > > > extension availability and skip the reinterleave step. We wouldn't
> > > > care if an extra copy was being done in the driver or not - that's
> > > > up to the internal representation.
> > > >
> > > > > > In other words, no matter what, we've already established that
> > > > > > the existing interop API returns separate fields and a client
> > > > > > must expect that. This reality caused grief for other vendors
> > > > > > trying to implement interop when their hardware returned whole
> > > > > > frames.
> > > > >
> > > > > At least the extra work is the easy way around; providing two
> > > > > separate sub-surfaces to interop with doubled "stride" is much
> > > > > easier than trying to transparently combine two separate
> > > > > surfaces into one for interop without being forced to just copy
> > > > > the surface and be
> > > > > inefficient:-)
> > > >
> > > > True, and this speaks to another point - if we assume that media
> > > > players will actively want to use interop2, then you should just
> > > > default to whole-frame layout in the driver whenever possible and
> > > > if you have to split fields for an interop1 client, it's fine in
> > > > practice, and if you have to reinterleave, that's work that was
> > > > going to happen anyway.
> > >
> > > The approach you suggest does indeed make life simple for simple
> > > clients. However, it also prevents any more advanced clients from
> > > operating in a way that completely avoids these driver-internal
> > > surface copies in all situations. I'd like the API to expose enough
> > > information and/or features to allow those copies to be avoided
> > > where possible. That requires deciding at surface creation time, or
> > > at least some time before any decode operation uses the surface, how
> > > interop will expose those surfaces. That's what drove the current
> > > API design.
> > >
> > > This approach doesn't prevent the later introduction of an interop
> > > v2 API that always exposes frame surfaces too; it's just that's not
> > > what this change is about.
> > >
> > > --
> > > nvpublic
> _______________________________________________
> VDPAU mailing list
> VDPAU at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/vdpau