[VDPAU] [PATCH 0/1] Add new frame and field mode chroma types. Add VdpDecoderQueryProfileCapability API

[Stephen Warren]

Philip Langdale wrote at Wednesday, October 31, 2018 10:20 PM:
> On Wed, 17 Oct 2018 17:46:25 +0530
> Manoj Bonda <mbonda at nvidia.com> wrote:
> > The new API proposed here addresses the request to provide a new mode,
> > where the VDPAU OpenGL interop surfaces are frame or field based.
> > (Reference : https://patchwork.ffmpeg.org/patch/4172/ )
> >
> > Existing VDPAU OpenGL interop supports field-based surfaces only.
> > Applications are not aware of the picture structure of the video
> > surface nor can they allocate the video surface with a specific
> > picture structure.
> >
> > The picture structure can be either "field" or "frame". In a video
> > surface with field picture structure, odd lines are part of the top
> > field and even lines are part of the bottom field. In the case of a
> > video surface with frame structure, there are no top/bottom fields;
> > all lines are stored sequentially as a progressive frame.
> >
> > The new API allows applications to allocate surfaces with the
> > required picture structure. Chroma types defined in the VDPAU API are
> > extended for this purpose.
> >
> > For the below chroma types that already exist,
> >     VDP_CHROMA_TYPE_420
> >     VDP_CHROMA_TYPE_422
> >     VDP_CHROMA_TYPE_444
> >
> > The VDPAU implementation can either directly decode into the video
> > surface passed or decode to another internal surface with a different
> > picture structure type it supports. When used with VDPAU OpenGL
> > interop, the decoded data may be copied from a driver internal
> > surface to the application allocated surface.
> >
> > With the existing interop API, the surface is always treated as
> > having field picture structure. With the updated interop API,
> > surfaces can be registered for either field or frame surface type and
> > used accordingly.
> >
> > Applications can allocate video surfaces with this enum and use those
> > video surfaces with the interop API. This usage allows more
> > flexibility for applications and VDPAU implementations to support
> > various use cases. E.g: The application expects a frame but vdpau
> > implementation can provide field.
> >
> > This change proposes new enums:
> >     VDP_CHROMA_TYPE_420_FIELD
> >     VDP_CHROMA_TYPE_422_FIELD
> >     VDP_CHROMA_TYPE_444_FIELD
> >     VDP_CHROMA_TYPE_420_FRAME
> >     VDP_CHROMA_TYPE_422_FRAME
> >     VDP_CHROMA_TYPE_444_FRAME
> 
> My big question is why extend the Chroma Type enum in this way? Why not
> add a separate enum that specifies whether you want Field or Frame
> layout. Fudging it into the chroma type feels like a hack to avoid
> having to extend various API calls.

Yes, this approach was taken to minimize the changes to the API. We
felt  it'd be easier for applications to simply change a parameter to an
existing API rather than to have to deal with many more entry points;
we'd likely have to duplicate most entry points related to video surfaces
in order to make this change by adding parameters rather than changing
the existing enum. Furthermore, applications wouldn't be able to outright
convert to the new APIs, but instead would have to support both, since
older VDPAU implementations would not support the new APIs.

> From my perspective, I would have imagined that you would not really
> need this at all, because the field/frame layout concept is already
> hidden as an implementation detail everywhere in VDPAU except with
> respect to the GL interop. So why is it not sufficient to have
> GL_NV_vdpau_interop2 as already described, where the picture structure
> can be specified? The driver can presumably react accordingly at that
> time. And to be honest, I think having the picture structure flag on
> the new method is probably unnecessary. Just have the new API call
> for whole frames and use the old API if you somehow want fields, and
> no one actually wants fields. The driver would only expose the new
> extension if it could actually return pictures as frames.

This would be technically feasible, but would be inefficient.

The problem is that when the actual surface format differs from the
format that interop mandates, the interop map/unmap APIs must perform a
format-converting copy. This both wastes time and requires that
additional memory be allocated to store the converted data.

Instead, if we allocate the video surfaces in a format that matches the
format we wish to expose with interop, then we don't need any format-
converting copy, and the system is as efficient as possible; interop map
and unmap operations are simply barriers and don't require copies to be
performed.

This is somewhat complicated by the fact that different hardware
generations support different subsets of format, and this even differs
between compressed media formats; in some cases we support decoding only
to field-based surfaces (which drove the initial definition of the
interop API), in some cases to only frame-based surfaces, and in some
cases to either. Hence the need to explicitly expose this information to
applications.

-- 
nvpublic