[PATCH dri2proto] RFC: video support for dri2
Rob Clark
robdclark at gmail.com
Fri Aug 19 06:43:04 PDT 2011
On Fri, Aug 19, 2011 at 5:18 AM, Pauli Nieminen
<pauli.nieminen at linux.intel.com> wrote:
> On Thu, Aug 18, 2011 at 09:58:07PM -0500, Rob Clark wrote:
>> From: Rob Clark <rob at ti.com>
>>
>> To allow the potential use of overlays to display video content, a few
>> extra parameters are required:
>>
>> + source buffer in different format (for example, various YUV formats)
>> and size as compared to destination drawable
>> + multi-planar formats where discontiguous buffers are used for
>> different planes. For example, luma and chroma split across
>> multiple memory banks or with different tiled formats.
>> + flipping between multiple back buffers, perhaps not in order (to
>> handle video formats with B-frames)
>> + cropping during swap.. in case of video, perhaps the required hw
>> buffers are larger than the visible picture to account for codec
>> borders (for example, reference frames where a block/macroblock
>> moves past the edge of the visible picture, but back again in
>> subsequent frames).
>>
>> Current solutions use the GPU to do a scaled/colorconvert into a DRI2
>> buffer from the client context. The goal of this protocol change is
>> to push the decision to use overlay or GPU blit to the xorg driver.
>> ---
>> Eventually this should replace Xv. With a few additions, like attributes,
>> it could perhaps be possible to implement the client side Xv API on top
>> of dri2.
>>
>> Note: video is not exactly the same as 3d, there are a number of other
>> things to consider (scaling, colorconvert, multi-planar formats). But
>> on the other hand the principle is similar (direct rendering from hw
>> video codecs). And a lot infrastructure of connection, authentication,
>> is same. So there are two options, either extend DRI2 or add a new
>> protocol which duplicates some parts. I'd like to consider extending
>> DRI2 first, but if people think the requirements for video are too
>> much different from 3d, then I could split this into a new protocol.
>>
>> In either case, I will implement the xserver side infrastructure, but
>> I wanted to get some feel for what is the preferred approach (extend
>> dri2 or new videoproto) first.
>>
>
> XvPutVideo is already existing alternative to use for video playback
> to overlays. But XvPutVideo API isn't perfect for the job either.
We have until now actually used an unholy marriage of v4l2 and
XvPutVideo.. part of the motivation to either extend dri2 or introduce
a direct-video-rendering proto is have a better solution for some of
the sort-comings of the current solution:
+ currently the client has to realize it is using an overlay.. and
which overlay.. switching between overlay and GPU blitting is not
transparent to the client
+ which results in a resource management issue if there are multiple
videos playing, or if you need to scale beyond what can be done w/
overlay, etc
+ in some cases, you might want some way for the window-manager to
signal to xorg driver to switch to blitting because there are some
transformational effects.. ie. wobbly windows and that sort of thing..
(I'm thinking maybe setting some property on the root window?)
Ideally this would be transparent to the video player app.
some of this could be accomplished w/ XvPutImage, but then you are
restricted to memory that can be obtained from shmem, which is not
sufficient for hw video codecs in most cases.
> How would client know which formats are supported by driver? Xv expose
> list of supported image formats to client.
well, in current version of the patch, the client would "just have to
know".. which is sort of how it is on the 3d side for dri2.
Although if there was an attribute mechanism, then I guess we could
actually make it possible to have a completely generic client. One
other issue to tackle then is what exactly is the buffer name.
Currently for dri2 this is left as an implementation detail between
the client and server. But we could decree that it is a GEM buffer
flink name, or possibly that it is a dmabuf file-descriptor (see
http://lists.linaro.org/pipermail/linaro-mm-sig/2011-August/000509.html
)
For a first step, I was ok that the client side of the dri2 connection
would have to know some implementation details of the xorg driver that
it is talking too, since this is how it already is for dri2, and it
could be handled by an approach like vaapi/vdpau where there is some
driver specific code on the client side, above dri2.
But on the other hand, allowing for a completely generic client using
the video/dri2 proto could be interesting. I guess it depends on what
folk's general feeling is.. if completely generic client side is
important, then I'll go back and start adding attributes and this sort
of thing. I think it is doable although it would require adding more
to dri2proto than what I have proposed so far.
>> dri2proto.txt | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>> 1 files changed, 59 insertions(+), 1 deletions(-)
>>
>> diff --git a/dri2proto.txt b/dri2proto.txt
>> index df763c7..aa83b1a 100644
>> --- a/dri2proto.txt
>> +++ b/dri2proto.txt
>> @@ -163,7 +163,8 @@ and DRI2InvalidateBuffers.
>> 6. Protocol Types
>>
>> DRI2DRIVER { DRI2DriverDRI
>> - DRI2DriverVDPAU }
>> + DRI2DriverVDPAU,
>> + DRI2DriverXV }
>>
>> These values describe the type of driver the client will want
>> to load. The server sends back the name of the driver to use
>> @@ -184,6 +185,10 @@ DRI2ATTACHMENT { DRI2BufferFrontLeft
>> These values describe various attachment points for DRI2
>> buffers.
>>
>> + In the case of video driver (DRI2DriverXV) the attachment,
>> + other than DRI2BufferFrontLeft, just indicates buffer
>> + number and has no other special significance.
>> +
>> DRI2BUFFER { attachment: CARD32
>> name: CARD32
>> pitch: CARD32
>> @@ -203,6 +208,16 @@ DRI2ATTACH_FORMAT { attachment: CARD32
>> format. 'attachment' describes the attachment point for the buffer,
>> 'format' describes an opaque, device-dependent format for the buffer.
>>
>> +
>> +DRI2ATTACH_VIDEO { attachment: CARD32
>> + format: CARD32,
>> + width, height: CARD32 }
>> +
>> + The DRI2ATTACH_VIDEO describes an attachment and the associated
>> + format for video buffers. 'attachment' describes the attachment
>> + point for the buffer, 'format' describes a fourcc value for the
>> + buffer.
>> +
>
> I think you will anyway need Xv port attribute like system to expose
> HW features like color conversion. Width and height for buffers source
> buffers could be one of attributes.
>
> If width and height are in attributes video can use
> DRI2GetBuffersWithFormat.
>
> Format should include already information about excepted memory layout
> in buffer. Format would also include information about memory layout
> (packed/planar and tiled/linear)
agreed, if we go the attribute route, then DRI2GetBuffersWithFormat
would be enough
>> ⚙ ⚙ ⚙ ⚙ ⚙ ⚙
>>
>>
>> @@ -367,6 +382,15 @@ The name of this extension is "DRI2".
>> later.
>>
>> ┌───
>> + DRI2GetVideoBuffers
>> + drawable: DRAWABLE
>> + attachments: LISTofDRI2ATTACH_VIDEO
>> + ▶
>> + width, height: CARD32
>> + buffers: LISTofDRI2BUFFER
>> +└───
>> +
>> +┌───
>> DRI2GetMSC
>> drawable: DRAWABLE
>> ▶
>> @@ -585,11 +609,21 @@ A.1 Common Types
>> 4 CARD32 pitch
>> 4 CARD32 cpp
>> 4 CARD32 flags
>> + 4 n extra names length
>> + 4n LISTof extra names
>> └───
>> A DRI2 buffer specifies the attachment, the kernel memory
>> manager name, the pitch and chars per pixel for a buffer
>> attached to a given drawable.
>>
>> + In case of multi-planar video formats, 'extra names' will give the
>> + list of additional buffer names if there is one buffer per plane.
>> + For example, I420 has one Y plane in with a 8bit luma value per
>> + pixel, followed by one U plane subsampled 2x2 (with one 8bit U value
>> + per 2x2 pixel block), followed by one V plane subsampled 2x2. This
>> + could either be represented as a single buffer name, or three
>> + separate buffer names, one each for Y, U, and V.
>> +
>
> I think buffer format should already describe the internal memory
> layout for buffer.
>
> Do you have example where client couldn't know memory layout from
> format, width, height and stride?
well, it isn't so much a format/width/height/stride issue. But it is
likely that the different planes would not be contiguous in a single
buffer. For example, some SoC's require that luma and chroma exist in
different memory banks (different range of physical addresses). For
TI OMAP, we use different tiling formats for luma and chroma in an
NV12 buffer. Having the option for multiple buffer names per
attachment point seemed to give the most flexibility in
implementation.
It would be possible to stuff these all under a single buffer name,
although that brings some wrinkles.. for example, does plane n+1
start immediately after the previous plane (which it would for
contiguous buffers) or on next page boundary (which would be more
likely for non-contiguous buffers). And if different tiling formats
are used, that further complicates the kernel side code in case the
buffer is mmap'd to userspace.
Having the option for either one buffer name, or one buffer name per
plane avoids those complications. But I'm open to hear other opinions
on the idea of 1 vs n names per attachment point.
BR,
-R
>> ┌───
>> DRI2ATTACH_FORMAT
>> 4 CARD32 attachment
>> @@ -599,6 +633,17 @@ A.1 Common Types
>> This data type is only available with protocol version 1.1 or
>> later.
>>
>> +┌───
>> + DRI2ATTACH_VIDEO
>> + 4 CARD32 attachment
>> + 4 CARD32 format
>> + 4 CARD32 width
>> + 4 CARD32 height
>> +└───
>> + Used to describe the attachment and format requested from the server.
>> + This data type is only available with protocol version 1.? or
>> + later.
>> +
>> A.2 Protocol Requests
>>
>> ┌───
>> @@ -745,6 +790,11 @@ A.2 Protocol Requests
>> 4 CARD32 divisor_lo
>> 4 CARD32 remainder_hi
>> 4 CARD32 remainder_lo
>> + 4 DRI2ATTACHMENT source
>> + 4 CARD32 x1
>> + 4 CARD32 y1
>> + 4 CARD32 x2
>> + 4 CARD32 y2
>> ▶
>> 1 1 Reply
>> 1 unused
>> @@ -754,6 +804,14 @@ A.2 Protocol Requests
>> 4 CARD32 swap_lo
>> 5n LISTofDRI2BUFFER buffers
>> └───
>> + The 'source', if not zero (DRI2BufferFrontLeft) indicates the
>> + attachment point of the buffer to swap w/ DRI2BufferFrontLeft.
>> + If zero is specified, DRI2BufferBackLeft is swapped with the
>> + DRI2BufferFrontLeft buffer, for compatibility.
>> +
>> + If 'source' is not zero, (x1,y1), (x2,y2) specify the bounding
>> + box in coordinates of the source buffer which should be scaled
>> + to (0,0), (width,height) of the destination drawable.
>>
>> ┌───
>> DRI2GetMSC
>> --
>> 1.7.5.4
>>
>> _______________________________________________
>> xorg-devel at lists.x.org: X.Org development
>> Archives: http://lists.x.org/archives/xorg-devel
>> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>
More information about the xorg-devel
mailing list