[gst-devel] proposal: support for row-stride in gstreamer

Thu Jul 23 02:37:19 CEST 2009

On Jul 22, 2009, at 3:23 AM, Jan Schmidt wrote:

> On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
>> btw, I noticed the proposal about GstBuffer metadata:
>>
>>  <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
>>>
>>
>> this provides a nice way to handle x,y vstab coordinates mentioned
>> below.
>>
>> In the cases that I can think of, rowstride would not change from
>> buffer to buffer, so I would propose to handle this in caps
>> negotation, as proposed originally.  But I would like to use per-
>> buffer meta-data to handle the per-buffer cropping/panning.
>
> Exactly what I've had in mind for a while - and exactly what we
> discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
>
> Now, we just need to find the time to implement stuff, and I need to
> write up my notes from GCDS in a more digestible form and post them.
>

I would be definitely interested to see the notes from GCDS..  I heard  
second-hand some proposal about caps-negotiation, but would like to  
know more..

 From my end, the relatively short term need is just for rowstride.   
Fortunately, I think this should not be such a big code-change as not  
all elements need to support it.  We'll add support for it in gst- 
openmax video encoder/decoder elements, and at least v4l2sink (which,  
btw, I have working now in my private fork of gst-plugins-good[1],  
although I need to do a bit more testing and it could use some eyes).   
I would be interested to eventually add this to xvimagesink too,  
although I'm not sure yet when I'll get to this, so if someone has a  
more short-term interest in this, please don't wait for me ;-)

We will anyways implement a transform element that can convert from  
strided to non-strided buffer, which would be useful between elements  
requiring stride and those that don't support it (and also, just for  
debugging, to dump out raw decoded frames without stride to a filesrc,  
and visa versa).

For slightly longer term (ie. between now and December-ish), if there  
are ways to divide up the work for per-buffer meta-data, maybe we can  
help with some parts of the implementation?  I haven't though about it  
too much, but if it there is a clean way to break it up into sub- 
tasks, I'm all for trying to help if we can.  At a minimum, I'll have  
some weekend and evening time where I'd be happy to help.

BR,
-R

[1] see: <http://github.com/robclark/gst-plugins-good/tree/4065c9e1ff927604d9a52c944eeee1047a24d283/sys/v4l2 
 >

> Cheers,
> Jan.
>
>>
>> Comments?
>>
>> BR,
>> -R
>>
>>
>> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
>>
>>> Hi gstreamer folks,
>>>
>>> The following is a proposal for how to add row-stride (and possibly
>>> some related changes) to gstreamer.  I gave a couple of possible
>>> examples of where this would be useful, but it is probably not
>>> exhaustive.  Please let me know if you see any cases that I  
>>> missed, or
>>> details that I overlooked, etc.
>>>
>>>
>>>
>>> Use-cases:
>>> ----------
>>> + display hardware with special constraints on image dimensions, for
>>> example
>>>   if the output buffer must have dimensions that are a power of two
>>> + zero-copy cropping of image / videoframe (at least for interleaved
>>> color
>>>   formats.. more on this later)
>>>
>>> One example to think about is rendering onto a 3d surface.  In some
>>> cases, graphics hardware could require that the surface dimensions  
>>> are
>>> a power of 2.  In this case, you would want the vsink to allocate a
>>> buffer with a rowstride that is the next larger power of 2 from the
>>> image width.
>>>
>>>
>>> Another example to think about is video stabilization.  In this use
>>> case, you would ask the camera to capture an oversized frame.  Your
>>> vstab algorithm would calculate an x,y offset of the stabilized
>>> image.  But if the decoder understands rowstride, you do not need to
>>> actually copy the image buffer.  Say, just to pick some numbers, you
>>> want your final output to be 640x480, and you want your oversized
>>> frame to be +20% in each dimension (768x576):
>>>
>>>   +--------+           +-------+           +------+
>>>   | camera |---------->| vstab |---------->| venc |
>>>   +--------+ width=768 +-------+ width=640 +------+
>>>             height=576          height=480
>>>          rowstride=768       rowstride=768
>>>
>>> In the case of an interleaved color format (RGB, UYVY, etc), you  
>>> could
>>> simply increment the 'data' pointer in the buffer by (y*rowstride) 
>>> +x.
>>> No memcpy() required.  As long as the video encoder respects the
>>> rowstride, it will see the stabilized frame correctly.
>>>
>>>
>>>
>>> Proposal:
>>> ---------
>>>
>>> In all cases that I can think of, the row-stride will not be  
>>> changing
>>> dynamically.  So this parameter can be negotiated thru caps
>>> negotiation in the same way as image width/height, colorformat, etc.
>>> However, we need to know conclusively that there is no element in  
>>> the
>>> pipeline that cares about the image format, but does not understand
>>> "rowstride", so we cannot use existing type strings (ex. "video/x- 
>>> raw-
>>> yuv").  And, at least in the cases that I can think of, the video  
>>> sink
>>> will dictate the row-stride.  So upstream caps-renegotiation will be
>>> used to arrive at the final "rowstride" value.
>>>
>>> For media types, I propose to continue using existing strings for  
>>> non-
>>> stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
>>> elements, they can support a second format, ex. "video/x-raw-yuv-
>>> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
>>> whatever the existing string is).  In the case that a strided format
>>> is negotiated, it is required for there to also be a "rowstride"  
>>> entry
>>> in the final negotiated caps.
>>>
>>> question: in general, most elements supporting rowstride will have  
>>> no
>>> constraint on what particular rowstride values are supported.  Do  
>>> they
>>> just list "rowstride=[0-4294967295]" in their caps template?  The
>>> video sink allocating the buffer will likely have some constraints  
>>> on
>>> rowstride, although this will be a function of the width (for  
>>> example,
>>> round the width up to next power of two).
>>>
>>> We will implement some sort of GstRowStrideTransform element to
>>> interface between stride-aware and non-stride-aware elements.
>>>
>>>
>>>
>>>
>>> Non-Interleaved Color Formats:
>>> ------------------------------
>>>
>>> So, everything I've said so far about zero-copy cropping works until
>>> you start considering planar/semi-planar color formats which do not
>>> have equal size planes.  For example, consider NV12:  the offset to
>>> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
>>> plane is ((y/2)*rowstride)+x.  There are only three ways that I can
>>> think of to deal with this (listed in order of my preference):
>>>
>>> 1) add fields to GstBuffer to pass additional pointers to the other
>>> color
>>>    planes within the same GstBuffer
>>> 2) add a field(s) to GstBuffer to pass an offset..  either an x,y
>>> offset, or a
>>>    single value that is (y*rowstride)+x.  Either way, the various
>>> elements in
>>>    the pipeline can use this to calculate the start of the
>>> individual planes
>>>    of data.
>>> 3) pass individual planes of a single image as separate
>>> GstBuffer's.. but I'm
>>>    not a huge fan of this because now every element needs to have
>>> some sort
>>>    of "I've got Y, but I'm waiting for UV" state.
>>>
>>> I'm not sure if anyone has any thoughts about which of these three
>>> approaches is preferred.  Or any alternative ideas?
>>>
>>>
>>>
>>> BR,
>>> -Rob
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>
>
> -- 
> Jan Schmidt <thaytan at noraisin.net>
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>