[gst-devel] proposal: support for row-stride in gstreamer

[Rob Clark]

Hi Jun,

I would like to avoid subclassing GstBuffer for this.. since a lot of  
videosink (and other) elements already subclass GstBuffer for various  
purposes.  Having to both subclass GstBuffer and GstVideoBuffer would  
make a mess.

If there was some case where the stride could be changing from frame  
to frame, the buffer metadata proposal would solve this:

   http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt

but, at least in the cases I can think of, the stride would not be  
changing from frame to frame.

I'm currently working on implementing support for it in gst-openmax  
video decoder elements.. which at least should serve as a good  
reference to the changes to make in other video decoder/encoder  
elements.  I think it won't be so complicated, although the decoder  
element would need to support re-negotiating the caps when it  
pad_alloc's a buffer from the video sink element.  But this shouldn't  
be a big deal.

----

as far as stridetransform.. I wouldn't expect it to be part of a  
"normal" pipeline.  I'm using it for now for testing.  If the  
consensus is that normal pipelines might need some stride  
transformation, I think it should be combined with colorspace  
conversion, to avoid multiple passes over the decoded video frame.   
(But I've not had a chance to implement this yet.)

BR,
-R


On Aug 11, 2009, at 12:37 AM, Jim Nie wrote:

>    I agree to use x-raw-yuv-stride, x-raw-rgb-stride to mark the
> buffer and pad capability. However, stridetransform element may impact
> the other elements and application usage. It will be big effort to
> adopt it.
>
>    I have a propose that may not be mature neither. Comments are  
> welcome.
>    Elements that support stride, or sub region of video buffers will
> support both x-raw-yuv-stride and x-raw-yuv. If successfully
> negotiated with downstream element, its src_pad will use
> gst_video_buffer that derived from gst_buffer. gst_video_buffer
> contains the offset/stride information.
>
>    If element fails to negotiate with downstream element with
> x-raw-yuv-stride, it will try x-raw-yuv as current model. Memory is
> copied from subregion to new malloced memory and push to downstream.
>
>    In this way, we can keep compatible with current framework. All
> effort introduced by supporting subregion/stride is limited to the
> elements that want to support it.
>
> Jun
>
> 2009/8/4, Rob Clark <rob at ti.com>:
>>
>> On Aug 4, 2009, at 4:49 AM, Jan Schmidt wrote:
>>
>>> Hi,
>>>
>>> On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
>>>> so, my next steps (at least my current thinking about next steps)  
>>>> are
>>>> to start adding some rowstride related stuff in gst-plugins-base:
>>>>
>>>>
>>>> gst-libs/gst/video/video.c:
>>>> 1) add gst_video_format_get_size_strided()
>>>> 2) change gst_video_format_parse_caps() to understand strided caps,
>>>> and add a gst_video_format_parse_caps_strided() function  (it seems
>>>> there are enough places already using gst_video_format_parse_caps()
>>>> that I probably don't want to change the signature of this  
>>>> function)
>>>>
>>>> and then add a GstStrideTransform element under gst/stride
>>>
>>> I haven't put enough thought into designing a stride system to say
>>> anything for certain. Particularly, I'm not sure whether we'll be  
>>> able
>>> to safely and successfully integrate it into the 0.10 series without
>>> thinking about it harder.
>>>
>>> I am pretty sure, however, that adding a separate stride adjust
>>> element
>>> is the wrong way to go. None of the existing pipelines will include
>>> it,
>>> so it would no be useful without changes to the applications/playbin
>>> etc.
>>>
>>> My hunch is that it will be better to add a new 'video/x-raw-yuv- 
>>> full'
>>> format and add stride as a new parameter, and adjust
>>> ffmpegcolorspace to
>>> support conversions to/from the unstrided/strided formats.
>>
>>
>> Hmm.. my idea for stridetransform was mainly just an element for
>> testing and debugging with manually constructed gst-launch pipelines.
>> We need some way to run codecs both with and without rowstide and
>> verifying the output.  For example:
>>
>>
>>   ... ! decoder ! video/x-raw-
>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !  
>> filesink
>> location=file1.dat
>>
>> and then
>>
>>   ... ! decoder ! video/x-raw-yuv-
>> strided
>> ,format
>> =(fourcc)YUY2,width=320,height=240,rowstride=700,framerate=30/1 !
>> stridetransform ! video/x-raw-
>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !  
>> filesink
>> location=file2.dat
>>
>>
>> But if you think this should be part of a normal pipeline, then I
>> think it would make sense to merge in with the colorspace conversion,
>> so you have one memory copy instead of two.
>>
>> But do you think this is required?  At least in the cases that I have
>> in mind, the video sink will also support non-strided buffers, but
>> will be falling back to a less optimal mechanism.
>>
>>
>> ------
>>
>> btw, at the risk of starting a bikeshed discussion, is 'video/x-raw-
>> yuv-full' preferred to 'video/x-raw-yuv-strided'?  So far I've been
>> using the latter, but I don't mind changing the code that I've  
>> written
>> so far.  Should the functions added to video.c have the _full suffix
>> instead of _strided (ie. gst_video_format_parse_caps_full() instead  
>> of
>> gst_video_format_parse_caps_strided())?
>>
>> If we go with -full, are there any other fields we should add to the
>> caps at the same time?  Offhand, the only thing I can think of that  
>> is
>> a mandatory field would be rowstride, but I haven't thought too much
>> about, for example, interlaced buffers.
>>
>> ------
>>
>> fyi, I've begun making some changes in my private tree at
>> http://github.com/robclark/gst-plugins-base
>>  ... but nothing that I've changed so far is set in stone.  But we do
>> have to start adding support in the codec and other elements that
>> we'll use pretty soon, so comments and suggestions now are greatly
>> appreciated.  We can stay on our own tree, or a special rowstride
>> branch, for now if integration to master is post-0.10.  But it would
>> be nice to not have to *completely* re-write things later ;-)
>>
>>
>>
>> BR,
>> -R