[gst-devel] proposal: support for row-stride in gstreamer
Jan Schmidt
thaytan at noraisin.net
Tue Aug 4 11:49:21 CEST 2009
Hi,
On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
> so, my next steps (at least my current thinking about next steps) are
> to start adding some rowstride related stuff in gst-plugins-base:
>
>
> gst-libs/gst/video/video.c:
> 1) add gst_video_format_get_size_strided()
> 2) change gst_video_format_parse_caps() to understand strided caps,
> and add a gst_video_format_parse_caps_strided() function (it seems
> there are enough places already using gst_video_format_parse_caps()
> that I probably don't want to change the signature of this function)
>
> and then add a GstStrideTransform element under gst/stride
I haven't put enough thought into designing a stride system to say
anything for certain. Particularly, I'm not sure whether we'll be able
to safely and successfully integrate it into the 0.10 series without
thinking about it harder.
I am pretty sure, however, that adding a separate stride adjust element
is the wrong way to go. None of the existing pipelines will include it,
so it would no be useful without changes to the applications/playbin
etc.
My hunch is that it will be better to add a new 'video/x-raw-yuv-full'
format and add stride as a new parameter, and adjust ffmpegcolorspace to
support conversions to/from the unstrided/strided formats.
- Jan.
> Comments? Suggestions? Objections?
>
>
> BR,
> -R
>
>
> On Jul 22, 2009, at 3:23 AM, Jan Schmidt wrote:
>
> > On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
> >> btw, I noticed the proposal about GstBuffer metadata:
> >>
> >> <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
> >>>
> >>
> >> this provides a nice way to handle x,y vstab coordinates mentioned
> >> below.
> >>
> >> In the cases that I can think of, rowstride would not change from
> >> buffer to buffer, so I would propose to handle this in caps
> >> negotation, as proposed originally. But I would like to use per-
> >> buffer meta-data to handle the per-buffer cropping/panning.
> >
> > Exactly what I've had in mind for a while - and exactly what we
> > discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
> >
> > Now, we just need to find the time to implement stuff, and I need to
> > write up my notes from GCDS in a more digestible form and post them.
> >
> > Cheers,
> > Jan.
> >
> >>
> >> Comments?
> >>
> >> BR,
> >> -R
> >>
> >>
> >> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
> >>
> >>> Hi gstreamer folks,
> >>>
> >>> The following is a proposal for how to add row-stride (and possibly
> >>> some related changes) to gstreamer. I gave a couple of possible
> >>> examples of where this would be useful, but it is probably not
> >>> exhaustive. Please let me know if you see any cases that I
> >>> missed, or
> >>> details that I overlooked, etc.
> >>>
> >>>
> >>>
> >>> Use-cases:
> >>> ----------
> >>> + display hardware with special constraints on image dimensions, for
> >>> example
> >>> if the output buffer must have dimensions that are a power of two
> >>> + zero-copy cropping of image / videoframe (at least for interleaved
> >>> color
> >>> formats.. more on this later)
> >>>
> >>> One example to think about is rendering onto a 3d surface. In some
> >>> cases, graphics hardware could require that the surface dimensions
> >>> are
> >>> a power of 2. In this case, you would want the vsink to allocate a
> >>> buffer with a rowstride that is the next larger power of 2 from the
> >>> image width.
> >>>
> >>>
> >>> Another example to think about is video stabilization. In this use
> >>> case, you would ask the camera to capture an oversized frame. Your
> >>> vstab algorithm would calculate an x,y offset of the stabilized
> >>> image. But if the decoder understands rowstride, you do not need to
> >>> actually copy the image buffer. Say, just to pick some numbers, you
> >>> want your final output to be 640x480, and you want your oversized
> >>> frame to be +20% in each dimension (768x576):
> >>>
> >>> +--------+ +-------+ +------+
> >>> | camera |---------->| vstab |---------->| venc |
> >>> +--------+ width=768 +-------+ width=640 +------+
> >>> height=576 height=480
> >>> rowstride=768 rowstride=768
> >>>
> >>> In the case of an interleaved color format (RGB, UYVY, etc), you
> >>> could
> >>> simply increment the 'data' pointer in the buffer by (y*rowstride)
> >>> +x.
> >>> No memcpy() required. As long as the video encoder respects the
> >>> rowstride, it will see the stabilized frame correctly.
> >>>
> >>>
> >>>
> >>> Proposal:
> >>> ---------
> >>>
> >>> In all cases that I can think of, the row-stride will not be
> >>> changing
> >>> dynamically. So this parameter can be negotiated thru caps
> >>> negotiation in the same way as image width/height, colorformat, etc.
> >>> However, we need to know conclusively that there is no element in
> >>> the
> >>> pipeline that cares about the image format, but does not understand
> >>> "rowstride", so we cannot use existing type strings (ex. "video/x-
> >>> raw-
> >>> yuv"). And, at least in the cases that I can think of, the video
> >>> sink
> >>> will dictate the row-stride. So upstream caps-renegotiation will be
> >>> used to arrive at the final "rowstride" value.
> >>>
> >>> For media types, I propose to continue using existing strings for
> >>> non-
> >>> stride-aware element caps, ex. "video/x-raw-yuv". For stride-aware
> >>> elements, they can support a second format, ex. "video/x-raw-yuv-
> >>> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
> >>> whatever the existing string is). In the case that a strided format
> >>> is negotiated, it is required for there to also be a "rowstride"
> >>> entry
> >>> in the final negotiated caps.
> >>>
> >>> question: in general, most elements supporting rowstride will have
> >>> no
> >>> constraint on what particular rowstride values are supported. Do
> >>> they
> >>> just list "rowstride=[0-4294967295]" in their caps template? The
> >>> video sink allocating the buffer will likely have some constraints
> >>> on
> >>> rowstride, although this will be a function of the width (for
> >>> example,
> >>> round the width up to next power of two).
> >>>
> >>> We will implement some sort of GstRowStrideTransform element to
> >>> interface between stride-aware and non-stride-aware elements.
> >>>
> >>>
> >>>
> >>>
> >>> Non-Interleaved Color Formats:
> >>> ------------------------------
> >>>
> >>> So, everything I've said so far about zero-copy cropping works until
> >>> you start considering planar/semi-planar color formats which do not
> >>> have equal size planes. For example, consider NV12: the offset to
> >>> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
> >>> plane is ((y/2)*rowstride)+x. There are only three ways that I can
> >>> think of to deal with this (listed in order of my preference):
> >>>
> >>> 1) add fields to GstBuffer to pass additional pointers to the other
> >>> color
> >>> planes within the same GstBuffer
> >>> 2) add a field(s) to GstBuffer to pass an offset.. either an x,y
> >>> offset, or a
> >>> single value that is (y*rowstride)+x. Either way, the various
> >>> elements in
> >>> the pipeline can use this to calculate the start of the
> >>> individual planes
> >>> of data.
> >>> 3) pass individual planes of a single image as separate
> >>> GstBuffer's.. but I'm
> >>> not a huge fan of this because now every element needs to have
> >>> some sort
> >>> of "I've got Y, but I'm waiting for UV" state.
> >>>
> >>> I'm not sure if anyone has any thoughts about which of these three
> >>> approaches is preferred. Or any alternative ideas?
> >>>
> >>>
> >>>
> >>> BR,
> >>> -Rob
> >>>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> gstreamer-devel at lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >>
> >
> >
> > --
> > Jan Schmidt <thaytan at noraisin.net>
> >
> >
> >
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
--
Jan Schmidt <thaytan at noraisin.net>
More information about the gstreamer-devel
mailing list