[External] Re: gstreamer videoconvert support gpu memory buffer

Wed Aug 18 13:38:36 UTC 2021

Le mercredi 18 août 2021 à 08:33 +0000, Gaurav Vohra via gstreamer-devel a
écrit :
> Hi,
>  
> As I can see glcolorconvert expect sink and src as video/x-
> raw(memory:GLMemory), which is not same in my case as input for me is x-raw
> and o/p is x-raw(memory:GBM)

note that memory:GBM is a downstream implementation, so none of us if familiar
with that. Though from my experience, memory:GBM support can be added to
glupload, it is going to be very similar to DMABuf (except that you don't need
to pass the modifier explicitly).

As you have a modified waylandsink that supports this custom memory:GBM, you
will also need gldownload modification. It's not clear who's doing the
allocation, but I would guess waylandsink considering that you need a display
compatible modifier.

This is all very custom direction, we are working on this slowly but through
DMABuf, which I believe will be a better layer of abstraction for Linux Media
and Linux GFX buffer sharing.

good luck,
Nicolas

>  
> In my case I am trying to access usb camera using /dev/video3 so input to
> customized videoconvert/or any other plugin would be
>  
> video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)15/2,
> format=(string)YUY2
>  
> and the aim is to convert using customized videoconvert/other element.
>  
> video/x-raw(memory:GBM), width=(int)1280, height=(int)720,
> framerate=(fraction)15/2, format=(string)NV12
>  
> the videoconvert customized is aimed to perform
>  
>    a. format conversion from YUY2 to NV12
>    b. frame buffers from video/x-raw(system memory) to video/x-
> raw(memory:GBM)(gpu backed memory)
>  
> Here GBM buffers are gpu backed memory which I am setting up in
> transform_frame of videoconvert.
>  
> Regards
> Gaurav
> From: Matthew Waters <ystreet00 at gmail.com> 
> Sent: Wednesday, August 18, 2021 1:33 PM
> To: Discussion of the development of and with GStreamer
> <gstreamer-devel at lists.freedesktop.org>
> Cc: Gaurav Vohra <Gaurav.Vohra at einfochips.com>
> Subject: [External] Re: gstreamer videoconvert support gpu memory buffer
>  
> videoconvert most likely isn't going to work with GPU buffers at all.  You
> will probably want your own element that deals with your data correctly (or
> use an existing element like glcolorconvert or maybe a v4l2 converter
> element).
> 
> Cheers
> -Matt
> On 18/8/21 4:18 pm, Gaurav Vohra via gstreamer-devel wrote:
> > Hello,
> >  
> > I am working to customize gstreamer plugin to support 
> >  
> > working to modify gstreamer plugin - videoconvert get obtain output data
> > allocated as gpu memory allocated buffers.
> > As part of first step have modified the plugin to update the src template to
> > include the gpu memory caps.
> > Allocation for gpu memory is done is transform_frame, but before that being
> > called stream error and negotition is failing
> > Commented part
> > // #define CSP_VIDEO_CAPS GST_VIDEO_CAPS_MAKE (GST_VIDEO_FORMATS_ALL) ";" \
> > GST_VIDEO_CAPS_MAKE_WITH_FEATURES ("ANY", GST_VIDEO_FORMATS_ALL)
> > Modified part
> > #define CSP_VIDEO_CAPS_SRC GST_VIDEO_CAPS_MAKE (GST_VIDEO_FORMATS_ALL) ";" \
> > GST_VIDEO_CAPS_MAKE_WITH_FEATURES ("memory:GBM", GST_VIDEO_FORMATS_ALL)
> >  
> > //#define CSP_VIDEO_CAPS_SRC GST_VIDEO_CAPS_MAKE_WITH_FEATURES
> > ("memory:GBM", GST_VIDEO_FORMATS_ALL)
> > static GstStaticPadTemplate gst_video_convert_src_template =
> > GST_STATIC_PAD_TEMPLATE ("src",
> >     GST_PAD_SRC,
> >     GST_PAD_ALWAYS,
> >     GST_STATIC_CAPS (CSP_VIDEO_CAPS_SRC)
> >     );
> > While trying to run the playback pipeline getting errors related to
> > negotitation and stream error like below
> > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal
> > data stream error.
> > Additional debug info:
> > ../../../../gstreamer-1.14.4/libs/gst/base/gstbasesrc.c(3055):
> > gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > streaming stopped, reason not-negotiated (-4)
> >  
> > Further debugging from logs suggest intersect of capsfilter input and the
> > caps filter is coming as EMPTY
> >  
> > 0:00:06.974521920  4320  0x10cde60 DEBUG             capsfilter
> > gstcapsfilter.c:296:gst_capsfilter_transform_caps:<capsfilter0> input:    
> > video/x-raw(memory:GBM), format=(string)NV12, width=(int)[ 1, 32767 ],
> > height=(int)[ 1, 32767 ], framerate=(fraction)[ 0/1, 2147483647/1 ];
> > video/x-raw(memory:GBM), width=(int)[ 1, 32767 ], height=(int)[ 1, 32767 ],
> > framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ I420, YV12,
> > YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR,
> > Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, GRAY8, GRAY16_BE, GRAY16_LE,
> > v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1,
> > ARGB64, AYUV64, r210, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE,
> > Y444_10LE, GBR, GBR_10BE, GBR_10LE, NV16, NV24, NV12_64Z32, A420_10BE,
> > A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, NV61, P010_10BE,
> > P010_10LE, IYU2, VYUY, GBRA, GBRA_10BE, GBRA_10LE, GBR_12BE, GBR_12LE,
> > GBRA_12BE, GBRA_12LE, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE,
> > Y444_12LE, GRAY10_LE32, NV12_10LE32, NV16_10LE32 }
> > 0:00:06.974569108  4320  0x10cde60 DEBUG             capsfilter
> > gstcapsfilter.c:297:gst_capsfilter_transform_caps:<capsfilter0> filter:   
> > (NULL)
> > 0:00:06.974616661  4320  0x10cde60 DEBUG             capsfilter
> > gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter0> caps
> > filter:    video/x-raw, width=(int)1280, height=(int)720,
> > framerate=(fraction)15/2, format=(string)YUY2
> > 0:00:06.974651037  4320  0x10cde60 DEBUG             capsfilter
> > gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter0> intersect:
> > EMPTY
> >  
> > Regards
> > Gaurav
>  