[External] Re: gstreamer videoconvert support gpu memory buffer

Fri Aug 20 13:25:37 UTC 2021

Le vendredi 20 août 2021 à 13:07 +0000, Gaurav Vohra a écrit :
> Hello Nicolas,
> 
> Yes correct waylandsink is able to accept x-raw(memory:GBM), the allocation of the GBM buffers is required to be done earlier in pipeline by upstream element.

Then I don't see the point of the memory:GBM over memory:DMABuf (already
supported in waylandsink). In both cases that GL stack will use the wrong
modifier for direct display and the compositor will be forced to do GL
composition. As you are forking GStreamer, some of this remains on your own I
most apology.

regards,
Nicolas

> 
> Please let me know if  something thing like below can be pipeline which can take care of requirements when using gldownload(customized to o/p gbm memory buffers)
> 
> 1. default videoconvert - convert YUV2 -> NV12
> 2. gldownload(customized for gbm buffers o/p) ->  for allocation of frames with GBM buffers
> 
> gst-launch-1.0 -v v4l2src device=/dev/video3  num-buffers=1 ! video/x-raw,width=1280,height=720,framerate=15/2,format=YUY2 ! videoconvert ! gldownload(gbm supported) ! qtimletflite config=/data/misc/camera/mle_tflite.config model=/data/misc/camera/detect.tflite labels=/data/misc/camera/labelmap.txt postprocessing=detection ! waylandsink
> 
> Regards
> Gaurav
> -----Original Message-----
> From: Nicolas Dufresne <nicolas at ndufresne.ca> 
> Sent: Wednesday, August 18, 2021 7:09 PM
> To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>; Matthew Waters <ystreet00 at gmail.com>
> Cc: Gaurav Vohra <Gaurav.Vohra at einfochips.com>
> Subject: Re: [External] Re: gstreamer videoconvert support gpu memory buffer
> 
> 
> CAUTION: This email originated from outside of the organization. This message might not be safe, use caution in opening it. If in doubt, do not open the attachment nor links in the message.
> 
> 
> Le mercredi 18 août 2021 à 08:33 +0000, Gaurav Vohra via gstreamer-devel a écrit :
> > Hi,
> > 
> > As I can see glcolorconvert expect sink and src as video/x- 
> > raw(memory:GLMemory), which is not same in my case as input for me is 
> > x-raw and o/p is x-raw(memory:GBM)
> 
> note that memory:GBM is a downstream implementation, so none of us if familiar with that. Though from my experience, memory:GBM support can be added to glupload, it is going to be very similar to DMABuf (except that you don't need to pass the modifier explicitly).
> 
> As you have a modified waylandsink that supports this custom memory:GBM, you will also need gldownload modification. It's not clear who's doing the allocation, but I would guess waylandsink considering that you need a display compatible modifier.
> 
> This is all very custom direction, we are working on this slowly but through DMABuf, which I believe will be a better layer of abstraction for Linux Media and Linux GFX buffer sharing.
> 
> good luck,
> Nicolas
> 
> > 
> > In my case I am trying to access usb camera using /dev/video3 so input 
> > to customized videoconvert/or any other plugin would be
> > 
> > video/x-raw, width=(int)1280, height=(int)720, 
> > framerate=(fraction)15/2,
> > format=(string)YUY2
> > 
> > and the aim is to convert using customized videoconvert/other element.
> > 
> > video/x-raw(memory:GBM), width=(int)1280, height=(int)720, 
> > framerate=(fraction)15/2, format=(string)NV12
> > 
> > the videoconvert customized is aimed to perform
> > 
> >    a. format conversion from YUY2 to NV12
> >    b. frame buffers from video/x-raw(system memory) to video/x- 
> > raw(memory:GBM)(gpu backed memory)
> > 
> > Here GBM buffers are gpu backed memory which I am setting up in 
> > transform_frame of videoconvert.
> > 
> > Regards
> > Gaurav
> > From: Matthew Waters <ystreet00 at gmail.com>
> > Sent: Wednesday, August 18, 2021 1:33 PM
> > To: Discussion of the development of and with GStreamer 
> > <gstreamer-devel at lists.freedesktop.org>
> > Cc: Gaurav Vohra <Gaurav.Vohra at einfochips.com>
> > Subject: [External] Re: gstreamer videoconvert support gpu memory 
> > buffer
> > 
> > videoconvert most likely isn't going to work with GPU buffers at all.  
> > You will probably want your own element that deals with your data 
> > correctly (or use an existing element like glcolorconvert or maybe a 
> > v4l2 converter element).
> > 
> > Cheers
> > -Matt
> > On 18/8/21 4:18 pm, Gaurav Vohra via gstreamer-devel wrote:
> > > Hello,
> > > 
> > > I am working to customize gstreamer plugin to support
> > > 
> > > working to modify gstreamer plugin - videoconvert get obtain output 
> > > data allocated as gpu memory allocated buffers.
> > > As part of first step have modified the plugin to update the src 
> > > template to include the gpu memory caps.
> > > Allocation for gpu memory is done is transform_frame, but before 
> > > that being called stream error and negotition is failing Commented 
> > > part // #define CSP_VIDEO_CAPS GST_VIDEO_CAPS_MAKE 
> > > (GST_VIDEO_FORMATS_ALL) ";" \ GST_VIDEO_CAPS_MAKE_WITH_FEATURES 
> > > ("ANY", GST_VIDEO_FORMATS_ALL) Modified part #define 
> > > CSP_VIDEO_CAPS_SRC GST_VIDEO_CAPS_MAKE (GST_VIDEO_FORMATS_ALL) ";" \ 
> > > GST_VIDEO_CAPS_MAKE_WITH_FEATURES ("memory:GBM", 
> > > GST_VIDEO_FORMATS_ALL)
> > > 
> > > //#define CSP_VIDEO_CAPS_SRC GST_VIDEO_CAPS_MAKE_WITH_FEATURES 
> > > ("memory:GBM", GST_VIDEO_FORMATS_ALL) static GstStaticPadTemplate 
> > > gst_video_convert_src_template = GST_STATIC_PAD_TEMPLATE ("src",
> > >     GST_PAD_SRC,
> > >     GST_PAD_ALWAYS,
> > >     GST_STATIC_CAPS (CSP_VIDEO_CAPS_SRC)
> > >     );
> > > While trying to run the playback pipeline getting errors related to 
> > > negotitation and stream error like below
> > > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: 
> > > Internal data stream error.
> > > Additional debug info:
> > > ../../../../gstreamer-1.14.4/libs/gst/base/gstbasesrc.c(3055):
> > > gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > > streaming stopped, reason not-negotiated (-4)
> > > 
> > > Further debugging from logs suggest intersect of capsfilter input 
> > > and the caps filter is coming as EMPTY
> > > 
> > > 0:00:06.974521920  4320  0x10cde60 DEBUG             capsfilter
> > > gstcapsfilter.c:296:gst_capsfilter_transform_caps:<capsfilter0> input:
> > > video/x-raw(memory:GBM), format=(string)NV12, width=(int)[ 1, 32767 
> > > ], height=(int)[ 1, 32767 ], framerate=(fraction)[ 0/1, 2147483647/1 
> > > ]; video/x-raw(memory:GBM), width=(int)[ 1, 32767 ], height=(int)[ 
> > > 1, 32767 ], framerate=(fraction)[ 0/1, 2147483647/1 ], 
> > > format=(string){ I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, 
> > > xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, 
> > > v210, v216, NV12, NV21, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, 
> > > BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, 
> > > AYUV64, r210, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, 
> > > Y444_10LE, GBR, GBR_10BE, GBR_10LE, NV16, NV24, NV12_64Z32, 
> > > A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, 
> > > NV61, P010_10BE, P010_10LE, IYU2, VYUY, GBRA, GBRA_10BE, GBRA_10LE, 
> > > GBR_12BE, GBR_12LE, GBRA_12BE, GBRA_12LE, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE, Y444_12LE, GRAY10_LE32, NV12_10LE32, NV16_10LE32 }
> > > 0:00:06.974569108  4320  0x10cde60 DEBUG             capsfilter
> > > gstcapsfilter.c:297:gst_capsfilter_transform_caps:<capsfilter0> filter:
> > > (NULL)
> > > 0:00:06.974616661  4320  0x10cde60 DEBUG             capsfilter
> > > gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter0> caps
> > > filter:    video/x-raw, width=(int)1280, height=(int)720,
> > > framerate=(fraction)15/2, format=(string)YUY2
> > > 0:00:06.974651037  4320  0x10cde60 DEBUG             capsfilter
> > > gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter0> intersect:
> > > EMPTY
> > > 
> > > Regards
> > > Gaurav
> > 
> 
> 