Gstreamer and vim2m with bayer capture formats

Fri Feb 1 17:03:49 UTC 2019

Le vendredi 01 février 2019 à 12:32 -0200, Mauro Carvalho Chehab a
écrit :
> Hi Nicolas,
> 
> I just added a patch for the vim2m Kernel driver to also support bayer formats,
> but only in capture mode:
> 
> 	https://git.linuxtv.org/mchehab/experimental.git/commit/?h=vim2m&id=7fd6ccf110b7c167a2304ffd482e6c04252c4909
> 
> The goal here is to be able to use vim2m to simulate a webcam. Creating
> a bayer output is trivial, but converting from bayer to RGB is a way more
> complex, and probably not too useful for this Kernel testing driver. So, 
> I opted to not implement bayer conversion only for the V4L2 output mode
> (with is actually the m2m input).
> 
> After such patch and the ones that I merged yesterday at the linux-media
> development git tree, what we have is:
> 
> 	$ v4l2-ctl --list-formats --list-formats-out
> 	ioctl: VIDIOC_ENUM_FMT
> 	Type: Video Capture
> 
> 	[0]: 'RGBP' (16-bit RGB 5-6-5)
> 	[1]: 'RGBR' (16-bit RGB 5-6-5 BE)
> 	[2]: 'RGB3' (24-bit RGB 8-8-8)
> 	[3]: 'BGR3' (24-bit BGR 8-8-8)
> 	[4]: 'YUYV' (YUYV 4:2:2)
> 	[5]: 'BA81' (8-bit Bayer BGBG/GRGR)
> 	[6]: 'GBRG' (8-bit Bayer GBGB/RGRG)
> 	[7]: 'GRBG' (8-bit Bayer GRGR/BGBG)
> 	[8]: 'RGGB' (8-bit Bayer RGRG/GBGB)
> 
> 	ioctl: VIDIOC_ENUM_FMT
> 	Type: Video Output
> 
> 	[0]: 'RGBP' (16-bit RGB 5-6-5)
> 	[1]: 'RGBR' (16-bit RGB 5-6-5 BE)
> 	[2]: 'RGB3' (24-bit RGB 8-8-8)
> 	[3]: 'BGR3' (24-bit BGR 8-8-8)
> 	[4]: 'YUYV' (YUYV 4:2:2)
> 
> With that, I got two problems with gstreamer V4L2 plugin:
> 
> 1) Right now, it only creates a v4l2video?convert source if a M2M device
> is symmetric, e. g. exactly the same set of formats should be supported by 
> both capture and output types. That was never a V4L2 API requirement. 

Not exactly a question of symmetry. The v4l2transform element is means
for RAW video processors, so the classification heuristic is that all
input and output formats must be RAW video.

What I believe the bug might be is that the bayer formats may not have
been classified as RAW video (they are not considered RAW in GStreamer
at the moment).

> 
> Actually, vim2m capture driver had always asymmetric formats since when m2m
> support was introduced. It only became symmetric for a short period of
> time (this week) when I added more formats to it. Yet, after my new patch
> (planned to be merged next week), it will be asymmetric again.

Again, it has nothing to do with symmetry.

> 
> So, the enclosed patch is required (or something similar) in order to fix
> gstreamer with regards to V4L2 M2M API support.
> 
> With such patch applied, and building it with --enable-v4l2-plugin, the
> gst V4L2 plugin works with a pipeline like:
> 
> 	$ gst-launch-1.0 videotestsrc ! video/x-raw,format=BGR ! v4l2video0convert disable-passthrough=1 extra-controls="s,horizontal_flip=1,vertical_flip=1" ! video/x-raw,format=BGR ! videoconvert ! fpsdisplaysink
> 
> 2) I'm not sure how to use gstreamer to use a bayer format for capture type.
> 
> I tried this:
> 
> 	$ gst-launch-1.0 videotestsrc ! video/x-raw,format=BGR ! v4l2video0convert disable-passthrough=1 extra-controls="s,horizontal_flip=1,vertical_flip=1" ! video/x-bayer,format=bggr ! bayer2rgb ! videoconvert ! fpsdisplaysink
> 
> But it failed:
> 	Setting pipeline to PAUSED ...
> 	failed to open /usr/lib64/dri/hybrid_drv_video.so
> 	Not using hybrid_drv_video.so
> 	failed to open /usr/lib64/dri/hybrid_drv_video.so
> 	Not using hybrid_drv_video.so
> 	Pipeline is PREROLLING ...
> 	Got context from element 'fps-display-video_sink-actual-sink-vaapi': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\ vaapidisplayglx1";
> 	ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
> 	Additional debug info:
> 	gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
> 	streaming stopped, reason not-negotiated (-4)
> 	ERROR: pipeline doesn't want to preroll.
> 	Setting pipeline to NULL ...
> 	Freeing pipeline ...
> 
> PS.: I'm sure that the vim2m is working fine with bayer, as I tested it with:
> 
> 	$ qvidcap -p &
> 	$ for i in BA81 GBRRG GRBG RGGB; do v4l2-ctl --stream-mmap --stream-out-mmap --stream-to-host localhost --stream-lossless --stream-out-hor-speed 1 -v pixelformat=$i --stream-count 40; done
> 
> Cheers,
> Mauro
> 
> diff --git a/sys/v4l2/gstv4l2.c b/sys/v4l2/gstv4l2.c
> index 2674d9cd3449..9031de657d71 100644
> --- a/sys/v4l2/gstv4l2.c
> +++ b/sys/v4l2/gstv4l2.c
> @@ -204,7 +204,7 @@ gst_v4l2_probe_and_register (GstPlugin * plugin)
>        if (gst_v4l2_is_vp9_enc (sink_caps, src_caps))
>          gst_v4l2_vp9_enc_register (plugin, basename, it->device_path,
>              sink_caps, src_caps);
> -    } else if (gst_v4l2_is_transform (sink_caps, src_caps)) {
> +    } else {
>        gst_v4l2_transform_register (plugin, basename, it->device_path,
>            sink_caps, src_caps);

Of course this by-pass the classification. The intention is not that
v4l2convert becomes a fallthrough, the reason is that the caps
negotiation has semantic related to the expected functions the m2m
driver will do (right now we support csc and scaling). So the intention
is that GstV4l2Transform becomes an abstract class, and GstV4l2Convert
would subclass it to implement csc and scaling. And then other
specialized subclass can be made to support other type of hardware.

But before this can happen, proper API for HW function classification
will be needed. Right now, this is all base on guessing. For the vim2m
case, we should simply properly classify bayer format as RAW video, and
that should solve your issue.

If my change you have resources or time to work on a proper patch, be
aware that patch submissions works through gitlab.freedesktop.org Merge
Request (basically pushing a branch on a fork there and doing couple of
webui clicks).

>      }
> diff --git a/sys/v4l2/gstv4l2transform.c b/sys/v4l2/gstv4l2transform.c
> index e92f984ff40a..487b7750d17b 100644
> --- a/sys/v4l2/gstv4l2transform.c
> +++ b/sys/v4l2/gstv4l2transform.c
> @@ -1171,17 +1171,6 @@ gst_v4l2_transform_subclass_init (gpointer g_class, gpointer data)
>  }
>  
>  /* Probing functions */
> -gboolean
> -gst_v4l2_is_transform (GstCaps * sink_caps, GstCaps * src_caps)
> -{
> -  gboolean ret = FALSE;
> -
> -  if (gst_caps_is_subset (sink_caps, gst_v4l2_object_get_raw_caps ())
> -      && gst_caps_is_subset (src_caps, gst_v4l2_object_get_raw_caps ()))
> -    ret = TRUE;
> -
> -  return ret;
> -}
>  
>  void
>  gst_v4l2_transform_register (GstPlugin * plugin, const gchar * basename,
> diff --git a/sys/v4l2/gstv4l2transform.h b/sys/v4l2/gstv4l2transform.h
> index 29f3f3c655b7..afdc289db545 100644
> --- a/sys/v4l2/gstv4l2transform.h
> +++ b/sys/v4l2/gstv4l2transform.h
> @@ -73,7 +73,6 @@ struct _GstV4l2TransformClass
>  
>  GType gst_v4l2_transform_get_type (void);
>  
> -gboolean gst_v4l2_is_transform       (GstCaps * sink_caps, GstCaps * src_caps);
>  void     gst_v4l2_transform_register (GstPlugin * plugin,
>                                        const gchar *basename,
>                                        const gchar *device_path,
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20190201/b4fea9b1/attachment.sig>