plugin writing: base class for format conversion

Sun Aug 1 19:28:45 UTC 2021

GstMeta worked beautifully, thanks a lot!

On Sat, 31 Jul 2021, Michael Gruner wrote:

> Hey Guennadi
>
> GStreamer has the GstMeta API, which allows you to attach metadata to
> the buffers which, from your last email, seems to be what you need. If
> you are using GStreamer 1.20 or later you can use
> gst_buffer_add_custom_meta. On previous versions, you’ll need to
> implement your own meta. If you need an example, here’s an
> implementation that adds video-related metadata to the buffers:
>
> https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/gstvideometa.c
>
> The meta approach is what I would personally go for. For completeness of
> the answer, there is a way to have elements within elements, it’s called
> a bin. Your element would inherit from GstBin and basically build a
> sub-pipeline.
>
> One final idea is to consider using a third party scaler in your
> element, opencv or libyuv, for example. Your transform_caps method would
> change, not only the format, but the size as well. Then on the
> transform_frame you’d call the scaler after your processing. however,
> i’d remain principled and avoid mixing the two responsibilities and go
> for the meta approach.
>
> Michael
>
> > On 31 Jul 2021, at 02:42, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> >
> > Hi Michael,
> >
> >> On Mon, 19 Jul 2021, Michael Gruner wrote:
> >>
> >> Glad it worked!
> >>
> >> If you want to overlay a picture (like the thermal bar in that GIF), you
> >> can check the gdkpixbufoverlay
> >> <https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-gdkpixbufoverlay.html>
> >> element. If you also want to add a border to the image so that the
> >> overlay doesn’t cover part of the image, you may look at the videobox
> >> <https://gstreamer.freedesktop.org/documentation/videobox/index.html?gi-language=c>
> >> element. By setting the “right” property to a negative value, you’ll add
> >> a black border to the right of the image.
> >
> > Thanks, I'll check those! Meanwhile I realised, that I have one more
> > difficulty with this: the image I'm getting from the camera is 32x24
> > pixels, so, if I just extend it similar to that GIF I still only have 24
> > pixels vertically which isn't a lot for temperature text :-) I usually add
> > a videoscale element to the pipeline after my conversion plugin, that
> > works well, but that doesn't help with that bar - I want to add it after
> > scaling, and the actual processing must happen before scaling, it must
> > access original camera data.
> >
> > One approach I could take is write a second plugin to apply after scaling,
> > but apart from the image that plugin would need some additional data from
> > the first one - at least the temperature range. My current plan is to pass
> > those two numbers in metadata lines below the image and hope that those
> > numbers aren't modified by the videoscale plugin...
> >
> > Are there better ways to achieve this? Can I "call" videoscale internally
> > in my conversion plugin, then I wouldn't need two plugins. Or can I pass
> > metadata in some way more robust than image pixels?
> >
> > Thanks
> > Guennadi
> >
> >>>> On 18 Jul 2021, at 05:09, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> >>>
> >>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
> >>>
> >>>> The default is to only allow the same format. You need to reimplement
> >>>> the transform_caps virtual method in order to change the format. (note
> >>>> that transform_caps is a basetransform method).
> >>>
> >>> Great, this works! Thanks very much! I've got a colour-coded image now.
> >>> Ideally - if not too much work - I'd also like to get numeric temperature
> >>> information like on the side bar in the GIF on
> >>> https://github.com/makerportal/raspi-thermal-cam or maybe in an additional
> >>> in-frame metadata channel. I found a discussion here
> >>> https://developer.ridgerun.com/wiki/index.php/GStreamer_and_in-band_metadata
> >>> but that doesn't seem to have made it to the mainline. I guess the most
> >>> user-friendly would be adding a side bar like in the former link. It
> >>> should be possible to extend the width of the image and overlay that kind
> >>> of information, right?
> >>>
> >>> Thanks
> >>> Guennadi
> >>>
> >>>>> On 17 Jul 2021, at 13:50, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> >>>>>
> >>>>> Hi Michael,
> >>>>>
> >>>>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
> >>>>>>
> >>>>>> Hi Guennadi
> >>>>>>
> >>>>>> The videoconvert element can already handle this conversion for you:
> >>>>>>
> >>>>>> gst-launch-1.0 v4l2src ! videoconvert ! video/x-raw,format=RGB ! …
> >>>>>
> >>>>> Yes, I know about videoconvert, but I need to do some special processing
> >>>>> of thermal data. For the algorithms to work I have to access the raw data,
> >>>>> which is output by the v4l2 driver in GRAY16_BE format (1-to-1 from the
> >>>>> camera), and I want to transform that to a colour gradient similar to well
> >>>>> known thermal images, so my plugin cannot just stay within GRAY16_BE and
> >>>>> use videoconvert to convert that to RGB - you'd just get a gray image of
> >>>>> course.
> >>>>>
> >>>>>> If you, for other reason, need to write your own element, you probably
> >>>>>> want to use the videofilter base class. It’s a specialization of the
> >>>>>> basetransform class for video. The important virtual method for you to
> >>>>>> implement is “transform_caps”, that will allow you to specify the input
> >>>>>> and output caps respectively.
> >>>>>
> >>>>> The videofilter class is "allowed" to convert between formats? I thought
> >>>>> it would be, so I actually tried it. But I was unable to get it to input
> >>>>> and output different formats. I was only able to get it to work with the
> >>>>> same format. Are you sure that is supposed to work?
> >>>>>
> >>>>> Thanks
> >>>>> Guennadi
> >>>>>
> >>>>>> Michael
> >>>>>>
> >>>>>>>> On 17 Jul 2021, at 05:05, Guennadi Liakhovetski via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I'm trying to write a plugin to convert data from a thermal camera. It
> >>>>>>> should take data from the v4l2src source in GRAY16_BE format and convert
> >>>>>>> it to an RGB video output. I'm looking at gst-element-maker but I'm not
> >>>>>>> finding any suitable base class that would take video/x-raw as input and
> >>>>>>> generate the same as output and would convert between format. Do I have to
> >>>>>>> write a plugin from scratch? Or should I use basetransform? Any tutorials
> >>>>>>> for that?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Guennadi
> >>>>>>> _______________________________________________
> >>>>>>> gstreamer-devel mailing list
> >>>>>>> gstreamer-devel at lists.freedesktop.org
> >>>>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >>>>>>
> >>>>
> >>
> >>
>