plugin writing: base class for format conversion

Sun Aug 1 03:53:25 UTC 2021

Hey Guennadi

GStreamer has the GstMeta API, which allows you to attach metadata to the buffers which, from your last email, seems to be what you need. If you are using GStreamer 1.20 or later you can use gst_buffer_add_custom_meta. On previous versions, you’ll need to implement your own meta. If you need an example, here’s an implementation that adds video-related metadata to the buffers:

https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/gstvideometa.c

The meta approach is what I would personally go for. For completeness of the answer, there is a way to have elements within elements, it’s called a bin. Your element would inherit from GstBin and basically build a sub-pipeline.

One final idea is to consider using a third party scaler in your element, opencv or libyuv, for example. Your transform_caps method would change, not only the format, but the size as well. Then on the transform_frame you’d call the scaler after your processing. however, i’d remain principled and avoid mixing the two responsibilities and go for the meta approach. 

Michael

> On 31 Jul 2021, at 02:42, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> 
> Hi Michael,
> 
>> On Mon, 19 Jul 2021, Michael Gruner wrote:
>> 
>> Glad it worked!
>> 
>> If you want to overlay a picture (like the thermal bar in that GIF), you
>> can check the gdkpixbufoverlay
>> <https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-gdkpixbufoverlay.html>
>> element. If you also want to add a border to the image so that the
>> overlay doesn’t cover part of the image, you may look at the videobox
>> <https://gstreamer.freedesktop.org/documentation/videobox/index.html?gi-language=c>
>> element. By setting the “right” property to a negative value, you’ll add
>> a black border to the right of the image.
> 
> Thanks, I'll check those! Meanwhile I realised, that I have one more
> difficulty with this: the image I'm getting from the camera is 32x24
> pixels, so, if I just extend it similar to that GIF I still only have 24
> pixels vertically which isn't a lot for temperature text :-) I usually add
> a videoscale element to the pipeline after my conversion plugin, that
> works well, but that doesn't help with that bar - I want to add it after
> scaling, and the actual processing must happen before scaling, it must
> access original camera data.
> 
> One approach I could take is write a second plugin to apply after scaling,
> but apart from the image that plugin would need some additional data from
> the first one - at least the temperature range. My current plan is to pass
> those two numbers in metadata lines below the image and hope that those
> numbers aren't modified by the videoscale plugin...
> 
> Are there better ways to achieve this? Can I "call" videoscale internally
> in my conversion plugin, then I wouldn't need two plugins. Or can I pass
> metadata in some way more robust than image pixels?
> 
> Thanks
> Guennadi
> 
>>>> On 18 Jul 2021, at 05:09, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
>>> 
>>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
>>> 
>>>> The default is to only allow the same format. You need to reimplement
>>>> the transform_caps virtual method in order to change the format. (note
>>>> that transform_caps is a basetransform method).
>>> 
>>> Great, this works! Thanks very much! I've got a colour-coded image now.
>>> Ideally - if not too much work - I'd also like to get numeric temperature
>>> information like on the side bar in the GIF on
>>> https://github.com/makerportal/raspi-thermal-cam or maybe in an additional
>>> in-frame metadata channel. I found a discussion here
>>> https://developer.ridgerun.com/wiki/index.php/GStreamer_and_in-band_metadata
>>> but that doesn't seem to have made it to the mainline. I guess the most
>>> user-friendly would be adding a side bar like in the former link. It
>>> should be possible to extend the width of the image and overlay that kind
>>> of information, right?
>>> 
>>> Thanks
>>> Guennadi
>>> 
>>>>> On 17 Jul 2021, at 13:50, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
>>>>>> 
>>>>>> Hi Guennadi
>>>>>> 
>>>>>> The videoconvert element can already handle this conversion for you:
>>>>>> 
>>>>>> gst-launch-1.0 v4l2src ! videoconvert ! video/x-raw,format=RGB ! …
>>>>> 
>>>>> Yes, I know about videoconvert, but I need to do some special processing
>>>>> of thermal data. For the algorithms to work I have to access the raw data,
>>>>> which is output by the v4l2 driver in GRAY16_BE format (1-to-1 from the
>>>>> camera), and I want to transform that to a colour gradient similar to well
>>>>> known thermal images, so my plugin cannot just stay within GRAY16_BE and
>>>>> use videoconvert to convert that to RGB - you'd just get a gray image of
>>>>> course.
>>>>> 
>>>>>> If you, for other reason, need to write your own element, you probably
>>>>>> want to use the videofilter base class. It’s a specialization of the
>>>>>> basetransform class for video. The important virtual method for you to
>>>>>> implement is “transform_caps”, that will allow you to specify the input
>>>>>> and output caps respectively.
>>>>> 
>>>>> The videofilter class is "allowed" to convert between formats? I thought
>>>>> it would be, so I actually tried it. But I was unable to get it to input
>>>>> and output different formats. I was only able to get it to work with the
>>>>> same format. Are you sure that is supposed to work?
>>>>> 
>>>>> Thanks
>>>>> Guennadi
>>>>> 
>>>>>> Michael
>>>>>> 
>>>>>>>> On 17 Jul 2021, at 05:05, Guennadi Liakhovetski via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I'm trying to write a plugin to convert data from a thermal camera. It
>>>>>>> should take data from the v4l2src source in GRAY16_BE format and convert
>>>>>>> it to an RGB video output. I'm looking at gst-element-maker but I'm not
>>>>>>> finding any suitable base class that would take video/x-raw as input and
>>>>>>> generate the same as output and would convert between format. Do I have to
>>>>>>> write a plugin from scratch? Or should I use basetransform? Any tutorials
>>>>>>> for that?
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Guennadi
>>>>>>> _______________________________________________
>>>>>>> gstreamer-devel mailing list
>>>>>>> gstreamer-devel at lists.freedesktop.org
>>>>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>>>> 
>>>> 
>> 
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210731/e4f52f08/attachment-0001.htm>