MLX90640 plugins (was Re: plugin writing: base class for format conversion)

Sun Sep 26 10:39:07 UTC 2021

Hi,

Just a quick note, that my 2 plugins are working and are uploaded at
https://github.com/lyakh/gst-mlx90640 I don't think it would make sense to
submit them to make them in any way official, they're rather simple
half-hacks, just announcing them here in case they can be useful to
anyone.

Thanks
Guennadi

On Sun, 12 Sep 2021, Guennadi Liakhovetski via gstreamer-devel wrote:

> Got it working!
>
> On Sat, 11 Sep 2021, Guennadi Liakhovetski wrote:
>
> > Hi Michael,
> >
> > As you notice, I might be slow (hey, it's a hobby in the end) but I'm
> > steady! I now have two plugins: the first one does the actual temperature
> > calculations on the original binary data, and exports temperature values
> > as metadata. Then comes a standard scaler in the pipeline, then my second
> > plugin, which should add the side bar with those temperature values. That
> > second plugin can already receive metadata from the first one, can add a
> > side border and can draw a colour gradient in it.
> >
> > Now I want to print those temperature values in that border next to that
> > gradient. You previously recommended an overlay plugin, but I want to do
> > that programmatically from my plugin, i.e. I need a C gstreamer API for
> > that. I've done some searching and I see pango (on Linux) being used in
> > gstbasetextoverlay and related code, but again, that's just building
> > plugins and not exporting an API for other gstreamer code. Do I understand
> > it correctly that there's no such API? Do I have to export those values in
> > text sinks and then use them for textoverlay (if that even would work...)?
> >
> > Thanks
> > Guennadi
> >
> > On Sun, 1 Aug 2021, Guennadi Liakhovetski wrote:
> >
> > > GstMeta worked beautifully, thanks a lot!
> > >
> > > On Sat, 31 Jul 2021, Michael Gruner wrote:
> > >
> > > > Hey Guennadi
> > > >
> > > > GStreamer has the GstMeta API, which allows you to attach metadata to
> > > > the buffers which, from your last email, seems to be what you need. If
> > > > you are using GStreamer 1.20 or later you can use
> > > > gst_buffer_add_custom_meta. On previous versions, you’ll need to
> > > > implement your own meta. If you need an example, here’s an
> > > > implementation that adds video-related metadata to the buffers:
> > > >
> > > > https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/gstvideometa.c
> > > >
> > > > The meta approach is what I would personally go for. For completeness of
> > > > the answer, there is a way to have elements within elements, it’s called
> > > > a bin. Your element would inherit from GstBin and basically build a
> > > > sub-pipeline.
> > > >
> > > > One final idea is to consider using a third party scaler in your
> > > > element, opencv or libyuv, for example. Your transform_caps method would
> > > > change, not only the format, but the size as well. Then on the
> > > > transform_frame you’d call the scaler after your processing. however,
> > > > i’d remain principled and avoid mixing the two responsibilities and go
> > > > for the meta approach.
> > > >
> > > > Michael
> > > >
> > > > > On 31 Jul 2021, at 02:42, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > >> On Mon, 19 Jul 2021, Michael Gruner wrote:
> > > > >>
> > > > >> Glad it worked!
> > > > >>
> > > > >> If you want to overlay a picture (like the thermal bar in that GIF), you
> > > > >> can check the gdkpixbufoverlay
> > > > >> <https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-gdkpixbufoverlay.html>
> > > > >> element. If you also want to add a border to the image so that the
> > > > >> overlay doesn’t cover part of the image, you may look at the videobox
> > > > >> <https://gstreamer.freedesktop.org/documentation/videobox/index.html?gi-language=c>
> > > > >> element. By setting the “right” property to a negative value, you’ll add
> > > > >> a black border to the right of the image.
> > > > >
> > > > > Thanks, I'll check those! Meanwhile I realised, that I have one more
> > > > > difficulty with this: the image I'm getting from the camera is 32x24
> > > > > pixels, so, if I just extend it similar to that GIF I still only have 24
> > > > > pixels vertically which isn't a lot for temperature text :-) I usually add
> > > > > a videoscale element to the pipeline after my conversion plugin, that
> > > > > works well, but that doesn't help with that bar - I want to add it after
> > > > > scaling, and the actual processing must happen before scaling, it must
> > > > > access original camera data.
> > > > >
> > > > > One approach I could take is write a second plugin to apply after scaling,
> > > > > but apart from the image that plugin would need some additional data from
> > > > > the first one - at least the temperature range. My current plan is to pass
> > > > > those two numbers in metadata lines below the image and hope that those
> > > > > numbers aren't modified by the videoscale plugin...
> > > > >
> > > > > Are there better ways to achieve this? Can I "call" videoscale internally
> > > > > in my conversion plugin, then I wouldn't need two plugins. Or can I pass
> > > > > metadata in some way more robust than image pixels?
> > > > >
> > > > > Thanks
> > > > > Guennadi
> > > > >
> > > > >>>> On 18 Jul 2021, at 05:09, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> > > > >>>
> > > > >>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
> > > > >>>
> > > > >>>> The default is to only allow the same format. You need to reimplement
> > > > >>>> the transform_caps virtual method in order to change the format. (note
> > > > >>>> that transform_caps is a basetransform method).
> > > > >>>
> > > > >>> Great, this works! Thanks very much! I've got a colour-coded image now.
> > > > >>> Ideally - if not too much work - I'd also like to get numeric temperature
> > > > >>> information like on the side bar in the GIF on
> > > > >>> https://github.com/makerportal/raspi-thermal-cam or maybe in an additional
> > > > >>> in-frame metadata channel. I found a discussion here
> > > > >>> https://developer.ridgerun.com/wiki/index.php/GStreamer_and_in-band_metadata
> > > > >>> but that doesn't seem to have made it to the mainline. I guess the most
> > > > >>> user-friendly would be adding a side bar like in the former link. It
> > > > >>> should be possible to extend the width of the image and overlay that kind
> > > > >>> of information, right?
> > > > >>>
> > > > >>> Thanks
> > > > >>> Guennadi
> > > > >>>
> > > > >>>>> On 17 Jul 2021, at 13:50, Guennadi Liakhovetski <g.liakhovetski at gmx.de> wrote:
> > > > >>>>>
> > > > >>>>> Hi Michael,
> > > > >>>>>
> > > > >>>>>> On Sat, 17 Jul 2021, Michael Gruner wrote:
> > > > >>>>>>
> > > > >>>>>> Hi Guennadi
> > > > >>>>>>
> > > > >>>>>> The videoconvert element can already handle this conversion for you:
> > > > >>>>>>
> > > > >>>>>> gst-launch-1.0 v4l2src ! videoconvert ! video/x-raw,format=RGB ! …
> > > > >>>>>
> > > > >>>>> Yes, I know about videoconvert, but I need to do some special processing
> > > > >>>>> of thermal data. For the algorithms to work I have to access the raw data,
> > > > >>>>> which is output by the v4l2 driver in GRAY16_BE format (1-to-1 from the
> > > > >>>>> camera), and I want to transform that to a colour gradient similar to well
> > > > >>>>> known thermal images, so my plugin cannot just stay within GRAY16_BE and
> > > > >>>>> use videoconvert to convert that to RGB - you'd just get a gray image of
> > > > >>>>> course.
> > > > >>>>>
> > > > >>>>>> If you, for other reason, need to write your own element, you probably
> > > > >>>>>> want to use the videofilter base class. It’s a specialization of the
> > > > >>>>>> basetransform class for video. The important virtual method for you to
> > > > >>>>>> implement is “transform_caps”, that will allow you to specify the input
> > > > >>>>>> and output caps respectively.
> > > > >>>>>
> > > > >>>>> The videofilter class is "allowed" to convert between formats? I thought
> > > > >>>>> it would be, so I actually tried it. But I was unable to get it to input
> > > > >>>>> and output different formats. I was only able to get it to work with the
> > > > >>>>> same format. Are you sure that is supposed to work?
> > > > >>>>>
> > > > >>>>> Thanks
> > > > >>>>> Guennadi
> > > > >>>>>
> > > > >>>>>> Michael
> > > > >>>>>>
> > > > >>>>>>>> On 17 Jul 2021, at 05:05, Guennadi Liakhovetski via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Hi,
> > > > >>>>>>>
> > > > >>>>>>> I'm trying to write a plugin to convert data from a thermal camera. It
> > > > >>>>>>> should take data from the v4l2src source in GRAY16_BE format and convert
> > > > >>>>>>> it to an RGB video output. I'm looking at gst-element-maker but I'm not
> > > > >>>>>>> finding any suitable base class that would take video/x-raw as input and
> > > > >>>>>>> generate the same as output and would convert between format. Do I have to
> > > > >>>>>>> write a plugin from scratch? Or should I use basetransform? Any tutorials
> > > > >>>>>>> for that?
> > > > >>>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>> Guennadi
> > > > >>>>>>> _______________________________________________
> > > > >>>>>>> gstreamer-devel mailing list
> > > > >>>>>>> gstreamer-devel at lists.freedesktop.org
> > > > >>>>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > >
> >
>