Adding MetaData to GstBuffer

Mon Sep 27 15:45:50 UTC 2021

If I am gonna take a quick guess, I think if you probe GstMeta at the
sender after `audioconvert` you will not find it!

This is basically addressed by metadata transformation which basically
relates to how a particular element will handle (either by
passing/dropping/transforming) the GstMeta attached with its incoming
buffer.
To break it down,  `audioconvert` inherits from `GstBaseTransform` which
has a default of implementation of the transform_meta() method. See
https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta

Now if `audioconvert` doesn't override GstBaseTransform's default
implementation (which is true by my guess), then `audioconvert` will only copy
all the metadata (GstMeta) without tags (default implementation).
These tags are the ones you pass to gst_meta_api_type_register() and if you
have tagged your GstMetaAPI (i.e. tags are not null) then `audioconvert`
will not copy (drop) the metadata into the output buffer.

Best Regards,
Eslam Ahmed

On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
wrote:

> Your code should work.
>
> However, I stumbled upon something like that but in a video branch. For me
> it was an element in the pipeline that was dropping the GstMeta before it
> gets payloaded. I think your case might be similar.
>
> Try probing the GstBuffer and check the existing metadata after every
> element after you have injected it and make sure that it enters the
> payloader as is. Please confirm that first! If so then we can dig into the
> solution.
>
> Best Regards,
> Eslam Ahmed
>
>
> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com> wrote:
>
>> So I have created sample application with sender adding metadata and
>> receiver reading it. I am using templates for gst_Add_meta from here:
>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>
>>
>> Relevant code from my sender and receiver as follows:
>> However, if I try to read meta data below in the same pipeline, it works.
>>
>> rtpsink is udpsink and rtpsrc is udpsrc.
>>
>> Sender:
>>
>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>
>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>> rtpgstpay, rtpsink, NULL)
>>
>> ADD_PAD_PROBE(source, "src", Callback,
>>                     GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>
>> GstPadProbeReturn Callback(GstPad *pad,
>>                                                          GstPadProbeInfo
>> *info,
>>                                                          gpointer
>> user_data) {
>>   GstBuffer *buffer = (GstBuffer *)info->data;
>>
>>   GstMapInfo inmap = GST_MAP_INFO_INIT;
>>
>>   g_print("\n data sending from source ");
>>   GstMapInfo map = GST_MAP_INFO_INIT;
>>
>>   GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>
>>   meta->in_timestamp = 99000;
>>
>>   // Below works
>>     /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>   if(meta1)
>>     g_print("\n data is %lu ", meta1->in_timestamp);
>> */
>>   return GST_PAD_PROBE_OK;
>>
>> }
>>
>>
>> Receiver:
>>
>> // capsfilterAFX are same as sender AUDIO_CAPS
>>
>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay, capsfilterAFX,
>> audioconver,audioresample, fakesink,NULL);
>> g_assert(res==TRUE);
>>
>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>                     GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>
>> GstPadProbeReturn Callback(GstPad *pad,
>>                            GstPadProbeInfo *info,
>>                            gpointer user_data) {
>>
>>   GstBuffer *buffer = (GstBuffer *)info->data;
>>   //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>   GstMapInfo inmap = GST_MAP_INFO_INIT;
>>
>>   GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>   if(!meta)
>>     g_print("\n it is null ");
>>   else
>>     g_print("\n data received %lu", meta->in_timestamp);
>>
>>   //g_print("\n data received");
>>   return GST_PAD_PROBE_OK;
>> }
>>
>>
>>
>> However, my receiver, always get data null.
>>
>>
>>
>> -
>> Thanks,
>> Roshan
>>
>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>> wrote:
>>
>>> You are welcome!
>>>
>>> With gstrtppay/gstrtpdepay you should be able to serialize/deserialize
>>> the GstBuffer over the network no matter the type of data it contains.
>>> GstBuffer at this point could contain text, encoded/raw frame or an audio
>>> frame.
>>>
>>> Best Regards,
>>> Eslam Ahmed
>>>
>>>
>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>> wrote:
>>>
>>>> Thank you Michael and Eslam!
>>>>
>>>> @Eslam, Seems like all the above options you mentioned are specific to
>>>> MPEG, or RTP?
>>>>
>>>> I am looking for a solution which could be applied to any data such as
>>>> text, audio, video data.
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Roshan
>>>>
>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>> wrote:
>>>>
>>>>> Roshan,
>>>>>
>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>> this out:
>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>
>>>>> Just make sure to push klv metadata every video frame otherwise the
>>>>> pipeline would stall. One useful thing to note is that your metadata need
>>>>> not be in a klv format, you can just push arbitrary bytes as long as you
>>>>> know how to interpret them after the demuxer at the receiver end.
>>>>>
>>>>> Another approach is to use payloaders (e.g. rtph264pay) and encode
>>>>> your metadata as RTP extension headers. see
>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>> you will need to handle where to insert your metadata. You can use the
>>>>> marker bit for that. see
>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>
>>>>> Final approach which is much more complicated which will require you
>>>>> to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>> video/x-mytype and restore the original stream.
>>>>>
>>>>> Hope that helps!
>>>>>
>>>>> Best Regards,
>>>>> Eslam Ahmed
>>>>>
>>>>>
>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via gstreamer-devel <
>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>
>>>>>> Hello Roshan
>>>>>>
>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>> has support for this.
>>>>>>
>>>>>> Another option may be to use a codec that allows you to insert custom
>>>>>> data within the compressed bitstream (like SEI in H264 or H265). That,
>>>>>> however is a bit more tricky to get right.
>>>>>>
>>>>>> Michael
>>>>>> www.ridgerun.con
>>>>>>
>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>
>>>>>> 
>>>>>> Hello,
>>>>>>
>>>>>> I am trying to append some metadata to the data transferred over udp
>>>>>> using gstreamer.
>>>>>>
>>>>>>
>>>>>>    1. I would like to know whether metadata in GstBuffer is
>>>>>>    transferred/retained when transferred over the network or it is only
>>>>>>    retained in that pipeline? I have tried using my custom metadata with
>>>>>>    gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>    udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>    present in GstBuffer.
>>>>>>    2.
>>>>>>    3. If this is not carried over the network, what would be the
>>>>>>    other way to add metadata? I could write custom plugin and append to actual
>>>>>>    data, so my custom encoder and decoder knows how to extract real data and
>>>>>>    pass it to next stage in the pipeline. But this way, it puts restriction on
>>>>>>    the receiver side to have decoder if my data contains metadata.
>>>>>>
>>>>>>
>>>>>> -
>>>>>> Thanks,
>>>>>> Roshan
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210927/4eb89fa6/attachment.htm>