Adding MetaData to GstBuffer
Roshan Chaudhari
rgc183 at gmail.com
Thu Nov 11 09:02:18 UTC 2021
I have uploaded the project here:
https://github.com/rochaudhari/gst_metadata
I am still not able to get it working :(
In my program, there are 2 pipelines running in parallel, sender and
receiver, sender pipeline has metadata added, receiver pipeline does not
contain metadata.
Please let me know what i am missing here.
-
Thanks,
Roshan
On Tue, Sep 28, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com> wrote:
> I am using 1.19.0.1.
>
> I will try with raw text data or using multiple pipelines in the same
> process instead using udp.
>
>
> Thanks,
> Roshan
>
> On Tue, Sep 28, 2021 at 12:04 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
> wrote:
>
>> That makes sense, Your metadata is present on the sink pad of
>> rtpstpay but not present afterwards because now that audio frame has been
>> converted into RTP packet(s).
>>
>> It's really weird that at the sender you attached rtpgstdepay and was
>> able to obtain the metadata back but not at the receiver side.
>>
>> Btw, I am using Gstreamer 1.16.2.
>>
>> Best Regards,
>> Eslam Ahmed
>>
>>
>> On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <rgc183 at gmail.com>
>> wrote:
>>
>>> Quick update, from sender, I probed it on every element before
>>> rtpgstpay, meta data is present. On sinkpad of rtpgstpay, it is present but
>>> not on srcpad. If I add rtpgstdepay after rtpgstpay, data is present on
>>> srcpad of rtpgstdepay, so looks like sender is fine and audioconvert is not
>>> messing with the data.
>>>
>>> However, on the receiver, if I query after rtpgstdepay, data is not
>>> present.
>>>
>>> On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com>
>>> wrote:
>>>
>>>> I see some weird behaviour:
>>>>
>>>> I add metadata on first element of sender:
>>>>
>>>> ADD_PAD_PROBE(source, "src", Callback1,
>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>
>>>> and then:
>>>>
>>>> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>> rtpgstpay, rtpsink, NULL)){
>>>> g_error("Failed to link ");
>>>> }
>>>>
>>>> ADD_PAD_PROBE(rtpsink, "sink", Callback,
>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>
>>>> with this, on probing on last element, I dont get metadata.. however,
>>>> removing rtpgstpay above, I see metadata on rtpsink..
>>>> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata
>>>> with or without rtpgstpay.
>>>>
>>>>
>>>>
>>>> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>> wrote:
>>>>
>>>>> If I am gonna take a quick guess, I think if you probe GstMeta at the
>>>>> sender after `audioconvert` you will not find it!
>>>>>
>>>>> This is basically addressed by metadata transformation which basically
>>>>> relates to how a particular element will handle (either by
>>>>> passing/dropping/transforming) the GstMeta attached with its incoming
>>>>> buffer.
>>>>> To break it down, `audioconvert` inherits from `GstBaseTransform`
>>>>> which has a default of implementation of the transform_meta() method. See
>>>>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>>>>
>>>>> Now if `audioconvert` doesn't override GstBaseTransform's default
>>>>> implementation (which is true by my guess), then `audioconvert` will only copy
>>>>> all the metadata (GstMeta) without tags (default implementation).
>>>>> These tags are the ones you pass to gst_meta_api_type_register() and
>>>>> if you have tagged your GstMetaAPI (i.e. tags are not null) then
>>>>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>>>>
>>>>> Best Regards,
>>>>> Eslam Ahmed
>>>>>
>>>>>
>>>>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>>> wrote:
>>>>>
>>>>>> Your code should work.
>>>>>>
>>>>>> However, I stumbled upon something like that but in a video branch.
>>>>>> For me it was an element in the pipeline that was dropping the GstMeta
>>>>>> before it gets payloaded. I think your case might be similar.
>>>>>>
>>>>>> Try probing the GstBuffer and check the existing metadata after every
>>>>>> element after you have injected it and make sure that it enters the
>>>>>> payloader as is. Please confirm that first! If so then we can dig into the
>>>>>> solution.
>>>>>>
>>>>>> Best Regards,
>>>>>> Eslam Ahmed
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> So I have created sample application with sender adding metadata and
>>>>>>> receiver reading it. I am using templates for gst_Add_meta from here:
>>>>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>>>>
>>>>>>>
>>>>>>> Relevant code from my sender and receiver as follows:
>>>>>>> However, if I try to read meta data below in the same pipeline, it
>>>>>>> works.
>>>>>>>
>>>>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>>>>
>>>>>>> Sender:
>>>>>>>
>>>>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>>>>
>>>>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>>> rtpgstpay, rtpsink, NULL)
>>>>>>>
>>>>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>
>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>>
>>>>>>> GstPadProbeInfo *info,
>>>>>>> gpointer
>>>>>>> user_data) {
>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>>
>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>
>>>>>>> g_print("\n data sending from source ");
>>>>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>>>>
>>>>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>>>>
>>>>>>> meta->in_timestamp = 99000;
>>>>>>>
>>>>>>> // Below works
>>>>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>>>>> if(meta1)
>>>>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>>>>> */
>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Receiver:
>>>>>>>
>>>>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>>>>
>>>>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay,
>>>>>>> capsfilterAFX, audioconver,audioresample, fakesink,NULL);
>>>>>>> g_assert(res==TRUE);
>>>>>>>
>>>>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>
>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>> GstPadProbeInfo *info,
>>>>>>> gpointer user_data) {
>>>>>>>
>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>
>>>>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>>>>> if(!meta)
>>>>>>> g_print("\n it is null ");
>>>>>>> else
>>>>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>>>>
>>>>>>> //g_print("\n data received");
>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> However, my receiver, always get data null.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> Thanks,
>>>>>>> Roshan
>>>>>>>
>>>>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <
>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>
>>>>>>>> You are welcome!
>>>>>>>>
>>>>>>>> With gstrtppay/gstrtpdepay you should be able to
>>>>>>>> serialize/deserialize the GstBuffer over the network no matter the type of
>>>>>>>> data it contains. GstBuffer at this point could contain text,
>>>>>>>> encoded/raw frame or an audio frame.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Eslam Ahmed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you Michael and Eslam!
>>>>>>>>>
>>>>>>>>> @Eslam, Seems like all the above options you mentioned are
>>>>>>>>> specific to MPEG, or RTP?
>>>>>>>>>
>>>>>>>>> I am looking for a solution which could be applied to any data
>>>>>>>>> such as text, audio, video data.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks,
>>>>>>>>> Roshan
>>>>>>>>>
>>>>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <
>>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>>
>>>>>>>>>> Roshan,
>>>>>>>>>>
>>>>>>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>>>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>>>>>>> this out:
>>>>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>>>>
>>>>>>>>>> Just make sure to push klv metadata every video frame otherwise
>>>>>>>>>> the pipeline would stall. One useful thing to note is that your
>>>>>>>>>> metadata need not be in a klv format, you can just push arbitrary bytes as
>>>>>>>>>> long as you know how to interpret them after the demuxer at the
>>>>>>>>>> receiver end.
>>>>>>>>>>
>>>>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and
>>>>>>>>>> encode your metadata as RTP extension headers. see
>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>>>>> marker bit for that. see
>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>>>>
>>>>>>>>>> Final approach which is much more complicated which will require
>>>>>>>>>> you to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>>>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>>>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>>>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>>>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>>>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>>>>>>> video/x-mytype and restore the original stream.
>>>>>>>>>>
>>>>>>>>>> Hope that helps!
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Eslam Ahmed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via
>>>>>>>>>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Roshan
>>>>>>>>>>>
>>>>>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>>>>> has support for this.
>>>>>>>>>>>
>>>>>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>>>>>
>>>>>>>>>>> Michael
>>>>>>>>>>> www.ridgerun.con
>>>>>>>>>>>
>>>>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to append some metadata to the data transferred over
>>>>>>>>>>> udp using gstreamer.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>>>>> present in GstBuffer.
>>>>>>>>>>> 2.
>>>>>>>>>>> 3. If this is not carried over the network, what would be
>>>>>>>>>>> the other way to add metadata? I could write custom plugin and append to
>>>>>>>>>>> actual data, so my custom encoder and decoder knows how to extract real
>>>>>>>>>>> data and pass it to next stage in the pipeline. But this way, it puts
>>>>>>>>>>> restriction on the receiver side to have decoder if my data contains
>>>>>>>>>>> metadata.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Roshan
>>>>>>>>>>>
>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211111/00995d0a/attachment-0001.htm>
More information about the gstreamer-devel
mailing list