Adding MetaData to GstBuffer
Roshan Chaudhari
rgc183 at gmail.com
Mon Nov 15 12:15:02 UTC 2021
Thank you for looking into my code and providing the guidance.
--
Thanks,
Roshan
On Thu, Nov 11, 2021 at 5:01 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
wrote:
> Just forgot to mention this:
> Hence it makes sense to use rtpgstpay/rtpgstdepay to serialize/deserialize
> the new media type (e.g. audio/x-ac3-with-metadata) over the network.
>
> Best Regards,
> Eslam Ahmed
>
>
> On Thu, Nov 11, 2021 at 1:26 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
> wrote:
>
>> Hi Roshan,
>>
>> I have tried your code, and it will not work like this. it might be that
>> I failed to make you understand me when I said, and I quote myself here:
>>
>> "With gstrtppay/gstrtpdepay you should be able to serialize/deserialize
>> the GstBuffer over the network no matter the type of data it contains.
>> GstBuffer at this point could contain text, encoded/raw frame or an audio
>> frame."
>>
>> By that I mean that rtpgstpay/rtpgstdepay will ONLY serialize/deserialize
>> the contents of the GstBuffer (GstMapInfo's data contents) excluding the
>> GstMeta attached to it. That means the text, encoded/raw video frame or the
>> audio frame. But you will lose all the attached meta once you are at the
>> receiver side.
>>
>> To Recap the solutions I listed earlier:
>>
>> - Use the MPEG-TS container to contain your encoded stream as well as
>> your metadata in a klv-format, this works well if you have a video stream
>> and want to add some metadata on it. Also it has the drawback of manually
>> synchronizing the video stream and klv-metadata stream. Hence, I didn't
>> proceed with this approach.
>> - Encode/Write your metadata as RTP extension headers. This works
>> well for any payloader but has the drawback that one audio/video frame
>> might be broken into multiple RTP packets based on its size. So if
>> your metadata is per-frame basis you must decide which RTP packet you
>> should use to add the header extension on. The marker bit should help with
>> that but I didn't find it very robust to be honest.
>> - Finally, create a new media type in gstreamer that has inherent
>> metadata handling capabilities. For this you must write 2 elements, one
>> that takes a known media type and produces the new media type by leaving
>> origina contents intact (GstBuffer's data - can really
>> resize/append/prepend data here) but perhaps append/prepend the metadata on
>> it. The other element simply reverses the previous operation, Receives the
>> new media type and produces whatever was there initially by removing the
>> added metadata and restoring the original content and perhaps attaching the
>> metadata again to the GstBuffer manually. It is the most complex solution
>> but eliminates synchronization problems and is robust enough based on the
>> quality of the 2 elements.
>>
>> I apologize if my previous answer confused you, and I hope this helps you.
>>
>> Best Regards,
>> Eslam Ahmed
>>
>>
>> On Thu, Nov 11, 2021 at 11:02 AM Roshan Chaudhari <rgc183 at gmail.com>
>> wrote:
>>
>>> I have uploaded the project here:
>>> https://github.com/rochaudhari/gst_metadata
>>>
>>> I am still not able to get it working :(
>>> In my program, there are 2 pipelines running in parallel, sender and
>>> receiver, sender pipeline has metadata added, receiver pipeline does not
>>> contain metadata.
>>>
>>> Please let me know what i am missing here.
>>>
>>> -
>>> Thanks,
>>> Roshan
>>>
>>> On Tue, Sep 28, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>> wrote:
>>>
>>>> I am using 1.19.0.1.
>>>>
>>>> I will try with raw text data or using multiple pipelines in the same
>>>> process instead using udp.
>>>>
>>>>
>>>> Thanks,
>>>> Roshan
>>>>
>>>> On Tue, Sep 28, 2021 at 12:04 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>> wrote:
>>>>
>>>>> That makes sense, Your metadata is present on the sink pad of
>>>>> rtpstpay but not present afterwards because now that audio frame has been
>>>>> converted into RTP packet(s).
>>>>>
>>>>> It's really weird that at the sender you attached rtpgstdepay and was
>>>>> able to obtain the metadata back but not at the receiver side.
>>>>>
>>>>> Btw, I am using Gstreamer 1.16.2.
>>>>>
>>>>> Best Regards,
>>>>> Eslam Ahmed
>>>>>
>>>>>
>>>>> On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Quick update, from sender, I probed it on every element before
>>>>>> rtpgstpay, meta data is present. On sinkpad of rtpgstpay, it is present but
>>>>>> not on srcpad. If I add rtpgstdepay after rtpgstpay, data is present on
>>>>>> srcpad of rtpgstdepay, so looks like sender is fine and audioconvert is not
>>>>>> messing with the data.
>>>>>>
>>>>>> However, on the receiver, if I query after rtpgstdepay, data is not
>>>>>> present.
>>>>>>
>>>>>> On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I see some weird behaviour:
>>>>>>>
>>>>>>> I add metadata on first element of sender:
>>>>>>>
>>>>>>> ADD_PAD_PROBE(source, "src", Callback1,
>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>
>>>>>>> and then:
>>>>>>>
>>>>>>> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>>> rtpgstpay, rtpsink, NULL)){
>>>>>>> g_error("Failed to link ");
>>>>>>> }
>>>>>>>
>>>>>>> ADD_PAD_PROBE(rtpsink, "sink", Callback,
>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>
>>>>>>> with this, on probing on last element, I dont get metadata..
>>>>>>> however, removing rtpgstpay above, I see metadata on rtpsink..
>>>>>>> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata
>>>>>>> with or without rtpgstpay.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <
>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>
>>>>>>>> If I am gonna take a quick guess, I think if you probe GstMeta at
>>>>>>>> the sender after `audioconvert` you will not find it!
>>>>>>>>
>>>>>>>> This is basically addressed by metadata transformation which
>>>>>>>> basically relates to how a particular element will handle (either by
>>>>>>>> passing/dropping/transforming) the GstMeta attached with its incoming
>>>>>>>> buffer.
>>>>>>>> To break it down, `audioconvert` inherits from `GstBaseTransform`
>>>>>>>> which has a default of implementation of the transform_meta() method. See
>>>>>>>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>>>>>>>
>>>>>>>> Now if `audioconvert` doesn't override GstBaseTransform's default
>>>>>>>> implementation (which is true by my guess), then `audioconvert` will only copy
>>>>>>>> all the metadata (GstMeta) without tags (default implementation).
>>>>>>>> These tags are the ones you pass to gst_meta_api_type_register()
>>>>>>>> and if you have tagged your GstMetaAPI (i.e. tags are not null) then
>>>>>>>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Eslam Ahmed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <
>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>
>>>>>>>>> Your code should work.
>>>>>>>>>
>>>>>>>>> However, I stumbled upon something like that but in a video
>>>>>>>>> branch. For me it was an element in the pipeline that was dropping the
>>>>>>>>> GstMeta before it gets payloaded. I think your case might be similar.
>>>>>>>>>
>>>>>>>>> Try probing the GstBuffer and check the existing metadata after
>>>>>>>>> every element after you have injected it and make sure that it enters the
>>>>>>>>> payloader as is. Please confirm that first! If so then we can dig into the
>>>>>>>>> solution.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Eslam Ahmed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> So I have created sample application with sender adding
>>>>>>>>>> metadata and receiver reading it. I am using templates for gst_Add_meta
>>>>>>>>>> from here:
>>>>>>>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Relevant code from my sender and receiver as follows:
>>>>>>>>>> However, if I try to read meta data below in the same pipeline,
>>>>>>>>>> it works.
>>>>>>>>>>
>>>>>>>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>>>>>>>
>>>>>>>>>> Sender:
>>>>>>>>>>
>>>>>>>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>>>>>>>
>>>>>>>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>>>>>> rtpgstpay, rtpsink, NULL)
>>>>>>>>>>
>>>>>>>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>>>>
>>>>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>>>>>
>>>>>>>>>> GstPadProbeInfo *info,
>>>>>>>>>> gpointer
>>>>>>>>>> user_data) {
>>>>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>>>>>
>>>>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>>>>
>>>>>>>>>> g_print("\n data sending from source ");
>>>>>>>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>>>>>>>
>>>>>>>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>>>>>>>
>>>>>>>>>> meta->in_timestamp = 99000;
>>>>>>>>>>
>>>>>>>>>> // Below works
>>>>>>>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>>>>>>>> if(meta1)
>>>>>>>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>>>>>>>> */
>>>>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Receiver:
>>>>>>>>>>
>>>>>>>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>>>>>>>
>>>>>>>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay,
>>>>>>>>>> capsfilterAFX, audioconver,audioresample, fakesink,NULL);
>>>>>>>>>> g_assert(res==TRUE);
>>>>>>>>>>
>>>>>>>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>>>>
>>>>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>>>>> GstPadProbeInfo *info,
>>>>>>>>>> gpointer user_data) {
>>>>>>>>>>
>>>>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>>>>
>>>>>>>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>>>>>>>> if(!meta)
>>>>>>>>>> g_print("\n it is null ");
>>>>>>>>>> else
>>>>>>>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>>>>>>>
>>>>>>>>>> //g_print("\n data received");
>>>>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> However, my receiver, always get data null.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -
>>>>>>>>>> Thanks,
>>>>>>>>>> Roshan
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <
>>>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> You are welcome!
>>>>>>>>>>>
>>>>>>>>>>> With gstrtppay/gstrtpdepay you should be able to
>>>>>>>>>>> serialize/deserialize the GstBuffer over the network no matter the type of
>>>>>>>>>>> data it contains. GstBuffer at this point could contain text,
>>>>>>>>>>> encoded/raw frame or an audio frame.
>>>>>>>>>>>
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Eslam Ahmed
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <
>>>>>>>>>>> rgc183 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you Michael and Eslam!
>>>>>>>>>>>>
>>>>>>>>>>>> @Eslam, Seems like all the above options you mentioned are
>>>>>>>>>>>> specific to MPEG, or RTP?
>>>>>>>>>>>>
>>>>>>>>>>>> I am looking for a solution which could be applied to any data
>>>>>>>>>>>> such as text, audio, video data.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Roshan
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <
>>>>>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Roshan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding Transport Stream, you can use the MPEG-TS
>>>>>>>>>>>>> container to contain your encoded stream as well as your metadata in a
>>>>>>>>>>>>> klv-format. Check this out:
>>>>>>>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just make sure to push klv metadata every video frame
>>>>>>>>>>>>> otherwise the pipeline would stall. One useful thing to note is that your
>>>>>>>>>>>>> metadata need not be in a klv format, you can just push arbitrary bytes as
>>>>>>>>>>>>> long as you know how to interpret them after the demuxer at the
>>>>>>>>>>>>> receiver end.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and
>>>>>>>>>>>>> encode your metadata as RTP extension headers. see
>>>>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>>>>>>>> marker bit for that. see
>>>>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>>>>>>>
>>>>>>>>>>>>> Final approach which is much more complicated which will
>>>>>>>>>>>>> require you to create a new custom gstreamer media type (e.g.
>>>>>>>>>>>>> video/x-mytype) and insert your custom metadata via
>>>>>>>>>>>>> gst_buffer_add_meta() in the GstBuffer and use
>>>>>>>>>>>>> rtpgstpay/rtpgstdepay which guarantee to fully serialize/deserialize the
>>>>>>>>>>>>> GstBuffer over the network. You would also need to develop 2 gstreamer
>>>>>>>>>>>>> elements, one to prepend/append your metadata to your au-aligned H264 and
>>>>>>>>>>>>> the other to extract the metadata from video/x-mytype and restore the
>>>>>>>>>>>>> original stream.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Eslam Ahmed
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via
>>>>>>>>>>>>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Roshan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The GstMeta is not transmitted through network. If your want
>>>>>>>>>>>>>> a standard way to transmit metadata over the network along with audio and
>>>>>>>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>>>>>>>> has support for this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another option may be to use a codec that allows you to
>>>>>>>>>>>>>> insert custom data within the compressed bitstream (like SEI in H264 or
>>>>>>>>>>>>>> H265). That, however is a bit more tricky to get right.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>> www.ridgerun.con
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via
>>>>>>>>>>>>>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am trying to append some metadata to the data transferred
>>>>>>>>>>>>>> over udp using gstreamer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>>>>>>>> present in GstBuffer.
>>>>>>>>>>>>>> 2.
>>>>>>>>>>>>>> 3. If this is not carried over the network, what would be
>>>>>>>>>>>>>> the other way to add metadata? I could write custom plugin and append to
>>>>>>>>>>>>>> actual data, so my custom encoder and decoder knows how to extract real
>>>>>>>>>>>>>> data and pass it to next stage in the pipeline. But this way, it puts
>>>>>>>>>>>>>> restriction on the receiver side to have decoder if my data contains
>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Roshan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211115/7953a80d/attachment-0001.htm>
More information about the gstreamer-devel
mailing list