Adding MetaData to GstBuffer
Eslam Ahmed
eslam.ahmed at avidbeam.com
Thu Nov 11 11:30:57 UTC 2021
Just forgot to mention this:
Hence it makes sense to use rtpgstpay/rtpgstdepay to serialize/deserialize
the new media type (e.g. audio/x-ac3-with-metadata) over the network.
Best Regards,
Eslam Ahmed
On Thu, Nov 11, 2021 at 1:26 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
wrote:
> Hi Roshan,
>
> I have tried your code, and it will not work like this. it might be that I
> failed to make you understand me when I said, and I quote myself here:
>
> "With gstrtppay/gstrtpdepay you should be able to serialize/deserialize
> the GstBuffer over the network no matter the type of data it contains.
> GstBuffer at this point could contain text, encoded/raw frame or an audio
> frame."
>
> By that I mean that rtpgstpay/rtpgstdepay will ONLY serialize/deserialize
> the contents of the GstBuffer (GstMapInfo's data contents) excluding the
> GstMeta attached to it. That means the text, encoded/raw video frame or the
> audio frame. But you will lose all the attached meta once you are at the
> receiver side.
>
> To Recap the solutions I listed earlier:
>
> - Use the MPEG-TS container to contain your encoded stream as well as
> your metadata in a klv-format, this works well if you have a video stream
> and want to add some metadata on it. Also it has the drawback of manually
> synchronizing the video stream and klv-metadata stream. Hence, I didn't
> proceed with this approach.
> - Encode/Write your metadata as RTP extension headers. This works well
> for any payloader but has the drawback that one audio/video frame might be
> broken into multiple RTP packets based on its size. So if your metadata is
> per-frame basis you must decide which RTP packet you should use to add the
> header extension on. The marker bit should help with that but I didn't find
> it very robust to be honest.
> - Finally, create a new media type in gstreamer that has inherent
> metadata handling capabilities. For this you must write 2 elements, one
> that takes a known media type and produces the new media type by leaving
> origina contents intact (GstBuffer's data - can really
> resize/append/prepend data here) but perhaps append/prepend the metadata on
> it. The other element simply reverses the previous operation, Receives the
> new media type and produces whatever was there initially by removing the
> added metadata and restoring the original content and perhaps attaching the
> metadata again to the GstBuffer manually. It is the most complex solution
> but eliminates synchronization problems and is robust enough based on the
> quality of the 2 elements.
>
> I apologize if my previous answer confused you, and I hope this helps you.
>
> Best Regards,
> Eslam Ahmed
>
>
> On Thu, Nov 11, 2021 at 11:02 AM Roshan Chaudhari <rgc183 at gmail.com>
> wrote:
>
>> I have uploaded the project here:
>> https://github.com/rochaudhari/gst_metadata
>>
>> I am still not able to get it working :(
>> In my program, there are 2 pipelines running in parallel, sender and
>> receiver, sender pipeline has metadata added, receiver pipeline does not
>> contain metadata.
>>
>> Please let me know what i am missing here.
>>
>> -
>> Thanks,
>> Roshan
>>
>> On Tue, Sep 28, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>> wrote:
>>
>>> I am using 1.19.0.1.
>>>
>>> I will try with raw text data or using multiple pipelines in the same
>>> process instead using udp.
>>>
>>>
>>> Thanks,
>>> Roshan
>>>
>>> On Tue, Sep 28, 2021 at 12:04 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>> wrote:
>>>
>>>> That makes sense, Your metadata is present on the sink pad of
>>>> rtpstpay but not present afterwards because now that audio frame has been
>>>> converted into RTP packet(s).
>>>>
>>>> It's really weird that at the sender you attached rtpgstdepay and was
>>>> able to obtain the metadata back but not at the receiver side.
>>>>
>>>> Btw, I am using Gstreamer 1.16.2.
>>>>
>>>> Best Regards,
>>>> Eslam Ahmed
>>>>
>>>>
>>>> On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>> wrote:
>>>>
>>>>> Quick update, from sender, I probed it on every element before
>>>>> rtpgstpay, meta data is present. On sinkpad of rtpgstpay, it is present but
>>>>> not on srcpad. If I add rtpgstdepay after rtpgstpay, data is present on
>>>>> srcpad of rtpgstdepay, so looks like sender is fine and audioconvert is not
>>>>> messing with the data.
>>>>>
>>>>> However, on the receiver, if I query after rtpgstdepay, data is not
>>>>> present.
>>>>>
>>>>> On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I see some weird behaviour:
>>>>>>
>>>>>> I add metadata on first element of sender:
>>>>>>
>>>>>> ADD_PAD_PROBE(source, "src", Callback1,
>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>
>>>>>> and then:
>>>>>>
>>>>>> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>> rtpgstpay, rtpsink, NULL)){
>>>>>> g_error("Failed to link ");
>>>>>> }
>>>>>>
>>>>>> ADD_PAD_PROBE(rtpsink, "sink", Callback,
>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>
>>>>>> with this, on probing on last element, I dont get metadata.. however,
>>>>>> removing rtpgstpay above, I see metadata on rtpsink..
>>>>>> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata
>>>>>> with or without rtpgstpay.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If I am gonna take a quick guess, I think if you probe GstMeta at
>>>>>>> the sender after `audioconvert` you will not find it!
>>>>>>>
>>>>>>> This is basically addressed by metadata transformation which
>>>>>>> basically relates to how a particular element will handle (either by
>>>>>>> passing/dropping/transforming) the GstMeta attached with its incoming
>>>>>>> buffer.
>>>>>>> To break it down, `audioconvert` inherits from `GstBaseTransform`
>>>>>>> which has a default of implementation of the transform_meta() method. See
>>>>>>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>>>>>>
>>>>>>> Now if `audioconvert` doesn't override GstBaseTransform's default
>>>>>>> implementation (which is true by my guess), then `audioconvert` will only copy
>>>>>>> all the metadata (GstMeta) without tags (default implementation).
>>>>>>> These tags are the ones you pass to gst_meta_api_type_register()
>>>>>>> and if you have tagged your GstMetaAPI (i.e. tags are not null) then
>>>>>>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Eslam Ahmed
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <
>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>
>>>>>>>> Your code should work.
>>>>>>>>
>>>>>>>> However, I stumbled upon something like that but in a video branch.
>>>>>>>> For me it was an element in the pipeline that was dropping the GstMeta
>>>>>>>> before it gets payloaded. I think your case might be similar.
>>>>>>>>
>>>>>>>> Try probing the GstBuffer and check the existing metadata after
>>>>>>>> every element after you have injected it and make sure that it enters the
>>>>>>>> payloader as is. Please confirm that first! If so then we can dig into the
>>>>>>>> solution.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Eslam Ahmed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> So I have created sample application with sender adding
>>>>>>>>> metadata and receiver reading it. I am using templates for gst_Add_meta
>>>>>>>>> from here:
>>>>>>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Relevant code from my sender and receiver as follows:
>>>>>>>>> However, if I try to read meta data below in the same pipeline, it
>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>>>>>>
>>>>>>>>> Sender:
>>>>>>>>>
>>>>>>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>>>>>>
>>>>>>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>>>>> rtpgstpay, rtpsink, NULL)
>>>>>>>>>
>>>>>>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>>>
>>>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>>>>
>>>>>>>>> GstPadProbeInfo *info,
>>>>>>>>> gpointer
>>>>>>>>> user_data) {
>>>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>>>>
>>>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>>>
>>>>>>>>> g_print("\n data sending from source ");
>>>>>>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>>>>>>
>>>>>>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>>>>>>
>>>>>>>>> meta->in_timestamp = 99000;
>>>>>>>>>
>>>>>>>>> // Below works
>>>>>>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>>>>>>> if(meta1)
>>>>>>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>>>>>>> */
>>>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Receiver:
>>>>>>>>>
>>>>>>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>>>>>>
>>>>>>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay,
>>>>>>>>> capsfilterAFX, audioconver,audioresample, fakesink,NULL);
>>>>>>>>> g_assert(res==TRUE);
>>>>>>>>>
>>>>>>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>>>>
>>>>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>>>> GstPadProbeInfo *info,
>>>>>>>>> gpointer user_data) {
>>>>>>>>>
>>>>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>>>>
>>>>>>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>>>>>>> if(!meta)
>>>>>>>>> g_print("\n it is null ");
>>>>>>>>> else
>>>>>>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>>>>>>
>>>>>>>>> //g_print("\n data received");
>>>>>>>>> return GST_PAD_PROBE_OK;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> However, my receiver, always get data null.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>> Thanks,
>>>>>>>>> Roshan
>>>>>>>>>
>>>>>>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <
>>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>>
>>>>>>>>>> You are welcome!
>>>>>>>>>>
>>>>>>>>>> With gstrtppay/gstrtpdepay you should be able to
>>>>>>>>>> serialize/deserialize the GstBuffer over the network no matter the type of
>>>>>>>>>> data it contains. GstBuffer at this point could contain text,
>>>>>>>>>> encoded/raw frame or an audio frame.
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Eslam Ahmed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <
>>>>>>>>>> rgc183 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you Michael and Eslam!
>>>>>>>>>>>
>>>>>>>>>>> @Eslam, Seems like all the above options you mentioned are
>>>>>>>>>>> specific to MPEG, or RTP?
>>>>>>>>>>>
>>>>>>>>>>> I am looking for a solution which could be applied to any data
>>>>>>>>>>> such as text, audio, video data.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Roshan
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <
>>>>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Roshan,
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding Transport Stream, you can use the MPEG-TS
>>>>>>>>>>>> container to contain your encoded stream as well as your metadata in a
>>>>>>>>>>>> klv-format. Check this out:
>>>>>>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>>>>>>
>>>>>>>>>>>> Just make sure to push klv metadata every video frame otherwise
>>>>>>>>>>>> the pipeline would stall. One useful thing to note is that your
>>>>>>>>>>>> metadata need not be in a klv format, you can just push arbitrary bytes as
>>>>>>>>>>>> long as you know how to interpret them after the demuxer at the
>>>>>>>>>>>> receiver end.
>>>>>>>>>>>>
>>>>>>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and
>>>>>>>>>>>> encode your metadata as RTP extension headers. see
>>>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>>>>>>> marker bit for that. see
>>>>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>>>>>>
>>>>>>>>>>>> Final approach which is much more complicated which will
>>>>>>>>>>>> require you to create a new custom gstreamer media type (e.g.
>>>>>>>>>>>> video/x-mytype) and insert your custom metadata via
>>>>>>>>>>>> gst_buffer_add_meta() in the GstBuffer and use
>>>>>>>>>>>> rtpgstpay/rtpgstdepay which guarantee to fully serialize/deserialize the
>>>>>>>>>>>> GstBuffer over the network. You would also need to develop 2 gstreamer
>>>>>>>>>>>> elements, one to prepend/append your metadata to your au-aligned H264 and
>>>>>>>>>>>> the other to extract the metadata from video/x-mytype and restore the
>>>>>>>>>>>> original stream.
>>>>>>>>>>>>
>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Eslam Ahmed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via
>>>>>>>>>>>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Roshan
>>>>>>>>>>>>>
>>>>>>>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>>>>>>> has support for this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Michael
>>>>>>>>>>>>> www.ridgerun.con
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel
>>>>>>>>>>>>> <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am trying to append some metadata to the data transferred
>>>>>>>>>>>>> over udp using gstreamer.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>>>>>>> present in GstBuffer.
>>>>>>>>>>>>> 2.
>>>>>>>>>>>>> 3. If this is not carried over the network, what would be
>>>>>>>>>>>>> the other way to add metadata? I could write custom plugin and append to
>>>>>>>>>>>>> actual data, so my custom encoder and decoder knows how to extract real
>>>>>>>>>>>>> data and pass it to next stage in the pipeline. But this way, it puts
>>>>>>>>>>>>> restriction on the receiver side to have decoder if my data contains
>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Roshan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211111/9f448e47/attachment-0001.htm>
More information about the gstreamer-devel
mailing list