Adding MetaData to GstBuffer
Roshan Chaudhari
rgc183 at gmail.com
Tue Sep 28 05:02:45 UTC 2021
Quick update, from sender, I probed it on every element before rtpgstpay,
meta data is present. On sinkpad of rtpgstpay, it is present but not on
srcpad. If I add rtpgstdepay after rtpgstpay, data is present on srcpad of
rtpgstdepay, so looks like sender is fine and audioconvert is not messing
with the data.
However, on the receiver, if I query after rtpgstdepay, data is not
present.
On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com> wrote:
> I see some weird behaviour:
>
> I add metadata on first element of sender:
>
> ADD_PAD_PROBE(source, "src", Callback1,
> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>
> and then:
>
> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
> rtpgstpay, rtpsink, NULL)){
> g_error("Failed to link ");
> }
>
> ADD_PAD_PROBE(rtpsink, "sink", Callback,
> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>
> with this, on probing on last element, I dont get metadata.. however,
> removing rtpgstpay above, I see metadata on rtpsink..
> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata with
> or without rtpgstpay.
>
>
>
> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
> wrote:
>
>> If I am gonna take a quick guess, I think if you probe GstMeta at the
>> sender after `audioconvert` you will not find it!
>>
>> This is basically addressed by metadata transformation which basically
>> relates to how a particular element will handle (either by
>> passing/dropping/transforming) the GstMeta attached with its incoming
>> buffer.
>> To break it down, `audioconvert` inherits from `GstBaseTransform` which
>> has a default of implementation of the transform_meta() method. See
>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>
>> Now if `audioconvert` doesn't override GstBaseTransform's default
>> implementation (which is true by my guess), then `audioconvert` will only copy
>> all the metadata (GstMeta) without tags (default implementation).
>> These tags are the ones you pass to gst_meta_api_type_register() and if
>> you have tagged your GstMetaAPI (i.e. tags are not null) then
>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>
>> Best Regards,
>> Eslam Ahmed
>>
>>
>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>> wrote:
>>
>>> Your code should work.
>>>
>>> However, I stumbled upon something like that but in a video branch. For
>>> me it was an element in the pipeline that was dropping the GstMeta before
>>> it gets payloaded. I think your case might be similar.
>>>
>>> Try probing the GstBuffer and check the existing metadata after every
>>> element after you have injected it and make sure that it enters the
>>> payloader as is. Please confirm that first! If so then we can dig into the
>>> solution.
>>>
>>> Best Regards,
>>> Eslam Ahmed
>>>
>>>
>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>> wrote:
>>>
>>>> So I have created sample application with sender adding metadata and
>>>> receiver reading it. I am using templates for gst_Add_meta from here:
>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>
>>>>
>>>> Relevant code from my sender and receiver as follows:
>>>> However, if I try to read meta data below in the same pipeline, it
>>>> works.
>>>>
>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>
>>>> Sender:
>>>>
>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>
>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>> rtpgstpay, rtpsink, NULL)
>>>>
>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>
>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>
>>>> GstPadProbeInfo *info,
>>>> gpointer
>>>> user_data) {
>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>
>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>
>>>> g_print("\n data sending from source ");
>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>
>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>
>>>> meta->in_timestamp = 99000;
>>>>
>>>> // Below works
>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>> if(meta1)
>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>> */
>>>> return GST_PAD_PROBE_OK;
>>>>
>>>> }
>>>>
>>>>
>>>> Receiver:
>>>>
>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>
>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay, capsfilterAFX,
>>>> audioconver,audioresample, fakesink,NULL);
>>>> g_assert(res==TRUE);
>>>>
>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>
>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>> GstPadProbeInfo *info,
>>>> gpointer user_data) {
>>>>
>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>
>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>> if(!meta)
>>>> g_print("\n it is null ");
>>>> else
>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>
>>>> //g_print("\n data received");
>>>> return GST_PAD_PROBE_OK;
>>>> }
>>>>
>>>>
>>>>
>>>> However, my receiver, always get data null.
>>>>
>>>>
>>>>
>>>> -
>>>> Thanks,
>>>> Roshan
>>>>
>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>> wrote:
>>>>
>>>>> You are welcome!
>>>>>
>>>>> With gstrtppay/gstrtpdepay you should be able to serialize/deserialize
>>>>> the GstBuffer over the network no matter the type of data it contains.
>>>>> GstBuffer at this point could contain text, encoded/raw frame or an audio
>>>>> frame.
>>>>>
>>>>> Best Regards,
>>>>> Eslam Ahmed
>>>>>
>>>>>
>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thank you Michael and Eslam!
>>>>>>
>>>>>> @Eslam, Seems like all the above options you mentioned are specific
>>>>>> to MPEG, or RTP?
>>>>>>
>>>>>> I am looking for a solution which could be applied to any data such
>>>>>> as text, audio, video data.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Roshan
>>>>>>
>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Roshan,
>>>>>>>
>>>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>>>> this out:
>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>
>>>>>>> Just make sure to push klv metadata every video frame otherwise the
>>>>>>> pipeline would stall. One useful thing to note is that your metadata need
>>>>>>> not be in a klv format, you can just push arbitrary bytes as long as you
>>>>>>> know how to interpret them after the demuxer at the receiver end.
>>>>>>>
>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and encode
>>>>>>> your metadata as RTP extension headers. see
>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>> marker bit for that. see
>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>
>>>>>>> Final approach which is much more complicated which will require you
>>>>>>> to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>>>> video/x-mytype and restore the original stream.
>>>>>>>
>>>>>>> Hope that helps!
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Eslam Ahmed
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via gstreamer-devel <
>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>
>>>>>>>> Hello Roshan
>>>>>>>>
>>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>> has support for this.
>>>>>>>>
>>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>>
>>>>>>>> Michael
>>>>>>>> www.ridgerun.con
>>>>>>>>
>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am trying to append some metadata to the data transferred over
>>>>>>>> udp using gstreamer.
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>> present in GstBuffer.
>>>>>>>> 2.
>>>>>>>> 3. If this is not carried over the network, what would be the
>>>>>>>> other way to add metadata? I could write custom plugin and append to actual
>>>>>>>> data, so my custom encoder and decoder knows how to extract real data and
>>>>>>>> pass it to next stage in the pipeline. But this way, it puts restriction on
>>>>>>>> the receiver side to have decoder if my data contains metadata.
>>>>>>>>
>>>>>>>>
>>>>>>>> -
>>>>>>>> Thanks,
>>>>>>>> Roshan
>>>>>>>>
>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210928/ff16bcd7/attachment-0001.htm>
More information about the gstreamer-devel
mailing list