<div dir="ltr">Hi Roshan,<div><br></div><div>I have tried your code, and it will not work like this. it might be that I failed to make you understand me when I said, and I quote myself here:</div><div><br></div><div>"With gstrtppay/gstrtpdepay you should be able to serialize/deserialize the GstBuffer over the network no matter the type of data it contains. GstBuffer at this point could contain text, encoded/raw frame or an audio frame."</div><div><br></div><div>By that I mean that rtpgstpay/rtpgstdepay will ONLY serialize/deserialize the contents of the GstBuffer (GstMapInfo's data contents) excluding the GstMeta attached to it. That means the text, encoded/raw video frame or the audio frame. But you will lose all the attached meta once you are at the receiver side.</div><div><br></div><div>To Recap the solutions I listed earlier:</div><div><ul><li>Use the MPEG-TS container to contain your encoded stream as well as your metadata in a klv-format, this works well if you have a video stream and want to add some metadata on it. Also it has the drawback of manually synchronizing the video stream and klv-metadata stream. Hence, I didn't proceed with this approach.<br></li><li>Encode/Write your metadata as RTP extension headers. This works well for any payloader but has the drawback that one audio/video frame might be broken into multiple RTP packets based on its size. So if your metadata is per-frame basis you must decide which RTP packet you should use to add the header extension on. The marker bit should help with that but I didn't find it very robust to be honest.</li><li>Finally, create a new media type in gstreamer that has inherent metadata handling capabilities. For this you must write 2 elements, one that takes a known media type and produces the new media type by leaving origina contents intact (GstBuffer's data - can really resize/append/prepend data here) but perhaps append/prepend the metadata on it. The other element simply reverses the previous operation, Receives the new media type and produces whatever was there initially by removing the added metadata and restoring the original content and perhaps attaching the metadata again to the GstBuffer manually. It is the most complex solution but eliminates synchronization problems and is robust enough based on the quality of the 2 elements. </li></ul></div><div>I apologize if my previous answer confused you, and I hope this helps you.</div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 11, 2021 at 11:02 AM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">I have uploaded the project here: <a href="https://github.com/rochaudhari/gst_metadata" target="_blank">https://github.com/rochaudhari/gst_metadata</a></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I am still not able to get it working :(</div><div class="gmail_default" style="font-family:verdana,sans-serif">In my program, there are 2 pipelines running in parallel, sender and receiver, sender pipeline has metadata added, receiver pipeline does not contain metadata.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Please let me know what i am missing here.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">-</div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:verdana,sans-serif">Roshan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 28, 2021 at 1:00 PM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com" target="_blank">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">I am using 1.19.0.1. </div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I will try with raw text data or using multiple pipelines in the same process instead using udp. </div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:verdana,sans-serif">Roshan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 28, 2021 at 12:04 PM Eslam Ahmed <<a href="mailto:eslam.ahmed@avidbeam.com" target="_blank">eslam.ahmed@avidbeam.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>That makes sense, Your metadata is present on the sink pad of rtpstpay but not present afterwards because now that audio frame has been converted into RTP packet(s).</div><div><br></div><div>It's really weird that at the sender you attached rtpgstdepay and was able to obtain the metadata back but not at the receiver side.</div><div><br></div><div>Btw, I am using Gstreamer 1.16.2.</div><br clear="all"><div><div dir="ltr"><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com" target="_blank">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Quick update, from sender, I probed it on every element before rtpgstpay, meta data is present. On sinkpad of rtpgstpay, it is present but not on srcpad. If I add rtpgstdepay after rtpgstpay, data is present on srcpad of rtpgstdepay, so looks like sender is fine and audioconvert is not messing with the data.<br><br>However, on the receiver, if I query after rtpgstdepay, data is not present. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com" target="_blank">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">I see some weird behaviour:</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I add metadata on first element of sender:<br> </div><div class="gmail_default" style="font-family:verdana,sans-serif">ADD_PAD_PROBE(source, "src", Callback1,<br> GST_PAD_PROBE_TYPE_BUFFER, nullptr);<br><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">and then:<br><br> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX, rtpgstpay, rtpsink, NULL)){<br> g_error("Failed to link ");<br> }<br><br> ADD_PAD_PROBE(rtpsink, "sink", Callback,<br> GST_PAD_PROBE_TYPE_BUFFER, nullptr);<br><br>with this, on probing on last element, I dont get metadata.. however, removing rtpgstpay above, I see metadata on rtpsink..<br>Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata with or without rtpgstpay.<br><br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <<a href="mailto:eslam.ahmed@avidbeam.com" target="_blank">eslam.ahmed@avidbeam.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>If I am gonna take a quick guess, I think if you probe GstMeta at the sender after `audioconvert` you will not find it!</div><div><br></div><div>This is basically addressed by metadata transformation which basically relates to how a particular element will handle (either by passing/dropping/transforming) the GstMeta attached with its incoming buffer.</div><div>To break it down, `audioconvert` inherits from `GstBaseTransform` which has a default of implementation of the transform_meta() method. See <a href="https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta" target="_blank">https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta</a></div><div><br></div><div>Now if `audioconvert` doesn't override GstBaseTransform's default implementation (which is true by my guess), then `audioconvert` will only <span style="background-color:rgb(34,36,38);font-family:"Source Sans Pro","Source Sans",sans-serif;font-size:14px">copy </span><span style="background-color:rgb(34,36,38);font-family:"Source Sans Pro","Source Sans",sans-serif">all the metadata (GstMeta) without tags (default implementation).</span></div><div><span style="background-color:rgb(34,36,38);font-family:"Source Sans Pro","Source Sans",sans-serif">These tags are the ones you pass to </span>gst_meta_api_type_register() and if you have tagged your GstMetaAPI (i.e. tags are not null) then `audioconvert` will not copy (drop) the metadata into the output buffer.</div><br clear="all"><div><div dir="ltr"><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <<a href="mailto:eslam.ahmed@avidbeam.com" target="_blank">eslam.ahmed@avidbeam.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Your code should work.</div><div><br></div>However, I stumbled upon something like that but in a video branch. For me it was an element in the pipeline that was dropping the GstMeta before it gets payloaded. I think your case might be similar.<div><br></div><div>Try probing the GstBuffer and check the existing metadata after every element after you have injected it and make sure that it enters the payloader as is. Please confirm that first! If so then we can dig into the solution.<br clear="all"><div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com" target="_blank">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">So I have created sample application with sender adding metadata and receiver reading it. I am using templates for gst_Add_meta from here: <a href="http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/" target="_blank">http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/</a> </div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Relevant code from my sender and receiver as follows: <br>However, if I try to read meta data below in the same pipeline, it works.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">rtpsink is udpsink and rtpsrc is udpsrc.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Sender:<br><br>#define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"<br><br>gst_element_link_many(source, wavparse, audioconv,capsfilterAFX, rtpgstpay, rtpsink, NULL)<br><br>ADD_PAD_PROBE(source, "src", Callback,<br> GST_PAD_PROBE_TYPE_BUFFER, nullptr);<br><br>GstPadProbeReturn Callback(GstPad *pad,<br> GstPadProbeInfo *info,<br> gpointer user_data) {<br> GstBuffer *buffer = (GstBuffer *)info->data;<br> <br> GstMapInfo inmap = GST_MAP_INFO_INIT;<br> <br> g_print("\n data sending from source ");<br> GstMapInfo map = GST_MAP_INFO_INIT;<br><br> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);<br><br> meta->in_timestamp = 99000;<br> <br> // Below works<br> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);<br> if(meta1)<br> g_print("\n data is %lu ", meta1->in_timestamp);<br> */<br> return GST_PAD_PROBE_OK; <br><br>}<br><br><br>Receiver:<br><br>// capsfilterAFX are same as sender AUDIO_CAPS<br><br>gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay, capsfilterAFX, audioconver,audioresample, fakesink,NULL);<br>g_assert(res==TRUE);<br><br>ADD_PAD_PROBE(capsfilterAFX, "src", Callback,<br> GST_PAD_PROBE_TYPE_BUFFER, nullptr);<br><br>GstPadProbeReturn Callback(GstPad *pad,<br> GstPadProbeInfo *info,<br> gpointer user_data) {<br> <br> GstBuffer *buffer = (GstBuffer *)info->data;<br> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;<br> GstMapInfo inmap = GST_MAP_INFO_INIT;<br><br> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);<br> if(!meta)<br> g_print("\n it is null ");<br> else<br> g_print("\n data received %lu", meta->in_timestamp);<br> <br> //g_print("\n data received");<br> return GST_PAD_PROBE_OK;<br>}<br><br><br><br>However, my receiver, always get data null.<br><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">-</div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:verdana,sans-serif">Roshan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <<a href="mailto:eslam.ahmed@avidbeam.com" target="_blank">eslam.ahmed@avidbeam.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>You are welcome!</div><div><br></div><div>With gstrtppay/gstrtpdepay you should be able to serialize/deserialize the GstBuffer over the network no matter the type of data it contains. GstBuffer at this point could contain text, encoded/raw frame or an audio frame.</div><div><br></div><div><div dir="ltr"><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <<a href="mailto:rgc183@gmail.com" target="_blank">rgc183@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Thank you Michael and Eslam!</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">@Eslam, Seems like all the above options you mentioned are specific to MPEG, or RTP?<br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I am looking for a solution which could be applied to any data such as text, audio, video data.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">--</div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:verdana,sans-serif">Roshan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <<a href="mailto:eslam.ahmed@avidbeam.com" target="_blank">eslam.ahmed@avidbeam.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Roshan,</div><div><br></div><div>Regarding Transport Stream, you can use the MPEG-TS container to contain your encoded stream as well as your metadata in a klv-format. Check this out: <a href="https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3" target="_blank">https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3</a></div><div><br></div><div>Just make sure to push klv metadata every video frame otherwise the pipeline would stall. One useful thing to note is that your metadata need not be in a klv format, you can just push arbitrary bytes as long as you know how to interpret them after the demuxer at the receiver end.</div><div><br></div><div>Another approach is to use payloaders (e.g. rtph264pay) and encode your metadata as RTP extension headers. see <a href="https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header" target="_blank">https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header</a>. If your metadata is per-frame basis, then most probably and based on the encoded frame size, one frame might end up having multiple RTP packets so you will need to handle where to insert your metadata. You can use the marker bit for that. see <a href="https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker" target="_blank">https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker</a></div><div><br></div><div>Final approach which is much more complicated which will require you to create a new custom gstreamer media type (e.g. video/x-mytype) and insert your custom metadata via <span style="color:rgb(232,230,227);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">gst_buffer_add_meta()</span> in the GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully serialize/deserialize the GstBuffer over the network. You would also need to develop 2 gstreamer elements, one to prepend/append your metadata to your au-aligned H264 and the other to extract the metadata from video/x-mytype and restore the original stream.</div><div><br></div><div>Hope that helps!</div><br clear="all"><div><div dir="ltr"><div dir="ltr">Best Regards,<br>Eslam Ahmed</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hello Roshan<div><br></div><div>The GstMeta is not transmitted through network. If your want a standard way to transmit metadata over the network along with audio and video you may look into Transport Stream. This will allow you to insert your custom metadata without breaking existing decoders. GStreamer already has support for this.</div><div><br></div><div>Another option may be to use a codec that allows you to insert custom data within the compressed bitstream (like SEI in H264 or H265). That, however is a bit more tricky to get right. </div><div><br><div dir="ltr">Michael</div><div id="gmail-m_6754563923581267169gmail-m_3694185518632340627gmail-m_-8124501107666192924gmail-m_-576461650351359416gmail-m_6991227839669836105gmail-m_8186088162079987691gmail-m_5416545608240255195gmail-m_-8570058427938580020gmail-m_-5587974118274190083gmail-m_-6161567490019311726gmail-m_-8342857539067565191gmail-m_1637966370249812558AppleMailSignature" dir="ltr">www.ridgerun.con</div><div dir="ltr"><br><blockquote type="cite">On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">Hello,</span><br></div><div class="gmail_quote"><div dir="ltr"><div style="font-family:verdana,sans-serif"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br></div><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">I am trying to append some metadata to the data transferred over udp using gstreamer.<span style="font-family:verdana,sans-serif;font-size:12pt"></span></div><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br></div><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><ol><li>I
would like to know whether metadata in GstBuffer is
transferred/retained when transferred over the network or it is only
retained in that pipeline? I have tried using my custom metadata with
gst_buffer_add_meta() at the udpsink, however, when I try to query it on
udpsrc on other machine using gst_buffer_get_meta(), metadata is not
present in GstBuffer. <br></li><li style="display:block"><br></li><li>If
this is not carried over the network, what would be the other way to
add metadata? I could write custom plugin and append to actual data, so
my custom encoder and decoder knows how to extract real data and pass it
to next stage in the pipeline. But this way, it puts restriction on the
receiver side to have decoder if my data contains metadata.</li></ol><div><br></div><div>-<br></div><div>Thanks,</div><div>Roshan<br></div></div><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br></div></div></div>
</div></div>
</div></blockquote></div></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>