<div dir="ltr"><div>Thanks Thiago - I appreciate your responsiveness!</div><div><br></div><div>I'm at the planning stage of a port-to-gstreamer project, so I'm trying to understand it better before I make any final design decisions.</div><div><br></div><div>My application is an interactive spectrogram with a Qt GUI that let's the user modify FFT (spectrogram) parameters (which will mean gstreamer caps changes) on-the-fly as audio data is streaming. I support both real-time streaming (e.g. from a microphone source) as well as manual audio file "browsing" where the user can use a scroll bar to seek through the audio file with the visible portion being pushed thru the processing pipeline in response to scroll movement. My out-of-order buffer processing requirement comes from the Qt GUI which maintains a display buffer of the stream chunks (buffers) it has received from the pipeline, and may need to render them (needing data format/caps) in any order per the Qt paintEvents it receives.</div><div><br></div><div>The application currently uses my own pipeline/element design, but I want to port to gstreamer in order to support a wider variety of audio sources as well as spectrogram-synced audio playback capability.</div><div><br></div><div>Right now the GUI is the final sink of my pipeline (audio-source -> audio-resampling -> audio-preemphasis -> FFT -> Qt-signal-adapter -> Qt GUI), but in the future I will be feeding the output into a machine learning program, and so would like to make my FFT/analysis component a greamer element rather than appsink in order to retain flexibility. I may also split this "FFT" component into multiple elements as it also does some compute-heavy FFT post-processing that could benefit from being multithreaded via gstreamer queue elements.</div><div><br></div><div>In the port to gstreamer, I'm planning to implement the GUI-driven FFT parameter changes via a caps filter inserted after the FFT element, via which I can set the desired caps and force the FTT to adapt. The downstream Qt GUI will therefore receive a stream of caps change events and buffers and per the out-of-order processing requirement I need an efficient way to map from buffers to corresponding caps/data format. I'm thinking that if I could add a "caps ID" tag to the GstBuffers (metadata or reuse some existing struct member) then I could maintain a corresponding "caps ID" to caps map in the GUI to make the association.</div><div><br></div><div>Hopefully this explains what I'm trying to do. Any suggestions or advice are is very welcome, especially regarding the out-of-order buffer-to-caps mapping.</div><div><br></div><div>Ben</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 29, 2015 at 11:46 AM, Thiago Santos <span dir="ltr"><<a href="mailto:thiagoss@osg.samsung.com" target="_blank">thiagoss@osg.samsung.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<div>On 01/29/2015 12:52 PM, Ben Bridgwater
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Thanks, Thiago.</div>
<div><br>
</div>
<div>Yes, in my application (because of the out-of-order buffer
processing) I do need to explicitly associate buffers with
caps rather than just rely on the caps event and buffer
ordering.</div>
</div>
</blockquote></span>
What is your application exactly? Are you using appsink, fakesink or
a custom sink? Or is this another type of element?<span><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I assume gstreamer pipeline elements can't actually pass
GstSamples rather than GstBuffers - this is just a convenience
class for a sink to associate buffers/caps/segments rather
than a type that would be passed from element to element?</div>
</div>
</blockquote></span>
Yes, only buffers are passed through pads.<span><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>It seems another option for me might be to use GstBuffer
metadata to add the tagging/caps info I need. Would I be right
in thinking that an element could receive a GstBuffer, add
some metadata, then pass the GstBuffer with added metadata
down the pipleline? Would an existing element/plugin care if
it received a buffer with added metadata that it wasn't
expecting? Presumably adding metadata involves an additional
malloc?</div>
<div><br>
</div>
<div>Is there any way to add any metadata (maybe just an integer
tag) to a GstBuffer without doing an additional alloc? Any
GstBuffer members than it would be harmless to repurpose as a
tag, perhaps?</div>
</div>
</blockquote></span>
There is GstMeta that is a flexible metadata system that can be
associated with buffers. You need to make sure that other elements
in the pipeline preserve the meta when passing buffers around
(specially the ones that create new buffers for outputing, like
decoders). There are also some flags in GstBuffer itself that might
be useful for you.<br>
<br>
In any case, associating a Caps to a Buffer is a bad idea as it is
already implicitly associated by the previous caps event.<br>
<br>
Perhaps you could provide more details on your use case so we can
provide better guidance.<div><div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jan 29, 2015 at 10:05 AM,
Thiago Santos <span dir="ltr"><<a href="mailto:thiagoss@osg.samsung.com" target="_blank">thiagoss@osg.samsung.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div>
<div>On 01/29/2015 11:53 AM, Ben Bridgwater wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,</div>
<div>I'm new to gstreamer development - converting
an existing application (interactive
spectrogram) from using my own pipeline to use
gstreamer instead.</div>
<div><br>
</div>
<div>I gather that in gstreamer 0.1 the caps of a
pad sourcing a buffer were also recorded in the
buffer itself, but since 1.0 that is no longer
the case, and it's up to elements to tracks caps
via caps events instead...</div>
<div><br>
</div>
<div>My question is what are the options and best
practices with gstreamer 1.x for tracking buffer
caps in the case where caps are changing rapidly
and buffers may be processed out of order? In my
case this requirement comes from a downstream
display buffer where paint events may request
arbitrary parts of the display to be updated -
corresponding to arbitrary sequences of buffers
needing to be processed, and the need to know
what the caps (data format) are for each buffer.</div>
</div>
</blockquote>
<br>
</div>
</div>
The GstCaps are not stored on buffers anymore, they are
only present on GstPads. Instead of calling
gst_pad_set_caps or pushing a buffer with a caps set, now
elements should push a caps event before the buffers and
all buffers following a caps event should have the caps of
that event. You can use GstPad's API to get the current
negotiated caps on a pad and all buffers flowing on that
pad should have that caps until a new event arrives. Check
<a href="http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps" target="_blank">http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps</a><br>
<br>
So, to keep track of buffers and caps, you need to look
for the caps event. For example, suppose you receive the
following events and buffers:<br>
<br>
CAPS_1 BUF_A BUF_B BUF_C CAPS_2 BUF_D BUF_E<br>
<br>
Buffers A B and C will be on format defined by CAPS_1,
while buffers D and E will be on format CAPS_2.<br>
<br>
I hope it makes it clear. If for some reason you need to
store buffer + caps together you can use the GstSample : <a href="https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html" target="_blank">https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html</a><br>
<br>
<blockquote type="cite"><span>
<div dir="ltr">
<div><br>
</div>
<div>Thanks for any pointers and advice!</div>
<div><br>
</div>
<div>Ben</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</span>
<pre>_______________________________________________
gstreamer-devel mailing list
<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>
<a href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<pre cols="72">--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley</pre>
</font></span></div>
<br>
_______________________________________________<br>
gstreamer-devel mailing list<br>
<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a><br>
<a href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
gstreamer-devel mailing list
<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>
<a href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley</pre>
</div></div></div>
<br>_______________________________________________<br>
gstreamer-devel mailing list<br>
<a href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a><br>
<a href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><br>
<br></blockquote></div><br></div>