<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 01/29/2015 02:35 PM, Ben Bridgwater
wrote:<br>
</div>
<blockquote
cite="mid:CABH16+mEuGNHBfFs-q9ev0Xxi6PaWyXS_p--2duKbtCxKx3-ew@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Thanks Thiago - I appreciate your responsiveness!</div>
<div><br>
</div>
<div>I'm at the planning stage of a port-to-gstreamer project,
so I'm trying to understand it better before I make any final
design decisions.</div>
<div><br>
</div>
<div>My application is an interactive spectrogram with a Qt GUI
that let's the user modify FFT (spectrogram) parameters (which
will mean gstreamer caps changes) on-the-fly as audio data is
streaming. I support both real-time streaming (e.g. from a
microphone source) as well as manual audio file "browsing"
where the user can use a scroll bar to seek through the audio
file with the visible portion being pushed thru the processing
pipeline in response to scroll movement. My out-of-order
buffer processing requirement comes from the Qt GUI
which maintains a display buffer of the stream chunks
(buffers) it has received from the pipeline, and may need to
render them (needing data format/caps) in any order per the Qt
paintEvents it receives.</div>
</div>
</blockquote>
You can store the buffers with caps in GstSamples, you don't need to
copy the caps or the buffer, just keep a ref to them in the
GstSample and the overhead will be on the allocation of a struct,
which shouldn't be that much.<br>
<br>
If you don't need to pass this buffer around you can also have some
other custom way of identifying the format internally in your
element if you believe performance would be improved.<br>
<blockquote
cite="mid:CABH16+mEuGNHBfFs-q9ev0Xxi6PaWyXS_p--2duKbtCxKx3-ew@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>The application currently uses my own pipeline/element
design, but I want to port to gstreamer in order to support a
wider variety of audio sources as well as spectrogram-synced
audio playback capability.</div>
<div><br>
</div>
<div>Right now the GUI is the final sink of my pipeline
(audio-source -> audio-resampling -> audio-preemphasis
-> FFT -> Qt-signal-adapter -> Qt GUI), but in the
future I will be feeding the output into a machine learning
program, and so would like to make my FFT/analysis component a
greamer element rather than appsink in order to retain
flexibility. I may also split this "FFT" component into
multiple elements as it also does some compute-heavy FFT
post-processing that could benefit from being multithreaded
via gstreamer queue elements.</div>
</div>
</blockquote>
Good idea, it will make it more reusable.<br>
<blockquote
cite="mid:CABH16+mEuGNHBfFs-q9ev0Xxi6PaWyXS_p--2duKbtCxKx3-ew@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>In the port to gstreamer, I'm planning to implement the
GUI-driven FFT parameter changes via a caps filter inserted
after the FFT element, via which I can set the desired caps
and force the FTT to adapt. The downstream Qt GUI will
therefore receive a stream of caps change events and buffers
and per the out-of-order processing requirement I need an
efficient way to map from buffers to corresponding caps/data
format. I'm thinking that if I could add a "caps ID" tag to
the GstBuffers (metadata or reuse some existing struct member)
then I could maintain a corresponding "caps ID" to caps map in
the GUI to make the association.</div>
</div>
</blockquote>
You don't necessarily need the capsfilter, your element can change
the requirements on its pads dynamically and push a 'reconfigure'
event upstream to notify other elements that they should try to
renegotiate to another format. This is essentially what the
capsfilter would be doing when you set a new caps to it.<br>
<blockquote
cite="mid:CABH16+mEuGNHBfFs-q9ev0Xxi6PaWyXS_p--2duKbtCxKx3-ew@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Hopefully this explains what I'm trying to do. Any
suggestions or advice are is very welcome, especially
regarding the out-of-order buffer-to-caps mapping.</div>
</div>
</blockquote>
Hope it helps, success on your porting!<br>
<blockquote
cite="mid:CABH16+mEuGNHBfFs-q9ev0Xxi6PaWyXS_p--2duKbtCxKx3-ew@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Ben</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jan 29, 2015 at 11:46 AM,
Thiago Santos <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:thiagoss@osg.samsung.com" target="_blank">thiagoss@osg.samsung.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<div>On 01/29/2015 12:52 PM, Ben Bridgwater wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Thanks, Thiago.</div>
<div><br>
</div>
<div>Yes, in my application (because of the
out-of-order buffer processing) I do need to
explicitly associate buffers with caps rather than
just rely on the caps event and buffer ordering.</div>
</div>
</blockquote>
</span> What is your application exactly? Are you using
appsink, fakesink or a custom sink? Or is this another
type of element?<span><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I assume gstreamer pipeline elements can't
actually pass GstSamples rather than GstBuffers -
this is just a convenience class for a sink to
associate buffers/caps/segments rather than a type
that would be passed from element to element?</div>
</div>
</blockquote>
</span> Yes, only buffers are passed through pads.<span><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>It seems another option for me might be to use
GstBuffer metadata to add the tagging/caps info I
need. Would I be right in thinking that an element
could receive a GstBuffer, add some metadata, then
pass the GstBuffer with added metadata down the
pipleline? Would an existing element/plugin care
if it received a buffer with added metadata that
it wasn't expecting? Presumably adding metadata
involves an additional malloc?</div>
<div><br>
</div>
<div>Is there any way to add any metadata (maybe
just an integer tag) to a GstBuffer without doing
an additional alloc? Any GstBuffer members than it
would be harmless to repurpose as a tag, perhaps?</div>
</div>
</blockquote>
</span> There is GstMeta that is a flexible metadata
system that can be associated with buffers. You need to
make sure that other elements in the pipeline preserve the
meta when passing buffers around (specially the ones that
create new buffers for outputing, like decoders). There
are also some flags in GstBuffer itself that might be
useful for you.<br>
<br>
In any case, associating a Caps to a Buffer is a bad idea
as it is already implicitly associated by the previous
caps event.<br>
<br>
Perhaps you could provide more details on your use case so
we can provide better guidance.
<div>
<div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks,</div>
<div>Ben</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jan 29, 2015 at
10:05 AM, Thiago Santos <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:thiagoss@osg.samsung.com"
target="_blank">thiagoss@osg.samsung.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div>
<div>On 01/29/2015 11:53 AM, Ben
Bridgwater wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,</div>
<div>I'm new to gstreamer
development - converting an
existing application (interactive
spectrogram) from using my own
pipeline to use gstreamer instead.</div>
<div><br>
</div>
<div>I gather that in gstreamer 0.1
the caps of a pad sourcing a
buffer were also recorded in the
buffer itself, but since 1.0 that
is no longer the case, and it's up
to elements to tracks caps via
caps events instead...</div>
<div><br>
</div>
<div>My question is what are the
options and best practices with
gstreamer 1.x for tracking buffer
caps in the case where caps are
changing rapidly and buffers may
be processed out of order? In my
case this requirement comes from a
downstream display buffer where
paint events may request arbitrary
parts of the display to be updated
- corresponding to arbitrary
sequences of buffers needing to be
processed, and the need to know
what the caps (data format) are
for each buffer.</div>
</div>
</blockquote>
<br>
</div>
</div>
The GstCaps are not stored on buffers
anymore, they are only present on GstPads.
Instead of calling gst_pad_set_caps or
pushing a buffer with a caps set, now
elements should push a caps event before the
buffers and all buffers following a caps
event should have the caps of that event.
You can use GstPad's API to get the current
negotiated caps on a pad and all buffers
flowing on that pad should have that caps
until a new event arrives. Check <a
moz-do-not-send="true"
href="http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps"
target="_blank">http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps</a><br>
<br>
So, to keep track of buffers and caps, you
need to look for the caps event. For
example, suppose you receive the following
events and buffers:<br>
<br>
CAPS_1 BUF_A BUF_B BUF_C CAPS_2 BUF_D BUF_E<br>
<br>
Buffers A B and C will be on format defined
by CAPS_1, while buffers D and E will be on
format CAPS_2.<br>
<br>
I hope it makes it clear. If for some reason
you need to store buffer + caps together you
can use the GstSample : <a
moz-do-not-send="true"
href="https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html"
target="_blank">https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html</a><br>
<br>
<blockquote type="cite"><span>
<div dir="ltr">
<div><br>
</div>
<div>Thanks for any pointers and
advice!</div>
<div><br>
</div>
<div>Ben</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</span>
<pre>_______________________________________________
gstreamer-devel mailing list
<a moz-do-not-send="true" href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>
<a moz-do-not-send="true" href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><span><font color="#888888">
</font></span></pre>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
<br>
<pre cols="72">--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley</pre>
</font></span></div>
<br>
_______________________________________________<br>
gstreamer-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:gstreamer-devel@lists.freedesktop.org"
target="_blank">gstreamer-devel@lists.freedesktop.org</a><br>
<a moz-do-not-send="true"
href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel"
target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
gstreamer-devel mailing list
<a moz-do-not-send="true" href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>
<a moz-do-not-send="true" href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a>
</pre>
</blockquote>
<br>
<br>
<pre cols="72">--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley</pre>
</div>
</div>
</div>
<br>
_______________________________________________<br>
gstreamer-devel mailing list<br>
<a moz-do-not-send="true"
href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a><br>
<a moz-do-not-send="true"
href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel"
target="_blank">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gstreamer-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a>
<a class="moz-txt-link-freetext" href="http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel">http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley</pre>
</body>
</html>