How to track buffer caps in gstreamer 1.x ?

Thu Jan 29 09:35:01 PST 2015

Thanks Thiago - I appreciate your responsiveness!

I'm at the planning stage of a port-to-gstreamer project, so I'm trying to
understand it better before I make any final design decisions.

My application is an interactive spectrogram with a Qt GUI that let's the
user modify FFT (spectrogram) parameters (which will mean gstreamer caps
changes) on-the-fly as audio data is streaming. I support both real-time
streaming (e.g. from a microphone source) as well as manual audio file
"browsing" where the user can use a scroll bar to seek through the audio
file with the visible portion being pushed thru the processing pipeline in
response to scroll movement. My out-of-order buffer processing requirement
comes from the Qt GUI which maintains a display buffer of the stream chunks
(buffers) it has received from the pipeline, and may need to render them
(needing data format/caps) in any order per the Qt paintEvents it receives.

The application currently uses my own pipeline/element design, but I want
to port to gstreamer in order to support a wider variety of audio sources
as well as spectrogram-synced audio playback capability.

Right now the GUI is the final sink of my pipeline (audio-source ->
audio-resampling -> audio-preemphasis -> FFT -> Qt-signal-adapter -> Qt
GUI), but in the future I will be feeding the output into a machine
learning program, and so would like to make my FFT/analysis component a
greamer element rather than appsink in order to retain flexibility. I may
also split this "FFT" component into multiple elements as it also does some
compute-heavy FFT post-processing that could benefit from being
multithreaded via gstreamer queue elements.

In the port to gstreamer, I'm planning to implement the GUI-driven FFT
parameter changes via a caps filter inserted after the FFT element, via
which I can set the desired caps and force the FTT to adapt. The downstream
Qt GUI will therefore receive a stream of caps change events and buffers
and per the out-of-order processing requirement I need an efficient way to
map from buffers to corresponding caps/data format. I'm thinking that if I
could add a "caps ID" tag to the GstBuffers (metadata or reuse some
existing struct member) then I could maintain a corresponding "caps ID" to
caps map in the GUI to make the association.

Hopefully this explains what I'm trying to do. Any suggestions or advice
are is very welcome, especially regarding the out-of-order buffer-to-caps
mapping.

Ben

On Thu, Jan 29, 2015 at 11:46 AM, Thiago Santos <thiagoss at osg.samsung.com>
wrote:

>  On 01/29/2015 12:52 PM, Ben Bridgwater wrote:
>
>  Thanks, Thiago.
>
>  Yes, in my application (because of the out-of-order buffer processing) I
> do need to explicitly associate buffers with caps rather than just rely on
> the caps event and buffer ordering.
>
> What is your application exactly? Are you using appsink, fakesink or a
> custom sink? Or is this another type of element?
>
>
>  I assume gstreamer pipeline elements can't actually pass GstSamples
> rather than GstBuffers - this is just a convenience class for a sink to
> associate buffers/caps/segments rather than a type that would be passed
> from element to element?
>
> Yes, only buffers are passed through pads.
>
>
>  It seems another option for me might be to use GstBuffer metadata to add
> the tagging/caps info I need. Would I be right in thinking that an element
> could receive a GstBuffer, add some metadata, then pass the GstBuffer with
> added metadata down the pipleline? Would an existing element/plugin care if
> it received a buffer with added metadata that it wasn't expecting?
> Presumably adding metadata involves an additional malloc?
>
>  Is there any way to add any metadata (maybe just an integer tag) to a
> GstBuffer without doing an additional alloc? Any GstBuffer members than it
> would be harmless to repurpose as a tag, perhaps?
>
> There is GstMeta that is a flexible metadata system that can be associated
> with buffers. You need to make sure that other elements in the pipeline
> preserve the meta when passing buffers around (specially the ones that
> create new buffers for outputing, like decoders). There are also some flags
> in GstBuffer itself that might be useful for you.
>
> In any case, associating a Caps to a Buffer is a bad idea as it is already
> implicitly associated by the previous caps event.
>
> Perhaps you could provide more details on your use case so we can provide
> better guidance.
>
>
>  Thanks,
> Ben
>
> On Thu, Jan 29, 2015 at 10:05 AM, Thiago Santos <thiagoss at osg.samsung.com>
> wrote:
>
>>   On 01/29/2015 11:53 AM, Ben Bridgwater wrote:
>>
>>  Hi,
>> I'm new to gstreamer development - converting an existing application
>> (interactive spectrogram) from using my own pipeline to use gstreamer
>> instead.
>>
>>  I gather that in gstreamer 0.1 the caps of a pad sourcing a buffer were
>> also recorded in the buffer itself, but since 1.0 that is no longer the
>> case, and it's up to elements to tracks caps via caps events instead...
>>
>>  My question is what are the options and best practices with gstreamer
>> 1.x for tracking buffer caps in the case where caps are changing rapidly
>> and buffers may be processed out of order? In my case this requirement
>> comes from a downstream display buffer where paint events may request
>> arbitrary parts of the display to be updated - corresponding to arbitrary
>> sequences of buffers needing to be processed, and the need to know what the
>> caps (data format) are for each buffer.
>>
>>
>>  The GstCaps are not stored on buffers anymore, they are only present on
>> GstPads. Instead of calling gst_pad_set_caps or pushing a buffer with a
>> caps set, now elements should push a caps event before the buffers and all
>> buffers following a caps event should have the caps of that event. You can
>> use GstPad's API to get the current negotiated caps on a pad and all
>> buffers flowing on that pad should have that caps until a new event
>> arrives. Check
>> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps
>>
>> So, to keep track of buffers and caps, you need to look for the caps
>> event. For example, suppose you receive the following events and buffers:
>>
>> CAPS_1 BUF_A BUF_B BUF_C CAPS_2 BUF_D BUF_E
>>
>> Buffers A B and C will be on format defined by CAPS_1, while buffers D
>> and E will be on format CAPS_2.
>>
>> I hope it makes it clear. If for some reason you need to store buffer +
>> caps together you can use the GstSample :
>> https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html
>>
>>
>>  Thanks for any pointers and advice!
>>
>>  Ben
>>
>>
>>
>>  _______________________________________________
>> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttp://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>>
>> --
>> Thiago Sousa Santos
>> Senior Multimedia Engineer, Open Source Group
>> Samsung Research America - Silicon Valley
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>
>
> _______________________________________________
> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttp://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
> --
> Thiago Sousa Santos
> Senior Multimedia Engineer, Open Source Group
> Samsung Research America - Silicon Valley
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20150129/763742ad/attachment-0001.html>