[gst-devel] Audio visualizations

[Stefan Kost]

Am 06.08.2010 14:48, schrieb Matteo Landi:
> On Thu, Aug 5, 2010 at 11:29 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>> Am 27.07.2010 18:53, schrieb Matteo Landi:
>>> Hi guys,
>>> these days I was wondering how to use gstreamer in order to create
>>> some audio visualizations like an oscilloscope or a spectrum analyzer.
>>>
>>> The application consist of a window with a drawing area used to
>>> display the visualizations.
>>> Regarding the pipeline, I thought about something like
>>>
>>> filesrc !
>>> decodebin !
>>> audioconvert !
>>> audio/x-raw-int, channels=1, rate=44100, width=16, depth=16, endianness=1234 !
>>> tee name=t !
>>>   queue !
>>>     fakesink name=fakesink signal-handoffs=true t. !
>>>   queue !
>>>     autoaudiosink
>>>
>>> with the handoff callback linked to a method which spawn a new thread,
>>> process the signal, and queue a redraw of the window.
>>
>> Have a look at gst-plugins-good/test/examples/spectrum/
> 
> Thank you, at first I haven't noticed the demo used to display bars
> values read from the spectrum built in plugin. I was trying to
> re-implement my own just for the curiosity-sake, nothing else :P
> 
>>
>> Anyway, don't draw to gtk-windows from a custom thread. Rather do processing in
>> a gst element and post the data via GstBus to the main thread and draw there.
> 
> Indeed, I suppose I was not been too clear while explaining my approach:
> 
> 1 the fakesink queue a callback linked to the event (this is equal to
> listening for messages on the bus): the only difference is that I need
> more time for audio processing (fft) inside that callback
> 2 invoke a redraw method using gobject.idle_add with the fft data
> processed at the previous step
> 3 draw the content on a backup cairo surface and invalidate the window
> 4 automatically redraw the window (paint the surface) using the
> generated expose event.
> 
> The main difference I see with the code you linked [1], is that I
> don't care at all about all those clocks operations (are they needed?)

The clock stuff just sync the drawing with the playback. Just imagine lots of
slow elements after the spectrum element.

> 
> I talked about threads, because I was thinking about spawning a new
> thread for the audio process:
> 1 fakesink callback which spawns a new thread
> 2 threaded audio processing
> 3 gobject.idleadd ..
> 4 ...
> 

So you want to do the fft yourself, instead of using gstreamers? Have a look at
http://versia.com/category/spek/ until 0.5 it used gstreamer like I described above.

> There last one obscure thing: why the demo example does not initialize
> threads for gobject? I thought it was necessary while working
> simultaneously with gtk and gst..
> 
It should, but gst_init does that also.

Stefan

>>
>> Stefan
>>
>>>
>>> I got few problems with the implementation:
>>> 1 - I often get core dumped, and I cannot understand why: all the
>>> drawing actions are done inside the main thread, so it should be ok
>>> ...
>>> 2 - Synchronization between audio and video: the two sources are not
>>> synchronized.. maybe is it something related to the queues elements?
>>>
>>> Have you got any advices regarding the chosen implementation, and how
>>> to solve the listed problems?
>>>
>>> Thanks in advance for the attention.
>>>
>>>
>>
>>
> 
> Regards,
> 
> 
> [1] http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/spectrum/demo-audiotest.c
> 
>