[gst-devel] Audio visualizations

[Matteo Landi]

On Thu, Aug 5, 2010 at 11:29 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
> Am 27.07.2010 18:53, schrieb Matteo Landi:
>> Hi guys,
>> these days I was wondering how to use gstreamer in order to create
>> some audio visualizations like an oscilloscope or a spectrum analyzer.
>>
>> The application consist of a window with a drawing area used to
>> display the visualizations.
>> Regarding the pipeline, I thought about something like
>>
>> filesrc !
>> decodebin !
>> audioconvert !
>> audio/x-raw-int, channels=1, rate=44100, width=16, depth=16, endianness=1234 !
>> tee name=t !
>>   queue !
>>     fakesink name=fakesink signal-handoffs=true t. !
>>   queue !
>>     autoaudiosink
>>
>> with the handoff callback linked to a method which spawn a new thread,
>> process the signal, and queue a redraw of the window.
>
> Have a look at gst-plugins-good/test/examples/spectrum/

Thank you, at first I haven't noticed the demo used to display bars
values read from the spectrum built in plugin. I was trying to
re-implement my own just for the curiosity-sake, nothing else :P

>
> Anyway, don't draw to gtk-windows from a custom thread. Rather do processing in
> a gst element and post the data via GstBus to the main thread and draw there.

Indeed, I suppose I was not been too clear while explaining my approach:

1 the fakesink queue a callback linked to the event (this is equal to
listening for messages on the bus): the only difference is that I need
more time for audio processing (fft) inside that callback
2 invoke a redraw method using gobject.idle_add with the fft data
processed at the previous step
3 draw the content on a backup cairo surface and invalidate the window
4 automatically redraw the window (paint the surface) using the
generated expose event.

The main difference I see with the code you linked [1], is that I
don't care at all about all those clocks operations (are they needed?)

I talked about threads, because I was thinking about spawning a new
thread for the audio process:
1 fakesink callback which spawns a new thread
2 threaded audio processing
3 gobject.idleadd ..
4 ...

There last one obscure thing: why the demo example does not initialize
threads for gobject? I thought it was necessary while working
simultaneously with gtk and gst..

>
> Stefan
>
>>
>> I got few problems with the implementation:
>> 1 - I often get core dumped, and I cannot understand why: all the
>> drawing actions are done inside the main thread, so it should be ok
>> ...
>> 2 - Synchronization between audio and video: the two sources are not
>> synchronized.. maybe is it something related to the queues elements?
>>
>> Have you got any advices regarding the chosen implementation, and how
>> to solve the listed problems?
>>
>> Thanks in advance for the attention.
>>
>>
>
>

Regards,


[1] http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/spectrum/demo-audiotest.c


-- 
Matteo Landi
http://www.matteolandi.net/