[gst-devel] Audio visualizations

[Matteo Landi]

On Sat, Aug 7, 2010 at 1:26 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
> Am 07.08.2010 13:47, schrieb Matteo Landi:
>> On Fri, Aug 6, 2010 at 9:01 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>>> Am 06.08.2010 14:48, schrieb Matteo Landi:
>>>> On Thu, Aug 5, 2010 at 11:29 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>>>>> Am 27.07.2010 18:53, schrieb Matteo Landi:
>>>>>> Hi guys,
>>>>>> these days I was wondering how to use gstreamer in order to create
>>>>>> some audio visualizations like an oscilloscope or a spectrum analyzer.
>>>>>>
>>>>>> The application consist of a window with a drawing area used to
>>>>>> display the visualizations.
>>>>>> Regarding the pipeline, I thought about something like
>>>>>>
>>>>>> filesrc !
>>>>>> decodebin !
>>>>>> audioconvert !
>>>>>> audio/x-raw-int, channels=1, rate=44100, width=16, depth=16, endianness=1234 !
>>>>>> tee name=t !
>>>>>>   queue !
>>>>>>     fakesink name=fakesink signal-handoffs=true t. !
>>>>>>   queue !
>>>>>>     autoaudiosink
>>>>>>
>>>>>> with the handoff callback linked to a method which spawn a new thread,
>>>>>> process the signal, and queue a redraw of the window.
>>>>>
>>>>> Have a look at gst-plugins-good/test/examples/spectrum/
>>>>
>>>> Thank you, at first I haven't noticed the demo used to display bars
>>>> values read from the spectrum built in plugin. I was trying to
>>>> re-implement my own just for the curiosity-sake, nothing else :P
>>>>
>>>>>
>>>>> Anyway, don't draw to gtk-windows from a custom thread. Rather do processing in
>>>>> a gst element and post the data via GstBus to the main thread and draw there.
>>>>
>>>> Indeed, I suppose I was not been too clear while explaining my approach:
>>>>
>>>> 1 the fakesink queue a callback linked to the event (this is equal to
>>>> listening for messages on the bus): the only difference is that I need
>>>> more time for audio processing (fft) inside that callback
>>>> 2 invoke a redraw method using gobject.idle_add with the fft data
>>>> processed at the previous step
>>>> 3 draw the content on a backup cairo surface and invalidate the window
>>>> 4 automatically redraw the window (paint the surface) using the
>>>> generated expose event.
>>>>
>>>> The main difference I see with the code you linked [1], is that I
>>>> don't care at all about all those clocks operations (are they needed?)
>>>
>>> The clock stuff just sync the drawing with the playback. Just imagine lots of
>>> slow elements after the spectrum element.
>>
>> I imagined it. I thought the sync=true property was enough.
>
> Setting sync=true on the sink is making the playback synced. In the example we
> sync drawing to our custom graphs the screen to the playback time.

Ok sorry. The achieved synchronization is in term of actually
displaying the fft data of the current audio buffer. This should
prevent from analyzing and drawing spectrum data from old audio
buffered.

>>
>>>>
>>>> I talked about threads, because I was thinking about spawning a new
>>>> thread for the audio process:
>>>> 1 fakesink callback which spawns a new thread
>>>> 2 threaded audio processing
>>>> 3 gobject.idleadd ..
>>>> 4 ...
>>>>
>>>
>>> So you want to do the fft yourself, instead of using gstreamers? Have a look at
>>> http://versia.com/category/spek/ until 0.5 it used gstreamer like I described above.
>>
>> Thanks, I'll give it a look asap.
>>
>>>
>>>> There last one obscure thing: why the demo example does not initialize
>>>> threads for gobject? I thought it was necessary while working
>>>> simultaneously with gtk and gst..
>>>>
>>> It should, but gst_init does that also.
>>
>> Reading the docs of gst_init ([1]), it comes out that g_threads_init()
>> should be invoked explicitly.. that is in contrast with the example
>> inside the documentation or not?
>
> As I said, the examples should do that. I will improve the spectrum example soon
> (already did that in buzztard).

Sorry again, I misunderstood your statement.

Thanks again,

>
> Stefan
>
>>
>>
>> Regards,
>>
>>
>> [1] http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gstreamer-Gst.html#gst-init
>>
>>>
>>> Stefan
>>>
>>>>>
>>>>> Stefan
>>>>>
>>>>>>
>>>>>> I got few problems with the implementation:
>>>>>> 1 - I often get core dumped, and I cannot understand why: all the
>>>>>> drawing actions are done inside the main thread, so it should be ok
>>>>>> ...
>>>>>> 2 - Synchronization between audio and video: the two sources are not
>>>>>> synchronized.. maybe is it something related to the queues elements?
>>>>>>
>>>>>> Have you got any advices regarding the chosen implementation, and how
>>>>>> to solve the listed problems?
>>>>>>
>>>>>> Thanks in advance for the attention.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>> [1] http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/spectrum/demo-audiotest.c
>>>>
>>>>
>>>
>>>
>>
>>
>
>



-- 
Matteo Landi
http://www.matteolandi.net/