[gst-devel] Google Summer of Code Summary

[Sebastian Dröge]

Hi,
as some of you might know I did a GStreamer related google summer of
code project in the last months, with the almost longest name of all
Gnome projects "GStreamer plugins and extensions for multimedia content
creation applications"[11] ;)

Here's the summary of everything that happened in the last months:

GstController[0]:
-----------------
The first goal of my project were larger improvements to the
GstController. First of all I implemented a cubic interpolation mode,
before only linear/none and a trigger "interpolation" mode was possible.
While doing this I found many places in the source which needed cleanup
or smaller fixes and this was then what I did next. This included
optimizations, checking if the control values are in the allowed range
of the property, possible memleaks, etc.

Afterwards I refactored the rather static controller into the core
controller + GstControlSource. Before only several interpolation modes
where possible by the API, no there's a control source base class which
can be used for control sources that can do anything to return values on
timestamps. This was obviously all done in a backward compatible way by
creating GstInterpolationControlSource as first control source
implementation.

The last thing I did with the controller for the project was another
control source, GstLFOControlSource, which provides several common
waveforms (like sine, saw, triangle) for the control values.

In the future the controller definitely needs another change (or
elements using it). Currently all elements sync their properties for
each buffer but for audio this is not enough in many cases, so there's
either the way that every element needs to subdivide it's processing
loop to provide more fine grained property values or something for this
has to be added in a base class somewhere. Also a defined way of
specifying how often the properties should be updated would be nice.
Well, that's something I'd like to look into again at some point in the
future...

AudioConvert[2]:
----------------
Next goal was to implement dithering and noise shaping in audioconvert.
Before audioconvert only truncated samples when going from a format with
higher bitdepth to one with lower which introduces artifacts. To prevent
this one can add random noise to the audio data, called dithering. To
make this noise more inaudible one can pass it through a filter which
moves it to inaudible frequency ranges, this is called noise shaping.
See [12] and [13] for more information on the theory.

In audioconvert I first refactored the quantization part into a separate
processing step and then implemented several dither types and noise
shaping filters. Nowadays the default is to apply TPDF dither always
when going from a larger bit depth to a bit depth smaller or equal to
20.

MP3 seeking:
------------
Better seeking in MP3 files was the next point on my list then,
especially as Christian asked me many times when I finally start with
this.

Better seeking for MP3 files was necessary for apps like pitivi or
jokosher as without seeking to the exact position MP3 files are rather
useless.

First part I did was implementing seeking based on the Xing seek table
if present in mpegaudioparse[3]. This is still not accurate but at least
provides a bit more accurate positions already and is used always if the
Xing seek table is present.
Then I implemented really accurate seeking in mpegaudioparse by filling
a seek table with exact position while buffers are pushed through the
element. As this is memory intensive and seeking to a later position
requires on to read the complete file until that position, this is only
used if the accurate seek flag is set for a seek.
Special care was required to actually seek to a position from which on
we can decode the samples we want as MP3 frames can depend on the frames
before.

And then it was necessary to make mad[4] segment aware and implement
buffer clipping based on the playback segments. Otherwise there would be
still too much pushed downstream unfortunately. This is still to be done
for flump3dec but should be fairly easy with the gst_audio_buffer_clip()
function I added to libgstaudio.

Elements:
---------
The next part of the project was to care for random effects or filters.
Unfortunately time only permitted to write one bandpass/bandreject and
highpass/lowpas filter, care for the stereo plugin to be ported to 0.10,
making bpwsinc and lpwsinc actually useful and ready for
gst-plugins-good and care a bit for the spectrum element to give far
better results.

The first two elements I created for this goal were Chebyshev
lowpass/highpass and bandpass/bandreject filters[5]. Doing this required
me to get some deeper understanding of the theory involved, including
the Laplace and Z transforms. In the end those two filters are now very
customizable, fast and have very good results.

Afterwards I started to clean up lpwsinc and bpwsinc[6], which are
currently still in gst-plugins-bad but I hope they can be moved to
audiofx in gst-plugins-good and be renamed for consistency reasons.[14]

For those elements I first fixed some memleaks and did many other
cleanups, actually made the bandpass filter a bandpass filter,
implemented a highpass mode for lpwsinc and a bandreject mode for
bpwsinc, made the windowing function to use a property that can be
changed and wrote unit tests and documentation.
Now those are the best lowpass/highpass/bandpass/bandreject filters we
have, the rolloff is incredible good with larger kernel sizes and the
speed is also ok in most cases.

Then the most difficult part for those elements were, to drop the first
$kernel_size/2 samples and attach them at the end again. This required a
fairly difficult to read algorithm unfortunately but at least works
perfect now. Before the first $kernel_size/2 samples were essentially
silence.

Somewhere inbetween I noticed, that audiotestsrc can only output int16
samples and this always required me to plug an audioconvert between it
and the filters above. So I changed it to also output int32, float and
double samples.[9]

Also, while looking through gst-plugins-bad, I noticed that there was a
really simple, small plugin that still wasn't ported to gst0.10 so I
ported it. It's a nice effect, called stereo, but I'm not yet sure if
this is how it was intended to sound.[7]

And the last thing I did was to make the spectrum plugin better. Before
the output wasn't exactly sharp and it could only accept int16 samples
as input and many other things[2]. While doing that I ported it to a new
FFT implementation, which will hopefully become libgstfft soon. This
library can do FFT and inverse FFT on int16, int32, float and double
data and is fairly fast. See [2] for more information.


Future:
-------
In the future I definitely plan to continue to work on GStreamer and
especially on audio related stuff. For example I plan a few more effect
elements (flange, chorus, echo, ... are planned), improving the
equalizer[10] and controller[0] and for everything else that looks
interesting and is fun :)

Ok, and in the end some stats of all the changes I did for this
project :) All except the spectrum/libgstfft stuff is already in CVS.

148 files changed, 23984 insertions(+), 3725 deletions(-)


Bye


[0]http://webcvs.freedesktop.org/gstreamer/gstreamer/libs/gst/controller/
[1]http://bugzilla.gnome.org/show_bug.cgi?id=468619
[2]http://webcvs.freedesktop.org/gstreamer/gst-plugins-base/gst/audioconvert/
[3]http://webcvs.freedesktop.org/gstreamer/gst-plugins-ugly/gst/mpegaudioparse/
[4]http://webcvs.freedesktop.org/gstreamer/gst-plugins-ugly/ext/mad/
[5]http://webcvs.freedesktop.org/gstreamer/gst-plugins-good/gst/audiofx/
[6]http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst/filter/
[7]http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst/stereo/
[8]http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst/spectrum/
[9]http://webcvs.freedesktop.org/gstreamer/gst-plugins-base/gst/audiotestsrc/
[10]http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst/equalizer/
[11]http://live.gnome.org/SummerOfCode2007/AcceptedProposals#head-a78f87de6df00a834c5e7fd6ae2800d5ad3536df
[12]http://en.wikipedia.org/wiki/Dithering
[13]http://en.wikipedia.org/wiki/Noise_shaping
[14]http://bugzilla.gnome.org/show_bug.cgi?id=467666
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20070825/2c6b6ef2/attachment.pgp>