[gst-devel] Linux audio is a mess? [was: JACK and GStreamer, from the horse's mouth]

[Erik Walthinsen]

I was asked to comment on this thread, so for better or worse, here goes:

Ronald S. Bultje wrote:
> You guys are making far too big a problem out of this.
I agree, but not in quite the way you mean.

The question originally posed was "why is there no [functional] Jack 
integration into GStreamer?".  That's not really what I'm seeing debated 
here.  Granted, someone changed the Subject to better reflect that, but I 
don't see anyone here (except Wingo's hints) actually answering the original 
question.  Doing so could eliminate many of the questions being raised.

Paul Davis wrote:
 > of course, most of the time, Joe A. User doesn't want to do this, and
 > just wants their app to run by itself, without JACK. so now, the app
 > developer has to handle two different backends. what is the developer to
 > do? the worst approach, of course, is for them to try to build
 > everything into their VOIP app.
 > ...
 > there is also the issue that lots of
 > developers want to write apps that port to OS X (and possibly windows)
 > so it becomes necessary to not have to redesign the entire data flow
 > through the application when doing so.
This is almost exactly the definition of GStreamer's problem space.  Inputs 
and outputs are all plugins, and the app-visible dataflow is the same either 
way (ignoring for the moment the details of the push- vs pull- argument, see 
below).  As has already been proven by test code and I believe some apps, 
portability at the *application* level between the original Linux platform 
and others like Windows and Sun is simply a matter of dropping in the 
appropriate plugin.  Heck, even GNOME does it with the gconf keys to select 
the preferred output plugins.

Paul Davis wrote:
 > Christian Schaller wrote:
 > > So until someone either starts writing a GStreamer application which has
 > > Jack output as a priority I don't see this changing.
 > i thought that the whole idea of gstreamer was to remove this kind of
 > pressure from app developers.
Not sure why the technical concept of GStreamer removes the supply-demand 
rule of software development (*especially* open-source development). 
Software is written when demand is present, regardless of whether that comes 
from users or developers (usually the same people in open-source, 
initially).  It has always been the rule, for *any* modular framework, that 
entirely optional components don't get spontaneously created unless someone 
has a use for them.  Most of the existing GStreamer plugins, especially the 
esoteric ones, are currently maintained by the people who wrote the first 
apps that needed them.

Ronald S. Bultje wrote:
> What use case is jack ("pro audio"? - come on, it's all or nothing) or
No, it is most definitely *not* "all or nothing".  In "consumer" 
applications latency is not particularly critical (generally +-1 video frame 
is acceptable, e.g. 16-32ms), nor is sample/track alignment a huge deal. 
The total datarates are also rather undemanding.  Pro environments must very 
strictly control all of these: ~5ms in-system latency is on the ragged edge 
for any live work, multitrack alignment must be maintained at the sample 
level (e.g. between multiple cards), and a medium-sized system could be 
running 32 in and 32 out at 32-bit float and 96KHz, or nearly 25MB/sec of 
just I/O not to mention processing.

> pulseaudio ("the non-linux desktop unix/gnome user"?) trying to address?
I agree that in "our" ideal world, GStreamer will be the basis for *all* 
audio applications and thus the output modularity it provides (as per above) 
makes PulseAudio moot, but that's not going to happen.  Thus it 
(specifically its cross-platform nature) most certainly *is* relevant.

Christian F.K. Schaller wrote:
 > ALSA API's are not nice
I have to heartily agree with this, having made several abortive/low-prio 
attempts to figure out how to do anything at all with it, let alone handle 
the Hammerfall card I have.  Unfortunately, significantly due to this 
complexity, the ALSA plugin in GStreamer seems to be very much hardcoded to 
handle *only* consumer-level cards (stereo, 16-bit, 44.1KHz and minor 
variations) and thus is completely useless for any other cards.

OTOH this is what Jack is supposed to solve.  My primary goal in starting 
GStreamer in the first place was to build a live-audio mixing environment. 
For all sorts of reasons I've never gotten around to it, but one of these 
days soon I'll really have to start.  *THE* missing feature is full Jack 
integration.  Without I/O, processing is irrelevant.  With it, building a 
basic mixer is pretty easy.  At a minimum level, a functional and full-spec 
mixmatrix element is needed.  The next step would be a custom scheduler 
that's designed to operate at very low latencies, in cooperation with Jack, 
with some consideration given to cache-thrash avoidance.  Finally, all 
manner of processing elements (EQ, compressor, reverb, etc.) can be added, 
many in the form of existing LADSPA plugins.

Andy Wingo wrote:
 > But I think
 > whenever somebody gets off their ass, we will have two kinds of jack
 > integration: jack elements that drive the pipeline from the jack thread,
 > and jack elements that are separated from the main pipeline via a ring
 > buffer.
This is something I haven't kept up on, unfortunately, but in earlier 
versions there was the abstraction of a pipeline scheduler that was 
responsible for handling the potentially bidirectional nature of elements. 
On the face of things I don't see such things in the 0.10 docs, but I'm 
hoping that some of the same functionality is still present.  The original 
intent was to support the normal, buffering-heavy scheduling for things like 
media players, and also support customized schedulers for things like the 
aforementioned low-latency environments.  Both of these models can be 
available, and integrate with Jack elements (possibly two sets) in order to 
handle both the "consumer" and "pro" style applications.

Christian Schaller wrote:
 > I do fear that the discussion at DAM-3 might end up just as unfruitful
 > unless OSDL have figured out what problem they feel needs addressing as
 > there are still a gigantic host of them we need to tackle at some point.
I will be at the DAM-3 meetings next week (30min drive across town <g>), and 
I know Paul Davis will be, but I don't know who else.  However, given your 
comments we should perhaps start out the multimedia side of things by 
addressing this exact issue.  As this discussion reveals, we have ourselves 
a big hairy mess with a lot of competing projects and agendas and egos 
(myself included), and I can only imagine what that big hairy mess could 
look like all in the same room.

The DAM-3 meeting looks pretty packed, leaving relatively little room for 
focusing in on this topic, but I suspect there will be a fair amount of 
"informal" discussions while eating and beering ("Coke please!"), so 
hopefully we can come to some reasonable consensus.

TTYL,
     Omega
     aka Erik Walthinsen
     omega at temple-baptist.com