[gst-devel] "bounties"/...

[Josh Green]

On Fri, 16 Jul 2004, Andy Wingo wrote:

> Hey Josh (and Jonathan),
>
> I really wonder what you want to do with MIDI. Do you want to write
> elements that deal with MIDI and alter MIDI events, or do you want to
> turn MIDI events into sound via synthesis? The "bounty" in this case
> isn't clear at all, but to be fair, neither is the idea of integrating
> MIDI into gstreamer.
>

What I would like to see happen with MIDI in GStreamer is the ability to
route MIDI from various sources (a MIDI file, ALSA sequencer ports
including external MIDI gear, etc). Along with MIDI support I am looking
into adding instrument file support (using libInstPatch), file formats
like SoundFont, DLS and GigaSampler. This latter library could provide
file format conversion/compression. Now doing something useful with those
instrument files involves plugging them into a soft synth (libswami
handles wavetable devices, currently FluidSynth, but others could be added
just by writing a new derivative object of SwamiWavetbl). So FluidSynth
could then be placed into a GStreamer pipeline with an audio output, an
instrument file input and MIDI input.

Essentially I'm looking to try and include instrument file support into
GStreamer, MIDI is just one of those things that would make it that much
more useful. I can also see that a GValue event network might be very useful
in GStreamer. How about the ability to connect a MIDI controller to a
property of a pad or other object? Connect a GUI control to a property,
etc. I guess I'm trying to see how much of my current work on Swami could
be more generally useful in a media framework like GStreamer. I'm still
not sure what parts of this project make sense to integrate into
GStreamer, so I'm still testing the water in this regard. It just seems
like so much of the work I am doing, could be generally useful to other
applications as well.

> In the first case, I would go by the timestamped-events model, or the
> filled-buffer model, but to be honest I don't care ;)
>

I still like the thought of adding a generic GValue GstData type. Anyone
have some comments on this particular thinking?

> I eventually plan on supporting MIDI in soundscrape[0], although it will
> be along the lines of the Voicer unit generator in SuperCollider (check
> the docs on www.audiosynth.com for details on that). What I think that
> means is that I will have a pipeline like voicer ! audiosink, where the
> voicer is actually a bin that contains a mixer, and when an event comes
> in, it plugs a new element chain into the mixer.
>
> This is effectively the "spawn" idea from SuperCollider. I have the
> spawner implemented in soundscrape, and it works quite well.
>
> I just wanted to let you know what I was thinking.
>

Would the mixer respond to MIDI events? How does MIDI enter into that
description?

> [0] http://ambient.2y.net/soundscrape/ -- the web site is out of date,
> but the arch archive lives :-)
>
> > - Should we use the GStreamer devel email list for discussion of this
> > topic?
>
> There are smart kids on this list, and I for one would be interested in
> hearing your designs.
>
> > - Defining GstMidiEvent is step one and getting it encapsulated in
> > GstData.
>
> In my studies of the Oshiwambo language, I came across the proverb,
> "londa omukwa noongaku." Literally it means "climb a baobab with shoes",
> but its figurative meaning is that you will experience problems with
> this way of doing things. (In the case of the baobab, you'll fall.)
>
> We've never, to my knowledge, had a pipeline that _only_ passes events.
> This will be tricky. I'm not saying that it's a bad idea, only that
> passing GstBuffer's whose data you could cast to GstMidiEvent would give
> you less problems.
>
> (Certainly your solution is cleaner.)
>

How about a new data type that is derived from GstData that is used to
pass events/control values. A GValue container would be the simplest and
most flexible way to do this (essentially SwamiControl). This would add a
lot of other nice features related to networking generic valued events
(ints, floats, objects, boxed structures, strings, etc that a GValue can
contain).

> > - GStreamer MIDI could be modeled after the ALSA sequencer, since its
> > pretty sweet.
>
> Yes, it's pretty sweet ;)
>
> (Which leads to the question, why duplicate it if it's so nice? Why pass
> MIDI data in a pipeline if the ALSA sequencer can do it much more
> generally?)
>

I have had this same thought before as well. I think portability is one
reason (GStreamer runs or will run where ALSA is not), the inability for
MIDI to be processed in a pipeline (no MIDI control of properties or
processing of MIDI files, etc). I suppose the question has already been
answered in regards to whether MIDI should be a part of GStreamer. It
sounds like it has been on the wish list for some time.

>
> > It seems like this is not much code initially and it will help get things
> > started. I'll get myself up to speed on current GStreamer CVS and API so I
> > can be of more help in actually coding on this. Cheers.
>
> Let me know if you have a question. I have my own project I'm sinking my
> time into (and we should collaborate when things are stable), but I do
> know gstreamer and I'd like to get more "pro" audio people looking at
> it. So keep me informed.
>

It seems like a lot of developers are working on their own projects. I
think a lot of technologies like JACK, the ALSA sequencer, GStreamer, etc
offer a great opportunity for co-operation and integration of individual
applications just by adding support for these technologies. Which I think
is the way in which Linux (and free software in general) can start to
really compete with other media platforms out there. I'm sure a lot of
this is already apparent to readers of this list, so I'll keep it
relatively short. I like the idea of being able to co-operate with other
developers, but still work on what we individually are most interested in.

> Cheers, and good luck,
> --
> Andy Wingo <wingo at pobox.com>
> http://ambient.2y.net/wingo/
>

Cheers.
	Josh Green
	http://swami.sourceforge.net