[gst-devel] potential for midi support in GStreamer

[Andy Wingo]

On Mon, 17 Jun 2002, Baker, Steve wrote:

> I've thought a bit about how to add midi support to GStreamer. I'm just
> going to do a brain dump of my ideas since I doubt I'll be coding any of
> this any time soon.

I've thought a little bit about this too. Lemme first give you my
vision, and then comment on yours.

However, before starting, I have to say that I'm not a midi expert. I
kinda understand the alsa seq api, and know what midi can do as far as
hardware integration, but I'm not an expert. It would be good to hear
what vektor has to say about all this (you listening, billy?)

Unfortunately, the full MIDI specification is only available for a fee
(around USD 60), so we have to go by other references... I found
http://www.midi.org/about-midi/tutorial/tutor.htm to be useful, although
you've got to up the font size a bit ;)

> Midi could be thought of in terms of dataflow as a sparse non-constant flow
> of bytes. GStreamer works best with near-constant data flow so a midi stream
> would probably have to consist mostly of filler events, sent at a constant
> tick-rate.

As I understand it, on-the-wire hardware midi connections run at a fixed
data rate, quoting from the page above:
  
  The MIDI data stream is a unidirectional asynchronous bit stream at
  31.25 Kbits/sec. with 10 bits transmitted per byte (a start bit, 8
  data bits, and one stop bit).

Which is to say, 3125 bytes/sec. I would assume that the rawmidi
interface would already filter out the stop and start bits? dunno. How
about the diagram on http://www.philrees.co.uk/#midi, I found that to be
useful.

Now, there's another form of MIDI (the common usage?), "Standard MIDI
files". We'll talk about that in a bit.

I don't know very much about the OSS MIDI interface; apparently there
exists an evil /dev/sequencer interface, and maybe a better /dev/midi*
one. I only know this from overhearing it from people.

ALSA has a couple ways to access MIDI devices. One way is the sequencer
api. There's a tutorial,
http://www.suse.de/~mana/alsa090_howto.html#sect04, and some example
code, http://www.suse.de/~mana/seqdemo.c -- the paradigm is 'wait on
some event fd's until you get an event, then process the event'. Not
very GStreamer-like. This api timestamps the events, much like Standard
MIDI files.

The other way to use MIDI with alsa is by the rawmidi interface. Here's
the canonical reference:
http://www.alsa-project.org/alsa-doc/alsa-lib/rawmidi.html#rawmidi
It seems there is example code, too:
http://www.alsa-project.org/alsa-doc/alsa-lib/_2test_2rawmidi_8c-example.html#example_test_rawmidi

This is much more like GStreamer. I do wonder about the ability to
connect to other sequencer clients, though...

> I think the following 3 elements would be most useful initially:
> - midifileparse
> takes midi file format data on sink pad, and produces timestamped midi data
> on output. A property will specify what the tick rate would be (default to
> 96 ticks per beat or something). If no data exists for a given tick, it can
> just send a filler event. Timestamp would be derived from the bpm property,
> and the time deltas of the midi file data.

This would parse so-called Standard MIDI files. This, to me, is the
least interesting application of MIDI. Let's be crude. Who gives a fuck
about shitty little MIDI tunes? I don't. I want to use MIDI as a
connection mechanism between cool softsynths and drum machines and
sequencers and the like.

Anyway, Standard MIDI files are just timestamped MIDI data; they don't
run at a constant bitrate, and for that reason you need this element.

> - ossmidisink
> could be added to the existing oss plugin dir, sends midi data to oss midi
> sequencer. Makes extensive use of GstClock to only send out data when the
> buffer/event timestamp says it should.

Yes. Or the raw midi device, doesn't matter which.

> - alsamidisink
> guess what this does. don't know whether alsa's sequencer interface would be
> better than its raw midi one. My money is on raw midi.

Yeah. I discussed this above.

> - ossmidisrc, alsamidisrc
> real time midi input.

This needs to be from the raw api, though...

> It would be nice to be able to transform midi to audio which can be further
> processed in a gstreamer pipeline.

You mean use GStreamer as some kind of softsynth. Yes, this would be nice.

> Timidity might be the best hope for this,

Maybe. See, there are two issues here.

The first is sending MIDI data to softsynths and getting audio data out.
There's a very, very nice way of doing this in ALSA, and that's the
sequencer api. Timidity can already register itself as a sequencer
client, as can amSynth, AlsaModularSynth, SpiralSynth, etc... and these
latter ones are *much* more interesting. This is the proper, imho, way
of doing things.

But, the other question is getting that data back for use by GStreamer.
In that sense a librafied timidity would be useful, I guess... see the
thing is that all of these sequencer clients probably want to output to
the sound card directly, although they are configurable. In this, the
musician's only hope is Jack. If the synth is jacked up, we can get its
output back into gstreamer. If not, oh well, it's gone...

> Once we have midi streams, we can start doing fun things like writing a
> midi2dparams element which would map midi data to control the dynamic
> parameters of other elements, but lets not get ahead of ourselves.

Yes!

Anyway, it's a long road, at this point...

regards,

wingo.