[gst-devel] MIDI in Gstreamer

[Stefan Kost]

Hi Andrew,

for me the question is, does anyone wants to write plugins that process midi 
data, such as a midi echo. If so we need midi as a stream format. The difficulty 
here is that there is not a lot of code that handles sparse streams. GStreamer 
is fine handling mostly continous data, like audio and video.

Personally I think playing midi or tracker music is more like setting up 
sound-generators and effect, wire them and connect to a sound-sink. Further set 
up GstControllers to the element properties that push in the control changes 
from the tracks at the right time.



> Hi all, I'm new to Gstreamer, so if I ask something completely stupid, 
> be gentle ;)
> 
> Seems like MIDI has been ignored for the most part with Gstreamer. I've 
> been shown a document detailing discussions so far regarding MIDI in 
> Gstreamer, and my initial ideas appear to coincide with those already 
> suggested, which is good.
> 
> I'm interested in implementing MIDI as:
> 1) I'm writing a program which will use MIDI and audio extensively
> 2) My own routing API - which I've given up to use Gstreamer instead - 
> was going to deal with MIDI
> 3) I've worked with MIDI for quite a while so know the standard fairly 
> extensively
> 
> I'm also interested in implementing a "CV" (Reason-speak... Control 
> Voltage/Value?) pad/data type, for the purpose of sending timestamped 
> controller values between elements. This is useful for example in the 
> case where you might want one element to generate values going from 0.0 
> to 1.0 and back, and use those values in another element to control 
> volume/panning, etc. This would look something like this:
> 
> Generator Output Pad -> Effect Input Pad
> 
> (Where both pads are in CV/controllervalue/controlvalue/whatever format)
> 
> This isn't essential, but it'd be useful for my app, and I imagine maybe 
> other apps too. It might be possible to sit this on top of the MIDI 
> protocol, using one of the general purpose or "effect" controllers which 
> aren't assigned to a specific function. Or even have elements generate 
> events/parameter changes occurring in other elements. I'm not too sure 
> of this though.
> 
> So we need formats. audio/midi might not be suitable, due to audio/* 
> having channel and rate properties - MIDI doesn't really have these. 
> Also we need one for the MIDI file format (I assume?) and one for the 
> MIDI stream. Any suggestions?
> 
> My main purpose for writing this message though is to get an idea what 
> people want from MIDI.
> 
> For those of you unfamiliar with MIDI, the format generally goes 
> something like this:
> Status Byte - Data 1 - Data 2
> 
> Tack a timestamp on the start of that and you have a MIDI event stream, 
> I guess.
> 
> Status Byte is always 0x80 or higher - this is to enable MIDI devices to 
> just spit out the data bytes, and the last status byte will be assumed. 
> Thus all data bytes must be under 0x80.
> 
> Status bytes between 0x80 and 0xEF have the low 4 bits used to indicate 
> which channel the message refers to (0x80 - channel 1, 0x81 - channel 2 
> ... up to 16)
> 
> Status bytes 0xF0 and above are system messages and not 
> channel-dependent. I'll not go into detail about these at the moment.
> 
> Currently, I have ideas for the following elements:
> SMF_IN - Standard MIDI File input - takes a filesrc (or other input?) 
> and provides a stream of MIDI events
> SMF_OUT - same as above, but takes MIDI events and spits out a MIDI file
> MIDI_IN - Hardware MIDI input - provides a stream of MIDI events from a 
> device
> MIDI_OUT - same as above, for output
> 
> MIDI_SPLIT - take 1 MIDI input - provide N MIDI outputs (repeats the 
> same stream N times)
> MIDI_MERGE - take N MIDI inputs - provide 1 MIDI output (to join 
> multiple MIDI streams)
> 
> MIDI_DEMUXER - take 1 MIDI input - provide 17 MIDI outputs (1 for each 
> channel, and 1 for system messages)
> MIDI_MUXER - take 17 MIDI inputs (as above) - provide 1 MIDI output
> 
> MIDI_MESSAGE_FILTER - take 1 MIDI input - provide 7 MIDI outputs - spits 
> out messagse on each pad depending on the category (note on/off, 
> controller value change, patch/program select, etc.)
> 
> MIDI_SYSEX_FILTER - take 1 MIDI input - provide 1 MIDI output - removes 
> messages between 0xF0 and 0xF7 (not really useful messages unless doing 
> advanced things like dumping hardware equipment configurations etc.)
> 
> 
> There's probably oh-so-much more that can be done with this, but those 
> are my initial ideas - mainly for the purpose of manipulating MIDI 
> messages so that they can be routed as you wish really.
> 
> Another possible element would be one that takes a MIDI input and 
> provides an output where all the messages refer to a specific channel. 
> This isn't really necessary as long as elements that deal with one 
> channel only ignore the low 4 bits of each message...
> 

Having windows sinks and source for audio and video would be nice. It just not 
yet be done. We are still waiting for takers ;)

> Also, as a side-note, has anyone considered implementing elements to 
> make use of PortAudio / PortMIDI? This might make Gstreamer a bit more 
> portable as I'd love to use it for Windows stuff as well as Linux. Just 
> a thought as it might help cut a few corners...
> 
> -Andrew
> 

Stefan