[gst-devel] float caps

[Benjamin Otte]

I'm not sure we need to seperate float caps, though it probably wouldn't
hurt either, you can accept more than one mimetype ;)

Anyway, my idea for a unified float caps is like this:
mimetype: audio/x-raw-float (doh)
properties:
- width (int): 32 for float, 64 for double
- endianness (int): BIG_ENDIAN or LITTLE_ENDIAN (we're still missing a
converter for this. (off topic: mms needs those conversion functions, too
as M$ sends out floats as timestamps)
- channels (int): same as for int audio
- rate (int): same as for int audio
- buffer-frames (int): number of samples per channel in one buffer. If set
to 0, the number of samples per buffer is undefined.
All of the above is used like this in current GStreamer. One property I'd
like to have:
- layout (string): "interleaved" or "sequential", which would describe how
the samples are put into the buffer on multichannel output.
"interleaved" would be like today (example for stereo):
| channel1,sample1 | channel2,sample1 | channel1,sample2 | ... |
channel2,sampleN |
"sequential" would be like this:
| channel1,sample1 | channel1,sample2 | ... | channel1,sampleN |
channel2,sample1 | ... | channel2,sampleN |

sequential is obviously better suited for pro-audio as it is easy to split
for oneton (just make subbuffers), while interleaved is easier for
conversions from/to int.
Currently we use interleaved everywhere.

Given that format, pro-audio would set the properties like this:
- channels to 1
- omit layout (it's only interesting for > 1 channel)
- buffer-frames to [1 - MAX]
consumer audio (like vorbisdec) use this:
- channels to [1, N]
- layout to what it can produce/consume (vorbis produces "sequential" for
example, it's currently converted inside the plugin)
- buffer-frames to 0

This obviously requires converters (floatconvert, ntoone, oneton (this
one needs to work with float, too), int2float, float2int and maybe
buffer-frames-adapt).

Does that make sense? Or do you see issues there?

Benjamin


On Thu, 29 Jan 2004, Andy Wingo wrote:

> Hi Benjamin,
>
> First off, thanks for not returning my grumpiness ;) Sorry about that.
> We can now return to our regularly-scheduled wingo.
>
> I was probing my mind and wondering why I didn't want a didn't want
> channels=1 or whatever. I guess the complexity of the old caps[0] made
> it a PITA. Also, I feel in "pro-audio"[1] space we should encourage
> uniformity. I have a pro-audio document that addresses this.
> Furthermore, the whole caps system is an inefficiency wrt state
> transitions when you know you're in a realm of restricted variables.
> That might be paranoia, though.
>
> One more point is that buffer-frames is a key part of ensuring the
> efficiency and operability of a graph. It allows graphs that aren't
> linear not have to worry about bytestream-type hacks (which are prone to
> lockups in circular pipes) and just chain their buffers with efficient
> function calls. It is the thing that makes multi-pad elements bearable
> to write.
>
> Nevertheless, I could prolly be convinced to introduce more caps into
> the float domain, but we'd have to look at some other options first.
>
> [0] The new caps, from what I've coded with it, looks wonderful. I
>     feel much more comfortable predicting the behavior of with the
>     new caps. It feels more Correct. Nice job, Dave :)
> [1] I don't like this phrase, but I'll use it because of existing
>     practice. Same goes for "consumer". Webs we weave!
>
> On Thu, 2004-01-22 at 18:28, in7y118 at public.uni-hamburg.de wrote:
> > - If we split int2float and float2int each into two plugins, one doing channel
> > splitting/merging, the other doing the actual conversion, we could use
> > float2int and int2float with autoplugging. That would also solve the current
> > problems for the people that have been reporting them with int2float !
> > float2int not working.
>
> int2float ! float2int should work. Why doesn't it? Sorry if I missed
> this one. It will drop all but the first channel though, in that parse
> syntax.
>
> > And yet another thing that moves us closer to conversion hell would be to
> > invent another float format.
>
> Consider this possibility, though. It has a number of advantages:
>
> (1) It reduces the complexity of capsnego in DSP pipelines (think many
>     many elements)
> (2) It cleanly separates DSP plugins from media-processing plugins,
>     allowing simplifications or flexibilities as desired
> (3) It removes the cause of these questions that seem to come up every 8
>     months or so
>
> The "consumer" format could even be called audio/x-raw-float, which
> would be more general, and the dsp format audio/x-dsp-float or
> something. The disadvantage would be that interoperability with the rest
> of GStreamer elements wouldn't be so good. You'd have to make int2dsp or
> float2dsp and their opposites.
>
> So, as I see it, we have to make int2float and the channel splitter into
> different elements anyway, for your purposes. Oneton is already done for
> us of course.
>
> The question is, should DSP plugins advertise x-raw-float caps or have
> their own caps? I'm leaning towards the second. Thoughts?
>
> Regards...
> --
> Andy Wingo <wingo at pobox.com>
>
>
> -------------------------------------------------------
> The SF.Net email is sponsored by EclipseCon 2004
> Premiere Conference on Open Tools Development and Integration
> See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
> http://www.eclipsecon.org/osdn
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>