[gst-devel] Use of capabilities (RFC)

[Wim Taymans]

During the course of a discussion on IRC, it turned out
that there are many possible ways to handle the capabilities.

A capabilitiy is bascially a set of properties attached to a 
mimetype in order to more closely describe the mimetype. 
Capabiltities are supposed to be attached to pads so that the
autoplugging algorithm has more specific information to connect
compatible pads.

We present 3 possible implementation for the capabilities. we need
to pick one of them.

1. static capabilities
----------------------

When an element is created, it creates its pads like:

  mpg123->sinkpad = gst_pad_new ("sink", GST_PAD_SINK);
  gst_element_add_pad (GST_ELEMENT (mpg123), mpg123->sinkpad);

  mpg123->srcpad = gst_pad_new ("src", GST_PAD_SRC);
  gst_element_add_pad (GST_ELEMENT (mpg123), mpg123->srcpad);

In the static capabilities case, it will attach a GstCaps* structure
to the pad. The GstCaps structure in the above example might look like:

  static GstCapsFactory mpg123_sink_caps = {
    "audio/mp3",
    "layer",   GST_CAPS_INT_RANGE (1, 3),
    "bitrate", GST_CAPS_INT_RANGE (8, 320),
    NULL
  };

with 

  mpg123sinkcaps  = gst_caps_register (mpg123_sink_caps);

the factory can be converted into a GstCaps* structure. The
GstCaps* structure is attached to the pad with:

  gst_pad_add_caps (mpg123->sinkpad, mpg123sinkcaps);

The GstElement would then have a sinkpad with the given 
mimetype (audio/mp3) and with the capabilitities of accepting
mpeg layer 1 to 3 and a bitrate from 8 up to 320 Kbps.

Likewise, the src pad could be set up in the same way. An
example capability factory could look like:

  static GstCapsFactory mpg123_src_caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (...),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT_RANGE (4000, 96000),
    "channels", GST_CAPS_INT_RANGE (1, 2),
    NULL
  };

All GstElements would present their pads with the appropriate
capabilities structure.

The autoplugger would then proceed (once the source media type
is known with a typefind function) in finding all the elements
with compatible pads and connecting them into a pipeline. 

All elements of the complete pipeline could then be constructed
with one single pass. No new elements should be added to the
pipeline because we can figure out all the possibilities using the
pad capabilities.

We call this the static case because the capabilities of the pads
are supposed to stay the same after creating the element.

While the ability to completly setup the pipeline before actually 
starting playback is an advantage regarding performance, one obvious
problem with this setup is that the static case may be too static in
some cases. We can illustrate this with the following setup:

  ----------)             (------------
   mpg123   !             !  audiosink
           src          sink
            !             !
  ----------)             (------------

The mpg123 element has its src capabilities set up as mpg123_src_caps
in the above example.

The audio renderer has its capabilities set up with the following
factory:

  static GstCapsFactory audio_sink_caps = {
    "audio/raw",
    "format",   GST_CAPS_BITFIELD (...),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT_RANGE (8000, 44000),
    "channels", GST_CAPS_INT_RANGE (1, 2),
    NULL
  };

The static autoplugger has to be carefull when connecting the mpg123
element with the audiosink because it is theoretically possible that
the mpg123 element outputs raw audio with a rate that cannot be
handled by the audiosink (ex. 4000KHz). In the absense of another
audiosink with more capabilities, the autoplugging of this simple 
pipeline will not be possible and would fail.

the autoplugging algorithm would probably select another element to
insert between the mpg123 element and the audiosink in order to handle
the (uncommon) case of a rate conversion (audioscaler).

It is clear that this static setup might even fail or work suboptimal
for even the common case and should therefore be considered as too 
restrictive.


2. dynamic capabilities
-----------------------

The idea of dynamic capabilities is that the capabilities are not set
at element create time but rather while the pipeline is running.

An element would still list its mime type using: 

  gst_pad_add_type_id(mpg123->sinkpad, mp3type);

The idea would then be that a rough draft of the pipeline would be
built afer the media type of the stream has been detected with the
typefind functions. The rough draft would consist of laying out a
global plan to reach the renderer(s). this plan would basically list
the set of conversions that have to be performed. (mime-type to
mime-type conversion).

Elements that accept the src mime-type are tried by giving it a buffer.
If the element accepts the buffer, it will set its capabilities for
both the sink pad and the src pad. At that time other elements can be
tried and added to the src pad, until we reach the renderer. As usual
one has to be carefull to add just the minimum amount of elements to
reach the renderer. The global plan will help with that.

Since we basically do not use the capabilities of the sink pad one has 
to question the need for sink pad capabilities in the first place.

We might also have a hard time trying different elements until we find
a compatible one that does not cause a dead end at some point.


3. combined setup
-----------------

This combined setup will minimise the effort needed to try different 
elements encountered by option 2 while still allowing a more dynamic
setup based on the actual media stream we are handling.

The combined setup will list/add the sink capabilities at create time.
It will only set the mime-type of its src pads.

As with option2, a global plan will be built. At runtime the src pads
will actually specify the capabilities they need for any element that
wants to be connected to its source pads.

In this case we specifiy the capabilities for all the sink pads of an
element at create time. The capabilities of the src pads would only
become available when data has been processed by the element.

The autoplugger would then be able to choose an element that can handle
the capability listed by the src pad.

in our previous example:

  ----------)             (------------
   mpg123   !             !  audiosink
           src          sink
            !             !
  ----------)             (------------
 
the audiosink element would specify its sink pad capabilities at create
time, while the mpg123 elements src pad would not yet have any capabilities
set.

When data is handled by the mpg123 element, a capability would be added to
the mpg123 src pad. This capability might be:

  static GstCapsFactory mpg123_src_caps = {
    "audio/raw",
    "format",   GST_CAPS_INT (S16),
    "depth",    GST_CAPS_INT (16),
    "rate",     GST_CAPS_INT (44000),
    "channels", GST_CAPS_INT (2),
    NULL
  };

This capability would be compatible with the audiosinks sinkpad
capabilities
and the autoplugger would therefore be able to connect the two elements.

While allowing a more flexible setup with option3, compared to option1, we 
introduce a slightly higher overhead when we need to dynamically connect 
elements. This overhead will not be as big as option2 because we don't 
have to 'try' elements.

so:

  src caps:  added at runtime to list the caps needed for an element that
             wants to connect to this pad.
  sink caps: the (static) capabilities that this sinkpad has. 


Any thoughts?

Wim
 


-- 
What we Are is God's gift to us.
What we Become is our gift to God.