[gst-devel] first caps negotiation draft (RFC)

[Wim Taymans]

This is a little outline of my current investigations for caps
negotiation. this turns out
to be, ..um.., not so trivial. This doc still has many holes and
inconsistencies in it but
I'm sure some of you can help me out...

caps negotiation
================

1) purpose
----------

The pads expose the media types they can handle using a mime
type and a set of properties. Before the pad is created or
used to pass buffers, we only know the global 'range' of media
data this pad can accept. When the element has had a chance to
look at the media data, only then it knows the exact values of the
properties.

example1:
!
! The mp3 decoder exposes the capabilities of its src pad
! with the following caps:
!
!     'mpg123_src':
!       MIME type: 'audio/raw':
!       format: Integer: 16
!       depth: Integer: 16
!       rate: Integer range: 11025 - 48000
!       channels: Integer range: 1 - 2

as you can see in example1, the padtemplate has both a range
(for the audio rate) and a list (for the number of channels)
for its properties.

only when the mpg123 element has decoded the first mpeg audio
header, it knows the exact values of the rate and channels
properties.

suppose that we want to connect this src pad to the sink pad
of an audiosink with the properties given in example2:

example2:
!
!     'audiosink_sink':
!       MIME type: 'audio/raw':
!       format: Integer: 16
!       depth: List:
!         Integer: 8
!         Integer: 16
!       rate: Integer range: 8000 - 44000
!       channels: Integer range: 1 - 2

we can see that connecting the mpg123 src pad with the
audiosinks sink pad can cause a potential problem with the
rate property.

When the mpg123 decoder decides to output raw audio with a
48000Hz samplerate, the audiosink will not be able to handle
it. The conservative approach would be to disallow the connection
between the two incompatible pads. This rules out any potential
problems but severely limits the amount of possible connections
between the elements.

Another approach would be to allow the connection (and mark it
as dangerous) and let the two elements figure out a suitable
media type at runtime. This procedure is called caps negotiation.


2) a bit of history
-------------------

The typing of the data that was attached to a buffer used to be
done using GstMeta* (and it still is as of 11 feb 2001). With
the new GstCaps and GstProps system this typing is gradually moved
to the pads and to the padtemplates. This has several advantages:

  - the typing of the data tends to be static. The type of media
    doesn't change for every buffer.

  - Moving the typing up to the pad(templates) allows us to save
    them into the registry and allows us to figure out what pads
    are compatible.


  - the current metadata implementation needs header files. this may
    change when we also use properties for metadata.

example3:
!
! This is the current GstMeta structure that travels with audio buffers
!
!  struct _MetaAudioRaw {
!    GstMeta meta;
!
!    /* formatting information */
!    gint format;
!    gint channels;
!    gint frequency;
!    gint bps;
!  };


The question still remains what purpose the metadata will serve
now that we expose the media type in the pads. Some possibilities:

  - interesting information, not describing the data itself but the
    context in which the data was generated (suggested buffer size,
    timestamps, etc...)

  - user app metadata.

In this proposal we also assume that the current use of metadata using
GstMeta is deprecated and that we move this information to the
properties
of the pads.


3) the pad/padtemplates caps
----------------------------

All elements have to provide a padtemplate for their pads.

The padtemplates provide a range of possible media types this pad can
src/sink. the main purpose for the padtemplates is to allow a
rough guess at which pads are compatible before even a single buffer
has been processed by the element.

pads are usually created from the templates. When the pad is created
it has no GstCaps* attached to it yet. The possible caps this pad
can have is exposed in the padtemplate. The caps are filled in by
the element when it knows the values for the caps.


4) the connect function
-----------------------

when two pads are connected the following steps will take
placei (not sure, FIXME):

 - if both pads have caps, the caps are checked. If the caps
   are incompatible, the padtemplates are checked, if they
   are compatible, caps negotiation is performed.

 - if one of the pads has caps, the caps is checked against
   the padtemplate of the peer pad. If they are incompatible,
   the padtemplates are compared, if they are incompatible,
   caps negotiation is performed.

 - if none of the pads have caps, the padtemplates are checked,
   if they are incompatible, a warning is issued.


5) when the element knows the media type it is handling
-------------------------------------------------------

When the element has received its first buffer it will know
the media type it is handling by inspecting the buffer.

before pushing the data out to its peer element(s), the element
will set its src pad with the appropriate caps and properties.
These caps must follow the following rules:


  - the caps must be compatible with the padtemplates of this
    pad.

  - the caps cannot contain ranges or lists.

when the element wants to change the caps of a pad, it has to
perform gst_pad_renegotiate (GstPad *pad). this will trigger
the caps negotiation procedure.

this will trigger the class method of the pad and calls the pads
gst_pad_negotiate function:

  GstCaps *gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint count);

This function takes a GstCaps *structure as an argument (typically the
current caps of the pad) and a negotiation counter. this counter can be
used to keep track of the negotiation process.

The pad then creates a new caps structure with the desired caps.
If the caps are accepted, it just returns the provided input caps. the
_renegotiate function will set the caps of both pads whenever the
input caps are the same (pointer wise) as the input caps.

the caps structure is checked against the padtemplate of the peer pad,
if it is incompatible the gst_pad_negotiate function is called again
and the element is supposed to create another caps structure.

the gst_pad_renegotiate function then calls the gst_pad_negotiate
function of the peer pad with the new caps as the argument. The peer
pad can adjust or create a new caps if it doesn't accept it.

the caps structure keeps on bouncing between the two pads until one
of the pads negotiation functions returns the caps unmodified.

The element can also return a NULL pointer if it has run out of
options for the caps structure. When this happens, both pads are set
the the NULL caps again and the pad connnection is broken.

The negotiation process is stopped after a fixed number of tries,
when the counter has reached some limit. This limit is typically
checked by the pads negotiate function.


6) caps negotiation function
----------------------------

the negotiate function of a pad is called whenever the pad or
peer pad has performed _renegotiate.

example5:
!
! this is the caps negotiation function implemented by an element on
! one of its sink pads.
!
!  static GstCaps*
!  gst_pad_negotiate (GstPad *pad, GstCaps *caps, guint counter)
!  {
!    /* we don't accept anything else than audio/raw */
!    if (strcmp (gst_caps_get_mime (caps), "audio/raw"))
!      return NULL;
!
!    if (gst_caps_get_int_prop (caps, "format") != AFMT_S16_LE)
!      return NULL;
!
!    /* we accept everything else */
!    return caps;
!  }

When the negotiate function returns NULL (it does not accept the
specified caps of the peer pad), the negotiation process is stopped.



APPENDIX A: use cases
=====================

1) mpg123 src!sink audiosink
----------------------------

When the pads are connected the padtemplates are checked and it
turns out that the pads might be incompatible (mpg123 can do
48000Hz while audiosink can do 44000Hz). Nothing happens at
connect time except for the user app that can mark this connection
as possibly dangerous and keep some spare elements ready for when
the pads turn out to be incompatible.

both elements start out with no caps at all (NULL). mpg123 wants
to output a buffer with specific properties. It calls
gst_pad_renegotiate (mpg123->srcpad).

The _renegotiate functions calls the negotiate function of the
mpg123->srcpad. the negotiate function would look like this:


/*
 * The mpg123 element cannot convert the decoded type into something
 * else so it has to force the caps of the src pad into the specific
 * type as defined by the mp3.
 */
static GstCaps*
gst_mpeg123_src_negotiate (GstPad *pad, GstCaps *caps, guint counter)
{
  GstMpg123 *mpg123;

  mpg123 = GST_MPG123 (gst_pad_get_parent (pad));

  /* we got caps in, check them */
  if (caps != NULL) {
    if (!strcmp (gst_caps_get_mime (caps), "audio/raw") &&
        (gst_caps_get_int_prop (caps, "format") == AFMT_S16_LE) &&
        (gst_caps_get_int_prop (caps, "depth") == 16) &&
        (gst_caps_get_int_prop (caps, "rate") == mpg123->rate) &&
        (gst_caps_get_int_prop (caps, "channels") == mpg123->channels))
{
      return caps;
    }
  }
  /* we didn't get caps, so we decide */
  else if (counter != 2) {
    GstCaps *new;

    /* fill in our desired caps */
    new = gst_caps_new_with_props (
            "src_caps",                       /* name */
            "audio/raw",                      /* mime */
            gst_props_new (
              "format",   GST_PROPS_INT (AFMT_S16_LE),
              "depth",    GST_PROPS_INT (16),
              "rate",     GST_PROPS_INT (mpg123->rate),
              "channels", GST_PROPS_INT (mpg123->channels),
              NULL
            )
          );
    return caps;
  }
  /* too many attempts at nogotiation, bail out */
  return NULL;
}


The audiosink pad negotiate function would look like this:

/*
 * The audiosink has a wide range of possible parameters for
 * its sink pad, based on the audio card capabilities and
 * possibly the element configuration.
 * we assume the audiosink element can be both the initiator of
 * the negotiations and the negotiated one.
 */
static GstCaps*
gst_audiosink_sink_negotiate (GstPad *pad, GstCaps *caps, guint counter)
{
  GstAudiosink *audiosink;
  gboolean accepted = TRUE;

  audiosink = GST_AUDIOSINK (gst_pad_get_parent (pad));

  /* we got caps in, we know they will match the padtemplate */
  if (caps != NULL) {
    return caps;
  }
  /* we didn't get caps, so we decide */
  else if (counter != 2) {
    GstCaps *new;

    /* fill in our desired caps */
    new = gst_caps_new_with_props (
            "sink_caps",                      /* name */
            "audio/raw",                      /* mime */
            gst_props_new (
              "format",   GST_PROPS_INT (audiosink->format),
              "depth",    GST_PROPS_INT (audiosink->depth),
              "rate",     GST_PROPS_INT (audiosink->rate),
              "channels", GST_PROPS_INT (audiosink->channels),
              NULL
            )
          );
    return caps;
  }
  /* too many attempts at nogotiation, bail out */
  return NULL;
}