[gst-devel] type system... (comment please)

[Wim Taymans]

On Mon, 27 Nov 2000 06:40:54 Erik Walthinsen wrote:
> On Sun, 26 Nov 2000, Wim Taymans wrote:
> 
> > Any comment that can make the system less complicated is
> > appreciated.
> Heh, you think I'm going to make this *less* complicated???  That sure 
> is some good crack you got there ;-)
> 

Almost, Holland is nearby :-)

> > 2. Type properties
> > ------------------
> > 
> > a) identification of the type
> > 
> >   Types are identified by one or more MIME types. 
> There's a potential problem here: the MIME type is not necessarily unique
> or sufficient.  Uniqueness is violated by MPEG, since video/mpeg could
> mean MPEG 1 or 2, elementary or system/program.  audio/raw is not
> sufficient to describe the data stream, since it ignores the all the
> type-specific options:
> 
> Format (aLaw/uLaw, little/big endian)
> Depth (8 - 32)
> Channels (1 - N)
> Channel arrangement (interleaved/concatenated)
> 

Sure, that's what metadata is for. Still, the types are also used to 
describe the metadata we are talking about. Not much different from the
current system.

> Video is even worse....  This seems to be the biggest hole in the type
> system (either what we have or what you describe).  For audio at least,
> we'd want to have the source and sink be able to describe whether they
> can
> deal with various forms of audio, or at least have veto over some
> attempted operation.

Do you feel the need for a negociation phase between connected pads?

We could make this quite easy:

 pad1: generate a list of possible output formats it supports. This would
       be a list of metadata with the type of the pads.

 pad2: pick one of the formats or return NULL, in which case negociation
       fails.

Since both connected paths would be of the same mime type, the metadata
would be the same so exchange between the pads/elements is possible.

> 
> Some kind of properties system on an instance of a type might work, but
> gets really messy when you're talking about trying to match arbitrary
> type
> properties.

true, only pads of equal types can be connected. The type will merely
describe
the metadata that will be passed. The metatdata will provide more specific
information.

> 
> > b) detection of types
> > 
> >   The type of any given GstBuffer can be detected using
> > 
> >     - a typefind function: a function that will inspect the bytes
> >            in the buffer and return a gboolean indicating the
> > 	   buffer is of the given type.
> > 
> >     - a template for the bytes/bits in the data that must be
> >            satisfied in order for the GstBuffer to be of the given 
> > 	   type.
> Sound like we'd want to get GNU file(1) to export a library, except
> file(1) outputs random strings.  If the magic(4) format could be extended
> to consistently output some form of MIME type too, that would be the end
> of that.  But until we (the community) manage to merge mime.types(4) and
> magic(4), as well as settle on MIME types and extension, that's kinda
> hard.
> 

Let's be an example then :-)

> >     - other properties that act more like a hint like:
> >            the extension of the source filename.
> > 	   the URI of the source.
> At least mime.types is easy to parse ;-)
> 
> > 3. Type registration
> > --------------------
> > 
> > The system will keep a directed graph of the types and the plugins
> > that operate on them.
> > 
> > example:
> > 
> >                      video/mpeg
> >                          ! 
> > 		     mpeg1parse
> > 		      /     \
> > 	      video/mpeg1   audio/mp3 -----\
> > 	      !        !           \        !
> > 	 mpeg_play   mpeg2dec      mpg123   xing  ...
> >               \        /              \     /		        
> >               video/raw              audio/raw-----\
> > 	      !      !                   !         !
> >         videosink   SDLsink           audiosink    alsasink ...
> What we have now should work fine, it just keeps a list of the elements
> that can handle a given type for input and ouptut.  It's just a different
> form of the above graph, and I think it's easier to use.  Following the
> graph around (if you meant actually having a graph data structure) seems
> wasteful.  The lists themselves sorta form a graph anyway, since the
> elements are pointers to elements that have the type ids, from which you
> can extract the lists for that type.

creating the graph from the current data structure is possible (since it
has
the same info) but not easy. It was much easier to create an optimised 
structure for that (look at gsttype.c). the shortest path algorithm does
a lot of 'get list of types that output x', so it might be a lot less
efficient 
if we keep the list datastructure.

> 
> > 4. type equivalence
> > -------------------
> > 
> > some types can have the same meaning for example:
> > 
> >   audio/mp3 and audio/mpeg 
> > 
> > or
> > 
> >   video/raw and image/raw
> >   
> > 
> > a plugin that exposes its output type as audio/mpeg and another plugins
> > with input type audio/mp3 can be connected. The system has to know
> about
> > the equivalence of both types, even it they have a different mime type.
> I think there's a tiny bit of this set up already, in that a given type's
> mime string can simply hold any number of MIME types.  However, I think
> it
> should be more explicit and more flexible simultaneously.  This makes it
> much more complex, but the type system at least shouldn't be a
> performance
> issue.

The current system is not very clear in what the multiple mime-type string
means. Are they equivalent? from less general to more general? Are they
maybe different types. 

To make the current system even more broken, as soon as you register a
string
with an allready existing type, the string as a whole receives the same 
typeid (?). the random order in which plugins are registered makes this
even
worse.

My suggestion would be to add more semantics to the string to explicitly
describe the type.

> 
> The idea would be to have a distinct type allocated to every unique MIME
> type, and have a list for each of which ones are equivalent.  This might
> even solve the type hierarchy system, since you could have a one-sided
> relationship of equivalence between audio/mp3 and audio/mp3-frame.
> mp3-frame is equivalent to mp3, but not vice versa.  The lists form the
> appropriate graph on their own.

So how about this:

string syntax:

a) "audio/mp3 audio/mp3-frame"

   to denote relationship


b) "audio/mp3 | audio/mpeg"

   to denote equivalency 

a multitype pad would be implemented as a list of types.

  gst_pad_set_type_id would vanish and would be replaced with
  gst_pad_add_type_id.

> 
> > 5. type hierarchy
> > -----------------
> > 
> > some plugins can ouput a specific subset of an allready existing type.
> > 
> > example:
> > 
> >   mp3parse inputs audio/mp3 and packs the stream into mp3 audio frames
> >   with mime type: audio/mp3-frame
> > 
> >   mpg123 only accepts audio/mp3-frame but not audio/mp3.
> > 
> >   another mp3 decoder (libmpg123) can accept audio/mp3 (and thus also 
> >   audio/mp3-frame)
> > 
> >   it must be possible to connect both libmpg123 and mpg123 to the
> mp3parse
> >   element.
> >   it must not be possible to connect mpg123 to an element that outputs
> only
> >   audio/mp3 but not audio/mp3-frame.
> > 
> > 
> > We say that audio/mp3-frame is a more specific subset of type
> audio/mp3.
> > 
> > we can create a type hierarchy:
> > 
> >           audio/mp3
> >          /        \
> >   audio/mp3-frame  audio/mp3-layer12
> >                      /        \
> > 	   audio/mp3-layer1    audio/mp3-layer2
> I'm starting to get a little hesitant to invent too more MIME types.  If
> we can work out a properties system, so we can describe the MPEG layer 
> separately, I think I'd be happier.  This makes things significantly more
> complex, though, since you might have to maintain an equivalency graph
> down at the properities level in addition to the MIME level.

I agreed that adding a lot of mime-types might not be a good idea. We might
eliminate the need for too many mime-types by the above syntax proposition.

We could even forget about the audio/mp3-layerxxyy cruft when we stick to a
general approach where the mime-types are just indication about possible
compatible elements. Since the elements exchange compatible metadata, we
could
make the negociation phase an integral part of autoplugging. That way the
low-level bits of the types are handled by the plugins themselves instead
of
the type system.

> 
> > 6. multi-type pads
> > ------------------
> > 
> > certain plugins might accept multiple non equivalent types in one of
> their
> > input pads. Consequently a plugin might output non equivalent types in
> > its output pad.
> > 
> > It must therefore be possible to specify multiple types for a pad.
> > 
> > example:
> > 
> >   mpegdemux may be able to demux both MPEG1 and MPEG2 system streams.
> >   we show the type hierarchy of the video/mpeg as follows:
> > 
> >                           video/mpeg
> > 		         /        \
> >                video/mpeg1         video/mpeg2 ---------------\
> > 	       /    \                    /    \               !
> >       mpeg1-system*  mpeg1-video    mpeg2-ts   mpeg2-ps*    mpeg2-video
> > 
> > 
> >   the mpegdemux element might specify the type of the input pad as 
> >   one of video/mpeg1-system and video/mpeg2-ts
> Not sure whether you're suggesting there be a list or a graph of
> supported
> types.  I'd say just a list, possibly with the option of forcing
> non-equivalence for any given one.  I.e. if you need to specify
> audio/mp3-frame without implying audio/mp3 too.

or in the mp3 case, just audio/mp3 and handle the details with a
negociation
phase. 

You are right that the extensive tree structure and abundance of mime types
will make the type system a bit too bloated.

> 
> Also, I'd think this would apply to sources pads as well as sinks.  This
> does make autoplugging harder, but so does having a list of possible sink
> types.  It just squares the problem, which is tractable.
> 
> > 7. definition of types
> > ----------------------
> > 
> > A plugin will provide the following information to the type system:
> > 
> >  - a mime type string describing the hierarchy and where the types
> >    they provide are located in that hierarchy.
> > 
> >  - typefind functions for some of the types.
> > 
> >  - extensions for some of the types
> > 
> > We will propose a syntax to define the type hierarchy
> > 
> > a) equivalent types :
> > 
> >    separated with a | sign
> > 
> >    ( audio/mp3 | audio/mpeg )
> > 
> > b) type hierarchy :
> > 
> >    in braces:
> > 
> >    ( audio/mp3 ( audio/mp3-frame))
> > 
> > c) multi-type pads
> > 
> >   ( mpegdemux ( video/mpeg1-system + video/mpeg2-ps) )
> > 
> >   the pad will have type mpegdemux which is the parent for
> >   type video/mpeg1-system or video/mpeg2-ps
> > 
> >                   mpegdemux
> >           	  /       \
> >  video/mpeg1-system      video/mpeg2-ps
> I'd rather have a programmatic interface instead of a textual format. 
> I'd
> hate to have to programatically construct one of these strings within the
> plugin.  And any property system is going to wreak havoc with a textual
> setup.

If we stick to simple lists with an optional | sign, to denote equivalence,
and a list of types for a pad, it might not be too difficult after all and
it looks like we have the best of both worlds.

comments?

Wim


-- 
All the big corporations depreciate their possessions, and you can, too,
provided you use them for business purposes.  For example, if you subscribe
to the Wall Street Journal, a business-related newspaper, you can deduct
the
cost of your house, because, in the words of U.S. Supreme Court Chief
Justice Warren Burger in a landmark 1979 tax decision: "Where else are you
going to read the paper?  Outside?  What if it rains?"
		-- Dave Barry, "Sweating Out Taxes"