[gst-devel] caps nego

[David Schleef]

On Thu, Jul 15, 2004 at 04:22:07PM +0200, Thomas Vander Stichele wrote:
> I was getting frustrated with my inability to express what doesn't feel
> right about the current caps nego system.
> 
> I attached the result from gst-launch for three pipelines.
> 
> The first one is videotestsrc ! identity ! ffmpegcolorspace ! identity !
> xvimagesink.
> 
> This is a very easy pipeline, anyone can see that the only ones who
> should ever decide about w, h and fps are the src and the sink element,
> and both of them implement a fixation function that also try to fixate
> to the exact same values for them.
> 
> Yet the pipeline ends up fixating to 16x16 at 1 fps.
> 
> I *think* I have some understanding of what goes wrong; some links
> between elements that have no preference for fixating do not fixate,
> hence the core feels it should help in fixating and ends up fixating to
> the lowest int value.

The core essentially chooses a random link to start performing the
fixation.  (It's not random, but it appears that way when constructing
pipelines.)  If it happens to choose a link with either videotestsrc
or xvimagesink, which both have fixate functions, it will use those
and get a decent image size and framerate.  Otherwise, you get the
core's attempt at fixation.

> However, this is a very simple pipeline example that should be fairly
> trivial to fixate.  If our system can't negotiate it correctly, I'd say
> it's a flaw in our system.

It's a flaw.  But not a flaw that is trivially fixed, otherwise it
would have been fixed already.  The best solution right now is to
connect to the fixate signal on each pad (or at least one pad per
link), and have the application decide the best format.

The future direction that I had envisioned basically involves adding
some kind of preference setting on each pad that would be obtained
by the core and travel down pipelines similar to getcaps().

One possibility is to extend GstCaps to add a GstStructure
*preference field -- this has the advantage of being ABI and API
compatible.  Elements that have a format preference would just return
a caps with the preference field set from the getcaps() function.
(Even better, if the pad has a fixate function, the preference field
could be created automatically by the core using pad->fixate().)
This preference field would travel around with the caps, getting
copied, modified, etc., as the negotiation travels down a pipeline.
Once the core actually decides on a format, it would look at the
available preference fields, and pick a format that is reasonable
given the information.  This is probably the best bet right now,
since very few elements would need changes, mainly converter
elements like colorspace and audioconvert -- they need to copy
the preference manually because they create a caps directly from
information, not by modifying an existing caps.

The big IFs about the previous idea are:

 - What does it mean to have a preference flow through converter
   elements?  If converter elements don't modify preferences, you
   get weird things happening in 'videotestsrc ! ffcolorspace !
   videoscale ! ximagesink' -- ximagesink sees a preference from
   videotestsrc for 'video/x-raw-yuv, format=(4)I420, width=640,
   height=480'.  This is trivially incompatible with the x-raw-rgb
   caps from ximagesink, but some fields make sense.  (My gut
   feeling here is that the core should assume fields with the
   same name have the same meaning.  Obviously, this breaks down
   for a visualizer, but a visualizer shouldn't be copying
   preferences verbatim.)

 - How does this fit in with the audioconvert debacle?  ('... !
   audioconvert ! osssink' would always convert to mono, because
   it could.)  Would this system inadvertantly create "smart"
   element combinations that make dumb decisions?

 - How do preferences interact with elements that currently have
   a very strong preference for pass-through?  videoscale will
   always attempt to set the same caps on both pads, but honestly,
   this is merely a preference that happens to work in the current
   system.

 - How do applications express preferences?  Should it express a
   preference on a link, or on all links in a bin, or?  (This is
   one of the problems with fixate -- the application need to
   work too hard to make pipelines do what it wants.)

 - Should a pad be allowed to specify a preferred range or list
   rather than a single value?

 - Is it possible to create a system where the core will come up
   with the same overall negotiation independent of which link
   it chooses to start with?

Ok, and lets get more complicated:

 - Should there be strength of preferences?  videotestsrc honestly
   shouldn't care how big it's output is, it just has a good hunch
   that it should be larger than 100x100 and smaller than 1000x1000,
   and probably have an aspect ratio of roughly 1.333.  ximagesink
   is pretty sure that the output should not be larger than the
   screen, and is reasonably sure that the output should match the
   existing window size, and has a hunch that it should be larger
   than 100x100.

 - How does one express an aspect ratio as a preference?  (hard
   without fundamental changes)  This is actually probably more
   important than always getting the size right.

 - How should preferences fit in with resource limiting?  An
   application may have a global preference to 

 - Do the combination of 2 medium preferences outweigh 1 strong
   preference?
   
 - I haven't imagined a system that allows a pad to express multiple
   preferences with different priorities.  That's _really_ hard, and
   would likely double or triple the complexity of the current caps
   system.

I have lots of other ideas/concerns, but my head hurts.



dave...