Gnome-2.6, GStreamer-0.8 and caps [was: Re: [gst-devel] more caps]

[Ronald Bultje]

Hi,

On Thu, 2003-12-25 at 11:20, Ronald Bultje wrote:
> I didn't get anywhere yet with gst_pad_get_allowed_caps() or
> GST_PAD_CAPS(), but I'll look at that too.

next step is ffmpegdec (or, in general: decoder/encoder) negotiation.
What currently happens:

sinklink is called. Since this doesn't necessarily contain size (think
of MPEG video, audio), we delay src negotiation to the first chain
function run. There, we decode data (size is now known), check if a caps
was set (if (!GST_PAD_CAPS (pad)) { ... }) and do capsnego if not.

There's a lot of issues here, and we cannot fix them unless we rewrite
the plugins or if we change capsnego (sorry Dave). First, GST_PAD_CAPS()
is broken and always returns NULL. There is no way to get the currently
negotiated caps. It wouldn't help us either. Here's why: let's assume
that we want to have two-sided capsnego (this is the end goal, see
Banjamin's email), then we could in theory have pipelines such as:

filesrc location=file.avi ! avidemux .video_00 ! ffdec_mpeg4 !
videoscale ! videosink
or
v4l2src ! ffdec_mpeg4 ! videoscale ! videosink

Now imagine two-sided capsnego. Our video window is resized, so we want
to rescale the video. videosink sets caps on videoscale. videoscale
first tries to set caps on its sink pad (ffdec_mpeg4), and if that fails
(avidemux case), it resizes itself, else it uses the new input (v4l2src
case with a hardware MPEG-4 card), right?

The issue is simple: this needs a rewrite of capsnego in all plugins.
They all need serialized bidirectional capsnego (so on srclink, I need
to do new try_set_caps or renegotiate on my sink pad, use that result,
and re-setup everything. On sinklink, I delay src-renego until the next
cycle so that I know data size). I also need to be prepared for failure
(filesrc case) and then I need to go back to where I was without losing
state. This sounds easy in theory, but it's hard to code. It's really
not easy. We'll have to think this out carefully before we adapt all
plugins, because if we do it half-right in one plugin, it'll take us
double the effort to fix it again in a possible second round. Yes,
indeed: this is non-trivial, especially if we want to do this correctly
for all plugins. We'll never get this right for 0.8.0, it's impossible.
We don't have the time or manpower. And don't forget that we're lagging
badly in documentation, iternationalization, all things that we promised
to work on in january before Gnome-2.6.

Now, for us to continue doing what we were able to do (which means using
the new caps system, but not implement it completely yet), we have an
issue: linkage is already taking place before we try_set_caps(), and the
caps are fixed (due to the fixate() stuff). This is plain wrong, we can
never check whether the current caps on the link is compatible to our
data. I see why fixate is useful, but it breaks too much in how we
currently use it. It's a cool system, but it's not right, at least not
now. We're not ready for it. We can work around that and continue what
we're doing, but: read my words. We're *working around it*. It means
that in practice, we've got a nice system, and we'll work around all of
its features so that it works for us in practice. Do you see how
illogical this would be?

As you see, I'm starting to see bad issues that are being caused by
incompatibility of our plugins with the oh-so-nice, but theoretical,
system of our new capsnego. Things that we cannot fix easily. Things
that will take time.

I've looked at it, I can see how it works, I'm in for Benjamin's new
comments (see all the previous emails), but I also think it'll be a
terrible amount of work to fix up everything, and we won't make it for
the Gnome feature freeze, probably not even for Gnome-2.6. I don't know
what to do next. I just know it'll be tough to complete it, and that I
don't want to spend my time on fixing something while the old system was
good enough for 0.8.0 (or: so it seemed). I don't want to get into the
same issues that we had with Gnome-2.2, where our system was really
cool, but only a few (mostly audio) plugins actually worked. People
thought GStreamer ruled, but couldn't actually see it, because nothing
was 100% ready.

OK, now on to what I really want to say:

I want to propose to branch off GSTREAMER_0_8 to right before the caps
branch was merged. We'll develop caps in HEAD/CVS, we'll develop a
stable Gnome-2.6 version of GStreamer-0.8 and use GStreamer-0.10 (with
fully implemented new caps system) for Gnome-2.8 or 3.0 (whichever comes
first), which gives us another half year to fix everything I've
enumerated above.

I suck. I'm sorry for that. I know I'm negative, but I'm also realistic.
I apologize for all flame-like comments in the past few days, I should
be happy because it's christmas. Please give this some consideration.

Of course, comments are welcome.

Merry xmas all,

Ronald

-- 
Ronald Bultje <rbultje at ronald.bitfreak.net>
Linux Video/Multimedia developer