[gst-devel] RGB mimetyping in gstreamer & lcs

[Ronald Bultje]

Hey all,

as announced earlier, I'm currently adding bits 'n pieces to
libcolorspace (from codecs.org) so we can actively use that instead of
the hermes-based colorspace-plugin we're using now. This one will have
more supported functions and will do *any* colorspace conversion without
in-between steps. So using colorspace twice because "rgb2yuy2 isn't
implemented" won't happen. Fortunately. ;).

OK, so now about how I'm going to do that: I have divided all
Gst-recognized raw video formats (RGB & YUV) in three categories: planar
Y'Cb'Cr, packed Y'Cb'Cr and RGB. Each has a set of properties (integers)
that uniquely identifies it, plus the LCS identification tag.
For planar Y'Cb'Cr (YUV from now on), this is the plane order (YUV or
YVU?) and horizontal/vertical downsampling of the U/V planes.
For packed YUV, this is the horizontal downsampling of the U/V
components plus a GOP code (so a string) that identifies one single tag.
We need this because for some formats, the tag is non-standard (e.g.
Y41P = UYVYUYVYYYYY).
For RGB, this is going to shock you, I've figured a new system to
type-define these. I hate our current approach in GStreamer, so that's
what I want to discuss further on.

What we currently use in GStreamer is R/G/B component masks, plus the
width/depth. I've kept the bpp/depth items (and renamed bpp to 'bytes'),
and am using the endianness order of the format, plus the component's
offset from LSB and width in bits ('bits per component'). So RGB24BE
looks like this:

bytes=3
depth=24
byteorder=BIG_ENDIAN
red_offset=16
green_offset=8
blue_offset=0
red_bpc=8
green_bpc=8
blue_bpc=8

The nice thing here is that masks can be generated on the fly
(((1<<color_bpc)-1)<<color_offset). We're still having the endianness
problem here. I chose for a non-bytewise approach because I want to
support 10bpc formats too. 8bpc is just 'accidently' suitable for
bytewise approach. If we're going to do 16bpc support later on
(endianness issues inside a component!), the current approach will get
even worse. With this approach, it will just work.

The cool thing of all this is that I can write a generic colorspace
RGB->RGB function for *any* defined RGB type to *any* defined RGB type,
because it just always works - it's perfect. ;). It's not optimized, of
course, so you'd want to write specific optimized functions for 'common'
formats like RGB24-to-32 and so on (Dave proposed that, I fully agree
here), but this allows us to just define the format and we'll
automatically support it right away. I have some code that does this
already, and will commit it to LCS CVS when I've got my changes
finished.

So, how do we want to move this over to GStreamer? I don't think we want
to. We use tags as identifiers in LCS, and I think that we just want to
support these tags (aliases, for that matter), since they're "used"
formats. Our current GStreamer approach allows us to support any format,
but I disagree with that. We also don't try to support *any* YUV format
out there, but just the ones that we've defined. Of course, it's
extendible, not limiting, but we're not going into a road that we cannot
keep up with. xvideosink, for example, is actually heavily broken. It
doesn't check rgb_masks at all! And most plugins don't! And we never
intended them to do so, imo.

Ok, so what I want to use in GStreamer is to go back to the
bpp/depth/endianness defines, and no more. So RGB24BE in GStreamer would
then look like this:

video/x-raw-rgb
endianness=G_BIG_ENDIAN
bpp=24
depth=24

And now that we know the depth+endianness, we automatically know that
the corresponding LCS-tag is "RGB24". LCS will fill in the rest if we
need to do conversion, and we don't need that information for any other
approach. Likewise, little-endian RGB24 would be
endianness=G_LITTLE_ENDIAN. This obsoletes the hard-to-use masks, that
everyone got wrong anyway, while still maintaining the cross-endianness
flexibility that we want and need. LCS on LE computers will understand
BE formats just fine, I've programmed all that already. GStreamer
plugins will not have to bother with this low-level stuff, they'll just
do what they want: simple stuff.

What do other people think of this new mimetype approach for GStreamer?
We're giving up some flexibility here, but we never ever used and I
don't think we ever will in large scale. For small-scale and simple
stuff, we can just define new RGB types and add them to the list of
supported formats in LCS and in docs/random/mimetypes, and we'll have a
new format either way. For GStreamer, it will be much simpler. The
disadvantage is that we're giving up our byte-wise approach of RGB24/32,
but I never liked that anyway, since it's specific to 8bpc RGB and won't
work for 5/6/10/16bpc RGB (note: 16bpc in our current approach has
endianness issues *per component*! That's fixed here, too).

So, opinions? Once I've changed LCS to do all this, can I change
GStreamer as described above too?

Thanks for reading,

Ronald

-- 
Ronald Bultje <rbultje at ronald.bitfreak.net>