[gst-devel] best way to internally represent an image

[Gustavo Sverzut Barbieri]

On Sun, Feb 24, 2008 at 8:33 PM, Florin Andrei <florin at andrei.myip.org> wrote:
> (apparently offtopic, but bear with me, you'll see it's not)
>
>  I'm writing a 3D raytracer to analyze relativistic phenomena. The
>  raytracer must be able to draw the 3D scene in real time, or to parse a
>  script and then generate a movie based on the scripted movements of the
>  camera. I'm in a very early stage, still planning the high level stuff.
>   From a user's perspective, it will superficially resemble Celestia
>  (display a view of the "cosmos" that the user can navigate with the
>  joystick or keyboard):
>
>  http://www.shatters.net/celestia/gallery.html
>
>  I think I may use Gstreamer to generate the movies.
>
>  But anyway, here's the question:
>  I need to decide on the internal format to represent the image being
>  rendered. It will probably be some kind of bitmap, a matrix of pixels.
>  But I have a hard time deciding between RGB, YUV, and all that stuff.
>
>  I am tempted to use a matrix of pixels, each pixel being a C structure
>  like this:
>
>  struct pixel
>  {
>         unsigned char red;
>         unsigned char green;
>         unsigned char blue;
>  };
>
>  That would be 24 bit RGB, I guess. I like it because it feels "natural",
>  and it's trivial to come up with algorithms that, e.g., give a blue tint
>  ("blue shift"), etc.
>
>  But I need to decide on a format that will make it easiest to accomplish
>  the 2 main goals of the program's output:
>
>  1. Project an interactive 3D scene on the screen in real time using SDL
>  or Xv, or whatever fast and portable method I can find
>
>  2. Interface with a program or a library that can dump a movie on the
>  disc in some nice universal format (MPEG2, MPEG4, etc, 480p, 1080p, or
>  user-defined such as 1024x768, etc.) regardless of whether I'll actually
>  use Gstreamer for that purpose or not (but the likeliest candidate at
>  this moment is Gstreamer, it's really nice and powerful)
>
>  Speed is crucial, as the project has some pretty ambitious goals (HD
>  resolutions up to 1080p, real time display for as high resolutions as
>  possible - ideally for 1080p on very good hardware, or whatever can be
>  achieved with the best hardware today). So choosing the internal image
>  representation has to take that in consideration.
>
>  I'm not an expert in video programming. I'm not even a professional
>  programmer. I'm just a hobbyist. So I would appreciate any advice from
>  experienced people.

Ok, so as not many people replied, I'll give my shot.

24bits is often not a good format if you need to access data much,
why? because we use 32bits (or multiple of) machines, using it will
mean more optimized access which is often faster. But going 32bpp
(bits per pixel) will waste more memory, so my advice is to do some
tests on your desired hardware and see for your use case which one
fits better.

Also, if going 32bpp maybe it's better to handle it as an int32_t
instead of a struct, but most compilers should optimize it anyway.

I'd also do YUV tests. Often used variations have half planes for
color channels, so you have less data and thus it tend to be faster.
Most video cards have accelerated interface to get this colorspace, so
you can get scale and other operations done by hardware.
    However YUV is not a good format if you want to mix it back into
your RGB color space. Say you want to do alpha blending of your video
on some RGB data (window or like), you'd have to convert YUV->RGB (or
the other way around) and it will be slow... also you loose Xv.
However I have to admit it's uncommon, but worth thinking about it.

As Florent already said, most systems do accept image buffers in most
colorspaces and do the conversion for you (often the video format is
YUV variant). If you choose a weird video encoding stack that does not
support it by default, you can use ffmpeg code, as in gstreamer's
ffmpegcolorspace.

-- 
Gustavo Sverzut Barbieri
http://profusion.mobi - Embedded and Mobile Software Development
--------------------------------------
MSN: barbieri at gmail.com
Skype: gsbarbieri
Mobile: +55 (81) 9927 0010