Video standards

Fri Apr 5 22:16:55 UTC 2024

On Fri, 5 Apr 2024 at 12:57, Pekka Paalanen <pekka.paalanen at haloniitty.fi>
wrote:

> On Fri, 5 Apr 2024 08:28:27 -0300
> salsaman <salsaman at gmail.com> wrote:
>
> > I don't think you are paying enough attention to the main points. Ir is
> not
> > simply a case of extending the fourcc values to include more. If I didn't
> > make it clear enough, the whole fourcc system is obscure, inadequate,
> > ambiguous. The only reason ever to use it would be when you don't have
> meta
> > data and you are forced to encode the format in the first 4 bytes.
>
> Right. You must be talking about some other fourcc system. There are
> many of them, and some combine multiple orthogonal things into a single
> enumeration, which then becomes very difficult to extend and work with.
>
> drm_fourcc.h is not one of those.
>

I am talking about any system which tries to enumerate palettes (pixel
formats) in four bytes in a non sequential way.
In my own system (Weed) for example, all RGB palettes are in the range 1 -
511, yuv palettes are 512 - 1023, alpha are 1024 +

In fact this header is enough to define every possible palette, there are
standard enumerations for the most commonly used palettes, and
advanced palettes allows for the composition of new ones. In there also I
have symbolic names for gamma types and yuv details,

interlacing and flags for pre-posr alpha are kept in another header,

>
> Metadata is always necessary anyway, either implied or explicit.
>

Exactly, so I don't know why you keep mentioning fourcc as if it were some
kind of complete solution.

>
> > Colorimetry is only relevant when displaying on a monitor. In the video
> > world we just have red, green and blue (plus alpha, y, u and v). These
> are
> > just labels for the colour channels, mapping them to bit formats.
>
> That is a very surprising opinion. Have you worked on HDR imagery?
> Or wide color gamut?
>

As I have mentioned several times, these are display output parameters,
 The only details which are relevant are the yuv/rgb conversion constants
and the gamma transfer values, With those I can convert berween any two
formats, which is all that is necessary for the steps between decoding and
encoding / display.

>
> > The values I mentioned are all necessary if you want to convert from one
> > colourspace to another. For example if I decode a video frame and the pix
> > format is YUV420P then to convert it to RGBA to display via openGL, I
> need
> > to know the YUV subspace (bt709 or itu601) and whether the values are
> > clamped or full range. Then I apply the standard conversion factors (Kr =
> > 0.2126, Kb = 0.0722 for bt709). This cannot be derived from the fourcc
> > (generally). No doubt there is a standard definition of definition of the
> > R,G,B primaries, but that isnr a concern.  I just feed the values into an
> > openGL texture buffer, and SDL buffer, a gdkpixbuf, QImage or whatever
> and
> > ask for it to be displayed. Now in an application I may optionally offer
> > the user filters to adjust the white balance, contrast, display gamma
> etc.
> > but that is outside of the scope of what I am proposing.
>
> Yes, those are all important properties, and not enough.
>
> Let's just say that the final display output is out of scope, what else is
missing ?
Pre / post alpha is required for conversion between formats, I hadn't
mentioned that because I was trying to avoid going into every little detail.

> > And no, it is not a case of "adding another standard" and confusing
> things,
> > there is no standard.
>
> There are standards. ITU-T H.273, coding-independent code points, for
> example. That combines well with drm_fourcc.h. Also ICC combines well
> with drm_fourcc.h. This works, because drm_fourcc.h does not attempt to
> define anything outside of the memory layout and abstract channels.
>
> Sorry what I meant is there are standards on paper, but there is no
standard set of enumerations (implementation vs specification).
Instead we have multiple implementations, each with their own definitions.
In fact somewhere above I actually linked to the ITU709 standard.

> > I just had a look at pipewire, there is nothing bad about it per se, they
> > mention their palette values are based on gstreamer. So fine, we have yet
> > another library specific set of definitions.
> >
> > It's like I am trying to invent Esperanto, and all you can say is...."oh
> > you don't like English, well have you considered speaking German instead
> ?"
>
> That does seem like an apt analogue.
>
> >
> > Well that is it, I am done. I was asked how XDG video could be useful. I
> > explained the shortcomings of what exists currently, and outlined various
> > ways in which having a standard could be useful.
>
> Sorry, but I haven't understood what gap there is that would need to be
> filled with yet another pixel format enumeration. Or is it perhaps the
> same gap that we are filling in Wayland?
>
> Yes, yet another standard (implementarion), but a common standard that
everyone agrees on, That is the entire basis for the existence of XDG, is
it not ?
I have no idea what you doing in Wayland but from what you have said the
focus is naturally on display devices, This is a logical place to include
colour temperature, white balance and so on.
All fine and good, but these are monitor standards, video processing itself
(converting between formats and applying effects) is independant of the
display device. If I want to make an RGB delay effect I will split the
image into R, G, and B components, add a delay and then recombine. I don't
care about the colour primaries because that is irrelevant. What I do care
about is, if my input image is in yuv format, what choice of values should
I use to convert it to RGB.
If I am sending to openGL for display, I only care that the image I send is
in RGBA format, post multiplied alpha, non interlaced, and sRGB gamma
because that is what the openGL standard calls for,
 For Wayland, libGL or whatever will get the monitor details and adjust the
primaries etc. The same as if I connect a projector to the device, the
display output can adjust the colours, but in my RGB delay effect it makes
no difference.

> We need to communicate a lot more about images than what pixel formats
> ever could. We are building that definition based on existing standards
> defining their own aspects: drm_fourcc.h, H.273 / ICC, and then adding
> what's missing like the how or if alpha channel is baked into RGB (the
> two ways to pre-multiply). Since these are all well-defined and
> orthogonal, there is no problem combining them.
>
> I totally agree ! I don't see what the argument is about. I just don't
think that fourcc, alone or even supplemented with metadata is
a good idea, I prefer just to use plain integer enumerations

Sorry do you mean literally there are two ways to pre-multiply, or are you
refering to pre multiplied vs. post multiplied ? The only way I know is to
multiply each colour (RGB) value by the alpha value. With pre multiplied
(such as what Cairo uses), the values have already been multiplied, with
post alpha they have not. Most applications prefer post as it can avoid
rounding errors, but pre can be faster when you have multiple layers,

ah here is my header

https://github.com/salsaman/LiVES/blob/LiVES-4.0/libweed/weed-palettes.h

the values are all described in the Weed standard (which I will split up
into libweed and weed fx, as they have grown into independent entities).
https://github.com/salsaman/LiVES/blob/d9cbe56c04569980a9c86b6df422a6bba1f451a8/libweed/docs/weedspec.txt#L1392

you will see nowhere a mention of the output display device, except for as
I mentioned, aspect ratio and PAR which are useful for fixed sizing.

> Wayland also already provides ways to handle some things, like pixel
> aspect ratio, so we don't want to define another conflicting way to
> define the same thing. That means the solution for Wayland is probably
> not applicable somewhere else, and vice versa.
>

Well I thought this was the XDG list, not the Wayland list, The most
community friendly way would be to develop these things as application
neutral xdg standards and then have Wayland be compatible with that.

I will point out again, Wayland standards are geared towards display
hardware, and whilst there is some overlap between that and video
processing, the two things are not precisely the same. The former being
device dependant, whilst the latter is a device independant abstraction
dealing purely with manipulating bits and bytes.

>
>
> Thanks,
> pq
>

Thankyou too, this is an interesting subject and something which I am
involved in on a day to day basis,

G,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/xdg/attachments/20240405/0291bf0a/attachment-0001.htm>