Video standards

[Pekka Paalanen]

On Thu, 11 Apr 2024 22:22:01 -0300
salsaman <salsaman at gmail.com> wrote:

> Sorry for the delay, there was a lot to respond to, most of it not relevant
> to the XDG discussion. I suggest we limit discussion on the mailing list to
> whatever is relevant. Pekka, if you want to continue this discussion, then
> we can chat in private off the mailing list.
> 

Sorry from my behalf as well, I've been spread too thin lately.

> 
> On Mon, 8 Apr 2024 at 08:42, Pekka Paalanen <pekka.paalanen at haloniitty.fi>
> wrote:
> 
> > On Fri, 5 Apr 2024 19:16:55 -0300
> > salsaman <salsaman at gmail.com> wrote:
> >  
> > > On Fri, 5 Apr 2024 at 12:57, Pekka Paalanen <
> > pekka.paalanen at haloniitty.fi>
> > > wrote:
> > >  
> > > > On Fri, 5 Apr 2024 08:28:27 -0300
> > > > salsaman <salsaman at gmail.com> wrote:
> > > >  
> > > > > I don't think you are paying enough attention to the main points. Ir  
> > is  
> > > > not  
> > > > > simply a case of extending the fourcc values to include more. If I  
> > didn't  
> > > > > make it clear enough, the whole fourcc system is obscure, inadequate,
> > > > > ambiguous. The only reason ever to use it would be when you don't  
> > have  
> > > > meta  
> > > > > data and you are forced to encode the format in the first 4 bytes.  


> Now to get back to my proposal , all I am suggesting is an agreed upon set
> of constants. It shouldn't stop anyone from doing anything they would
> normally do; what I am suggesting now is that projects that want to provide
> full interoperability provide an API to convert between project constants
> and XDG constants.  My preference would be to implement the conversions as
> macros in a compatibility header to avoid having to link with libraries.
> Then it is a simple matter of, if I wanted to send metadata to Wayland, I
> translate to XDG constants using my compatibility macro, then from XDG
> constants to Wayland values using the Wayland XDG compatibility API. libav
> can implement their own compatibility header, or someone could contribute a
> patch. Pipewire can make an XDG compat header and now suddenly all these
> apps and frameworks become compatible with each other ! Due to the network
> effect, each new project which participates increases the number of
> possibilities exponentially.

...

> 
> None of that will change, all I am suggesting is adding glue to help the
> building blocks gel together better.
> All pf this is optional, nobody will be forced to include this project <->
> XDG compatibiliity but as more projects join the advantages of
> participating increase.
> 
> Just to reiterate - nothing internal to a projecr need change in any way,
> Projects participate by publishing exrenal APIs for two way conversion.
> We are talking only about constant enumerations. The meanings will be
> defined with reference to existing standards, so XDG_VIDEO_GAMMA_BT709
> could have value 2, and it is defined as representing the gamma transfer
> function specified in the bt709 standard.
> 
> Then suppose in my project I have WEED_GAMMA_BT709 with value 3, then I
> make a comoat macro for gamma XDG_GAMMA_TO_PROJECT(gamma_type, which given
> value 2, returns 3, and the corresponding PROJECT_GAMMA_TO_XDG converts 3
> to 2.
> 
> It's that simple.

Right, as long as the things are enumerated rather than numeric, and
the common enumerations are each for strictly orthogonal aspects of
color encoding. It gets complicated when one enumeration combines
things A and B, and another enumeration combines things B and C
instead.

What about all the practicalities for allowing lots of programs to
interoperate:

- Defining an API or IPC protocol between programs so that they can
  actually pass messages and images from one to another?

- How to find available sources and sinks in a system?

- How to decide where to connect, which program feeds into which program?

- How to negotiate a common set of image parameters so that both sides
  of a connection can handle them?


> > > >  
> > > > > Colorimetry is only relevant when displaying on a monitor. In the  
> > video  
> > > > > world we just have red, green and blue (plus alpha, y, u and v).  
> > These  
> > > > are  
> > > > > just labels for the colour channels, mapping them to bit formats.  
> > > >
> > > > That is a very surprising opinion. Have you worked on HDR imagery?
> > > > Or wide color gamut?
> > > >  
> > >
> > > As I have mentioned several times, these are display output parameters,
> > >  The only details which are relevant are the yuv/rgb conversion constants
> > > and the gamma transfer values, With those I can convert berween any two
> > > formats, which is all that is necessary for the steps between decoding  
> > and  
> > > encoding / display.  
> >
> > I am puzzled. Let's say we have BT.601 525-line video, and it needs to
> > be re-coded for a BT.2020 container. The yuv/rgb conversion matrices
> > are different, sure. But the primaries are different as well. If you
> > ignore the difference in primaries, does that not result in unnaturally
> > color saturated image when the video is eventually displayed?
> >
> >  
> I was not aware of the colour primary differences. The yuv -> rgb
> conversion functions ought to produce the same perceptual colours after
> conversion, and gamma adjustment, notwithstanding rounding differences due
> to range (gamut) differences.
> So are you implying there is some kind of non-linear factor involved
> resulting in what you would call primaries adjustments ?

It's not about non-linearity. Primaries are a whole another concept
than YUV-RGB matrices. Specifications like BT.709 and BT.2020 just
happen to define both properties, but fundamentally they are independent
properties of color encoding.

For example, when you have BT.709 primaries (the same are used by
sRGB), then RGB=(1.0, 0.0, 0.0) is a certain red stimulus.

When you take RGB=(1.0, 0.0, 0.0) in BT.2020 primaries, it is a
significantly more saturated red than the BT.709 one. The BT.2020 red
is literally laser-pure, while BT.709 red is much less so. Brightness
may be the same in both, saturation or purity is the important aspect
here.

I think the post linked from
https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/learn.md#color-spaces-bartosz-ciechanowski
is nice. The section "Primary Differences" is perhaps the most relevant.

In short, one can define an infinite number of different RGB color
spaces, where the same optical (a.k.a linear gamma) RGB-tuple value
would produce a different (color) stimulus depending which color space
is used to define it. It can be even a different hue.

> I have theoretical support for bt2020 in my app and standards, but I have
> not come across any material as yet that utilises it.
> 

I think BT.2020 primaries are most commonly found in HDR videos.

The trouble is, if one reinterprets RGB values from one set of
primaries in another set of primaries, the result may not look
obviously wrong. You might need to have a very good BT.2020 and an sRGB
monitor side-by-side when feeding both the same optical RGB data to see
the difference, or you would need to have an excellent color memory and
know what the image should usually look like.

If you have an average or a poor BT.2020 consumer monitor, the monitor
might do enough color mangling that it won't show much of a difference
anyway. The HDMI or DisplayPort signal also needs to explicitly put the
monitor into BT.2020 mode, and some monitors may conflate that with a
HDR mode. There are many pitfalls in trying to seethe difference.

It tends to be artists and other people professionally working with
wide color gamut displays that notice the difference, and complain that
non-color-managed content is displayed far too saturated on their
expensive monitors.

> > Ok, this our difference. Final display is always in scope for me,
> > because digital images are meant to be displayed on displays. To my
> > understanding, a formed digital image is always intended for a specific
> > display viewed in a specific environment (e.g. a room), because that is
> > where it was color graded or intended to be viewed.
> >
> > When a destination standard or actual viewing differs from the
> > original, I would need to know what I am converting from and what I am
> > converting to, in order to convert well.
> >
> Yes this is different indeed. If I am encoding video to clip, then there  
> is simply no way of knowing the display environment.
> All I know is that the user wants a specific codec, frame size, bitrate,
> letter boxing or scaling. Then after that is up to the video player
> software or hardware. The same way as if you were using an image editor to
> make a png file. Generally you wouldn't target it at a specific monitor,
> but when you view the image in a photo editor, then you might adjust it for
> whatever monitor you are viewing. Production and reproduction are different
> but related processes.

Yes, now we are talking. Are the concepts of "display-referred" and
"scene-referred" familiar to you?

I'm working exclusively with display-referred images. The other kind
images tend to still care about the primaries and white point and more,
they just have a slightly different meaning. It's also possible to
completely let go of all those, and just have "data", and let whoever
gets it to figure out what colors it should map to. I guess that's
quite hard though, and having a defined starting point for color
grading might help.

Taking this back to the interoperating video programs design, will
those programs be sending display-referred or not content to each
other? Or perhaps both?

If it's display-referred, we need the expected display parameters. If
it's not, we would probably want something so that whatever is doing
the color grading or image formation has a starting point or some
reference of what kind of stimulus there originally was, if that's
significant. If there is no human doing the color grading, then those
parameters are definitely needed.

...

I suspect that the LiVES system you have built may be relying on the
user or administrator to ensure that all content and all displays agree
by convention. That may be a perfectly valid design choice for you. Or
perhaps "unintended" extra color saturation is just a nice bonus for
your audience, making things look more vivid? You mentioned DJs. I
guess something like a laser projector would look quite dull if it was
limited to sRGB color gamut, even if it was bright.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/xdg/attachments/20240503/c1fd411d/attachment.sig>