[RFC] Plane color pipeline KMS uAPI

[Pekka Paalanen]

On Tue, 09 May 2023 20:22:30 +0000
Simon Ser <contact at emersion.fr> wrote:

> On Tuesday, May 9th, 2023 at 21:53, Dave Airlie <airlied at gmail.com> wrote:
> 
> > There are also other vendor side effects to having this in userspace.
> > 
> > Will the library have a loader?
> > Will it allow proprietary plugins?
> > Will it allow proprietary reimplementations?
> > What will happen when a vendor wants distros to ship their
> > proprietary fork of said library?
> > 
> > How would NVIDIA integrate this with their proprietary stack?  
> 
> Since all color operations exposed by KMS are standard, the library
> would just be a simple one: no loader, no plugin, no proprietary pieces,
> etc.

Hi,

that's certainly the long term goal, and *if* Linux software can in any
way guide hardware design, then I believe it is an achievable goal. I
understand "standard" as something that is widely implemented in
various hardware rather than only "well-defined and documented and
free to implement in any hardware if its vendor cared".

However, like I mentioned in my other reply to Steven, I expect there
will be a time period when each hardware has custom processing blocks
no other hardware (same or different vendor) has. I might not call them
outright proprietary though, because in order have them exposed via
UAPI, the mathematical model of the processing block must be documented
with its UAPI. This means there cannot be secrets on what the hardware
does, which means there cannot be a requirement for secret sauce in
userspace either.

I wonder if we can also require new COLOROP elements to be freely
implementable by anyone anywhere in any way one wants? Or do kernel
maintainers just need to NAK proposals for elements that might not be
that free?

Anything that is driver-chosen or automatic can also be proprietary,
because today's KMS UAPI rules do not require documenting how automatic
features work, e.g. the existing YUV-to-RGB conversion. Hardware could
have whatever wild skin tone improvement algorithms hidden in there for
example. In this new proposal, there cannot be undocumented behaviour.

Dave, if we went with a descriptive UAPI model, everything behind it
could be proprietary and secret. That's not open in the least.

On Wed, 10 May 2023 at 00:31, Harry Wentland <harry.wentland at amd.com> wrote:
>
> I am debating whether we need to be serious about a userspace library
> (or maybe a user-mode driver) to provide an abstraction from the
> descriptive to the prescriptive model. HW vendors need a way to provide
> timely support for new HW generations without requiring updates to a
> large number of compositors.  

Drivers can always map old COLOROP elements to new style hardware
blocks if they can achieve the same mathematical operation up to
whatever precision was promised before. I think that should be the main
form of supporting hardware evolution. Then also add new alternative
COLOROP elements that can better utilize the hardware block.

Naturally that means that COLOROP elements must be designed to be
somewhat generic to have a reasonable life time. They cannot be
extremely tightly married to the hardware implementation that might
cease to exist in the very next hardware revision.

Let's say some vendor has a hardware block that does a series of
operations in an optimized fashion, perhaps with hardwired constants.
This is exposed as a custom COLOROP element. The next hardware revision
no longer has this block, but it has a bunch of new blocks that can
produce the exact same result. The driver for this hardware can expose
two different pipelines: one using the old COLOROP element, and another
using a bunch of other COLOROP elements which exposes the new
flexibility of the hardware design better. If userspace chooses the
former pipeline, the driver just programs the bunch of blocks to behave
accordingly. Hopefully the other COLOROP elements will be more standard
than the old element.

Over time, I hope this causes an evolution where hardware implements
only the most standard COLOROP elements, and special-case compound
elements will eventually fall out of use over the decades.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20230510/68ee69f5/attachment.sig>