[RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed

[Harry Wentland]

On 2023-11-08 11:19, Pekka Paalanen wrote:
> On Wed, 8 Nov 2023 09:31:17 -0500
> Harry Wentland <harry.wentland at amd.com> wrote:
> 
>> On 2023-11-08 06:40, Sebastian Wick wrote:
>>> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <ppaalanen at gmail.com> wrote:  
>>>>
>>>> On Tue, 7 Nov 2023 11:58:26 -0500
>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>  
>>>>> On 2023-11-07 04:55, Pekka Paalanen wrote:  
>>>>>> On Mon, 6 Nov 2023 11:19:27 -0500
>>>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>>>  
>>>>>>> On 2023-10-20 06:36, Pekka Paalanen wrote:  
>>>>>>>> On Thu, 19 Oct 2023 10:56:40 -0400
>>>>>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>>>>>  
>>>>>>>>> On 2023-10-10 12:13, Melissa Wen wrote:  
>>>>>>>>>> O 09/08, Harry Wentland wrote:  
>>>>>>>>>>> Signed-off-by: Harry Wentland <harry.wentland at amd.com>  
>>>>>>>>
>>>>>>>> ...
>>>>>>>>  
>>>>>>>>>> Also, with this new plane API in place, I understand that we will
>>>>>>>>>> already need think on how to deal with the mixing between old drm color
>>>>>>>>>> properties (color encoding and color range) and these new way of setting
>>>>>>>>>> plane color properties. IIUC, Pekka asked a related question about it
>>>>>>>>>> when talking about CRTC automatic RGB->YUV (?)
>>>>>>>>>>  
>>>>>>>>>
>>>>>>>>> We'll still need to confirm whether we'll want to deprecate these
>>>>>>>>> existing properties. If we do that we'd want a client prop. Things
>>>>>>>>> should still work without deprecating them, if drivers just pick up
>>>>>>>>> after the initial encoding and range CSC.
>>>>>>>>>
>>>>>>>>> But realistically it might be better to deprecate them and turn them
>>>>>>>>> into explicit colorops.  
>>>>>>>>
>>>>>>>> The existing properties would need to be explicitly reflected in the
>>>>>>>> new pipelines anyway, otherwise there would always be doubt at which
>>>>>>>> point of a pipeline the old properties apply, and they might even
>>>>>>>> need to change positions between pipelines.
>>>>>>>>
>>>>>>>> I think it is simply easier to just hide all old color related
>>>>>>>> properties when userspace sets the client-cap to enable pipelines. The
>>>>>>>> problem is to make sure to hide all old properties on all drivers that
>>>>>>>> support the client-cap.
>>>>>>>>
>>>>>>>> As a pipeline must be complete (describe everything that happens to
>>>>>>>> pixel values), it's going to be a flag day per driver.
>>>>>>>>
>>>>>>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
>>>>>>>> as well. Maybe it's purely informative and non-configurable, keyed by
>>>>>>>> FB pixel format, but still.
>>>>>>>>
>>>>>>>> We also need a colorop to represent sample filtering, e.g. bilinear,
>>>>>>>> like I think Sebastian may have mentioned in the past. Everything
>>>>>>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
>>>>>>>> everything after it happens on the sample that was computed as a
>>>>>>>> weighted average of the filter tap inputs (texels).
>>>>>>>>
>>>>>>>> There could be colorops other than sample filtering that operate on
>>>>>>>> more than one sample at a time, like blur or sharpness. There could
>>>>>>>> even be colorops that change the image size like adding padding that
>>>>>>>> the following colorop hardware requires, and then yet another colorop
>>>>>>>> that clips that padding away. For an example, see
>>>>>>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
>>>>>>>>
>>>>>>>> If that padding and its color can affect the pipeline results of the
>>>>>>>> pixels near the padding (e.g. some convolution is applied with them,
>>>>>>>> which may be the reason why padding is necessary to begin with), then
>>>>>>>> it would be best to model it.
>>>>>>>>  
>>>>>>>
>>>>>>> I hear you but I'm also somewhat shying away from defining this at this point.  
>>>>>>
>>>>>> Would you define them before the new UAPI is released though?
>>>>>>
>>>>>> I agree there is no need to have them in this patch series, but I think
>>>>>> we'd hit the below problems if the UAPI is released without them.
>>>>>>  
>>>>>>> There are already too many things that need to happen and I will focus on the
>>>>>>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
>>>>>>> (read-only) colorop type to define scaling and tap behavior at any point and
>>>>>>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
>>>>>>> info in it.  
>>>>>>
>>>>>> How would userspace know to look for tap/scale info, if there is no
>>>>>> upstream definition even on paper?
>>>>>>  
>>>>>
>>>>> So far OSes did not care about this. Whether that's good or bad is
>>>>> something everyone can answer for themselves.
>>>>>
>>>>> If you write a compositor and really need this you can just ignore
>>>>> color pipelines that don't have this, i.e., you'll probably want
>>>>> to wait with implementing color pipeline support until you have what
>>>>> you need from DRM/KMS.
>>>>>
>>>>> This is not to say I don't want to have support for Weston. But I'm
>>>>> wondering if we place too much importance on getting every little
>>>>> thing figured out whereas we could be making forward progress and
>>>>> address more aspects of a color pipeline in the future. There is a
>>>>> reason gamescope has made a huge difference in driving the color
>>>>> management work forward.
>>>>>  
>>>>>> And the opposite case, if someone writes userspace without tap/scale
>>>>>> colorops, and then drivers add those, and there is no pipeline without
>>>>>> them, because they always exist. Would that userspace disregard all
>>>>>> those pipelines because it does not understand tap/scale colorops,
>>>>>> leaving no usable pipelines? Would that not be kernel regressing
>>>>>> userspace?
>>>>>>  
>>>>>
>>>>> The simple solution is to leave previously advertised pipelines
>>>>> untouched and add a new version that does include scaling information.
>>>>>  
>>>>>> If the kernel keeps on exposing pipelines without the colorops, it
>>>>>> fails the basic promise of the whole design: that all pixel value
>>>>>> affecting operations are at least listed if not controllable.
>>>>>>
>>>>>> How will we avoid painting ourselves in a corner?
>>>>>>
>>>>>> Maybe we need a colorop for "here be dragons" documented as having
>>>>>> unknown and unreliable effects, until driver authors are sure that
>>>>>> everything has been modelled in the pipeline and there are no unknowns?
>>>>>> Or a flag on the pipelines, if we can have that. Then we can at least
>>>>>> tell when the pipeline does not fulfil the basic promise.
>>>>>>  
>>>>>
>>>>> The will always be dragons at some level.  
>>>>
>>>> Do I understand right that the goal of fully understood color pipelines
>>>> is a lost cause?
>>>>
>>>> That every pipeline might always have something unknown and there is no
>>>> way for userspace to know if it does? Maybe because driver developers
>>>> don't know either?
>>>>
>>>> By something unknown I refer to anything outside of basic precision
>>>> issues. Doing interpolation or mixing of inputs on the wrong side of a
>>>> known non-linear colorop, for example.  
>>>
>>> I don't think that's the case. Hardware vendors should understand the
>>> hardware and exposing everything that affects the values is the goal
>>> here. There will be a transitional period where the pipelines might
>>> not expose every detail yet but that's fine. It's better than what we
>>> have now and should become even better with time. It would maybe be
>>> helpful in the future to have a cap, or property, or whatever, to
>>> indicate that the pipelines are "complete" descriptions of what
>>> happens to the values but we can discuss it when it becomes relevant.
>>>   
>>
>> I agree, for the most part. But how do you then define "complete" if
>> you exclude precision issues?
> 
> If someone can develop some kind of precision indication in the UAPI,
> we might be able to answer that question then.
> 
>>>> An incremental UAPI development approach is fine by me, meaning that
>>>> pipelines might not be complete at first, but I believe that requires
>>>> telling userspace whether the driver developers consider the pipeline
>>>> complete (no undescribed operations that would significantly change
>>>> results from the expected results given the UAPI exposed pipeline).
>>>>
>>>> The prime example of what I would like to know is that if a FB
>>>> contains PQ-encoded image and I use a color pipeline to scale that
>>>> image up, will the interpolation happen before or after the non-linear
>>>> colorop that decodes PQ. That is a significant difference as pointed
>>>> out by Joshua.
>>>>  
>>
>> That's fair and I want to give that to you. My concern stems from
>> the sentiment that I hear that any pipeline that doesn't explicitly
>> advertise this is useless. I don't agree there. Let's not let perfect
>> be the enemy of good.
> 
> It's up to the use case. The policy of what is sufficient should reside
> in userspace.
> 
> What about matching compositor shader composition with KMS?
> 
> Can we use that as a rough precision threshold? If userspace implements
> the exact same color pipeline as the KMS UAPI describes, then that and
> the KMS composited result should be indistinguishable in side-by-side
> or alternating visual inspection on any monitor in isolation.
> 
> Did this whole effort not start from wanting to off-load things to
> display hardware but still maintain visual equivalence to software/GPU
> composition?
> 

I agree with you and I want all that as well.

All I'm saying is that every userspace won't have the same policy of
what is sufficient. Just because Weston has a very high threshold
doesn't mean we can't move forward with a workable solution for other
userspace, as long as we don't do something that prevents us from
doing the perfect solution eventually.

And yes, I do want a solution that works for Weston and hear your
comments on what that requires.

Harry

> 
> Thanks,
> pq