[RFC PATCH 01/10] drm/doc/rfc: Describe why prescriptive color pipeline is needed

[Harry Wentland]

On 2023-11-08 06:40, Sebastian Wick wrote:
> On Wed, Nov 8, 2023 at 11:16 AM Pekka Paalanen <ppaalanen at gmail.com> wrote:
>>
>> On Tue, 7 Nov 2023 11:58:26 -0500
>> Harry Wentland <harry.wentland at amd.com> wrote:
>>
>>> On 2023-11-07 04:55, Pekka Paalanen wrote:
>>>> On Mon, 6 Nov 2023 11:19:27 -0500
>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>
>>>>> On 2023-10-20 06:36, Pekka Paalanen wrote:
>>>>>> On Thu, 19 Oct 2023 10:56:40 -0400
>>>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>>>
>>>>>>> On 2023-10-10 12:13, Melissa Wen wrote:
>>>>>>>> O 09/08, Harry Wentland wrote:
>>>>>>>>> Signed-off-by: Harry Wentland <harry.wentland at amd.com>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>>> Also, with this new plane API in place, I understand that we will
>>>>>>>> already need think on how to deal with the mixing between old drm color
>>>>>>>> properties (color encoding and color range) and these new way of setting
>>>>>>>> plane color properties. IIUC, Pekka asked a related question about it
>>>>>>>> when talking about CRTC automatic RGB->YUV (?)
>>>>>>>>
>>>>>>>
>>>>>>> We'll still need to confirm whether we'll want to deprecate these
>>>>>>> existing properties. If we do that we'd want a client prop. Things
>>>>>>> should still work without deprecating them, if drivers just pick up
>>>>>>> after the initial encoding and range CSC.
>>>>>>>
>>>>>>> But realistically it might be better to deprecate them and turn them
>>>>>>> into explicit colorops.
>>>>>>
>>>>>> The existing properties would need to be explicitly reflected in the
>>>>>> new pipelines anyway, otherwise there would always be doubt at which
>>>>>> point of a pipeline the old properties apply, and they might even
>>>>>> need to change positions between pipelines.
>>>>>>
>>>>>> I think it is simply easier to just hide all old color related
>>>>>> properties when userspace sets the client-cap to enable pipelines. The
>>>>>> problem is to make sure to hide all old properties on all drivers that
>>>>>> support the client-cap.
>>>>>>
>>>>>> As a pipeline must be complete (describe everything that happens to
>>>>>> pixel values), it's going to be a flag day per driver.
>>>>>>
>>>>>> Btw. the plane FB YUV->RGB conversion needs a colorop in every pipeline
>>>>>> as well. Maybe it's purely informative and non-configurable, keyed by
>>>>>> FB pixel format, but still.
>>>>>>
>>>>>> We also need a colorop to represent sample filtering, e.g. bilinear,
>>>>>> like I think Sebastian may have mentioned in the past. Everything
>>>>>> before the sample filter happens "per tap" as Joshua Ashton put it, and
>>>>>> everything after it happens on the sample that was computed as a
>>>>>> weighted average of the filter tap inputs (texels).
>>>>>>
>>>>>> There could be colorops other than sample filtering that operate on
>>>>>> more than one sample at a time, like blur or sharpness. There could
>>>>>> even be colorops that change the image size like adding padding that
>>>>>> the following colorop hardware requires, and then yet another colorop
>>>>>> that clips that padding away. For an example, see
>>>>>> https://lists.freedesktop.org/archives/dri-devel/2023-October/427015.html
>>>>>>
>>>>>> If that padding and its color can affect the pipeline results of the
>>>>>> pixels near the padding (e.g. some convolution is applied with them,
>>>>>> which may be the reason why padding is necessary to begin with), then
>>>>>> it would be best to model it.
>>>>>>
>>>>>
>>>>> I hear you but I'm also somewhat shying away from defining this at this point.
>>>>
>>>> Would you define them before the new UAPI is released though?
>>>>
>>>> I agree there is no need to have them in this patch series, but I think
>>>> we'd hit the below problems if the UAPI is released without them.
>>>>
>>>>> There are already too many things that need to happen and I will focus on the
>>>>> actual color blocks (LUTs, matrices) first. We'll always be able to add a new
>>>>> (read-only) colorop type to define scaling and tap behavior at any point and
>>>>> a client is free to ignore a color pipeline if it doesn't find any tap/scale
>>>>> info in it.
>>>>
>>>> How would userspace know to look for tap/scale info, if there is no
>>>> upstream definition even on paper?
>>>>
>>>
>>> So far OSes did not care about this. Whether that's good or bad is
>>> something everyone can answer for themselves.
>>>
>>> If you write a compositor and really need this you can just ignore
>>> color pipelines that don't have this, i.e., you'll probably want
>>> to wait with implementing color pipeline support until you have what
>>> you need from DRM/KMS.
>>>
>>> This is not to say I don't want to have support for Weston. But I'm
>>> wondering if we place too much importance on getting every little
>>> thing figured out whereas we could be making forward progress and
>>> address more aspects of a color pipeline in the future. There is a
>>> reason gamescope has made a huge difference in driving the color
>>> management work forward.
>>>
>>>> And the opposite case, if someone writes userspace without tap/scale
>>>> colorops, and then drivers add those, and there is no pipeline without
>>>> them, because they always exist. Would that userspace disregard all
>>>> those pipelines because it does not understand tap/scale colorops,
>>>> leaving no usable pipelines? Would that not be kernel regressing
>>>> userspace?
>>>>
>>>
>>> The simple solution is to leave previously advertised pipelines
>>> untouched and add a new version that does include scaling information.
>>>
>>>> If the kernel keeps on exposing pipelines without the colorops, it
>>>> fails the basic promise of the whole design: that all pixel value
>>>> affecting operations are at least listed if not controllable.
>>>>
>>>> How will we avoid painting ourselves in a corner?
>>>>
>>>> Maybe we need a colorop for "here be dragons" documented as having
>>>> unknown and unreliable effects, until driver authors are sure that
>>>> everything has been modelled in the pipeline and there are no unknowns?
>>>> Or a flag on the pipelines, if we can have that. Then we can at least
>>>> tell when the pipeline does not fulfil the basic promise.
>>>>
>>>
>>> The will always be dragons at some level.
>>
>> Do I understand right that the goal of fully understood color pipelines
>> is a lost cause?
>>
>> That every pipeline might always have something unknown and there is no
>> way for userspace to know if it does? Maybe because driver developers
>> don't know either?
>>
>> By something unknown I refer to anything outside of basic precision
>> issues. Doing interpolation or mixing of inputs on the wrong side of a
>> known non-linear colorop, for example.
> 
> I don't think that's the case. Hardware vendors should understand the
> hardware and exposing everything that affects the values is the goal
> here. There will be a transitional period where the pipelines might
> not expose every detail yet but that's fine. It's better than what we
> have now and should become even better with time. It would maybe be
> helpful in the future to have a cap, or property, or whatever, to
> indicate that the pipelines are "complete" descriptions of what
> happens to the values but we can discuss it when it becomes relevant.
> 

I agree, for the most part. But how do you then define "complete" if
you exclude precision issues?

>> An incremental UAPI development approach is fine by me, meaning that
>> pipelines might not be complete at first, but I believe that requires
>> telling userspace whether the driver developers consider the pipeline
>> complete (no undescribed operations that would significantly change
>> results from the expected results given the UAPI exposed pipeline).
>>
>> The prime example of what I would like to know is that if a FB
>> contains PQ-encoded image and I use a color pipeline to scale that
>> image up, will the interpolation happen before or after the non-linear
>> colorop that decodes PQ. That is a significant difference as pointed
>> out by Joshua.
>>

That's fair and I want to give that to you. My concern stems from
the sentiment that I hear that any pipeline that doesn't explicitly
advertise this is useless. I don't agree there. Let's not let perfect
be the enemy of good.

Harry

>>
>> Thanks,
>> pq
>