[Intel-gfx] [RFC 01/33] drm/doc/rfc: Add RFC document for proposed Plane Color Pipeline

Harry Wentland harry.wentland at amd.com
Tue Aug 29 19:40:15 UTC 2023



On 2023-08-29 12:03, Uma Shankar wrote:
> Add the documentation for the new proposed Plane Color Pipeline.
> 
> Co-developed-by: Chaitanya Kumar Borah <chaitanya.kumar.borah at intel.com>
> Signed-off-by: Chaitanya Kumar Borah <chaitanya.kumar.borah at intel.com>
> Signed-off-by: Uma Shankar <uma.shankar at intel.com>
> ---
>   .../gpu/rfc/plane_color_pipeline.rst          | 394 ++++++++++++++++++
>   1 file changed, 394 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/plane_color_pipeline.rst
> 
> diff --git a/Documentation/gpu/rfc/plane_color_pipeline.rst b/Documentation/gpu/rfc/plane_color_pipeline.rst
> new file mode 100644
> index 000000000000..60ce515b6ea7
> --- /dev/null
> +++ b/Documentation/gpu/rfc/plane_color_pipeline.rst
> @@ -0,0 +1,394 @@
> +=======================================
> + Plane Color Pipeline: A UAPI proposal
> +=======================================
> +
> +To build the proposal on, lets take the premise of a color pipeline as shown
> +below.
> +
> + +-------------------------------------------+
> + |                RAM                        |
> + |  +------+    +---------+    +---------+   |
> + |  | FB 1 |    |  FB 2   |    | FB N    |   |
> + |  +------+    +---------+    +---------+   |
> + +-------------------------------------------+
> +       |  Plane Color Hardware Block |
> + +--------------------------------------------+
> + | +---v-----+   +---v-------+   +---v------+ |
> + | | Plane A |   | Plane B   |   | Plane N  | |
> + | | Pre-CSC |   | Pre-CSC   |   | Pre-CSC  | |
> + | +---+-----+   +---+-------+   +---+------+ |
> + |     |             |               |        |
> + | +---v-----+   +---v-------+   +---v------+ |
> + | |Plane A  |   | Plane B   |   | Plane N  | |
> + | |CSC/CTM  |   | CSC/CTM   |   | CSC/CTM  | |
> + | +---+-----+   +----+------+   +----+-----+ |
> + |     |              |               |       |
> + | +---v-----+   +----v------+   +----v-----+ |
> + | | Plane A |   | Plane B   |   | Plane N  | |
> + | |Post-CSC |   | Post-CSC  |   | Post-CSC | |
> + | +---+-----+   +----+------+   +----+-----+ |
> + |     |              |               |       |
> + +--------------------------------------------+
> ++------v--------------v---------------v-------|
> +||                                           ||
> +||           Pipe Blender                    ||
> ++--------------------+------------------------+
> +|                    |                        |
> +|        +-----------v----------+             |
> +|        |  Pipe Pre-CSC        |             |
> +|        |                      |             |
> +|        +-----------+----------+             |
> +|                    |            Pipe Color  |
> +|        +-----------v----------+ Hardware    |
> +|        |  Pipe CSC/CTM        |             |
> +|        |                      |             |
> +|        +-----------+----------+             |
> +|                    |                        |
> +|        +-----------v----------+             |
> +|        |  Pipe Post-CSC       |             |
> +|        |                      |             |
> +|        +-----------+----------+             |
> +|                    |                        |
> ++---------------------------------------------+
> +                     |
> +                     v
> +                Pipe Output
> +
> +Each plane consists of the following color blocks
> + * Pre-CSC : This block can used to linearize the input frame buffer data.
> +             The linear data then can be further acted on by the following
> +             color hardware blocks in the display hardware pipeline
> +
> + * CSC/CTM: Used to program color transformation matrix, this block is used
> +            to perform color space conversions like BT2020 to BT709 or BT601
> +            etc. This block acts on the linearized data coming from the
> +            Pre-CSC HW block.
> +
> + * Post-CSC: This HW block can be used to non-linearize frame buffer data to
> +             match the sink. Another use case of it could be to perform Tone
> +             mapping for HDR use-cases.
> +
> +Data from multiple planes will then be fed to pipe/crtc where it will get blended.
> +There is a similar set of HW blocks available at pipe/crtc level which acts on
> +this blended data.
> +
> +Below is a sample usecase fo video playback with sub-titles and playback
> +controls
> +
> +┌────────────┐    ┌─────────────┐     ┌─────────────┐    ┌─────────────┐
> +│FB1         │    │PRE-CSC      │     │ CTM Matrix  │    │ POST-CSC    │
> +│            ├───►│Linearize    ├────►│ BT709 to    ├───►│ SDR to HDR  │
> +│BT709 SDR   │    │             │     │ BT2020      │    │ Tone Mapping├─────┐
> +└────────────┘    └─────────────┘     └─────────────┘    └─────────────┘     │
> +(subtitles)                                                                  │
> +                                                                             │
> +┌────────────┐    ┌─────────────┐     ┌─────────────┐    ┌─────────────┐     │
> +│FB2         │    │PRE-CSC      │     │ CTM Matrix  │    │ POST-CSC    │     │
> +│            ├───►│Linearize    ├────►│ BT601 to    ├───►│ SDR to HDR  ├───┐ │
> +│BT601 SDR   │    │             │     │ BT2020      │    │ Tone Mapping│   │ │
> +└────────────┘    └─────────────┘     └─────────────┘    └─────────────┘   │ │
> +(Playback controls UI)                                                     │ │
> +                                                                           │ │
> +┌────────────┐    ┌─────────────┐     ┌─────────────┐    ┌─────────────┐   │ │
> +│FB2         │    │PRE-CSC      │     │ CTM Matrix  │    │ POST-CSC    │   │ │
> +│            ├───►│Linearize    ├────►│ BT601 to    ├───►│ SDR to HDR  ├─┐ │ │
> +│BT2020 HDR  │    │             │     │ pass-through│    │ pass-through│ │ │ │
> +└────────────┘    └─────────────┘     └─────────────┘    └─────────────┘ │ │ │
> +(video frame)                                                            │ │ │
> +                                                                         │ │ │
> +┌────────────────────────────────────────────────────────────────────────┴─┴─┘
> +│
> +│ ┌─────────────┐      ┌─────────────┐      ┌───────────────┐
> +│ │ CRTC        │      │ CRTC        │      │ CRTC          │
> +└─┤ PRE-CSC     ├─────►│ CSC/CTM     ├─────►│ POST-CSC      ├─────► TO Port
> +  │             │      │             │      │               │
> +  └─────────────┘      └─────────────┘      └───────────────┘
> +
> +This RFC is intended to propose an uAPI for the pre-blending color pipeline
> +(however, can be also extended to post blending pipeline).
> +
> +Below are the design considerations while designing the uAPI.
> +
> +1. Individual HW blocks can be muxed. (e.g. out of two HW blocks only one can be used)
> +2. Position of the HW block can be programmable
> +3. LUTs can be one dimentional or three dimentional
> +4. Number of LUT entries can vary across platforms
> +5. Precision of LUT entries can vary across platforms
> +6. Distribution of LUT entries may vary. e.g Mutli-segmented, Logarithmic,
> +   Piece-Wise Linear(PWL) etc
> +7. There can be parameterized/non-parameterized fixed function HW blocks.
> +   e.g. Just a hardware bit, to convert from one color space to another.
> +
> +Plane Color Pipeline: Design Proposal
> +=====================================
> +Each Color Hardware block will be represented by the structure drm_color_op.
> +
> +struct drm_color_op {
> +	enum color_op_block name;
> +	enum color_op_type type;
> +	u32 blob_id;
> +	u32 private_flags;
> +};
> +
> +The members of which will constitute:
> +1. name: A standardised enum for the color hardware block
> +2. type: The type of mathematical operation done by the hardware block.
> +         e.g. 1D Curve, 3D Curve, Matrix etc.
> +3. blob id: Id  pointing to a blob containing information about the hardware
> +         block advertising the respective capabilities to the userspace.
> +         It can be an optional field depending on the members "name" and "type".
> +4. private_flags: This can be used to provide vendor specific hints
> +         to user space
> +
> +
> +   For example to represent LUTs, we introduce the drm_color_lut_range
> +   structure. It can represent LUTs with varied number of entries and
> +   distributions (Multi segmented, Logarithmic etc).
> +
> +   struct drm_color_lut_range {
> +		/* DRM_MODE_LUT_* */
> +		__u32 flags;
> +		/* number of points on the curve */
> +		__u16 count;
> +		/* input/output bits per component */
> +		__u8 input_bpc, output_bpc;
> +		/* input start/end values */
> +		__s32 start, end;
> +		/* output min/max values */
> +		__s32 min, max;
> +   };
> +
> +Note: More details on exact usage and implementation of this structure can be
> +      found in the RFC. This structure is taken as is from the series.
> +      https://patchwork.freedesktop.org/series/90825/
> +      However, we can add more members to it to encompass all use-cases.
> +      For example. we can add a precision field to advertise the
> +      bitdepth of the LUTs. Similarly, we can reserve some bits in the flag
> +      field for vendor specific use cases.
> +
> +      At the same time, we don't need to pass any additional information for the
> +      CSC block as userspace and driver already agrees struct drm_color_ctm as
> +      a uAPI to pass on data.
> +
> +These color operations will form the building blocks of a color pipeline which
> +best represents the underlying Hardware. Color operations can be re-arranged,
> +substracted or added to create distinct color pipelines to accurately describe
> +the Hardware blocks present in the display engine.
> +

What do you mean when you're talking of re-arranging, or subtracting 
color operations?

> +In this proposal, a color pipeline is represented as an array of
> +struct drm_color_op.
> +
> +struct drm_color_op color_pipeline_1[]
> +
> +For example to represent the pre-blending color pipeline as described above
> +
> +We can create a color pipeline like this.
> +
> +struct drm_color_op color_pipeline_1[] = {
> +	{
> +		.name = DRM_CB_PRE_CSC,
> +		.type = CURVE_1D,
> +		.blob_id = 0; /* actual value to be populated during plane
> +						 initialization*/
> +	},
> +	{
> +		.name = DRM_CB_CSC,
> +		.type = MATRIX,
> +		.blob_id = 0;
> +	},
> +	{
> +		.name = DRM_CB_POST_CSC,
> +		.type = CURVE_1D,
> +		.blob_id = 0;
> +	},
> +};
> +
> +Then, for individual color operation, we add blobs to advertise the capability
> +of the particular Hardware block. In case of the example pipeline, we add
> +blobs of type drm_color_lut_range for the "pre-csc" and "post-csc".
> +For the "csc" block we pass no blob id to user space as it is known to both
> +user space and driver that drm_color_ctm structure is to be used for such
> +operation.
> +
> +To represent, this in a diagram.
> +
> +   struct drm_color_op color_pipeline_1[]
> +      +---------------------------+
> +      |                           |           drm_color_op
> +      |  +---------------------+--+-----------+---------------------+
> +      |  |                     |  |           |                     |
> +      |  |                     |  |           | +-----------------+ |
> +      |  |                     |  |           | |     name        | |
> +      |  |                     |  |           | +-----------------+ |
> +      |  |                     |  |           | |     type        | |
> +      |  |    color_op_1       |  |           | +-----------------+ |
> +      |  |                     |  |           | |     blob id     | +--------+
> +      |  |                     |  |           | +-----------------+ |        |
> +      |  |                     |  |           | |     private     | |        |
> +      |  |                     |  |           | |      flags      | |        |
> +      |  |                     |  |           | +-----------------+ |        |
> +      |  |                     |  |           |                     |        |
> +      |  +---------------------+--+-----------+---------------------+        |
> +      |                           |                                          |
> +      |                           |                                          |
> +      |  +---------------------+  |                                          |
> +      |  |                     |  |           drm_color_lut_range            |
> +      |  |    color_op_2       |  |           +-------------------------+    |
> +      |  |                     |  |           |                         |    |
> +      |  |                     |  |           | +---------------------+ |    |
> +      |  +---------------------+  |           | | segment 1 {         | |<---+
> +      |                           |           | |  ...                | |
> +      |  +---------------------+  |           | |  .input_bpc = 16,   | |
> +      |  |                     |  |           | |  .output_bpc = 16,  | |
> +      |  |    color_op_3       |  |           | |  ...                | |
> +      |  |                     |  |           | | }                   | |
> +      |  |                     |  |           | +---------------------+ |
> +      |  +---------------------+  |           |                         |
> +      |             .             |           | +---------------------+ |
> +      |             .             |           | | segment 2 {         | |
> +      |             .             |           | | ...                 | |
> +      +---------------------------+           | | }                   | |
> +                                              | |                     | |
> +                                              | |                     | |
> +                                              | |                     | |
> +                                              | +---------------------+ |
> +                                              |            .            |
> +                                              |            .            |
> +                                              |            .            |
> +                                              +-------------------------+
> +
> +
> +
> +This color pipeline is then packaged within a blob for the user space to
> +retrieve it. Details can be found in the next section
> +

Not sure I like blobs that contain other blob ids.

> +Exposing a color pipeline to user space
> +=======================================
> +
> +To advertise the available color pipelines, an immutable ENUM property
> +"GET_COLOR_PIPELINE" is introduced.
> +This enum property has blob id's as values. With each blob id representing
> +a distinct color pipeline based on underlying HW capabilities and their
> +respective combinations.
> +
> +The following output of drm_info [1], shows how color pipelines are visible
> +to userspace.
> +
> +├───Plane 0
> +    │   ├───Object ID: 31
> +    │   ├───CRTCs: {0}
> +    │   ├───Legacy info
> +        ...
> +    │       ├───"GET_COLOR_PIPELINE" (immutable): enum {no color pipeline,
> +						color pipeline 1, color pipeline 2}= no color pipeline
> +
> +To understand the capabilities of individual pipelines, first the userspace
> +need to retrieve the pipeline blob with the blob ids retrieved by reading the
> +enum property.
> +
> +Once the color pipeline is retrieved, user can then parse through
> +individual drm_color_op blocks to understand the capabilities of each
> +hardware block.
> +
> +Check IGT series to see how user space can parse through color pipelines.
> +Refer the IGT series here: https://patchwork.freedesktop.org/series/123018/
> +
> +Setting up a color pipeline
> +===========================
> +
> +Once the user space decides on a color pipeline, it can set the pipeline and
> +the corresponding data for the hardware blocks within the pipeline with
> +the BLOB property "SET_COLOR_PIPELINE".
> +
> +To achieve this two structures are introduced
> +
> +1.	struct drm_color_op_data: It represents data to be passed onto individual
> +							  color hardware blocks. It contains three members
> +                      a) name: to identify the color operation block
> +                      b) blob_id: pointing to the blob with data for the
> +                                  corresponding hardware block
> +
> +	struct drm_color_op_data {
> +		enum color_op_block name;
> +		u32 blob_id;
> +	};
> +
> +2.	struct drm_color_pipeline: This structure represents the aggregate
> +                                   pipeline to be set. it contains the following  members
> +					  a) num: pipeline number to be selected
> +					  b) size: size of the data to be passed onto the driver
> +					  c) data: array of struct drm_color_op_data with data for
> +                                                   the hardware block/s that userspace wants to
> +                                                   set values for.
> +
> +	struct drm_color_pipeline {
> +		int num;
> +		int size;
> +		struct drm_color_op_data *data;
> +	};
> +
> +	User can either:
> +	1. send data for all the color operations in a pipeline as shown in [2].
> +	   The color operation data need not be in order that the pipeline advertises
> +	   however, it should not contain data for any
> +	   color operation that is not present in the pipeline.
> +
> +	   Note: This check for pipeline is not yet implemented but if the
> +	   wider proposal is accepted we have few solutions in mind.
> +
> +	2. send data for a subset of color operations as shown in [3].For the
> +	   color operation that userspace does not send data will retain their
> +	   older state.
> +
> +	3. reset/disable the pipeline by setting the "SET_COLOR_PIPELINE" blob
> +	   property to NULL as shown in both [2] and [3]
> +
> +	4. change the color pipeline as demonstrated in [3].
> +	   On the new pipeline, the user is expected to setup all the color hardware block
> +	   Once the user requests a pipeline change, the driver will provide it a clean slate
> +           which means that all the data previously set by the user will be discarded even if
> +           there are common hardware blocks between the two(previous and current) pipelines.
> +
> +IGT implementation can be found here [4]
> +
> +Representing Fixed function hardware
> +====================================
> +
> +To provide support for fixed function hardware, the driver could expose vendor
> +specific struct drm_color_op with parameters that both the userspace and
> +driver agrees on. To demonstrate, let's consider a hyphothetical fixed
> +function hardware block that converts BT601 to BT2020.
> +The driver can choose to advertise the block as such.
> +
> +struct drm_color_op color_pipeline_X[] = {
> +	{
> +		.name = DRM_CB_PRIVATE,
> +		.type = FIXED_FUNCTION,
> +		.blob_id = 45;
> +	},
> +}
> +
> +Where the blob represents some vendor specific enums, strings or any other
> +appropriate data types which both the user-space and drivers are aligned on.
> +
> +blob:45 {
> +	VENDORXXX_BT602_TO_BT2020,
> +};
> +
> +For enabling or passing parameters to such blocks, the user can send data
> +to the driver wrapped within a blob like any other color operation block.
> +
> +	struct drm_color_op_data {
> +		.name = DRM_CB_PRIVATE;
> +		.blob_id = 46;
> +	} ;
> +
> +where blob with id 46 contains data to enable the fixed function hardware(s).
> +

One major thing missing from the RFC is an explanation on why we want to 
go with a prescriptive model, rather than a descriptive model. This 
question came up in Dave's response to Simon's RFC:
https://lore.kernel.org/dri-devel/CAPM=9tz54Jc1HSjdh5A7iG4X8Gvgg46qu7Ezvgnmj4N6gbY+Kw@mail.gmail.com/

This is a rough attempt at such an explanation:
https://gitlab.freedesktop.org/hwentland/linux/-/merge_requests/5/diffs?commit_id=bc99737623796b39925767d6f8cbc097ad0b4d8d

Harry

> +References
> +==========
> +
> +[1] https://gitlab.freedesktop.org/emersion/drm_info
> +[2] https://patchwork.freedesktop.org/patch/554827/?series=123018&rev=1
> +[3] https://patchwork.freedesktop.org/patch/554826/?series=123018&rev=1
> +[4] https://patchwork.freedesktop.org/series/123018/


More information about the Intel-gfx mailing list