<html><body><p>
<pre>
On Mon, 2025-07-07 at 09:31 +0800, shangyao lin wrote:
> From: "shangyao.lin" <shangyao.lin@mediatek.com>
>
> This document covers:
> - The hardware pipeline and firmware interface
> - Integration with the V4L2 CAMSYS/media controller framework
> - The list of custom V4L2 controls, their usage, valid ranges, and typical usage flow
You may drop some advanced function, but I don't know which are advanced function, so I would review all.
>
> Signed-off-by: shangyao.lin <shangyao.lin@mediatek.com>
> ---
> .../media/v4l/mtk-isp7x-camsys.rst | 94 +++++++++
> .../media/v4l/mtk-isp7x-controls.rst | 199 ++++++++++++++++++
> 2 files changed, 293 insertions(+)
> create mode 100755 Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
> create mode 100755 Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
>
> diff --git a/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst b/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
> new file mode 100755
> index 000000000000..04a1fe1c448d
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
> @@ -0,0 +1,94 @@
> +==========================
> +MediaTek ISP7x Camera System
> +==========================
> +
> +Overview
> +========
> +
> +The MediaTek ISP7x camera system is designed to provide a flexible, high-performance image capture pipeline for SoCs, leveraging the Linux V4L2 framework. The system is composed of multiple hardware modules (Sensor, SENINF, RAW ISP, etc.) and a firmware backend, with a clear separation of control and data flow. This document describes the hardware architecture from the perspective of the interface exposed to the firmware (backend), and how it integrates with the V4L2 CAMSYS framework.
> +
> +V4L2 CAMSYS Framework Overview
> +==============================
> +
> +The V4L2 CAMSYS framework models the camera pipeline as a set of interconnected devices and subdevices:
> +
> +- **Video Device**: Exposes buffer queue/dequeue (enque/deque) operations to user space, allowing applications to stream image data.
> +- **Subdevice**: Represents a hardware block (e.g., sensor, SENINF, RAW ISP) and handles control/configuration operations.
> +- **Pads and Links**: Each subdevice has pads (input/output endpoints). Media links connect pads, representing the physical and logical data flow between hardware blocks.
> +
> +The topology and link state are established at probe time and can be dynamically enabled/disabled by user space, depending on the desired pipeline configuration.
> +
> +Hardware Pipeline
> +=================
> +
> +A typical pipeline consists of:
> +
> +.. code-block::
> +
> + Sensor --> SENINF --> RAW ISP --> (optional: YUV ISP) --> Memory
> +
> +- **Sensor**: Provides MIPI CSI-2 data.
> +- **SENINF**: Receives and demultiplexes sensor data, supports virtual channel (VC) and data type (DT) routing.
> +- **RAW ISP**: Main image signal processor, handles demosaicing, noise reduction, and other image enhancements.
> +- **YUV ISP** (optional): Further processing for YUV output.
> +- **CAMSYS**: System controller, manages pipeline state, buffer flow, and synchronization.
> +
> +Firmware Interface
> +==================
> +
> +The firmware (backend, e.g., CCD/SCP) is responsible for:
> +
> +- Calculating hardware settings (3A, tuning, DMA, FBC, Twin, etc.) for each frame.
> +- Generating command queues (CQ) and descriptors to program the hardware.
> +- Abstracting hardware-specific details from the kernel driver.
> +
> +The kernel driver communicates with the firmware via an inter-processor interface (IPI), typically using RPMSG. The main interactions are:
> +
> +1. **Session Management**: The kernel establishes a session with the firmware at stream-on, and destroys it at stream-off.
> +2. **Per-frame Request Scheduling**: For each user request (frame), the kernel sends a request to the firmware, including sensor settings, buffer addresses, and control parameters.
> +3. **CQ Generation**: The firmware generates the necessary command queues and returns them to the kernel.
> +4. **Event Notification**: The firmware notifies the kernel of CQ completion, errors, or other events.
> +
> +Request Scheduling and Buffer Flow
> +==================================
> +
> +The request scheduling flow is as follows:
> +
> +1. **User space** submits a request (with buffer and control settings) via V4L2.
> +2. **PipeMgr** (user space) collects requests, groups them by file descriptor, and sends them to the kernel CAMSYS driver.
> +3. **CAMSYS** schedules requests, manages buffer flow, and coordinates with the firmware for hardware programming.
> +4. **Firmware** processes each request, generates CQ, and signals completion.
> +5. **Kernel** dequeues the buffer and notifies user space.
> +
> +The system supports per-frame control, allowing sensor settings and ISP parameters to be updated on a frame-by-frame basis.
> +
> +Media Controller Topology
> +=========================
> +
> +The media controller graph models the hardware pipeline as a set of entities (video devices, subdevices) connected by links. Each entity exposes pads, and links are established at probe time based on the hardware configuration and device tree.
> +
> +.. code-block::
> +
> + +---------+ +--------+ +--------+ +------+
> + | Sensor | --> | SENINF | --> | RAW ISP| --> | Video|
> + +---------+ +--------+ +--------+ +------+
> +
> +- Solid lines: active links (enabled)
> +- Dashed lines: inactive links (disabled)
> +
> +User space can query and configure the topology using standard V4L2 and media controller APIs.
> +
> +Backend Abstraction
> +===================
> +
> +The backend firmware abstracts hardware-specific programming, allowing the kernel driver to remain generic and maintainable. All hardware register programming, tuning, and advanced features are handled in the firmware, with the kernel acting as a control and data flow manager.
> +
> +This separation ensures that proprietary hardware details remain in the firmware, while the kernel driver exposes a standard V4L2 interface.
> +
> +Summary
> +=======
> +
> +- The MediaTek ISP7x camera system is built on the V4L2 media controller framework, with a modular pipeline of video devices and subdevices.
> +- The kernel driver manages pipeline state, buffer flow, and per-frame control, while delegating hardware programming to the backend firmware via IPI.
> +- The firmware interface is session-based, supports per-frame CQ generation, and abstracts all hardware-specific details from the kernel.
> +- The media controller graph models the hardware pipeline, allowing flexible configuration and dynamic topology management.
> \ No newline at end of file
> diff --git a/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst b/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
> new file mode 100755
> index 000000000000..0e8d41d339e7
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
> @@ -0,0 +1,199 @@
> +==========================================
> +MediaTek ISP Camsys User Space Interface
> +==========================================
> +
> +Overview
> +--------
> +
> +The MediaTek ISP camsys driver provides a set of custom V4L2 controls for user space
> +applications to configure and query the ISP hardware. These controls allow
> +fine-grained management of resource allocation, feature enablement, and
> +hardware-specific settings.
> +
> +Typical Usage Flow
> +------------------
> +
> +1. Open the video device node (e.g., ``/dev/video0``).
> +2. Set required V4L2 controls using ``VIDIOC_S_CTRL`` or ``VIDIOC_S_EXT_CTRLS``.
> +3. Set up video format and buffers (``VIDIOC_S_FMT``, ``VIDIOC_REQBUFS``, etc.).
> +4. Start streaming with ``VIDIOC_STREAMON``.
> +5. Queue and dequeue buffers for image processing (``VIDIOC_QBUF``, ``VIDIOC_DQBUF``).
> +6. Stop streaming with ``VIDIOC_STREAMOFF``.
> +
> +Control List and Documentation
> +-----------------------------
> +
> +Below is a list of custom V4L2 controls provided by the camsys driver, with detailed descriptions.
> +
> +.. list-table::
> + :header-rows: 1
> +
> + * - Control Name
> + - Type
> + - Description
> + - Valid Values / Range
> + - When to Set
> +
> + * - ``V4L2_CID_MTK_CAM_USED_ENGINE_LIMIT``
> + - Integer
> + - Limits the maximum number of RAW engines (hardware pipelines) that can be used for this stream.
> + - 1 ~ 2 (default: 2)
I don't know why you have this interface.
You have V4L2_CID_MTK_CAM_USED_ENGINE which would decide the engine number, why have this interface to limit it?
If I want 1 engine, I just use V4L2_CID_MTK_CAM_USED_ENGINE to set. V4L2_CID_MTK_CAM_USED_ENGINE_LIMIT is redundant.
So drop this.
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_BIN_LIMIT``
> + - Integer
> + - Sets the maximum binning ratio allowed for the stream.
> + - 0 ~ 0xFFF (default: 0)
Is the binning equal to pixel binning [1]?
If it is, does 0 mean no binning effect?
What does the level mean?
I guess is:
The maximum is 16x16 binning, so 0xFFF means 16x16 binning, and 0x800 means 4x4 binning.
This is just my guess, please provide more information what the level mean.
[1] https://en.wikipedia.org/wiki/Pixel_binning
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_FRZ_LIMIT``
> + - Integer
> + - Sets the maximum resizer (FRZ) ratio allowed (as a percentage).
> + - 70 ~ 100 (default: 100)
I don't know why you have this interface.
You have V4L2_CID_MTK_CAM_FRZ which would decide the resize ratio, why have this interface to limit it?
If I want 0.8 ratio, I just use V4L2_CID_MTK_CAM_FRZ to set. V4L2_CID_MTK_CAM_FRZ_LIMIT is redundant.
So drop this.
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_RESOURCE_PLAN_POLICY``
> + - Integer (Enum)
> + - Selects the resource planning policy for the stream.
> + - 0: Default, 1: Power-saving, 2: Performance, ... (0~10)
You just list 0, 1, 2.
What about 3 ~ 10?
Add the rest information.
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_USED_ENGINE``
> + - Integer
> + - Reports or sets the number of RAW engines actually used for this stream after negotiation.
> + - 1 ~ 2 (default: 2)
> + - After resource negotiation
What's the negotiation?
Is it V4L2_CID_MTK_CAM_USED_ENGINE_TRY?
or all XXX_TRY interface?
Please explicitly list all these interface.
> +
> + * - ``V4L2_CID_MTK_CAM_BIN``
> + - Boolean
> + - Enables or disables binning for the stream.
> + - 0 (disable), 1 (enable, default)
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_FRZ``
> + - Integer
> + - Sets the resizer (FRZ) ratio (as a percentage) for the stream.
> + - 70 ~ 100 (default: 100)
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_USED_ENGINE_TRY``
> + - Integer
> + - Reports the number of RAW engines that would be used for the current "try" format negotiation.
> + - 1 ~ 2 (default: 2)
> + - After VIDIOC_TRY_FMT
> +
> + * - ``V4L2_CID_MTK_CAM_BIN_TRY``
> + - Boolean
> + - Reports whether binning would be enabled for the current "try" format negotiation.
> + - 0, 1 (default: 1)
> + - After VIDIOC_TRY_FMT
> +
> + * - ``V4L2_CID_MTK_CAM_FRZ_TRY``
> + - Integer
> + - Reports the resizer ratio that would be used for the current "try" format negotiation.
> + - 70 ~ 100 (default: 100)
> + - After VIDIOC_TRY_FMT
> +
> + * - ``V4L2_CID_MTK_CAM_PIXEL_RATE``
> + - Integer64
> + - Sets the pixel rate (in pixels per second) of the sensor for resource calculation.
> + - 0 ~ 0xFFFFFFFF
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_FEATURE``
> + - Bitmask (Integer64)
> + - Bitmask of enabled features for the stream (e.g., HDR, 3A, etc.). See driver header for details.
In patch [12/13], I does not see any information about these feature.
Add these feature definition in patch [12/13].
> + - Bitmask (0 ~ RAW_FUNCTION_END)
> + - Before streaming or per-frame
> +
> + * - ``V4L2_CID_MTK_CAM_SYNC_ID``
> + - Integer64
> + - Frame synchronization ID for multi-sensor or multi-pipeline synchronization.
Describe how to use this ID for multi-sensor or multi-pipeline.
I think every frame has a timestamp which could be used as ID.
Why not use timestamp instead of this ID?
> + - -1 ~ 0x7FFFFFFF (default: -1)
> + - Before streaming or per-frame
> +
> + * - ``V4L2_CID_MTK_CAM_RAW_PATH_SELECT``
> + - Integer
> + - Selects the RAW path (e.g., main or secondary) for the stream.
> + - 0, 1 (default: 1)
You have draw a graph like this, but it does not show two raw path.
+---------+ +--------+ +--------+ +------+
| Sensor | --> | SENINF | --> | RAW ISP| --> | Video|
+---------+ +--------+ +--------+ +------+
Does it mean this?
+---------+ +--------+ +---------------+ +------+
| Sensor | --> | SENINF | +-> | MAIN RAW ISP | --> | Video|
+---------+ +--------+ | +---------------+ +------+
|
| +---------------+ +------+
+-> | SECOND RAW ISP| --> | Video|
+---------------+ +------+
If so, change your graph.
How do we decide which should we select?
What's the difference of these two RAW?
When we configure camsys driver, both RAW has the same setting or each RAW has different setting?
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_HSF_EN``
> + - Integer
> + - Enables or disables hardware secure flow (HSF) for the stream.
What is HSF?
Does it mean output video buffer is accessible only by trusted CPU or hardware?
Does it mean camsys hardware could be configured only by trusted CPU or firmware?
If so, add this information.
> + - 0 (disable), 1 (enable)
> + - Before streaming
> +
> + * - ``V4L2_CID_MTK_CAM_PDE_INFO``
> + - Integer (struct mtk_cam_pde_info)
> + - Provides or sets phase detection (PDE) information for PDAF/AF features.
> + - See struct definition
After seeing the structure, I still don't understand what it is.
What is pdo? What is pdi? What is pd table offset?
Try to add more information to let us know how to configure it.
> + - Before streaming or per-frame
> +
> + * - ``V4L2_CID_MTK_CAM_MSTREAM_EXPOSURE``
> + - Integer/Integer64
> + - Controls multi-stream exposure settings for advanced HDR or multi-exposure modes.
> + - See driver header
I does not see this in driver header, so drop this.
> + - Before streaming or per-frame
> +
> + * - ``V4L2_CID_MTK_CAM_RAW_RESOURCE_CALC``
> + - Compound (struct mtk_cam_resource)
> + - Triggers resource calculation for the current format and settings.
> + - See struct definition
> + - During format negotiation
Call this between some XXX_TRY interface?
The negotiation interface could be called in a random sequence?
If there are dependencies, explicitly describe it.
> +
> + * - ``V4L2_CID_MTK_CAM_TG_FLASH_CFG``
> + - Integer/struct
> + - Configures timing generator (TG) and flash synchronization.
> + - See driver header
I could not find this in driver, so drop it.
> + - Before streaming or per-frame
> +
> + * - ``V4L2_CID_MTK_CAM_RAW_RESOURCE_UPDATE``
> + - Integer
> + - Notifies the driver of a sensor mode update (e.g., seamless switch).
> + - 0 ~ 0xF (default: 0)
What is sensor mode?
I still don't understand to set value 3 or value 7?
Describe more about how to set these value.
> + - When sensor mode changes
> +
> + * - ``V4L2_CID_MTK_CAM_CAMSYS_HW_MODE``
> + - Integer64
> + - Selects the hardware mode for the CAMSYS pipeline (e.g., normal, secure, etc.).
> + - 0 ~ 0x7FFFFFFF
What is hardware mode?
I still don't understand to set value 3 or value 7?
Describe more about how to set these value.
Do you understand the meaning of upstream?
Upstream is not a cloud server to store your code.
Upstream is to share your code so others could use your code to control this hardware.
Try to let us understand how to use this code to control hardware.
Regards,
CK
> + - Before streaming
> +
> +Control Usage Patterns
> +---------------------
> +
> +- Controls with ``_TRY`` suffix are used to query the result of a "try" format negotiation (``VIDIOC_TRY_FMT``), without committing changes.
> +- Controls without ``_TRY`` are used to set or get the actual configuration (``VIDIOC_S_FMT``, ``VIDIOC_G_CTRL``).
> +- Use ``V4L2_CID_MTK_CAM_RAW_RESOURCE_CALC`` to check if the requested format and settings are feasible, and to get recommended values.
> +- Use ``V4L2_CID_MTK_CAM_RAW_RESOURCE_UPDATE`` to notify the driver of a sensor mode change (e.g., seamless switch).
> +- Use ``V4L2_CID_MTK_CAM_SYNC_ID`` to synchronize frames across multiple pipelines.
> +- Use ``V4L2_CID_MTK_CAM_FEATURE`` to enable/disable advanced features (bitmask).
> +- Use ``V4L2_CID_MTK_CAM_PDE_INFO`` to set/get phase detection information for PDAF/AF.
> +- Use ``V4L2_CID_MTK_CAM_PIXEL_RATE`` to inform the driver of the sensor's pixel rate for bandwidth/resource calculation.
> +
> +Example: Setting a Control
> +--------------------------
> +
> +.. code-block:: c
> +
> + int fd = open("/dev/video0", O_RDWR);
> + struct v4l2_control ctrl = {
> + .id = V4L2_CID_MTK_CAM_BIN_LIMIT,
> + .value = 2,
> + };
> + if (ioctl(fd, VIDIOC_S_CTRL, &ctrl) < 0) {
> + perror("Failed to set BIN_LIMIT");
> + }
> +
> +Notes
> +-----
> +
> +- All controls should be set before starting the video stream unless otherwise specified.
> +- Refer to the driver header file for detailed struct definitions and bitmask values.
> +- The driver will reject invalid values or unsupported configurations with -EINVAL.
> +
> +References
> +----------
> +
> +- V4L2 documentation: https://urldefense.com/v3/__https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/v4l2.html__;!!CTRNKA9wMg0ARbw!koPmGRLtqjG7_WecTl2EBNDMmuZO-_VzYEAbKzsTJhueUGekgy_ky5-FeOERXvI6377LscGrivr2cMlRuXbU3pM$
> \ No newline at end of file
</pre>
</p></body></html><!--type:text--><!--{--><pre>************* MEDIATEK Confidentiality Notice
********************
The information contained in this e-mail message (including any
attachments) may be confidential, proprietary, privileged, or otherwise
exempt from disclosure under applicable laws. It is intended to be
conveyed only to the designated recipient(s). Any use, dissemination,
distribution, printing, retaining or copying of this e-mail (including its
attachments) by unintended recipient(s) is strictly prohibited and may
be unlawful. If you are not an intended recipient of this e-mail, or believe
that you have received this e-mail in error, please notify the sender
immediately (by replying to this e-mail), delete any and all copies of
this e-mail (including any attachments) from your system, and do not
disclose the content of this e-mail to any other person. Thank you!
</pre><!--}-->