Plan to support Rockchip VPU in DRM, is it a good idea

[Randy Li]

On 08/26/2016 06:56 PM, Hans Verkuil wrote:
> On 08/26/2016 12:05 PM, Randy Li wrote:
>>
>> On 08/26/2016 05:34 PM, Hans Verkuil wrote:
>>> Hi Randi,
>>>
>>> On 08/26/2016 04:13 AM, Randy Li wrote:
>>>> Hello,
>>>>     We always use some kind of hack work to make our Video Process
>>>> Unit(Multi-format Video Encoder/Decoder) work in kernel. From a
>>>> customize driver(vpu service) to the customize V4L2 driver. The V4L2
>>>> subsystem is really not suitable for the stateless Video process or it
>>>> could make driver too fat.
>>>>     After talking to some kindness Intel guys and moving our userspace
>>>> library to VA-API driver, I find the DRM may the good choice for us.
>>>> But I don't know whether it is welcome to to submit a video driver in
>>>> DRM subsystem?
>>>>     Also our VPU(Video process unit) is not just like the Intel's, we
>>>> don't have VCS, we based on registers to set the encoder/decoder. I
>>>> think we may need a lots of IOCTL then. Also we do have a IOMMU in VPU
>>>> but also not a isolated memory for VPU, I don't know I should use TT
>>>> memory or GEM memory.
>>>>     I am actually not a member of the department in charge of VPU, and I
>>>> am just beginning to learning DRM(thank the help from Intel again), I am
>>>> not so good at memory part as well(I am more familiar with CMA not the
>>>> IOMMU way), I may need know guide about the implementations when I am
>>>> going to submit driver, I hope I could get help from someone.
>>>>
>>> It makes no sense to do this in the DRM subsystem IMHO. There are already
>>> quite a few HW codecs implemented in the V4L2 subsystem and more are in the
>>> pipeline. Putting codec support in different subsystems will just make
>>> userspace software much harder to write.
>>>
>>> One of the codecs that was posted to linux-media was actually from Rockchip:
>>>
>>> https://lkml.org/lkml/2016/2/29/861
>>>
>>> There is also a libVA driver (I think) that sits on top of it:
>>>
>>> https://github.com/rockchip-linux/rockchip-va-driver/tree/v4l2-libvpu
>> It is old version, I am the author of this
>> https://github.com/rockchip-linux/rockchip-va-driver
>>> For the Allwinner a patch series was posted yesterday:
>>>
>>> https://lkml.org/lkml/2016/8/25/246
>>>
>>> They created a pretty generic libVA userspace that looks very promising at
>>> first glance.
>>>
>>> What these have in common is that they depend on the Request API and Frame API,
>>> neither of which has been merged. The problem is that the Request API requires
>>> more work since not only controls have to be part of a request, but also formats,
>>> selection rectangles, and even dynamic routing changes. While that is not relevant
>>> for codecs, it is relevant for Android CameraHAL in general and complex devices
>>> like Google's Project Ara.
>> Actually just as the Intel did, our hardware decoder/encoder need full
>> settings for them, most of them are relevant to the codec. You may
>> notice that there is four extra control need to be set before. If the
>> libvpu(a helper library we offered to parse each slice to generate
>> decoder settings) is remove(in process now, only three decoder settings
>> can't got from VA-API directly), it would be more clearly.
>> We really a lots decoder settings information to make the decoder work.
>>> This is being worked on, but it is simply not yet ready. The core V4L2 developers
>>> involved in this plan to discuss this on the Monday before the ELCE in Berlin,
>>> to see if we can fast track this work somehow so this support can be merged.
>>>
>> I am glad to hear that. I hope that I could have an opportunity to show
>> our problems.
>>> If there are missing features in V4L2 (other that the two APIs discussed above)
>>> that prevent you from creating a good driver, then please discuss that with us.
>>> We are always open to suggestions and improvements and want to work with you on
>>> that.
>> I have a few experience with the s5p-mfc, and I do wrote a V4L2 encoder
>> plugin for Gstreamer.  I don't think the V4L2 is good place for us
>> stateless video processor, unless it would break the present implementation.
>>
>>     The stateful and stateless are operated quite differently. The
>> stateless must parse the header and set those settings for every frames.
>> The request data may quite different from vendor to vendor, even chip to
>> chip. It is impossible to make a common way to send those settings to
>> driver.For the samsung MFC, you don't need to do any parse work at all.
>>     Anyway, I would like to follow what Intel does now, we are both
>> stateless video processor.
> I don't see the problem. As I understand it what the hardware needs is the
> video data and settings (i.e. 'state'). It will process the video data (encode
> or decode) and return the result (probably with additional settings/state).
>
> V4L2 + Request API does exactly that. What does DRM offer you that makes life
Actually I don't reject the new framework, I also have heard this new 
API before. But the last update of Request API is 2015 and still not 
been merged yet. The DRM looks more stable and less limit. I still 
doesn't know how to implement a new  V4L2 Request API. I would like to 
last prototype of V4L2 Request API.
> easier for you compared to V4L2? I am not aware of Intel upstreaming any of
> their codec solutions, if you have pointers to patches from them attempting
> to do that, then please let me know.
I just like the Intel way, they move to support of any new format to 
userspace.  Also I don't think it would looks a streaming operation, one 
shot stop doesn't really like what V4L2 does.

Anyway I really want to know more detail about V4L2 request API, I am a 
junior member in Rockchip, I need my supervisor to make final decision. 
So I hope I could know about more this. Also the state of framework 
should be count into. I am too fresh to handle the bleeding state.

I would be ayaka in #dri-devel or #linux-rockchip @freenode for the 
weekend and stdint in the workday.
>
> If your goal is to get your code into the upstream kernel, then I am not
> sure Intel is the best place to look: I have yet to see patches from their
> media (camera/isp/codec) team. They do not seem to care about kernel
> upstreaming.
>
> I am trying to avoid that support for these devices gets fragmented over
> various subsystems and with various userspace solutions. I don't think that
> is in anyone's interest.
>
> Regards,
>
> 	Hans
>