v4l2 mem2mem compose support?

[Tim Harvey]

On Sat, Feb 16, 2019 at 1:14 PM Nicolas Dufresne <nicolas at ndufresne.ca> wrote:
>
> Le sam. 16 févr. 2019 à 13:40, Hans Verkuil <hverkuil at xs4all.nl> a écrit :
> >
> > On 2/16/19 4:42 PM, Nicolas Dufresne wrote:
> > > Le sam. 16 févr. 2019 à 04:48, Hans Verkuil <hverkuil at xs4all.nl> a écrit :
> > >>
> > >> On 2/16/19 10:42 AM, Hans Verkuil wrote:
> > >>> On 2/16/19 1:16 AM, Tim Harvey wrote:
> > >>>> Greetings,
> > >>>>
> > >>>> What is needed to be able to take advantage of hardware video
> > >>>> composing capabilities and make them available in something like
> > >>>> GStreamer?
> > >>>
> > >>> Are you talking about what is needed in a driver or what is needed in
> > >>> gstreamer? Or both?
> > >>>
> > >>> In any case, the driver needs to support the V4L2 selection API, specifically
> > >>> the compose target rectangle for the video capture.
> > >>
> > >> I forgot to mention that the driver should allow the compose rectangle to
> > >> be anywhere within the bounding rectangle as set by S_FMT(CAPTURE).
> > >>
> > >> In addition, this also means that the DMA has to be able to do scatter-gather,
> > >> which I believe is not the case for the imx m2m hardware.
> > >
> > > I believe the 2D blitter can take an arbitrary source rectangle and
> > > compose it to an arbitrary destination rectangle (a lot of these will
> > > in fact use Q16 coordinate, allowing for subpixel rectangle, something
> > > that V4L2 does not support).
> >
> > Not entirely true. I think this can be done through the selection API,
> > although it would require some updating of the spec and perhaps the
> > introduction of a field or flag. The original VIDIOC_CROPCAP and VIDIOC_CROP
> > ioctls actually could do this since with analog video (e.g. S-Video) you
> > did not really have the concept of a 'pixel'. It's an analog waveform after
> > all. In principle the selection API works in the same way, even though the
> > documentation always assumes that the selection rectangles map directly on
> > the digitized pixels. I'm not sure if there are still drivers that report
> > different crop bounds in CROPCAP compared to actual number of digitized pixels.
> > The bttv driver is most likely to do that, but I haven't checked.
> >
> > Doing so made it very hard to understand, though.
> >
> >  I don't think this driver exist in any
> > > form upstream on IMX side. The Rockchip dev tried to get one in
> > > recently, but the discussion didn't go so well with  the rejection of
> > > the proposed porter duff controls was probably devoting, as picking
> > > the right blending algorithm is the basic of such driver.
> >
> > I tried to find the reason why the Porter Duff control was dropped in v8
> > of the rockchip RGA patch series back in 2017.
> >
> > I can't find any discussion about it on the mailinglist, so perhaps it
> > was discussed on irc.
> >
> > Do you remember why it was removed?
>
> I'll try and retrace what happened, it was not a nack, and I realize
> that "rejection" wasn't the right word, but if I remember, the focus
> in the review went fully around this and the fact that it was doing
> blending which such API, while the original intention with the driver
> was to have CSC, so removing this was basically a way forward.
>
> >
> > >
> > > I believe a better approach for upstreaming such driver would be to
> > > write an M2M spec specific to that type of m2m drivers. That spec
> > > would cover scalers and rotators, since unlike the IPUv3 (which I
> > > believe you are referring too) a lot of the CSC and Scaler are
> > > blitters.
> >
> > No, I was referring to the imx m2m driver that Phillip is working on.
>
> I'll need  to check what driver Veolab was using, but if it's the
> same, then maybe it only do source-over operations using SELECTION as
> you described. If I remember their use case, they where doing simple
> source-over blending of two video feeds.
>
> Could it be this ?
> https://gitlab.com/veo-labs/linux/tree/veobox/drivers/staging/media/imx6/m2m
> Is it an ancester of Philipp's driver ?
>

It does look like this was an ancestor of Philipp's mem2mem driver
which is currently under review
(https://patchwork.kernel.org/project/linux-media/list/?series=67977)

Hans, can you give me a little more detail about what would be needed
in Phillipp's mem2mem driver to do this (or if its already there). I
imagine what we are talking about is being able to specify the
destination buffer and a rect within it.

I will take a look at the gstreamer plugin at
https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/308
and see if I can get that building on top of master.

It sounds like that's a good path towards hardware accelerated composing.

Thanks!

Tim