Support for 2D engines/blitters in V4L2 and DRM

[Nicolas Dufresne]

Le jeudi 18 avril 2019 à 10:18 +0200, Daniel Vetter a écrit :
> > It would be cool if both could be used concurrently and not just return
> > -EBUSY when the device is used with the other subsystem.
> 
> We live in this world already :-) I think there's even patches (or merged
> already) to add fences to v4l, for Android.

This work is currently suspended. It will require some feature on DRM
display to really make this useful, but there is also a lot of
challanges in V4L2. In GFX space, most of the use case are about
rendering as soon as possible. Though, in multimedia we have two
problems, we need to synchronize the frame rendering with the audio,
and output buffers may comes out of order due to how video CODECs are
made.

In the first, we'd need a mechanism where we can schedule a render at a
specific time or vblank. We can of course already implement this in
software, but with fences, the scheduling would need to be done in the
driver. Then if the fence is signalled earlier, the driver should hold
on until the delay is met. If the fence got signalled late, we also
need to think of a workflow. As we can't schedule more then one render
in DRM at one time, I don't really see yet how to make that work.

For the second, it's complicated on V4L2 side. Currently we signal
buffers when they are ready in the display order. With fences, we
receive early pairs buffer and fence (in decoding order). There exist
cases where reordering is done by the driver (stateful CODEC). We
cannot schedule these immediately we would need a new mechanism to know
which one come next. If we just reuse current mechnism, it would void
the fence usage since the fence will always be signalled by the time it
reaches DRM or other v4l2 component.

There also other issues, for video capture pipeline, if you are not
rendering ASAP, you need the HW timestamp in order to schedule. Again,
we'd get the fence early, but the actual timestamp will be signalled at
the very last minutes, so we also risk of turning the fence into pure
overhead. Note that as we speak, I have colleagues who are
experimenting with frame timestamp prediction that slaves to the
effective timestamp (catching up over time). But we still have issues
when the capture driver skipped a frame (missed a capture window).


I hope this is useful reflection data,
Nicolas