Implementing Miracast?

[Daniel Vetter]

On Thu, Dec 03, 2015 at 07:26:31PM +0200, Martin Peres wrote:
> On 03/12/15 18:38, Ilia Mirkin wrote:
> >On Thu, Dec 3, 2015 at 11:10 AM, Laurent Pinchart
> ><laurent.pinchart at ideasonboard.com> wrote:
> >>Hi Ilia,
> >>
> >>On Thursday 03 December 2015 11:03:28 Ilia Mirkin wrote:
> >>>On Thu, Dec 3, 2015 at 10:53 AM, Laurent Pinchart wrote:
> >>>>On Thursday 03 December 2015 10:42:50 Ilia Mirkin wrote:
> >>>>>On Thu, Dec 3, 2015 at 10:34 AM, Laurent Pinchart wrote:
> >>>>>>On Thursday 03 December 2015 14:42:51 Hannikainen, Jaakko wrote:
> >>>>>>>Hello,
> >>>>>>>
> >>>>>>>We're developing Miracast (HDMI over Wireless connections). The
> >>>>>>>current progress is that it 'works' in the userspace but doesn't have
> >>>>>>>any integration with X/Wayland and can only mirror the current desktop
> >>>>>>>using gstreamer.
> >>>>>>>
> >>>>>>>We're looking into extending the implementation so that we would be
> >>>>>>>able to use the remote screens just as any other connected screen, but
> >>>>>>>we're not quite sure where we should implement it.
> >>>>>>>
> >>>>>>>The DRM interface seems like the perfect fit since we wouldn't need to
> >>>>>>>patch every compositor.
> >>>>>>>
> >>>>>>>Right now, gstreamer is the equivalent of the crtc/encoder, in the DRM
> >>>>>>>model. Screens / crtcs are discovered using a WiFi's p2p protocol
> >>>>>>>which means that screens should be hotpluggable. Since we cannot
> >>>>>>>change the number of crtcs of a driver on the fly, we propose adding
> >>>>>>>and removing gpus with one crtc attached and no rendering
> >>>>>>>capabilities.
> >>>>>>>
> >>>>>>>Compositors and X currently use udev to list gpus and get run-time
> >>>>>>>events for gpu hot-plugging (see the work from Dave Airlie for USB
> >>>>>>>GPUs, using the modesetting X driver). We did not find a way to tell
> >>>>>>>udev that we have a new device and it seems like the only way to get
> >>>>>>>it to pick up our driver is from a uevent which can only be generated
> >>>>>>>from the kernel.
> >>>>>>>
> >>>>>>>Since we have so many userspace components, it doesn't make sense to
> >>>>>>>implement the entire driver in the kernel.
> >>>>>>>
> >>>>>>>We would thus need to have a communication from the kernel space to
> >>>>>>>the userspace at least to send the flip commands to the fake crtc.
> >>>>>>>Since we need this, why not implement everything in the userspace and
> >>>>>>>just redirect the ioctls to the userspace driver?
> >>>>>>>
> >>>>>>>This is exactly what fuse / cuse [1] does, with the minor catch that
> >>>>>>>it creates devices in /sys/class/cuse instead of drm. This prevents
> >>>>>>>the wayland compositors and X to pick it up as a normal drm driver...
> >>>>>>>
> >>>>>>>We would thus need to have the drm subsystem create the device nodes
> >>>>>>>for us when the userspace needs to create a new gpu. We could create a
> >>>>>>>node named /dev/dri/cuse_card that, when opened, would allocate a node
> >>>>>>>(/dev/dri/cardX) and would use cuse/fuse to redirect the ioctls to the
> >>>>>>>process who opened /dev/dri/cuse_card.
> >>>>>>>
> >>>>>>>The process would then be responsible for decoding the ioctl and
> >>>>>>>implementing the drm API.
> >>>>>>>
> >>>>>>>Since this is a major change which would allow proprietary drivers to
> >>>>>>>be implemented in the userspace and since we may have missed something
> >>>>>>>obvious, we would like to start a discussion on this. What are your
> >>>>>>>thoughts?
> >>>>>>As you raise the issue, how would you prevent proprietary userspace
> >>>>>>drivers to be implemented ? Anything that would allow vendors to
> >>>>>>destroy the Linux graphics ecosystem would receive a big nack from me.
> >>>>>AFAIK the displaylink people already have precisely such a driver -- a
> >>>>>(open-source) kernel module that allows their (closed-source)
> >>>>>userspace blob to present a drm node to pass through modesetting/etc
> >>>>>ioctl's.
> >>>>Are you talking about the drivers/gpu/drm/udl/ driver ? I might be wrong
> >>>>but I'm not aware of that kernel driver requiring a closed-source
> >>>>userspace blob.
> >>>Nope. That driver only works for their USB2 parts. This is what I mean:
> >>>
> >>>https://github.com/DisplayLink/evdi
> >>>http://support.displaylink.com/knowledgebase/articles/679060
> >>>http://support.displaylink.com/knowledgebase/articles/615714#ubuntu
> >>Right. That's out-of-tree, people are free to screw up on their own there ;-)
> >Sure, but it's identical to Jaakko's proposal from what I can
> >(quickly) tell. And it's an example of someone taking an interface
> >like that and writing a proprietary driver on top.
> >
> >   -ilia
> >
> 
> You are right Ilia, this is indeed what Jaakko and I had in mind, but they
> did not re-use the fuse/cuse framework to do the serialization of the
> ioctls.
> 
> Not sure what we can do against allowing proprietary drivers to use this
> feature though :s To be fair, nothing prevents any vendor to do this shim
> themselves and nvidia definitely did it, and directly called their
> closed-source driver.
> 
> Any proposition on how to handle this case? I guess we could limit that to
> screens only, no rendering. That would block any serious GPU manufacturer
> from using this code even if any sane person would never write a driver in
> the userspace...

Hm for virtual devices like this I figured there's no point exporting the
full kms api to userspace, but instead we'd just need a simple kms driver
with just 1 crtc and 1 connector per drm_device. Plus a special device
node (v4l is probably inappropriate since it doesn't do damage) where the
miracast userspace can receive events with just the following information:
- virtual screen size
- fd to the underlying shmem node for the current fb. Or maybe a dma-buf
  (but then we'd need the dma-buf mmap stuff to land first).
- damage tracking

If we want fancy, we could allow userspace to reply (through an ioctl)
when it's done reading the previous image, which the kernel could then
forward as vblank complete events.

Connector configuration could be done by forcing the outputs (we'll send
out uevents nowadays for that), so the only thing we need is some configfs
to instantiate new copies of this.

At least for miracst (as opposed to full-blown hw drivers in userspace) I
don't think we need to export everything.
Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch