[RFC] remoting KMS

[Daniel Vetter]

On Thu, May 17, 2018 at 11:18 PM, Thomas Hellstrom <thomas at shipmail.org> wrote:
>
> On 05/17/2018 09:20 PM, Daniel Vetter wrote:
>>
>> On Thu, May 17, 2018 at 8:23 PM, Thomas Hellstrom <thomas at shipmail.org>
>> wrote:
>>>
>>> Hi!
>>>
>>> I'm currently working on a remoting KMS backend, and now I thought it
>>> would
>>> be a good time to get some feedback on a very rough design:
>>>
>>> The idea is that we want to be get enough information on the backend side
>>> of
>>> KMS to be able to remote the display system over VNC or something
>>> similar.
>>> Initially I'd plug in an open-source VNC server as the "reference"
>>> user-space.
>>>
>>> The solution should be general enough to plug into any driver, including
>>> VKMS, either mirroring an existing crtc/connector or adding a virtual
>>> one.
>>> The driver would typically only have to implement an efficient readback
>>> command, optionally with a diff-map or damage rect that tracks changed
>>> content for the VNC server to be able to do its work efficiently.
>>>
>>> This would all be forwarded to a user-space interface and protocol with a
>>> slightly more elaborate user-space event-mechanism than the one we have
>>> in
>>> drm now. In particular we want to be able to read partial events, and
>>> resume
>>> event reading after a partial event. We would also want to be able to
>>> send
>>> commands either through an ioctl() or using write(). Current test code is
>>> using write().
>>>
>>> Since the VNC server in our application typically will be system-wide, we
>>> need to resurrect a control-node functionality. I was trying to use sysfs
>>> attributes for this, but while sysfs supports rudimentary poll
>>> functionality. It's a bit too rudimentary and tracking client lifetime
>>> becomes racy. Perhaps control nodes renamed to "remoting" nodes or
>>> something
>>> similar. It might be that masters want to attach a VNC server to their
>>> current display and in that case we probably would want to expand primary
>>> nodes with this functionality as well.
>>>
>>> Now to the open-source  problem. As mentioned, the reference user-space
>>> will
>>> be a solution based on libvncserver. In fact there will be a small
>>> user-space interface library that libvncserver can interact with, similar
>>> to
>>> libdrm. However, I envision in the long run VMWare would be interested to
>>> use other remoting protocols with closed source protocol encoders, using
>>> the
>>> open source reference library. In theory, this solution would also allow
>>> for
>>> simple (open- or closed source) user-space display drivers, very similar
>>> to
>>> DisplayLink's evdi solution, even if that's not a use-case VMWare is
>>> currently interested in.
>>>
>>> Thoughts and comments appreciated.
>>
>> Just two very quick comments:
>>
>> - For the uapi - can't we just use v4l for the output side? That gives
>> you buffer handoff, syncing (v4l is gaining dma_fence support) and
>> pretty much everything you might want. And we've talked about building
>> kms->v4l pipelines in a bunch of other use-cases already (like feeding
>> into video encoders for widi and stuff like that). Would give you the
>> userspace for free, only thing we really need is to (maybe, not sure
>> you'd even need that) expose what kms output corresponds to what v4l
>> pipeline.
>
>
> Hi!
> Thanks for the comments.
>
> I'll take a look whether this is possible to accomplish with v4l. Hadn't
> thought of that. However, I doubt it will be possible to support all things
> we need to support. Initially we started looking at this in the context of
> readback connectors but buffer management and readbacks are only a subset of
> what we want to do. We also need to pipe a subset of the (atomic) state and
> its updates, the handling of which actually becomes the vast major part of
> the code.
> We also need to feed data back, similar to the vmwgfx update_layout ioctl,
> that sets suggested x and y position, preferred mode, and, on readback,
> obtain damage information. Perhaps it's possible to construct some kind of
> side-channel.

Ok, that's a slightly different use-case than what I thought off,
sounds like you want to keep the special cursor meaning for
virtualization. And that wouldn't go through v4l I think.

>> - You say "The solution should be general enough to plug into any
>> driver" - is the idea that any hw driver would automatically support
>> this?
>
> No, the initial idea is that any hw driver that wants to could use helpers
> and a readback implementation to support this

Is the helper idea just what you have in mind for vmwgfx, or does your
use-case require that you patch this to all kms drivers (kinda like
all kms drivers have fbcon)? +/- a bunch you'll probably not care
about. If you want this I guess you need to explain a bit more what's
the intended use-case, since it seems to be more than just piping any
random compositor than can run on kms into something like a vnc server
for forwarding.

>> Given that prime-like setups are getting ever more common
>> (mobile is essentially all prime with display and offloaded rendering)
>> I think it'd be cleanest if we don't tie that virtual output to other
>> drivers, but bind things together using buffer sharing/dma-fences and
>> prime.
>
>
> We do need a front end, though, I assume you're suggestion is that we add
> support to vkms only. That would work for the main VMware use-case, but not
> for a general use-case where a user would want to share his existing screen
> using VNC.

Fire up vkms, set up as a cloning output? Of course that needs minimal
cooperation of the compositor, so depending upon what you're trying to
do it's not good enough.
-Daniel

>>   Adding v4l output to vkms is actually something we've already
>> discussed a bit on irc, but with the use-case that you could just
>> watch what your desktop/window manager is doing under a specific
>> simulated setup (e.g. pll sharing constraints, or special plane
>> features, or whatever other specific thing you want to test your
>> compositor against but isn't supported on all physical hw/drivers).
>>
>> Cheers, Daniel
>
>
> Thanks,
>
> Thomas
>
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch