Introduction and updates from NVIDIA

[Daniel Vetter]

On Mon, May 16, 2016 at 11:12:35AM -0700, James Jones wrote:
> On 05/16/2016 02:36 AM, Daniel Vetter wrote:
> >On Sat, May 14, 2016 at 05:46:51PM +0100, Daniel Stone wrote:
> >>On 12 May 2016 at 00:08, James Jones <jajones at nvidia.com> wrote:
> >>>The EGLStream encapsulation takes into consideration the new use cases
> >>>EGLImage, GBM, etc. were intended to address, and restores what I believe to
> >>>be the minimal amount of the traditional GL+GLX/EGL/etc. model, while still
> >>>allowing as much of the flexibility of the "a bunch of buffers" mental model
> >>>as possible.  We can re-invent that with GBM API adjustments, a set of
> >>>restrictions on how the buffers it allocates can be used, and another layer
> >>>of metadata being pumped into drivers on top of that, but I suspect we'd
> >>>wind up with something that looks very similar to streams.
> >>
> >>The only allocation GBM does is for buffers produced by the compositor
> >>and used for scanout, so in this regard it's quite straightforward.
> >>Client buffers are a separate topic, and I don't buy that the
> >>non-Streams model precludes things like render compression. In fact,
> >>Ben Widawsky, Dan Vetter, and some others are as we speak working on
> >>support for render compression within both Wayland EGL and GBM itself
> >>(for direct scanout from compressed buffers with an auxiliary plane).
> >>So far, the only external impact has been a very small extension to
> >>the GBM API to allow use of multiple planes and FB modifiers: a far
> >>smaller change than implementing the whole of Streams and all its
> >>future extensions (Switch et al).
> >
> >Just a quick correction: For render compression we also do need some
> >allocation hinting interface, since on intel gpus you can't always scan
> >out render compressed buffers. So exactly what EGLstreams tries to also
> >solve (at least if my understanding is correct). So we need a bit more in
> >gbm than just be able to pass fb modifiers around.
> 
> Yes, this, and it goes beyond just hinting at allocation time for us if you
> intend to reconfigure the output without reallocating the surface (E.g.,
> switch to a different plane, start rotating the output, etc.).
> 
> >I still think it's the better approach though since it's still fairly
> >incremental. And exposing the allocation hints and making them explicitly
> >will avoid the need to teach everything in the world about EGLstreams (vk,
> >v4l, drm, ...). Which as Daniel Stone pointed out, doesn't really work
> >well if you have IP blocks from multiple vendors on your SoC.
> >-Daniel
> 
> Yeah, IP blocks from multiple vendors are hard.  I don't see how they're any
> harder with streams though Vs. the alternate GBM-based proposals that have
> been suggested thus far.  We're not entirely immune to this at NVIDIA.
> Sometimes we want to present to an Intel display engine, for example.  An
> EGL-based solution doesn't necessarily mean a single vendor's EGL driver
> (GLVND is coming, slowly), and even if it does, it only requires explicit
> cooperation if both vendors share some more optimal layout than basic
> pitch-linear with minimal alignment requirements and whatnot, no
> compression, either fully-coherent caches or no caching.
> 
> However, there are two ways to solve this:
> 
> -Always resort to the lowest common denominator when the producer/consumer
> aren't from the same vendor, as mentioned above.
> 
> -Have some sort of coordination, either handled by the application and a
> bunch of capability bits, or handled by a driver<->driver API below the
> level of the application API.
> 
> Neither of these seem specific to either a streams-based or EGL-based
> solution to me.  The important part is to standardize the interfaces exposed
> to applications or drivers to coordinate the right formats.
> 
> As to needing to teach everything about EGLStreams, I think there's a
> misconception that this means every component vendor needs to get on the EGL
> bandwagon and start writing a bunch of no-op eglGetConfig() entry points and
> whatnot.  Even with all our in-house IP, that's not the case at NVIDIA.  Our
> media codecs aren't baked into the same driver module as our OpenGL drivers
> for example, and the drivers and engineers maintaining them know very little
> about eachother.  Our EGL driver allows stream producers/consumers to plug
> into it using some internal-standard interfaces and a relatively minimal
> amount of code, and without even including any Khronos EGL headers.

I think this is the crux here - we need to standardize those hints/minimal
set of shared metadata, in a public, cross-vendor interface. Since
obviously without that it doesn't even work for you in your one-vendor
case.

And as soon as we have that standard, I don't really see the benefit of
EGLstreams any more. At least my understanding is that the entire point of
EGLstreams is to hide these hints and metadata and allow them to be vendor
specific.

The other argument for EGLstreams seems to be that once you use EGLstreams
everywhere, there's less wheel reinventing or well protocol re-typing
going on. But right now we (at least the open source community) are in a
world where EGLstreams doesn't exist, so we need to do all that typing
anyway. And typing the protocol itself has the advantage that you can more
easily fit it into whatever's there already. E.g. for
xfree86-video-modesetting we want to add any metadata we standardize to
DRI3 (probably, haven't looked at the details). On wayland we can just add
a new protocol object and reuse all the fancy stuff wayland provides, but
put that proto within libEGL (or gbm) since that fits more with how wl
works right now. And for Android we can do an Android (again haven't
looked into details there). So afaics more flexibility, with roughly the
same amount of work.

Or is there something else I'm not seeing?

> The current Khronos EGL API doesn't need to be the only interface through
> which drivers plug in to a libEGL or vendor EGL implementation.  The
> proposal to expose a vendor-agnostic set of hooks to allow writing EGL
> platform implementations without EGL vendor involvement is one example of a
> non-application facing EGL API.  EGLStream producer and consumer hooks could
> be handled with another non-application facing API.
> 
> > As Kristian says, I really don't see where the existing non-Streams
> > solutions, being GBM on the compositor side and private frame-based
> > protocols between compositor and client, leave you unable to reach
> > full performance potential. Do you have any concrete usecases that you
> > can point to in as much detail as possible, outlining exactly how the
> > GBM/private-Wayland-protocol model forces you to compromise
> > performance?
> 
> Unfortunately, the only realistic way to get to the full patchset is
> incrementally.  We haven't even finished the EGLSwitch extension, let alone
> writing Weston code to use it.  This is why I believe temporary co-existance
> of the two paths is a reasonable path for now.  Not all the benefits of
> streams are demonstrable yet, nor is GBM in its final form.
> 
> Daniel Stone, I'd like to hear more about how you envision a GBM library
> communicating with an EGL producer in a remote process.  Would GBM be
> sending wayland protocol directly?  If so, this is really starting to sound
> like streams-rewritten-using-wayland-protocol, and I don't think wayland is
> the right domain to solve these non-wayland-specific issues in.  If, on the
> other hand, GBM is going to gain its own set of per-vendor cross-process
> communication mechanisms, that really sounds like a re-invention of
> EGLStreams.
> 
> Perhaps both of my assumed solutions above are way off the mark, and it does
> seem like we're talking past eachother at times.  It seems like you have a
> pretty strong understanding of how this would all work in GBM, even if it's
> not there in the code yet.  I understand you're quite busy, but perhaps we
> could have a brief real-time communication session (IRC?  Phone?) where we
> can talk through some of your ideas for GBM, so we can at least start from
> the same basic understanding when talking about this stuff.  Let me know if
> you want to schedule something.

Yeah, I think we're at the point where code is probably much easier to
understand. I'm trying to type up the gbm based idea in hopefully the near
feature. That hopefully grounds the discussion a lot more.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch