Best practices for client side buffer management

[Carsten Haitzler (The Rasterman)]

On Wed, 24 Jun 2020 19:17:57 +1000 Brad Robinson <brobinson at toptensoftware.com>
said:

EFL has the same model - it tracks damage rects (either sent by expose/damage
events from windowing system combined with a set of rect regions it calculates
itself based on previous frame state vs current frame state and which
exposed/visible regions needed updating and it can do "just re-render a
blinking cursor" region without a full buffer render. you just needs to handle
a set of buffers and partial updates properly. it can also be done with opengl
(using the  buffer age extension and also hinting to the compositor/display
system which regions changed with swap with damage regions too, but it's the
same idea as with software rendering so i'll just cover software).

so you keep a set of buffers. A, B, C. at a minimum A and B, but ABC is the
best.

you fully render A, then display it. you then render all of B and display, then
all of C and display. you now have 3 fully rendered buffers. you will be in a
steady state like this through 99.999% of the life of a window (unless it
resizes where we begin with a new set of buffers, or we do fun things like
drop all buffers but the visible one to compact memory usage).

from now on, let us call the set of rectangles that define the regions rendered
as Ar, Br and Cr per buffer. every time we need to render a frame we are doing
frames in order ABCABCABCABCA... etc.

that means we have the calculated "these areas need updating" ArBrCrArBrCr...
etc. ..

so those are the regions you actually calculate to ne re-rendered. it's the
minimum update area to cover changes. to do this right though you have to
actually over-render. so let us call the set of rects that you REALLY render
and tell the compositor you updated as u. so we have AuBuCuAuBuCu... let us say
+ is the UNION of 2 rect regions.

current u == current r + previous r + previous - 1 r.

so: Cu = Cr + Br + Ar
then next frame is A again: Au = Ar + Cr + Br
then: Br = Br + Ar + Cr

etc....

you keep a sliding window of the last 2 frames with of rect regions you union
(merge with) the current frame's update rects... then render that. you can play
tricks like copy back some regions from a previous frame instead of
re-rendering them (as it's a read from, not a write to that buffer it's safe..
but beware that readbacks may have issues especially if regions are not mapped
as non-cachable because they are being scanned out/displayed for example. it
depends on hardware entirely so the safe thing is to their shadow them and make
a copy before you display or just re-render them. it turns out the just
re-render is simpler to do and generally performant).

so basically you need to start thinking of a ring of buffers and managing
those and tracking updates over multiple frames... then you can happily do
fairly minimal "only update this region" drawing. i've been doing this for
many years .. even in X11. it's not a wayland specific thing. you will HAVE
to do this if you want to do partial updates with acceleration like with
opengl :) it works...

this all removes the need for copies and allows zero-copy minimum region
partial updates. :)

p.s.  reason for 3 buffers is to allow for the compositor having not released
the current front buffer yet. compositor can hold onto its buffer. you can have
already sent/submitted a new buffer that's waiting on the compositor to notice
and pick up, and in the meantime you still have a 3rd buffer you can prepare
another frame on while the one in flight arrives and the compositor releases
what it has.

you will need to come up with solutions for if you render this, submit it then
the compositor is still hanging onto its buffer now with 2 in flight. at this
point you can block or "defer" the frame until the 3rd buffer is released, or
perhaps just don't submit a new buffer until the first buffer was released.
then you can just render AGAIN to the 3rd buffer that you have not sent yet,
updating it and now you only render the partial update regions on this one
since the last frame... keep going until compositor releases the first buffer.

i prefer the first method of "defer the rendering and don't render anything
until we get the in-flight buffer to the compositor AND it releases the current
one as this avoids getting too far ahead of the compositor and avoids rendering
possibly useless frames that will never be displayed because the compositor is
having a hiccup and blocking for a few hundred ms - on a heavily loaded system
either cpu or gpu wise its better to back off and not just render more frames
and give things space to get past the spike of work. using the frame requests
from the compositor is a way of this all working out well as compositor calls
you saying "i need a new frame now" and this serves as a kind of throttling in
these cases anyway.

> Hi All,
> 
> @Guillermo: yep, that's exactly the same problem I'm thinking about.
> 
> I had an idea that I'm wondering about....  the rendering model in my
> toolkit is essentially the same as Windows/OSX - where the application
> invalidates part of the window, and then at some later point the OS calls
> back with a message prompting the app to paint that part of the screen.
> ie: multiple invalidate calls are coalesced into a single paint call at a
> later time - typically just before entering the message loop's idle state.
> 
> Could a Wayland app mimic this behaviour by having a single background
> buffer that contains the current window content?  When the application
> invalidates the window it does one of two things depending on whether the
> buffer has been returned from the compositor yet.
> 
> * If the buffer has been returned and no longer in use by the compositor,
> it queues a paint message to itself.
> 
> * If the buffer hasn't been returned and is still in use by the compositor,
> set a flag on the window.  When the buffer is returned from the compositor,
> it checks that flag and if set then it clears the flag and queues the paint
> message.
> 
> The paint message redraws all the invalidated (damaged) rectangles and
> re-submits it to the compositor and the whole cycle starts again.
> 
> Obviously there's more logic around maintaining the dirty region,
> coalescing multiple invalidate/damage calls, an assumption that there's a
> mechanism to post a message via the message loop etc... but I hope this
> explains the idea.
> 
> Would this work?  I think the main requirement would be that:
> 
> 1. the compositor doesn't change the contents of the buffer and that when
> it's returned it's still got the old content.
> 2. that the compositor returns buffers it's finished with in a timely manner
> 
> It's not clear to me from what I've read that either of these points are
> true or safe assumptions.
> 
> This would eliminate the need to manage multiple buffers, multiple dirty
> regions, the need to copy previously rendered content between buffers and
> fits nicely with the rendering model of Windows/OSX giving similar
> semantics on all platforms.
> 
> Brad
> 
> On Wed, Jun 24, 2020 at 6:11 PM Guillermo Rodriguez Garcia <
> guille.rodriguez at gmail.com> wrote:
> 
> > Hi Brad,
> >
> > El vie., 19 jun. 2020 a las 5:24, Brad Robinson
> > (<brobinson at toptensoftware.com>) escribió:
> > [...]
> > >
> > > Finally, the toolkit already maintains an off-screen buffer with the
> > window's current contents rendered into it.  I'll probably replace that
> > with a Wayland buffer, but wondering about partial updates.  eg: if the
> > client only needs to redraw a part of the window what's the correct process
> > to update just that part with Wayland.  Can I just update the existing
> > buffer and prompt Wayland to just redraw that part?
> >
> > I am facing a similar situation with a toolkit that maintains an
> > off-screen buffer with the current window contents, and will update
> > that buffer at arbitrary times, when the application needs to update
> > the UI.
> >
> > You must consider the following:
> >
> > Once you submit a buffer to Wayland you cannot touch it anymore until
> > the compositor releases it back (wl_buffer.release callback). From
> > Pekka's email:
> >
> > > When you submit a buffer to a Wayland compositor, it gives the
> > > compositor permission to read the buffer at any time, as many times as
> > > it wants to, until it tells you with wl_buffer.release that it will not
> > > be looking at that buffer again. You must not write to a buffer that
> > > might be read by the compositor, as that can cause misrendering on
> > > screen (e.g. your repaint is shown unfinished).
> >
> > This means you cannot simply use the same off-screen buffer that the
> > toolkit maintains, since the toolkit should not update the buffer once
> > it submits it to Wayland.
> >
> > At some point I thought that I could have separate Wayland buffers and
> > copy damaged regions from the toolkit buffer to one of these Wayland
> > buffers before submitting it to the compositor.
> > The idea would be to copy the set of regions that were damaged since
> > the last update.
> >
> > But just copying damaged regions is not enough either. From Pekka's email:
> >
> > > Also, every buffer you submit must be fully drawn, also outside of the
> > > areas you mark as damage. The compositor may be reading more than just
> > > the damage you submit.
> >
> > So this means that everytime the toolkit updates the off-screen
> > buffer, I may need to copy _the entire thing_ to a Wayland buffer and
> > submit it to the compositor... not very efficient I guess.
> >
> > I have the feeling that Wayland is designed for a model where the
> > client renders the entire frame on demand (most of the examples I've
> > found so far work like this) and not for a model where the client is
> > updating a buffer asynchronously and you just want to tell the
> > compositor to update parts of the frame.
> >
> > I would be very interested to learn about your solution to this problem.
> >
> > Best,
> >
> > Guillermo Rodriguez Garcia
> > guille.rodriguez at gmail.com
> >


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - raster at rasterman.com