[PATCH v2 2/3] drm: Add variable refresh property to DRM CRTC

[Kazlauskas, Nicholas]

On 10/10/2018 03:14 AM, Pekka Paalanen wrote:
> On Fri, 5 Oct 2018 12:21:20 -0400
> "Kazlauskas, Nicholas" <nicholas.kazlauskas at amd.com> wrote:
> 
>> On 10/05/2018 04:10 AM, Pekka Paalanen wrote:
>>> Hi,
>>>
>>> I have a somewhat of my own view on what would be involved with VRR,
>>> and I'd like to hear what you think of it. Comments inline.
>>>
>>>
>>> On Tue, 25 Sep 2018 09:51:37 -0400
>>> "Kazlauskas, Nicholas" <nicholas.kazlauskas at amd.com> wrote:
>>>    
>>>> On 09/24/2018 04:26 PM, Ville Syrjälä wrote:
>>>>> On Mon, Sep 24, 2018 at 03:06:02PM -0400, Kazlauskas, Nicholas wrote:
>>>>>> On 09/24/2018 02:38 PM, Ville Syrjälä wrote:
>>>>>>> On Mon, Sep 24, 2018 at 02:15:36PM -0400, Nicholas Kazlauskas wrote:
>>>>>>>> Variable refresh rate algorithms have typically been enabled only
>>>>>>>> when the display is covered by a single source of content.
>>>>>>>>
>>>>>>>> This patch introduces a new default CRTC property that helps
>>>>>>>> hint to the driver when the CRTC composition is suitable for variable
>>>>>>>> refresh rate algorithms. Userspace can set this property dynamically
>>>>>>>> as the composition changes.
>>>>>>>>
>>>>>>>> Whether the variable refresh rate algorithms are active will still
>>>>>>>> depend on the CRTC being suitable and the connector being capable
>>>>>>>> and enabled by the user for variable refresh rate support.
>>>>>>>>
>>>>>>>> It is worth noting that while the property is atomic it isn't filtered
>>>>>>>> from legacy userspace queries. This allows for Xorg userspace drivers
>>>>>>>> to implement support in non-atomic setups.
>>>>>>>>
>>>>>>>> Signed-off-by: Nicholas Kazlauskas <nicholas.kazlauskas at amd.com>
> 
> ...
> 
>>>> Whenever I mentioned variable refresh "features", what I really meant
>>>> was operating in one of two modes:
>>>>
>>>> (1) Letting the driver and hardware adjust refresh automatically based
>>>> on the flip rate on a CRTC from a single application
>>>>
>>>> (2) Setting a fixed frame duration based on the flip rate on a CRTC from
>>>> a single application
>>>
>>> I wonder if that's too much magic in the kernel... what would be wrong
>>> with simply flipping ASAP when VRR is active?
>>>
>>> How will userspace be able to predict coming flip opportunities if the
>>> kernel does so much magic?
>>
>> The kernel driver doesn't need to do much more than let the hardware
>> know the variable refresh range. The "magic" is performed by hardware.
>>
>> Most games would like to render as fast as possible to deliver a more
>> responsive and smoother image to the user. Many of these are also
>> resource intensive and won't always be able to render at the fixed
>> refresh rate of the panel (especially for higher refresh rates like
>> 144Hz). The user will experience stuttering if the game takes too long
>> to render and misses the vblank window for the flip.
>>
>> Dynamic VRR adjustment can resolve this problem. The hardware can lower
>> the refresh rate and increase the vblank window in response to this so
>> the user doesn't experience stuttering (or latency).
>>
>> Userspace shouldn't predict anything.
> 
> ...
> 
>>>>>> The reasoning for the split is because not all content is suitable for
>>>>>> variable refresh. Desktop environments, web browsers, etc only typically
>>>>>> flip when needed - which will result in display flickering.
>>>
>>> Flickering? What do you mean?
>>
>> This is a property of how panels work.
>>
>> The luminance for a panel will vary based on how long the vrefresh is.
>> Since the vrefresh length is changing as part of VRR you're more likely
>> to notice the difference in luminance the bigger the difference is.
>>
>> The difference will be largest when switching from the min vrefresh to
>> the max vrefresh duration.
>>
>> Large differences can occur for applications that render on demand like
>> a web browser (and why you wouldn't want VRR enabled for those). The
>> hardware would continuously wait for a flip that isn't coming. Then if
>> the user moves their cursor or the page updates it's going to happen
>> "randomly" in that window and the hardware will adjust to that.
> 
> Hi Nicholas,
> 
> it seems I have very much mis-guessed what VRR aims to achieve, and the
> effect on luminance sounds horrible.

It really depends on the panel characteristics - like the VRR range and 
panel brightness. Some panels can be particularly unpleasant but others 
are fine.

> 
> People have worked for years to make display timings more explicit,
> giving better control and predictability over them. It sounds like VRR
> is not an improvement that allows new smarter software to take control
> of timings even better. Instead, VRR seems to be a step backwards,
> introducing more uncertainty into the timings. The expectation of a
> fixed unknown refresh rate must be built into software for the software
> to work reasonably while VRR is active. >
>  From your comments I understood that the VRR hardware still very much
> depends on a consistent refresh rate, except the hardware (not the
> software!) can additionally slew the refresh rate over time. Abrupt
> changes in frame timings must still be prevented, but I wasn't quite
> sure if you meant the hardware will do that or if the software must do
> that, since you are worried about on-demand updating applications
> causing flickering.
> 
> Hence, VRR looks like a band-aid for old and simple applications that
> use brute force (more fps, maximize the work load) in an attempt to
> make things smoother and to reduce latency. There are lots of such
> applications, so VRR has its place. It is my mistake of assuming it
> could do more. Yes, I am disappointed to realize this.
> 
> I believe that future display servers and applications that actually
> care about display timings will just opt out from VRR to gain better
> predictability.
> 
> I also believe that you will need to blacklist applications like video
> players whose developers have spent a great deal of effort in making a
> smart scheduling algorithm for fixed rate display. I assume that a
> smart sheduling algorithm that is based on prediciting the next display
> time instant will interact poorly with VRR. A video player will need to
> know if it is getting fixed rate or VRR to choose the appropriate
> timing algorithm - wherther it needs to adapt to the display rate or
> will the display rate adapt to it.
> 
> In summary, VRR is only good for exactly the use cases you listed, and
> for other uses it can be harmful, just like you said.
> 
> Given the above, I think we are now very much on the same page about
> what the KMS UABI should look like.


The use-case for the dynamic adjustment mode is largely games, yes. But 
I don't think that's something to be entirely dismissive of.

The old and simple applications are the ones that would benefit the 
least from this because they probably run fairly well on new hardware.

Modern games benefit the most from this since they have high performance 
requirements. There's also a fair amount of these titles coming out in 
the last few years largely because of the cross-platform deployment 
capabilities in popular game engines and frameworks.

While I don't have actual numbers, I think that most modern games that 
come out on Linux are scalable when it comes to refresh rate. VRR panels 
with a "decent" range will see nearly all stuttering gone in practice 
with no negative impact to the experience at all. This also benefits 
games that render lower than the monitor's refresh all the time, like 
45Hz on a VRR monitor with a 40-60Hz range.

The dynamic adjustment also works as expected with implicit vsync on/off 
via GLX swap control. Lightweight games don't need to burn a ton of 
power if they're capable of rendering above the refresh rate of the 
monitor. Vsync on will still cap you at the mode's refresh as expected 
(like 60Hz).

The patches I've put out target this use case mostly because of the 
benefit for a relatively simple interface. VRR can and has been used in 
more ways that this, however.

An example usecase that differs from this would actually be video 
playback. The monitor's refresh rate likely doesn't align with the video 
content rate. An API that exposes direct control over VRR (like the 
range, refresh duration, presentation timestamp) could allow the 
application to specify the content rate directly to deliver a smoother 
playback experience. For example, if you had a 24fps video and the VRR 
range was 40-60Hz you could target 48Hz via some API here.

> 
> 
>> Most compositors function well under this stack. It will vary
>> depending on compositor support for window unredirection to let the
>> window flip via the Present extension. Mutter handles this without
>> any configuration and kwin can be configured to work with these
>> patches. Compton can be configured to support unredirection as well.
> 
> You are thinking about the applications. What about the compositor
> itself? Is the COW not hitting the Present direct scanout path,
> triggering VRR, when what is actually showing is the desktop with
> on-demand randomly updating windows?
> 
>> These (among others) are covered as part of the blacklist for the
>> mesa patches. They do need to be explicitly excluded.
> 
> Right, you blacklist all compositing managers to prevent them from
> regressing. That feels a bit ugly to me, needing exceptions to not
> regress things, but maybe that's business as usual in Mesa?
> 
> Fortunately that problem with compositing managers won't exist with
> Wayland.

Compositors do bring out the worst when it comes to flickering for VRR 
with the dynamic mode. A more explicit API may help here in the future.

I think the blacklist approach is still the best when compared with the 
alternatives. At least when it comes to supporting the vast majority of 
existing applications without any explicit configuration from the user 
or developers.

I would imagine you're right about a Wayland implementation being 
cleaner when dealing with some of these issues.

> 
> In any case, there must also be a way for an application to explicitly
> say if it supports VRR or not and to know if it gets VRR or not. Are
> there no EGL or Vulkan extensions for that?

I think this is an interesting idea given what's already out there in 
terms of swap and present control. There was some discussion about this 
and the blacklist on the mesa mailing list.

For the current implementation the easiest way to override the blacklist 
is via including an explicit driconf file for the application. A 
semi-recent change in mesa lets applications do this without overriding 
the global or user conf.

> 
> 
> Thanks,
> pq
> 

Nicholas Kazlauskas