[Bug 101739] An issue with alpha-to-coverage handling is causing Arma 3 64-bit Linux port to render trees incorrectly

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=101739

            Bug ID: 101739
           Summary: An issue with alpha-to-coverage handling is causing
                    Arma 3 64-bit Linux port to render trees incorrectly
           Product: Mesa
           Version: git
          Hardware: Other
                OS: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: Drivers/Gallium/radeonsi
          Assignee: dri-devel at lists.freedesktop.org
          Reporter: krystian.galaj at vpltd.com
        QA Contact: dri-devel at lists.freedesktop.org

The game is sometimes rendering trees. It has multiple tree mesh models, and
when the tree gets close, the game is switching between different LODs/models
by rendering both models overlapping on a grid of pixels, as if checkerboard,
pixels of old LOD in black fields of it, new one in white.

As long as it’s doing that using single-sampled buffers as render targets, all
is fine. However in higher quality modes, the game switches to multisampled
buffers. When using multisampled buffers, if ATOC is turned on in settings, the
game is using alpha-to-coverage technique without polygon sorting to make sure
that the grass is rendered correctly. At the same time it’s using depth test
and depth write to fill up the depth buffer. In the same execution of fragment
shader it sets the output color and alpha value, and the depth buffer value,
and it expects the alpha value to cause the color value to end up only in some
samples of multisampled texture, and - the important bit - the depth value to
end up only in the corresponding samples of the depth buffer.

This technique works on DX11 on Windows, on all drivers on Mac OS X, and on
NVidia drivers on Linux, but on Mesa Radeonsi drivers on Linux it makes the
rendering go bad, rendering most of the tree pixels white until the LOD
transition ends.

The white is visible on the screen because it is the initial color of the
multisampled render target, and for some reason this color is allowed to leak
through, which means that in some cases the depth buffer value is set, when the
corresponding color buffer value isn’t.

Both depth and color buffer have the same number of samples (8), and were
created with fixedSamples = true in OpenGL call.

It looks as if the depth buffer values in case of Mesa Radeonsi driver were:
 - correctly not written when the depth test fails for the fragment shader,
 - correctly written to all samples of the depth buffer when alpha coverage
directs the draw to fill them all, but
 - INCORRECTLY assigned not only to those samples to which alpha coverage
directed the fragment output color, but either to all the samples in depth
buffer, or to samples in depth buffer that don’t correspond to the samples in
color attachment buffer.

We encountered the same issue in 2015 in fglrx drivers, contacted AMD team
about it, and received confirmation that it is a bug, and that it was fixed.
Unfortunately, shortly after that fglrx drivers went out of use.

The issue can be easily reproduced in Arma 3 by:
 - launching Arma 3 game,
 - switching to High quality or higher to turn on multisampled buffers,
 - enabling ATOC (alpha to coverage) in Video settings, or making sure it's
enabled,
 - launching first level of Drawdown 2035 campaign, ie. starting a new
campaign, bypassing optional tutorial, and observing as the main character
walks into the base, the bushes flicker white from time to time as they get
closer.

The issue is present in Mesa 17.2.0-devel from padoka PPA, on at least Radeon
R7 260X/360.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20170710/b412504e/attachment.html>