[Mesa-users] [Mesa-dev] GraphicsFuzz metamorphic testing of shader compilers

Thu Mar 1 06:55:01 UTC 2018

Hi;

Some feedback below ..

On 02/28/2018 02:21 PM, Hugues Evrard wrote:
> Hi Brian, all,
> 
> Thanks for the follow-up, and congrats to the team for the prompt bug 
> fix! We've just tweeted about it:
> https://twitter.com/GraphicsFuzz/status/968815766681333760
> 
> We'll be pleased to report further Mesa issues as we find them, yet as a 
> three-people team we unfortunately do not have the human resources to do 
> proper testing of Mesa on various platforms right now.

It would be good for this test-suite to provide more meaningful report 
of the results. When a test fails, it could list what kind of shaders 
were given to the browser (before browser goes and mangles them before 
giving them to driver). This would help debugging individual failures.

> Meanwhile, our public demo gives a nice excerpt of our test suite, and 
> is easily ran via a web browser, so don't hesitate to try it on your 
> devices:
> http://www.graphicsfuzz.com/#demo
> 
> Please let us know if it triggers other issues in Mesa!
> 
> Many thanks,
> Hugues
> 
> 
> On 27 February 2018 at 23:40, Brian Paul <brianp at vmware.com 
> <mailto:brianp at vmware.com>> wrote:
> 
>     On 02/27/2018 10:27 AM, Hugues Evrard wrote:
> 
>         Hi all,
> 
>         I have just reported a Mesa (i965) crash which was triggered by
>         a shader from the GraphicsFuzz demo (bug ID 105271), and I
>         wanted to give a broader context on that bug report.
> 
>         We are three academics (Alastair, Paul and myself) from Imperial
>         College London who work on metamorphic testing of shader
>         compilers, last year we reported drivers bugs across all major
>         GPU vendors and wrote some blog posts about this
>         (https://medium.com/@afd_icl/689d15ce922b
>         <https://medium.com/@afd_icl/689d15ce922b>
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40afd-5Ficl_689d15ce922b&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=cSYEtIjkXjH08DSPfDfe8muZ-dBkGxI-hJIFGGujT9I&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40afd-5Ficl_689d15ce922b&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=cSYEtIjkXjH08DSPfDfe8muZ-dBkGxI-hJIFGGujT9I&e=>>).
>         We also had the chance to visit some driver developers,
>         including the Intel Mesa team in Portland -- thanks again for
>         hosting us!
> 
>         After months of further development and tedious paperwork, we
>         are now spinning GraphicsFuzz out of academia with the aim to
>         raise graphics drivers reliability across the board. Our first
>         effort focuses on the mobile landscape, you can see wrong images
>         and crashes due to graphics driver bugs in the Samsung S8s,
>         Nvidia Shields, Google NexusTV and Pixels, Huawei Honors and
>         Apple iPhones here (more to come!):
>         http://www.graphicsfuzz.com/#results
>         <http://www.graphicsfuzz.com/#results>
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_-23results&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=j3eMhe_uZIVNtxkBTwgU0xcl6vvyPHN9lbg6qa3wihw&e=
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_-23results&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=j3eMhe_uZIVNtxkBTwgU0xcl6vvyPHN9lbg6qa3wihw&e=>>
> 
>         On the technical side, a summary of our testing approach is
>         here: http://www.graphicsfuzz.com/howitworks.html
>         <http://www.graphicsfuzz.com/howitworks.html>
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_howitworks.html&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=tQS-TpnGyPrWxUzZZFY4UE-Q1Ta0QkcptHGtzjSx3uU&e=
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_howitworks.html&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=tQS-TpnGyPrWxUzZZFY4UE-Q1Ta0QkcptHGtzjSx3uU&e=>>
> 
>         We are looking forward to cover the Mesa drivers, but not
>         immediately given our current focus on mobile devices.
>         Meanwhile, anyone can easily try our demo, which executes 15 of
>         our test shaders, on any WebGL2-capable web browser. Today's bug
>         report comes from this demo, which crashes i956 (Mesa 17.3.3) on
>         my Intel HD 520:
>         http://www.graphicsfuzz.com/#demo
>         <http://www.graphicsfuzz.com/#demo>
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_-23demo&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=qWeYO3iPLlVvvHP2MJIQSnvHh7BTqxQBDNd4Uu_Q0NY&e=
>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.graphicsfuzz.com_-23demo&d=DwMFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=Ie7_encNUsqxbSRbqbNgofw0ITcfE8JKfaUjIQhncGA&m=IXuO7p4yYdvVl_wrG-_cBRZTgrbYT-aP0C5xQNl3hlU&s=qWeYO3iPLlVvvHP2MJIQSnvHh7BTqxQBDNd4Uu_Q0NY&e=>>
> 
>         Please let me know if the demo crashes other Mesa setups, we
>         will be happy to help providing good reproduction examples!
> 
>         Many thanks, and if you have questions or feedback, we are all
>         ears :)
>         Hugues
> 
> 
>     Hi Hughes,
> 
>     I've been followed your work since I came across it a few months
>     ago. It's a great project.  I look forward to finding/fixing any
>     issues in Mesa.
> 
>     I'm cc'ing your message to the mesa-dev list.  That's probably the
>     better place to get the attention of the i965 driver developers. 
>     Also, you should probably file a Mesa bug for any issues you find.
> 
>     -Brian
> 
>     PS: Sorry about mangling your URLs above.  It's a VMware mail thing. :-(
> 
> 
> 
> 
> 
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev
> 