[Mesa-dev] KWin and Mesa

[Martin Gräßlin]

Hi Mesa-devs,

yesterday I published a rant about Mesa breaking KWin and given some 
comments on Phoronix Forums it seems like there is the wish for more 
communication between our development groups and so I want to start it. Please 
note that I am not subscribed to this mailing list, so please keep me in CC (I 
might not be able to reply this week at all). It is my wish to never have to 
rant about the state of Linux drivers any more and that I never have to see 
Mesa breaking KWin again.

First of all I want to give a little bit personal history to help you 
understand why I so far did not contact you and up to now wrote two rants 
about Mesa breaking KWin. Let's go back to 2010 and the 4.5 release. In March 
2010 I finished my Master Thesis and in April 2010 I started my first job. KDE 
hat the feature freeze for the 4.5 release around End of April, beginning of 
May. All new functionality including the Blur and Lanczos filter were 
implemented at that time. Given the change in my life due to end of my studies 
I did not contribute much to the release.

At the time when we implemented the features the current Mesa version was 7.7. 
Version 7.8 was under development and when it got released marked as a 
"development version". I read this information and considered "ok we don't 
have to deal with 7.8 - it is development". At that time I had a notebook with 
NVIDIA graphics and an old system with a rather modern Ati card, with a 
crashing X server if I tried to use the radeon driver. I had no chance to test 
Mesa drivers at that time!

In the time leading to the 4.5 release KWin had no active maintainer. Lubos 
had been inactive for quite some time and made me maintainer in November. At 
that time I considered Lubos still to be the maintainer and to be responsible 
for decisions whether to ship the new features or not. I considered myself 
only responsible for my own code (which did not cause any problems in that 
release). I was also running the stable version of KDE (4.4) at that time and 
the development only for testing.

During the beta phase we realized that we had a problem. Mesa 7.8.2 was 
marked as stable (which I did not expect due to the fact that 7.8.0 and 7.8.1 
were unstable) and distributions started to include it. Users were complaining 
about broken features mostly concerning blur and lanczos filters and mostly 
with Ati and Intel drivers. Nobody in our development team had an Intel system 
at that time. I had had access to a system before through a friend of mine, 
but unfortunately it broke down in exactly that important time. Later on my 
friend got a new Intel powered device but run Debian Testing on it which did 
not show any of the problems the users reported with Mesa 7.7. Concerning Ati 
I knew that Fredrik had been in contact with Mesa developers and that all the 
new functionality had been implemented on his Ati systems. So we knew that the 
functionality worked at least with some systems.

With the looming release and more and more obvious problems we faced two 
possible solutions: remove the code completely (disable by default would not 
have solved it) or try to get it working somehow. I did not see any reason why 
we should have punished the users of working drivers (e.g. NVIDIA) because 
other drivers did not work. After all the new functionality was an important 
feature for our provided user experience and our designers and the Plasma team 
were demanding it. So we had only the option to make it work. Now you can 
imagine how difficult it is to workaround bugs in hardware you do not have. My 
solution was to implement a black list and to crowd source the creation to all 
our users. Another of the changes was to have the test whether to use direct 
rendering in an external application (due to drivers crashing when trying to 
create a GLContext) and there the "hack" was introduced which now backfired.

Now why did I not contact you when we were facing these problems? Given that I 
had a day job, 2 h travel each day and trying at the same time to make the 
experience as smooth as possible for our users in the evenings, I seriously 
had not the time to even think about it. And what would it have changed? We 
need to get KDE supporting the drivers out there and not the next version! I 
am sorry that I did not contact you at that time but I think everybody will 
understand that sometimes as a volunteer developer you don't have the time to 
do everything which would help. Last but not least I did not feel responsible 
as I was not the maintainer of KWin. 

When we released 4.5 I was positive that we had successfully established a 
black list which ensures that no user would face issues. Unfortunately I was 
wrong. Users still faced the issue and even worse: the desktop started to 
freeze if you changed settings in KWin. A problem which had been completely 
unknown to us before the release (there was a bug report by one user but it 
was unconfirmed). None of us was able to reproduce because we did not have 
the hardware. At the time when finally a user was able to investigate it, he 
discovered that it was a bug introduced in Mesa 7.8 and not present in the 
development version of Mesa 7.9 - one of the reasons why Maverick shipped a 
development version of Mesa 7.9.

Due to the bugs KWin had hit in Mesa, KDE got a pretty bad Media coverage. 
This was something I did not like. The bugs were not in our software. I have 
no problems with admitting bugs in my part and I take blame for it, but I 
cannot stand being blamed for problems due to 3rd party breakage. So I wrote a 
blog post explaining the situation. Explaining the steps we did to ensure that 
there won't be problems and I did not want to have it as a rant, though it 
seems to have been read as such by some people. I really did not want to piss 
you off, I just had to explain that the bugs are not in KDE, but in a 
different layer of the stack and that to a certain degree we did not even know 
about the bugs prior to the release.

Now let's fast forward to last week and why I again wrote a rant (and this 
time it was intended as a rant) without contacting you. Again it is quite 
human: I was informed about the change which broke KWin last week. Last week 
I was unfortunately ill (I am still fighting the last parts of my cold) and 
nevertheless went to work and was in the evenings so exhausted that I most of 
the days just dropped to bed. On Saturday I flew to New York for a business 
trip and just now I am spending my evening sitting in a hotel room writing 
this mail. Seriously I just did not have the time to contact you.

So why a rant? Seriously: I am disappointed. I am very, very sad about what 
happened here. After all the buzz about my blog post more than half a year ago 
it happened again that Mesa developers broke our software. I think I cannot 
describe in words how I feel about it. Now the situation has changed and I am 
maintainer of KWin and I do feel responsible for the complete experience. I 
hope this somewhat explains why I did not contact you prior to writing the 
blog post. It is important to know that I also wanted to get the right people 
to read it: that is the Kubuntu devs to find a solution ASAP to not have a 
broken release. Kubuntu is one of our most important downstreams and I do 
care that they have a great release. Given the combination of illness and 
business trip I had not seen any other way than to write a blog post. Sorry 
that I put my frustration into it. This is unprofessional, but also part of my 
character. You know we Germans are direct and I speak the truth even if it 
hurts. Now feel free to shoot the messenger of bad news, but sorry it is the 
truth that Mesa just broke KWin.

So this is already rather long and should be enough for the history part. 
Let's now look on how to improve the situation. Therefore I try to explain how 
I see this. Mesa is our most important upstream and KWin (together with Compiz 
and Mutter) are your most important downstreams. I am sometimes not sure that 
you are aware of this. KWin is probably the OpenGL compositor with the largest 
user base or the second largest after Compiz. You should be aware that there 
is nothing worse than a broken compositor. It doesn't matter if you now 
support game foo or game bar if the compositor breaks. Please keep this in 
mind. Given how important KWin is for you as a downstream I am really 
surprised to see that you do not do regression tests against KWin. What can we 
do to help you there?

Please understand that KWin has only three regular developers. We are all 
volunteers and are doing development in our spare time. Please understand that 
I personally do neither have the time nor the patience to follow the upstream 
development. In a perfect world I would do KWin development as my day job and 
would have the time to follow the upstream development. In reality I have to 
choose to spend my time on developing KWin or follow upstream development - I 
don't have time for both. Please also understand that I am not able to test 
your drivers. On my notebook I was using nouveau for some time, but reducing 
the battery life by more than an hour and not being able to suspend is a high 
price for running free drivers. On my primary system I tried to use the radeon 
driver. I even tried to compile it from sources. After not being able to 
succeed after spending half a weekend I am waiting for new drivers to hit 
Debian testing. Please understand that I am not willing to switch the 
distribution just to get newer drivers. This is my main work system and for 
development I want to use a distribution I know and trust. Yes Kubuntu might 
be an obvious choice but I am unhappy with parts of the underlying Ubuntu 
stack. On my third system which actually has an Intel GPU I am not able to 
install development drivers as it is a company notebook. I am glad that I can 
use the latest version of OpenSUSE so that I have a decent combination of KDE 
and Mesa.

Anyway even if I were able to follow the upstream development it would not 
help much. The combinations of hardware, driver, mesa, kernel and distribution 
are way too high that our team can ensure that it does not break. This is 
something only you can ensure. I don't know if you are aware of it, but our 
latest version of KWin is currently used by users from Mesa 7.7 to Mesa 7.11. 
Our users are hitting bugs you fixed more than a year ago and we cannot just 
ignore our users and tell them to compile the latest Mesa version. Yes there 
are hacks in KWin - one of the hacks backfired last week, but this is the 
reason why we have hacks. We need to support yesterdays, todays and 
tomorrows Mesa versions. If we needed a hack to workaround an issue in an old 
version or KWin would crash, nobody is helped if we contact you and you tell 
us that the bug is fixed. Please understand that our users have the drivers 
the distributions give them, it is not under our control.

We have developed a very complex system to ensure that our users get the best 
possible experience. Due to the problems we had with 4.5 and Mesa announcing 
support for features not supported by the chip (consider emulating GLSL in 
software - great for games deadly for compositors) Fredrik developed an 
elaborated system to detect what chipset is installed in the system and 
enabling features based on what the chipset really supports instead of 
listening to what Mesa tells us. As far as I understood you recently changed 
the way how the renderer and version string looks like. This means that you 
completely broke that system. After all that had been said about the 4.5 
release I was sure that you know what KWin reads from your announced strings. 
Now it might be that we should not parse this information and of course you 
are right with saying so, but you know what Linus would answer? "Never break 
your userspace!" This is what just is happening. Now we are lucky and there is 
time to adjust our code base till our next major release, but again we need to 
at least support both Mesa 7.10 and 7.11 in one code base. Kubuntu and 
OpenSUSE users (both using Mesa 7.10) will upgrade to 4.7 as soon as it will 
be released and Kubuntu 11.10 will be 4.7 with Mesa 7.10. I think you get the 
problem we are in. We cannot just remove the detection code due to the 
experience we had with 4.5 and due to the fact that we want to give the users 
a good experience. If the hardware does not support shaders the CPU is most 
likely not powerful enough to do useful emulation. Assumptions which are 
correct for a game are completely wrong for a compositor. The compositor may 
not cause high CPU usage, the compositor may not block the system. It is a 
background service which just has to work and should never be visible to the 
user. If the user sees KWin in top something is badly wrong. So we have no 
other choice than to parse the renderer information as we cannot rely on the 
extension announced. The ideal world is one thing, but the reality looks 
different.

I hope you better understand now my position and I hope we can both work on 
better collaboration in future. I still have the plan to submit our complete 
compositor and all effects as a set of piglit tests but we still need some 
refactoring and I don't know whether it is possible to take 60 kSLOC C++ code 
as a piglit test. Please try to get your prioritize right. It is great if the 
wine developers have the time and manpower to work together with you, but 
please also think about that others might not have the time. Please consider 
if it is more important to keep compatibility for an important downstream than 
to support some more games. Think about what you can do to ensure that I never 
have to notice about important changes in Upstream Mesa from upset 
bugreports. The only thing what we need is stability: the drivers should not 
break KWin. That's all I'm asking for.

Let's work together on a better composited experience on X11.

Cheers
Martin Gräßlin
KWin Maintainer