[Mesa-dev] [RFC PATCH 00/16] A new IR for Mesa

[Christian König]

Am 26.08.2014 um 18:00 schrieb Jose Fonseca:
> On 26/08/14 10:04, Christian König wrote:
>> Am 26.08.2014 um 03:54 schrieb Rob Clark:
>>> On Wed, Aug 20, 2014 at 2:13 PM, Kenneth Graunke
>>> <kenneth at whitecape.org> wrote:
>>>>>> we've already had problems where distros refused to ship newer Mesa
>>>>>> releases because radeon depended on a version of LLVM newer than the
>>>>>> one they were shipping, [...]
>>>>> That's news to me, can you be more specific?
>>>>>
>>>>> That sounds like basically a distro issue though, since different 
>>>>> LLVM
>>>>> versions can be installed in parallel (and the one used by default
>>>>> doesn't have to be the newest one). And it even works if another
>>>>> part of
>>>>> the same process uses a different version of LLVM.
>>>> Yes, one can argue that it's a distribution issue - but it's an
>>>> extremely painful problem for distributions.
>>>>
>>>> For example, Debian was stuck on Mesa 9.2.2 for 4 months (2013-12-08
>>>> to 2014-03-22), and I was told this was because of LLVM versioning
>>>> changes in the other drivers (primarily radeon, I believe, but
>>>> probably also llvmpipe).
>>>>
>>>> Mesa 9.2.2 hung the GPU every 5-10 minutes on Sandybridge, and we
>>>> fixed that in Mesa 9.2.3.  But we couldn't get people to actually
>>>> ship it, and had to field tons of bug reports from upset users for
>>>> several months.
>>>>
>>>> Gentoo has also had trouble updating for similar reasons; Matt (the
>>>> Gentoo Mesa package mantainer) can probably comment more.
>>>>
>>>> I've also heard stories from friends of mine who use radeonsi that
>>>> they couldn't get new GL features or compiler fixes unless they
>>>> upgrade both Mesa /and/ LLVM, and that LLVM was usually either not
>>>> released or not available in their distribution for a few months.
>>>>
>>>> Those are the sorts of things I'd like to avoid.  The compiler is
>>>> easily the most crucial part of a modern graphics stack; splitting it
>>>> out into a separate repository and project seems like a nightmare for
>>>> people who care about getting new drivers released and shipped in
>>>> distributions in a timely fashion.
>>>>
>>>> Or, looking at it the other way: today, everything you need as an
>>>> Intel or (AFAIK) Nouveau 3D user is nicely contained within Mesa.
>>>> Our community has complete control over when we do those releases.
>>>> New important bug fixes, performance improvements, or features?  Ship
>>>> a new Mesa, and you're done.  That's a really nice feature I'd hate
>>>> to lose.
>>>
>>> tbh, it sounds a lot to me like if we start using LLVM more
>>> heavily/widely in mesa we should import LLVM (or at least the parts we
>>> need) into the mesa src tree.. as it is, the logistical/distro issues
>>> of LLVM have been what has scared me off the most about using it.
>>
>> Yes, agree totally. But I think we would always have that problem if we
>> want to use a compiler infrastructure outside of mesa.
>
> Not sure in what way would having LLVM in mesa's source tree be an 
> improvement upon building and linking LLVM from an external repository.

I'm not talking about pulling LLVM into mesa, but rather giving the 
build system an option to download a known good version of LLVM and 
using that one to build mesa. This would for example simplify bisecting, 
since when you build a certain mesa version it would automatically build 
the known good LLVM version as well.

>
> Just so we fool packagers into believing it is a single project? It's 
> cumbersome and won't change anything.
>
> For example, Apitrace bundles several thirdparty libraries, so it 
> statically links them into the wrappers shared objects to prevent 
> symbol collision when tracing proprietary applications which sometimes 
> also bundle their own dependencies.  Yet I note most distros patch 
> apitrace source tree to use the system's libraries instead of the 
> bundled tree.
>
> I'm pretty sure they would do precisely the same with Mesa if LLVM was 
> imported into Mesa tree.

Yeah, exactly what I suggested as well. Just that you don't need to 
patch it to build it separately, but rather provide an upstream compile 
time switch to do so.

>
>> Basically I see only two options:
>> a) We do everything inside Mesa and so also need to maintain and develop
>> it further by ourself.
>> b) We use a compiler infrastructure outside of Mesa with the well known
>> problems and advantages.
>>
>> Also what closed source drivers usually do is pulling a certain LLVM
>> version into their code and so build the compiler infrastructure
>> directly into their binaries.  This obviously isn't such a good idea for
>> distributions who seems to be concerned about the size of the resulting
>> binaries, but completely avoids the version clash.
>
> I think it is unreasonable for distros to expect to have both ways: 
> either they package multiple versions of LLVM (or at least one suited 
> to Mesa), or they accept upgrading LLVM whenever they upgrade Mesa.  
> They can't expect us to not depend on LLVM just to make packaging 
> easier...
>
>
> And I must say: basing the decision to use LLVM or any other 
> compilation framework based on how easy it is to package is having 
> priorities _completely_ backwards.
>
> If LLVM was a useless piece of junk, we wouldn't have any trouble 
> adding it as a dependency, as we would be the sole user.  But 
> precisely because LLVM is successful in so many use cases, hence 
> several packages depend on it, we shouldn't depend, so we can avoid 
> the dealing with the oh-so-hard dependency issue!?  I find it 
> ridiculous: it's precisely because LLVM is potentially that good that 
> it makes sense for us/distros/everybody to put up with the 
> dependencies issues it may bring, and worth considering.
>
> And the same reasoning goes for using LLVM vs writing a compiler 
> framework in-house.  Deciding on technical legal merits (or sometimes 
> legal constraints) is one thing.  But deciding to write things 
> in-house just to avoid dealing with an external dependency seems wrong 
> in principle.

That's indeed a pretty good argument.

>
>> But as far as I can see this could be avoid by maintaining both shared
>> and static ways of compiling LLVM into Mesa.
>>
>> Using shared LLVM would mean that LLVM must be installed in one of the
>> supported versions to use the resulting drivers. Using static LLVM can
>> be used for somebody who just wants to build the drivers manually. We
>> could even provide an option to download the necessary LLVM code,
>> compile it locally and use it only for Mesa.
>
> Yes, this sounds a good compromise to me.
>
> Though I still think that dynamically LLVM into Mesa is a bad idea 
> regardless, as last time I checked there were many global variables 
> due to the historical predominant use of LLVM as an off-line compiler.

Tom is the expert on this, but I think they are more and more moving 
away from this because of exactly those problems.

Christian.

>
> Jose