[Mesa-dev] [RFC PATCH 00/16] A new IR for Mesa

[Jose Fonseca]

On 26/08/14 10:04, Christian König wrote:
> Am 26.08.2014 um 03:54 schrieb Rob Clark:
>> On Wed, Aug 20, 2014 at 2:13 PM, Kenneth Graunke
>> <kenneth at whitecape.org> wrote:
>>>>> we've already had problems where distros refused to ship newer Mesa
>>>>> releases because radeon depended on a version of LLVM newer than the
>>>>> one they were shipping, [...]
>>>> That's news to me, can you be more specific?
>>>>
>>>> That sounds like basically a distro issue though, since different LLVM
>>>> versions can be installed in parallel (and the one used by default
>>>> doesn't have to be the newest one). And it even works if another
>>>> part of
>>>> the same process uses a different version of LLVM.
>>> Yes, one can argue that it's a distribution issue - but it's an
>>> extremely painful problem for distributions.
>>>
>>> For example, Debian was stuck on Mesa 9.2.2 for 4 months (2013-12-08
>>> to 2014-03-22), and I was told this was because of LLVM versioning
>>> changes in the other drivers (primarily radeon, I believe, but
>>> probably also llvmpipe).
>>>
>>> Mesa 9.2.2 hung the GPU every 5-10 minutes on Sandybridge, and we
>>> fixed that in Mesa 9.2.3.  But we couldn't get people to actually
>>> ship it, and had to field tons of bug reports from upset users for
>>> several months.
>>>
>>> Gentoo has also had trouble updating for similar reasons; Matt (the
>>> Gentoo Mesa package mantainer) can probably comment more.
>>>
>>> I've also heard stories from friends of mine who use radeonsi that
>>> they couldn't get new GL features or compiler fixes unless they
>>> upgrade both Mesa /and/ LLVM, and that LLVM was usually either not
>>> released or not available in their distribution for a few months.
>>>
>>> Those are the sorts of things I'd like to avoid.  The compiler is
>>> easily the most crucial part of a modern graphics stack; splitting it
>>> out into a separate repository and project seems like a nightmare for
>>> people who care about getting new drivers released and shipped in
>>> distributions in a timely fashion.
>>>
>>> Or, looking at it the other way: today, everything you need as an
>>> Intel or (AFAIK) Nouveau 3D user is nicely contained within Mesa.
>>> Our community has complete control over when we do those releases.
>>> New important bug fixes, performance improvements, or features?  Ship
>>> a new Mesa, and you're done.  That's a really nice feature I'd hate
>>> to lose.
>>
>> tbh, it sounds a lot to me like if we start using LLVM more
>> heavily/widely in mesa we should import LLVM (or at least the parts we
>> need) into the mesa src tree.. as it is, the logistical/distro issues
>> of LLVM have been what has scared me off the most about using it.
>
> Yes, agree totally. But I think we would always have that problem if we
> want to use a compiler infrastructure outside of mesa.

Not sure in what way would having LLVM in mesa's source tree be an 
improvement upon building and linking LLVM from an external repository.

Just so we fool packagers into believing it is a single project?  It's 
cumbersome and won't change anything.

For example, Apitrace bundles several thirdparty libraries, so it 
statically links them into the wrappers shared objects to prevent symbol 
collision when tracing proprietary applications which sometimes also 
bundle their own dependencies.  Yet I note most distros patch apitrace 
source tree to use the system's libraries instead of the bundled tree.

I'm pretty sure they would do precisely the same with Mesa if LLVM was 
imported into Mesa tree.

> Basically I see only two options:
> a) We do everything inside Mesa and so also need to maintain and develop
> it further by ourself.
> b) We use a compiler infrastructure outside of Mesa with the well known
> problems and advantages.
>
> Also what closed source drivers usually do is pulling a certain LLVM
> version into their code and so build the compiler infrastructure
> directly into their binaries.  This obviously isn't such a good idea for
> distributions who seems to be concerned about the size of the resulting
> binaries, but completely avoids the version clash.

I think it is unreasonable for distros to expect to have both ways: 
either they package multiple versions of LLVM (or at least one suited to 
Mesa), or they accept upgrading LLVM whenever they upgrade Mesa.  They 
can't expect us to not depend on LLVM just to make packaging easier...


And I must say: basing the decision to use LLVM or any other compilation 
framework based on how easy it is to package is having priorities 
_completely_ backwards.

If LLVM was a useless piece of junk, we wouldn't have any trouble adding 
it as a dependency, as we would be the sole user.  But precisely because 
LLVM is successful in so many use cases, hence several packages depend 
on it, we shouldn't depend, so we can avoid the dealing with the 
oh-so-hard dependency issue!?  I find it ridiculous: it's precisely 
because LLVM is potentially that good that it makes sense for 
us/distros/everybody to put up with the dependencies issues it may 
bring, and worth considering.

And the same reasoning goes for using LLVM vs writing a compiler 
framework in-house.  Deciding on technical legal merits (or sometimes 
legal constraints) is one thing.  But deciding to write things in-house 
just to avoid dealing with an external dependency seems wrong in principle.

> But as far as I can see this could be avoid by maintaining both shared
> and static ways of compiling LLVM into Mesa.
>
> Using shared LLVM would mean that LLVM must be installed in one of the
> supported versions to use the resulting drivers. Using static LLVM can
> be used for somebody who just wants to build the drivers manually. We
> could even provide an option to download the necessary LLVM code,
> compile it locally and use it only for Mesa.

Yes, this sounds a good compromise to me.

Though I still think that dynamically LLVM into Mesa is a bad idea 
regardless, as last time I checked there were many global variables due 
to the historical predominant use of LLVM as an off-line compiler.

Jose