request for comments - standardization of python's purelib and platlib

Jan Matejek jan.matejek at novell.com
Thu Aug 13 11:23:14 PDT 2009


Hello,

I'm cross-posting this to distributions at freedesktop and python-dev,
because the topic is relevant to both groups and should be solved in
cooperation.

The issue:

In Python's default configuration (on linux), both purelib (location for
pure python modules) and platlib (location for platform-dependent binary
extensions) point to $prefix/lib/pythonX.Y/site-packages.
That is no good for two main reasons.

One, python depends on the "lib" directory. (from distro's point of
view, prefix is /usr, so let's talk /usr/lib) Due to this, it's
impossible to install python under /usr/lib64 without heavy patching.
Repeated attempts to bring python developers to acknowledge and rectify
the situation have all failed (common argument here is "that would mean
redesign of distutils and huge parts of whatnot").

Conversely, that also means that multiarch setup (/usr/lib or lib32 with
32bit python and /usr/lib64 with 64bit python) is not possible with
stock python.

Two, the default configuration makes purelib and platlib identical,
which somehow defeats the purpose of the distinction in the first place.
You either need to patch the default, or supply some alternate
configuration to take advantage of this feature.
And that's not the end of it - the next step is to make python aware of
two different locations on sys.path, one for purelib and one for
platlib, which is a different story altogether.

As distributors, we like to take advantage of purelib/platlib separation
to package pure python modules as platform-independent (noarch for
rpm-speakers). And that's not easy to do properly.

The proposal:

Let's put our heads together and choose good default locations for
purelib and platlib. Then add support to python for recognizing the
locations by default, and possibly leave note in FHS that "this is the
place".

This is IMO a good first step to making python multiarch-aware, and it
would also help a bit with LSB integration [1].

I've come up with three basic options for the configuration (substitute
"/usr" with "$prefix" if you're not a distributor). This list is by no
means comprehensive, it's just what looked reasonable at the time of
writing.

1 - the traditional way
purelib = /usr/lib/pythonX.Y/site-packages
platlib = /usr/lib(64)/pythonX.Y/site-packages

pros:
+ this is already the default for 32bit systems
+ major distributions (including Fedora, Mandriva and now finally
openSUSE too) do this
cons:
- 32bit systems have no separation, poor they!
- with multiarch setup, /usr/lib is "cluttered" by both
platform-dependent files for 32bit and platform-independent files shared
by the platforms. Also, 64bit python can pick up 32bit modules. That
doesn't cause problems in practice, but doesn't fell like a clean design.

2 - the sharedir way
purelib = /usr/share/python/X.Y
platlib = /usr/lib(64)/pythonX.Y/site-packages

pros:
+ clean separation of purelib - nice!
+ unheard of - a good place to start anew
cons:
- FHS states that /usr/share is for data. But OTOH, they don't say much
about platform-independent bytecode. We could probably get an exception
for this.
- unheard of - everyone will be surprised

3 - the perl way
purelib = /usr/lib/pythonX.Y
platlib = /usr/lib/pythonX.Y/lib-dynload-(platform-identifier)/site-packages

pros:
+ possibility of multiarch packages that would install pure python parts
into purelib and extensions or accelerators for more platforms at once -
and therefore, possibility to split large modules into
platform-dependent and platform-independent parts and save space on
installation media
+ "idea compatibility" with perl and ruby, one less install layout to learn
cons:
- completely different from what we have now - would require the most
work from both python developers and distributions

comments?

regards
jan matejek
python packager for SUSE Linux

[1] http://www.linuxfoundation.org/en/LsbPython



More information about the Distributions mailing list