Universal package specification (the long way to go...)

[Stanislav Brabec]

Hallo Eugen.

I worked on the idea of a generic build description many times in last
10 years. However, I always failed for reasons that were obvious ex post
but not before starting the implementation. (Skip to "Summary of all my
attempts..." to read my opinion of correct way to go.)


"Abstract Package Build Description" SoC 2006 project by Rajagopal
Natarajan[1] was just a concept very similar to your idea: A XML file
that will generate DEB and SPEC file after processing.

However the basic implementation succeeded, the project never reach a
wide audience. I can only guess reasons why it never reach commercially
usable state.


I guess that there are serious technical problems of this concept:


Names and versions of dependencies

Distro dependent package naming and splitting extremely complicates
section of build requirements and install requirements. You have to
maintain 10-20 supported distros in a single XML file. You often need a
complicated expressions: "If it is an openSUSE distro older that 10.3 or
Fedora older than 10 or SUSE Linux enterprise less than 11, then you
need package A, otherwise you need package B." "In distros older than
xxx you need to disable feature yyy, because dependent packages are too
old."


Splitting and file lists

Each distro has a different need for packages splitting. One-purpose
distros preffer all-in-one packages, embedded distros prefer cutting
into smallest possible pieces. Some distros have mandatory packaging
conventions for libraries. You have to follow them.


Scripts

DEB and RPM have a completely different concept of installation scripts.
Even each RPM based distro and distro version use a different scripts.

It is not easy to maintain scripts for all distros up to date.

Depending on sub-package splitting, scripts may be needed to be called
inside a different package.


Init scripts, files paths and prefixes

Even init scripts, files paths and prefixes may be distro specific. You
again have to follow them.



Build description conventions

Different distros need different build description abbreviation. Nobody
wants to maintain spec files thousands lines long. Abbreviation of file
lists, calling %configure instead of ./configure, using %make_install or
%makeinstall instead of "make install DESTDIR=..." etc. These minor, but
non-trivial things to implement discriminate good spec file from an ugly
one.


Multiplies need for QA resources

Edits in a particular per-distro-version-spec need to build and re-test
just for a single distro version.

Each edit of the global XML description needs to be re-test in all
supported products.


Social problems

People do not feel a need to join to a project, that "just adds one
level of abstraction". Why learn a new format and edit XML, if I can
edit just a spec and deb separately in a comparable time.


Your concept has even more problems:


Redundant information

SONAME is intended to provide API versioning of libraries. Manual adding
of this redundant information into a package description is just a waste
of work. The same is valid for D-Bus interfaces. D-Bus introspection is
the way to get API information.


[1] http://en.opensuse.org/Abstract_Package_Build_Description

===============

Summary of all my attempts to implement universal package description:


Spec and deb generators are not way to go

To create a high quality .spec or .deb build description, you have to
invest extreme amount of implementation work. It will be wasted again
and again with each new packaging convention.

If the generator create sub-par spec files, nobody will take such tool
into account.

If the generator would not be able to incorporate existing spec-level
file changes, nobody will take such tool into account.



General package descriptions are not way to go

General package description may require to track and test tens of
possible branches and tests for each requirement of option.

Maintenance of such package-dinosaur would be extremely slow,
complicated and expensive. Each edit may potentially require tens of
compilation and installation tests on different platforms.


New package binary format is not way to go

Binary package format change by any of large Linux players means
estimated expenses of several millions of dollars for a complete change
of the infrastructure (imagine migration support, build systems rewrite,
packaging systems front-ends rewrite, testings, education of package
maintainers, education of customers, rewrite of documentation etc.).
Nobody wants a new format, except the case if the change would promise a
really huge potential benefit (several millions of dollars in next few
years).


Save peoples' work

Current packaging systems require large amount of mechanic manual work.
It is not a good situation.

New approach should minimize required amount of the package maintainer
edits. Anything that goes in a reverse direction is not acceptable.



Possible future approaches
==========================

Just now I see two possible future approaches:


Autotools

Massive improvement of autotools (adding packagers' support in autoconf,
automake, libtool) may start new era of packaging packagers. Thousands
of packages may automagically start to support new way of packaging.

Last change in this direction was DESTDIR ten years ago. Now we need:

- List of checked files and features for package requirements.

- Unified support for installation scripts and association of scripts
with particular files.

- Split proposals and dependency generators

- Description of plugin-ability and directories assigned for plug-ins.

- Unified way to discriminate between compiling for build and compiling
for target.

- Complete suport for sysroot.

...


Change package build spec but not binary format

Users don't care of a way packages are built. If the binary package has
the correct format, users don't care in package spec. Change of package
description format would cause much less pain than change of package
binary format.


Meta packages

Meta packages which describe packaging class and divergences from its
standard model instead of manually describing each steps of the process
are way to go.

BitBake[2] (used in OpenEmbedded[3] build system) seems to be the most
advanced tools of this type.


Wizards

Semi-automatic tools capable to prepare new simple package or version
upgrade. Package maintainer would just review and accept them.


[2] http://bitbake.berlios.de/
[3] http://www.openembedded.org/

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec at suse.cz
Lihovarská 1060/12           tel: +420 284 028 966, +49 911 740538747
190 00 Praha 9                                  fax: +420 284 028 951
Czech Republic                                    http://www.suse.cz/