Release criteria for X11 releases

[Eric Anholt]

On Tue, 2007-09-25 at 08:56 -0700, Alan Coopersmith wrote:
> We seem to be loosening and loosening what it takes to declare
> an X11 release shipped, and X11R7.3 seems to have lowered the
> bar even further.  I'm sure Eric & Daniel worked hard on this
> release, but it could have gone better.
> 
> Unfortunately, I don't know of anywhere we recorded the release
> criteria, but from what I remember from the 6.x series and the
> 6.9/7.0 release, the criteria for shipping were at least:
> 
> 1) Blocker bug list cleared

Yeah, in the past we would just decide at some point to shuffle the bugs
off of the current release's blocker list onto the next release's list.
What was different this time is that I just didn't shuffle them to the
next list, or I removed the blocker tag entirely.  Once you've not
actually blocked a release on a bug for 3 releases, maybe it's not
actually a blocker bug.

> 2) Entire tree or entire set of released modules builds on at
> 	least one platform (usually Linux/x86), more preferred
> 	(usually the major BSDs & Solaris)

Everything relevant (i.e. not a bunch of serial input drivers) built on
my system.  I should have just kicked the non-building input drivers out
of the 7.3 list entirely and explained the situation in relnotes.  We
also left xprint pieces in this release, despite the fact that they
don't build.

> 3) XTS run and passed on at least one platform (usually Linux/x86)

If someone could fix XTS to not have an insane build system, I would
probably have run it.  But the investment each time I want to bring it
up on a system currently is not worth it (and I've nearly autotooled it
before, but that work got lost somewhere, and I've autotooled enough
projects in the past to not get excited about doing so any more).  XTS
is also only useful as a delta from previous releases, since all X
releases fail some testcases, which is additional effort.  And XTS of
course doesn't actually test what most of my desktop is drawn with.

> 4) Documentation updated and released

Compare:
http://xorg.freedesktop.org/archive/X11R7.3/
to
http://xorg.freedesktop.org/archive/X11R7.2/

http://xorg.freedesktop.org/archive/X11R7.1/doc/RELNOTES.html looks like
it's the relnotes from a previous release with a new intro, rather than
being "what's new and important in this release"

The doc blocker bug was also full of bugs from previous releases that
never got fixed.

So, was there important katamari-time documentation that I missed?  If
so, please update the MakingReleases page so that the next release
manager doesn't miss it, too.

> (Frankly I would have considered the failure to light keyboard LED's
>   known about for months before the release as a blocker just to
>   save the developers from wasting time on the inevitable flood of bug
>   reports, but the release manager seems to not agree.)

We're shipping a release with intel's 2D driver with around 175 open
bugs against it, most of them valid and a massive number of which would
be reasonably marked as "critical".  Some of those bugs are server bugs
exposed by or inside the new modes code, which are just mis-assigned.
The ATI driver is in bad shape, with us shipping 6.6.3, which as I
understood it has many critical bugs in it, but is better than the
current branch is expected to be for some time.  So, when I look at how
things are today, a new failure to light keyboard LEDs doesn't seem like
a big deal to me.

> Was making our already-delayed release date so important that the
> lessened quality of the release was worth it?   Is this good enough
> for an X.Org final release or do we want to go back to the higher bars
> of our previous release criteria for future releases?

Here's my issue.  The release isn't a surprise.  It's time-based.  We
told everyone what month it was going to be, and then I told everyone
when it was going to be released that month.  I even slipped the
schedule a bit for important bugs that I knew we could get fixed in an
amount of time if I pushed hard and was kind of a jerk to some friends.
For the bugs that I couldn't attach a time to because nobody was
planning on fixing them soon, or because I decided they were otherwise
insufficient to block a release on, I didn't block the release.
Keyboard LEDs were the former.  If you're concerned about what's going
to happen to the release if we ship with these bugs that have been known
about for months, do something about those bugs in those months in
advance of when the release is going to occur.

(By the way, server 1.5 should be in February.  Drivers need to be
ported to pci-rework or they won't work.  I expect to not hear about
"but we didn't block the release for cirrus to get ported forward!" in
March, or if you're planning on arguing that, start pushing for
pci-rework to be un-merged today.  I'm done pushing on that,
personally.)

Ultimately, my experience with this release at the distribution level
was that the one user who complained about keyboard LEDs wasn't terribly
bothered, while the users whose video mode setting magically worked were
rather pleased.  And, just like when XFree86-3 left our tree despite
having some unique driver support, the sky didn't fall when we removed a
bunch of those input drivers.

Some distributors will not be satisfied with this quality of .0 server
release.  For them, I highly recommend helping maintain the stable
server branches, and putting effort forward to get releases of those
done. 

-- 
Eric Anholt                             anholt at FreeBSD.org
eric at anholt.net                         eric.anholt at intel.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part
URL: <http://lists.x.org/archives/xorg/attachments/20070926/970e46a2/attachment.pgp>