[Mesa-dev] A proposal for new testing requirements for stable releases

[Carl Worth]

Michel Dänzer <michel at daenzer.net> writes:
>> 	configurations are. All I need is something along the lines of:
>> 
>> 		"The radeon team is OK with releasing commit <foo>"
...
[snip of the rest of my proposal]

> This sounds good, but...

Good! Thanks for reading this and giving me some feedback.

>> 	And in order to put some teeth into this scheme:
>> 
>> 	I propose that on the day of the release, the release manager
>> 	drop all driver-specific patches for any driver for which the
>> 	driver team has not provided an affirmative testing report.
>
> ... this seems a bit harsh.
>
> A lot of backported fixes are specific to one driver and have zero
> impact on anything else. Surely it should be up to the discretion of the
> driver maintainers whether or not such changes should be backported.

Yes. I do want to give the driver maintainers a lot of discretion. They
are already the people who are doing the proposal of which changes
should be backported.

Quite frankly, my real concern with all of this is not that the driver
maintainers will propose something bad, but that I will inadvertently
botch something while cherry-picking or merging a conflict, etc. that I
won't be able to notice in my touch testing.

> OTOH, who's supposed to give that sort of OK for changes to say st/mesa
> or Mesa core? The former may affect all Gallium based drivers, the
> latter basically anything.

Previously, for each stable release, I was doing piglit testing against
the hardware driver for my personal laptop. My hope was that that would
give at least some coverage for any mesa-core changes.

With the current release cycle, I've expanded that testing to also
include piglit runs against swrast (in both dri and gallium flavors).
I hope that that will give us sufficient confidence in
non-driver-specific changes.

With this additional testing, it's sufficiently time-intensive that I
don't plan to repeat it in a single cycle, (I'm performing that testing
before pushing out a proposed branch like I did yesterday). That's why
my email yesterday said I could still accept driver-specific changes,
(but any core changes will have to wait for the next release).

> It's unlikely that any individual or organization can test all
> affected setups in advance.

I agree. And I won't require that.

All I want is what I said above:

	"We're OK with releasing commit <foo>"

From a reasonably representative of each driver team.

I'm not going to ask for documentation to backup that statement, (no
piglit reports or anything else).

So every driver team can come up with their own criteria for comfort
level. If you're happy to continue to just defer to me (or any other
stable-release manager), then all I'm asking for is one quick email
each release to give your OK.

If driver teams want to do some actual testing, then this new process
gives them time to do that testing and a way to communicate those
results prior to release.

And I think that's the most-important change here. I'm committing to
providing a 3-day window in which testing can be performed prior to each
stable release.

> We at AMD are working towards having such regular testing, but we're not
> quite there yet, nor do we have an exact schedule yet for when we will
> be.

It's great that you're working toward it. I think that's a valuable goal
that every driver team should work toward.

But my proposal was not intended to make that a requirement. You can
continue to get your commits in with no additional testing on your part,
by just affirming for each release that you're OK with what the
stable-branch maintainer has put together.

When I phrase things that way, does it seem more reasonable to you?

And if this feels like a more bureaucratic means for doing nothing
effectually different than what we did before, please humor me. I think
it can help reduce some psychological pressure for the release
maintainer.

(Maybe I'm just a wimp, but it's been tough at times to trust myself to
resolve conflicts in code that I know I don't have any way to test. I do
ask for help if the conflict looks really messy. But I don't like to
bother people for things that look trivial. And even the trivial, manual
conflict resolution can give me misgivings that I might be breaking the
release.)

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/mesa-dev/attachments/20140715/85d22a49/attachment.sig>