Invoking gla11y [was: core.git: bin/gla11y config_host.mk.in configure.ac solenv/gbuild]

[Stephan Bergmann]

On 22.02.2018 17:08, Samuel Thibault wrote:
> Stephan Bergmann, on jeu. 22 févr. 2018 16:43:37 +0100, wrote:
>> On 22.02.2018 13:54, Samuel Thibault wrote:
>>> Miklos Vajna, on jeu. 22 févr. 2018 09:53:43 +0100, wrote:
>>>> On Wed, Feb 21, 2018 at 10:11:18AM +0100, Samuel Thibault <sthibault at hypra.fr> wrote:
>>>>> Rene Engelhard, on mer. 21 févr. 2018 10:02:08 +0100, wrote:
>>>>>> OK, thanks - and then I wonder why this is ran in "normal" UIConfig targets and
>>>>>> not only in make check...
>>>>> For the "error" cases (-W none), it makes sense:
>>>>>
>>>>> « Add to the build process error checking (only the hard errors such as
>>>>> bogus target names). »
>>>>>
>>>>> That's really the kind of issues one should get as soon as compilation
>>>>> time.
>>>>>
>>>>> Later on I'll add a call in "make check" time for the warnings.
>>>>
>>>> For source code we have warnings for these linter type problems
>>>
>>> Err, these are not just "linter type problems". They are undefined
>>> references, just like undefined C references. So the failure belongs to
>>> compilation time. If somebody modifies a .ui file in a way that gets
>>> such undefined reference, compilation itself should really fail, just
>>> like it does when a variable defintion is missing.
>>>
>>> The "make check" warnings I mentioned above are *other* kinds of issues,
>>> which would indeed only be tested within make check, made warnings by
>>> default, and turned into errors by Jenkins.
>>
>> I don't understand the "turned into errors by Jenkins" part.  How would that
>> be done?
>>
>> (And `make check` should ideally be "silent".  We do not want it to produce
>> any non-fatal warnings.)
> 
> Well, perhaps I didn't understand what Miklos meant, then.
> 
> Miklos Vajna, on jeu. 22 févr. 2018 09:53:43 +0100, wrote:
>> For source code we have warnings for these linter type problems and then
>> --enable-werror (Jenkins uses it) turns those warnings into errors.
>>
>> Probably the best would be if the .ui checker would follow the same
>> pattern, instead of running it twice during 'make check' (which is a
>> superset of 'make').
> 
> What I had understood from that was that at "make" time there are only
> compiler errors, and at "make check" time there are linter warnings, and
> in Jenkins, --enable-werror is used to turn these warnings into errors.
> 
> Now that I re-read it with what you said, I understand that
> 
> - at "make" time there are some compiler warnings which are turned into
> errors in Jenkins with --enable-werror
> - Miklos suggested that we only call gla11y at "make" time, and not a
> second time at "make check" time.
> 
> I'm fine with emitting accessibility warnings at "make" time and let
> Jenkins turning them into errors, I just thought from the discussions at
> FOSDEM that they should be done at "make check" time.

`make` vs. `make check` is completely orthogonal to 
--{en,dis}able-werror:  `make check` just runs more tests than `make`; 
some tests are expensive (and some are a bit brittle), so there are 
reasons why some builds want to be done without running the extra tests 
(though in an ideal world, they would always be included).

--enable-werror causes C/C++ compiler warnings to be turned into build 
failures.  We have a policy of a warning-free git master (and release 
branches), so that developers can --enable-werror and immediately find 
new issues they are introducing for which compilers emit warnings.  (And 
to keep master warning free on all platforms, Gerrit/Jenkins builds are 
also done with --enable-werror.)  Again, in an ideal world, 
--enable-werror would always be enabled for every build, but there are 
practical reasons (like some compiler emitting some hard-to-silence 
warnings only in -O2 production builds) why some builds use 
--disable-werror.

Now, gla11y shall do two things:

1  Find newly introduced bugs in .ui files.  Those should always cause 
the build to fail, regardless of `make` vs. `make check` or 
--enable-werror vs. --disable-werror.  I see no reason to have a build 
mode in which those bugs are either not detected at all, or do not fail 
the build.

2  Find existing bugs in .ui files (and maybe also minor newly 
introduced issues that are not severe enough to be considered fatal bugs 
upfront; I do not know).  Those should initially be suppressed by (1), 
but should eventually all be fixed (IIUC).  So for somebody working on 
fixing those issues, it would be useful to have a way to get a report of 
all those issues.  Maybe a new make target would be useful for that.

Does that make sense?