[Piglit] [Patch v2 1/4] log_tests.py: Add tests from framework/log.py

[Ilia Mirkin]

On Wed, Feb 19, 2014 at 4:47 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> On Wednesday, February 19, 2014 04:39:40 PM Ilia Mirkin wrote:
>> On Wed, Feb 19, 2014 at 4:24 PM, Dylan Baker <baker.dylan.c at gmail.com>
> wrote:
>> > On Wednesday, February 19, 2014 03:44:03 PM Ilia Mirkin wrote:
>> >> On Wed, Feb 19, 2014 at 3:13 PM, Dylan Baker <baker.dylan.c at gmail.com>
>> >
>> > wrote:
>> >> > On Wednesday, February 19, 2014 02:25:15 PM Ilia Mirkin wrote:
>> >> >> On Wed, Feb 19, 2014 at 9:57 AM, Dylan Baker <baker.dylan.c at gmail.com>
>> >> >
>> >> > wrote:
>> >> >> > On Tuesday, February 18, 2014 07:07:19 PM Ilia Mirkin wrote:
>> >> >> >> On Tue, Feb 18, 2014 at 6:41 PM, Tom Stellard <tom at stellard.net>
>> >
>> > wrote:
>> >> >> >> > Hi Dylan,
>> >> >> >> >
>> >> >> >> > I've tested version 2 of this series, and I have a few
>> >> >> >> > questions/comments:
>> >> >> >> >
>> >> >> >> > + I really like being able to see how many tests have run and
>> >> >> >> >
>> >> >> >> >   how many have completed.  The pass/fail rates are nice and
>> >> >> >> >   help me identify bad test runs right away.
>> >> >> >> >
>> >> >> >> > + Would it be possible to print the test names in the non-verbose
>> >> >> >> >
>> >> >> >> >   output mode?  Would this work in concurrency mode?
>> >> >> >> >
>> >> >> >> > + When I use the verbose output, the output looks like this:
>> >> >> >> >
>> >> >> >> > running :: Program/Execute/get-global-id, skip: 11, warn: 1
>> >> >> >> > Running
>> >> >> >> > Test(s): 253
>> >> >> >>
>> >> >> >> Bah, the test name is too short... the OpenCL tests need to get
>> >> >> >> with
>> >> >> >> the program -- at least 50 chars per test name, like the OpenGL
>> >> >> >> tests
>> >> >> >
>> >> >> > Yeah, that's why the short mode has the running test numbers. I
>> >> >> > guess
>> >> >> > we
>> >> >> > could do something like have piglit catch SIGQUIT and print the
>> >> >> > running
>> >> >> > test names before exiting. Of course, that's the kind of evil thing
>> >> >> > yum
>> >> >> > does and drives me crazy
>> >> >>
>> >> >> Well, apparently that's the preferred way of killing piglit right now.
>> >> >> I'm also not a big fan of doing unexpected things with signals.
>> >> >>
>> >> >> >> :) The old contents of the line aren't being fully overwritten...
>> >> >> >>
>> >> >> >> I wonder if instead of relying on \r + overwriting, we should
>> >> >> >> instead
>> >> >> >> emit a ^L (clear screen) -- although obviously not for the verbose
>> >> >> >> output. I also wonder if this style of output should only happen if
>> >> >> >> the output is a tty. This would also allow one to show all the
>> >> >> >> currently-running tests, one per line or something (to satisfy the
>> >> >> >> previous request). And it would resolve the issue when the line
>> >> >> >> becomes longer than the terminal width and you get wrapping.
>> >> >> >
>> >> >> > we could also just emit blank space to pad up to 80 characters(since
>> >> >> > that's
>> >> >> > the standard width for a terminal), or we could do some magic to
>> >> >> > find
>> >> >> > the
>> >> >> > width of the terminal, and do some more magic to fill space pad
>> >> >> > that.
>> >> >> >
>> >> >> > If we really want to get fancy we could use curses to have
>> >> >> > multi-line
>> >> >> > rewrites. I looked into it initially and there are python bindings
>> >> >> > that
>> >> >> > work on all of the major versions of curses, but ewww... curses
>> >> >>
>> >> >> Well, we wouldn't have to go full-on curses, just a couple of commands
>> >> >> that would be easy enough to just emit directly. Actually I guess ^L
>> >> >> doesn't "just work" -- the shell must handle it somehow :( The
>> >> >> internet says that print "\033[2J" "\033[0;0H" should work (clear
>> >> >> screen, reset coordinates). But we can obviously only play such tricks
>> >> >> at all in the first place if stdout is a tty, i.e.
>> >> >> sys.stdout.isatty().
>> >> >
>> >> > I'm concerned about portability. We do have windows users, and then
>> >> > there's
>> >> > the bash, dash, tcsh, zsh, ksh, tmux, screen, etc. \r is pretty
>> >> > standard.
>> >>
>> >> That sequence should work in any terminal written in the past 30
>> >> years, AFAIK. However windows is an interesting question -- I would
>> >> guess that isatty() returns false there. And then you would fall back
>> >> on the terser output without test names which can still fit on one
>> >> line. If isatty() is true there, we could just have an explicit check.
>> >> A question is whether isatty() == false should imply verbose mode or
>> >> not. (e.g. what should happen if you do ./piglit-run.py > foo.log, or
>> >> even | tee foo.log)
>> >
>> > The other thing about the terminal codes is that they don't solve the
>> > problem, since clearing the screen in verbose mode would defeat the
>> > purpose of having a verbose mode (lots of terminal spew), and in terse
>> > mode the line will only ever get longer.
>>
>> This would obviously not be done for verbose mode, only non-verbose.
>> Otherwise, as you mention, it would completely defeat the point of
>> verbose.
>>
>> But for the non-verbose mode, you could have the output like
>>
>> [1234/5000] pass: 1000, fail: 123, etc. Running tests:
>>   [1233] some/long/test-name with/many-parameters that/make-no-sense
>>   [1222] another/similarly-randomly-named/test
>>
>> And to refresh it, you'd clear the screen, and "redraw".
>>
>> > The other option is to have a -o/--out option that logs the output to a
>> > file, and a -s/--silent option that silences sys.stdout (which could be
>> > used in conjunction to get only file logging)
>>
>> I dunno if _more_ options are the answer :) I'd rather find the
>> smallest number of options that maximizes people's satisfaction with
>> the system. I was hoping that -v/not-v would be enough. If enough
>> people would do -v by default, perhaps we should flip it and make it
>> -q for quiet mode. (I've never seen -s for silent, -q is pretty
>> standard, e.g. wget uses it, and a ton of other tools.) And using
>> isatty() to seed it wouldn't be so bad -- tty's get quiet, non-tty's
>> get verbose. Perhaps that's too confusing though.
>
> That is an option, I'll code something up.
>
> I acutally do have a use for a quite mode, I'm setting up a few headless
> jenkins servers, and printing anything is utterly useless for that
> application.
>

Oh, I see what you meant by "silent". I misunderstood. You meant "no
output at all". So then we have "regular", "verbose", and "quiet".
Makes sense. tty's get regular by default, non-tty's get verbose.
(Where "regular" is the thing described above.)

  -ilia