[Piglit] [PATCH] framework: keep track of running stats and display them

[Dylan Baker]

On Saturday, February 15, 2014 01:34:09 PM Ilia Mirkin wrote:
> On Sat, Feb 15, 2014 at 6:43 AM, Dylan Baker <baker.dylan.c at gmail.com> 
wrote:
> > On Saturday, February 15, 2014 04:48:37 AM Ilia Mirkin wrote:
> >> On Sat, Feb 15, 2014 at 4:45 AM, Dylan Baker <baker.dylan.c at gmail.com>
> > 
> > wrote:
> >> > On Saturday, February 15, 2014 02:28:11 AM Ilia Mirkin wrote:
> >> >> This will display a line that looks like:
> >> >> [00076/11064] fail: 1, pass: 55, skip: 19 Running Test(s): 00075
> >> >> 
> >> >> Signed-off-by: Ilia Mirkin <imirkin at alum.mit.edu>
> >> >> ---
> >> >> 
> >> >>  framework/core.py | 11 +++++++----
> >> >>  framework/log.py  | 19 +++++++++++++++----
> >> >>  2 files changed, 22 insertions(+), 8 deletions(-)
> >> >> 
> >> >> diff --git a/framework/core.py b/framework/core.py
> >> >> index ac25917..4bcaa82 100644
> >> >> --- a/framework/core.py
> >> >> +++ b/framework/core.py
> >> >> 
> >> >> @@ -435,6 +435,7 @@ class Test(object):
> >> >>          '''
> >> >>          
> >> >>          log_current = log.get_current()
> >> >> 
> >> >> +        counts = {}
> >> >> 
> >> >>          # Run the test
> >> >>          
> >> >>          if env.execute:
> >> >>              try:
> >> >> @@ -462,15 +463,17 @@ class Test(object):
> >> >>                  result['traceback'] = \
> >> >>                  
> >> >>                      "".join(traceback.format_tb(sys.exc_info()[2]))
> >> >> 
> >> >> -            if 'subtest' in result and len(result['subtest'].keys())
> >> >> >
> >> >> 1:
> >> >> -                for test in result['subtest'].keys():
> >> >> -                    result['result'] = result['subtest'][test]
> >> >> +            if 'subtest' in result and len(result['subtest']) > 1:
> >> >> +                for test in result['subtest']:
> >> >> +                    res = result['result'] = result['subtest'][test]
> >> >> +                    counts[res] = counts.get(res, 0) + 1
> >> >> 
> >> >>                      json_writer.write_dict_item(os.path.join(path,
> >> >>                      test),
> >> >> 
> >> >> result) else:
> >> >> +                counts[result['result']] = 1
> >> >> 
> >> >>                  json_writer.write_dict_item(path, result)
> >> >>          
> >> >>          else:
> >> >>              log.log()
> >> >> 
> >> >> -        log.mark_complete(log_current)
> >> >> +        log.mark_complete(log_current, counts)
> >> >> 
> >> >>  class Group(dict):
> >> >> diff --git a/framework/log.py b/framework/log.py
> >> >> index 01c3a32..ba045da 100644
> >> >> --- a/framework/log.py
> >> >> +++ b/framework/log.py
> >> >> 
> >> >> @@ -37,22 +37,30 @@ class Log(object):
> >> >>          self.__running = []
> >> >>          self.__generator = (x for x in xrange(self.__total))
> >> >>          self.__pad = len(str(self.__total))
> >> >> 
> >> >> +        self.__summary = {}
> >> >> +
> >> >> +    def _summary(self):
> >> >> +        return ", ".join("{0}: {1}".format(k, self.__summary[k])
> >> >> +                         for k in sorted(self.__summary))
> >> >> 
> >> >>      def _running(self):
> >> >>          return "Running Test(s): {}".format(
> >> >>          
> >> >>              " ".join([str(x).zfill(self.__pad) for x in
> >> >>              self.__running]))
> >> >>      
> >> >>      def _percent(self):
> >> >> -        return
> >> >> "[{0}/{1}]".format(str(self.__complete).zfill(self.__pad),
> >> >> -                                 
> >> >> str(self.__total).zfill(self.__pad))
> >> >> +        return "[{0}/{1}] {2}".format(
> >> >> +            str(self.__complete).zfill(self.__pad),
> >> >> +            str(self.__total).zfill(self.__pad),
> >> >> +            self._summary())
> >> >> 
> >> >>      @synchronized_self
> >> >> 
> >> >> -    def mark_complete(self, value):
> >> >> 
> >> >> +    def mark_complete(self, value, counts):
> >> >>          """ Used to mark a test as complete in the log
> >> >>          
> >> >>          Arguments:
> >> >>          value -- the test number to mark complete
> >> >> 
> >> >> -
> >> >> +        counts -- a map from result to count of tests with that
> >> >> result
> >> >> +
> >> >> 
> >> >>          """
> >> >>          # Mark running complete
> >> >>          assert value in self.__running
> >> >> 
> >> >> @@ -61,6 +69,9 @@ class Log(object):
> >> >>          # increment the number of completed tests
> >> >>          self.__complete += 1
> >> >> 
> >> >> +        for k, v in counts.iteritems():
> >> >> +            self.__summary[k] = self.__summary.get(k, 0) + v
> >> >> +
> >> >> 
> >> >>      @synchronized_self
> >> >>      
> >> >>      def log(self):
> >> >>          """ Print to the screen
> >> > 
> >> > Keeping track of subtests is going to make all of the accounting in log
> >> > inaccurate, correct? Since subtests have never been displayed by the
> >> > logger
> >> > (not this one or the previous one), counting them into your total will
> >> > end
> >> > up with a count greater than the total, or am I totally missing
> >> > something?
> >> 
> >> Not missing anything. I can redo this to just add 1/test, rather than
> >> counting the subtests. Of course if you can add in your head _that_
> >> fast... oh, and piglit-summary-html, at least, shows data by subtest.
> >> 
> >>   -ilia
> > 
> > I just think it'll look weird when the report says: [104/100] (or
> > similar), so
> That shouldn't be possible -- that should still be the completed
> count, which is 1/test. The subtests only affect the summary bit.

I completely misunderstood what was happening here, my bad.

> 
> > I'd prefer not counting subtests, but just incrementing the total every
> > time a subtest is counted would be fine by me too.
> 
> Sorry, having trouble parsing that...

Yeah, I should know better than to write emails late at night, sigh.

> 
> Anyways, I'll just kill the subtest stuff and use the overall result
> and be done with it.
> 
> Any changes I make will conflict with your fixes to the logging stuff
> too -- how do you want to handle it? It seems like your latest RFC is
> based on my changes?


> 
>   -ilia
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20140216/644b76ca/attachment.pgp>