[Ezbench-dev] ezBench errors & error handling

Fri Apr 28 15:45:25 UTC 2017

On 27/04/17 18:45, Eero Tamminen wrote:
> Hi,
> 
> Run-time errors...
> 
> First got following:
> ----------------------------------------------
>   ./core.sh -P mesa -r 1 -b gputest:fur:1080p:fullscreen -b 
> gputest:pixmark_volplosion:1080p:fullscreen -b 
> gputest:pixmark_julia_fp32:1080p:fullscreen -b 
> gputest:pixmark_piano:1080p:fullscreen -b gputest:tess_x64:576p:window 
> -b unigine:heaven:1080p:fullscreen HEAD
> Configuration scripts used:
> Repo type = git, directory = work/upstream/mesa, version = 4e1f3af, 
> deployed version =
> Tests that will be run: gputest:fur:1080p:fullscreen 
> gputest:pixmark_volplosion:1080p:fullscreen 
> gputest:pixmark_julia_fp32:1080p:fullscreen 
> gputest:pixmark_piano:1080p:fullscreen gputest:tess_x64:576p:window 
> unigine:heaven:1080p:fullscreen
> Testing 1 versions: 4e1f3af
> Estimated finish date: 17-04-27 - 18:14:32 (00h:09m:15s)
> 
> WARNING: automatic sudo rights are missing for function 
> 'xserver_setup_start'
> WARNING: automatic sudo rights are missing for function 
> 'cpu_reclocking_disable_start'
> WARNING: automatic sudo rights are missing for function 
> 'cpu_reclocking_disable_stop'
> rm: cannot remove 'logs/2017-04-27-18:05:17/lock': No such file or 
> directory
> Exiting with error code 6
> ----------------------------------------------
> 
> This was with just cloned ezBench repository i.e. empty logs directory 
> before calling core.sh.  The script created 
> "logs/2017-04-27-18\:05\:17/" and "results" file under it before breaking.
> 
> If WARNING issues prevent it from running, they should be reported as 
> ERRORs.
> 
> 
> Accidentally giving bad repository to ezbench results in nested errors. 
> Maybe error checking for that could also be improved:
> ----------------------------------------------
> $ ./ezbench -p ~/work/upstream/mesa/ -r 3 -b 
> gputest:fur:1080p:fullscreen -b 
> gputest:pixmark_volplosion:1080p:fullscreen -b 
> gputest:pixmark_julia_fp32:1080p:fullscreen -b 
> gputest:pixmark_piano:1080p:fullscreen -b gputest:tess_x64:576p:window 
> -b unigine:heaven:1080p:fullscreen -c "911391b 637e571 bf3cdf0 c942faf 
> 2f8d6bd 8307124 f7b1371 39644fa ec30149" mesa-regressions
> 2017-04-27 17:56:19: (II) Created report 'mesa-regressions'
> Traceback (most recent call last):
>    File "ezbench/python-modules/ezbench/smartezbench.py", line 420, in 
> set_profile
>      runner = self.__create_ezbench(profile=profile)
>    File "ezbench/python-modules/ezbench/smartezbench.py", line 341, in 
> __create_ezbench
>      runner.set_profile(profile)
>    File "ezbench/python-modules/ezbench/runner.py", line 442, in 
> set_profile
>      self.__send_command__("profile,{}".format(profile))
>    File "ezbench/python-modules/ezbench/runner.py", line 390, in 
> __send_command__
>      "err_code":err_code, "err_str":errstr}))
> ezbench.runner.RunnerError: {'err_str': 'The profile does not exist', 
> 'msg': "The runner returned the error code 12 (The profile does not 
> exist) for the command 'profile,~/work/upstream/mesa/'", 'err_code': 
> <RunnerErrorCode.CMD_PROFILE_INVALID: 12>}
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>    File 
> "/home/testrunner/work/ezbench/python-modules/ezbench/smartezbench.py", 
> line 426, in set_profile
>      if e.args['err_code'] == RunnerErrorCode.ARG_PROFILE_INVALID:
> TypeError: tuple indices must be integers or slices, not str
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>    File "./ezbench", line 146, in <module>
>      sbench.set_profile(args.profile)
>    File 
> "/home/testrunner/work/ezbench/python-modules/ezbench/smartezbench.py", 
> line 434, in set_profile
>      del runner
> UnboundLocalError: local variable 'runner' referenced before assignment
> -------------------------------------------2504---

Wow, that was a lot of bugs in one trace. ARG_PROFILE_INVALID is a typo 
for CMD_PROFILE_INVALID.

The second bug (local variable runner referenced before assignment) was 
already fixed in my dev branch. I squashed the two changes and pushed it 
to master.

> 
> This should probably be fatal error:
> ----------------------------------------------
> Traceback (most recent call last):
>    File "./ezbenchd.py", line 357, in <module>
>      sbench.run()
>    File "ezbench/python-modules/ezbench/smartezbench.py", line 806, in run
>      runner.start_testing()
>    File "ezbench/python-modules/ezbench/runner.py", line 476, in 
> start_testing
>      self.__send_command__("start_testing")
>    File "ezbench/python-modules/ezbench/runner.py", line 399, in 
> __send_command__
>      "err_str":"Stream ended before we got '<--'"}))
> ezbench.runner.RunnerError: {'err_str': "Stream ended before we got 
> '<--'", 'msg': 'Incomplete command:\n["WARNING: automatic sudo rights 
> are missing for function \'xserver_setup_start\'", "WARNING: automatic 
> sudo rights are missing for function 
> \'cpu_reclocking_disable_start\'"]', 'err_code': 
> <RunnerErrorCode.UNKNOWN: -1>}
> 
> Exception ignored in: <bound method Runner.__del__ of 
> <ezbench.runner.Runner object at 0x7fca98d8da58>>
> Traceback (most recent call last):
>    File "ezbench/python-modules/ezbench/runner.py", line 356, in __del__
>      self.done()
>    File "ezbench/python-modules/ezbench/runner.py", line 418, in done
>      self.__send_command__("done")
>    File "ezbench/python-modules/ezbench/runner.py", line 374, in 
> __send_command__
>      self.runner.stdin.flush()
> ----------------------------------------------
> 
> It's also pretty nasty that the script requires sudo rights for 
> everything.  Things that are done with sudo should be separate, so that 
> just them need to be granted sudo rights.

Yes, it is super nasty, but it cannot be that easily. What we really 
need is a root brocker. I started writing it but the work has stalled as 
IGT testing became more important. I will get back to it ASAP.

> 
> 
> And at end of ezbenchd.py build run there were more exceptions:
> ----------------------------------------------
> 2017-04-27 18:34:32: (II) Ezbench running m2504ode set to 'ERROR'
> Traceback (most recent call last):
>    File "./ezbenchd.py", line 357, in <module>
>      sbench.run()
>    File "ezbench/python-modules/ezbench/smartezbench.py", line 843, in run
>      time, cmd_output = runner.run(e.commit, e.test, False)
>    File "ezbench/python-modules/ezbench/runner.py", line 472, in run
>      r = self.__send_command__("run,{},{},{}".format(commit, test, 
> verbose))
>    File "ezbench/python-modules/ezbench/runner.py", line 374, in 
> __send_command__
>      self.runner.stdin.flush()
> BrokenPipeError: [Errno 32] Broken pipe
> 
> Exception ignored in: <bound method Runner.__del__ of 
> <ezbench.runner.Runner object at 0x7f8baa57a668>>
> Traceback (most recent call last):
>    File "ezbench/python-modules/ezbench/runner.py", line 356, in __del__
>      self.done()
>    File "ezbench/python-modules/ezbench/runner.py", line 418, in done
>      self.__send_command__("done")
>    File "ezbench/python-modules/ezbench/runner.py", line 374, in 
> __send_command__
>      self.runner.stdin.flush()
> BrokenPipeError: [Errno 32] Broken pipe
> ----------------------------------------------

I push an update to take care of these messages. But I do not understand 
why runner.sh just exits. I tried to find any exit that may still be in 
the code, but fell out short :s

> 
> 
> Build errors...
> 
> ezBench should have a configuration option for where suitable version of 
> libdrm is located, or it should build one itself.

The latter is the ultimate plan. I guess we don't need to wait for multi 
component bisect to compile libdrm already.

In the mean time, I suggest putting in user_parameters.sh all the 
necessary environment variables you want accessible during the building 
process:

	export LD_LIBRARY_PATH=babla
	etc...

So sorry for the current state, we have been going through a lot of 
changes recently!

Martin