[igt-dev] [PATCH i-g-t] runner/resultgen: Provide output when test output is completely empty

[Petri Latvala]

On Wed, Feb 19, 2020 at 01:59:55PM +0000, Chris Wilson wrote:
> Quoting Petri Latvala (2020-02-19 13:56:59)
> > On Wed, Feb 19, 2020 at 01:50:14PM +0000, Chris Wilson wrote:
> > > Quoting Chris Wilson (2020-02-19 13:46:38)
> > > > Quoting Petri Latvala (2020-02-19 13:45:03)
> > > > > On Wed, Feb 19, 2020 at 01:33:15PM +0000, Chris Wilson wrote:
> > > > > > Quoting Petri Latvala (2020-02-19 13:27:58)
> > > > > > > On Wed, Feb 19, 2020 at 01:23:22PM +0000, Chris Wilson wrote:
> > > > > > > > Quoting Petri Latvala (2020-02-19 12:19:40)
> > > > > > > > > If a machine is hard-hanging or otherwise rebooted at the correct
> > > > > > > > > time, intermediary output files get created but nothing ever gets
> > > > > > > > > written to them. That yields results that are completely empty and
> > > > > > > > > hard to categorize or even sometimes detect automatically. Handle this
> > > > > > > > > corner case explicitly with a custom text explaining what might have
> > > > > > > > > happened to prod result analysis towards fixing the real issue instead
> > > > > > > > > of wondering if test result processing is faulty.
> > > > > > > > > 
> > > > > > > > > The race for getting empty files is easier to hit than it seems. The
> > > > > > > > > files get created by the runner before calling exec(), and there's
> > > > > > > > > plenty of time to hit a really hard crash.
> > > > > > > > 
> > > > > > > > Speaking of which, it would not be terrible if igt_runner periodically
> > > > > > > > called sync() everytime it decided there were new logs to preserve.
> > > > > > > > We should also probably remove/reduce the writeback delay.
> > > > > > > 
> > > > > > > 
> > > > > > > Does calling sync() sync more than fsyncdata()?
> > > > > > 
> > > > > > No manual entry for fsyncdata(), fdatasync()?
> > > > > > sync/fsync includes metadata, fdatasync is just contents, unless
> > > > > > metadata is required for data retrieval.
> > > > > > 
> > > > > > So for us, there really shouldn't be any difference between fdatasync
> > > > > > and fsync (with the slight exception of last access timestamps).
> > > > > 
> > > > > Then what you're asking for is already there with --sync.
> > > > 
> > > > Do we use it? :)
> > > 
> > > I see that fdatasync() is used on the runner's output; are they all the
> > > files that are retrieved after a crash? I'd feel more comfortable with
> > > sync() as we purposefully do crash the system.
> > 
> > Yeah, any output written gets fdatasync()d immediately.
> 
> What about dmesg? Is that pulled from the runner's logs or from syslog?
> I ask because we quite frequently see lost output in dmesg.txt

>From /dev/kmsg directly. Reason for the discrepancy between runner's
kernel logs and dmesg.txt is beyond me.

Btw I've seen the difference go both ways so it's even more confusing.


-- 
Petri Latvala