tracing on demand

[Peter Lohrmann]

glCreateShaderProgramv - ah, thanks for the warning. I knew you could query shader source, but didn't realize you could compile sources directly into a program object. 

Unconditionally tracing some of the calls could definitely be a useful.

I guess I meant that I wouldn't be tracking ALL the state throughout the start of the trace, but will track the few cases which can't be queried.  OpenGLES may need more tracking than desktop GL. Once I get there, I'll figure out the best approach. I did see the retrace/state* classes, and will probably want to utilize those classes. I haven't fully investigated how the retrace captures the state, but I'll be digging into that in the next day or so.

Another question: is there a particular reason the _gl* api call is made between Begin\EndEnter() and Begin\EndLeave() calls to the localWriter? Do the Enter and Leave sections get utilized in the retrace to help with profiling? or is it primarily for organization of the trace file? I may have to have to reorganize the wgltrace.cpp file so that I can take advantage of the trace file writing without having duplicated code in various places, and don't want to break anything that I'm not yet aware of.

- Peter

-----Original Message-----
From: José Fonseca [mailto:jose.r.fonseca at gmail.com] 
Sent: Tuesday, March 26, 2013 1:08 AM
To: Peter Lohrmann
Cc: apitrace at lists.freedesktop.org
Subject: Re: tracing on demand

On Mon, Mar 25, 2013 at 10:49 PM, Peter Lohrmann <peterl at valvesoftware.com> wrote:
>
> I've spent the past week getting up to speed on apitrace, mostly looking into the trace implementation and getting it to trace API calls for a single frame, specified by frame number on the CLI. On its own I don't think this is particularly interesting (and the same result can already be obtained by a full trace & trim to single frame), but there were some bugs in the code generation that I had to fix if tracing is not enabled (look at wglGetProcAddress not exposing the GREMEDY entrypoints and glGetStringi not callling glGetStringi_override(), as two examples).
>
> I'm now planning to implement the tracing on demand, which I saw requested here: https://github.com/apitrace/apitrace/issues/31 My ultimate goal is to allow for a keypress to snapshot the state at the start of the next frame, and then record that frame's API calls. Initially this is just to capture a single frame which can then be replayed, but that can easily be extended to multiple frames in the future. I have implemented similar functionality in the past, so it won't be a difficult feat, but will take time.


One thing to be aware is that not all OpenGL can be queried after it has been set. For example, if an app uses glCreateShaderProgramv, the shader source can't be recovered. We have workarounds in glretrace for that so that we can dump state later (e.g.
https://github.com/apitrace/apitrace/commit/3db3b2f03af32da02faa59b9bcbdd0e2879e280a
). You could mimic such workarounds before tracing starts, or you could unconditionally trace such calls immediately even before the trace starts. In particular I suspect that most GLX initialization calls should be invariably recorded, as I don't think there is any way to enumerate all contexts, or determine whether they share any state, and they are very infrequent.

I'm not sure if it's relevant for you, but OpenGL ES is even more problematic -- one basically needs to shadow a lot of state because ARB left out major chunks of functionality.

> I don't plan to track state throughout the applications execution, which you have suggested would be a hard task (and I agree), but instead to take a thorough snapshot of all state and buffers and then generate API calls to reproduce that state.

Concerning the emission of the calls, how exactly do envision to implement it? I suppose that JSON would be unnecessary here, but there are several helper functions in retrace/glstate* which would probably be useful for you to extract the state, so we should consider moving that code to a separate library and subdir so it can be reused for trace as well, and refactor its implementations to expose more functionality independent of JSON output.

> As part of this implementation, distinguishing between real calls and fake ones to replay correctly (https://github.com/apitrace/apitrace/issues/59 ) can be resolved by wrapping the fake calls in a documented string marker. I don't know how many of those calls exist currently, but there will definitely be some to replicate the state snapshot at the start of a single-frame  trace.


There several places where we currently emit fake calls (e.g, memcpy,
glVertexAttribPointer*) I actually planned to add a new "flag"
property to call events. The flags already exists (see CALL_FLAG_FAKE in trace_model.hpp) but I never actually added the flag to the trace file format. It's an easy thing to do, so I'd suggest you ignore the fake calls business for now, and I'll take on to myself to finish the implementation in the near term.

>
> Let me know if you have any suggestions or concerns based on the 
> above,
>
>    Peter


Overall sounds good. It's a great technical challenge, but with equally great usefulness. Hopefully one can strike some middle ground where most apps  work well. We could also enhance apitrace to produce helpful messages when unsupported corner cases are detected, so that app developers known when and how to avoid them.

Jose