[Libva] [RFC] OpenGL API extension (v3)

[Gwenole Beauchesne]

Hi Jonathan,

On Tue, 22 Sep 2009, Bian, Jonathan wrote:

> I didn't see follow-ups to the following thread when catching up on 
> emails after coming back from vacation ...

Sorry, I was busy implementing and testing the thing. ;-)

> I am thinking along similar lines as Austin on this proposal. If we can 
> have the app pass the GLX context at VA init time, then perhaps only a 
> new vaPutSurface interface is needed for the purpose of getting the VA 
> surface into a GL texture, without having to add a number of new 
> functions.

Actually, this was the way (v1) of the API operated:
1) One function to tell we are in GLX mode: vaGetDisplayGLX()
2) One function to transfer VA surface to GL texture: 
vaCopySurfaceToTexture()

This worked for my small hwdecode-demos. However, while integrating into 
real-world applications (e.g. MPlayer), this exposed several problems.

1) GLX context is generally tied to a window. So, propagating this context 
to vaGetDisplay() is not a good option. A vaInitializeGLX() seemed a 
better location, especially to let the VA driver initialize extra VTable 
hooks, but it's not really convenient either from a user point of view.

For instance, MPlayer is creating/destroying the GLX context whenever the 
video is changed. vaInitialize()'ing there, or changing the VADisplay, is 
not a good approach, IMHO. So, at this stage, it became apparent I needed 
a new function to tell "hey, we use GLX, here is the GLX context".

Next, I came to think about the actual storage of the GLX context.

- A pointer to the GLXContext? If GLXContext is changed, libVA 
automatically knows about the change. However, if GLXContext is destroyed, 
and user does not set it to GL_NONE or whatever, libVA would not have any 
chance to know about it. Well, we have the same problem as with the X11 
display anyway, so this probably bearable and documentable. e.g. GLX 
context shall be live from X to Y.

- A copy of the GLXContext? If GLXContext is changed, libVA wouldn't know 
about the changes since it had a previous copy. We then tend to an 
explicit API to notify the GLXContext.

Finally, I decided that we may just delegate this to the user and let 
libVA assume to operate under the currently live GLXContext. This works 
with current implementations.

2) Some implementations (like NVIDIA) are limited to the GLX 
texture-from-pixmap extension. They have no means to directly transfer the 
VA surface to a GL texture. If I want a VA surface to be transferred to a 
GL texture, I can only use an FBO. Extra work that I don't want users to 
support.

BTW, TFP works by binding/releasing the GLX pixmap whenever we need to 
render a texture where that pixmap is attached to. So, in order to avoid 
FBO, I wanted to see whether there would be any performance gain by adding 
bind/release functions directly at the VA level and have this usage 
scenario:

vaBindSurface
// gl operations to render
vaReleaseSurface

It turns out the gain is really minimal but that was on an NVIDIA 8600GT 
discrete card. Around 3 points gain (e.g. 13% -> 10% CPU time).

BTW, it would be interesting to know performance on a Moorestown platform. 
If using MPlayer, you can try the following:
$ ./mplayer -vo vaapi:gl -va vaapi -fs /path/to/some/1080p/video
$ ./mplayer -vo vaapi:gl:bind -va vaapi -fs /path/to/some/1080p/video

Would a bind/unbind approach be faster than an FBO approach on Moorestown? 
You can privately tell me that. In the meantime, I will try again to get a 
platform.

Why Associate/Deassociate functions? This was a convenience to avoid 
repeating GL texture target, id and probably GLX context too in function 
arguments. So, it became apparent a dedicated VA/GLX surface would be 
useful to hold that information.

3) I only care about transferring the VA surface to a GL texture as my 
needs are for Clutter. I also want a user to do minimal work. So, I'd 
really want to keep only the "transfer VA surface to GL texture" API and 
possibly drop the bind/unbind stuff, unless it's really necessary 
performance-wise e.g. on Moorestown and Poulsbo.

4) I wanted a generic VA/GLX implementation within libVA for drivers only 
supporting vaPutSurface(Pixmap,...). So, this point was IMHO a stronger 
reason to have a dedicated VA/GLX surface. Otherwise, that would have 
tended to some awful machinery to my taste. And also because there is no 
means to attach "user" information to an existing VA surface.

In summary, I would like:

(a) Less pain for the users. i.e. don't have him duplicate TFP(+FBO) 
machinery in his code. Move that directly into libVA for the generic 
implementation.

(b) A single "Copy VA surface to GL texture" API. This will remove the 
need for associate, deassociate, bind, release functions. Unless this is 
really necessary to keep those for other implementations for performance.

With the (b) condition, we would only keep: vaGetDisplayGLX(), 
vaCreateSurfaceGLX(), vaDestroySurfaceGLX(), vaCopySurfaceGLX(). This has 
the advantage that vaCreateSurfaceGLX() could also be the API to receive 
the GLX context.

5) I want to keep the existing functions and control flow as is. That is, 
I would like to avoid the need to call an extra function in-between just 
to pass some GLX info/context to the VA driver. The reason is e.g. for 
FFmpeg, I want to move more stuff to it so that a user only has to:
- dpy = vaGetDisplay{X11,GLX,whatever}()
- av_create_hwaccel_context(avctx, dpy);
- vaPutSurface()/vaCopySurfaceToTexture() when it needs to
- av_destroy_hwaccel_context(avctx);

i.e. vaCreateContext(), vaCreateConfig(), vaCreateSurfaces() and all those 
things (display-independent) are to be moved to FFmpeg itself. This would 
terribly ease FFmpeg-based applications enabling to VA API.

Regards,
Gwenole.