gstreamer with cairo plugin

[Guillaume Emont]

On Fri, Jan 11, 2013 at 03:33:22PM +0100, Sebastian Dröge wrote:
> On Do, 2013-01-10 at 22:18 +0000, Henry (Yu) Song - SISA wrote:
> > Well
> > 
> > First, we would like to update cairo plugin such that it is up-to-date,
> > and maintained.  Second we like to to use cairo_gl_surface instead of
> > image surface.  As far as I know our cairo-gles backend is the fastest
> > one among all cairo backend.  It is maintained by me and is being
> > actively optimized/bug fixed.
> 
> Which parts of the cairo plugin would this be? It contains (contained)
> multiple elements.

I think that the most important goal is to have video playback well integrated
in an interface that uses cairo. So I believe that a sink is the part most
needed. Not sure there ever was a cairo sink previously, apart from Benjamin
Otte's experiment[1], or was there?

> 
> Switching it unconditionally to GL doesn't sound like a good idea
> though, it should only use GL if GL is used elsewhere in the pipeline.
> Otherwise the overhead of uploading/downloading to/from the GPU is
> probably too high.
> But this can be done, is mostly unrelated to cairo though.

Yes, ideally, we should be able to:
 - know whether cairo is using gl or not (well, rather easy)
 - if we use gl, be able to negotiate with upstream whether to use buffers in
   GPU memory or in CPU memory

I am currently looking at various other sinks that seem to try to do that or
similar things, such as gst-vaapi[2]. Are there other examples out there where
I should get inspiration?

> 
> > Second, I would like to decode image into GPU memory directly with cairo
> > gl backend as sink, such that we can either play it back much quicker
> > instead of upload to texture and we can add overlay, which may also be
> > in GPU memory, quicker.
> 
> This would require the decoders to either be able to decode directly
> into GPU memory somehow or use GL... or it would be necessary for the
> cairo GL backend to be able to map the GPU memory to normal system
> memory for usage by other GStreamer elements.
I believe he was thinking of extensions such as KHR_lock_surface[3], which
"allows mapping color buffers of EGL surfaces into the client address space".

Also, in the case of hardware decoders, I think we can often hope to get the
video decoded into GPU memory. When such a case happens, we obviously want to
keep buffers in the GPU until they are rendered.
By the way, I am slightly confused: is there a "standardised" way (with a
GstMeta I guess) in GStreamer 1.0 of saying things like "this buffer is in GPU
memory" or "this function can give you this buffer mapped to an opengl(es)
texture"?
For instance, the metas in gstreamer-vaapi[2] only seem to provide info that is
useful if you can "speak" vaapi.

Guillaume

[1] http://cgit.freedesktop.org/~company/gst-plugins-cairo/
[2] http://cgit.freedesktop.org/~sree/gstreamer-vaapi-1.0/
[3] http://www.khronos.org/registry/egl/extensions/KHR/EGL_KHR_lock_surface.txt

> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel