gstreamer with cairo plugin

[Sebastian Dröge]

On So, 2013-01-13 at 11:00 +1100, Matthew Waters wrote:
> On 12/01/13 04:59, Guillaume Emont wrote: 
> > On Fri, Jan 11, 2013 at 03:33:22PM +0100, Sebastian Dröge wrote: 
> > > On Do, 2013-01-10 at 22:18 +0000, Henry (Yu) Song - SISA wrote: 
> > > > Well 
> > > > 
> > > > First, we would like to update cairo plugin such that it is
> > > > up-to-date, 
> > > > and maintained.  Second we like to to use cairo_gl_surface
> > > > instead of 
> > > > image surface.  As far as I know our cairo-gles backend is the
> > > > fastest 
> > > > one among all cairo backend.  It is maintained by me and is
> > > > being 
> > > > actively optimized/bug fixed. 
> > > Which parts of the cairo plugin would this be? It contains
> > > (contained) 
> > > multiple elements. 
> > I think that the most important goal is to have video playback well
> > integrated 
> > in an interface that uses cairo. So I believe that a sink is the
> > part most 
> > needed. Not sure there ever was a cairo sink previously, apart from
> > Benjamin 
> > Otte's experiment[1], or was there? 
> > 
> > > Switching it unconditionally to GL doesn't sound like a good idea 
> > > though, it should only use GL if GL is used elsewhere in the
> > > pipeline. 
> > > Otherwise the overhead of uploading/downloading to/from the GPU
> > > is 
> > > probably too high. 
> > > But this can be done, is mostly unrelated to cairo though. 
> > Yes, ideally, we should be able to: 
> >   - know whether cairo is using gl or not (well, rather easy) 
> >   - if we use gl, be able to negotiate with upstream whether to use
> > buffers in 
> >     GPU memory or in CPU memory 
> > 
> > I am currently looking at various other sinks that seem to try to do
> > that or 
> > similar things, such as gst-vaapi[2]. Are there other examples out
> > there where 
> > I should get inspiration? 
> You may want to look at [1] as well. 
> > > > Second, I would like to decode image into GPU memory directly
> > > > with cairo 
> > > > gl backend as sink, such that we can either play it back much
> > > > quicker 
> > > > instead of upload to texture and we can add overlay, which may
> > > > also be 
> > > > in GPU memory, quicker. 
> > > This would require the decoders to either be able to decode
> > > directly 
> > > into GPU memory somehow or use GL... or it would be necessary for
> > > the 
> > > cairo GL backend to be able to map the GPU memory to normal
> > > system 
> > > memory for usage by other GStreamer elements. 
> > I believe he was thinking of extensions such as KHR_lock_surface[3],
> > which 
> > "allows mapping color buffers of EGL surfaces into the client
> > address space". 
> > 
> > Also, in the case of hardware decoders, I think we can often hope to
> > get the 
> > video decoded into GPU memory. When such a case happens, we
> > obviously want to 
> > keep buffers in the GPU until they are rendered. 
> > By the way, I am slightly confused: is there a "standardised" way
> > (with a 
> > GstMeta I guess) in GStreamer 1.0 of saying things like "this buffer
> > is in GPU 
> > memory" or "this function can give you this buffer mapped to an
> > opengl(es) 
> > texture"? 
> > For instance, the metas in gstreamer-vaapi[2] only seem to provide
> > info that is 
> > useful if you can "speak" vaapi. 
> No, there isn't a standardised way yet for notifying of GPU buffers
> yet. [1] contains a basic attempt at some kind of standard
> (GstGLMemory).  A more comprehensive approach should involve specific
> GstMeta for both the GPU context and GPU buffers.  There also needs to
> be a way of passing the GPU context between elements before buffers
> are created (query/event?) if we are to implement context sharing. 

At least for Linux it would probably make sense to have a generic
dma_buf meta that can be used to pass data between different devices
directly. And a corresponding GstMemory/GstAllocator.

The same should probably be done for EGLImages. Putting generic GL
"resources" into buffers is IMHO calling for trouble as it's not
guaranteed at all that they can be used in different threads.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20130113/897b7d5b/attachment.pgp>