[gst-devel] Video Hackfest conclusions

[Robert Krakora]

Message: 1
Date: Tue, 24 Nov 2009 12:47:38 +0100
From: Benjamin Otte <otte at gnome.org>
Subject: [gst-devel] Video Hackfest conclusions
To: gstreamer-devel at lists.sourceforge.net,      Cairo list
       <cairo at cairographics.org>
Cc: mesa3d-dev at lists.sourceforge.net, libva at lists.freedesktop.org,
       desktop-devel-list at gnome.org, xorg at lists.freedesktop.org
Message-ID:
       <a4ed255b0911240347g7c238c1fm507f9a051f3729f8 at mail.gmail.com
>
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi,
>
> As you may be aware, we held a video hackfest last week in Barcelona.
> Developers met to discuss how best to improve GPU support for video
> applications. See http://gstreamer.freedesktop.org/wiki/VideoHackfest
> for more details. In particular, you might be interested in the notes
> some people took while the hackfest was going on. These can be found
> at http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes
>
> What follows is the results we arrived at after the hackfest. They are
> taken from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions
> but certainly deserve wide circulation. We're interested in any
> comments (or questions) you might have about them, so please don't
> heistate to reply.
>
> If you do reply, please make sure to reduce the recipient list
> appropriately.
>
> Cheers,
> Benjamin,
>
> on behalf of all hackfest paticipants.
>
>
> == YUV in pixman ==
> It has been a goal for a while to make video a first-class citizen in
> the image stack. A concrete API proposal on how to integrate video
> into pixman was reached and reviewed.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/YCbCr_Formats
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixman
> Actions:
>  * GStreamer developers to provide a specification of important YUV formats
>  * Write implementation
>
> == YUV in Cairo ==
> To make it easy for applications to use these different formats, the
> Cairo API needs to be extended to allow them. A proposal does exist
> and has been reviewed, details need to be finalized.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo
> Actions:
>  * Finalize proposal
>  * Write implementation
>
> == locking in Cairo ==
> GStreamer takes a lot of freedom in deciding which threads to schedule
> elements in. Most hardware backends however require proper
> serialization of commands. The current gst-plugins-cairo and
> gst-plugins-gl code provide different, but ugly ways to achieve this.
> Cairo's internal thread safety guarantees match these requirements
> very well. But the different Cairo backends don't always keep these
> guarantees. Interaction with these locking mechanisms from outside
> applications is not possible yet either.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo
> Actions:
>  * Improve Cairo backend implementations, in particular X11 and GL
>  * Expose Cairo's locking API to allow interaction with it
>  * Write testcases to squash bugs
>
> == locking in Mesa ==
> The GLX specification does not allow binding the same GLX context in
> multiple threads at the same time. This is a requirement for both
> Cairo's and GStreamer's threading model. An extension was proposed and
> initial code developed to support this requirement in the same way as
> Apple's GL does by default. Windows does not support this and would
> require potentially expensive fallback code.
> Links:
>  * http://people.freedesktop.org/~anholt/MESA_multithread_makecurrent.spec<http://people.freedesktop.org/%7Eanholt/MESA_multithread_makecurrent.spec>
> Actions:
>  * Get review for suggested extension and include it in future Mesa
> releases
>  * Make Windows developers investigate the situation
>
> == switch GStreamer to Cairo as default video transport model ==
> The current approach to handling video in GStreamer is very outdated.
> It does not allow hw-accelerated buffers nor does it provide a unified
> API to modify video buffers, which leads to fragmentation and
> duplication. It was agreed that using Cairo to provide an abstraction
> as a drawing API and abstraction over different backends was a good
> idea.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes
>  * http://cgit.freedesktop.org/~company/gst-plugins-cairo<http://cgit.freedesktop.org/%7Ecompany/gst-plugins-cairo>
> Actions:
>  * Rework gst-plugins-cairo to match improvements listed in previous points
>  * Get more review on API and to avoid regressions
>  * Merge into gst-plugins-base
>  * Switch plugins gradually to use new Cairo code
>
> == gst-plugins-gl ==
> The general consensus was that gst-plugins-gl is a hack. It was
> necessary in the past to get things to work, but is not a good way
> forward. However, the functionality or performance provided by the
> current elements needs to continue working. Developers shared the
> opinion that gst-plugins-cairo with cairo-gl surfaces is the best way
> to achieve this.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes
> Actions:
>  * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl
> developers
>  * Port gst-plugins-gl elements to use Cairo early to ensure required
> features are available
>
> == XRenderPutImage ==
> Currently there is no way to upload video data to the X server for
> later use. The XV extension is not sufficient for anything more
> complicated than a simple video player.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo
> Actions:
>  * Figure out the best way to move YUV data into hw-accelerated Cairo
> surfaces (GL vs X)
>  * Possibly draft an extension to XRender to handle this in the no-GL case
>
> == JIT in Pixman ==
> ORC was investigated as a potential JIT for pixman. A lot of talk has
> happened, but no clear conclusions were reached. The idea to share a
> JIT with Mesa's GLSL stack was also brought up.
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes
> Actions:
>  * Continue investigating JIT options
>  * Prototype ORC integration into Pixman
>
> == Video decoding Acceleration ==
> Hardware video decoding is a feature that everybody is very excited
> about and wants to integrate into GStreamer as soon as possible. But
> developers feel that the current APIs available - the focus was on
> VAAPI and VDPAU - to do hardware acceleration fail to integrate well.
> No developer managed to create even demo-quality integration with
> gst-plugins-cairo, even though quite a few tried. In particular, both
> VDPAU and VAAPI...
>
> ... do not integrate with existing frameworks
> Both libraries fail to use (or allow easy conversion to) existing
> objects (GEM handles, GL textures, ...) but only provide access to an
> encapsulation. This makes it hard for a flexible framework like
> GStreamer to make use of its features without limiting itself.
>
> ... do not have a clear scope
> In particular VDPAU advertises itself as a video playback framework
> using hardware acceleration. Unfortunately, they both limit themselves
> to only the formats and functionalities supported by the hardware and
> see fullscreen video playback as the single usecase. This needlessly
> complicates integration with existing frameworks
>
> ... provide a very complicated API
> For decoding video, an API like provided by libtheora, libmpeg2 or
> even ffmpeg is very simple and allows easy integration with both the
> demuxing  and the postprocessing/display side. VAAPI and VDPAU require
> complex setups and preprocessing of the data to work properly.
>
> ... overlap with GL functionality
> It was felt that a lot of processing features provided by these
> libraries are equally well available using shaders in GL or by
> providing GL extensions. It was unclear why it is necessary to use a
> separate way to do this.
>
> Links:
>  * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes
>  *
> http://gstreamer.freedesktop.org/wiki/VideoHackfest?action=AttachFile&do=view&target=VAinGST.pdf
> Actions:
>  * Get in contact with developers of the APIs
>  * Encourage revision of public
>  * Work closer together in integrating the provided functionality into
> GStreamer
>
> = Timeline =
> The end goal of all of this is to get the code into users' hands
> quickly, but allow a smooth and non-disrupting upgrade. Desired dates
> have been put forth to achieve this:
>  * End of January: release new versions of Cairo and Pixman that
> contain the new APIs reuired by GStreamer
>  * End of March: X server and Mesa releases are due - make sure the
> required Mesa extension is part of this. Ideally XrenderPutImage would
> be included, too.
>  * April/May, after next Fedora/Ubuntu releases: Merge Cairo support
> into gst-plugins-base and start porting elements to it. Encourage
> application developers (browser, Flash players) to make use of the new
> APIs
>  * October: another Fedora/Ubuntu release that switches all users to
> the new APIs



Ben:

Thanks for your very insightful summary of the Video Hackfest and thanks to
the GStreamer team for a very well architected and extendible piece of
software.  Recently ffmpeg and MPlayer introduced VDPAU support which works
very well and allows one to employ lower cost CPUs in Linux based machines
that are to be used primarily for multimedia purposes.  Can anything be
learned from their integration of VDPAU that might benefit the GStreamer
team?  My end goal is to employ GStreamer with VDPAU support for video
decode with Fluendo codecs for audio decode.

Best Regards,

-- 
Rob Krakora
Senior Software Engineer
MessageNet Systems
101 East Carmel Dr. Suite 105
Carmel, IN 46032
(317)566-1677 Ext. 206
(317)663-0808 Fax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20091125/24ca4923/attachment.htm>