No subject
Wed Dec 2 15:13:03 PST 2009
<br>
"The GLX specification does not allow binding the same GLX context in
multiple threads at the same time. This is a requirement for both
Cairo's and GStreamer's threading model. An extension was proposed =
and
initial code developed to support this requirement in the same way as
Apple's GL does by default. Windows does not support this and would
require potentially expensive fallback code."<br><br></div>-->The o=
ther solution would be to use OpenGL context sharing. But it requires to ha=
ve this:<br><br>(from <a href=3D"http://gstreamer.freedesktop.org/wiki/Vide=
oHackfest/Notes" target=3D"_blank">http://gstreamer.freedesktop.org/wiki/Vi=
deoHackfest/Notes</a>)<br>
"Eric: One thing I'd like is if two neighboring elements could <sp=
an></span>determine whether they are running in the same thread"<br><b=
r>I already asked gstreamer developers in the past to add support for this =
but this is not an easy task and only useful for opengl right now.<br>
The result would be to remove this :<br>(from <a href=3D"http://gstreamer.f=
reedesktop.org/wiki/VideoHackfest/Notes" target=3D"_blank">http://gstreamer=
.freedesktop.org/wiki/VideoHackfest/Notes</a>)<br>"So what gst-plugins=
-gl does is to proxy all OpenGL communication through a <span></span>single=
thread associated with the context"<br>
<br>For example in the pipeline:<br>videotestsrc ! glupload ! glfilterA ! q=
ueue ! glfilterB ! glimagesink <br>It would have one opengl context for glu=
pload ! glfilterA since their chain func is running in the same thread (if =
the user does not manually push buffer itself form an other thread, I do no=
t know all the possible case but we could ensure that they are running in t=
he same thread).<br>
This opengl context would be shared with the opengl context used by glfilte=
rB ! glimagesink<br>Moreover, thoses 2 opengl contexst (shared together) co=
uld work at the
same time (they would work on different textures at the same time)<br><br>A=
nd so no need the overhead due to the gl thread is avoid, (and the property=
"each opengl context is made current only one time" is kept. Not=
e that this property is already true in the current gst-plugins-gl implemen=
tation) <br>
<br>I have already demonstrated the validity of using OpenGL context sharin=
g in the implementation of the glmixer base class.<br>The glmosaic element =
(which will be renamed to glmixercube) is a glmixer. (availaible in gst-plu=
gins-gl/git)<br>
For example in the following pipeline:<br><br>videotestsrc ! glupload ! glf=
ilterA ! queue ! glmixer name=3Dm ! glfilterC ! glimagesink<br>=A0=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 glupload ! glfilterB ! queue =
! .m<br><br>There are 3 opengl contexts shared together. (there is a glthre=
ad per opengl context)<br>
One used by glupload ! glfilterA, one other for glupload ! glfilterB, and a=
last one used by glmixer=A0 ! glfilterC ! glimagesink<br><br>Finally, if w=
e would have the following support: <br>"Eric: One thing I'd like =
is if two neighboring elements could <span></span>determine whether they ar=
e running in the same thread"<br>
We could avoid the use of the a thread per opengl context.<br>Moreover it w=
ould not need what you suggested:<div class=3D"im"><br>"The GLX specif=
ication does not allow binding the same GLX context in
multiple threads at the same time"<br><br></div>Well, internally, mayb=
e the mecanism/result is exactly the same:<br>I mean, maybe binding the sam=
e GLX context in
multiple threads at the same time is equivalent to have several opengl cont=
exts shared together and each one used in only one thread (a gstreamer stre=
aming thread).<br>If this is really equivalent (at least for our use) maybe=
it would be more portable to use the context sharing solution, because the=
feature is already there.<br>
<br>Actually I think using opengl context sharing is more efficient because=
several context can be used at the same time. (not possible with the same =
context in multiple-thread. Well it depends on the implementation that you =
are going to do)<br>
<br>I could help in the future about that:<br>
(from <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclu=
sions" target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfes=
t/Conclusions</a>)<br><div class=3D"im">"Actions: <span></span><ul><li=
>
Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developer=
s <span></span></li>
<li>Port gst-plugins-gl elements to use Cairo early to ensure required feat=
ures are available"</li></ul><br></div>Sincerely<br><font color=3D"#88=
8888">Julien Isorce<br></font><br><br><div class=3D"gmail_quote">
2009/11/24 Benjamin Otte <span dir=3D"ltr"><<a href=3D"mailto:otte at gnome=
.org" target=3D"_blank">otte at gnome.org</a>></span><div><div></div><div c=
lass=3D"h5"><br><blockquote class=3D"gmail_quote" style=3D"border-left: 1px=
solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
As you may be aware, we held a video hackfest last week in Barcelona.<br>
Developers met to discuss how best to improve GPU support for video<br>
applications. See <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHac=
kfest" target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfes=
t</a><br>
for more details. In particular, you might be interested in the notes<br>
some people took while the hackfest was going on. These can be found<br>
at <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes" ta=
rget=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes</=
a><br>
<br>
What follows is the results we arrived at after the hackfest. They are<br>
taken from <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/C=
onclusions" target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHa=
ckfest/Conclusions</a><br>
but certainly deserve wide circulation. We're interested in any<br>
comments (or questions) you might have about them, so please don't<br>
heistate to reply.<br>
<br>
If you do reply, please make sure to reduce the recipient list appropriatel=
y.<br>
<br>
Cheers,<br>
Benjamin,<br>
<br>
on behalf of all hackfest paticipants.<br>
<br>
<br>
=3D=3D YUV in pixman =3D=3D<br>
It has been a goal for a while to make video a first-class citizen in<br>
the image stack. A concrete API proposal on how to integrate video<br>
into pixman was reached and reviewed.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/YCbCr_F=
ormats" target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfe=
st/YCbCr_Formats</a><br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixman"=
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixm=
an</a><br>
Actions:<br>
=A0* GStreamer developers to provide a specification of important YUV forma=
ts<br>
=A0* Write implementation<br>
<br>
=3D=3D YUV in Cairo =3D=3D<br>
To make it easy for applications to use these different formats, the<br>
Cairo API needs to be extended to allow them. A proposal does exist<br>
and has been reviewed, details need to be finalized.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo=
</a><br>
Actions:<br>
=A0* Finalize proposal<br>
=A0* Write implementation<br>
<br>
=3D=3D locking in Cairo =3D=3D<br>
GStreamer takes a lot of freedom in deciding which threads to schedule<br>
elements in. Most hardware backends however require proper<br>
serialization of commands. The current gst-plugins-cairo and<br>
gst-plugins-gl code provide different, but ugly ways to achieve this.<br>
Cairo's internal thread safety guarantees match these requirements<br>
very well. But the different Cairo backends don't always keep these<br>
guarantees. Interaction with these locking mechanisms from outside<br>
applications is not possible yet either.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo=
</a><br>
Actions:<br>
=A0* Improve Cairo backend implementations, in particular X11 and GL<br>
=A0* Expose Cairo's locking API to allow interaction with it<br>
=A0* Write testcases to squash bugs<br>
<br>
=3D=3D locking in Mesa =3D=3D<br>
The GLX specification does not allow binding the same GLX context in<br>
multiple threads at the same time. This is a requirement for both<br>
Cairo's and GStreamer's threading model. An extension was proposed =
and<br>
initial code developed to support this requirement in the same way as<br>
Apple's GL does by default. Windows does not support this and would<br>
require potentially expensive fallback code.<br>
Links:<br>
=A0* <a href=3D"http://people.freedesktop.org/%7Eanholt/MESA_multithread_ma=
kecurrent.spec" target=3D"_blank">http://people.freedesktop.org/~anholt/MES=
A_multithread_makecurrent.spec</a><br>
Actions:<br>
=A0* Get review for suggested extension and include it in future Mesa relea=
ses<br>
=A0* Make Windows developers investigate the situation<br>
<br>
=3D=3D switch GStreamer to Cairo as default video transport model =3D=3D<br=
>
The current approach to handling video in GStreamer is very outdated.<br>
It does not allow hw-accelerated buffers nor does it provide a unified<br>
API to modify video buffers, which leads to fragmentation and<br>
duplication. It was agreed that using Cairo to provide an abstraction<br>
as a drawing API and abstraction over different backends was a good<br>
idea.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes=
</a><br>
=A0* <a href=3D"http://cgit.freedesktop.org/%7Ecompany/gst-plugins-cairo" t=
arget=3D"_blank">http://cgit.freedesktop.org/~company/gst-plugins-cairo</a>=
<br>
Actions:<br>
=A0* Rework gst-plugins-cairo to match improvements listed in previous poin=
ts<br>
=A0* Get more review on API and to avoid regressions<br>
=A0* Merge into gst-plugins-base<br>
=A0* Switch plugins gradually to use new Cairo code<br>
<br>
=3D=3D gst-plugins-gl =3D=3D<br>
The general consensus was that gst-plugins-gl is a hack. It was<br>
necessary in the past to get things to work, but is not a good way<br>
forward. However, the functionality or performance provided by the<br>
current elements needs to continue working. Developers shared the<br>
opinion that gst-plugins-cairo with cairo-gl surfaces is the best way<br>
to achieve this.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes=
</a><br>
Actions:<br>
=A0* Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl deve=
lopers<br>
=A0* Port gst-plugins-gl elements to use Cairo early to ensure required<br>
features are available<br>
<br>
=3D=3D XRenderPutImage =3D=3D<br>
Currently there is no way to upload video data to the X server for<br>
later use. The XV extension is not sufficient for anything more<br>
complicated than a simple video player.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo=
</a><br>
Actions:<br>
=A0* Figure out the best way to move YUV data into hw-accelerated Cairo<br>
surfaces (GL vs X)<br>
=A0* Possibly draft an extension to XRender to handle this in the no-GL cas=
e<br>
<br>
=3D=3D JIT in Pixman =3D=3D<br>
ORC was investigated as a potential JIT for pixman. A lot of talk has<br>
happened, but no clear conclusions were reached. The idea to share a<br>
JIT with Mesa's GLSL stack was also brought up.<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes=
</a><br>
Actions:<br>
=A0* Continue investigating JIT options<br>
=A0* Prototype ORC integration into Pixman<br>
<br>
=3D=3D Video decoding Acceleration =3D=3D<br>
Hardware video decoding is a feature that everybody is very excited<br>
about and wants to integrate into GStreamer as soon as possible. But<br>
developers feel that the current APIs available - the focus was on<br>
VAAPI and VDPAU - to do hardware acceleration fail to integrate well.<br>
No developer managed to create even demo-quality integration with<br>
gst-plugins-cairo, even though quite a few tried. In particular, both<br>
VDPAU and VAAPI...<br>
<br>
... do not integrate with existing frameworks<br>
Both libraries fail to use (or allow easy conversion to) existing<br>
objects (GEM handles, GL textures, ...) but only provide access to an<br>
encapsulation. This makes it hard for a flexible framework like<br>
GStreamer to make use of its features without limiting itself.<br>
<br>
... do not have a clear scope<br>
In particular VDPAU advertises itself as a video playback framework<br>
using hardware acceleration. Unfortunately, they both limit themselves<br>
to only the formats and functionalities supported by the hardware and<br>
see fullscreen video playback as the single usecase. This needlessly<br>
complicates integration with existing frameworks<br>
<br>
... provide a very complicated API<br>
For decoding video, an API like provided by libtheora, libmpeg2 or<br>
even ffmpeg is very simple and allows easy integration with both the<br>
demuxing =A0and the postprocessing/display side. VAAPI and VDPAU require<br=
>
complex setups and preprocessing of the data to work properly.<br>
<br>
... overlap with GL functionality<br>
It was felt that a lot of processing features provided by these<br>
libraries are equally well available using shaders in GL or by<br>
providing GL extensions. It was unclear why it is necessary to use a<br>
separate way to do this.<br>
<br>
Links:<br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes" =
target=3D"_blank">http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes=
</a><br>
=A0* <a href=3D"http://gstreamer.freedesktop.org/wiki/VideoHackfest?action=
=3DAttachFile&do=3Dview&target=3DVAinGST.pdf" target=3D"_blank">htt=
p://gstreamer.freedesktop.org/wiki/VideoHackfest?action=3DAttachFile&do=
=3Dview&target=3DVAinGST.pdf</a><br>
Actions:<br>
=A0* Get in contact with developers of the APIs<br>
=A0* Encourage revision of public<br>
=A0* Work closer together in integrating the provided functionality into GS=
treamer<br>
<br>
=3D Timeline =3D<br>
The end goal of all of this is to get the code into users' hands<br>
quickly, but allow a smooth and non-disrupting upgrade. Desired dates<br>
have been put forth to achieve this:<br>
=A0* End of January: release new versions of Cairo and Pixman that<br>
contain the new APIs reuired by GStreamer<br>
=A0* End of March: X server and Mesa releases are due - make sure the<br>
required Mesa extension is part of this. Ideally XrenderPutImage would<br>
be included, too.<br>
=A0* April/May, after next Fedora/Ubuntu releases: Merge Cairo support<br>
into gst-plugins-base and start porting elements to it. Encourage<br>
application developers (browser, Flash players) to make use of the new<br>
APIs<br>
=A0* October: another Fedora/Ubuntu release that switches all users to<br>
the new APIs<br>
<br>
---------------------------------------------------------------------------=
---<br>
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day=
<br>
trial. Simplify your report design, integration and deployment - and focus =
on<br>
what you do best, core application coding. Discover what's new with<br>
Crystal Reports now. =A0<a href=3D"http://p.sf.net/sfu/bobj-july" target=3D=
"_blank">http://p.sf.net/sfu/bobj-july</a><br>
_______________________________________________<br>
gstreamer-devel mailing list<br>
<a href=3D"mailto:gstreamer-devel at lists.sourceforge.net" target=3D"_blank">=
gstreamer-devel at lists.sourceforge.net</a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/gstreamer-devel" ta=
rget=3D"_blank">https://lists.sourceforge.net/lists/listinfo/gstreamer-deve=
l</a><br>
</blockquote></div></div></div><br>
</div><br>
--00032555b2f29c370a0479437924--
More information about the Libva
mailing list