[st-ericsson] v4l2 vs omx for camera

[Edward Hervey]

Hi,

On Fri, 2011-02-18 at 17:39 +0100, Robert Fekete wrote:
> Hi,
> 
> In order to expand this knowledge outside of Linaro I took the Liberty
> of inviting both linux-media at vger.kernel.org and
> gstreamer-devel at lists.freedesktop.org. For any newcomer I really
> recommend to do some catch-up reading on
> http://lists.linaro.org/pipermail/linaro-dev/2011-February/thread.html
> ("v4l2 vs omx for camera" thread) before making any comments. And sign
> up for Linaro-dev while you are at it :-)
> 
> To make a long story short:
> Different vendors provide custom OpenMax solutions for say Camera/ISP.
> In the Linux eco-system there is V4L2 doing much of this work already
> and is evolving with mediacontroller as well. Then there is the
> integration in Gstreamer...Which solution is the best way forward.
> Current discussions so far puts V4L2 greatly in favor of OMX.
> Please have in mind that OpenMAX as a concept is more like GStreamer
> in many senses. The question is whether Camera drivers should have OMX
> or V4L2 as the driver front end? This may perhaps apply to video
> codecs as well. Then there is how to in best of ways make use of this
> in GStreamer in order to achieve no copy highly efficient multimedia
> pipelines. Is gst-omx the way forward?
> 
> Let the discussion continue...
> 

  I'll try to summarize here my perspective from a GStreamer point of
view. You wanted some, here it is :) This is a summary to answering
everything in this mail thread at this time. You can go straight to the
last paragraphs for a summary.

  The question to be asked, imho, is not "omx or v4l2 or gstreamer", but
rather "What purpose does each of those API/interface serve, when do
they make sense, and how can they interact in the most efficient way
possible"

  Looking at the bigger picture, the end goal to all of us is to make
best usage of what hardware/IP/silica is available all the way up to
end-user applications/use-cases, and do so in the most efficient way
possible (whether in terms of memory/cpu/power usage at the lower
levels, but also in terms of manpower and flexibility at the higher
levels).

  Will GStreamer be as cpu/memory efficient as a pure OMX solution ? No,
I seriously doubt we'll break down all the fundamental notions in
GStreamer to make it use 0 cpu when running some processing.

  Can GStreamer provide higher flexibility than a pure OMX solution ?
Definitely, unless you have all the plugins for accesing all other hw
systems out there,  (de)muxers, rtp (de)payloaders, jitter buffers,
network components, auto-pluggers, convenience elements, application
interaction that GStreamer has been improving over the past 10 years.
All that is far from trivial.
  And as Rob Clark said that you could drop HW specific gst plugins in
and have it work with all existing applications, the same applies to all
the other peripheral existing *and* future plugins you need to make a
final application. So there you benefit from all the work done from the
non-hw-centric community.

  Can we make GStreamer use as little cpu/overhead as possible without
breaking the fundamental concepts it provides ? Definitely.
  There are quite a few examples out there of zero-memcpy gst plugins
wrapping hw accelerated systems for a ridiculous amount of cpu (they
just take a opaque buffer and pass it down. That's 300-500 cpu
instructions for a properly configured setup if my memory serves me
right). And efforts have been going on for the past 2 years to carry on
to make GStreamer overall consume as little cpu as possible, making it
as lockless as possible and so forth. The undergoing GStreamer 0.11/1.0
effort will allow breaking down even more barriers for even more
efficient usage.

  Can OMX provide a better interface than v4l2 for video sources ?
Possible, but doubtful, The V4L2 people have been working at it for ages
and works for a *lot* of devices out there. It is the interface one
expects to use on Linux based systems, you write your kernel drivers
with a v4l2 interface and people can use it straight away on any linux
setup.

  Do Hardware/Silica vendors want to write kernel/userspace drivers for
their hw-accelerated codecs in all variants available out there ? No
way, they've got better things to do, they need to chose one.
  Is OMX the best API out there for providing hw-accelerated codecs ?
Not in my opinion. Efforts like libva/vdpau are better in that regards,
but for most ARM SoC ... OMX is the closest thing to a '''standard'''.
And they (Khronos) don't even provide reference implementations, so you
end up with a bunch of header files that everybody {mis|ab}uses.



  So where does this leave us ?

  * OMX is here for HW-accelerated codecs and vendors are doubtfully
going to switch from it, but there are other system popping up that will
use other APIs (libva, vdpau, ...).
  * V4L2 has an long standing and evolving interface people expect for
video sources on linux-based systems. Making OMX provide an
as-robust/tested interface as that is going to be hard.
  * GStreamer can wrap all existing APIs (including the two mentionned
above), adds the missing blocks to go from standalone components to
full-blown future-looking applications/use-cases.

  * The main problem... is making all those components talk to eachother
in the most cpu/mem efficient way possible.

  No, GStreamer can't solve all of that last problem. We are working
hard on reducing as much as possible the overhead GStreamer brings in
while offering the most flexible solution out there and you can join in
making sure the plugins exposing those various APIs mentionned above
make the best usage of it. There is a point where we are going to reach
our limit.

  What *needs* to be solved is an API for data allocation/passing at the
kernel level which v4l2,omx,X,GL,vdpau,vaapi,... can use and that
userspace (like GStreamer) can pass around, monitor and know about.
  That is a *massive* challenge on its own. The choice of using
GStreamer or not ... is what you want to do once that challenge is
solved.

  Regards,

    Edward

P.S. GStreamer for Android already works :
http://www.elinux.org/images/a/a4/Android_and_Gstreamer.ppt