[gst-devel] Re: Common Opensource codec API

[Artem Baguinski]

On Mon, Dec 29, 2003 at 10:22:29PM +0200, Tuukka Toivonen wrote:
> >Beside imho it wouldn't be a good idea. The power of opensource
> >comes from diversity, if you restrict that by using a common plugin
> 

reminds me of:

  "once I thought that his was, not the Adamic language that a happy
  mankind had spoken, all unified by a single tongue from the origin of
  the world to the tower of Babel, or one of the languages that arose
  after the dire event of their division, but precisely the Babelish
  language of the first day after the divine chastisement, the language
  of primal confusion."

although i don't have much faith in universal solutions either. if you
look at my ~/.signature you'll notice i try to speak a bit Babelian
myself ;-)

there was a meeting a couple of month ago in Bergen Norway
(http://www.piksel.no/) where real time video artists and coders
exchanged some ideas on make their software interoperable, not uniform.
but one of the conclusions was to work on common effect plugins
architecture similar to that of LADSPA in audio domain. the project is
located at http://savannah.nongnu.org/projects/piksel/ only part of
piksel framework is common effect plugins, there is also some [less
relevant for this discussion] work going on. 

i'm afraid i'm a bit too late but there's also piksel mailing list
(http://plot.bek.no/mailman/listinfo/piksel) concerned with open source
tools for real time video processing and among other issues problems of
their interoperability and avoiding double work. i though [too late?] the
discussion of common codec architecture could happen there.

> If this diversity means that things just don't work, I disagree. And I feel
> that this would be the result, because there are devices that produce video
> in their own formats. Before any application can do anything with
> these devices, they must understand the format. This can be achieved by
> either doing format conversion in the kernel--which isn't a good idea nor
> even allowed by the master kernel hackers--or by a plugin system that is
> adhered by all applications.
> 
> A plugin must be delivered along with a device driver, but it makes no
> sense to make one for each application. That would defeat the whole purpose
> of device-independent API of drivers.
> 
> (Yet another method would be vloopback device, but I don't know much about
> it, wouldn't it be quite inefficient?)

vloopback is linux only driver that allows a software to act like a
video4linux device. among other things this allows construction of video
processing pipelines. it is:

- not maintained anymore
- not portable 
- loads the kernel with things it shouldn't be doing

its purposes were:

- to make it possible to feed any video into programs that only expect
  v4l input
- to add some pre/post processing to v4l devices outside of their
  drivers [so that any "in software" processing happens outside the
  driver and can be used with virtually any device]. 

success stories of vloopback use:
- exposing firewire camera as a video4linux device to xawtv [by some
  bizarre reason different firewire cameras provide two different APIs
  then video4linux, i'm not the only "lost in Babilon", it seems] 
- sending output of my video processing application to ffmpeg and
  streaming the encoded with ffserver

not sure how vloopback's relevant to the discussion [my explanation aims
to show its irrelevance ;)]

> >interface code, so porting code from one interface to another
> >(w/o optimizing to the new interface) is quite easy.
> 
> Maybe, if somebody does it. In many cases, nobody would do it, and there
> would be cases "this web camera is supported by MEncoder but not xawtv".
> Too little devices are supported in Linux anyway.

yep, and 

- this camera is firewire dv camera, it has this interface
- that camera is firewire dc camera, it has that interface
- yet another camera is usb webcam, it has yet another interface

but this is more a problem of the kernel i think. either video4linux API
weren't good enough for firewire or people how work on firewire drivers
were too creative...

may be the solution though is another layer, some video access library,
that would provide uniform access to various video inputs and having OS
dependant backends. e.g. OpenML (http://www.khronos.org/openml/) API
implementation that wraps existing APIs... analogous to mesa as it was
in the beginning [emulating OpenGL in software and allowing to render to
not-OpenGL capable hardware or drivers providing no OpenGL support for
capable hardware].

again, a nice issue to discuss but i dont see how it relates to the
Subject...

> >The only place where a common api makes imho sense are the codecs
> >as this code is quite complex and only a few people have enough
>
> Hmm, maybe I'm ignorant, but if there's a common API for codecs why not
> other even simpler things too?

i can name three video applications with very different requirements to
"other simpler things": 

- non-linear editing
- "normal" video playback
- realtime video manipulation

it's quite difficult to find common denominator for all three,
especially the universal one which would allow simple Lego like front
end coding. 

on the other hand. projects like mplayer and xine has started with the
only goal in mind - to provide an open source media player that will
play all video sources, play it good and not crash. that's a noble goal
and the projects are quite successful in moving towards it.

the projects have developed the infrastructure and internal plugins APIs
optimized for "normal video playback", perfect audio/video
synchronization, onscreen display, subtitles support... any attempt to
find the least common denominator, invent common API and retrofit the
existing sucesful players to it would require a lot of work and will
make xine not xine anymore and mplayer - not mplayer. and their only
commons will ensure the new API is perfect for video playback and
useless [or hard to use] for other applications. 

is such result worth the hassle?

> >knowledge to handle that stuff properly. But there, we already have
> >a quite common api for codecs: libavcodec

and why every project includes its "personal" copy of the api's
implementation then?

> Yeah, and Gstreamer. Probably others too. Multiplicity is the problem.

Gstreamer isn't a codec api, it's a glue layer implented in C.

> >> >What I'd love to see is the codec API not being an actual lib, but just
> >> >a protocol, like X.
> >That's imho a quite bad idea. If you have ever written anything
> >time critical with using X11 than you know how much time you loose
> >just for the protokol conversion. Nevertheless X11 is a great protokol

are you sure "codec" is the correct word? i think the uniform media
access layer is missing, in xine terms - input, demux and codec part. 

i really feel the need for such layer for the new open source media
applications, not for trying to recreate existing ones. 

i guess before trying to invent it people should look at OpenML i
suggest. not for the implementation but for the protocol.

-- 
gr{oe|ee}t{en|ings}
artm