[gst-devel] GStreamer and gnome 2.2

[Thomas Vander Stichele]

Hi guys,

taaz asked me to send some info regarding the inclusion of GStreamer into
Gnome 2.2 and what that means for us.

Firstly, the current line of thought seems to be that GStreamer is going
to be included in one form or another in Gnome 2.2.  It mostly depends on
whether apps use it ;)

Second, there are four or five categories of "platforms" in Gnome 2.2.
Each one has different requirements and contents.

Development Platform
	- contains core libraries used by lots of apps
	- ABI/API stable for all of Gnome 2.x lifecycle

Desktop Platform
	- contains core apps, as well as some of the libraries used by them
	- probably has to be UI- and feature-stable during each 2.x cycle
	- less stringent ABI requirement, but don't do silly stuff

Fifth Toe (& Multimedia)
	- cool fringe gnome apps
	- likely candidates could be galeon, gnomemeeting, and probably others
	- could contain GStreamer-based apps like rhythmbox and have a
          multimedia edge

Powertools
	- a newly created platform for cool geek toys.


Now, where would GStreamer fit in ?

For now, we probably shouldn't try to get it in the Development Platform.
First of all, there's no real need, second, the requirements are a lot more
stringent and while we believe we may be able to fulfill them, it's probably
better not to stifle our development just because we are going into Gnome.
More likely than not, we will still uncover issues for which we have
better solutions when people really test the library more.
(This is not necessarily always bad ;) xine is just completing a complete
library API rewrite in the middle of their 0.9.x cycle I've been told)

Powertools is also not the place for us.  Currently it contains fringe
hardcore hacker tools and eyecandy ;)

So that leaves us with two platforms.  Of these, we were originally going
to go into fifth toe/multimedia.

Now, originally rhythmbox was going to go into FT(/MM), but for various
reasons jorn has decided to make rhythmbox more mature before putting it
into the core desktop.

The current line of thought is that we are going to go into Desktop Platform,
where we would be supporting (at the very least) iain's gnome media package.

The choice of if GStreamer would go into the Desktop Platform has to be made
in (roughly) the next two days.  Feature freeze and module list is 
scheduled
for Thursday 31st (which probably is Wednesday since it's on Oceanic time ;))

It depends on the applications using GStreamer that are proposed for the module
list.

So, here's a shortlist of possibilities that could be made ready for inclusion
in Gnome 2.2 :

1) gnome-media
  contains the sound recorder which iain has moved to GStreamer and
  rumour has it he is happy with the transition.  "no more sox" is a good
  reason too.
2) gst-player
  works with gstreamer, of course.  It is less HIG-compliant than totem
  and that is mostly a UI problem, but one that needs fixing on a pretty
  short notice.  the 31st is a module/feature freeze, so that still leaves
  us time to fix up the UI.
3) totem
  currently uses xine, but HIG-compliant.  Totem could be ported to use
  GStreamer, Bastien is all for it (but is not going to do it himself).
  I had offered to help out in that area, but I'm not sure I could get
  that done on such short notice with my still limited video skills ;)
  Someone here should outline the work that needs to be done to make this
  happen - from what I can see it shouldn't have to be too hard to support
  at least a first subset of what xine provides - ie, basic movie playback
  using libgstplay
4) a video thumbnailing application
  nautilus 2.2 features pluggable thumbnailing, which basically launches
  an app for each mime type it encounters.  So, for example, the mime type
  video/avi would use the thumbnailer "gst-snapshot %i %o" which would
  create a thumbnail png of the input video.
  This thumbnailing app is stupendously easy to write since basically
  you can pretty much do that using a gst-launch line with snapshot at this
  point.
  We have discussed changing snapshot's strategy because it depends on two
  libs instead of one.  We might investigate this a little so we can solve
  this properly, but it's easy to make this app happen in any case. 
5) nautilus-media
  there IS a music view in nautilus currently, but
  - it only does mp3
  - it contains a hacked-up version of (I believe) mpg123
  - Red Hat doesn't even ship it due to licensing issues
  - it is album-centered; it shows the dir, tries to pretend the dir is
    an album, and even has room for display of an album cover and so on.
    A bit overkill in a file browser perhaps.
  I started learning some nautilus and wrote an audio view for nautilus that
  basically works, but needs some improving UI-wise.  It'll focus on
  providing features that allow you to easily browse and scan through a bunch
  of songs.
  I intend to make roughly the same view for video as well, but with a
  fixed-size small video window in the view somewhere.
6) nautilus property pages
  James Willcox has created a system to have pluggable property pages as well
  in nautilus.  I'm currently focussing on getting the necessary stuff in
  place to have a streaminfo library that would fetch metadata and caps
  on files for display in properties and so on.
  My hope is to make this happen so that the property pages use these
  functions.  One of the planned features for this lib is to implement
  a streaminfo cache on-disk similar to the nautilus thumbnail cache.
  Wim convinced me that there is no problem in having GStreamer doing efficient
  handling of metadata and stream info, and after implementing it in vorbisfile
  I have to agree with him.
7) nautilus mouse-over play
  Alex Graveley from Ximian has worked on a nautilus hack that basically
  "drew" a video widget over the thumbnail of a video in the icon view
  in nautilus, and after 1 sec of mouseover the video itself would play
  "in the thumbnail".  That's a pretty cool hack ;) I hope to somehow
  get that functionality in nautilus somehow.  We've discussed it and
  it's actually in theory not that hard to do.  We have to figure out
  though how to provide this without making nautilus depend on GStreamer
  at compile-time for now.  James Willcox seems to be willing to
  see how a mechanism could be provided for nautilus to allow the "thumbnail
  widget" to be replaced with a video widget.  If we can pull this off,
  this will be a kick-ass eye candy feature.
8) rhythmbox
  is currently focusing on stability and some extra features and probably
  is not going to propose itself for inclusion anywhere until the groups
  have been implemented.  I hope they can get it to work as soon as possible,
  because rhythmbox rocks ;) but it's already been decided not to propose
  rhythmbox for the module list this week.  It might make a fifth toe
  or multimedia release.

So, that's a pretty good list.  Not all of them will make it, I think,
and I have to check up on when everything needs to be ready.  The upcoming
freeze this week is a "feature and module decision" freeze.  It means that
we should have a good idea of the features we want to provide in gnome 2.2,
and make sure we don't bite off more than we can chew.

So currently, what seems likely to me to get done is :

- (1) goes into desktop platform
- (2) gets proposed with commitment on our side to make it HIG-compliant
      (which means somebody has to READ it in the first place here - come on,
      don't be scared).  We will get help from usability guys (Calum, 
      Nils, etc...)
      This probably depends on Steve Baker's ability and willingness to
      make the changes and someone to assist him in translating the HIG to
      gst-player.  Someone should also just ask the usability guys to give
      us a UI review so we can start somewhere.
- (3) seems like a more long-term plan, especially since I am currently
      focussed on the other stuff
- (4) should be easy to write, and KConger (on irc) has done this over the 
      weekend.  I'll prod him to see how it's doing.  So I consider this
      "going in" as well, possibly as a part of nautilus-media.
- (5) the audio view part of nautilus can be made ready in relatively short
      time.  The biggest things to get right here are
      - writing a static autoplugger that actually knows when it can't play
        files
      - getting the streaminfo system basics in place so that it can
        do queries on the audio types we currently want to support
        (which I'm probably going to limit at this point to ogg, mp3, wav and
        flac)
- (6) I suggested to James to make these use the new streaminfo system and add
      these pages to nautilus-media, and he seemed to think that was a good
      idea, so if the streaminfo works, then these pages can also go in.
- (7) depends on James' ability to a) provide the mechanism and b) convince
      Alex Larsson and Dave Camp of the sensibility of it.
      I don't want to push James into doing it, it depends on his availability.
      I do hope he finds the time to help us out there though.
- (8) is not going in at this point and will need to be reconsidered in the
      future.  I hope to move the metadata handling in monkey-media over to
      GStreamer so that monkey-media itself doesn't need to depend on any
      decoding library and is thus fully pluggable.


Now, for the bad news.  What does this mean we should do in GStreamer ?

We currently have three major problems that need thought.
1) We cannot fully trust our current pthread schedulers.  We have seen
   mysterious crashes on reuse of pthreads, and the glibc people confirmed
   that you're not allowed to reuse the stack even after the thread using
   it has been joined.  For all uses we have had up to now, this isn't a big
   problem, but it seems that in the nautilus music view the code I currently
   use starts getting major problems when playing the 8th track.
   Having stack sizes of 2MB be wasted for each thread that gets created is
   thus probably not a good idea if you cannot reuse those 2MB in the same
   process ;)

   ways to fix this :
   - the optimal scheduler, which Wim claims is almost ready
   - move the omega scheduler to use GThreads.  David Schleef says this can
     be done easily, he just needs to change some API in the core for that to
     happen.
   - reuse pthreads instead of joining them.  This would make our use of
     pthreads more efficient and avoids crashes on stack reuse.
   - make stack space per thread smaller - this would imply making each
     cothread stack space a bit smaller, or allowing for less elements in one
     thread.  Both will probably work (the 2MB is largely arbitrary) but
     are workarounds, not fixes

2) Our most-used autoplugger, spider, has MAJOR issues.  Among these are
   - it locks up when given a file it can't play.  Try playing a text file
     in gst-player.
   - sometimes it outright crashes when doing a recursive function nesting
     because it can't figure out it can't play the file
   - it is currently unmaintained and Company has been around again but
     has not himself stated any willingness to work on it after careful
     prodding on our part ;)

   ways to fix this :
   - somebody bites the bullet and investigates the code.  All that needs to be
     added is an option to spider that tells it how many times it needs to
     loop the typefind, and set it to default 1.  After that it should
     send an element error.  Basically ;)
   - a static autoplugger; it would use typefind to determine the file type,
     then replace itself with a static pipeline a la gst-launch-ext.
     Currently the most feasible option.  Would even be sensible to write
     a fixed audio-only autoplugger.

3) being in gnome means having to run on all of the archs and platforms
   gnome supports.  I have doubts that out current thread code will work
   on every platform.  Jacob Berkman from Ximian has uncovered issues on
   mac osx, and I doubt it will reliably work on Sun.
   We need access to machines to test somehow and someone needs to set us
   up with that.
   Also, our code will have to become more portable, define out the
   arch-specific bits (preferably in one place and not all over the code),
   and generally be more robust.
   Sun is rumoured to be setting up a machine outside their firewall for
   testing, so we should be able to get access to that.
   We need a complete list of archs gnome supports and find the right 
   testers for it.
   I am going over our current set of media and writing up a simple script
   to run these media files through gstreamer so we can get usable 
   feedback on how well we do.

4) We want to maintain some form of ABI stability in our library.  Wim says
   he can get this done in about a week, which means it would be done
   by the time the second round of tarballs needs to go in.  Jeff Waugh
   is ok with that.

5) We need to do and/or figure out the following things :
   - proper library versioning from now on
   - define where all of our headers will get installed
     currently, it's (prefix)/include/(module)-(version)/gst
     My current idea is to bump us up to 0.5.0, and put ALL headers in
     include/gstreamer-0.5 for the whole of the 0.5 devel tree used by gnome
     2.1.x.
     We use this tree to work up to a "stable" 0.6 tree for gnome 2.2
   - it would also mean that plugin libraries also put their headers in
     gstreamer-0.5 instead of gst-plugins-0.5 like now.
   - the library base name would also be libgstreamer-0.5.so
     Basically, we either drop the micro number or drop the whole "version
     in the name" scheme
   - I will check these changes with experienced hackers who've seen this
     before like owen, hp, jody and others.

6) We'll be doing a prerelease today for a version to be included this 31st.
   We need this for iain's wavenc plugin.  Basically, I could also do
   a separate tarball with the plugin and throw it in with the 0.4.1 release,
   but it'd be good to have a new release.  It will depend on some of the
   features that have been added, and Uraeus is of course going to help out ;)

So, a big set of possible changes.  Exciting times are ahead, but PLEASE
comment on this mail because it WILL involve some changes and we need to
make sure we understand what this will involve.

On the plus side, we will be providing a cool desktop with great features
and we will put the wonder that is GStreamer to the test ;)

It's also good to mention that this is a major addition to Gnome 2.2, will 
likely be a controversial issue, will get lots of discussion.  We might 
not make Gnome 2.2 desktop platform, but that's no reason to be 
dissapointed if we don't.  We still have the Fifth Toe release which we 
can go in, and we can use this opportunity to check if we can make the 
proper commitments and put GStreamer to the test.

Steady on,
Thomas

-- 

The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/
<-*- thomas (dot) apestaart (dot) org -*->
I'd like to tell you how I feel
I'll probably keep it till a Saturday
<-*- thomas  (at) apestaart (dot) org -*->
URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/