[gst-devel] Plugin split FINAL VOTE

[Erik Walthinsen]

OK, here's the deal:  We've come to the conclusion that there is no other
plugin split that makes sense (at least that we can figure out), but have
also discussed the specific usefulness of the top-level directories
currently proposed.  So thomasvs and I have summarized our positions, and
we'll all vote.  Whatever is voted is final, and we'll start on it
immediately.

Voting is open to anyone who cares, and votes must be mailed to the list
in the next 24 hours (from the timestamp in this mail).  The vote is a
simple boolean, no comments allowed.

Make sure you *READ* Thomas's proposal, as well as both his and my
summaries, before you decide.

Votes are:

dirs	Use gst/sys/ext directories for plugins
flat	Use flat arrangement for plugins


The proposal (thomasvs):
======================================================================
My dir proposal intends to solve two basic problems :
a) where does one put a plugin unambiguously ?
b) how does one name the dir where the plugins end up ?

I think we all agree that the current layout is lacking in both regards;
take filters and effects as an example.  Also, the dir names themselves
are messy, why have mp3decode/mp3encode next to vorbis, for example.

gst-plugins/
	gst/	self-contained plugins, no lib dependencies (other than
		glib and optional XML), build out of the box

		Naming of the dirs should be fairly standard but up to
		the plugin developer, hopefully incorporating some sense
		of function and type.  If it's an enc *only*, name it
		that.  If it's a src *only*, name it that.  Only when
		the code allows for sharing between enc/dec or src/sink
		should you drop it.
		prime examples are wavparse, adder, volume, ...

	sys/	self-contained as well, but system-dependant.  Typically
		these plugins need a kernel header or a specific kernel
		feature, or even a specific architecture.  No external
		lib dependency however

		Naming should be same as in gst, but please choose
		something that references the system function you're
		using.

		prime examples are oss, v4l, vgasink

	ext/	plugins that cannot function without the corresponding
		third-party library.

		Name the dir according to the canonical name of the lib,
		and let's try to give test variables the same name and
		stuff.  Makes it easy for writing build scripts.

		prime examples are the colorspace plugin (which
		should probably be renamed to hermes and put in the
		dir hermes, but I'm still not sure if colorspace does
		anything without hermes, and if it does, please split it)

	gst-libs/	Our own libraries used for specific plugins.
		audio is a good example; it should contain routines
		for standard frame rate/length/... calculation.

	ext-libs/	external and UNSTABLE libraries.  People have
		argued that, as long as a lib we depend on has not yet
		had a stable release, we should include the source to it.
		This way we can keep the plugin and the lib in sync and
		make sure the plugin functions.
		As soon as a lib HAS a stable release, we can remove it
		from this dir.

		Examples : mplib, avifile

	docs/, tools/, scripts/, .... : the usual, no discussion necessary ;)

At first I was afraid that I would have a hard time splitting the plugins
up for real like this, as has been the case with all previous schemes.
But bear in mind that I actually did it, and the tarball is sitting on my
laptop and on the net, so it's possible.  And for about 100 dirs I only
got two comments of wrong placements which were easily fixed.  So it is
possible.
======================================================================


thomasvs's summary:
======================================================================
To make it easier on the quick deciders, I've marked my sentences with
"*"   for This explains my reasoning behind this
"**"  this is a point to consider
"***" this is the crux of the matter

I've chosen to give a lot of explanation because I did over the last few
days anyway and I don't like it when people miss the point I make because
I want to be terse.  So if you care deeply, read it all.  If you do not
care but just want to decide, read the "*" paragraphs ;)

** As the one who got very intimate with the build structure of GStreamer
throughout the last two weeks, I find our layout very lacking.  I've spent
a few days on tracking libraries.  I still don't have a few libraries, I
still don't know a few of the plugins we have, there are even plugins that
didn't ever compile and should probably have been killed long ago.

* For example, I learned that we had FOUR mp3 decoders; that the lib for
one of them (xaudio) isn't downloadable atm because the "site is under
construction", that xing - which everyone knows sucks - was still a
plugin, and so on.  I am still missing libraries but I'm not sure for what
plugins because probably a few have never even been included in the build
system.  Just to say that the current layout makes it really hard on a
build maintainer and that's probably exactly the reason <boldface>Why no
one was maintaining it</boldface>.

*** Now, we need to consider three typical people in deciding this layout
and the problems/needs they solve for them.

a) an old hat gstreamer developer :
  * If he's been at it long enough, he might have an idea of about 60-70 %
  of the plugins.  Just like me, he probably hasn't found a few of the
  unmaintained ones which could maybe easily be polished up.
  *** As for if he will find his way in the new layout, I think having all
  the plugins on the SAME hierarchy level will make it easy, and having
  spent time on tracking down libs for plugins, he'll probably already
  know what libs are required for what plugins and thus find his way.

b) the build maintainer :
  ** Speaking from personal experience, the argument that bothers me the
  most
  against my proposal is "it doesn't give us any noticeable advantage."
  The us here means "me".  And I totally agree with you : you DON'T see
  the advantage because NO ONE here likes touching the build stuff unless
  it's with a ten feet pole wearing medieval armour.  This approach causes
  even more breakage every step we take further.  So I dove in and while
  working at it for FOUR DAYS, I had ample time to think about how a good
  dir layout would make my life easier.
  *** I'm sure you all appreciate the
  end-of-configure status layout describing what you will and won't get.
  It's just one of the things that is simple to do when the layout makes
  sense.

  * I suppose it is hard for you guys to understand WHY it would make my
  maintainer life easier, but you can gain some perspective by taking
  the 0.2.1 tarball and trying to fix it's build, just for fun.

  * In short : it is easy for me to write scripts to check if I have all
  the dependencies mapped out if the ones with dependencies are in the
  same dir.  It is easy for me to get all the vars in all the scripts
  consistent if I have something set to base myself on (as opposed to
  the case where one has _LAME in configure.ac and _LIBLAME in Makefile.am
  and you spend more than an hour tracking it down because autogenning
  takes so long.  And it's easy for me to set up automatic building tests
  on my few machines, on machines zaheer is going to offer, and machines
  of friends with exotic archs (who I can't even recommend gstreamer
  because it would be a full time job to help them get the deps right)

  ** The one argument against this offered is that you can do the same
  with little files in each dir giving package and dependency hints.
  *SIGH* Yes, you're right, that would work.  So why isn't it there ?

  *** You were all not even touching the build and all of a sudden you are
  going to create a new format for maintenance HINTS, and fill each of
  the 100 dirs with these files, and expect not to miss one ?

  * Will each plugin writer actually do this ? No, because he didn't even
  bother to get tests in configure.ac and get symbols right and stuff, so
  why would he write this HINTS file ?
  Why not take the simple way so we get it done instead of spouting ideas ?
  *** Why not use the implied maintenance HINTS you get from a sensible
  dir layout ?

  *** So, once and for all, the advantage you gain that you don't see
  atm is
  simple : if you choose this sensible layout then you make my job as
  a maintainer a whole lot EASIER since I have a picture of where I'm
  going with the build, and I can ensure a lot better that the build
  will WORK and not get in your way as a developer.  I think that,
  given the past, that is a really nice advantage to have.

  * Please don't
  try to solve the maintenance problem by putting MORE work in the hands
  of developers and maintainers who have to generate more little files
  everywhere.  It's just NOT going to happen and you know it.


c) a new developer :
  *** The learning curve for gstreamer is really high.

  * As Michael Meeks
  pointed out and experienced by me personally, you see the power when you
  read the docs, but it takes quite some time before you actually get a
  part of gstreamer to a working state.  Luckily I started with the docs
  because I'm a doc-type person.  I'd have disregarded gstreamer if I
  started out by actually trying it.

  *** The awesome power of gstreamer is hidden behind a very steep curve.

  ** Anything we can do to make this curve (which is not even a *learning*
  curve, which would be half ok, but just a *use* curve) less steep is
  really good, because it attracts developers.  Issues that need to be
  solved to achieve this are : a robust mediaplayer, a more graceful
  -launch, and a build layout from which you can easily tell what you NEED
  to get it working.

  * We have only about thirty people asking questions on
  IRC up till now.  When GStreamer takes off and you get triple or ten
  times that amount for some reason, because it gets into Gnome or
  whatever, do you really want to answer questions all the time like,
  "hey, your avi stuff doesn't work ?" and tell them "you need to get
  avifile" ?

  *** No, you want to have a top-level README file that says :
  "if the plugin is in gst/, it should compile out of the box. If it's in
  sys/, it'll probably work, but if you're on sparc and you don't have
  oss, you'll have a problem.  And if it's in ext/, please take a look
  at the dir name and figure out if you have that lib installed."

  *** You want a new developer to download a few tarballs, read the
  README, and know that he can at least get a good picture of what
  gstreamer is capable of without the hassle of tracking down libs and
  stuff.  The core will get you fake stuff and cp/dd/ ...
  The gst/ part of gst-plugins will allow you to mix wavs, resample,
  add effects, ... sox-stuff really.
  The really fancy stuff you'll need libs for.
  So BEFORE he starts asking questions like "why doesn't gstreamer work"
  you have given him an easy glimpse of the power behind it.

  ** I see you debian people laughing at RedHat's RPM
  dependency hunting, but you don't seem to think it's a problem for
  GStreamer ? At least with RPM I can go to one place to get them,
  there's no one place for gstreamer dependencies (yet).

  * So now you may think, well then why don't we put a README in each dir
  saying what lib it needs, that would solve the problem.  And I say, well
  good question, why don't you have that README right now ? Because YOU
  DON'T HAVE TIME, you're developing ! A default README which would at
  least help new people solve their issues with plugins BEFORE they
  start asking us questions is a good idea any which way you twist it.
  It'll always work, always be current and always at least help out new
  developers.

  *** It will not depend on our laziness to document what we are doing.


  So, in closing :

  ** I think the argument "let's make it flat because I don't see
  the point in another layout" just sucks.  It'll be a big directory,
  we haven't even scratched the surface of what plugins are possible.
  We won't scratch it further unless we get gstreamer cleaner.
  It's not because YOU can't come up with something better that flat
  is the best option.  I see the merit in my system, and choosing flat
  by default is silly.

  *** If you can live with a flat structure, I'm sure you can live
  with mine as well.  It won't make your live harder, will it ? Plugins
  are on the same level (ext/oss instead of src/oss, for example).
  They're easy to find since you as a developer know what the plugin
  does and what it uses.  And if you don't, you will as soon as you
  find it, for which we have the simple "find" command ;)

  ** I'm sure I forgot some points because I'm tired because I stayed up
  too long wrapped up in the politics of this, but the gist is here.  By
  now you should all realize I CARE about how we split it up and I CARE
  enough about gstreamer to commit myself to getting the build sorted.
  (it isn't fully atm).

  *** Please help me help you.  You're all developers,
  none of you want to touch autotools, you want to get on developing.
  So do I ;) I like all the ideas with the README's and the little files
  offering hints, but no one is DOING it, and no one will.  While I'm
  DOING the build repairage at the moment.  I think actually doing it
  should at least count for something.
======================================================================

omega's summary:
======================================================================
The given reason for the proposed directories is to provide a distinction
between plugins that either do or do not have external dependencies of one
kind or another, so as to aid in constructing the build system.  However,
I'm unsure as to why putting plugins that need external libraries in a
separate directory will make managing configure.ac any easier, or that
having them in a flat structure will make it harder.  The suggestion is
that one can determine which plugins need configure.ac checks by doing `ls
ext`.

However, there is only one time when this needs to be done in bulk: now.
The current configure.ac is a mess, we all know this.  In the process of
splitting the plugins off, we'll be constructing a new configure.ac.  At
that time we'll sort through all the plugins and determine which need
checks, and lay them out (alphabetically would make sense, IMO, as long as
there aren't inter-dependencies...) in configure.ac.  Assuming whatever
mechanism we decide (control file[s], whatever) is fully in place, all
future modifications to gst-plugins will be incremental, and will be dealt
with within that framework.

The arguments against using control files are based on the assumption that
no one will maintain them, which itself is based on the fact that no one's
written any, which is because we have no infrastructure in place right
now.  As soon as we have infrastructure for it (and depend on it),
developers will be forced to use the build system that's there.  For
instance, if a new plugin is written and a control file isn't, it simply
won't build.

As part of the infrastructure that would be built, various documentation
would be required for each plugin, and scripts can trivially be written to
validate things.  For instance, if we have a REQUIREMENTS file of a given
format, a script can check for -l's in a plugin's Makefile (or something
more accurate like a control file), and if there are dependencies but no
REQUIREMENTS entry, it deems that plugin to be incomplete.  Add it to the
nightly build...

Basically I'm worried that categorizing the plugins on a scheme that is
only really useful once (IMO, but I've gone through several cycles of
build hell too) is going to get in the way later.  The distinction isn't
relevant to users of packages, is only vaguely relevant to builders (given
REQUIREMENTS and README maintained fully, and enforced), doesn't mean
anything to developers afaict, and given the right build infrastructure
shouldn't matter at all to a build maintainer in the long run.

The naming of plugins (and thus their directories, which should be the
same IMO) is a separate issue, and is another problem we need to tackle at
the same time.  The main question I have is how much we use the 'gst'
prefix in plugins and elements.  Its use is quite random right now, and we
need to decide if we're ever going to use it in the actual plugin names,
because that must be reflected immediately in the process of moving
plugins around in CVS.

Some of the plugins and elements need rearranging as they are moved, but
I've already posted a list of the various merges and other changes that
are pretty obvious (like merging a lot of groups of separate plugins into
single plugins with multiple elements).

As for the naming of the elements, they must be globally unique, and
reasonably descriptive to anyone who'd ever care about the actual element
name.  This means not very many people, because anyone who'd be messing
with the elements is either a) explicitly hardcoding them, b) using the
editor to find them, and thus element klasses, or c) using the autoplugger
anyway...  On that score, I think we're doing quite good, except in the
system-streams cases of MPEG, AVI, and QT, which are easily fixable as we
move stuff into gst-plugins.

Other random comments:

1) The old mp3 decoders should be scrapped, but no one ever decided to.
That's cause they never built, and no one cared.  If we make it part of
normal procedure to revisit such things for each release (for instance,
with a script I've needed recently that would simply search the *source
tree* for all plugins and list them), these things will be taken care of.

2) Only having any clue of some percentage of the available plugins is no
longer an issue thanks to the printout at the end of configure, which also
will help solve the previous issue ("hey, 'xing' isn't ever building, do
we even need that anymore???").  The ability to print out a list of
plugins that will and won't built based on available libs is not a
function of what directory they're in, but whether there's any
configure.ac check for them.

3) Variable names in configure.ac are random because they've evolved over
time.  There was and is no standard, which is rather easily solvable by
decree.  "All configure.ac variables for plugin xyz must be called
..._XYZ", for instance.

Proposed directory structure:

gst-plugins/
	src/				(plugins/ ?  or is that redundant?)
		plugin1/
			plugin1.c
			libplugin1.so
			test1.c
			test1
			control
		plugin2/
			plugin2.c
			plugin2.h
			libplugin2.so
			control
	tests/
		xyzplayer.c
		xyzplayer
	scripts/
		buildconfigure.sh
		buildREQUIREMENTS.sh
	README
	REQUIREMENTS

1) Validating tests reside within plugin directories, with standardized
names like testN, etc.  Non-validating tests (i.e. more examples than
anything) go into a top-level tests/ directory.

2) Various files like REQUIREMENTS could optionally be generated from the
control files.  Heck, even configure.ac's main bulk could be generated
from control files.  It would even let us quick-test a plugin build by
only building configure.ac for one plugin at a time.
======================================================================

/me is really tired and is going to sleep now.

      Erik Walthinsen <omega at temple-baptist.com> - System Administrator
        __
       /  \                GStreamer - The only way to stream!
      |    | M E G A        ***** http://gstreamer.net/ *****
      _\  /_