Shared documentation system

[Shaun McCance]

NOTE:  A shorter version of this email is stuck in moderation, because
I'm a doofus and was subscribed with the wrong email address.  Normally
I'd wait for the moderators, but I had stuff to add.

I'd like to get some discussion started about having a shared system for
documentation in GNOME, KDE, and other desktop environments.  I am the
maintainer of Yelp, the GNOME help browser.  The current stable version
of Yelp finds documentation through ScrollKeeper.  As far as I can
tell,the KDE Help Center uses the DocPath field of .dekstop files, and
all documentation is installed in

<prefix>/doc/HTML/<lang>/<app>

That's mostly a guess on my part, based on what I see in KDE 3.1.  If
I'm wrong or out of date, feel free to smack me.  The KDE Help Center
also seems able to read from ScrollKeeper, but it's not used as the
primary means of installing documentation for KDE applications.

Why we need a shared system
---------------------------
It's my observation that only core GNOME apps use the GNOME help system
(even though ScrollKeeper is supposed to be neutral), and only core KDE
apps use the KDE help system.  This is a problem.  If I'm using KMail
inside of GNOME, I should still be able to view KMail's documentation.

In the case of mixing applications from different desktops, the proble
misn't devastating.  Most users will get the help for an application by
selecting it from the Help menu.  In KMail, that will open up the KDE
Help Center.  In Evolution, that will open up Yelp.

This presents a problem for third-party software, though.  When ISVs
develop documentation for their applications, what do they do?  If they
have to jump through hoops to make it work with different desktops, then
they probably just won't bother.  They'll put it in HTML and open a
webbrowser.

Goals of the help system
------------------------
* Users should be able to choose the help browser of their choice.  If
I'm a GNOME user, I should be able to see all my documentation in Yelp. 
If I'm a KDE user, I should be able to see all my documentation in the
KDE Help Center.

* The help system should impose as little policy as possible on the help
browser, and should expose as little policy as possible to documenters.

* It should be simple to place a document in the help system.  The help
system shouldn't require anything to be run to register or unregister a
document.  That just begs for problems.

* The help system should not assume anything about the viewing program. 
The viewer may not be a graphical program like Yelp or the KDE Help
Center.  It may be a print system, a console program, or some sort of
documentation server.

* Documents should be able to supply meaningful metadata, which hel
pviewers may be able to use.  The metadata format for documents should
be easily extended in a well-defined way.

* Applications should have a simple and consistent way of accessing
their documentation, independant of the desktop environment.

* Documents shoud have a simple and consistent way of referencing other
pieces of documentation, independant of the desktop environment.

* The help system should be capable of pulling information from other
existing help systems.  This way we can have legacy compatibility with
KDE's and GNOME's current systems.  Also, this can provide a consistent
means for help viewers to integrate man and info, if they choose to.

What I'm proposing
------------------
I would like to see something very similar to the way applications are
registered.  Every piece of documentation would place a metadata file in
a predetermined location.  Then the help browser would use something
like the Menu spec to construct a listing of installed documentation.  

In fact, the applications Menu and the documentation Menu could probably
even be merged.  Code that's constructing a menu of applications would
ignore elements relating to documentation, and vice versa.  Doing this
has the advantage that changing the organization of the applications
automatically changes the organization of the documentation to match.

As far as I know, this is similar to how KDE deals with documentation
right now.  I want to make it more sophisticated.  Instead of using the
DocPath field in .desktop files, I want to use a specialized file format
that's capable of actually describing the document.  RDF/XML seems like
a natural fit for this, except that it's a royal PITA.

NOTE: End of original email.

The metadata format
-------------------
So after playing around, I came up with a fairly simple XML format that
could be used for metadata.  This file format is not RDF/XML, though it
has a clear mapping onto RDF.  An example:

<Document id="http://www.gnome.org/learn/user-guide/"
    xmlns="http://freedesktop.org/Standards/docs-spec/0.1"
    xmlns:dc="http://purl.org/dc/elements/1.1">
  <dc:title>GNOME User Guide</dc:title>
  <dc:title xml:lang="de">GNOME Benutzerhandbuch</dc:title>

  <dc:creator>gdocteam at sun.com (Sun Microsystems)</dc:creator>
  <dc:publisher>GNOME Documentation Project</dc:publisher>

  <dc:subject>Core</dc:subject>
  <dc:subject>GNOME</dc:subject>

  <dc:rights>http://www.gnu.org/licenses/fdl</dc:rights>

  <Copy>
    <dc:date>2003-09-08</dc:date>
    <dc:format>text/x-docbook+xml</cd:format>
    <dc:identifier>
      file:///usr/share/gnome/help/user-guide/C/user-guide.xml
    </dc:identifier>
    <dc:language>C</dc:language>
  </Copy>
  <Copy>
    <dc:date>2003-09-08</dc:date>
    <dc:format>text/x-docbook+xml</cd:format>
    <dc:identifier>
      file:///usr/share/gnome/help/user-guide/de/user-guide.xml
    </dc:identifier>
    <dc:language>de</dc:language>
  </Copy>
</Document>

In RDF terms, there are three resource types:  Collection, Document, and
Copy.  (I'll get to Collection later.)  Everything else is a property. 
The properties shown here come from Dublin Core, and it seems to cover
the basics of what's needed.  Properties don't have to come from Dublin
Core, and this format is easily extended by adding new properties.

This format is very simple.  A Collection can contain any number of
properties and any number of Document elements.  A Document can contain
any number of properties and any number of Copy elements.  A Copy can
contain any number of properties.

There are some built-in semantics in this format.  When a Collection
contains a Document or a Document contains a Copy, in RDF terms, there
is an implicit magical property connecting them.  Also, all resources
implicitly get all properties of their parent resources.  Thus, all the
properties on the Document above (dc:title, dc:creator, etc.) are also
properties of each Copy.

A Document refers to the abstract concept of a piece of documentation. 
It includes all versions, translations, and formats of that piece of
documentation.  A Copy refers to a specific rendering of a Document.

A Copy will most often be identified by the combination of version,
language, and format.  A help viewer can choose whichever Copy it
considers most suitable for display.

A Copy can have any number of dc:identifier properties, even zero.  Help
viewers such as Yelp and the KDE Help Center will typically prefer those
Copy elements that have a file:// identifier.  However, the format may
be used for things other than local help viewers, so all copies can be
included in the metadata file.  If the documentation is online, each
Copy can have an http:// identifier.  Then those copies that are locally
installed can also have a file:// identifier.

Each Document should have an id attribute, which should be a URL that
references this piece of documentation.  It needn't point to any actual
copy of the documentation, though there should be something at that URL
that references it.  The URL is intended as an identifier for all copies
of the documentation, so it shouldn't have any references to language,
version, or format.

Where the Menu spec for applications can match against Filename and
Category, the Menu spec for documentation could match against id and
dc:subject.

Now, Collection.  An example:

<Collection
    xmlns="http://freedesktop.org/Standards/docs-spec/0.1"
    xmlns:dc="http://purl.org/dc/elements/1.1">
  <dc:title>Glade Documentation</dc:title>

  <Document id="http://www.gnome.org/projects/glade/faq/">
    <dc:title>Glade FAQ</dc:title>
  </Document>
  <Document id="http://www.gnome.org/projects/glade/turbo-start/">
    <dc:title>Glade Turbo Start</dc:title>
  </Document>
  <Document id="http://www.gnome.org/projects/glade/user-guide/">
    <dc:title>Glade User Guide</dc:title>
  </Document>
</Collection>

A Collection can be used to group Documents.  If a help viewer shows
HTML pages for the documentation listing (as Yelp does), then it may
choose to put these on a separate page, or to list them under a common
header on the containing page.  If the viewer shows a tree sidebar of
all the documentation, it may choose to put these under one header.

Basically, the Collection doesn't strictly say "You must group these in
such-and-such manner."  It just says that these documents all belong to
some collection, and the help viewer may want to show them accordingly.

Back to the goals
-----------------
1) Users should be able to choose their help browser.
If these metadata files are given a unique MIME type, and documentation
is always launched through them, choosing your help browser is as simple
as setting a MIME handler.

2) The help browser should impose little policy.
There's a great deal of flexibility in this system, and I think it could
be used for a number of different types of documentation systems.

3) It should be simple to place a document in the help system.
These files are relatively easy to write, far simpler than equivalent
RDF/XML would be.  And there is no document registration system here.

4) The help system should not assume anything about the viewer.
See (2).

5) Documents should be able to supply meaningful metadata.
These files have simple and clearly-defined semantics, and a simple
means of adding new properties.

6) Applications should have a consistent way to access documentation.
Simply launching the MIME handler for their metadata file would work. 
For applications using GNOME libraries or KDE libraries, convenience
functions may be in place to do this.

7) Documents should have a consistent way to access documentation.
This one is a bit trickier, but still doable.  If documentation needs to
link to other documentation, it could use the id of that documentation
as its link target.  The help viewer could check link targets against
its list of documentation.  This avoids the creation of yet another
ad-hoc URI scheme, though it has some disadvantages.

8) The help system should be able to work with other systems.
I think there's a simple mapping from OMF onto this format, so there
could be a specified means of cooperating with ScrollKeeper.  The same
is probably true for .desktop files.  Perhaps properties could be added
that effectively say "I replace such-and-such piece of documentation
that you may find in and OMF or .desktop file."  That way we don't get
duplicate entries of pieces of documentation that are registered in both
places.

For man and info, probably the best way to handle these is just to have
special elements in the Menu format that reference them explicitly.


--
Shaun