[Clipart] dms status

Mon Feb 14 14:45:47 PST 2005

Bryce Harrington wrote:
> Hi all,
> 
> Tobias asked me to write a summary of where things are with dms, what
> needs to be worked on, etc.
> 
> As a quick recap, the architecture for dms involves a daemon that runs
> on the server and provides a SOAP API that remote scripts and tools can
> use to interact with the document system.  The daemon is composed of a
> short SOAP daemon script, that simply directs calls to a perl module
> called Document::Manager.  Document::Manager provides a set of routines
> for checking documents in and out, adding properties, querying for lists
> of things in different ways (author stats, system stats, change history,
> etc.)  
> 
> It also allows retrieval of a Document::Object which provides detailed
> access to the document in question, including metadata, comments,
> workflow state, etc.  A 'Document' is a collection of one or more files;
> for instance, this could include an SVG file, a thumbnail png, a
> metadata xml file, and a log of discussion about it (imagine something
> like an mbox file).

So cool!!!

> Document::Manager works through a lower level module called
> Document::Repository, that handles the actual storage and retrieval of
> information from the file system.
> 
> There is also a piece WebService::TicketAuth, which handles the
> authentication of the user.  Another (TBD) piece is needed for handling
> authorization (i.e., to identify if the user has administrative or just
> normal access).  This was going to tie into Mantis so that we wouldn't
> have to write our own account management code, but since we're not using
> Mantis anymore, we'd need to figure something else out.

Well, will this allow for secure login/logout without Mantis being 
installed? Or, would we need something else implemented for that?

> In theory, these modules will work with *only* the file system (i.e., no
> database required), but will allow use of a database for caching
> purposes.  Because the system does not rely on the existance of the
> database, this means that backups or replications could be done by just
> mirroring the repository filesystem itself; no database synchronization
> is needed.  However, none of this db stuff is implemented yet so it's
> entirely theoretical.

Well, doing it at the file system level would be fine for a year or so, 
while we could work out db support while the system scales up.

> Prior to the freedesktop.org rebuild, I had a functional daemon
> installed and running that permitted checking in documents and
> retrieving them.  There were several dependencies required to be
> installed to make this work, so one of the next steps would be to
> re-install these and get the daemon running again.

This seems like the first step. Can you either set this up with fdo, or 
provide the steps necessary so that I could set it up on fdo please?

> The next item on the todo list is to implement the property handling
> code.  I think I had this about half-way done when the rebuild
> happened.  Basically, there just needs to be some code implemented in
> Document::Object to permit storing of arbitrary properties into a file
> of some sort (XML::Simple, Config::Simple, or so forth.)  Then routines
> hooked up for adding a new property, editing/deleting a property, etc.
> Also needed would be a way to query the repository as a whole based on
> properties (e.g., select all documents where author='niku').

> Once property handling is available, a whole host of things could be
> implemented on top of that.  I think the most worthwhile thing to do
> first would be workflow handling.  This would allow tagging a document
> as to whether it is "new", "accepted", "problem", etc.  This would allow
> us to track and filter for items that have copyright questions, have bad
> metadata, or whatever, so they can be kept out of the releases until the
> problems are resolved.

Yes...or if there are updates to a file, right?

> For the database side of things, the major todo item is to implement a
> routine that can take a Document::Object, render its contents into SQL,
> and store it.  Obviously, this also requires some SQL design; ideally
> this would be done in a way that is generic such that the db schema
> would not need to be customized if someone wished to use the system for
> something other than SVG files.

So right now it is specific to SVG, and not generalized?

> I had assumed the system would perform fairly well even without the
> database, however I've found that actually it's quite slow.  However,
> there are a couple places where I took some particularly dumb design
> approaches, so I think that if these were redesigned they could be much
> better optimized.  (Basically, it hits the file system far too much, and
> it doesn't store info in memory between calls, as it should.)

Well, hopefully we can get the system work and then optimize...don't wanna 
prematurely optimize....;)

> For the interface side of things, the next step is hooking up PHP to the
> SOAP calls.  I had assumed this would be a no-brainer since PHP supports
> SOAP.  Unfortunately, the way PHP supports SOAP is different than how
> Perl does, and I found that the way I had coded Document::Manager won't
> work with PHP.  This was pretty disappointing.  The two options would be
> to recode OCAL's website to use Perl instead of PHP (such as through
> Template Toolkit), or to recode Document::Manager into a non-OOP style
> that will be compatible with PHP.  Neither solution would be much fun to
> implement, and would take a good bit of time to do.  

Well, PHP supports OOP as well, and it would be easier to recode the site 
into either PERL, or OOP PHP. I would prefer to do OOP PHP if possible and 
necessary.

> Anyway, that's where things are at.  The freedesktop.org rebuild and the
> Perl/PHP SOAP incompatibility got me off track, and things with Inkscape
> such as gtkmm and the 0.40 and 0.41 releases have kept me busy since
> then.  However, I'm pretty confident that the parts that have been
> implemented so far are good and solid, and that the overall design
> approach is right.  There's a lot left to do, though.

Okay, well, I've put some space in the roadmap to not stress out plans. As 
I said previously, our project is successful as long as people continue to 
contribute clip art. I would be great though to get back on track with 
using DMS to build our infrastructure upon. If you could help us get DMS 
daemon up and running on the FDO server, and then possibly provide a little 
direction, I would be more than happy to hack on this.

All of our tools were cobbled together as temporary solutions for the more 
powerful DMS solution, so I think it best if we devote more resources to 
working on DMS, metadata tools on the site to take our project to the next 
level.

Thanks for all your work Bryce...where is all the code for DMS at? Also, 
maybe it would be best to make a page on the OCAL wiki to keep track of 
what is done, what needs to be done, and resources on DMS (this is more for 
other people).

Jon

> _______________________________________________
> clipart mailing list
> clipart at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/clipart
> 
> 

-- 
Jon Phillips

USA PH 510.499.0894
jon at rejon.org
http://www.rejon.org

Inkscape (http://inkscape.org)
Open Clip Art Library (www.openclipart.org)
CVS Book (http://cvsbook.ucsd.edu)
Scale Journal (http://scale.ucsd.edu)	