shared wasabi implementation

[Joe Shaw]

Hi,

Mikkel Kamstrup Erlandsen wrote:
>     For external filters, I have shamelessly borrowed Beagle's
>     external-filters.xml
>     with some modifications. Built-in filters register what MIME types
>     they support
>     when the corresponding dynamic library is loaded. 
> 
> 
> For reference: http://beagle-project.org/ExternalFiltersRepository
> 
> It still appears not to cover the two cases I mention - emails in a 
> database in a hidden directory, indexing of webpages+urls as you browse. 
> Anyway - a good starting point. Perhaps Joe can shed some light on why 
> this was left out..?

The external filters were added so that people could index file types 
not supported internally without having to code up support for them.

The two cases you mention aren't file types, they're data sources. 
(Mail is handed by our mail filter, and web pages by our HTML filter 
already.)  For people who want to index their data externally we provide 
an indexing service.  Apps can do one of two things: they can make an 
RPC call and pass in a document and metadata to be indexed, or they can 
drop the file into ~/.beagle/ToIndex with a control file that describes 
its metadata and Beagle will automatically index it.  (This latter 
method is how the Beagle Firefox extension works.)

We could maybe create an external data source backend, but since the 
sources are so specific, all it would amount to would be calling some 
sort of script that did the crawling and used one of the two methods 
above to signal Beagle.  Unlike the external filters, there hasn't been 
any demand for it, and fitting it in to the scheduler so that it didn't 
peg the indexer or fill up the disk would be tough to do externally.

Joe