2006/12/21, Jean-Francois Dockes <<a href="mailto:jean-francois.dockes@wanadoo.fr">jean-francois.dockes@wanadoo.fr</a>>:<div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Mikkel Kamstrup Erlandsen writes: > Having a unique handle for each document is a really handy thing. A unique > handle could be many other things than an uri, fx a unique integer or > anything as well. The way you describe this handle is just a number > specifying an entry in an array and is in this way not unique (except in the > context of a query). That is right, the result sequence entry number is only usable in the context of a given query. > The uri is just a more convenient handle than fx an integer - for > applications at least (the toolkit and platform libs often handle uris > directly). It depends, this is only true in the case of documents with a well-known URI-based access method, which is not the general case (ie: not true for email messages).</blockquote><div> Well URL and file URIs are readily usable for the apps, and they account for a very large portion of the results in many cases. Also, if there ever comes a standard URI for pointing at emails, apps could start using that right away.  </div> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> > I respect your disagreement, and would really like to hear what the other > guys think... Me too ... I am going to restate the problem as I see it, in a way which is probably somewhat xapian-specific. Hopefully someone will be able to concur or contradict this for other engines. After it has received and processed a query, the backend has some kind of structure that represents the results. This is an ordered sequence of documents, the order depends on the kind of sorting that was requested. Typically and by default this will be by order of relevance. The result set is mostly organized to be accessed by sequence number, and things are set up so that you can retrieve auxiliary data for a given result: a result entry will have some kind of document number which can be used in turn to access document data efficiently. Access by URI is not part of the scheme. This means that a backend for the current WasabiSimple api would have to: - In response to Query(), run a query, then walk the result set to request    document data, select URIS, and return these only (while it has access to    all the documents field at the same time at no additional cost). - In response to GetProperties() or GetSnippets() either run another,    different query to get the corresponding document data (this is an    actual db query using the URI as a unique term, not the simpler and more    efficient access to data by document number), or use some kind of cache    that it would hopefully have built in response to the first Query()    call, hoping that the  requested URI->document association is    actually cached. This is very awkward, and offers no real benefit to the application, which would as well decide initially what kind of metadata it wants back, and get it all in response to the Query() call. </blockquote><div> I think the much the same applies to Lucene. I'll have to ask some of the Lucene experts in the office in the new year. Anyway, it is not a herculean task for the search engine to keep a ring-buffer-like hashmap of the last 100 hits, uri->doc_id. Easier said than done of course since hashmaps has no obvious way of acting as a ring buffer. There is also synchronization issues when the underlying index changes of course...  </div> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> > Pasting in Jean-Francois'  follow up mail: >  > As an afterthought to my previous message (sorry), the result list could >  > change if the query has to be re-run. This is a good reason for keeping >  > the uris as document identifiers for getSnippets(). > > It would feel akward if you had to request a specific property (the uri) to > be able to obtain a snippet IMHO. Sorry, but I can't see why. It would be vastly less awkward than the current proposal, where you initially get a set of data for which you have no use (try displaying a list of bare URIS to the user and see how they like it), and then immediately make other calls to request data for each and every of the initial results (because there is no way to select among them). > Ok, my central point is: We need a unique handle for each document/object in > store - this should be used to identify the returned hits from Query(). > Whether or not an opaque handle of some undefined sort or it is defined to > be the uri is another matter. The nice thing with the URI is that it is independant of the query. The fact that queries are only identified by the query string sort of implies that the query result list might not be fully stable across calls, so that using result sequence numbers as identifiers would be inappropriate. So either we need a unique *query* identifier which would at least enable detecting that the result set is stale, or we use URIS as document identifiers for calls subsequent to the initial Query(). If we really want to keep a separate GetSnippets() call, my proposal would be to: - Return all desired metadata with the initial Query() call as an    appropriately ordered sequence (as determined by the user query sort    parameters). - Request snippets by URI (which may imply running a different query for    each snippet on the backend side, but hopefully not for all initial    results).</blockquote><div> Ok. In this case I suggest always including a Object.URI property in the return values. This way platform bindings can have a Hit.GetUri() method that is guaranteed to make sense. With platform-binding-niceness in mind, there could even be a *small* set of obligatory properties to return (including uri) - I can't think of any other right now.  </div> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Note that both in this approach and in the current WasabiSimple proposal, it may probably happen under some circonstances that the result sequence obtained by successive Query() calls would be unperfect, with duplicates or holes (if the database has changed and the backend had to run the query again for some reason). This forces requesting Snippets by URI.</blockquote><div> Yeah right. The simple api has inevitable shortcomings when it comes to query integrity. I suspect most apps using the simple query to only call Query() once per actual user query (ie don't do paging), in this case query integrity is less compromised.  </div> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> > To the sorting problem I see two solutions. > 1) Always return a score property as part of the response properties as >    defined in my proposal. > 2) Always include the UniqueHandle property as part of the response as > defined by your proposal. 1)  would have to be extended to some sort of ordinal value (not always representing a score, the results might have been requested ordered by ie, date), and it burdens the application with sorting the results again. Why make things so complicated ? About 2), URI *is* an appropriate handle, and probably the best as long as we can't guarantee the stability of the result set (that is: *if* we need separate Query() and GetSnippets() calls, *then* the URI is probably the best identifier to ensure consistency).</blockquote><div> The thing is that I think we should keep an eye out for keeping the Simple and Live interfaces as consistent as possible. Returning and using similar data structures and such. In the Live interface you simply have to include some sorting data in the hits  because they can be added and removed dynamically. I suggested making URI a obligatory property to return. Maybe we have to  have some sorting info available too... In the simple api the sorting property(ies) would be redundant though. Cheers, Mikkel </div></div>