[SyncEvolution] SyncEvolution in Fremantle

[Patrick Ohly]

On Wed, 2009-12-30 at 06:42 +0000, Ove Kaaven wrote:
> Patrick Ohly skrev:
> > This is becoming more SyncEvolution specific. Should we continue
> > cross-posting to maemo-developers?
> 
> Dunno, I wasn't subscribed to the syncevolution list yet. I suppose I
> should subscribe to it. But I think some Maemo developers may have some
> interest in the progress of this anyway?

Might be; so let's continue cross-posting until someone objects.

> > On Sat, 2009-12-26 at 13:53 +0000, Ove Kaaven wrote:
> >> Patrick Ohly skrev:
> >>> I was planning to do a 0.9.2 update release in a few weeks. Should we
> >>> merge your code and include the Maemo support in the release
> >>> announcement?
> >> If you want.
> > 
> > It's your call. Given that you raised some questions below about
> > improvements which cannot be done in backwards-compatible way between
> > releases of the calendar backend (change tracking!), it might be better
> > to give it some more time and release with 1.0 (tentative goal: end of
> > March).
> 
> Well, I think many people could benefit from synchronization before
> March, even if an incompatible change is introduced later. Is then the
> best way to just keep my "unofficial" builds available, without upstream
> integration before 1.0?

I'm fine either way. I'm still on vacation until January 11th, then I'll
have a closer look at the current state. In the meantime, let's continue
the discussion and perhaps see what users think about it.

> >> It's not clear from the website whether this "Linux Foundation license
> >> agreement" is really to be sent to the moblin guys, or to some other
> >> address.
> > 
> > My reading is that it needs to go to the Linux Foundation. I'll check.
> 
> OK. I'm not sure what's up with having to sign "Linux Foundation Moblin
> Workgroup Individual or Corporate Contribute License Agreement v1.0"
> (phew) if SyncEvolution isn't a Moblin project. Perhaps I'll just stick
> with the other options for now.

There's no "SyncEvolution Foundation". Instead of creating one it seemed
easier to just use what was already established, even if it is not a
perfect fit.

> >> - Do I need to worry about getSupportedTypes() or anything
> > 
> > You only need to implement the pure virtual methods, everything else has
> > reasonable defaults.
> 
> Well, I just imagine "reasonable default" is not always "perfect" or
> "efficient" or anything...

That's something that should be pointed out during a code review. I
appreciate your diligence, though ;-}

> >> - The Maemo's calendar-backend can return entries that have changed
> >> after a particular date (typically you'd use the time of the previous
> >> sync). Is there a way to use this to improve on the TrackingSyncSource
> >> method, so it won't be necessary to load and generate revision strings
> >> for the whole database every time?
> > 
> > As Congwu said, this can be done by inheriting from SyncSource directly
> > and adding the SyncSource* building blocks that you want to reuse. You
> > can use the TrackingSyncSource as an example how that is done.
> 
> Well, as an example, it's really not very enlightening. Its
> implementation has no explanation. What is all this multiple-inheritance
> mess really doing?

First, avoiding code duplication. A sync source must implement multiple
different, orthogonal aspects (change tracking, item access,
backup/restore, ...). There are multiple different possible
implementations for each, which must be combined somehow without copying
the source code. Second, multiple-inheritance is used to introduce pure
virtual methods. Derive from TrackingSyncSource and once your code
compiles, you know that you have provided all the necessary custom code.

I agree that the way how this was done in C++ is not particularly nice.
I couldn't come up with a more elegant approach despite thinking about
this quite a bit, so you may have a point about C++ making it worse than
in other languages.

> How many (and which) virtual base classes would I
> have to derive from to implement a more efficient change tracker, with
> the least amount of code?

You need to understand the SyncSource base class and what functionality
it can and should provide. Then you need to know which existing
implementations of that functionality exist. An overview page in Doxygen
with links to more detailed API descriptions would be useful here, but
doesn't exist yet.

>  Would I have to spend a month digging into the
> internals of every single class of this huge forest of
> multiple-inheriting classes and take notes about how they work and
> interact, before I'd be able to write code that does something as simple
> as considering a known subset of the database as potentially modified
> since last sync, instead of checking the whole database?

No. You would need to understand only one part, the one currently
implemented by SyncSourceRevisions.

> > The "time of last sync" is something that could be stored either in an
> > internal source property or (better) in the SyncML anchor string,
> > handled by the Synthesis engine. However, this will require changes to
> > the Synthesis/SyncEvolution and SyncSource interfaces. We have to work
> > on this anyway for resume support in the SyncML server, so my suggestion
> > is to address this in 1.0 like this:
> >       * make those API changes
> >       * create a new DateSyncSource which requires the following
> >         information from derived classes:
> >               * complete list of all local item IDs
> >               * list of changed item IDs since a certain time
> >       * change FileSyncSource so that it is derived from DateSyncSource
> 
> Works for me.

Looking at the code again and in particular
SyncSourceRevisions::updateRevision(), I remembered one problem with the
"changes since last sync" approach of change tracking. Suppose the
following chain of events:
     1. sync starts at time T1, with no local changes
     2. a changed item A is sent by the SyncML server and updated
        locally
     3. the user edits a different item B while the sync runs
     4. sync ends at time T2

Now in the next sync, which time stamp should be used to find changed
items that need to be sent to the server?

When using T1, the changed item A would be sent to the server, which is
wrong (additional traffic, unnecessary risk of conflicts on the server).
When using T2, the locally modified item B would not be sent.

Updates of the database would have to be disabled while a sync runs to
avoid this problem. Is that possible?

The SyncSourceRevisions class allows concurrent editing by
distinguishing changes made by SyncEvolution and others. It's not
perfect at the moment, because an API change in EDS would be needed to
avoid race conditions - we have patches pending for that.

> > Do you have easy access to all IDs of existing items? This is necessary
> > to detect deleted items.
> 
> Sure, a full list of IDs can be queried easily, with minimal overhead
> (unlike their associated last-modified-times). I'm not totally sure
> whether deleted IDs are included or not, though, might have to test.

They don't have to when we keep a list of known IDs in the last sync.
Any ID no longer listed then is known to be from a deleted item.


-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.