From patrick.ohly at intel.com Mon Aug 22 13:22:31 2016 From: patrick.ohly at intel.com (Patrick Ohly) Date: Mon, 22 Aug 2016 15:22:31 +0200 Subject: survey: providing SyncEvolution binaries Message-ID: <1471872151.24499.9.camel@intel.com> Hello! I really should do a new SyncEvolution update, including the patches that distros are using to build with more recent tools and/or libraries. But... what distro should I provide binaries for? That's the big roadblock at the moment that prevents a new release. I could leave everything as it is and do a release, but that's not very useful, because recent distros wouldn't be supported. Does anyone still need pre-compiled binaries at all? Distro had quite a lot of time to catch up with adding and updating SyncEvolution and SyncEvolution development has slowed down, so there is no need anymore to always offer the latest and greatest upstream version. One exception is perhaps Debian Stable, which only has an old version (1.4.99). But even that isn't too bad. So let me ask: which distro do you need syncevolution.org binaries for? Why? -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From tino.keitel+syncevolution at tikei.de Wed Aug 31 21:12:17 2016 From: tino.keitel+syncevolution at tikei.de (Tino Mettler) Date: Wed, 31 Aug 2016 23:12:17 +0200 Subject: [SyncEvolution] survey: providing SyncEvolution binaries In-Reply-To: <1471872151.24499.9.camel@intel.com> References: <1471872151.24499.9.camel@intel.com> Message-ID: <20160831211217.nhqooolxbpplwtaz@mac.home> On Mon, Aug 22, 2016 at 15:22:31 +0200, Patrick Ohly wrote: > One exception > is perhaps Debian Stable, which only has an old version (1.4.99). But > even that isn't too bad. Hi Patrick, I could check if a backport of 1.5.1 for Stable is trivial. Users would have to manually add the backports repository, but installing binaries from syncevolution.org requires some manual steps, too. Regards, Tino _______________________________________________ SyncEvolution mailing list SyncEvolution at syncevolution.org https://lists.syncevolution.org/mailman/listinfo/syncevolution From patrick.ohly at intel.com Mon Aug 22 14:02:25 2016 From: patrick.ohly at intel.com (Patrick Ohly) Date: Mon, 22 Aug 2016 16:02:25 +0200 Subject: [SyncEvolution] syncevolution debug help needed In-Reply-To: References: <1469786988.23795.18.camel@intel.com> Message-ID: <1471874545.24499.16.camel@intel.com> On Fri, 2016-07-29 at 21:27 +0200, deloptes wrote: > Patrick Ohly wrote: > > > On Thu, 2016-07-28 at 19:22 +0200, deloptes wrote: > >> Hi, > >> from time to time I'm getting in following situation. > >> Do you have some hint how I can understand what is wrong here. It works > >> fine, but at once I start getting those messages. > >> > >> The following output was > >> created with SYNCEVOLUTION_DEBUG=3 syncevolution nokia_N9 calendar+todo > > > > The only unusual message that I noticed is: > > > >> [2016-07-28 18:49:04.473] to-be-added item already exists -> trying > >> [replace > >> (=conflict resolved by client winning) > >> [2016-07-28 18:49:04.473] TCustomImplDS::implProcessItem 0x27e1620 > >> [starting, > >> SyncOp=replace, RemoteID='496', LocalID='' > >> [2016-07-28 18:49:04.473] Executing Script 'beforewritescript' > >> [2016-07-28 18:49:04.473] todo: updating "Juli call ptotesi gel" > >> [2016-07-28 18:49:04.473] todo: Item delete: ( > >> 00287b7f-207e-476b-96ad-6d0758d563e1 ) > >> [2016-07-28 18:49:04.764] todo: Item saved: ( > >> 00287b7f-207e-476b-96ad-6d0758d563e1 ) > >> [2016-07-28 18:49:04.764] todo: Item ( > >> 00287b7f-207e-476b-96ad-6d0758d563e1 : 20160704T075430Z ) done. > >> [2016-07-28 18:49:04.764] todo: > >> [aID=(00287b7f-207e-476b-96ad-6d0758d563e1,) > >> res=208 > >> [2016-07-28 18:49:04.764] - Operation replace failed with SyncML > >> [status=208 > > > > I'm not entirely sure why it is marked as "failed" - does that really > > break syncing? > > > > DB_DataReplaced = 208 = ITEM_REPLACED in SyncEvolution. > > > > I'm not exactly sure where the "Item delete" and "Item saved" messages > > come from. Is that perhaps from your backend? > > > > Yes it comes from the backend. Unfortunately I am not exactly sure if I have > to propagate the change immediately as it has a kind of cache, but I think > it does not work properly, so I will save immediately > > > > The underlying problem here is that calendar items are supposed to be > > unique, based on their UID. That is a semantic that needs to be enforced > > in calendar applications. However, some clients (in particular older, > > vCalendar 1.0 based phones) do not support the same semantic. What could > > happen is that (for whatever reason - that's often what needs to be > > investigated) a phone asks to *add* an item with a UID that already is > > in use by an existing item. In that case, the existing item needs to be > > updated. > > This is also my understanding and the way I think I implemented it Looking again and based on your code I've come to a slightly different conclusion: in the log above, the libsynthesis engine itself already detects that it needs to update an item. From the libsynthesis code: // in normal sync in the server case, this can happen when a previous session // was aborted (and already applied adds not rolled back) // --> reprocess it as a replace PDEBUGPRINTFX(DBG_DATA,("to-be-added item already exists -> trying replace (=conflict resolved by client winning)")); > I have implemented this logic in insertItem the following way > > ... > InsertItemResultState state = ITEM_OKAY; > ... > > KCal::Incidence *oldinc = calendarPtr->incidence(uid); > if (oldinc) { > if ( ! calendarPtr->deleteIncidence(oldinc)) > Exception::throwError(SE_HERE, "internal error, unable to delete item > from calendar"); > > ---> I add now these 2 lines here > if ( ! calendarPtr->save() ) // It looks like we have to propagate the > change immediately > Exception::throwError(SE_HERE, "internal error, unable to save > calendar"); > ---> > state = ITEM_REPLACED; > SE_LOG_DEBUG(getDisplayName(), "Item delete: ( %s )", uid.latin > () ); > } > ... > ... > > This was one of the last doubts I have at this stage, but I am still not > confident. Could be that it needs improvement. Is the full code available somewhere? In the situation from the log file, your insertItem is called with a non-empty uid and thus needs update the corresponding item. Returning ITEM_REPLACED for an update operation is unexpected and thus leads to the sync error that you encountered. The code that you are showing shouldn't be used in this case, or if it is used (for example, because "update" always equals "delete + add"), it should return ITEM_OKAY. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From deloptes at gmail.com Mon Aug 22 22:29:45 2016 From: deloptes at gmail.com (deloptes) Date: Tue, 23 Aug 2016 00:29:45 +0200 Subject: [SyncEvolution] syncevolution debug help needed References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> Message-ID: Thanks again for the reply! Patrick Ohly wrote: > Is the full code available somewhere? > No, I still don't know what to do with it and I'm not sure in which form I should upload it. Uploading it means maintain it ... etc. I was thinking to upload it to the trinity desktop, but still it is not clear how it would fit with the different distros. Another option would be to add it to syncevolution - perhaps it is even more proper ... anyway short answer is no - I could pass a zip file of it .. it's really not much code. The last but not least is I do not feel confident because of this issue and do not want to "release" is until it is solved. > In the situation from the log file, your insertItem is called with a > non-empty uid and thus needs update the corresponding item. Returning > ITEM_REPLACED for an update operation is unexpected and thus leads to > the sync error that you encountered. > > The code that you are showing shouldn't be used in this case, or if it > is used (for example, because "update" always equals "delete + add"), it > should return ITEM_OKAY. Ah good. Update is indeed Del + Add (but item has the old ID). In the opensync code it was the same and I didn't challenge my imagination that much to invent something new. So it looks like I misunderstood the ITEM_REPLACED meaning? Could you please confirm once again. I have been struggling with these part of the code for some time already. My questions 1. even when update is implemented as del+add and the item has the old ID it should return ITEM_OKAY? 2. does ITEM_REPLACED means that item has new ID and is to replace the old item with the old ID? IMO both can be implemented. Case/Question 2 would even save some lines of code - as I am now explicitly setting the old id to the item. Strange is only that I have such issues with calendar+todo only and a fact is that it messes totally up in slow sync. However (if you are right and I think you probably are) even if I end up in slow sync for some reason, I would come out clean by using ITEM_OKAY. Let me know what you think. thanks in advance and regards _______________________________________________ SyncEvolution mailing list SyncEvolution at syncevolution.org https://lists.syncevolution.org/mailman/listinfo/syncevolution From patrick.ohly at intel.com Tue Aug 23 06:53:32 2016 From: patrick.ohly at intel.com (Patrick Ohly) Date: Tue, 23 Aug 2016 08:53:32 +0200 Subject: [SyncEvolution] syncevolution debug help needed In-Reply-To: References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> Message-ID: <1471935212.24499.28.camel@intel.com> On Tue, 2016-08-23 at 00:29 +0200, deloptes wrote: > Patrick Ohly wrote: > > > Is the full code available somewhere? > > > > No, I still don't know what to do with it and I'm not sure in which form I > should upload it. Uploading it means maintain it ... etc. I was thinking to > upload it to the trinity desktop, but still it is not clear how it would > fit with the different distros. Another option would be to add it to > syncevolution - perhaps it is even more proper ... anyway short answer is > no - I could pass a zip file of it .. it's really not much code. > The last but not least is I do not feel confident because of this issue and > do not want to "release" is until it is solved. Fair enough. The common wisdom is to "release early, release often", but in this case I agree that it is better to ensure that there is no data loss before making it available to others. > So it looks like I misunderstood the ITEM_REPLACED meaning? Could you > please confirm once again. I have been struggling with these part of the > code for some time already. > > My questions > 1. even when update is implemented as del+add and the item has the old ID it > should return ITEM_OKAY? Yes. > 2. does ITEM_REPLACED means that item has new ID and is to replace the old > item with the old ID? ITEM_REPLACED, ITEM_MERGED and ITEM_NEEDS_MERGE all are to be used only when adding a new item, i.e. uid is empty. In this case, the engine doesn't know about the existing item in the database and the backend has to tell the engine about it. ITEM_NEEDS_MERGE is probably best for handling this situation because merely replacing the existing item (ITEM_REPLACED) can loose data and merging in the backend (ITEM_MERGED) is more work to implement. > Strange is only that I have such issues with calendar+todo only and a fact > is that it messes totally up in slow sync. Not sure either. Perhaps it is because calendar items have a UID and thus additional constraints that don't apply to contacts. ITEM_REPLACED, ITEM_MERGED and ITEM_NEEDS_MERGE should never be needed for contacts, because a contact backend typically will never be able to detect when a new contact that is to be added already exists. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From deloptes at gmail.com Tue Aug 23 23:22:28 2016 From: deloptes at gmail.com (deloptes) Date: Wed, 24 Aug 2016 01:22:28 +0200 Subject: [SyncEvolution] syncevolution debug help needed References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> <1471935212.24499.28.camel@intel.com> Message-ID: Patric, thanks a lot! This was a great explanation (as usual). I will rework the relevant parts when there is some time available soon (I think I will have to only replace ITEM_REPLACED with ITEM_OKAY in the contacts backend and ITEM_NEEDS_MERGE in the calendar/todo backend). Patrick Ohly wrote: > On Tue, 2016-08-23 at 00:29 +0200, deloptes wrote: >> Patrick Ohly wrote: >> >> > Is the full code available somewhere? >> > >> >> No, I still don't know what to do with it and I'm not sure in which form >> I should upload it. Uploading it means maintain it ... etc. I was >> thinking to upload it to the trinity desktop, but still it is not clear >> how it would fit with the different distros. Another option would be to >> add it to syncevolution - perhaps it is even more proper ... anyway short >> answer is no - I could pass a zip file of it .. it's really not much >> code. The last but not least is I do not feel confident because of this >> issue and do not want to "release" is until it is solved. > > Fair enough. The common wisdom is to "release early, release often", but > in this case I agree that it is better to ensure that there is no data > loss before making it available to others. > The question is where. We've fixed some UTF related parts of the TDE code, which was bugging me since the time of opensync and smart phones started supporting UTF or something more than iso8859. Slowly it gets mature. I had a wish to implement support for vcal but gave it up as testing showed there are some issues with handling vcal in the TDE lib. I still suspect some issues there perhaps UTF related, but no time to work on this recently. So while the backend gets mature there are some challenges on the opposite side, but it is still good to have a plan what to do next. IMO it would be best to push into syncevolution tree and TDE could apply own debian rules for building a package which would replace the debian native package offering tdepim. Does someone has better ideas? >> So it looks like I misunderstood the ITEM_REPLACED meaning? Could you >> please confirm once again. I have been struggling with these part of the >> code for some time already. >> >> My questions >> 1. even when update is implemented as del+add and the item has the old ID >> it should return ITEM_OKAY? > > Yes. > >> 2. does ITEM_REPLACED means that item has new ID and is to replace the >> old item with the old ID? > > ITEM_REPLACED, ITEM_MERGED and ITEM_NEEDS_MERGE all are to be used only > when adding a new item, i.e. uid is empty. In this case, the engine > doesn't know about the existing item in the database and the backend has > to tell the engine about it. > > ITEM_NEEDS_MERGE is probably best for handling this situation because > merely replacing the existing item (ITEM_REPLACED) can loose data and > merging in the backend (ITEM_MERGED) is more work to implement. > >> Strange is only that I have such issues with calendar+todo only and a >> fact is that it messes totally up in slow sync. > > Not sure either. Perhaps it is because calendar items have a UID and > thus additional constraints that don't apply to contacts. ITEM_REPLACED, > ITEM_MERGED and ITEM_NEEDS_MERGE should never be needed for contacts, > because a contact backend typically will never be able to detect when a > new contact that is to be added already exists. > The story about ITEM_NEEDS_MERGE however is still a bit unclear to me. Is it a suggestion - at least I understand it as such? I think I lack understanding about it and will look into some of the other backends to get better insight. I see return InsertItemResult(luid, "", ITEM_NEEDS_MERGE); in ./src/backends/webdav/WebDAVSource.cpp and ./src/backends/webdav/CalDAVSource.cpp and there is no delete or add around. It means we find out there is item with such ID and do nothing - just curious what happens next in theory. I'll ask here if I need some assistance. Ideas and enlightening is always welcome of course. regards _______________________________________________ SyncEvolution mailing list SyncEvolution at syncevolution.org https://lists.syncevolution.org/mailman/listinfo/syncevolution From patrick.ohly at intel.com Wed Aug 24 07:09:18 2016 From: patrick.ohly at intel.com (Patrick Ohly) Date: Wed, 24 Aug 2016 09:09:18 +0200 Subject: [SyncEvolution] syncevolution debug help needed In-Reply-To: References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> <1471935212.24499.28.camel@intel.com> Message-ID: <1472022558.20290.4.camel@intel.com> On Wed, 2016-08-24 at 01:22 +0200, deloptes wrote: > IMO it would be best to push into syncevolution tree and TDE could apply own > debian rules for building a package which would replace the debian native > package offering tdepim. > Does someone has better ideas? Sounds like the right approach. I can't promise that I will do anything with the new backend code (like at least compile-testing it when making changes) because I'm not sure how much work it would be to make the dependencies available on my build host, but at least the code would be available for those who want to use it. > The story about ITEM_NEEDS_MERGE however is still a bit unclear to me. Is it > a suggestion - at least I understand it as such? It's a bit more than a suggestion. It's a request to the engine to do the merge. What happens is: 1. add new item -> ITEM_NEEDS_MERGE without changing the database 2. read old item with luid as provided with ITEM_NEEDS_MERGE 3. update old item with merged data -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. From deloptes at gmail.com Thu Aug 25 06:50:48 2016 From: deloptes at gmail.com (deloptes) Date: Thu, 25 Aug 2016 08:50:48 +0200 Subject: [SyncEvolution] syncevolution debug help needed References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> <1471935212.24499.28.camel@intel.com> <1472022558.20290.4.camel@intel.com> Message-ID: Patrick Ohly wrote: > On Wed, 2016-08-24 at 01:22 +0200, deloptes wrote: >> IMO it would be best to push into syncevolution tree and TDE could apply >> own debian rules for building a package which would replace the debian >> native package offering tdepim. >> Does someone has better ideas? > > Sounds like the right approach. I can't promise that I will do anything > with the new backend code (like at least compile-testing it when making > changes) because I'm not sure how much work it would be to make the > dependencies available on my build host, but at least the code would be > available for those who want to use it. > Last part is nice to have, so that it goes into the distros. I had some discussion with Tino here regarding debian package builds. As I am using debian it would be easy to get a package if the code is in there and just rebuild for trinity desktop. >> The story about ITEM_NEEDS_MERGE however is still a bit unclear to me. Is >> it a suggestion - at least I understand it as such? > > It's a bit more than a suggestion. It's a request to the engine to do > the merge. What happens is: > 1. add new item -> ITEM_NEEDS_MERGE without changing the database > 2. read old item with luid as provided with ITEM_NEEDS_MERGE > 3. update old item with merged data > So in this case I should not DEL+ADD in the backend, but just notify with ITEM_NEEDS_MERGE? Correct? (Sorry I did not have time to look in the other backends how this is exactly implemented). thanks and regards _______________________________________________ SyncEvolution mailing list SyncEvolution at syncevolution.org https://lists.syncevolution.org/mailman/listinfo/syncevolution From patrick.ohly at intel.com Thu Aug 25 08:52:27 2016 From: patrick.ohly at intel.com (Patrick Ohly) Date: Thu, 25 Aug 2016 10:52:27 +0200 Subject: [SyncEvolution] syncevolution debug help needed In-Reply-To: References: <1469786988.23795.18.camel@intel.com> <1471874545.24499.16.camel@intel.com> <1471935212.24499.28.camel@intel.com> <1472022558.20290.4.camel@intel.com> Message-ID: <1472115147.396.6.camel@intel.com> On Thu, 2016-08-25 at 08:50 +0200, deloptes wrote: > Patrick Ohly wrote: > >> The story about ITEM_NEEDS_MERGE however is still a bit unclear to me. Is > >> it a suggestion - at least I understand it as such? > > > > It's a bit more than a suggestion. It's a request to the engine to do > > the merge. What happens is: > > 1. add new item -> ITEM_NEEDS_MERGE without changing the database > > 2. read old item with luid as provided with ITEM_NEEDS_MERGE > > 3. update old item with merged data > > > > So in this case I should not DEL+ADD in the backend, but just notify with > ITEM_NEEDS_MERGE? Correct? Correct. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.