AppStream / Software-Center GSoC project #4

[Matthias Klumpp]

Hi!

Some more information about what I'm doing at time to make the
Software-Center cross-distro and usable with PackageKit.
At time I am in a pretty intense exam phase, so there are a little
less changes happening - more will be done after my biggest two exams
on 5th July are over.

Of course working hasn't stopped. Here's a short summary (I'll tell
some things again for better understanding)
The key element to get a working Software Center is PackageKit. When I
started the project, SC was incredibly slow. After some modifications,
it is now usable, which means that it will run with acceptable speed.
However, it is still much slower than it's native aptd version.
The current design of PackageKit allows only sequential processing of
transactions, meaning if one transaction (InstallPackages()) is
running, you can't run another one in parallel (GetDetails()). This
results in that odd case that users will not be able to browse the
software catalog while software is installed.

To solve that issue, I introduced the package cache, a SQLite cache of
all installed packages and their details. This worked well, but also
had all issues of a cache and was not very elegant and generally
disliked by many people, including me. So we looked for other
solutions and decided to implement "parallel transactions" after a
longer discussion.
"Parallel transactions" means that you will be able to run some
transactions, e.g. everything which reads the cache, in parallel to
others, for example one which installs packages. This will enable us
to not only show package details while installing stuff, but also to
fire many read transactions at the same time at get them processed
without delay. The final speed of PK will - of course - depend on how
well the backend is written.
Downside of this solution is that all backends have to be threadsafe
(or created processes and do IPC) and that this feature is non-trivial
to implement.

Richard Hughes (PackageKit maintainer) and I drafted the
specifications of parallel transactions, including some test-cases.
Essential for planning the implementation was that a running
transactions may never ever block existing ones. (Avoiding situations
like everything is waiting to release locks)
After planning, implementation started, where we now completely broke
backend API. First idea way to create one PkBackend instance per
PkTransaction, but after we did many parts of the implementation and I
had a working prototype already, Richard stated that initializing one
new backend per transaction is a bad idea and a big overhead. Another
problem which I had discussed with Daniel (aptcc backend maintainer)
before was that with this approach backends can't share cache
resources, as they don't have control over the running threads.
At first, we wanted to do a PkBackendPool, which would've contained a
pool of "activated" backend instances, but once again we avoided the
easy way to do it the right way(tm) (at least that's our impression
:P).
New solution is that there will be only one backend, but multiple
PkBackendJobs, which are submitted to the backend and executed. A
backend job is something like "install X,Y" or "get me the details of
package X".
Implementation is going on right now, and Richard is really amazing
with that, as he already updated some of the most difficult parts (I'm
writing exams, so not much time ^^). Of course this is still not done
yet.
There will be a PackageKit release of the new 0.8.x series we've been
working on the last weeks. This release will already contain many of
the SoC improvements, but I'd like to warn you not to use it in stable
distributions.
As soon as that is done, the PkBackendJob thing will be merged &
tweaked. - A lot of work from backend author will be needed too.

After PK is working properly (and probably even before all pieces are
at place) I'll go into SC again to fix some issues which were
bothering me a long time already. At time, PK is the number-one
priority, because without doing the work on PackageKit, a well-working
Software-Center is impossible.
I haven't seen that coming when I started my SoC, but hey, the changes
are awesome! Many other projects will also benefit from them, as there
are already some other efforts to write a SC on top of PK. (but
they're all in a pretty early state)

Next time, I won't write that much text, I somehow always feel the
need to explain every single step - probably a sideeffect from working
on cross-distro projects :-)
Expect some more news and a blogpost soon after I finished exams :-)
(I wanted have written a blogpost a month ago already, and the text is
nearly done...)

If you have questions, please ask or catch me on IRC (ximion,
Freenode) - I'll be 100% available efter 6th July again :-)
Cheers and thanks for reading!
    Matthias