AppStream/SC/PK GSoC report #1

[Matthias Klumpp]

Hi!

This is my first report of the work I'm doing on AppStream and
PackageKit / the Software-Center.
It took a little longer to write it, university is giving me a hard
time, sorry for that!

Before the SoC development started, I already familiarized with the SC
code and learned a bit more Python, which will definitely help a lot
:-)
I ported the SC from using PkClient to PkTask, because PkTask is a
much more generic interface and the thing we want to use in the SC.
Unfortunately, the SC was still extremely slow, so I applied some
PackageKit-tricks to make it over 420% faster, I introduced some new
PK API for that.
Still this was not enough: If you install software, you are - due to
PK's design - unable to also query package details, which makes it
impossible to view application details while installing stuff. This
makes using the SC a pain and also has some problems when it comes to
searching stuff.

In the last two weeks, I discussed all these issues with Richard
Hughes, the PackageKit maintainer. I wrote a prototype of a SQLite
based package-cache, which will make querying data super-fast and will
make it possible to do this in parallel to PK transactions.
First, the cache was super-slow, but with some optimizations in
PkPackageSack routines, it now runs with acceptable speed so we can
use it. I also found some other optimizations together with Richard,
which will make PK itself a lot faster on some operations. (it will
save from a few msecs up to 10secs, depending on what you're trying to
do)

In the process of talking about changes and reviewing patches and
suggestions, Richard decided that it's time now to _really_ break
stuff and make it more sane. So right now we're doing a new PackageKit
0.8.x series, which will contain lots of cool improvements, not only
relevant for the stuff I want to do, but also for other new features,
like systemd-offline-upgrade.
My changes are already merged upstream and I will do a public API to
access the cache now, after discussing that upstream. Also, there are
some other changes on PKs DBus-API I and some other people would like
to have, discussion is going on there right now.
We're also improving backend interaction and broke plugin API, so
there's lots of stuff going on right now which will make developing a
SC based on PackageKit a lot easier.

At time, all PackageKit backends are broken and there's lots of stuff
which needs to be fixed and even more to be discussed.
Doing cross-distro-work is sometimes even more discussion than coding,
but I am certain right now that we will have a usable AppStream
Software-Center with the end of my SoC project.

If you read through this now, I'd like to ask you if you have anything
you want to do with PackageKit (which is not very backend-specific)
and haven't been able to do until now. - If there is any of these
issues, we might now be able to fix it.

Next steps are even more discussion and work on PackageKit to get the
API right, as well as some changes on SC and probably a C++
implementation of AppStream Xapian generator, so other implementations
of an AppStream-SC are possible (hey, KDE!) and we can use it even
without the SC.

Also, backend developers, please fix your backends soon! I want to
find people responsible for the Zypper backend, so they can take a
look at it and adjust it to our changes. (I have no knowledge about
Zypper, and developing in a VM is a pain, but I could of course make
some changes by myself later, if the basic stuff is working)
Test results about overall performance is helpful too.

--

Overall the start of this SoC was promising. First after experiencing
the first issues, I thought it might become extremely difficult to
complete this project, as there were many issues to work around, but
now problems are solved and we're really doing amazing stuff! Working
with Richard and the PK crows is a pleasure, as always and very
productive.
I'm looking forward to the next weeks, as everything is now working as
planned again. :-)
(Especially I need feedback from other distributions later)

Thanks,
   Matthias