[packagekit] [Ubuntu-appstore-developers] D-Bus API for Click Packages (PackageKit/AptDaemon)

[Colin Watson]

On Thu, Jun 27, 2013 at 05:32:02PM +0200, Sebastian Heinlein wrote:
> At the virtual UDS session about click packages there was a short side
> discussion about the API for the communication between the GUI
> components and the higher privileged click package system. I would like
> to raise this issue again.

Thanks for raising this; I've been meaning to talk with you for a while
now.

> There are currently several existing IPC methods for managing software
> in Ubuntu (AptDaemon system D-Bus, PackageKit system and session D-Bus
> or command line interfaces e.g. apt-add-repository or the dash/s-c glue
> code). It would be nice to avoid a wider fragmentation with the
> introduction of click packages.
> 
> AFAIK the required API of the AppStore would be quite small and the
> package downloading and update calculation is done in the GUI component,
> right? So we could end up with a very small set of methods e.g.
> InstallFile() RemovePackages(), GetPackages(FILTER_INSTALLED)?

I believe so, at least as a first pass.  These would roughly map to
"click install", "click remove", and (forthcoming) "click list".

There are some complexities relating to per-user installation of
packages, which is a requirement: that is, all packages will be unpacked
as a privileged user, but it's possible to unpack multiple parallel
versions of a package and each user will be able to register their own
"current version" links.  Do you have any thoughts on whether this kind
of dual operation (unpack as privileged user, link as ordinary user)
should be done as two D-Bus methods or one?

> Aptdaemon is used by Software Center, Update-Manager and
> SessionInstaller for managing software in Ubuntu. Since APT is only
> accessed in the aptdaemon.worker.Worker class - the class could be
> replaced by a worker that makes use of click packages. The disadvantage
> of this solution is a missing non-Python based client library.

I was indeed looking at aptdaemon for this.  As an alternative to
replacing the whole worker, do you think it might make sense to modify
apt.debfile.DebPackage.install to understand click packages (a rather
small change, at least if it only handles the privileged side of
things)?  Since they use the .deb container format, that seems like a
convenient class to reuse.  Part of the point of using the same
container format was to reduce the amount of work we needed to do at
this kind of layer, for code that's fundamentally not that interesting.

I'm not wedded to that idea.  If it makes sense to use a separate
backend and maybe just share some common libraries, that's fine too.

For illustration, I've been playing with things like this:

=== modified file 'apt/debfile.py'
--- apt/debfile.py	2012-10-01 14:32:35 +0000
+++ apt/debfile.py	2013-07-03 12:41:58 +0000
@@ -621,8 +621,15 @@ class DebPackage(object):
         if level <= self.debug:
             print >> sys.stderr, msg
 
+    @property
+    def is_click_package(self):
+        return "click-version" in self
+
     def install(self, install_progress=None):
         """Install the package."""
+        if self.is_click_package:
+            return os.spawnlp(
+                os.P_WAIT, "click", "click", "install", self.filename)
         if install_progress is None:
             return os.spawnlp(os.P_WAIT, "dpkg", "dpkg", "-i", self.filename)
         else:

=== modified file 'aptdaemon/worker.py'
--- aptdaemon/worker.py	2012-12-29 19:27:25 +0000
+++ aptdaemon/worker.py	2013-07-03 12:56:23 +0000
@@ -1321,6 +1321,16 @@ class AptWorker(GObject.GObject):
         return (depends, required_download, required_space, unauthenticated,
                 high_trust_packages)
 
+    def _is_click_package(self, path):
+        """Check whether a Debian package is really a Click package."""
+        try:
+            deb = apt.debfile.DebPackage(path, self._cache)
+            # '"Click-Version" in deb' requires python-apt 0.8.9.1
+            deb["Click-Version"]
+            return deb
+        except Exception:
+            return None
+
     def _check_deb_file(self, trans, path, force):
         """Perform some basic checks for the Debian package.
 
@@ -1330,6 +1340,9 @@ class AptWorker(GObject.GObject):
         """
         if not os.path.isfile(path):
             raise TransactionFailed(ERROR_UNREADABLE_PACKAGE_FILE, path)
+        deb = self._is_click_package(path)
+        if deb is not None:
+            return deb
         if not force and os.path.isfile("/usr/bin/lintian"):
             with DaemonLintianProgress(trans) as progress:
                 progress.run(path)

Obviously this is incomplete, but I don't think it necessarily has to be
all that much code.  Thoughts welcome.

> Furthermore it limits the click package implementation to Python or
> requires a Python library to access the click packages system.

Not really.  It would make sense to invoke /usr/bin/click as a
subprocess, which the relevant parts of aptdaemon can already do without
much difficulty, and as mentioned above we can reuse existing Python
libraries for things like handling some metadata.  But it depends how
much sophistication you think aptdaemon would need.

In any case, click is currently in Python.  I do want to reserve the
option to rewrite it in C at some point, but for the moment I don't see
the urgent need to do so; rapid prototyping is valuable, it should be
able to meet its performance requirements while remaining in Python, and
the limitations on using Python on the Ubuntu phone relate much more to
long-running processes than to things like this that will be relatively
ephemeral.

> On the other hand there is the PackageKit daemon and its D-Bus API. The
> PackageKit daemon allows to write backends in different languages: the
> current Python based click package system could use a spawned backend.
> If it is still the plan, click packages could be re-written in C/C++
> later without having to do any changes on the communication between the
> daemon and the client.

I'm not sure I understand how this isn't already the case with
aptdaemon.  Can you elaborate?

> Furthermore PackageKit features a nice GLib/GObject introspection
> client library - and even a QT based one.
> 
> Since aptdaemon provides a PackageKit D-Bus interface implementation
> next to the native aptdaemon interface you could theoretically use the
> packagekit daemon on the phone and keep aptdaemon on the desktop. I
> don't know what the future of software-center will be at all and how it
> fits into the scenario.
> 
> The support for the latest PackageKit 0.8 system D-Bus interface landed
> in aptdaemon's trunk today. I plan to release and upload a 1.1 version
> of aptdaemon soon. This will unblock the transition to packagekit 0.8.9
> for saucy.

That would be great, and let me know if I can help with that.  I'm
really agnostic as to whether we use aptdaemon or PackageKit as the
backend; I think it definitely makes sense to use one of them rather
than inventing some new service, but I wanted to avoid getting caught up
in the PackageKit transition because it isn't really something I have a
stake in either way and I want to be able to deliver click packages
independently of it.  For that matter, picking one for now doesn't
necessarily tie us down either.

As for software-center, I don't know either (not really my field).  The
immediate plans for click packages for the Ubuntu phone centre around
calling some appropriate D-Bus interfaces directly from the Unity Dash,
so that's what I'm focusing on.

It's probably best to use the PackageKit interfaces if at all possible,
even if aptdaemon is currently responsible for implementing them.  Some
concepts won't transfer: there isn't really a repository list as such
(it's all search-based), and I believe that we want the phone's download
service to be primarily responsible for package acquisition (though I
suppose it might be possible to hook
org.freedesktop.PackageKit.Transaction.DownloadPackages up to that; I
haven't seen what interfaces it will provide yet).  But presumably we
can just stub out things that don't apply.

One thing I think is important is to make sure that we don't end up
incidentally loading the apt cache.  Firstly, this will be relatively
slow and memory-hungry; secondly, in the phone's default configuration
outside developer mode any attempts to install packages using apt will
fail since the system partition will be read-only.  Would avoiding the
apt cache require a separate worker backend?

> I could assist in modifying aptdaemon's worker or in writing a Python
> PackageKit backend for click packages. The old and somehow obsoleted
> Python based apt backend (in contrast to the aptcc backend) also
> demonstrates how to write a test suite for Python backends.

I would definitely appreciate advice and assistance on doing this the
right way.  I've yet to start on the system/user split in click, which
is probably going to be influenced by how we want the
aptdaemon/PackageKit APIs to look, so it would be good to get that
nailed down.

> PackageKit provides two D-Bus interfaces to manage software. The system
> D-Bus interface should be used by the native client libraries. For
> applications that cannot access the client libraries or don't want to
> provide a separate graphical user interface that handles errors or shows
> the progress there is also a small session D-Bus interface which is
> implemented by gnome-packagekit and apper.

Do we need to use a session interface for the kinds of per-user work I
described?

Thanks,

-- 
Colin Watson                                       [cjwatson at ubuntu.com]