Research for app purchases

Fri Oct 4 09:27:55 UTC 2019

I've started doing some research on support for purchases/donations,
and I plan to start writing the code for the lowlevel parts of
this. There is an initial proposed desing document written by Damián
which I'm using as a base:

https://docs.google.com/document/d/1zE_QbB6mtdhjH5bsFdf9kPrYvjKAMBRl4uSRYDt-BqQ

There are some initial issues first though that we need to figure
out. I want to start with support for:

 * Allow configuring an authenticator for remotes.
 * Allow the authenticator to provide a bearer token for pulls
   and/or getting the summary.
 * Store some kind of metadata in the repo to signal which refs need
   a bearer token so that we can mix and match pay and non-pay in
   one repo without a lot of overhead or privacy issues.
 * Add to the the APIs of libflatpak as needed to support the above.
 * Support/use the new APIs in the CLI commands.

Note, this initial work deliberately leaves out some important things:
 * Ability to purchase or donate, we just assume this happened elsewhere.
 * Ability query if you needs to buy an app before installing (for
   e.g. turning the install button to a buy button).
These will come later.

Having done some research on this I've noticed two problems:

First of all, how do we mark which refs need a bearer token. I
initially thought this would just need some extra data in the summary
file which we could use during the resolve phase in the
FlatpakTransaction code. However, in the collection-id/p2p case the
summary is not used/available during resolve but instead it downloads
each actual commit object and gets the metadata directly from that.

Unfortunately the commit object is what we expect to protect with a
bearer token, so we can't really do the resolve before getting the
bearer token. And additionally, once we download the commit object
further resolving using that may produce new things we need to
download (like e.g.  an Locale extension), which may need even more
bearer tokens.

Ideally how I would like transaction to work is:
 1. Take a list of input ops (ie. install/update these refs)
 2. Resolve this to a final list of which refs (& commits) needs to be
    downloaded.
 3. Request bearer tokens for all the refs in one go per remote.
 4. Pull each ref with the combined per-remote bearer token.

But the p2p case is throwing a spanner in all this as we need to bearer
token in the middle of step 2 already.

I see two alternatives here. Either we store the
does-this-need-a-bearer-token somewhere else and make getting the
bearer token an iterative thing where we may request tokens multiple
times during the resolve operation. Or, we make the commit object
freely downloadable and protect some other part of the app with the
bearer token. For instance the toplevel dirtree object (as well as the
deltas).

I'm leaning towards the second here, as this would make things much
more natural and efficient on the client side. However, that might
cause problems on the server side, because we can't just redirect all
files matching *.commit to the server that handles token verification,
which may cause CDN support to be inefficient.

Opinions on this?

The second issue I have is with the interactive parts of the
workflows. History has shown that anything that adds unexpected
interactions in the middle of what the caller would otherwise expect to be
"pure i/o" operations is a poor idea. You'll end up with weird
reentrancy issues, incorrectly parented modal dialogs blocking each
other, and just a generally poor UI. So, in terms of APIs I think we need
to be very explicit in where interaction will happen.

There are two major sources of interaction. The first is the actual
"purchase" operation. This is imho the simplest thing to solve, and
the existing design doc workflow describes this well. Basically we
will have some kind of API call that gnome-software (or equivalent)
calls to know whether it should show an "install" or "purchase"
button. And when the button is pressed it can initiate a full interactive
operation, with an API designed for this.

Once the ref is bought we can just trigger a regular install operation
which will verify the purchase and send the right tokens. If anything
goes wrong here we'll just return an error, assuming that the caller
did the correct check-and-purchase-if-needed before installin. So,
for the purchase itself, install (i.e. FlatpakTransaction) doesn't
need to do any interaction, it just calls out to the authenticator and
either gets a token that works or gets a "hasn't bought this ref" error.

For the CLI we can't really expect to do a web interaction flow for
the purchase, so I think we'll have to just print a URL and have the
user do the purchase on the side and then install again. (Or for donations,
just show the donation link with a message.)

However, there are still some cases where the FlatpakTransaction
operation will run into issues where it needs some form of
interaction.

For example, suppose flathub has purchasable refs, and your local
flathub remote is set up with a standardized authenticator (generic
config we got from the flatpakrepo file). Now we want to install a ref
marked as needing authentication, so we ask the authenticator for a
token for it. In the regular case this would just do some network call
and return either "need-to-buy" or the token.

However, here are some cases where this will not be enough:

 * This is the first time the user tried to install an authenticated ref, so
   we have no auth data. The user needs to create or select a user
   account on the flathub account servers and log in to it.

 * We have account information (ie. username), but it is not
   authenticated (at all or recently) so we need to (re)authenticate with
   the flathub account server.

The question is, how do we expose these in a sane way in the API and in
the CLI?

Some possible approaches for the workflow:

 1) Purely web-based, the authenticator hands out a URL to the app which
    shows the url in some embedded webkit widget, with some standardized
    way for the webpage to indicate that the operation is completed
    (successfully or not).
 2) Have a list of pre-defined interaction operations (like
    the typical username/password, etc) dialogs. Maybe with an
    additional link you can click on to open a completely separate
    browser page for more complicated thing like creating a new user.
 3) Have the authenticator itself open up a UI for doing whatever it
    needs, blocking the GetToken call until it is ready.

1 is more flexible, whereas 2 would be more limited but allow a nicer
native UI experience and useful CLI.

Alternative 3 has all the typical problems with an unepected sync call
doing UI stuff like reentrancy, multiple modal dialog, incorrect
parenting and generally unconnected UIs. However, it is possible to
make this at least a bit better by making it more explicit in the API,
allowing the app to be aware of the blocking and pass in things like
parent window references. Still, from historical experience I think
this is a bad solution.

Obviously we could also use any combination of these.

I'm leaning towards model 2 for the authentication, although we're going
to require a fully generic web flow for the actual purchase part.

The ideal workflow for FlatpakTransaction would work something like
this:
 1. Complete resolving the transaction so we know which refs require
    tokens.
 2. For each authenticator involved in the transaction, call some
    IsReady() method on them to see which need to do some kind
    of interaction to be ready.
 3. Emit a signal like "prepare-authenticators" which is given
    a list of object, each one representing a non-ready authenticator,
    and having some API to allow initiating and completing the
    required interaction.
 4. Request bearer tokens for all the refs, if the app failed to
    handle step 3 here we'll return a non-prepared error.
 5. Pull each ref with the combined per-remote bearer token.

This kind of assumes we can resolve the transactions and get the full
list of refs before doing authentication, which is a problem as per
above. If that part is instead iterative, then we have to make this
stuff also iterative, possibly emitting the prepare-authenticators
signal multiple times. Another problem is authenticators that require
a bearer token for the summary file (which will be some OCI servers).

So, a more pragmatic approach is to do the authenticator preparation
*before* resolving the initial ref set. This assumes that the resolve
doesn't add any new remotes that need authentication. In other words,
all runtimes/extensions are either in the same remote as its app, or
does not need authentication. I think this is realistically going to
be true in practice, however not ideal. Also, with this we might
require authentication for a "flatpak update" operation even if there
isn't an update available for the refs that need authentication.

Opinions? Ideas?

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl at redhat.com         alexander.larsson at gmail.com