Research for app purchases

[Dan Nicholson]

On Fri, Oct 4, 2019 at 3:28 AM Alexander Larsson <alexl at redhat.com> wrote:
>
> I've started doing some research on support for purchases/donations,
> and I plan to start writing the code for the lowlevel parts of
> this. There is an initial proposed desing document written by Damián
> which I'm using as a base:
>
> https://docs.google.com/document/d/1zE_QbB6mtdhjH5bsFdf9kPrYvjKAMBRl4uSRYDt-BqQ

Thanks for looking at this, and sorry for the slow feedback.

> There are some initial issues first though that we need to figure
> out. I want to start with support for:
>
>  * Allow configuring an authenticator for remotes.
>  * Allow the authenticator to provide a bearer token for pulls
>    and/or getting the summary.

Do you really want to require authentication for the summary? As you
note below, you want to resolve the refs needed for the transaction
before authenticating, and this would be difficult without the
summary.

>  * Store some kind of metadata in the repo to signal which refs need
>    a bearer token so that we can mix and match pay and non-pay in
>    one repo without a lot of overhead or privacy issues.

In the design above we had decided to leave the repo out of this
decision since it would require coordination between the repo and the
auth server. Instead we preferred to have the auth server solely
responsible for the authorization policy. For the case of mixed pay
and non-pay commits, we decided that the repo server would have a
simple boolean needs auth configuration. Then it would require tokens
for all refs and the auth server would just hand out tokens to all the
free commits regardless of what the user has purchased.

That does run into one issue you pointed out in the comments, though.
If you want to add pay apps to an existing repo and therefore require
token authentication for it, then you'll break old clients.

So, let's assume for now that the repo server is involved in the policy.

>  * Add to the the APIs of libflatpak as needed to support the above.
>  * Support/use the new APIs in the CLI commands.
>
> Note, this initial work deliberately leaves out some important things:
>  * Ability to purchase or donate, we just assume this happened elsewhere.
>  * Ability query if you needs to buy an app before installing (for
>    e.g. turning the install button to a buy button).
> These will come later.
>
> Having done some research on this I've noticed two problems:
>
> First of all, how do we mark which refs need a bearer token. I
> initially thought this would just need some extra data in the summary
> file which we could use during the resolve phase in the
> FlatpakTransaction code. However, in the collection-id/p2p case the
> summary is not used/available during resolve but instead it downloads
> each actual commit object and gets the metadata directly from that.

In the p2p case, the same information should be available from the
ostree-metadata commit. That's the entire purpose it exists - to
supplant the summary in p2p cases. Like the summary, I'd assume that
the ostree-metadata commit can be fetched without authentication. If
some information is only available in the summary but not in the
ostree-metadata commit, then that should be addressed.

> Ideally how I would like transaction to work is:
>  1. Take a list of input ops (ie. install/update these refs)
>  2. Resolve this to a final list of which refs (& commits) needs to be
>     downloaded.
>  3. Request bearer tokens for all the refs in one go per remote.
>  4. Pull each ref with the combined per-remote bearer token.

Yeah, I agree with this and we designed the auth server interface with
this in mind. I.e., allow the client to make a single call to the
remote's auth server with all desired refsand get back one token.

> But the p2p case is throwing a spanner in all this as we need to bearer
> token in the middle of step 2 already.
>
> I see two alternatives here. Either we store the
> does-this-need-a-bearer-token somewhere else and make getting the
> bearer token an iterative thing where we may request tokens multiple
> times during the resolve operation. Or, we make the commit object
> freely downloadable and protect some other part of the app with the
> bearer token. For instance the toplevel dirtree object (as well as the
> deltas).
>
> I'm leaning towards the second here, as this would make things much
> more natural and efficient on the client side. However, that might
> cause problems on the server side, because we can't just redirect all
> files matching *.commit to the server that handles token verification,
> which may cause CDN support to be inefficient.
>
> Opinions on this?

I believe it would be better to fix whatever is needed for resolving
refs from the ostree-metadata commit in the p2p case and continue
requiring authorization for the commit objects.

As you note, it would be more difficult to enumerate which dirtree
objects need authorization on the server side. I actually think using
the toplevel dirtree object might cause issues, too. Although the
metadata file in the root dirtree likely makes each root dirtree
unique, I don't think it's as robust as using the commit object.

> The second issue I have is with the interactive parts of the
> workflows. History has shown that anything that adds unexpected
> interactions in the middle of what the caller would otherwise expect to be
> "pure i/o" operations is a poor idea. You'll end up with weird
> reentrancy issues, incorrectly parented modal dialogs blocking each
> other, and just a generally poor UI. So, in terms of APIs I think we need
> to be very explicit in where interaction will happen.

For sure this was the part that was the most hand-wavey when we were
discussing the design. Ultimately you just need to pass a token to the
repo, but the process of acquiring the token in a user friendly way is
tricky.

> There are two major sources of interaction. The first is the actual
> "purchase" operation. This is imho the simplest thing to solve, and
> the existing design doc workflow describes this well. Basically we
> will have some kind of API call that gnome-software (or equivalent)
> calls to know whether it should show an "install" or "purchase"
> button. And when the button is pressed it can initiate a full interactive
> operation, with an API designed for this.
>
> Once the ref is bought we can just trigger a regular install operation
> which will verify the purchase and send the right tokens. If anything
> goes wrong here we'll just return an error, assuming that the caller
> did the correct check-and-purchase-if-needed before installin. So,
> for the purchase itself, install (i.e. FlatpakTransaction) doesn't
> need to do any interaction, it just calls out to the authenticator and
> either gets a token that works or gets a "hasn't bought this ref" error.

Maybe I'm missing something, but in order for the API to know whether
a ref is purchased or not, doesn't the user already need to be
authenticated? In other words, I don't think the purchase flow is that
different than the install flow. In either case, I think the API has
to return an error that authentication is needed which would trigger
the client to initiate that process.

> For the CLI we can't really expect to do a web interaction flow for
> the purchase, so I think we'll have to just print a URL and have the
> user do the purchase on the side and then install again. (Or for donations,
> just show the donation link with a message.)

Yes, I think that's the best you can probably do for the CLI. However,
if you just print a URL and the user handles that in their web
browser, the auth data still won't be available to flatpak or the
authenticator. So, I believe you'll still end up having to handle
authentication in the CLI later.

> However, there are still some cases where the FlatpakTransaction
> operation will run into issues where it needs some form of
> interaction.
>
> For example, suppose flathub has purchasable refs, and your local
> flathub remote is set up with a standardized authenticator (generic
> config we got from the flatpakrepo file). Now we want to install a ref
> marked as needing authentication, so we ask the authenticator for a
> token for it. In the regular case this would just do some network call
> and return either "need-to-buy" or the token.
>
> However, here are some cases where this will not be enough:
>
>  * This is the first time the user tried to install an authenticated ref, so
>    we have no auth data. The user needs to create or select a user
>    account on the flathub account servers and log in to it.
>
>  * We have account information (ie. username), but it is not
>    authenticated (at all or recently) so we need to (re)authenticate with
>    the flathub account server.
>
> The question is, how do we expose these in a sane way in the API and in
> the CLI?
>
> Some possible approaches for the workflow:
>
>  1) Purely web-based, the authenticator hands out a URL to the app which
>     shows the url in some embedded webkit widget, with some standardized
>     way for the webpage to indicate that the operation is completed
>     (successfully or not).
>  2) Have a list of pre-defined interaction operations (like
>     the typical username/password, etc) dialogs. Maybe with an
>     additional link you can click on to open a completely separate
>     browser page for more complicated thing like creating a new user.
>  3) Have the authenticator itself open up a UI for doing whatever it
>     needs, blocking the GetToken call until it is ready.
>
> 1 is more flexible, whereas 2 would be more limited but allow a nicer
> native UI experience and useful CLI.
>
> Alternative 3 has all the typical problems with an unepected sync call
> doing UI stuff like reentrancy, multiple modal dialog, incorrect
> parenting and generally unconnected UIs. However, it is possible to
> make this at least a bit better by making it more explicit in the API,
> allowing the app to be aware of the blocking and pass in things like
> parent window references. Still, from historical experience I think
> this is a bad solution.
>
> Obviously we could also use any combination of these.
>
> I'm leaning towards model 2 for the authentication, although we're going
> to require a fully generic web flow for the actual purchase part.

I am far from the expert on designing this part of the interaction.
One thing I worry about, though, is that authentication scheme and
data is inherently specific to the authenticator. That's why we
specified the authenticator as a separate component. Although there
are some common authentication data formats (e.g., an OAuth2 token),
it's entirely reasonable that the authenticator has its own way of
managing authentication data.

So, it seems to me that the authenticator needs to be the owner of the
authentication data and store it on disk in the format that is
appropriate for it. That to me implies that the authenticator likely
needs to handle the UI so that it can get back the data appropriately
and do whatever is needed so that it can re-use that data on a
subsequent interaction. If the app is managing the UI, then it would
need a way to pass back the data from the dialog to the authenticator.
That would require the app to have detailed knowledge about the
authentication scheme, I believe.

I'm probably missing something here, though.

> The ideal workflow for FlatpakTransaction would work something like
> this:
>  1. Complete resolving the transaction so we know which refs require
>     tokens.
>  2. For each authenticator involved in the transaction, call some
>     IsReady() method on them to see which need to do some kind
>     of interaction to be ready.
>  3. Emit a signal like "prepare-authenticators" which is given
>     a list of object, each one representing a non-ready authenticator,
>     and having some API to allow initiating and completing the
>     required interaction.
>  4. Request bearer tokens for all the refs, if the app failed to
>     handle step 3 here we'll return a non-prepared error.
>  5. Pull each ref with the combined per-remote bearer token.

That makes sense to me. Obviously at step 4 you may get a failure from
the authenticator if the ref hasn't been purchased or the user is
otherwise unauthorized to access that ref.

> This kind of assumes we can resolve the transactions and get the full
> list of refs before doing authentication, which is a problem as per
> above. If that part is instead iterative, then we have to make this
> stuff also iterative, possibly emitting the prepare-authenticators
> signal multiple times. Another problem is authenticators that require
> a bearer token for the summary file (which will be some OCI servers).

If the summary and ostree-metadata commit are both available without
authentication, then I think you can resolve the refs before
authentication. However, those do require authentication, then I
believe you're right that the authentication has to happen first.

> So, a more pragmatic approach is to do the authenticator preparation
> *before* resolving the initial ref set. This assumes that the resolve
> doesn't add any new remotes that need authentication. In other words,
> all runtimes/extensions are either in the same remote as its app, or
> does not need authentication. I think this is realistically going to
> be true in practice, however not ideal. Also, with this we might
> require authentication for a "flatpak update" operation even if there
> isn't an update available for the refs that need authentication.

I don't think it will always be true that all runtimes are in the same
remote as the app. I think it would be pretty reasonable that you have
an app in your repo that requires the freedesktop or GNOME runtime,
for instance.

So, I think if both the repo metadata requires authentication and you
might resolve required refs from other remotes that require
authentication, then you likely need an iterative approach in the
resolving phase. The only way to avoid authentication during resolving
is if the repo metadata is always available without authentication.

--
Dan