Research for app purchases

[Alexander Larsson]

On Thu, Oct 10, 2019 at 8:01 PM Dan Nicholson <nicholson at endlessm.com> wrote:
>
> On Fri, Oct 4, 2019 at 3:28 AM Alexander Larsson <alexl at redhat.com> wrote:
> >
> > I've started doing some research on support for purchases/donations,
> > and I plan to start writing the code for the lowlevel parts of
> > this. There is an initial proposed desing document written by Damián
> > which I'm using as a base:
> >
> > https://docs.google.com/document/d/1zE_QbB6mtdhjH5bsFdf9kPrYvjKAMBRl4uSRYDt-BqQ
>
> Thanks for looking at this, and sorry for the slow feedback.

Also, I've got some very work-in-progress code now, at:
  https://github.com/flatpak/flatpak/pull/3167

Its obviously not *useful* yet, but it lets us progress.

> > There are some initial issues first though that we need to figure
> > out. I want to start with support for:
> >
> >  * Allow configuring an authenticator for remotes.
> >  * Allow the authenticator to provide a bearer token for pulls
> >    and/or getting the summary.
>
> Do you really want to require authentication for the summary? As you
> note below, you want to resolve the refs needed for the transaction
> before authenticating, and this would be difficult without the
> summary.

I think you're using "authenticating" here for two different things. There is
1) Supplying a user id and password to log in to whatever service
handles purchases.
2) Determine, if the above user is allowed to download a ref and if so
give out a token for it.

Now, the request for protected summaries was raised for the OCI case,
which is a bit weird i guess because there is no real "summary file"
in the OCI case. However, what happens there is that we use some HTTP
call to get a bunch of json metadata which we combine into a local
file in the format of a summary file, so in an API sense its just
"getting the summary file". The issue there is that many OCI repos
will not let you get such metadata at all using a public API, *all*
requests have to be authenticated as a user. There probably will not
really be some *permissions* on the listing methods used, but you
still need a login on the service.

However, I don't think this is naturally an OCI-only thing. I can
imagine an OSTree repo where you have to sign in with your name and
password to even list what is available in the repo. Obviously this is
a different thing than the "per-ref" access, so it will be a different
kind of get_token() call in the API, one that we'll call before doing
anything at all in the transaction.

> >  * Store some kind of metadata in the repo to signal which refs need
> >    a bearer token so that we can mix and match pay and non-pay in
> >    one repo without a lot of overhead or privacy issues.
>
> In the design above we had decided to leave the repo out of this
> decision since it would require coordination between the repo and the
> auth server. Instead we preferred to have the auth server solely
> responsible for the authorization policy. For the case of mixed pay
> and non-pay commits, we decided that the repo server would have a
> simple boolean needs auth configuration. Then it would require tokens
> for all refs and the auth server would just hand out tokens to all the
> free commits regardless of what the user has purchased.
>
> That does run into one issue you pointed out in the comments, though.
> If you want to add pay apps to an existing repo and therefore require
> token authentication for it, then you'll break old clients.

I don't think it only affects old clients. It will fundamentally
change the behaviour of a download. Currently a flathub download is
just a few "mostly anonymous" http requests to a cdn. If we switch the
need_auth bit on then instead every user would need to create a user
on the flathub service with a password or whatnot, and every time you
want to download an update you'd have to log in and tell us about it.
This is bad for two reasons, first you add this complicated "create a
user on flathub" stage which is pretty ass if you were only going to
download free software, and secondly it will de-anonymize every
request even when not needed which seems like a poor privacy choice.

> > First of all, how do we mark which refs need a bearer token. I
> > initially thought this would just need some extra data in the summary
> > file which we could use during the resolve phase in the
> > FlatpakTransaction code. However, in the collection-id/p2p case the
> > summary is not used/available during resolve but instead it downloads
> > each actual commit object and gets the metadata directly from that.
>
> In the p2p case, the same information should be available from the
> ostree-metadata commit. That's the entire purpose it exists - to
> supplant the summary in p2p cases. Like the summary, I'd assume that
> the ostree-metadata commit can be fetched without authentication. If
> some information is only available in the summary but not in the
> ostree-metadata commit, then that should be addressed.

The problem with the ostree-metadata file is that it may not be 100%
in sync with all the refs in a p2p mirror. On a p2p mirror, the
summary file is generated locally, based on what exactly is available
in the repo, but the ostree-metadata ref is pulled from the master
repo at some point. For example, it could easily be pointing to the
very latest upstream ostree-metadata revision, which has a different
commit of some ref than what is locally available.

This is fine for certain kinds of operations, like tab completion or
listing things. However, the resolve phase requires certain things to
be 100% correct, such as permissions and dependencies. We can't claim
the app needs a set of permissions then install one with a different
set, or download a runtime for the app which isn't the one it will
actually use.

This is why the p2p case downloads the actual commit that will be
used, and I unfortunately can't see a way out of this.

> > I'm leaning towards the second here, as this would make things much
> > more natural and efficient on the client side. However, that might
> > cause problems on the server side, because we can't just redirect all
> > files matching *.commit to the server that handles token verification,
> > which may cause CDN support to be inefficient.
> >
> > Opinions on this?
>
> I believe it would be better to fix whatever is needed for resolving
> refs from the ostree-metadata commit in the p2p case and continue
> requiring authorization for the commit objects.
>
> As you note, it would be more difficult to enumerate which dirtree
> objects need authorization on the server side. I actually think using
> the toplevel dirtree object might cause issues, too. Although the
> metadata file in the root dirtree likely makes each root dirtree
> unique, I don't think it's as robust as using the commit object.

Yeah, I'm backpaddling on this one. I think we still need to protect
the commit objects. We just have to make this an iterative process. It
is unfortunate, as it makes the design a bit ugly and may in some
cases cause multiple dialogs where it might not otherwise be needed.
However, we can still do our best in batching get_token() calls.

> > There are two major sources of interaction. The first is the actual
> > "purchase" operation. This is imho the simplest thing to solve, and
> > the existing design doc workflow describes this well. Basically we
> > will have some kind of API call that gnome-software (or equivalent)
> > calls to know whether it should show an "install" or "purchase"
> > button. And when the button is pressed it can initiate a full interactive
> > operation, with an API designed for this.
> >
> > Once the ref is bought we can just trigger a regular install operation
> > which will verify the purchase and send the right tokens. If anything
> > goes wrong here we'll just return an error, assuming that the caller
> > did the correct check-and-purchase-if-needed before installin. So,
> > for the purchase itself, install (i.e. FlatpakTransaction) doesn't
> > need to do any interaction, it just calls out to the authenticator and
> > either gets a token that works or gets a "hasn't bought this ref" error.
>
> Maybe I'm missing something, but in order for the API to know whether
> a ref is purchased or not, doesn't the user already need to be
> authenticated? In other words, I don't think the purchase flow is that
> different than the install flow. In either case, I think the API has
> to return an error that authentication is needed which would trigger
> the client to initiate that process.

On a very high level, you're not wrong. However, the two flows are
pretty different. One involves in the un-common case something like
typing a password and in the common case just sending the
pre-authenticated token we have stored somewhere. The other involves a
complicated web flow with dollar amounts and entering of credit card
details.

These are justifiable different enough that we should perhaps handle
them different in the
API, so that for instance we could handle basic authentication in the
CLI, but punt on the full purchase webflow.

> > For the CLI we can't really expect to do a web interaction flow for
> > the purchase, so I think we'll have to just print a URL and have the
> > user do the purchase on the side and then install again. (Or for donations,
> > just show the donation link with a message.)
>
> Yes, I think that's the best you can probably do for the CLI. However,
> if you just print a URL and the user handles that in their web
> browser, the auth data still won't be available to flatpak or the
> authenticator. So, I believe you'll still end up having to handle
> authentication in the CLI later.

Yes, this is related to the different kinds of flow i'm talking about
above. If we can do the simpler "log in to the server" auth in the CLI
then this will work out fine.

Another way for this to work is to treat it more like gnome online
accounts work. I.e. the authenticator would have its own ui that you
explicitly run to set up your account and authenticate to it. Then
both the CLI and gnome-software would just error out saying you need
to enable/authenticate the account, which you do somewhere else.
Maybe we could even use goa, although that's a bit gnome specific,
does kde have something similar?

> > I'm leaning towards model 2 for the authentication, although we're going
> > to require a fully generic web flow for the actual purchase part.
>
> I am far from the expert on designing this part of the interaction.
> One thing I worry about, though, is that authentication scheme and
> data is inherently specific to the authenticator. That's why we
> specified the authenticator as a separate component. Although there
> are some common authentication data formats (e.g., an OAuth2 token),
> it's entirely reasonable that the authenticator has its own way of
> managing authentication data.
>
> So, it seems to me that the authenticator needs to be the owner of the
> authentication data and store it on disk in the format that is
> appropriate for it. That to me implies that the authenticator likely
> needs to handle the UI so that it can get back the data appropriately
> and do whatever is needed so that it can re-use that data on a
> subsequent interaction. If the app is managing the UI, then it would
> need a way to pass back the data from the dialog to the authenticator.
> That would require the app to have detailed knowledge about the
> authentication scheme, I believe.

Yes, the authenticator needs to *drive* the authentication. But that
doesn't necessarily mean it has to "render" it.

The way the simple case would work is that there would be some
standardized fields that the authenticator tells the app to ask for in
a dialog, and the result is then passed back to the authenticator
where it can store it or whatever, and then this happens in a
sequence. So, things are driven by  the authenticator and it gets all
info, and the app just renders the fields in a nice way with matching
fonts/themes, correct parenting and modality, use the desktop-specific
keyring for passwords, etc.

In the webflow case things are a bit more difficult, as the api is
much more generic. But one can imagine solutions. For example, the
authenticator could set up the operation ahead of time with the server
and when the operation is done go back to the server and ask for the
results. Or, we could define a standard way that the webflow results
gets encoded and passed to the authenticator. Or the authenticator can
proxy things via itself. Clearly needs research here, someone must
have done this before?

> So, I think if both the repo metadata requires authentication and you
> might resolve required refs from other remotes that require
> authentication, then you likely need an iterative approach in the
> resolving phase. The only way to avoid authentication during resolving
> is if the repo metadata is always available without authentication.

Yes, I think there really is no getting away from an iterative solution....

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl at redhat.com         alexander.larsson at gmail.com