Research for app purchases

[Dan Nicholson]

On Fri, Oct 11, 2019 at 2:02 AM Alexander Larsson <alexl at redhat.com> wrote:
>
> On Thu, Oct 10, 2019 at 8:01 PM Dan Nicholson <nicholson at endlessm.com> wrote:
> >
> > On Fri, Oct 4, 2019 at 3:28 AM Alexander Larsson <alexl at redhat.com> wrote:
> > >
> > > I've started doing some research on support for purchases/donations,
> > > and I plan to start writing the code for the lowlevel parts of
> > > this. There is an initial proposed desing document written by Damián
> > > which I'm using as a base:
> > >
> > > https://docs.google.com/document/d/1zE_QbB6mtdhjH5bsFdf9kPrYvjKAMBRl4uSRYDt-BqQ
> >
> > Thanks for looking at this, and sorry for the slow feedback.
>
> Also, I've got some very work-in-progress code now, at:
>   https://github.com/flatpak/flatpak/pull/3167
>
> Its obviously not *useful* yet, but it lets us progress.

Sure, I'll take a look.

> > > There are some initial issues first though that we need to figure
> > > out. I want to start with support for:
> > >
> > >  * Allow configuring an authenticator for remotes.
> > >  * Allow the authenticator to provide a bearer token for pulls
> > >    and/or getting the summary.
> >
> > Do you really want to require authentication for the summary? As you
> > note below, you want to resolve the refs needed for the transaction
> > before authenticating, and this would be difficult without the
> > summary.
>
> I think you're using "authenticating" here for two different things. There is
> 1) Supplying a user id and password to log in to whatever service
> handles purchases.
> 2) Determine, if the above user is allowed to download a ref and if so
> give out a token for it.

Indeed, I was being short with authentication vs authorization. I
think the design doc may even play a little loose with them.

That said, the concern of the email was with presenting an
authentication dialog to the user. Once the authenticator has valid
auth data (e.g., a non-expired OAuth2 token), then we can interact
with the auth server and the repo server without further interaction
with the user. Obviously, flatpak and/or the authenticator need to be
able to deal with failed authorization, but I think that's just a case
of passing back appropriate errors so that the application can present
the appropriate options to the user.

> Now, the request for protected summaries was raised for the OCI case,
> which is a bit weird i guess because there is no real "summary file"
> in the OCI case. However, what happens there is that we use some HTTP
> call to get a bunch of json metadata which we combine into a local
> file in the format of a summary file, so in an API sense its just
> "getting the summary file". The issue there is that many OCI repos
> will not let you get such metadata at all using a public API, *all*
> requests have to be authenticated as a user. There probably will not
> really be some *permissions* on the listing methods used, but you
> still need a login on the service.
>
> However, I don't think this is naturally an OCI-only thing. I can
> imagine an OSTree repo where you have to sign in with your name and
> password to even list what is available in the repo. Obviously this is
> a different thing than the "per-ref" access, so it will be a different
> kind of get_token() call in the API, one that we'll call before doing
> anything at all in the transaction.

At Endless we had decided to punt on the idea of limiting access to
the ref listing at the server level and do some things client side to
not show apps the user wasn't authorized to install. But I think
you're right that the process should likely allow for an
authentication/authorization requirement at that level.

> > >  * Store some kind of metadata in the repo to signal which refs need
> > >    a bearer token so that we can mix and match pay and non-pay in
> > >    one repo without a lot of overhead or privacy issues.
> >
> > In the design above we had decided to leave the repo out of this
> > decision since it would require coordination between the repo and the
> > auth server. Instead we preferred to have the auth server solely
> > responsible for the authorization policy. For the case of mixed pay
> > and non-pay commits, we decided that the repo server would have a
> > simple boolean needs auth configuration. Then it would require tokens
> > for all refs and the auth server would just hand out tokens to all the
> > free commits regardless of what the user has purchased.
> >
> > That does run into one issue you pointed out in the comments, though.
> > If you want to add pay apps to an existing repo and therefore require
> > token authentication for it, then you'll break old clients.
>
> I don't think it only affects old clients. It will fundamentally
> change the behaviour of a download. Currently a flathub download is
> just a few "mostly anonymous" http requests to a cdn. If we switch the
> need_auth bit on then instead every user would need to create a user
> on the flathub service with a password or whatnot, and every time you
> want to download an update you'd have to log in and tell us about it.
> This is bad for two reasons, first you add this complicated "create a
> user on flathub" stage which is pretty ass if you were only going to
> download free software, and secondly it will de-anonymize every
> request even when not needed which seems like a poor privacy choice.

Right. For our purposes we had no intention of bolting this on to an
existing free repo. But since that's the intention for flathub, let's
just forget about this model.

> > > First of all, how do we mark which refs need a bearer token. I
> > > initially thought this would just need some extra data in the summary
> > > file which we could use during the resolve phase in the
> > > FlatpakTransaction code. However, in the collection-id/p2p case the
> > > summary is not used/available during resolve but instead it downloads
> > > each actual commit object and gets the metadata directly from that.
> >
> > In the p2p case, the same information should be available from the
> > ostree-metadata commit. That's the entire purpose it exists - to
> > supplant the summary in p2p cases. Like the summary, I'd assume that
> > the ostree-metadata commit can be fetched without authentication. If
> > some information is only available in the summary but not in the
> > ostree-metadata commit, then that should be addressed.
>
> The problem with the ostree-metadata file is that it may not be 100%
> in sync with all the refs in a p2p mirror. On a p2p mirror, the
> summary file is generated locally, based on what exactly is available
> in the repo, but the ostree-metadata ref is pulled from the master
> repo at some point. For example, it could easily be pointing to the
> very latest upstream ostree-metadata revision, which has a different
> commit of some ref than what is locally available.
>
> This is fine for certain kinds of operations, like tab completion or
> listing things. However, the resolve phase requires certain things to
> be 100% correct, such as permissions and dependencies. We can't claim
> the app needs a set of permissions then install one with a different
> set, or download a runtime for the app which isn't the one it will
> actually use.
>
> This is why the p2p case downloads the actual commit that will be
> used, and I unfortunately can't see a way out of this.

Hmm, I wasn't aware of that but I see what you're saying. Can you
point me to any discussion/issues for that? I don't want to get off
onto a tangent, but I think this needs to be fixed for p2p. Here's a
strawman I thought of:

* p2p server generates a summary file, which will contain the refs
that it's mirroring. If the mirrored refs in the repo don't match
those in the ostree-metadata commit from the mirrored repo, the
summary generation fails.
* p2p client pulls both the summary and the ostree-metadata commit,
verifying only the ostree-metadata commit since its signature from the
mirrored remote
* p2p client validates the mirrored refs in the summary match the refs
in the ostree-metadata commit. If they don't match, it's a warning. If
they don't match and the remote defines an authenticator, then it's an
error. I haven't thought that all the way through, but the idea is
that old p2p clients and servers can still operate but if you want to
participate in purchasing by declaring an authenticator, then the p2p
server needs to DTRT.
* p2p client pulls only mirrored refs from the p2p server and falls
back to the remote for non-mirrored refs.

> > > I'm leaning towards the second here, as this would make things much
> > > more natural and efficient on the client side. However, that might
> > > cause problems on the server side, because we can't just redirect all
> > > files matching *.commit to the server that handles token verification,
> > > which may cause CDN support to be inefficient.
> > >
> > > Opinions on this?
> >
> > I believe it would be better to fix whatever is needed for resolving
> > refs from the ostree-metadata commit in the p2p case and continue
> > requiring authorization for the commit objects.
> >
> > As you note, it would be more difficult to enumerate which dirtree
> > objects need authorization on the server side. I actually think using
> > the toplevel dirtree object might cause issues, too. Although the
> > metadata file in the root dirtree likely makes each root dirtree
> > unique, I don't think it's as robust as using the commit object.
>
> Yeah, I'm backpaddling on this one. I think we still need to protect
> the commit objects. We just have to make this an iterative process. It
> is unfortunate, as it makes the design a bit ugly and may in some
> cases cause multiple dialogs where it might not otherwise be needed.
> However, we can still do our best in batching get_token() calls.
>
> > > There are two major sources of interaction. The first is the actual
> > > "purchase" operation. This is imho the simplest thing to solve, and
> > > the existing design doc workflow describes this well. Basically we
> > > will have some kind of API call that gnome-software (or equivalent)
> > > calls to know whether it should show an "install" or "purchase"
> > > button. And when the button is pressed it can initiate a full interactive
> > > operation, with an API designed for this.
> > >
> > > Once the ref is bought we can just trigger a regular install operation
> > > which will verify the purchase and send the right tokens. If anything
> > > goes wrong here we'll just return an error, assuming that the caller
> > > did the correct check-and-purchase-if-needed before installin. So,
> > > for the purchase itself, install (i.e. FlatpakTransaction) doesn't
> > > need to do any interaction, it just calls out to the authenticator and
> > > either gets a token that works or gets a "hasn't bought this ref" error.
> >
> > Maybe I'm missing something, but in order for the API to know whether
> > a ref is purchased or not, doesn't the user already need to be
> > authenticated? In other words, I don't think the purchase flow is that
> > different than the install flow. In either case, I think the API has
> > to return an error that authentication is needed which would trigger
> > the client to initiate that process.
>
> On a very high level, you're not wrong. However, the two flows are
> pretty different. One involves in the un-common case something like
> typing a password and in the common case just sending the
> pre-authenticated token we have stored somewhere. The other involves a
> complicated web flow with dollar amounts and entering of credit card
> details.
>
> These are justifiable different enough that we should perhaps handle
> them different in the
> API, so that for instance we could handle basic authentication in the
> CLI, but punt on the full purchase webflow.

I see what you're saying. I was focused on the "something needs to
show an authentication dialog" and not the "something needs to show a
purchasing dialog" part. Because flatpak has to be prepared to show an
authentication dialog in many scenarios if it either doesn't have any
auth data or the auth data has expired.

> > > For the CLI we can't really expect to do a web interaction flow for
> > > the purchase, so I think we'll have to just print a URL and have the
> > > user do the purchase on the side and then install again. (Or for donations,
> > > just show the donation link with a message.)
> >
> > Yes, I think that's the best you can probably do for the CLI. However,
> > if you just print a URL and the user handles that in their web
> > browser, the auth data still won't be available to flatpak or the
> > authenticator. So, I believe you'll still end up having to handle
> > authentication in the CLI later.
>
> Yes, this is related to the different kinds of flow i'm talking about
> above. If we can do the simpler "log in to the server" auth in the CLI
> then this will work out fine.

There are definitely cases where you could just get the auth details
from the user and the authenticator will be able to validate the
credentials in a non-interactive way. However, thinking about the
prominent HTTP authentication schemes like OpenID Connect and SAML,
there's a fairly complex back and forth. And one of the major design
goals of those authentication schemes is that you don't give your
credentials to the application. The application opens a connection to
the identity provider (i.e., a web page) where you enter your
credentials. The identity provider hands back a token (i.e. OAuth2) so
that the application can act on behalf of the user.

The simpler "log in to server" dialog where you provide your
credentials to the application implies that the application will act
*as* you. Likely it will want to store your credentials locally so
that it can authenticate non-interactively again later. I found
https://www.oauth.com/oauth2-servers/background/ to be useful when
thinking about this process.

Anyways, the point is that I think the authentication process should
cater to these types of authentication schemes where you do not enter
your credentials directly to the flatpak application.

> Another way for this to work is to treat it more like gnome online
> accounts work. I.e. the authenticator would have its own ui that you
> explicitly run to set up your account and authenticate to it. Then
> both the CLI and gnome-software would just error out saying you need
> to enable/authenticate the account, which you do somewhere else.
> Maybe we could even use goa, although that's a bit gnome specific,
> does kde have something similar?

GOA was immediately what I had in mind when we were discussing
authentication at Endless. I don't know if you'd want to use GOA
itself (particularly if you want to cater to custom authenticators),
but it would definitely be worth looking at the webkit code in there
for initiating typical authentication schemes.

I would hope that the flatpak application could call an API in the
authenticator to bring up the dialog, but you're far more versed in
how these types of interactions work.

> > > I'm leaning towards model 2 for the authentication, although we're going
> > > to require a fully generic web flow for the actual purchase part.
> >
> > I am far from the expert on designing this part of the interaction.
> > One thing I worry about, though, is that authentication scheme and
> > data is inherently specific to the authenticator. That's why we
> > specified the authenticator as a separate component. Although there
> > are some common authentication data formats (e.g., an OAuth2 token),
> > it's entirely reasonable that the authenticator has its own way of
> > managing authentication data.
> >
> > So, it seems to me that the authenticator needs to be the owner of the
> > authentication data and store it on disk in the format that is
> > appropriate for it. That to me implies that the authenticator likely
> > needs to handle the UI so that it can get back the data appropriately
> > and do whatever is needed so that it can re-use that data on a
> > subsequent interaction. If the app is managing the UI, then it would
> > need a way to pass back the data from the dialog to the authenticator.
> > That would require the app to have detailed knowledge about the
> > authentication scheme, I believe.
>
> Yes, the authenticator needs to *drive* the authentication. But that
> doesn't necessarily mean it has to "render" it.
>
> The way the simple case would work is that there would be some
> standardized fields that the authenticator tells the app to ask for in
> a dialog, and the result is then passed back to the authenticator
> where it can store it or whatever, and then this happens in a
> sequence. So, things are driven by  the authenticator and it gets all
> info, and the app just renders the fields in a nice way with matching
> fonts/themes, correct parenting and modality, use the desktop-specific
> keyring for passwords, etc.
>
> In the webflow case things are a bit more difficult, as the api is
> much more generic. But one can imagine solutions. For example, the
> authenticator could set up the operation ahead of time with the server
> and when the operation is done go back to the server and ask for the
> results. Or, we could define a standard way that the webflow results
> gets encoded and passed to the authenticator. Or the authenticator can
> proxy things via itself. Clearly needs research here, someone must
> have done this before?

I don't have a lot of experience on the client side here, and
definitely not in a desktop UI.

--
Dan