[Freedesktop-sdk] Freedesktop sdk aka 'tiny base runtime' project

Mon Oct 30 13:15:29 UTC 2017

Hi,

I can provide some background.

On Mon, 2017-10-30 at 10:49 +0000, Daniel Stone wrote:
> Hi Laurence,
> 
> On 27 October 2017 at 15:45, Laurence Urhegyi
> > <laurence.urhegyi at codethink.co.uk> wrote:
> > So i wanted to kick off this mailing list by trying to round up a few things
> > which have been discussed recently, at GUADEC, in f2f conversations
> > afterwards, or shot over emails between a few people. It's been discussed
> > for a while that we want to create a tiny base runtime with bst. The upshot
> > is this will homogenize the metadata used in the build process for Flatpak,
> > GNOME Continuous and eventually many other projects too, we hope.
> 
> Thiago's questions would be good to have an elaborated answer on. In
> particular, whether BuildStream - which seems like it's only a few
> months old - is a core part of the 'API' for the SDK, or whether it's
> just incidentally used as part of the build process. I hadn't heard of
> it until now, and would be curious to see if it's really used at all
> outside of GNOME and Codethink.

Yes BuildStream is new, it's one year in the making and close to our
1.0 first stable release.

Before starting, we discussed this with the GNOME release team and the
Flatpak maintainer, Alex - one of our main goals when starting the
BuildStream project was to unify the build metadata used to deliver
GNOME releases, run GNOME continuous integration and deliver Flatpak
SDKs and Flatpaks.

Consider that for GNOME, we currently had 3 formats which the release
team would basically have to be responsible for:
  o JHBuild XML
  o flatpak-builder JSON
  o GNOME continuous JSON

All of which represent... basically different ways to build the same
things.

Now we have the ball rolling and will start by migrating the existing
JHBuild releases to use buildstream, consensus on that is pretty much
reached in this thread[0], aside from talks we had face to face at
GUADEC.

Next, we will be delivering the GNOME Flatpak SDK with BuildStream,
from the *same* build data maintained by the GNOME release team.

At the same time, we will be migrating builds of the
org.freedesktop.Sdk and org.freedesktop.BaseSdk Flatpak runtimes to use
BuildStream build metadata and tooling - overall we will be maintaining
less tooling, and having multiple BuildStream projects which logically
depend on eachother helps to leverage the feature set (shared artifact
caches, avoid export/import dances, etc).

This will include a tiny runtime for the base SDK, and something a bit
bigger to match the Flatpak org.freedesktop.Sdk runtime.

In order to tie up the loop, it will make sense to have a minimal
kernel build so that we can create bootable images - this will help us
tie up the loop and allow GNOME developers to boot VMs with full GNOME
builds and test the nitty gritty stuff like GDM and gnome-session.

So the bottom line here, is that we have a shared runtime to build with
BuildStream, and neither the GNOME release team mailing list, or
Flatpak mailing lists are suitable venues for discussion and planning
of this work - since *both* projects have a common interest in making
this happen.

Of course, we hope that this base runtime and potentially bootable
minimal system layer can be of use to other projects besides the GNOME
and Flatpak projects, so we will welcome other participants to
contribute and be flexible - so long as the topic remains about
building approximately the same stack using BuildStream (for instance,
some parties might be interested in generating Docker images as output
or other kinds of deployments of pretty much the same builds).

> > ## Objectives
> > 
> > * Create and maintain a minimal base runtime using BuildStream definitions.
> 
> Same question as above: is the goal of the SDK to use BuildStream
> definitions, or is BuildStream just something which happens to be a
> part of it?

The goal is to have a common base built with BuildStream, and the
project itself will consist of BuildStream YAML.

There are other ways of building parts of stacks and deploying those,
however tying them together to the same tooling is certainly the big
picture here - so that we do have a unified picture of everything we
build, and so they can all leverage the same features (e.g. projects
which depend on other projects, and sharing of artifacts/build output
using the same artifact sharing mechanisms).

Development of this is underway, we just need a proper venue to discuss
it. Since this particular project should remain desktop/distro
agnostic, and since our first goals are to build the freedesktop
flatpak SDKs with BuildStream, freedesktop.org seems to be the logical
venue for this.

> > * Establish neutrally located infrastructure to host the base runtime.
> > * Implement an effective strategy for security updates to the runtime.
> > * Ensure that the base runtime works with the relevant tooling in Flatpak
> > and GNOME Continuous.
> > * Replace flatpak builder with BuildStream to generate flatpak SDKs.
> > * Replace Freedesktop Base Runtimes (based on YOCTO) with the base runtime.
> > * Lastly, replace the GNOME Continuous (based on YOCTO) with the base
> > runtime.
> 
> I'd ideally suggest establishing the goals of the SDK in terms of what
> it's meant to provide to external users. That's something which is
> actually compelling and useful: fd.o doesn't usually get involved in
> projects whose sole goal is to _use_ a new build system.

I dont think that continuing to build GNOME Continuous images and base
Flatpak runtimes with Yocto is an option.

It makes no sense to use many different tools which produce
approximately same things, we're just tripling our maintenance burden
for no reason with that thinking.

I don't mind really how people want to phrase this, but if it sounds
better this way:

  "The purpose of this project is rather to provide build metadata for
   building a commonly maintained base runtime for both GNOME and
   Flatpak projects.

   Others are welcome to join and consume the same metadata and add to
   this, so long as there is not too much scope creep in terms of what
   we build."

> > Some points on the above:
> > * We're on gitlab.com for now as an interim solution before we migrate to
> > freedesktop infra, or freedesktop migrates to gitlab.
> 
> For the record, I would very much like fd.o to migrate to GitLab.
> 
> > * Hardware will required at some point for the infrastructure.
> 
> What kind of hardware? Is that something you'd expect from fd.o, or
> would you reuse, e.g. flathub and GNOME Continuous? Do you need
> hosting, or build machines, or test runners, or ... ?

I'll try not to get too deep on this one, my assumption is that
Laurence is referring to resources needed to run gitlab and runners to
run CI, or at least continuous builds, on a variety of supported
hardware - For this, I dont want to make an official statement but I'm
quite certain that we can reuse existing hardware (we currently build
the org.freedesktop.Sdk and friends on some arm machines donated to the
GNOME project by Codethink[1], and I believe the Intel machine(s) used
build the same are a Red Hat donation).

Asides from this, I'll leave it to others to reply, I suppose a simple
static website could be nice (but gitlab's "pages" gives us this if we
have fd.o migrate to gitlab anyway).

> > ## Licence
> > 
> > X / MIT
> > 
> > No CLA to sign: All contributors hold their own copyright.
> 
> Is this just for the SDK - which I understand at this point to be a
> bunch of build recipes and a conversion script from JSON/Yocto to
> YAML/BuildStream - or are you talking about BuildStream itself?

This point is about the actual YAML build metadata.

I.e. we would hope to avoid copyright / legal notices all over the
place and instead have a single, ultra-permissive license, we think the
X / MIT license is good for this mostly because it's popular and well
understood.

Hope I've cleared some things up here.

Best Regards,
    -Tristan

[0]: https://mail.gnome.org/archives/release-team/2017-October/msg00018.html
[1]: https://blogs.gnome.org/tvb/2016/05/23/endless-and-codethink-team-up-for-gnome-on-arm/