[Freedesktop-sdk] Freedesktop sdk aka 'tiny base runtime' project

Mon Oct 30 18:23:31 UTC 2017

Thiago,

Seems the below mail from Tristan also didn't reach the xdg list.

On 30/10/17 13:15, Tristan Van Berkom wrote:
> Hi,
> 
> I can provide some background.
> 
> On Mon, 2017-10-30 at 10:49 +0000, Daniel Stone wrote:
>> Hi Laurence,
>>
>> On 27 October 2017 at 15:45, Laurence Urhegyi
>>> <laurence.urhegyi at codethink.co.uk> wrote:
>>> So i wanted to kick off this mailing list by trying to round up a few things
>>> which have been discussed recently, at GUADEC, in f2f conversations
>>> afterwards, or shot over emails between a few people. It's been discussed
>>> for a while that we want to create a tiny base runtime with bst. The upshot
>>> is this will homogenize the metadata used in the build process for Flatpak,
>>> GNOME Continuous and eventually many other projects too, we hope.
>>
>> Thiago's questions would be good to have an elaborated answer on. In
>> particular, whether BuildStream - which seems like it's only a few
>> months old - is a core part of the 'API' for the SDK, or whether it's
>> just incidentally used as part of the build process. I hadn't heard of
>> it until now, and would be curious to see if it's really used at all
>> outside of GNOME and Codethink.
> 
> Yes BuildStream is new, it's one year in the making and close to our
> 1.0 first stable release.
> 
> Before starting, we discussed this with the GNOME release team and the
> Flatpak maintainer, Alex - one of our main goals when starting the
> BuildStream project was to unify the build metadata used to deliver
> GNOME releases, run GNOME continuous integration and deliver Flatpak
> SDKs and Flatpaks.
> 
> Consider that for GNOME, we currently had 3 formats which the release
> team would basically have to be responsible for:
>    o JHBuild XML
>    o flatpak-builder JSON
>    o GNOME continuous JSON
> 
> All of which represent... basically different ways to build the same
> things.
> 
> Now we have the ball rolling and will start by migrating the existing
> JHBuild releases to use buildstream, consensus on that is pretty much
> reached in this thread[0], aside from talks we had face to face at
> GUADEC.
> 
> Next, we will be delivering the GNOME Flatpak SDK with BuildStream,
> from the *same* build data maintained by the GNOME release team.
> 
> At the same time, we will be migrating builds of the
> org.freedesktop.Sdk and org.freedesktop.BaseSdk Flatpak runtimes to use
> BuildStream build metadata and tooling - overall we will be maintaining
> less tooling, and having multiple BuildStream projects which logically
> depend on eachother helps to leverage the feature set (shared artifact
> caches, avoid export/import dances, etc).
> 
> This will include a tiny runtime for the base SDK, and something a bit
> bigger to match the Flatpak org.freedesktop.Sdk runtime.
> 
> In order to tie up the loop, it will make sense to have a minimal
> kernel build so that we can create bootable images - this will help us
> tie up the loop and allow GNOME developers to boot VMs with full GNOME
> builds and test the nitty gritty stuff like GDM and gnome-session.
> 
> So the bottom line here, is that we have a shared runtime to build with
> BuildStream, and neither the GNOME release team mailing list, or
> Flatpak mailing lists are suitable venues for discussion and planning
> of this work - since *both* projects have a common interest in making
> this happen.
> 
> Of course, we hope that this base runtime and potentially bootable
> minimal system layer can be of use to other projects besides the GNOME
> and Flatpak projects, so we will welcome other participants to
> contribute and be flexible - so long as the topic remains about
> building approximately the same stack using BuildStream (for instance,
> some parties might be interested in generating Docker images as output
> or other kinds of deployments of pretty much the same builds).
> 
>>> ## Objectives
>>>
>>> * Create and maintain a minimal base runtime using BuildStream definitions.
>>
>> Same question as above: is the goal of the SDK to use BuildStream
>> definitions, or is BuildStream just something which happens to be a
>> part of it?
> 
> The goal is to have a common base built with BuildStream, and the
> project itself will consist of BuildStream YAML.
> 
> There are other ways of building parts of stacks and deploying those,
> however tying them together to the same tooling is certainly the big
> picture here - so that we do have a unified picture of everything we
> build, and so they can all leverage the same features (e.g. projects
> which depend on other projects, and sharing of artifacts/build output
> using the same artifact sharing mechanisms).
> 
> Development of this is underway, we just need a proper venue to discuss
> it. Since this particular project should remain desktop/distro
> agnostic, and since our first goals are to build the freedesktop
> flatpak SDKs with BuildStream, freedesktop.org seems to be the logical
> venue for this.
> 
>>> * Establish neutrally located infrastructure to host the base runtime.
>>> * Implement an effective strategy for security updates to the runtime.
>>> * Ensure that the base runtime works with the relevant tooling in Flatpak
>>> and GNOME Continuous.
>>> * Replace flatpak builder with BuildStream to generate flatpak SDKs.
>>> * Replace Freedesktop Base Runtimes (based on YOCTO) with the base runtime.
>>> * Lastly, replace the GNOME Continuous (based on YOCTO) with the base
>>> runtime.
>>
>> I'd ideally suggest establishing the goals of the SDK in terms of what
>> it's meant to provide to external users. That's something which is
>> actually compelling and useful: fd.o doesn't usually get involved in
>> projects whose sole goal is to _use_ a new build system.
> 
> I dont think that continuing to build GNOME Continuous images and base
> Flatpak runtimes with Yocto is an option.
> 
> It makes no sense to use many different tools which produce
> approximately same things, we're just tripling our maintenance burden
> for no reason with that thinking.
> 
> I don't mind really how people want to phrase this, but if it sounds
> better this way:
> 
>    "The purpose of this project is rather to provide build metadata for
>     building a commonly maintained base runtime for both GNOME and
>     Flatpak projects.
> 
>     Others are welcome to join and consume the same metadata and add to
>     this, so long as there is not too much scope creep in terms of what
>     we build."
> 
> 
>>> Some points on the above:
>>> * We're on gitlab.com for now as an interim solution before we migrate to
>>> freedesktop infra, or freedesktop migrates to gitlab.
>>
>> For the record, I would very much like fd.o to migrate to GitLab.
>>
>>> * Hardware will required at some point for the infrastructure.
>>
>> What kind of hardware? Is that something you'd expect from fd.o, or
>> would you reuse, e.g. flathub and GNOME Continuous? Do you need
>> hosting, or build machines, or test runners, or ... ?
> 
> I'll try not to get too deep on this one, my assumption is that
> Laurence is referring to resources needed to run gitlab and runners to
> run CI, or at least continuous builds, on a variety of supported
> hardware - For this, I dont want to make an official statement but I'm
> quite certain that we can reuse existing hardware (we currently build
> the org.freedesktop.Sdk and friends on some arm machines donated to the
> GNOME project by Codethink[1], and I believe the Intel machine(s) used
> build the same are a Red Hat donation).
> 
> Asides from this, I'll leave it to others to reply, I suppose a simple
> static website could be nice (but gitlab's "pages" gives us this if we
> have fd.o migrate to gitlab anyway).
> 
> 
>>> ## Licence
>>>
>>> X / MIT
>>>
>>> No CLA to sign: All contributors hold their own copyright.
>>
>> Is this just for the SDK - which I understand at this point to be a
>> bunch of build recipes and a conversion script from JSON/Yocto to
>> YAML/BuildStream - or are you talking about BuildStream itself?
> 
> This point is about the actual YAML build metadata.
> 
> I.e. we would hope to avoid copyright / legal notices all over the
> place and instead have a single, ultra-permissive license, we think the
> X / MIT license is good for this mostly because it's popular and well
> understood.
> 
> 
> Hope I've cleared some things up here.
> 
> Best Regards,
>      -Tristan
> 
> 
> [0]: https://mail.gnome.org/archives/release-team/2017-October/msg00018.html
> [1]: https://blogs.gnome.org/tvb/2016/05/23/endless-and-codethink-team-up-for-gnome-on-arm/
> 
> 
> 