Rust in our code base

[Karol Herbst]

On Wed, Aug 24, 2022 at 5:18 PM Jason Ekstrand
<jason.ekstrand at collabora.com> wrote:
>
> +mesa-dev and my jlekstrand.net e-mail
>
> On Sun, 2022-08-21 at 20:44 +0200, Karol Herbst wrote:
> > On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark at gmail.com>
> > wrote:
> > >
> > > On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst at redhat.com>
> > > wrote:
> > > >
> > > > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst at redhat.com>
> > > > wrote:
> > > > >
> > > > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark at gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst
> > > > > > <kherbst at redhat.com> wrote:
> > > > > > >
> > > > > > > Hey everybody,
> > > > > > >
> > > > > > > so I think it's time to have this discussion for real.
> > > > > > >
> > > > > > > I am working on Rusticl
> > > > > > > (
> > > > > > > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15
> > > > > > > 439)
> > > > > > > which I would like to merge quite soon.
> > > > > > >
> > > > > > > Others might also plan on starting kernel drivers written
> > > > > > > in Rust (and
> > > > > > > if people feel comfortable to discuss this as well, they
> > > > > > > might reply
> > > > > > > here)
> > > > > > >
> > > > > > > The overall implication of that is: if we are doing this,
> > > > > > > people (that
> > > > > > > is we) have to accept that touching Rust code will be part
> > > > > > > of our
> > > > > > > development process. There is no other sane way of doing
> > > > > > > it.
> > > > > > >
> > > > > > > I am not willing to wrap things in Rusticl so changing
> > > > > > > gallium APIs
> > > > > > > won't involve touching Rust code, and we also can't expect
> > > > > > > people to
> > > > > > > design their kernel drivers in weird ways "just because
> > > > > > > somebody
> > > > > > > doesn't want to deal with Rust"
> > > > > > >
> > > > > > > If we are going to do this, we have to do it for real,
> > > > > > > which means,
> > > > > > > Rust code will call C APIs directly and a change in those
> > > > > > > APIs will
> > > > > > > also require changes in Rust code making use of those APIs.
> > > > > > >
> > > > > > > I am so explicit on this very point, because we had some
> > > > > > > discussion on
> > > > > > > IRC where this was seen as a no-go at least from some
> > > > > > > people, which
> > > > > > > makes me think we have to find a mutual agreement on how it
> > > > > > > should be
> > > > > > > going forward.
> > > > > > >
> > > > > > > And I want to be very explicit here about the future of
> > > > > > > Rusticl as
> > > > > > > well: if the agreement is that people don't want to have to
> > > > > > > deal with
> > > > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am
> > > > > > > not
> > > > > > > willing to come up with some trashy external-internal API
> > > > > > > just to
> > > > > > > maintain Rusticl outside of the mesa git repo.
> > > > > > > And doing it on a kernel level is even more of a no-go.
> > > > > > >
> > > > > > > So what are we all thinking about Rust in our core repos?
> > > > > >
> > > > > > I think there has to be willingness on the part of rust folks
> > > > > > to help
> > > > > > others who aren't so familiar with rust with these sorts of
> > > > > > API
> > > > > > changes.  You can't completely impose the burden on others
> > > > > > who have
> > > > > > never touched rust before.  That said, I expect a lot of API
> > > > > > changes
> > > > > > over time are simple enough that other devs could figure out
> > > > > > the
> > > > > > related rust side changes.
> > > > > >
> > > > >
> > > > > yeah, I agree here. I wouldn't say it's all the responsibility
> > > > > of
> > > > > developers changing APIs to also know how to change the code.
> > > > > So e.g.
> > > > > if an MR fails to compile and it's because of rusticl, I will
> > > > > help out
> > > > > and do the changes myself if necessary. But long term we have
> > > > > to
> > > > > accept that API changes also come with the implication of also
> > > > > having
> > > > > to touch Rust code.
> > > > >
> > > > > Short term it might be a learning opportunity for some/most,
> > > > > but long
> > > > > term it has to be accepted as a part of development to deal
> > > > > with Rust.
> > > > >
> > > > > > As long as folks who want to start introducing rust in mesa
> > > > > > and drm
> > > > > > realize they are also signing up to play the role of rust
> > > > > > tutor and
> > > > > > technical assistance, I don't see a problem.  But if they
> > > > > > aren't
> > > > > > around and willing to help, I could see this going badly.
> > > > > >
> > > > >
> > > > > Yep, I fully agree here. This is also the main reason I am
> > > > > bringing
> > > > > this up. Nobody should be left alone with having to deal with
> > > > > changing
> > > > > the code. On the other hand a "not having to touch Rust code
> > > > > when
> > > > > changing APIs" guarantee is something which is simply
> > > > > impossible to
> > > > > have in any sane architecture. So we should figure out under
> > > > > which
> > > > > circumstances it will be okay for everybody.
> > >
> > > Yeah, this sounds fine to me.
> > >
> > > > > At least I don't see a way how I can structure Rusticl so that
> > > > > somebody working on gallium won't have to also deal with
> > > > > rusticl. One
> > > > > possibility would be to have a libgallium.so file I can link
> > > > > to, but
> > > > > then it's all about "stable gallium API" vs "not having to
> > > > > touch rust
> > > > > code" and I hope everybody prefers the second :)
> > > > >
> > > >
> > > > uhm... I meant "stable gallium API" vs "dealing with Rust code on
> > > > API changes"
> > >
> > > Yeah, stable gallium ABI makes as much sense as stable drm (in-
> > > kernel)
> > > ABI (ie. it doesn't).  So even if we tried to make it so that folks
> > > who haven't used rust much (or at all) yet on the gallium side,
> > > we'll
> > > still have to deal with it on the kernel side.  So let's not design
> > > our APIs to avoid a learning curve.. lets' just recognize that the
> > > learning curve will be there.
>
> I think this is mostly fine.  If you're making a gallium change that
> touches all the drivers, you need cross-project review as well.
> Learning a bit of rust or asking one of the rust-knowledgable people to
> update rusticl's wrappers seems reasonable at that point.
>
> Where I would take issue is if various wrappers or inlines are
> duplicated in rust such that a change which wouldn't break drivers
> because they're all using the wrappers do break rust code.  I know
> that, at least at one point, rusticl had some of this which is why I'm
> pointing it out specifically.  I've not looked at the branch in a
> while, though.
>

The only thing I am aware of is that I have an iterator written in
rust for glsl's list implementation in order to iterate over variables
inside nir shaders. But yeah, that has the potential to blow up if
somebody decides to change how the list works without changing the
definition of the list. But as I am actually running piglit on rusticl
with llvmpipe, this might not be a problem we wouldn't catch.

> > > > > > I do also wonder a bit about code tooling (indexers, etc)..
> > > > > > I'm not
> > > > > > sure what the state of things when it comes to cross c<->rust
> > > > > > integration.  Ie. it is usually straightforward enough to
> > > > > > track down
> > > > > > all the spots in C code which would be affected by some
> > > > > > change.  It
> > > > > > might be easier to overlook things on the rust side.  On the
> > > > > > mesa
> > > > > > side, pre-merge CI jobs help to catch these issues.  Less
> > > > > > sure about
> > > > > > how to handle that on the kernel side.
> > > > > >
> > > > >
> > > > > At least for Rusticl it's all within meson/ninja. We use
> > > > > bindgen to
> > > > > generate the bindings automatically so you simply run into
> > > > > compilation
> > > > > issues. And for the kernel side I think that Linus wanted Rust
> > > > > to be
> > > > > non optional. If something uses it, you also make sure the Rust
> > > > > side
> > > > > compiles. And the build system is dynamic enough that you can
> > > > > also
> > > > > wire up bindgen and make it part of the normal development
> > > > > model.
> > > > >
> > > > > In regards to code tooling, for rust you usually rely on
> > > > > rust-analyzer. I've already figured out with dcbaker on how to
> > > > > integrate our meson setup with that. Atm I am able to get the
> > > > > full
> > > > > experience with VScode. Not sure if we also need to figure out
> > > > > how
> > > > > that can work with e.g. vim and how to deal with C <=> Rust
> > > > > boundaries.
> > >
> > > Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> > > project that includes the combination of a significant amount of
> > > both
> > > .c and .rs.
> > >
> >
> > we have meson generate a rust-project.json file which you can point
> > rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682
> >
> > Sadly there is no "real" integration of rust and C code, so all you
> > get are references to the bindgen generated files, but you can't jump
> > to the actual C headers from there.
>
> I would be very surprised if any sort of C refactoring tool could see
> through bindgen now, or in the future.  Seeing across language
> boundaries is tractable in C because it's basically a superset of C++.
> (Yes, I know it's not really.)  I wouldn't expect a static analyzer to
> see across a C/Python or C/Java boundary, so I don't expect it to be
> able to see across C/rust.  (Sadly, the boundary is sharp enough that
> that's a reasonable comparison.)
>
> > > If bindgen is integrated with the build system, that sounds fine as
> > > far as helping to not overlook changes that are needed on the .rs
> > > side
> > > of things.. crosvm seems to mainly make bindgen a manual step, but
> > > then again it is mostly used for external dependencies like the
> > > interface to virglrenderer
> > >
> > > Side note, from my experience with crosvm, please be careful about
> > > any
> > > external rust dependencies that are used.. rust still seems to be
> > > young enough and evolving quickly enough that thing can quickly
> > > turn
> > > into a big headache
> > >
> >
> > That's easy: meson doesn't support external crates, so no external
> > rust dependencies are used at the moment :)
> >
> > But yeah, I was planning on using some where it really makes sense,
> > but we can have this discussion once we get there. So far it all
> > works
> > without any external dependency besides bindgen.
>
> I think most of us are agreed here.  Mesa has pretty strict rules about
> pulling in piles of external dependencies and we don't want that to
> change just because we added a bit of rust.  There may be a crate or
> two that we want to pull in in the future but, for now, everything is
> core.
>
> One point not mentioned above is Rust compiler versions.  Because Rust
> is a rapidly evolving language, new features are being added to the
> language and standard library every day.  If we impose the same
> requirements on Rust that we're imposing on GCC where we still compile
> with a version from 10-15 years ago, that's going to really hamper
> development.  We probably don't want to require people to always pull
> the very latest compiler from nightly but I do think we want to be able
> to bump the compiler version once a year or something.  IDK how distros
> and others will feel about that.  Hopefully, they feel about the same
> way towards compiler requirement bumps as they feel towards rust
> itself.
>

yeah.. atm I actually require 1.59 which is quite new (Feb 2022). I
think we might want to make sure that at least debians rust compiles
rusticl in the future, but they also seem to be willing to packaging
specific versions if they need it for other things like Firefox. They
have rustc (1.48) and rustc-mozilla (1.51), where the latter is even
available in stretch. So we might actually decide to simply depend on
the same rust version Firefox uses, because that's something
distributions will have to deal with anyway.

> --Jason
>
> > > BR,
> > > -R
> > >
> > > > > > BR,
> > > > > > -R
> > > > > >
> > > > > > > CCing a bunch of people who think are the most impacted by
> > > > > > > it either
> > > > > > > from a reviewer/maintainer or developer perspective. If you
> > > > > > > think some
> > > > > > > others should be explicitly aware of this, please point
> > > > > > > them to this
> > > > > > > discussion here.
> > > > > > >
> > > > > > > Karol
> > > > > > >
> > > > > >
> > > >
> > >
> >
>