Rust in our code base

Wed Aug 24 15:18:39 UTC 2022

+mesa-dev and my jlekstrand.net e-mail

On Sun, 2022-08-21 at 20:44 +0200, Karol Herbst wrote:
> On Sun, Aug 21, 2022 at 8:34 PM Rob Clark <robdclark at gmail.com>
> wrote:
> > 
> > On Sun, Aug 21, 2022 at 10:45 AM Karol Herbst <kherbst at redhat.com>
> > wrote:
> > > 
> > > On Sun, Aug 21, 2022 at 7:43 PM Karol Herbst <kherbst at redhat.com>
> > > wrote:
> > > > 
> > > > On Sun, Aug 21, 2022 at 5:46 PM Rob Clark <robdclark at gmail.com>
> > > > wrote:
> > > > > 
> > > > > On Sat, Aug 20, 2022 at 5:23 AM Karol Herbst
> > > > > <kherbst at redhat.com> wrote:
> > > > > > 
> > > > > > Hey everybody,
> > > > > > 
> > > > > > so I think it's time to have this discussion for real.
> > > > > > 
> > > > > > I am working on Rusticl
> > > > > > (
> > > > > > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15
> > > > > > 439)
> > > > > > which I would like to merge quite soon.
> > > > > > 
> > > > > > Others might also plan on starting kernel drivers written
> > > > > > in Rust (and
> > > > > > if people feel comfortable to discuss this as well, they
> > > > > > might reply
> > > > > > here)
> > > > > > 
> > > > > > The overall implication of that is: if we are doing this,
> > > > > > people (that
> > > > > > is we) have to accept that touching Rust code will be part
> > > > > > of our
> > > > > > development process. There is no other sane way of doing
> > > > > > it.
> > > > > > 
> > > > > > I am not willing to wrap things in Rusticl so changing
> > > > > > gallium APIs
> > > > > > won't involve touching Rust code, and we also can't expect
> > > > > > people to
> > > > > > design their kernel drivers in weird ways "just because
> > > > > > somebody
> > > > > > doesn't want to deal with Rust"
> > > > > > 
> > > > > > If we are going to do this, we have to do it for real,
> > > > > > which means,
> > > > > > Rust code will call C APIs directly and a change in those
> > > > > > APIs will
> > > > > > also require changes in Rust code making use of those APIs.
> > > > > > 
> > > > > > I am so explicit on this very point, because we had some
> > > > > > discussion on
> > > > > > IRC where this was seen as a no-go at least from some
> > > > > > people, which
> > > > > > makes me think we have to find a mutual agreement on how it
> > > > > > should be
> > > > > > going forward.
> > > > > > 
> > > > > > And I want to be very explicit here about the future of
> > > > > > Rusticl as
> > > > > > well: if the agreement is that people don't want to have to
> > > > > > deal with
> > > > > > Rust changing e.g. gallium, Rusticl is a dead project. I am
> > > > > > not
> > > > > > willing to come up with some trashy external-internal API
> > > > > > just to
> > > > > > maintain Rusticl outside of the mesa git repo.
> > > > > > And doing it on a kernel level is even more of a no-go.
> > > > > > 
> > > > > > So what are we all thinking about Rust in our core repos?
> > > > > 
> > > > > I think there has to be willingness on the part of rust folks
> > > > > to help
> > > > > others who aren't so familiar with rust with these sorts of
> > > > > API
> > > > > changes.  You can't completely impose the burden on others
> > > > > who have
> > > > > never touched rust before.  That said, I expect a lot of API
> > > > > changes
> > > > > over time are simple enough that other devs could figure out
> > > > > the
> > > > > related rust side changes.
> > > > > 
> > > > 
> > > > yeah, I agree here. I wouldn't say it's all the responsibility
> > > > of
> > > > developers changing APIs to also know how to change the code.
> > > > So e.g.
> > > > if an MR fails to compile and it's because of rusticl, I will
> > > > help out
> > > > and do the changes myself if necessary. But long term we have
> > > > to
> > > > accept that API changes also come with the implication of also
> > > > having
> > > > to touch Rust code.
> > > > 
> > > > Short term it might be a learning opportunity for some/most,
> > > > but long
> > > > term it has to be accepted as a part of development to deal
> > > > with Rust.
> > > > 
> > > > > As long as folks who want to start introducing rust in mesa
> > > > > and drm
> > > > > realize they are also signing up to play the role of rust
> > > > > tutor and
> > > > > technical assistance, I don't see a problem.  But if they
> > > > > aren't
> > > > > around and willing to help, I could see this going badly.
> > > > > 
> > > > 
> > > > Yep, I fully agree here. This is also the main reason I am
> > > > bringing
> > > > this up. Nobody should be left alone with having to deal with
> > > > changing
> > > > the code. On the other hand a "not having to touch Rust code
> > > > when
> > > > changing APIs" guarantee is something which is simply
> > > > impossible to
> > > > have in any sane architecture. So we should figure out under
> > > > which
> > > > circumstances it will be okay for everybody.
> > 
> > Yeah, this sounds fine to me.
> > 
> > > > At least I don't see a way how I can structure Rusticl so that
> > > > somebody working on gallium won't have to also deal with
> > > > rusticl. One
> > > > possibility would be to have a libgallium.so file I can link
> > > > to, but
> > > > then it's all about "stable gallium API" vs "not having to
> > > > touch rust
> > > > code" and I hope everybody prefers the second :)
> > > > 
> > > 
> > > uhm... I meant "stable gallium API" vs "dealing with Rust code on
> > > API changes"
> > 
> > Yeah, stable gallium ABI makes as much sense as stable drm (in-
> > kernel)
> > ABI (ie. it doesn't).  So even if we tried to make it so that folks
> > who haven't used rust much (or at all) yet on the gallium side,
> > we'll
> > still have to deal with it on the kernel side.  So let's not design
> > our APIs to avoid a learning curve.. lets' just recognize that the
> > learning curve will be there.

I think this is mostly fine.  If you're making a gallium change that
touches all the drivers, you need cross-project review as well. 
Learning a bit of rust or asking one of the rust-knowledgable people to
update rusticl's wrappers seems reasonable at that point.

Where I would take issue is if various wrappers or inlines are
duplicated in rust such that a change which wouldn't break drivers
because they're all using the wrappers do break rust code.  I know
that, at least at one point, rusticl had some of this which is why I'm
pointing it out specifically.  I've not looked at the branch in a
while, though.

> > > > > I do also wonder a bit about code tooling (indexers, etc)..
> > > > > I'm not
> > > > > sure what the state of things when it comes to cross c<->rust
> > > > > integration.  Ie. it is usually straightforward enough to
> > > > > track down
> > > > > all the spots in C code which would be affected by some
> > > > > change.  It
> > > > > might be easier to overlook things on the rust side.  On the
> > > > > mesa
> > > > > side, pre-merge CI jobs help to catch these issues.  Less
> > > > > sure about
> > > > > how to handle that on the kernel side.
> > > > > 
> > > > 
> > > > At least for Rusticl it's all within meson/ninja. We use
> > > > bindgen to
> > > > generate the bindings automatically so you simply run into
> > > > compilation
> > > > issues. And for the kernel side I think that Linus wanted Rust
> > > > to be
> > > > non optional. If something uses it, you also make sure the Rust
> > > > side
> > > > compiles. And the build system is dynamic enough that you can
> > > > also
> > > > wire up bindgen and make it part of the normal development
> > > > model.
> > > > 
> > > > In regards to code tooling, for rust you usually rely on
> > > > rust-analyzer. I've already figured out with dcbaker on how to
> > > > integrate our meson setup with that. Atm I am able to get the
> > > > full
> > > > experience with VScode. Not sure if we also need to figure out
> > > > how
> > > > that can work with e.g. vim and how to deal with C <=> Rust
> > > > boundaries.
> > 
> > Yeah, I've used vscode with rust codebase (crosvm).. just not for a
> > project that includes the combination of a significant amount of
> > both
> > .c and .rs.
> > 
> 
> we have meson generate a rust-project.json file which you can point
> rust-analyzer to: https://github.com/mesonbuild/meson/pull/10682
> 
> Sadly there is no "real" integration of rust and C code, so all you
> get are references to the bindgen generated files, but you can't jump
> to the actual C headers from there.

I would be very surprised if any sort of C refactoring tool could see
through bindgen now, or in the future.  Seeing across language
boundaries is tractable in C because it's basically a superset of C++.
(Yes, I know it's not really.)  I wouldn't expect a static analyzer to
see across a C/Python or C/Java boundary, so I don't expect it to be
able to see across C/rust.  (Sadly, the boundary is sharp enough that
that's a reasonable comparison.)

> > If bindgen is integrated with the build system, that sounds fine as
> > far as helping to not overlook changes that are needed on the .rs
> > side
> > of things.. crosvm seems to mainly make bindgen a manual step, but
> > then again it is mostly used for external dependencies like the
> > interface to virglrenderer
> > 
> > Side note, from my experience with crosvm, please be careful about
> > any
> > external rust dependencies that are used.. rust still seems to be
> > young enough and evolving quickly enough that thing can quickly
> > turn
> > into a big headache
> > 
> 
> That's easy: meson doesn't support external crates, so no external
> rust dependencies are used at the moment :)
> 
> But yeah, I was planning on using some where it really makes sense,
> but we can have this discussion once we get there. So far it all
> works
> without any external dependency besides bindgen.

I think most of us are agreed here.  Mesa has pretty strict rules about
pulling in piles of external dependencies and we don't want that to
change just because we added a bit of rust.  There may be a crate or
two that we want to pull in in the future but, for now, everything is
core.

One point not mentioned above is Rust compiler versions.  Because Rust
is a rapidly evolving language, new features are being added to the
language and standard library every day.  If we impose the same
requirements on Rust that we're imposing on GCC where we still compile
with a version from 10-15 years ago, that's going to really hamper
development.  We probably don't want to require people to always pull
the very latest compiler from nightly but I do think we want to be able
to bump the compiler version once a year or something.  IDK how distros
and others will feel about that.  Hopefully, they feel about the same
way towards compiler requirement bumps as they feel towards rust
itself.

--Jason

> > BR,
> > -R
> > 
> > > > > BR,
> > > > > -R
> > > > > 
> > > > > > CCing a bunch of people who think are the most impacted by
> > > > > > it either
> > > > > > from a reviewer/maintainer or developer perspective. If you
> > > > > > think some
> > > > > > others should be explicitly aware of this, please point
> > > > > > them to this
> > > > > > discussion here.
> > > > > > 
> > > > > > Karol
> > > > > > 
> > > > > 
> > > 
> > 
> 