# //tools/rust
This directory contains scripts for building, packaging, and distributing the
Rust toolchain (the Rust compiler, and also C++/Rust FFI tools like
[Crubit](https://github.com/google/crubit)).
[TOC]
## Background
Like with Clang, Chromium uses bleeding edge Rust tooling. We track the upstream
projects' latest development as closely as possible. However, Chromium cannot
use official Rust builds for various reasons which require us to match the Rust
LLVM backend version with the Clang we use.
It would not be reasonable to build the tooling for every Chromium build, so we
build it centrally (with the scripts here) and distribute it for all to use
(also fetched with the scripts here).
Similar to the Clang package which exists as a tarball that is unpacked into
`third_party/llvm-build`, the Rust package exists as a tarball that is unpacked
into `third_party/rust-toolchain`.
## Rust build overview
Each Rust package is built from an Rust git, usually from HEAD directly, along
with the current Clang/LLVM revision in use in Chromium. Hence a new Rust
package must be built whenever either Rust or Clang is updated. When building
Rust we also build additional tools such as clippy and rustfmt, and interop
tools including bindgen and crubit.
The Rust build also includes building LLVM for rustc to use, and Clang for
bindgen and crubit to use.
The `*_upload_clang` and `*_upload_rust` trybots are used to build Clang and
Rust respectively from the revisions specified in the Chromium source tree.
These are uploaded to a storage bucket when the build succeeds. After being
copied from staging to production by a developer (see
[cs/copy_staging_to_prod_and_goma.sh](
http://cs/copy_staging_to_prod_and_goma.sh)), they can then be fetched by
`gclient sync`.
The `update_rust.py` script is used by `gclient sync` to fetch the Rust
toolchain for the revisions specified in the script.
## Rolling Rust
Follow the directions in [//docs/updating_clang.md](
../../docs/updating_clang.md) to roll Clang and Rust together. To just
roll Rust on its own, use the `--skip-clang` argument when running
`upload_revision.py`.
The upload_revision.py script will update the revision of Rust to be
built and used in `update_rust.py` and will start the trybots that
will build the Rust toolchain.
After the build has succeeded and the new toolchain has been copied to
production, the CQ will run trybots to verify that our code still builds
and tests pass with the new Rust toolchain.
### An overview of what is updated in a Rust roll
During Rust packaging, the upstream Rust sources are checked out into
`third_party/rust-src`.
During a Rust roll, a couple of things get updated. The most obvious one is
various toolchain binaries like `rustc` that live in
`third_party/rust-toolchain/bin`. These are the direct outputs of a Rust
toolchain build.
We also update the Rust standard library. We actually provide two copies of the
standard library in Chromium: a prebuilt version only for use in host tools
(e.g. build scripts, proc macros), and a version built from source as part of
the normal Chromium build for use in target artifacts. These are the same
version of the standard library that the Rust toolchain revision provides.
The reason we have a prebuilt version of the standard library for use in host
tools is that they are often loaded into `rustc` as a module, so to be safe we
use the same prebuilts that the toolchain linked against. These are copied from
the Rust toolchain build to
`third_party/rust-toolchain/lib/rustlib/$PLATFORM/lib/*.rlib`. We use these
when the gn arg `rust_prebuilt_stdlib` is true, which is manually set to true
for gn host toolchains.
The sources of the standard library we build from source for target artifacts
live in `third_party/rust-toolchain/lib/rustlib/src/rust`. These are copied
from `third_party/rust-src`. Since Chromium uses gn as its build system, we
need some way to translate build files from Rust's build system, cargo, to gn
rules. This is the responsibility of `gnrt`, which is a Chromium-specific tool
that lives in [`tools/crates/gnrt`](https://crsrc.org/c/tools/crates/gnrt/),
written in Rust. `gnrt gen` takes a cargo workspace, runs `cargo metadata` on
it to get information about sources and dependencies, and outputs gn rules
corresponding to the cargo build rules. Rust has a
[`sysroot`](https://github.com/rust-lang/rust/tree/master/library/sysroot)
crate roughly corresponding to a top level cargo workspace we want for the
standard library. However, we want a couple of customizations without having to
patch the Rust sources, so we have another crate
[`fake_root`](https://crsrc.org/c/build/rust/std/fake_root/) above that depends
on `sysroot`.
[`tools/rust/gnrt_stdlib.py`](https://crsrc.org/c/tools/rust/gnrt_stdlib.py)
fetches and invokes the pinned `cargo` (see [`rustc` bootstrapping
explanation](https://rustc-dev-guide.rust-lang.org/building/bootstrapping/what-bootstrapping-does.html),
"pinned" is the "stage0" toolchain) to build and run `gnrt` with `fake_root` as
the base workspace, generating an updated
[`build/rust/std/rules/BUILD.gn`](https://crsrc.org/c/build/rust/std/rules/BUILD.gn)
that has gn rules for the new standard library sources. For convenience when
rolling Rust, this is one big `BUILD.gn` file as opposed to multiple files per
crate. Note that because we do not ship cargo build files in
`third_party/rust-toolchain`, we must run `gnrt` against `third_party/rust-src`
instead of `third_party/rust-toolchain`. But end users do not have
`third_party/rust-src` checked out, so we must rewrite the
`third_party/rust-src` paths to the copies of the sources in
`third_party/rust-toolchain/lib/rustlib/src/rust`, which is checked out by end
users as part of the Rust toolchain.
As an aside, `gnrt` is also used to generate gn build rules for
non-standard-library Rust packages in `third_party/rust` used in Chromium's
build. It uses
[`third_party/rust/chromium_crates_io`](https://crsrc.org/c/third_party/rust/chromium_crates_io)
as the base workspace and vendors sources into
`third_party/rust/chromium_crates_io/vendor`. The `--for-std` argument to `gnrt
gen` does different things for creating gn rules for the standard library
versus for various non-standard-library packages, such as producing a single
BUILD.gn file.
### Possible failure: Missing dependencies
`build_rust.py` will vendor all dependencies before starting the build. To do
this it first initializes git submodules. Then it runs `cargo vendor`. However
some parts of the compiler build are excluded from the top level Cargo.toml
workspace. Thus it passes `--sync dir` for a number of subdirectories, based
on [dist.rs, the nightly tarball packager](
https://github.com/rust-lang/rust/blob/master/src/bootstrap/dist.rs#L986-L995).
If another Cargo.toml is required in the future, and not part of the workspace
it would produce missing dependencies, and the set of directories in
`build_rust.py` would need to be updated.
### Possible failure: Missing sources or inputs
A build error when building the stdlib in Chromium may look like:
```
FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
...build command...
ERROR: file not in GN sources: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../portable-simd/crates/std_float/src/lib.rs
```
Or:
```
FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
...build command...
ERROR: file not in GN inputs: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../stdarch/crates/core_arch/src/core_arch_docs.md
```
When building the stdlib in Chromium, the GN rules must have every rust source
or other input file that makes up the crate listed in the `sources` and
`inputs` GN variables. gnrt will walk the directory tree from the root of the
crate and put every relevant file into the set. But sometimes a crate includes
modules from paths outside the crate root's directory tree, with a path
directive such as
```rs
#[path = "../../stuff.rs"]
mod stuff;
```
or will `include!()` a file from another path, which is common for `.md` files:
```rs
include!("../../other_place.md")
```
The first error is saying the source file `std_float/src/lib.rs` did not
appear in the `sources` variable. The `../../` part of the path shows that
this is outside the crate root's directory tree. The second error is saying
that `core_arch/src/core_arch_docs.md` did not appear in the `inputs` variable.
To fix the error:
* Determine the path that is missing, relative to the crate root. In the above
example this is `../../portable-simd/crates/std_float/src`. We could also use
`../../portable-simd` or anything in between, though that would add a lot
more sources to the GN rules than is necessary in this case. It's best to
point to the directory of the module root (where the `lib.rs` or `mod.rs`
is located).
* Download the roll CL (on Gerrit, click on the 3 dots in the upper right
corner and click on "Download patch").
* Find the failing build target crate's rules in
`//build/rust/std/gnrt_config.toml`. The failing crate in the above example
is `libstd.rlib`, so we want the `[crate.std]` section of the config file.
* Determine if the missing file should go in `sources` or `inputs`.
* For `sources`, add the path to a `extra_src_roots` list in the crate's
rules. For the above example, we could add
`extra_src_roots = ['../../portable-simd/crates/std_float/src']`.
* For `inputs`, add the path to a `extra_input_roots` list in the crate's
rules. For the above example, we could add
`extra_input_roots = ['../../stdarch/crates/core_arch/src']`.
* With the roll CL checked out, run `gclient sync`.
*** note
NOTE: `gclient sync` will download the version of the rust toolchain from the
roll CL. In order for this to work, the upload_rust bots should've completed and
`copy_staging_to_prod_and_goma.sh should've been run.
***
* Run `tools/rust/gnrt_stdlib.py` to use gnrt to rebuild the stdlib GN rules
using the updated config.
### Generating `BUILD.gn` files for stdlib crates
If the build structure changes in any way during a roll, the GN files need
to be regenerated.
#### Simple way:
Run `tools/rust/gnrt_stdlib.py`.
#### Longer way
This requires Rust to be installed and available in your system, typically
through [https://rustup.rs](https://rustup.rs).
To generate `BUILD.gn` files for the crates with the `gnrt` tool:
1. Change directory to the root `src/` dir of Chromium.
1. Build `gnrt` to run on host machine: `cargo build --release --manifest-path
tools/crates/gnrt/Cargo.toml --target-dir out/gnrt`.
1. Ensure you have a checkout of the Rust source tree in `third_party/rust-src`
which can be done with `tools/rust/build_rust.py --sync-for-gnrt`.
1. Run `gnrt` with the `gen` action:
`out/gnrt/release/gnrt gen --for-std third_party/rust-src`.
This will generate the `//build/rust/std/rules/BUILD.gn` file, with the changes
visible in `git status` and can be added with `git add`.
## Local development
To build the Rust toolchain locally, run `//tools/rust/build_rust.py`. It
has additional flags to skip steps if you're making local changes and want
to retry a build. The script will produce its outputs in
`//third_party/rust-toolchain/`, which is the same place that `gclient sync`
places them.
Building the `rust_build_tests` GN target is a good way to quickly verify the
toolchain is working.
## Rolling Crubit tools
Steps to roll the Crubit tools (e.g. `rs_bindings_from_cc` tool)
to a new version:
- Locally, update `CRUBIT_REVISION` in `update_rust.py`.
(Update `CRUBIT_SUB_REVISION` when the build or packaging is changed, but
the upstream Rust revision we build from is not changed.)
- Locally, update `crubit_revision` in `//DEPS`, so that it matches
the revision from the previous bullet item.
- Run manual tests locally (see the "Building and testing the tools locally"
section below).
TODO(crbug.com/40226863): These manual steps should
be made obsolete once Rust-specific tryjobs cover Crubit
tests.
## Building and testing Crubit locally
### Prerequisites
#### Bazel
`build_crubit.py` depends on Bazel.
To get Bazel, ensure that you have `checkout_bazel` set in your `.gclient` file
and then rerun `gclient sync`:
```sh
$ cat ../.gclient
solutions = [
{
"name": "src",
"url": "https://chromium.googlesource.com/chromium/src.git",
...
"custom_vars": {
"checkout_bazel": True,
"checkout_crubit": True,
},
},
]
```
### Building
Just run `tools/rust/build_crubit.py`. So far `build_crubit.py` has only been
tested on Linux hosts.
### Deploying
`build_crubit.py` will copy files into the directory specified in the
(optional) `--install-to` cmdline parameter - for example:
```
$ tools/rust/build_crubit.py --install-to=third_party/rust-toolchain/bin/
```
### Testing
Crubit tests are under `//build/rust/tests/test_rs_bindings_from_cc`. Until
Crubit is built on the bots, the tests are commented out in
`//build/rust/tests/BUILD.gn`, but they should still be built and run before
rolling Crubit. TODO(crbug.com/40226863): Rephrase this paragraph
after Crubit is built and tested on the bots.
Codebase Browser
chromium