This commit expands the bootstrap build system's `README.md` as well as ensuring
that all API documentation is present and up-to-date. Additionally a new
`config.toml.example` file is checked in with commented out versions of all
possible configuration values.
When downloading a file in the bootstrap phase - get it to a temp
location first. Verify it there and only if downloaded properly move it
to the `cache` directory.
This should prevent `make` being stuck if the download was interrupted
or otherwise corrupted.
The temporary files are deleted in case of an exception.
Forcing them to be embedded in makefiles precludes being able to run them in
rustbuild, and adding them to compiletest gives us a great way to leverage
future enhancements to our "all encompassing test suite runner" as well as just
moving more things into Rust.
All tests are still Makefile-based in the sense that they rely on `make` being
available to run them, but there's no longer any Makefile-trickery to run them
and rustbuild can now run them out of the box as well.
rustbuild: Run all markdown documentation tests
This commit adds support to rustbuild to run all documentation tests, basically
running `rustdoc --test` over all our documentation.
Sanity check Python on OSX for LLDB tests
Two primary changes:
* Don't get past the configure stage if `python` isn't coming from `/usr/bin`
* Call `debugger.Terminate()` to prevent segfaults on newer versions of LLDB.
Closes#32994
rustbuild: Package librustc & co for cross-hosts
Currently the `rust-std` package produced by rustbuild only contains the
standard library plus libtest, but the makefiles actually produce a `rust-std`
package with all known target libraries (including libsyntax, librustc, etc).
Tweak the behavior so the dependencies of the `dist-docs` step in rustbuild
depend on the compiler libraries as well (so that they're all packaged).
Closes#32984
mk: Bootstrap from stable instead of snapshots
This commit removes all infrastructure from the repository for our so-called
snapshots to instead bootstrap the compiler from stable releases. Bootstrapping
from a previously stable release is a long-desired feature of distros because
they're not fans of downloading binary stage0 blobs from us. Additionally, this
makes our own CI easier as we can decommission all of the snapshot builders and
start having a regular cadence to when we update the stage0 compiler.
A new `src/etc/get-stage0.py` script was added which shares some code with
`src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists
the current stage0 compiler as well as cargo that we bootstrap from. This script
will download the relevant `rustc` package an unpack it into `$target/stage0` as
we do today.
One problem of bootstrapping from stable releases is that we're not able to
compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd).
To overcome this we employ two strategies:
* The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt`
(enabled as a result of #32731) and exported by the build system. This enables
nightly features in the compiler we download.
* The standard library and compiler are pinned to a specific stage0, which
doesn't change, so we're guaranteed that we'll continue compiling as we start
from a known fixed source.
The process for making a release will also need to be tweaked now to continue to
cadence of bootstrapping from the previous release. This process looks like:
1. Merge `beta` to `stable`
2. Produce a new stable compiler.
3. Change `master` to bootstrap from this new stable compiler.
4. Merge `master` to `beta`
5. Produce a new beta compiler
6. Change `master` to bootstrap from this new beta compiler.
Step 3 above should involve very few changes as `master` was previously
bootstrapping from `beta` which is the same as `stable` at that point in time.
Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and
get to use new features. This also shouldn't slow the release too much as steps
1-5 requires little work other than waiting and step 6 just needs to happen at
some point during a release cycle, it's not time sensitive.
Closes#29555Closes#29557
rustbuild: Fix --enable-rpath usage
This commit fixes the `--enable-rpath` configure flag in rustbuild to work
despite the compile-time directories being different than the runtime
directories. This unfortunately means that we can't use `-C rpath` out of the
box but hopefully the portability story here isn't too bad as
`src/librustc_back/rpath.rs` isn't *too* complicated.
Closes#32886
This commit removes all infrastructure from the repository for our so-called
snapshots to instead bootstrap the compiler from stable releases. Bootstrapping
from a previously stable release is a long-desired feature of distros because
they're not fans of downloading binary stage0 blobs from us. Additionally, this
makes our own CI easier as we can decommission all of the snapshot builders and
start having a regular cadence to when we update the stage0 compiler.
A new `src/etc/get-stage0.py` script was added which shares some code with
`src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists
the current stage0 compiler as well as cargo that we bootstrap from. This script
will download the relevant `rustc` package an unpack it into `$target/stage0` as
we do today.
One problem of bootstrapping from stable releases is that we're not able to
compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd).
To overcome this we employ two strategies:
* The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt`
(enabled as a result of #32731) and exported by the build system. This enables
nightly features in the compiler we download.
* The standard library and compiler are pinned to a specific stage0, which
doesn't change, so we're guaranteed that we'll continue compiling as we start
from a known fixed source.
The process for making a release will also need to be tweaked now to continue to
cadence of bootstrapping from the previous release. This process looks like:
1. Merge `beta` to `stable`
2. Produce a new stable compiler.
3. Change `master` to bootstrap from this new stable compiler.
4. Merge `master` to `beta`
5. Produce a new beta compiler
6. Change `master` to bootstrap from this new beta compiler.
Step 3 above should involve very few changes as `master` was previously
bootstrapping from `beta` which is the same as `stable` at that point in time.
Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and
get to use new features. This also shouldn't slow the release too much as steps
1-5 requires little work other than waiting and step 6 just needs to happen at
some point during a release cycle, it's not time sensitive.
Closes#29555Closes#29557
This commit adds support to rustbuild to run all documentation tests, basically
running `rustdoc --test` over all our documentation. This also includes support
for running the error index tests.
This commit adds support in rustbuild for running all of the compiletest test
suites as part of `make check`. The `compiletest` program was moved to
`src/tools` (like `rustbook` and others) and is now just compiled like any other
old tool. Each test suite has a pretty standard set of dependencies and just
tweaks various parameters to the final compiletest executable.
Note that full support is lacking in terms of:
* Once a test suite has passed, that's not remembered. When a test suite is
requested to be run, it's always run.
* The arguments to compiletest probably don't work for every possible
combination of platforms and testing environments just yet. There will likely
need to be future updates to tweak various pieces here and there.
* Cross compiled test suites probably don't work just yet, support for that will
come in a follow-up patch.
Clean out old documentation as well as the new test/tools directories. Should
prevent a problem that happened this morning where a PR bounced and then it left
docs with "broken links" so all future PRs bounced.
Currently the `rust-std` package produced by rustbuild only contains the
standard library plus libtest, but the makefiles actually produce a `rust-std`
package with all known target libraries (including libsyntax, librustc, etc).
Tweak the behavior so the dependencies of the `dist-docs` step in rustbuild
depend on the compiler libraries as well (so that they're all packaged).
Closes#32984
rustbuild: Verify sha256 of downloaded tarballs
Here's a quick first pass at this.
I don't use Python often enough to claim that this is totally Pythonic. I've left off some (almost certainly unnecessary) error handling regarding opening and processing files. The whole tarball is read into memory to calculate the hash, but the file isn't *so* large so that should be fine. I don't care for the output from `raise RuntimeError`, but that's how `run()` does it so I'm following precedent.
Tested by manually changing the value of `expected`, and by modifying the tarball then forcing `rustc_out_of_date()`. Both cases tripped the error.
Closes https://github.com/rust-lang/rust/issues/32902
cargotest: Put output in build directory
Right now cargotest uses `TempDir` to place output into the system temp
directory, but unfortunately this means that if the process is interrupted then
it'll leak the directory and that'll never get cleaned up. One of our bots
filled up its disk space and there were 20 cargotest directories lying around so
seems prudent to clean them up!
By putting the output in the build directory it should ensure that we don't leak
too many extra builds.
Right now cargotest uses `TempDir` to place output into the system temp
directory, but unfortunately this means that if the process is interrupted then
it'll leak the directory and that'll never get cleaned up. One of our bots
filled up its disk space and there were 20 cargotest directories lying around so
seems prudent to clean them up!
By putting the output in the build directory it should ensure that we don't leak
too many extra builds.
rustbuild: Fix handling of the bootstrap key
Bring the calculation logic in line with the makefiles and also set the
RUSTC_BOOTSTRAP_KEY environment variable to enable the bootstrap on the stable
compiler.
Add rustbuild option to use Ninja for LLVM build
This change adds support for a `ninja` option in the `[llvm]` section of rustbuild's `config.toml`. When `true`, the option enables use of the Ninja build tool. Note that this change does not add support for Ninja to the old makefile based build system.
Closes https://github.com/rust-lang/rust/issues/32809
r? @alexcrichton
It looks like before these config variables weren't actually taken
into account. This patch should make the build system skip over the
documentation steps correctly.
This verifies that the crates listed in the `[dependencies]` section of
`Cargo.toml` are a subset of the crates listed in `lib.rs` for our in-tree
crates. This should help ensure that when we refactor crates over time we keep
these dependency lists in sync.
This commit rewrites all of the tidy checks we have, namely:
* featureck
* errorck
* tidy
* binaries
into Rust under a new `tidy` tool inside of the `src/tools` directory. This at
the same time deletes all the corresponding Python tidy checks so we can be sure
to only have one source of truth for all the tidy checks.
cc #31590
This commit fixes the `--enable-rpath` configure flag in rustbuild to work
despite the compile-time directories being different than the runtime
directories. This unfortunately means that we can't use `-C rpath` out of the
box but hopefully the portability story here isn't too bad as
`src/librustc_back/rpath.rs` isn't *too* complicated.
Closes#32886
Bring the calculation logic in line with the makefiles and also set the
RUSTC_BOOTSTRAP_KEY environment variable to enable the bootstrap on the stable
compiler.
Right now if you configure multiple hosts rustbuild will only build
documentation for the build triple, but we've got all the support necessary to
build documentation for different architectures as well. This commit
reinterprets the `target` field of doc `Step` instances to be the target of the
documentation rather than the target of the rustdoc/tool being run.
This should enable `make dist` to start producing a bunch of `rust-docs`
packages for all the cross architectures that rustbuild is producing now.
rustbuild: Fix compile on OSX for 10.7
This commit should help configure our OSX rustbuild builder for targeting 10.7.
A key part of this is using `libc++` instead of `libstdc++` as apparently it's
more filled out and otherwise LLVM's cmake configuration would fail.
The `rust-std` package that we produce is expected to have not only the standard
library but also libtest for compiling unit tests. Unfortunately this does not
currently happen due to the way rustbuild is structured.
There are currently two main stages of compilation in rustbuild, one for the
standard library and one for the compiler. This is primarily done to allow us to
fill in the sysroot right after the standard library has finished compiling to
continue compiling the rest of the crates. Consequently the entire compiler does
not have to explicitly depend on the standard library, and this also should
allow us to pull in crates.io dependencies into the build in the future because
they'll just naturally build against the std we just produced.
These phases, however, do not represent a cross-compiled build. Target-only
builds also require libtest, and libtest is currently part of the
all-encompassing "compiler build". There's unfortunately no way to learn about
just libtest and its dependencies (in a great and robust fashion) so to ensure
that we can copy the right artifacts over this commit introduces a new build
step, libtest.
The new libtest build step has documentation, dist, and link steps as std/rustc
already do. The compiler now depends on libtest instead of libstd, and all
compiler crates can now assume that test and its dependencies are implicitly
part of the sysroot (hence explicit dependencies being removed). This makes the
build a tad less parallel as in theory many rustc crates can be compiled in
parallel with libtest, but this likely isn't where we really need parallelism
either (all the time is still spent in the compiler).
All in all this allows the `dist-std` step to depend on both libstd and libtest,
so `rust-std` packages produced by rustbuild should start having both the
standard library and libtest.
Closes#32523
This commit should help configure our OSX rustbuild builder for targeting 10.7.
A key part of this is using `libc++` instead of `libstdc++` as apparently it's
more filled out and otherwise LLVM's cmake configuration would fail.
This is a new suite of tests that verifies that the compiler
builds specific revisions of select crates from crates.io.
It does not run by default. It is intended that buildbot
runs these tests against all PRs, and gate on them.
This commit implements the `make dist` command in the new rustbuild build
system, porting over `dist.mk` and `prepare.mk` into Rust. There's a huge amount
of complexity between those two files, not all of which is likely justified, so
the Rust implementation is *much* smaller.
Currently the implementation still shells out to rust-installer as well as some
python scripts, but ideally we'd rewrite it all in the future to not shell out
and be in Rust proper.
rustbuild: Fix cross compiling to FreeBSD
This commit fixes our support for cross compiling a compiler to run on FreeBSD.
Over the weekend I managed to get a cross compiler from Linux to FreeBSD [1]
which I hope to soon use to start producing FreeBSD nightly compilers. With the
`make dist` support added in #32237 we should be able to produce standard
rustc/rust-std packages for FreeBSD through a new slave with this cross compiler.
Currently, however, we don't "Just Work" when cross compiling FreeBSD and a
number of changes were required (part of this PR). They include:
* A few build fixes were needed in LLVM. Our own branch has been rebased on the
actual 3.8 release and I applied one extra commit [2] which contains two fixes:
1. The LLVM CMake build system passes the `-Wl,-z,defs` flag on many
platforms, but *not* when `CMAKE_SYSTEM_NAME` is "FreeBSD". Unfortunately
this doesn't take into account when we're cross compiling, and as predicted
the build will fail if `-Wl,-z,defs` is passed (see [3] for more info). To
fix this we test `TARGET_TRIPLE` instead of the `CMAKE_SYSTEM_NAME` which
is what we're compiling for which fixes the problem.
2. The `PATH_MAX` constant is apparently defined in a different location than
many other Unix systems, so a file which required this just needed some
help to keep compiling.
* Support for compiling compiler-rt with CMake has been added to rustbuild. It
looks like it just emulates Linux in what it compiles as it didn't seem to
naturally produce anything else... At least the architecture is right, so
seems good for now at least!
[1]: https://github.com/alexcrichton/port-of-rust/blob/master/prebuilt/freebsd/Dockerfile
[2]: https://github.com/rust-lang/llvm/commit/be89e4b5
[3]: https://bugs.webkit.org/show_bug.cgi?id=138420
This commit fixes our support for cross compiling a compiler to run on FreeBSD.
Over the weekend I managed to get a cross compiler from Linux to FreeBSD [1]
which I hope to soon use to start producing FreeBSD nightly compilers. With the
`make dist` support added in #32237 we should be able to produce standard
rustc/rust-std packages for FreeBSD through a new slave with this cross compiler.
Currently, however, we don't "Just Work" when cross compiling FreeBSD and a
number of changes were required (part of this PR). They include:
* A few build fixes were needed in LLVM. Our own branch has been rebased on the
actual 3.8 release and I applied one extra commit [2] which contains two fixes:
1. The LLVM CMake build system passes the `-Wl,-z,defs` flag on many
platforms, but *not* when `CMAKE_SYSTEM_NAME` is "FreeBSD". Unfortunately
this doesn't take into account when we're cross compiling, and as predicted
the build will fail if `-Wl,-z,defs` is passed (see [3] for more info). To
fix this we test `TARGET_TRIPLE` instead of the `CMAKE_SYSTEM_NAME` which
is what we're compiling for which fixes the problem.
2. The `PATH_MAX` constant is apparently defined in a different location than
many other Unix systems, so a file which required this just needed some
help to keep compiling.
* Support for compiling compiler-rt with CMake has been added to rustbuild. It
looks like it just emulates Linux in what it compiles as it didn't seem to
naturally produce anything else... At least the architecture is right, so
seems good for now at least!
[1]: https://github.com/alexcrichton/port-of-rust/blob/master/prebuilt/freebsd/Dockerfile
[2]: https://github.com/rust-lang/llvm/commit/be89e4b5
[3]: https://bugs.webkit.org/show_bug.cgi?id=138420
The facet of a stage is rarely relevant when running a tool or building
something, it's all a question of what stage the *compiler* is built in. We've
already got a nice handy `Compiler` structure to carry this information, so
let's use it!
This refactors the signature of the `Build::cargo` function two ways:
1. The `stage` argument is removed, this was just duplicated with the `compiler`
argument's stage field.
2. The `target` argument is now required. This was a bug where if the `--target`
flag isn't passed then the snapshot stage0 compiler is always used, so we
won't pick up any changes.
Much of the other changes in this commit are just propagating these decisions
outwards. For example many of the `Step` variants no longer have a stage
argument as they're baked into the compiler.
Add a script to get run which verifies that `href` links in documents are
correct. We're always getting a steady stream of "fix a broken link" PRs and
issue reports, and we should probably just nip them all in the bud.
Run `cargo doc` to generate all documentation for the standard library, and also
add a target which generates documentation for the compiler as well (but don't
enable it by default).
This commit implements documentation generation of the nomicon, the book, the
style guide, and the standalone docs. New steps were added for each one as well
as appropriate makefile targets for each one as well.
When cross compiling for a new host, we can't actually run the host compiler to
generate its own libs. In theory, however, all stage2 compilers (for any host)
will produce the same libraries, so we just require the build compiler to
produce the necessary host libraries and then we link those into place.
This switches the defaults to ensure that everything is built with the build
compiler rather than the host compiler itself (which we're not guaranteed to be
able to run)
Currently all multi-host builds assume the the build platform can run the
`llvm-config` binary generated for each host platform we're creating a compiler
for. Unfortunately this assumption isn't always true when cross compiling, so we
need to handle this case.
This commit alters the build script of `rustc_llvm` to understand when it's
running an `llvm-config` which is different than the platform we're targeting for.
This commit fixes a longstanding issue with the makefiles where all host
platforms bootstrap themselves. This commit alters the build logic for the
bootstrap to instead only bootstrap the build triple, and all other compilers
are compiled from that one compiler.
The benefit of this change is that we can cross-compile compilers which cannot
run on the build platform. For example our builders could start creating
`arm-unknown-linux-gnueabihf` compilers.
This reduces the amount of bootstrapping we do, reducing the amount of test
coverage, but overall it should largely just end in faster build times for
multi-host compiles as well as enabling a feature which can't be done today.
cc #5258
This commit implements documentation generation of the nomicon, the book, the
style guide, and the standalone docs. New steps were added for each one as well
as appropriate makefile targets for each one as well.
Also add a `clean` target for the makefiles to blow away everything related to
the build. Note that this specifically does not tamper with:
* the LLVM build directory
* the directory of the bootstrap system
* the cached downloads of cargo/rustc
This commit adds a `--enable-rustbuild` option to the configure script which
will copy a different `Makefile.in` into place to intercept all `make`
invocations.
Currently this makefile only has one target, but it's expected to be filled out
quite a bit over time!
During the transition period where we're still using ./configure and makefiles,
read some extra configuration from `config.mk` if it's present. This means that
the bootstrap build should be configured the same as the original ./configure
invocation.
Eventually this will all be removed in favor of only storing information in
`config.toml` (e.g. the configure script will generate config.toml), but for now
this should suffice.
This commit is the start of a series of commits which start to replace the
makefiles with a Cargo-based build system. The aim is not to remove the
makefiles entirely just yet but rather just replace the portions that invoke the
compiler to do the bootstrap. This commit specifically adds enough support to
perform the bootstrap (and all the cross compilation within) along with
generating documentation.
More commits will follow up in this series to actually wire up the makefiles to
call this build system, so stay tuned!
Steve Klabnik
Nerijus Arlauskas
Niko Matsakis
Brian Anderson
Georg Brandl
Alex Crichton
Cyryl Płotnicki-Chudyk
Tshepang Lekhonkhobe
bors
Manish Goregaokar
Nick Platt
Timon Van Overveldt
Björn Steinbrink
Jorge Aparicio