rustbuild: Actually test musl on the musl bot
A typo unfortunately meant that we haven't been testing musl for a bit, so now
it's time to discover if we accidentally introduced a regression!
rustbuild: Run debuginfo tests by default
This fixes an accidental regression in rustbuild which stopped running debuginfo
tests by default. Here we flag the test suites as `default(true)` to ensure that
they're run on bots, for example.
rustbuild: Update Cargo download location
I updated the beta compiler used to bootstrap the master branch in #38438 with
the intention of fixing Travis OSX linkage issues but I mistakenly forgot that
the PR only updated rustc, not Cargo itself. Cargo has a new release process
with downloads in a different location, so this commit updates rustbuild to
download from this new location by tracking revisions instead of Cargo nightly
dates.
rustbuild: Eliminate duplication of dist tarballs
Fixes#38365 by not constructing the duplicate steps in the first place, as suggested. The source package step is lacking the check as in other steps, so it is added as well.
Tested locally with the `alexcrichton/rust-slave-linux-cross:2016-11-11` container (with the build slave init replaced with no-op, of course).
r? @alexcrichton
adaptation to rustbuild for openbsd
Since the switch to rustbuild, the build for openbsd is broken:
- [X] `ar` inference based on compiler name is wrong (OpenBSD usually use `egcc`, but `ear` doesn't exist)
- [X] `make` isn't GNU-make under OpenBSD (and others BSD platforms)
- [x] `stdc++` isn't the right stdc++ library to link with (it should be `estdc++`)
- [x] corrects tests that don't pass anymore (problems related to rustbuild)
r? @alexcrichton
rustbuild: Stop building docs for libtest by default
They cause the search index from the std docs to get overwritten just like #34800.
Part of #38319.
Add prefix to config.toml
This allows `rustbuild` to be used to install to a prefix.
```toml
[build]
prefix = "/path/to/install"
```
For example, the following `config.toml` will cause `x.py dist --install` to install to `/path/to/install`
This fixes an accidental regression in rustbuild which stopped running debuginfo
tests by default. Here we flag the test suites as `default(true)` to ensure that
they're run on bots, for example.
We only want to package each host/target once for `dist`. The obvious
solution takes the form of step dependency, which is implemented at
least for the `dist-rustc` step. Unfortunately since the steps are
created from `hosts x targets` during planning and *not* de-duplicated
afterwards, the problem still persists.
We therefore move the check inside `plan()` instead, to avoid creating
the duplicate steps in the first place.
I updated the beta compiler used to bootstrap the master branch in #38438 with
the intention of fixing Travis OSX linkage issues but I mistakenly forgot that
the PR only updated rustc, not Cargo itself. Cargo has a new release process
with downloads in a different location, so this commit updates rustbuild to
download from this new location by tracking revisions instead of Cargo nightly
dates.
add preliminary support for incremental compilation to rustbuild.py
This implements the integration described in #37929. It requires the use of a local nightly as your bootstrap compiler. The setup is described in `src/bootstrap/README.md`.
This does NOT implement the "copy stage0 libs to stage1" optimization described in #37929, just because that seems orthogonal to me.
In local testing, I do not yet see any incremental re-use when building rustc. I'm not sure why that is, more investigation needed.
(For these reasons, this is not marked as fixing the relevant issue.)
r? @alexcrichton -- I included one random cleanup (`Step::noop()`) that turned out to not be especially relevant. Feel free to tell me you liked it better the old way.
rustbuild: Fix `copy` helper with existing files
This erroneously truncated files when the destination already existed and was an
existing hard link to the source. This in turn caused weird bugs!
Closes#37745
This not only avoids the small – and unnecessary – constant overhead for each compiler invocation,
but also helps somewhat by only having “correct” rustc processes to look for in `/proc/`.
This also makes the wrapper behave effectively as a regular exec wrapper its intended to be.
rustbuild: Fix LC_ID_DYLIB directives on OSX
Currently libraries installed by rustbuild on OSX have an incorrect
`LC_ID_DYLIB` directive located in the dynamic libraries that are
installed. The directive we expect looks like:
@rpath/libstd.dylib
Which means that if you want to find that dynamic library you should
look at the dylib's other `@rpath` directives. Typically our `@rpath`
directives look like `@loader_path/../lib` for the compiler as that's
where the installed libraries will be located. Currently, though,
rustbuild produces dylibs with the directive that looks like:
/Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/build/x86_64-apple-darwin/stage1-std/x86_64-apple-darwin/release/deps/libstd-713ad88203512705.dylib
In other words, the build directory is encoded erroneously. The compiler
already [knows how] to change this directive, but it only passes that
argument when `-C rpath` is also passed. The rustbuild system, however,
explicitly [does not pass] this option explicitly and instead bakes its
own. This logic then also erroneously didn't pass `-Wl,-install_name`
like the compiler.
[knows how]: 4a008cccaa/src/librustc_trans/back/linker.rs (L210-L214)
[does not pass]: 4a008cccaa/src/bootstrap/bin/rustc.rs (L133-L158)
To fix this regression this patch introduces a new `-Z` flag, `-Z
osx-rpath-install-name` which basically just forces the compiler to take
the previous `-install_name` branch when creating a dynamic library.
Hopefully we can sort out a better rpath story in the future, but for
now this "hack" should suffice in getting our nightly builds back to the
same state as before.
Closes#38430
rustbuild: Create directories in mingw dist
Previously we accidentally relied on the mingw dist step running last, but the
step just needed to ensure the directories were created.
Currently libraries installed by rustbuild on OSX have an incorrect
`LC_ID_DYLIB` directive located in the dynamic libraries that are
installed. The directive we expect looks like:
@rpath/libstd.dylib
Which means that if you want to find that dynamic library you should
look at the dylib's other `@rpath` directives. Typically our `@rpath`
directives look like `@loader_path/../lib` for the compiler as that's
where the installed libraries will be located. Currently, though,
rustbuild produces dylibs with the directive that looks like:
/Users/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-mac/build/build/x86_64-apple-darwin/stage1-std/x86_64-apple-darwin/release/deps/libstd-713ad88203512705.dylib
In other words, the build directory is encoded erroneously. The compiler
already [knows how] to change this directive, but it only passes that
argument when `-C rpath` is also passed. The rustbuild system, however,
explicitly [does not pass] this option explicitly and instead bakes its
own. This logic then also erroneously didn't pass `-Wl,-install_name`
like the compiler.
[knows how]: 4a008cccaa/src/librustc_trans/back/linker.rs (L210-L214)
[does not pass]: 4a008cccaa/src/bootstrap/bin/rustc.rs (L133-L158)
To fix this regression this patch introduces a new `-Z` flag, `-Z
osx-rpath-install-name` which basically just forces the compiler to take
the previous `-install_name` branch when creating a dynamic library.
Hopefully we can sort out a better rpath story in the future, but for
now this "hack" should suffice in getting our nightly builds back to the
same state as before.
Closes#38430
rustbuild: Add sccache support
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
---
Note that this does not currently change Windows [due to previously written up issues](https://github.com/rust-lang/rust/issues/38119#issuecomment-266631585). Despite that, however, I was curious to get timings for the builds on Travis to see what ranges we're working with. As a result, this is a WIP PR I'm using to gauge build times and such.
This erroneously truncated files when the destination already existed and was an
existing hard link to the source. This in turn caused weird bugs!
Closes#37745
rustbuild: Package rust-mingw by default
This fixes the `make dist` step on MinGW to package the `rust-mingw` component
by default. This should hopefully be the last step in fixing nightlies.
rustbuild: Add cli option --keep-stage
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries or when re-running stage 1 tests in libraries in general.
Fixes#38326
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
Document --test-args for rustbuild
There are three changes:
* Replace --filter with --test-args
* Delete `./x.py test src/test/run-pass/assert-*` example, which doesn't work
* As driveby, update Buildbot URLs to https
Fix#38275.
r? @alexcrichton
This commit skips a few more dist tragets during compilation which shouldn't be
necessary.
* First, when packaging std we only take action when the host target is the
build target. Otherwise we package the same artifacts a number of times, which
shouldn't be necessary.
* Next, we apply the same logic to the save-analysis build. This is actually
required for correctness as the build compiler is the only one which actually
has save analysis information. This should fix an error seen on nightlies.
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries, or when re-running stage 1 tests in libraries in general.
Git worktrees have this as a file and typically won't work inside docker
containers, but that's ok, so instead of just checking for existence check for a
directory to see if the git commands will succeed.
Make configure message re x.py not assume build dir == src dir
Fix#38251 but perhaps not BEST fix for it.
As driveby, fix copyright year in `Makefile.in`
Prevent Windows from displaying UI on errors.
Otherwise tests like run-pass/out-of-stack get wedged on Windows error reporting dialog (unless error reporting has been disabled, of course).
This commit implements the `distcheck` target for rustbuild which is only ever
run on our nightly bots. This essentially just creates a tarball, un-tars it,
and then runs a full build, validating that the release tarballs do indeed have
everything they need to build Rust.
rustbuild: Use src/rustc for assembled compilers
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
Add rustbuild command `bench`
Add command bench to rustbuild, so that `./x.py bench <path>` can compile and run benchmarks.
`./x.py bench --stage 1 src/libcollections` and `./x.py bench --stage 1 src/libstd` should both compile well. Just `./x.py bench` runs all benchmarks for the libstd crates.
Fixes#37897
with this feature disabled, you can (Cargo) compile std with
"panic=abort"
rustbuild will build std with this feature enabled, to maintain the
status quo
fixes#37252
rustbuild: allow dynamically linking LLVM
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Update: There are now four static/shared scenarios that can happen
for the supported LLVM versions:
- 3.9+: By default use `llvm-config --link-static`
- 3.9+ and `--enable-llvm-link-shared`: Use `--link-shared` instead.
- 3.8: Use `llvm-config --shared-mode` and go with its answer.
- 3.7: Just assume static, maintaining the status quo.
rustbuild: Allow configuration of python interpreter
Add a configuration key to `config.toml`, read it from `./configure`, and add
auto-detection if none of those were specified.
Closes#35760
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Add llvm debuginfo configure option
CC @nnethercote @Mark-Simulacrum
We add a new configure option, `--enable-llvm-debuginfo`, to do exactly what you'd think.
Re: #31033Fixes#37738
If a rule is flagged with `default(true)` then the pseudo-rule `default:foo`
will include that. If a rule is also flagged with `.host(true)`, however, then
the rule shouldn't be included for targets that aren't in the host array. This
adds a filter to ensure we don't pull in host rules for targets by accident.
update_with_config_mk() needs to read the new llvm debuginfo config
option from config.mk. Other than that, rustbuild already supports
LLVM's RelWithDebInfo build type.
compile-test: allow overriding nodejs binary location
Add a command-line argument to manually specify which nodejs binary should be used,
which disables the default search.
Original work done by @tari.
Fixes#34188.
rustbuild: enable an initial local cargo
This allows the initial build of src/bootstrap itself to use a local
cargo taken from `configure --local-rust-root`. It was already finding
rustc this way, but was always downloading cargo since it didn't know
where to find it.
It now matches the same logic that `config.rs` will use for stage0,
where both rustc and cargo are taken from `CFG_LOCAL_RUST_ROOT`.
Move all Linux/OSX CI infastructure to Travis
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
This allows the initial build of src/bootstrap itself to use a local
cargo taken from `configure --local-rust-root`. It was already finding
rustc this way, but was always downloading cargo since it didn't know
where to find it.
It now matches the same logic that `config.rs` will use for stage0,
where both rustc and cargo are taken from `CFG_LOCAL_RUST_ROOT`.
Vendor all rustbuild dependencies in this repo
This commit vendors all crates.io dependencies into the rust-lang/rust repository using the `cargo-vendor` tool. This is done in an effort to make rustbuild distro-ready by ensuring that our source tarballs are self-contained units which don't need extraneous network downloads.
A new `src/vendor` directory is created with all vendored crates, and Cargo, when using rustbuild, is configured to use this directory. Over time we can deduplicate this directory with the actual src tree (e.g. src/librustc_serialize, src/liblibc, src/libgetopts, ...). For now though that's left to a separate commit.
A few changes are included here:
* The `winapi` and `url` dependencies were dropped. The source code for these
projects is pretty weighty, and we're about to vendor them, so let's not
commit to that intake just yet. If necessary we can vendor them later but for
now it shouldn't be necessary.
* The `--frozen` flag is now always passed to Cargo, obviating the need for
tidy's `cargo_lock` check.
* Tidy was updated to not check the vendor directory
Closes#34687
There is a *little* code duplication, but primarily for sake of "match
exhaustiveness". Let's blame Linux/MIPS for not exposing endianness
explicitly in `uname -m` (that's user-space interface and as such is
frozen).
Currently the build won't work as we have to wait for a new stage0 for
the MIPS host compilers, but this paves the way to self-hosted Rust on
MIPS. The cross-compiled MIPS binaries are confirmed to work on the
Loongson 3A2000 (MIPS64r2-compatible) so we have plenty of confidence
that they'll work on other MIPS platforms too, as Linux/MIPS user-space
ABI is consistent across machines of the same bitness.
This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:
# build everything
./x.py build
# document everyting
./x.py doc
# test everything
./x.py test
# test libstd
./x.py test src/libstd
# build libcore stage0
./x.py build src/libcore --stage 0
# run stage1 run-pass tests
./x.py test src/test/run-pass --stage 1
The `src/bootstrap/bootstrap.py` script is now aliased as a top-level `x.py`
script. This `x` was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.
Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.
The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).
The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.
The *intention* is that this support is extended for arbitrary files (e.g.
`src/test/run-pass/my-test-case.rs`), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.
Enable line number debuginfo in releases
This commit enables by default passing the `-C debuginfo=1` argument to the
compiler for the stable, beta, and nightly release channels. A new configure
option was also added, `--enable-debuginfo-lines`, to enable this behavior in
developer builds as well.
Closes#36452
This commit enables by default passing the `-C debuginfo=1` argument to the
compiler for the stable, beta, and nightly release channels. A new configure
option was also added, `--enable-debuginfo-lines`, to enable this behavior in
developer builds as well.
Closes#36452
A new point-release shouldn't change any language semantics, so a local
stage0 that matches MAJOR.MINOR version should still be considered a
local-rebuild as far as `--cfg stageN` features go.
e.g. `1.14.0` should be considered a local-rebuild for any `1.14.X`.
(Bootstrap keys used to be an issue too, until #37265.)
Allow bootstrapping without a key. Fixes#36548
This will make it easier for packagers to bootstrap rustc when they happen
to have a bootstrap compiler with a slightly different version number.
It's not ok for anything other than the build system to set this environment variable.
r? @alexcrichton
This will make it easier for packagers to bootstrap rustc when they happen
to have a bootstrap compiler with a slightly different version number.
It's not ok for anything other than the build system to set this environment variable.
We hope to move to AppVeyor in the near future off of Buildbot + EC2. This adds
an `appveyor.yml` configuration file which is ready to run builds on the auto
branch. This is also accompanied with a few minor fixes to the build system and
such to accomodate AppVeyor.
The intention is that we're not switching over to AppVeyor entirely just yet,
but rather we'll watch the builds for a week or so. If everything checks out
then we'll start gating on AppVeyor instead of Buildbot!
configure: Add options for separate musl roots
This allows using the `./configure` script to enable rustbuild to compile
multiple musl targets at once. We'll hopefully use this soon on our bots to
produce a bunch of targets.
As the entry point for building the Rust compiler, a good user experience hinges
on this compiling quickly to get to the meat of the problem. To that end use
`#[cfg]`-specific dependencies to avoid building Windows crates on Unix and drop
the `regex` crate for now which was easily replacable with some string
searching.
This updates the commit to use workspaces to use `cargo metadata` instead of
hardcoded lists about what to test. This should help us be resilient to updates
in the future on behalf of the crate DAG and minimize the amount of files that
need to be touched.
This involves hacking the code used to run cargo test on various
packages, because it reads Cargo.lock to determine which packages should
be tested. This change implements a blacklist, since that will catch new
crates when they are added in the future.
rustbuild: Add install target. #34675
It just prints to the screen currently.
r? @alexcrichton
I'm working on the next commit to actually have it install.
This allows using the `./configure` script to enable rustbuild to compile
multiple musl targets at once. We'll hopefully use this soon on our bots to
produce a bunch of targets.
The previous panic message delivered when a musl target was specified
but musl-root was not specified incorrectly instructed the user to add
the musl-root key to the "build" section of config.toml. The key
actually needs to be added to the "rust" section.
In #36292, support was added to target musl libc for ARM targets using
rustbuild. Specifically, that change allowed the addition of per-target
"musl-root" options in the rustbuild config.toml so that multiple
targets depending on musl could be built. However, that implementation
contained a couple of omissions: the musl-root option was added to the
config, but was never added to the TOML parsing, and therefore was not
actually being loaded from config.toml. This commit rectifies that and
allows successful building of musl-based ARM targets.
rustdoc: Fix documenting rustc-macro crates
This commit adds a "hack" to the session to track whether we're a rustdoc
session or not. If we're rustdoc then we skip the expansion to add the
rustc-macro infrastructure.
Closes#36820
This is a hack to support building targets that don't support jemalloc
alongside hosts that do. The jemalloc build is controlled by a feature
of the std crate, and if that feature changes between targets, it
invalidates the fingerprint of std's build script (this is a cargo
bug); so we must ensure that the feature set used by std is the same
across all targets, which means we have to build the alloc_jemalloc
crate for targets like emscripten, even if we don't use it.
This commit adds a "hack" to the session to track whether we're a rustdoc
session or not. If we're rustdoc then we skip the expansion to add the
rustc-macro infrastructure.
Closes#36820
rustbuild: Use current_dir instead of -C
Apparently some versions of git don't support the `-C` flag, so let's use the
guaranteed-to-work `current_dir` function.
