Support non-stage0 check
Seems to work locally - a full stage 1 check succeeds, building std (because we can't get away with checking it), and then checking the compiler and other tools. This ran into the problem that a unconditional x.py check in stage 1 *both* checks and builds stage 1 std, and then has to clean up because for some reason the rmeta and rlib artifacts conflict (though I'm not actually entirely sure why, but it doesn't seem worth digging in in too much detail).
Ideally we wouldn't be building and checking like that but it's a minor worry as checking std is pretty fast and you can avoid it if you're aiming for speed by passing the compiler (e.g., compiler/rustc) explicitly.
r? ```@jyn514```
Allow downloading LLVM on Windows and MacOS
- Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked
statically
- Add `link-type.txt` so bootstrap knows whether llvm was linked
statically or dynamically
- Don't assume CI LLVM is linked dynamically in `bootstrap::config`
- Fall back to dynamic linking if `link-type.txt` doesn't exist
- Fix existing bug that split the output of `llvm-config` on lines, not spaces
- Only special case MacOS when dynamic linking. Static linking works fine.
- Enable building LLVM tests
This works around the following llvm bug:
```
llvm-config: error: component libraries and shared library
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a
thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles"
```
I'm not sure why llvm-config thinks these are required, but to avoid
the error, this builds them anyway.
- Bump version of `download-ci-llvm-stamp`
`src/llvm-project` hasn't changed, but the generated tarball has.
Fixes https://github.com/rust-lang/rust/issues/77084.
# Current Status
This works on both MacOS and Windows! 🎉🎉 Thanks to ```@nagisa,``` ```@halkcyon,``` ```@Lokathor,``` ```@jryans,``` and ```@poliorcetics``` for helping me test!
The `if-available` check now supports all tier 1 platforms. Although only x64 apple and x64 msvc have been tested, none of the changes here are Windows or Mac specific, and I expect this to work anywhere that LLVM artifacts are uploaded to CI (i.e. the `rust-dev` component exists).
## Windows
Note that if you have an old version of MSVC build tools you'll need to update them. VS Build Tools 2019 14.28 and later are known to work. With old tools, you may see an error like the following:
```
error LNK2001: unresolved external symbol __imp___std_init_once_complete
```
- Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked
statically
- Add `link-type.txt` so bootstrap knows whether llvm was linked
statically or dynamically
- Don't assume CI LLVM is linked dynamically in `bootstrap::config`
- Fall back to dynamic linking if `link-type.txt` doesn't exist
- Fix existing bug that split the output of `llvm-config` on lines, not spaces
- Enable building LLVM tests
This works around the following llvm bug:
```
llvm-config: error: component libraries and shared library
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a
thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles"
```
I'm not sure why llvm-config thinks these are required, but to avoid
the error, this builds them anyway.
- Temporarily set windows as the try builder. This should be reverted
before merging.
- Bump version of `download-ci-llvm-stamp`
`src/llvm-project` hasn't changed, but the generated tarball has.
- Only special case MacOS when dynamic linking. Static linking works fine.
- Store `link-type.txt` to the top-level of the tarball
This allows writing the link type unconditionally. Previously, bootstrap
had to keep track of whether the file IO *would* succeed (it would fail
if `lib/` didn't exist), which was prone to bugs.
- Make `link-type.txt` required
Anyone downloading this from CI should be using a version of bootstrap
that matches the version of the uploaded artifacts. So a missing
link-type indicates a bug in x.py.
Support `download-ci-llvm` on NixOS
In particular, the CI built `libLLVM-*.so` needs to have `libz.so`
RPATHed so that binaries like `llvm-config` work at all.
Rollup of 12 pull requests
Successful merges:
- #80442 (Mention Arc::make_mut and Rc::make_mut in the documentation of Cow)
- #80533 (bootstrap: clippy fixes)
- #80538 (Add check for `[T;N]`/`usize` mismatch in astconv)
- #80612 (Remove reverted change from relnotes)
- #80627 (Builder: Warn if test file does not exist)
- #80637 (Use Option::filter instead of open-coding it)
- #80643 (Move variable into the only branch where it is relevant)
- #80656 (Fixed documentation error for `std::hint::spin_loop`)
- #80666 (Fix missing link for "fully qualified syntax")
- #80672 (./x.py clippy: allow the most noisy lints)
- #80677 (doc -- list edit for consistency)
- #80696 (make sure that promoteds which fail to evaluate in dead const code behave correctly)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
./x.py clippy: allow the most noisy lints
This silences the following clippy lints in ./x.py clippy:
many_single_char_names (there are a lot of warnings caused by stdarch)
collapsible_if (can reduce readability)
type_complexity
missing_safety_doc (there are almost 3K warnings issued)
too_many_arguments
needless_lifetimes (people want 'tcx lifetimes etc)
wrong_self_convention (warns about from_..(), to_..(), into_..().. fns that do or do not take self by reference.
Just for clarification; this only changes the output of `x.py clippy` inside the rustc repo and does not change anything about clippy or how `cargo clippy` is run on peoples crates.
Builder: Warn if test file does not exist
Running `./x.py test` with a file that does not exists (but where the path belongs to a test suite) silently ignores the missing file and runs the whole test suite. This PR prints a warning to reduce the potential surprise factor.
Closes#80621
Don't use `self.date` unconditionally for `program_out_of_date()`
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it in https://github.com/rust-lang/rust/pull/79540
because I wasn't adding `self.date` to the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache key.
This silences the following clippy lints in ./x.py clippy:
many_single_char_names (there are a lot of warnings caused by stdarch)
collapsible_if (can reduce readability)
type_complexity
missing_safety_doc (there are almost 3K warnings issued)
too_many_arguments
needless_lifetimes (people want 'tcx lifetimes etc)
wrong_self_convention (warns about from_..(), to_..(), into_..().. fns that do or do not take self by reference.
Enable ASan, TSan, UBSan for aarch64-apple-darwin.
I confirmed ASan, TSan, UBSan all work for me locally with `clang` on my new Macbook Air.
~This requires https://github.com/rust-lang/llvm-project/pull/86~
Only produce .xz tarballs on CI
This PR adds a `./configure` option to choose which tarball compression formats to produce, and changes our CI configuration to only produce `.xz` tarballs. The release process will then recompress everything into `.gz` when producing a release.
This will drastically reduce our storage costs for CI artifacts, as we'd stop storing the same data twice. **Stable, beta and nightly releases will not be affected by this at all.**
Before landing this we'll need to increase the VM size of our release process, to recompress everything in a reasonable amount of time.
r? `@Mark-Simulacrum`
Rollup of 9 pull requests
Successful merges:
- #78934 (refactor: removing library/alloc/src/vec/mod.rs ignore-tidy-filelength)
- #79479 (Add `Iterator::intersperse`)
- #80128 (Edit rustc_ast::ast::FieldPat docs)
- #80424 (Don't give an error when creating a file for the first time)
- #80458 (Some Promotion Refactoring)
- #80488 (Do not create dangling &T in Weak<T>::drop)
- #80491 (Miri: make size/align_of_val work for dangling raw ptrs)
- #80495 (Rename kw::Invalid -> kw::Empty)
- #80513 (Add regression test for #80062)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Don't give an error when creating a file for the first time
Previously, `os.remove` would always give a FileNotFound error the first
time you called it, causing bootstrap to make unnecessary copies. This
now only calls `remove()` if the file exists, avoiding the unnecessary
error.
This is a pretty small cleanup but I think it's useful. Taken from https://github.com/rust-lang/rust/pull/79540.
Add llvm-libunwind change to bootstrap CHANGELOG
From #77703.
This doesn't need a `changelog-seen` version bump because the old values aren't accepted anymore, meaning anyone who was using this had to change it already.
r? ``@Mark-Simulacrum``
bootstrap: put the component name in the tarball temp dir path
This should not matter right now, but if we ever parallelize rustbuild this will avoid tarball contents being merged together.
r? `@Mark-Simulacrum`
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it because I wasn't adding `self.date` to
the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache
key.
Previously, `os.remove` would always give a FileNotFound error the first
time you called it, causing bootstrap to make unnecessary copies. This
now only calls `remove()` if the file exists, avoiding the unnecessary
error.
Utilize PGO for rustc linux dist builds
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater implementation time commitment,
implementing for platforms outside of x86_64-unknown-linux-gnu is skipped.
In practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour with this PR, up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
r? `@michaelwoerister` (but tell me if you don't want to / don't feel comfortable approving and we can find others)
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.
Unfortunately, #80087 forgot to update the ci-llvm stamp, so the updated
ci-llvm tarball with `llvm-dwp` wasn't downloaded by users. This commit
updates the ci-llvm stamp to resolve that problem.
Signed-off-by: David Wood <david@davidtw.co>
This commit includes the `llvm-dwp` tool in the CI LLVM (which rustc
developers can download instead of building LLVM locally) - `llvm-dwp`
is required by Split DWARF which landed in PR #77117.
Signed-off-by: David Wood <david@davidtw.co>
This commit adds a Split DWARF compare mode to compiletest so that
debuginfo tests are also tested using Split DWARF in split mode (and
manually in single mode).
Signed-off-by: David Wood <david@davidtw.co>
`llvm-dwp` is required for linking the DWARF objects into DWARF packages
when using Split DWARF, especially given that rustc produces multiple
DWARF objects (one for each codegen unit).
Signed-off-by: David Wood <david@davidtw.co>
fix more clippy::complexity findings
fix clippy::unnecessary_filter_map
use if let Some(x) = .. instead of ...map(|x|) to conditionally run fns that return () (clippy::option_map_unit_fn)
fix clippy::{needless_bool, manual_unwrap_or}
don't clone types that are copy (clippy::clone_on_copy)
don't convert types into identical types with .into() (clippy::useless_conversion)
use strip_prefix over slicing (clippy::manual_strip)
r? ``@Dylan-DPC``
Fix rustup support in default_build_triple for python3
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Follow-up to https://github.com/rust-lang/rust/pull/78513.
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Validate lint docs separately.
This addresses some concerns raised in https://github.com/rust-lang/rust/pull/76549#issuecomment-727638552 about errors with the lint docs being confusing and cumbersome. Errors from validating the lint documentation were being generated during `x.py doc` (and `x.py dist`), since extraction and validation are being done in a single step. This changes it so that extraction and validation are separated, so that `x.py doc` will not error if there is a validation problem, and tests are moved to `x.py test src/tools/lint-docs`.
This includes the following changes:
* Separate validation to `x.py test`.
* Added some more documentation on how to more easily modify and test the docs.
* Added more help to the error messages to hopefully provide more information on how to fix things.
The first commit just moves the code around, so you may consider looking at the other commits for a smaller diff.
x.py: allow a custom string appended to the version
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
Make compiletest testing use the local sysroot
We already set `compiletest` to use the local sysroot in #68019, but
that missed the configuration for testing `compiletest` itself.
fix handling the default config for profiler and sanitizers
#78354 don't handle the case that user don't add any target-specific config in `[target.*]` of `config.toml`:
```toml
changelog-seen = 2
[llvm]
link-shared = true
[build]
sanitizers = true
profiler = true
[install]
[rust]
[dist]
```
The previes code handle the default config in `Config::prase()`:
```rust
target.sanitizers = cfg.sanitizers.unwrap_or(build.sanitizers.unwrap_or_default());
target.profiler = cfg.profiler.unwrap_or(build.profiler.unwrap_or_default());
config.target_config.insert(TargetSelection::from_user(&triple), target);
```
In this case, `toml.target` don't contain any target, so the above code won't execute. Instead, a default `Target` is insert in c919f490bb/src/bootstrap/sanity.rs (L162-L166)
The default value for `bool` is false, hence the issue in #79124
This fix change the type of `sanitizers` and `profiler` to `Option<bool>`, so the default value is `None`, and fallback config is handled in `Config::sanitizers_enabled` and `Config::profiler_enabled`
fix#79124
cc `@Mark-Simulacrum` `@richkadel`
Historically the stable tarballs were named after the version number of
the specific tool, instead of the version number of Rust. For example,
both of the following tarballs were part of the same release:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-0.49.0-x86_64-unknown-linux-gnu.tar.xz
PR #77336 changed the dist code to instead use Rust's version number for
all the tarballs, regardless of the tool they contain:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-1.48.0-x86_64-unknown-linux-gnu.tar.xz
Because of that there is no need anymore to have a separate `cargo`
field in src/stage0.txt, as the Cargo version will always be the same as
the rustc version. This PR removes the field, simplifying the code and
the maintenance work required while producing releases.
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
Install CI llvm into the library directory
In other words, my concern in https://github.com/rust-lang/rust/issues/78932#issuecomment-725781767 was perfectly justified by something we were already doing. For now just special case CI LLVM, but in the future we may want a more general fix.
Fixes#79071.
r? `@alexcrichton`
Avoid installing external LLVM dylibs
If the LLVM was externally provided, then we don't currently copy artifacts into
the sysroot. This is not necessarily the right choice (in particular, it will
require the LLVM dylib to be in the linker's load path at runtime), but the
common use case for external LLVMs is distribution provided LLVMs, and in that
case they're usually in the standard search path (e.g., /usr/lib) and copying
them here is going to cause problems as we may end up with the wrong files and
isn't what distributions want.
This behavior may be revisited in the future though.
Fixes#78932.
Include llvm-as in llvm-tools-preview component
Including `llvm-as` adds the ability to include assembly language fragments that can be inlined using LTO while making sure the correct version of LLVM is always used.
If the LLVM was externally provided, then we don't currently copy artifacts into
the sysroot. This is not necessarily the right choice (in particular, it will
require the LLVM dylib to be in the linker's load path at runtime), but the
common use case for external LLVMs is distribution provided LLVMs, and in that
case they're usually in the standard search path (e.g., /usr/lib) and copying
them here is going to cause problems as we may end up with the wrong files and
isn't what distributions want.
This behavior may be revisited in the future though.
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix#78329
Infer the default host target from the host toolchain if possible
- `beta-x86_64-unknown-linux-gnu` has beta stripped
- `rustc2` is ignored
This fixes ongoing issues where x.py will detect the wrong host triple
between MSVC and GNU.
I don't think this will break anyone's workflow - I'd be very surprised if you a) had no `[build]` section in `config.toml`, b) had rustc installed, and c) expected the default target to be something other than the default target used by `rustc`. But I could be wrong - I'm happy to hear user stories :)
Fixes https://github.com/rust-lang/rust/issues/78150.
r? ``@Mark-Simulacrum``
cc ``@Lokathor``
Compile rustc crates with the initial-exec TLS model
This should produce more efficient code, with fewer calls to
__tls_get_addr. The tradeoff is that libraries using it won't work with
dlopen, but that shouldn't be a problem for rustc's internal libraries.
Honor the rustfmt setting in config.toml
Prior to this, setting the rustfmt configuration was ignored:
```
% mkdir example
% cd example
% ../configure --set build.rustfmt=/usr/bin/true
% ../x.py fmt
./x.py fmt is not supported on this channel
failed to run: /Users/shep/Projects/rust/example/build/bootstrap/debug/bootstrap fmt
Build completed unsuccessfully in 0:00:01
```
And after:
```
% ../x.py fmt
Build completed successfully in 0:00:11
```
r? `@Mark-Simulacrum`
Fix run-make tests running when LLVM is disabled
The `--cc`, `--cxx`, `--cflags` and `--ar` flags were only passed to compiletest when `builder.config.llvm_enabled()` returned true. This is preventing me from running the tests on cg_clif.
Prior to this, setting the rustfmt configuration was ignored:
```
% mkdir example
% cd example
% ../configure --set build.rustfmt=/usr/bin/true
% ../x.py fmt
./x.py fmt is not supported on this channel
failed to run: /Users/shep/Projects/rust/example/build/bootstrap/debug/bootstrap fmt
Build completed unsuccessfully in 0:00:01
```
And after:
```
% ../x.py fmt
Build completed successfully in 0:00:11
```
This should produce more efficient code, with fewer calls to
__tls_get_addr. The tradeoff is that libraries using it won't work with
dlopen, but that shouldn't be a problem for tools or for our own
internal libraries.
Co-authored-by: Mark Rousskov <mark.simulacrum@gmail.com>
Print a summary of which test suite failed
Especially on CI, where cross-compiling is common and single builder may end up
with multiple hosts and multiple targets, it can be annoying to scroll back to
the nearest start of test marker. This prints out a summary of the test suite
being run directly in compiletest.
For example, on a mir-opt failure, this would show something like this:
```
failures:
[mir-opt] mir-opt/while-storage.rs
test result: FAILED. 140 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out
Some tests failed in compiletest suite=mir-opt mode=mir-opt host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu
```
Fixes#78517
Fix `x.py clippy`
I don't think this ever worked.
Fixes https://github.com/rust-lang/rust/issues/77309. `--fix` support is a work in progress, but works for a very small subset of `libtest`.
This works by using the host `cargo-clippy` driver; it does not use `stage0.txt` at all. To mitigate confusion from this, it gives an error if you don't have `rustc +nightly` as the default rustc in `$PATH`. Additionally, it means that bootstrap can't set `RUSTC`; this makes it no longer possible for clippy to detect the sysroot itself. Instead, bootstrap passes the sysroot to cargo.
r? `@ghost`
Rune Tynan
Mara Bos
Joshua Nelson
Mark Rousskov
Yuki Okushi
Simonas Kazlauskas
Noah
Pietro Albini
bors
Dániel Buga
Matthias Krüger
Corey Farwell
Dylan DPC
Tomasz Miąsko
Vadim Petrochenkov
Tyler Mandry
Eric Huss
Yuxuan Shui
J. Ryan Stinnett
David Wood
bjorn3
Nixon Enraght-Moony
nooberfsh
Joseph Rafael Ferrer
Josh Stone
12101111
Jonas Schievink
DevJPM
Zachary Catlin
dalance
Alexis Beingessner
Jake Goulding
Josh Triplett
