Utilize PGO for rustc linux dist builds
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater implementation time commitment,
implementing for platforms outside of x86_64-unknown-linux-gnu is skipped.
In practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour with this PR, up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
r? `@michaelwoerister` (but tell me if you don't want to / don't feel comfortable approving and we can find others)
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.
Unfortunately, #80087 forgot to update the ci-llvm stamp, so the updated
ci-llvm tarball with `llvm-dwp` wasn't downloaded by users. This commit
updates the ci-llvm stamp to resolve that problem.
Signed-off-by: David Wood <david@davidtw.co>
This commit includes the `llvm-dwp` tool in the CI LLVM (which rustc
developers can download instead of building LLVM locally) - `llvm-dwp`
is required by Split DWARF which landed in PR #77117.
Signed-off-by: David Wood <david@davidtw.co>
This commit adds a Split DWARF compare mode to compiletest so that
debuginfo tests are also tested using Split DWARF in split mode (and
manually in single mode).
Signed-off-by: David Wood <david@davidtw.co>
`llvm-dwp` is required for linking the DWARF objects into DWARF packages
when using Split DWARF, especially given that rustc produces multiple
DWARF objects (one for each codegen unit).
Signed-off-by: David Wood <david@davidtw.co>
fix more clippy::complexity findings
fix clippy::unnecessary_filter_map
use if let Some(x) = .. instead of ...map(|x|) to conditionally run fns that return () (clippy::option_map_unit_fn)
fix clippy::{needless_bool, manual_unwrap_or}
don't clone types that are copy (clippy::clone_on_copy)
don't convert types into identical types with .into() (clippy::useless_conversion)
use strip_prefix over slicing (clippy::manual_strip)
r? ``@Dylan-DPC``
Fix rustup support in default_build_triple for python3
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Follow-up to https://github.com/rust-lang/rust/pull/78513.
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
Validate lint docs separately.
This addresses some concerns raised in https://github.com/rust-lang/rust/pull/76549#issuecomment-727638552 about errors with the lint docs being confusing and cumbersome. Errors from validating the lint documentation were being generated during `x.py doc` (and `x.py dist`), since extraction and validation are being done in a single step. This changes it so that extraction and validation are separated, so that `x.py doc` will not error if there is a validation problem, and tests are moved to `x.py test src/tools/lint-docs`.
This includes the following changes:
* Separate validation to `x.py test`.
* Added some more documentation on how to more easily modify and test the docs.
* Added more help to the error messages to hopefully provide more information on how to fix things.
The first commit just moves the code around, so you may consider looking at the other commits for a smaller diff.
x.py: allow a custom string appended to the version
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
Make compiletest testing use the local sysroot
We already set `compiletest` to use the local sysroot in #68019, but
that missed the configuration for testing `compiletest` itself.
fix handling the default config for profiler and sanitizers
#78354 don't handle the case that user don't add any target-specific config in `[target.*]` of `config.toml`:
```toml
changelog-seen = 2
[llvm]
link-shared = true
[build]
sanitizers = true
profiler = true
[install]
[rust]
[dist]
```
The previes code handle the default config in `Config::prase()`:
```rust
target.sanitizers = cfg.sanitizers.unwrap_or(build.sanitizers.unwrap_or_default());
target.profiler = cfg.profiler.unwrap_or(build.profiler.unwrap_or_default());
config.target_config.insert(TargetSelection::from_user(&triple), target);
```
In this case, `toml.target` don't contain any target, so the above code won't execute. Instead, a default `Target` is insert in c919f490bb/src/bootstrap/sanity.rs (L162-L166)
The default value for `bool` is false, hence the issue in #79124
This fix change the type of `sanitizers` and `profiler` to `Option<bool>`, so the default value is `None`, and fallback config is handled in `Config::sanitizers_enabled` and `Config::profiler_enabled`
fix#79124
cc `@Mark-Simulacrum` `@richkadel`
Historically the stable tarballs were named after the version number of
the specific tool, instead of the version number of Rust. For example,
both of the following tarballs were part of the same release:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-0.49.0-x86_64-unknown-linux-gnu.tar.xz
PR #77336 changed the dist code to instead use Rust's version number for
all the tarballs, regardless of the tool they contain:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-1.48.0-x86_64-unknown-linux-gnu.tar.xz
Because of that there is no need anymore to have a separate `cargo`
field in src/stage0.txt, as the Cargo version will always be the same as
the rustc version. This PR removes the field, simplifying the code and
the maintenance work required while producing releases.
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
Install CI llvm into the library directory
In other words, my concern in https://github.com/rust-lang/rust/issues/78932#issuecomment-725781767 was perfectly justified by something we were already doing. For now just special case CI LLVM, but in the future we may want a more general fix.
Fixes#79071.
r? `@alexcrichton`
Avoid installing external LLVM dylibs
If the LLVM was externally provided, then we don't currently copy artifacts into
the sysroot. This is not necessarily the right choice (in particular, it will
require the LLVM dylib to be in the linker's load path at runtime), but the
common use case for external LLVMs is distribution provided LLVMs, and in that
case they're usually in the standard search path (e.g., /usr/lib) and copying
them here is going to cause problems as we may end up with the wrong files and
isn't what distributions want.
This behavior may be revisited in the future though.
Fixes#78932.
Include llvm-as in llvm-tools-preview component
Including `llvm-as` adds the ability to include assembly language fragments that can be inlined using LTO while making sure the correct version of LLVM is always used.
If the LLVM was externally provided, then we don't currently copy artifacts into
the sysroot. This is not necessarily the right choice (in particular, it will
require the LLVM dylib to be in the linker's load path at runtime), but the
common use case for external LLVMs is distribution provided LLVMs, and in that
case they're usually in the standard search path (e.g., /usr/lib) and copying
them here is going to cause problems as we may end up with the wrong files and
isn't what distributions want.
This behavior may be revisited in the future though.
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix#78329
Infer the default host target from the host toolchain if possible
- `beta-x86_64-unknown-linux-gnu` has beta stripped
- `rustc2` is ignored
This fixes ongoing issues where x.py will detect the wrong host triple
between MSVC and GNU.
I don't think this will break anyone's workflow - I'd be very surprised if you a) had no `[build]` section in `config.toml`, b) had rustc installed, and c) expected the default target to be something other than the default target used by `rustc`. But I could be wrong - I'm happy to hear user stories :)
Fixes https://github.com/rust-lang/rust/issues/78150.
r? ``@Mark-Simulacrum``
cc ``@Lokathor``
Compile rustc crates with the initial-exec TLS model
This should produce more efficient code, with fewer calls to
__tls_get_addr. The tradeoff is that libraries using it won't work with
dlopen, but that shouldn't be a problem for rustc's internal libraries.
Honor the rustfmt setting in config.toml
Prior to this, setting the rustfmt configuration was ignored:
```
% mkdir example
% cd example
% ../configure --set build.rustfmt=/usr/bin/true
% ../x.py fmt
./x.py fmt is not supported on this channel
failed to run: /Users/shep/Projects/rust/example/build/bootstrap/debug/bootstrap fmt
Build completed unsuccessfully in 0:00:01
```
And after:
```
% ../x.py fmt
Build completed successfully in 0:00:11
```
r? `@Mark-Simulacrum`
Fix run-make tests running when LLVM is disabled
The `--cc`, `--cxx`, `--cflags` and `--ar` flags were only passed to compiletest when `builder.config.llvm_enabled()` returned true. This is preventing me from running the tests on cg_clif.
Prior to this, setting the rustfmt configuration was ignored:
```
% mkdir example
% cd example
% ../configure --set build.rustfmt=/usr/bin/true
% ../x.py fmt
./x.py fmt is not supported on this channel
failed to run: /Users/shep/Projects/rust/example/build/bootstrap/debug/bootstrap fmt
Build completed unsuccessfully in 0:00:01
```
And after:
```
% ../x.py fmt
Build completed successfully in 0:00:11
```
This should produce more efficient code, with fewer calls to
__tls_get_addr. The tradeoff is that libraries using it won't work with
dlopen, but that shouldn't be a problem for tools or for our own
internal libraries.
Co-authored-by: Mark Rousskov <mark.simulacrum@gmail.com>
Print a summary of which test suite failed
Especially on CI, where cross-compiling is common and single builder may end up
with multiple hosts and multiple targets, it can be annoying to scroll back to
the nearest start of test marker. This prints out a summary of the test suite
being run directly in compiletest.
For example, on a mir-opt failure, this would show something like this:
```
failures:
[mir-opt] mir-opt/while-storage.rs
test result: FAILED. 140 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out
Some tests failed in compiletest suite=mir-opt mode=mir-opt host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu
```
Fixes#78517
Fix `x.py clippy`
I don't think this ever worked.
Fixes https://github.com/rust-lang/rust/issues/77309. `--fix` support is a work in progress, but works for a very small subset of `libtest`.
This works by using the host `cargo-clippy` driver; it does not use `stage0.txt` at all. To mitigate confusion from this, it gives an error if you don't have `rustc +nightly` as the default rustc in `$PATH`. Additionally, it means that bootstrap can't set `RUSTC`; this makes it no longer possible for clippy to detect the sysroot itself. Instead, bootstrap passes the sysroot to cargo.
r? `@ghost`
Here's the error for rustdoc:
```
Checking rustdoc artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
error: no library targets found in package `rustdoc-tool`
```
Fix panic in bootstrap for non-workspace path dependencies.
If you add a `path` dependency to a `Cargo.toml` that is located outside of the workspace, then the `in_tree_crates` function can panic because it finds a path dependency that is not defined (since it uses `cargo metadata --no-deps`). This fixes it by skipping over those entries, which are usually not things you select on the command-line.
Fixes#78617
Suggest library/std when running all stage 0 tests
r? ``@Mark-Simulacrum``
cc ``@ijackson``
For context, this came out of a discord conversation where ``@ijackson`` was running `test --stage 1` when they were only adding doc-tests to the standard library.
Especially on CI, where cross-compiling is common and single builder may end up
with multiple hosts and multiple targets, it can be annoying to scroll back to
the nearest start of test marker. This prints out a summary of the test suite
being run directly in compiletest.
Clippy does its own runtime detection of the sysroot, which was
incorrect in this case (it used the beta sysroot). This overrides the
sysroot to use `stage0-sysroot` instead.
- Get `x.py clippy` to work on nightly
- Give a nice error message if nightly clippy isn't installed
Add cg_clif as optional codegen backend
Rustc_codegen_cranelift is an alternative codegen backend for rustc based on Cranelift. It has the potential to improve compilation times in debug mode. In my experience the compile time improvements over debug mode LLVM for a clean build are about 20-30% in most cases.
This PR adds cg_clif as optional codegen backend. By default it is only enabled for `./x.py check`. It can be enabled for `./x.py build` too by adding `cranelift` to the `rust.codegen-backends` array in `config.toml`.
MCP: https://github.com/rust-lang/compiler-team/issues/270
r? `@Mark-Simulacrum`
Sync LLVM submodule if it has been initialized
Since having enabled the download-ci-llvm option,
and having rebased on top of #76864,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
Haiku: explicitly set CMAKE_SYSTEM_NAME when cross-compiling
This resolves issues where the cross-build of LLVM fails because it tries to
link to the host's system libraries instead of the target's system libraries.
Since having enabled the download-ci-llvm option,
and having rebased on top of f05b47ccdf,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
Modify executable checking to be more universal
This uses a dummy file to check if the filesystem being used supports the executable bit in general.
Supersedes #74753.
This resolves issues where the cross-build of LLVM fails because it tries to
link to the host's system libraries instead of the target's system libraries.
Detect configuration for LLVM during setup
This is a first draft to address #77579, setting `download-ci-llvm` to true on Linux, but I could also implement the `if-available` setting mentioned in the issue.
On other platforms I was thinking about using [the which crate](https://crates.io/crates/which), if adding a dependency on it is considered okay of course, to detect the presence of `llvm-config` in the path, and use it if found. Still a work in progress of course.
We need a fresh input buffer each time, or we reuse the previous
data (since `read_line` appends).
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Put all()'s otuput in the order we want to print things in, and add a
comment about why they are in this order. Provide purpose() and
all_for_help(). Use these things everywhere.
Move all the abbrev character ("a", "b", etc.) processing into
interactive_path.
No functional change.
Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
bootstrap: only use compiler-builtins-c if they exist
The assignment of `features` above was added in rust-lang#60981, but
never used. Presumably the intent was to replace the string literal here
with it.
While I'm in the area, `compiler_builtins_c_feature` doesn't need to be
a `String`.
I'm not entirely sure of a great way to locally test this -- `./x.py test`
passed on my machine, but 🤷♂️.
r? @alexcrichton
bootstrap: set correct path for the build-manifest binary
This changes the path of the binary inside the tarball to be:
```
build-manifest-{release}-{target}/build-manifest/bin/build-manifest
```
...instead of:
```
build-manifest-{release}-{target}/build-manifest/bin/build-manifest/build-manifest
```
r? @Mark-Simulacrum
Include `llvm-dis`, `llc` and `opt` in `llvm-tools-preview` component
Fixes#55890
It's useful to have `llc` and `opt` available when debugging an LLVM
miscompilation,.
Mention rustdoc in `x.py setup`
This lets new contributors know which option they should pick; previously it wasn't clear 'compiler' also included rustdoc.
Unresolved questions: should this say 'compiler and tools' instead? I don't know of any tools that are modified in-tree other than rustdoc, though.
r? @Mark-Simulacrum
Give an error when running `x.py test --stage 0 src/test/ui`
The error can be overridden with `COMPILETEST_FORCE_STAGE0=1`. In practice I don't know why anyone would do this.
r? @Mark-Simulacrum
Closes https://github.com/rust-lang/rust/issues/77711
Dist build manifest
This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`:
* `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
* A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system.
* The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`.
* The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime.
This PR is best reviewed commit-by-commit.
r? `@Mark-Simulacrum`
This commit changes the way build-manifest is invoked, to let it accept
the Rust version directly instead of requiring the path of the Rust
monorepo and letting build-manifest figure out the path on its own.
This allows to run build-manifest without a clone of the monorepo.
More print statementsstatements lol
Solved the basic case of eliminating check_version ifk_version if subcommand = setup
Finished v1
checking out old bootstrap.py
checked out old irrelevant files
fixed tidy
Moved VERSION from bin/main.rs to lib.rs
Fixed semicolon return issue
x.py fmt
The assignment of `features` above was added in rust-lang#60981, but
never used. Presumably the intent was to replace the string literal here
with it.
While I'm in the area, `compiler_builtins_c_feature` doesn't need to be
a `String`.
The format of the tarballs produced by CI is roughly the following:
{component}-{release}-{target}.{ext}
While on the beta and nightly channels `{release}` is just the channel
name, on the stable channel is either the Rust version or the version of
the component we're shipping:
cargo-0.47.0-x86_64-unknown-linux-gnu.tar.xz
clippy-0.0.212-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-1.46.0-x86_64-unknown-linux-gnu.tar.xz
miri-0.1.0-x86_64-unknown-linux-gnu.tar.xz
rls-1.41.0-x86_64-unknown-linux-gnu.tar.xz
rust-1.46.0-x86_64-unknown-linux-gnu.tar.xz
...
This makes it really hard to get the package URL without having access
to the manifest (and there is no manifest on ci-artifacts.rlo), as there
is no consistent version number to use.
This commit addresses the problem by always using the Rust version
number as `{release}` for the stable channel, regardless of the version
number of the component we're shipping. I chose that instead of "stable"
to avoid breaking the URL scheme *that* much.
Rustup should not be affected by this change, as it fetches the URLs
from the manifest. Unfortunately we don't have a way to test other
clients before making a stable release, as this change only affects the
stable channel.
Add c as a shorthand check alternative for new options #77603
There is a missing "c" that is a shorthand for "check" in newly added match arm for handling check-specific options.
Fix suggestions for x.py setup
#76631 introduced a new `setup` command to x.py
By default the command prompts for a profile to use:
```
Welcome to the Rust project! What do you want to do with x.py?
a) Contribute to the standard library
b) Contribute to the compiler
c) Contribute to the compiler, and also modify LLVM or codegen
d) Install Rust from source
```
and then displays command suggestions, depending on which profile was chosen. However [the mapping between chosen profile](9cba260df0/src/bootstrap/setup.rs (L75-L85)) and [suggestion](9cba260df0/src/bootstrap/setup.rs (L42-L47)) isn't exact, leading to suggestions not being shown if the user presses `c` or `d`. (because "c" is translated to "llvm" and "d" to "maintainer", but suggestions trigger for "codegen" and "user" respectively)
A more thorough refactor would stop using "strings-as-type" to make sure this kind of error doesn't happen, but it may be overkill for that kind of "script" program?
Tagging the setup command author: @jyn514
Add a command to install a git hook to automatically run `x.py test tidy --bless`
Some folks (such as myself) would probably find a lot of convenience in a pre-commit hook that automatically runs tidy before committing, to avoid burning CI time learning that your commit wasn't tidy.
I'm absolutely positive I have missed some stuff. I basically just got this to where you can run `./x.py run install-git-hook` and then clicked the commit button. Please let me know what else you'd like me to add before this can be merged!
[rustc-dev-guide companion PR](https://github.com/rust-lang/rustc-dev-guide/pull/848)
Rename bootstrap/defaults/{config.toml.PROFILE => config.PROFILE.toml}
This allows these files to have okay syntax highlighting in editors, and helps avoid nagging from editors which want to suggest that I install a plugin for `*.library` files to view the `config.toml.library` or whatever.
It's a very minor change.
r?@jyn514
Improve build-manifest to work with the improved promote-release
This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to [promote-release](https://github.com/rust-lang/promote-release).
A new way to invoke the tool was added: `./x.py run src/tools/build-manifest`. The new invocation disables the generation of `.sha256` files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them in `promote-release` will improve the maintenability of our release process. Invocations through the old command (`./x.py dist hash-and-sign`) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server.
Improvements were also made on how the checksums included in the manifest are generated:
* The manifest is first generated with placeholder checksums, and then a function walks through the manifes and calculates only the needed hashes. Before this PR, all the hashes were calculated beforehand, including the hashes of unused files.
* Calculating the hashes is now done in parallel with rayon, to better utilize all the available disk bandwidth.
* The `sha2` crate is now used instead of the `sha256sum` CLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (the `sha256sum` CLI tool doesn't support it at all).
r? @Mark-Simulacrum
This PR is best reviewed commit-by-commit.
This matches Cargo behavior and avoids the (somewhat expensive) double checking,
as well as the unfortunate duplicate error messages (#76822,
rust-lang/cargo#5128).
Remove --cfg dox from rustdoc.rs
This was added in https://github.com/rust-lang/rust/pull/53076 because
several dependencies were using `cfg(dox)` instead of `cfg(rustdoc)` (now `cfg(doc)`).
I ran `rg 'cfg\(dox\)'` on the source tree with no matches, so I think
this is now safe to remove.
r? `@Mark-Simulacrum`
cc `@QuietMisdreavus` :)
Don't warn if the config file is somewhere other than `config.toml`
Previously, `config.config` was always hardcoded as `"config.toml"`.
I thought that it was being overridden with the actual value later, but
it turns out `flags.config` was being completely discarded. This keeps
`config.config` in sync with `flags.config`.
Fixes https://github.com/rust-lang/rust/issues/77293
r? `@Mark-Simulacrum`
cc `@davidtwco`
bootstrap: Always build for host, even when target is given
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
r? `@Mark-Simulacrum`
This was added in https://github.com/rust-lang/rust/pull/53076 because
several dependencies were using `cfg(dox)` instead of `cfg(rustdoc)`.
I ran `rg 'cfg\(dox\)'` on the source tree with no matches, so I think
this is now safe to remove.
Refactor versions detection in build-manifest
This PR refactors how `build-manifest` handles versions, making the following changes:
* `build-manifest` now detects the "package releases" on its own, without relying on rustbuild providing them through CLI arguments. This drastically simplifies calling the tool outside of `x.py`, and will allow to ship the prebuilt tool in a tarball in the future, with the goal of stopping to invoke `x.py` during `promote-release`.
* The `tar` command is not used to extract the version and the git hash from tarballs anymore. The `flate2` and `tar` crates are used instead. This makes detecting those pieces of data way faster, as the archive is decompressed just once and we stop parsing the archive once all the information is retrieved.
* The code to extract the version and the git hash now stores all the collected data dynamically, without requiring to add new fields to the `Builder` struct every time.
I tested the changes locally and it should behave the same as before.
r? `@Mark-Simulacrum`
This changes the behavior from *not* building for host whenever an
explicit target is specified. I find this much less confusing.
You can still disable host steps by passing an explicit empty list for
host.
Fixes#76990.
Previously, `config.config` was always hardcoded as `"config.toml"`.
I thought that it was being overridden with the actual value later, but
it turns out `flags.config` was being completely discarded. This keeps
`config.config` in sync with `flags.config`.
Mark Rousskov
bors
Joshua Nelson
Pietro Albini
Eric Huss
Tomasz Miąsko
Yuxuan Shui
J. Ryan Stinnett
David Wood
Yuki Okushi
bjorn3
Matthias Krüger
Corey Farwell
Nixon Enraght-Moony
Mara Bos
nooberfsh
Joseph Rafael Ferrer
Dylan DPC
Josh Stone
12101111
Jonas Schievink
DevJPM
Zachary Catlin
dalance
Alexis Beingessner
Jake Goulding
Josh Triplett
Guillaume Gomez
Nelson J Morais
Marc-Antoine Perennou
est31
Simon Sapin
Camelid
Niels Sascha Reedijk
Ian Jackson
Antoine Martin
Aaron Hill
Winnie Xiao
George Burgess IV
Alexander Koptelov
Cassandra Fridkin
Thom Chiovoloni
ecstatic-morse
Tyler Mandry
