This commit changes the way build-manifest is invoked, to let it accept
the Rust version directly instead of requiring the path of the Rust
monorepo and letting build-manifest figure out the path on its own.
This allows to run build-manifest without a clone of the monorepo.
The format of the tarballs produced by CI is roughly the following:
{component}-{release}-{target}.{ext}
While on the beta and nightly channels `{release}` is just the channel
name, on the stable channel is either the Rust version or the version of
the component we're shipping:
cargo-0.47.0-x86_64-unknown-linux-gnu.tar.xz
clippy-0.0.212-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-1.46.0-x86_64-unknown-linux-gnu.tar.xz
miri-0.1.0-x86_64-unknown-linux-gnu.tar.xz
rls-1.41.0-x86_64-unknown-linux-gnu.tar.xz
rust-1.46.0-x86_64-unknown-linux-gnu.tar.xz
...
This makes it really hard to get the package URL without having access
to the manifest (and there is no manifest on ci-artifacts.rlo), as there
is no consistent version number to use.
This commit addresses the problem by always using the Rust version
number as `{release}` for the stable channel, regardless of the version
number of the component we're shipping. I chose that instead of "stable"
to avoid breaking the URL scheme *that* much.
Rustup should not be affected by this change, as it fetches the URLs
from the manifest. Unfortunately we don't have a way to test other
clients before making a stable release, as this change only affects the
stable channel.
Improve build-manifest to work with the improved promote-release
This PR makes some changes to build-manifest to have it work better with the other improvements I'm making to [promote-release](https://github.com/rust-lang/promote-release).
A new way to invoke the tool was added: `./x.py run src/tools/build-manifest`. The new invocation disables the generation of `.sha256` files and the generation of GPG signatures, as those steps are not tied to the Rust version we're building the manifest of: handling them in `promote-release` will improve the maintenability of our release process. Invocations through the old command (`./x.py dist hash-and-sign`) are referred inside the source code as "legacy". The new invocation also enables internal parallelism, disabled on legacy to avoid overloading our old server.
Improvements were also made on how the checksums included in the manifest are generated:
* The manifest is first generated with placeholder checksums, and then a function walks through the manifes and calculates only the needed hashes. Before this PR, all the hashes were calculated beforehand, including the hashes of unused files.
* Calculating the hashes is now done in parallel with rayon, to better utilize all the available disk bandwidth.
* The `sha2` crate is now used instead of the `sha256sum` CLI tool: this avoids the overhead of calling another process, but more importantly enables hardware acceleration whenever available (the `sha256sum` CLI tool doesn't support it at all).
r? @Mark-Simulacrum
This PR is best reviewed commit-by-commit.
This commit improves the way build-manifest calculates the checksums
included in the manifest, speeding it up:
* Instead of calculating all the hashes beforehand and then using the
ones we need, the manifest is first generated with placeholder hashes,
and then a function walks through the manifest and calculates only the
needed checksums.
* Calculating the checksums is now done in parallel with rayon, to
better utilize all the available disk bandwidth.
* Calculating the checksums now uses the sha2 crate instead of the
sha256sum CLI tool: this avoids the overhead of calling another
process, but more importantly uses hardware acceleration whenever
available (the CLI tool doesn't support it at all).
This connects to https://github.com/rust-lang/rustup/issues/794.
It's hard to remember if there have been patch releases for old versions
when you'd like to install the latest in a MAJOR.MINOR series.
When we're doing a stable release, we write duplicate manifests to
`stable`. With this change, only when we're doing a stable release, also
write duplicate manifests to `MAJOR.MINOR` to eventually enable rustup
(and any other tooling that builds Rust release URLs) to request, say,
`1.45` and get `1.45.2` (assuming `1.45.2` is the latest available
`1.45` and assuming that we never publish patch releases out of order).
Refactor versions detection in build-manifest
This PR refactors how `build-manifest` handles versions, making the following changes:
* `build-manifest` now detects the "package releases" on its own, without relying on rustbuild providing them through CLI arguments. This drastically simplifies calling the tool outside of `x.py`, and will allow to ship the prebuilt tool in a tarball in the future, with the goal of stopping to invoke `x.py` during `promote-release`.
* The `tar` command is not used to extract the version and the git hash from tarballs anymore. The `flate2` and `tar` crates are used instead. This makes detecting those pieces of data way faster, as the archive is decompressed just once and we stop parsing the archive once all the information is retrieved.
* The code to extract the version and the git hash now stores all the collected data dynamically, without requiring to add new fields to the `Builder` struct every time.
I tested the changes locally and it should behave the same as before.
r? `@Mark-Simulacrum`
Previously it was listed as a package but wasn't available in the component
lists in rustup, so wasn't actually installable.
rustc-docs is also only present for x86_64-unknown-linux-gnu. Eventually it'll
also be shipped for aarch64-gnu with current CI configuration, but that builder
isn't quite up and running yet.
We probably want to ship compiler docs for other platforms as well, though, but
this commit doesn't enable that quite yet. A future PR may do so by adding
--enable-compiler-docs to the relevant builders (but it would also need to
decide the set of builders which we'd ship on).
Add the rust-docs component to toolchain x86_64-unknown-linux-musl, which allows
people using rustup on their musl-based linux distribution to download the
rust-docs.
Generating and uploading the docs was enabled in b5d143b.
This change creates a new Docker image, "dist-x86_64-illumos", and sets
things up to build the full set of "dist" packages for illumos hosts, so
that illumos users can use "rustup" to install packages. It also
adjusts the manifest builder to expect complete toolchains for this
platform.
add bare metal ARM Cortex-A targets to rustc
-> `rustc --target armv7a-none-eabi` will work
also build rust-std (rustup) components for them
-> `rustup target add armv7a-none-eabi` will work
this completes our bare-metal support of ARMv7 cores on stable Rust (by 1.42 or 1.43)
(these target specifications have been tested on a real (no emulation / QEMU) [Cortex-A7 core](https://github.com/iqlusioninc/usbarmory.rs/))
It was already filtered from other branches, but we only kept it in
nightly's default to ease the transition. Now that the separation of
rust-std/rustc-dev has reached the 1.40 release, it seems like a good
time for that transition to end.
This splits out a rustc-dev component with the compiler crates, and
keeps the status quo of default installed files on nightly. The default
changing to not install compiler libraries by default is left for a
future pull request.
However, on stable and beta, this does remove the compiler libraries
from the set of libraries installed by default, as they are never needed
there (per our stability story, they "cannot" be used).
Pietro Albini
Dylan DPC
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
bors
Ralf Jung
Alistair Francis
Jens Reidel
Mark Rousskov
Matthias Krüger
David Sonder
msizanoen1
Joshua M. Clulow
Aleksey Kladov
Eric Huss
yuqio
Patrick Mooney
Andre Richter
Tyler Mandry
Jorge Aparicio
msizanoen1
Josh Stone
Wang Xuerui
Stefan Lankes