This commit improves the way build-manifest calculates the checksums
included in the manifest, speeding it up:
* Instead of calculating all the hashes beforehand and then using the
ones we need, the manifest is first generated with placeholder hashes,
and then a function walks through the manifest and calculates only the
needed checksums.
* Calculating the checksums is now done in parallel with rayon, to
better utilize all the available disk bandwidth.
* Calculating the checksums now uses the sha2 crate instead of the
sha256sum CLI tool: this avoids the overhead of calling another
process, but more importantly uses hardware acceleration whenever
available (the CLI tool doesn't support it at all).
This commit adds a new tool, `build-manifest`, which is used to generate a
distribution manifest of all produced artifacts. This tool is intended to
replace the `build-rust-manifest.py` script that's currently located on the
buildmaster. The intention is that we'll have a builder which periodically:
* Downloads all artifacts for a commit
* Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files
as well as TOML manifests.
* Upload all generated hashes and manifests to the directory the artifacts came
from.
* Upload *all* artifacts (tarballs and hashes and manifests) to an archived
location.
* If necessary, upload all artifacts to the main location.
This script is intended to just be the second step here where orchestrating
uploads and such will all happen externally from the build system itself.
Pietro Albini
Ralf Jung
Eric Huss
Andy Russell
Hirokazu Hata
Mark Simulacrum
Alex Crichton