This commit removes the `rand` crate from the standard library facade as
well as the `__rand` module in the standard library. Neither of these
were used in any meaningful way in the standard library itself. The only
need for randomness in libstd is to initialize the thread-local keys of
a `HashMap`, and that unconditionally used `OsRng` defined in the
standard library anyway.
The cruft of the `rand` crate and the extra `rand` support in the
standard library makes libstd slightly more difficult to port to new
platforms, namely WebAssembly which doesn't have any randomness at all
(without interfacing with JS). The purpose of this commit is to clarify
and streamline randomness in libstd, focusing on how it's only required
in one location, hashmap seeds.
Note that the `rand` crate out of tree has almost always been a drop-in
replacement for the `rand` crate in-tree, so any usage (accidental or
purposeful) of the crate in-tree should switch to the `rand` crate on
crates.io. This then also has the further benefit of avoiding
duplication (mostly) between the two crates!
Make sure to clear out the stageN-{rustc,std,tools} directories.
We copy built tool binaries into a dedicated directory to avoid deleting them,
stageN-tools-bin. These aren't ever cleared out by code, since there should be
no reason to do so, and we'll simply overwrite them as necessary.
When clearing out the stageN-{std,rustc,tools} directories, make sure to delete
both Cargo directories -- per-target and build scripts. This ensures that
changing libstd doesn't cause problems due to build scripts not being rebuilt,
even though they should be.
Fixes https://github.com/rust-lang/rust/issues/44739.
This commit enables configuring the RLS/rustfmt tools to the "broken" state and
actually get it past CI. The main changes here were to update all dist-related
code to handle the situation where the RLS isn't available. This in turn
involved a homegrown preprocessor-like-function to edit the configuration files
we pass to the various combined installer tools.
Apparently `File::create` was called when there was an existing hard link or the
like, causing an existing file to get accidentally truncated!
Closes#44487
The `copy` function historically in rustbuild used hard links to speed up the
copy operations that it does. This logic was backed out, however, in #39518 due
to a bug that only showed up on Windows, described in #39504. The cause
described in #39504 happened because Cargo, on a fresh build, would overwrite
the previous artifacts with new hard links that Cargo itself manages.
This behavior in Cargo was fixed in rust-lang/cargo#4390 where it no longer
should overwrite files on fresh builds, opportunistically leaving the filesystem
intact and not touching it.
Hopefully this can help speed up local builds by doing fewer copies all over the
place!
We wanted `src/compiler-rt/test` filtered from the `rust-src` package,
but that path is now `src/libcompiler_builtins/compiler-rt/test`. This
saves over half of the installed rust-src size. (50MB -> 22MB)
include Cargo.{toml,lock} in rust-src tarball
The lock file is interesting because e.g. xargo could use it to build libstd against the same dependencies that were used for the main build. More generally speaking, just documenting in this form which exact dependencies should be used IMHO makes lots of sense.
I added the Cargo.toml mostly because having the lock without the toml feels odd. Of course, the toml contains references to paths that don't actually exist in the rust-src tarball. Not sure if that is considered a problem.
This commit rewrites our ancient `./configure` script from shell into Python.
The impetus for this change is to remove `config.mk` which is just a vestige of
the old makefile build system at this point. Instead all configuration is now
solely done through `config.toml`.
The python script allows us to more flexibly program (aka we can use loops
easily) and create a `config.toml` which is based off `config.toml.example`.
This way we can preserve comments and munge various values as we see fit.
It is intended that the configure script here is a drop-in replacement for the
previous configure script, no functional change is intended. Also note that the
rationale for this is also because our build system requires Python, so having a
python script a bit earlier shouldn't cause too many problems.
Closes#40730
This introduces a slight change in behavior, where we unilaterally
respect the --host and --target parameters passed for all sanity
checking and runtime configuration.
When copying libstd for the stage 2 compiler, the builder ignores the
configured libdir/libdir_relative configuration parameters. This causes
the compiler to fail to find libstd, which cause any tools built with the
stage 2 compiler to fail.
To fix this, make the copy steps of rustbuild aware of the libdir_relative
parameter when the stage >= 2. Also update the dist target to be aware of
the new location of libstd.
Switch to rust-lang-nursery/compiler-builtins
This commit migrates the in-tree `libcompiler_builtins` to the upstream version
at https://github.com/rust-lang-nursery/compiler-builtins. The upstream version
has a number of intrinsics written in Rust and serves as an in-progress rewrite
of compiler-rt into Rust. Additionally it also contains all the existing
intrinsics defined in `libcompiler_builtins` for 128-bit integers.
It's been the intention since the beginning to make this transition but
previously it just lacked the manpower to get done. As this PR likely shows it
wasn't a trivial integration! Some highlight changes are:
* The PR rust-lang-nursery/compiler-builtins#166 contains a number of fixes
across platforms and also some refactorings to make the intrinsics easier to
read. The additional testing added there also fixed a number of integration
issues when pulling the repository into this tree.
* LTO with the compiler-builtins crate was fixed to link in the entire crate
after the LTO process as these intrinsics are excluded from LTO.
* Treatment of hidden symbols was updated as previously the
`#![compiler_builtins]` crate would mark all symbol *imports* as hidden
whereas it was only intended to mark *exports* as hidden.
This commit migrates the in-tree `libcompiler_builtins` to the upstream version
at https://github.com/rust-lang-nursery/compiler-builtins. The upstream version
has a number of intrinsics written in Rust and serves as an in-progress rewrite
of compiler-rt into Rust. Additionally it also contains all the existing
intrinsics defined in `libcompiler_builtins` for 128-bit integers.
It's been the intention since the beginning to make this transition but
previously it just lacked the manpower to get done. As this PR likely shows it
wasn't a trivial integration! Some highlight changes are:
* The PR rust-lang-nursery/compiler-builtins#166 contains a number of fixes
across platforms and also some refactorings to make the intrinsics easier to
read. The additional testing added there also fixed a number of integration
issues when pulling the repository into this tree.
* LTO with the compiler-builtins crate was fixed to link in the entire crate
after the LTO process as these intrinsics are excluded from LTO.
* Treatment of hidden symbols was updated as previously the
`#![compiler_builtins]` crate would mark all symbol *imports* as hidden
whereas it was only intended to mark *exports* as hidden.
This commit deletes the in-tree `getopts` crate in favor of the crates.io-based
`getopts` crate. The main difference here is with a new builder-style API, but
otherwise everything else remains relatively standard.
Add the RLS to .exe, .msi, and .pkg installers
This directly addresses issue #42157, adding the RLS as a non-default component in the mentioned installers. The windows installers appear to have the right functionality added, but I don't have a machine that runs OSX, so it would be great if someone could test whether my .pkg commit adds the RLS correctly. The final commit also fixes some formatting issues I'd noticed while working on the installers, but I don't know if that's within the scope of this PR, so input would be appreciated.
rustbuild: don't create a source tarball when installing
This splits Install out of Dist as it is not a full dist anymore, and creates the source tarball only for the Dist command.
This will allow splitting install in a few rules if we want as it's done for other phases.
only create source tarball for the Dist subcommand
mark install rule as default for Kind::Install
split install-docs
split install-std
factor out empty_dir handling
split install-cargo
split install-analysis
split install-src
rework install-rustc
properly handle cross-compilation setups for install
use pkgname in install
split plain source tarball generation from rust-src dist
document src-tarball in config.toml.exmaple
Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
There's no shell interpreting the file paths under the new Rusty
rust-installer, so we don't need to use `sanitize_sh` for it. Plus,
the drive-letter transformation is actually harmful for the now-native
Windows rust-installer to understand those paths.
Add a distcheck for rust-src completeness
This is for the last commit of #41546. For some reason, @bors only saw the first two commits, and wouldn't approve the last even when explicitly directed so.
r? @alexcrichton
We've got a freshly minted beta compiler, let's update to use that on nightly!
This has a few other changes associated with it as well
* A bump to the rustc version number (to 1.19.0)
* Movement of the `cargo` and `rls` submodules to their "proper" location in
`src/tools/{cargo,rls}`. Now that Cargo workspaces support the `exclude`
option this can work.
* Updates of the `cargo` and `rls` submodules to their master branches.
* Tweak to the `src/stage0.txt` format to be more amenable for Cargo version
numbers. On the beta channel Cargo will bootstrap from a different version
than rustc (e.g. the version numbers are different), so we need different
configuration for this.
* Addition of `dev` as a readable key in the `src/stage0.txt` format. If present
then stage0 compilers are downloaded from `dev-static.rust-lang.org` instead
of `static.rust-lang.org`. This is added to accomodate our updated release
process with Travis and AppVeyor.
Shrink the rust-src component
Before this change, the installable rust-src component had essentially the same contents as the rustc-src dist tarball, just additionally wrapped in a rust-installer. As discussed on [internals], rust-src is only meant to support uses for the standard library, so it doesn't really need the rest of the compiler sources.
Now rust-src only contains libstd and its path dependencies, which roughly matches the set of crates that have rust-analysis data. The result is **significantly** smaller, from 36MB to 1.3MB compressed, and from 247MB to 8.5MB uncompressed.
[internals]: https://internals.rust-lang.org/t/minimizing-the-rust-src-component/5117
* Use the right version when building combined installer
* Update dependencies of rls as it depends on rustc and plugins
* Fix build-manifest and the versions it uses for the rls
rustbuild: Make save-analysis an option
This makes save-analysis an option independent from the release channel.
The CI build scripts have been modified to enable the flag.
*Merge with caution.* I haven't tested this, and this can cause nightly breakage.
This causes problems when first cloning and bootstrapping the repository
unfortunately, so let's ensure that Cargo sticks around in its own workspace.
Because Cargo is a submodule it's not available by default on the inital clone
of the rust-lang/rust repository. Normally it's the responsibility of the
rustbuild to take care of this, but unfortunately to build rustbuild itself we
need to resolve the workspace conflicts.
To deal with this we'll just have to ensure that all submodules are in their own
workspace, which sort of makes sense anyway as updates to dependencies as
bugfixes to Cargo should go to rust-lang/cargo instead of rust-lang/rust. In any
case this commit removes Cargo from the global workspace which should resolve
the issues that we've been seeing.
To actually perform this the `cargo` submodule has been moved to the top
directory to ensure it's outside the scope of `src/Cargo.toml` as a workspace.
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
travis: Disable source tarballs on most builders
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
* Add version info to channel.rs as main.mk is no longer available
* Update `Makefile.in` used with bootstrap to not try to require `mk/util.mk`
* Update the `dist` target to avoid the makefile pieces
This commit adds a new tool, `build-manifest`, which is used to generate a
distribution manifest of all produced artifacts. This tool is intended to
replace the `build-rust-manifest.py` script that's currently located on the
buildmaster. The intention is that we'll have a builder which periodically:
* Downloads all artifacts for a commit
* Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files
as well as TOML manifests.
* Upload all generated hashes and manifests to the directory the artifacts came
from.
* Upload *all* artifacts (tarballs and hashes and manifests) to an archived
location.
* If necessary, upload all artifacts to the main location.
This script is intended to just be the second step here where orchestrating
uploads and such will all happen externally from the build system itself.
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
The source tarball creation step would attempt to skip a number of files that we
want to ignore ourselves, but once we've hit the vendor directory we don't want
to skip anything so be sure to vendor everything inside that directory.
Closes#38690
This commit skips a few more dist tragets during compilation which shouldn't be
necessary.
* First, when packaging std we only take action when the host target is the
build target. Otherwise we package the same artifacts a number of times, which
shouldn't be necessary.
* Next, we apply the same logic to the save-analysis build. This is actually
required for correctness as the build compiler is the only one which actually
has save analysis information. This should fix an error seen on nightlies.
Git worktrees have this as a file and typically won't work inside docker
containers, but that's ok, so instead of just checking for existence check for a
directory to see if the git commands will succeed.
This commit implements the `distcheck` target for rustbuild which is only ever
run on our nightly bots. This essentially just creates a tarball, un-tars it,
and then runs a full build, validating that the release tarballs do indeed have
everything they need to build Rust.
As the entry point for building the Rust compiler, a good user experience hinges
on this compiling quickly to get to the meat of the problem. To that end use
`#[cfg]`-specific dependencies to avoid building Windows crates on Unix and drop
the `regex` crate for now which was easily replacable with some string
searching.
The organization in rustbuild was a little odd at the moment where the `lib.rs`
was quite small but the binary `main.rs` was much larger. Unfortunately as well
there was a `build/` directory with the implementation of the build system, but
this directory was ignored by GitHub on the file-search prompt which was a
little annoying.
This commit reorganizes rustbuild slightly where all the library files (the
build system) is located directly inside of `src/bootstrap` and all the binaries
now live in `src/bootstrap/bin` (they're small). Hopefully this should allow
GitHub to index and allow navigating all the files while maintaining a
relatively similar layout to the other libraries in `src/`.
Nick Cameron
Alex Crichton
leonardo.yvens
bors
Mark Simulacrum
Marc-Antoine Perennou
Vadim Petrochenkov
Oliver Schneider
est31
Jonathan A. Kollasch
Simon Sapin
Josh Stone
Ariel Ben-Yehuda
Ralf Jung
Matthew Dawson
Murarth
Marco Castelluccio
Corey Farwell
Edward Yang
Wesley Wiser
Andrew Gaspar
Tatsuyuki Ishi
Shawn Walker-Salas
Eduard-Mihai Burtescu
Wang Xuerui
Ahmed Charles
Joseph Dunne
Diggory Blake