When copying libstd for the stage 2 compiler, the builder ignores the
configured libdir/libdir_relative configuration parameters. This causes
the compiler to fail to find libstd, which cause any tools built with the
stage 2 compiler to fail.
To fix this, make the copy steps of rustbuild aware of the libdir_relative
parameter when the stage >= 2. Also update the dist target to be aware of
the new location of libstd.
Switch to rust-lang-nursery/compiler-builtins
This commit migrates the in-tree `libcompiler_builtins` to the upstream version
at https://github.com/rust-lang-nursery/compiler-builtins. The upstream version
has a number of intrinsics written in Rust and serves as an in-progress rewrite
of compiler-rt into Rust. Additionally it also contains all the existing
intrinsics defined in `libcompiler_builtins` for 128-bit integers.
It's been the intention since the beginning to make this transition but
previously it just lacked the manpower to get done. As this PR likely shows it
wasn't a trivial integration! Some highlight changes are:
* The PR rust-lang-nursery/compiler-builtins#166 contains a number of fixes
across platforms and also some refactorings to make the intrinsics easier to
read. The additional testing added there also fixed a number of integration
issues when pulling the repository into this tree.
* LTO with the compiler-builtins crate was fixed to link in the entire crate
after the LTO process as these intrinsics are excluded from LTO.
* Treatment of hidden symbols was updated as previously the
`#![compiler_builtins]` crate would mark all symbol *imports* as hidden
whereas it was only intended to mark *exports* as hidden.
This commit migrates the in-tree `libcompiler_builtins` to the upstream version
at https://github.com/rust-lang-nursery/compiler-builtins. The upstream version
has a number of intrinsics written in Rust and serves as an in-progress rewrite
of compiler-rt into Rust. Additionally it also contains all the existing
intrinsics defined in `libcompiler_builtins` for 128-bit integers.
It's been the intention since the beginning to make this transition but
previously it just lacked the manpower to get done. As this PR likely shows it
wasn't a trivial integration! Some highlight changes are:
* The PR rust-lang-nursery/compiler-builtins#166 contains a number of fixes
across platforms and also some refactorings to make the intrinsics easier to
read. The additional testing added there also fixed a number of integration
issues when pulling the repository into this tree.
* LTO with the compiler-builtins crate was fixed to link in the entire crate
after the LTO process as these intrinsics are excluded from LTO.
* Treatment of hidden symbols was updated as previously the
`#![compiler_builtins]` crate would mark all symbol *imports* as hidden
whereas it was only intended to mark *exports* as hidden.
bootstrap: Fix all the pep-8 issues reported by flake8
This commit also adds a few missing docstrings.
Today, after reading this [article](https://blog.rust-lang.org/2017/06/27/Increasing-Rusts-Reach.html), I downloaded this project and started building from source. In the meantime, I began to read the `bootstrap.py`, to know more about the building process, and I made a few changes, this is my first contribution to the project, hope you like it.
BTW, I have a few doubts about the `bootstrap.py`, any guidance is more than welcome:
* Where can I find the unit tests for this script? In case it doesn't exist yet, do you like to include some unit tests with pytest?
* Some methods like `fix_executable`, `get_string`, and `exe_suffix` in the `RustBuild` class should be converted to a function because it doesn't use `self` anywhere. What do you think?
Make wasm32 buildbot test LLVM backend
This adds the experimental targets option to configure so it can be used
by the builders and changes the wasm32 Dockerfile accordingly. Instead
of using LLVM from the emsdk, the builder's emscripten tools now uses
the Rust in-tree LLVM, since this is the one built with wasm support.
This adds the experimental targets option to configure so it can be used
by the builders and changes the wasm32 Dockerfile accordingly. Instead
of using LLVM from the emsdk, the builder's emscripten tools now uses
the Rust in-tree LLVM, since this is the one built with wasm support.
Pass path to python from bootstrap.py to bootstrap.rs
When bootstrap is executed with python not in `$PATH`, (e. g.
`c:\Python27\python.exe x.py test`) bootstrap cannot find python
and crashes.
This commit passes path to python in `BOOTSTRAP_PYTHON` env var.
Make rustc errors colorful.
Rustbuild passes --message-format=json to Cargo to learn about the
dependencies for a given build, which then makes Cargo steal the
stderr/stdout for the compiler process, leading to non colorful output.
To avoid this, detection of stderr being a tty is added to rustbuild,
and an environment variable is used to communicate with the rustc shim.
Fixes https://github.com/rust-lang/rust/issues/42801.
r? @alexcrichton
Fixes bootstrapping with custom cargo/rustc.
config.mk is now always read when parsing the configuration to prevent
this from reoccurring in the future, hopefully.
Fixes https://github.com/rust-lang/rust/issues/42543.
r? @alexcrichton
cc @infinity0 @kyrias
Rustbuild passes --message-format=json to Cargo to learn about the
dependencies for a given build, which then makes Cargo steal the
stderr/stdout for the compiler process, leading to non colorful output.
To avoid this, detection of stderr being a tty is added to rustbuild,
and an environment variable is used to communicate with the rustc shim.
When bootstrap is executed with python not in `$PATH`, (e. g.
`c:\Python27\python.exe x.py test`) bootstrap cannot find python
and crashes.
This commit passes path to python in `BOOTSTRAP_PYTHON` env var.
This commit deletes the in-tree `getopts` crate in favor of the crates.io-based
`getopts` crate. The main difference here is with a new builder-style API, but
otherwise everything else remains relatively standard.
Enable wasm LLVM backend
Enables compilation to WebAssembly with the LLVM backend using the target triple "wasm32-unknown-unknown". This is the beginning of my work on #38804.
**edit:** The new new target is now wasm32-experimental-emscripten instead of wasm32-unknown-unknown.
Use custom cargo/rustc paths when parsing flags.
Fixes https://github.com/rust-lang/rust/issues/41779, probably also https://github.com/rust-lang/rust/issues/42543 (I think they're duplicates).
I'm not entirely happy with the implementation, since it means we parse the configuration twice, but it's the minimal solution. I think the other choice is to move both calls to Config::parse inside Flags::parse and merge them, but I don't know if that's a good idea.
r? @alexcrichton
The new target is wasm32-experimental-emscripten. Adds a new
configuration option to opt in to building experimental LLVM backends
such as the WebAssembly backend. The target name was chosen to be
similar to the existing wasm32-unknown-emscripten target so that the
build and tests would work with minimal other code changes. When/if the
new target replaces the old target, simply renaming it should just work.
Autogenerate stubs and SUMMARY.md in the unstable book
Removes a speed bump in compiler development by autogenerating stubs for features in the unstable book. See #42454 for discussion.
The PR contains three commits, separated in order to make review easy:
* The first commit converts the tidy tool from a binary crate to a crate that contains both a library and a binary. In the second commit, we'll use the tidy library
* The second and main commit introduces autogeneration of SUMMARY.md and feature stub files
* The third commit turns off the tidy lint that checks for features without a stub, and removes the stub files. A separate commit due to the large number of files touched
Members of the doc team who wish to document some features can either do this (where `$rustsrc` is the root of the rust repo git checkout):
1. cd to `$rustsrc/src/tools/unstable-book-gen` and then do `cargo run $rustsrc/src $rustsrc/src/doc/unstable-book` to put the stubs into the unstable book
2. cd to `$rustsrc` and run `git ls-files --others --exclude-standard` to list the newly added stubs
3. choose a file to edit, then `git add` it and `git commit`
4. afterwards, remove all changes by the tool by doing `git --reset hard` and `git clean -f`
Or they can do this:
1. remove the comment marker in `src/tools/tidy/src/unstable_book.rs` line 122
2. run `./x.py test src/tools/tidy` to list the unstable features which only have stubs
3. revert the change in 1
3. document one of the chosen unstable features
The changes done by this PR also allow for further development:
* tidy obtains information about tracking issues. We can now forbid differing tracking issues between differing `#![unstable]` annotations. I haven't done this but plan to in a future PR
* we now have a general framework for generating stuff for the unstable book at build time. Further changes can autogenerate a list of the API a given library feature exposes.
The old way to simply click through the documentation after it has been uploaded to rust-lang.org works as well.
r? @nagisa
Fixes#42454
Add a travis builder for wasm32-unknown-emscripten
This commits add an entry to travis matrix that will execute wasm32-unknown-emscripten tests suites.
- Emscripten for asmjs was updated to sdk-1.37.13-64bit
- The tests are run with node 8.0.0 (it can execute wasm)
- A wrapper script is used to run each test from the directory where it is (workaround for https://github.com/kripken/emscripten/issues/4542)
- Some tests are ignore, see #42629 and #42630
Build instruction profiler runtime as part of compiler-rt
r? @alexcrichton
This is #38608 with some fixes.
Still missing:
- [x] testing with profiler enabled on some builders (on which ones? Should I add the option to some of the already existing configurations, or create a new configuration?);
- [x] enabling distribution (on which builders?);
- [x] documentation.
bootstrap: improve 'build --help' by explaining what exactly the last example does
I recently found myself confused about what exactly gets built how often when I run which command; I think this would have helped me.
One thing I did not touch, but I feel could also be improved, is the wording just above: "For a quick build with a usable compile, you can pass". I am not a native speaker, but this sounds odd to me. Do you mean "For a quick build of a usable compiler" (but then it should say where the usable compiler is produced)? Or do you mean "For a quick build testing if the compiler is usable"? I can reword this, but I'd like to make sure I understand the intent of the message.
What about
```
For a quick build of a usable compiler, you can pass:
./x.py build --stage 1 src/libtest
This will first build everything once (like --stage 0 without further
arguments would), and then use the compiler built in stage 0 to build
src/libtest and its dependencies.
Once this is done, build/$ARCH/stage1 contains a usable compiler.
```
However, I am not sure this is actually true. In particular, why even bother building the libstd in stage 1? AFAIK that ends up in `build/*/stage1-std`, not in `build/*/stage1` (which is filled from `build/*/stage0-*`).
rustbuild: Remove RUSTFLAGS logic in rustc shim
This was added in #38072 but I can't recall why and AFAIK Cargo already handles
this. This was discovered through #42146 where passing duplicate flags was
causing problems.
rustbuild: Add `./x.py test --no-fail-fast`
This option forwards to each `cargo test` invocation, and applies the
same logic across all test steps to keep going after failures. At the
end, a brief summary line reports how many commands failed, if any.
Note that if a test program fails to even start at all, or if an
auxiliary build command related to testing fails, these are still left
to stop everything right away.
Fixes#40219.
Reduce verbosity of build logs
This does two separate things.
- Sets sccache logging to warn instead of info.
- Makes tests when running for a given PR (not on auto branch) quiet. (cc @eddyb)
r? @alexcrichton
This was added in #38072 but I can't recall why and AFAIK Cargo already handles
this. This was discovered through #42146 where passing duplicate flags was
causing problems.
This option forwards to each `cargo test` invocation, and applies the
same logic across all test steps to keep going after failures. At the
end, a brief summary line reports how many commands failed, if any.
Note that if a test program fails to even start at all, or if an
auxiliary build command related to testing fails, these are still left
to stop everything right away.
Fixes#40219.
Support VS 2017
Fixes#38584
This replaces all the MSVC linker logic with that from the 'gcc' crate. The code looks the same, but there could be regressions.
I've only tested this with x86_64.
r? @alexcrichton
cc @vadimcn @retep998
When `--quiet` is passed to rustbuild, suppress rustdoc test output unless
failure.
Added a `--quiet` flag to `tidy`, which suppresses the features table.
The actual `--quiet` flag is enabled in #42354.
Since details of failed tests will still be printed, and the name of slow
tests taking >60 to runtime will also be printed, the debugging difficulty
caused by information loss should be minimal; but it is very worthwhile to
keep the log under 10000 lines on Travis CI so that common errors can be
spotted without reading the raw log.
* Bring back colors on Travis, which was disabled since #39036.
Append --color=always to cargo when running in CI environment.
* Removed `set -x` in the shell scripts. The `retry` function already
prints which command it is running, add `-x` just add noise to the
output.
* Support travis_fold/travis_time. Matching pairs of these allow Travis CI
to collapse the output in between. This greatly cut down the unnecessary
"successful" output one need to scroll through before finding the failed
statement.
rustbuild: Fix copying duplicate crates into the sysroot
After compiling a project (e.g. libstd, libtest, or librustc) rustbuild needs to
copy over all artifacts into the sysroot of the compiler it's assembling.
Unfortunately rustbuild doesn't know precisely what files to copy! Today it has
a heuristic where it just looks at the most recent version of all files that
look like rlibs/dylibs and copies those over. This unfortunately leads to bugs
with different versions of the same crate as seen in #42261.
This commit updates rustbuild's strategy of copying artifacts to work off the
list of artifacts produced by `cargo build --message-format=json`. The build
system will now parse json messages coming out of Cargo to watch for files being
generated, and then it'll only copy over those precise files.
Note that there's still a bit of weird logic where Cargo prints that it's
creating `libstd.rlib` where we actually want `libstd-xxxxx.rlib`, so we still
do a bit of "most recent file" probing for those. This commit should take care
of the crates.io dependency issues, however, as they're all copied over
precisely.
Closes#42261
Add the RLS to .exe, .msi, and .pkg installers
This directly addresses issue #42157, adding the RLS as a non-default component in the mentioned installers. The windows installers appear to have the right functionality added, but I don't have a machine that runs OSX, so it would be great if someone could test whether my .pkg commit adds the RLS correctly. The final commit also fixes some formatting issues I'd noticed while working on the installers, but I don't know if that's within the scope of this PR, so input would be appreciated.
After compiling a project (e.g. libstd, libtest, or librustc) rustbuild needs to
copy over all artifacts into the sysroot of the compiler it's assembling.
Unfortunately rustbuild doesn't know precisely what files to copy! Today it has
a heuristic where it just looks at the most recent version of all files that
look like rlibs/dylibs and copies those over. This unfortunately leads to bugs
with different versions of the same craet as seen in #42261.
This commit updates rustbuild's strategy of copying artifacts to work off the
list of artifacts produced by `cargo build --message-format=json`. The build
system will now parse json messages coming out of Cargo to watch for files being
generated, and then it'll only copy over those precise files.
Note that there's still a bit of weird logic where Cargo prints that it's
creating `libstd.rlib` where we actually want `libstd-xxxxx.rlib`, so we still
do a bit of "most recent file" probing for those. This commit should take care
of the crates.io dependency issues, however, as they're all copied over
precisely.
Closes#42261
rustbuild: don't create a source tarball when installing
This splits Install out of Dist as it is not a full dist anymore, and creates the source tarball only for the Dist command.
This will allow splitting install in a few rules if we want as it's done for other phases.
Use the improved submodule handling
r? @alexcrichton
That was a crap...
```
Updating submodules
Traceback (most recent call last):
File "./x.py", line 20, in <module>
bootstrap.main()
File "/home/ishitatsuyuki/Documents/rust/src/bootstrap/bootstrap.py", line 684, in main
bootstrap()
File "/home/ishitatsuyuki/Documents/rust/src/bootstrap/bootstrap.py", line 662, in bootstrap
rb.update_submodules()
File "/home/ishitatsuyuki/Documents/rust/src/bootstrap/bootstrap.py", line 566, in update_submodules
path = line[1:].split(' ')[1]
TypeError: a bytes-like object is required, not 'str'
```
Maybe we need to confirm the compatibility of git options, such as `git config` or `git -C` (I believe they existed long before, though). This is tested locally.
only create source tarball for the Dist subcommand
mark install rule as default for Kind::Install
split install-docs
split install-std
factor out empty_dir handling
split install-cargo
split install-analysis
split install-src
rework install-rustc
properly handle cross-compilation setups for install
use pkgname in install
split plain source tarball generation from rust-src dist
document src-tarball in config.toml.exmaple
Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
