Allow writing metadata without llvm
# Todo:
* [x] Rebase
* [x] Fix eventual errors
* [x] <strike>Find some crate to write elf files</strike> (will do it later)
Cc #43842
* Adjust bootstrap to provide useful output on failure
* Add missing package dependencies in the build environment
* Fix permission bits on prebuilt toolchain files
If config.toml doesn't exist, then an IOError will be raised
on the `with open(...)` line. Prior to e788fa7, this was
caught because the `except` clause didn't specify what
exceptions it caught, so both IOError and OSError were
caught
Run the miri test suite on the aux builder and travis
Reopen of #38350
see https://github.com/rust-lang/rust/pull/43340#issuecomment-316940762 for earlier discussion
Rationale for running miri's test suite in rustc's CI is that miri currently contains many features that we want in const eval in the future, and these features would break if the test suite is not run.
fixes#44077
r? @nikomatsakis
cc @eddyb
rustbuild: Don't pass `-j` if called by `make`
In these situations Cargo just prints out a warning about ignoring the flag
anyway, so let `make` take care of jobs and whatnot instead of getting warnings
printed.
In these situations Cargo just prints out a warning about ignoring the flag
anyway, so let `make` take care of jobs and whatnot instead of getting warnings
printed.
ci: Upload/download from a new S3 bucket
Moving buckets from us-east-1 to us-west-1 because us-west-1 is where
rust-central-station itself runs and in general is where we have all our other
buckets.
rustbuild: Fix test "test rustdoc" invocation
Previously it would use the librustc output directory which would cause rustdoc
to get entirely recompiled, whereas the intention is that it uses the
already-compiled artifacts from building rustdoc itself, using the tool output
directory
bump gcc for bootstrap
On Windows, the gcc crate would send /Wall to msvc, which would cause
builds to get flooded with warnings, exploding compile times from one
hour to more than 72! The gcc crate version 0.3.54 changes this behavior
to send /W4 instead, which greatly cuts down on cl.exe flooding the
command prompt window with warnings.
Moving buckets from us-east-1 to us-west-1 because us-west-1 is where
rust-central-station itself runs and in general is where we have all our other
buckets.
Previously it would use the librustc output directory which would cause rustdoc
to get entirely recompiled, whereas the intention is that it uses the
already-compiled artifacts from building rustdoc itself, using the tool output
directory
Right now we comiple rustdoc in stage 2 and the error index in stage 0, which
ends up compiling rustdoc twice! To avoid compiling rustdoc twice (which takes
awhile) let's just compile it once in stage 2.
Explicitly run perl for OpenSSL Configure
OpenSSL's Configure script is missing a shebang. On some platforms,
execve falls back to execution with the shell. Some other platforms,
like musl, will fail with an exec format error. Avoid this by calling
perl explicitly (since it's a perl script).
On Windows, the gcc crate would send /Wall to msvc, which would cause
builds to get flooded with warnings, exploding compile times from one
hour to more than 72! The gcc crate version 0.3.54 changes this behavior
to send /W4 instead, which greatly cuts down on cl.exe flooding the
command prompt window with warnings.
Apparently `File::create` was called when there was an existing hard link or the
like, causing an existing file to get accidentally truncated!
Closes#44487
rustbuild: Switch back to using hard links
The `copy` function historically in rustbuild used hard links to speed up the
copy operations that it does. This logic was backed out, however, in #39518 due
to a bug that only showed up on Windows, described in #39504. The cause
described in #39504 happened because Cargo, on a fresh build, would overwrite
the previous artifacts with new hard links that Cargo itself manages.
This behavior in Cargo was fixed in rust-lang/cargo#4390 where it no longer
should overwrite files on fresh builds, opportunistically leaving the filesystem
intact and not touching it.
Hopefully this can help speed up local builds by doing fewer copies all over the
place!
The `copy` function historically in rustbuild used hard links to speed up the
copy operations that it does. This logic was backed out, however, in #39518 due
to a bug that only showed up on Windows, described in #39504. The cause
described in #39504 happened because Cargo, on a fresh build, would overwrite
the previous artifacts with new hard links that Cargo itself manages.
This behavior in Cargo was fixed in rust-lang/cargo#4390 where it no longer
should overwrite files on fresh builds, opportunistically leaving the filesystem
intact and not touching it.
Hopefully this can help speed up local builds by doing fewer copies all over the
place!
Include rustc in the default `./x.py install`
The default install used to include rustc, rust-std, and rust-docs, but
the refactoring in commit 6b3413d825 make rustc only default in
extended builds. This commit makes rustc installed by default again.
use gcc::Build rather than deprecated gcc::Config
I did `cargo update -p gcc` to upgrade only this package. Is there further process that should be follwoed when updating a build dependency from crates.io?
r? @alexcrichton
Fixes#43973
The default install used to include rustc, rust-std, and rust-docs, but
the refactoring in commit 6b3413d825 make rustc only default in
extended builds. This commit makes rustc installed by default again.
Doc tests are temporarily disabled until next release cycle, since
current beta Cargo errors on them. Upgrade should be smooth as the
relevant tests are already fixed in this commit.
Update rls
And expose the `CFG_VERSION` env var to tools so they can determine the version of Rust.
This gets the RLS back on master and so completes the PR dance for the generators PR.
r? @alexcrichton
Add clippy as a submodule
~~This builds clippy as part of `./x.py build` (locally and in CI).~~
This allows building clippy with `./x.py build src/tools/clippy`
~~Needs https://github.com/nrc/dev-tools-team/issues/18#issuecomment-322456461 to be resolved before it can be merged.~~ Contributers can simply open a PR to clippy and point the submodule at the `pull/$pr_number/head` branch.
This does **not** build clippy or test the clippy test suite at all as per https://github.com/nrc/dev-tools-team/issues/18#issuecomment-321411418
r? @nrc
cc @Manishearth @llogiq @mcarton @alexcrichton
rustbuild: update the rust-src filter for compiler-rt
We wanted `src/compiler-rt/test` filtered from the `rust-src` package,
but that path is now `src/libcompiler_builtins/compiler-rt/test`. This
saves over half of the installed rust-src size. (50MB -> 22MB)
We wanted `src/compiler-rt/test` filtered from the `rust-src` package,
but that path is now `src/libcompiler_builtins/compiler-rt/test`. This
saves over half of the installed rust-src size. (50MB -> 22MB)
rustbuild: Avoid some extraneous rustc compiles on cross builds
This tweaks a few locations here and there to avoid compiling rustc too many times on our cross-builders on CI.
Closes https://github.com/rust-lang/rust/issues/44132
Fail ./x.py on invalid command
Make the ./x.py script fail when run with an invalid command, like:
```
./x.py nonsense
```
This helps in case of chaining multiple runs, eg.:
```
./x.py biuld && ./x.py test
```
include Cargo.{toml,lock} in rust-src tarball
The lock file is interesting because e.g. xargo could use it to build libstd against the same dependencies that were used for the main build. More generally speaking, just documenting in this form which exact dependencies should be used IMHO makes lots of sense.
I added the Cargo.toml mostly because having the lock without the toml feels odd. Of course, the toml contains references to paths that don't actually exist in the rust-src tarball. Not sure if that is considered a problem.
OpenSSL's Configure script is missing a shebang. On some platforms,
execve falls back to execution with the shell. Some other platforms,
like musl, will fail with an exec format error. Avoid this by calling
perl explicitly (since it's a perl script).
rustbuild: Rewrite the configure script in Python
This commit rewrites our ancient `./configure` script from shell into Python.
The impetus for this change is to remove `config.mk` which is just a vestige of
the old makefile build system at this point. Instead all configuration is now
solely done through `config.toml`.
The python script allows us to more flexibly program (aka we can use loops
easily) and create a `config.toml` which is based off `config.toml.example`.
This way we can preserve comments and munge various values as we see fit.
It is intended that the configure script here is a drop-in replacement for the
previous configure script, no functional change is intended. Also note that the
rationale for this is also because our build system requires Python, so having a
python script a bit earlier shouldn't cause too many problems.
Closes#40730Closes#43295Closes#42255Closes#38058Closes#32176
This commit rewrites our ancient `./configure` script from shell into Python.
The impetus for this change is to remove `config.mk` which is just a vestige of
the old makefile build system at this point. Instead all configuration is now
solely done through `config.toml`.
The python script allows us to more flexibly program (aka we can use loops
easily) and create a `config.toml` which is based off `config.toml.example`.
This way we can preserve comments and munge various values as we see fit.
It is intended that the configure script here is a drop-in replacement for the
previous configure script, no functional change is intended. Also note that the
rationale for this is also because our build system requires Python, so having a
python script a bit earlier shouldn't cause too many problems.
Closes#40730
Make the ./x.py script fail when run with an invalid command, like:
./x.py nonsense
This helps in case of chaining multiple runs, eg.:
./x.py biuld && ./x.py test
Discovered in #43767 it turns out the default MSBuild generator in CMake for
whatever reason isn't supporting many of the configuration options we give to
LLVM. To improve the contributor experience automatically enable Ninja if we
find it to ensure that "flavorful" configurations of LLVM work by default in
more situations.
Closes#43767
This controls the value of the crt-static feature used when building the
standard library for a target, as well as the compiler itself when that
target is the host.
They are required for linking it, even though it is a library, because
crtn.o in post_link_objects, as hardcoded in src/librustc_back/target/
linux_musl_base.rs, is added to the linker command line for both
executables and libraries.
Implement a temp redirect for cargo docs
As discussed in
https://github.com/rust-lang/cargo/issues/4040#issuecomment-321639074
This is a redirect meant to be replaced once cargo docs have been
converted to mdbook. We just want *a* URL to ride the trains for now so
that we can print doc.rust-lang.org/cargo in the paper book and
guarantee that it will go *somewhere* useful by the time the book is
printed.
Implemented as a meta redirect in HTML because we don't currently have
any google juice at doc.rust-lang.org/cargo to lose.
When I run `./x.py doc`, this creates a `build/x86_64-apple-darwin/doc/cargo/index.html` file that contains a meta redirect to doc.crates.io. As I understand rust-central-station to work, this should be what we need to make `doc.rust-lang.org/cargo` to work.
r? @alexcrichton and/or @steveklabnik
As discussed in
https://github.com/rust-lang/cargo/issues/4040#issuecomment-321639074
This is a redirect meant to be replaced once cargo docs have been
converted to mdbook. We just want *a* URL to ride the trains for now so
that we can print doc.rust-lang.org/cargo in the paper book and
guarantee that it will go *somewhere* useful by the time the book is
printed.
Implemented as a meta redirect in HTML because we don't currently have
any google juice at doc.rust-lang.org/cargo to lose.
Cleanup for "Support compiling rustc without LLVM (try 2)"
This includes a small patch to allow running tests without llvm. Also check if you are not trying to compile a dylib.
cc #42932
r? @alexcrichton
This fixes the bug we previously had where we'd build a libtest tool
after building a libstd tool and clear out the libstd tool. Since we
clear out all tools for a given stage on invocations of CleanTools after
lib{std, test, rustc} change, we need to make sure that all tools built
with that stage will be built after the clearing is done.
The fix contained here technically isn't perfect; there is still an edge
case of compiling a libstd tool, then compiling libtest, which will
clear out the libstd tool and it won't ever get rebuilt within that
session of rustbuild. This is where the caching system used today shows
it's problems -- in effect, all tools depend on a global counter of the
stage being cleared out. We can implement such a counter in a future
patch to ensure that tools are rebuilt as needed, but it is deemed
unlikely that it will be required in practice, since most if not all
tools are built after the relevant stage's std/test/rustc are built,
though this is only an opinion and hasn't been verified.
Some users of the build system change the git sha on every build due to
utilizing git to push changes to a remote server. This allows them to
simply configure that away instead of depending on custom patches to
rustbuild.
This introduces a slight change in behavior, where we unilaterally
respect the --host and --target parameters passed for all sanity
checking and runtime configuration.
Link LLVM tools dynamically
Set `LLVM_LINK_LLVM_DYLIB=ON` -- "If enabled, tools will be linked with
the libLLVM shared library." Rust doesn't ship any of the LLVM tools,
and only needs a few at all for some test cases, so statically linking
the tools is just a waste of space. I've also had memory issues on
slower machines with LLVM debuginfo enabled, when several tools start
linking in parallel consuming several GBs each.
With the default configuration, `build/x86_64-unknown-linux-gnu/llvm`
was 1.5GB before, now down to 731MB. The difference is more drastic
with `--enable-llvm-release-debuginfo`, from 28GB to "only" 13GB.
This does not change the linking behavior of `rustc_llvm`.
extend config.toml doc for debug-assertions
Even after I knew that I had to change config.toml to get any printing from debug! and trace!, going over the entire fail did not make it clear to me that `debug-assertions` is the option controlling that.
Set `LLVM_LINK_LLVM_DYLIB=ON` -- "If enabled, tools will be linked with
the libLLVM shared library." Rust doesn't ship any of the LLVM tools,
and only needs a few at all for some test cases, so statically linking
the tools is just a waste of space. I've also had memory issues on
slower machines with LLVM debuginfo enabled, when several tools start
linking in parallel consuming several GBs each.
With the default configuration, `build/x86_64-unknown-linux-gnu/llvm`
was 1.5GB before, now down to 731MB. The difference is more drastic
with `--enable-llvm-release-debuginfo`, from 28GB to "only" 13GB.
This does not change the linking behavior of `rustc_llvm`.
This was intended for bots back in the day where we'd persist caches of LLVM
builds across runs, but nowadays we don't do that on any of the bots so this
option is no longer necessary
rustbuild: Use Cargo's "target runner"
This commit leverages a relatively new feature in Cargo to execute
cross-compiled tests, the `target.$target.runner` configuration. We configure it
through environment variables in rustbuild and this avoids the need for us to
locate and run tests after-the-fact, instead relying on Cargo to do all that
execution for us.
This commit leverages a relatively new feature in Cargo to execute
cross-compiled tests, the `target.$target.runner` configuration. We configure it
through environment variables in rustbuild and this avoids the need for us to
locate and run tests after-the-fact, instead relying on Cargo to do all that
execution for us.
In preparation for upgrading to LLVM 5.0 it looks like we need to tweak how we
cross compile LLVM slightly. It's using `CMAKE_SYSTEM_NAME` to infer whether to
build libFuzzer which only works on some platforms, and then once we configure
that it needs to apparently reach into the host build area to try to compile
`llvm-config` as well. Once these are both configured, though, it looks like we
can successfully cross-compile LLVM.
Add a disabled builder for aarch64 emulated tests
This commit adds a disabled builder which will run all tests for the standard
library for aarch64 in a QEMU instance. Once we get enough capacity to run this
on Travis this can be used to boost our platform coverage of AArch64
bootstrap: Add doctests and unitests
This commit includes the following changes:
* Include more docstrings in classes, methods, and functions
* Add doctests, which are great for self-documenting our source code
* Add some unit tests with the `unittest` module
* Remove `WindowsError` reference on non-windows systems
* Rename some variables to be more explicit about their meaning
* Move all the attributes defined outside of `__init__`
* Add initial support for Python 3
r? @alexcrichton
Bump master to 1.21.0
This commit bumps the master branch's version to 1.21.0 and also updates the
bootstrap compiler from the freshly minted beta release.
This commit includes the following:
* Fix syntax errors in Python 3
* Include more docstrings in classes, methods, and functions
* Include unit tests using `unittest`
* Merge implementation of `{rustc,cargo}_out_of_date`
* Merge implementation of `RustBuild.{cargo,rustc}`
* Remove unnecessary source code
* Move all the attributes defined outside of `__init__`
* Remove remaining `%s` from print function
* Remove `WindowsError` reference on non-windows systems
* Rename some variables to be more explicit avoid their meaning
* Run bootstrap tests in the CI process
* Remove non-pythonic getters
* Remove duplicate code in `download_stage0` method
* Reduce the number of branches in `build_bootstrap` method
* Re-raise exception when we cannot execute uname in non-windows systems
* Avoid long lines
rustbuild: Fix the --build argument to bootstrap.py
This makes the --build argument also apply for the downloading of the stage0 toolchain and building rustbuild.
Fixes#42116
This commit adds a disabled builder which will run all tests for the standard
library for aarch64 in a QEMU instance. Once we get enough capacity to run this
on Travis this can be used to boost our platform coverage of AArch64
Fix stage 2 builds with a custom libdir.
When copying libstd for the stage 2 compiler, the builder ignores the
configured libdir/libdir_relative configuration parameters. This causes
the compiler to fail to find libstd, which cause any tools built with the
stage 2 compiler to fail.
To fix this, make the copy steps of rustbuild aware of the libdir_relative
parameter when the stage >= 2. Also update the dist target to be aware of
the new location of libstd.
