Disable auto-detection of libxml2 when compiling llvm.
This broke cross-compiling rustc with internal llvm (with both the host and target being executable on the machine), because llvm's build system detected libxml2 on the host, therefore auto-enabled libxml2 support, but wouldn't compile as the target didn't have libxml2 installed.
Add src/test/ui regression testing for NLL
This PR changes `x.py test` so that when you are running the `ui` test suite, it will also always run `compiletest` in the new `--compare-mode=nll`, which just double-checks that when running under the experimental NLL mode, the output matches the `<source-name>.nll.stderr` file, if present.
In order to reduce the chance of a developer revolt in response to this change, this PR also includes some changes to make the `--compare-mode=nll` more user-friendly:
1. It now generates nll-specific .stamp files, and uses them (so that repeated runs can reuse previously cached results).
2. Each line of terminal output distinguishes whether we are running under `--compare-mode=nll` by printing with the prefix `[ui (nll)]` instead of just the prefix `[ui]`.
Subtask of rust-lang/rust#48879
Add rustc_trans to x.py check
r? @Mark-Simulacrum
I looked at `bootstrap/compile.rs` and `bootstrap/check.rs` to try to work out which steps were appropriate, but I'm sure I've overlooked some details here, so it's worth checking carefully I've got all the steps right (e.g. I wasn't sure whether we want to build LLVM if necessary with `x.py check`, though I thought it was probably better to than to not).
From a quick test, it seems to be working, though.
Only emit save-analysis data for `cargo build` tasks
Previously, we were emittinng analysis data for all tasks, including `doc`. That meant we got two sets of save-analysis data, one from the normal build and one from the docs. That means indexing with the RLS took twice as long and made downloads larger and build times longer.
cc https://github.com/rust-lang-nursery/rls/issues/826
r? @Mark-Simulacrum
Previously, we were emittinng analysis data for all tasks, including `doc`. That meant we got two sets of save-analysis data, one from the normal build and one from the docs. That means indexing with the RLS took twice as long and made downloads larger and build times longer.
cc https://github.com/rust-lang-nursery/rls/issues/826
rustbuild: allow building tools with debuginfo
Debugging information for the extended tools is currently disabled for
concerns about the size. This patch adds `--enable-debuginfo-tools` to
let one opt into having that debuginfo.
This is useful for debugging the tools in distro packages. We always
strip debuginfo into separate packages anyway, so the extra size is not
a concern in regular use.
Add docs for the test crate with the std docs
If the compiler docs aren't going to include the test crate then it may as well be included with std.
Fixes#49388
Debugging information for the extended tools is currently disabled for
concerns about the size. This patch adds `--enable-debuginfo-tools` to
let one opt into having that debuginfo.
This is useful for debugging the tools in distro packages. We always
strip debuginfo into separate packages anyway, so the extra size is not
a concern in regular use.
rustbuild: canonicalize prefix `install_sh`
Testing:
```
$ git diff
diff --git a/config.toml.example b/config.toml.example
index 9dd3002506..b47bc490cd 100644
--- a/config.toml.example
+++ b/config.toml.example
@@ -196,7 +196,7 @@
[install]
# Instead of installing to /usr/local, install to this path instead.
-#prefix = "/usr/local"
+prefix = "install-prefix"
# Where to install system configuration files
# If this is a relative path, it will get installed in `prefix` above
$ mkdir install-prefix
$ ./x.py install -i --stage 0 --config config.toml.example
...
$ ls install-prefix/
bin lib share
```
Closes#36989.
r? @Mark-Simulacrum
Testing:
```
$ git diff
diff --git a/config.toml.example b/config.toml.example
index 9dd3002506..b47bc490cd 100644
--- a/config.toml.example
+++ b/config.toml.example
@@ -196,7 +196,7 @@
[install]
# Instead of installing to /usr/local, install to this path instead.
-#prefix = "/usr/local"
+prefix = "install-prefix"
# Where to install system configuration files
# If this is a relative path, it will get installed in `prefix` above
$ mkdir install-prefix
$ ./x.py install -i --stage 0 --config config.toml.example
...
$ ls install-prefix/
bin lib share
```
Closes#36989.
configure.py --tools should set a list instead of a string
Currently the --tools option does not work because it is setting a string value
for 'build.tools'. It should be a list of strings instead.
Host compiler documentation: Include private items
Fixes#29893. Now that compiler documentation is being hosted, including private items seems sensible as these types are going to be being used by contributors working on the compiler.
However, including this means that doc comments that contain codeblocks with invalid Rust and can fail the documenting of a given crate (as evidenced by the changes in the second commit included in this PR). We'd need some way of ensuring that this cannot happen so that these failures don't cause documenting to fail. I'm unsure whether this change to documentation steps will cause this to happen already or if something new will be required.
r? @alexcrichton
add a dist builder to build rust-std components for the THUMB targets
the rust-std component only contains the core and compiler-builtins (+c +mem) crates
cc #49382
- I'm not entirely sure if this PR alone will produce rust-std components installable by rustup or if something else needs to be changed
- I could have done the THUMB builds in an existing builder / image; I wasn't sure if that was a good idea so I added a new image
- I could build other crates like alloc into the rust-std component but, AFAICT, that would require calling Cargo a second time (one for alloc and one for compiler-builtins), or have alloc depend on compiler-builtins (#49503 will perform that change) *and* have alloc resurface the "c" and "mem" Cargo features.
r? @alexcrichton
the goal is to build, in a single Cargo invocation, several no-std crates that we want to put in the
rust-std component of no-std targets. The nostd crate builds these crates:
- core
- compiler-builtin (with the "c" and "mem" features enabled)
- alloc
- std_unicode
This is too likely to cause spurious bounces on CI; what we run may be
dependent on what ran successfully before hand (e.g. RLS features with
Clippy), which makes this not tenable. There's no good way to ignore
specifically these problematic steps so we'll just ignore everything for
the time being. We still test that a dry run worked though so largely
this is the same from a ensure-that-tests-work perspective.
Eventually we'll want to undo this commit, though, to make our tests
more accurate.
This ensures that each build will support the testing design of "dry
running" builds. It's also checked that a dry run build is equivalent
step-wise to a "wet" run build; the graphs we generate when running are
directly compared node/node and edge/edge, both for order and contents.
In order to run tests, previous commits have cfg'd out various parts of
rustbuild. Generally speaking, these are filesystem-related operations
and process-spawning related parts. Then, rustbuild is run "as normal"
and the various steps that where run are retrieved from the cache and
checked against the expected results.
Note that this means that the current implementation primarily tests
"what" we build, but doesn't actually test that what we build *will*
build. In other words, it doesn't do any form of dependency verification
for any crate. This is possible to implement, but is considered future
work.
This implementation strives to cfg out as little code as possible; it
also does not currently test anywhere near all of rustbuild. The current
tests are also not checked for "correctness," rather, they simply
represent what we do as of this commit, which may be wrong.
Test cases are drawn from the old implementation of rustbuild, though
the expected results may vary.
This ensures that the working directory of rustbuild has no effect on
it's run; since tests will run with a different cwd this is required for
consistent behavior.
Also update some `Cargo.lock` dependencies, finishing up some final steps of our
[release process]!
This doesn't update the bootstrap compiler just yet but that will come in a
follow-up PR.
[release process]: https://forge.rust-lang.org/release-process.html
Only include space in RUSTFLAGS extra flags if not empty
When the RUSTFLAGS_STAGE_{1,2} is not set, including a space means
the string will always be non-empty and RUSTFLAGS will be always be
reset which breaks other ways of setting these such as through config
in CARGO_HOME.
When the RUSTFLAGS_STAGE_{1,2} is not set, including a space means
the string will always be non-empty and RUSTFLAGS will be always be
reset which breaks other ways of setting these such as through config
in CARGO_HOME.
This option was introduced in 72cb109bec, but it uses two different
spellings (fast-submodule vs fast-submodules) and isn't handled by
Rust bootstrap which means that any attempt to set this flag fails.
This fixes building the compiler docs because stage1-rustc\x86_64-pc-windows-msvc\doc is used twice which
doesn't work if we still have a handle from the first time.
This commit updates the `ToolBuild` step to stream Cargo's JSON messages, parse
them, and record all libraries built. If we build anything twice (aka Cargo)
it'll most likely happen due to dependencies being recompiled which is caught by
this check.
This commit disables building documentation on cross-compiled compilers, for
example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out
of these documentation artifacts and they often take 10-15 minutes total to
build as it requires building rustdoc/rustbook and then also generating all the
documentation, especially for the reference and the book itself.
In an effort to cut down on the amount of work that we're doing on dist CI
builders in light of recent timeouts this was some relatively low hanging fruit
to cut which in theory won't have much impact on the ecosystem in the hopes that
the documentation isn't used too heavily anyway.
While initial analysis in #48827 showed only shaving 5 minutes off local builds
the same 5 minute conclusion was drawn from #48826 which ended up having nearly
a half-hour impact on the bots. In that sense I'm hoping that we can land this
and test out what happens on CI to see how it affects timing.
Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to
generate documentation.
appveyor: Move run-pass-fulldeps to extra builders
We've made headway towards splitting the test suite across two appveyor builders
and this moves one more tests suite between builders. The last [failed
build][fail] had its longest running test suite and I've moved that to the
secondary builder.
cc #48844
[fail]: https://ci.appveyor.com/project/rust-lang/rust/build/1.0.6782
We've made headway towards splitting the test suite across two appveyor builders
and this moves one more tests suite between builders. The last [failed
build][fail] had its longest running test suite and I've moved that to the
secondary builder.
cc #48844
[fail]: https://ci.appveyor.com/project/rust-lang/rust/build/1.0.6782
Allow installing rustfmt without config.extended
This assertion was preventing `./x.py install rustfmt` if attempted
without an "extended" build configuration, but it actually builds and
installs just fine.
rustbuild: Tweak where timing information goes
This commit tweaks where timing and step information is printed out as part of
the build, ensuring that we do it as close to the location where work happens as
possible. In rustbuild various functions may perform long blocking work as
dependencies are assembled, so if we print out timing information early on we
may accidentally time more than just the step we were intending to time!
Use the same RUSTFLAGS for building and testing `bootstrap`
This avoids recompiling the whole dependency graph twice for every `./x.py test` run.
Fixes#49215
This assertion was preventing `./x.py install rustfmt` if attempted
without an "extended" build configuration, but it actually builds and
installs just fine.
This commit is an implementation of adding custom sections to wasm artifacts in
rustc. The intention here is to expose the ability of the wasm binary format to
contain custom sections with arbitrary user-defined data. Currently neither our
version of LLVM nor LLD supports this so the implementation is currently custom
to rustc itself.
The implementation here is to attach a `#[wasm_custom_section = "foo"]`
attribute to any `const` which has a type like `[u8; N]`. Other types of
constants aren't supported yet but may be added one day! This should hopefully
be enough to get off the ground with *some* custom section support.
The current semantics are that any constant tagged with `#[wasm_custom_section]`
section will be *appended* to the corresponding section in the final output wasm
artifact (and this affects dependencies linked in as well, not just the final
crate). This means that whatever is interpreting the contents must be able to
interpret binary-concatenated sections (or each constant needs to be in its own
custom section).
To test this change the existing `run-make` test suite was moved to a
`run-make-fulldeps` folder and a new `run-make` test suite was added which
applies to all targets by default. This test suite currently only has one test
which only runs for the wasm target (using a node.js script to use `WebAssembly`
in JS to parse the wasm output).
ci: Print out how long each step takes on CI
This commit updates CI configuration to inform rustbuild that it should print
out how long each step takes on CI. This'll hopefully allow us to track the
duration of steps over time and follow regressions a bit more closesly (as well
as have closer analysis of differences between two builds).
cc #48829
This commit updates CI configuration to inform rustbuild that it should print
out how long each step takes on CI. This'll hopefully allow us to track the
duration of steps over time and follow regressions a bit more closesly (as well
as have closer analysis of differences between two builds).
cc #48829
This commit tweaks where timing and step information is printed out as part of
the build, ensuring that we do it as close to the location where work happens as
possible. In rustbuild various functions may perform long blocking work as
dependencies are assembled, so if we print out timing information early on we
may accidentally time more than just the step we were intending to time!
Faster submodule updating
For the common case when there are no submodules which need updating, this takes 0.48 seconds instead of 47 seconds.
r? @alexcrichton
Support extra-verbose builds
- The bootstrap crate currently passes -v to Cargo if itself invoked with -vv. But Cargo supports -vv (to show build script output), so make bootstrap pass that if itself invoked with -vvv. (More specifically, pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when building bootstrap, but incorrectly ('-v' is marked as 'store_true', so argparse stores either False or True, ignoring multiple '-v's). Fix this, allow passing any number of '-v's, and make it consistent with bootstrap's invocation of Cargo (i.e. subtract one from the number of '-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary verbosity levels, + allow command line to override it.
rustc: Enable embedding LLVM bitcode for iOS
This commit updates rustc to embed bitcode in each object file generated by
default when compiling for iOS. This was determined in #35968 as a step
towards better compatibility with the iOS toolchain, so let's give it a spin and
see how it turns out!
Note that this also updates the `cc` dependency which should propagate this
change of embedding bitcode for C dependencies as well.
- The bootstrap crate currently passes -v to Cargo if itself invoked
with -vv. But Cargo supports -vv (to show build script output), so make
bootstrap pass that if itself invoked with -vvv. (More specifically,
pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when
building bootstrap, but incorrectly ('-v' is marked as 'store_true', so
argparse stores either False or True, ignoring multiple '-v's). Fix
this, allow passing any number of '-v's, and make it consistent with
bootstrap's invocation of Cargo (i.e. subtract one from the number of
'-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary
verbosity levels, + allow command line to override it.
rustbuild: Remove ThinLTO-related configuration
This commit removes some ThinLTO/codegen unit cruft primarily only needed during
the initial phase where we were adding ThinLTO support to rustc itself. The
current bootstrap compiler knows about ThinLTO and has it enabled by default for
multi-CGU builds which are also enabled by default. One CGU builds (aka
disabling ThinLTO) can be achieved by configuring the number of codegen units to
1 for a particular builds.
This also changes the defaults for our dist builders to go back to multiple
CGUs. Unfortunately we're seriously bleeding for cycle time on the bots right
now so we need to recover any time we can.
* Pass `opt_level(2)` when calculating CFLAGS to get the right flags on iOS
* Unconditionally pass `-O2` when compiling libbacktrace
This should...
Close#48903Close#48906
Required moving all fulldeps tests depending on `rand` to different locations as
now there's multiple `rand` crates that can't be implicitly linked against.
Remove ONLY_BUILD and ONLY_BUILD_TARGETS
Primarily removes `ONLY_BUILD` and `ONLY_BUILD_TARGETS`. These aren't actually needed in the new system since we can simply not take the relevant `host` and `target` fields if we don't want to run with them in `Step::make_run`.
This PR also includes a few other commits which generally clean up the state of rustbuild, but are not related to the `Step` changes.
We explicitly do this when compiling OpenSSL itself due to weird racy issues in
its build system, and now we've started seeing issues in the `make install` step
so let's try and see what ratcheting down the parallelism does here...
travis: Upgrade dist builders for OSX
This commit upgrades the dist builders for OSX to Travis's new `xcode9.3-moar`
image which has 3 cores available to it instead of 2. This should help us
provide speedier builds on OSX and hit timeouts less in theory!
Note that historically the dist builders for OSX have been a different version
than the ones that are running tests. I had forgotten why this was the case and
digging around brought up 307615567 where apparently Xcode 8 wasn't able to
compile LLVM with `MACOSX_DEPLOYMENT_TARGET=10.7` which we desired. On a whim I
gave this PR a spin and it [looks like][green] this has since been fixed (maybe
in LLVM?). In any case those green builds should hopefully mean that we can
safely upgrade and get faster infrastructure to boot.
This commit also includes an upgrade of OpenSSL. This is not done for security
reasons but rather build system reasons. Originally builds with the new image
[did not succeed][red] due to weird build failures in OpenSSL, but upgrading
seems to have made the spurious errors go away to here's to also hoping that's
fixed!
[green]: https://travis-ci.org/rust-lang/rust/builds/351353412
[red]: https://travis-ci.org/rust-lang/rust/builds/350969248
This commit removes some ThinLTO/codegen unit cruft primarily only needed during
the initial phase where we were adding ThinLTO support to rustc itself. The
current bootstrap compiler knows about ThinLTO and has it enabled by default for
multi-CGU builds which are also enabled by default. One CGU builds (aka
disabling ThinLTO) can be achieved by configuring the number of codegen units to
1 for a particular builds.
This also changes the defaults for our dist builders to go back to multiple
CGUs. Unfortunately we're seriously bleeding for cycle time on the bots right
now so we need to recover any time we can.
This commit upgrades the dist builders for OSX to Travis's new `xcode9.3-moar`
image which has 3 cores available to it instead of 2. This should help us
provide speedier builds on OSX and hit timeouts less in theory!
Note that historically the dist builders for OSX have been a different version
than the ones that are running tests. I had forgotten why this was the case and
digging around brought up 307615567 where apparently Xcode 8 wasn't able to
compile LLVM with `MACOSX_DEPLOYMENT_TARGET=10.7` which we desired. On a whim I
gave this PR a spin and it [looks like][green] this has since been fixed (maybe
in LLVM?). In any case those green builds should hopefully mean that we can
safely upgrade and get faster infrastructure to boot.
This commit also includes an upgrade of OpenSSL. This is not done for security
reasons but rather build system reasons. Originally builds with the new image
[did not succeed][red] due to weird build failures in OpenSSL, but upgrading
seems to have made the spurious errors go away to here's to also hoping that's
fixed!
[green]: https://travis-ci.org/rust-lang/rust/builds/351353412
[red]: https://travis-ci.org/rust-lang/rust/builds/350969248
For LLD integration the path to `llvm-config` needed to change to inside the
build directory itself (for whatever reason) but the build directory is
different on MSBuild than it is on `ninja` for MSVC builds, so the path to
`llvm-config.exe` was actually wrong and not working!
This commit removes the `Build::llvm_config` function in favor of the source of
truth, the `Llvm` build step itself. The build step was then updated to find the
right build directory for MSBuild as well as `ninja` for where `llvm-config.exe`
is located.
Closes#48749
rustbuild: Pass `ccache` to build scripts
This is a re-attempt at #48192 hopefully this time with 100% less randomly
[blocking builds for 20 minutes][block]. To work around #48192 the sccache
server is started in the `run.sh` script very early on in the compilation
process.
[block]: https://github.com/rust-lang/rust/issues/48192
This commit refactors how the path to the linker that we're going to invoke is
selected. Previously all targets listed *both* a `LinkerFlavor` and a `linker`
(path) option, but this meant that whenever you changed one you had to change
the other. The purpose of this commit is to avoid coupling these where possible.
Target specifications now only unconditionally define the *flavor* of the linker
that they're using by default. If not otherwise specified each flavor now
implies a particular default linker to run. As a result, this means that if
you'd like to test out `ld` for example you should be able to do:
rustc -Z linker-flavor=ld foo.rs
whereas previously you had to do
rustc -Z linker-flavor=ld -C linker=ld foo.rs
This will hopefully make it a bit easier to tinker around with variants that
should otherwise be well known to work, for example with LLD, `ld` on OSX, etc.
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
Don't produce TOCs for doc markdown files
Currently, we are producing headers for markdown files,
which is generally not what we want. As such, passing this
flag causes them to render normally.
https://doc.rust-lang.org/nightly/book/ is an example page currently where this is done incorrectly.
Restore the download of rust-mingw
The build might otherwise break due to mixing MinGW object files from rust-std and the local MinGW which might be newer/older than the version used to build rust-std.
Fixes#48272
r? @alexcrichton
Improve --help performance for x.py
Since compiling the bootstrap command doesn't require any submodules,
we can skip updating submodules when a --help command is passed in.
On my machine, this saves 1 minute if the submodules are already
downloaded, and 10 minutes if run on a clean repo.
This commit also adds a message before compiling/downloading anything
when a --help command is passed in, to tell the user WHY --help
takes so long to complete. It also points the user to the bootstrap
README.md for faster help.
Finally, this fixes one warning message that still referenced using
make instead of x.py, even though x.py is now the standard way of
building rust.
Closes#37305
Currently, we are producing headers for markdown files,
which is generally not what we want. As such, passing this
flag causes them to render normally.
It was an existing solution to tell the user why a --help command
takes a long time to process. However, it would only print if the
stage0 rust compiler needed to be downloaded, it came after
update_submodules (which took a long time), and it was immediately
followed by download messages and loading bars, meaning users could
easily gloss over the message.
This commit also moves the help message out of main(), and instead
puts it at the top of bootstrap(). main() is intended to be minimal,
only handling error messages.
This is a re-attempt at #48192 hopefully this time with 100% less randomly
[blocking builds for 20 minutes][block]. To work around #48192 the sccache
server is started in the `run.sh` script very early on in the compilation
process.
[block]: https://github.com/rust-lang/rust/issues/48192
The build might otherwise break due to mixing MinGW object files from
rust-std and the local MinGW which might be newer/older than the version
used to build rust-std.
Fixes#48272
Fix error-format to properly send JSON to stdout
Since we take Cargo's JSON messages as well we need to specifically send
rustc's messages out so we don't hide them.
r? @Manishearth
Since compiling the bootstrap command doesn't require any submodules,
we can skip updating submodules when a --help command is passed in.
On my machine, this saves 1 minute if the submodules are already
downloaded, and 10 minutes if run on a clean repo.
This commit also adds a message before compiling/downloading anything
when a --help command is passed in, to tell the user WHY --help
takes so long to complete. It also points the user to the bootstrap
README.md for faster help.
Finally, this fixes one warning message that still referenced using
make instead of x.py, even though x.py is now the standard way of
building rust.
rustbuild: Restore Config.libdir_relative
This re-introduces a `Config.libdir_relative` field, now derived from
`libdir` and made relative to `prefix` if necessary.
This fixes a regression from #46592 when `--libdir` is given an absolute
path. `Builder::sysroot_libdir` should always use a relative path so
its callers don't clobber system locations, and `librustc` also asserts
that `CFG_LIBDIR_RELATIVE` is really relative.
Add missing pieces for sparc-linux-gnu support
I noticed that while Rust has CABI support for 32-bit SPARC, there are still some pieces missing to be able to use Rust on a 32-Bit SPARC system like Gentoo which still defaults to a 32-bit port unlike Debian's sparc64 port.
This PR is an attempt to add the missing pieces. I will send the necessary changes for libc in a separate PR.
CC @jrtc27
This re-introduces a `Config.libdir_relative` field, now derived from
`libdir` and made relative to `prefix` if necessary.
This fixes a regression from #46592 when `--libdir` is given an absolute
path. `Builder::sysroot_libdir` should always use a relative path so
its callers don't clobber system locations, and `librustc` also asserts
that `CFG_LIBDIR_RELATIVE` is really relative.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?
-----
So, timeline for those who need to catch up:
* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](https://github.com/rust-lang/rust/issues/38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](https://github.com/rust-lang/rust/pull/40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](https://github.com/rust-lang/rust/issues/40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](https://github.com/rust-lang/rust/pull/41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](https://github.com/rust-lang/rust/pull/41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](https://github.com/rust-lang/rust/pull/41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
* In that time, [bootstrap was completely reworked](https://github.com/rust-lang/rust/pull/43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](https://github.com/rust-lang/rust/pull/43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
* The warnings were not perfect, and revealed a few [spurious](https://github.com/rust-lang/rust/pull/44368) [differences](https://github.com/rust-lang/rust/pull/45421) between how we handled the renderers.
* Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](https://github.com/rust-lang/rust/pull/45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
* Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](https://github.com/rust-lang/rust/pull/47398), making Pulldown the default but still offering the option to use Hoedown.
And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
We'd previously assumed that these paths would be relative to the src
dir, and that for example our various CI scripts would, when calling
x.py, use `../x.py build ../src/tools/...` but this isn't the case --
they use `../x.py` without using the relevant source-relative path.
We eventually may want to make this (actually somewhat logical) change,
but this is not that time.
Primarily for CI purposes; this is intended to avoid cases where we
update rustbuild and unintentionally make CI stop running some builds to
the arguments being passed no longer applying for some reason.
Implement excluding a build-step via --exclude
First step to fixing https://github.com/rust-lang/rust/issues/47911. This doesn't change any CI configuration, but implements what I believe necessary to make that feasible in rustbuild.
In theory this should be sufficient to allow someone to open a PR against .travis.yml and appveyor.yml which splits the Windows 32-bit tests and maybe the OS X tests into multiple builders (depending on what our cost-concerns are) to reduce runtimes.
r? @alexcrichton
cc @kennytm
Work around LLVM OCAML binding installation failure
Hello,
I have OCaml installed on my machine, and compiling rust systematically fails when LLVM attempts installing the OCaml bindings in `/usr/lib/ocaml`, which is write-protected. Here are the logs: https://gist.github.com/roblabla/3f147914c5df627c9d97ab311ba133ad
Some digging around the issue reveals:
- The code that finds if OCaml is installed, and sets the bindings to be compiled/installed: b24a45d2e9/cmake/config-ix.cmake (L612)
- b24a45d2e9/bindings/ocaml/llvm/CMakeLists.txt Some code that does the installation.
The problem seems to be that `LLVM_OCAML_INSTALL_PATH` is set to `OCAML_STDLIB_PATH` by default, which is in `/usr/lib/ocaml`, instead of the prefix.
This PR "fixes" the issue by setting `LLVM_OCAML_INSTALL_PATH` to `usr/lib/ocaml`. I haven't found a way to make LLVM not build OCaml, which would probably be a superior fix.
Some Steps are by-default run but don't have any paths associated with
them. We need to have at least one PathSet per each Step, though, so we
add an empty one on calls to `never()`.
Previously, a Step would be able to tell on its own when it was invoked
"by-default" (that is, `./x.py test` was called instead of `./x.py test
some/path`). This commit replaces that functionality, invoking each Step
with each of the paths it has specified as "should be invoked by."
For example, if a step calls `path("src/tools/cargo")` and
`path("src/doc/cargo")` then it's make_run will be called twice, with
"src/tools/cargo" and "src/doc/cargo." This makes it so that default
handling logic is in builder, instead of spread across various Steps.
However, this meant that some Step specifications needed to be updated,
since for example `rustdoc` can be built by `./x.py build
src/librustdoc` or `./x.py build src/tools/rustdoc`. A `PathSet`
abstraction is added that handles this: now, each Step can not only list
`path(...)` but also `paths(&[a, b, ...])` which will make it so that we
don't invoke it with each of the individual paths, instead invoking it
with the first path in the list (though this shouldn't be depended on).
Future work likely consists of implementing a better/easier way for a
given Step to work with "any" crate in-tree, especially those that want
to run tests, build, or check crates in the std, test, or rustc crate
trees. Currently this is rather painful to do as most of the logic is
duplicated across should_run and make_run. It seems likely this can be
abstracted away into builder somehow.
ci: Actually bootstrap on i686 dist
Right now the `--build` option was accidentally omitted, so we're bootstraping
from `x86_64` to `i686`. In addition to being slower (more compiles) that's not
actually bootstrapping!
rustbuild: Pass `ccache` to build scripts
Right now the ccache setting is only used for LLVM, but this tweaks it to also
be used for build scripts so C++ builds like `librustc_llvm` can be a bit
speedier.
Customizable extended tools
This PR adds `build.tools` option to manage installation of extended rust tools.
By default it doesn't change installation. All tools are built and `rls` and `rustfmt` allowed to fail installation.
If some set of tools chosen only those tools are built and installed without any fails allowed.
It solves some slotting issues with extended build enabled: https://bugs.gentoo.org/show_bug.cgi?id=645498
The following submodules have been updated for a new version of LLVM:
- `src/llvm`
- `src/libcompiler_builtins` - transitively contains compiler-rt
- `src/dlmalloc`
This also updates the docker container for dist-i686-freebsd as the old 16.04
container is no longer capable of building LLVM. The
compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without
much interesting happening, but the LLVM update here is of particular note.
Unlike previous updates I haven't cherry-picked all existing patches we had on
top of our LLVM branch as we have a [huge amount][patches4] and have at this
point forgotten what most of them are for. Instead I started from the current
`release_60` branch in LLVM and only applied patches that were necessary to get
our tests working and building.
The current set of custom rustc-specific patches included in this LLVM update are:
* rust-lang/llvm@1187443 - this is how we actually implement
`cfg(target_feature)` for now and continues to not be upstreamed. While a
hazard for SIMD stabilization this commit is otherwise keeping the status
quo of a small rustc-specific feature.
* rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't
upstreamed, notably teaching LLVM about our allocation-related routines (which
aren't malloc/free). Once we stabilize the global allocator routines we will
likely want to upstream this patch, but for now it seems reasonable to keep it
on our fork.
* rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in
our 32-bit linux container. I'm not really sure why it's necessary but my
guess is that it's because of the absolutely ancient glibc that we're using.
In any case it's only updating pieces we're not actually using in LLVM so I'm
hoping it'll turn out alright. This doesn't seem like something we'll want to
upstream.c
* rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our
i686-freebsd container, I'm not really sure what's going on but we for sure
probably don't want to upstream this and otherwise it seems not too bad for
now at least.
* rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug]
which although diagnosed to a particular revision isn't currently fixed
upstream (and the bug itself doesn't seem too active). This commit is a
partial revert of the suspected cause of this regression (found via a
bisection). I'm sort of hoping that this eventually gets fixed upstream with a
similar fix (which we can replace in our branch), but for now I'm also hoping
it's a relatively harmless change to have.
After applying these patches (plus one [backport] which should be [backported
upstream][llvm-back]) I believe we should have all tests working on all
platforms in our current test suite. I'm like 99% sure that we'll need some more
backports as issues are reported for LLVM 6 when this propagates through
nightlies, but that's sort of just par for the course nowadays!
In any case though some extra scrutiny of the patches here would definitely be
welcome, along with scrutiny of the "missing patches" like a [change to pass
manager order](rust-lang/llvm@2717444753), [another change to pass manager
order](rust-lang/llvm@c782febb7b), some [compile fixes for
sparc](rust-lang/llvm@1a83de63c4), and some [fixes for
solaris](rust-lang/llvm@c2bfe0abb).
[patches4]: https://github.com/rust-lang/llvm/compare/5401fdf23...rust-llvm-release-4-0-1
[backport]: https://github.com/rust-lang/llvm/commit/5c54c252db
[llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114
[upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096
---
The update to LLVM 6 is desirable for a number of reasons, notably:
* This'll allow us to keep up with the upstream wasm backend, picking up new
features as they start landing.
* Upstream LLVM has fixed a number of SIMD-related compilation errors,
especially around AVX-512 and such.
* There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed
in the LLVM 4 branch we're using.
* Overall it's not a great idea to stagnate with our codegen backend!
This update is mostly powered by #47730 which is allowing us to update LLVM
*independent* of the version of LLVM that Emscripten is locked to. This means
that when compiling code for Emscripten we'll still be using the old LLVM 4
backend, but when compiling code for any other target we'll be using the new
LLVM 6 target. Once Emscripten updates we may no longer need this distinction,
but we're not sure when that will happen!
Closes#43370Closes#43418Closes#47015Closes#47683Closesrust-lang-nursery/stdsimd#157Closesrust-lang-nursery/rust-wasm#3
Right now the `--build` option was accidentally omitted, so we're bootstraping
from `x86_64` to `i686`. In addition to being slower (more compiles) that's not
actually bootstrapping!
Right now the ccache setting is only used for LLVM, but this tweaks it to also
be used for build scripts so C++ builds like `librustc_llvm` can be a bit
speedier.
Dist builds should always be as fast as we can make them, and since
those run on CI we don't care quite as much for the build being somewhat
slower. As such, we don't automatically enable ThinLTO on builds for the
dist builders.
Implement extensible syscall interface for wasm
Currently it's possible to run tests with the native wasm target, but it's not possible to tell whether they pass or to capture the output, because libstd throws away stdout, stderr and the exit code. While advanced libstd features should probably require more specific targets (eg. wasm-unknown-web) I think even the unknown target should at least support basic I/O.
Any solution is constrained by these factors:
- It must not be javascript specific
- There must not be too strong coupling between libstd and the host environment (because it's an "unknown" target)
- WebAssembly does not allow "optional" imports - all imports *must* be resolved.
- WebAssembly does not support calling the host environment through any channel *other* than imports.
The best solution I could find to these constraints was to give libstd a single required import, and implement a syscall-style interface through that import. Each syscall is designed such that a no-op implementation gives the most reasonable fallback behaviour. This means that the following import table would be perfectly valid:
```javascript
imports.env = { rust_wasm_syscall: function(index, data) {} }
```
Currently I have implemented these system calls:
- Read from stdin
- Write to stdout/stderr
- Set the exit code
- Get command line arguments
- Get environment variable
- Set environment variable
- Get time
It need not be extended beyond this set if being able to run tests for this target is the only goal.
edit:
As part of this PR I had to make a further change. Previously, the rust entry point would be automatically called when the webassembly module was instantiated. This was problematic because from the javascript side it was impossible to call exported functions, access program memory or get a reference to the instance.
To solve this, ~I changed the default behaviour to not automatically call the entry point, and added a crate-level attribute to regain the old behaviour. (`#![wasm_auto_run]`)~ I disabled this behaviour when building tests.
Right now this directory is located under:
$sysroot/lib/rustlib/$target/lib/codegen-backends
but after seeing what we do in a few other places it seems that a more
appropriate location would be:
$sysroot/lib/rustlib/$target/codegen-backends
so this commit moves it!
This commit introduces a separately compiled backend for Emscripten, avoiding
compiling the `JSBackend` target in the main LLVM codegen backend. This builds
on the foundation provided by #47671 to create a new codegen backend dedicated
solely to Emscripten, removing the `JSBackend` of the main codegen backend in
the process.
A new field was added to each target for this commit which specifies the backend
to use for translation, the default being `llvm` which is the main backend that
we use. The Emscripten targets specify an `emscripten` backend instead of the
main `llvm` one.
There's a whole bunch of consequences of this change, but I'll try to enumerate
them here:
* A *second* LLVM submodule was added in this commit. The main LLVM submodule
will soon start to drift from the Emscripten submodule, but currently they're
both at the same revision.
* Logic was added to rustbuild to *not* build the Emscripten backend by default.
This is gated behind a `--enable-emscripten` flag to the configure script. By
default users should neither check out the emscripten submodule nor compile
it.
* The `init_repo.sh` script was updated to fetch the Emscripten submodule from
GitHub the same way we do the main LLVM submodule (a tarball fetch).
* The Emscripten backend, turned off by default, is still turned on for a number
of targets on CI. We'll only be shipping an Emscripten backend with Tier 1
platforms, though. All cross-compiled platforms will not be receiving an
Emscripten backend yet.
This commit means that when you download the `rustc` package in Rustup for Tier
1 platforms you'll be receiving two trans backends, one for Emscripten and one
that's the general LLVM backend. If you never compile for Emscripten you'll
never use the Emscripten backend, so we may update this one day to only download
the Emscripten backend when you add the Emscripten target. For now though it's
just an extra 10MB gzip'd.
Closes#46819
Building on the work of # 45684 this commit updates the compiler to
unconditionally load the `rustc_trans` crate at runtime instead of linking to it
at compile time. The end goal of this work is to implement # 46819 where rustc
will have multiple backends available to it to load.
This commit starts off by removing the `extern crate rustc_trans` from the
driver. This involved moving some miscellaneous functionality into the
`TransCrate` trait and also required an implementation of how to locate and load
the trans backend. This ended up being a little tricky because the sysroot isn't
always the right location (for example `--sysroot` arguments) so some extra code
was added as well to probe a directory relative to the current dll (the
rustc_driver dll).
Rustbuild has been updated accordingly as well to have a separate compilation
invocation for the `rustc_trans` crate and assembly it accordingly into the
sysroot. Finally, the distribution logic for the `rustc` package was also
updated to slurp up the trans backends folder.
A number of assorted fallout changes were included here as well to ensure tests
pass and such, and they should all be commented inline.
We can't use git commands to compute a prerelease version when we're
building from a source tarball, or if git is otherwise unavailable.
We'll just call such builds `x.y.z-beta`, without a prerelease.
This currently only supports a limited subset of the full compilation,
but is likely 90% of what people will want and is possible without
building a full compiler (i.e., running LLVM). In theory, this means
that contributors who don't want to build LLVM now have an easy way to
compile locally, though running tests won't work.
Automaticaly calculate beta prerelease numbers
This is a forward-port of:
* 9426dda83d7a928d6ced377345e14b84b0f11c21
* cbfb9858951da7aee22d82178405306fca9decb1
from the beta branch which is used to automatically calculate the beta number
based on the number of merges to the beta branch so far.
This is a forward-port of:
* 9426dda83d7a928d6ced377345e14b84b0f11c21
* cbfb9858951da7aee22d82178405306fca9decb1
from the beta branch which is used to automatically calculate the beta number
based on the number of merges to the beta branch so far.
Don't include DefIndex in proc-macro registrar function symbol.
There can only ever be one registrar function per plugin or proc-macro crate, so adding the `DefIndex` to the function's symbol name does not serve a real purpose. Remove the `DefIndex` from the symbol name makes it stable across incremental compilation sessions.
This should fix issue #47292.
Test rustdoc js
Add tests for the rustdoc search. It was heavily required because of all the recent breaking changes that happened while I went through improvements in doc search (add search in/for generic search for example).
Fix 45345
There is a fix for https://github.com/rust-lang/rust/issues/45345
It re-introduces `CFG_LIBDIR_RELATIVE` which was broken when migration from `configure` script to `x.py`.
Other commits fix errors which happen after rustbuild cleanups.
Bump to 1.25.0
* Bump the release version to 1.25
* Bump the bootstrap compiler to the recent beta
* Allow using unstable rustdoc features on beta - this fix has been applied to
the beta branch but needed to go to the master branch as well.
* Bump the release version to 1.25
* Bump the bootstrap compiler to the recent beta
* Allow using unstable rustdoc features on beta - this fix has been applied to
the beta branch but needed to go to the master branch as well.
rustdoc: add option to abort the process on markdown differences
In the efforts of keeping the std docs free of markdown warnings, this PR adds a stopgap measure to make sure the CI fails if it detects a markdown difference. It does this by adding a new unstable flag to rustdoc, `--deny-render-differences`, which bootstrap then passes to rustdoc when documenting std and friends.
The implementation is... probably not the cleanest option. It currently adds an extra branch after it prints the markdown warnings, which just prints a final line and calls `::std::process::abort(1)`. I did it like this because if it just panics regularly, it looks like an ICE, an even though `html::render::run` returns a Result, that Result is also just `expect`ed immediately, generating the same problem. This way bypasses the panic handler at the top of the thread and looks like a proper failure. Since i don't have a real error Handler there, this is the best i can do without pulling in a real error system for rustdoc.
This PR is blocked on https://github.com/rust-lang/rust/pull/46853, which will fix the rendering differences that were present on master when i started this branch.
bors
kennytm
Alex Crichton
Johannes Nixdorf
varkor
Felix S. Klock II
Mark Simulacrum
Nick Cameron
John Kåre Alsaker
Guillaume Gomez
Josh Stone
Fabio B
Chris Coulson
Oliver Schneider
steveklabnik
Tamir Duberstein
Dan Callaghan
Oliver Middleton
Tatsuyuki Ishi
Jorge Aparicio
David Wood
Petr Hosek
Simon Sapin
Maxim Nazarenko
Oliver Schneider
QuietMisdreavus
comex
Marc-Antoine Perennou
Mark Mansi
Phlosioneer
Manish Goregaokar
Segev Finer
John Paul Adrian Glaubitz
penpalperson
Vadim Petrochenkov
Matthias Krüger
Jonathan A. Kollasch
roblabla
O01eg
Onur Aslan
Diggory Blake
Marco A L Barbosa
Santiago Pastorino
Michael Woerister
projektir