Use #!/usr/bin/env as shebang for Bash scripts
On some systems, the bash command could be available in another
directory than /bin. As such, to offer an env shebang is more
convenient.
This make sense even for docker scripts, as you can use Docker
on FreeBSD or SmartOS for example.
On some systems, the bash command could be available in another
directory than /bin. As such, to offer an env shebang is more
convenient.
This make sense even for docker scripts, as you can use Docker
on FreeBSD or SmartOS for example.
* SDK tools is upgraded to 27.0.0.
- Refactored to use `sdkmanager`/`avdmanager` instead of the deprecated
`android` tool.
* The Java version used by Android SDK is downgraded to OpenJDK-8, in order
to download the SDK through HTTPS.
* NDK is upgrade to r15c.
- Dropped support for android-9 (2.3 / Gingerbread), the minimal
supported version is now android-14 (4.0 / Ice Cream Sandwich).
- Changed the default Android compiler from GCC to clang.
- For details of change introduced by NDK r15, see
https://github.com/android-ndk/ndk/wiki/Changelog-r15.
ci: Update musl with new release
Apparently there's at least one CVE fixed in the new version of musl, and
because we're distributing it seems like a good opportunity to update!
Unfortunately it looks like #38618 still hasn't been fixed.
Apparently there's at least one CVE fixed in the new version of musl, and
because we're distributing it seems like a good opportunity to update!
Unfortunately it looks like #38618 still hasn't been fixed.
test: Update Emscripten failures/passing
All tests should now have annotation for *why* they're ignored on emscripten. A
few tests no longer need such an annotation as well!
Closes#41299
Make Solaris builder compatible with Solaris 10 retry
Unfortunately, #45255 does not quite cut it,
so use a different approach to have Solaris 10 compatibility
by tricking libbacktrace's autoconf tests.
The sysroot download routine is slightly changed, too.
Unfortunately, #45255 does not quite cut it,
so use a different approach to have Solaris 10 compatibility
by tricking libbacktrace's autoconf tests.
The sysroot download routine is slightly changed, too.
Apparently the https urls are broken due to some certificate validation
whatnots, and so far the least intrusive solution I've found is to just disable
that.
This can be used to build rust-std.
The dilos illumos distribution was chosen, because illumos is free software
as opposed to Oracle Solaris and dilos is the only illumos distribution that
supports x86_64 and sparcv9 at the same level.
Add aarch64-unknown-linux-musl target
This adds support for the aarch64-unknown-linux-musl target in the build and CI systems.
This addresses half of issue #42520.
The new file `aarch64_unknown_linux_musl.rs` is a copy of `aarch64_unknown_linux_gnu.rs` with "gnu" replaced by "musl", and the added logic in `build-arm-musl.sh` is similarly a near-copy of the arches around it, so overall the changes were straightforward.
Testing:
```
$ sudo ./src/ci/docker/run.sh cross
...
Dist std stage2 (x86_64-unknown-linux-gnu -> aarch64-unknown-linux-musl)
Building stage2 test artifacts (x86_64-unknown-linux-gnu -> aarch64-unknown-linux-musl)
Compiling getopts v0.2.14
Compiling term v0.0.0 (file:///checkout/src/libterm)
Compiling test v0.0.0 (file:///checkout/src/libtest)
Finished release [optimized] target(s) in 16.91 secs
Copying stage2 test from stage2 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / aarch64-unknown-linux-musl)
...
Build completed successfully in 0:55:22
```
```
$ rustup toolchain link local obj/build/x86_64-unknown-linux-gnu/stage2
$ rustup default local
```
After setting the local toolchain as default, and adding this in ~/.cargo/config:
```
[target.aarch64-unknown-linux-musl]
linker = "aarch64-linux-musl-gcc"
```
...then the toolchain was able to build a working ripgrep as a test:
```
$ readelf -a target/aarch64-unknown-linux-musl/debug/rg | grep -i interpreter
$ readelf -a target/aarch64-unknown-linux-musl/debug/rg | grep NEEDED
$ file target/aarch64-unknown-linux-musl/debug/rg
target/aarch64-unknown-linux-musl/debug/rg: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=be11036b0988fac5dccc9f6487eb780b05186582, not stripped
```
* Adjust bootstrap to provide useful output on failure
* Add missing package dependencies in the build environment
* Fix permission bits on prebuilt toolchain files
ci: Upload/download from a new S3 bucket
Moving buckets from us-east-1 to us-west-1 because us-west-1 is where
rust-central-station itself runs and in general is where we have all our other
buckets.
ci: Disable rustc debug assertions on OSX
This commit disables debug assertions for OSX in an effort to improve cycle time
on OSX. It looks like #44610 didn't shave off quite as much time as desired so
let's see how much this helps.
This commit disables debug assertions for OSX in an effort to improve cycle time
on OSX. It looks like #44610 didn't shave off quite as much time as desired so
let's see how much this helps.
Moving buckets from us-east-1 to us-west-1 because us-west-1 is where
rust-central-station itself runs and in general is where we have all our other
buckets.
Most of the other rust-lang buckets are in us-west-1 and I think the original
bucket was just accidentally created in the us-east-1 region. Let's consolidate
by moving it to the same location as the rest of our buckets.
This was intended for bots back in the day where we'd persist caches of LLVM
builds across runs, but nowadays we don't do that on any of the bots so this
option is no longer necessary
Flag docker invocations as --privileged on CI
When upgrading to LLVM 5.0 it was found that the leak sanitizer tests were
failing with fatal errors, but they were passing locally when run. Turns out it
looks like they may be using new ptrace-like syscalls so the docker container
now needs `--privileged` when executing to complete the test.
When upgrading to LLVM 5.0 it was found that the leak sanitizer tests were
failing with fatal errors, but they were passing locally when run. Turns out it
looks like they may be using new ptrace-like syscalls so the docker container
now needs `--privileged` when executing to complete the test.
When working through bugs for the LLVM 5.0 upgrade it looks like the FreeBSD
cross compilers we're currently using are unable to build LLVM, failing with
references to the function `std::to_string` claiming it doesn't exist. I don't
actually know what this function is, but assuming that it was added in a more
recent version of a C++ standard I've updated the gcc versions for the
toolchains we're using. This made the error go away!
Make compiletest set cwd before running js tests
Proposed fix for #38800.
Not all tests pass yet - I will mention failures here once the test suite has finished.
Add a disabled builder for aarch64 emulated tests
This commit adds a disabled builder which will run all tests for the standard
library for aarch64 in a QEMU instance. Once we get enough capacity to run this
on Travis this can be used to boost our platform coverage of AArch64
This commit includes the following:
* Fix syntax errors in Python 3
* Include more docstrings in classes, methods, and functions
* Include unit tests using `unittest`
* Merge implementation of `{rustc,cargo}_out_of_date`
* Merge implementation of `RustBuild.{cargo,rustc}`
* Remove unnecessary source code
* Move all the attributes defined outside of `__init__`
* Remove remaining `%s` from print function
* Remove `WindowsError` reference on non-windows systems
* Rename some variables to be more explicit avoid their meaning
* Run bootstrap tests in the CI process
* Remove non-pythonic getters
* Remove duplicate code in `download_stage0` method
* Reduce the number of branches in `build_bootstrap` method
* Re-raise exception when we cannot execute uname in non-windows systems
* Avoid long lines
This commit adds a disabled builder which will run all tests for the standard
library for aarch64 in a QEMU instance. Once we get enough capacity to run this
on Travis this can be used to boost our platform coverage of AArch64
Use the `-f` flag to indicate that, for example, a 500 response code is to be
considered a failure, triggering the normal retry logic. Also ignore errors
where we check the date from google.com, as a failure there shouldn't fail the
build.
This commit migrates the in-tree `libcompiler_builtins` to the upstream version
at https://github.com/rust-lang-nursery/compiler-builtins. The upstream version
has a number of intrinsics written in Rust and serves as an in-progress rewrite
of compiler-rt into Rust. Additionally it also contains all the existing
intrinsics defined in `libcompiler_builtins` for 128-bit integers.
It's been the intention since the beginning to make this transition but
previously it just lacked the manpower to get done. As this PR likely shows it
wasn't a trivial integration! Some highlight changes are:
* The PR rust-lang-nursery/compiler-builtins#166 contains a number of fixes
across platforms and also some refactorings to make the intrinsics easier to
read. The additional testing added there also fixed a number of integration
issues when pulling the repository into this tree.
* LTO with the compiler-builtins crate was fixed to link in the entire crate
after the LTO process as these intrinsics are excluded from LTO.
* Treatment of hidden symbols was updated as previously the
`#![compiler_builtins]` crate would mark all symbol *imports* as hidden
whereas it was only intended to mark *exports* as hidden.
This modifies the builder to download and use the LLVM tools from the
last known good build on the WebAssembly buildbot waterfall, since these
tools are built with the WebAssembly LLVM backend enabled.
This adds the experimental targets option to configure so it can be used
by the builders and changes the wasm32 Dockerfile accordingly. Instead
of using LLVM from the emsdk, the builder's emscripten tools now uses
the Rust in-tree LLVM, since this is the one built with wasm support.
Add a travis builder for wasm32-unknown-emscripten
This commits add an entry to travis matrix that will execute wasm32-unknown-emscripten tests suites.
- Emscripten for asmjs was updated to sdk-1.37.13-64bit
- The tests are run with node 8.0.0 (it can execute wasm)
- A wrapper script is used to run each test from the directory where it is (workaround for https://github.com/kripken/emscripten/issues/4542)
- Some tests are ignore, see #42629 and #42630
* Bring back colors on Travis, which was disabled since #39036.
Append --color=always to cargo when running in CI environment.
* Removed `set -x` in the shell scripts. The `retry` function already
prints which command it is running, add `-x` just add noise to the
output.
* Support travis_fold/travis_time. Matching pairs of these allow Travis CI
to collapse the output in between. This greatly cut down the unnecessary
"successful" output one need to scroll through before finding the failed
statement.
In rustbuild itself we download from our mirror but in the containers we don't
do this yet. The OpenSSL download url changes from time to time (it breaks when
they release a new version) so let's download from our mirror instead.
Add disabled android host builders
Introduce the concept of disabled builder. A disabled builder is one that is not run by travis. It is intended to be run by the user who wants a rustc for a tier 2 or 3 platform. Off corse, there is no guarantee that it will work.
Now that the final bug fixes have been merged into sccache we can start
leveraging sccache on the MSVC builders on AppVeyor instead of relying on the
ad-hoc caching strategy of trigger files and whatnot.
LLVM 4.0 Upgrade
Since nobody has done this yet, I decided to get things started:
**Todo:**
* [x] push the relevant commits to `rust-lang/llvm` and `rust-lang/compiler-rt`
* [x] cleanup `.gitmodules`
* [x] Verify if there are any other commits from `rust-lang/llvm` which need backporting
* [x] Investigate / fix debuginfo ("`<optimized out>`") failures
* [x] Use correct emscripten version in docker image
---
Closes#37609.
---
**Test results:**
Everything is green 🎉
Add a comment for disabling errexit, try to debug appveyor cache
Comments added as requested.
Also, lets add some cache debugging to appveyor. I *think* this is how to ignore errors in cmd.exe (and I did try it on my own machine), but I'm not 100% sure how appveyor runs them. In the worst case it'll fail but I guess that isn't the end of the world since the build has already failed by this point.
r? @TimNN
This commit enables the `rust-analysis` package to be produced for all targets
that are part of the `dist-*` suite of docker images on Travis. Currently
these packages are showing up with `available = false` in the
`channel-rust-nightly.toml` manifest where we'd prefer to have them show up for
all targets.
Unfortunately rustup isn't handling the `available = false` section well right
now, so this should also inadvertently fix the nightly regression.
Add the RLS as a submodule and build a package out of it
r? @brson (and cc @alexcrichton) Please review closely, I am not at all convinced I've done the right things here. I did run `x.py dist` and it makes an rls package which looks right to my eyes, but I haven't tested on non-linux platforms nor am I really sure what it should look like.
This does not attempt to run tests for the RLS yet.
travis: Use upstream LLVM repositories for Fuchsia
The Fuchsia copies of LLVM repositories contain additional patches
for work-in-progress features and there is some amount of churn that
may break Rust. Use upstream LLVM repositories instead for building
the toolchain used by the Fuchsia builder.
dist-powerpc-linux: use a pure 32-bit CPU profile
With `-mcpu=power4`, code might use instructions like `fcfid`, excluding
older CPUs like the PowerPC G4, which apparently some users would like
to use. The generic `-mcpu=powerpc` should stick to pure 32-bit PowerPC
instructions.
Fixesrust-lang/cargo#3852.
Re-enable appveyor cache
After breaking the queue last time, I'm cautiously back with a PR to re-enable caching on appveyor.
If you look at https://ci.appveyor.com/project/rust-lang/rust/build/1.0.2623/job/46o90by4ari6gege (one of the multiple runs that started failed, there are actually two errors - one for restoring the cache, one right at the bottom for creating a directory. I only noticed the restore error at the time as I was a bit rushed to revert and didn't stop to wonder why it continued - turns out appveyor [does not abort on cache restore failure](https://github.com/appveyor/ci/issues/723).
Turns out the cause of the build failures was the cache directory existing and me being thinking that because mkdir on windows is [recursive by default](http://stackoverflow.com/a/905239/2352259), it ignores the error if the directory already exists. Apparently this is not true, so now it checks if the directory exists before attempting to create.
In addition, I've added some more paranoia to double check everything is sane.
The Fuchsia copies of LLVM repositories contain additional patches
for work-in-progress features and there is some amount of churn that
may break Rust. Use upstream LLVM repositories instead for building
the toolchain used by the Fuchsia builder.
With `-mcpu=power4`, code might use instructions like `fcfid`, excluding
older CPUs like the PowerPC G4, which apparently some users would like
to use. The generic `-mcpu=powerpc` should stick to pure 32-bit PowerPC
instructions.
Fixesrust-lang/cargo#3852.
I've tracked down what I believe is the last spurious sccache failure on #40240
to behavior in mio (carllerche/mio#583), and this commit updates the binaries to
a version which has that fix incorporated.
Previously we would use one builder on Travis to produce two sets of host
compilers for two different targets. Unfortunately though we've recently
increased how much we're building for each target so this is starting to take
unnecessarily long (#40804). This commit splits the dist builders in two by
ensuring that we only dist one target on each builder, which should take a much
shorter amount of time. This should also unblock other work such as landing the
RLS (#40584).
Attempt to cache git modules
Partial resolution of #40772, appveyor remains to be done once travis looks like it's working ok.
The approach in this PR is based on the `--reference` flag to `git-clone`/`git-submodule --update` and is a compromise based on the current limitations of the tools we're using.
The ideal would be:
1. have a cached pristine copy of rust-lang/rust master in `$HOME/rustsrc` with all submodules initialised
2. clone the PR branch with `git clone --recurse-submodules --reference $HOME/rustsrc git@github.com:rust-lang/rust.git`
This would (in the nonexistent ideal world) use the pristine copy as an object cache for the top level repo and all submodules, transferring over the network only the changes on the branch. Unfortunately, a) there is no way to manually control the initial clone with travis and b) even if there was, cloned submodules don't use the submodules of the reference as an object cache. So the steps we end up with are:
1. have a cached pristine copy of rust-lang/rust master in `$HOME/rustsrc` with all submodules initialised
2. have a cloned PR branch
3. extract the path of each submodule, and explicitly `git submodule update --init --reference $HOME/rustsrc/$module $module` (i.e. point directly to the location of the pristine submodule repo) for each one
I've also taken some care to make this forward compatible, both for adding and removing submodules.
r? @alexcrichton
Let's have this layer be towards the end as we're emprically changing sccache
more than we're changing the rest of the image, so this'll allow us to reuse as
much of the cached image as possible.
Use the "official" cross compiler for NetBSD
The current NetBSD cross compiler is lacking, for example `std::thread` is not available (which causes problems for LLVM 4.0). This PR uses the official netbsd build system to compiler the cross compiler.
@alexcrichton: Can you please mirror `ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-7.0/source/sets/{src,gnusrc,sharesrc,syssrc}.tgz`. (Optionally you may want to use NetBSD versions 7.0.2 or 7.1, in that case you'll probably want to update the binary downloads used today as well).
I'll update the URL's afterwards (or feel free to use "allow edits from maintainers").
r? @alexcrichton
travis: Add timestamps to all build messages
When debugging why builds are taking so long it's often useful to get the
timestamp of all log messages as we're not always timing every tiny step of the
build. I wrote a [utility] for prepending a relative timestamp from the start of
a process which is now downloaded to the builders and is what we wrap the entire
build invocation in.
[utility]: https://github.com/alexcrichton/stamp-rsCloses#40577
travis: Ensure cargo links libcurl statically
We don't want a dynamic dependency in the library that we ship, so link it
statically by configuring curl-sys's build script to not pick up the system
version via pkg-config.
rustbuild: Make save-analysis an option
This makes save-analysis an option independent from the release channel.
The CI build scripts have been modified to enable the flag.
*Merge with caution.* I haven't tested this, and this can cause nightly breakage.
We don't want a dynamic dependency in the library that we ship, so link it
statically by configuring curl-sys's build script to not pick up the system
version via pkg-config.
rustbuild: Build documentation for `proc_macro`
This commit fixes#38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes#38749
travis: Attempt to debug sccache failures
I can't find anything that'd cause unexpected EOF in the source, so let's try
taking a look at the error logs on failures.
This commit fixes#38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes#38749
This change introduces a Dockerfile and script which builds a complete
Fuchsia toolchain which can be used to build Rust distribution for
Fuchsia. We only support cross-compiling at the moment, hence only
setting the target.
Travis only gives us 30GB disk space and we don't currently have an option to
increase that. Each musl target generates "hello world" binaries of about 3.5MB
in size, and we're testing two targets in the same image. We have around 3k
run-pass tests and 2 musl targets which works out to around 20GB. That's
dangerously close to the limit and is causing PRs to bounce.
This PR splits up the builder in two, one for x86_64 musl and the other for
i686. Hopefully that'll keep us under the disk limit.
Closes#40359
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
PRs can't land againt beta right now because the android bot is filling up on
disk space. I don't really know what's going on but the android bot is the
longest one to run anyway so it'll benefit from being split up regardless.
Now that mozilla/sccache#43 is fixed the caching works for MinGW on Windows. We
still can't use it for MSVC just yet, but I'll try to revive that branch at some
point.
This commit attempts to move more network operations to being retryable through
various operations. For example git submodule updates, downloading snapshots,
etc, are now all in retryable steps.
Hopefully this commit can cut down on the number of network failures we've been
seeing!
travis: Compile a more compatible libc.a for musl
The mitigations for #34978 involve passing `-Wa,-mrelax-relocations=no` to all C
code we compile, and we just forgot to pass it when compiling musl itself.
Closes#39979
Currently CI builds can fail spuriously during the LLVM build (#39003). I
believe this is due to sccache, and I believe that in turn was due to the fact
that the sccache server used to just be a raw mio server. Historically raw mio
servers are quite complicated to get right, but this is why we built Tokio! The
sccache server has been migrated to Tokio which I suspect would fix any latent
issues.
I have no confirmation of this (never been able to reproduce the deadlock
locally), but my hunch is that updating sccache to the master branch will fix
the timeouts during the LLVM build.
The binaries previously came from Gecko's infrastructure, but I've built new
ones by hand for Win/Mac/Linux and uploaded them to our CI bucket.
The mitigations for #34978 involve passing `-Wa,-mrelax-relocations=no` to all C
code we compile, and we just forgot to pass it when compiling musl itself.
Closes#39979
travis: Disable source tarballs on most builders
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
travis: Add builders without assertions
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.
Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.
Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.
Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.
Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
build std for sparc64-unknown-linux-gnu
r? @alexcrichton
panicking / unwinding is broken (#39646) but with std available at least people
will be able to debug that issue on real hardware
travis: Fix build order of dist-x86-linux
I just tried to build this container locally but it looks like connecting to
ftp.gnu.org requires SNI, so let's build curl/OpenSSL first to ensure that we've
got an SNI-capable client to download gcc/binutils with.
I just tried to build this container locally but it looks like connecting to
ftp.gnu.org requires SNI, so let's build curl/OpenSSL first to ensure that we've
got an SNI-capable client to download gcc/binutils with.
Delete the makefile build system
This PR deletes the makefile build system in favor of the rustbuild build system. The beta has now been branched so 1.16 will continue to be buildable from the makefiles, but going forward 1.17 will only be buildable with rustbuild.
Rustbuild has been the default build system [since 1.15.0](https://github.com/rust-lang/rust/pull/37817) and the makefiles were [proposed for deletion](https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368) at this time back in November of last year.
And now with the deletion of these makefiles we can start getting those sweet sweet improvements of using crates.io crates in the compiler!
Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
travis: Gate on some minimal support for incremental compilation.
This commit adds a travis job that
1. builds a stage2 compiler in incremental mode (but with empty incremental compilation cache), and
2. builds and runs the run-pass test suite also in incremental mode.
Building incrementally with an empty cache makes sure that the compiler doesn't crash in dependency tracking during bootstrapping. Executing the incrementally built test suite gives some measure of confidence that we generate valid code.
Note, however, that the above does not give strong guarantees about the validity of incremental compilation, it just provides a basis for being able to rely on from-scratch incr. comp. builds as reference values in further tests (which then do actual incremental compilation).
r? @alexcrichton
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
Using Ubuntu's cross-toolchains for powerpc* and s390x meant they were
depending on glibc symbols from Ubuntu 16.04. And if that host is ever
updated to a new release, the toolchains would raise the bar too.
This switches powerpc, powerpc64, and s390x to use crosstool-ng
toolchains, configured approximately like RHEL6 with kernel 2.6.32 and
glibc 2.12. This ABI level should also be compatible with Debian 7
(wheezy) and Ubuntu 12.04 (precise).
For powerpc64le, the challenge was that only glibc-2.19 officially added
support, but RHEL7 backported those changes to glibc-2.17. The backport
patches are complex and numerous, so instead of trying to push those
into crosstool-ng, this just uses glibc binaries directly from CentOS 7
and builds the toolchain manually.
This is ported from rust-lang/rust-buildbot#149.
r? @alexcrichton
rustbuild: Start building --enable-extended
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
cc #38531
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
travis: Pass --enable-llvm-static-stdcpp
All our releases are compiled with this, so let's be sure to do so whenever
`DEPLOY` is set. This'll ensure that we don't have dynamic dependencies on
libstdc++ which LLVM depends on, but instead we link it all statically to have
more portable binaries.
travis: Get an emscripten builder online
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
travis: Expand the `cross` linux image
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:
* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl
This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
cc #38531
travis: Add i586 linux and i686 musl
This commit expands the existing x86_64-musl entry in the Travis matrix to also
build/test i586-unknown-linux-gnu and i686-unknown-linux-musl.
cc #38531Closes#35599Closes#39053
This commit updates the compilers for many of the artifacts that we're producing
on Travis. These compilers are all compiled by crosstool-ng as they're currently
done for the images in which we're building all our cross compiled compilers.
The purpose of this commit is that when we ship binaries the artifacts won't
require a newer glibc, but rather be as compatible as possible with Linux
distributions by working with a very old version of glibc.
This commit always allocates a new matrix entry for the i686/x86_64 builder.
This builder is dedicated to just producing artifacts and eventually we'll
expand it to building other tools like Cargo and the RLS. The other builders
testing i686 and x86_64 won't use these historical toolchains.
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
This commit adds a travis job that builds a stage2 compiler in
incremental mode (but with empty incremental compilation cache).
Building incrementally with an empty cache makes sure that the
compiler doesn't crash in dependency tracking during bootstrapping.
All our releases are compiled with this, so let's be sure to do so whenever
`DEPLOY` is set. This'll ensure that we don't have dynamic dependencies on
libstdc++ which LLVM depends on, but instead we link it all statically to have
more portable binaries.
This commit passes the `--release-channel=nightly` flag to all images which have
the `DEPLOY` flag set. This means that we'll name artifacts and the compiler
appropriately.
This reworks a bit how arguments are passed, but for now doesn't change what's
already being passed. Eventually we'll want to avoid enabling debug assertions
and llvm assertions for *all* releases, but I figure we can tackle that a little
bit more down the road.
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:
* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl
This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
This commit expands the existing x86_64-musl entry in the Travis matrix to also
build/test i586-unknown-linux-gnu and i686-unknown-linux-musl.
cc #38531Closes#39053
travis: Expand dist builder coverage
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
This commit adds a new entry to the Travis matrix which performs a "distcheck",
which basically means that we create a tarball, extract that tarball, and then
build/test inside there. This ensures that the tarballs we produce are actually
able to be built/tested!
Along the way this also updates the rustbuild distcheck definition to propagate
the configure args from the top-level invocation.
Closes#38691
rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
This commit relegates all pretty tests to not get run by default and rather get
run as part of an "aux" test suite. This "aux" suite is renamed from the old
"cargotest" suite to just collect tests that don't need to run everywhere but
should at least pass on Unix/Windows.
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
There's been some flaky runs on Travis where the Android emulator is having
problems staying alive... presumably? For example:
* https://travis-ci.org/rust-lang/rust/jobs/186736745
This commit spawns the emulator in the same way as buildbot with `nohup` to hope
that it goes into the background successfully, followed by a `wait-for-device`
command. I'm not actually sure if this'll fix the problems we're seeing, but I
figure it can't hurt to test out.
This commit configures local sccache directory to get used if you're running
builds locally to enjoy the benefits of caching when running inside the
containers.
Recently the Cargo revision running on cargotest was updated in #38051, and the
updated version of Cargo pulls in the openssl-sys 0.9 crate instead of the old
openssl-sys 0.7 crate. The 0.9 release of openssl-sys has slightly different
requirements and logic for detecting OpenSSL, namely it requires `pkg-config` to
be present on Linux typically.
This commit fixes this problem by installing pkg-config in the cargotest
container that's running on the bots. This in turn should hopefully fix the
build script and allow it to find the already-installed local OpenSSL libraries.
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
After reading some articles [1] [2] yesterday about Docker and the "init"
process I got to thinking about the problems that we've been seeing on Travis.
The basic problem is that a Linux system may need an "init" process to work
properly when processes become zombies. Docker by default doesn't handle this
and the root process typically isn't an init process, so this can occasionally
cause quite a few problems.
We've been seeing spurious errors on Travis inside containers which look like
OOM and such, but my guess is that zombie processes were being reparented to the
top-level shell. The shell didn't expect the zombies and then behaved very
strangely.
This commit fixes these problems by using Yelp's "dumb-init" program [2] as the
init process in all of our containers. This ensures that there's a valid init
ready to reap children when they're reparented, which our test suite apparently
generates a bunch of throughout the tests and such.
[1]: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
[2]: https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
gnzlbg
Guillaume Gomez
Sébastien Santoro
bors
Michael Neumann
Simon Sapin
kennytm
Alex Crichton
bgermann
Josh Stone
Marco A L Barbosa
Mark Simulacrum
Alexander von Gluck IV
James Tucker
Petr Hosek
P.Y. Laligand
Ben Cressey
Oliver Schneider
Oliver Schneider
Tim Neumann
Jessica Hamilton
Jeremy Sorensen
Jeremy Soller
Richard Dodd
Milton Mazzarri
Thomas Lively
Ty Coghlan
Aidan Hobson Sayers
Marco Castelluccio
Venkata Giri Reddy
Corey Farwell
Nick Cameron
Ariel Ben-Yehuda
Vadim Petrochenkov
Tatsuyuki Ishi
Steve Klabnik
Eduard-Mihai Burtescu
Jorge Aparicio
Michael Woerister