This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
This commit adds a new entry to the Travis matrix which performs a "distcheck",
which basically means that we create a tarball, extract that tarball, and then
build/test inside there. This ensures that the tarballs we produce are actually
able to be built/tested!
Along the way this also updates the rustbuild distcheck definition to propagate
the configure args from the top-level invocation.
Closes#38691
rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
travis: Don't use -9 on gzip
I timed this locally and plain old `gzip` took 2m06s while `gzip -9` took a
whopping 6m23s to save a mere 4MB out of 1.2GB. Let's shave a few minutes off
the Android builder by turning down the compression level.
This commit relegates all pretty tests to not get run by default and rather get
run as part of an "aux" test suite. This "aux" suite is renamed from the old
"cargotest" suite to just collect tests that don't need to run everywhere but
should at least pass on Unix/Windows.
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
The current image is `xcode7.3`, Travis's current default. Unfortunately this
has a version of LLDB which doesn't support debuginfo-lldb tests (see #32520),
so we're not running LLDB tests on Travis yet.
This switches us to the newest image from Travis, `xcode8.2`, which should have
a newer version of LLDB we can run tests against.
I timed this locally and plain old `gzip` took 2m06s while `gzip -9` took a
whopping 6m23s to save a mere 4MB out of 1.2GB. Let's shave a few minutes off
the Android builder by turning down the compression level.
travis: Fix testing 32-bit OSX target
We passed --target when we meant to pass --build, meaning we tested only the
standard library for 32-bit, not the whole compiler like we intended.
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
A few changes are included here:
* The `winapi` and `url` dependencies were dropped. The source code for these
projects is pretty weighty, and we're about to vendor them, so let's not
commit to that intake just yet. If necessary we can vendor them later but for
now it shouldn't be necessary.
* The `--frozen` flag is now always passed to Cargo, obviating the need for
tidy's `cargo_lock` check.
* Tidy was updated to not check the vendor directory
Closes#34687
Travis CI has new infrastructure using the Google Compute Engine which has both
faster CPUs and more memory, and we've been encouraged to switch as it should
help our build times! The only downside currently, however, is that IPv6 is
disabled, causing a number of standard library tests to fail.
Consequently this commit tweaks our travis config in a few ways:
* ccache is disabled as it's not working on GCE just yet
* Docker is used to run tests inside which reportedly will get IPv6 working
* A system LLVM installation is used instead of building LLVM itself. This is
primarily done to reduce build times, but we want automation for this sort of
behavior anyway and we can extend this in the future with building from source
as well if needed.
* gcc-specific logic is removed as the docker image for Ubuntu gives us a
recent-enough gcc by default.
It is very difficult to find tidy problems in the midst of the output of
the LLVM/jemalloc/etc. build, and travis is great for the former, so
lets remove that problem.
Tidy is still run first for failing fast on the easy stuff.
To accomplish this we have travis actually persist ccache across builds. This
has LLVM built within 6 minutes, and all of stage1 built within 18.
Caching should work on fresh PRs (cache acquired from the master branch).
Because all we persist is ccache, there is minimal danger of persisting corrupt
build state.
I had to mangle `configure` a bit to make --enable-ccache work when custom
compilers are provide via CC and CXX.
Instead, only run `make tidy`. The tidy script can run quite quickly, and it's
super annoying to run tests for 50 minutes only to have bors fail with a
"trailing whitespace" error.
Instead, only run `make tidy`. The tidy script can run quite quickly, and it's
super annoying to run tests for 50 minutes only to have bors fail with a
"trailing whitespace" error.
It's looking like we're still timing out all over the place with travis_wait
because the entire `make -j4 rustc-stage1` command is taking too long. Instead,
achieve roughly the same idea by just having `-Z time-passes` printing
information. We shouldn't have a pass that takes longer than 10 minutes in
isolation.
The most frequent failure for our travis builds is running into the timeout
limits when building the compiler itself. Building librustc takes a very long
amount of time, often hitting the 10 minutes with no output threshold that
travis imposes on us.
This commit switches the relevant `make` step to being wrapped in the
`travis_wait` command [1]. This command will print something once a minute so as
to not time out a build.
This will hopefully enable us to have fewer flaky builds on travis!
[1]: http://docs.travis-ci.com/user/build-timeouts/
This version is slightly more up to date and is closer to the 3.5 that we're
using. This also updates the travis config to have a build matrix which tests
rust against LLVM 3.3 and 3.4. For pull requests only LLVM 3.4 is tested to
reduce the load on travis.
This is mostly just fluff, there's no real reason to gate rust on these results,
it's more of just a nice thing to know when we break compatibility with LLVM 3.3
and 3.4 (and eventually 3.5). This turns off notifications of failed commits
(which are sent out for pushes to master).
Travis CI provides an easy-to-use continuous integration infrastructure for
github repos to use. Travis will automatically test all PRs which are opened
against the rust repository, informing PR owners of the test results.
I believe that this will be a very convenient piece of infrastructure as we'll
be able to reduce the load on bors quite a bit. In theory all PRs opened have
had the full test suite run against them, but unfortunately this is rarely the
case (I'm a prime suspect). Travis will be able to provide easy and relatively
quick (~30min) feedback for PRs. By ensuring fewer failures on bors, we can
hopefully feed more successful jobs to bors.
Overall, I expect this to be very helpful for new contributors as well as
regular contributors as it's another layer of tests being run which will
hopefully catch things sooner. One of the most convenient parts about using
Travis is that there's very little burden in terms of maintenance, and if things
go wrong we can easily turn travis completely off.
Note that this is *not* the metric by which a PR will be merged with. Using
travis will purely be another source for running tests, we will continue to gate
all PRs on bors.
Alex Crichton