Let's have this layer be towards the end as we're emprically changing sccache
more than we're changing the rest of the image, so this'll allow us to reuse as
much of the cached image as possible.
Use the "official" cross compiler for NetBSD
The current NetBSD cross compiler is lacking, for example `std::thread` is not available (which causes problems for LLVM 4.0). This PR uses the official netbsd build system to compiler the cross compiler.
@alexcrichton: Can you please mirror `ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-7.0/source/sets/{src,gnusrc,sharesrc,syssrc}.tgz`. (Optionally you may want to use NetBSD versions 7.0.2 or 7.1, in that case you'll probably want to update the binary downloads used today as well).
I'll update the URL's afterwards (or feel free to use "allow edits from maintainers").
r? @alexcrichton
travis: Add timestamps to all build messages
When debugging why builds are taking so long it's often useful to get the
timestamp of all log messages as we're not always timing every tiny step of the
build. I wrote a [utility] for prepending a relative timestamp from the start of
a process which is now downloaded to the builders and is what we wrap the entire
build invocation in.
[utility]: https://github.com/alexcrichton/stamp-rsCloses#40577
travis: Ensure cargo links libcurl statically
We don't want a dynamic dependency in the library that we ship, so link it
statically by configuring curl-sys's build script to not pick up the system
version via pkg-config.
rustbuild: Make save-analysis an option
This makes save-analysis an option independent from the release channel.
The CI build scripts have been modified to enable the flag.
*Merge with caution.* I haven't tested this, and this can cause nightly breakage.
We don't want a dynamic dependency in the library that we ship, so link it
statically by configuring curl-sys's build script to not pick up the system
version via pkg-config.
rustbuild: Build documentation for `proc_macro`
This commit fixes#38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes#38749
travis: Attempt to debug sccache failures
I can't find anything that'd cause unexpected EOF in the source, so let's try
taking a look at the error logs on failures.
This commit fixes#38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes#38749
This change introduces a Dockerfile and script which builds a complete
Fuchsia toolchain which can be used to build Rust distribution for
Fuchsia. We only support cross-compiling at the moment, hence only
setting the target.
Travis only gives us 30GB disk space and we don't currently have an option to
increase that. Each musl target generates "hello world" binaries of about 3.5MB
in size, and we're testing two targets in the same image. We have around 3k
run-pass tests and 2 musl targets which works out to around 20GB. That's
dangerously close to the limit and is causing PRs to bounce.
This PR splits up the builder in two, one for x86_64 musl and the other for
i686. Hopefully that'll keep us under the disk limit.
Closes#40359
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
PRs can't land againt beta right now because the android bot is filling up on
disk space. I don't really know what's going on but the android bot is the
longest one to run anyway so it'll benefit from being split up regardless.
Now that mozilla/sccache#43 is fixed the caching works for MinGW on Windows. We
still can't use it for MSVC just yet, but I'll try to revive that branch at some
point.
This commit attempts to move more network operations to being retryable through
various operations. For example git submodule updates, downloading snapshots,
etc, are now all in retryable steps.
Hopefully this commit can cut down on the number of network failures we've been
seeing!
travis: Compile a more compatible libc.a for musl
The mitigations for #34978 involve passing `-Wa,-mrelax-relocations=no` to all C
code we compile, and we just forgot to pass it when compiling musl itself.
Closes#39979
Currently CI builds can fail spuriously during the LLVM build (#39003). I
believe this is due to sccache, and I believe that in turn was due to the fact
that the sccache server used to just be a raw mio server. Historically raw mio
servers are quite complicated to get right, but this is why we built Tokio! The
sccache server has been migrated to Tokio which I suspect would fix any latent
issues.
I have no confirmation of this (never been able to reproduce the deadlock
locally), but my hunch is that updating sccache to the master branch will fix
the timeouts during the LLVM build.
The binaries previously came from Gecko's infrastructure, but I've built new
ones by hand for Win/Mac/Linux and uploaded them to our CI bucket.
The mitigations for #34978 involve passing `-Wa,-mrelax-relocations=no` to all C
code we compile, and we just forgot to pass it when compiling musl itself.
Closes#39979
travis: Disable source tarballs on most builders
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
travis: Add builders without assertions
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.
Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.
Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.
Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.
Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
build std for sparc64-unknown-linux-gnu
r? @alexcrichton
panicking / unwinding is broken (#39646) but with std available at least people
will be able to debug that issue on real hardware
travis: Fix build order of dist-x86-linux
I just tried to build this container locally but it looks like connecting to
ftp.gnu.org requires SNI, so let's build curl/OpenSSL first to ensure that we've
got an SNI-capable client to download gcc/binutils with.
I just tried to build this container locally but it looks like connecting to
ftp.gnu.org requires SNI, so let's build curl/OpenSSL first to ensure that we've
got an SNI-capable client to download gcc/binutils with.
Delete the makefile build system
This PR deletes the makefile build system in favor of the rustbuild build system. The beta has now been branched so 1.16 will continue to be buildable from the makefiles, but going forward 1.17 will only be buildable with rustbuild.
Rustbuild has been the default build system [since 1.15.0](https://github.com/rust-lang/rust/pull/37817) and the makefiles were [proposed for deletion](https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368) at this time back in November of last year.
And now with the deletion of these makefiles we can start getting those sweet sweet improvements of using crates.io crates in the compiler!
Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
travis: Gate on some minimal support for incremental compilation.
This commit adds a travis job that
1. builds a stage2 compiler in incremental mode (but with empty incremental compilation cache), and
2. builds and runs the run-pass test suite also in incremental mode.
Building incrementally with an empty cache makes sure that the compiler doesn't crash in dependency tracking during bootstrapping. Executing the incrementally built test suite gives some measure of confidence that we generate valid code.
Note, however, that the above does not give strong guarantees about the validity of incremental compilation, it just provides a basis for being able to rely on from-scratch incr. comp. builds as reference values in further tests (which then do actual incremental compilation).
r? @alexcrichton
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
Using Ubuntu's cross-toolchains for powerpc* and s390x meant they were
depending on glibc symbols from Ubuntu 16.04. And if that host is ever
updated to a new release, the toolchains would raise the bar too.
This switches powerpc, powerpc64, and s390x to use crosstool-ng
toolchains, configured approximately like RHEL6 with kernel 2.6.32 and
glibc 2.12. This ABI level should also be compatible with Debian 7
(wheezy) and Ubuntu 12.04 (precise).
For powerpc64le, the challenge was that only glibc-2.19 officially added
support, but RHEL7 backported those changes to glibc-2.17. The backport
patches are complex and numerous, so instead of trying to push those
into crosstool-ng, this just uses glibc binaries directly from CentOS 7
and builds the toolchain manually.
This is ported from rust-lang/rust-buildbot#149.
r? @alexcrichton
rustbuild: Start building --enable-extended
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
cc #38531
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
travis: Pass --enable-llvm-static-stdcpp
All our releases are compiled with this, so let's be sure to do so whenever
`DEPLOY` is set. This'll ensure that we don't have dynamic dependencies on
libstdc++ which LLVM depends on, but instead we link it all statically to have
more portable binaries.
travis: Get an emscripten builder online
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
travis: Expand the `cross` linux image
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:
* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl
This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
cc #38531
travis: Add i586 linux and i686 musl
This commit expands the existing x86_64-musl entry in the Travis matrix to also
build/test i586-unknown-linux-gnu and i686-unknown-linux-musl.
cc #38531Closes#35599Closes#39053
This commit updates the compilers for many of the artifacts that we're producing
on Travis. These compilers are all compiled by crosstool-ng as they're currently
done for the images in which we're building all our cross compiled compilers.
The purpose of this commit is that when we ship binaries the artifacts won't
require a newer glibc, but rather be as compatible as possible with Linux
distributions by working with a very old version of glibc.
This commit always allocates a new matrix entry for the i686/x86_64 builder.
This builder is dedicated to just producing artifacts and eventually we'll
expand it to building other tools like Cargo and the RLS. The other builders
testing i686 and x86_64 won't use these historical toolchains.
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
This commit adds a travis job that builds a stage2 compiler in
incremental mode (but with empty incremental compilation cache).
Building incrementally with an empty cache makes sure that the
compiler doesn't crash in dependency tracking during bootstrapping.
All our releases are compiled with this, so let's be sure to do so whenever
`DEPLOY` is set. This'll ensure that we don't have dynamic dependencies on
libstdc++ which LLVM depends on, but instead we link it all statically to have
more portable binaries.
This commit passes the `--release-channel=nightly` flag to all images which have
the `DEPLOY` flag set. This means that we'll name artifacts and the compiler
appropriately.
This reworks a bit how arguments are passed, but for now doesn't change what's
already being passed. Eventually we'll want to avoid enabling debug assertions
and llvm assertions for *all* releases, but I figure we can tackle that a little
bit more down the road.
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:
* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl
This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
This commit expands the existing x86_64-musl entry in the Travis matrix to also
build/test i586-unknown-linux-gnu and i686-unknown-linux-musl.
cc #38531Closes#39053
travis: Expand dist builder coverage
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
This commit adds a new entry to the Travis matrix which performs a "distcheck",
which basically means that we create a tarball, extract that tarball, and then
build/test inside there. This ensures that the tarballs we produce are actually
able to be built/tested!
Along the way this also updates the rustbuild distcheck definition to propagate
the configure args from the top-level invocation.
Closes#38691
rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
This commit relegates all pretty tests to not get run by default and rather get
run as part of an "aux" test suite. This "aux" suite is renamed from the old
"cargotest" suite to just collect tests that don't need to run everywhere but
should at least pass on Unix/Windows.
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
There's been some flaky runs on Travis where the Android emulator is having
problems staying alive... presumably? For example:
* https://travis-ci.org/rust-lang/rust/jobs/186736745
This commit spawns the emulator in the same way as buildbot with `nohup` to hope
that it goes into the background successfully, followed by a `wait-for-device`
command. I'm not actually sure if this'll fix the problems we're seeing, but I
figure it can't hurt to test out.
This commit configures local sccache directory to get used if you're running
builds locally to enjoy the benefits of caching when running inside the
containers.
Recently the Cargo revision running on cargotest was updated in #38051, and the
updated version of Cargo pulls in the openssl-sys 0.9 crate instead of the old
openssl-sys 0.7 crate. The 0.9 release of openssl-sys has slightly different
requirements and logic for detecting OpenSSL, namely it requires `pkg-config` to
be present on Linux typically.
This commit fixes this problem by installing pkg-config in the cargotest
container that's running on the bots. This in turn should hopefully fix the
build script and allow it to find the already-installed local OpenSSL libraries.
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
After reading some articles [1] [2] yesterday about Docker and the "init"
process I got to thinking about the problems that we've been seeing on Travis.
The basic problem is that a Linux system may need an "init" process to work
properly when processes become zombies. Docker by default doesn't handle this
and the root process typically isn't an init process, so this can occasionally
cause quite a few problems.
We've been seeing spurious errors on Travis inside containers which look like
OOM and such, but my guess is that zombie processes were being reparented to the
top-level shell. The shell didn't expect the zombies and then behaved very
strangely.
This commit fixes these problems by using Yelp's "dumb-init" program [2] as the
init process in all of our containers. This ensures that there's a valid init
ready to reap children when they're reparented, which our test suite apparently
generates a bunch of throughout the tests and such.
[1]: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
[2]: https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
Alex Crichton
bors
Corey Farwell
Tim Neumann
Vadim Petrochenkov
Tatsuyuki Ishi
Petr Hosek
Steve Klabnik
Eduard-Mihai Burtescu
Jorge Aparicio
Josh Stone
Michael Woerister
Aidan Hobson Sayers