Previously, logging at DEBUG or below would always be silenced, because
rustc compiles tracing with the `static_max_level_info` feature. That
makes sense for release artifacts, but not for developing rustdoc.
Instead, this compiles two different versions of tracing: one in the
release artifacts, distributed in the sysroot, and a new version
compiled by rustdoc. Since `rustc_driver` is always linked to the
version of sysroot, this copy/pastes `init_env_logging` into rustdoc.
The builds the second version of tracing unconditionally; see the code
for details on why.
* Reuse memory
* simplify `next_def_id`, avoid multiple hashing and unnecessary lookups
* remove `all_fake_def_ids`, use the global map instead (probably not a good step toward parallelization, though...)
* convert `add_deref_target` to iterative implementation
* use `ArrayVec` where we know the max number of elements
* minor touchups here and there
* avoid building temporary vectors that get appended to other vectors
At most places I may or may not be doing the compiler's job is this PR.
Thanks to marcusklaas' hard work in https://github.com/raphlinus/pulldown-cmark/pull/469, this fixes a lot of rustdoc bugs!
- Get rid of unnecessary `RefCell`
- Fix duplicate warnings for broken implicit reference link
- Remove unnecessary copy of links
The only change is that `expect_file` now uses path relative to the
current file (same as `include!`). Before, it used paths relative to
the workspace root, which makes no sense.
It's a unit-test in a sense that it only checks syntax highlighting.
However, the resulting HTML is written to disk and can be easily
inspected in the browser.
To update the test, run with `--bless` argument or set
`UPDATE_EXPEC=1` env var
This drops the parking_lot dependency; the ReentrantMutex type appeared
to be unused (at least, no compilation failures occurred).
This is technically a possible change in behavior of its users, as
lock() would wait on other threads releasing their guards, but since we
didn't actually remove any threading or such in this code, it appears
that we never used that behavior (the behavior change is only noticeable
if the type previously was used in two threads, in a single thread
ReentrantMutex is useless).
* Move fs::create_dir_all calls into DocFS to provide a clean
extension point if async extension there is needed.
* Convert callsites of create_dir_all to ensure_dir to reduce syscalls.
* Convert fs::write usage to DocFS.write
(which also removes a lot of try_err! usage for easier reading)
* Convert File::create calls to use Vec buffers and then DocFS.write
in order to consistently reduce syscalls as well, make
deferring to threads cleaner and avoid leaving dangling content if
writing to existing files....
* Convert OpenOptions usage similarly - I could find no discussion on
the use of create_new for that one output file vs all the other
files render creates, if link redirection attacks are a concern
DocFS will provide a good central point to introduce systematic
create_new usage. (fs::write/File::create is vulnerable to link
redirection attacks).
* DocFS::write defers to rayon for IO on Windows producing a modest
speedup: before this patch on my development workstation:
$ time cargo +mystg1 doc -p winapi:0.3.7
Documenting winapi v0.3.7
Finished dev [unoptimized + debuginfo] target(s) in 6m 11s
real 6m11.734s
Afterwards:
$ time cargo +mystg1 doc -p winapi:0.3.7
Compiling winapi v0.3.7
Documenting winapi v0.3.7
Finished dev [unoptimized + debuginfo] target(s) in 49.53s
real 0m49.643s
I haven't measured how much time is in the compilation logic vs in the
IO and outputting etc, but this takes it from frustating to tolerable
for me, at least for now.
Update parking_lot to 0.7
Unfortunately this'll dupe parking_lot until the data_structures crate
is published and be updated in rls in conjunction with crossbeam-channel
constraints:
- clean/inline.rs needs this map to fill in traits when inlining
- fold.rs needs this map to allow passes to fold trait items
- html/render.rs needs this map to seed the Cache.traits map of all
known traits
The first two are the real problem, since `DocFolder` only operates on
`clean::Crate` but `clean/inline.rs` only sees the `DocContext`. The
introduction of early passes means that these two now exist at the same
time, so they need to share ownership of the map. Even better, the use
of `Crate` in a rustc thread pool means that it needs to be Sync, so it
can't use `Lrc<Lock>` to manually activate thread-safety.
`parking_lot` is reused from elsewhere in the tree to allow use of its
`ReentrantMutex`, as the relevant parts of rustdoc are still
single-threaded and this allows for easier use in that context.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?
-----
So, timeline for those who need to catch up:
* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](https://github.com/rust-lang/rust/issues/38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](https://github.com/rust-lang/rust/pull/40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](https://github.com/rust-lang/rust/issues/40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](https://github.com/rust-lang/rust/pull/41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](https://github.com/rust-lang/rust/pull/41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](https://github.com/rust-lang/rust/pull/41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
* In that time, [bootstrap was completely reworked](https://github.com/rust-lang/rust/pull/43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](https://github.com/rust-lang/rust/pull/43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
* The warnings were not perfect, and revealed a few [spurious](https://github.com/rust-lang/rust/pull/44368) [differences](https://github.com/rust-lang/rust/pull/45421) between how we handled the renderers.
* Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](https://github.com/rust-lang/rust/pull/45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
* Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](https://github.com/rust-lang/rust/pull/47398), making Pulldown the default but still offering the option to use Hoedown.
And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Remove librustdoc dependency on log
This change should have been included in PR #46386.
Since librustdoc doesn't explicitly depend on internal crates
(such as librustc_driver) through its Cargo.toml, it ends up using the
sysroot to resolve them. By removing the dependency on 'log',
we ensure that the syroot is used to resolve `log` as well. This ensures
that only one version of log is in use, so that `env_logger::init()`
enables all uses of `log!` macros.
This change should have been included in PR #46386.
Since librustdoc doesn't explicitly depend on internal crates
(such as librustc_driver) through its Cargo.toml, it ends up using the
sysroot to resolve them. By removing the dependency on 'log',
we ensure that the syroot is used to resolve `log` as well. This ensures
that only one version of log is in use, so that `env_logger::init()`
enables all uses of `log!` macros.
We want librustdoc to pickup the env_logger dependency from
the sysroot. This ensures that the same copy of env_logger is used
for both internal crates (e.g. librustc_driver, libsyntax) and
librustdoc
Closes#46383
Oli Scherer
Guillaume Gomez
Joshua Nelson
klensy
bors
Rune Tynan
Andreas Jonson
Aleksey Kladov
Vadim Petrochenkov
Ohad Ravid
Andy Russell
Josh Stone
Mark Rousskov
Robert Collins
Hirokazu Hata
Bastien Orivel
QuietMisdreavus
Manish Goregaokar
Aaron Hill
Irina-Gabriela Popa
Ariel Ben-Yehuda