Snapshot testing is a technique for writing maintainable unit tests.
Unlike usual `assert_eq!` tests, snapshot tests allow
to *automatically* upgrade expected values on test failure.
In a sense, snapshot tests are inline-version of our beloved
UI-tests.
Example:
A particular library we use, `expect_test` provides an `expect!`
macro, which creates a sort of self-updating string literal (by using
`file!` macro). Self-update is triggered by setting `UPDATE_EXPECT`
environmental variable (this info is printed during the test failure).
This library was extracted from rust-analyzer, where we use it for
most of our tests.
There are some other, more popular snapshot testing libraries:
* https://github.com/mitsuhiko/insta
* https://github.com/aaronabramov/k9
The main differences of `expect` are:
* first-class snapshot objects (so, tests can be written as functions,
rather than as macros)
* focus on inline-snapshots (but file snapshots are also supported)
* restricted feature set (only `assert_eq` and `assert_debug_eq`)
* no extra runtime (ie, no `cargo insta`)
See https://github.com/rust-analyzer/rust-analyzer/pull/5101 for a
an extended comparison.
It is unclear if this testing style will stick with rustc in the long
run. At the moment, rustc is mainly tested via integrated UI tests.
But in the library-ified world, unit-tests will become somewhat more
important (that's why use use `rustc_lexer` library-ified library as
an example in this PR). Given that the cost of removal shouldn't be
too high, it probably makes sense to just see if this flies!
Move doc comment parsing to rustc_lexer
Plain comments are trivia, while doc comments are not, so it feels
like this belongs to the rustc_lexer.
The specific reason to do this is the desire to use rustc_lexer in
rustdoc for syntax highlighting, without duplicating "is this a doc
comment?" logic there.
r? @ghost
All other tokens are named by the punctuation they use, rather than
by semantics operation they stand for. `!` is the only exception to
the rule, let's fix it.
Plain comments are trivial, while doc comments are not, so it feels
like this belongs to the rustc_lexer.
The specific reason to do this is the desire to use rustc_lexer in
rustdoc for syntax highlighting, without duplicating "is this a doc
comment?" logic there.
Shebang handling was too agressive in stripping out the first line in cases where it is actually _not_ a shebang, but instead, valid rust (#70528). This is a second attempt at resolving this issue (the first attempt was flawed, for, among other reasons, causing an ICE in certain cases (#71372, #71471).
The behavior is now codified by a number of UI tests, but simply:
For the first line to be a shebang, the following must all be true:
1. The line must start with `#!`
2. The line must contain a non whitespace character after `#!`
3. The next character in the file, ignoring comments & whitespace must not be `[`
I believe this is a strict superset of what we used to allow, so perhaps a crater run is unnecessary, but probably not a terrible idea.
The modified code to handle parsing raw strings didn't properly account for the case where there was no "#" on either end and erroneously reported this strings as complete. This lead to a panic trying to read off the end of the file.
This diff improves error messages around raw strings in a few ways:
- Catch extra trailing `#` in the parser. This can't be handled in the lexer because we could be in a macro that actually expects another # (see test)
- Refactor & unify error handling in the lexer between ByteStrings and RawByteStrings
- Detect potentially intended terminators (longest sequence of "#*" is suggested)
They are only used by rustc_lexer, and are not needed elsewhere.
So we move the relevant definitions into rustc_lexer (while the actual
unicode data comes from the unicode-xid crate) and make the rest of
the compiler use it.
add rustc_private as a proper language feature gate
At the moment, `rustc_private` as a (library) feature exists by
accident: `char::is_xid_start`, `char::is_xid_continue` methods in
libcore define it.
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/How.20to.20declare.20new.20langauge.20feature.3F
I don't know if this is at all reasonable, but at least tests seem to pass locally. That probably means that we can remove/rename to something more resonable the feature in libcore in the next release?
The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.
So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
Aleksey Kladov
bors
Vadim Petrochenkov
Manish Goregaokar
pierwill
Matthias Krüger
Julian Wollersberger
Russell Cohen
Eduard-Mihai Burtescu
Ayush Kumar Mishra
Drew Ripberger
Mark Rousskov
Mazdak Farrokhzad
Igor Aleksanov
