4 Commits

Author	SHA1	Message	Date
Aleksey Kladov	a0c186c34f	remove XID and Pattern_White_Space unicode tables from libcore ... They are only used by rustc_lexer, and are not needed elsewhere. So we move the relevant definitions into rustc_lexer (while the actual unicode data comes from the unicode-xid crate) and make the rest of the compiler use it.	2019-09-04 13:11:11 +03:00
topecongiro	98f29f5e38	Add comment	2019-07-27 22:10:09 +09:00
topecongiro	13b41000ea	Add lib section to rustc_lexer's Cargo.toml	2019-07-27 17:21:42 +09:00
Aleksey Kladov	395ee0b79f	Introduce rustc_lexer ... The idea here is to make a reusable library out of the existing rust-lexer, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token.	2019-07-20 21:12:34 +03:00