tests: Add htmldocck.py script for the use of Rustdoc tests.

The script is intended as a tool for doing every sort of verifications
amenable to Rustdoc's HTML output. For example, link checkers would go
to this script. It already parses HTML into a document tree form (with
a slight caveat), so future tests can make use of it.

As an example, relevant `rustdoc-*` run-make tests have been updated
to use `htmldocck.py` and got their `verify.sh` removed. In the future
they may go to a dedicated directory with htmldocck running by default.
The detailed explanation of test scripts is provided as a docstring of
htmldocck.

cc #19723
This commit is contained in:
Kang Seonghoon 2015-01-18 02:24:04 +09:00
parent ee2bfae011
commit de6f520192
16 changed files with 350 additions and 99 deletions

View File

@ -1011,7 +1011,8 @@ $(3)/test/run-make/%-$(1)-T-$(2)-H-$(3).ok: \
$$(LD_LIBRARY_PATH_ENV_NAME$(1)_T_$(2)_H_$(3)) \
"$$(LD_LIBRARY_PATH_ENV_HOSTDIR$(1)_T_$(2)_H_$(3))" \
"$$(LD_LIBRARY_PATH_ENV_TARGETDIR$(1)_T_$(2)_H_$(3))" \
$(1)
$(1) \
$$(S)
@touch $$@
else
# FIXME #11094 - The above rule doesn't work right for multiple targets

316
src/etc/htmldocck.py Normal file
View File

@ -0,0 +1,316 @@
# Copyright 2015 The Rust Project Developers. See the COPYRIGHT
# file at the top-level directory of this distribution and at
# http://rust-lang.org/COPYRIGHT.
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.
r"""
htmldocck.py is a custom checker script for Rustdoc HTML outputs.
# How and why?
The principle is simple: This script receives a path to generated HTML
documentation and a "template" script, which has a series of check
commands like `@has` or `@matches`. Each command can be used to check if
some pattern is present or not present in the particular file or in
the particular node of HTML tree. In many cases, the template script
happens to be a source code given to rustdoc.
While it indeed is possible to test in smaller portions, it has been
hard to construct tests in this fashion and major rendering errors were
discovered much later. This script is designed for making the black-box
and regression testing of Rustdoc easy. This does not preclude the needs
for unit testing, but can be used to complement related tests by quickly
showing the expected renderings.
In order to avoid one-off dependencies for this task, this script uses
a reasonably working HTML parser and the existing XPath implementation
from Python 2's standard library. Hopefully we won't render
non-well-formed HTML.
# Commands
Commands start with an `@` followed by a command name (letters and
hyphens), and zero or more arguments separated by one or more whitespace
and optionally delimited with single or double quotes. The `@` mark
cannot be preceded by a non-whitespace character. Other lines (including
every text up to the first `@`) are ignored, but it is recommended to
avoid the use of `@` in the template file.
There are a number of supported commands:
* `@has PATH` checks for the existence of given file.
`PATH` is relative to the output directory. It can be given as `-`
which repeats the most recently used `PATH`.
* `@has PATH PATTERN` and `@matches PATH PATTERN` checks for
the occurrence of given `PATTERN` in the given file. Only one
occurrence of given pattern is enough.
For `@has`, `PATTERN` is a whitespace-normalized (every consecutive
whitespace being replaced by one single space character) string.
The entire file is also whitespace-normalized including newlines.
For `@matches`, `PATTERN` is a Python-supported regular expression.
The file remains intact but the regexp is matched with no `MULTILINE`
and `IGNORECASE` option. You can still use a prefix `(?m)` or `(?i)`
to override them, and `\A` and `\Z` for definitely matching
the beginning and end of the file.
(The same distinction goes to other variants of these commands.)
* `@has PATH XPATH PATTERN` and `@matches PATH XPATH PATTERN` checks for
the presence of given `XPATH` in the given HTML file, and also
the occurrence of given `PATTERN` in the matching node or attribute.
Only one occurrence of given pattern in the match is enough.
`PATH` should be a valid and well-formed HTML file. It does *not*
accept arbitrary HTML5; it should have matching open and close tags
and correct entity references at least.
`XPATH` is an XPath expression to match. This is fairly limited:
`tag`, `*`, `.`, `//`, `..`, `[@attr]`, `[@attr='value']`, `[tag]`,
`[POS]` (element located in given `POS`), `[last()-POS]`, `text()`
and `@attr` (both as the last segment) are supported. Some examples:
- `//pre` or `.//pre` matches any element with a name `pre`.
- `//a[@href]` matches any element with an `href` attribute.
- `//*[@class="impl"]//code` matches any element with a name `code`,
which is an ancestor of some element which `class` attr is `impl`.
- `//h1[@class="fqn"]/span[1]/a[last()]/@class` matches a value of
`class` attribute in the last `a` element (can be followed by more
elements that are not `a`) inside the first `span` in the `h1` with
a class of `fqn`. Note that there cannot be no additional elements
between them due to the use of `/` instead of `//`.
Do not try to use non-absolute paths, it won't work due to the flawed
ElementTree implementation. The script rejects them.
For the text matches (i.e. paths not ending with `@attr`), any
subelements are flattened into one string; this is handy for ignoring
highlights for example. If you want to simply check the presence of
given node or attribute, use an empty string (`""`) as a `PATTERN`.
All conditions can be negated with `!`. `@!has foo/type.NoSuch.html`
checks if the given file does not exist, for example.
"""
import sys
import os.path
import re
import shlex
from collections import namedtuple
from HTMLParser import HTMLParser
from xml.etree import cElementTree as ET
# &larrb;/&rarrb; are not in HTML 4 but are in HTML 5
from htmlentitydefs import entitydefs
entitydefs['larrb'] = u'\u21e4'
entitydefs['rarrb'] = u'\u21e5'
# "void elements" (no closing tag) from the HTML Standard section 12.1.2
VOID_ELEMENTS = set(['area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'keygen',
'link', 'menuitem', 'meta', 'param', 'source', 'track', 'wbr'])
# simplified HTML parser.
# this is possible because we are dealing with very regular HTML from rustdoc;
# we only have to deal with i) void elements and ii) empty attributes.
class CustomHTMLParser(HTMLParser):
def __init__(self, target=None):
HTMLParser.__init__(self)
self.__builder = target or ET.TreeBuilder()
def handle_starttag(self, tag, attrs):
attrs = dict((k, v or '') for k, v in attrs)
self.__builder.start(tag, attrs)
if tag in VOID_ELEMENTS: self.__builder.end(tag)
def handle_endtag(self, tag):
self.__builder.end(tag)
def handle_startendtag(self, tag, attrs):
attrs = dict((k, v or '') for k, v in attrs)
self.__builder.start(tag, attrs)
self.__builder.end(tag)
def handle_data(self, data):
self.__builder.data(data)
def handle_entityref(self, name):
self.__builder.data(entitydefs[name])
def handle_charref(self, name):
code = int(name[1:], 16) if name.startswith(('x', 'X')) else int(name, 10)
self.__builder.data(unichr(code).encode('utf-8'))
def close(self):
HTMLParser.close(self)
return self.__builder.close()
Command = namedtuple('Command', 'negated cmd args lineno')
LINE_PATTERN = re.compile(r'(?<=(?<!\S)@)(?P<negated>!?)(?P<cmd>[A-Za-z]+(?:-[A-Za-z]+)*)(?P<args>.*)$')
def get_commands(template):
with open(template, 'rUb') as f:
for lineno, line in enumerate(f):
m = LINE_PATTERN.search(line.rstrip('\r\n'))
if not m: continue
negated = (m.group('negated') == '!')
cmd = m.group('cmd')
args = m.group('args')
if args and not args[:1].isspace():
raise RuntimeError('Invalid template syntax at line {}'.format(lineno+1))
args = shlex.split(args)
yield Command(negated=negated, cmd=cmd, args=args, lineno=lineno+1)
def _flatten(node, acc):
if node.text: acc.append(node.text)
for e in node:
_flatten(e, acc)
if e.tail: acc.append(e.tail)
def flatten(node):
acc = []
_flatten(node, acc)
return ''.join(acc)
def normalize_xpath(path):
if path.startswith('//'):
return '.' + path # avoid warnings
elif path.startswith('.//'):
return path
else:
raise RuntimeError('Non-absolute XPath is not supported due to \
the implementation issue.')
class CachedFiles(object):
def __init__(self, root):
self.root = root
self.files = {}
self.trees = {}
self.last_path = None
def resolve_path(self, path):
if path != '-':
path = os.path.normpath(path)
self.last_path = path
return path
elif self.last_path is None:
raise RuntimeError('Tried to use the previous path in the first command')
else:
return self.last_path
def get_file(self, path):
path = self.resolve_path(path)
try:
return self.files[path]
except KeyError:
try:
with open(os.path.join(self.root, path)) as f:
data = f.read()
except Exception as e:
raise RuntimeError('Cannot open file {!r}: {}'.format(path, e))
else:
self.files[path] = data
return data
def get_tree(self, path):
path = self.resolve_path(path)
try:
return self.trees[path]
except KeyError:
try:
f = open(os.path.join(self.root, path))
except Exception as e:
raise RuntimeError('Cannot open file {!r}: {}'.format(path, e))
try:
with f:
tree = ET.parse(f, CustomHTMLParser())
except Exception as e:
raise RuntimeError('Cannot parse an HTML file {!r}: {}'.format(path, e))
else:
self.trees[path] = tree
return self.trees[path]
def check_string(data, pat, regexp):
if not pat:
return True # special case a presence testing
elif regexp:
return re.search(pat, data) is not None
else:
data = ' '.join(data.split())
pat = ' '.join(pat.split())
return pat in data
def check_tree_attr(tree, path, attr, pat, regexp):
path = normalize_xpath(path)
ret = False
for e in tree.findall(path):
try:
value = e.attrib[attr]
except KeyError:
continue
else:
ret = check_string(value, pat, regexp)
if ret: break
return ret
def check_tree_text(tree, path, pat, regexp):
path = normalize_xpath(path)
ret = False
for e in tree.findall(path):
try:
value = flatten(e)
except KeyError:
continue
else:
ret = check_string(value, pat, regexp)
if ret: break
return ret
def check(target, commands):
cache = CachedFiles(target)
for c in commands:
if c.cmd == 'has' or c.cmd == 'matches': # string test
regexp = (c.cmd == 'matches')
if len(c.args) == 1 and not regexp: # @has <path> = file existence
try:
cache.get_file(c.args[0])
ret = True
except RuntimeError:
ret = False
elif len(c.args) == 2: # @has/matches <path> <pat> = string test
ret = check_string(cache.get_file(c.args[0]), c.args[1], regexp)
elif len(c.args) == 3: # @has/matches <path> <pat> <match> = XML tree test
tree = cache.get_tree(c.args[0])
pat, sep, attr = c.args[1].partition('/@')
if sep: # attribute
ret = check_tree_attr(cache.get_tree(c.args[0]), pat, attr, c.args[2], regexp)
else: # normalized text
pat = c.args[1]
if pat.endswith('/text()'): pat = pat[:-7]
ret = check_tree_text(cache.get_tree(c.args[0]), pat, c.args[2], regexp)
else:
raise RuntimeError('Invalid number of @{} arguments \
at line {}'.format(c.cmd, c.lineno))
elif c.cmd == 'valid-html':
raise RuntimeError('Unimplemented @valid-html at line {}'.format(c.lineno))
elif c.cmd == 'valid-links':
raise RuntimeError('Unimplemented @valid-links at line {}'.format(c.lineno))
else:
raise RuntimeError('Unrecognized @{} at line {}'.format(c.cmd, c.lineno))
if ret == c.negated:
raise RuntimeError('@{}{} check failed at line {}'.format('!' if c.negated else '',
c.cmd, c.lineno))
if __name__ == '__main__':
if len(sys.argv) < 3:
print >>sys.stderr, 'Usage: {} <doc dir> <template>'.format(sys.argv[0])
raise SystemExit(1)
else:
check(sys.argv[1], get_commands(sys.argv[2]))

View File

@ -46,6 +46,8 @@ putenv('LD_LIB_PATH_ENVVAR', sys.argv[8]);
putenv('HOST_RPATH_DIR', os.path.abspath(sys.argv[9]));
putenv('TARGET_RPATH_DIR', os.path.abspath(sys.argv[10]));
putenv('RUST_BUILD_STAGE', sys.argv[11])
putenv('S', os.path.abspath(sys.argv[12]))
putenv('PYTHON', sys.executable)
if not filt in sys.argv[1]:
sys.exit(0)

View File

@ -7,8 +7,7 @@ all:
@echo $(RUSTDOC)
$(HOST_RPATH_ENV) $(RUSTDOC) --test foo.rs
$(HOST_RPATH_ENV) $(RUSTDOC) -w html -o $(TMPDIR)/doc foo.rs
cp verify.sh $(TMPDIR)
$(call RUN,verify.sh) $(TMPDIR)
$(HTMLDOCCK) $(TMPDIR)/doc foo.rs
else
all:

View File

@ -30,3 +30,7 @@
/// }
/// ```
pub fn foo() {}
// @!has foo/fn.foo.html invisible
// @matches - //pre '#.*\[.*derive.*\(.*Eq.*\).*\].*//.*Bar'

View File

@ -1,8 +0,0 @@
#!/bin/sh
file="$1/doc/foo/fn.foo.html"
grep -v 'invisible' $file &&
grep '#.*\[.*derive.*(.*Eq.*).*\].*//.*Bar' $file
exit $?

View File

@ -7,9 +7,7 @@ source=index.rs
all:
$(HOST_RPATH_ENV) $(RUSTDOC) -w html -o $(TMPDIR)/doc $(source)
cp $(source) $(TMPDIR)
cp verify.sh $(TMPDIR)
$(call RUN,verify.sh) $(TMPDIR)
$(HTMLDOCCK) $(TMPDIR)/doc $(source)
else
all:

View File

@ -10,20 +10,17 @@
#![crate_name = "rustdoc_test"]
// In: Foo
// @has search-index.js Foo
pub use private::Foo;
mod private {
pub struct Foo;
impl Foo {
// In: test_method
pub fn test_method() {}
// Out: priv_method
fn priv_method() {}
pub fn test_method() {} // @has - test_method
fn priv_method() {} // @!has - priv_method
}
pub trait PrivateTrait {
// Out: priv_method
fn trait_method() {}
fn trait_method() {} // @!has - priv_method
}
}

View File

@ -1,33 +0,0 @@
#!/bin/sh
source="$1/index.rs"
index="$1/doc/search-index.js"
if ! [ -e $index ]
then
echo "Could not find the search index (looked for $index)"
exit 1
fi
ins=$(grep -o 'In: .*' $source | sed 's/In: \(.*\)/\1/g')
outs=$(grep -o 'Out: .*' $source | sed 's/Out: \(.*\)/\1/g')
for p in $ins
do
if ! grep -q $p $index
then
echo "'$p' was erroneously excluded from search index."
exit 1
fi
done
for p in $outs
do
if grep -q $p $index
then
echo "'$p' was erroneously included in search index."
exit 1
fi
done
exit 0

View File

@ -1,5 +1,4 @@
-include ../tools.mk
all:
$(HOST_RPATH_ENV) $(RUSTDOC) -w html -o $(TMPDIR)/doc foo.rs
cp verify.sh $(TMPDIR)
$(call RUN,verify.sh) $(TMPDIR)
$(HTMLDOCCK) $(TMPDIR)/doc foo.rs

View File

@ -8,22 +8,29 @@
// option. This file may not be copied, modified, or distributed
// except according to those terms.
// @has foo/index.html
#![crate_name = "foo"]
//! Very docs
// @has foo/bar/index.html
pub mod bar {
/// So correct
// @has foo/bar/baz/index.html
pub mod baz {
/// Much detail
// @has foo/bar/baz/fn.baz.html
pub fn baz() { }
}
/// *wow*
// @has foo/bar/trait.Doge.html
pub trait Doge { }
// @has foo/bar/struct.Foo.html
pub struct Foo { x: int, y: uint }
// @has foo/bar/fn.prawns.html
pub fn prawns((a, b): (int, uint), Foo { x, y }: Foo) { }
}

View File

@ -1,17 +0,0 @@
#!/bin/sh
# $1 is the TMPDIR
dirs="doc doc/foo doc/foo/bar doc/foo/bar/baz doc/src doc/src/foo"
for dir in $dirs; do if [ ! -d $1/$dir ]; then
echo "$1/$dir is not a directory!"
exit 1
fi done
files="doc/foo/index.html doc/foo/bar/index.html doc/foo/bar/baz/fn.baz.html doc/foo/bar/trait.Doge.html doc/src/foo/foo.rs.html"
for file in $files; do if [ ! -f $1/$file ]; then
echo "$1/$file is not a file!"
exit 1
fi done

View File

@ -1,6 +1,6 @@
-include ../tools.mk
all: verify.sh foo.rs
all: foo.rs
$(HOST_RPATH_ENV) $(RUSTDOC) -w html -o $(TMPDIR)/doc foo.rs
cp verify.sh $(TMPDIR)
$(call RUN,verify.sh) $(TMPDIR)
$(HTMLDOCCK) $(TMPDIR)/doc foo.rs

View File

@ -10,17 +10,25 @@
pub trait MyTrait {}
// @matches foo/struct.Alpha.html '//pre' "Alpha.*where.*A:.*MyTrait"
pub struct Alpha<A> where A: MyTrait;
// @matches foo/trait.Bravo.html '//pre' "Bravo.*where.*B:.*MyTrait"
pub trait Bravo<B> where B: MyTrait {}
// @matches foo/fn.charlie.html '//pre' "charlie.*where.*C:.*MyTrait"
pub fn charlie<C>() where C: MyTrait {}
pub struct Delta<D>;
// @matches foo/struct.Delta.html '//*[@class="impl"]//code' "impl.*Delta.*where.*D:.*MyTrait"
impl<D> Delta<D> where D: MyTrait {
pub fn delta() {}
}
pub struct Echo<E>;
// @matches foo/struct.Echo.html '//*[@class="impl"]//code' "impl.*MyTrait.*for.*Echo.*where.*E:.*MyTrait"
// @matches foo/trait.MyTrait.html '//*[@id="implementors-list"]//code' "impl.*MyTrait.*for.*Echo.*where.*E:.*MyTrait"
impl<E> MyTrait for Echo<E> where E: MyTrait {}
pub enum Foxtrot<F> {}
// @matches foo/enum.Foxtrot.html '//*[@class="impl"]//code' "impl.*MyTrait.*for.*Foxtrot.*where.*F:.*MyTrait"
// @matches foo/trait.MyTrait.html '//*[@id="implementors-list"]//code' "impl.*MyTrait.*for.*Foxtrot.*where.*F:.*MyTrait"
impl<F> MyTrait for Foxtrot<F> where F: MyTrait {}

View File

@ -1,23 +0,0 @@
#!/bin/sh
set -e
# $1 is the TMPDIR
DOC=$1/doc/foo
grep "Alpha.*where.*A:.*MyTrait" $DOC/struct.Alpha.html > /dev/null
echo "Alpha"
grep "Bravo.*where.*B:.*MyTrait" $DOC/trait.Bravo.html > /dev/null
echo "Bravo"
grep "charlie.*where.*C:.*MyTrait" $DOC/fn.charlie.html > /dev/null
echo "Charlie"
grep "impl.*Delta.*where.*D:.*MyTrait" $DOC/struct.Delta.html > /dev/null
echo "Delta"
grep "impl.*MyTrait.*for.*Echo.*where.*E:.*MyTrait" $DOC/struct.Echo.html > /dev/null
echo "Echo"
grep "impl.*MyTrait.*for.*Foxtrot.*where.*F:.*MyTrait" $DOC/enum.Foxtrot.html > /dev/null
echo "Foxtrot"
# check "Implementors" section of MyTrait
grep "impl.*MyTrait.*for.*Echo.*where.*E:.*MyTrait" $DOC/trait.MyTrait.html > /dev/null
grep "impl.*MyTrait.*for.*Foxtrot.*where.*F:.*MyTrait" $DOC/trait.MyTrait.html > /dev/null
echo "Implementors OK"

View File

@ -7,6 +7,7 @@ TARGET_RPATH_ENV = \
RUSTC := $(HOST_RPATH_ENV) $(RUSTC) --out-dir $(TMPDIR) -L $(TMPDIR)
CC := $(CC) -L $(TMPDIR)
HTMLDOCCK := $(PYTHON) $(S)/src/etc/htmldocck.py
# This is the name of the binary we will generate and run; use this
# e.g. for `$(CC) -o $(RUN_BINFILE)`.