TestingISO C++testtestsuiteperformanceconformanceABIexception safety
The libstdc++ testsuite includes testing for standard conformance,
regressions, ABI, and performance.
Test OrganizationDirectory Layout
The directory
gccsrcdir/libstdc++-v3/testsuite
contains the individual test cases organized in sub-directories
corresponding to clauses of the C++ standard (detailed below),
the DejaGnu test harness support files, and sources to various
testsuite utilities that are packaged in a separate testing library.
All test cases for functionality required by the runtime components
of the C++ standard (ISO 14882) are files within the following
directories:
17_intro
18_support
19_diagnostics
20_util
21_strings
22_locale
23_containers
24_iterators
25_algorithms
26_numerics
27_io
28_regex
29_atomics
30_threads
In addition, the following directories include test files:
tr1Tests for components as described by the Technical Report
on Standard Library Extensions (TR1).
backwardTests for backwards compatibility and deprecated features.
demangleTests for __cxa_demangle, the IA-64 C++ ABI
demangler.
extTests for extensions.performanceTests for performance analysis, and performance regressions.
Some directories don't have test files, but instead contain
auxiliary information:
configFiles for the DejaGnu test harness.libFiles for the DejaGnu test harness.libstdc++*Files for the DejaGnu test harness.dataSample text files for testing input and output.utilFiles for libtestc++, utilities and testing routines.
Within a directory that includes test files, there may be
additional subdirectories, or files. Originally, test cases
were appended to one file that represented a particular section
of the chapter under test, and was named accordingly. For
instance, to test items related to 21.3.6.1 -
basic_string::find [lib.string::find]
in the standard, the following was used:
21_strings/find.cc
However, that practice soon became a liability as the test cases
became huge and unwieldy, and testing new or extended
functionality (like wide characters or named locales) became
frustrating, leading to aggressive pruning of test cases on some
platforms that covered up implementation errors. Now, the test
suite has a policy of one file, one test case, which solves the
above issues and gives finer grained results and more manageable
error debugging. As an example, the test case quoted above
becomes:
21_strings/basic_string/find/char/1.cc
21_strings/basic_string/find/char/2.cc
21_strings/basic_string/find/char/3.cc
21_strings/basic_string/find/wchar_t/1.cc
21_strings/basic_string/find/wchar_t/2.cc
21_strings/basic_string/find/wchar_t/3.cc
All new tests should be written with the policy of "one test
case, one file" in mind.
Naming Conventions
In addition, there are some special names and suffixes that are
used within the testsuite to designate particular kinds of
tests.
_xin.cc
This test case expects some kind of interactive input in order
to finish or pass. At the moment, the interactive tests are not
run by default. Instead, they are run by hand, like:
g++ 27_io/objects/char/3_xin.cc
cat 27_io/objects/char/3_xin.in | a.out.in
This file contains the expected input for the corresponding
_xin.cc test case.
_neg.cc
This test case is expected to fail: it's a negative test. At the
moment, these are almost always compile time errors.
char
This can either be a directory name or part of a longer file
name, and indicates that this file, or the files within this
directory are testing the char instantiation of a
template.
wchar_t
This can either be a directory name or part of a longer file
name, and indicates that this file, or the files within this
directory are testing the wchar_t instantiation of
a template. Some hosts do not support wchar_t
functionality, so for these targets, all of these tests will not
be run.
thread
This can either be a directory name or part of a longer file
name, and indicates that this file, or the files within this
directory are testing situations where multiple threads are
being used.
performance
This can either be an enclosing directory name or part of a
specific file name. This indicates a test that is used to
analyze runtime performance, for performance regression testing,
or for other optimization related analysis. At the moment, these
test cases are not run by default.
Running the TestsuiteBasic
You can check the status of the build without installing it
using the DejaGnu harness, much like the rest of the gcc
tools, i.e.
make check
in the
libbuilddir
directory, or
make check-target-libstdc++-v3
in the
gccbuilddir
directory.
These commands are functionally equivalent and will create a
'testsuite' directory underneath
libbuilddir
containing the results of the
tests. Two results files will be generated:
libstdc++.sum, which is a PASS/FAIL summary
for each test, and
libstdc++.log which is a log of
the exact command-line passed to the compiler, the compiler
output, and the executable output (if any) for each test.
Archives of test results for various versions and platforms are
available on the GCC website in the build
status section of each individual release, and are also
archived on a daily basis on the gcc-testresults
mailing list. Please check either of these places for a similar
combination of source version, operating system, and host CPU.
Variations
There are several options for running tests, including testing
the regression tests, testing a subset of the regression tests,
testing the performance tests, testing just compilation, testing
installed tools, etc. In addition, there is a special rule for
checking the exported symbols of the shared library.
To debug the DejaGnu test harness during runs, try invoking with a
specific argument to the variable RUNTESTFLAGS,
like so:
make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
or
make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
To run a subset of the library tests, you can either generate the
testsuite_files file (described below) by running
make testsuite_files in the
libbuilddir/testsuite
directory, then edit the
file to remove the tests you don't want and then run the testsuite as
normal, or you can specify a testsuite and a subset of tests in the
RUNTESTFLAGS variable.
For example, to run only the tests for containers you could use:
make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*"
When combining this with other options in RUNTESTFLAGS
the options must come first.
There are two ways to run on a simulator: set up DEJAGNU
to point to a specially crafted site.exp,
or pass down flags.
Example flags to pass down for various embedded builds are as follows:
--target=powerpc-eabisim (libgloss/sim)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
--target=calmrisc32 (libgloss/sid)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
--target=xscale-elf (newlib/sim)
make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
Also, here is an example of how to run the libstdc++ testsuite
for a multilibed build directory with different ABI settings:
make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
You can run the tests with a compiler and library that have
already been installed. Make sure that the compiler (e.g.,
g++) is in your PATH. If you are
using shared libraries, then you must also ensure that the
directory containing the shared version of libstdc++ is in your
LD_LIBRARY_PATH, or
equivalent.
If your GCC source tree is at
/path/to/gcc,
then you can run the tests as follows:
runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
The testsuite will create a number of files in the directory in
which you run this command,. Some of those files might use the
same name as files created by other testsuites (like the ones
for GCC and G++), so you should not try to run all the
testsuites in parallel from the same directory.
In addition, there are some testing options that are mostly of
interest to library maintainers and system integrators. As such,
these tests may not work on all CPU and host combinations, and
may need to be executed in the
libbuilddir/testsuite
directory. These
options include, but are not necessarily limited to, the
following:
make testsuite_files
>
Five files are generated that determine what test files
are run. These files are:
testsuite_files
This is a list of all the test cases that will be run. Each
test case is on a separate line, given with an absolute path
from the
libsrcdir/testsuite
directory.
testsuite_files_interactive
This is a list of all the interactive test cases, using the
same format as the file list above. These tests are not run
by default.
testsuite_files_performance
This is a list of all the performance test cases, using the
same format as the file list above. These tests are not run
by default.
testsuite_thread
This file indicates that the host system can run tests which
involved multiple threads.
testsuite_wchar_t
This file indicates that the host system can run the
wchar_t tests, and corresponds to the macro
definition _GLIBCXX_USE_WCHAR_T in the
file c++config.h.
make check-abi
>
The library ABI can be tested. This involves testing the shared
library against a baseline list of symbol exports that defines the
previous version of the ABI. The tests require that no exported
symbols are removed, no new symbols are added to the old symbol
versions, and any new symbols have the latest symbol version.
See Versioning for more details
of the ABI version history.
make new-abi-baseline
>
Generate a new baseline set of symbols exported from the library
(written to a file under
libsrcdir/config/abi/post/target/).
A different baseline symbols file is needed for each architecture and
is used by the check-abi target described above.
The files are usually re-generated by target maintainers for releases.
make check-compile
>
This rule compiles, but does not link or execute, the
testsuite_files test cases and displays the
output on stdout.
make check-performance
>
This rule runs through the
testsuite_files_performance test cases and
collects information for performance analysis and can be used to
spot performance regressions. Various timing information is
collected, as well as number of hard page faults, and memory
used. This is not run by default, and the implementation is in
flux.
make check-debug
>
This rule runs through the test suite under the
debug mode.
make check-parallel
>
This rule runs through the test suite under the
parallel mode.
We are interested in any strange failures of the testsuite;
please email the main libstdc++ mailing list if you see
something odd or have questions.
Permutations
The tests will be compiled with a set of default compiler flags defined
by the
libbuilddir/scripts/testsuite_flags
file, as well as options specified in individual tests. You can run
the tests with different options by adding them to the output of
the option of that script, or by setting
the CXXFLAGS variable when running
make, or via options for the DejaGnu test framework
(described below). The latter approach uses the
option that was shown earlier,
but requires DejaGnu version 1.5.3 or newer to work reliably, so that the
dg-options in the test aren't overridden.
For example, to run the tests with
you could use:
make check RUNTESTFLAGS=--target_board=unix/-O1/-D_GLIBCXX_ASSERTIONS
The option can also be used to run the
tests multiple times in different variations. For example, to run the
entire testsuite three times using but with
different options:
make check 'RUNTESTFLAGS=--target_board=unix/-O3\"{-std=gnu++98,-std=gnu++11,-std=gnu++14}\"'
N.B. that set of variations could also be written as
unix/-O3\"{-std=gnu++98,-std=gnu++11,}\" so that
the third variation would use the default for
(which is as of GCC 6).
To run the libstdc++ test suite under the
debug mode, use
make check-debug. Alternatively, edit
libbuilddir/scripts/testsuite_flags
to add the compile-time flag to the
result printed by the
option. Additionally, add the
flag to turn on
pedantic checking. The libstdc++ test suite should produce
the same results under debug mode that it does under release mode:
any deviation indicates an error in either the library or the test suite.
Note, however, that the number of tests that PASS may change, because
some test cases are skipped in normal mode, and some are skipped in
debug mode, as determined by the
dg-require-support
directives described below.
The parallel
mode can be tested using
make check-parallel, or in much the same manner
as the debug mode, substituting
for
in the previous paragraph.
Or, just run the testsuite
or
in CXXFLAGS or RUNTESTFLAGS.
Writing a new test case
The first step in making a new test case is to choose the correct
directory and file name, given the organization as previously
described.
Historically all test files were copyright the FSF, and GPL licensed.
We no longer require that, because most tests are uninteresting
and contain no "original authorship", and so would not be protected
by copyright anyway.
If you do want to add the FSF copyright notice and GPL licence text,
then the first copyright year should correspond to the date
the file was checked in to version control. If a test is copied from
an existing file it should retain the copyright years from the
original file.
The DejaGnu instructions say to always return 0
from main to indicate success. Strictly speaking
this is redundant in C++, since returning from main
is defined to return 0. Most tests still have an
explicit return.
A bunch of utility functions and classes have already been
abstracted out into the testsuite utility library,
libtestc++. To use this functionality, just include the
appropriate header file: the library or specific object files will
automatically be linked in as part of the testsuite run.
Tests that need to perform runtime checks should use the
VERIFY macro, defined in the
<testsuite_hooks.h> header.
This expands to a custom assertion using
__builtin_printf and
__builtin_abort
(to avoid using assert and being affected by
NDEBUG).
Prior to GCC 7.1, VERIFY was defined differently.
It usually expanded to the standard assert macro, but
allowed targets to define it to something different. In order to support
the alternative expansions of VERIFY, before any use
of the macro there needed to be a variable called test
in scope, which was usually defined like so (the attribute avoids
warnings about an unused variable):
bool test __attribute__((unused)) = true;
This is no longer needed, and should not be added to new tests.
The testsuite uses the DejaGnu framework to compile and run the tests.
Test cases are normal C++ files which contain special directives in
comments. These directives look like { dg-* ... }
and tell DejaGnu what to do and what kinds of behavior are to be expected
for a test. The core DejaGnu directives are documented in the
dg.exp file installed by DejaGnu.
The GCC testsuites support additional directives
as described in the GCC internals documentation, see Syntax
and Descriptions of test directives. GCC also defines many
Keywords describing target attributes (a.k.a effective targets)
which can be used where a target selector can
appear.
Some directives commonly used in the libstdc++ testsuite are:
{ dg-do do-what-keyword [{ target/xfail selector }] }Where do-what-keyword is usually
one of run (which is the default),
compile, or link,
and typical selectors are targets such as *-*-gnu*
or an effective target such as c++11.
{ dg-require-support args }Skip the test if the target does not provide the required support.
See below for values of support.
{ dg-options options [{ target selector }] }{ dg-error regexp [ comment [{ target/xfail selector } [line] ]] }{ dg-excess-errors comment [{ target/xfail selector }] }
For full details of these and other directives see the main GCC DejaGnu
documentation in the internals manual.
Test cases that use features of a particular C++ standard should specify
the minimum required standard as an effective target:
// { dg-do run { target c++11 } }
or
// { dg-require-effective-target c++11 }
Specifying the minimum required standard for a test allows it to be run
using later standards, so that we can verify that C++11 components still
work correctly when compiled as C++14 or later. Specifying a minimum also
means the test will be skipped if the test is compiled using
an older standard, e.g. using
.
It is possible to indicate that a test should only
be run for a specific standard (and not later standards) using an
effective target like c++11_only. However, this means
the test will be skipped by default (because the default mode is
gnu++14), and so will only run when
or is used
explicitly. For tests that require a specific standard it is better to
use a dg-options directive:
// { dg-options "-std=gnu++11" }
This means the test will not get skipped by default, and will always use
the specific standard dialect that the test requires. This isn't needed
often, and most tests should use an effective target to specify a
minimum standard instead, to allow them to be tested for all
possible variations.
Similarly, tests which depend on a newer standard than the default
must use dg-options instead of (or in addition to)
an effective target, so that they are not skipped by default.
For example, tests for C++17 features should use
// { dg-options "-std=gnu++17" }
before any dg-do such as:
// { dg-do run "c++17" }
The dg-options directive must come first, so that
the -std flag has already been added to the options
before checking the c++17 target.
Examples of Test Directives
Example 1: Testing compilation only:
// { dg-do compile }
Example 2: Testing for expected warnings on line 36, which all targets fail:
// { dg-warning "string literals" "" { xfail *-*-* } 36 }
Example 3: Testing for expected warnings on line 36:
// { dg-warning "string literals" "" { target *-*-* } 36 }
Example 4: Testing for compilation errors on line 41:
// { dg-do compile }
// { dg-error "no match for" "" { target *-*-* } 41 }
Example 5: Testing with special command line settings, or without the
use of pre-compiled headers, in particular the
stdc++.h.gch file. Any
options here will override the DEFAULT_CXXFLAGS and
PCH_CXXFLAGS set up in the normal.exp
file:
// { dg-options "-O0" { target *-*-* } }
Example 6: Compiling and linking a test only for C++14 and later, and only
if Debug Mode is active:
// { dg-do link { target c++14 } }
// { dg-require-debug-mode "" }
Example 7: Running a test only on x86 targets, and only for C++11 and later,
with specific options, and additional options for 32-bit x86:
// { dg-options "-fstrict-enums" }
// { dg-additional-options "-march=i486" { target ia32 } }
// { dg-do run { target { ia32 || x86_64-*-* } } }
// { dg-require-effective-target "c++11" }
More examples can be found in the
libstdc++-v3/testsuite/*/*.cc files.
Directives Specific to Libstdc++ Tests
In addition to the usual Variants
of dg-require-support
several more directives are available for use in libstdc++ tests,
including the following:
dg-require-namedlocalenameThe named locale must be available.
dg-require-debug-mode ""Skip the test if the Debug Mode is not active
(as determined by the _GLIBCXX_DEBUG macro).
dg-require-parallel-mode ""Skip the test if the Parallel Mode is not active
(as determined by the _GLIBCXX_PARALLEL macro).
dg-require-normal-mode ""Skip the test if Debug or Parallel Mode is active.
dg-require-atomic-builtins ""Skip the test if atomic operations on bool
and int are not lock-free.
dg-require-gthreads ""Skip the test if the C++11 thread library is not
supported, as determined by the _GLIBCXX_HAS_GTHREADS
macro.
dg-require-gthreads-timed ""Skip the test if C++11 timed mutexes are not supported,
as determined by the _GLIBCXX_HAS_GTHREADS and
_GTHREAD_USE_MUTEX_TIMEDLOCK macros.
dg-require-string-conversions ""Skip the test if the C++11 to_string
and stoi, stod etc. functions
are not fully supported (including wide character versions).
dg-require-filesystem-ts ""Skip the test if the Filesystem TS is not supported.
Test Harness and UtilitiesDejaGnu Harness Details
Underlying details of testing for conformance and regressions are
abstracted via the GNU DejaGnu package. This is similar to the
rest of GCC.
This is information for those looking at making changes to the testsuite
structure, and/or needing to trace DejaGnu's actions with
.
This will not be useful to people who are "merely" adding new tests
to the existing structure.
The first key point when working with DejaGnu is the idea of a "tool".
Files, directories, and functions are all implicitly used when they are
named after the tool in use. Here, the tool will always be "libstdc++".
The lib subdir contains support routines. The
lib/libstdc++.exp file ("support library") is loaded
automagically, and must explicitly load the others. For example, files can
be copied from the core compiler's support directory into lib.
Some routines in lib/libstdc++.exp are callbacks, some are
our own. Callbacks must be prefixed with the name of the tool. To easily
distinguish the others, by convention our own routines are named "v3-*".
The next key point when working with DejaGnu is "test files". Any
directory whose name starts with the tool name will be searched for test files.
(We have only one.) In those directories, any .exp file is
considered a test file, and will be run in turn. Our main test file is called
normal.exp; it runs all the tests in testsuite_files using the
callbacks loaded from the support library.
The config directory is searched for any particular "target
board" information unique to this library. This is currently unused and sets
only default variables.
Utilities
The testsuite directory also contains some files that implement
functionality that is intended to make writing test cases easier,
or to avoid duplication, or to provide error checking in a way that
is consistent across platforms and test harnesses. A stand-alone
executable, called abi_check, and a static
library called libtestc++ are
constructed. Both of these items are not installed, and only used
during testing.
These files include the following functionality:
testsuite_abi.h,
testsuite_abi.cc,
testsuite_abi_check.cc
Creates the executable abi_check.
Used to check correctness of symbol versioning, visibility of
exported symbols, and compatibility on symbols in the shared
library, for hosts that support this feature. More information
can be found in the ABI documentation here
testsuite_allocator.h,
testsuite_allocator.cc
Contains specialized allocators that keep track of construction
and destruction. Also, support for overriding global new and
delete operators, including verification that new and delete
are called during execution, and that allocation over max_size
fails.
testsuite_character.h
Contains std::char_traits and
std::codecvt specializations for a user-defined
POD.
testsuite_hooks.h,
testsuite_hooks.cc
A large number of utilities, including:
VERIFYset_memory_limitsverify_demanglerun_tests_wrapped_localerun_tests_wrapped_envtry_named_localetry_mkfifofunc_callbackcountercopy_trackercopy_constructorassignment_operatordestructorpod_char, pod_int and associated char_traits specializationstestsuite_io.h
Error, exception, and constraint checking for
std::streambuf, std::basic_stringbuf, std::basic_filebuf.
testsuite_iterators.h
Wrappers for various iterators.
testsuite_performance.h
A number of class abstractions for performance counters, and
reporting functions including:
time_counterresource_counterreport_performanceSpecial Topics
Qualifying Exception Safety Guarantees
TestException SafetyOverview
Testing is composed of running a particular test sequence,
and looking at what happens to the surrounding code when
exceptions are thrown. Each test is composed of measuring
initial state, executing a particular sequence of code under
some instrumented conditions, measuring a final state, and
then examining the differences between the two states.
Test sequences are composed of constructed code sequences
that exercise a particular function or member function, and
either confirm no exceptions were generated, or confirm the
consistency/coherency of the test subject in the event of a
thrown exception.
Random code paths can be constructed using the basic test
sequences and instrumentation as above, only combined in a
random or pseudo-random way.
To compute the code paths that throw, test instruments
are used that throw on allocation events
(__gnu_cxx::throw_allocator_random
and __gnu_cxx::throw_allocator_limit)
and copy, assignment, comparison, increment, swap, and
various operators
(__gnu_cxx::throw_type_random
and __gnu_cxx::throw_type_limit). Looping
through a given test sequence and conditionally throwing in
all instrumented places. Then, when the test sequence
completes without an exception being thrown, assume all
potential error paths have been exercised in a sequential
manner.
Existing tests
Ad Hoc
For example,
testsuite/23_containers/list/modifiers/3.cc.
Policy Based Data Structures
For example, take the test
functor rand_reg_test in
in testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc. This uses container_rand_regression_test in
testsuite/util/regression/rand/assoc/container_rand_regression_test.h.
Which has several tests for container member functions,
Includes control and test container objects. Configuration includes
random seed, iterations, number of distinct values, and the
probability that an exception will be thrown. Assumes instantiating
container uses an extension
allocator, __gnu_cxx::throw_allocator_random,
as the allocator type.
C++11 Container Requirements.
Coverage is currently limited to testing container
requirements for exception safety,
although __gnu_cxx::throw_type meets
the additional type requirements for testing numeric data
structures and instantiating algorithms.
Of particular interest is extending testing to algorithms and
then to parallel algorithms. Also io and locales.
The test instrumentation should also be extended to add
instrumentation to iterator
and const_iterator types that throw
conditionally on iterator operations.
C++11 Requirements Test Sequence Descriptions
Basic
Basic consistency on exception propagation tests. For
each container, an object of that container is constructed,
a specific member function is exercised in
a try block, and then any thrown
exceptions lead to error checking in the appropriate
catch block. The container's use of
resources is compared to the container's use prior to the
test block. Resource monitoring is limited to allocations
made through the container's allocator_type,
which should be sufficient for container data
structures. Included in these tests are member functions
are iterator and const_iterator
operations, pop_front, pop_back, push_front, push_back, insert, erase, swap, clear,
and rehash. The container in question is
instantiated with two instrumented template arguments,
with __gnu_cxx::throw_allocator_limit
as the allocator type, and
with __gnu_cxx::throw_type_limit as
the value type. This allows the test to loop through
conditional throw points.
The general form is demonstrated in
testsuite/23_containers/list/requirements/exception/basic.cc
. The instantiating test object is __gnu_test::basic_safety and is detailed in testsuite/util/exception/safety.h.
Generation Prohibited
Exception generation tests. For each container, an object of
that container is constructed and all member functions
required to not throw exceptions are exercised. Included in
these tests are member functions
are iterator and const_iterator operations, erase, pop_front, pop_back, swap,
and clear. The container in question is
instantiated with two instrumented template arguments,
with __gnu_cxx::throw_allocator_random
as the allocator type, and
with __gnu_cxx::throw_type_random as
the value type. This test does not loop, an instead is sudden
death: first error fails.
The general form is demonstrated in
testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
. The instantiating test object is __gnu_test::generation_prohibited and is detailed in testsuite/util/exception/safety.h.
Propagation Consistent
Container rollback on exception propagation tests. For
each container, an object of that container is constructed,
a specific member function that requires rollback to a previous
known good state is exercised in
a try block, and then any thrown
exceptions lead to error checking in the appropriate
catch block. The container is compared to
the container's last known good state using such parameters
as size, contents, and iterator references. Included in these
tests are member functions
are push_front, push_back, insert,
and rehash. The container in question is
instantiated with two instrumented template arguments,
with __gnu_cxx::throw_allocator_limit
as the allocator type, and
with __gnu_cxx::throw_type_limit as
the value type. This allows the test to loop through
conditional throw points.
The general form demonstrated in
testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
. The instantiating test object is __gnu_test::propagation_coherent and is detailed in testsuite/util/exception/safety.h.