mirror of
https://github.com/dlang/dmd.git
synced 2025-04-26 05:00:16 +03:00
04025fdba6
* Add --ftime-trace to dmd * Refactor timetrace code for dmd. Pass Outbuffer by ref, move timetrace filename logic to main.d
472 lines
18 KiB
Markdown
472 lines
18 KiB
Markdown
Running DMD's test suite
|
|
-------------------------
|
|
|
|
This is the documentation and guide for DMD's test suite. See
|
|
[src/README.md](../src/README.md#major-targets) for additional
|
|
checks performed alongside these dedicated tests.
|
|
|
|
Maybe you are looking for the [contributing guide](../CONTRIBUTING.md)
|
|
too?
|
|
|
|
---
|
|
|
|
- [Quick guide](#quick-guide)
|
|
- [Types of Tests](#types-of-tests)
|
|
- [`run.d` targets](#rund-targets)
|
|
- [Test Configuration](#test-configuration)
|
|
- [Environment variables](#environment-variables)
|
|
- [Bash Tests](#bash-tests)
|
|
- [Test configuration variables](#test-configuration-variables)
|
|
- [`TEST_OUTPUT`](#test_output)
|
|
- [Test Coding Practices](#test-coding-practices)
|
|
|
|
---
|
|
|
|
Quick guide
|
|
-----------
|
|
|
|
### Run all tests
|
|
|
|
```console
|
|
./run.d
|
|
```
|
|
|
|
Note:
|
|
|
|
- [`run.d`](./run.d) will automatically use all available threads. Use e.g. `-j4` if you need a lower parallelism
|
|
- all commands below assume that you are in the `test` directory
|
|
|
|
### Run only a specific subset
|
|
|
|
```console
|
|
./run.d fail
|
|
./run.d compilable
|
|
```
|
|
|
|
As linking is slow the [`runnable`](runnable/README.md) tests take a bit longer to run:
|
|
|
|
```console
|
|
./run.d runnable
|
|
```
|
|
|
|
### Run only an individual test
|
|
|
|
```console
|
|
./run.d fail_compilation/diag10089.d
|
|
```
|
|
|
|
Multiple arguments are supported too.
|
|
You can use [`./run.d`](./run.d) to quickly run a custom subset of tests.
|
|
For example, all diagnostic tests in [`fail_compilation`](fail_compilation/README.md):
|
|
|
|
```console
|
|
./run.d fail_compilation/diag*.d
|
|
```
|
|
|
|
### Automatically update the `TEST_OUTPUT` segments
|
|
|
|
Often, when you add a new error message, a few tests need to be updated as their
|
|
`TEST_OUTPUT` has changed. This is tedious work and `AUTO_UPDATE` can be to automate it:
|
|
|
|
```console
|
|
AUTO_UPDATE=1 ./run.d fail
|
|
```
|
|
|
|
Updating the `TEST_OUTPUT` can also be done for a custom subset of tests:
|
|
|
|
```console
|
|
./run.d fail_compilation/diag*.d AUTO_UPDATE=1
|
|
```
|
|
|
|
Note:
|
|
- you might need to run this command twice if you add a new error message(s) as then the line numbers of the following error messages will change
|
|
- `AUTO_UPDATE` doesn't work with tests that have no, empty or multiple `TEST_OUTPUT` segments
|
|
- `AUTO_UPDATE` can be set as an environment variable or as Makefile-like argument assignment
|
|
|
|
### Running the Unit Tests
|
|
|
|
The unit tests will automatically run when all tests are run using `./run.d`.
|
|
To only run the unit tests the `./run.d unit_tests` command can be used.
|
|
For a more finer grain control over the unit tests the `./run.d -u` command can
|
|
be used:
|
|
|
|
To run all unit tests:
|
|
|
|
```console
|
|
./run.d -u
|
|
```
|
|
|
|
To only run the unit tests in one or more specific files:
|
|
|
|
```console
|
|
./run.d -u unit/deinitialization.d
|
|
```
|
|
|
|
To only run a subset of the unit tests in a single file:
|
|
|
|
```console
|
|
./run.d -u unit/deinitialization.d --filter Expression
|
|
```
|
|
|
|
In the above example, the `--filter` flag will filter to only run the tests with
|
|
a UDA matching the given value, in this case `Expression`.
|
|
|
|
```d
|
|
@("Target.deinitialize")
|
|
unittest {}
|
|
|
|
@("Expression.deinitialize")
|
|
unittest {}
|
|
```
|
|
|
|
Of the above unit tests, only the second one will be run, since
|
|
`--filter Expression` was specified.
|
|
|
|
The `--filter` flag works when no files are specified as well.
|
|
|
|
## Types of Tests
|
|
|
|
There are two types of tests in the DMD test suite:
|
|
|
|
* **End-to-end test**. These are tests that invokes the compiler as an external
|
|
process in some kind of way. Then it asserts either the exit code or the output
|
|
of the compiler. These tests are located in
|
|
|
|
- [`compilable`](compilable/README.md)
|
|
- [`fail_compilation`](fail_compilation/README.md)
|
|
- [`runnable`](runnable/README.md)
|
|
- [`runnable_cxx`](runnable_cxx/README.md)
|
|
- [`dshell`](dshell/README.md)
|
|
|
|
* **Unit tests**. These tests are more of a unit test, integration or
|
|
functional style tests. These tests are using the compiler as a library. They
|
|
are more flexible because they can assert state internal to the compiler which
|
|
the end-to-end tests would never have access to. The unit test runner will
|
|
compile all files in the [`unit`](unit/README.md) directory into a single executable and run the
|
|
tests. This should make it quick to run the tests since only a single process
|
|
need to be started.
|
|
|
|
`run.d` targets
|
|
----------------
|
|
|
|
default | all: run all unit tests that haven't been run yet
|
|
|
|
run_tests: run all tests
|
|
run_runnable_tests: run just the runnable tests
|
|
run_compilable_tests: run just the compilable tests
|
|
run_fail_compilation_tests: run just the fail compilation tests
|
|
unit_test: run all unit tests (those in the "unit" directory)
|
|
|
|
quick: run all tests with no default permuted args
|
|
(individual test specified options still honored)
|
|
|
|
clean: remove all temporary or result files from previous runs
|
|
|
|
test_results/compilable/json.d.out runs an individual test
|
|
(run log of the test is stored)
|
|
|
|
Test Configuration
|
|
------------------
|
|
|
|
All tests defined within `.d` source files may use various settings to configure how they are to be run, i.e.
|
|
|
|
`compilable/hellotest.d`:
|
|
```d
|
|
/*
|
|
REQUIRED_ARGS: -version=Foo
|
|
TEST_OUTPUT:
|
|
---
|
|
Hello, World!
|
|
---
|
|
*/
|
|
void main(string[] args)
|
|
{
|
|
version(Foo)
|
|
{
|
|
pragma(msg, "Hello World");
|
|
}
|
|
}
|
|
```
|
|
|
|
Test parameters can be restricted to certain targets by adding a brace-enclosed
|
|
condition after the name, i.e. `REQUIRED_ARGS(<condition>): ...`. The `<condition>`
|
|
consists of the target operating system followed by an optional model suffix,
|
|
e.g. `linux`, `win32`, `freebsd64`.
|
|
|
|
Valid platforms:
|
|
- win
|
|
- linux
|
|
- osx
|
|
- freebsd
|
|
- dragonflybsd
|
|
- netbsd
|
|
|
|
Valid models:
|
|
- 32
|
|
- 64
|
|
|
|
Note that test parameters *MUST* be followed by a colon (intermediate whitespace is allowed).
|
|
The test runner will issue an error for a missing colon (e.g. `REQUIRED_ARGS foo`)
|
|
to avoid ambiguities. Test directives embedded within other words (e.g. `OPTLINK`)
|
|
will be ignored.
|
|
|
|
The following is a list of all available settings:
|
|
|
|
ARG_SETS: sets off extra arguments to invoke $(DMD) with (seperated by ';').
|
|
default: (none)
|
|
|
|
COMPILE_SEPARATELY: if present, forces each .d file to compile separately and linked
|
|
together in an extra setup. May specify additional parameters which
|
|
are passed to $(DMD) when linking the generated object files.
|
|
default: (none, aka compile/link all in one step)
|
|
|
|
COMPILED_IMPORTS: list of modules files that are imported by the main source file that
|
|
should be included in compilation; this differs from the EXTRA_SOURCES
|
|
variable in that these files could be compiled by either explicitly
|
|
passing them to the compiler or by using the "-i" option. Using this
|
|
option will cause the test to be compiled twice, once using "-i" and
|
|
once by explicitly passing the modules to the compiler.
|
|
default: (none)
|
|
|
|
CXXFLAGS: list of extra arguments passed to $(CC) when compiling C++ sources
|
|
defined in EXTRA_CPP_SOURCES.
|
|
default: (none)
|
|
|
|
DFLAGS: Overrides the DFLAGS environment variable if specified in the test.
|
|
No values are permitted; an error will be emitted if the value is not
|
|
empty.
|
|
|
|
DISABLED: selectively disable the test on specific platforms (if empty, the test is
|
|
considered to be enabled on all platform). Target platforms are specified
|
|
using nearly the same syntax as conditions of optional parameters, except for
|
|
`win` instead of `windows`.
|
|
Potential filters are `win32`, `linux`, ...
|
|
default: (none, enabled)
|
|
|
|
EXECUTE_ARGS: parameters to add to the execution of the test
|
|
default: (none)
|
|
|
|
EXTRA_CPP_SOURCES: list of extra C++ files to build and link along with the test
|
|
default: (none).
|
|
|
|
EXTRA_FILES: list of extra files and sources used by the test, either during
|
|
compilation or execution of the test. It is currently ignored by the test
|
|
runner, but serves as documentation of the test itself.
|
|
default: (none)
|
|
|
|
EXTRA_OBJC_SOURCES: list of extra Objective-C files to build and link along with the test
|
|
default: (none). Test files with this variable will be ignored unless
|
|
the D_OBJC environment variable is set to "1"
|
|
|
|
EXTRA_SOURCES: list of extra files to build and link along with the test
|
|
default: (none)
|
|
|
|
GDB_MATCH: a regular expression describing the expected output of GDB_SCRIPT. The test
|
|
will fail if it does not match the actual output.
|
|
default: (none)
|
|
|
|
GDB_SCRIPT: if present, starts a `gdb` session for the compiled executable to run the commands
|
|
specified in the corresponding section. GDB_MATCH may be used to used to verfiy
|
|
expected output using a regex.
|
|
note: restricted to `runnable` tests, the executable will not be run outside of the
|
|
gdb session.
|
|
default: (none)
|
|
|
|
LINK: enables linking (used for the compilable and fail_compilable tests).
|
|
default: (none)
|
|
|
|
OUTPUT_FILES: files generated during the compilation (separated by ';').
|
|
The content of each file is appended to the output of the
|
|
compilation (in the order of this list) according to the HAR
|
|
format (https://code.dlang.org/packages/har).
|
|
Example:
|
|
------------------------------------------
|
|
<Compilation Output>
|
|
=== <FILENAME_1>
|
|
<CONTENT_1>
|
|
=== <FILENAME_2>
|
|
<CONTENT_2>
|
|
[...]
|
|
------------------------------------------
|
|
The merged output will then be prepared and compared to the
|
|
expected TEST_OUTPUT as defined below.
|
|
default: (none)
|
|
|
|
PERMUTE_ARGS: the set of arguments to permute in multiple $(DMD) invocations.
|
|
An empty set means only one permutation with no arguments.
|
|
default: the make variable ARGS (see below)
|
|
|
|
POST_SCRIPT: name of script to execute after test run
|
|
note: arguments to the script may be included after the name.
|
|
additionally, the name of the file that contains the output
|
|
of the compile/link/run steps is added as the last parameter.
|
|
default: (none)
|
|
|
|
REQUIRED_ARGS: arguments to add to the $(DMD) command line
|
|
default: (none)
|
|
note: the make variable REQUIRED_ARGS is also added to the $(DMD)
|
|
command line (see below)
|
|
|
|
RUN_OUTPUT: output expected from running the compiled executable which must match
|
|
the actual output. The comparison adheres to the rules defined for
|
|
TEST_OUTPUT and allow e.g. using special sequences as defined below.
|
|
|
|
TEST_OUTPUT: the output is expected from the compilation (if the
|
|
output of the compilation doesn't match, the test
|
|
fails). You can use the this format for multi-line
|
|
output:
|
|
TEST_OUTPUT:
|
|
---
|
|
Some
|
|
Output
|
|
---
|
|
note: if not given, it is assumed that the compilation will be silent.
|
|
default: (none)
|
|
|
|
TEST_OUTPUT_FILE: file containing the expected output as defined for TEST_OUTPUT.
|
|
note: Further TEST_OUTPUT sections in the test are ignored.
|
|
default: (none)
|
|
|
|
TRANSFORM_OUTPUT: steps to apply to the output of the compilation before it
|
|
is compared to the expected TEST_OUTPUT. A step may take
|
|
arguments akin to a function call, e.g. `step(arg)` and arguments
|
|
may be quoted using "".
|
|
|
|
Supported transformations:
|
|
- sanitize_json: Remove compiler specific information from output
|
|
of -Xi (see test/tools/sanitize_json.d)
|
|
arguments: none
|
|
|
|
- sanitize_timetrace: Parse output of -ftime-trace profiling data,
|
|
extract event name / detail strings, and sort them.
|
|
Raw profiler output can't be tested against because it
|
|
contains non-deterministic timing data, or implementation
|
|
details such as semantic analysis order and memory usage,
|
|
so use this when writing a test for -ftime-trace.
|
|
|
|
- remove_lines: Remove lines matching a given regex
|
|
arguments: the regex
|
|
note: patterns containing ')' must be quoted
|
|
|
|
UNICODE_NAMES: file containing symbols with unicode characters in their name, which might
|
|
not be supported on some specific platforms. It is currently ignored by the
|
|
test runner, but serves as documentation of the test itself.
|
|
default: (none)
|
|
|
|
|
|
Environment variables
|
|
------------------------------
|
|
|
|
[`run.d`](./run.d) uses environment variables to store test settings and as a way to pass these settings to the test wrapper tool [`d_do_test.d`](tools/d_do_test.d).
|
|
|
|
> Note: These variables are also available inside any Bash test.
|
|
|
|
ARGS: set to execute all combinations of
|
|
AUTO_UPDATE: set to 1 to auto-update mismatching test output
|
|
CC: C++ compiler to use, ex: cl, g++
|
|
DMD: compiler to use, ex: ../src/dmd (required)
|
|
MODEL: 32 or 64 (required)
|
|
OS: windows, linux, freebsd, osx, netbsd, dragonflybsd
|
|
REQUIRED_ARGS: arguments always passed to the compiler
|
|
RESULTS_DIR: base directory for test results
|
|
|
|
Windows vs non-windows portability env vars:
|
|
|
|
DSEP: \\ or /
|
|
EXE: .exe or <null> (required)
|
|
OBJ: .obj or .o (required)
|
|
SEP: \ or / (required)
|
|
|
|
Bash Tests
|
|
----------
|
|
|
|
Along with the environment variables provided by [`run.d`](./run.d) (see above), an additional set of environment variables are made available to Bash tests. These variables are defined in `tools/exported_vars.sh`:
|
|
|
|
EXTRA_FILES directory for extra files of this test type, e.g. runnable/extra-files
|
|
|
|
LIBEXT platform-specific extension for library files, e.g. .a or .lib
|
|
|
|
RESULTS_TEST_DIR the results directory for tests of this type, e.g. test_results/runnable
|
|
|
|
OUTPUT_BASE the prefix used for test output files, e.g. test_results/runnable/mytest
|
|
|
|
SOEXT platform-specific extension for shared object files (aka. dynamic libraries),
|
|
e.g. .so, .dll or .dylib
|
|
|
|
TEST_DIR the name of the test directory
|
|
(one of compilable, fail_compilation or runnable)
|
|
|
|
TEST_NAME the base name of the test file without the extension, e.g. test15897
|
|
|
|
Test configuration variables
|
|
----------------------------
|
|
|
|
Sometimes test configuration arguments must be dynamic.
|
|
For example, the output of all tests should be placed into `RESULTS_DIR`:
|
|
|
|
```
|
|
// REQUIRED_ARGS: -mixin=${RESULTS_DIR}/fail_compilation/mixin_test.mixin
|
|
```
|
|
|
|
Currently these variables are exposed:
|
|
|
|
RESULTS_DIR Path to `test_results`
|
|
|
|
`TEST_OUTPUT`
|
|
-------------
|
|
|
|
A few operations are done on the output of a test before the comparison with `TEST_OUTPUT`:
|
|
|
|
- newlines get unified for consistent `TEST_OUTPUT` between platforms
|
|
- DMD's debug message (e.g. `DMD v2.084.0-255-g86b608a15-dirty DEBUG`) gets stripped away
|
|
- paths to `test_results` will be replaced with `{{RESULTS_DIR}}`
|
|
|
|
`TEST_OUTPUT` offers the following special sequences to match error messages which
|
|
depend on the current platform and target:
|
|
|
|
$n$ arbitrary amount of digits
|
|
|
|
$p:<tail>$ paths ending with <tail> (which must refer to an existing file or directory)
|
|
|
|
$?:<choices>$ selection based on the current environment where a choice is either
|
|
conditional `<condition>=<content>` or a fallback value `<default>`.
|
|
Multiple choices are separated by `|` and the leftmost satisfied condition
|
|
or fallback is chosen if multiple choices apply.
|
|
|
|
Supported conditions:
|
|
- OS: posix, windows, ...
|
|
- Model: 64, 32
|
|
|
|
$r:<regex>$ any text matching <regex> (using $ inside of <regex> is not
|
|
supported, use multiple regexes instead)
|
|
|
|
Both stderr and stdout of the DMD are captured for output comparison.
|
|
|
|
## Test Coding Practices
|
|
|
|
The purpose of the test suite is to test the compiler only. This means:
|
|
|
|
* do not import modules from Phobos
|
|
* keep imports from druntime to the interface to the C standard library
|
|
* use `core.stdc.stdio.printf`, not `std.stdio.writef`
|
|
|
|
In order to make the test suite run faster, multiple unrelated tests can
|
|
be aggregated into a single file, for example `test/runnable/test42.d`
|
|
|
|
Each test should be in the following form:
|
|
|
|
```d
|
|
/*******************************/
|
|
// https://issues.dlang.org/show_bug.cgi?id=NNNN
|
|
|
|
void testNNNN()
|
|
{
|
|
}
|
|
```
|
|
|
|
The NNNN is the bugzilla issue number this test ensures is fixed.
|
|
The test code should be self-contained. The test code should be
|
|
minimized to focus on the test.
|
|
|
|
As usual, test source code should be LF terminated lines, and not
|
|
contain any tab characters.
|