diff options
| author | Sean Silva <silvas@purdue.edu> | 2012-11-14 21:09:30 +0000 |
|---|---|---|
| committer | Sean Silva <silvas@purdue.edu> | 2012-11-14 21:09:30 +0000 |
| commit | ac99eed043e84905bc2fb299ccaf5809e9c0e90f (patch) | |
| tree | f3e8b22b7ff8dfdb6acdf545f8b158172141a22c /docs | |
| parent | 70a870add82dab944b98ee1840fafff33795fc95 (diff) | |
| download | llvm-ac99eed043e84905bc2fb299ccaf5809e9c0e90f.tar.gz llvm-ac99eed043e84905bc2fb299ccaf5809e9c0e90f.tar.bz2 llvm-ac99eed043e84905bc2fb299ccaf5809e9c0e90f.tar.xz | |
docs: Sphinxify TestingGuide
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167979 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/DeveloperPolicy.rst | 4 | ||||
| -rw-r--r-- | docs/GettingStarted.rst | 4 | ||||
| -rw-r--r-- | docs/MakefileGuide.rst | 2 | ||||
| -rw-r--r-- | docs/Projects.rst | 6 | ||||
| -rw-r--r-- | docs/TestingGuide.html | 921 | ||||
| -rw-r--r-- | docs/TestingGuide.rst | 713 | ||||
| -rw-r--r-- | docs/userguides.rst | 3 |
7 files changed, 723 insertions, 930 deletions
diff --git a/docs/DeveloperPolicy.rst b/docs/DeveloperPolicy.rst index e35e729556..390901289d 100644 --- a/docs/DeveloperPolicy.rst +++ b/docs/DeveloperPolicy.rst @@ -180,8 +180,8 @@ Developers are required to create test cases for any bugs fixed and any new features added. Some tips for getting your testcase approved: * All feature and regression test cases are added to the ``llvm/test`` - directory. The appropriate sub-directory should be selected (see the `Testing - Guide <TestingGuide.html>`_ for details). + directory. The appropriate sub-directory should be selected (see the + :doc:`Testing Guide <TestingGuide>` for details). * Test cases should be written in `LLVM assembly language <LangRef.html>`_ unless the feature or regression being tested requires another language diff --git a/docs/GettingStarted.rst b/docs/GettingStarted.rst index 68768921f6..4ad763650d 100644 --- a/docs/GettingStarted.rst +++ b/docs/GettingStarted.rst @@ -1073,8 +1073,8 @@ module that must be checked out (usually to ``projects/test-suite``). This module contains a comprehensive correctness, performance, and benchmarking test suite for LLVM. It is a separate Subversion module because not every LLVM user is interested in downloading or building such a comprehensive test suite. For -further details on this test suite, please see the `Testing -Guide <TestingGuide.html>`_ document. +further details on this test suite, please see the :doc:`Testing Guide +<TestingGuide>` document. .. _tools: diff --git a/docs/MakefileGuide.rst b/docs/MakefileGuide.rst index d2bdd24a9e..2c1d33e962 100644 --- a/docs/MakefileGuide.rst +++ b/docs/MakefileGuide.rst @@ -339,7 +339,7 @@ the invocation of ``make check-local`` in the ``test`` directory. The intended usage for this is to assist in running specific suites of tests. If ``TESTSUITE`` is not set, the implementation of ``check-local`` should run all normal tests. It is up to the project to define what different values for -``TESTSUTE`` will do. See the `Testing Guide <TestingGuide.html>`_ for further +``TESTSUTE`` will do. See the :doc:`Testing Guide <TestingGuide>` for further details. ``check-local`` diff --git a/docs/Projects.rst b/docs/Projects.rst index 63132887a5..c5d03d33a0 100644 --- a/docs/Projects.rst +++ b/docs/Projects.rst @@ -156,9 +156,9 @@ Underneath your top level directory, you should have the following directories: * LLVM provides a ``tcl`` procedure that is used by ``Dejagnu`` to run tests. It can be found in ``llvm/lib/llvm-dg.exp``. This test procedure uses ``RUN`` lines in the actual test case to determine how to run the test. See the - `TestingGuide <TestingGuide.html>`_ for more details. You can easily write - Makefile support similar to the Makefiles in ``llvm/test`` to use ``Dejagnu`` - to run your project's tests. + :doc:`TestingGuide` for more details. You can easily write Makefile + support similar to the Makefiles in ``llvm/test`` to use ``Dejagnu`` to + run your project's tests. * LLVM contains an optional package called ``llvm-test``, which provides benchmarks and programs that are known to compile with the Clang front diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html deleted file mode 100644 index 1be7551122..0000000000 --- a/docs/TestingGuide.html +++ /dev/null @@ -1,921 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" - "http://www.w3.org/TR/html4/strict.dtd"> -<html> -<head> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> - <title>LLVM Testing Infrastructure Guide</title> - <link rel="stylesheet" href="_static/llvm.css" type="text/css"> -</head> -<body> - -<h1> - LLVM Testing Infrastructure Guide -</h1> - -<ol> - <li><a href="#overview">Overview</a></li> - <li><a href="#requirements">Requirements</a></li> - <li><a href="#org">LLVM testing infrastructure organization</a> - <ul> - <li><a href="#regressiontests">Regression tests</a></li> - <li><a href="#testsuite"><tt>test-suite</tt></a></li> - <li><a href="#debuginfotests">Debugging Information tests</a></li> - </ul> - </li> - <li><a href="#quick">Quick start</a> - <ul> - <li><a href="#quickregressiontests">Regression tests</a></li> - <li><a href="#quickdebuginfotests">Debugging Information tests</a></li> - </ul> - </li> - <li><a href="#rtstructure">Regression test structure</a> - <ul> - <li><a href="#rtcustom">Writing new regression tests</a></li> - <li><a href="#FileCheck">The FileCheck utility</a></li> - <li><a href="#rtvars">Variables and substitutions</a></li> - <li><a href="#rtfeatures">Other features</a></li> - </ul> - </li> - <li><a href="#testsuiteoverview"><tt>test-suite</tt> Overview</a> - <ul> - <li><a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a></li> - <li><a href="#testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></li> - </ul> - </li> -</ol> - -<div class="doc_author"> - <p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p> -</div> - -<!--=========================================================================--> -<h2><a name="overview">Overview</a></h2> -<!--=========================================================================--> - -<div> - -<p>This document is the reference manual for the LLVM testing infrastructure. It -documents the structure of the LLVM testing infrastructure, the tools needed to -use it, and how to add and run tests.</p> - -</div> - -<!--=========================================================================--> -<h2><a name="requirements">Requirements</a></h2> -<!--=========================================================================--> - -<div> - -<p>In order to use the LLVM testing infrastructure, you will need all of the -software required to build LLVM, as well -as <a href="http://python.org">Python</a> 2.4 or later.</p> - -</div> - -<!--=========================================================================--> -<h2><a name="org">LLVM testing infrastructure organization</a></h2> -<!--=========================================================================--> - -<div> - -<p>The LLVM testing infrastructure contains two major categories of tests: -regression tests and whole programs. The regression tests are contained inside -the LLVM repository itself under <tt>llvm/test</tt> and are expected to always -pass -- they should be run before every commit.</p> - -<p>The whole programs tests are referred to as the "LLVM test suite" (or -"test-suite") and are in the <tt>test-suite</tt> module in subversion. For -historical reasons, these tests are also referred to as the "nightly tests" in -places, which is less ambiguous than "test-suite" and remains in use although we -run them much more often than nightly.</p> - -<!-- _______________________________________________________________________ --> -<h3><a name="regressiontests">Regression tests</a></h3> -<!-- _______________________________________________________________________ --> - -<div> - -<p>The regression tests are small pieces of code that test a specific feature of -LLVM or trigger a specific bug in LLVM. They are usually written in LLVM -assembly language, but can be written in other languages if the test targets a -particular language front end (and the appropriate <tt>--with-llvmgcc</tt> -options were used at <tt>configure</tt> time of the <tt>llvm</tt> module). These -tests are driven by the 'lit' testing tool, which is part of LLVM.</p> - -<p>These code fragments are not complete programs. The code generated -from them is never executed to determine correct behavior.</p> - -<p>These code fragment tests are located in the <tt>llvm/test</tt> -directory.</p> - -<p>Typically when a bug is found in LLVM, a regression test containing -just enough code to reproduce the problem should be written and placed -somewhere underneath this directory. In most cases, this will be a small -piece of LLVM assembly language code, often distilled from an actual -application or benchmark.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="testsuite"><tt>test-suite</tt></a></h3> -<!-- _______________________________________________________________________ --> - -<div> - -<p>The test suite contains whole programs, which are pieces of code which can be -compiled and linked into a stand-alone program that can be executed. These -programs are generally written in high level languages such as C or C++.</p> - -<p>These programs are compiled using a user specified compiler and set of flags, -and then executed to capture the program output and timing information. The -output of these programs is compared to a reference output to ensure that the -program is being compiled correctly.</p> - -<p>In addition to compiling and executing programs, whole program tests serve as -a way of benchmarking LLVM performance, both in terms of the efficiency of the -programs generated as well as the speed with which LLVM compiles, optimizes, and -generates code.</p> - -<p>The test-suite is located in the <tt>test-suite</tt> Subversion module.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="debuginfotests">Debugging Information tests</a></h3> -<!-- _______________________________________________________________________ --> - -<div> - -<p>The test suite contains tests to check quality of debugging information. -The test are written in C based languages or in LLVM assembly language. </p> - -<p>These tests are compiled and run under a debugger. The debugger output -is checked to validate of debugging information. See README.txt in the -test suite for more information . This test suite is located in the -<tt>debuginfo-tests</tt> Subversion module. </p> - -</div> - -</div> - -<!--=========================================================================--> -<h2><a name="quick">Quick start</a></h2> -<!--=========================================================================--> - -<div> - - <p>The tests are located in two separate Subversion modules. The regressions - tests are in the main "llvm" module under the directory - <tt>llvm/test</tt> (so you get these tests for free with the main llvm - tree). Use "make check-all" to run the regression tests after building - LLVM.</p> - - <p>The more comprehensive test suite that includes whole programs in C and C++ - is in the <tt>test-suite</tt> - module. See <a href="#testsuitequickstart"><tt>test-suite</tt> Quickstart</a> - for more information on running these tests.</p> - -<!-- _______________________________________________________________________ --> -<h3><a name="quickregressiontests">Regression tests</a></h3> -<div> -<!-- _______________________________________________________________________ --> -<p>To run all of the LLVM regression tests, use master Makefile in - the <tt>llvm/test</tt> directory:</p> - -<div class="doc_code"> -<pre> -% gmake -C llvm/test -</pre> -</div> - -<p>or</p> - -<div class="doc_code"> -<pre> -% gmake check -</pre> -</div> - -<p>If you have <a href="http://clang.llvm.org/">Clang</a> checked out and built, -you can run the LLVM and Clang tests simultaneously using:</p> - -<p>or</p> - -<div class="doc_code"> -<pre> -% gmake check-all -</pre> -</div> - -<p>To run the tests with Valgrind (Memcheck by default), just append -<tt>VG=1</tt> to the commands above, e.g.:</p> - -<div class="doc_code"> -<pre> -% gmake check VG=1 -</pre> -</div> - -<p>To run individual tests or subsets of tests, you can use the 'llvm-lit' -script which is built as part of LLVM. For example, to run the -'Integer/BitPacked.ll' test by itself you can run:</p> - -<div class="doc_code"> -<pre> -% llvm-lit ~/llvm/test/Integer/BitPacked.ll -</pre> -</div> - -<p>or to run all of the ARM CodeGen tests:</p> - -<div class="doc_code"> -<pre> -% llvm-lit ~/llvm/test/CodeGen/ARM -</pre> -</div> - -<p>For more information on using the 'lit' tool, see 'llvm-lit --help' or the -'lit' man page.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="quickdebuginfotests">Debugging Information tests</a></h3> -<div> -<!-- _______________________________________________________________________ --> -<div> - -<p> To run debugging information tests simply checkout the tests inside -clang/test directory. </p> - -<div class="doc_code"> -<pre> -%cd clang/test -% svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests -</pre> -</div> - -<p> These tests are already set up to run as part of clang regression tests.</p> - -</div> - -</div> - -</div> - -<!--=========================================================================--> -<h2><a name="rtstructure">Regression test structure</a></h2> -<!--=========================================================================--> -<div> - <p>The LLVM regression tests are driven by 'lit' and are located in - the <tt>llvm/test</tt> directory. - - <p>This directory contains a large array of small tests - that exercise various features of LLVM and to ensure that regressions do not - occur. The directory is broken into several sub-directories, each focused on - a particular area of LLVM. A few of the important ones are:</p> - - <ul> - <li><tt>Analysis</tt>: checks Analysis passes.</li> - <li><tt>Archive</tt>: checks the Archive library.</li> - <li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li> - <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li> - <li><tt>CodeGen</tt>: checks code generation and each target.</li> - <li><tt>Features</tt>: checks various features of the LLVM language.</li> - <li><tt>Linker</tt>: tests bitcode linking.</li> - <li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility - transforms to ensure they make the right transformations.</li> - <li><tt>Verifier</tt>: tests the IR verifier.</li> - </ul> - -<!-- _______________________________________________________________________ --> -<h3><a name="rtcustom">Writing new regression tests</a></h3> -<!-- _______________________________________________________________________ --> -<div> - <p>The regression test structure is very simple, but does require some - information to be set. This information is gathered via <tt>configure</tt> and - is written to a file, <tt>lit.site.cfg</tt> - in <tt>llvm/test</tt>. The <tt>llvm/test</tt> Makefile does this work for - you.</p> - - <p>In order for the regression tests to work, each directory of tests must - have a <tt>lit.local.cfg</tt> file. Lit looks for this file to determine how - to run the tests. This file is just Python code and thus is very flexible, - but we've standardized it for the LLVM regression tests. If you're adding a - directory of tests, just copy <tt>lit.local.cfg</tt> from another directory to - get running. The standard <tt>lit.local.cfg</tt> simply specifies which files - to look in for tests. Any directory that contains only directories does not - need the <tt>lit.local.cfg</tt> file. Read the - <a href="http://llvm.org/cmds/lit.html">Lit documentation</a> for more - information. </p> - - <p>The <tt>llvm-runtests</tt> function looks at each file that is passed to - it and gathers any lines together that match "RUN:". These are the "RUN" lines - that specify how the test is to be run. So, each test script must contain - RUN lines if it is to do anything. If there are no RUN lines, the - <tt>llvm-runtests</tt> function will issue an error and the test will - fail.</p> - - <p>RUN lines are specified in the comments of the test program using the - keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline) - to execute. Together, these lines form the "script" that - <tt>llvm-runtests</tt> executes to run the test case. The syntax of the - RUN lines is similar to a shell's syntax for pipelines including I/O - redirection and variable substitution. However, even though these lines - may <i>look</i> like a shell script, they are not. RUN lines are interpreted - directly by the Tcl <tt>exec</tt> command. They are never executed by a - shell. Consequently the syntax differs from normal shell script syntax in a - few ways. You can specify as many RUN lines as needed.</p> - - <p>lit performs substitution on each RUN line to replace LLVM tool - names with the full paths to the executable built for each tool (in - $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not - invoke any stray LLVM tools in the user's path during testing.</p> - - <p>Each RUN line is executed on its own, distinct from other lines unless - its last character is <tt>\</tt>. This continuation character causes the RUN - line to be concatenated with the next one. In this way you can build up long - pipelines of commands without making huge line lengths. The lines ending in - <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is - found. This concatenated set of RUN lines then constitutes one execution. - Tcl will substitute variables and arrange for the pipeline to be executed. If - any process in the pipeline fails, the entire line (and test case) fails too. - </p> - - <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p> - -<div class="doc_code"> -<pre> -; RUN: llvm-as < %s | llvm-dis > %t1 -; RUN: llvm-dis < %s.bc-13 > %t2 -; RUN: diff %t1 %t2 -</pre> -</div> - - <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection - to be used. However, the usage is slightly different than for Bash. To check - what's legal, see the documentation for the - <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a> - command and the - <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>. - The major differences are:</p> - <ul> - <li>You can't do <tt>2>&1</tt>. That will cause Tcl to write to a - file named <tt>&1</tt>. Usually this is done to get stderr to go through - a pipe. You can do that in tcl with <tt>|&</tt> so replace this idiom: - <tt>... 2>&1 | grep</tt> with <tt>... |& grep</tt></li> - <li>You can only redirect to a file, not to another descriptor and not from - a here document.</li> - <li>tcl supports redirecting to open files with the @ syntax but you - shouldn't use that here.</li> - </ul> - - <p>There are some quoting rules that you must pay attention to when writing - your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any - quote characters so they will get passed to the invoked program. For - example:</p> - -<div class="doc_code"> -<pre> -... | grep 'find this string' -</pre> -</div> - - <p>This will fail because the ' characters are passed to grep. This would - instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and - <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should - treat everything enclosed as one value. So our example would become:</p> - -<div class="doc_code"> -<pre> -... | grep {find this string} -</pre> -</div> - - <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated - specially by Tcl. They tell Tcl to interpret the content as a command to - execute. Since these characters are often used in regular expressions this can - have disastrous results and cause the entire test run in a directory to fail. - For example, a common idiom is to look for some basicblock number:</p> - -<div class="doc_code"> -<pre> -... | grep bb[2-8] -</pre> -</div> - - <p>This, however, will cause Tcl to fail because its going to try to execute - a program named "2-8". Instead, what you want is this:</p> - -<div class="doc_code"> -<pre> -... | grep {bb\[2-8\]} -</pre> -</div> - - <p>Finally, if you need to pass the <tt>\</tt> character down to a program, - then it must be doubled. This is another Tcl special character. So, suppose - you had: - -<div class="doc_code"> -<pre> -... | grep 'i32\*' -</pre> -</div> - - <p>This will fail to match what you want (a pointer to i32). First, the - <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off - by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match - anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like - this:</p> - -<div class="doc_code"> -<pre> -... | grep {i32\\*} -</pre> -</div> - -<p>If your system includes GNU <tt>grep</tt>, make sure -that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise, -you may get invalid results (both false positives and false -negatives).</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="FileCheck">The FileCheck utility</a></h3> -<!-- _______________________________________________________________________ --> - -<div> - -<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands - to be executed as part of the test harness. While standard (portable) unix - tools like 'grep' work fine on run lines, as you see above, there are a lot - of caveats due to interaction with Tcl syntax, and we want to make sure the - run lines are portable to a wide range of systems. Another major problem is - that grep is not very good at checking to verify that the output of a tools - contains a series of different output in a specific order. The FileCheck - tool was designed to help with these problems.</p> - -<p>FileCheck (whose basic command line arguments are described in <a - href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page</a> is - designed to read a file to check from standard input, and the set of things - to verify from a file specified as a command line argument. A simple example - of using FileCheck from a RUN line looks like this:</p> - -<div class="doc_code"> -<pre> -; RUN: llvm-as < %s | llc -march=x86-64 | <b>FileCheck %s</b> -</pre> -</div> - -<p>This syntax says to pipe the current file ("%s") into llvm-as, pipe that into -llc, then pipe the output of llc into FileCheck. This means that FileCheck will -be verifying its standard input (the llc output) against the filename argument -specified (the original .ll file specified by "%s"). To see how this works, -let's look at the rest of the .ll file (after the RUN line):</p> - -<div class="doc_code"> -<pre> -define void @sub1(i32* %p, i32 %v) { -entry: -; <b>CHECK: sub1:</b> -; <b>CHECK: subl</b> - %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v) - ret void -} - -define void @inc4(i64* %p) { -entry: -; <b>CHECK: inc4:</b> -; <b>CHECK: incq</b> - %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1) - ret void -} -</pre> -</div> - -<p>Here you can see some "CHECK:" lines specified in comments. Now you can see -how the file is piped into llvm-as, then llc, and the machine code output is -what we are verifying. FileCheck checks the machine code output to verify that -it matches what the "CHECK:" lines specify.</p> - -<p>The syntax of the CHECK: lines is very simple: they are fixed strings that -must occur in order. FileCheck defaults to ignoring horizontal whitespace -differences (e.g. a space is allowed to match a tab) but otherwise, the contents -of the CHECK: line is required to match some thing in the test file exactly.</p> - -<p>One nice thing about FileCheck (compared to grep) is that it allows merging -test cases together into logical groups. For example, because the test above -is checking for the "sub1:" and "inc4:" labels, it will not match unless there -is a "subl" in between those labels. If it existed somewhere else in the file, -that would not count: "grep subl" matches if subl exists anywhere in the -file.</p> - -<!-- _______________________________________________________________________ --> -<h4> - <a name="FileCheck-check-prefix">The FileCheck -check-prefix option</a> -</h4> - -<div> - -<p>The FileCheck -check-prefix option allows multiple test configurations to be -driven from one .ll file. This is useful in many circumstances, for example, -testing different architectural variants with llc. Here's a simple example:</p> - -<div class="doc_code"> -<pre> -; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \ -; RUN: | <b>FileCheck %s -check-prefix=X32</b> -; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \ -; RUN: | <b>FileCheck %s -check-prefix=X64</b> - -define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind { - %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1 - ret <4 x i32> %tmp1 -; <b>X32:</b> pinsrd_1: -; <b>X32:</b> pinsrd $1, 4(%esp), %xmm0 - -; <b>X64:</b> pinsrd_1: -; <b>X64:</b> pinsrd $1, %edi, %xmm0 -} -</pre> -</div> - -<p>In this case, we're testing that we get the expected code generation with -both 32-bit and 64-bit code generation.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h4> - <a name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a> -</h4> - -<div> - -<p>Sometimes you want to match lines and would like to verify that matches -happen on exactly consecutive lines with no other lines in between them. In -this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If -you specified a custom check prefix, just use "<PREFIX>-NEXT:". For -example, something like this works as you'd expect:</p> - -<div class="doc_code"> -<pre> -define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) { - %tmp3 = load <2 x double>* %A, align 16 - %tmp7 = insertelement <2 x double> undef, double %B, i32 0 - %tmp9 = shufflevector <2 x double> %tmp3, - <2 x double> %tmp7, - <2 x i32> < i32 0, i32 2 > - store <2 x double> %tmp9, <2 x double>* %r, align 16 - ret void - -; <b>CHECK:</b> t2: -; <b>CHECK:</b> movl 8(%esp), %eax -; <b>CHECK-NEXT:</b> movapd (%eax), %xmm0 -; <b>CHECK-NEXT:</b> movhpd 12(%esp), %xmm0 -; <b>CHECK-NEXT:</b> movl 4(%esp), %eax -; <b>CHECK-NEXT:</b> movapd %xmm0, (%eax) -; <b>CHECK-NEXT:</b> ret -} -</pre> -</div> - -<p>CHECK-NEXT: directives reject the input unless there is exactly one newline -between it an the previous directive. A CHECK-NEXT cannot be the first -directive in a file.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h4> - <a name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a> -</h4> - -<div> - -<p>The CHECK-NOT: directive is used to verify that a string doesn't occur -between two matches (or the first match and the beginning of the file). For -example, to verify that a load is removed by a transformation, a test like this -can be used:</p> - -<div class="doc_code"> -<pre> -define i8 @coerce_offset0(i32 %V, i32* %P) { - store i32 %V, i32* %P - - %P2 = bitcast i32* %P to i8* - %P3 = getelementptr i8* %P2, i32 2 - - %A = load i8* %P3 - ret i8 %A -; <b>CHECK:</b> @coerce_offset0 -; <b>CHECK-NOT:</b> load -; <b>CHECK:</b> ret i8 -} -</pre> -</div> - -</div> - -<!-- _______________________________________________________________________ --> -<h4> - <a name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a> -</h4> - -<div> - -<!-- {% raw %} --> - -<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most -uses of FileCheck, fixed string matching is perfectly sufficient. For some -things, a more flexible form of matching is desired. To support this, FileCheck -allows you to specify regular expressions in matching strings, surrounded by -double braces: <b>{{yourregex}}</b>. Because we want to use fixed string -matching for a majority of what we do, FileCheck has been designed to support -mixing and matching fixed string matching with regular expressions. This allows -you to write things like this:</p> - -<div class="doc_code"> -<pre> -; CHECK: movhpd <b>{{[0-9]+}}</b>(%esp), <b>{{%xmm[0-7]}}</b> -</pre> -</div> - -<p>In this case, any offset from the ESP register will be allowed, and any xmm -register will be allowed.</p> - -<p>Because regular expressions are enclosed with double braces, they are -visually distinct, and you don't need to use escape characters within the double -braces like you would in C. In the rare case that you want to match double -braces explicitly from the input, you can use something ugly like -<b>{{[{][{]}}</b> as your pattern.</p> - -<!-- {% endraw %} --> - -</div> - -<!-- _______________________________________________________________________ --> -<h4> - <a name="FileCheck-Variables">FileCheck Variables</a> -</h4> - -<div> - - -<!-- {% raw %} --> - -<p>It is often useful to match a pattern and then verify that it occurs again -later in the file. For codegen tests, this can be useful to allow any register, -but verify that that register is used consistently later. To do this, FileCheck -allows named variables to be defined and substituted into patterns. Here is a -simple example:</p> - -<div class="doc_code"> -<pre> -; CHECK: test5: -; CHECK: notw <b>[[REGISTER:%[a-z]+]]</b> -; CHECK: andw {{.*}}<b>[[REGISTER]]</b> -</pre> -</div> - -<p>The first check line matches a regex (<tt>%[a-z]+</tt>) and captures it into -the variables "REGISTER". The second line verifies that whatever is in REGISTER -occurs later in the file after an "andw". FileCheck variable references are -always contained in <tt>[[ ]]</tt> pairs, are named, and their names can be -formed with the regex "<tt>[a-zA-Z][a-zA-Z0-9]*</tt>". If a colon follows the -name, then it is a definition of the variable, if not, it is a use.</p> - -<p>FileCheck variables can be defined multiple times, and uses always get the -latest value. Note that variables are all read at the start of a "CHECK" line -and are all defined at the end. This means that if you have something like -"<tt>CHECK: [[XYZ:.*]]x[[XYZ]]</tt>" that the check line will read the previous -value of the XYZ variable and define a new one after the match is performed. If -you need to do something like this you can probably take advantage of the fact -that FileCheck is not actually line-oriented when it matches, this allows you to -define two separate CHECK lines that match on the same line. -</p> - -<!-- {% endraw %} --> - -</div> - -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="rtvars">Variables and substitutions</a></h3> -<!-- _______________________________________________________________________ --> -<div> - <p>With a RUN line there are a number of substitutions that are permitted. In - general, any Tcl variable that is available in the <tt>substitute</tt> - function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line. - To make a substitution just write the variable's name preceded by a $. - Additionally, for compatibility reasons with previous versions of the test - library, certain names can be accessed with an alternate syntax: a % prefix. - These alternates are deprecated and may go away in a future version. - </p> - <p>Here are the available variable names. The alternate syntax is listed in - parentheses.</p> - - <dl style="margin-left: 25px"> - <dt><b>$test</b> (%s)</dt> - <dd>The full path to the test case's source. This is suitable for passing - on the command line as the input to an llvm tool.</dd> - - <dt><b>%(line), %(line+<i>number</i>), %(line-<i>number</i>)</b></dt> - <dd>The number of the line where this variable is used, with an optional - integer offset. This can be used in tests with multiple RUN: lines, which - reference test file's line numbers.</dd> - - <dt><b>$srcdir</b></dt> - <dd>The source directory from where the "<tt>make check</tt>" was run.</dd> - - <dt><b>objdir</b></dt> - <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd> - - <dt><b>subdir</b></dt> - <dd>A partial path from the <tt>test</tt> directory that contains the - sub-directory that contains the test source being executed.</dd> - - <dt><b>srcroot</b></dt> - <dd>The root directory of the LLVM src tree.</dd> - - <dt><b>objroot</b></dt> - <dd>The root directory of the LLVM object tree. This could be the same - as the srcroot.</dd> - - <dt><b>path</b><dt> - <dd>The path to the directory that contains the test case source. This is - for locating any supporting files that are not generated by the test, but - used by the test.</dd> - - <dt><b>tmp</b></dt> - <dd>The path to a temporary file name that could be used for this test case. - The file name won't conflict with other test cases. You can append to it if - you need multiple temporaries. This is useful as the destination of some - redirected output.</dd> - - <dt><b>target_triplet</b> (%target_triplet)</dt> - <dd>The target triplet that corresponds to the current host machine (the one - running the test cases). This should probably be called "host".<dd> - - <dt><b>link</b> (%link)</dt> - <dd>This full link command used to link LLVM executables. This has all the - configured -I, -L and -l options.</dd> - - <dt><b>shlibext</b> (%shlibext)</dt> - <dd>The suffix for the host platforms share library (dll) files. This - includes the period as the first character.</dd> - </dl> - <p>To add more variables, two things need to be changed. First, add a line in - the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will - "set" the variable as a global in the site.exp file. Second, in the - <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name - to the list of "global" declarations at the beginning of the proc. That's it, - the variable can then be used in test scripts.</p> -</div> - -<!-- _______________________________________________________________________ --> -<h3><a name="rtfeatures">Other Features</a></h3> -<!-- _______________________________________________________________________ --> -<div> - <p>To make RUN line writing easier, there are several shell scripts located - in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH - when running tests, so you can just call these scripts using their name. For - example:</p> - <dl> - <dt><b>ignore</b></dt> - <dd>This script runs its arguments and then always returns 0. This is useful - in cases where the test needs to cause a tool to generate an error (e.g. to - check the error output). However, any program in a pipeline that returns a - non-zero result will cause the test to fail. This script overcomes that - issue and nicely documents that the test case is purposefully ignoring the - result code of the tool</dd> - - <dt><b>not</b></dt> - <dd>This script runs its arguments and then inverts the result code from - it. Zero result codes become 1. Non-zero result codes become 0. This is - useful to invert the result of a grep. For example "not grep X" means - succeed only if you don't find X in the input.</dd> - </dl> - - <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. - You can easily mark a test as XFAIL just by including <tt>XFAIL: </tt> on a - line near the top of the file. This signals that the test case should succeed - if the test fails. Such test cases are counted separately by the testing - tool. To specify an expected fail, use the XFAIL keyword in the comments of - the test program followed by a colon and one or more failure patterns. Each - failure pattern can be either '*' (to specify fail everywhere), or a part of a - target triple (indicating the test should fail on that platform), or the name - of a configurable feature (for example, "loadable_module"). If there is a - match, the test is expected to fail. If not, the test is expected to - succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. Here is an - example of an <tt>XFAIL</tt> line:</p> - -<div class="doc_code"> -<pre> -; XFAIL: darwin,sun -</pre> -</div> - - <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil - scan the lines of the test case for ones that contain a pattern that matches - PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that - is related to the test case. The number after "PR" specifies the LLVM bugzilla - number. When a PR number is specified, it will be used in the pass/fail - reporting. This is useful to quickly get some context when a test fails.</p> - - <p>Finally, any line that contains "END." will cause the special - interpretation of lines to terminate. This is generally done right after the - last RUN: line. This has two side effects: (a) it prevents special - interpretation of lines that are part of the test program, not the - instructions to the test case, and (b) it speeds things up for really big test - cases by avoiding interpretation of the remainder of the file.</p> - -</div> - -</div> - -<!--=========================================================================--> -<h2><a name="testsuiteoverview"><tt>test-suite</tt> Overview</a></h2> -<!--=========================================================================--> - -<div> - -<p>The <tt>test-suite</tt> module contains a number of programs that can be -compiled and executed. The <tt>test-suite</tt> includes reference outputs for -all of the programs, so that the output of the executed program can be checked -for correctness.</p> - -<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource, -SingleSource, and External.</p> - -<ul> -<li><tt>test-suite/SingleSource</tt> -<p>The SingleSource directory contains test programs that are only a single -source file in size. These are usually small benchmark programs or small -programs that calculate a particular value. Several such programs are grouped -together in each directory.</p></li> - -<li><tt>test-suite/MultiSource</tt> -<p>The MultiSource directory contains subdirectories which contain entire -programs with multiple source files. Large benchmarks and whole applications -go here.</p></li> - -<li><tt>test-suite/External</tt> -<p>The External directory contains Makefiles for building code that is external -to (i.e., not distributed with) LLVM. The most prominent members of this -directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt> -directory does not contain these actual tests, but only the Makefiles that know -how to properly compile these programs from somewhere else. When -using <tt>LNT</tt>, use the <tt>--test-externals</tt> option to include these -tests in the results.</p></li> -</ul> -</div> - -<!--=========================================================================--> -<h2><a name="testsuitequickstart"><tt>test-suite</tt> Quickstart</a></h2> -<!--=========================================================================--> - -<div> -<p>The modern way of running the <tt>test-suite</tt> is focused on testing and -benchmarking complete compilers using -the <a href="http://llvm.org/docs/lnt">LNT</a> testing infrastructure.</p> - -<p>For more information on using LNT to execute the <tt>test-suite</tt>, please -see the <a href="http://llvm.org/docs/lnt/quickstart.html">LNT Quickstart</a> -documentation.</p> -</div> - -<!--=========================================================================--> -<h2><a name="testsuitemakefiles"><tt>test-suite</tt> Makefiles</a></h2> -<!--=========================================================================--> - -<div> -<p>Historically, the <tt>test-suite</tt> was executed using a complicated setup -of Makefiles. The LNT based approach above is recommended for most users, but -there are some testing scenarios which are not supported by the LNT approach. In -addition, LNT currently uses the Makefile setup under the covers and so -developers who are interested in how LNT works under the hood may want to -understand the Makefile based setup.</p> - -<p>For more information on the <tt>test-suite</tt> Makefile setup, please see -the <a href="TestSuiteMakefileGuide.html">Test Suite Makefile Guide.</a></p> -</div> - -<!-- *********************************************************************** --> - -<hr> -<address> - <a href="http://jigsaw.w3.org/css-validator/check/referer"><img - src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a> - <a href="http://validator.w3.org/check/referer"><img - src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a> - - John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br> - <a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br> - Last modified: $Date$ -</address> -</body> -</html> diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst new file mode 100644 index 0000000000..db779185a9 --- /dev/null +++ b/docs/TestingGuide.rst @@ -0,0 +1,713 @@ +================================= +LLVM Testing Infrastructure Guide +================================= + +Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya +Lattner + +.. contents:: + :local: + +Overview +======== + +This document is the reference manual for the LLVM testing +infrastructure. It documents the structure of the LLVM testing +infrastructure, the tools needed to use it, and how to add and run +tests. + +Requirements +============ + +In order to use the LLVM testing infrastructure, you will need all of +the software required to build LLVM, as well as +`Python <http://python.org>`_ 2.4 or later. + +LLVM testing infrastructure organization +======================================== + +The LLVM testing infrastructure contains two major categories of tests: +regression tests and whole programs. The regression tests are contained +inside the LLVM repository itself under ``llvm/test`` and are expected +to always pass -- they should be run before every commit. + +The whole programs tests are referred to as the "LLVM test suite" (or +"test-suite") and are in the ``test-suite`` module in subversion. For +historical reasons, these tests are also referred to as the "nightly +tests" in places, which is less ambiguous than "test-suite" and remains +in use although we run them much more often than nightly. + +Regression tests +---------------- + +The regression tests are small pieces of code that test a specific +feature of LLVM or trigger a specific bug in LLVM. They are usually +written in LLVM assembly language, but can be written in other languages +if the test targets a particular language front end (and the appropriate +``--with-llvmgcc`` options were used at ``configure`` time of the +``llvm`` module). These tests are driven by the 'lit' testing tool, +which is part of LLVM. + +These code fragments are not complete programs. The code generated from +them is never executed to determine correct behavior. + +These code fragment tests are located in the ``llvm/test`` directory. + +Typically when a bug is found in LLVM, a regression test containing just +enough code to reproduce the problem should be written and placed +somewhere underneath this directory. In most cases, this will be a small +piece of LLVM assembly language code, often distilled from an actual +application or benchmark. + +``test-suite`` +-------------- + +The test suite contains whole programs, which are pieces of code which +can be compiled and linked into a stand-alone program that can be +executed. These programs are generally written in high level languages +such as C or C++. + +These programs are compiled using a user specified compiler and set of +flags, and then executed to capture the program output and timing +information. The output of these programs is compared to a reference +output to ensure that the program is being compiled correctly. + +In addition to compiling and executing programs, whole program tests +serve as a way of benchmarking LLVM performance, both in terms of the +efficiency of the programs generated as well as the speed with which +LLVM compiles, optimizes, and generates code. + +The test-suite is located in the ``test-suite`` Subversion module. + +Debugging Information tests +--------------------------- + +The test suite contains tests to check quality of debugging information. +The test are written in C based languages or in LLVM assembly language. + +These tests are compiled and run under a debugger. The debugger output +is checked to validate of debugging information. See README.txt in the +test suite for more information . This test suite is located in the +``debuginfo-tests`` Subversion module. + +Quick start +=========== + +The tests are located in two separate Subversion modules. The +regressions tests are in the main "llvm" module under the directory +``llvm/test`` (so you get these tests for free with the main llvm tree). +Use "make check-all" to run the regression tests after building LLVM. + +The more comprehensive test suite that includes whole programs in C and +C++ is in the ``test-suite`` module. See ```test-suite`` +Quickstart <#testsuitequickstart>`_ for more information on running +these tests. + +Regression tests +---------------- + +To run all of the LLVM regression tests, use master Makefile in the +``llvm/test`` directory: + +.. code-block:: bash + + % gmake -C llvm/test + +or + +.. code-block:: bash + + % gmake check + +If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you +can run the LLVM and Clang tests simultaneously using: + +or + +.. code-block:: bash + + % gmake check-all + +To run the tests with Valgrind (Memcheck by default), just append +``VG=1`` to the commands above, e.g.: + +.. code-block:: bash + + % gmake check VG=1 + +To run individual tests or subsets of tests, you can use the 'llvm-lit' +script which is built as part of LLVM. For example, to run the +'Integer/BitPacked.ll' test by itself you can run: + +.. code-block:: bash + + % llvm-lit ~/llvm/test/Integer/BitPacked.ll + +or to run all of the ARM CodeGen tests: + +.. code-block:: bash + + % llvm-lit ~/llvm/test/CodeGen/ARM + +For more information on using the 'lit' tool, see 'llvm-lit --help' or +the 'lit' man page. + +Debugging Information tests +--------------------------- + +To run debugging information tests simply checkout the tests inside +clang/test directory. + +.. code-block:: bash + + % cd clang/test + % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests + +These tests are already set up to run as part of clang regression tests. + +Regression test structure +========================= + +The LLVM regression tests are driven by 'lit' and are located in the +``llvm/test`` directory. + +This directory contains a large array of small tests that exercise +various features of LLVM and to ensure that regressions do not occur. +The directory is broken into several sub-directories, each focused on a +particular area of LLVM. A few of the important ones are: + +- ``Analysis``: checks Analysis passes. +- ``Archive``: checks the Archive library. +- ``Assembler``: checks Assembly reader/writer functionality. +- ``Bitcode``: checks Bitcode reader/writer functionality. +- ``CodeGen``: checks code generation and each target. +- ``Features``: checks various features of the LLVM language. +- ``Linker``: tests bitcode linking. +- ``Transforms``: tests each of the scalar, IPO, and utility transforms + to ensure they make the right transformations. +- ``Verifier``: tests the IR verifier. + +Writing new regression tests +---------------------------- + +The regression test structure is very simple, but does require some +information to be set. This information is gathered via ``configure`` +and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The +``llvm/test`` Makefile does this work for you. + +In order for the regression tests to work, each directory of tests must +have a ``lit.local.cfg`` file. Lit looks for this file to determine how +to run the tests. This file is just Python code and thus is very +flexible, but we've standardized it for the LLVM regression tests. If +you're adding a directory of tests, just copy ``lit.local.cfg`` from +another directory to get running. The standard ``lit.local.cfg`` simply +specifies which files to look in for tests. Any directory that contains +only directories does not need the ``lit.local.cfg`` file. Read the `Lit +documentation <http://llvm.org/cmds/lit.html>`_ for more information. + +The ``llvm-runtests`` function looks at each file that is passed to it +and gathers any lines together that match "RUN:". These are the "RUN" +lines that specify how the test is to be run. So, each test script must +contain RUN lines if it is to do anything. If there are no RUN lines, +the ``llvm-runtests`` function will issue an error and the test will +fail. + +RUN lines are specified in the comments of the test program using the +keyword ``RUN`` followed by a colon, and lastly the command (pipeline) +to execute. Together, these lines form the "script" that +``llvm-runtests`` executes to run the test case. The syntax of the RUN +lines is similar to a shell's syntax for pipelines including I/O +redirection and variable substitution. However, even though these lines +may *look* like a shell script, they are not. RUN lines are interpreted +directly by the Tcl ``exec`` command. They are never executed by a +shell. Consequently the syntax differs from normal shell script syntax +in a few ways. You can specify as many RUN lines as needed. + +lit performs substitution on each RUN line to replace LLVM tool names +with the full paths to the executable built for each tool (in +$(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not +invoke any stray LLVM tools in the user's path during testing. + +Each RUN line is executed on its own, distinct from other lines unless +its last character is ``\``. This continuation character causes the RUN +line to be concatenated with the next one. In this way you can build up +long pipelines of commands without making huge line lengths. The lines +ending in ``\`` are concatenated until a RUN line that doesn't end in +``\`` is found. This concatenated set of RUN lines then constitutes one +execution. Tcl will substitute variables and arrange for the pipeline to +be executed. If any process in the pipeline fails, the entire line (and +test case) fails too. + +Below is an example of legal RUN lines in a ``.ll`` file: + +.. code-block:: llvm + + ; RUN: llvm-as < %s | llvm-dis > %t1 + ; RUN: llvm-dis < %s.bc-13 > %t2 + ; RUN: diff %t1 %t2 + +As with a Unix shell, the RUN: lines permit pipelines and I/O +redirection to be used. However, the usage is slightly different than +for Bash. To check what's legal, see the documentation for the `Tcl +exec <http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2>`_ command and the +`tutorial <http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html>`_. The +major differences are: + +- You can't do ``2>&1``. That will cause Tcl to write to a file named + ``&1``. Usually this is done to get stderr to go through a pipe. You + can do that in tcl with ``|&`` so replace this idiom: + ``... 2>&1 | grep`` with ``... |& grep`` +- You can only redirect to a file, not to another descriptor and not + from a here document. +- tcl supports redirecting to open files with the @ syntax but you + shouldn't use that here. + +There are some quoting rules that you must pay attention to when writing +your RUN lines. In general nothing needs to be quoted. Tcl won't strip +off any quote characters so they will get passed to the invoked program. +For example: + +.. code-block:: bash + + ... | grep 'find this string' + +This will fail because the ' characters are passed to grep. This would +instruction grep to look for ``'find`` in the files ``this`` and +``string'``. To avoid this use curly braces to tell Tcl that it should +treat everything enclosed as one value. So our example would become: + +.. code-block:: bash + + ... | grep {find this string} + +Additionally, the characters ``[`` and ``]`` are treated specially by +Tcl. They tell Tcl to interpret the content as a command to execute. +Since these characters are often used in regular expressions this can +have disastrous results and cause the entire test run in a directory to +fail. For example, a common idiom is to look for some basicblock number: + +.. code-block:: bash + + ... | grep bb[2-8] + +This, however, will cause Tcl to fail because its going to try to +execute a program named "2-8". Instead, what you want is this: + +.. code-block:: bash + + ... | grep {bb\[2-8\]} + +Finally, if you need to pass the ``\`` character down to a program, then +it must be doubled. This is another Tcl special character. So, suppose +you had: + +.. code-block:: bash + + ... | grep 'i32\*' + +This will fail to match what you want (a pointer to i32). First, the +``'`` do not get stripped off. Second, the ``\`` gets stripped off by +Tcl so what grep sees is: ``'i32*'``. That's not likely to match +anything. To resolve this you must use ``\\`` and the ``{}``, like this: + +.. code-block:: bash + + ... | grep {i32\\*} + +If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is +not set in your environment. Otherwise, you may get invalid results +(both false positives and false negatives). + +The FileCheck utility +--------------------- + +A powerful feature of the RUN: lines is that it allows any arbitrary +commands to be executed as part of the test harness. While standard +(portable) unix tools like 'grep' work fine on run lines, as you see +above, there are a lot of caveats due to interaction with Tcl syntax, +and we want to make sure the run lines are portable to a wide range of +systems. Another major problem is that grep is not very good at checking +to verify that the output of a tools contains a series of different +output in a specific order. The FileCheck tool was designed to help with +these problems. + +FileCheck (whose basic command line arguments are described in `the +FileCheck man page <http://llvm.org/cmds/FileCheck.html>`_ is designed +to read a file to check from standard input, and the set of things to +verify from a file specified as a command line argument. A simple +example of using FileCheck from a RUN line looks like this: + +.. code-block:: llvm + + ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s + +This syntax says to pipe the current file ("%s") into llvm-as, pipe that +into llc, then pipe the output of llc into FileCheck. This means that +FileCheck will be verifying its standard input (the llc output) against +the filename argument specified (the original .ll file specified by +"%s"). To see how this works, let's look at the rest of the .ll file +(after the RUN line): + +.. code-block:: llvm + + define void @sub1(i32* %p, i32 %v) { + entry: + ; CHECK: sub1: + ; CHECK: subl + %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v) + ret void + } + + define void @inc4(i64* %p) { + entry: + ; CHECK: inc4: + ; CHECK: incq + %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1) + ret void + } + +Here you can see some "CHECK:" lines specified in comments. Now you can +see how the file is piped into llvm-as, then llc, and the machine code +output is what we are verifying. FileCheck checks the machine code +output to verify that it matches what the "CHECK:" lines specify. + +The syntax of the CHECK: lines is very simple: they are fixed strings +that must occur in order. FileCheck defaults to ignoring horizontal +whitespace differences (e.g. a space is allowed to match a tab) but +otherwise, the contents of the CHECK: line is required to match some +thing in the test file exactly. + +One nice thing about FileCheck (compared to grep) is that it allows +merging test cases together into logical groups. For example, because +the test above is checking for the "sub1:" and "inc4:" labels, it will +not match unless there is a "subl" in between those labels. If it +existed somewhere else in the file, that would not count: "grep subl" +matches if subl exists anywhere in the file. + +The FileCheck -check-prefix option +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The FileCheck -check-prefix option allows multiple test configurations +to be driven from one .ll file. This is useful in many circumstances, +for example, testing different architectural variants with llc. Here's a +simple example: + +.. code-block:: llvm + + ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \ + ; RUN: | FileCheck %s -check-prefix=X32 + ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \ + ; RUN: | FileCheck %s -check-prefix=X64 + + define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind { + %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1 + ret <4 x i32> %tmp1 + ; X32: pinsrd_1: + ; X32: pinsrd $1, 4(%esp), %xmm0 + + ; X64: pinsrd_1: + ; X64: pinsrd $1, %edi, %xmm0 + } + +In this case, we're testing that we get the expected code generation +with both 32-bit and 64-bit code generation. + +The "CHECK-NEXT:" directive +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sometimes you want to match lines and would like to verify that matches +happen on exactly consecutive lines with no other lines in between them. +In this case, you can use CHECK: and CHECK-NEXT: directives to specify +this. If you specified a custom check prefix, just use "<PREFIX>-NEXT:". +For example, something like this works as you'd expect: + +.. code-block:: llvm + + define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) { + %tmp3 = load <2 x double>* %A, align 16 + %tmp7 = insertelement <2 x double> undef, double %B, i32 0 + %tmp9 = shufflevector <2 x double> %tmp3, + <2 x double> %tmp7, + <2 x i32> < i32 0, i32 2 > + store <2 x double> %tmp9, <2 x double>* %r, align 16 + ret void + + ; CHECK: t2: + ; CHECK: movl 8(%esp), %eax + ; CHECK-NEXT: movapd (%eax), %xmm0 + ; CHECK-NEXT: movhpd 12(%esp), %xmm0 + ; CHECK-NEXT: movl 4(%esp), %eax + ; CHECK-NEXT: movapd %xmm0, (%eax) + ; CHECK-NEXT: ret + } + +CHECK-NEXT: directives reject the input unless there is exactly one +newline between it an the previous directive. A CHECK-NEXT cannot be the +first directive in a file. + +The "CHECK-NOT:" directive +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The CHECK-NOT: directive is used to verify that a string doesn't occur +between two matches (or the first match and the beginning of the file). +For example, to verify that a load is removed by a transformation, a +test like this can be used: + +.. code-block:: llvm + + define i8 @coerce_offset0(i32 %V, i32* %P) { + store i32 %V, i32* %P + + %P2 = bitcast i32* %P to i8* + %P3 = getelementptr i8* %P2, i32 2 + + %A = load i8* %P3 + ret i8 %A + ; CHECK: @coerce_offset0 + ; CHECK-NOT: load + ; CHECK: ret i8 + } + +FileCheck Pattern Matching Syntax +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The CHECK: and CHECK-NOT: directives both take a pattern to match. For +most uses of FileCheck, fixed string matching is perfectly sufficient. +For some things, a more flexible form of matching is desired. To support +this, FileCheck allows you to specify regular expressions in matching +strings, surrounded by double braces: **{{yourregex}}**. Because we want +to use fixed string matching for a majority of what we do, FileCheck has +been designed to support mixing and matching fixed string matching with +regular expressions. This allows you to write things like this: + +.. code-block:: llvm + + ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}} + +In this case, any offset from the ESP register will be allowed, and any +xmm register will be allowed. + +Because regular expressions are enclosed with double braces, they are +visually distinct, and you don't need to use escape characters within +the double braces like you would in C. In the rare case that you want to +match double braces explicitly from the input, you can use something +ugly like **{{[{][{]}}** as your pattern. + +FileCheck Variables +^^^^^^^^^^^^^^^^^^^ + +It is often useful to match a pattern and then verify that it occurs +again later in the file. For codegen tests, this can be useful to allow +any register, but verify that that register is used consistently later. +To do this, FileCheck allows named variables to be defined and +substituted into patterns. Here is a simple example: + +.. code-block:: llvm + + ; CHECK: test5: + ; CHECK: notw [[REGISTER:%[a-z]+]] + ; CHECK: andw {{.*}}[[REGISTER]] + +The first check line matches a regex (``%[a-z]+``) and captures it into +the variables "REGISTER". The second line verifies that whatever is in +REGISTER occurs later in the file after an "andw". FileCheck variable +references are always contained in ``[[ ]]`` pairs, are named, and their +names can be formed with the regex "``[a-zA-Z][a-zA-Z0-9]*``". If a +colon follows the name, then it is a definition of the variable, if not, +it is a use. + +FileCheck variables can be defined multiple times, and uses always get +the latest value. Note that variables are all read at the start of a +"CHECK" line and are all defined at the end. This means that if you have +something like "``CHECK: [[XYZ:.*]]x[[XYZ]]``" that the check line will +read the previous value of the XYZ variable and define a new one after +the match is performed. If you need to do something like this you can +probably take advantage of the fact that FileCheck is not actually +line-oriented when it matches, this allows you to define two separate +CHECK lines that match on the same line. + +Variables and substitutions +--------------------------- + +With a RUN line there are a number of substitutions that are permitted. +In general, any Tcl variable that is available in the ``substitute`` +function (in ``test/lib/llvm.exp``) can be substituted into a RUN line. +To make a substitution just write the variable's name preceded by a $. +Additionally, for compatibility reasons with previous versions of the +test library, certain names can be accessed with an alternate syntax: a +% prefix. These alternates are deprecated and may go away in a future +version. + +Here are the available variable names. The alternate syntax is listed in +parentheses. + +``$test`` (``%s``) + The full path to the test case's source. This is suitable for passing on + the command line as the input to an llvm tool. + +``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` + The number of the line where this variable is used, with an optional + integer offset. This can be used in tests with multiple RUN: lines, + which reference test file's line numbers. + +``$srcdir`` + The source directory from where the "``make check``" was run. + +``objdir`` + The object directory that corresponds to the ``$srcdir``. + +``subdir`` + A partial path from the ``test`` directory that contains the + sub-directory that contains the test source being executed. + +``srcroot`` + The root directory of the LLVM src tree. + +``objroot`` + The root directory of the LLVM object tree. This could be the same as + the srcroot. + +``path`` + The path to the directory that contains the test case source. This is + for locating any supporting files that are not generated by the test, + but used by the test. + +``tmp`` + The path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it + if you need multiple temporaries. This is useful as the destination of + some redirected output. + +``target_triplet`` (``%target_triplet``) + The target triplet that corresponds to the current host machine (the one + running the test cases). This should probably be called "host". + +``link`` (``%link``) + This full link command used to link LLVM executables. This has all the + configured -I, -L and -l options. + +``shlibext`` (``%shlibext``) + The suffix for the host platforms share library (dll) files. This + includes the period as the first character. + +To add more variables, two things need to be changed. First, add a line +in the ``test/Makefile`` that creates the ``site.exp`` file. This will +"set" the variable as a global in the site.exp file. Second, in the +``test/lib/llvm.exp`` file, in the substitute proc, add the variable +name to the list of "global" declarations at the beginning of the proc. +That's it, the variable can then be used in test scripts. + +Other Features +-------------- + +To make RUN line writing easier, there are several shell scripts located +in the ``llvm/test/Scripts`` directory. This directory is in the PATH +when running tests, so you can just call these scripts using their name. +For example: + +``ignore`` + This script runs its arguments and then always returns 0. This is useful + in cases where the test needs to cause a tool to generate an error (e.g. + to check the error output). However, any program in a pipeline that + returns a non-zero result will cause the test to fail. This script + overcomes that issue and nicely documents that the test case is + purposefully ignoring the result code of the tool +``not`` + This script runs its arguments and then inverts the result code from it. + Zero result codes become 1. Non-zero result codes become 0. This is + useful to invert the result of a grep. For example "not grep X" means + succeed only if you don't find X in the input. + +Sometimes it is necessary to mark a test case as "expected fail" or +XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` +on a line near the top of the file. This signals that the test case +should succeed if the test fails. Such test cases are counted separately +by the testing tool. To specify an expected fail, use the XFAIL keyword +in the comments of the test program followed by a colon and one or more +failure patterns. Each failure pattern can be either ``*`` (to specify +fail everywhere), or a part of a target triple (indicating the test +should fail on that platform), or the name of a configurable feature +(for example, ``loadable_module``). If there is a match, the test is +expected to fail. If not, the test is expected to succeed. To XFAIL +everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` +line: + +.. code-block:: llvm + + ; XFAIL: darwin,sun + +To make the output more useful, the ``llvm_runtest`` function wil scan +the lines of the test case for ones that contain a pattern that matches +``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number +that is related to the test case. The number after "PR" specifies the +LLVM bugzilla number. When a PR number is specified, it will be used in +the pass/fail reporting. This is useful to quickly get some context when +a test fails. + +Finally, any line that contains "END." will cause the special +interpretation of lines to terminate. This is generally done right after +the last RUN: line. This has two side effects: + +(a) it prevents special interpretation of lines that are part of the test + program, not the instructions to the test case, and + +(b) it speeds things up for really big test cases by avoiding + interpretation of the remainder of the file. + +``test-suite`` Overview +======================= + +The ``test-suite`` module contains a number of programs that can be +compiled and executed. The ``test-suite`` includes reference outputs for +all of the programs, so that the output of the executed program can be +checked for correctness. + +``test-suite`` tests are divided into three types of tests: MultiSource, +SingleSource, and External. + +- ``test-suite/SingleSource`` + + The SingleSource directory contains test programs that are only a + single source file in size. These are usually small benchmark + programs or small programs that calculate a particular value. Several + such programs are grouped together in each directory. + +- ``test-suite/MultiSource`` + + The MultiSource directory contains subdirectories which contain + entire programs with multiple source files. Large benchmarks and + whole applications go here. + +- ``test-suite/External`` + + The External directory contains Makefiles for building code that is + external to (i.e., not distributed with) LLVM. The most prominent + members of this directory are the SPEC 95 and SPEC 2000 benchmark + suites. The ``External`` directory does not contain these actual + tests, but only the Makefiles that know how to properly compile these + programs from somewhere else. When using ``LNT``, use the + ``--test-externals`` option to include these tests in the results. + +``test-suite`` Quickstart +------------------------- + +The modern way of running the ``test-suite`` is focused on testing and +benchmarking complete compilers using the +`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. + +For more information on using LNT to execute the ``test-suite``, please +see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ +documentation. + +``test-suite`` Makefiles +------------------------ + +Historically, the ``test-suite`` was executed using a complicated setup +of Makefiles. The LNT based approach above is recommended for most +users, but there are some testing scenarios which are not supported by +the LNT approach. In addition, LNT currently uses the Makefile setup +under the covers and so developers who are interested in how LNT works +under the hood may want to understand the Makefile based setup. + +For more information on the ``test-suite`` Makefile setup, please see +the `Test Suite Makefile Guide. <TestSuiteMakefileGuide.html>`_ diff --git a/docs/userguides.rst b/docs/userguides.rst index 8c1554dfce..8f184205c8 100644 --- a/docs/userguides.rst +++ b/docs/userguides.rst @@ -20,6 +20,7 @@ User Guides HowToSubmitABug SphinxQuickstartTemplate Phabricator + TestingGuide * :ref:`getting_started` @@ -77,7 +78,7 @@ User Guides A template + tutorial for writing new Sphinx documentation. It is meant to be read in source form. -* `LLVM Testing Infrastructure Guide <TestingGuide.html>`_ +* :doc:`LLVM Testing Infrastructure Guide <TestingGuide>` A reference manual for using the LLVM testing infrastructure. |