diff options
| author | Sean Silva <silvas@purdue.edu> | 2012-11-14 23:11:10 +0000 |
|---|---|---|
| committer | Sean Silva <silvas@purdue.edu> | 2012-11-14 23:11:10 +0000 |
| commit | d5f4b4ca0de666ea3ad754a7a1e9c1ddc93ef252 (patch) | |
| tree | 0241598b613f59b15c1c820b8e14e9f71ed548be /docs | |
| parent | 8762a6bc2b83dfc11e784772311ad0b8fa5211e2 (diff) | |
| download | llvm-d5f4b4ca0de666ea3ad754a7a1e9c1ddc93ef252.tar.gz llvm-d5f4b4ca0de666ea3ad754a7a1e9c1ddc93ef252.tar.bz2 llvm-d5f4b4ca0de666ea3ad754a7a1e9c1ddc93ef252.tar.xz | |
docs: Sphinxify TestSuiteMakefileGuide
Some small related fixups to TestingGuide too.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167996 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/TestSuiteMakefileGuide.html | 351 | ||||
| -rw-r--r-- | docs/TestSuiteMakefileGuide.rst | 279 | ||||
| -rw-r--r-- | docs/TestingGuide.rst | 16 |
3 files changed, 290 insertions, 356 deletions
diff --git a/docs/TestSuiteMakefileGuide.html b/docs/TestSuiteMakefileGuide.html deleted file mode 100644 index 1b24250380..0000000000 --- a/docs/TestSuiteMakefileGuide.html +++ /dev/null @@ -1,351 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" - "http://www.w3.org/TR/html4/strict.dtd"> -<html> -<head> - <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> - <title>LLVM test-suite Makefile Guide</title> - <link rel="stylesheet" href="_static/llvm.css" type="text/css"> -</head> -<body> - -<h1> - LLVM test-suite Makefile Guide -</h1> - -<ol> - <li><a href="#overview">Overview</a></li> - <li><a href="#testsuitestructure">Test suite structure</a></li> - <li><a href="#testsuiterun">Running the test suite</a> - <ul> - <li><a href="#testsuiteexternal">Configuring External Tests</a></li> - <li><a href="#testsuitetests">Running different tests</a></li> - <li><a href="#testsuiteoutput">Generating test output</a></li> - <li><a href="#testsuitecustom">Writing custom tests for test-suite</a></li> - </ul> - </li> -</ol> - -<div class="doc_author"> - <p>Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner</p> -</div> - -<!--=========================================================================--> -<h2><a name="overview">Overview</a></h2> -<!--=========================================================================--> - -<div> - -<p>This document describes the features of the Makefile-based LLVM -test-suite. This way of interacting with the test-suite is deprecated in favor -of running the test-suite using LNT, but may continue to prove useful for some -users. See the Testing -Guide's <a href="TestingGuide.html#testsuitequickstart">test-suite -Quickstart</a> section for more information.</p> - -</div> - -<!--=========================================================================--> -<h2><a name="testsuitestructure">Test suite Structure</a></h2> -<!--=========================================================================--> - -<div> - -<p>The <tt>test-suite</tt> module contains a number of programs that can be compiled -with LLVM and executed. These programs are compiled using the native compiler -and various LLVM backends. The output from the program compiled with the -native compiler is assumed correct; the results from the other programs are -compared to the native program output and pass if they match.</p> - -<p>When executing tests, it is usually a good idea to start out with a subset of -the available tests or programs. This makes test run times smaller at first and -later on this is useful to investigate individual test failures. To run some -test only on a subset of programs, simply change directory to the programs you -want tested and run <tt>gmake</tt> there. Alternatively, you can run a different -test using the <tt>TEST</tt> variable to change what tests or run on the -selected programs (see below for more info).</p> - -<p>In addition for testing correctness, the <tt>test-suite</tt> directory also -performs timing tests of various LLVM optimizations. It also records -compilation times for the compilers and the JIT. This information can be -used to compare the effectiveness of LLVM's optimizations and code -generation.</p> - -<p><tt>test-suite</tt> tests are divided into three types of tests: MultiSource, -SingleSource, and External.</p> - -<ul> -<li><tt>test-suite/SingleSource</tt> -<p>The SingleSource directory contains test programs that are only a single -source file in size. These are usually small benchmark programs or small -programs that calculate a particular value. Several such programs are grouped -together in each directory.</p></li> - -<li><tt>test-suite/MultiSource</tt> -<p>The MultiSource directory contains subdirectories which contain entire -programs with multiple source files. Large benchmarks and whole applications -go here.</p></li> - -<li><tt>test-suite/External</tt> -<p>The External directory contains Makefiles for building code that is external -to (i.e., not distributed with) LLVM. The most prominent members of this -directory are the SPEC 95 and SPEC 2000 benchmark suites. The <tt>External</tt> -directory does not contain these actual tests, but only the Makefiles that know -how to properly compile these programs from somewhere else. The presence and -location of these external programs is configured by the test-suite -<tt>configure</tt> script.</p></li> -</ul> - -<p>Each tree is then subdivided into several categories, including applications, -benchmarks, regression tests, code that is strange grammatically, etc. These -organizations should be relatively self explanatory.</p> - -<p>Some tests are known to fail. Some are bugs that we have not fixed yet; -others are features that we haven't added yet (or may never add). In the -regression tests, the result for such tests will be XFAIL (eXpected FAILure). -In this way, you can tell the difference between an expected and unexpected -failure.</p> - -<p>The tests in the test suite have no such feature at this time. If the -test passes, only warnings and other miscellaneous output will be generated. If -a test fails, a large <program> FAILED message will be displayed. This -will help you separate benign warnings from actual test failures.</p> - -</div> - -<!--=========================================================================--> -<h2><a name="testsuiterun">Running the test suite</a></h2> -<!--=========================================================================--> - -<div> - -<p>First, all tests are executed within the LLVM object directory tree. They -<i>are not</i> executed inside of the LLVM source tree. This is because the -test suite creates temporary files during execution.</p> - -<p>To run the test suite, you need to use the following steps:</p> - -<ol> - <li><tt>cd</tt> into the <tt>llvm/projects</tt> directory in your source tree. - </li> - - <li><p>Check out the <tt>test-suite</tt> module with:</p> - -<div class="doc_code"> -<pre> -% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite -</pre> -</div> - <p>This will get the test suite into <tt>llvm/projects/test-suite</tt>.</p> - </li> - <li><p>Configure and build <tt>llvm</tt>.</p></li> - <li><p>Configure and build <tt>llvm-gcc</tt>.</p></li> - <li><p>Install <tt>llvm-gcc</tt> somewhere.</p></li> - <li><p><em>Re-configure</em> <tt>llvm</tt> from the top level of - each build tree (LLVM object directory tree) in which you want - to run the test suite, just as you do before building LLVM.</p> - <p>During the <em>re-configuration</em>, you must either: (1) - have <tt>llvm-gcc</tt> you just built in your path, or (2) - specify the directory where your just-built <tt>llvm-gcc</tt> is - installed using <tt>--with-llvmgccdir=$LLVM_GCC_DIR</tt>.</p> - <p>You must also tell the configure machinery that the test suite - is available so it can be configured for your build tree:</p> -<div class="doc_code"> -<pre> -% cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR] -</pre> -</div> - <p>[Remember that <tt>$LLVM_GCC_DIR</tt> is the directory where you - <em>installed</em> llvm-gcc, not its src or obj directory.]</p> - </li> - - <li><p>You can now run the test suite from your build tree as follows:</p> -<div class="doc_code"> -<pre> -% cd $LLVM_OBJ_ROOT/projects/test-suite -% make -</pre> -</div> - </li> -</ol> -<p>Note that the second and third steps only need to be done once. After you -have the suite checked out and configured, you don't need to do it again (unless -the test code or configure script changes).</p> - -<!-- _______________________________________________________________________ --> -<h3> - <a name="testsuiteexternal">Configuring External Tests</a> -</h3> -<!-- _______________________________________________________________________ --> - -<div> -<p>In order to run the External tests in the <tt>test-suite</tt> - module, you must specify <i>--with-externals</i>. This - must be done during the <em>re-configuration</em> step (see above), - and the <tt>llvm</tt> re-configuration must recognize the - previously-built <tt>llvm-gcc</tt>. If any of these is missing or - neglected, the External tests won't work.</p> -<dl> -<dt><i>--with-externals</i></dt> -<dt><i>--with-externals=<<tt>directory</tt>></i></dt> -</dl> - This tells LLVM where to find any external tests. They are expected to be - in specifically named subdirectories of <<tt>directory</tt>>. - If <tt>directory</tt> is left unspecified, - <tt>configure</tt> uses the default value - <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>. - Subdirectory names known to LLVM include: - <dl> - <dt>spec95</dt> - <dt>speccpu2000</dt> - <dt>speccpu2006</dt> - <dt>povray31</dt> - </dl> - Others are added from time to time, and can be determined from - <tt>configure</tt>. -</div> - -<!-- _______________________________________________________________________ --> -<h3> - <a name="testsuitetests">Running different tests</a> -</h3> -<!-- _______________________________________________________________________ --> -<div> -<p>In addition to the regular "whole program" tests, the <tt>test-suite</tt> -module also provides a mechanism for compiling the programs in different ways. -If the variable TEST is defined on the <tt>gmake</tt> command line, the test system will -include a Makefile named <tt>TEST.<value of TEST variable>.Makefile</tt>. -This Makefile can modify build rules to yield different results.</p> - -<p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to -create the nightly test reports. To run the nightly tests, run <tt>gmake -TEST=nightly</tt>.</p> - -<p>There are several TEST Makefiles available in the tree. Some of them are -designed for internal LLVM research and will not work outside of the LLVM -research group. They may still be valuable, however, as a guide to writing your -own TEST Makefile for any optimization or analysis passes that you develop with -LLVM.</p> - -</div> - -<!-- _______________________________________________________________________ --> -<h3> - <a name="testsuiteoutput">Generating test output</a> -</h3> -<!-- _______________________________________________________________________ --> -<div> - <p>There are a number of ways to run the tests and generate output. The most - simple one is simply running <tt>gmake</tt> with no arguments. This will - compile and run all programs in the tree using a number of different methods - and compare results. Any failures are reported in the output, but are likely - drowned in the other output. Passes are not reported explicitly.</p> - - <p>Somewhat better is running <tt>gmake TEST=sometest test</tt>, which runs - the specified test and usually adds per-program summaries to the output - (depending on which sometest you use). For example, the <tt>nightly</tt> test - explicitly outputs TEST-PASS or TEST-FAIL for every test after each program. - Though these lines are still drowned in the output, it's easy to grep the - output logs in the Output directories.</p> - - <p>Even better are the <tt>report</tt> and <tt>report.format</tt> targets - (where <tt>format</tt> is one of <tt>html</tt>, <tt>csv</tt>, <tt>text</tt> or - <tt>graphs</tt>). The exact contents of the report are dependent on which - <tt>TEST</tt> you are running, but the text results are always shown at the - end of the run and the results are always stored in the - <tt>report.<type>.format</tt> file (when running with - <tt>TEST=<type></tt>). - - The <tt>report</tt> also generate a file called - <tt>report.<type>.raw.out</tt> containing the output of the entire test - run. -</div> - -<!-- _______________________________________________________________________ --> -<h3> - <a name="testsuitecustom">Writing custom tests for the test suite</a> -</h3> -<!-- _______________________________________________________________________ --> - -<div> - -<p>Assuming you can run the test suite, (e.g. "<tt>gmake TEST=nightly report</tt>" -should work), it is really easy to run optimizations or code generator -components against every program in the tree, collecting statistics or running -custom checks for correctness. At base, this is how the nightly tester works, -it's just one example of a general framework.</p> - -<p>Lets say that you have an LLVM optimization pass, and you want to see how -many times it triggers. First thing you should do is add an LLVM -<a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which -will tally counts of things you care about.</p> - -<p>Following this, you can set up a test and a report that collects these and -formats them for easy viewing. This consists of two files, a -"<tt>test-suite/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your -test) and a "<tt>test-suite/TEST.XXX.report</tt>" file that indicates how to -format the output into a table. There are many example reports of various -levels of sophistication included with the test suite, and the framework is very -general.</p> - -<p>If you are interested in testing an optimization pass, check out the -"libcalls" test as an example. It can be run like this:<p> - -<div class="doc_code"> -<pre> -% cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level -% make TEST=libcalls report -</pre> -</div> - -<p>This will do a bunch of stuff, then eventually print a table like this:</p> - -<div class="doc_code"> -<pre> -Name | total | #exit | -... -FreeBench/analyzer/analyzer | 51 | 6 | -FreeBench/fourinarow/fourinarow | 1 | 1 | -FreeBench/neural/neural | 19 | 9 | -FreeBench/pifft/pifft | 5 | 3 | -MallocBench/cfrac/cfrac | 1 | * | -MallocBench/espresso/espresso | 52 | 12 | -MallocBench/gs/gs | 4 | * | -Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * | -Prolangs-C/agrep/agrep | 33 | 12 | -Prolangs-C/allroots/allroots | * | * | -Prolangs-C/assembler/assembler | 47 | * | -Prolangs-C/bison/mybison | 74 | * | -... -</pre> -</div> - -<p>This basically is grepping the -stats output and displaying it in a table. -You can also use the "TEST=libcalls report.html" target to get the table in HTML -form, similarly for report.csv and report.tex.</p> - -<p>The source for this is in test-suite/TEST.libcalls.*. The format is pretty -simple: the Makefile indicates how to run the test (in this case, -"<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for -each column of the output. The first value is the header for the column and the -second is the regex to grep the output of the command for. There are lots of -example reports that can do fancy stuff.</p> - -</div> - -</div> - -<!-- *********************************************************************** --> - -<hr> -<address> - <a href="http://jigsaw.w3.org/css-validator/check/referer"><img - src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS"></a> - <a href="http://validator.w3.org/check/referer"><img - src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01"></a> - - John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner<br> - <a href="http://llvm.org/">The LLVM Compiler Infrastructure</a><br> - Last modified: $Date$ -</address> -</body> -</html> diff --git a/docs/TestSuiteMakefileGuide.rst b/docs/TestSuiteMakefileGuide.rst new file mode 100644 index 0000000000..b10379ef4d --- /dev/null +++ b/docs/TestSuiteMakefileGuide.rst @@ -0,0 +1,279 @@ +============================== +LLVM test-suite Makefile Guide +============================== + +Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya +Lattner + +.. contents:: + :local: + +Overview +======== + +This document describes the features of the Makefile-based LLVM +test-suite. This way of interacting with the test-suite is deprecated in +favor of running the test-suite using LNT, but may continue to prove +useful for some users. See the Testing Guide's :ref:`test-suite Quickstart +<test-suite-quickstart>` section for more information. + +Test suite Structure +==================== + +The ``test-suite`` module contains a number of programs that can be +compiled with LLVM and executed. These programs are compiled using the +native compiler and various LLVM backends. The output from the program +compiled with the native compiler is assumed correct; the results from +the other programs are compared to the native program output and pass if +they match. + +When executing tests, it is usually a good idea to start out with a +subset of the available tests or programs. This makes test run times +smaller at first and later on this is useful to investigate individual +test failures. To run some test only on a subset of programs, simply +change directory to the programs you want tested and run ``gmake`` +there. Alternatively, you can run a different test using the ``TEST`` +variable to change what tests or run on the selected programs (see below +for more info). + +In addition for testing correctness, the ``test-suite`` directory also +performs timing tests of various LLVM optimizations. It also records +compilation times for the compilers and the JIT. This information can be +used to compare the effectiveness of LLVM's optimizations and code +generation. + +``test-suite`` tests are divided into three types of tests: MultiSource, +SingleSource, and External. + +- ``test-suite/SingleSource`` + + The SingleSource directory contains test programs that are only a + single source file in size. These are usually small benchmark + programs or small programs that calculate a particular value. Several + such programs are grouped together in each directory. + +- ``test-suite/MultiSource`` + + The MultiSource directory contains subdirectories which contain + entire programs with multiple source files. Large benchmarks and + whole applications go here. + +- ``test-suite/External`` + + The External directory contains Makefiles for building code that is + external to (i.e., not distributed with) LLVM. The most prominent + members of this directory are the SPEC 95 and SPEC 2000 benchmark + suites. The ``External`` directory does not contain these actual + tests, but only the Makefiles that know how to properly compile these + programs from somewhere else. The presence and location of these + external programs is configured by the test-suite ``configure`` + script. + +Each tree is then subdivided into several categories, including +applications, benchmarks, regression tests, code that is strange +grammatically, etc. These organizations should be relatively self +explanatory. + +Some tests are known to fail. Some are bugs that we have not fixed yet; +others are features that we haven't added yet (or may never add). In the +regression tests, the result for such tests will be XFAIL (eXpected +FAILure). In this way, you can tell the difference between an expected +and unexpected failure. + +The tests in the test suite have no such feature at this time. If the +test passes, only warnings and other miscellaneous output will be +generated. If a test fails, a large <program> FAILED message will be +displayed. This will help you separate benign warnings from actual test +failures. + +Running the test suite +====================== + +First, all tests are executed within the LLVM object directory tree. +They *are not* executed inside of the LLVM source tree. This is because +the test suite creates temporary files during execution. + +To run the test suite, you need to use the following steps: + +#. ``cd`` into the ``llvm/projects`` directory in your source tree. +#. Check out the ``test-suite`` module with: + + .. code-block:: bash + + % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite + + This will get the test suite into ``llvm/projects/test-suite``. + +#. Configure and build ``llvm``. + +#. Configure and build ``llvm-gcc``. + +#. Install ``llvm-gcc`` somewhere. + +#. *Re-configure* ``llvm`` from the top level of each build tree (LLVM + object directory tree) in which you want to run the test suite, just + as you do before building LLVM. + + During the *re-configuration*, you must either: (1) have ``llvm-gcc`` + you just built in your path, or (2) specify the directory where your + just-built ``llvm-gcc`` is installed using + ``--with-llvmgccdir=$LLVM_GCC_DIR``. + + You must also tell the configure machinery that the test suite is + available so it can be configured for your build tree: + + .. code-block:: bash + + % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR] + + [Remember that ``$LLVM_GCC_DIR`` is the directory where you + *installed* llvm-gcc, not its src or obj directory.] + +#. You can now run the test suite from your build tree as follows: + + .. code-block:: bash + + % cd $LLVM_OBJ_ROOT/projects/test-suite + % make + +Note that the second and third steps only need to be done once. After +you have the suite checked out and configured, you don't need to do it +again (unless the test code or configure script changes). + +Configuring External Tests +-------------------------- + +In order to run the External tests in the ``test-suite`` module, you +must specify *--with-externals*. This must be done during the +*re-configuration* step (see above), and the ``llvm`` re-configuration +must recognize the previously-built ``llvm-gcc``. If any of these is +missing or neglected, the External tests won't work. + +* *--with-externals* + +* *--with-externals=<directory>* + +This tells LLVM where to find any external tests. They are expected to +be in specifically named subdirectories of <``directory``>. If +``directory`` is left unspecified, ``configure`` uses the default value +``/home/vadve/shared/benchmarks/speccpu2000/benchspec``. Subdirectory +names known to LLVM include: + +* spec95 + +* speccpu2000 + +* speccpu2006 + +* povray31 + +Others are added from time to time, and can be determined from +``configure``. + +Running different tests +----------------------- + +In addition to the regular "whole program" tests, the ``test-suite`` +module also provides a mechanism for compiling the programs in different +ways. If the variable TEST is defined on the ``gmake`` command line, the +test system will include a Makefile named +``TEST.<value of TEST variable>.Makefile``. This Makefile can modify +build rules to yield different results. + +For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to +create the nightly test reports. To run the nightly tests, run +``gmake TEST=nightly``. + +There are several TEST Makefiles available in the tree. Some of them are +designed for internal LLVM research and will not work outside of the +LLVM research group. They may still be valuable, however, as a guide to +writing your own TEST Makefile for any optimization or analysis passes +that you develop with LLVM. + +Generating test output +---------------------- + +There are a number of ways to run the tests and generate output. The +most simple one is simply running ``gmake`` with no arguments. This will +compile and run all programs in the tree using a number of different +methods and compare results. Any failures are reported in the output, +but are likely drowned in the other output. Passes are not reported +explicitly. + +Somewhat better is running ``gmake TEST=sometest test``, which runs the +specified test and usually adds per-program summaries to the output +(depending on which sometest you use). For example, the ``nightly`` test +explicitly outputs TEST-PASS or TEST-FAIL for every test after each +program. Though these lines are still drowned in the output, it's easy +to grep the output logs in the Output directories. + +Even better are the ``report`` and ``report.format`` targets (where +``format`` is one of ``html``, ``csv``, ``text`` or ``graphs``). The +exact contents of the report are dependent on which ``TEST`` you are +running, but the text results are always shown at the end of the run and +the results are always stored in the ``report.<type>.format`` file (when +running with ``TEST=<type>``). The ``report`` also generate a file +called ``report.<type>.raw.out`` containing the output of the entire +test run. + +Writing custom tests for the test suite +--------------------------------------- + +Assuming you can run the test suite, (e.g. +"``gmake TEST=nightly report``" should work), it is really easy to run +optimizations or code generator components against every program in the +tree, collecting statistics or running custom checks for correctness. At +base, this is how the nightly tester works, it's just one example of a +general framework. + +Lets say that you have an LLVM optimization pass, and you want to see +how many times it triggers. First thing you should do is add an LLVM +`statistic <ProgrammersManual.html#Statistic>`_ to your pass, which will +tally counts of things you care about. + +Following this, you can set up a test and a report that collects these +and formats them for easy viewing. This consists of two files, a +"``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of +your test) and a "``test-suite/TEST.XXX.report``" file that indicates +how to format the output into a table. There are many example reports of +various levels of sophistication included with the test suite, and the +framework is very general. + +If you are interested in testing an optimization pass, check out the +"libcalls" test as an example. It can be run like this: + +.. code-block:: bash + + % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level + % make TEST=libcalls report + +This will do a bunch of stuff, then eventually print a table like this: + +:: + + Name | total | #exit | + ... + FreeBench/analyzer/analyzer | 51 | 6 | + FreeBench/fourinarow/fourinarow | 1 | 1 | + FreeBench/neural/neural | 19 | 9 | + FreeBench/pifft/pifft | 5 | 3 | + MallocBench/cfrac/cfrac | 1 | * | + MallocBench/espresso/espresso | 52 | 12 | + MallocBench/gs/gs | 4 | * | + Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * | + Prolangs-C/agrep/agrep | 33 | 12 | + Prolangs-C/allroots/allroots | * | * | + Prolangs-C/assembler/assembler | 47 | * | + Prolangs-C/bison/mybison | 74 | * | + ... + +This basically is grepping the -stats output and displaying it in a +table. You can also use the "TEST=libcalls report.html" target to get +the table in HTML form, similarly for report.csv and report.tex. + +The source for this is in ``test-suite/TEST.libcalls.*``. The format is +pretty simple: the Makefile indicates how to run the test (in this case, +"``opt -simplify-libcalls -stats``"), and the report contains one line +for each column of the output. The first value is the header for the +column and the second is the regex to grep the output of the command +for. There are lots of example reports that can do fancy stuff. diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index db779185a9..7ca49ceac0 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -8,6 +8,11 @@ Lattner .. contents:: :local: +.. toctree:: + :hidden: + + TestSuiteMakefileGuide + Overview ======== @@ -98,10 +103,9 @@ regressions tests are in the main "llvm" module under the directory ``llvm/test`` (so you get these tests for free with the main llvm tree). Use "make check-all" to run the regression tests after building LLVM. -The more comprehensive test suite that includes whole programs in C and -C++ is in the ``test-suite`` module. See ```test-suite`` -Quickstart <#testsuitequickstart>`_ for more information on running -these tests. +The more comprehensive test suite that includes whole programs in C and C++ +is in the ``test-suite`` module. See :ref:`test-suite Quickstart +<test-suite-quickstart>` for more information on running these tests. Regression tests ---------------- @@ -688,6 +692,8 @@ SingleSource, and External. programs from somewhere else. When using ``LNT``, use the ``--test-externals`` option to include these tests in the results. +.. _test-suite-quickstart: + ``test-suite`` Quickstart ------------------------- @@ -710,4 +716,4 @@ under the covers and so developers who are interested in how LNT works under the hood may want to understand the Makefile based setup. For more information on the ``test-suite`` Makefile setup, please see -the `Test Suite Makefile Guide. <TestSuiteMakefileGuide.html>`_ +the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. |