| Message ID | 1434570533.14663.126.camel@surprise |
|---|---|
| State | New |
| Headers | show |
On 06/17/2015 01:48 PM, David Malcolm wrote: > On Thu, 2015-06-11 at 00:18 +0200, Jakub Jelinek wrote: > > I wrote a custom formatter for the output (class deja_gnu_printer within > unittests-plugin.c) which generates lines like this on stderr: > > PASS: ggc_test.tree_marking > PASS: ggc_test.custom_struct > PASS: ggc_test.finalization > PASS: ggc_test.inheritance > PASS: ggc_test.chain_next > > These get detected on stderr by some new logic inside > testsuite/lib/prune.exp, which prefixes them with > [testname-for-summary], and emitting them at the Tcl level, so they end > up in the .log and .sum files as lines like this: > > PASS: c-c++-common/torture/run-unittests-plugin.c -O0 ggc_test.tree_marking > PASS: c-c++-common/torture/run-unittests-plugin.c -O0 ggc_test.custom_struct > PASS: c-c++-common/torture/run-unittests-plugin.c -O0 ggc_test.finalization > PASS: c-c++-common/torture/run-unittests-plugin.c -O0 ggc_test.inheritance > PASS: c-c++-common/torture/run-unittests-plugin.c -O0 ggc_test.chain_next > > (if doing it for all testcase output is an issue, perhaps this prune.exp > logic could be made conditional, so that we only do it for testcases > using the plugin, or that contain a dg command) Yea, this is roughly what I was expecting -- you don't have access to the .log/.sum files from within the test, so you have to scan the output from the test and emit the pass/fail/xfail/whatever messages. Presumably there's a mechanism for mark expected failures in the gtest framework that we can turn into XFAIL/XPASS as needed. What about the UNSUPPORTED result -- perhaps useful if a test depends on some attribute of the target. > This works, and means there's no extra external dependency, and I was > able to bootstrap with this. I ran into an issue with "make check": the > plugin is linked against the previous stage's libstdc++, but "make > check" doesn't seem to set up LD_LIBRARY_PATH to point at a fresh > libstdc++ and uses the system copy, leading to > cc1: error: cannot load plugin ../../unittests_plugin.so > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required > by ../../unittests_plugin.so) > > on this Fedora 20 box. Manually setting LD_LIBRARY_PATH to point at the > built libstdc++ fixes it. You might look at the libstdc++ testsuite. It has to arrange to get the just built library used for testing. Jeff
This is a followup to these proposals: * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html The following patch kit adds a unit tests framework for gcc, as a new subdirectory below gcc/testsuite. The aim is to give us direct coverage of low level implementation details that aren't well covered by our existing tests, such as our data structures, gengtype, etc. This will let us have tests that directly poke at specific internal APIs and verify that the right thing happens, rather than having to write source code and option combinations that we hope touches the relevant state. It should make it trivial to add new unit tests. I see it as complementary to our existing test approaches. Like previous versions of the patch kit it uses the Google Test framework, since I've had good experiences with it: http://code.google.com/p/googletest/ and like v2 of the kit it embeds a two-file copy of v1.7 of the kit, to avoid adding extra dependencies (one .h file and one .c file). v1 of the kit was structured as a frontend, v2 of the kit as a plugin that was built as if it were a frontend. Both of these approaches were problematic, so this version of the patch kit simply builds as a test case within the plugin.exp suites. In this approach, each of gcc.dg and g++.dg have a plugin.exp that lists plugins to be built and source files to then compile with the plugin. This patch kit inserts into them references to the plugin within ../../unittests, so that we can run the tests within both cc1 and cc1plus without having to list everything twice. The plugin is built by plugin.exp and plugin-support.exp as per the existing test cases. Doing so requires #include-ing everything relevant within unittests-plugin.c. The plugin uses a custom gtest reporter that reports results in DejaGnu format. The kit adds some logic to testsuite/lib/prune.exp to parse the DejaGnu output on stderr e.g.: PASS: ggc_test.inheritance and re-emits it at the Tcl level, so that the unit test results from the plugin are reported within the regular test output like this: PASS: gcc.dg/plugin/unittests.c -fplugin=./unittests-plugin.so ggc_test.inheritance Successfully bootstrapped®rtested on x86_64-pc-linux-gnu (on top of r229364); adds 61 new PASS results to each of gcc.sum and g++.sum. I've broken up the patches into logical chunks in the hope of making review easier, but this is effectively one change, and would be committed as such. Though if any individual testcases are problematic, they could be deferred, if that will help get the rest of the work approved. [There are some FIXMEs in the testcases indicating room for further testing, and I've not fully reduced the #includes in them; is the "blessed" #include order documented yet, and do we have tooling in "contrib" for this yet?] With those caveats, OK for trunk? David Malcolm (16): Add unittest infrastructure Add embedded copy of gtest-1.7 to unittests Add test-bitmap.c to unittests Add test-cfg.c to unittests Add test-et-forest.c to unittests Add test-folding.c to unittests Add test-functions.c to unittests Add test-ggc.c to unittests Add test-gimple.c to unittests Add test-hash-map.c to unittests Add test-hash-set.c to unittests Add test-locations.c to unittests Add test-rtl.c to unittests Add test-tree.c to unittests Add test-vec.c to unittests Add test-wide-int.c to unittests gcc/testsuite/g++.dg/plugin/plugin.exp | 4 +- gcc/testsuite/g++.dg/plugin/unittests.C | 9 + gcc/testsuite/gcc.dg/plugin/plugin.exp | 1 + gcc/testsuite/gcc.dg/plugin/unittests.c | 9 + gcc/testsuite/lib/plugin-support.exp | 29 +- gcc/testsuite/lib/prune.exp | 43 + gcc/testsuite/unittests/gtest-all.c | 9592 +++++++++++++ gcc/testsuite/unittests/gtest/gtest.h | 20061 +++++++++++++++++++++++++++ gcc/testsuite/unittests/test-bitmap.c | 116 + gcc/testsuite/unittests/test-cfg.c | 319 + gcc/testsuite/unittests/test-et-forest.c | 121 + gcc/testsuite/unittests/test-folding.c | 120 + gcc/testsuite/unittests/test-functions.c | 645 + gcc/testsuite/unittests/test-ggc.c | 299 + gcc/testsuite/unittests/test-gimple.c | 178 + gcc/testsuite/unittests/test-hash-map.c | 77 + gcc/testsuite/unittests/test-hash-set.c | 53 + gcc/testsuite/unittests/test-locations.c | 145 + gcc/testsuite/unittests/test-rtl.c | 94 + gcc/testsuite/unittests/test-tree.c | 101 + gcc/testsuite/unittests/test-vec.c | 161 + gcc/testsuite/unittests/test-wide-int.c | 185 + gcc/testsuite/unittests/unittests-plugin.c | 213 + 23 files changed, 32573 insertions(+), 2 deletions(-) create mode 100644 gcc/testsuite/g++.dg/plugin/unittests.C create mode 100644 gcc/testsuite/gcc.dg/plugin/unittests.c create mode 100644 gcc/testsuite/unittests/gtest-all.c create mode 100644 gcc/testsuite/unittests/gtest/gtest.h create mode 100644 gcc/testsuite/unittests/test-bitmap.c create mode 100644 gcc/testsuite/unittests/test-cfg.c create mode 100644 gcc/testsuite/unittests/test-et-forest.c create mode 100644 gcc/testsuite/unittests/test-folding.c create mode 100644 gcc/testsuite/unittests/test-functions.c create mode 100644 gcc/testsuite/unittests/test-ggc.c create mode 100644 gcc/testsuite/unittests/test-gimple.c create mode 100644 gcc/testsuite/unittests/test-hash-map.c create mode 100644 gcc/testsuite/unittests/test-hash-set.c create mode 100644 gcc/testsuite/unittests/test-locations.c create mode 100644 gcc/testsuite/unittests/test-rtl.c create mode 100644 gcc/testsuite/unittests/test-tree.c create mode 100644 gcc/testsuite/unittests/test-vec.c create mode 100644 gcc/testsuite/unittests/test-wide-int.c create mode 100644 gcc/testsuite/unittests/unittests-plugin.c
On Tue, 2015-10-27 at 15:48 -0400, David Malcolm wrote: [...snip...] Looks like [Patch 02/16] was too big (1.2MB) for the list; it can be seen here: https://dmalcolm.fedorapeople.org/gcc/2015-10-27/unittests/0002-Add-embedded-copy-of-gtest-1.7-to-unittests.patch [..snip...]
On 10/27/2015 08:48 PM, David Malcolm wrote: > The following patch kit adds a unit tests framework for gcc, > as a new subdirectory below gcc/testsuite. So, as a general comment I think this would be a very good thing to have, and from a quick look through the tests they look pretty sensible. > Like previous versions of the patch kit it uses the Google Test > framework, since I've had good experiences with it: > http://code.google.com/p/googletest/ > and like v2 of the kit it embeds a two-file copy of v1.7 of > the kit, to avoid adding extra dependencies (one .h file and one > .c file). How much of this are you actually using? How much effort would be involved in removing the extra prerequisite? Is it just the EXPECT_TRUE etc. macros? > v1 of the kit was structured as a frontend, v2 of the kit as > a plugin that was built as if it were a frontend. Both of > these approaches were problematic, so this version > of the patch kit simply builds as a test case within the > plugin.exp suites. This is the part I'm least certain about, for two reasons: * They are more like source files than tests, and I think I'd prefer to have them alongside the source, in gcc/ rather than in the testsuite. This way people are invariable going to fail to notice them when they grep for something. * This uses a plugin into whatever compiler was built, but sometimes you can't really set up unit tests that way because what you want to test depends on target specifics. What I've often wanted is a special test target that gets built with a special machine description that has whatever patterns are needed to replicate tricky situations in reload or other optimization passes. The tests you have so far are focused mostly on high-level gimple/tree tests where this limitation is probably not showing up very much, but I think it would be better to have something that allows us to have more in-depth tests. Bernd
On 10/28/2015 12:38 PM, Bernd Schmidt wrote: > * This uses a plugin into whatever compiler was built, but sometimes > you can't really set up unit tests that way because what you want > to test depends on target specifics. What I've often wanted is a > special test target that gets built with a special machine > description that has whatever patterns are needed to replicate > tricky situations in reload or other optimization passes. Continuing to think out loud, this could be built in a separate top-level directory alongside the normal build. We've briefly had machinery to do something like that for OpenACC accel-gcc builds; that was removed but it shows that the concept is viable. We should ensure that our framework for this is powerful enough to validate not just basic data structures like wide-int, but also questions like "I just set up an rtl insn in a specific way, does it produce the right set of reloads?" Bernd
On 10/27/2015 01:48 PM, David Malcolm wrote: > This is a followup to these proposals: > * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html > * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html > > The following patch kit adds a unit tests framework for gcc, > as a new subdirectory below gcc/testsuite. > > The aim is to give us direct coverage of low level implementation > details that aren't well covered by our existing tests, such as > our data structures, gengtype, etc. This will let us have tests > that directly poke at specific internal APIs and verify that the > right thing happens, rather than having to write source code and > option combinations that we hope touches the relevant state. > It should make it trivial to add new unit tests. > > I see it as complementary to our existing test approaches. Right. And I feel I ought to chime in since I've been the source of confusion WRT unit testing. I have often mis-used the term "unit testing" to mean feed state/code into a pass and test what comes out. But that's not actually unit testing and not the focus of this patchkit from David. As David states, the idea here is to get direct testing coverage of low level internals. I've been pondering this problem a fair amount lately as I look at our codebase and ponder how to improve its maintainability over time. This kind of low level testing has value. I don't doubt that in the slightest anymore. I fear the real problem in this space is getting enough scaffolding around those low level things we want to test so that we can write tests around them. I wouldn't at all be surprised if we go through multiple iterations on the overall design & implementation. Hell, you're already on your 3rd :-) > > Like previous versions of the patch kit it uses the Google Test > framework, since I've had good experiences with it: > http://code.google.com/p/googletest/ > and like v2 of the kit it embeds a two-file copy of v1.7 of > the kit, to avoid adding extra dependencies (one .h file and one > .c file). > > v1 of the kit was structured as a frontend, v2 of the kit as > a plugin that was built as if it were a frontend. Both of > these approaches were problematic, so this version > of the patch kit simply builds as a test case within the > plugin.exp suites. Using a plugin approach allows us to avoid at least some of the problems in getting a test that can be built because ultimately the bits get sucked into the address space of the compiler as a DSO. I suspect that's the only sensible model in the short/medium term. I just don't see us getting to a point where (for example) we could instantiate some internal class as an independent test totally independent of the rest of the compiler without a ton more work. I will note that large blobs of what Andrew has been doing would be a prerequisite for getting to that point. Anyway, it's a bit of rambling, but in the end, I think the summary is the plugin approach seems right for where we are today and expect to be for a while. > > In this approach, each of gcc.dg and g++.dg have a plugin.exp that > lists plugins to be built and source files to then compile with > the plugin. This patch kit inserts into them references to the plugin > within ../../unittests, so that we can run the tests within both cc1 > and cc1plus without having to list everything twice. The plugin > is built by plugin.exp and plugin-support.exp as per the existing > test cases. Doing so requires #include-ing everything relevant > within unittests-plugin.c. So as I ponder how we'll use this in practice, I think we'll probably need a way to control which tests run with a fair amount of granularity. So for example, if I'm doing something like cleaning up the SSA_NAME manager, I'd really like to be able to unit test that and do so quickly & repeatedly during development before moving on to a larger test like bootstrap & regression test. Similarly if I had this framework in place earlier, I would have wanted run tests around just the scoped hashtable interface and implementation. Conceptually this might work in the same manner we've done with c-torture & the dg frameworks. ie RUNTESTFLAGS=unittest.exp=scopedtables If you've already got that capability in place, then well, that's awesome. > > The plugin uses a custom gtest reporter that reports results in > DejaGnu format. The kit adds some logic to testsuite/lib/prune.exp > to parse the DejaGnu output on stderr e.g.: > PASS: ggc_test.inheritance > and re-emits it at the Tcl level, so that the unit test results from > the plugin are reported within the regular test output like this: > PASS: gcc.dg/plugin/unittests.c -fplugin=./unittests-plugin.so ggc_test.inheritance I hate to note it, but isn't the pruning code fairly performance sensitive? Any way to do this outside of prune.exp that doesn't effect running the rest of the testsuite? Or show that it just "doesn't matter" having these extra bits in the pruning code. > Successfully bootstrapped®rtested on x86_64-pc-linux-gnu > (on top of r229364); adds 61 new PASS results to each of gcc.sum and > g++.sum. > > I've broken up the patches into logical chunks in the hope of making > review easier, but this is effectively one change, and would be > committed as such. Though if any individual testcases are > problematic, they could be deferred, if that will help get the rest > of the work approved. You know me. I always try to carve off the easiest parts of the review first, then iterate on what's left. It's usually the case that each iteration makes something in the next iteration easier to understand. So you should expect reviews to arrive in a semi-random order. > > [There are some FIXMEs in the testcases indicating room for further > testing, and I've not fully reduced the #includes in them; is the > "blessed" #include order documented yet, and do we have tooling > in "contrib" for this yet?] The tooling isn't finished, but the last patch from Andrew has usable versions. I've used show-headers extensively, but order-headers and reduce/replace seem to work when I've done light poking. I'd certainly recommend using them for the reduction. Jeff
On Oct 27, 2015, at 12:48 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> The following patch kit adds a unit tests framework for gcc,
I’m supportive of the patch set and the concept in general. I don’t know who wants to review this, Jeff seems on track to do this. My only guidance would be to say we should allow more latitude and leeway here and let people experiment in tree with what works and what doesn’t and evolve things as fast as they want. As the patches go in, I think they should have to latitude to evolve things as they see fit. If they would like to be listed as maintainer of unit test, I’d support that.
On Wed, 2015-10-28 at 12:38 +0100, Bernd Schmidt wrote: > On 10/27/2015 08:48 PM, David Malcolm wrote: > > The following patch kit adds a unit tests framework for gcc, > > as a new subdirectory below gcc/testsuite. > > So, as a general comment I think this would be a very good thing to > have, and from a quick look through the tests they look pretty sensible. Thanks. > > Like previous versions of the patch kit it uses the Google Test > > framework, since I've had good experiences with it: > > http://code.google.com/p/googletest/ > > and like v2 of the kit it embeds a two-file copy of v1.7 of > > the kit, to avoid adding extra dependencies (one .h file and one > > .c file). > > How much of this are you actually using? How much effort would be > involved in removing the extra prerequisite? Is it just the EXPECT_TRUE > etc. macros? As well as the EXPECT_FOO macros, there's test registration, sometimes done via TEST, sometimes via TEST_F to support fixtures. I used TYPED_TEST for type-parameterizing the wide-int tests (though, to be fair, I didn't manage to get this properly working; see the comment in test-wide-int.c). There may be useful things in gtest for us that I'm not using yet. For example, the support for death tests may be useful for testing that e.g. our checking macros kill the program sanely in the presence of malformed internal data. I wanted to reuse the gtest code on the grounds that: * the code exists * it's maintained (https://github.com/google/googletest shows the last commit was 10 days ago) * it has documentation: https://github.com/google/googletest/blob/master/googletest/docs/Primer.md (there's much more that just that) * it will be familiar to some people from other projects That said, if it's a blocker, I can attempt a minimal reimplementation of the subset of the API that I'm using. One wart in the v3 patchkit was that I used the default configuration for gtest. For example, this implicitly required threading support. This could be suppressed by defining: #define GTEST_HAS_PTHREAD 0 before including gtest.h. > > v1 of the kit was structured as a frontend, v2 of the kit as > > a plugin that was built as if it were a frontend. Both of > > these approaches were problematic, so this version > > of the patch kit simply builds as a test case within the > > plugin.exp suites. > > This is the part I'm least certain about, for two reasons: > * They are more like source files than tests, and I think I'd prefer > to have them alongside the source, in gcc/ rather than in the > testsuite. This way people are invariable going to fail to notice > them when they grep for something. So something like: gcc/test-vec.c or even: gcc/vec-tests.c so that it sorts next to gcc/vec.h in a listing? i.e. gcc/FOO-tests.c for gcc/FOO.c Maybe the unittests-plugin.c should live within the "gcc" directory? > * This uses a plugin into whatever compiler was built, but sometimes > you can't really set up unit tests that way because what you want > to test depends on target specifics. What I've often wanted is a > special test target that gets built with a special machine > description that has whatever patterns are needed to replicate > tricky situations in reload or other optimization passes. > > The tests you have so far are focused mostly on high-level gimple/tree > tests where this limitation is probably not showing up very much, but I > think it would be better to have something that allows us to have more > in-depth tests. I think our long-term goal should be at least 5 styles of test: (A) end-to-end tests of the compiler: running it on some source and verifying properties of the diagnostics and/or generated code (potentially running it). This is what we have today. (B) unit tests of individual subsystems: tests that exercise specific internal APIs e.g. containers like vec<>, hash_map<> etc, also things like gengtype. This is a gap in our current test coverage, and what this patch kit aims to start filling. (C) and (D): tests of specific passes: tests that accept IR (either gimple or RTL), and run just one pass, and verify properties of the IR emitted by the pass, or messages emitted by the pass. LLVM has a "-R" remark option, so that a pass can issue remarks about what it's doing (a new kind of diagnostic): http://llvm.org/releases/3.5.0/tools/clang/docs/UsersManual.html#opt-rpass We could implement this kind of testing by implementing gimple and RTL frontends, and a -R option, which could integrate nicely with DejaGnu, with /* { dg-remark "inlined here" } */ and the like. This would require finishing the gimple FE, and writing an RTL FE (independent goals). Hopefully that would give us test coverage for dealing with e.g. tricky reload situations. (E) performance testing The purpose of this patch kit is (B). I'm interested in tackling (C) and (D), but I think those are going to have to wait until next stage 1 at this point. Dave
On 10/28/2015 05:38 AM, Bernd Schmidt wrote: > On 10/27/2015 08:48 PM, David Malcolm wrote: >> The following patch kit adds a unit tests framework for gcc, >> as a new subdirectory below gcc/testsuite. > > So, as a general comment I think this would be a very good thing to > have, and from a quick look through the tests they look pretty sensible. > >> Like previous versions of the patch kit it uses the Google Test >> framework, since I've had good experiences with it: >> http://code.google.com/p/googletest/ >> and like v2 of the kit it embeds a two-file copy of v1.7 of >> the kit, to avoid adding extra dependencies (one .h file and one >> .c file). > > How much of this are you actually using? How much effort would be > involved in removing the extra prerequisite? Is it just the EXPECT_TRUE > etc. macros? > >> v1 of the kit was structured as a frontend, v2 of the kit as >> a plugin that was built as if it were a frontend. Both of >> these approaches were problematic, so this version >> of the patch kit simply builds as a test case within the >> plugin.exp suites. > > This is the part I'm least certain about, for two reasons: > * They are more like source files than tests, and I think I'd prefer > to have them alongside the source, in gcc/ rather than in the > testsuite. This way people are invariable going to fail to notice > them when they grep for something. Excellent point. I think this is worth some serious thought. Given the state of GCC's sources, tests of this nature are going to be inherently tied to implementation details/sources rather than interfaces. That's obviously not ideal, but it is where we are. Combined with the cleanups/refactoring I think we ought to be doing, we've got a fairly strong argument to set these along side the sources. The counter is that when grepping, you should probably be using find/xargs grep :-) > * This uses a plugin into whatever compiler was built, but sometimes > you can't really set up unit tests that way because what you want > to test depends on target specifics. What I've often wanted is a > special test target that gets built with a special machine > description that has whatever patterns are needed to replicate > tricky situations in reload or other optimization passes. > > The tests you have so far are focused mostly on high-level gimple/tree > tests where this limitation is probably not showing up very much, but I > think it would be better to have something that allows us to have more > in-depth tests. Yes. But I think this level of testing is on another point in the testing continuum and will probably require some significant work beyond the unit testing being proposed by David. jeff
On 10/29/2015 10:10 AM, David Malcolm wrote: > > There may be useful things in gtest for us that I'm not using yet. For > example, the support for death tests may be useful for testing that e.g. > our checking macros kill the program sanely in the presence of malformed > internal data. Please check on this. I want to build a set of tests around the SSA_NAME manager, and that would include setting up can't happen situations and seeing if they're properly diagnosed by the checking code. So obviously I have an interest in how the framework handles this kind of scenario. > > I wanted to reuse the gtest code on the grounds that: > > * the code exists > > * it's maintained (https://github.com/google/googletest > shows the last commit was 10 days ago) > > * it has documentation: > https://github.com/google/googletest/blob/master/googletest/docs/Primer.md > (there's much more that just that) > > * it will be familiar to some people from other projects > > > That said, if it's a blocker, I can attempt a minimal reimplementation > of the subset of the API that I'm using. I'd probably lean towards continuing to use gtest. I don't currently see a compelling reason to build another framework. > >>> v1 of the kit was structured as a frontend, v2 of the kit as >>> a plugin that was built as if it were a frontend. Both of >>> these approaches were problematic, so this version >>> of the patch kit simply builds as a test case within the >>> plugin.exp suites. >> >> This is the part I'm least certain about, for two reasons: >> * They are more like source files than tests, and I think I'd prefer >> to have them alongside the source, in gcc/ rather than in the >> testsuite. This way people are invariable going to fail to notice >> them when they grep for something. > > So something like: > > gcc/test-vec.c > > or even: > > gcc/vec-tests.c > > so that it sorts next to gcc/vec.h in a listing? > > i.e. gcc/FOO-tests.c for gcc/FOO.c > > Maybe the unittests-plugin.c should live within the "gcc" directory? It's a bit of bike-shedding :-) I'd be happy with gcc/test-whatever or gcc/whatever-test. Both have pluses/minuses and in the end I doubt it'd matter much. My only worry here is doubling the number of files in gcc/. But I'd claim that we want to have more subdirectories for major logical areas. gimple/ rtl/ whatever, which would significantly mitigate the problem of too many files in gcc/ > >> * This uses a plugin into whatever compiler was built, but sometimes >> you can't really set up unit tests that way because what you want >> to test depends on target specifics. What I've often wanted is a >> special test target that gets built with a special machine >> description that has whatever patterns are needed to replicate >> tricky situations in reload or other optimization passes. >> >> The tests you have so far are focused mostly on high-level gimple/tree >> tests where this limitation is probably not showing up very much, but I >> think it would be better to have something that allows us to have more >> in-depth tests. > > I think our long-term goal should be at least 5 styles of test: > > (A) end-to-end tests of the compiler: running it on some source and > verifying properties of the diagnostics and/or generated code > (potentially running it). This is what we have today. > > (B) unit tests of individual subsystems: tests that exercise specific > internal APIs e.g. containers like vec<>, hash_map<> etc, also things > like gengtype. This is a gap in our current test coverage, and what > this patch kit aims to start filling. Right. It's essentially data structure and other very low level testing (gengtype). That's even true if I think about the SSA_NAME manager -- in an ideal world, it'd just be a container for objects that we want to release & recycle, whether they're SSA_NAMEs, pseudos, whatever we decide has the right properties to benefit from recycling. The fact that it's recycling SSA_NAMEs isn't for the most part all that important. > > (C) and (D): tests of specific passes: tests that accept IR (either > gimple or RTL), and run just one pass, and verify properties of the IR > emitted by the pass, or messages emitted by the pass. LLVM has a "-R" > remark option, so that a pass can issue remarks about what it's doing (a > new kind of diagnostic): > > http://llvm.org/releases/3.5.0/tools/clang/docs/UsersManual.html#opt-rpass > > We could implement this kind of testing by implementing gimple and RTL > frontends, and a -R option, which could integrate nicely with DejaGnu, > with > /* { dg-remark "inlined here" } */ > and the like. > > This would require finishing the gimple FE, and writing an RTL FE > (independent goals). Hopefully that would give us test coverage for > dealing with e.g. tricky reload situations. Insert note from Richi (was it?) where he wants the gimple FE made real and integrated. I think you'll have technical support if you want to tackle that next year. Something similar for RTL would be cool, but probably even harder given the amount of state that's traditionally been kept out of the IL stream. I'm sure some things are better today than in the past, but it's probably a very tanged mess to get to where we want to go with RTL. jeff
On Oct 29, 2015, at 12:32 PM, Jeff Law <law@redhat.com> wrote:
> Something similar for RTL would be cool, but probably even harder given the amount of state that's traditionally been kept out of the IL stream. I'm sure some things are better today than in the past, but it's probably a very tanged mess to get to where we want to go with RTL.
I had a bit of an rtl front end once. It was never complete, and likely was the 20% effort that did 20% of the actual job that needs doing even for a simple hello world program. Ah, too bad you didn’t ask around 10-15 years ago. I likely could have just pull that code out of a bag, as it is, likely lost in time.
On 10/29/2015 04:22 PM, Mike Stump wrote: > On Oct 29, 2015, at 12:32 PM, Jeff Law <law@redhat.com> wrote: >> Something similar for RTL would be cool, but probably even harder >> given the amount of state that's traditionally been kept out of the >> IL stream. I'm sure some things are better today than in the past, >> but it's probably a very tanged mess to get to where we want to go >> with RTL. > > I had a bit of an rtl front end once. It was never complete, and > likely was the 20% effort that did 20% of the actual job that needs > doing even for a simple hello world program. Ah, too bad you didn’t > ask around 10-15 years ago. I likely could have just pull that code > out of a bag, as it is, likely lost in time. Back then an official RTL front-end would have been strictly verboten. jeff
On 10/27/2015 01:58 PM, David Malcolm wrote: > On Tue, 2015-10-27 at 15:48 -0400, David Malcolm wrote: > [...snip...] > > Looks like [Patch 02/16] was too big (1.2MB) for the list; > it can be seen here: > https://dmalcolm.fedorapeople.org/gcc/2015-10-27/unittests/0002-Add-embedded-copy-of-gtest-1.7-to-unittests.patch DO we really need to embed a copy of gtest? Can we just make it a prereq? Jeff
On 10/29/2015 08:21 PM, Jeff Law wrote: > Excellent point. I think this is worth some serious thought. Given the > state of GCC's sources, tests of this nature are going to be inherently > tied to implementation details/sources rather than interfaces. That's > obviously not ideal, but it is where we are. Combined with the > cleanups/refactoring I think we ought to be doing, we've got a fairly > strong argument to set these along side the sources. > > The counter is that when grepping, you should probably be using > find/xargs grep :-) There's actually a tool called ack which automates that. But we've often seen cases where people fail to spot occurrences in config/ directories. I think tests for things like bitmap or wide-int could well live at the end of the respective source files - this would be the most convenient location so that if you add a new bitmap function, you can immediately add tests as well. Do we even support plugins on every host? >> The tests you have so far are focused mostly on high-level gimple/tree >> tests where this limitation is probably not showing up very much, but I >> think it would be better to have something that allows us to have more >> in-depth tests. > Yes. But I think this level of testing is on another point in the > testing continuum and will probably require some significant work beyond > the unit testing being proposed by David. Not sure about significant - it shouldn't take long to set up an extra test-gcc build at the top-level if we decide to go with that and add a -ftest option. I think it's worth spending some time thinking long-term about what the best way to go about this would be. Bernd
On 10/30/2015 04:54 AM, Bernd Schmidt wrote: >> The counter is that when grepping, you should probably be using >> find/xargs grep :-) > > There's actually a tool called ack which automates that. But we've often > seen cases where people fail to spot occurrences in config/ directories. We certainly have. Never saw "ack" before, I wonder if I can retrain myself to use it over find/grep :-) And it does seem damn fast. > I think tests for things like bitmap or wide-int could well live at the > end of the respective source files - this would be the most convenient > location so that if you add a new bitmap function, you can immediately > add tests as well. It has benefits as you noted above. What I don't like is the ifdefs we'll need to ensure the testing bits don't bloat cc1 and friends. I can live with any of the 3 proposals (in testsuite directory, alongside sources, inside existing sources) with the explicit assumption that if we try one and it does't work out that we might experiment with another. In my mind we're very early in this process, so course corrections may be necessary. > > Do we even support plugins on every host? I would think not :-) > >>> The tests you have so far are focused mostly on high-level gimple/tree >>> tests where this limitation is probably not showing up very much, but I >>> think it would be better to have something that allows us to have more >>> in-depth tests. >> Yes. But I think this level of testing is on another point in the >> testing continuum and will probably require some significant work beyond >> the unit testing being proposed by David. > > Not sure about significant - it shouldn't take long to set up an extra > test-gcc build at the top-level if we decide to go with that and add a > -ftest option. I think it's worth spending some time thinking long-term > about what the best way to go about this would be. If we look at the cfg tests you'll see one that we take down to generation of RTL. But the pain in setting that up is significant IMHO. Enough that I doubt many folks are willing to go to those lengths to write RTL (or even most gimple) tests. The first high level question is how do we envision writing those tests. I don't think the style shown in the gimple test is sustainable long term. We've got to get past hand creating cfgs, cfun, basic rtl and tree nodes, etc etc. Which argues that true gimple & rtl testing ought to be sucking in gimple/rtl to start (ie, gimple & rtl front-ends). So until those are in place, I just don't see going very far with testing at a level much higher than David's doing right now. The other way would be to look at the pain of setting up all the required state and say it's a failing in the APIs of GCC itself and that if we had the right APIs in place, constructing enough state to do this kind of testing would be relatively easy. David has mentioned working on the gimple front-end as his next significant project. I suspect we'll learn a ton from that work and wiring it into a test harness -- lessons I'd like to have under our belt prior to tacking the RTL testing side. Similarly I suspect we'll learn a lot about weaknesses in our API interfaces trying to write tests at a level just above what David's targeting right now. Jeff
So Jeff and I just had a chat, and we came up with some thoughts about how to proceed. I think we both agree that it would be good to have a special testing backend, along with frontends designed to be able to read in gimple or rtl that can be operated on. That's more of a long-term thing. For some of the simpler infrastructure tests such as the ones in this patch kit (bitmap, vec or wide-int functionality testing and such), we had the idea of putting these into every ENABLE_CHECKING compiler, and run them after building stage1, controlled by a -fself-test flag. It's better to detect such basic failures early rather than complete a full bootstrap and test cycle. It also keeps the tests alongside the rest of the implementation, which I consider desirable for such relatively simple data structures. Thoughts? Bernd
On Mon, 2015-11-16 at 19:17 +0100, Bernd Schmidt wrote: > So Jeff and I just had a chat, and we came up with some thoughts about > how to proceed. I think we both agree that it would be good to have a > special testing backend, along with frontends designed to be able to > read in gimple or rtl that can be operated on. That's more of a > long-term thing. (nods) FWIW, I'm interesting in trying to get the gimple FE into gcc 7. > For some of the simpler infrastructure tests such as the ones in this > patch kit (bitmap, vec or wide-int functionality testing and such), we > had the idea of putting these into every ENABLE_CHECKING compiler, and > run them after building stage1, controlled by a -fself-test flag. It's > better to detect such basic failures early rather than complete a full > bootstrap and test cycle. It also keeps the tests alongside the rest of > the implementation, which I consider desirable for such relatively > simple data structures. Would it be reasonable to run them at each stage? My hope is that they will be fast. If we're building the tests into the compiler itself, guarded by #if ENABLE_CHECKING then presumably it makes sense to put the tests directly into the pertinent source files? (rather than in a "foo-tests.c" file). Possibly even to interleave them, to be next to the code in question. Given that this patch kit has seen a fair amount of discussion, and parts of it are already approved, and that it's designed to improve our test coverage, is it reasonable to continue pursuing this within stage 3? (I hope so) Should I attempt a patch for the above? (once I've fixed the AIX bootstrap issue, of course) Any thoughts on embedded gtest vs external gtest vs building our own? If we're embedding, it may make most sense to build our own minimal implementation, with a similar API (to avoid relying on the C++ stdlib, which gtest does; that was the most awkward part of dealing with it, iirc); this would be simpler, I suspect. Dave
>> For some of the simpler infrastructure tests such as the ones in this >> patch kit (bitmap, vec or wide-int functionality testing and such), we >> had the idea of putting these into every ENABLE_CHECKING compiler, and >> run them after building stage1, controlled by a -fself-test flag. It's >> better to detect such basic failures early rather than complete a full >> bootstrap and test cycle. It also keeps the tests alongside the rest of >> the implementation, which I consider desirable for such relatively >> simple data structures. > > Would it be reasonable to run them at each stage? My hope is that they > will be fast. Depends on how fast, I guess. I don't think testing them more than once gains very much; if there's a suspicion that stage3 was miscompiled one could still run -fself-test manually. > If we're building the tests into the compiler itself, guarded by > #if ENABLE_CHECKING > then presumably it makes sense to put the tests directly into the > pertinent source files? (rather than in a "foo-tests.c" file). Possibly > even to interleave them, to be next to the code in question. Yes, I was thinking same source file for the most part. I don't think there has to be any kind of rule, we just do whatever makes sense. > Given that this patch kit has seen a fair amount of discussion, and > parts of it are already approved, and that it's designed to improve our > test coverage, is it reasonable to continue pursuing this within stage > 3? (I hope so) Should I attempt a patch for the above? (once I've > fixed the AIX bootstrap issue, of course) As far as I'm concerned this can still proceed given that it was submitted well in advance of stage 3 (unless someone objects). > Any thoughts on embedded gtest vs external gtest vs building our own? I think for -fself-test we can mostly operate with gcc_assert, IMO there's no need to use an elaborate framework. We can revisit this issue when we get to more extensive tests that require multiple compiler invocations. Bernd
On 11/16/2015 11:48 AM, David Malcolm wrote: >> For some of the simpler infrastructure tests such as the ones in this >> patch kit (bitmap, vec or wide-int functionality testing and such), we >> had the idea of putting these into every ENABLE_CHECKING compiler, and >> run them after building stage1, controlled by a -fself-test flag. It's >> better to detect such basic failures early rather than complete a full >> bootstrap and test cycle. It also keeps the tests alongside the rest of >> the implementation, which I consider desirable for such relatively >> simple data structures. > > Would it be reasonable to run them at each stage? My hope is that they > will be fast. But with what value? The point here is basic testing coverage of things like low level data structures and such. The odds that we break something in those that show up under unit testing during stage2/stage3, but not during stage1 or during the compiler itself would be small I would think. > > If we're building the tests into the compiler itself, guarded by > #if ENABLE_CHECKING > then presumably it makes sense to put the tests directly into the > pertinent source files? (rather than in a "foo-tests.c" file). Possibly > even to interleave them, to be next to the code in question. I'm of two minds. I really don't like the idea of lots of #ifs mucking up the sources. So I'd tend to want them either at the end of the file with a single #if CHECKING_P or as a separate foo-tests file. We may want to get at static objects/functions from within a test (to either set or query state), so that would tend to argue that within the existing .c files. > Given that this patch kit has seen a fair amount of discussion, and > parts of it are already approved, and that it's designed to improve our > test coverage, is it reasonable to continue pursuing this within stage > 3? (I hope so) Should I attempt a patch for the above? (once I've > fixed the AIX bootstrap issue, of course) I'm leaning towards suggesting we get on a branch and look to merge it into the next stage1. This isn't something that's going to have a user impact. > > Any thoughts on embedded gtest vs external gtest vs building our own? > If we're embedding, it may make most sense to build our own minimal > implementation, with a similar API (to avoid relying on the C++ stdlib, > which gtest does; that was the most awkward part of dealing with it, > iirc); this would be simpler, I suspect. I'd lean towards external gtest. I realize it's another dependency, but it's a testing only dependency, so it's not any worse in my mind than dejagnu. jeff
On Nov 16, 2015, at 3:12 PM, Jeff Law <law@redhat.com> wrote: > So I'd tend to want them either at the end of the file with a single #if CHECKING_P or as a separate foo-tests file. Hum… I kinda don’t want the main files mucked up with tests. I think I’d rather have #if CHECKING_P #include "test/expr-test.h" #endif at the end, and punt the whole lot into a single subdirectory that most people, most of the time, can simply ignore. Wading through a ton of code that you aren’t interested in, is, well, annoying. We so rarely change apis that I don’t see the harm in the separation. Since we default to testing, and since the test suite will horrifically fail (if it works) if people do it wrong, I don’t think people will forget to update the test suite as the apis change. > I'm leaning towards suggesting we get on a branch and look to merge it into the next stage1. This isn't something that's going to have a user impact. Depends on the patch set, but, I’d like to think most would be fairly safe to put in post stage 1.
On 11/17/2015 02:53 AM, Mike Stump wrote: > On Nov 16, 2015, at 3:12 PM, Jeff Law <law@redhat.com> wrote: >> So I'd tend to want them either at the end of the file with a >> single #if CHECKING_P or as a separate foo-tests file. > > Hum… I kinda don’t want the main files mucked up with tests. I > think I’d rather have > > #if CHECKING_P #include "test/expr-test.h" #endif > > at the end, and punt the whole lot into a single subdirectory that > most people, most of the time, can simply ignore. Wading through a > ton of code that you aren’t interested in, is, well, annoying. Most of the tests submitted so far are relatively tiny, sometimes the list of #includes in the testcase is longer than the tests themselves. If they are at the end of a file you'd hardly be wading through them either. Let's just use common sense and make separate files if we ever get huge amounts of test code and keep it simple otherwise. Bernd
On 11/17/2015 05:51 AM, Bernd Schmidt wrote: > On 11/17/2015 02:53 AM, Mike Stump wrote: >> On Nov 16, 2015, at 3:12 PM, Jeff Law <law@redhat.com> wrote: >>> So I'd tend to want them either at the end of the file with a >>> single #if CHECKING_P or as a separate foo-tests file. >> >> Hum… I kinda don’t want the main files mucked up with tests. I >> think I’d rather have >> >> #if CHECKING_P #include "test/expr-test.h" #endif >> >> at the end, and punt the whole lot into a single subdirectory that >> most people, most of the time, can simply ignore. Wading through a >> ton of code that you aren’t interested in, is, well, annoying. > > Most of the tests submitted so far are relatively tiny, sometimes the > list of #includes in the testcase is longer than the tests themselves. > If they are at the end of a file you'd hardly be wading through them > either. Let's just use common sense and make separate files if we ever > get huge amounts of test code and keep it simple otherwise. I'd been pondering a #include solution too. But at this stage I don't think it buys us anything significant beyond just putting them at the end of the source files. Obviously if we find that the tests are intrusive, then we can adjust. One could legitimately ask what about tests that hit multiple source files. The snarky response is that such tests aren't really suitable for unit testing :-) But for those we can create a separate source file, or put them into the most logical location. Jeff
On Mon, 2015-11-16 at 19:17 +0100, Bernd Schmidt wrote: > So Jeff and I just had a chat, and we came up with some thoughts about > how to proceed. I think we both agree that it would be good to have a > special testing backend, along with frontends designed to be able to > read in gimple or rtl that can be operated on. That's more of a > long-term thing. > > For some of the simpler infrastructure tests such as the ones in this > patch kit (bitmap, vec or wide-int functionality testing and such), we > had the idea of putting these into every ENABLE_CHECKING compiler, and > run them after building stage1, controlled by a -fself-test flag. It's > better to detect such basic failures early rather than complete a full > bootstrap and test cycle. It also keeps the tests alongside the rest of > the implementation, which I consider desirable for such relatively > simple data structures. > > Thoughts? I like the idea. Here's another iteration of the patch kit, which implements it (mostly). David Malcolm (15): Selftest framework (unittests v4) Add selftests to bitmap.c Add selftests to tree-cfg.c Add selftests to et-forest.c Add selftests to fold-const.c Add function-tests.c Fix warning in function-tests.c Add selftests to gimple.c Add hash-map-tests.c Add hash-set-tests.c Add selftests to input.c Add rtl-tests.c Add selftests to tree.c Add selftests to vec.c RFC: Add ggc-tests.c gcc/Makefile.in | 9 +- gcc/bitmap.c | 92 ++++++++ gcc/common.opt | 4 + gcc/et-forest.c | 99 ++++++++ gcc/fold-const.c | 66 ++++++ gcc/function-tests.c | 632 +++++++++++++++++++++++++++++++++++++++++++++++++++ gcc/ggc-tests.c | 302 ++++++++++++++++++++++++ gcc/gimple.c | 103 +++++++++ gcc/hash-map-tests.c | 81 +++++++ gcc/hash-set-tests.c | 57 +++++ gcc/input.c | 107 +++++++++ gcc/rtl-tests.c | 98 ++++++++ gcc/selftest.c | 152 +++++++++++++ gcc/selftest.h | 270 ++++++++++++++++++++++ gcc/toplev.c | 51 +++++ gcc/toplev.h | 2 + gcc/tree-cfg.c | 264 +++++++++++++++++++++ gcc/tree.c | 47 ++++ gcc/vec.c | 142 ++++++++++++ 19 files changed, 2575 insertions(+), 3 deletions(-) create mode 100644 gcc/function-tests.c create mode 100644 gcc/ggc-tests.c create mode 100644 gcc/hash-map-tests.c create mode 100644 gcc/hash-set-tests.c create mode 100644 gcc/rtl-tests.c create mode 100644 gcc/selftest.c create mode 100644 gcc/selftest.h
This is effectively v5 of the unittests proposal; for the earlier versions see: * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html * v3: https://gcc.gnu.org/ml/gcc-patches/2015-10/msg02947.html * v4: https://gcc.gnu.org/ml/gcc-patches/2015-11/msg02379.html Bernd said (in https://gcc.gnu.org/ml/gcc-patches/2015-11/msg01981.html ): > For some of the simpler infrastructure tests such as the ones in this > patch kit (bitmap, vec or wide-int functionality testing and such), > we had the idea of putting these into every ENABLE_CHECKING compiler, > and run them after building stage1, controlled by a -fself-test flag. > It's better to detect such basic failures early rather than complete > a full bootstrap and test cycle. It also keeps the tests alongside the > rest of the implementation, which I consider desirable for such > relatively simple data structures." So the main difference is this version of the patch kit is that the tests are run much earlier: rather than have a DejaGnu test below gcc.dg that compiles a dummy file with -fself-test, in this iteration, gcc/Makefile.in is updated so that the selftests are run during the build. The patch kit adds a "selftests" phony target to gcc/Makefile.in ensuring that the selftests must be successfully run for the build to complete. The time taken to run the testsuite is also printed. A "s-selftests" file is created, so that the selftests need only be run once each time "cc1" is rebuilt in a working copy. They are run at each stage in a bootstrap. For example, looking at a bootstrap log: $ grep "fself-test:" test/experiment/x86_64-pc-linux-gnu/build/make.log -fself-test: 522 pass(es); 0 failure(s) in 0.013000 seconds -fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds -fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds shows that the selftests are indeed run at each stage of a bootstrap, and that they're fast (well under a second). There's also a debug target, so that we can type: make selftests-gdb to run the selftests under gdb. I also patched things so that it aborts on the first failure, making failures easy to track down in the debugger, though requiring us to make the selftests robust. Assuming that other people like these changes, it has a few implications: * selftests ought to be fast. We don't want to slow down the build. * running -fself-test should be terse by default; we may want an option for it to be more verbose * poorly written selftests could break the build for other people (e.g. accidentally making target-specific assumptions). The patch kit adds some new tests: * some self-tests for diagnostic-show-locus.c. This shows how to unit-test an internal API. * conversion of the Levenshtein selftest from a plugin to using -fself-test (we could move various other tests from plugins to run earlier). There is a bug with GNU ld in which it doesn't run tests in source files that purely contain tests (hence I was #include-ing these from toplev.c). I tracked down the issue, which I've filed against GNU ld here: https://sourceware.org/bugzilla/show_bug.cgi?id=20152 and a minimal reproducer can be seen here: https://github.com/davidmalcolm/test-ctor The issue is that these don't work with GNU ld if they're linked into a .a file (libbackend.a in our case) and then linked from there into cc1. They work if they're linked directly into cc1 without going through an archive. We could link these files directly into cc1, or we could move to manual test registration instead. Or we could not have registration, and simply hardcode what tests run, and what order they run in. Testing: * successfully bootstrapped®ression tested on x86_64-pc-linux-gnu. * successfully built with 193 configurations using contrib/config-list.mk. The remaining configurations fail for various reasons: * 9 vxworks configurations fail to run the selftests with: xgcc: fatal error: environment variable ‘WIND_BASE’ not defined * 4 configurations fail due to pre-existing warnings: fails on 3 configurations (microblaze and rl78) due to warnings which I've posted fixes for), and on avr-rtemsOPT-enable-obsolete due to a different pair of warnings. * 2 failures: rs6000-ibm-aix4.3 and rs6000-ibm-aix5.1.0 due to a pre-existing bug on trunk (I've filed this as PR target/71375). * I haven't yet tested this with build != host. Patch 1 is the main part of the implementation; it's similar to v4. Patch 2 adds the new Makefile.in integration. Patch 3 reduces the verbosity of the tests by moving almost all logging to a new optional logfile (without providing a way to set it at this stage). Patch 4 shows a way of filtering the tests using a command-line regex. Patch 5 adds a new example test, showing how to unit-test an implementation detail of diagnostic-show-locus.c Patch 6 shows an example of porting one of our existing plugin-based tests to be within -fself-test instead. Patches 7-21 are the previously posted tests, albeit with some bug fixes. My recollection is that the major unresolved discussion areas in v4 of the patch kit were: (a) do we want to stop running at the first failure, or to run the testsuite to completion? (b) should tests be automatically registered, or should this be manual? (c) is this over-engineered; should we simply use gcc_assert etc? For (a), this version of the patch kit switches to stopping at the first failure. For (b), this version of the patch kit stays with auto-registration. For (c), this version stays with a "lite" version of GTest. Thoughts? Does the "running tests early" approach suggest answers to these? David Malcolm (21): Selftest framework (unittests v5) Makefile.in integration Various selftest::runner tweaks. Add -fself-test-regex= Add selftests for diagnostic-show-locus.c Convert Levenshtein test from a plugin to a selftest Add selftests to bitmap.c Add selftests to tree-cfg.c Add selftests to et-forest.c Add selftests to fold-const.c Add function-tests.c Fix warning in function-tests.c Fixup to function-tests.c Remove x86_64-isms in function-tests.c Add selftests to gimple.c Add hash-map-tests.c Add hash-set-tests.c Add selftests to input.c Add rtl-tests.c Add selftests to tree.c Add selftests to vec.c gcc/Makefile.in | 26 +- gcc/bitmap.c | 92 ++++ gcc/common.opt | 8 + gcc/diagnostic-show-locus.c | 113 ++++ gcc/et-forest.c | 99 ++++ gcc/fold-const.c | 66 +++ gcc/function-tests.c | 629 +++++++++++++++++++++++ gcc/gimple.c | 103 ++++ gcc/hash-map-tests.c | 81 +++ gcc/hash-set-tests.c | 57 ++ gcc/input.c | 110 ++++ gcc/rtl-tests.c | 98 ++++ gcc/selftest.c | 210 ++++++++ gcc/selftest.h | 267 ++++++++++ gcc/spellcheck.c | 48 ++ gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c | 9 - gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c | 64 --- gcc/testsuite/gcc.dg/plugin/plugin.exp | 1 - gcc/toplev.c | 55 ++ gcc/toplev.h | 2 + gcc/tree-cfg.c | 264 ++++++++++ gcc/tree.c | 47 ++ gcc/vec.c | 142 +++++ 23 files changed, 2512 insertions(+), 79 deletions(-) create mode 100644 gcc/function-tests.c create mode 100644 gcc/hash-map-tests.c create mode 100644 gcc/hash-set-tests.c create mode 100644 gcc/rtl-tests.c create mode 100644 gcc/selftest.c create mode 100644 gcc/selftest.h delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c
On 06/01/2016 03:19 PM, David Malcolm wrote: > This is effectively v5 of the unittests proposal; for the earlier > versions see: > * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html > * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html > * v3: https://gcc.gnu.org/ml/gcc-patches/2015-10/msg02947.html > * v4: https://gcc.gnu.org/ml/gcc-patches/2015-11/msg02379.html > > Bernd said (in https://gcc.gnu.org/ml/gcc-patches/2015-11/msg01981.html ): >> For some of the simpler infrastructure tests such as the ones in this >> patch kit (bitmap, vec or wide-int functionality testing and such), >> we had the idea of putting these into every ENABLE_CHECKING compiler, >> and run them after building stage1, controlled by a -fself-test flag. >> It's better to detect such basic failures early rather than complete >> a full bootstrap and test cycle. It also keeps the tests alongside the >> rest of the implementation, which I consider desirable for such >> relatively simple data structures." > > So the main difference is this version of the patch kit is that the tests > are run much earlier: rather than have a DejaGnu test below gcc.dg that > compiles a dummy file with -fself-test, in this iteration, gcc/Makefile.in > is updated so that the selftests are run during the build. > > [snip] I don't see any documentation here for the new command-line options. If these are not intended to be user-visible, I think you should set the "Undocumented" flag for them in the .opt file instead. -Sandra
On 06/01/2016 11:19 PM, David Malcolm wrote: > This is effectively v5 of the unittests proposal; for the earlier > versions see: In general: nice to see this moving forward. There are some issues with the patch kit, some patches seem to fix issues with earlier ones (#3 or #13). One patch adds a sorry, the next changes it to inform; patch #1 adds includes top toplev.c without adding the included files. All these issues should be resolved: any patch series should compile if it is applied only partially up to a point; no early patch should depend on later ones. Also, bugfixes and behaviour changes should be merged directly into the code so as to not hopelessly confuse the reviewers. Some of the cover letters, in particular #1 seem to contain outdated information. > I also patched things so that it aborts on the first failure, making > failures easy to track down in the debugger, though requiring us to > make the selftests robust. Is there any kind of doubt that this is a good requirement? > * conversion of the Levenshtein selftest from a plugin to using > -fself-test (we could move various other tests from plugins to run > earlier). That sounds good. > Patch 4 shows a way of filtering the tests using a command-line regex. What's the use case for this? > For (a), this version of the patch kit switches to stopping at the > first failure. > For (b), this version of the patch kit stays with auto-registration. > For (c), this version stays with a "lite" version of GTest. > > Thoughts? Does the "running tests early" approach suggest answers to > these? I think this is mostly a good resolution, although I have one particular issue that rubs me the wrong way where I'd go even "liter" on the GTest. In my view, tests are functions, and using object-orientation leads to oddly contorted code. An example can be found in patch #5: +class range_contains_point_tests : public ::selftest::test +{ + protected: + layout_range + make_range (int start_line, int start_col, + int end_line, int end_col) (note also that this defeats the purpose of the GNU function formatting which is to enable grep ^function_name to find them) + { + const expanded_location start_exploc + = {"test.c", start_line, start_col, NULL, false}; + expanded_location finish_exploc + = {"test.c", end_line, end_col, NULL, false}; + + return layout_range (&start_exploc, &finish_exploc, false, + &start_exploc); + } +}; I think I raised this before, but defining a class only to define functions inside them seems relatively pointless; if anything you want namespaces. This one doesn't even seem to contain any references to the selftest framework so it could immediately be converted to a function. Other such cases appear to use the EXPECT_EQ etc. macros, which call pass and fail methods, but these just delegate immediately to the runner's pass and fail. Which then raises the question whether runner needs to be an object if there's only ever going to be one - once again I think a functional rather than object-oriented style would be more suitable to the problem. The main (only?) reason to have tests declared as objects is for the auto-registration, beyond that it serves very little purpose. There are some inconsistent spellings of an end comment across the series: +} // anon namespace +} /* anon namespace. */ I wasn't sure we were doing these at all, but it turns out the former is the canonical one (with a single space). Bernd
On Wed, 2016-06-01 at 15:20 -0600, Sandra Loosemore wrote: > On 06/01/2016 03:19 PM, David Malcolm wrote: > > This is effectively v5 of the unittests proposal; for the earlier > > versions see: > > * v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html > > * v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html > > * v3: https://gcc.gnu.org/ml/gcc-patches/2015-10/msg02947.html > > * v4: https://gcc.gnu.org/ml/gcc-patches/2015-11/msg02379.html > > > > Bernd said (in > > https://gcc.gnu.org/ml/gcc-patches/2015-11/msg01981.html ): > > > For some of the simpler infrastructure tests such as the ones in > > > this > > > patch kit (bitmap, vec or wide-int functionality testing and > > > such), > > > we had the idea of putting these into every ENABLE_CHECKING > > > compiler, > > > and run them after building stage1, controlled by a -fself-test > > > flag. > > > It's better to detect such basic failures early rather than > > > complete > > > a full bootstrap and test cycle. It also keeps the tests > > > alongside the > > > rest of the implementation, which I consider desirable for such > > > relatively simple data structures." > > > > So the main difference is this version of the patch kit is that the > > tests > > are run much earlier: rather than have a DejaGnu test below gcc.dg > > that > > compiles a dummy file with -fself-test, in this iteration, > > gcc/Makefile.in > > is updated so that the selftests are run during the build. > > > > [snip] > > I don't see any documentation here for the new command-line options. > If > these are not intended to be user-visible, I think you should set the > "Undocumented" flag for them in the .opt file instead. Thanks; I'll do that in the next iteration of the patches.
On Thu, 2016-06-02 at 12:29 +0200, Bernd Schmidt wrote: > On 06/01/2016 11:19 PM, David Malcolm wrote: > > This is effectively v5 of the unittests proposal; for the earlier > > versions see: > > In general: nice to see this moving forward. Thanks. > There are some issues with the patch kit, some patches seem to fix > issues with earlier ones (#3 or #13). One patch adds a sorry, the > next > changes it to inform; patch #1 adds includes top toplev.c without > adding > the included files. All these issues should be resolved: any patch > series should compile if it is applied only partially up to a point; > no > early patch should depend on later ones. Also, bugfixes and behaviour > changes should be merged directly into the code so as to not > hopelessly > confuse the reviewers. > Some of the cover letters, in particular #1 seem to contain outdated > information. Sorry about that. I was hoping to emphasize the changes from the last version, but clearly it made things more confusing. > > I also patched things so that it aborts on the first failure, > > making > > failures easy to track down in the debugger, though requiring us to > > make the selftests robust. > > Is there any kind of doubt that this is a good requirement? I was uncertain about how acceptable the "run the tests from within gcc/Makefile.in" approach is, so I was hedging my bets somewhat in the kit. It seems like you do like the gcc/Makefile.in approach, so this implies each of: (1) that tests should be fast (which is why I left out the test-ggc tests in this iteration for now, as one of them is relatively slow) (2) that tests must be 100% reliable (3) it must be trivially easy to track down failures (since in this approach test failure means build failure); "make selftests-gdb" is the solution here Given (3), this implies that we should halt on the first failure (as in this version of the kit). My experience when writing new tests and experimenting with the gcc/Makefile.in approach was that "halt on first failure" was much easier to work with than "run everything and report". > > * conversion of the Levenshtein selftest from a plugin to using > > -fself-test (we could move various other tests from plugins to > > run > > earlier). > > That sounds good. > > > Patch 4 shows a way of filtering the tests using a command-line > > regex. > > What's the use case for this? Say there's a problem with many tests and that you suspect an issue in, say, bitmap.c. This would let you run e.g. ".*bitmap.*" to better isolate the issue. Maybe this suggests that we should go even simpler, and hard-code the order in which tests run, manually encoding the dependencies (e.g. test the fundamental types first, then test the things that build on top of them, etc). This would be losing auto-registration, but has the virtue of simplicity. Thought experiment: imagine trying to bring up gcc on a new host, and 3/4 of the self tests are failing; it turns out to be some sizeof() assumption about int vs long or somesuch in one of the fundamental data structures. How do we make it easy to isolate such a problem? > > For (a), this version of the patch kit switches to stopping at the > > first failure. > > For (b), this version of the patch kit stays with auto > > -registration. > > For (c), this version stays with a "lite" version of GTest. > > > > Thoughts? Does the "running tests early" approach suggest answers > > to > > these? > > I think this is mostly a good resolution, Given that we're going with "halt on first failure", that means that there's no longer a distinction between EXPECT_EQ and ASSERT_EQ. I'll eliminate the former in favor of the latter. > although I have one particular > issue that rubs me the wrong way where I'd go even "liter" on the > GTest. > In my view, tests are functions, and using object-orientation leads > to > oddly contorted code. An example can be found in patch #5: > > +class range_contains_point_tests : public ::selftest::test > +{ > + protected: > + layout_range > + make_range (int start_line, int start_col, > + int end_line, int end_col) > (note also that this defeats the purpose of the GNU function > formatting > which is to enable grep ^function_name to find them) > + { > + const expanded_location start_exploc > + = {"test.c", start_line, start_col, NULL, false}; > + expanded_location finish_exploc > + = {"test.c", end_line, end_col, NULL, false}; > + > + return layout_range (&start_exploc, &finish_exploc, false, > + &start_exploc); > + } > +}; > > I think I raised this before, but defining a class only to define > functions inside them seems relatively pointless; if anything you > want > namespaces. This one doesn't even seem to contain any references to > the > selftest framework so it could immediately be converted to a > function. > > Other such cases appear to use the EXPECT_EQ etc. macros, which call > pass and fail methods, but these just delegate immediately to the > runner's pass and fail. Which then raises the question whether runner > needs to be an object if there's only ever going to be one - once > again > I think a functional rather than object-oriented style would be more > suitable to the problem. The main (only?) reason to have tests > declared > as objects is for the auto-registration, beyond that it serves very > little purpose. I agree that there's plenty of scope for simplification here (I think the choice to go with halt-on-first-failure makes the runner class redundant). I'll have a go at simplifying things; I'll try to keep auto-registration for now, but I'm in two minds about it. > There are some inconsistent spellings of an end comment across the > series: > +} // anon namespace > +} /* anon namespace. */ > > I wasn't sure we were doing these at all, but it turns out the former > is > the canonical one (with a single space). Will fix. Thanks Dave
Given that we're now using an "abort on first failure" model, I renamed all of the EXPECT_ macros to ASSERT_ (for consistency with GTest). As per Bernd's suggestions I eliminated the runner class and moved to a more function-based rather than class-based approach. At this point, the only reasons left for "class test" were auto-registration, and giving names to tests. So I tried removing it, and invoking tests manually. I like the resulting code; it seems much simpler and clearer, with very little "magic". The following is an updated version of the patch kit that uses this simpler approach. In theory there's a slight risk to the manual-invocation approach. If you forget a test within a file, the compiler tells you: ../../src/gcc/vec.c:204:1: warning: ‘void test_quick_push()’ defined but not used [-Wunused-function] but if you forget to call the file from selftests.c, there's no warning. I believe that adding a new test file will be a rare event, so this kind of mistake will (I hope) be unlikely. I've verified that the pass count before/after the change matches up. By constrast, the auto-registration approach put us at the mercy of the implementation of C++ global constructors, and I ran into at least one surprise with that ( https://sourceware.org/bugzilla/show_bug.cgi?id=20152 ). I've added the wide-int tests back. These are parametrized by type, and it was fairly easy to do this manually using templates once I eliminated test registration. I also added some new tests to diagnostic-show-locus.c. Although I'm posted this as a patch kit, it would be applied in one commit: the initial patch makes reference to tests added in later patches. I split it up so that each test file is in its own patch, to make review easier (I hope). As in v5, the tests are run in gcc/Makefile.in at each stage of a bootstrap: $ grep "fself-test:" test/experiment/x86_64-pc-linux-gnu/build/make.log -fself-test: 621 pass(es) in 0.013000 seconds -fself-test: 621 pass(es) in 0.006000 seconds -fself-test: 621 pass(es) in 0.007000 seconds Successfully bootstrapped®ression tested on x86_64-pc-linux-gnu. A test against all configurations using contrib/config-list.mk is in progress. OK for trunk? David Malcolm (16): Core of selftest framework (v6) diagnostic-show-locus.c: add selftests spellcheck.c: convert Levenshtein test from a plugin to a selftest bitmap.c: add selftests tree-cfg.c: add selftests et-forest.c: add selftests fold-const.c: add selftests Add function-tests.c gimple.c: add selftests Add hash-map-tests.c Add hash-set-tests.c input.c: add selftests Add rtl-tests.c tree.c: add selftests vec.c: add selftests wide-int.cc: add selftests gcc/Makefile.in | 31 +- gcc/bitmap.c | 110 ++++ gcc/common.opt | 4 + gcc/diagnostic-show-locus.c | 156 ++++++ gcc/et-forest.c | 112 ++++ gcc/fold-const.c | 75 +++ gcc/function-tests.c | 639 +++++++++++++++++++++++ gcc/gimple.c | 119 +++++ gcc/hash-map-tests.c | 88 ++++ gcc/hash-set-tests.c | 64 +++ gcc/input.c | 112 ++++ gcc/rtl-tests.c | 108 ++++ gcc/selftest-run-tests.c | 77 +++ gcc/selftest.c | 49 ++ gcc/selftest.h | 153 ++++++ gcc/spellcheck.c | 45 ++ gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c | 9 - gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c | 64 --- gcc/testsuite/gcc.dg/plugin/plugin.exp | 1 - gcc/toplev.c | 26 + gcc/toplev.h | 2 + gcc/tree-cfg.c | 277 ++++++++++ gcc/tree.c | 60 +++ gcc/vec.c | 162 ++++++ gcc/wide-int.cc | 152 ++++++ 25 files changed, 2616 insertions(+), 79 deletions(-) create mode 100644 gcc/function-tests.c create mode 100644 gcc/hash-map-tests.c create mode 100644 gcc/hash-set-tests.c create mode 100644 gcc/rtl-tests.c create mode 100644 gcc/selftest-run-tests.c create mode 100644 gcc/selftest.c create mode 100644 gcc/selftest.h delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c
commit f843bf0ed5764a36b0c0188c6d367b3164dfd34c
Author: David Malcolm <dmalcolm@redhat.com>
Date: Wed Jun 17 15:27:53 2015 -0400
Rewrite unittests to be a plugin
diff --git a/gcc/testsuite/c-c++-common/torture/run-unittests-plugin.c b/gcc/testsuite/c-c++-common/torture/run-unittests-plugin.c
new file mode 100644
index 0000000..297d78b
--- /dev/null
+++ b/gcc/testsuite/c-c++-common/torture/run-unittests-plugin.c
@@ -0,0 +1,2 @@
+/* This source file leads to the unittests_plugin being loaded and run.
+ { dg-options "-fplugin=../../unittests_plugin.so" } */
diff --git a/gcc/testsuite/lib/prune.exp b/gcc/testsuite/lib/prune.exp
index 8e4c203..c5d24be 100644
--- a/gcc/testsuite/lib/prune.exp
+++ b/gcc/testsuite/lib/prune.exp
@@ -68,6 +68,48 @@ proc prune_gcc_output { text } {
# Ignore harmless warnings from Xcode 4.0.
regsub -all "(^|\n)\[^\n\]*ld: warning: could not create compact unwind for\[^\n\]*" $text "" text
+ # Process any DejaGnu output from the test, and convert into results at
+ # this level, prefixing with the testcase.
+ # For example, given the line
+ # PASS: vec_test.quick_push
+ # from the test, we want to call "pass", giving a line like this in
+ # our .log/.sum, depending on [testname-for-summary]:
+ # PASS: c-c++-common/torture/run-unittests-plugin.c -O0 vec_test.quick_push
+ verbose "Would handle: $text\n"
+ # TODO: generalize for other kinds of test result:
+ foreach status_pair { {"PASS" pass} {"FAIL" fail} } {
+ set kind [lindex $status_pair 0]
+ set handler [lindex $status_pair 1]
+ #verbose "kind: $kind handler: $handler"
+ set pattern "^$kind: (.*?)$"
+ #verbose "pattern: $pattern"
+ set matches [regexp -all -inline -lineanchor $pattern $text]
+ #verbose "matches: $matches"
+ foreach {matched_line detail} $matches {
+ #verbose "matched_line: $matched_line"
+ #verbose "detail: $detail"
+ set testname [testname-for-summary]
+
+ # Call the handler ("pass"/"fail" etc) to convert to a message
+ # at this level, prefixed with [testname-for-summary]:
+ $handler "$testname $detail"
+
+ # Prune $matched_line from $text (or, at least, the first
+ # instance of it).
+ # Is there a less clunky way to do this? (regsub would require
+ # escaping any regex special characters within $matched_line).
+ # Locate first instance of matched_line:
+ set idx [string first $matched_line $text]
+ # Get length, adding one for the trailing newline char:
+ set linelen [expr [string length $matched_line] + 1]
+ # Cut it out from the middle of $text:
+ set textlen [string length $text]
+ set before [string range $text 0 [expr $idx - 1] ]
+ set after_ [string range $text [expr $idx + $linelen] $textlen]
+ set text $before$after_
+ }
+ }
+
#send_user "After:$text\n"
return $text
diff --git a/gcc/unittests/Make-lang.in b/gcc/unittests/Make-lang.in
index 50d6e4f..26a0cb9 100644
--- a/gcc/unittests/Make-lang.in
+++ b/gcc/unittests/Make-lang.in
@@ -40,43 +40,33 @@
# into the unittests rule, but that needs a little bit of work
# to do the right thing within all.cross.
-LIBGCCUNITTESTS_LINKER_NAME = libgccunittests.so
-LIBGCCUNITTESTS_VERSION_NUM = 0
-LIBGCCUNITTESTS_MINOR_NUM = 0
-LIBGCCUNITTESTS_RELEASE_NUM = 1
-LIBGCCUNITTESTS_SONAME = $(LIBGCCUNITTESTS_LINKER_NAME).$(LIBGCCUNITTESTS_VERSION_NUM)
-LIBGCCUNITTESTS_FILENAME = \
- $(LIBGCCUNITTESTS_SONAME).$(LIBGCCUNITTESTS_MINOR_NUM).$(LIBGCCUNITTESTS_RELEASE_NUM)
+UNITTESTS_PLUGIN_SO = unittests_plugin.so
-LIBGCCUNITTESTS_LINKER_NAME_SYMLINK = $(LIBGCCUNITTESTS_LINKER_NAME)
-LIBGCCUNITTESTS_SONAME_SYMLINK = $(LIBGCCUNITTESTS_SONAME)
-
-TESTPROGRAMS = unittests.exe
-
-RUN_TESTPROGRAMS = run-unittests.exe
-
-unittests: $(LIBGCCUNITTESTS_FILENAME) \
- $(LIBGCCUNITTESTS_SYMLINK) \
- $(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK) \
+unittests: \
$(FULL_DRIVER_NAME) \
- $(TESTPROGRAMS) \
- $(RUN_TESTPROGRAMS)
+ $(UNITTESTS_PLUGIN_SO)
# Tell GNU make to ignore these if they exist.
-.PHONY: unittests $(RUN_TESTPROGRAMS)
+.PHONY: unittests
-# "Frontend" objects, which go into libgccunittests.so
-unittests_OBJS = attribs.o \
- unittests/unittests-frontend.o
+# gengtype integration
+gt-unittests-test-ggc.h: ./gengtype $(srcdir)/unittests/test-ggc.c
+ ./gengtype \
+ --plugin $@ \
+ --read-state gtype.state \
+ $(srcdir)/unittests/test-ggc.c
# Build various files against gtest
# Files that are linked into unittests.exe
-UNITTESTS_EXE_OBJS = \
+
+UNITTESTS_PLUGIN_SO_OBJS = \
+ gtest-all.o \
test-bitmap.o \
test-cfg.o \
test-folding.o \
test-functions.o \
+ test-ggc.o \
test-gimple.o \
test-hash-map.o \
test-hash-set.o \
@@ -85,11 +75,11 @@ UNITTESTS_EXE_OBJS = \
test-tree.o \
test-vec.o \
test-wide-int.o \
- unittests-main.o
+ unittests-plugin.o
# Files to be built against gtest
-# test-ggc.o has to be linked in to libgccunittests.so
-OBJS_BUILT_AGAINST_GTEST = $(UNITTESTS_EXE_OBJS) test-ggc.o
+# FIXME: do we still need this?
+OBJS_BUILT_AGAINST_GTEST = $(UNITTESTS_PLUGIN_SO_OBJS)
# All objects (for the main Makefile.in's dependency tracking)
#extra_OBJS += $(OBJS_BUILT_AGAINST_GTEST)
@@ -97,23 +87,6 @@ OBJS_BUILT_AGAINST_GTEST = $(UNITTESTS_EXE_OBJS) test-ggc.o
# Use strict warnings for this front end.
unittests-warn = $(STRICT_WARN)
-# We avoid using $(BACKEND) from Makefile.in in order to avoid pulling
-# in main.o
-$(LIBGCCUNITTESTS_FILENAME): $(unittests_OBJS) \
- libbackend.a libcommon-target.a libcommon.a \
- $(CPPLIB) $(LIBDECNUMBER) \
- $(LIBDEPS) test-ggc.o
- +$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ -shared \
- $(unittests_OBJS) libbackend.a libcommon-target.a libcommon.a \
- $(CPPLIB) $(LIBDECNUMBER) $(LIBS) $(BACKENDLIBS) test-ggc.o \
- -Wl,-soname,$(LIBGCCUNITTESTS_SONAME)
-
-$(LIBGCCUNITTESTS_SONAME_SYMLINK): $(LIBGCCUNITTESTS_FILENAME)
- ln -sf $(LIBGCCUNITTESTS_FILENAME) $(LIBGCCUNITTESTS_SONAME_SYMLINK)
-
-$(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK): $(LIBGCCUNITTESTS_SONAME_SYMLINK)
- ln -sf $(LIBGCCUNITTESTS_SONAME_SYMLINK) $(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK)
-
#
# Build hooks:
@@ -121,24 +94,31 @@ unittests.all.cross:
unittests.start.encap:
unittests.rest.encap:
-$(OBJS_BUILT_AGAINST_GTEST): %.o: $(srcdir)/unittests/%.c
+# Needed by gtest-all.c:
+# -Wno-missing-field-initializers
+# -Wno-conversion-null
+# -Wno-suggest-attribute=format
+# Needed by various test-*.c:
+# -Wno-sign-compare
+
+$(OBJS_BUILT_AGAINST_GTEST): %.o: $(srcdir)/unittests/%.c \
+ gt-unittests-test-ggc.h
$(COMPILER) \
- $(shell gtest-config --cppflags --cxxflags) \
$(INCLUDES) \
$(ALL_COMPILERFLAGS) \
- -c -o $@ $<
-
-unittests.exe: $(UNITTESTS_EXE_OBJS) $(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK)
+ -c -o $@ $< \
+ -fPIC \
+ -Wno-missing-field-initializers \
+ -Wno-conversion-null \
+ -Wno-suggest-attribute=format \
+ -Wno-sign-compare
+
+$(UNITTESTS_PLUGIN_SO): $(UNITTESTS_PLUGIN_SO_OBJS)
$(LINKER) \
- $(shell gtest-config --ldflags --libs) \
- $(ALL_LINKERFLAGS) \
- -L. -lgccunittests \
-o $@ \
- $(UNITTESTS_EXE_OBJS) \
- $(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK)
-
-run-unittests.exe: unittests.exe
- LD_LIBRARY_PATH=. ./unittests.exe
+ -shared \
+ $(UNITTESTS_PLUGIN_SO_OBJS) \
+ -lpthread
# Documentation build hooks.
unittests.info:
@@ -149,19 +129,7 @@ lang_checks += check-unittests
#
# Install hooks:
-unittests.install-common: installdirs
- $(INSTALL_PROGRAM) $(LIBGCCUNITTESTS_FILENAME) \
- $(DESTDIR)/$(libdir)/$(LIBGCCUNITTESTS_FILENAME)
- ln -sf \
- $(LIBGCCUNITTESTS_FILENAME) \
- $(DESTDIR)/$(libdir)/$(LIBGCCUNITTESTS_SONAME_SYMLINK)
- ln -sf \
- $(LIBGCCUNITTESTS_SONAME_SYMLINK)\
- $(DESTDIR)/$(libdir)/$(LIBGCCUNITTESTS_LINKER_NAME_SYMLINK)
- $(INSTALL_PROGRAM) $(srcdir)/unittests/libgccunittests.h \
- $(DESTDIR)/$(includedir)/libgccunittests.h
- $(INSTALL_PROGRAM) $(srcdir)/unittests/libgccunittests++.h \
- $(DESTDIR)/$(includedir)/libgccunittests++.h
+unittests.install-common:
unittests.install-man:
diff --git a/gcc/unittests/config-lang.in b/gcc/unittests/config-lang.in
index 1e0bfd7..0d24609 100644
--- a/gcc/unittests/config-lang.in
+++ b/gcc/unittests/config-lang.in
@@ -29,6 +29,6 @@ compilers="unittest-suite"
target_libs=""
-gtfiles="\$(srcdir)/unittests/unittests-frontend.c \$(srcdir)/unittests/test-ggc.c"
+gtfiles=""
build_by_default="no"
diff --git a/gcc/unittests/test-ggc.c b/gcc/unittests/test-ggc.c
index 197b4b5..4418a21 100644
--- a/gcc/unittests/test-ggc.c
+++ b/gcc/unittests/test-ggc.c
@@ -125,6 +125,12 @@ TEST_F (ggc_test, finalization)
static GTY((deletable)) test_struct *test_of_deletable;
+/* FIXME: we can't do this test via a plugin as it stands.
+ The list of deletable roots is fixed by the main gengtype
+ run; there isn't yet a way to add extra
+ deletable roots (PLUGIN_REGISTER_GGC_ROOTS is for regular
+ roots). */
+#if 0
TEST_F (ggc_test, deletable_global)
{
test_of_deletable = ggc_cleared_alloc <test_struct> ();
@@ -134,6 +140,7 @@ TEST_F (ggc_test, deletable_global)
EXPECT_EQ (NULL, test_of_deletable);
}
+#endif
/* Verify that gengtype etc can cope with inheritance. */
diff --git a/gcc/unittests/unittests-plugin.c b/gcc/unittests/unittests-plugin.c
new file mode 100644
index 0000000..83dc4f3
--- /dev/null
+++ b/gcc/unittests/unittests-plugin.c
@@ -0,0 +1,182 @@
+/* Plugin that process internal tests for sreal. */
+#include "config.h"
+#include "gtest/gtest.h"
+#include "gcc-plugin.h"
+#include "system.h"
+#include "coretypes.h"
+#include "tree.h"
+#include "tm.h"
+#include "toplev.h"
+#include "hash-table.h"
+#include "vec.h"
+#include "ggc.h"
+#include "basic-block.h"
+#include "tree-ssa-alias.h"
+#include "internal-fn.h"
+#include "gimple-fold.h"
+#include "tree-eh.h"
+#include "gimple-expr.h"
+#include "is-a.h"
+#include "gimple.h"
+#include "tree-pass.h"
+#include "intl.h"
+#include "context.h"
+#include "diagnostic.h"
+#include "bitmap.h"
+
+int plugin_is_GPL_compatible;
+
+/* Print test output in DejaGnu form. */
+class deja_gnu_printer : public ::testing::EmptyTestEventListener
+{
+ public:
+ deja_gnu_printer (FILE *outfile, int verbose)
+ : m_outfile (outfile),
+ m_verbose (verbose)
+ {}
+
+ private:
+ virtual void
+ OnTestCaseStart(const ::testing::TestCase& test_case)
+ {
+ if (m_verbose)
+ fprintf (m_outfile, "NOTE: %s: case starting\n",
+ test_case.name ());
+ }
+
+ /* Vfunc called before a test starts. */
+ virtual void
+ OnTestStart (const ::testing::TestInfo& test_info)
+ {
+ if (m_verbose)
+ fprintf (m_outfile,
+ "NOTE: %s.%s: test starting\n",
+ test_info.test_case_name (), test_info.name ());
+ m_per_test_fails = 0;
+ }
+
+ /* Vfunc called after a failed assertion or a SUCCEED() invocation. */
+ virtual void
+ OnTestPartResult (const ::testing::TestPartResult& test_part_result)
+ {
+ fprintf (m_outfile,
+ "%s: %s:%d: %s\n",
+ test_part_result.failed () ? "FAIL" : "PASS",
+ test_part_result.file_name (),
+ test_part_result.line_number (),
+ test_part_result.summary ());
+ if (test_part_result.failed ())
+ m_per_test_fails++;
+ }
+
+ /* Vfunc called after a test ends. */
+ virtual void
+ OnTestEnd (const ::testing::TestInfo& test_info)
+ {
+ if (m_verbose)
+ fprintf (m_outfile,
+ "NOTE: %s.%s: test ending: %i failure(s)\n",
+ test_info.test_case_name (), test_info.name (),
+ m_per_test_fails);
+ fprintf (m_outfile,
+ "%s: %s.%s\n",
+ m_per_test_fails > 0 ? "FAIL" : "PASS",
+ test_info.test_case_name (), test_info.name ());
+ }
+
+ virtual void
+ OnTestCaseEnd (const ::testing::TestCase& test_case)
+ {
+ if (m_verbose)
+ fprintf (m_outfile, "NOTE: %s: case ending\n",
+ test_case.name ());
+ }
+
+ private:
+ FILE *m_outfile;
+ int m_verbose;
+ int m_per_test_fails;
+};
+
+/* Get rid of the default gtest result printer, and instead use LISTENER,
+ taking ownership of the ptr. */
+
+static void
+replace_default_gtest_result_printer (::testing::TestEventListener *listener)
+{
+ ::testing::TestEventListeners& listeners =
+ ::testing::UnitTest::GetInstance()->listeners();
+ delete listeners.Release(listeners.default_result_printer());
+ listeners.Append(listener);
+}
+
+/* Callback handler for the PLUGIN_FINISH event.
+ At this point, all GCC subsystems should be initialized and
+ "warmed up"; this is where we run our unit tests. */
+
+static void
+on_finish (void */*gcc_data*/, void */*user_data*/)
+{
+ /* FIXME: get these from the passed-in args to the plugin?
+ would we need to store them? */
+ int argc = 0;
+ char **argv = NULL;
+ ::testing::InitGoogleTest (&argc, argv);
+
+ /* Use our custom result-printer. */
+ replace_default_gtest_result_printer (new deja_gnu_printer (stderr, 0));
+
+ /* Reset some state. */
+ input_location = UNKNOWN_LOCATION;
+ bitmap_obstack_initialize (NULL);
+
+ /* Run the tests. */
+ int result = RUN_ALL_TESTS();
+
+ /* Ensure that a test failure leads to the process exiting with
+ a non-zero exit code. */
+ if (result)
+ error ("at least one test failure occurred");
+
+ /* Cleanup. */
+ bitmap_obstack_release (NULL);
+}
+
+/* Declared in "gt-unittests-test-ggc.h". */
+extern const struct ggc_root_tab gt_ggc_r_gt_unittests_test_ggc_h[];
+#if 0
+extern const struct ggc_root_tab gt_ggc_rd_gt_unittests_test_ggc_h[];
+#endif
+
+int
+plugin_init (struct plugin_name_args *plugin_info,
+ struct plugin_gcc_version */*version*/)
+{
+ if (0)
+ fprintf (stderr, "got here\n");
+
+ register_callback (plugin_info->base_name,
+ PLUGIN_REGISTER_GGC_ROOTS,
+ NULL,
+ const_cast <ggc_root_tab *> (
+ gt_ggc_r_gt_unittests_test_ggc_h));
+ /* FIXME: we'd use this for test-ggc.c's test_of_deletable.
+ However we'd need to register it as a different kind of roottab;
+ doing it with PLUGIN_REGISTER_GGC_ROOTS leads to a segfault on
+ collection due to the NULL cb.
+ It looks like we need another hook; gt_ggc_deletable_rtab is
+ currently not expandable. */
+#if 0
+ register_callback (plugin_info->base_name,
+ PLUGIN_REGISTER_GGC_ROOTS,
+ NULL,
+ const_cast <ggc_root_tab *> (
+ gt_ggc_rd_gt_unittests_test_ggc_h));
+#endif
+ register_callback (plugin_info->base_name,
+ PLUGIN_FINISH,
+ on_finish,
+ NULL); /* void *user_data */
+
+ return 0;
+}