diff mbox

Don't run guality.exp tests with LTO_TORTURE_OPTIONS.

Message ID 1404295545-17107-1-git-send-email-mjw@redhat.com
State New
Headers show

Commit Message

Mark Wielaard July 2, 2014, 10:05 a.m. UTC 
Hi,

While writing new guality.exp tests I noticed they often just fail
when ran with -flto, even though they PASS in all other cases. When
I asked on irc about this I was told that LTO was known to not play
well with DWARF debuginfo anyway. If that is the case, it seems better
to disable -flto for guality.exp for now since it just adds noise
(and unfortunately guality.exp already has lots of FAILs). The attached
patch does this. It changes the make check-c RUNTESTFLAGS=guality.exp
results (against gdb git master) from:

# of expected passes            3121
# of unexpected failures        180
# of unexpected successes       33
# of expected failures          15
# of unsupported tests          78

to:

# of expected passes            2389
# of unexpected failures        110
# of unexpected successes       26
# of expected failures          10
# of unsupported tests          54

That looks like an improvement to me, even though the number of FAILs
is still a bit high. Does this look a good thing to apply? Or are people
actively working on reducing the number of LTO guality.exp failures and
can we expect them to turn into PASSes soon anyway?

Thanks,

Mark

gcc/testsuite/ChangeLog

	* gcc.dg/guality/guality.exp: Remove LTO_TORTURE_OPTIONS from
	set-torture-options. Only use DG_TORTURE_OPTIONS.
---
 gcc/testsuite/ChangeLog                  |    5 +++++
 gcc/testsuite/gcc.dg/guality/guality.exp |    6 ++++++
 2 files changed, 11 insertions(+), 0 deletions(-)

Comments

Jason Merrill July 2, 2014, 11:06 p.m. UTC | #1 
I think that makes sense; I'm not aware of anyone working on improving 
LTO debugging.

Jason
Jakub Jelinek July 3, 2014, 5:37 a.m. UTC | #2 
On Wed, Jul 02, 2014 at 04:06:30PM -0700, Jason Merrill wrote:
> I think that makes sense; I'm not aware of anyone working on improving LTO
> debugging.

I think at this point all we care about is that with -flto we don't ICE on
those, perhaps we should arrange to change all the tests into dg-do compile
with -flto and ignore all gdb-test and have some env var override which
would force full testing also with -flto?

	Jakub
Richard Biener July 3, 2014, 6:55 a.m. UTC | #3 
On July 3, 2014 1:06:30 AM CEST, Jason Merrill <jason@redhat.com> wrote:
>I think that makes sense; I'm not aware of anyone working on improving 
>LTO debugging.

I've done that in the past.  So it would be nice to verify we don't regress existing tests.

Richard.

>Jason
Richard Biener July 3, 2014, 7:41 a.m. UTC | #4 
On July 3, 2014 7:37:13 AM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
>On Wed, Jul 02, 2014 at 04:06:30PM -0700, Jason Merrill wrote:
>> I think that makes sense; I'm not aware of anyone working on
>improving LTO
>> debugging.
>
>I think at this point all we care about is that with -flto we don't ICE
>on
>those, perhaps we should arrange to change all the tests into dg-do
>compile
>with -flto and ignore all gdb-test and have some env var override which
>would force full testing also with -flto?

I think the individual tests that currently fail can be appropriately changed, no?

It would be bad to lose the lto regression testing here.

Richard.

>	Jakub
Jakub Jelinek July 3, 2014, 7:55 a.m. UTC | #5 
On Thu, Jul 03, 2014 at 09:41:15AM +0200, Richard Biener wrote:
> On July 3, 2014 7:37:13 AM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
> >On Wed, Jul 02, 2014 at 04:06:30PM -0700, Jason Merrill wrote:
> >> I think that makes sense; I'm not aware of anyone working on
> >improving LTO
> >> debugging.
> >
> >I think at this point all we care about is that with -flto we don't ICE
> >on
> >those, perhaps we should arrange to change all the tests into dg-do
> >compile
> >with -flto and ignore all gdb-test and have some env var override which
> >would force full testing also with -flto?
> 
> I think the individual tests that currently fail can be appropriately changed, no?

That is hard, as whether a test fails heavily depends on the optimization flags and
targets, so maintaining xfails would be a nightmare.

BTW, the trunk has lots of guality regressions even on x86_64-linux compared to 4.9 branch
now :(, some of them are LTO only, but others are not.

+FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 18 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 16 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 14 arg7 == 30
+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 arg7 == 30
-XPASS: gcc.dg/guality/pr41353-1.c  -O1  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -O2  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -O3 -fomit-frame-pointer  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -O3 -g  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -Os  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 28 j == 28 + 37
-XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 28 j == 28 + 37
+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 35 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 36 e == &a[1]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 39 c == &a[0]
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 40 v == 1
+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 41 e == &a[1]
+FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 24 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 15 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 15 x == 7
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 24 l == 10
+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 24 x == 7
+FAIL: gcc.dg/guality/pr43479.c  -O1  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O1  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -Os  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -Os  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 18 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 13 h == 9
+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 18 h == 9
+FAIL: gcc.dg/guality/pr45003-2.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 10 a == 0x8078
+FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 d == 112
+FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 e == 142
+FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 20 x == 36
+FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 23 x == 98
+FAIL: gcc.dg/guality/pr54519-4.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 17 x == 6
+FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -O2  line 43 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 43 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 43 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -Os  line 43 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 43 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 21 a.i == 4
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 21 a.j == 14
+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 43 a.i == 4
+FAIL: gcc.dg/guality/vla-2.c  -O2  line 25 sizeof (a) == 6 * sizeof (int)
+FAIL: gcc.dg/guality/vla-2.c  -O3 -fomit-frame-pointer  line 25 sizeof (a) == 6 * sizeof (int)
+FAIL: gcc.dg/guality/vla-2.c  -O3 -g  line 25 sizeof (a) == 6 * sizeof (int)
+FAIL: gcc.dg/guality/vla-2.c  -Os  line 25 sizeof (a) == 6 * sizeof (int)
+FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 25 sizeof (a) == 6 * sizeof (int)
+FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 25 sizeof (a) == 6 * sizeof (int)

	Jakub
Richard Biener July 3, 2014, 6:37 p.m. UTC | #6 
On July 3, 2014 9:55:36 AM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
>On Thu, Jul 03, 2014 at 09:41:15AM +0200, Richard Biener wrote:
>> On July 3, 2014 7:37:13 AM CEST, Jakub Jelinek <jakub@redhat.com>
>wrote:
>> >On Wed, Jul 02, 2014 at 04:06:30PM -0700, Jason Merrill wrote:
>> >> I think that makes sense; I'm not aware of anyone working on
>> >improving LTO
>> >> debugging.
>> >
>> >I think at this point all we care about is that with -flto we don't
>ICE
>> >on
>> >those, perhaps we should arrange to change all the tests into dg-do
>> >compile
>> >with -flto and ignore all gdb-test and have some env var override
>which
>> >would force full testing also with -flto?
>> 
>> I think the individual tests that currently fail can be appropriately
>changed, no?
>
>That is hard, as whether a test fails heavily depends on the
>optimization flags and
>targets, so maintaining xfails would be a nightmare.

Well, simply removing the regression testing for LTO is a maintainance nightmare as well.

The guality testsuite is very noisy anyway with all the xfail and xpass.

Richard.

>BTW, the trunk has lots of guality regressions even on x86_64-linux
>compared to 4.9 branch
>now :(, some of them are LTO only, but others are not.
>
>+FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 16
>arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 18
>arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 18 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 14
>arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 16
>arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 16 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 14 arg7 == 30
>+FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 16 arg7 == 30
>-XPASS: gcc.dg/guality/pr41353-1.c  -O1  line 28 j == 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -O2  line 28 j == 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -O3 -fomit-frame-pointer  line 28 j
>== 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -O3 -g  line 28 j == 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -Os  line 28 j == 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 28 j == 28 + 37
>-XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 28 j == 28 + 37
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 35 v
>== 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 36 e
>== &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 39 c
>== &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 40 v
>== 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 41 e
>== &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-loops  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-loops  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-loops  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-loops  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-loops  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-all-loops -finline-functions  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-all-loops -finline-functions  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-all-loops -finline-functions  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-all-loops -finline-functions  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer
>-funroll-all-loops -finline-functions  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 35 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 36 e == &a[1]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 39 c == &a[0]
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 40 v == 1
>+FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 41 e == &a[1]
>+FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 l ==
>10
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 x ==
>7
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 l ==
>10
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 x ==
>7
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 15 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 15 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 24 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 24 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 15 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 15 x == 7
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 24 l == 10
>+FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 24 x == 7
>+FAIL: gcc.dg/guality/pr43479.c  -O1  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O1  line 18 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2  line 18 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 13 h ==
>9
>+FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 18 h ==
>9
>+FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 18 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -Os  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -Os  line 18 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 18 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 13 h == 9
>+FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 18 h == 9
>+FAIL: gcc.dg/guality/pr45003-2.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 10 a == 0x8078
>+FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 16 d == 112
>+FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 16 e == 142
>+FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 20 x == 36
>+FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 23 x == 98
>+FAIL: gcc.dg/guality/pr54519-4.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 17 x == 6
>+FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.j == 14
>+FAIL: gcc.dg/guality/sra-1.c  -O2  line 43 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.i ==
>4
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.j ==
>14
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 43 a.i ==
>4
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.j == 14
>+FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 43 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.j == 14
>+FAIL: gcc.dg/guality/sra-1.c  -Os  line 43 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 21 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 21 a.j == 14
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 43 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 21 a.i == 4
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 21 a.j == 14
>+FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 43 a.i == 4
>+FAIL: gcc.dg/guality/vla-2.c  -O2  line 25 sizeof (a) == 6 * sizeof
>(int)
>+FAIL: gcc.dg/guality/vla-2.c  -O3 -fomit-frame-pointer  line 25 sizeof
>(a) == 6 * sizeof (int)
>+FAIL: gcc.dg/guality/vla-2.c  -O3 -g  line 25 sizeof (a) == 6 * sizeof
>(int)
>+FAIL: gcc.dg/guality/vla-2.c  -Os  line 25 sizeof (a) == 6 * sizeof
>(int)
>+FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fno-use-linker-plugin
>-flto-partition=none  line 25 sizeof (a) == 6 * sizeof (int)
>+FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fuse-linker-plugin
>-fno-fat-lto-objects  line 25 sizeof (a) == 6 * sizeof (int)
>
>	Jakub
Jakub Jelinek July 3, 2014, 6:38 p.m. UTC | #7 
On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
> Well, simply removing the regression testing for LTO is a maintainance nightmare as well.
> 
> The guality testsuite is very noisy anyway with all the xfail and xpass.

Let's keep it as is then?

	Jakub
Richard Biener July 3, 2014, 7:52 p.m. UTC | #8 
On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
>On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
>> Well, simply removing the regression testing for LTO is a
>maintainance nightmare as well.
>> 
>> The guality testsuite is very noisy anyway with all the xfail and
>xpass.
>
>Let's keep it as is then?

That works for me.

Richard.

>	Jakub
Mark Wielaard July 3, 2014, 8:04 p.m. UTC | #9 
On Thu, 2014-07-03 at 21:52 +0200, Richard Biener wrote:
> On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
> >On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
> >> Well, simply removing the regression testing for LTO is a
> >maintainance nightmare as well.
> >> 
> >> The guality testsuite is very noisy anyway with all the xfail and
> >xpass.
> >
> >Let's keep it as is then?
> 
> That works for me.

I don't find that very satisfactory. I want to add more guality tests,
but the fact that they are unreliable and by default introduce even more
FAILs when lto is enabled makes that not very attractive. I do like
Jakub's suggestion to disable the guality tests be run with lto by
default, but provide an environment variable to enable them for those
that want to try them anyway. Shall I implement that?

Thanks,

Mark
Jakub Jelinek July 3, 2014, 8:14 p.m. UTC | #10 
On Thu, Jul 03, 2014 at 10:04:35PM +0200, Mark Wielaard wrote:
> On Thu, 2014-07-03 at 21:52 +0200, Richard Biener wrote:
> > On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
> > >On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
> > >> Well, simply removing the regression testing for LTO is a
> > >maintainance nightmare as well.
> > >> 
> > >> The guality testsuite is very noisy anyway with all the xfail and
> > >xpass.
> > >
> > >Let's keep it as is then?
> > 
> > That works for me.
> 
> I don't find that very satisfactory. I want to add more guality tests,
> but the fact that they are unreliable and by default introduce even more
> FAILs when lto is enabled makes that not very attractive. I do like
> Jakub's suggestion to disable the guality tests be run with lto by
> default, but provide an environment variable to enable them for those
> that want to try them anyway. Shall I implement that?

They aren't that unrealiable (at least, if people committing patches don't
ignore regressions in there).  Just one should diff contrib/test_summary
output from earlier builds to the latest, that way it is clear what is a
regression and what is not.

	Jakub
Mark Wielaard July 3, 2014, 8:47 p.m. UTC | #11 
On Thu, 2014-07-03 at 22:14 +0200, Jakub Jelinek wrote:
> On Thu, Jul 03, 2014 at 10:04:35PM +0200, Mark Wielaard wrote:
> > On Thu, 2014-07-03 at 21:52 +0200, Richard Biener wrote:
> > > On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
> > > >On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
> > > >> Well, simply removing the regression testing for LTO is a
> > > >maintainance nightmare as well.
> > > >> 
> > > >> The guality testsuite is very noisy anyway with all the xfail and
> > > >xpass.
> > > >
> > > >Let's keep it as is then?
> > > 
> > > That works for me.
> > 
> > I don't find that very satisfactory. I want to add more guality tests,
> > but the fact that they are unreliable and by default introduce even more
> > FAILs when lto is enabled makes that not very attractive. I do like
> > Jakub's suggestion to disable the guality tests be run with lto by
> > default, but provide an environment variable to enable them for those
> > that want to try them anyway. Shall I implement that?
> 
> They aren't that unrealiable (at least, if people committing patches don't
> ignore regressions in there).  Just one should diff contrib/test_summary
> output from earlier builds to the latest, that way it is clear what is a
> regression and what is not.

The are much more unreliable than any other test. With guality.exp
disabled one can just eyeball the results and investigate new FAILS.
There are only a handful. When you include guality.exp you can easily
get the impression the gcc testsuite is really bad (and it isn't!) And
the problem is that it makes adding new tests a pain. See my new tests,
they introduce new FAILs because LTO is enabled by default for
guality.exp at the moment. It just results in a slow increase of FAILs
that people have to ignore. And I am afraid that will just result in
people missing real regressions.

I don't mind if there is active work to fix LTO DWARF debuginfo
generation issues and the guality.exp LTO failures will soon disappear,
but if there is no active work on reducing the amount of failures and
introducing new guality.exp testcases will keep adding more FAILs I
think we are much better off disabling them for now.

Thanks,

Mark
Richard Biener July 4, 2014, 8:12 a.m. UTC | #12 
On July 3, 2014 10:47:52 PM CEST, Mark Wielaard <mjw@redhat.com> wrote:
>On Thu, 2014-07-03 at 22:14 +0200, Jakub Jelinek wrote:
>> On Thu, Jul 03, 2014 at 10:04:35PM +0200, Mark Wielaard wrote:
>> > On Thu, 2014-07-03 at 21:52 +0200, Richard Biener wrote:
>> > > On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <jakub@redhat.com>
>wrote:
>> > > >On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote:
>> > > >> Well, simply removing the regression testing for LTO is a
>> > > >maintainance nightmare as well.
>> > > >> 
>> > > >> The guality testsuite is very noisy anyway with all the xfail
>and
>> > > >xpass.
>> > > >
>> > > >Let's keep it as is then?
>> > > 
>> > > That works for me.
>> > 
>> > I don't find that very satisfactory. I want to add more guality
>tests,
>> > but the fact that they are unreliable and by default introduce even
>more
>> > FAILs when lto is enabled makes that not very attractive. I do like
>> > Jakub's suggestion to disable the guality tests be run with lto by
>> > default, but provide an environment variable to enable them for
>those
>> > that want to try them anyway. Shall I implement that?
>> 
>> They aren't that unrealiable (at least, if people committing patches
>don't
>> ignore regressions in there).  Just one should diff
>contrib/test_summary
>> output from earlier builds to the latest, that way it is clear what
>is a
>> regression and what is not.
>
>The are much more unreliable than any other test. With guality.exp
>disabled one can just eyeball the results and investigate new FAILS.
>There are only a handful. When you include guality.exp you can easily
>get the impression the gcc testsuite is really bad (and it isn't!) And
>the problem is that it makes adding new tests a pain. See my new tests,
>they introduce new FAILs because LTO is enabled by default for
>guality.exp at the moment. It just results in a slow increase of FAILs
>that people have to ignore. And I am afraid that will just result in
>people missing real regressions.
>
>I don't mind if there is active work to fix LTO DWARF debuginfo
>generation issues and the guality.exp LTO failures will soon disappear,
>but if there is no active work on reducing the amount of failures and
>introducing new guality.exp testcases will keep adding more FAILs I
>think we are much better off disabling them for now.

Can't you simply add proper xfails for lto when you add new tests that fail with lto?
(Please quickly check why - usually the tests are just optimized in an unexpected way - compare with -fwhole-program behavior for example). BTW, reducing the number of lto variants checked to -O2 -flto would be fine with me.

Richard.

>Thanks,
>
>Mark
Richard Biener July 7, 2014, 8:21 a.m. UTC | #13 
On Thu, 3 Jul 2014, Jakub Jelinek wrote:

> On Thu, Jul 03, 2014 at 09:41:15AM +0200, Richard Biener wrote:
> > On July 3, 2014 7:37:13 AM CEST, Jakub Jelinek <jakub@redhat.com> wrote:
> > >On Wed, Jul 02, 2014 at 04:06:30PM -0700, Jason Merrill wrote:
> > >> I think that makes sense; I'm not aware of anyone working on
> > >improving LTO
> > >> debugging.
> > >
> > >I think at this point all we care about is that with -flto we don't ICE
> > >on
> > >those, perhaps we should arrange to change all the tests into dg-do
> > >compile
> > >with -flto and ignore all gdb-test and have some env var override which
> > >would force full testing also with -flto?
> > 
> > I think the individual tests that currently fail can be appropriately changed, no?
> 
> That is hard, as whether a test fails heavily depends on the optimization flags and
> targets, so maintaining xfails would be a nightmare.
> 
> BTW, the trunk has lots of guality regressions even on x86_64-linux compared to 4.9 branch
> now :(, some of them are LTO only, but others are not.

Most of the "noise" (in terms of line-count) are the FAILs that are
not LTO exclusive because we run for all of the torture options
(IMHO a bit excessive).

Btw, it seems that nobody files bugs / triages guality regressions
which means we could as well remove the guality as a whole (it's
slow anyway).

Richard.

> +FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O1  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O3 -fomit-frame-pointer  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O3 -g  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -Os  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 18 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O1  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O3 -fomit-frame-pointer  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O3 -g  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -Os  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 16 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 14 arg7 == 30
> +FAIL: gcc.dg/guality/pr36728-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 arg7 == 30
> -XPASS: gcc.dg/guality/pr41353-1.c  -O1  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -O2  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -O3 -fomit-frame-pointer  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -O3 -g  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -Os  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 28 j == 28 + 37
> -XPASS: gcc.dg/guality/pr41353-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 28 j == 28 + 37
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-loops  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -fomit-frame-pointer -funroll-all-loops -finline-functions  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O3 -g  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -Os  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 35 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 36 e == &a[1]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 39 c == &a[0]
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 40 v == 1
> +FAIL: gcc.dg/guality/pr43051-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 41 e == &a[1]
> +FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -fomit-frame-pointer  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O3 -g  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -Os  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -Os  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 15 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 15 x == 7
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 24 l == 10
> +FAIL: gcc.dg/guality/pr43177.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 24 x == 7
> +FAIL: gcc.dg/guality/pr43479.c  -O1  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O1  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O3 -fomit-frame-pointer  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O3 -g  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -Os  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -Os  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 18 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 13 h == 9
> +FAIL: gcc.dg/guality/pr43479.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 18 h == 9
> +FAIL: gcc.dg/guality/pr45003-2.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 10 a == 0x8078
> +FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 d == 112
> +FAIL: gcc.dg/guality/pr45882.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 16 e == 142
> +FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 20 x == 36
> +FAIL: gcc.dg/guality/pr54519-3.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 23 x == 98
> +FAIL: gcc.dg/guality/pr54519-4.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 17 x == 6
> +FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O2  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -O2  line 43 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -fomit-frame-pointer  line 43 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -O3 -g  line 43 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -Os  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -Os  line 43 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 43 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 21 a.i == 4
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 21 a.j == 14
> +FAIL: gcc.dg/guality/sra-1.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 43 a.i == 4
> +FAIL: gcc.dg/guality/vla-2.c  -O2  line 25 sizeof (a) == 6 * sizeof (int)
> +FAIL: gcc.dg/guality/vla-2.c  -O3 -fomit-frame-pointer  line 25 sizeof (a) == 6 * sizeof (int)
> +FAIL: gcc.dg/guality/vla-2.c  -O3 -g  line 25 sizeof (a) == 6 * sizeof (int)
> +FAIL: gcc.dg/guality/vla-2.c  -Os  line 25 sizeof (a) == 6 * sizeof (int)
> +FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fno-use-linker-plugin -flto-partition=none  line 25 sizeof (a) == 6 * sizeof (int)
> +FAIL: gcc.dg/guality/vla-2.c  -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects  line 25 sizeof (a) == 6 * sizeof (int)
> 
> 	Jakub
> 
>
diff mbox

Patch

diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
index b364f40..421e006 100644
--- a/gcc/testsuite/ChangeLog
+++ b/gcc/testsuite/ChangeLog
@@ -1,3 +1,8 @@ 
+2014-07-02  Mark Wielaard  <mjw@redhat.com>
+
+	* gcc.dg/guality/guality.exp: Remove LTO_TORTURE_OPTIONS from
+	set-torture-options. Only use DG_TORTURE_OPTIONS.
+
 2014-07-02  Uros Bizjak  <ubizjak@gmail.com>
 
 	* gfortran.dg/ieee/ieee_1.F90 (dg-additional-options): Remove -O0.
diff --git a/gcc/testsuite/gcc.dg/guality/guality.exp b/gcc/testsuite/gcc.dg/guality/guality.exp
index 5e714dd..7991661 100644
--- a/gcc/testsuite/gcc.dg/guality/guality.exp
+++ b/gcc/testsuite/gcc.dg/guality/guality.exp
@@ -47,6 +47,12 @@  if {[check_guality "
     return 0;
   }
 "]} {
+  # Override default torture-options which include LTO_TORTURE_OPTIONS
+  # Currently LTO and the guality tests that depend on debuginfo don't mix.
+  global DG_TORTURE_OPTIONS
+  torture-init
+  set-torture-options $DG_TORTURE_OPTIONS
+
   gcc-dg-runtest [lsort [glob $srcdir/$subdir/*.c]] ""
   gcc-dg-runtest [lsort [glob $srcdir/c-c++-common/guality/*.c]] "-Wc++-compat"
 }