| Message ID | 1459211926.12955.37.camel@oc8801110288.ibm.com |
|---|---|
| State | New |
| Headers | show |
On Mar 28, 2016, at 5:38 PM, Bill Schmidt <wschmidt@linux.vnet.ibm.com> wrote: > For a long time we've had hundreds of failing guality tests. These > failures don't seem to have any correlation with gdb functionality for > POWER, which is working fine. > Verified to remove hundreds of failure messages on > powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? So, I think this should be up to the target maintainer and/or the quality people, but I think it is reasonable to do this. If all or most target maintainers do this, then we flip to not run, unless the target triplet asks for them. On my (random) target, I don’t fail any quality tests: newlib/newlib/libc/posix/popen.c:138: undefined reference to `vfork’ newlib/newlib/libc/posix/popen.c:218: undefined reference to `waitpid’ so they all turn off. I have a fork (it forks my simulator) and a wait, but no vfork or waitpid. This might explain why some people don’t trip on the issue.
On Mon, Mar 28, 2016 at 07:38:46PM -0500, Bill Schmidt wrote: > For a long time we've had hundreds of failing guality tests. These > failures don't seem to have any correlation with gdb functionality for > POWER, which is working fine. At this point the value of these tests to > us seems questionable. Fixing these is such low priority that it is > unlikely we will ever get around to it. In the meanwhile, the failures > simply clutter up our regression test reports. Thus I'd like to disable > them, and that's what this test does. > > Verified to remove hundreds of failure messages on > powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? This is IMNSHO very wrong, you then lose tracking of regressions in the debug info quality. It is true that the debug info quality is already pretty bad on powerpc*, it would be really very much desirable if anyone had time to analyze some of them and improve stuff, but we at least shouldn't regress. Guality testsuite has various FAILs and/or XFAILs on lots of architectures, the problem is that the testing matrix is simply too large to have them in the testcases - it depends on the target, various ISA settings on the target, on the optimization level (most of the guality tests are torture tested through -O0 up to -O3 with extra flags), and in some cases also on the version of the used GDB. For guality, the most effective test for regressions is simply always running contrib/test_summary after all your bootstraps and then just diffing up that against the same from earlier bootstrap. Jakub
Jakub Jelinek <jakub@redhat.com> writes: > For guality, the most effective test for regressions is simply always > running contrib/test_summary after all your bootstraps and then just > diffing up that against the same from earlier bootstrap. Or use contrib/compare_tests. Andreas.
Hi Jakub, On Tue, 2016-03-29 at 08:53 +0200, Jakub Jelinek wrote: > On Mon, Mar 28, 2016 at 07:38:46PM -0500, Bill Schmidt wrote: > > For a long time we've had hundreds of failing guality tests. These > > failures don't seem to have any correlation with gdb functionality for > > POWER, which is working fine. At this point the value of these tests to > > us seems questionable. Fixing these is such low priority that it is > > unlikely we will ever get around to it. In the meanwhile, the failures > > simply clutter up our regression test reports. Thus I'd like to disable > > them, and that's what this test does. > > > > Verified to remove hundreds of failure messages on > > powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? > > This is IMNSHO very wrong, you then lose tracking of regressions in the > debug info quality. It is true that the debug info quality is already > pretty bad on powerpc*, it would be really very much desirable if > anyone had time to analyze some of them and improve stuff, > but we at least shouldn't regress. Guality testsuite has various FAILs > and/or XFAILs on lots of architectures, the problem is that the testing > matrix is simply too large to have them in the testcases > - it depends on the target, various ISA settings on the target, on the > optimization level (most of the guality tests are torture tested through > -O0 up to -O3 with extra flags), and in some cases also on the version of > the used GDB. > > For guality, the most effective test for regressions is simply always > running contrib/test_summary after all your bootstraps and then just > diffing up that against the same from earlier bootstrap. And of course we do this, and we can keep doing it. My main purpose in opening this issue is to try to understand whether we are getting any benefit from these tests, rather than just noise. When you say that "the debug info quality is already pretty bad on powerpc*," do you mean that it is known to be bad, or simply that we have a lot of guality failures that may or may not indicate that the debug info is bad? I don't have experiential evidence of bad debug info that shows up during debugging sessions. Perhaps these are corner cases that I will never encounter in practice? Or perhaps the tests are just badly formed? The failing tests have all been bit-rotten (or never worked) since before I joined this project, and from what others tell me, for at least a decade. As you suggest here, others have always told me just to ignore the existing guality failures. However, this can easily lead to a culture of "ignore any guality failure, that stuff is junk" which can cause regressions to be missed. (I can't say that I've actually observed this, but it is a concern I have.) I have been consistently told that the same situation exists on most of the supported targets, again because of the size of the testing matrix. I'd be interested in knowing if this is true, or just anecdotal. The other point, "it would be really very much desirable if anyone had time to analyze some of them and improve stuff," has to be answered by "apparently nobody does." I am currently tracking well over 200 improvements I would like to see made to the powerpc64le target alone. Investigating old guality failures isn't even on that list. Our team won't have time for it, and if we have bounty money to spend, it will be spent on more important things. That's just the economic reality, not a desire to disrespect the guality tests or anyone associated with them. From my limited perspective, it seems like the guality tests are unique within the test suite as a set of tests that everyone just expects to have lots of failures. Is that healthy? Will it ever change? That said, it is clear that you feel the guality tests provide at least some value in their present state, so we can continue to live with things as they are. I'm just curious how others feel about the state of these tests. Thanks, Bill > > Jakub >
On Tue, Mar 29, 2016 at 3:19 PM, Bill Schmidt <wschmidt@linux.vnet.ibm.com> wrote: > Hi Jakub, > > On Tue, 2016-03-29 at 08:53 +0200, Jakub Jelinek wrote: >> On Mon, Mar 28, 2016 at 07:38:46PM -0500, Bill Schmidt wrote: >> > For a long time we've had hundreds of failing guality tests. These >> > failures don't seem to have any correlation with gdb functionality for >> > POWER, which is working fine. At this point the value of these tests to >> > us seems questionable. Fixing these is such low priority that it is >> > unlikely we will ever get around to it. In the meanwhile, the failures >> > simply clutter up our regression test reports. Thus I'd like to disable >> > them, and that's what this test does. >> > >> > Verified to remove hundreds of failure messages on >> > powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? >> >> This is IMNSHO very wrong, you then lose tracking of regressions in the >> debug info quality. It is true that the debug info quality is already >> pretty bad on powerpc*, it would be really very much desirable if >> anyone had time to analyze some of them and improve stuff, >> but we at least shouldn't regress. Guality testsuite has various FAILs >> and/or XFAILs on lots of architectures, the problem is that the testing >> matrix is simply too large to have them in the testcases >> - it depends on the target, various ISA settings on the target, on the >> optimization level (most of the guality tests are torture tested through >> -O0 up to -O3 with extra flags), and in some cases also on the version of >> the used GDB. >> >> For guality, the most effective test for regressions is simply always >> running contrib/test_summary after all your bootstraps and then just >> diffing up that against the same from earlier bootstrap. > > And of course we do this, and we can keep doing it. My main purpose in > opening this issue is to try to understand whether we are getting any > benefit from these tests, rather than just noise. > > When you say that "the debug info quality is already pretty bad on > powerpc*," do you mean that it is known to be bad, or simply that we > have a lot of guality failures that may or may not indicate that the > debug info is bad? I don't have experiential evidence of bad debug info > that shows up during debugging sessions. Perhaps these are corner cases > that I will never encounter in practice? Or perhaps the tests are just > badly formed? > > The failing tests have all been bit-rotten (or never worked) since > before I joined this project, and from what others tell me, for at least > a decade. As you suggest here, others have always told me just to > ignore the existing guality failures. However, this can easily lead to > a culture of "ignore any guality failure, that stuff is junk" which can > cause regressions to be missed. (I can't say that I've actually > observed this, but it is a concern I have.) > > I have been consistently told that the same situation exists on most of > the supported targets, again because of the size of the testing matrix. > I'd be interested in knowing if this is true, or just anecdotal. > > The other point, "it would be really very much desirable if > anyone had time to analyze some of them and improve stuff," has to be > answered by "apparently nobody does." I am currently tracking well over > 200 improvements I would like to see made to the powerpc64le target > alone. Investigating old guality failures isn't even on that list. Our > team won't have time for it, and if we have bounty money to spend, it > will be spent on more important things. That's just the economic > reality, not a desire to disrespect the guality tests or anyone > associated with them. > > From my limited perspective, it seems like the guality tests are unique > within the test suite as a set of tests that everyone just expects to > have lots of failures. Is that healthy? Will it ever change? > > That said, it is clear that you feel the guality tests provide at least > some value in their present state, so we can continue to live with > things as they are. I'm just curious how others feel about the state of > these tests. I agree with Jakub that disabling the tests is not good. Just look at a random testcase that FAILs on powerpc but not on x86_64-linux for all optimization levels. You can literally "debug" this manually as the guality would - there may be ABI issues that make handling the case hard or there may be simple bugs (like a target reorg pass not properly caring for debug insns). Richard. > Thanks, > Bill > >> >> Jakub >> > >
On Mon, Mar 28, 2016 at 8:38 PM, Bill Schmidt <wschmidt@linux.vnet.ibm.com> wrote: > Hi, > > For a long time we've had hundreds of failing guality tests. These > failures don't seem to have any correlation with gdb functionality for > POWER, which is working fine. At this point the value of these tests to > us seems questionable. Fixing these is such low priority that it is > unlikely we will ever get around to it. In the meanwhile, the failures > simply clutter up our regression test reports. Thus I'd like to disable > them, and that's what this test does. > > Verified to remove hundreds of failure messages on > powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? > > Thanks, > Bill > > > 2016-03-28 Bill Schmidt <wschmidt@linux.vnet.ibm.com> > > * g++.dg/guality/guality.exp: Disable for powerpc*-linux*. > * gcc.dg/guality/guality.exp: Likewise. Thanks for everyone else's suggestions. As far as we understand, debugging quality on POWER is equivalent to other targets. There is an issue with PPC64 BE and AIX requiring an extra frame push when debugging is enabled, which will cause differences between code with debugging enabled and debugging disabled. THIS WILL NOT BE CHANGED. We have no plans to make code generation a slave to the testsuite. The testsuite is a tool, successful results from the testsuite is not a goal unto itself. This patch is okay. Thanks, David
On March 29, 2016 4:45:44 PM GMT+02:00, David Edelsohn <dje.gcc@gmail.com> wrote: >On Mon, Mar 28, 2016 at 8:38 PM, Bill Schmidt ><wschmidt@linux.vnet.ibm.com> wrote: >> Hi, >> >> For a long time we've had hundreds of failing guality tests. These >> failures don't seem to have any correlation with gdb functionality >for >> POWER, which is working fine. At this point the value of these tests >to >> us seems questionable. Fixing these is such low priority that it is >> unlikely we will ever get around to it. In the meanwhile, the >failures >> simply clutter up our regression test reports. Thus I'd like to >disable >> them, and that's what this test does. >> >> Verified to remove hundreds of failure messages on >> powerpc64le-unknown-linux-gnu. :) Is this ok for trunk? >> >> Thanks, >> Bill >> >> >> 2016-03-28 Bill Schmidt <wschmidt@linux.vnet.ibm.com> >> >> * g++.dg/guality/guality.exp: Disable for powerpc*-linux*. >> * gcc.dg/guality/guality.exp: Likewise. > >Thanks for everyone else's suggestions. > >As far as we understand, debugging quality on POWER is equivalent to >other targets. > >There is an issue with PPC64 BE and AIX requiring an extra frame push >when debugging is enabled, which will cause differences between code >with debugging enabled and debugging disabled. THIS WILL NOT BE >CHANGED. Guality does not check for this but guality tests are in essence debug info tests (using gdb). So definitely for those test cases failing debug quality is _not_ on par with x86 Linux. Richard. >We have no plans to make code generation a slave to the testsuite. >The testsuite is a tool, successful results from the testsuite is not >a goal unto itself. > >This patch is okay. > >Thanks, David
Hi Jakub, Thanks for the information; I really do appreciate it! On Tue, 2016-03-29 at 17:33 +0200, Jakub Jelinek wrote: > On Tue, Mar 29, 2016 at 08:19:39AM -0500, Bill Schmidt wrote: > > When you say that "the debug info quality is already pretty bad on > > powerpc*," do you mean that it is known to be bad, or simply that we > > have a lot of guality failures that may or may not indicate that the > > debug info is bad? I don't have experiential evidence of bad debug info > > that shows up during debugging sessions. Perhaps these are corner cases > > A lot of effort has been spent on x86_64/i?86 to improve the debug info > for optimized code, while far less effort has been spent on it for say > powerpc* or s390*. Many of the guality testcases are derived from > real-world programs, such as the Linux kernel or python or other packages > where the lack of QoI of debug info (or sometimes even wrong debug info) > caused some tool failures or has been a major obstackle to users so that > they couldn't debug something important. > And for evidence, we have e.g. in redhat.com bugzilla, significantly more > complains about debug info on powerpc*/s390* than on i?86/x86_64. This is good information. Unfortunately, this is the first time I've been made aware of it. If these bugs aren't posted to the FSF bugzilla, or mirrored to us, we are ignorant that there is even a problem. At the moment I'm not aware of any bug reports about debug info on powerpc*. Please pass these along to me as they arise. We can't prioritize what we can't see. > > before I joined this project, and from what others tell me, for at least > > a decade. As you suggest here, others have always told me just to > > ignore the existing guality failures. However, this can easily lead to > > Then you've been told wrong suggestions. You should just keep comparing > the results against older ones. That is what I meant. I apologize for the unclear language. > > > The other point, "it would be really very much desirable if > > anyone had time to analyze some of them and improve stuff," has to be > > answered by "apparently nobody does." I am currently tracking well over > > That would be a wrong answer, several man-years have been spent on analyzing > and improving those, by Alex, myself, Richi, various others. Again, this is good information to know about. But the "stuff" we were talking about was the failures on powerpc*, and I took what you said to mean that nobody was working on those. It sounds like you're saying that the community has spent time on debug improvements for optimized code on x86_64/i?86, but only for that target. Is that a fair statement? If so, it seems unsurprising that you would get more bug reports for the debug information on powerpc* and s/390. I'm not trying to be critical here. I'm trying to understand the value offered by these tests (which I do much better now, thanks), since we have to prioritize our work carefully for the resources that we have. Thanks, Bill
On Tue, Mar 29, 2016 at 12:01:20PM -0500, Bill Schmidt wrote: > Again, this is good information to know about. But the "stuff" we were > talking about was the failures on powerpc*, and I took what you said to > mean that nobody was working on those. It sounds like you're saying > that the community has spent time on debug improvements for optimized > code on x86_64/i?86, but only for that target. Is that a fair > statement? If so, it seems unsurprising that you would get more bug Well, most of the analysis has been done on x86_64/i?86. The bug fixes, DWARF enhancements etc. were then in various areas, if something has been improved through some GIMPLE change, then likely all targets benefited, if it was something at the RTL level (or var-tracking pass itself), then it really depends on the various properties of the machine descriptions, argument passing etc. I'm not saying it is possible to have all the guality tests pass at all optimization levels on all targets, sometimes the value of some variable is really lost through optimizations and can't be reconstructed in any way, sometimes it is too costly to track it, etc. In other cases we have yet to create new DWARF extensions, known stuff is e.g. debugging vectorized loops, what kind of user experience we want for users if single set of instructions handles multiple iterations of the loop? Do we want user to think he is seeing e.g. the first iteration, then the fifth one, then ninth etc., or provide enough info for the debuggers so that the user could find out he is in vectorized loop and explicitly request he is e.g. interested in the 3rd iteration instead of 1st? Then there will be certainly cases where even without adding any extensions one can just add some smarts to var-tracking, or change other GCC internals to handle some stuff better. Jakub
On Mar 29, 2016, at 7:45 AM, David Edelsohn <dje.gcc@gmail.com> wrote: > We have no plans to make code generation a slave to the testsuite. > The testsuite is a tool, successful results from the testsuite is not > a goal unto itself. > > This patch is okay. We look forward to the day when someone can find the time and energy and desire to make subsets of this work better and reenable those as they bring them back online. I view it as I do for turning off C++ testing on a PIC target, if no one wants to make it work nicely, then it is better to just turn it off. Anyone with the desire to make these tests work nicely will step forward and donate as they are able to. If someone would like that work done, you can edit up a TODO or projects file to describe the work you’d like done, and try and find someone that would like to do the work, or, just do the work yourself. If someone has the free time, and wants to tackle this project, merely step forward and let others know. This is how we make progress.
On Tue, Mar 29, 2016 at 11:34:17AM -0700, Mike Stump wrote: > On Mar 29, 2016, at 7:45 AM, David Edelsohn <dje.gcc@gmail.com> wrote: > > We have no plans to make code generation a slave to the testsuite. > > The testsuite is a tool, successful results from the testsuite is not > > a goal unto itself. > > > > This patch is okay. > > We look forward to the day when someone can find the time and energy and > desire to make subsets of this work better and reenable those as they > bring them back online. I view it as I do for turning off C++ testing on > a PIC target, if no one wants to make it work nicely, then it is better to > just turn it off. Anyone with the desire to make these tests work nicely > will step forward and donate as they are able to. If someone would like > that work done, you can edit up a TODO or projects file to describe the > work you’d like done, and try and find someone that would like to do the > work, or, just do the work yourself. If someone has the free time, and > wants to tackle this project, merely step forward and let others know. > This is how we make progress. The problem with the disabling is not in those tests that don't pass right now on whatever target you are testing on, but with any regressions in tests that pass right now but will not pass in half a year or year because of GCC changes; if the tests are disabled, nobody will notice that, one can't look at gcc-regressions or elsewhere to find out quickly where it regressed, etc. Jakub
On Tue, Mar 29, 2016 at 11:42 AM, Jakub Jelinek <jakub@redhat.com> wrote: > On Tue, Mar 29, 2016 at 11:34:17AM -0700, Mike Stump wrote: >> On Mar 29, 2016, at 7:45 AM, David Edelsohn <dje.gcc@gmail.com> wrote: >> > We have no plans to make code generation a slave to the testsuite. >> > The testsuite is a tool, successful results from the testsuite is not >> > a goal unto itself. >> > >> > This patch is okay. >> >> We look forward to the day when someone can find the time and energy and >> desire to make subsets of this work better and reenable those as they >> bring them back online. I view it as I do for turning off C++ testing on >> a PIC target, if no one wants to make it work nicely, then it is better to >> just turn it off. Anyone with the desire to make these tests work nicely >> will step forward and donate as they are able to. If someone would like >> that work done, you can edit up a TODO or projects file to describe the >> work you’d like done, and try and find someone that would like to do the >> work, or, just do the work yourself. If someone has the free time, and >> wants to tackle this project, merely step forward and let others know. >> This is how we make progress. > > The problem with the disabling is not in those tests that don't pass right > now on whatever target you are testing on, but with any regressions in tests > that pass right now but will not pass in half a year or year because of GCC > changes; if the tests are disabled, nobody will notice that, one can't look > at gcc-regressions or elsewhere to find out quickly where it regressed, etc. > > Jakub One issue with gcc.dg/guality/guality.exp is when there is an ICE regression during guality init, all sudden there is no failure in guality tests: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68545 Next time, when ICE is fixed, a bunch of guality failures show up.
On Tue, Mar 29, 2016 at 7:16 PM, Jakub Jelinek <jakub@redhat.com> wrote: > On Tue, Mar 29, 2016 at 12:01:20PM -0500, Bill Schmidt wrote: >> Again, this is good information to know about. But the "stuff" we were >> talking about was the failures on powerpc*, and I took what you said to >> mean that nobody was working on those. It sounds like you're saying >> that the community has spent time on debug improvements for optimized >> code on x86_64/i?86, but only for that target. Is that a fair >> statement? If so, it seems unsurprising that you would get more bug > > Well, most of the analysis has been done on x86_64/i?86. The bug fixes, > DWARF enhancements etc. were then in various areas, if something has been > improved through some GIMPLE change, then likely all targets benefited, > if it was something at the RTL level (or var-tracking pass itself), then > it really depends on the various properties of the machine descriptions, > argument passing etc. > I'm not saying it is possible to have all the guality tests pass at all > optimization levels on all targets, sometimes the value of some variable > is really lost through optimizations and can't be reconstructed in any way, > sometimes it is too costly to track it, etc. > In other cases we have yet to create new DWARF extensions, known stuff is > e.g. debugging vectorized loops, what kind of user experience we want for > users if single set of instructions handles multiple iterations of the loop? > Do we want user to think he is seeing e.g. the first iteration, then the > fifth one, then ninth etc., or provide enough info for the debuggers so that > the user could find out he is in vectorized loop and explicitly request > he is e.g. interested in the 3rd iteration instead of 1st? > Then there will be certainly cases where even without adding any extensions > one can just add some smarts to var-tracking, or change other GCC internals > to handle some stuff better. Yes, I think we _do_ need some dg-effective-target stuff for guality as some tests currently fail on arm (IIRC, I've only once did some non-x86 digging into guality fails) because of ABI issues that make a middle-end debuginfo fix incomplete (or impossible, don't remember). For powerpc somebody needs to look at a few regressions towards x86_64 and see if there's a similar pattern - adding arch specific xfails (or adding effective targets) is a good way to make progress as well. Hell, even slapping a xfail powerpc*-*-* on all current ppc FAILs would be better than simply disabling all of guality for ppc. Richard. > Jakub
Richard Biener <richard.guenther@gmail.com> writes: > Hell, even slapping a xfail powerpc*-*-* on all current ppc FAILs > would be better > than simply disabling all of guality for ppc. FWIW, I agree. While working on the debug early project, I found at least two legitimate bugs affecting all architectures with guality tests on ppc64. Aldy
Thanks to all for the helpful explanations. We plan to leave things as they are. I hope someday we can make some time to do some basic investigations here. Bill On Fri, 2016-04-01 at 00:09 -0500, Aldy Hernandez wrote: > Richard Biener <richard.guenther@gmail.com> writes: > > > Hell, even slapping a xfail powerpc*-*-* on all current ppc FAILs > > would be better > > than simply disabling all of guality for ppc. > > FWIW, I agree. While working on the debug early project, I found at > least two legitimate bugs affecting all architectures with guality tests > on ppc64. > > Aldy >
Index: gcc/testsuite/g++.dg/guality/guality.exp =================================================================== --- gcc/testsuite/g++.dg/guality/guality.exp (revision 234476) +++ gcc/testsuite/g++.dg/guality/guality.exp (working copy) @@ -13,6 +13,11 @@ if { [istarget "powerpc-ibm-aix*"] } { return } +if { [istarget "powerpc*-linux*"] } { + set torture_execute_xfail "powerpc*-linux*" + return +} + proc check_guality {args} { # Don't count check_guality as PASS, or FAIL etc., that would make # the total PASS count dependent on how many parallel runtest invocations Index: gcc/testsuite/gcc.dg/guality/guality.exp =================================================================== --- gcc/testsuite/gcc.dg/guality/guality.exp (revision 234476) +++ gcc/testsuite/gcc.dg/guality/guality.exp (working copy) @@ -13,6 +13,11 @@ if { [istarget "powerpc-ibm-aix*"] } { return } +if { [istarget "powerpc*-linux*"] } { + set torture_execute_xfail "powerpc*-linux*" + return +} + proc check_guality {args} { # Don't count check_guality as PASS, or FAIL etc., that would make # the total PASS count dependent on how many parallel runtest invocations