diff mbox series

gcc/testsuite: Scale down long-running tree-prof.exp tests on slow targets

Message ID 699ee787-c7da-1cac-3be1-ec2fb6ffc88e@codesourcery.com
State New
Headers show
Series gcc/testsuite: Scale down long-running tree-prof.exp tests on slow targets | expand

Commit Message

Sandra Loosemore July 20, 2020, 4:47 a.m. UTC 
I was looking at some timeout failures in nios2-linux-gnu test results 
and found several tree-prof.exp tests were doing what appears to be an 
excessive number of iterations (350 million?).  Even though this is 
hardware and not a simulator, I thought it would be reasonable to tell 
the test harness to treat it like a simulator, and make these tests pay 
attention to the is_simulator flag to run fewer iterations, as a number 
of other test cases do to scale down long-running tests.

I somewhat randomly chose to reduce the counts by a factor of 100 so the 
longest-running one takes just over 10 seconds on this target.  But, the 
original numbers seem pretty random to me as well.  Is there actually 
any benefit to running more iterations even on a fast target?

-Sandra

Comments

Richard Biener July 20, 2020, 8:15 a.m. UTC | #1 
On Mon, Jul 20, 2020 at 6:48 AM Sandra Loosemore
<sandra@codesourcery.com> wrote:
>
> I was looking at some timeout failures in nios2-linux-gnu test results
> and found several tree-prof.exp tests were doing what appears to be an
> excessive number of iterations (350 million?).  Even though this is
> hardware and not a simulator, I thought it would be reasonable to tell
> the test harness to treat it like a simulator, and make these tests pay
> attention to the is_simulator flag to run fewer iterations, as a number
> of other test cases do to scale down long-running tests.
>
> I somewhat randomly chose to reduce the counts by a factor of 100 so the
> longest-running one takes just over 10 seconds on this target.  But, the
> original numbers seem pretty random to me as well.  Is there actually
> any benefit to running more iterations even on a fast target?

I think at least parts of tree-prof.exp exercises sample-based profiling
which might require more iterations.  For example cold_partition_label.c
was changed by

commit f63ba78ce6d50bf627dd18018179eb03bf89716f
Author: Andi Kleen <ak@linux.intel.com>
Date:   Thu Jul 14 02:14:56 2016 +0000

    Some fixes for profile test cases for autofdo

    This fixes some basic issues with the profile test cases with autofdo.

    - Disable checking for value transformations that autofdo does not
      support.
    - Disable checking for fixed hit counts which autofdo does not support
    - Enable dumping of afdo log file and check right log file.
    - Increase run time of test cases to 1M iterations because autofdo needs
      a few samples to make sense of a program. The test case don't run
      noticeable slower with that.

    There are still failures unfortunately, especially the indirect call
    transformations do not trigger because autofdo thinks they are not hot.
    This can be addressed later.

so the change to a larger number of iterations was intended.  Maybe
we can arrange to pass -DFOR_AUTOFDO_TESTING for the
autofdo compiles and gate the larger number of iterations on that
(most targets do not support autofdo and to not run that mode)?

Richard.

> -Sandra
Sandra Loosemore July 20, 2020, 11:14 p.m. UTC | #2 
On 7/20/20 2:15 AM, Richard Biener wrote:

> I think at least parts of tree-prof.exp exercises sample-based profiling
> which might require more iterations.  For example cold_partition_label.c
> was changed by
> 
> commit f63ba78ce6d50bf627dd18018179eb03bf89716f
> Author: Andi Kleen <ak@linux.intel.com>
> Date:   Thu Jul 14 02:14:56 2016 +0000
> 
>      Some fixes for profile test cases for autofdo
> 
>      This fixes some basic issues with the profile test cases with autofdo.
> 
>      - Disable checking for value transformations that autofdo does not
>        support.
>      - Disable checking for fixed hit counts which autofdo does not support
>      - Enable dumping of afdo log file and check right log file.
>      - Increase run time of test cases to 1M iterations because autofdo needs
>        a few samples to make sense of a program. The test case don't run
>        noticeable slower with that.
> 
>      There are still failures unfortunately, especially the indirect call
>      transformations do not trigger because autofdo thinks they are not hot.
>      This can be addressed later.
> 
> so the change to a larger number of iterations was intended.  Maybe
> we can arrange to pass -DFOR_AUTOFDO_TESTING for the
> autofdo compiles and gate the larger number of iterations on that
> (most targets do not support autofdo and to not run that mode)?

Something like the attached updated patch?  Unfortunately I'm not set up 
to test that this actually works on an autofdo target, maybe somebody 
else could give it a try?

-Sandra
Richard Biener July 21, 2020, 7:37 a.m. UTC | #3 
On Tue, Jul 21, 2020 at 1:14 AM Sandra Loosemore
<sandra@codesourcery.com> wrote:
>
> On 7/20/20 2:15 AM, Richard Biener wrote:
>
> > I think at least parts of tree-prof.exp exercises sample-based profiling
> > which might require more iterations.  For example cold_partition_label.c
> > was changed by
> >
> > commit f63ba78ce6d50bf627dd18018179eb03bf89716f
> > Author: Andi Kleen <ak@linux.intel.com>
> > Date:   Thu Jul 14 02:14:56 2016 +0000
> >
> >      Some fixes for profile test cases for autofdo
> >
> >      This fixes some basic issues with the profile test cases with autofdo.
> >
> >      - Disable checking for value transformations that autofdo does not
> >        support.
> >      - Disable checking for fixed hit counts which autofdo does not support
> >      - Enable dumping of afdo log file and check right log file.
> >      - Increase run time of test cases to 1M iterations because autofdo needs
> >        a few samples to make sense of a program. The test case don't run
> >        noticeable slower with that.
> >
> >      There are still failures unfortunately, especially the indirect call
> >      transformations do not trigger because autofdo thinks they are not hot.
> >      This can be addressed later.
> >
> > so the change to a larger number of iterations was intended.  Maybe
> > we can arrange to pass -DFOR_AUTOFDO_TESTING for the
> > autofdo compiles and gate the larger number of iterations on that
> > (most targets do not support autofdo and to not run that mode)?
>
> Something like the attached updated patch?  Unfortunately I'm not set up
> to test that this actually works on an autofdo target, maybe somebody
> else could give it a try?

Looks good to me, thus OK without further tests.

Thanks,
Richard.

> -Sandra
diff mbox series

Patch

commit 8163c5309d9663d75dbeadf7e04ab463c1c07f5d
Author: Sandra Loosemore <sandra@codesourcery.com>
Date:   Sun Jul 19 21:33:41 2020 -0700

    Scale down long-running tree-prof.exp tests on slow targets.
    
    2020-07-19  Sandra Loosemore  <sandra@codesourcery.com>
    
    	gcc/testsuite/
    	* gcc.dg/tree-prof/cold_partition_label.c: Scale down for slow
    	target.
    	* gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c: Likewise.
    	* gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c: Likewise.
    	* gcc.dg/tree-prof/indir-call-prof-topn.c: Likewise.
    	* gcc.dg/tree-prof/section-attr-1.c: Likewise.
    	* gcc.dg/tree-prof/section-attr-2.c: Likewise.
    	* gcc.dg/tree-prof/section-attr-3.c: Likewise.

diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
index 7e8dc55..0230012 100644
--- a/gcc/testsuite/ChangeLog
+++ b/gcc/testsuite/ChangeLog
@@ -1,3 +1,14 @@ 
+2020-07-19  Sandra Loosemore  <sandra@codesourcery.com>
+
+	* gcc.dg/tree-prof/cold_partition_label.c: Scale down for slow
+	target.
+	* gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c: Likewise.
+	* gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c: Likewise.
+	* gcc.dg/tree-prof/indir-call-prof-topn.c: Likewise.
+	* gcc.dg/tree-prof/section-attr-1.c: Likewise.
+	* gcc.dg/tree-prof/section-attr-2.c: Likewise.
+	* gcc.dg/tree-prof/section-attr-3.c: Likewise.
+
 2020-07-19  H.J. Lu  <hjl.tools@gmail.com>
 
 	PR target/95973
diff --git a/gcc/testsuite/gcc.dg/tree-prof/cold_partition_label.c b/gcc/testsuite/gcc.dg/tree-prof/cold_partition_label.c
index 450308d..ce7ff0a 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/cold_partition_label.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/cold_partition_label.c
@@ -2,6 +2,13 @@ 
    gets a label.  */
 /* { dg-require-effective-target freorder } */
 /* { dg-options "-O2 -freorder-blocks-and-partition -save-temps -fdump-tree-optimized" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 10000
+#else
+#define MAXITER 1000000
+#endif
 
 #define SIZE 10000
 
@@ -32,7 +39,7 @@  main (int argc, char *argv[])
   int i;
   buf_hot =  "hello";
   buf_cold = "world";
-  for (i = 0; i < 1000000; i++)
+  for (i = 0; i < MAXITER; i++)
     foo (argc);
   return 0;
 }
diff --git a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c
index a13b08c..0bbe3f2 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-1.c
@@ -2,6 +2,13 @@ 
 /* { dg-additional-sources "crossmodule-indir-call-topn-1a.c" } */
 /* { dg-require-profiling "-fprofile-generate" } */
 /* { dg-options "-O2 -flto -DDOJOB=1 -fdump-ipa-profile_estimate" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 3500000
+#else
+#define MAXITER 350000000
+#endif
 
 #include <stdio.h>
 
@@ -22,7 +29,7 @@  main()
 
   x = one (3);
 
-  for (i = 0; i < 350000000; i++)
+  for (i = 0; i < MAXITER; i++)
     {
       x = (*p) (3);
       p = table[x];
diff --git a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c
index 9b996fc..a66b1ee 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/crossmodule-indir-call-topn-2.c
@@ -2,6 +2,13 @@ 
 /* { dg-additional-sources "crossmodule-indir-call-topn-1a.c" } */
 /* { dg-require-profiling "-fprofile-generate" } */
 /* { dg-options "-O2 -flto -DDOJOB=1 -fdump-ipa-profile_estimate" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 3500000
+#else
+#define MAXITER 350000000
+#endif
 
 #include <stdio.h>
 
@@ -21,7 +28,7 @@  int foo ()
 
   x = one (3);
 
-  for (i = 0; i < 350000000; i++)
+  for (i = 0; i < MAXITER; i++)
     {
       x = (*p) (3);
       p = table[x];
diff --git a/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof-topn.c b/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof-topn.c
index 063996c..24416bc 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof-topn.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/indir-call-prof-topn.c
@@ -1,5 +1,12 @@ 
 /* { dg-require-profiling "-fprofile-generate" } */
 /* { dg-options "-O2 -fdump-ipa-profile_estimate" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 3500000
+#else
+#define MAXITER 350000000
+#endif
 
 #include <stdio.h>
 
@@ -26,7 +33,7 @@  main()
 
   one (3);
 
-  for (i = 0; i < 350000000; i++)
+  for (i = 0; i < MAXITER; i++)
     {
       x = (*p) (3);
       p = table[x];
diff --git a/gcc/testsuite/gcc.dg/tree-prof/section-attr-1.c b/gcc/testsuite/gcc.dg/tree-prof/section-attr-1.c
index 89ecc1c..15ac3d9 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/section-attr-1.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/section-attr-1.c
@@ -2,6 +2,13 @@ 
    all later functions from being partitioned into hot and cold blocks.  */
 /* { dg-require-effective-target freorder } */
 /* { dg-options "-O2 -fno-profile-reorder-functions -freorder-blocks-and-partition -save-temps" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 10000
+#else
+#define MAXITER 1000000
+#endif
 
 #define SIZE 10000
 
@@ -24,7 +31,7 @@  main (int argc, char *argv[])
   int i;
   buf_hot =  "hello";
   buf_cold = "world";
-  for (i = 0; i < 1000000; i++)
+  for (i = 0; i < MAXITER; i++)
     foo (argc);
   return 0;
 }
diff --git a/gcc/testsuite/gcc.dg/tree-prof/section-attr-2.c b/gcc/testsuite/gcc.dg/tree-prof/section-attr-2.c
index b856457..1b44fd6 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/section-attr-2.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/section-attr-2.c
@@ -3,6 +3,13 @@ 
    blocks.  */
 /* { dg-require-effective-target freorder } */
 /* { dg-options "-O2 -fno-profile-reorder-functions -freorder-blocks-and-partition -save-temps" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 10000
+#else
+#define MAXITER 1000000
+#endif
 
 #define SIZE 10000
 
@@ -20,7 +27,7 @@  main (int argc, char *argv[])
   int i;
   buf_hot =  "hello";
   buf_cold = "world";
-  for (i = 0; i < 1000000; i++)
+  for (i = 0; i < MAXITER; i++)
     foo (argc);
   return 0;
 }
diff --git a/gcc/testsuite/gcc.dg/tree-prof/section-attr-3.c b/gcc/testsuite/gcc.dg/tree-prof/section-attr-3.c
index 8d85cf3..2e347ef 100644
--- a/gcc/testsuite/gcc.dg/tree-prof/section-attr-3.c
+++ b/gcc/testsuite/gcc.dg/tree-prof/section-attr-3.c
@@ -3,6 +3,13 @@ 
    blocks.  */
 /* { dg-require-effective-target freorder } */
 /* { dg-options "-O2 -fno-profile-reorder-functions -freorder-blocks-and-partition -save-temps" } */
+/* { dg-additional-options "-DSLOW_TARGET" { target simulator } } */
+
+#ifdef SLOW_TARGET
+#define MAXITER 10000
+#else
+#define MAXITER 1000000
+#endif
 
 #define SIZE 10000
 
@@ -24,7 +31,7 @@  main (int argc, char *argv[])
   int i;
   buf_hot =  "hello";
   buf_cold = "world";
-  for (i = 0; i < 1000000; i++)
+  for (i = 0; i < MAXITER; i++)
     foo (argc);
   return 0;
 }