[{"id":1775665,"web_url":"http://patchwork.ozlabs.org/comment/1775665/","msgid":"","list_archive_url":null,"date":"2017-09-26T16:16:06","subject":"ACK: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","submitter":{"id":2898,"url":"http://patchwork.ozlabs.org/api/people/2898/","name":"Stefan Bader","email":"stefan.bader@canonical.com"},"content":"On 20.09.2017 11:32, Joseph Salisbury wrote:\n> From: Srikar Dronamraju \n> \n> BugLink: http://bugs.launchpad.net/bugs/1713576\n> \n> If the child domain prefers tasks to go siblings, the local group could\n> end up pulling tasks to itself even if the local group is almost equally\n> loaded as the source group.\n> \n> Lets assume a 4 core,smt==2 machine running 5 thread ebizzy workload.\n> Everytime, local group has capacity and source group has atleast 2 threads,\n> local group tries to pull the task. This causes the threads to constantly\n> move between different cores. This is even more profound if the cores have\n> more threads, like in Power 8, smt 8 mode.\n> \n> Fix this by only allowing local group to pull a task, if the source group\n> has more number of tasks than the local group.\n> \n> Here are the relevant perf stat numbers of a 22 core,smt 8 Power 8 machine.\n> \n> Without patch:\n> Performance counter stats for 'ebizzy -t 22 -S 100' (5 runs):\n> \n> 1,440 context-switches # 0.001 K/sec ( +- 1.26% )\n> 366 cpu-migrations # 0.000 K/sec ( +- 5.58% )\n> 3,933 page-faults # 0.002 K/sec ( +- 11.08% )\n> \n> Performance counter stats for 'ebizzy -t 48 -S 100' (5 runs):\n> \n> 6,287 context-switches # 0.001 K/sec ( +- 3.65% )\n> 3,776 cpu-migrations # 0.001 K/sec ( +- 4.84% )\n> 5,702 page-faults # 0.001 K/sec ( +- 9.36% )\n> \n> Performance counter stats for 'ebizzy -t 96 -S 100' (5 runs):\n> \n> 8,776 context-switches # 0.001 K/sec ( +- 0.73% )\n> 2,790 cpu-migrations # 0.000 K/sec ( +- 0.98% )\n> 10,540 page-faults # 0.001 K/sec ( +- 3.12% )\n> \n> With patch:\n> \n> Performance counter stats for 'ebizzy -t 22 -S 100' (5 runs):\n> \n> 1,133 context-switches # 0.001 K/sec ( +- 4.72% )\n> 123 cpu-migrations # 0.000 K/sec ( +- 3.42% )\n> 3,858 page-faults # 0.002 K/sec ( +- 8.52% )\n> \n> Performance counter stats for 'ebizzy -t 48 -S 100' (5 runs):\n> \n> 2,169 context-switches # 0.000 K/sec ( +- 6.19% )\n> 189 cpu-migrations # 0.000 K/sec ( +- 12.75% )\n> 5,917 page-faults # 0.001 K/sec ( +- 8.09% )\n> \n> Performance counter stats for 'ebizzy -t 96 -S 100' (5 runs):\n> \n> 5,333 context-switches # 0.001 K/sec ( +- 5.91% )\n> 506 cpu-migrations # 0.000 K/sec ( +- 3.35% )\n> 10,792 page-faults # 0.001 K/sec ( +- 7.75% )\n> \n> Which show that in these workloads CPU migrations get reduced significantly.\n> \n> Signed-off-by: Srikar Dronamraju \n> Signed-off-by: Peter Zijlstra (Intel) \n> Cc: Linus Torvalds \n> Cc: Mike Galbraith \n> Cc: Peter Zijlstra \n> Cc: Thomas Gleixner \n> Cc: Vincent Guittot \n> Link: http://lkml.kernel.org/r/1490205470-10249-1-git-send-email-srikar@linux.vnet.ibm.com\n> Signed-off-by: Ingo Molnar \n> (cherry picked from commit 05b40e057734811ce452344fb3690d09965a7b6a)\n> Signed-off-by: Joseph Salisbury \nAcked-by: Stefan Bader \n\n> ---\n> kernel/sched/fair.c | 7 ++++---\n> 1 file changed, 4 insertions(+), 3 deletions(-)\n> \n> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c\n> index 6559d19..a6dd010 100644\n> --- a/kernel/sched/fair.c\n> +++ b/kernel/sched/fair.c\n> @@ -7496,6 +7496,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> {\n> \tstruct sched_domain *child = env->sd->child;\n> \tstruct sched_group *sg = env->sd->groups;\n> +\tstruct sg_lb_stats *local = &sds->local_stat;\n> \tstruct sg_lb_stats tmp_sgs;\n> \tint load_idx, prefer_sibling = 0;\n> \tbool overload = false;\n> @@ -7512,7 +7513,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> \t\tlocal_group = cpumask_test_cpu(env->dst_cpu, sched_group_cpus(sg));\n> \t\tif (local_group) {\n> \t\t\tsds->local = sg;\n> -\t\t\tsgs = &sds->local_stat;\n> +\t\t\tsgs = local;\n> \n> \t\t\tif (env->idle != CPU_NEWLY_IDLE ||\n> \t\t\t time_after_eq(jiffies, sg->sgc->next_update))\n> @@ -7536,8 +7537,8 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> \t\t * the tasks on the system).\n> \t\t */\n> \t\tif (prefer_sibling && sds->local &&\n> -\t\t group_has_capacity(env, &sds->local_stat) &&\n> -\t\t (sgs->sum_nr_running > 1)) {\n> +\t\t group_has_capacity(env, local) &&\n> +\t\t (sgs->sum_nr_running > local->sum_nr_running + 1)) {\n> \t\t\tsgs->group_no_capacity = 1;\n> \t\t\tsgs->group_type = group_classify(sg, sgs);\n> \t\t}\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com\n\t(client-ip=91.189.94.19; helo=huckleberry.canonical.com;\n\tenvelope-from=kernel-team-bounces@lists.ubuntu.com;\n\treceiver=)","Received":["from huckleberry.canonical.com (huckleberry.canonical.com\n\t[91.189.94.19])\n\tby ozlabs.org (Postfix) with ESMTP id 3y1mKW4Blsz9t5C;\n\tWed, 27 Sep 2017 02:16:15 +1000 (AEST)","from localhost ([127.0.0.1] helo=huckleberry.canonical.com)\n\tby huckleberry.canonical.com with esmtp (Exim 4.86_2)\n\t(envelope-from )\n\tid 1dwsWo-0008Tq-B8; Tue, 26 Sep 2017 16:16:10 +0000","from youngberry.canonical.com ([91.189.89.112])\n\tby huckleberry.canonical.com with esmtps\n\t(TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128)\n\t(Exim 4.86_2) (envelope-from )\n\tid 1dwsWm-0008Ti-Qd\n\tfor kernel-team@lists.ubuntu.com; Tue, 26 Sep 2017 16:16:08 +0000","from rrcs-69-193-151-51.nyc.biz.rr.com ([69.193.151.51]\n\thelo=[10.155.52.21]) by youngberry.canonical.com with esmtpsa\n\t(TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.76) (envelope-from )\n\tid 1dwsWm-0006a7-H2\n\tfor kernel-team@lists.ubuntu.com; Tue, 26 Sep 2017 16:16:08 +0000"],"Subject":"ACK: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","To":"kernel-team@lists.ubuntu.com","References":"\n\t<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","From":"Stefan Bader ","Message-ID":"","Date":"Tue, 26 Sep 2017 12:16:06 -0400","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","X-BeenThere":"kernel-team@lists.ubuntu.com","X-Mailman-Version":"2.1.20","Precedence":"list","List-Id":"Kernel team discussions ","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"multipart/mixed;\n\tboundary=\"===============5310202091260479742==\"","Errors-To":"kernel-team-bounces@lists.ubuntu.com","Sender":"\"kernel-team\" "}},{"id":1779611,"web_url":"http://patchwork.ozlabs.org/comment/1779611/","msgid":"<42fd6d98-2fa9-4379-4fa0-992b49ee4fa2@canonical.com>","list_archive_url":null,"date":"2017-10-04T10:12:37","subject":"ACK: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","submitter":{"id":2900,"url":"http://patchwork.ozlabs.org/api/people/2900/","name":"Colin Ian King","email":"colin.king@canonical.com"},"content":"On 20/09/17 16:32, Joseph Salisbury wrote:\n> From: Srikar Dronamraju \n> \n> BugLink: http://bugs.launchpad.net/bugs/1713576\n> \n> If the child domain prefers tasks to go siblings, the local group could\n> end up pulling tasks to itself even if the local group is almost equally\n> loaded as the source group.\n> \n> Lets assume a 4 core,smt==2 machine running 5 thread ebizzy workload.\n> Everytime, local group has capacity and source group has atleast 2 threads,\n> local group tries to pull the task. This causes the threads to constantly\n> move between different cores. This is even more profound if the cores have\n> more threads, like in Power 8, smt 8 mode.\n> \n> Fix this by only allowing local group to pull a task, if the source group\n> has more number of tasks than the local group.\n> \n> Here are the relevant perf stat numbers of a 22 core,smt 8 Power 8 machine.\n> \n> Without patch:\n> Performance counter stats for 'ebizzy -t 22 -S 100' (5 runs):\n> \n> 1,440 context-switches # 0.001 K/sec ( +- 1.26% )\n> 366 cpu-migrations # 0.000 K/sec ( +- 5.58% )\n> 3,933 page-faults # 0.002 K/sec ( +- 11.08% )\n> \n> Performance counter stats for 'ebizzy -t 48 -S 100' (5 runs):\n> \n> 6,287 context-switches # 0.001 K/sec ( +- 3.65% )\n> 3,776 cpu-migrations # 0.001 K/sec ( +- 4.84% )\n> 5,702 page-faults # 0.001 K/sec ( +- 9.36% )\n> \n> Performance counter stats for 'ebizzy -t 96 -S 100' (5 runs):\n> \n> 8,776 context-switches # 0.001 K/sec ( +- 0.73% )\n> 2,790 cpu-migrations # 0.000 K/sec ( +- 0.98% )\n> 10,540 page-faults # 0.001 K/sec ( +- 3.12% )\n> \n> With patch:\n> \n> Performance counter stats for 'ebizzy -t 22 -S 100' (5 runs):\n> \n> 1,133 context-switches # 0.001 K/sec ( +- 4.72% )\n> 123 cpu-migrations # 0.000 K/sec ( +- 3.42% )\n> 3,858 page-faults # 0.002 K/sec ( +- 8.52% )\n> \n> Performance counter stats for 'ebizzy -t 48 -S 100' (5 runs):\n> \n> 2,169 context-switches # 0.000 K/sec ( +- 6.19% )\n> 189 cpu-migrations # 0.000 K/sec ( +- 12.75% )\n> 5,917 page-faults # 0.001 K/sec ( +- 8.09% )\n> \n> Performance counter stats for 'ebizzy -t 96 -S 100' (5 runs):\n> \n> 5,333 context-switches # 0.001 K/sec ( +- 5.91% )\n> 506 cpu-migrations # 0.000 K/sec ( +- 3.35% )\n> 10,792 page-faults # 0.001 K/sec ( +- 7.75% )\n> \n> Which show that in these workloads CPU migrations get reduced significantly.\n> \n> Signed-off-by: Srikar Dronamraju \n> Signed-off-by: Peter Zijlstra (Intel) \n> Cc: Linus Torvalds \n> Cc: Mike Galbraith \n> Cc: Peter Zijlstra \n> Cc: Thomas Gleixner \n> Cc: Vincent Guittot \n> Link: http://lkml.kernel.org/r/1490205470-10249-1-git-send-email-srikar@linux.vnet.ibm.com\n> Signed-off-by: Ingo Molnar \n> (cherry picked from commit 05b40e057734811ce452344fb3690d09965a7b6a)\n> Signed-off-by: Joseph Salisbury \n> ---\n> kernel/sched/fair.c | 7 ++++---\n> 1 file changed, 4 insertions(+), 3 deletions(-)\n> \n> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c\n> index 6559d19..a6dd010 100644\n> --- a/kernel/sched/fair.c\n> +++ b/kernel/sched/fair.c\n> @@ -7496,6 +7496,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> {\n> \tstruct sched_domain *child = env->sd->child;\n> \tstruct sched_group *sg = env->sd->groups;\n> +\tstruct sg_lb_stats *local = &sds->local_stat;\n> \tstruct sg_lb_stats tmp_sgs;\n> \tint load_idx, prefer_sibling = 0;\n> \tbool overload = false;\n> @@ -7512,7 +7513,7 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> \t\tlocal_group = cpumask_test_cpu(env->dst_cpu, sched_group_cpus(sg));\n> \t\tif (local_group) {\n> \t\t\tsds->local = sg;\n> -\t\t\tsgs = &sds->local_stat;\n> +\t\t\tsgs = local;\n> \n> \t\t\tif (env->idle != CPU_NEWLY_IDLE ||\n> \t\t\t time_after_eq(jiffies, sg->sgc->next_update))\n> @@ -7536,8 +7537,8 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd\n> \t\t * the tasks on the system).\n> \t\t */\n> \t\tif (prefer_sibling && sds->local &&\n> -\t\t group_has_capacity(env, &sds->local_stat) &&\n> -\t\t (sgs->sum_nr_running > 1)) {\n> +\t\t group_has_capacity(env, local) &&\n> +\t\t (sgs->sum_nr_running > local->sum_nr_running + 1)) {\n> \t\t\tsgs->group_no_capacity = 1;\n> \t\t\tsgs->group_type = group_classify(sg, sgs);\n> \t\t}\n> \nClean cherry pick and positive test results.\n\nAcked-by: Colin Ian King ","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com\n\t(client-ip=91.189.94.19; helo=huckleberry.canonical.com;\n\tenvelope-from=kernel-team-bounces@lists.ubuntu.com;\n\treceiver=)","Received":["from huckleberry.canonical.com (huckleberry.canonical.com\n\t[91.189.94.19])\n\tby ozlabs.org (Postfix) with ESMTP id 3y6WtN1CMGz9t2W;\n\tWed, 4 Oct 2017 21:12:44 +1100 (AEDT)","from localhost ([127.0.0.1] helo=huckleberry.canonical.com)\n\tby huckleberry.canonical.com with esmtp (Exim 4.86_2)\n\t(envelope-from )\n\tid 1dzgfQ-0003N2-2O; Wed, 04 Oct 2017 10:12:40 +0000","from youngberry.canonical.com ([91.189.89.112])\n\tby huckleberry.canonical.com with esmtps\n\t(TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128)\n\t(Exim 4.86_2) (envelope-from )\n\tid 1dzgfO-0003Ly-6N\n\tfor kernel-team@lists.ubuntu.com; Wed, 04 Oct 2017 10:12:38 +0000","from 1.general.cking.uk.vpn ([10.172.193.212])\n\tby youngberry.canonical.com with esmtpsa\n\t(TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.76) (envelope-from )\n\tid 1dzgfN-00029u-TX; Wed, 04 Oct 2017 10:12:37 +0000"],"Subject":"ACK: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","To":"kernel-team@lists.ubuntu.com","References":"\n\t<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","From":"Colin Ian King ","Message-ID":"<42fd6d98-2fa9-4379-4fa0-992b49ee4fa2@canonical.com>","Date":"Wed, 4 Oct 2017 11:12:37 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.8.0","MIME-Version":"1.0","In-Reply-To":"<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","X-BeenThere":"kernel-team@lists.ubuntu.com","X-Mailman-Version":"2.1.20","Precedence":"list","List-Id":"Kernel team discussions ","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"kernel-team-bounces@lists.ubuntu.com","Sender":"\"kernel-team\" "}},{"id":1782880,"web_url":"http://patchwork.ozlabs.org/comment/1782880/","msgid":"<20171009141521.dn227oqzxh5ffwvg@calabresa>","list_archive_url":null,"date":"2017-10-09T14:15:22","subject":"APPLIED: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","submitter":{"id":70574,"url":"http://patchwork.ozlabs.org/api/people/70574/","name":"Thadeu Lima de Souza Cascardo","email":"cascardo@canonical.com"},"content":"Applied to zesty master-next branch.\n\nThanks.\nCascardo.\n\nApplied-to: zesty/master-next","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com\n\t(client-ip=91.189.94.19; helo=huckleberry.canonical.com;\n\tenvelope-from=kernel-team-bounces@lists.ubuntu.com;\n\treceiver=)","Received":["from huckleberry.canonical.com (huckleberry.canonical.com\n\t[91.189.94.19])\n\tby ozlabs.org (Postfix) with ESMTP id 3y9j2C6kCXz9tXF;\n\tTue, 10 Oct 2017 01:15:31 +1100 (AEDT)","from localhost ([127.0.0.1] helo=huckleberry.canonical.com)\n\tby huckleberry.canonical.com with esmtp (Exim 4.86_2)\n\t(envelope-from )\n\tid 1e1Yq8-0003cG-4j; Mon, 09 Oct 2017 14:15:28 +0000","from youngberry.canonical.com ([91.189.89.112])\n\tby huckleberry.canonical.com with esmtps\n\t(TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128)\n\t(Exim 4.86_2) (envelope-from )\n\tid 1e1Yq7-0003c9-0t\n\tfor kernel-team@lists.ubuntu.com; Mon, 09 Oct 2017 14:15:27 +0000","from [177.94.55.56] (helo=calabresa)\n\tby youngberry.canonical.com with esmtpsa\n\t(TLS1.0:RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.76) (envelope-from )\n\tid 1e1Yq6-0008Vh-Ba; Mon, 09 Oct 2017 14:15:26 +0000"],"Date":"Mon, 9 Oct 2017 11:15:22 -0300","From":"Thadeu Lima de Souza Cascardo ","To":"Joseph Salisbury ","Subject":"APPLIED: [SRU][Zesty][PATCH 1/1] sched/fair: Prefer sibiling only if\n\tlocal group is under-utilized","Message-ID":"<20171009141521.dn227oqzxh5ffwvg@calabresa>","References":"\n\t<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<3c194524dadc3bda6a04632e71f68e0fe6ac72a0.1505920769.git.joseph.salisbury@canonical.com>","User-Agent":"NeoMutt/20170113 (1.7.2)","X-BeenThere":"kernel-team@lists.ubuntu.com","X-Mailman-Version":"2.1.20","Precedence":"list","List-Id":"Kernel team discussions ","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"kernel-team@lists.ubuntu.com","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"kernel-team-bounces@lists.ubuntu.com","Sender":"\"kernel-team\" "}}]