From patchwork Thu Nov 23 19:41:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 840885 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yjV8M54dlz9ryv for ; Fri, 24 Nov 2017 06:42:11 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 11AB2ABC; Thu, 23 Nov 2017 19:42:08 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 71C8EAB9 for ; Thu, 23 Nov 2017 19:42:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 773181E1 for ; Thu, 23 Nov 2017 19:42:05 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CB9EBC0587DD; Thu, 23 Nov 2017 19:42:04 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6E5945D6A5; Thu, 23 Nov 2017 19:42:02 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, aserdean@ovn.org, i.maximets@samsung.com, billy.o.mahony@intel.com, ian.stokes@intel.com Date: Thu, 23 Nov 2017 19:41:55 +0000 Message-Id: <1511466117-27191-1-git-send-email-ktraynor@redhat.com> In-Reply-To: <1511375946-28054-1-git-send-email-ktraynor@redhat.com> References: <1511375946-28054-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 23 Nov 2017 19:42:05 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 1/3] dpif-netdev: Add port/queue tiebreaker to rxq_cycle_sort. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org rxq_cycle_sort is used to compare rx queues by their measured number of cycles. In the event that they are equal, 0 could be returned. However, it is observed that returning 0 results in a different sort order on Windows/Linux. This is ok in practice but it causes a unit test failure for "1007: PMD - pmd-cpu-mask/distribution of rx queues" when running on different OS's. In order to have a consistent sort result across multiple OS's, introduce a tiebreaker of port/queue. Fixes: 655856ef39b9 ("dpif-netdev: Change rxq_scheduling to use rxq processing cycles.") Reported-by: Alin Gabriel Serdean Tested-by: Alin Gabriel Serdean Co-authored-by: Ilya Maximets Signed-off-by: Ilya Maximets Signed-off-by: Kevin Traynor --- V4: Added patch into series after Alin tested it as RFC lib/dpif-netdev.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 0a62630..f5cdd92 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -3464,8 +3464,17 @@ rxq_cycle_sort(const void *a, const void *b) dp_netdev_rxq_set_cycles(qb, RXQ_CYCLES_PROC_HIST, total_qb); - if (total_qa >= total_qb) { - return -1; + if (total_qa != total_qb) { + return (total_qa < total_qb) ? 1 : -1; + } else { + /* Cycles are the same so tiebreak on port/queue id. + * Tiebreaking (as opposed to return 0) ensures consistent + * sort results across multiple OS's. */ + if (qa->port->port_no != qb->port->port_no) { + return (qa->port->port_no > qb->port->port_no) ? 1 : -1; + } else { + return netdev_rxq_get_queue_id(qa->rx) + - netdev_rxq_get_queue_id(qb->rx); + } } - return 1; } From patchwork Thu Nov 23 19:41:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 840886 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yjV8x5hxQz9ryv for ; Fri, 24 Nov 2017 06:42:41 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 083BCAE7; Thu, 23 Nov 2017 19:42:12 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8B5B0ACB for ; Thu, 23 Nov 2017 19:42:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 50564224 for ; Thu, 23 Nov 2017 19:42:10 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BA3607EA9D; Thu, 23 Nov 2017 19:42:09 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id F286D5D6A5; Thu, 23 Nov 2017 19:42:05 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, aserdean@ovn.org, i.maximets@samsung.com, billy.o.mahony@intel.com, ian.stokes@intel.com Date: Thu, 23 Nov 2017 19:41:56 +0000 Message-Id: <1511466117-27191-2-git-send-email-ktraynor@redhat.com> In-Reply-To: <1511466117-27191-1-git-send-email-ktraynor@redhat.com> References: <1511375946-28054-1-git-send-email-ktraynor@redhat.com> <1511466117-27191-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Thu, 23 Nov 2017 19:42:09 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 2/3] dpif-netdev: Rename rxq_cycle_sort to compare_rxq_cycles. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This function is used for comparison between queues as part of the sort. It does not do the sort itself. As such, give it a more appropriate name. Suggested-by: Billy O'Mahony Signed-off-by: Kevin Traynor Acked-by: Billy O'Mahony --- V4: Added patch into series after suggestion by Billy lib/dpif-netdev.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index f5cdd92..657df71 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -3446,5 +3446,5 @@ rr_numa_list_destroy(struct rr_numa_list *rr) /* Sort Rx Queues by the processing cycles they are consuming. */ static int -rxq_cycle_sort(const void *a, const void *b) +compare_rxq_cycles(const void *a, const void *b) { struct dp_netdev_rxq *qa; @@ -3535,5 +3535,5 @@ rxq_scheduling(struct dp_netdev *dp, bool pinned) OVS_REQUIRES(dp->port_mutex) /* Sort the queues in order of the processing cycles * they consumed during their last pmd interval. */ - qsort(rxqs, n_rxqs, sizeof *rxqs, rxq_cycle_sort); + qsort(rxqs, n_rxqs, sizeof *rxqs, compare_rxq_cycles); } From patchwork Thu Nov 23 19:41:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 840887 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yjV9Y254cz9ryv for ; Fri, 24 Nov 2017 06:43:13 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id E92A2B14; Thu, 23 Nov 2017 19:42:16 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id AB754B08 for ; Thu, 23 Nov 2017 19:42:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 54AE41E1 for ; Thu, 23 Nov 2017 19:42:15 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D53A0C04D318; Thu, 23 Nov 2017 19:42:14 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id B5FCA5D6A5; Thu, 23 Nov 2017 19:42:10 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, aserdean@ovn.org, i.maximets@samsung.com, billy.o.mahony@intel.com, ian.stokes@intel.com Date: Thu, 23 Nov 2017 19:41:57 +0000 Message-Id: <1511466117-27191-3-git-send-email-ktraynor@redhat.com> In-Reply-To: <1511466117-27191-1-git-send-email-ktraynor@redhat.com> References: <1511375946-28054-1-git-send-email-ktraynor@redhat.com> <1511466117-27191-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 23 Nov 2017 19:42:14 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4 3/3] dpif-netdev: Calculate rxq cycles prior to compare_rxq_cycles calls. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org compare_rxq_cycles sums the latest cycles from each queue for comparison with each other. While each comparison correctly gets the latest cycles, the cycles could change between calls to compare_rxq_cycle. In order to use consistent values through each call of compare_rxq_cycles, sum the cycles before qsort is called. Requested-by: Ilya Maximets Signed-off-by: Kevin Traynor --- V4: Rebased to apply after patches 1/3 and 2/3 V3: - Drop V2 1/2 as merged - Removed rxq_cycle_sort return changes for a later patch - Some minor variable rename and comment changes lib/dpif-netdev.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 657df71..c31c09e 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -3450,20 +3450,14 @@ compare_rxq_cycles(const void *a, const void *b) struct dp_netdev_rxq *qa; struct dp_netdev_rxq *qb; - uint64_t total_qa, total_qb; - unsigned i; + uint64_t cycles_qa, cycles_qb; qa = *(struct dp_netdev_rxq **) a; qb = *(struct dp_netdev_rxq **) b; - total_qa = total_qb = 0; - for (i = 0; i < PMD_RXQ_INTERVAL_MAX; i++) { - total_qa += dp_netdev_rxq_get_intrvl_cycles(qa, i); - total_qb += dp_netdev_rxq_get_intrvl_cycles(qb, i); - } - dp_netdev_rxq_set_cycles(qa, RXQ_CYCLES_PROC_HIST, total_qa); - dp_netdev_rxq_set_cycles(qb, RXQ_CYCLES_PROC_HIST, total_qb); + cycles_qa = dp_netdev_rxq_get_cycles(qa, RXQ_CYCLES_PROC_HIST); + cycles_qb = dp_netdev_rxq_get_cycles(qb, RXQ_CYCLES_PROC_HIST); - if (total_qa != total_qb) { - return (total_qa < total_qb) ? 1 : -1; + if (cycles_qa != cycles_qb) { + return (cycles_qa < cycles_qb) ? 1 : -1; } else { /* Cycles are the same so tiebreak on port/queue id. @@ -3521,4 +3515,6 @@ rxq_scheduling(struct dp_netdev *dp, bool pinned) OVS_REQUIRES(dp->port_mutex) } } else if (!pinned && q->core_id == OVS_CORE_UNSPEC) { + uint64_t cycle_hist = 0; + if (n_rxqs == 0) { rxqs = xmalloc(sizeof *rxqs); @@ -3526,4 +3522,10 @@ rxq_scheduling(struct dp_netdev *dp, bool pinned) OVS_REQUIRES(dp->port_mutex) rxqs = xrealloc(rxqs, sizeof *rxqs * (n_rxqs + 1)); } + /* Sum the queue intervals and store the cycle history. */ + for (unsigned i = 0; i < PMD_RXQ_INTERVAL_MAX; i++) { + cycle_hist += dp_netdev_rxq_get_intrvl_cycles(q, i); + } + dp_netdev_rxq_set_cycles(q, RXQ_CYCLES_PROC_HIST, cycle_hist); + /* Store the queue. */ rxqs[n_rxqs++] = q;