From patchwork Wed Aug 23 13:34:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 804979 X-Patchwork-Delegate: dlu998@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xcpRl3Nwhz9s83 for ; Wed, 23 Aug 2017 23:38:59 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 6B134AD2; Wed, 23 Aug 2017 13:36:01 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 1C234AD2 for ; Wed, 23 Aug 2017 13:35:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 0EE13458 for ; Wed, 23 Aug 2017 13:35:58 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 64F13C047B78; Wed, 23 Aug 2017 13:35:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 64F13C047B78 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=ktraynor@redhat.com Received: from ktraynor.remote.csb (unknown [10.36.118.65]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FA32614D3; Wed, 23 Aug 2017 13:35:53 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, dball@vmware.com, ian.stokes@intel.com, jan.scheurich@ericsson.com, bhanuprakash.bodireddy@intel.com, mark.b.kavanagh@intel.com, gvrose8192@gmail.com, fbl@redhat.com Date: Wed, 23 Aug 2017 14:34:40 +0100 Message-Id: <1503495281-1741-6-git-send-email-ktraynor@redhat.com> In-Reply-To: <1503495281-1741-1-git-send-email-ktraynor@redhat.com> References: <1502293530-10783-1-git-send-email-ktraynor@redhat.com> <1503495281-1741-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Wed, 23 Aug 2017 13:35:57 +0000 (UTC) Subject: [ovs-dev] [PATCH v5 5/6] dpif-netdev: Change pmd selection order. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Up to his point rxqs are sorted by processing cycles they consumed and assigned to pmds in a round robin manner. Ian pointed out that on wrap around the most loaded pmd will be the next one to be assigned an additional rxq and that it would be better to reverse the pmd order when wraparound occurs. In other words, change from assigning by rr to assigning in a forward and reverse cycle through pmds. Also, now that the algothim has finalised, document an example. Suggested-by: Ian Stokes Signed-off-by: Kevin Traynor --- Documentation/howto/dpdk.rst | 16 ++++++++++++++++ lib/dpif-netdev.c | 21 ++++++++++++++++++++- tests/pmd.at | 2 +- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index 44737e4..493e215 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -124,4 +124,20 @@ will be used where known to assign rxqs with the highest consumption of processing cycles to different pmds. +For example, in the case where here there are 5 rxqs and 3 cores (e.g. 3,7,8) +available, and the measured usage of core cycles per rxq over the last +interval is seen to be: + +- Queue #0: 30% +- Queue #1: 80% +- Queue #3: 60% +- Queue #4: 70% +- Queue #5: 10% + +The rxqs will be assigned to cores 3,7,8 in the following order: + +Core 3: Q1 (80%) | +Core 7: Q4 (70%) | Q5 (10%) +core 8: Q0 (60%) | Q0 (30%) + Rxq to pmds assignment takes place whenever there are configuration changes. diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index afbf591..6cc0a1e 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -3285,4 +3285,5 @@ struct rr_numa { int cur_index; + bool idx_inc; }; @@ -3341,4 +3342,7 @@ rr_numa_list_populate(struct dp_netdev *dp, struct rr_numa_list *rr) numa->pmds = xrealloc(numa->pmds, numa->n_pmds * sizeof *numa->pmds); numa->pmds[numa->n_pmds - 1] = pmd; + /* At least one pmd so initialise curr_idx and idx_inc. */ + numa->cur_index = 0; + numa->idx_inc = true; } } @@ -3347,5 +3351,20 @@ static struct dp_netdev_pmd_thread * rr_numa_get_pmd(struct rr_numa *numa) { - return numa->pmds[numa->cur_index++ % numa->n_pmds]; + int numa_idx = numa->cur_index; + + if (numa->idx_inc == true) { + if (numa->cur_index == numa->n_pmds-1) { + numa->idx_inc = false; + } else { + numa->cur_index++; + } + } else { + if (numa->cur_index == 0) { + numa->idx_inc = true; + } else { + numa->cur_index--; + } + } + return numa->pmds[numa_idx]; } diff --git a/tests/pmd.at b/tests/pmd.at index b6732ea..e39a23a 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -54,5 +54,5 @@ m4_define([CHECK_PMD_THREADS_CREATED], [ m4_define([SED_NUMA_CORE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/"]) -m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/;s/\(queue-id: \)0 2 4 6/\1/;s/\(queue-id: \)1 3 5 7/\1/"]) +m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/;s/\(queue-id: \)1 2 5 6/\1/;s/\(queue-id: \)0 3 4 7/\1/"]) m4_define([DUMMY_NUMA], [--dummy-numa="0,0,0,0"])