From patchwork Thu Jan 11 14:32:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 859160 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zHSys6Z6dz9s8J for ; Fri, 12 Jan 2018 01:32:53 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 46AD2FE0; Thu, 11 Jan 2018 14:32:52 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id D2588EEF for ; Thu, 11 Jan 2018 14:32:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id D98CA17E for ; Thu, 11 Jan 2018 14:32:49 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 48B0C4E4C2; Thu, 11 Jan 2018 14:32:44 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0DC077553A; Thu, 11 Jan 2018 14:32:22 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, jan.scheurich@ericsson.com, ian.stokes@intel.com, i.maximets@samsung.com, billy.o.mahony@intel.com, aconole@redhat.com, fbl@redhat.com Date: Thu, 11 Jan 2018 14:32:11 +0000 Message-Id: <1515681132-15693-1-git-send-email-ktraynor@redhat.com> In-Reply-To: <1513171637-9215-3-git-send-email-ktraynor@redhat.com> References: <1513171637-9215-3-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 11 Jan 2018 14:32:49 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 1/2] dpif-netdev: Add percentage of pmd/core used by each rxq. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is based on the length of history that is stored about an rxq (currently 1 min). $ ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 0 core_id 4: isolated : false port: dpdkphy1 queue-id: 0 pmd usage: 70 % port: dpdkvhost0 queue-id: 0 pmd usage: 0 % pmd thread numa_id 0 core_id 6: isolated : false port: dpdkphy0 queue-id: 0 pmd usage: 64 % port: dpdkvhost1 queue-id: 0 pmd usage: 0 % These values are what would be used as part of rxq to pmd assignment due to a reconfiguration event e.g. adding pmds, adding rxqs or with the command: ovs-appctl dpif-netdev/pmd-rxq-rebalance Signed-off-by: Jan Scheurich Co-authored-by: Jan Scheurich Signed-off-by: Kevin Traynor --- V2: - Re-worked to calculate and store cycles needed for stats directly in pmd. Documentation/howto/dpdk.rst | 11 +++++++++ NEWS | 1 + lib/dpif-netdev.c | 53 +++++++++++++++++++++++++++++++++----------- tests/pmd.at | 51 +++++++++++++++++++++++++++++++----------- 4 files changed, 90 insertions(+), 26 deletions(-) diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index 587aaed..40f9d96 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -140,4 +140,15 @@ Core 7: Q4 (70%) | Q5 (10%) core 8: Q3 (60%) | Q0 (30%) +To see the current measured usage history of pmd core cycles for each rxq:: + + $ ovs-appctl dpif-netdev/pmd-rxq-show + +.. note:: + + A history of one minute is recorded and shown for each rxq to allow for + traffic pattern spikes. An rxq's pmd core cycles usage changes due to traffic + pattern or reconfig changes will take one minute before they are fully + reflected in the stats. + Rxq to pmds assignment takes place whenever there are configuration changes or can be triggered by using:: diff --git a/NEWS b/NEWS index aff0671..088dca3 100644 --- a/NEWS +++ b/NEWS @@ -41,4 +41,5 @@ Post-v2.8.0 - ovs-ofctl dump-ports command now prints new of set custom statistics if available (for OpenFlow 1.4+). + * Add rxq utilization of pmd to appctl 'dpif-netdev/pmd-rxq-show'. - vswitchd: * Datapath IDs may now be specified as 0x1 (etc.) instead of 16 digits. diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 57f6570..d960a3a 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -602,4 +602,9 @@ struct dp_netdev_pmd_thread { struct dp_netdev_pmd_cycles cycles; + /* Last interval timestamp. */ + uint64_t intrvl_tsc_prev; + /* Last interval cycles. */ + atomic_ullong intrvl_cycles; + /* Current context of the PMD thread. */ struct dp_netdev_pmd_thread_ctx ctx; @@ -996,7 +1001,7 @@ pmd_info_show_rxq(struct ds *reply, struct dp_netdev_pmd_thread *pmd) { if (pmd->core_id != NON_PMD_CORE_ID) { - const char *prev_name = NULL; struct rxq_poll *list; - size_t i, n; + size_t n_rxq; + uint64_t total_cycles = 0; ds_put_format(reply, @@ -1006,20 +1011,32 @@ pmd_info_show_rxq(struct ds *reply, struct dp_netdev_pmd_thread *pmd) ovs_mutex_lock(&pmd->port_mutex); - sorted_poll_list(pmd, &list, &n); - for (i = 0; i < n; i++) { - const char *name = netdev_rxq_get_name(list[i].rxq->rx); + sorted_poll_list(pmd, &list, &n_rxq); - if (!prev_name || strcmp(name, prev_name)) { - if (prev_name) { - ds_put_cstr(reply, "\n"); - } - ds_put_format(reply, "\tport: %s\tqueue-id:", name); + /* Get the total pmd cycles for an interval. */ + atomic_read_relaxed(&pmd->intrvl_cycles, &total_cycles); + /* Estimate the cycles to cover all intervals. */ + total_cycles *= PMD_RXQ_INTERVAL_MAX; + + for (int i = 0; i < n_rxq; i++) { + struct dp_netdev_rxq *rxq = list[i].rxq; + const char *name = netdev_rxq_get_name(rxq->rx); + uint64_t proc_cycles = 0; + + for (int j = 0; j < PMD_RXQ_INTERVAL_MAX; j++) { + proc_cycles += dp_netdev_rxq_get_intrvl_cycles(rxq, j); } - ds_put_format(reply, " %d", + ds_put_format(reply, "\tport: %16s\tqueue-id: %2d", name, netdev_rxq_get_queue_id(list[i].rxq->rx)); - prev_name = name; + ds_put_format(reply, "\tpmd usage: "); + if (total_cycles) { + ds_put_format(reply, "%2"PRIu64"", + proc_cycles * 100 / total_cycles); + ds_put_cstr(reply, " %"); + } else { + ds_put_format(reply, "%s", "NOT AVAIL"); + } + ds_put_cstr(reply, "\n"); } ovs_mutex_unlock(&pmd->port_mutex); - ds_put_cstr(reply, "\n"); free(list); } @@ -4157,4 +4174,6 @@ reload: } + pmd->intrvl_tsc_prev = 0; + atomic_store_relaxed(&pmd->intrvl_cycles, 0); cycles_count_start(pmd); for (;;) { @@ -6138,4 +6157,5 @@ dp_netdev_pmd_try_optimize(struct dp_netdev_pmd_thread *pmd, if (pmd->ctx.now > pmd->rxq_next_cycle_store) { + uint64_t curr_tsc; /* Get the cycles that were used to process each queue and store. */ for (unsigned i = 0; i < poll_cnt; i++) { @@ -6146,4 +6166,11 @@ dp_netdev_pmd_try_optimize(struct dp_netdev_pmd_thread *pmd, 0); } + curr_tsc = cycles_counter(); + if (pmd->intrvl_tsc_prev) { + /* There is a prev timestamp, store a new intrvl cycle count. */ + atomic_store_relaxed(&pmd->intrvl_cycles, + curr_tsc - pmd->intrvl_tsc_prev); + } + pmd->intrvl_tsc_prev = curr_tsc; /* Start new measuring interval */ pmd->rxq_next_cycle_store = pmd->ctx.now + PMD_RXQ_INTERVAL_LEN; diff --git a/tests/pmd.at b/tests/pmd.at index fcb007c..09d09cb 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -7,5 +7,13 @@ m4_divert_push([PREPARE_TESTS]) # port_name rxq_id numa_id core_id parse_pmd_rxq_show () { - awk '/pmd/ {numa=$4; core=substr($6, 1, length($6) - 1)} /\t/{for (i=4; i<=NF; i++) print $2, $i, numa, core}' | sort + awk '/pmd thread/ {numa=$4; core=substr($6, 1, length($6) - 1)} /\tport:/ {print $2, $4, numa, core}' | sort +} + +# Given the output of `ovs-appctl dpif-netdev/pmd-rxq-show`, +# and with queues for each core on one line, prints the rxqs +# of the core on one line +# 'port:' port_name 'queue_id:' rxq_id rxq_id rxq_id rxq_id +parse_pmd_rxq_show_group () { + awk '/port:/ {print $1, $2, $3, $4, $12, $20, $28}' } @@ -54,5 +62,5 @@ m4_define([CHECK_PMD_THREADS_CREATED], [ m4_define([SED_NUMA_CORE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/"]) -m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/;s/\(queue-id: \)1 2 5 6/\1/;s/\(queue-id: \)0 3 4 7/\1/"]) +m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/1 2 5 6//;s/0 3 4 7//"]) m4_define([DUMMY_NUMA], [--dummy-numa="0,0,0,0"]) @@ -66,5 +74,5 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 + port: p0 queue-id: 0 pmd usage: NOT AVAIL ]) @@ -97,5 +105,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ]) @@ -121,5 +136,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ]) @@ -128,11 +150,7 @@ AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3]) CHECK_PMD_THREADS_CREATED([2], [], [+$TMP]) -AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_QUEUE_PATTERN], [0], [dnl -pmd thread numa_id core_id : - isolated : false - port: p0 queue-id: -pmd thread numa_id core_id : - isolated : false - port: p0 queue-id: +AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed ':a;/AVAIL$/{N;s/\n//;ba}' | parse_pmd_rxq_show_group | sed SED_NUMA_CORE_QUEUE_PATTERN], [0], [dnl +port: p0 queue-id: +port: p0 queue-id: ]) @@ -144,5 +162,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ]) From patchwork Thu Jan 11 14:32:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 859162 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zHSzd3Cnwz9s75 for ; Fri, 12 Jan 2018 01:33:33 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 417EB1077; Thu, 11 Jan 2018 14:32:53 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 96144EEF for ; Thu, 11 Jan 2018 14:32:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 571F617E for ; Thu, 11 Jan 2018 14:32:51 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 54F9451EFF; Thu, 11 Jan 2018 14:32:45 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id B9ADF75549; Thu, 11 Jan 2018 14:32:35 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, jan.scheurich@ericsson.com, ian.stokes@intel.com, i.maximets@samsung.com, billy.o.mahony@intel.com, aconole@redhat.com, fbl@redhat.com Date: Thu, 11 Jan 2018 14:32:12 +0000 Message-Id: <1515681132-15693-2-git-send-email-ktraynor@redhat.com> In-Reply-To: <1515681132-15693-1-git-send-email-ktraynor@redhat.com> References: <1513171637-9215-3-git-send-email-ktraynor@redhat.com> <1515681132-15693-1-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Thu, 11 Jan 2018 14:32:50 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 2/2] NEWS: Add entry for new appctl dpif-netdev/pmd-rxq-rebalance. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This feature was added earlier but we thought it better to advertize in NEWS after there was stats provided to help the user decide whether they should use it. Signed-off-by: Kevin Traynor --- V2 - Newly added. NEWS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/NEWS b/NEWS index 088dca3..1e72562 100644 --- a/NEWS +++ b/NEWS @@ -41,4 +41,6 @@ Post-v2.8.0 - ovs-ofctl dump-ports command now prints new of set custom statistics if available (for OpenFlow 1.4+). + * New appctl command 'dpif-netdev/pmd-rxq-rebalance' to rebalance rxq to + pmd assignments. * Add rxq utilization of pmd to appctl 'dpif-netdev/pmd-rxq-show'. - vswitchd: