From patchwork Thu Jan 11 14:32:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 859160 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zHSys6Z6dz9s8J for ; Fri, 12 Jan 2018 01:32:53 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 46AD2FE0; Thu, 11 Jan 2018 14:32:52 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id D2588EEF for ; Thu, 11 Jan 2018 14:32:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id D98CA17E for ; Thu, 11 Jan 2018 14:32:49 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 48B0C4E4C2; Thu, 11 Jan 2018 14:32:44 +0000 (UTC) Received: from ktraynor.remote.csb (unknown [10.36.118.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0DC077553A; Thu, 11 Jan 2018 14:32:22 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org, jan.scheurich@ericsson.com, ian.stokes@intel.com, i.maximets@samsung.com, billy.o.mahony@intel.com, aconole@redhat.com, fbl@redhat.com Date: Thu, 11 Jan 2018 14:32:11 +0000 Message-Id: <1515681132-15693-1-git-send-email-ktraynor@redhat.com> In-Reply-To: <1513171637-9215-3-git-send-email-ktraynor@redhat.com> References: <1513171637-9215-3-git-send-email-ktraynor@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 11 Jan 2018 14:32:49 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 1/2] dpif-netdev: Add percentage of pmd/core used by each rxq. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is based on the length of history that is stored about an rxq (currently 1 min). $ ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 0 core_id 4: isolated : false port: dpdkphy1 queue-id: 0 pmd usage: 70 % port: dpdkvhost0 queue-id: 0 pmd usage: 0 % pmd thread numa_id 0 core_id 6: isolated : false port: dpdkphy0 queue-id: 0 pmd usage: 64 % port: dpdkvhost1 queue-id: 0 pmd usage: 0 % These values are what would be used as part of rxq to pmd assignment due to a reconfiguration event e.g. adding pmds, adding rxqs or with the command: ovs-appctl dpif-netdev/pmd-rxq-rebalance Signed-off-by: Jan Scheurich Co-authored-by: Jan Scheurich Signed-off-by: Kevin Traynor --- V2: - Re-worked to calculate and store cycles needed for stats directly in pmd. Documentation/howto/dpdk.rst | 11 +++++++++ NEWS | 1 + lib/dpif-netdev.c | 53 +++++++++++++++++++++++++++++++++----------- tests/pmd.at | 51 +++++++++++++++++++++++++++++++----------- 4 files changed, 90 insertions(+), 26 deletions(-) diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index 587aaed..40f9d96 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -140,4 +140,15 @@ Core 7: Q4 (70%) | Q5 (10%) core 8: Q3 (60%) | Q0 (30%) +To see the current measured usage history of pmd core cycles for each rxq:: + + $ ovs-appctl dpif-netdev/pmd-rxq-show + +.. note:: + + A history of one minute is recorded and shown for each rxq to allow for + traffic pattern spikes. An rxq's pmd core cycles usage changes due to traffic + pattern or reconfig changes will take one minute before they are fully + reflected in the stats. + Rxq to pmds assignment takes place whenever there are configuration changes or can be triggered by using:: diff --git a/NEWS b/NEWS index aff0671..088dca3 100644 --- a/NEWS +++ b/NEWS @@ -41,4 +41,5 @@ Post-v2.8.0 - ovs-ofctl dump-ports command now prints new of set custom statistics if available (for OpenFlow 1.4+). + * Add rxq utilization of pmd to appctl 'dpif-netdev/pmd-rxq-show'. - vswitchd: * Datapath IDs may now be specified as 0x1 (etc.) instead of 16 digits. diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 57f6570..d960a3a 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -602,4 +602,9 @@ struct dp_netdev_pmd_thread { struct dp_netdev_pmd_cycles cycles; + /* Last interval timestamp. */ + uint64_t intrvl_tsc_prev; + /* Last interval cycles. */ + atomic_ullong intrvl_cycles; + /* Current context of the PMD thread. */ struct dp_netdev_pmd_thread_ctx ctx; @@ -996,7 +1001,7 @@ pmd_info_show_rxq(struct ds *reply, struct dp_netdev_pmd_thread *pmd) { if (pmd->core_id != NON_PMD_CORE_ID) { - const char *prev_name = NULL; struct rxq_poll *list; - size_t i, n; + size_t n_rxq; + uint64_t total_cycles = 0; ds_put_format(reply, @@ -1006,20 +1011,32 @@ pmd_info_show_rxq(struct ds *reply, struct dp_netdev_pmd_thread *pmd) ovs_mutex_lock(&pmd->port_mutex); - sorted_poll_list(pmd, &list, &n); - for (i = 0; i < n; i++) { - const char *name = netdev_rxq_get_name(list[i].rxq->rx); + sorted_poll_list(pmd, &list, &n_rxq); - if (!prev_name || strcmp(name, prev_name)) { - if (prev_name) { - ds_put_cstr(reply, "\n"); - } - ds_put_format(reply, "\tport: %s\tqueue-id:", name); + /* Get the total pmd cycles for an interval. */ + atomic_read_relaxed(&pmd->intrvl_cycles, &total_cycles); + /* Estimate the cycles to cover all intervals. */ + total_cycles *= PMD_RXQ_INTERVAL_MAX; + + for (int i = 0; i < n_rxq; i++) { + struct dp_netdev_rxq *rxq = list[i].rxq; + const char *name = netdev_rxq_get_name(rxq->rx); + uint64_t proc_cycles = 0; + + for (int j = 0; j < PMD_RXQ_INTERVAL_MAX; j++) { + proc_cycles += dp_netdev_rxq_get_intrvl_cycles(rxq, j); } - ds_put_format(reply, " %d", + ds_put_format(reply, "\tport: %16s\tqueue-id: %2d", name, netdev_rxq_get_queue_id(list[i].rxq->rx)); - prev_name = name; + ds_put_format(reply, "\tpmd usage: "); + if (total_cycles) { + ds_put_format(reply, "%2"PRIu64"", + proc_cycles * 100 / total_cycles); + ds_put_cstr(reply, " %"); + } else { + ds_put_format(reply, "%s", "NOT AVAIL"); + } + ds_put_cstr(reply, "\n"); } ovs_mutex_unlock(&pmd->port_mutex); - ds_put_cstr(reply, "\n"); free(list); } @@ -4157,4 +4174,6 @@ reload: } + pmd->intrvl_tsc_prev = 0; + atomic_store_relaxed(&pmd->intrvl_cycles, 0); cycles_count_start(pmd); for (;;) { @@ -6138,4 +6157,5 @@ dp_netdev_pmd_try_optimize(struct dp_netdev_pmd_thread *pmd, if (pmd->ctx.now > pmd->rxq_next_cycle_store) { + uint64_t curr_tsc; /* Get the cycles that were used to process each queue and store. */ for (unsigned i = 0; i < poll_cnt; i++) { @@ -6146,4 +6166,11 @@ dp_netdev_pmd_try_optimize(struct dp_netdev_pmd_thread *pmd, 0); } + curr_tsc = cycles_counter(); + if (pmd->intrvl_tsc_prev) { + /* There is a prev timestamp, store a new intrvl cycle count. */ + atomic_store_relaxed(&pmd->intrvl_cycles, + curr_tsc - pmd->intrvl_tsc_prev); + } + pmd->intrvl_tsc_prev = curr_tsc; /* Start new measuring interval */ pmd->rxq_next_cycle_store = pmd->ctx.now + PMD_RXQ_INTERVAL_LEN; diff --git a/tests/pmd.at b/tests/pmd.at index fcb007c..09d09cb 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -7,5 +7,13 @@ m4_divert_push([PREPARE_TESTS]) # port_name rxq_id numa_id core_id parse_pmd_rxq_show () { - awk '/pmd/ {numa=$4; core=substr($6, 1, length($6) - 1)} /\t/{for (i=4; i<=NF; i++) print $2, $i, numa, core}' | sort + awk '/pmd thread/ {numa=$4; core=substr($6, 1, length($6) - 1)} /\tport:/ {print $2, $4, numa, core}' | sort +} + +# Given the output of `ovs-appctl dpif-netdev/pmd-rxq-show`, +# and with queues for each core on one line, prints the rxqs +# of the core on one line +# 'port:' port_name 'queue_id:' rxq_id rxq_id rxq_id rxq_id +parse_pmd_rxq_show_group () { + awk '/port:/ {print $1, $2, $3, $4, $12, $20, $28}' } @@ -54,5 +62,5 @@ m4_define([CHECK_PMD_THREADS_CREATED], [ m4_define([SED_NUMA_CORE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/"]) -m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/\(numa_id \)[[0-9]]*\( core_id \)[[0-9]]*:/\1\2:/;s/\(queue-id: \)1 2 5 6/\1/;s/\(queue-id: \)0 3 4 7/\1/"]) +m4_define([SED_NUMA_CORE_QUEUE_PATTERN], ["s/1 2 5 6//;s/0 3 4 7//"]) m4_define([DUMMY_NUMA], [--dummy-numa="0,0,0,0"]) @@ -66,5 +74,5 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 + port: p0 queue-id: 0 pmd usage: NOT AVAIL ]) @@ -97,5 +105,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ]) @@ -121,5 +136,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ]) @@ -128,11 +150,7 @@ AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3]) CHECK_PMD_THREADS_CREATED([2], [], [+$TMP]) -AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_QUEUE_PATTERN], [0], [dnl -pmd thread numa_id core_id : - isolated : false - port: p0 queue-id: -pmd thread numa_id core_id : - isolated : false - port: p0 queue-id: +AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed ':a;/AVAIL$/{N;s/\n//;ba}' | parse_pmd_rxq_show_group | sed SED_NUMA_CORE_QUEUE_PATTERN], [0], [dnl +port: p0 queue-id: +port: p0 queue-id: ]) @@ -144,5 +162,12 @@ AT_CHECK([ovs-appctl dpif-netdev/pmd-rxq-show | sed SED_NUMA_CORE_PATTERN], [0], pmd thread numa_id core_id : isolated : false - port: p0 queue-id: 0 1 2 3 4 5 6 7 + port: p0 queue-id: 0 pmd usage: NOT AVAIL + port: p0 queue-id: 1 pmd usage: NOT AVAIL + port: p0 queue-id: 2 pmd usage: NOT AVAIL + port: p0 queue-id: 3 pmd usage: NOT AVAIL + port: p0 queue-id: 4 pmd usage: NOT AVAIL + port: p0 queue-id: 5 pmd usage: NOT AVAIL + port: p0 queue-id: 6 pmd usage: NOT AVAIL + port: p0 queue-id: 7 pmd usage: NOT AVAIL ])