From patchwork Mon Nov 2 13:10:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1392258 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=RGeAc93v; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CPtb4320kz9sRR for ; Tue, 3 Nov 2020 00:11:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id CF61F8737B; Mon, 2 Nov 2020 13:11:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zRd74+y8hJxL; Mon, 2 Nov 2020 13:11:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 450F687228; Mon, 2 Nov 2020 13:11:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 31D01C088B; Mon, 2 Nov 2020 13:11:04 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0EC36C0051 for ; Mon, 2 Nov 2020 13:11:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 023FE8669E for ; Mon, 2 Nov 2020 13:11:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z1lE3hayQj8G for ; Mon, 2 Nov 2020 13:11:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by whitealder.osuosl.org (Postfix) with ESMTPS id 6E91F8667D for ; Mon, 2 Nov 2020 13:11:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604322660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Na6GQLT7e3Emw7Sh7Swiehzd687JOqsXuQEm9LtVgM0=; b=RGeAc93vvz1JogMjopX3wBZheF75JKB6s8pnKMAkj20W3ALjmJKjQiYY3MqJdxTZMWauEn zllQEvy0b5Optow0LCYe64LW3MWTeiCX4BxM9jG5XjVVwX4pyzy3YFjch1PN3e99L8h4qU R3WL1ILvwWQ8o7l+aE5jLC/TOWRsCNs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-491-LrDkG6HdNNOnS4Wbo2sB5A-1; Mon, 02 Nov 2020 08:10:53 -0500 X-MC-Unique: LrDkG6HdNNOnS4Wbo2sB5A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DEA42186841E; Mon, 2 Nov 2020 13:10:51 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.193.82]) by smtp.corp.redhat.com (Postfix) with ESMTP id 219625DA71; Mon, 2 Nov 2020 13:10:49 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Mon, 2 Nov 2020 14:10:45 +0100 Message-Id: <20201102131045.10410-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david.marchand@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH dpdk-latest v4] dpdk: Support running PMD threads on any core. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" DPDK 20.08 introduced a new API that associates a non-EAL thread to a free lcore. This new API does not change the thread characteristics (like CPU affinity). Using this new API, there is no assumption on lcore X running on cpu X anymore which leaves OVS free from running its PMD thread on any cpu. DPDK still limits the number of lcores to RTE_MAX_LCORE (128 on x86_64) which should be enough for OVS pmd threads (hopefully). DPDK lcore/OVS pmd threads mapping are logged at threads creation and destruction. A new command is added to help get DPDK point of view of the DPDK lcores: $ ovs-appctl dpdk/lcores-list lcore 0, socket 0, role RTE, cpuset 0 lcore 1, socket 0, role NON_EAL, cpuset 1 lcore 2, socket 0, role NON_EAL, cpuset 15 Signed-off-by: David Marchand --- Changes since v3: - rebased on current HEAD, - switched back to simple warning rather than abort when registering a thread fails, Changes since v2: - introduced a new api in DPDK 20.08 (still being discussed), inbox thread at http://inbox.dpdk.org/dev/20200610144506.30505-1-david.marchand@redhat.com/T/#t - this current patch depends on a patch on master I sent: https://patchwork.ozlabs.org/project/openvswitch/patch/20200626122738.28163-1-david.marchand@redhat.com/ - dropped 'dpdk-lcore-mask' compat handling, Changes since v1: - rewired existing configuration 'dpdk-lcore-mask' to use --lcores, - switched to a bitmap to track lcores, - added a command to dump current mapping (Flavio): used an experimental API to get DPDK lcores cpuset since it is the most reliable/portable information, - used the same code for the logs when starting DPDK/PMD threads, - addressed Ilya comments, --- lib/dpdk-stub.c | 8 +++++++- lib/dpdk.c | 36 ++++++++++++++++++++++++++++++++++-- lib/dpdk.h | 3 ++- lib/dpif-netdev.c | 3 ++- 4 files changed, 45 insertions(+), 5 deletions(-) diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c index b7d577870d..5bc996b665 100644 --- a/lib/dpdk-stub.c +++ b/lib/dpdk-stub.c @@ -39,7 +39,13 @@ dpdk_init(const struct smap *ovs_other_config) } void -dpdk_set_lcore_id(unsigned cpu OVS_UNUSED) +dpdk_init_thread_context(unsigned cpu OVS_UNUSED) +{ + /* Nothing */ +} + +void +dpdk_uninit_thread_context(void) { /* Nothing */ } diff --git a/lib/dpdk.c b/lib/dpdk.c index 319540394b..7d3c0637c9 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -14,6 +14,10 @@ * limitations under the License. */ +#ifndef ALLOW_EXPERIMENTAL_API +#define ALLOW_EXPERIMENTAL_API +#endif + #include #include "dpdk.h" @@ -525,6 +529,8 @@ dpdk_init__(const struct smap *ovs_other_config) dpdk_unixctl_mem_stream, rte_log_dump); unixctl_command_register("dpdk/log-set", "{level | pattern:level}", 0, INT_MAX, dpdk_unixctl_log_set, NULL); + unixctl_command_register("dpdk/lcores-list", "", 0, 0, + dpdk_unixctl_mem_stream, rte_lcore_dump); /* We are called from the main thread here */ RTE_PER_LCORE(_lcore_id) = NON_PMD_CORE_ID; @@ -601,11 +607,37 @@ dpdk_available(void) } void -dpdk_set_lcore_id(unsigned cpu) +dpdk_init_thread_context(unsigned cpu) { /* NON_PMD_CORE_ID is reserved for use by non pmd threads. */ ovs_assert(cpu != NON_PMD_CORE_ID); - RTE_PER_LCORE(_lcore_id) = cpu; + + if (!dpdk_available()) { + return; + } + + if (rte_thread_register() < 0) { + VLOG_WARN("This OVS pmd thread will share resources with the non-pmd " + "thread: %s.", rte_strerror(rte_errno)); + } else { + VLOG_INFO("PMD thread uses DPDK lcore %u.", rte_lcore_id()); + } +} + +void +dpdk_uninit_thread_context(void) +{ + unsigned int lcore_id; + + if (!dpdk_available()) { + return; + } + + lcore_id = rte_lcore_id(); + rte_thread_unregister(); + if (lcore_id != LCORE_ID_ANY) { + VLOG_INFO("PMD thread released DPDK lcore %u.", lcore_id); + } } void diff --git a/lib/dpdk.h b/lib/dpdk.h index 445a51d065..1bd16b31db 100644 --- a/lib/dpdk.h +++ b/lib/dpdk.h @@ -36,7 +36,8 @@ struct smap; struct ovsrec_open_vswitch; void dpdk_init(const struct smap *ovs_other_config); -void dpdk_set_lcore_id(unsigned cpu); +void dpdk_init_thread_context(unsigned cpu); +void dpdk_uninit_thread_context(void); const char *dpdk_get_vhost_sock_dir(void); bool dpdk_vhost_iommu_enabled(void); bool dpdk_vhost_postcopy_enabled(void); diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 300861ca59..da9e1e8fcf 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -5926,7 +5926,7 @@ pmd_thread_main(void *f_) /* Stores the pmd thread's 'pmd' to 'per_pmd_key'. */ ovsthread_setspecific(pmd->dp->per_pmd_key, pmd); ovs_numa_thread_setaffinity_core(pmd->core_id); - dpdk_set_lcore_id(pmd->core_id); + dpdk_init_thread_context(pmd->core_id); poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); dfc_cache_init(&pmd->flow_cache); pmd_alloc_static_tx_qid(pmd); @@ -6061,6 +6061,7 @@ reload: dfc_cache_uninit(&pmd->flow_cache); free(poll_list); pmd_free_cached_ports(pmd); + dpdk_uninit_thread_context(); return NULL; }