From patchwork Fri Jun 26 12:30:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1317615 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=P3dInWzD; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49tbp51bFmz9sRW for ; Fri, 26 Jun 2020 22:30:53 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A30AC231CB; Fri, 26 Jun 2020 12:30:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qVmQWn9e+nfY; Fri, 26 Jun 2020 12:30:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 579A820117; Fri, 26 Jun 2020 12:30:49 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4ACC9C0863; Fri, 26 Jun 2020 12:30:49 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1CDCFC016F for ; Fri, 26 Jun 2020 12:30:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0711B231CB for ; Fri, 26 Jun 2020 12:30:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mL0N6d2taPpx for ; Fri, 26 Jun 2020 12:30:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by silver.osuosl.org (Postfix) with ESMTPS id A9BA620117 for ; Fri, 26 Jun 2020 12:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593174644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UcYSFNMMfAontDNbl/UNSykICVDOqLWtAK0Tc2E9u1U=; b=P3dInWzDK1m3766AUcEC7itSktqS+ZPRpjLaBoTkug34HgPJHYu7LognawuwnmLT1LH12o KMmSpkwJHsoanti5j/m1HspkHzN62pjVK7e93VtGuE/HsD7C5Y05HfL+sxF5xk2FyzVzsW S4X/z22eijD7ysyznQG/KOXx23y8biE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-393-7xA2qlfJMOuJ2VHXueqIDQ-1; Fri, 26 Jun 2020 08:30:36 -0400 X-MC-Unique: 7xA2qlfJMOuJ2VHXueqIDQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BF23D3626D; Fri, 26 Jun 2020 12:30:25 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.193.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1FD851010428; Fri, 26 Jun 2020 12:30:23 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Fri, 26 Jun 2020 14:30:17 +0200 Message-Id: <20200626123017.28555-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH dpdk-latest v3] dpdk: Support running PMD threads on any core. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" DPDK 20.08 introduced a new API that associates a non-EAL thread to a free lcore. This new API does not change the thread characteristics (like CPU affinity). Using this new API, there is no assumption on lcore X running on cpu X anymore which leaves OVS free from running its PMD thread on any cpu. DPDK still limits the number of lcores to RTE_MAX_LCORE (128 on x86_64) which should be enough for OVS (hopefully). lcore/pmd threads mapping are logged at threads creation and destruction. A new command is added to help get DPDK point of view of the lcores: $ ovs-appctl dpdk/lcores-list lcore 0, socket 0, role RTE, cpuset 0 lcore 1, socket 0, role NON_EAL, cpuset 1 lcore 2, socket 0, role NON_EAL, cpuset 15 Signed-off-by: David Marchand --- Changes since v2: - introduced a new api in DPDK 20.08 (still being discussed), inbox thread at http://inbox.dpdk.org/dev/20200610144506.30505-1-david.marchand@redhat.com/T/#t - this current patch depends on a patch on master I sent: https://patchwork.ozlabs.org/project/openvswitch/patch/20200626122738.28163-1-david.marchand@redhat.com/ - dropped 'dpdk-lcore-mask' compat handling, Changes since v1: - rewired existing configuration 'dpdk-lcore-mask' to use --lcores, - switched to a bitmap to track lcores, - added a command to dump current mapping (Flavio): used an experimental API to get DPDK lcores cpuset since it is the most reliable/portable information, - used the same code for the logs when starting DPDK/PMD threads, - addressed Ilya comments, --- lib/dpdk-stub.c | 8 +++++++- lib/dpdk.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- lib/dpdk.h | 3 ++- lib/dpif-netdev.c | 3 ++- 4 files changed, 57 insertions(+), 4 deletions(-) diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c index c332c217cb..90473bc8e7 100644 --- a/lib/dpdk-stub.c +++ b/lib/dpdk-stub.c @@ -39,7 +39,13 @@ dpdk_init(const struct smap *ovs_other_config) } void -dpdk_set_lcore_id(unsigned cpu OVS_UNUSED) +dpdk_init_thread_context(unsigned cpu OVS_UNUSED) +{ + /* Nothing */ +} + +void +dpdk_uninit_thread_context(void) { /* Nothing */ } diff --git a/lib/dpdk.c b/lib/dpdk.c index 55ce9a9221..2a97786d14 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -358,6 +358,31 @@ dpdk_unixctl_log_set(struct unixctl_conn *conn, int argc, const char *argv[], unixctl_command_reply(conn, NULL); } +#ifdef ALLOW_EXPERIMENTAL_API +static void +dpdk_unixctl_lcore_list(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + char *response = NULL; + FILE *stream; + size_t size; + + stream = open_memstream(&response, &size); + if (!stream) { + response = xasprintf("Unable to open memstream: %s.", + ovs_strerror(errno)); + unixctl_command_reply_error(conn, response); + goto out; + } + + rte_lcore_dump(stream); + fclose(stream); + unixctl_command_reply(conn, response); +out: + free(response); +} +#endif + static bool dpdk_init__(const struct smap *ovs_other_config) { @@ -537,6 +562,10 @@ dpdk_init__(const struct smap *ovs_other_config) dpdk_unixctl_log_list, NULL); unixctl_command_register("dpdk/log-set", "pattern:level", 0, INT_MAX, dpdk_unixctl_log_set, NULL); +#ifdef ALLOW_EXPERIMENTAL_API + unixctl_command_register("dpdk/lcores-list", "", 0, 0, + dpdk_unixctl_lcore_list, NULL); +#endif /* We are called from the main thread here */ RTE_PER_LCORE(_lcore_id) = NON_PMD_CORE_ID; @@ -613,18 +642,34 @@ dpdk_available(void) } void -dpdk_set_lcore_id(unsigned cpu) +dpdk_init_thread_context(unsigned cpu) { /* NON_PMD_CORE_ID is reserved for use by non pmd threads. */ ovs_assert(cpu != NON_PMD_CORE_ID); +#ifdef ALLOW_EXPERIMENTAL_API + rte_thread_register(); +#else if (cpu >= RTE_MAX_LCORE) { cpu = LCORE_ID_ANY; } RTE_PER_LCORE(_lcore_id) = cpu; +#endif if (rte_lcore_id() == LCORE_ID_ANY) { ovs_abort(0, "PMD thread init failed, trying to use more cores than " "DPDK supports (RTE_MAX_LCORE %u).", RTE_MAX_LCORE); } + VLOG_INFO("PMD thread is associated to DPDK lcore %u.", rte_lcore_id()); +} + +void +dpdk_uninit_thread_context(void) +{ + unsigned int lcore_id = rte_lcore_id(); + +#ifdef ALLOW_EXPERIMENTAL_API + rte_thread_unregister(); +#endif + VLOG_INFO("PMD thread released DPDK lcore %u.", lcore_id); } void diff --git a/lib/dpdk.h b/lib/dpdk.h index 736a64279e..404ac1a4bf 100644 --- a/lib/dpdk.h +++ b/lib/dpdk.h @@ -36,7 +36,8 @@ struct smap; struct ovsrec_open_vswitch; void dpdk_init(const struct smap *ovs_other_config); -void dpdk_set_lcore_id(unsigned cpu); +void dpdk_init_thread_context(unsigned cpu); +void dpdk_uninit_thread_context(void); const char *dpdk_get_vhost_sock_dir(void); bool dpdk_vhost_iommu_enabled(void); bool dpdk_vhost_postcopy_enabled(void); diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 1086efd47e..cd7e1569e0 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -5701,7 +5701,7 @@ pmd_thread_main(void *f_) /* Stores the pmd thread's 'pmd' to 'per_pmd_key'. */ ovsthread_setspecific(pmd->dp->per_pmd_key, pmd); ovs_numa_thread_setaffinity_core(pmd->core_id); - dpdk_set_lcore_id(pmd->core_id); + dpdk_init_thread_context(pmd->core_id); poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); dfc_cache_init(&pmd->flow_cache); pmd_alloc_static_tx_qid(pmd); @@ -5821,6 +5821,7 @@ reload: dfc_cache_uninit(&pmd->flow_cache); free(poll_list); pmd_free_cached_ports(pmd); + dpdk_uninit_thread_context(); return NULL; }