From patchwork Fri Jun 4 21:18:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Traynor X-Patchwork-Id: 1488119 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZCq1hAGp; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FxbJ546j2z9sW4 for ; Sat, 5 Jun 2021 07:19:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C092E41DD9; Fri, 4 Jun 2021 21:19:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4sfPmQajOrhD; Fri, 4 Jun 2021 21:19:44 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id 9CBBB41DC2; Fri, 4 Jun 2021 21:19:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 80571C0019; Fri, 4 Jun 2021 21:19:34 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8769BC000D for ; Fri, 4 Jun 2021 21:19:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 66B5D415CB for ; Fri, 4 Jun 2021 21:19:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X_ROHpFBWsWb for ; Fri, 4 Jun 2021 21:19:32 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 7380E406B2 for ; Fri, 4 Jun 2021 21:19:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622841568; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UOmSiDiSe5+kN6Ig4IIsZDXysXZbB+kEEXOJFlgD3Bc=; b=ZCq1hAGpiAShX/rAKcvxK+ZRb8FW8SlqvvIL1KIAxXwvvCe8jVl87/il/AQ5cGOU9IeEXj XM2xo6bCKuoRHd22R+Sa0swt2eVqs0Y6c1oKMTeJ8l8k31LJA+eBfM84bMr1yldNPRCk+Z TJkxcQQETkJwOBTq7ckDqXr5R7jHBM8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-42-QtVG2nhFPa2U0un5JwqTpQ-1; Fri, 04 Jun 2021 17:19:25 -0400 X-MC-Unique: QtVG2nhFPa2U0un5JwqTpQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5A56F501F5; Fri, 4 Jun 2021 21:19:24 +0000 (UTC) Received: from rh.redhat.com (ovpn-114-242.ams2.redhat.com [10.36.114.242]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2B6EF10023AC; Fri, 4 Jun 2021 21:19:23 +0000 (UTC) From: Kevin Traynor To: dev@openvswitch.org Date: Fri, 4 Jun 2021 22:18:56 +0100 Message-Id: <20210604211856.915563-6-ktraynor@redhat.com> In-Reply-To: <20210604211856.915563-1-ktraynor@redhat.com> References: <20210604211856.915563-1-ktraynor@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ktraynor@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: david.marchand@redhat.com Subject: [ovs-dev] [PATCH 5/5] dpif-netdev: Allow pin rxq and non-isolate PMD. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Pinning an rxq to a PMD with pmd-rxq-affinity may be done for various reasons such as reserving a full PMD for an rxq, or to ensure that multiple rxqs from a port are handled on different PMDs. Previously pmd-rxq-affinity always isolated the PMD so no other rxqs could be assigned to it by OVS. There may be cases where there is unused cycles on those pmds and the user would like other rxqs to also be able to be assigned to it be OVS. Add an option to pin the rxq and non-isolate. The default behaviour is unchanged, which is pin and isolate. In order to pin and non-isolate: ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false Note this is available only with group assignment type. Signed-off-by: Kevin Traynor --- Documentation/topics/dpdk/pmd.rst | 9 ++++++-- lib/dpif-netdev.c | 37 +++++++++++++++++++++++++------ vswitchd/vswitch.xml | 19 ++++++++++++++++ 3 files changed, 56 insertions(+), 9 deletions(-) diff --git a/Documentation/topics/dpdk/pmd.rst b/Documentation/topics/dpdk/pmd.rst index 29ba53954..a24a59430 100644 --- a/Documentation/topics/dpdk/pmd.rst +++ b/Documentation/topics/dpdk/pmd.rst @@ -102,6 +102,11 @@ like so: - Queue #3 pinned to core 8 -PMD threads on cores where Rx queues are *pinned* will become *isolated*. This -means that this thread will only poll the *pinned* Rx queues. +PMD threads on cores where Rx queues are *pinned* will become *isolated* by +default. This means that this thread will only poll the *pinned* Rx queues. + +If using ``pmd-rxq-assign=group`` PMD threads with *pinned* Rxqs can be +*non-isolated* by setting:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-isolate=false .. warning:: diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 377573233..cf592a23e 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -375,4 +375,5 @@ struct dp_netdev { /* Use measured cycles for rxq to pmd assignment. */ enum sched_assignment_type pmd_rxq_assign_cyc; + bool pmd_iso; /* Protects the access of the 'struct dp_netdev_pmd_thread' @@ -4370,4 +4371,22 @@ dpif_netdev_set_config(struct dpif *dpif, const struct smap *other_config) } + bool pmd_iso = smap_get_bool(other_config, "pmd-rxq-isolate", true); + + if (pmd_rxq_assign_cyc != SCHED_GROUP && pmd_iso == false) { + /* Invalid combination*/ + VLOG_WARN("pmd-rxq-isolate can only be set false " + "when using pmd-rxq-assign=group"); + pmd_iso = true; + } + if (dp->pmd_iso != pmd_iso) { + dp->pmd_iso = pmd_iso; + if (pmd_iso) { + VLOG_INFO("pmd-rxq-affinity isolates PMD core"); + } else { + VLOG_INFO("pmd-rxq-affinity does not isolate PMD core"); + } + dp_netdev_request_reconfigure(dp); + } + struct pmd_auto_lb *pmd_alb = &dp->pmd_alb; bool cur_rebalance_requested = pmd_alb->auto_lb_requested; @@ -5107,5 +5126,5 @@ sched_numa_list_assignments(struct sched_numa_list *numa_list, sched_pmd = find_sched_pmd_by_pmd(numa_list, rxq->pmd); if (sched_pmd) { - if (rxq->core_id != OVS_CORE_UNSPEC) { + if (rxq->core_id != OVS_CORE_UNSPEC && dp->pmd_iso) { sched_pmd->isolated = true; } @@ -5417,4 +5436,5 @@ sched_numa_list_schedule(struct sched_numa_list *numa_list, struct dp_netdev_pmd_thread *pmd; struct sched_numa *numa; + bool iso = dp->pmd_iso; uint64_t proc_cycles; char rxq_cyc_log[MAX_RXQ_CYC_STRLEN]; @@ -5437,10 +5457,13 @@ sched_numa_list_schedule(struct sched_numa_list *numa_list, continue; } - /* Mark PMD as isolated if not done already. */ - if (sched_pmd->isolated == false) { - sched_pmd->isolated = true; - numa = sched_numa_list_find_numa(numa_list, - sched_pmd); - numa->n_iso++; + /* Check if isolating PMDs with pinned rxqs.*/ + if (iso) { + /* Mark PMD as isolated if not done already. */ + if (sched_pmd->isolated == false) { + sched_pmd->isolated = true; + numa = sched_numa_list_find_numa(numa_list, + sched_pmd); + numa->n_iso++; + } } proc_cycles = dp_netdev_rxq_get_cycles(rxq, diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 14cb8a2c6..dca334961 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -545,4 +545,23 @@ + +

+ Specifies if a CPU core will be isolated after being pinned with + an Rx queue. +

+ Set this value to false to non-isolate a CPU core after + it is pinned with an Rxq using pmd-rxq-affinity. This + will allow OVS to assign other Rxqs to that CPU core. +

+

+ The default value is true. +

+

+ This can only be false when pmd-rxq-assign + is set to group. +

+ +