From patchwork Wed Jun 1 16:44:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Santana X-Patchwork-Id: 1638043 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=eXe10zhv; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LCw3s6Xlsz9s0w for ; Thu, 2 Jun 2022 02:44:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5337740FF0; Wed, 1 Jun 2022 16:44:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 62AhuIhmLBXh; Wed, 1 Jun 2022 16:44:53 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 68E1740FD3; Wed, 1 Jun 2022 16:44:52 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 35BBDC0032; Wed, 1 Jun 2022 16:44:52 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1FABCC002D for ; Wed, 1 Jun 2022 16:44:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id F164740FCA for ; Wed, 1 Jun 2022 16:44:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Qz7bhcL08Im for ; Wed, 1 Jun 2022 16:44:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1F11740FD3 for ; Wed, 1 Jun 2022 16:44:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1654101888; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FhMmHqKwd3sm7d/hDlPg8XPEAC89si6wvdNdZd0JA+U=; b=eXe10zhv3coWrTFC3PHE9Xdh2g+shKDgQ7ypAu/FPdW8ams3DgYFoqxtK6DVwcAN6SDrpn zPE6LVGpC1m3eCThj7XAn4E1ERxuK+qxkvUFo4v9bdrX098XEOlYE2QuSK4N4szxi6Q4qc p11X8Gq5vWcCgbAYFVT7CtANFH+yRoc= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-240-y4Db8oyJPBK9lkzuE4yOWA-1; Wed, 01 Jun 2022 12:44:46 -0400 X-MC-Unique: y4Db8oyJPBK9lkzuE4yOWA-1 Received: by mail-oi1-f197.google.com with SMTP id d67-20020acab446000000b0032ae7767585so1273651oif.6 for ; Wed, 01 Jun 2022 09:44:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FhMmHqKwd3sm7d/hDlPg8XPEAC89si6wvdNdZd0JA+U=; b=DpTIBFvjt3g7u+vNIwd7xV4PhQefkgOH4WIIxVG0pz/AFV0BoQosHPhzwZGuZH0zGg BgUTRkP+sUoKBalqH0x7LGjmtsF0hFDDuhVvfH6KEclogbnaHiDxxz73Blw4Wrj/pkDx hJ7L2TcEyLfeNBkAwxr1m4zPXqm4e7jdfYT5DCOY1ND7Jgj59m2PDZ2YGQDFY3YBtBlW AzDtKMFY+TPqznMPQ/MjB/AlEtoliPoORwo8w5qaz0rMSIX/TK4Etmtjap9vo+BbpsoB XuOztELfbl3LGXL8/GdlhIO0kqBqdl7CbrUJkjHMcgg9f3JRbpksAHXVdKEz8y3jQJAf wjzA== X-Gm-Message-State: AOAM533vF+QGEcg/tGC8FD7gSxhmOwdVuOkr3gRgOoJ8r4O9MtpmEtBQ 99RLPpxZKOdM1CA/6HIiDH0+iC+4HgDnprUjVCeGeTLBFdpQhoaf3bqdjUY55d8mVACyLwu3ZUZ sRDYp8delNCGeN9WbpFwZRW1HTfclXNPKVNv2YIl00Ir7xBHhQKDngwsYQ7PkEZ5HmovsKA== X-Received: by 2002:a05:6808:300a:b0:2f9:4722:fffc with SMTP id ay10-20020a056808300a00b002f94722fffcmr16271317oib.31.1654101886008; Wed, 01 Jun 2022 09:44:46 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxvc4M/noBPy1T8NUK5J+AElH4ubIq8Oj9uSoPJYDlIA3zuczS2N1F4QGdwq7G08ChLGAVDHQ== X-Received: by 2002:a05:6808:300a:b0:2f9:4722:fffc with SMTP id ay10-20020a056808300a00b002f94722fffcmr16271298oib.31.1654101885641; Wed, 01 Jun 2022 09:44:45 -0700 (PDT) Received: from work.redhat.com (c-73-60-151-108.hsd1.ma.comcast.net. [73.60.151.108]) by smtp.gmail.com with ESMTPSA id i3-20020a9d6103000000b0060603221248sm1008892otj.24.2022.06.01.09.44.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 09:44:45 -0700 (PDT) From: Michael Santana To: ovs-dev@openvswitch.org, i.maximets@ovn.org, Flavio Leitner , Aaron Conole Date: Wed, 1 Jun 2022 12:44:28 -0400 Message-Id: <20220601164429.759054-1-msantana@redhat.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220527171817.662264-1-msantana@redhat.com> References: <20220527171817.662264-1-msantana@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=msantana@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH 1/2] handlers: Create additional handler threads when using CPU isolation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Additional threads are required to service upcalls when we have CPU isolation (in per-cpu dispatch mode). The formula used to calculate the number of handler threads to create is as follows handlers_n = min(next_prime(active_cores+1), total_cores) Where next_prime is a function that returns the next numeric prime number that is strictly greater than active_cores Assume default behavior when total_cores <= 2, that is do not create additional threads when we have less than 2 total cores on the system Fixes: b1e517bd2f81 ("dpif-netlink: Introduce per-cpu upcall dispatch.") Signed-off-by: Michael Santana Acked-by: Aaron Conole --- lib/dpif-netlink.c | 87 ++++++++++++++++++++++++++++++++++++++++++++-- lib/ovs-thread.c | 16 +++++++++ lib/ovs-thread.h | 1 + 3 files changed, 102 insertions(+), 2 deletions(-) diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 06e1e8ca0..355ad1d09 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "bitmap.h" #include "dpif-netlink-rtnl.h" @@ -2506,6 +2507,88 @@ dpif_netlink_handler_uninit(struct dpif_handler *handler) } #endif +/* + * Returns 1 if num is a prime number, + * Otherwise return 0 + */ +static uint32_t +is_prime(uint32_t num) +{ + if ((num < 2) || ((num % 2) == 0)) { + return num == 2; + } + + uint32_t i; + uint32_t limit = sqrt((float) num); + for (i = 3; i <= limit; i += 2) { + if (num % i == 0) { + return 0; + } + } + + return 1; +} + +/* + * Returns num if num is a prime number. Otherwise returns the next + * prime greater than num. Search is limited by the limit variable. + * + * Returns 0 if num >= limit, or if no prime has been found between + * num and limit + */ +static uint32_t +next_prime(uint32_t num, uint32_t limit) +{ + if (num < 2) { + return 2; + } + if (num == 2) { + return 3; + } + if (num % 2 == 0) { + num++; + } + + uint32_t i; + for (i = num; i < limit; i += 2) { + if (is_prime(i)) { + return i; + } + } + + return 0; +} + +/* + * Calcuales and returns the number of handler threads needed based + * the following formula: + * + * handlers_n = min(next_prime(active_cores+1), total_cores) + * + * Where next_prime is a function that returns the next numeric prime + * number that is strictly greater than active_cores + */ +static uint32_t +dpif_netlink_calculate_handlers_num(void) +{ + uint32_t next_prime_num; + uint32_t n_handlers = count_cpu_cores(); + uint32_t total_cores = count_total_cores(); + + /* + * If we have isolated cores, add additional handler threads to + * service inactive cores in the unlikely event that traffic goes + * through inactive cores + */ + if (n_handlers < total_cores && total_cores > 2) { + next_prime_num = next_prime(n_handlers +1, 10000000); + n_handlers = next_prime_num >= total_cores ? + total_cores : next_prime_num; + } + + return n_handlers; +} + static int dpif_netlink_refresh_handlers_cpu_dispatch(struct dpif_netlink *dpif) OVS_REQ_WRLOCK(dpif->upcall_lock) @@ -2515,7 +2598,7 @@ dpif_netlink_refresh_handlers_cpu_dispatch(struct dpif_netlink *dpif) uint32_t n_handlers; uint32_t *upcall_pids; - n_handlers = count_cpu_cores(); + n_handlers = dpif_netlink_calculate_handlers_num(); if (dpif->n_handlers != n_handlers) { VLOG_DBG("Dispatch mode(per-cpu): initializing %d handlers", n_handlers); @@ -2755,7 +2838,7 @@ dpif_netlink_number_handlers_required(struct dpif *dpif_, uint32_t *n_handlers) struct dpif_netlink *dpif = dpif_netlink_cast(dpif_); if (dpif_netlink_upcall_per_cpu(dpif)) { - *n_handlers = count_cpu_cores(); + *n_handlers = dpif_netlink_calculate_handlers_num(); return true; } diff --git a/lib/ovs-thread.c b/lib/ovs-thread.c index 805cba622..2172b3d3f 100644 --- a/lib/ovs-thread.c +++ b/lib/ovs-thread.c @@ -663,6 +663,22 @@ count_cpu_cores(void) return n_cores > 0 ? n_cores : 0; } +/* Returns the total number of cores on the system, or 0 if the + * number cannot be determined. */ +int +count_total_cores(void) { + long int n_cores; + +#ifndef _WIN32 + n_cores = sysconf(_SC_NPROCESSORS_CONF); +#else + n_cores = 0; + errno = ENOTSUP; +#endif + + return n_cores > 0 ? n_cores : 0; +} + /* Returns 'true' if current thread is PMD thread. */ bool thread_is_pmd(void) diff --git a/lib/ovs-thread.h b/lib/ovs-thread.h index 3b444ccdc..aac5e19c9 100644 --- a/lib/ovs-thread.h +++ b/lib/ovs-thread.h @@ -522,6 +522,7 @@ bool may_fork(void); /* Useful functions related to threading. */ int count_cpu_cores(void); +int count_total_cores(void); bool thread_is_pmd(void); #endif /* ovs-thread.h */