From patchwork Fri Nov 4 22:02:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1699885 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EOC+UCVR; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4N3vkd2Czjz23ls for ; Sat, 5 Nov 2022 09:02:49 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1269A40BDF; Fri, 4 Nov 2022 22:02:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 1269A40BDF Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EOC+UCVR X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M3ciYz2LN8RA; Fri, 4 Nov 2022 22:02:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2202440523; Fri, 4 Nov 2022 22:02:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 2202440523 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E4886C0032; Fri, 4 Nov 2022 22:02:44 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6768BC002D for ; Fri, 4 Nov 2022 22:02:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 40C7D40527 for ; Fri, 4 Nov 2022 22:02:43 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 40C7D40527 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Uxp5pUeCsgfT for ; Fri, 4 Nov 2022 22:02:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6FB3E40523 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6FB3E40523 for ; Fri, 4 Nov 2022 22:02:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667599361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dg8bfTYq6KIBcCbHP/iD0/7vIV2lbyUHSG4vZGUfiSY=; b=EOC+UCVRw/Yn2UfWtgapDPoJE2cuFcE6L9gI5OX7VpzJh1uUvqrQRTSM6jn9Xmaj0pBXa4 /jp9Vr6fvGSSOA4SJH5ru9HWFWqtRSBm1DDMw/FqdLSnpn0ycvNrSceUG1a3DEAN9o3u6b SVO+qE7XZ+imV/qmXKgJPLoNr2MRWAU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-265-su8hTjRpNVqyXkxbT48Wqg-1; Fri, 04 Nov 2022 18:02:37 -0400 X-MC-Unique: su8hTjRpNVqyXkxbT48Wqg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 583E7858F13 for ; Fri, 4 Nov 2022 22:02:36 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id ED70E39D7C; Fri, 4 Nov 2022 22:02:34 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 4 Nov 2022 23:02:30 +0100 Message-Id: <20221104220230.781507-1-amorenoz@redhat.com> In-Reply-To: <20220930153027.2601058-1-amorenoz@redhat.com> References: <20220930153027.2601058-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: david.marchand@redhat.com Subject: [ovs-dev] [PATCH v3] ovs-thread: Detect changes in number of cpus X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Currently, things like the number of handler and revalidator threads are calculated based on the number of available CPUs. However, this number is considered static and only calculated once, hence ignoring events such as cpus being hotplugged, switched on/off or affinity mask changing. On the other hand, checking the number of available CPUs multiple times per second seems like an overkill. Affinity should not change that often and, even if it does, the impact of destroying and recreating all the threads so often is probably a price too expensive to pay. This patch makes the number of cpus be calculated every time 5 seconds which seems a reasonable middle point. It generates an impact in the main loop duration of <1% and a worst-case scenario impact in throughput of < 5% [1]. As a result of these changes (assuming the patch is backported): - >=2.16: a change in the cpu affinity reflects on the number of threads in (at most) 5 seconds. - < 2.16: a change in the cpu affinity will be reflected on the number of threads the next time there is a call to bridge_reconfigure() (e.g: on the next DB change), and 5 seconds have passed. The difference in behavior is because on older versions the thread number calculation was done on bridge reconfiguration while newer versions moved this logic down to the dpif layer and is run on dpif->run() stage. Considering it has not been a huge problem up to today and that the cpu change would be reflected sooner or later (e.g the user could force a recalculation with a simple ovs-vsctl command), I think it might be OK to leave like that. [1] Tested in the worst-case scenario of disabling the kernel cache (other_config:flow-size=0), modifying ovs-vswithd's affinity so the number of handlers go up and down every 5 seconds and calculated the difference in netperf's ops/sec. Fixes: be15ec48d766 ("lib: Use a more accurate value for CPU count (sched_getaffinity).") Cc: david.marchand@redhat.com Signed-off-by: Adrian Moreno Acked-by: Mike Pattrick --- lib/ovs-thread.c | 61 +++++++++++++++++++++++++++++------------------- 1 file changed, 37 insertions(+), 24 deletions(-) diff --git a/lib/ovs-thread.c b/lib/ovs-thread.c index 78ed3e970..d1deb9c52 100644 --- a/lib/ovs-thread.c +++ b/lib/ovs-thread.c @@ -31,6 +31,7 @@ #include "openvswitch/poll-loop.h" #include "seq.h" #include "socket-util.h" +#include "timeval.h" #include "util.h" #ifdef __CHECKER__ @@ -627,42 +628,54 @@ ovs_thread_stats_next_bucket(const struct ovsthread_stats *stats, size_t i) } -/* Returns the total number of cores available to this process, or 0 if the - * number cannot be determined. */ -int -count_cpu_cores(void) +static int +count_cpu_cores__(void) { - static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; - static long int n_cores; + long int n_cores; - if (ovsthread_once_start(&once)) { #ifndef _WIN32 - n_cores = sysconf(_SC_NPROCESSORS_ONLN); + n_cores = sysconf(_SC_NPROCESSORS_ONLN); +#else + SYSTEM_INFO sysinfo; + GetSystemInfo(&sysinfo); + n_cores = sysinfo.dwNumberOfProcessors; +#endif #ifdef __linux__ - if (n_cores > 0) { - cpu_set_t *set = CPU_ALLOC(n_cores); + if (n_cores > 0) { + cpu_set_t *set = CPU_ALLOC(n_cores); - if (set) { - size_t size = CPU_ALLOC_SIZE(n_cores); + if (set) { + size_t size = CPU_ALLOC_SIZE(n_cores); - if (!sched_getaffinity(0, size, set)) { - n_cores = CPU_COUNT_S(size, set); - } - CPU_FREE(set); + if (!sched_getaffinity(0, size, set)) { + n_cores = CPU_COUNT_S(size, set); } + CPU_FREE(set); } -#endif -#else - SYSTEM_INFO sysinfo; - GetSystemInfo(&sysinfo); - n_cores = sysinfo.dwNumberOfProcessors; -#endif - ovsthread_once_done(&once); } - +#endif return n_cores > 0 ? n_cores : 0; } +/* It's unlikely that the available cpus change several times per second and + * even if it does, it's not needed (or desired) to react to such changes so + * quickly.*/ +#define COUNT_CPU_UPDATE_TIME_MS 5000 +/* Returns the current total number of cores available to this process, or 0 + * if the number cannot be determined. + * It is assumed that this function is only called from the main thread.*/ +int count_cpu_cores(void) { + static long long int last_updated = 0; + long long int now = time_msec(); + static int cpu_cores; + + if (now - last_updated >= COUNT_CPU_UPDATE_TIME_MS) { + last_updated = now; + cpu_cores = count_cpu_cores__(); + } + return cpu_cores; +} + /* Returns the total number of cores on the system, or 0 if the * number cannot be determined. */ int