From patchwork Tue Jan 16 22:52:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1887227 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=hTTMnHGh; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TF45b0cCmz23dx for ; Wed, 17 Jan 2024 09:52:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 4C81143575; Tue, 16 Jan 2024 22:52:16 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 4C81143575 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=hTTMnHGh X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RpVxR_Hl1F0D; Tue, 16 Jan 2024 22:52:14 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id A076E43555; Tue, 16 Jan 2024 22:52:13 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org A076E43555 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A49ABC0DD6; Tue, 16 Jan 2024 22:52:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id EEA84C0077 for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C7B6241C5A for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org C7B6241C5A Authentication-Results: smtp4.osuosl.org; dkim=pass (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=hTTMnHGh X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KzZBGJx1_8tc for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp4.osuosl.org (Postfix) with ESMTPS id CA8AA41C2D for ; Tue, 16 Jan 2024 22:52:08 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org CA8AA41C2D Received: from fnordahl.lxd (ti0189a430-2338.bb.online.no [88.90.42.44]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 61BD9413C0; Tue, 16 Jan 2024 22:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1705445526; bh=+RA1oiH/kfSS8JUhEU4RMZYAPIwKuqWh2W49EvsRc+E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hTTMnHGh2qdq+nLhMIg6fBq5AD4qF/nK7RRE5ydxoaSUmSKutEu1YkFAF8GrJGWp6 53z/jquL2scoLv7Phnx+T0+7mJmqbEJGhH2geOXFJqeqltxxpSDQqMnWfp5NsFQ3N3 jCISnT0fOKZ67NM9ux9sz/nLKppcXwC+A5pNm33menssancTitBruYhqpkovCQ39fu hJjmf8Sx2QP2VTualIjLb8NWuiJLRexeKUMn+jzhjQNtAmVMV42eb2yuLF8Ukzn7aV mN52SBKpp8nvJBeOxNijjEG85olS8PNGZr3EBxb+Mkjn6aK5dYpv/rdtU33JXEOVZW lkDDBJwYKvnCg== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 16 Jan 2024 22:52:01 +0000 Message-Id: <20240116225205.38112-2-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240116225205.38112-1-frode.nordahl@canonical.com> References: <20240116225205.38112-1-frode.nordahl@canonical.com> MIME-Version: 1.0 Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 1/5] timeval: Add internal timewarp interface. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It may be desirable to make use of time warp functionality in unit tests. Separate logic from time/stop unixctl into timeval_stop() and add a new timeval_warp() interface for directing monotonic clock into slow path and advancing the current monotonic directly. This will be used in a patch that implements unit tests for the cooperative multitasking module. Signed-off-by: Frode Nordahl --- lib/timeval.c | 28 ++++++++++++++++++++++++---- lib/timeval.h | 3 +++ 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/lib/timeval.c b/lib/timeval.c index 0abe7e555..10c1b9ca1 100644 --- a/lib/timeval.c +++ b/lib/timeval.c @@ -767,17 +767,22 @@ get_cpu_usage(void) /* "time/stop" stops the monotonic time returned by e.g. time_msec() from * advancing, except due to later calls to "time/warp". */ -static void -timeval_stop_cb(struct unixctl_conn *conn, - int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, - void *aux OVS_UNUSED) +void +timeval_stop(void) { ovs_mutex_lock(&monotonic_clock.mutex); atomic_store_relaxed(&monotonic_clock.slow_path, true); monotonic_clock.stopped = true; xclock_gettime(monotonic_clock.id, &monotonic_clock.cache); ovs_mutex_unlock(&monotonic_clock.mutex); +} +static void +timeval_stop_cb(struct unixctl_conn *conn, + int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, + void *aux OVS_UNUSED) +{ + timeval_stop(); unixctl_command_reply(conn, NULL); } @@ -818,6 +823,21 @@ timeval_warp_cb(struct unixctl_conn *conn, timewarp_work(); } +/* Direct monotonic clock into slow path and advance the current monotonic + * time by 'msecs' milliseconds directly. This is for use in unit tests. */ +void +timeval_warp(long long int msecs) +{ + struct clock *c = &monotonic_clock; + struct timespec warp; + + ovs_mutex_lock(&monotonic_clock.mutex); + atomic_store_relaxed(&monotonic_clock.slow_path, true); + msec_to_timespec(msecs, &warp); + timespec_add(&c->warp, &c->warp, &warp); + ovs_mutex_unlock(&monotonic_clock.mutex); +} + void timeval_dummy_register(void) { diff --git a/lib/timeval.h b/lib/timeval.h index 502f703d4..1c40530e2 100644 --- a/lib/timeval.h +++ b/lib/timeval.h @@ -81,6 +81,9 @@ long long int time_boot_msec(void); void timewarp_run(void); +void timeval_stop(void); +void timeval_warp(long long int msecs); + #ifdef __cplusplus } #endif From patchwork Tue Jan 16 22:52:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1887230 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Q/hrZJrr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TF45k035Sz23dx for ; Wed, 17 Jan 2024 09:52:26 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3E87983AA5; Tue, 16 Jan 2024 22:52:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 3E87983AA5 Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Q/hrZJrr X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RHv2nXfibQFT; Tue, 16 Jan 2024 22:52:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 33BD283926; Tue, 16 Jan 2024 22:52:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 33BD283926 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 35DC0C0DE1; Tue, 16 Jan 2024 22:52:14 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1D011C0DD6 for ; Tue, 16 Jan 2024 22:52:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id E32FA41C5A for ; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org E32FA41C5A Authentication-Results: smtp4.osuosl.org; dkim=pass (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Q/hrZJrr X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N_9RLxv6mvf9 for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp4.osuosl.org (Postfix) with ESMTPS id EE53841C59 for ; Tue, 16 Jan 2024 22:52:08 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org EE53841C59 Received: from fnordahl.lxd (ti0189a430-2338.bb.online.no [88.90.42.44]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id C1028413D5; Tue, 16 Jan 2024 22:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1705445526; bh=sixR/Ap/a2LxPPDLYtKt9xvSvkehFq0p9ALLj3zQ1E0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q/hrZJrrfoUhbC8iLCaQMs3SEmV9tQLL7quZMjlCo6p8U5MBWFvbFkdMGoDT8fcHK dwpF+KP6+ryme4+hspBxemTuG9nSPpqgEqG2iN2kjD9jXR3/uVdWhwhVmlZPCRSScU dJsgdiHu41nJg7uT3FW3xcYE/hv01rp/l6lepzRaT0CzNvx8UAc0bdZkPvMahsmQnt xvlUzEGAMdlEpux1gLhe1qFXwmyhQ7ko2bT92ffJZOVFUmZlEkYILjVxBiiHW+HUuc 7PQZYUlORjTxq0/PZehMvwNXLFM6v25JzdhvX/f44zEUb1QZtqugzBx4GU/luyfCoL yF0ujKJcDInRA== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 16 Jan 2024 22:52:02 +0000 Message-Id: <20240116225205.38112-3-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240116225205.38112-1-frode.nordahl@canonical.com> References: <20240116225205.38112-1-frode.nordahl@canonical.com> MIME-Version: 1.0 Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 2/5] lib: Introduce cooperative multitasking module. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" One of the goals of Open vSwitch is to be as resource efficient as possible. Core parts of the program has been implemented as asynchronous state machines, and when absolutely necessary additional threads are used. Introduce cooperative multitasking module which allow us to interleave important processing with long running tasks while avoiding the additional resource consumption of threads and complexity of asynchronous state machines. We will use this module to ensure long running processing in the OVSDB server does not interfere with stable maintenance of the RAFT cluster in subsequent patches. Suggested-by: Ilya Maximets Signed-off-by: Frode Nordahl --- lib/automake.mk | 3 + lib/cooperative-multitasking-private.h | 33 +++ lib/cooperative-multitasking.c | 157 +++++++++++++ lib/cooperative-multitasking.h | 115 +++++++++ tests/automake.mk | 1 + tests/library.at | 10 + tests/ovsdb-server.at | 1 + tests/test-cooperative-multitasking.c | 307 +++++++++++++++++++++++++ 8 files changed, 627 insertions(+) create mode 100644 lib/cooperative-multitasking-private.h create mode 100644 lib/cooperative-multitasking.c create mode 100644 lib/cooperative-multitasking.h create mode 100644 tests/test-cooperative-multitasking.c diff --git a/lib/automake.mk b/lib/automake.mk index 0dc8a35cc..8596171c6 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -94,6 +94,9 @@ lib_libopenvswitch_la_SOURCES = \ lib/conntrack-other.c \ lib/conntrack.c \ lib/conntrack.h \ + lib/cooperative-multitasking.c \ + lib/cooperative-multitasking.h \ + lib/cooperative-multitasking-private.h \ lib/coverage.c \ lib/coverage.h \ lib/cpu.c \ diff --git a/lib/cooperative-multitasking-private.h b/lib/cooperative-multitasking-private.h new file mode 100644 index 000000000..cb8382377 --- /dev/null +++ b/lib/cooperative-multitasking-private.h @@ -0,0 +1,33 @@ +/* + * Copyright (c) 2024 Canonical Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef COOPERATIVE_MULTITASKING_PRIVATE_H +#define COOPERATIVE_MULTITASKING_PRIVATE_H 1 + +#include "openvswitch/hmap.h" + +extern struct hmap cooperative_multitasking_callbacks; + +struct cm_entry { + struct hmap_node node; + void (*cb)(void *); + void *arg; + long long int threshold; + long long int last_run; + const char *name; +}; + +#endif /* COOPERATIVE_MULTITASKING_PRIVATE_H */ diff --git a/lib/cooperative-multitasking.c b/lib/cooperative-multitasking.c new file mode 100644 index 000000000..3a91af26f --- /dev/null +++ b/lib/cooperative-multitasking.c @@ -0,0 +1,157 @@ +/* + * Copyright (c) 2024 Canonical Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "backtrace.h" +#include "cooperative-multitasking-private.h" +#include "cooperative-multitasking.h" +#include "hash.h" +#include "openvswitch/hmap.h" +#include "openvswitch/vlog.h" +#include "timeval.h" + +VLOG_DEFINE_THIS_MODULE(cooperative_multitasking); + +struct hmap cooperative_multitasking_callbacks = HMAP_INITIALIZER( + &cooperative_multitasking_callbacks); + +/* Free any data allocated by calls to cooperative_multitasking_set(). */ +void +cooperative_multitasking_destroy(void) +{ + struct cm_entry *cm_entry; + HMAP_FOR_EACH_SAFE (cm_entry, node, &cooperative_multitasking_callbacks) { + hmap_remove(&cooperative_multitasking_callbacks, &cm_entry->node); + free(cm_entry); + } +} + +/* Set/update callback as identified by 'cb' and 'arg'. + * + * 'name' is used for logging events related to this callback. + * + * The value for 'last_run' must be updated each time the callback is run. + * + * Updating the value for 'threshold' may be necessary as a consequence of + * change in runtime configuration or requirements of the part of the program + * serviced by the callback. + * + * Providing a value of 0 for 'last_run' or 'threshold' will leave the stored + * value untouched. */ +void +cooperative_multitasking_set(void (*cb)(void *), void *arg, + long long int last_run, long long int threshold, + const char *name) +{ + struct cm_entry *cm_entry; + + HMAP_FOR_EACH_WITH_HASH (cm_entry, node, hash_pointer((void *) cb, 0), + &cooperative_multitasking_callbacks) { + if (cm_entry->cb == cb && cm_entry->arg == arg) { + if (last_run) { + cm_entry->last_run = last_run; + } + + if (threshold) { + cm_entry->threshold = threshold; + } + return; + } + } + + cm_entry = xzalloc(sizeof *cm_entry); + cm_entry->cb = cb; + cm_entry->arg = arg; + cm_entry->threshold = threshold; + cm_entry->last_run = last_run ? last_run : time_msec(); + cm_entry->name = name; + + hmap_insert(&cooperative_multitasking_callbacks, + &cm_entry->node, hash_pointer((void *) cm_entry->cb, 0)); +} + +/* Remove callback identified by 'cb' and 'arg'. */ +void +cooperative_multitasking_remove(void (*cb)(void *), void *arg) +{ + struct cm_entry *cm_entry; + + HMAP_FOR_EACH_WITH_HASH (cm_entry, node, hash_pointer((void *) cb, 0), + &cooperative_multitasking_callbacks) { + if (cm_entry->cb == cb && cm_entry->arg == arg) { + hmap_remove(&cooperative_multitasking_callbacks, &cm_entry->node); + free(cm_entry); + return; + } + } +} + +static void +cooperative_multitasking_yield_at__(const char *source_location) +{ + long long int start = time_msec(); + struct cm_entry *cm_entry; + long long int elapsed; + bool warn; + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + elapsed = time_msec() - cm_entry->last_run; + + if (elapsed >= cm_entry->threshold) { + warn = elapsed - cm_entry->threshold > cm_entry->threshold / 8; + + VLOG(warn ? VLL_WARN : VLL_DBG, "%s: yield for %s(%p): " + "elapsed(%lld) >= threshold(%lld), overrun: %lld", + source_location, cm_entry->name, cm_entry->arg, elapsed, + cm_entry->threshold, elapsed - cm_entry->threshold); + + if (warn && VLOG_IS_DBG_ENABLED()) { + log_backtrace(); + } + + (*cm_entry->cb)(cm_entry->arg); + } + } + + elapsed = time_msec() - start; + if (elapsed > 1000) { + VLOG_WARN("Unreasonably long %lldms runtime for callbacks.", elapsed); + } +} + +/* Iterate over registered callbacks and execute callbacks as demanded by the + * recorded time threshold. */ +void +cooperative_multitasking_yield_at(const char *source_location) +{ + static bool yield_in_progress = false; + + if (yield_in_progress) { + VLOG_ERR_ONCE("Nested yield avoided, this is a bug! " + "Enable debug logging for more details."); + if (VLOG_IS_DBG_ENABLED()) { + VLOG_DBG("%s: nested yield.", source_location); + log_backtrace(); + } + return; + } + yield_in_progress = true; + + cooperative_multitasking_yield_at__(source_location); + + yield_in_progress = false; +} diff --git a/lib/cooperative-multitasking.h b/lib/cooperative-multitasking.h new file mode 100644 index 000000000..1bbb59944 --- /dev/null +++ b/lib/cooperative-multitasking.h @@ -0,0 +1,115 @@ +/* + * Copyright (c) 2024 Canonical Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef COOPERATIVE_MULTITASKING_H +#define COOPERATIVE_MULTITASKING_H 1 + +/* + * cooperative-multitasking, interleaved execution for Open vSwitch. + * + * Overview + * ======== + * + * One of the goals of Open vSwitch is to be as resource efficient as + * possible. Core parts of the program has been implemented as asynchronous + * state machines, and when absolutely necessary additional threads are used. + * + * Modules with mostly synchronous and single threaded code that are expected + * to have heavy processing, can make use of the cooperative-multitasking + * interface to yield to modules that have registered callbacks at a time + * threshold. + * + * Typical Usage + * ============= + * + * The module that provides the callback typically has a run() function that is + * already part of the main processing loop and can then register like this: + * + * static void my_run_cb(void *arg); + * + * static void + * my_run(struct data *my_data) + * { + * ... + * + * cooperative_multitasking_set(&my_run_cb, (void *) my_data, + * time_msec(), 1000, "my_run"); + * } + * + * static void + * my_run_cb (void *arg) + * { + * struct data *my_data = (struct data *) arg; + * + * my_run(my_data); + * } + * + * static void + * my_destroy(struct data *my_data) + * { + * ... + * + * cooperatrive_multitasking_remove(&my_run_cb, (void *) my_data); + * } + * + * The module that is expected to have heavy processing can yield like this: + * + * HMAP_FOR_EACH (row, hmap_node, &src_table->rows) { + * cooperative_multitasking_yield(); + * + * ... + * } + * + * Rules for implementation + * ======================== + * + * - The module that registers itself with a callback must not use the yield + * functionality inside nor should it be possible to do so via calls to other + * modules. + * + * - The module that registers the callback should be self-sufficient, i.e. + * the internal state of that module should not matter to the outside world, + * at least it should not matter for the call stack that enters the + * cooperative_multitasking_yield(). + * + * - cooperative_multitasking_yield() must not be called from places that can + * loop indefinitely, only in places that eventually end, otherwise it may + * give a false impression that the server is working fine while it is stuck + * and not actually doing any useful work. + * + * Thread-safety + * ============= + * + * The cooperative-multitasking module and functions therein are not thread + * safe and must only be used by one thread. + */ + +struct hmap; + +void cooperative_multitasking_destroy(void); + +void cooperative_multitasking_set(void (*cb)(void *), void *arg, + long long int last_run, + long long int threshold, + const char *name); + +void cooperative_multitasking_remove(void (*cb)(void *), void *arg); + +void cooperative_multitasking_yield_at(const char *source_location); +#define cooperative_multitasking_yield() \ + cooperative_multitasking_yield_at(OVS_SOURCE_LOCATOR) + +#endif /* COOPERATIVE_MULTITASKING_H */ diff --git a/tests/automake.mk b/tests/automake.mk index 10c9fbb01..08c9b74d4 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -456,6 +456,7 @@ tests_ovstest_SOURCES = \ tests/test-ccmap.c \ tests/test-cmap.c \ tests/test-conntrack.c \ + tests/test-cooperative-multitasking.c \ tests/test-csum.c \ tests/test-flows.c \ tests/test-hash.c \ diff --git a/tests/library.at b/tests/library.at index 3f9df2f87..7b4acebb8 100644 --- a/tests/library.at +++ b/tests/library.at @@ -296,3 +296,13 @@ AT_CLEANUP AT_SETUP([uuidset module]) AT_CHECK([ovstest test-uuidset], [0], [], [ignore]) AT_CLEANUP + +AT_SETUP([cooperative-multitasking module]) +AT_CHECK([ovstest test-cooperative-multitasking], [0], []) +AT_CLEANUP + +AT_SETUP([cooperative-multitasking module nested yield detection]) +AT_CHECK([ovstest test-cooperative-multitasking-nested-yield], [0], [], [dnl +cooperative_multitasking|ERR|Nested yield avoided, this is a bug! Enable debug logging for more details. +]) +AT_CLEANUP diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index 036e4cc3b..e152a153c 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -2845,6 +2845,7 @@ m4_define([CLEAN_LOG_FILE], [sed 's/[[0-9\-]]*T[[0-9:\.]]*Z|[[0-9]]*\(|.*$\)/\1/g' $1 | dnl sed '/|poll_loop|/d' | dnl sed '/|socket_util|/d' | dnl + sed '/|cooperative_multitasking|DBG|/d' | dnl sed 's/[[0-9]]*\.ctl/\.ctl/g'> $2]) CLEAN_LOG_FILE([1.log], [1.log.clear]) diff --git a/tests/test-cooperative-multitasking.c b/tests/test-cooperative-multitasking.c new file mode 100644 index 000000000..f7407bb03 --- /dev/null +++ b/tests/test-cooperative-multitasking.c @@ -0,0 +1,307 @@ +/* + * Copyright (c) 2023 Canonical Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#undef NDEBUG +#include "cooperative-multitasking.h" +#include "cooperative-multitasking-private.h" +#include "openvswitch/hmap.h" +#include "ovstest.h" +#include "timeval.h" +#include "util.h" +#include "openvswitch/vlog.h" + +struct fixture_arg { + bool called; +}; + +static void fixture_run_wrap(void *arg); + +#define FIXTURE_RUN_NAME "fixture_run" + +static void +fixture_run(struct fixture_arg *arg) +{ + cooperative_multitasking_set(&fixture_run_wrap, (void *) arg, + time_msec(), 0, FIXTURE_RUN_NAME); + if (arg) { + arg->called = true; + } +} + +static void +fixture_run_wrap(void *arg) +{ + struct fixture_arg *fixture_arg = (struct fixture_arg *) arg; + + fixture_run(fixture_arg); +} + + +static void fixture_other_run_wrap(void *arg); + +#define FIXTURE_OTHER_RUN_NAME "fixture_other_run" + +static void +fixture_other_run(struct fixture_arg *arg) +{ + cooperative_multitasking_set(&fixture_other_run_wrap, (void *) arg, + time_msec(), 0, FIXTURE_OTHER_RUN_NAME); + if (arg) { + arg->called = true; + } +} + +static void +fixture_other_run_wrap(void *arg) +{ + struct fixture_arg *fixture_arg = (struct fixture_arg *) arg; + + fixture_other_run(fixture_arg); +} + +static void +test_cm_set_registration(void) +{ + struct cm_entry *cm_entry; + struct fixture_arg arg1 = { + .called = false, + }; + struct fixture_arg arg2 = { + .called = false, + }; + + timeval_stop(); + long long int now = time_msec(); + + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg1, 0, 1000, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg2, 0, 2000, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_other_run_wrap, NULL, 0, 3000, + FIXTURE_OTHER_RUN_NAME); + + ovs_assert(hmap_count(&cooperative_multitasking_callbacks) == 3); + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + if (cm_entry->arg == (void *) &arg1) { + ovs_assert(cm_entry->cb == &fixture_run_wrap); + ovs_assert(cm_entry->threshold == 1000); + ovs_assert(cm_entry->last_run == now); + } else if (cm_entry->arg == (void *) &arg2) { + ovs_assert(cm_entry->cb == &fixture_run_wrap); + ovs_assert(cm_entry->threshold == 2000); + ovs_assert(cm_entry->last_run == now); + } else if (cm_entry->cb == &fixture_other_run_wrap) { + ovs_assert(cm_entry->arg == NULL); + ovs_assert(cm_entry->threshold == 3000); + ovs_assert(cm_entry->last_run == now); + } else { + OVS_NOT_REACHED(); + } + } + + cooperative_multitasking_remove(&fixture_other_run_wrap, NULL); + ovs_assert(hmap_count(&cooperative_multitasking_callbacks) == 2); + cooperative_multitasking_remove(&fixture_run_wrap, (void *) &arg2); + ovs_assert(hmap_count(&cooperative_multitasking_callbacks) == 1); + + cooperative_multitasking_destroy(); +} + +static void +test_cm_set_update(void) +{ + struct cm_entry *cm_entry; + struct fixture_arg arg1 = { + .called = false, + }; + struct fixture_arg arg2 = { + .called = false, + }; + + timeval_stop(); + long long int now = time_msec(); + + /* First register a couple of callbacks. */ + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg1, 0, 0, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg2, 0, 0, + FIXTURE_RUN_NAME); + + ovs_assert(hmap_count(&cooperative_multitasking_callbacks) == 2); + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + if (cm_entry->arg == (void *) &arg1) { + ovs_assert(cm_entry->threshold == 0); + ovs_assert(cm_entry->last_run == now); + } else if (cm_entry->arg == (void *) &arg2) { + ovs_assert(cm_entry->threshold == 0); + ovs_assert(cm_entry->last_run == now); + } else { + OVS_NOT_REACHED(); + } + } + + /* Update 'last_run' and 'threshold' for each callback and validate + * that the correct entry was actually updated. */ + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg1, 1, 2, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg2, 3, 4, + FIXTURE_RUN_NAME); + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + if (cm_entry->arg == (void *) &arg1) { + ovs_assert(cm_entry->threshold == 2); + ovs_assert(cm_entry->last_run == 1); + } else if (cm_entry->arg == (void *) &arg2) { + ovs_assert(cm_entry->threshold == 4); + ovs_assert(cm_entry->last_run == 3); + } else { + OVS_NOT_REACHED(); + } + } + + /* Confirm that providing 0 for 'last_run' or 'threshold' leaves the + * existing value untouched. */ + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg1, 0, 5, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg2, 6, 0, + FIXTURE_RUN_NAME); + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + if (cm_entry->arg == (void *) &arg1) { + ovs_assert(cm_entry->threshold == 5); + ovs_assert(cm_entry->last_run == 1); + } else if (cm_entry->arg == (void *) &arg2) { + ovs_assert(cm_entry->threshold == 4); + ovs_assert(cm_entry->last_run == 6); + } else { + OVS_NOT_REACHED(); + } + } + + cooperative_multitasking_destroy(); +} + +static void +test_cm_yield(void) +{ + struct cm_entry *cm_entry; + struct fixture_arg arg1 = { + .called = false, + }; + struct fixture_arg arg2 = { + .called = false, + }; + + timeval_stop(); + long long int now = time_msec(); + + /* First register a couple of callbacks. */ + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg1, 0, 1000, + FIXTURE_RUN_NAME); + cooperative_multitasking_set(&fixture_run_wrap, (void *) &arg2, 0, 2000, + FIXTURE_RUN_NAME); + + ovs_assert(hmap_count(&cooperative_multitasking_callbacks) == 2); + + /* Call to yield should not execute callbacks until time threshold. */ + cooperative_multitasking_yield(); + ovs_assert(arg1.called == false); + ovs_assert(arg2.called == false); + + HMAP_FOR_EACH (cm_entry, node, &cooperative_multitasking_callbacks) { + ovs_assert(cm_entry->last_run == now); + } + + /* Move clock forward and confirm the expected callbacks to be executed. */ + timeval_warp(1000); + timeval_stop(); + cooperative_multitasking_yield(); + ovs_assert(arg1.called == true); + ovs_assert(arg2.called == false); + + /* Move clock forward and confirm the expected callbacks to be executed. */ + arg1.called = arg2.called = false; + timeval_warp(1000); + timeval_stop(); + cooperative_multitasking_yield(); + ovs_assert(arg1.called == true); + ovs_assert(arg2.called == true); + + cooperative_multitasking_destroy(); +} + +static void fixture_buggy_run_wrap(void *arg); + +#define FIXTURE_BUGGY_RUN_NAME "fixture_buggy_run" + +static void +fixture_buggy_run(struct fixture_arg *arg) +{ + cooperative_multitasking_set(&fixture_buggy_run_wrap, (void *) arg, + time_msec(), 0, FIXTURE_BUGGY_RUN_NAME); + if (arg) { + arg->called = true; + } + /* A real run function MUST NOT directly or indirectly call yield, this is + * here to test the detection of such a programming error. */ + cooperative_multitasking_yield(); +} + +static void +fixture_buggy_run_wrap(void *arg) +{ + struct fixture_arg *fixture_arg = (struct fixture_arg *) arg; + + fixture_buggy_run(fixture_arg); +} + +static void +test_cooperative_multitasking_nested_yield(int argc OVS_UNUSED, char *argv[]) +{ + struct fixture_arg arg1 = { + .called = false, + }; + + set_program_name(argv[0]); + vlog_set_pattern(VLF_CONSOLE, "%c|%p|%m"); + vlog_set_levels(NULL, VLF_SYSLOG, VLL_OFF); + + time_msec(); /* Ensure timeval is initialized. */ + + cooperative_multitasking_set(&fixture_buggy_run_wrap, (void *) &arg1, + 0, 1000, FIXTURE_BUGGY_RUN_NAME); + timeval_warp(1000); + cooperative_multitasking_yield(); + cooperative_multitasking_destroy(); +} + +static void +test_cooperative_multitasking(int argc OVS_UNUSED, char *argv[] OVS_UNUSED) +{ + time_msec(); /* Ensure timeval is initialized. */ + + test_cm_set_registration(); + test_cm_set_update(); + test_cm_yield(); +} + +OVSTEST_REGISTER("test-cooperative-multitasking", + test_cooperative_multitasking); +OVSTEST_REGISTER("test-cooperative-multitasking-nested-yield", + test_cooperative_multitasking_nested_yield); From patchwork Tue Jan 16 22:52:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1887226 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=kK91tD5N; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TF45X6hM0z23dx for ; Wed, 17 Jan 2024 09:52:15 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D46D341D92; Tue, 16 Jan 2024 22:52:12 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D46D341D92 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=kK91tD5N X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HaNGKaNhLYa4; Tue, 16 Jan 2024 22:52:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id E76DB41D82; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org E76DB41D82 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A4C06C0077; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id E0495C0037 for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id BBBBA60C06 for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org BBBBA60C06 Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=kK91tD5N X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HcOktWWWfs1n for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp3.osuosl.org (Postfix) with ESMTPS id E620B607E1 for ; Tue, 16 Jan 2024 22:52:08 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org E620B607E1 Received: from fnordahl.lxd (ti0189a430-2338.bb.online.no [88.90.42.44]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 182EB416D2; Tue, 16 Jan 2024 22:52:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1705445527; bh=vlc06hJY48sPk/Nw/wCcp03uvWOfjmhIH7TRu0cUcB8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kK91tD5NKX+uLzO+1JL8mfOFLdTakF+5dBA3xnQ54HzOFwLCyLkF8GHhN/6rNSwSf +cZyiJoCWPzzcUlrfCKiaFQoaI02l1UK1SLAM7InLDd7na3iUmPKW1uGIngt8cTzcI FBH+Mp3aftZ8Q0gWo3UOvuwV54N/i9jQo+r1TTowHoD4vy2/kad9P/JVG0tVZqRoz+ 4FUVe2bawcqLbKofBXybbzx3pMkeFI1v7cCtaTunHRv5C/6lKejFX718hsp2P58MxD /k8/u87IasMQA3H4InWs48f0yvApcTyGSuZ9NduGfBgnDIj/LWbXTlcT+LNC9cEEZq fz1s1gwro6SuA== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 16 Jan 2024 22:52:03 +0000 Message-Id: <20240116225205.38112-4-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240116225205.38112-1-frode.nordahl@canonical.com> References: <20240116225205.38112-1-frode.nordahl@canonical.com> MIME-Version: 1.0 Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 3/5] ovsdb raft: Enable cooperative multitasking. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The OVSDB server is mostly synchronous and single threaded. The OVSDB RAFT storage engine operate under strict deadlines with operational impact should the deadline be overrun. Register for cooperative multitasking so that long running processing elsewhere in the program may yield to allow stable maintenance of the cluster. Signed-off-by: Frode Nordahl --- ovsdb/raft.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/ovsdb/raft.c b/ovsdb/raft.c index 8effd9ad1..f463afcb3 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -22,6 +22,7 @@ #include #include +#include "cooperative-multitasking.h" #include "hash.h" #include "jsonrpc.h" #include "lockfile.h" @@ -993,10 +994,13 @@ raft_reset_election_timer(struct raft *raft) raft->election_timeout = raft->election_base + duration; } +#define RAFT_TIMER_THRESHOLD(t) (t / 3) + static void raft_reset_ping_timer(struct raft *raft) { - raft->ping_timeout = time_msec() + raft->election_timer / 3; + raft->ping_timeout = + time_msec() + RAFT_TIMER_THRESHOLD(raft->election_timer); } static void @@ -1371,6 +1375,8 @@ raft_take_leadership(struct raft *raft) } } +static void raft_run_cb(void *arg); + /* Closes everything owned by 'raft' that might be visible outside the process: * network connections, commands, etc. This is part of closing 'raft'; it is * also used if 'raft' has failed in an unrecoverable way. */ @@ -1397,6 +1403,8 @@ raft_close__(struct raft *raft) LIST_FOR_EACH_SAFE (conn, list_node, &raft->conns) { raft_conn_close(conn); } + + cooperative_multitasking_remove(&raft_run_cb, raft); } /* Closes and frees 'raft'. @@ -2114,6 +2122,11 @@ raft_run(struct raft *raft) raft_reset_ping_timer(raft); } + cooperative_multitasking_set( + &raft_run_cb, (void *) raft, time_msec(), + RAFT_TIMER_THRESHOLD(raft->election_timer) + + RAFT_TIMER_THRESHOLD(raft->election_timer) / 10, "raft_run"); + /* Do this only at the end; if we did it as soon as we set raft->left or * raft->failed in handling the RemoveServerReply, then it could easily * cause references to freed memory in RPC sessions, etc. */ @@ -2122,6 +2135,14 @@ raft_run(struct raft *raft) } } +static void +raft_run_cb(void *arg) +{ + struct raft *raft = (struct raft *) arg; + + raft_run(raft); +} + static void raft_wait_session(struct jsonrpc_session *js) { From patchwork Tue Jan 16 22:52:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1887229 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=vAvYqy+0; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TF45f1S4jz23dx for ; Wed, 17 Jan 2024 09:52:22 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D37FF43581; Tue, 16 Jan 2024 22:52:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D37FF43581 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=vAvYqy+0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Afub-QQfrEKJ; Tue, 16 Jan 2024 22:52:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 817634357D; Tue, 16 Jan 2024 22:52:16 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 817634357D Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3FD87C007C; Tue, 16 Jan 2024 22:52:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id F1628C0DD0 for ; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A52BC8308D for ; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A52BC8308D Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=vAvYqy+0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id divYzmEKtDJE for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp1.osuosl.org (Postfix) with ESMTPS id 83D4C8291D for ; Tue, 16 Jan 2024 22:52:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 83D4C8291D Received: from fnordahl.lxd (ti0189a430-2338.bb.online.no [88.90.42.44]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 5BACC41BB1; Tue, 16 Jan 2024 22:52:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1705445527; bh=iRKfkhtdYV0lJETGN0l29CEjtd4ed7+RfsSp+ACLlbA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=vAvYqy+0sqfG39vD57AjgCGx/GzqYJITwzXxJ6EYYD1Yb0QkDB6+dUQ3RXz7lNq0J H7s7M9ZKVxAswY7yzr5RSSgnImwiOqtJv3MOjSiuZuB/Rm9gul2hJJLy+JwsGcltnb R5Zpu4nE+XBjkgL0aKTXK9lVqW9oDG32vBJ1mWYUA8u7iCmbNwKeZodR/kXikp2H41 YOIokd+Upig0RwhtLLtwN2K0uU+GXRdDenVSx2w9ER0m3fxHABSdF0jbAAmHbqRRVh Ae3yrizcBdGWARaE57MUxvuQnVkmxtbAUfB+U8sBW8+cDBsD/zMBX1LV4eQMHgmoir HWtPpCSEo4g3g== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 16 Jan 2024 22:52:04 +0000 Message-Id: <20240116225205.38112-5-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240116225205.38112-1-frode.nordahl@canonical.com> References: <20240116225205.38112-1-frode.nordahl@canonical.com> MIME-Version: 1.0 Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 4/5] json: Add yielding json create/destroy functions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Creating and destroying JSON objects may be time consuming. Add json_serialized_object_create_with_yield() and json_destroy_with_yield() functions that make use of the cooperative multitasking module to yield during processing, allowing time sensitive tasks in other parts of the program to be completed during processing. We keep these new functions private to OVS by adding a new lib/json.h header file. The include guard in the public include/openvswitch/json.h is updated to contain the OPENVSWITCH prefix to be in line with the other public header files, allowing us to use the non-prefixed version in our private lib/json.h. Signed-off-by: Frode Nordahl --- include/openvswitch/json.h | 10 +++---- lib/automake.mk | 1 + lib/json.c | 58 ++++++++++++++++++++++++++++++++------ lib/json.h | 32 +++++++++++++++++++++ 4 files changed, 87 insertions(+), 14 deletions(-) create mode 100644 lib/json.h diff --git a/include/openvswitch/json.h b/include/openvswitch/json.h index eb92c6a91..555440760 100644 --- a/include/openvswitch/json.h +++ b/include/openvswitch/json.h @@ -14,8 +14,8 @@ * limitations under the License. */ -#ifndef JSON_H -#define JSON_H 1 +#ifndef OPENVSWITCH_JSON_H +#define OPENVSWITCH_JSON_H 1 /* This is an implementation of JavaScript Object Notation (JSON) as specified * by RFC 4627. It is intended to fully comply with RFC 4627, with the @@ -159,14 +159,14 @@ json_clone(const struct json *json_) return json; } -void json_destroy__(struct json *json); +void json_destroy__(struct json *json, bool); /* Frees 'json' and everything it points to, recursively. */ static inline void json_destroy(struct json *json) { if (json && !--json->count) { - json_destroy__(json); + json_destroy__(json, false); } } @@ -174,4 +174,4 @@ json_destroy(struct json *json) } #endif -#endif /* json.h */ +#endif /* OPENVSWITCH_JSON_H */ diff --git a/lib/automake.mk b/lib/automake.mk index 8596171c6..78d6e6516 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -178,6 +178,7 @@ lib_libopenvswitch_la_SOURCES = \ lib/jhash.c \ lib/jhash.h \ lib/json.c \ + lib/json.h \ lib/jsonrpc.c \ lib/jsonrpc.h \ lib/lacp.c \ diff --git a/lib/json.c b/lib/json.c index 9411eeda7..001f6e6ab 100644 --- a/lib/json.c +++ b/lib/json.c @@ -24,13 +24,21 @@ #include #include +#include "cooperative-multitasking.h" #include "openvswitch/dynamic-string.h" #include "hash.h" +#include "json.h" #include "openvswitch/shash.h" #include "unicode.h" #include "util.h" #include "uuid.h" +/* Non-public JSSF_* flags. Must not overlap with public ones defined + * in include/openvswitch/json.h. */ +enum { + JSSF_YIELD = 1 << 7, +}; + /* The type of a JSON token. */ enum json_token_type { T_EOF = 0, @@ -189,6 +197,14 @@ json_serialized_object_create(const struct json *src) return json; } +struct json * +json_serialized_object_create_with_yield(const struct json *src) +{ + struct json *json = json_create(JSON_SERIALIZED_OBJECT); + json->string = json_to_string(src, JSSF_SORT | JSSF_YIELD); + return json; +} + struct json * json_array_create_empty(void) { @@ -375,20 +391,20 @@ json_integer(const struct json *json) return json->integer; } -static void json_destroy_object(struct shash *object); -static void json_destroy_array(struct json_array *array); +static void json_destroy_object(struct shash *object, bool yield); +static void json_destroy_array(struct json_array *array, bool yield); /* Frees 'json' and everything it points to, recursively. */ void -json_destroy__(struct json *json) +json_destroy__(struct json *json, bool yield) { switch (json->type) { case JSON_OBJECT: - json_destroy_object(json->object); + json_destroy_object(json->object, yield); break; case JSON_ARRAY: - json_destroy_array(&json->array); + json_destroy_array(&json->array, yield); break; case JSON_STRING: @@ -410,14 +426,22 @@ json_destroy__(struct json *json) } static void -json_destroy_object(struct shash *object) +json_destroy_object(struct shash *object, bool yield) { struct shash_node *node; + if (yield) { + cooperative_multitasking_yield(); + } + SHASH_FOR_EACH_SAFE (node, object) { struct json *value = node->data; - json_destroy(value); + if (yield) { + json_destroy_with_yield(value); + } else { + json_destroy(value); + } shash_delete(object, node); } shash_destroy(object); @@ -425,12 +449,20 @@ json_destroy_object(struct shash *object) } static void -json_destroy_array(struct json_array *array) +json_destroy_array(struct json_array *array, bool yield) { size_t i; + if (yield) { + cooperative_multitasking_yield(); + } + for (i = 0; i < array->n; i++) { - json_destroy(array->elems[i]); + if (yield) { + json_destroy_with_yield(array->elems[i]); + } else { + json_destroy(array->elems[i]); + } } free(array->elems); } @@ -1664,6 +1696,10 @@ json_serialize_object(const struct shash *object, struct json_serializer *s) s->depth++; indent_line(s); + if (s->flags & JSSF_YIELD) { + cooperative_multitasking_yield(); + } + if (s->flags & JSSF_SORT) { const struct shash_node **nodes; size_t n, i; @@ -1697,6 +1733,10 @@ json_serialize_array(const struct json_array *array, struct json_serializer *s) ds_put_char(ds, '['); s->depth++; + if (s->flags & JSSF_YIELD) { + cooperative_multitasking_yield(); + } + if (array->n > 0) { indent_line(s); diff --git a/lib/json.h b/lib/json.h new file mode 100644 index 000000000..4ad440b39 --- /dev/null +++ b/lib/json.h @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2024 Canonical Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef JSON_H +#define JSON_H 1 + +#include "openvswitch/json.h" + +static inline void +json_destroy_with_yield(struct json *json) +{ + if (json && !--json->count) { + json_destroy__(json, true); + } +} + +struct json *json_serialized_object_create_with_yield(const struct json *); + +#endif /* JSON_H */ From patchwork Tue Jan 16 22:52:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1887231 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=O8TQW8OU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TF45m4W1xz23dx for ; Wed, 17 Jan 2024 09:52:28 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 8328641E7B; Tue, 16 Jan 2024 22:52:26 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 8328641E7B Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=O8TQW8OU X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LWVCfXwTjrQj; Tue, 16 Jan 2024 22:52:24 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id DDF7541E32; Tue, 16 Jan 2024 22:52:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org DDF7541E32 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 08F0DC0DE4; Tue, 16 Jan 2024 22:52:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 22343C0077 for ; Tue, 16 Jan 2024 22:52:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 400F641C59 for ; Tue, 16 Jan 2024 22:52:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 400F641C59 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OT1t6fFeakyK for ; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp4.osuosl.org (Postfix) with ESMTPS id 69E9A41C2D for ; Tue, 16 Jan 2024 22:52:10 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 69E9A41C2D Received: from fnordahl.lxd (ti0189a430-2338.bb.online.no [88.90.42.44]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id A635441BB2; Tue, 16 Jan 2024 22:52:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1705445527; bh=yEaERbBLXutlSCTT/rYSTfkE3q4J16UbC4r9tbfyEYU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O8TQW8OU73VdfNpxaOIM3VEMSiJdHJVSw5F3vMidT44OqotwTUBEX0NVqgZDijjEl gCcy5VH9gPFh4nrONt2zHKhfbVzI75YBeYchVgHIL4bT8CkKMZwNmGPzX97WkYNFKr RGsRD1DqBwlmN+dsz6E6mRFEQAE6q84tEOwcKTI8hjmjozvpwGhsYcloKJ+ZcPKE3H Z4A7q06XS+RqDK5xJBxG8MKrrGY336uGSLopF6N/2NyODPNIK/yr6hgAbYe/vZSVqb X9kwsYYAezbho3wiP0XYHsHOsa7OxywDQfLZ68HHmEi0DBweMIJJrVQY97gV1FP78n T1w2oc1QJDf2g== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 16 Jan 2024 22:52:05 +0000 Message-Id: <20240116225205.38112-6-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240116225205.38112-1-frode.nordahl@canonical.com> References: <20240116225205.38112-1-frode.nordahl@canonical.com> MIME-Version: 1.0 Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 5/5] ovsdb-server: Make use of cooperative multitasking. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Initialize the cooperative multitasking module for the ovsdb-server. The server side schema conversion process used for storage engines such as RAFT is time consuming, yield during processing. After the schema conversion is done, the processing of JSON-RPC sessions and OVSDB monitors for reconnecting clients can overrun the configured election timer. The destruction of JSON objects representing the database contents has been identified as one of the primary offenders. Make use of yielding version of the JSON object destroy function to mitigate. This series has been tested by checking success of schema conversion, ensuring no involuntary leader change occurs with election timer configurations as low as 750 msec, on a 75MB database with ~ 100 connected clients as produced by the ovn-heater ocp-120-density-light test scenario. Signed-off-by: Frode Nordahl --- NEWS | 2 ++ ovsdb/file.c | 3 +++ ovsdb/jsonrpc-server.c | 3 +++ ovsdb/monitor.c | 15 +++++++++++---- ovsdb/ovsdb-server.c | 2 ++ ovsdb/trigger.c | 3 +++ 6 files changed, 24 insertions(+), 4 deletions(-) diff --git a/NEWS b/NEWS index 410dd68e5..1b42383d0 100644 --- a/NEWS +++ b/NEWS @@ -10,6 +10,8 @@ Post-v3.2.0 remotes and database configuration, including setting options for connection methods for relays and active-backup replication. For more details see ovsdb-server(1) and ovsdb(7). + * Make use of cooperative multitasking to improve maintenance of RAFT + cluster during long running processing such as online schema conversion. - OpenFlow: * NXT_CT_FLUSH extension is updated to support flushing connections based on mark and labels. 'ct-flush' command of ovs-ofctl updated diff --git a/ovsdb/file.c b/ovsdb/file.c index 77a89fd1a..66ef87a1f 100644 --- a/ovsdb/file.c +++ b/ovsdb/file.c @@ -23,6 +23,7 @@ #include "bitmap.h" #include "column.h" +#include "cooperative-multitasking.h" #include "log.h" #include "openvswitch/json.h" #include "lockfile.h" @@ -321,6 +322,8 @@ ovsdb_convert_table(struct ovsdb_txn *txn, struct ovsdb_row *dst_row = ovsdb_row_create(dst_table); *ovsdb_row_get_uuid_rw(dst_row) = *ovsdb_row_get_uuid(src_row); + cooperative_multitasking_yield(); + SHASH_FOR_EACH (node, &src_table->schema->columns) { const struct ovsdb_column *src_column = node->data; const struct ovsdb_column *dst_column; diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c index 817997677..26a53898f 100644 --- a/ovsdb/jsonrpc-server.c +++ b/ovsdb/jsonrpc-server.c @@ -21,6 +21,7 @@ #include "bitmap.h" #include "column.h" +#include "cooperative-multitasking.h" #include "openvswitch/dynamic-string.h" #include "monitor.h" #include "openvswitch/json.h" @@ -694,6 +695,8 @@ ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *remote) struct ovsdb_jsonrpc_session *s; LIST_FOR_EACH_SAFE (s, node, &remote->sessions) { + cooperative_multitasking_yield(); + int error = ovsdb_jsonrpc_session_run(s); if (error) { ovsdb_jsonrpc_session_close(s); diff --git a/ovsdb/monitor.c b/ovsdb/monitor.c index d1e466faa..c3bfae3d2 100644 --- a/ovsdb/monitor.c +++ b/ovsdb/monitor.c @@ -20,8 +20,10 @@ #include "bitmap.h" #include "column.h" +#include "cooperative-multitasking.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/json.h" +#include "json.h" #include "jsonrpc.h" #include "ovsdb-error.h" #include "ovsdb-parser.h" @@ -262,7 +264,7 @@ ovsdb_monitor_json_cache_flush(struct ovsdb_monitor *dbmon) struct ovsdb_monitor_json_cache_node *node; HMAP_FOR_EACH_POP(node, hmap_node, &dbmon->json_cache) { - json_destroy(node->json); + json_destroy_with_yield(node->json); free(node); } } @@ -278,7 +280,7 @@ ovsdb_monitor_json_cache_destroy(struct ovsdb_monitor *dbmon, = ovsdb_monitor_json_cache_search(dbmon, v, change_set); if (node) { hmap_remove(&dbmon->json_cache, &node->hmap_node); - json_destroy(node->json); + json_destroy_with_yield(node->json); free(node); } } @@ -1172,6 +1174,8 @@ ovsdb_monitor_compose_update( struct ovsdb_monitor_table *mt = mcst->mt; HMAP_FOR_EACH_SAFE (row, hmap_node, &mcst->rows) { + cooperative_multitasking_yield(); + struct json *row_json; row_json = (*row_update)(mt, condition, OVSDB_MONITOR_ROW, row, initial, changed, mcst->n_columns); @@ -1217,6 +1221,8 @@ ovsdb_monitor_compose_cond_change_update( HMAP_FOR_EACH (row, hmap_node, &mt->table->rows) { struct json *row_json; + cooperative_multitasking_yield(); + row_json = ovsdb_monitor_compose_row_update2(mt, condition, OVSDB_ROW, row, false, changed, @@ -1286,8 +1292,9 @@ ovsdb_monitor_get_update( /* Pre-serializing the object to avoid doing this * for every client. */ - json_serialized = json_serialized_object_create(json); - json_destroy(json); + json_serialized = + json_serialized_object_create_with_yield(json); + json_destroy_with_yield(json); json = json_serialized; } ovsdb_monitor_json_cache_insert(dbmon, version, mcs, diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index d45c9e5f3..b51fd42fe 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -24,6 +24,7 @@ #include "column.h" #include "command-line.h" +#include "cooperative-multitasking.h" #include "daemon.h" #include "dirs.h" #include "dns-resolve.h" @@ -915,6 +916,7 @@ main(int argc, char *argv[]) } dns_resolve_destroy(); perf_counters_destroy(); + cooperative_multitasking_destroy(); service_stop(); return 0; } diff --git a/ovsdb/trigger.c b/ovsdb/trigger.c index 2a48ccc64..8c00fec18 100644 --- a/ovsdb/trigger.c +++ b/ovsdb/trigger.c @@ -20,6 +20,7 @@ #include #include +#include "cooperative-multitasking.h" #include "file.h" #include "openvswitch/json.h" #include "jsonrpc.h" @@ -181,6 +182,8 @@ ovsdb_trigger_run(struct ovsdb *db, long long int now) bool disconnect_all = false; LIST_FOR_EACH_SAFE (t, node, &db->triggers) { + cooperative_multitasking_yield(); + if (run_triggers || now - t->created >= t->timeout_msec || t->progress || t->txn_forward) {