From patchwork Mon Apr 12 22:00:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465452 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2jg0m4sz9sRK for ; Tue, 13 Apr 2021 08:00:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E22B2404FC; Mon, 12 Apr 2021 22:00:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4Zf7LgryFag1; Mon, 12 Apr 2021 22:00:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id E165940298; Mon, 12 Apr 2021 22:00:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1A0EBC0013; Mon, 12 Apr 2021 22:00:32 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6589DC0011 for ; Mon, 12 Apr 2021 22:00:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5BB53404DB for ; Mon, 12 Apr 2021 22:00:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwHkfj-dtokP for ; Mon, 12 Apr 2021 22:00:29 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp2.osuosl.org (Postfix) with ESMTPS id 940F540298 for ; Mon, 12 Apr 2021 22:00:28 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 214EFE0009; Mon, 12 Apr 2021 22:00:25 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:12 +0200 Message-Id: <20210412220020.2286954-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 1/9] ovs-replay: New library to create and manage replay files. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This library provides interfaces to open replay files and read/write records. Will be used later for stream record/replay functionality, i.e. to record all the incoming connections and data and replay it later for debugging and performance analysis purposes. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- lib/automake.mk | 5 + lib/ovs-replay-syn.man | 3 + lib/ovs-replay.c | 237 +++++++++++++++++++++++++++++++++++++++++ lib/ovs-replay.h | 163 ++++++++++++++++++++++++++++ lib/ovs-replay.man | 16 +++ lib/ovs-replay.xml | 35 ++++++ 6 files changed, 459 insertions(+) create mode 100644 lib/ovs-replay-syn.man create mode 100644 lib/ovs-replay.c create mode 100644 lib/ovs-replay.h create mode 100644 lib/ovs-replay.man create mode 100644 lib/ovs-replay.xml diff --git a/lib/automake.mk b/lib/automake.mk index 39901bd6d..b558692c6 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -236,6 +236,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/ovs-numa.h \ lib/ovs-rcu.c \ lib/ovs-rcu.h \ + lib/ovs-replay.c \ + lib/ovs-replay.h \ lib/ovs-router.h \ lib/ovs-router.c \ lib/ovs-thread.c \ @@ -532,6 +534,7 @@ EXTRA_DIST += \ lib/daemon.xml \ lib/dirs.c.in \ lib/db-ctl-base.xml \ + lib/ovs-replay.xml \ lib/ssl.xml \ lib/ssl-bootstrap.xml \ lib/ssl-peer-ca-cert.xml \ @@ -554,6 +557,8 @@ MAN_FRAGMENTS += \ lib/dpif-netdev-unixctl.man \ lib/ofp-version.man \ lib/ovs.tmac \ + lib/ovs-replay.man \ + lib/ovs-replay-syn.man \ lib/service.man \ lib/service-syn.man \ lib/ssl-bootstrap.man \ diff --git a/lib/ovs-replay-syn.man b/lib/ovs-replay-syn.man new file mode 100644 index 000000000..f0c78656b --- /dev/null +++ b/lib/ovs-replay-syn.man @@ -0,0 +1,3 @@ +.IP "Replay options:" +[\fB\-\-record-replay\fR[\fB=\fIdirectory\fR]] +[\fB\-\-replay\fR[\fB=\fIdirectory\fR]] diff --git a/lib/ovs-replay.c b/lib/ovs-replay.c new file mode 100644 index 000000000..fcdb4edbb --- /dev/null +++ b/lib/ovs-replay.c @@ -0,0 +1,237 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "dirs.h" +#include "ovs-atomic.h" +#include "ovs-replay.h" +#include "util.h" +#include "openvswitch/vlog.h" + +VLOG_DEFINE_THIS_MODULE(ovs_replay); + +static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(10, 25); + +static struct ovs_mutex replay_mutex = OVS_MUTEX_INITIALIZER; +static int replay_seqno OVS_GUARDED_BY(replay_mutex) = 0; +static atomic_int replay_state = ATOMIC_VAR_INIT(OVS_REPLAY_NONE); + +static char *dirname = NULL; + +void +ovs_replay_set_state(enum ovs_replay_state state) +{ + atomic_store_relaxed(&replay_state, state); +} + +enum ovs_replay_state +ovs_replay_get_state(void) +{ + int state; + + atomic_read_relaxed(&replay_state, &state); + return state; +} + +void +ovs_replay_set_dirname(const char *new_dirname) +{ + if (new_dirname) { + free(dirname); + dirname = xstrdup(new_dirname); + } +} + + +void +ovs_replay_lock(void) + OVS_ACQUIRES(replay_mutex) +{ + ovs_mutex_lock(&replay_mutex); +} + +void +ovs_replay_unlock(void) + OVS_RELEASES(replay_mutex) +{ + ovs_mutex_unlock(&replay_mutex); +} + +int +ovs_replay_seqno(void) + OVS_REQUIRES(replay_mutex) +{ + return replay_seqno; +} + +static char * +ovs_replay_file_name(const char *name, int seqno) +{ + char *local_name = xstrdup(name); + char *filename, *p, *c; + bool skip = false; + + /* Replace all the numbers and special symbols with single underscore. + * Numbers might be PIDs or port numbers that could change between record + * and replay phases, special symbols might be not good as a filename. + * We have a unique seuqence number as part of the name, so we don't care + * keeping too much information. */ + for (c = p = local_name; *p; p++) { + if (!isalpha((unsigned char) *p)) { + if (!skip) { + *c++ = '_'; + skip = true; + } + } else { + *c++ = *p; + skip = false; + } + } + if (skip) { + c--; + } + *c = '\0'; + filename = xasprintf("%s/replay_%s_%d", dirname ? dirname : "", + local_name, seqno); + VLOG_DBG("Constructing replay filename: '%s' --> '%s' --> '%s'.", + name, local_name, filename); + free(local_name); + + return filename; +} + +int +ovs_replay_file_open(const char *name, replay_file_t *f, int *seqno) + OVS_REQUIRES(replay_mutex) +{ + char *file_path, *filename; + int state = ovs_replay_get_state(); + + ovs_assert(state != OVS_REPLAY_NONE); + + filename = ovs_replay_file_name(name, replay_seqno); + if (filename[0] != '/') { + file_path = abs_file_name(ovs_rundir(), filename); + free(filename); + } else { + file_path = filename; + } + + *f = fopen(file_path, state == OVS_REPLAY_WRITE ? "wb" : "rb"); + if (!*f) { + VLOG_ERR("%s: fopen failed: %s", file_path, ovs_strerror(errno)); + free(file_path); + return errno; + } + free(file_path); + + if (state == OVS_REPLAY_READ + && fread(seqno, sizeof *seqno, 1, *f) != 1) { + VLOG_INFO("%s: failed to read seqno: replay might be empty.", name); + *seqno = INT_MAX; + } + replay_seqno++; /* New file opened. */ + return 0; +} + +void +ovs_replay_file_close(replay_file_t f) +{ + fclose(f); +} + +int +ovs_replay_write(replay_file_t f, const void *buffer, int n, bool is_read) + OVS_EXCLUDED(replay_mutex) +{ + int state = ovs_replay_get_state(); + int seqno_to_write; + int retval = 0; + + if (OVS_LIKELY(state != OVS_REPLAY_WRITE)) { + return 0; + } + + ovs_replay_lock(); + + seqno_to_write = is_read ? replay_seqno : -replay_seqno; + if (fwrite(&seqno_to_write, sizeof seqno_to_write, 1, f) != 1) { + VLOG_ERR_RL(&rl, "Failed to write seqno."); + retval = -1; + goto out; + } + if (fwrite(&n, sizeof n, 1, f) != 1) { + VLOG_ERR_RL(&rl, "Failed to write length."); + retval = -1; + goto out; + } + if (n > 0 && is_read && fwrite(buffer, 1, n, f) != n) { + VLOG_ERR_RL(&rl, "Failed to write data."); + retval = -1; + } +out: + replay_seqno++; /* Write completed. */ + ovs_replay_unlock(); + fflush(f); + return retval; +} + +int +ovs_replay_read(replay_file_t f, void *buffer, int buffer_size, + int *len, int *seqno, bool is_read) + OVS_REQUIRES(replay_mutex) +{ + int retval = EINVAL; + + if (fread(len, sizeof *len, 1, f) != 1 + || (is_read && *len > buffer_size)) { + VLOG_ERR("Failed to read replay length."); + goto out; + } + + if (*len > 0 && is_read && fread(buffer, 1, *len, f) != *len) { + VLOG_ERR("Failed to read replay buffer."); + goto out; + } + + if (fread(seqno, sizeof *seqno, 1, f) != 1) { + *seqno = INT_MAX; /* Most likely EOF. */ + if (ferror(f)) { + VLOG_INFO("Failed to read replay seqno."); + goto out; + } + } + + retval = 0; +out: + replay_seqno++; /* Read completed. */ + return retval; +} + +void +ovs_replay_usage(void) +{ + printf("\nReplay options:\n" + " --replay-record[=DIR] turn on writing replay files\n" + " --replay[=DIR] run from replay files\n"); +} diff --git a/lib/ovs-replay.h b/lib/ovs-replay.h new file mode 100644 index 000000000..03cddeb13 --- /dev/null +++ b/lib/ovs-replay.h @@ -0,0 +1,163 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVS_REPLAY_H +#define OVS_REPLAY_H 1 + +/* + * Library to work with 'replay' files. + * + * ovs_replay_file_open() should be used to open a new replay file. + * 'replay' file contains records. If current state is OVS_REPLAY_WRITE, + * files are opened in a write mode and new records could be written by + * ovs_replay_write(). If current mode is OVS_REPLAY_READ, files are + * opened in a read mode and records could be read with ovs_replay_read(). + * + * Each record has several fields: + * [] + * + * Here is a global sequence number of the record, it is unique + * across all the replay files. By comparing normalized version of this + * number (ovs_replay_normalized_seqno()) with the current global sequence + * number (ovs_replay_seqno()) users may detect if this record should be + * replayed now. + * + * Non-normalized versions of seqno are used to distinguish 'read' and 'write' + * records. 'read' records are records that corresponds to incoming events. + * Only 'read' records contains . 'write' records contains outgoing + * events, i.e. stream_write() and contains only the size of outgoing message. + * + * For 'read' records, is a size of a stored in this record in + * bytes. For 'write' records, it is a size of outgoing message, but there + * is no . If it contains negative value, it means that this record + * holds some recorded error and no data available. + */ + +#include +#include + +typedef FILE *replay_file_t; + +/* Replay state. */ +enum ovs_replay_state { + OVS_REPLAY_NONE, + OVS_REPLAY_WRITE, + OVS_REPLAY_READ, +}; + +void ovs_replay_set_state(enum ovs_replay_state); +enum ovs_replay_state ovs_replay_get_state(void); + +static inline bool +ovs_replay_is_active(void) +{ + return ovs_replay_get_state() != OVS_REPLAY_NONE; +} + +/* Returns 'true' if provided sequence number belongs to 'read' record. */ +static inline bool +ovs_replay_seqno_is_read(int seqno) +{ + return seqno >= 0; +} + +/* Normalizes sequence number, so it can be used to compare with result of + * ovs_replay_seqno(). */ +static inline int +ovs_replay_normalized_seqno(int seqno) +{ + return seqno >= 0 ? seqno : -seqno; +} + +/* Locks the replay module. + * Locking required to use ovs_replay_file_open() and ovs_replay_read(). */ +void ovs_replay_lock(void); + +/* Unlocks the replay module. */ +void ovs_replay_unlock(void); + +/* Returns current global replay sequence number. */ +int ovs_replay_seqno(void); + +/* In write mode creates a new replay file to write stream replay. + * In read mode opens an existing replay file. + * + * Requires replay being locked with ovs_replay_lock(). + * + * On success returns 0, 'f' points to the opened file. If current mode is + * OVS_REPLAY_READ, sets 'seqno' to the sequence number of the first record in + * the file. + * + * On failure returns positive errno. */ +int ovs_replay_file_open(const char *name, replay_file_t *f, int *seqno); + +/* Closes replay file. */ +void ovs_replay_file_close(replay_file_t f); + +/* Writes a new record of 'n' bytes from 'buffer' to a replay file. + * 'is_read' should be true if the record belongs to 'read' operation + * Depending on 'is_read', creates 'read' or 'write' record. 'write' records + * contains only the size of a bufer ('n'). + * If 'n' is negative, writes 'n' as an error status. + * + * On success returns 0. Otherwise, positive errno. */ +int ovs_replay_write(replay_file_t f, const void *buffer, int n, bool is_read); + +/* Reads one record from a replay file to 'buffer'. 'buffer_size' should be + * equal to the size of a memory available. + * + * On success, actual size of the read record will be set to 'len', 'seqno' + * will be set to the sequence number of the next record in the file. If it + * was the last record, sets 'seqno' to INT_MAX. + * Negative 'len' means that record contained an error status. + * + * Depending on 'is_read', tries to read 'read' or 'write' record. For the + * 'write' record, only 'len' and 'seqno' updated, no data read to 'buffer'. + * + * On success returns 0. Otherwise, positive errno. */ +int ovs_replay_read(replay_file_t f, void *buffer, int buffer_size, + int *len, int *seqno, bool is_read); + +/* Helpers for cmdline options. */ +#define OVS_REPLAY_OPTION_ENUMS \ + OPT_OVS_REPLAY_REC, \ + OPT_OVS_REPLAY + +#define OVS_REPLAY_LONG_OPTIONS \ + {"replay-record", optional_argument, NULL, OPT_OVS_REPLAY_REC}, \ + {"replay", optional_argument, NULL, OPT_OVS_REPLAY} + +#define OVS_REPLAY_OPTION_HANDLERS \ + case OPT_OVS_REPLAY_REC: \ + ovs_replay_set_state(OVS_REPLAY_WRITE); \ + ovs_replay_set_dirname(optarg); \ + break; \ + \ + case OPT_OVS_REPLAY: \ + ovs_replay_set_state(OVS_REPLAY_READ); \ + ovs_replay_set_dirname(optarg); \ + break; + +#define OVS_REPLAY_CASES \ + case OPT_OVS_REPLAY_REC: case OPT_OVS_REPLAY: + +/* Prints usage information. */ +void ovs_replay_usage(void); + +/* Sets path to the directory where replay files should be stored. */ +void ovs_replay_set_dirname(const char *new_dirname); + +#endif /* OVS_REPLAY_H */ diff --git a/lib/ovs-replay.man b/lib/ovs-replay.man new file mode 100644 index 000000000..f81b9fbed --- /dev/null +++ b/lib/ovs-replay.man @@ -0,0 +1,16 @@ +.IP "\fB\-\-record-replay[=\fIdirectory\fR]" +Sets the process in "recording" mode, in which it will record all the +connections, data from streams (Unix domain and network sockets) and some other +important necessary bits, so they could be replayed later. +Recorded data is stored in replay files in specified \fIdirectory\fR. +If \fIdirectory\fR does not begin with \fB/\fR, it is interpreted as relative +to \fB@RUNDIR@\fR. If \fIdirectory\fR is not specified, \fB@RUNDIR@\fR will +be used. +. +.IP "\fB\-\-replay[=\fIdirectory\fR]" +Sets the process in "replay" mode, in which it will read information about +connections, data from streams (Unix domain and network sockets) and some +other necessary bits directly from replay files instead of using real sockets. +Replay files from the \fIdirectory\fR will be used. If \fIdirectory\fR does +not begin with \fB/\fR, it is interpreted as relative to \fB@RUNDIR@\fR. +If \fIdirectory\fR is not specified, \fB@RUNDIR@\fR will be used. diff --git a/lib/ovs-replay.xml b/lib/ovs-replay.xml new file mode 100644 index 000000000..7392bd540 --- /dev/null +++ b/lib/ovs-replay.xml @@ -0,0 +1,35 @@ + +
+
--record-replay[=directory]
+
+

+ Sets the process in "recording" mode, in which it will record all the + connections, data from streams (Unix domain and network sockets) and some + other necessary bits, so they could be replayed later. +

+

+ Recorded data is stored in replay files in specified + directory. If directory does not begin with + /, it is interpreted as relative to @RUNDIR@. + If directory is not specified, @RUNDIR@ will + be used. +

+
+ +
--replay[=directory]
+
+

+ Sets the process in "replay" mode, in which it will read information + about connections, data from streams (Unix domain and network sockets) + and some other necessary bits directly from replay files instead of using + real sockets. +

+

+ Replay files from the directory will be used. + If directory does not begin with /, it is + interpreted as relative to @RUNDIR@. If + directory is not specified, @RUNDIR@ will be + used. +

+
+
From patchwork Mon Apr 12 22:00:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465453 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2jh1D7vz9sVv for ; Tue, 13 Apr 2021 08:00:40 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A569E60B22; Mon, 12 Apr 2021 22:00:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LcUe7uxldMvN; Mon, 12 Apr 2021 22:00:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTP id 529D960B18; Mon, 12 Apr 2021 22:00:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 291FFC000C; Mon, 12 Apr 2021 22:00:34 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id C510DC0019 for ; Mon, 12 Apr 2021 22:00:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A7E80404E7 for ; Mon, 12 Apr 2021 22:00:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HQ8PMzQvH9aP for ; Mon, 12 Apr 2021 22:00:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp2.osuosl.org (Postfix) with ESMTPS id DBA0540298 for ; Mon, 12 Apr 2021 22:00:30 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 7C287E000A; Mon, 12 Apr 2021 22:00:28 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:13 +0200 Message-Id: <20210412220020.2286954-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 2/9] stream: Add record/replay functionality. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" For debugging purposes it is useful to be able to record all the incoming transactions and commands and replay them locally under debugger or with additional logging enabled. This patch introduces ability to record all the incoming stream data and replay it via new stream provider named 'stream-replay'. During the record phase all the incoming stream data written to special replay_* files in the application rundir. On replay phase instead of opening real streams application will open replay_* files and read all the incoming data directly from them. If enabled for ovsdb-server, for example, this allows to record all the connections and transactions from the big setup and replay them locally afterwards to debug the behaviour or test performance. To start application in recording mode there is a --replay-record cmdline option. --replay is to replay previously recorded streams. Current version doesn't work well with time-based stream events like inactivity probes or any other events generated internally. This is a point for further improvement. Signed-off-by: Ilya Maximets --- lib/automake.mk | 1 + lib/stream-provider.h | 5 + lib/stream-replay.c | 459 ++++++++++++++++++++++++++++++++++++++++++ lib/stream.c | 35 +++- lib/stream.h | 12 ++ 5 files changed, 506 insertions(+), 6 deletions(-) create mode 100644 lib/stream-replay.c diff --git a/lib/automake.mk b/lib/automake.mk index b558692c6..db9017591 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -312,6 +312,7 @@ lib_libopenvswitch_la_SOURCES = \ lib/stream-fd.c \ lib/stream-fd.h \ lib/stream-provider.h \ + lib/stream-replay.c \ lib/stream-ssl.h \ lib/stream-tcp.c \ lib/stream.c \ diff --git a/lib/stream-provider.h b/lib/stream-provider.h index 75f4f059b..44e3c6431 100644 --- a/lib/stream-provider.h +++ b/lib/stream-provider.h @@ -18,6 +18,7 @@ #define STREAM_PROVIDER_H 1 #include +#include "ovs-replay.h" #include "stream.h" /* Active stream connection. */ @@ -29,6 +30,7 @@ struct stream { const struct stream_class *class; int state; int error; + replay_file_t replay_wfd; char *name; char *peer_id; }; @@ -133,6 +135,7 @@ struct pstream { const struct pstream_class *class; char *name; ovs_be16 bound_port; + replay_file_t replay_wfd; }; void pstream_init(struct pstream *, const struct pstream_class *, char *name); @@ -200,5 +203,7 @@ extern const struct pstream_class pwindows_pstream_class; extern const struct stream_class ssl_stream_class; extern const struct pstream_class pssl_pstream_class; #endif +extern const struct stream_class replay_stream_class; +extern const struct pstream_class preplay_pstream_class; #endif /* stream-provider.h */ diff --git a/lib/stream-replay.c b/lib/stream-replay.c new file mode 100644 index 000000000..ef591b920 --- /dev/null +++ b/lib/stream-replay.c @@ -0,0 +1,459 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "ovs-atomic.h" +#include "ovs-replay.h" +#include "util.h" +#include "stream-provider.h" +#include "stream.h" +#include "openvswitch/poll-loop.h" +#include "openvswitch/vlog.h" + +VLOG_DEFINE_THIS_MODULE(stream_replay); + +/* Active replay stream. */ + +struct stream_replay +{ + struct stream stream; + replay_file_t f; + int seqno; +}; + +const struct stream_class replay_stream_class; + +/* Creates a new stream named 'name' that will emulate sending and receiving + * data using replay file and stores a pointer to the stream in '*streamp'. + * + * Takes ownership of 'name'. + * + * Returns 0 if successful, otherwise a positive errno value. */ +static int +new_replay_stream(char *name, struct stream **streamp) +{ + struct stream_replay *s; + int seqno = 0, error = 0, open_result; + replay_file_t f; + + ovs_replay_lock(); + error = ovs_replay_file_open(name, &f, &seqno); + if (error) { + VLOG_ERR("%s: failed to open stream.", name); + goto unlock; + } + + error = ovs_replay_read(f, NULL, 0, &open_result, &seqno, true); + if (error) { + VLOG_ERR("%s: failed to read 'open' record.", name); + ovs_replay_file_close(f); + goto unlock; + } + + if (open_result) { + error = -open_result; + ovs_replay_file_close(f); + goto unlock; + } + + s = xmalloc(sizeof *s); + stream_init(&s->stream, &replay_stream_class, 0, name); + s->f = f; + s->seqno = seqno; + *streamp = &s->stream; +unlock: + ovs_replay_unlock(); + return error; +} + +static struct stream_replay * +stream_replay_cast(struct stream *stream) +{ + stream_assert_class(stream, &replay_stream_class); + return CONTAINER_OF(stream, struct stream_replay, stream); +} + +void +stream_replay_open_wfd(struct stream *s, int open_result, const char *name) +{ + int state = ovs_replay_get_state(); + int error = 0; + replay_file_t f; + + if (OVS_LIKELY(state != OVS_REPLAY_WRITE)) { + return; + } + + ovs_replay_lock(); + error = ovs_replay_file_open(name, &f, NULL); + if (error) { + VLOG_ERR("%s: failed to open replay file for stream.", name); + ovs_replay_unlock(); + return; + } + ovs_replay_unlock(); + + if (ovs_replay_write(f, NULL, -open_result, true)) { + VLOG_ERR("%s: failed to write 'open' failure: %d", + s->name, open_result); + } + if (open_result) { + /* We recorded failure to open the stream. */ + ovs_replay_file_close(f); + } else { + s->replay_wfd = f; + } +} + +void +stream_replay_write(struct stream *s, const void *buffer, int n, bool is_read) +{ + int state = ovs_replay_get_state(); + + if (OVS_LIKELY(state != OVS_REPLAY_WRITE)) { + return; + } + + if (ovs_replay_write(s->replay_wfd, buffer, n, is_read)) { + VLOG_ERR("%s: failed to write buffer.", s->name); + } +} + +void +stream_replay_close_wfd(struct stream *s) +{ + if (s->replay_wfd) { + ovs_replay_file_close(s->replay_wfd); + } +} + +static int +stream_replay_open(const char *name, char *suffix OVS_UNUSED, + struct stream **streamp, uint8_t dscp OVS_UNUSED) +{ + return new_replay_stream(xstrdup(name), streamp); +} + +static void +stream_replay_close(struct stream *stream) +{ + struct stream_replay *s = stream_replay_cast(stream); + ovs_replay_file_close(s->f); + free(s); +} + +static ssize_t +stream_replay_recv(struct stream *stream, void *buffer, size_t n) +{ + struct stream_replay *s = stream_replay_cast(stream); + int norm_seqno = ovs_replay_normalized_seqno(s->seqno); + int error, len; + + ovs_replay_lock(); + ovs_assert(norm_seqno >= ovs_replay_seqno()); + + if (norm_seqno != ovs_replay_seqno() + || !ovs_replay_seqno_is_read(s->seqno)) { + error = EAGAIN; + goto unlock; + } + + error = ovs_replay_read(s->f, buffer, n, &len, &s->seqno, true); + if (error) { + VLOG_ERR("%s: failed to read from replay file.", stream->name); + goto unlock; + } + +unlock: + ovs_replay_unlock(); + return error ? -error : len; +} + +static ssize_t +stream_replay_send(struct stream *stream OVS_UNUSED, + const void *buffer OVS_UNUSED, size_t n) +{ + struct stream_replay *s = stream_replay_cast(stream); + int norm_seqno = ovs_replay_normalized_seqno(s->seqno); + int error, len; + + ovs_replay_lock(); + ovs_assert(norm_seqno >= ovs_replay_seqno()); + + if (norm_seqno != ovs_replay_seqno() + || ovs_replay_seqno_is_read(s->seqno)) { + error = EAGAIN; + goto unlock; + } + + error = ovs_replay_read(s->f, NULL, 0, &len, &s->seqno, false); + if (error) { + VLOG_ERR("%s: failed to read from replay file.", stream->name); + goto unlock; + } + ovs_assert(len < 0 || len <= n); + +unlock: + ovs_replay_unlock(); + return error ? -error : len; +} + +static void +stream_replay_wait(struct stream *stream, enum stream_wait_type wait) +{ + struct stream_replay *s = stream_replay_cast(stream); + switch (wait) { + case STREAM_CONNECT: + /* Connect does nothing and always available. */ + poll_immediate_wake(); + break; + + case STREAM_SEND: + if (s->seqno != INT_MAX && !ovs_replay_seqno_is_read(s->seqno)) { + /* Stream waits for write. */ + poll_immediate_wake(); + } + break; + + case STREAM_RECV: + if (s->seqno != INT_MAX && ovs_replay_seqno_is_read(s->seqno)) { + /* We still have something to read. */ + poll_immediate_wake(); + } + break; + + default: + OVS_NOT_REACHED(); + } +} + +const struct stream_class replay_stream_class = { + "replay", /* name */ + false, /* needs_probes */ + stream_replay_open, /* open */ + stream_replay_close, /* close */ + NULL, /* connect */ + stream_replay_recv, /* recv */ + stream_replay_send, /* send */ + NULL, /* run */ + NULL, /* run_wait */ + stream_replay_wait, /* wait */ +}; + +/* Passive replay stream. */ + +struct replay_pstream +{ + struct pstream pstream; + replay_file_t f; + int seqno; +}; + +const struct pstream_class preplay_pstream_class; + +static struct replay_pstream * +replay_pstream_cast(struct pstream *pstream) +{ + pstream_assert_class(pstream, &preplay_pstream_class); + return CONTAINER_OF(pstream, struct replay_pstream, pstream); +} + +/* Creates a new pstream named 'name' that will accept new replay connections + * reading them from the replay file and stores a pointer to the stream in + * '*pstreamp'. + * + * Takes ownership of 'name'. + * + * Returns 0 if successful, otherwise a positive errno value. */ +static int +pstream_replay_listen(const char *name, char *suffix OVS_UNUSED, + struct pstream **pstreamp, uint8_t dscp OVS_UNUSED) +{ + int seqno = 0, error = 0, listen_result; + replay_file_t f; + + ovs_replay_lock(); + error = ovs_replay_file_open(name, &f, &seqno); + if (error) { + VLOG_ERR("%s: failed to open pstream.", name); + goto unlock; + } + + error = ovs_replay_read(f, NULL, 0, &listen_result, &seqno, true); + if (error) { + VLOG_ERR("%s: failed to read 'listen' record.", name); + ovs_replay_file_close(f); + goto unlock; + } + + if (listen_result) { + error = -listen_result; + ovs_replay_file_close(f); + goto unlock; + } + + struct replay_pstream *ps = xmalloc(sizeof *ps); + pstream_init(&ps->pstream, &preplay_pstream_class, xstrdup(name)); + ps->f = f; + ps->seqno = seqno; + *pstreamp = &ps->pstream; +unlock: + ovs_replay_unlock(); + return error; +} + +void +pstream_replay_open_wfd(struct pstream *ps, int listen_result, + const char *name) +{ + int state = ovs_replay_get_state(); + int error = 0; + replay_file_t f; + + if (OVS_LIKELY(state != OVS_REPLAY_WRITE)) { + return; + } + + ovs_replay_lock(); + error = ovs_replay_file_open(name, &f, NULL); + if (error) { + VLOG_ERR("%s: failed to open replay file for pstream.", name); + ovs_replay_unlock(); + return; + } + ovs_replay_unlock(); + + if (ovs_replay_write(f, NULL, -listen_result, true)) { + VLOG_ERR("%s: failed to write 'listen' result: %d", + ps->name, listen_result); + } + + if (listen_result) { + /* We recorded failure to open the stream. */ + ovs_replay_file_close(f); + } else { + ps->replay_wfd = f; + } +} + + +void +pstream_replay_write_accept(struct pstream *ps, const struct stream *s, + int accept_result) +{ + int state = ovs_replay_get_state(); + int len; + + if (OVS_LIKELY(state != OVS_REPLAY_WRITE)) { + return; + } + + if (!accept_result) { + len = strlen(s->name); + if (ovs_replay_write(ps->replay_wfd, s->name, len, true)) { + VLOG_ERR("%s: failed to write accept name: %s", ps->name, s->name); + } + } else if (ovs_replay_write(ps->replay_wfd, NULL, -accept_result, true)) { + VLOG_ERR("%s: failed to write 'accept' failure: %d", + ps->name, accept_result); + } +} + +void +pstream_replay_close_wfd(struct pstream *ps) +{ + if (ps->replay_wfd) { + ovs_replay_file_close(ps->replay_wfd); + } +} + + +static void +pstream_replay_close(struct pstream *pstream) +{ + struct replay_pstream *ps = replay_pstream_cast(pstream); + + ovs_replay_file_close(ps->f); + free(ps); +} + +#define MAX_NAME_LEN 65536 + +static int +pstream_replay_accept(struct pstream *pstream, struct stream **new_streamp) +{ + struct replay_pstream *ps = replay_pstream_cast(pstream); + int norm_seqno = ovs_replay_normalized_seqno(ps->seqno); + int retval, len; + char name[MAX_NAME_LEN]; + + ovs_replay_lock(); + ovs_assert(norm_seqno >= ovs_replay_seqno()); + + if (norm_seqno != ovs_replay_seqno() + || !ovs_replay_seqno_is_read(ps->seqno)) { + retval = EAGAIN; + ovs_replay_unlock(); + goto exit; + } + + retval = ovs_replay_read(ps->f, name, MAX_NAME_LEN - 1, + &len, &ps->seqno, true); + if (retval) { + VLOG_ERR("%s: failed to read from replay file.", pstream->name); + ovs_replay_unlock(); + goto exit; + } + + ovs_replay_unlock(); + + if (len > 0) { + name[len] = 0; + retval = new_replay_stream(xstrdup(name), new_streamp); + } else { + retval = -len; + } +exit: + return retval; +} + +static void +pstream_replay_wait(struct pstream *pstream) +{ + struct replay_pstream *ps = replay_pstream_cast(pstream); + + if (ps->seqno != INT_MAX) { + /* Replay always has something to say. */ + poll_immediate_wake(); + } +} + +const struct pstream_class preplay_pstream_class = { + "preplay", + false, + pstream_replay_listen, + pstream_replay_close, + pstream_replay_accept, + pstream_replay_wait, +}; diff --git a/lib/stream.c b/lib/stream.c index e246b3773..1e3c8a24e 100644 --- a/lib/stream.c +++ b/lib/stream.c @@ -33,6 +33,7 @@ #include "openvswitch/ofp-print.h" #include "openvswitch/ofpbuf.h" #include "openvswitch/vlog.h" +#include "ovs-replay.h" #include "ovs-thread.h" #include "packets.h" #include "openvswitch/poll-loop.h" @@ -185,6 +186,9 @@ stream_lookup_class(const char *name, const struct stream_class **classp) if (strlen(class->name) == prefix_len && !memcmp(class->name, name, prefix_len)) { *classp = class; + if (ovs_replay_get_state() == OVS_REPLAY_READ) { + *classp = &replay_stream_class; + } return 0; } } @@ -227,6 +231,8 @@ stream_open(const char *name, struct stream **streamp, uint8_t dscp) suffix_copy = xstrdup(strchr(name, ':') + 1); error = class->open(name, suffix_copy, &stream, dscp); free(suffix_copy); + + stream_replay_open_wfd(stream, error, name); if (error) { goto error; } @@ -295,6 +301,7 @@ stream_close(struct stream *stream) if (stream != NULL) { char *name = stream->name; char *peer_id = stream->peer_id; + stream_replay_close_wfd(stream); (stream->class->close)(stream); free(name); free(peer_id); @@ -367,9 +374,13 @@ int stream_recv(struct stream *stream, void *buffer, size_t n) { int retval = stream_connect(stream); - return (retval ? -retval - : n == 0 ? 0 - : (stream->class->recv)(stream, buffer, n)); + + retval = retval ? -retval + : n == 0 ? 0 + : (stream->class->recv)(stream, buffer, n); + + stream_replay_write(stream, buffer, retval, true); + return retval; } /* Tries to send up to 'n' bytes of 'buffer' on 'stream', and returns: @@ -385,9 +396,12 @@ int stream_send(struct stream *stream, const void *buffer, size_t n) { int retval = stream_connect(stream); - return (retval ? -retval - : n == 0 ? 0 - : (stream->class->send)(stream, buffer, n)); + retval = retval ? -retval + : n == 0 ? 0 + : (stream->class->send)(stream, buffer, n); + + stream_replay_write(stream, buffer, retval, false); + return retval; } /* Allows 'stream' to perform maintenance activities, such as flushing @@ -483,6 +497,9 @@ pstream_lookup_class(const char *name, const struct pstream_class **classp) if (strlen(class->name) == prefix_len && !memcmp(class->name, name, prefix_len)) { *classp = class; + if (ovs_replay_get_state() == OVS_REPLAY_READ) { + *classp = &preplay_pstream_class; + } return 0; } } @@ -544,6 +561,8 @@ pstream_open(const char *name, struct pstream **pstreamp, uint8_t dscp) suffix_copy = xstrdup(strchr(name, ':') + 1); error = class->listen(name, suffix_copy, &pstream, dscp); free(suffix_copy); + + pstream_replay_open_wfd(pstream, error, name); if (error) { goto error; } @@ -571,6 +590,7 @@ pstream_close(struct pstream *pstream) { if (pstream != NULL) { char *name = pstream->name; + pstream_replay_close_wfd(pstream); (pstream->class->close)(pstream); free(name); } @@ -588,9 +608,12 @@ pstream_accept(struct pstream *pstream, struct stream **new_stream) int retval = (pstream->class->accept)(pstream, new_stream); if (retval) { *new_stream = NULL; + pstream_replay_write_accept(pstream, NULL, retval); } else { ovs_assert((*new_stream)->state != SCS_CONNECTING || (*new_stream)->class->connect); + pstream_replay_write_accept(pstream, *new_stream, 0); + stream_replay_open_wfd(*new_stream, 0, (*new_stream)->name); } return retval; } diff --git a/lib/stream.h b/lib/stream.h index 77bffa498..e30c51275 100644 --- a/lib/stream.h +++ b/lib/stream.h @@ -94,4 +94,16 @@ enum stream_content_type { void stream_report_content(const void *, ssize_t, enum stream_content_type, struct vlog_module *, const char *stream_name); + +/* Stream replay helpers. */ +void stream_replay_open_wfd(struct stream *, int open_result, + const char *name); +void pstream_replay_open_wfd(struct pstream *, int listen_result, + const char *name); +void stream_replay_close_wfd(struct stream *); +void pstream_replay_close_wfd(struct pstream *); +void stream_replay_write(struct stream *, const void *, int, bool is_read); +void pstream_replay_write_accept(struct pstream *, const struct stream *, + int accept_result); + #endif /* stream.h */ From patchwork Mon Apr 12 22:00:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465454 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2jj6jRcz9sRK for ; Tue, 13 Apr 2021 08:00:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5463D40525; Mon, 12 Apr 2021 22:00:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HIqkc8iLfsYn; Mon, 12 Apr 2021 22:00:38 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 0EB7D404FE; Mon, 12 Apr 2021 22:00:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DF0C3C000F; Mon, 12 Apr 2021 22:00:36 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id EC676C000F for ; Mon, 12 Apr 2021 22:00:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D437B4052B for ; Mon, 12 Apr 2021 22:00:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mmta9Mald-9P for ; Mon, 12 Apr 2021 22:00:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp4.osuosl.org (Postfix) with ESMTPS id 95597405AA for ; Mon, 12 Apr 2021 22:00:33 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id E2939E0013; Mon, 12 Apr 2021 22:00:30 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:14 +0200 Message-Id: <20210412220020.2286954-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 3/9] uuid: Allow record/replay of generated UUIDs. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This is required for the stream record/replay functionality of ovsdb-server. With record/replay of UUIDs we could record all incoming transactions and replay them later while being sure that ovsdb-server will generate exactly same UUIDs for all the data updates. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- lib/uuid.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/lib/uuid.c b/lib/uuid.c index 13d20ac64..8a16606da 100644 --- a/lib/uuid.c +++ b/lib/uuid.c @@ -26,11 +26,16 @@ #include "aes128.h" #include "entropy.h" +#include "fatal-signal.h" +#include "openvswitch/vlog.h" +#include "ovs-replay.h" #include "ovs-thread.h" #include "sha1.h" #include "timeval.h" #include "util.h" +VLOG_DEFINE_THIS_MODULE(uuid); + static struct aes128 key; static uint64_t counter[2]; BUILD_ASSERT_DECL(sizeof counter == 16); @@ -54,6 +59,63 @@ uuid_init(void) pthread_once(&once, do_init); } +/* Record/replay of uuid generation. */ +static replay_file_t uuid_replay_file; +static int uuid_replay_seqno; + +static void +uuid_replay_file_close(void *aux OVS_UNUSED) +{ + ovs_replay_file_close(uuid_replay_file); +} + +static void +uuid_replay_file_open(void) +{ + int error; + + ovs_replay_lock(); + error = ovs_replay_file_open("__uuid_generate", &uuid_replay_file, + &uuid_replay_seqno); + ovs_replay_unlock(); + if (error) { + VLOG_FATAL("failed to open uuid replay file: %s.", + ovs_strerror(error)); + } + fatal_signal_add_hook(uuid_replay_file_close, NULL, NULL, true); +} + +static void +uuid_replay_file_read(struct uuid *uuid) +{ + int norm_seqno = ovs_replay_normalized_seqno(uuid_replay_seqno); + int retval, len; + + ovs_replay_lock(); + ovs_assert(norm_seqno == ovs_replay_seqno()); + ovs_assert(ovs_replay_seqno_is_read(uuid_replay_seqno)); + + retval = ovs_replay_read(uuid_replay_file, uuid, sizeof *uuid, + &len, &uuid_replay_seqno, true); + if (retval || len != sizeof *uuid) { + VLOG_FATAL("failed to read from replay file: %s.", + ovs_strerror(retval)); + } + ovs_replay_unlock(); +} + +static void +uuid_replay_file_write(struct uuid *uuid) +{ + int retval; + + retval = ovs_replay_write(uuid_replay_file, uuid, sizeof *uuid, true); + if (retval) { + VLOG_FATAL("failed to write uuid to replay file: %s.", + ovs_strerror(retval)); + } +} + /* Generates a new random UUID in 'uuid'. * * We go to some trouble to ensure as best we can that the generated UUID has @@ -82,10 +144,16 @@ void uuid_generate(struct uuid *uuid) { static struct ovs_mutex mutex = OVS_MUTEX_INITIALIZER; + enum ovs_replay_state replay_state = ovs_replay_get_state(); uint64_t copy[2]; uuid_init(); + if (replay_state == OVS_REPLAY_READ) { + uuid_replay_file_read(uuid); + return; + } + /* Copy out the counter's current value, then increment it. */ ovs_mutex_lock(&mutex); copy[0] = counter[0]; @@ -99,6 +167,10 @@ uuid_generate(struct uuid *uuid) aes128_encrypt(&key, copy, uuid); uuid_set_bits_v4(uuid); + + if (replay_state == OVS_REPLAY_WRITE) { + uuid_replay_file_write(uuid); + } } struct uuid @@ -276,6 +348,10 @@ do_init(void) uint8_t random_seed[16]; struct timeval now; + if (ovs_replay_is_active()) { + uuid_replay_file_open(); + } + /* Get seed data. */ get_entropy_or_die(random_seed, sizeof random_seed); xgettimeofday(&now); From patchwork Mon Apr 12 22:00:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465458 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2k91fClz9sRK for ; Tue, 13 Apr 2021 08:01:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1D6C3404FD; Mon, 12 Apr 2021 22:01:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fZgsS56NloxX; Mon, 12 Apr 2021 22:01:02 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id A9F6E40562; Mon, 12 Apr 2021 22:00:57 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 80B90C000C; Mon, 12 Apr 2021 22:00:57 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 088DDC000C for ; Mon, 12 Apr 2021 22:00:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0BBB383DD5 for ; Mon, 12 Apr 2021 22:00:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jrOpUfH4tBuw for ; Mon, 12 Apr 2021 22:00:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3005B83DD1 for ; Mon, 12 Apr 2021 22:00:35 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 66588E0014; Mon, 12 Apr 2021 22:00:33 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:15 +0200 Message-Id: <20210412220020.2286954-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 4/9] ovsdb-server: Integrate stream replay engine. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This change adds support of stream record/replay functionality to ovsdb-server. Since current replay engine doesn't work well with time-based events generated locally, it will work only with standalone databases for now (raft heavily depends on time). To use this functionality run: Recording: # create a directory for replay files. mkdir replay_dir # copy current db for later use by replay cp my_db ./replay_dir/my_db ovsdb-server --replay-record=./replay_dir my_db # connect some clients and run some ovsdb transactions ovs-appctl -t ovsdb-server exit Replay: # restore db from the copy cp ./replay_dir/my_db my_db.for_replay ovsdb-server --replay=./replay_dir my_db.for_replay At this point ovsdb-server should execute all the same commands and transactions. Since the last command was 'exit' via unixctl, ovsdb-server will exit in the end. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/ovsdb-server.1.in | 2 ++ ovsdb/ovsdb-server.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in index 5a7f3ba13..fdd52e8f6 100644 --- a/ovsdb/ovsdb-server.1.in +++ b/ovsdb/ovsdb-server.1.in @@ -24,6 +24,7 @@ ovsdb\-server \- Open vSwitch database server .so lib/ssl-peer-ca-cert-syn.man .so lib/ssl-connect-syn.man .so lib/unixctl-syn.man +.so lib/ovs-replay-syn.man .so lib/common-syn.man . .SH DESCRIPTION @@ -188,6 +189,7 @@ one row in \fItable\fR.) .so lib/ssl-connect.man .SS "Other Options" .so lib/unixctl.man +.so lib/ovs-replay.man .so lib/common.man .SH "RUNTIME MANAGEMENT COMMANDS" \fBovs\-appctl\fR(8) can send commands to a running diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 29a2bace8..564085e77 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -36,6 +36,7 @@ #include "openvswitch/list.h" #include "memory.h" #include "monitor.h" +#include "ovs-replay.h" #include "ovsdb.h" #include "ovsdb-data.h" #include "ovsdb-types.h" @@ -1797,6 +1798,7 @@ parse_options(int argc, char *argv[], VLOG_OPTION_ENUMS, DAEMON_OPTION_ENUMS, SSL_OPTION_ENUMS, + OVS_REPLAY_OPTION_ENUMS, }; static const struct option long_options[] = { @@ -1812,6 +1814,7 @@ parse_options(int argc, char *argv[], {"bootstrap-ca-cert", required_argument, NULL, OPT_BOOTSTRAP_CA_CERT}, {"peer-ca-cert", required_argument, NULL, OPT_PEER_CA_CERT}, STREAM_SSL_LONG_OPTIONS, + OVS_REPLAY_LONG_OPTIONS, {"sync-from", required_argument, NULL, OPT_SYNC_FROM}, {"sync-exclude-tables", required_argument, NULL, OPT_SYNC_EXCLUDE}, {"active", no_argument, NULL, OPT_ACTIVE}, @@ -1887,6 +1890,8 @@ parse_options(int argc, char *argv[], stream_ssl_set_peer_ca_cert_file(optarg); break; + OVS_REPLAY_OPTION_HANDLERS + case OPT_SYNC_FROM: *sync_from = xstrdup(optarg); break; @@ -1945,6 +1950,7 @@ usage(void) daemon_usage(); vlog_usage(); replication_usage(); + ovs_replay_usage(); printf("\nOther options:\n" " --run COMMAND run COMMAND as subprocess then exit\n" " --unixctl=SOCKET override default control socket name\n" From patchwork Mon Apr 12 22:00:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465455 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2jn2dSyz9sRK for ; Tue, 13 Apr 2021 08:00:45 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 9D98C60B37; Mon, 12 Apr 2021 22:00:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jMFEJEiJncMK; Mon, 12 Apr 2021 22:00:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id C7D5060B24; Mon, 12 Apr 2021 22:00:41 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9529CC000C; Mon, 12 Apr 2021 22:00:41 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id DA453C000A for ; Mon, 12 Apr 2021 22:00:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B740F405E8 for ; Mon, 12 Apr 2021 22:00:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Xn9aGYYYbbu for ; Mon, 12 Apr 2021 22:00:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp4.osuosl.org (Postfix) with ESMTPS id 26B2B405CB for ; Mon, 12 Apr 2021 22:00:37 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id BB9BFE000D; Mon, 12 Apr 2021 22:00:35 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:16 +0200 Message-Id: <20210412220020.2286954-6-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 5/9] ovsdb-server: Don't update manager status if replay engine is active. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Current version or replay engine doesn't handle correctly internal time-based events that ends up in stream events. For example, updates of a database status that happens each 2.5 seconds results in updates on client monitors. Disable updates for now if replay engine is active. The very first update kept to store the initial information about the server. The proper solution would be to record time and replay it, probably, with time warping or in some other way. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/ovsdb-server.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 564085e77..b09232c65 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -258,8 +258,10 @@ main_loop(struct server_config *config, } } - /* update Manager status(es) every 2.5 seconds */ - if (time_msec() >= status_timer) { + /* update Manager status(es) every 2.5 seconds. Don't update if we're + * recording or performing replay. */ + if (status_timer == LLONG_MIN || + (!ovs_replay_is_active() && time_msec() >= status_timer)) { status_timer = time_msec() + 2500; update_remote_status(jsonrpc, remotes, all_dbs); } @@ -285,7 +287,9 @@ main_loop(struct server_config *config, if (*exiting) { poll_immediate_wake(); } - poll_timer_wait_until(status_timer); + if (!ovs_replay_is_active()) { + poll_timer_wait_until(status_timer); + } poll_block(); if (should_service_stop()) { *exiting = true; From patchwork Mon Apr 12 22:00:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465456 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2jy1HCyz9sRK for ; Tue, 13 Apr 2021 08:00:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id AB77840551; Mon, 12 Apr 2021 22:00:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QL6yyHH1tBre; Mon, 12 Apr 2021 22:00:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id DD48940526; Mon, 12 Apr 2021 22:00:47 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A4F35C000C; Mon, 12 Apr 2021 22:00:47 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id C1AFCC000A for ; Mon, 12 Apr 2021 22:00:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 60C2083DEF for ; Mon, 12 Apr 2021 22:00:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vZALpxt6sVdy for ; Mon, 12 Apr 2021 22:00:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp1.osuosl.org (Postfix) with ESMTPS id 7AC3983DD8 for ; Mon, 12 Apr 2021 22:00:40 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 126ACE000A; Mon, 12 Apr 2021 22:00:37 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:17 +0200 Message-Id: <20210412220020.2286954-7-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 6/9] jsonrpc: Disable inactivity probes if replay engine is active. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Current version of replay engine doesn't handle time-based internal events that results in stream send/receive. Disabling jsonrpc inactivity probes for now to not block process waiting for probe being sent. The proper solution would be to implement correct record/replay of time, probably, by recording time and using the time warping. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- lib/jsonrpc.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/jsonrpc.c b/lib/jsonrpc.c index 8c5126ffc..09eeb902c 100644 --- a/lib/jsonrpc.c +++ b/lib/jsonrpc.c @@ -26,6 +26,7 @@ #include "openvswitch/json.h" #include "openvswitch/list.h" #include "openvswitch/ofpbuf.h" +#include "ovs-replay.h" #include "ovs-thread.h" #include "openvswitch/poll-loop.h" #include "reconnect.h" @@ -888,7 +889,7 @@ jsonrpc_session_open_multiple(const struct svec *remotes, bool retry) reconnect_set_backoff(s->reconnect, INT_MAX, INT_MAX); } - if (!stream_or_pstream_needs_probes(name)) { + if (!stream_or_pstream_needs_probes(name) || ovs_replay_is_active()) { reconnect_set_probe_interval(s->reconnect, 0); } @@ -913,6 +914,11 @@ jsonrpc_session_open_unreliably(struct jsonrpc *jsonrpc, uint8_t dscp) reconnect_set_quiet(s->reconnect, true); reconnect_set_name(s->reconnect, jsonrpc_get_name(jsonrpc)); reconnect_set_max_tries(s->reconnect, 0); + + if (ovs_replay_is_active()) { + reconnect_set_probe_interval(s->reconnect, 0); + } + reconnect_connected(s->reconnect, time_msec()); s->dscp = dscp; s->rpc = jsonrpc; @@ -1279,6 +1285,9 @@ void jsonrpc_session_set_probe_interval(struct jsonrpc_session *s, int probe_interval) { + if (ovs_replay_is_active()) { + return; + } reconnect_set_probe_interval(s->reconnect, probe_interval); } From patchwork Mon Apr 12 22:00:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465457 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2k33QBrz9sRK for ; Tue, 13 Apr 2021 08:00:59 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 11EDB40532; Mon, 12 Apr 2021 22:00:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7UgECq6-b3gO; Mon, 12 Apr 2021 22:00:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id C329040548; Mon, 12 Apr 2021 22:00:53 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 97F53C000C; Mon, 12 Apr 2021 22:00:53 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1A400C000F for ; Mon, 12 Apr 2021 22:00:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 54EC060B13 for ; Mon, 12 Apr 2021 22:00:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EboRubwN8ZLQ for ; Mon, 12 Apr 2021 22:00:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp3.osuosl.org (Postfix) with ESMTPS id D6BF360B32 for ; Mon, 12 Apr 2021 22:00:42 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 75019E0014; Mon, 12 Apr 2021 22:00:40 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:18 +0200 Message-Id: <20210412220020.2286954-8-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 7/9] ovsdb-server.at: Add unit test for record/replay. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- tests/ovsdb-server.at | 149 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index 926abce3a..de390fe3b 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -2014,3 +2014,152 @@ six 6 ]) AT_CLEANUP + +AT_BANNER([OVSDB -- ovsdb-server stream record/replay]) + +AT_SETUP([ovsdb-server record/replay]) +AT_KEYWORDS([ovsdb server record replay]) +on_exit 'kill `cat *.pid`' +ordinal_schema > schema +AT_CHECK([ovsdb-tool create db schema], [0], [ignore], [ignore]) + +dnl Create a directory for replay files. +AT_CHECK([mkdir replay_dir]) + +dnl Make a copy of a database for later replay. +AT_CHECK([cp db ./replay_dir/db.copy]) + +dnl Starting a dummy server only to reserve some tcp port. +AT_CHECK([cp db db.tmp]) +AT_CHECK([ovsdb-server -vfile -vvlog:off --log-file=listener.log dnl + --detach --no-chdir dnl + --pidfile=pid2 --unixctl=unixctl2 dnl + --remote=ptcp:0:127.0.0.1 dnl + db.tmp], [0], [stdout], [stderr]) +PARSE_LISTENING_PORT([listener.log], [BAD_TCP_PORT]) + +dnl Start ovsdb-server with recording enabled. +dnl Trying to start a tcp session on already used port to record the error. +AT_CHECK([ovsdb-server --replay-record=./replay_dir dnl + -vfile -vvlog:off -vjsonrpc:file:dbg --log-file=1.log dnl + --detach --no-chdir --pidfile dnl + --remote=punix:db.sock dnl + --remote=ptcp:$BAD_TCP_PORT:127.0.0.1 dnl + --remote=ptcp:0:127.0.0.1 dnl + db], [0], [stdout], [stderr]) +CHECK_DBS([ordinals +]) +PARSE_LISTENING_PORT([1.log], [TCP_PORT]) + +dnl Start a monitor on the 'ordinals' db to check recording of this kind +dnl of messages. +AT_CHECK([ovsdb-client -vfile -vvlog:off --detach --no-chdir dnl + --pidfile=monitor.pid --log-file=monitor.log dnl + --db-change-aware --no-headings dnl + monitor tcp:127.0.0.1:$TCP_PORT dnl + ordinals ordinals number name dnl + > monitor.stdout 2> monitor.stderr]) +OVS_WAIT_UNTIL([test -e monitor.pid]) + +dnl Do a bunch of random transactions. +AT_CHECK( + [[for pair in 'zero 0' 'one 1' 'two 2' 'three 3' 'four 4' 'five 5'; do + set -- $pair + if test "$2" -eq "5"; then + # killing the monitor to check if this correctly recorded. + kill -9 $(cat monitor.pid) + fi + ovsdb-client --db-change-aware transact unix:db.sock ' + ["ordinals", + {"op": "insert", + "table": "ordinals", + "row": {"name": "'$1'", "number": '$2'}}, + {"op": "comment", + "comment": "add row for '"$pair"'"}]' + ovsdb-client transact unix:db.sock ' + ["ordinals", + {"op": "delete", + "table": "ordinals", + "where": [["number", "==", '$2']]}, + {"op": "comment", + "comment": "delete row for '"$2"'"}]' + ovsdb-client transact unix:db.sock ' + ["ordinals", + {"op": "insert", + "table": "ordinals", + "row": {"name": "'$1'", "number": '$2'}}, + {"op": "comment", + "comment": "add back row for '"$pair"'"}]' + done]], + [0], [stdout]) + +AT_CHECK([ovsdb-client dump unix:db.sock ordinals | uuidfilt], 0, [dnl +ordinals table +_uuid name number +------------------------------------ ----- ------ +<0> five 5 +<1> four 4 +<2> one 1 +<3> three 3 +<4> two 2 +<5> zero 0 +]) + +AT_CHECK([uuidfilt monitor.stdout | sed '/^$/d'], [0], [dnl +<0> insert 0 zero +<0> delete 0 zero +<1> insert 0 zero +<2> insert 1 one +<2> delete 1 one +<3> insert 1 one +<4> insert 2 two +<4> delete 2 two +<5> insert 2 two +<6> insert 3 three +<6> delete 3 three +<7> insert 3 three +<8> insert 4 four +<8> delete 4 four +<9> insert 4 four +]) +OVSDB_SERVER_SHUTDOWN +OVSDB_SERVER_SHUTDOWN2 + +dnl Starting a replay. +AT_CHECK([ovsdb-server --replay=./replay_dir dnl + -vfile -vvlog:off -vjsonrpc:file:dbg --log-file=2.log dnl + --detach --no-chdir --pidfile dnl + --remote=punix:db.sock dnl + --remote=ptcp:$BAD_TCP_PORT:127.0.0.1 dnl + --remote=ptcp:0:127.0.0.1 dnl + ./replay_dir/db.copy], [0], [stdout], [stderr]) + +dnl Waiting for process termination. Process should exit after correct +dnl processing of the 'exit' unixctl command from the recorded session. +OVS_WAIT_WHILE([test -e ovsdb-server.pid]) + +dnl Stripping out timestamps from database files. Also clearing record +dnl hashes in database files, since dates inside are different. +m4_define([CLEAN_DB_FILE], + [sed 's/\(OVSDB JSON [[0-9]]*\).*$/\1/g' $1 | dnl + sed 's/"_date":[[0-9]]*/"_date":/g' > $2]) + +CLEAN_DB_FILE([db], [db.clear]) +CLEAN_DB_FILE([./replay_dir/db.copy], [./replay_dir/db.copy.clear]) + +dnl Stripping out timestamps, PIDs and poll_loop warnings from the log. +dnl Also stripping socket_util errors as sockets are not used in replay. +m4_define([CLEAN_LOG_FILE], + [sed 's/[[0-9\-]]*T[[0-9:\.]]*Z|[[0-9]]*\(|.*$\)/\1/g' $1 | dnl + sed '/|poll_loop|/d' | dnl + sed '/|socket_util|/d' | dnl + sed 's/[[0-9]]*\.ctl/\.ctl/g'> $2]) + +CLEAN_LOG_FILE([1.log], [1.log.clear]) +CLEAN_LOG_FILE([2.log], [2.log.clear]) + +dnl Checking that databases and logs are equal. +AT_CHECK([diff db.clear ./replay_dir/db.copy.clear]) +AT_CHECK([diff 1.log.clear 2.log.clear]) + +AT_CLEANUP From patchwork Mon Apr 12 22:00:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465459 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2kV0dDYz9sRK for ; Tue, 13 Apr 2021 08:01:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 93CAC405B8; Mon, 12 Apr 2021 22:01:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KGoHl1Br-3wJ; Mon, 12 Apr 2021 22:01:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTP id 822E440644; Mon, 12 Apr 2021 22:01:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 59776C000A; Mon, 12 Apr 2021 22:01:18 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2DC3EC000F for ; Mon, 12 Apr 2021 22:01:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 73CA683DE6 for ; Mon, 12 Apr 2021 22:00:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c2dbA1Oof3g4 for ; Mon, 12 Apr 2021 22:00:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp1.osuosl.org (Postfix) with ESMTPS id 4E73283E07 for ; Mon, 12 Apr 2021 22:00:44 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id C8461E0013; Mon, 12 Apr 2021 22:00:42 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:19 +0200 Message-Id: <20210412220020.2286954-9-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 8/9] ovsdb-client: Integrate record/replay functionality. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This is primarily to be able to test recording of client connections. Unit test added accordingly. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/ovsdb-client.1.in | 2 + ovsdb/ovsdb-client.c | 5 +++ tests/ovsdb-client.at | 89 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+) diff --git a/ovsdb/ovsdb-client.1.in b/ovsdb/ovsdb-client.1.in index ee329d2d7..91e0b3e25 100644 --- a/ovsdb/ovsdb-client.1.in +++ b/ovsdb/ovsdb-client.1.in @@ -71,6 +71,7 @@ ovsdb\-client \- command-line interface to \fBovsdb-server\fR(1) .so lib/ssl-syn.man .so lib/ssl-bootstrap-syn.man .so lib/ssl-connect-syn.man +.so lib/ovs-replay-syn.man .so lib/common-syn.man . .SH DESCRIPTION @@ -370,6 +371,7 @@ effect. .SS "SSL Connection Options" .so lib/ssl-connect.man .SS "Other Options" +.so lib/ovs-replay.man .so lib/common.man .SH "SEE ALSO" . diff --git a/ovsdb/ovsdb-client.c b/ovsdb/ovsdb-client.c index ba28e36d7..ffa8f8df2 100644 --- a/ovsdb/ovsdb-client.c +++ b/ovsdb/ovsdb-client.c @@ -38,6 +38,7 @@ #include "jsonrpc.h" #include "lib/table.h" #include "log.h" +#include "ovs-replay.h" #include "ovsdb.h" #include "ovsdb-data.h" #include "ovsdb-error.h" @@ -307,6 +308,7 @@ parse_options(int argc, char *argv[]) DAEMON_OPTION_ENUMS, TABLE_OPTION_ENUMS, SSL_OPTION_ENUMS, + OVS_REPLAY_OPTION_ENUMS, }; static const struct option long_options[] = { {"help", no_argument, NULL, 'h'}, @@ -325,6 +327,7 @@ parse_options(int argc, char *argv[]) STREAM_SSL_LONG_OPTIONS, #endif TABLE_LONG_OPTIONS, + OVS_REPLAY_LONG_OPTIONS, {NULL, 0, NULL, 0}, }; char *short_options = ovs_cmdl_long_options_to_short_options(long_options); @@ -352,6 +355,7 @@ parse_options(int argc, char *argv[]) DAEMON_OPTION_HANDLERS TABLE_OPTION_HANDLERS(&table_style) STREAM_SSL_OPTION_HANDLERS + OVS_REPLAY_OPTION_HANDLERS case OPT_BOOTSTRAP_CA_CERT: stream_ssl_set_ca_cert_file(optarg, true); @@ -467,6 +471,7 @@ usage(void) printf(" --timestamp timestamp \"monitor\" output"); daemon_usage(); vlog_usage(); + ovs_replay_usage(); printf("\nOther options:\n" " -h, --help display this help message\n" " -V, --version display version information\n"); diff --git a/tests/ovsdb-client.at b/tests/ovsdb-client.at index 5e3b26aea..e4f1fcfc4 100644 --- a/tests/ovsdb-client.at +++ b/tests/ovsdb-client.at @@ -195,3 +195,92 @@ _uuid,name,number OVS_APP_EXIT_AND_WAIT([ovsdb-server]) AT_CLEANUP + +AT_SETUP([ovsdb-client record/replay]) +AT_KEYWORDS([ovsdb client record replay]) + +on_exit 'kill `cat *.pid`' + +dnl Create a database. +ordinal_schema > schema +touch .db.~lock~ +AT_CHECK([ovsdb-tool create db schema]) + +dnl Start the database server. +AT_CHECK([ovsdb-server -vfile -vvlog:off --detach --no-chdir --pidfile dnl + --log-file --remote=punix:db.sock db], [0]) +AT_CAPTURE_FILE([ovsdb-server.log]) + +dnl Start a monitor on the 'ordinals' db with recording enabled. +AT_CHECK([mkdir replay_dir]) +AT_CHECK([ovsdb-client --replay-record=./replay_dir dnl + -vfile -vvlog:off --detach --no-chdir dnl + --pidfile --log-file=monitor.log dnl + --db-change-aware --no-headings dnl + monitor unix:db.sock dnl + ordinals ordinals number name dnl + > monitor.stdout 2> monitor.stderr]) +AT_CAPTURE_FILE([monitor.log]) + +dnl Put some data in the database. +AT_CHECK( + [[for txn in 'transact zero 0' \ + 'transact two 2' \ + 'transact four 4' + do + set -- $txn + ovsdb-client $1 ' + ["ordinals", + {"op": "insert", + "table": "ordinals", + "row": {"name": "'$2'", "number": '$3'}}, + {"op": "comment", + "comment": "add row for '"$pair"'"}]' + done | uuidfilt]], [0], +[[[{"uuid":["uuid","<0>"]},{}] +[{"uuid":["uuid","<1>"]},{}] +[{"uuid":["uuid","<2>"]},{}] +]], [ignore]) + +AT_CHECK([ovsdb-client -f csv dump | sort -t, -k 3 | uuidfilt], [0], [dnl +ordinals table +<0>,zero,0 +<1>,two,2 +<2>,four,4 +_uuid,name,number +]) + +dnl Stopping the server. +OVS_APP_EXIT_AND_WAIT([ovsdb-server]) +dnl ovsdb-client should exit by itself after disconnection form the server. +OVS_WAIT_WHILE([test -e ovsdb-client.pid]) + +dnl Starting replay. +AT_CHECK([ovsdb-client --replay=./replay_dir dnl + -vfile -vvlog:off --detach --no-chdir dnl + --pidfile --log-file=monitor-replay.log dnl + --db-change-aware --no-headings dnl + monitor unix:db.sock dnl + ordinals ordinals number name dnl + > monitor-replay.stdout 2> monitor-replay.stderr]) + +dnl Waiting for client to exit the same way as it exited during recording. +OVS_WAIT_WHILE([test -e ovsdb-client.pid]) + +AT_CHECK([diff monitor.stdout monitor-replay.stdout]) +AT_CHECK([diff monitor.stderr monitor-replay.stderr]) + +dnl Stripping out timestamps, PIDs and poll_loop warnings from the log. +dnl Also stripping socket_util errors as sockets are not used in replay. +m4_define([CLEAN_LOG_FILE], + [sed 's/[[0-9\-]]*T[[0-9:\.]]*Z|[[0-9]]*\(|.*$\)/\1/g' $1 | dnl + sed '/|poll_loop|/d' | dnl + sed '/|socket_util|/d' | dnl + sed 's/[[0-9]]*\.ctl/\.ctl/g'> $2]) + +CLEAN_LOG_FILE([monitor.log], [monitor.log.clear]) +CLEAN_LOG_FILE([monitor-replay.log], [monitor-replay.log.clear]) + +AT_CHECK([diff monitor.log.clear monitor-replay.log.clear]) + +AT_CLEANUP From patchwork Mon Apr 12 22:00:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1465461 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FK2l03vkXz9sRK for ; Tue, 13 Apr 2021 08:01:48 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1992A4056A; Mon, 12 Apr 2021 22:01:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aqEhhZTnamnn; Mon, 12 Apr 2021 22:01:41 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id 8284C404F7; Mon, 12 Apr 2021 22:01:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 58D72C000C; Mon, 12 Apr 2021 22:01:40 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 52343C000A for ; Mon, 12 Apr 2021 22:01:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 3836F4065F for ; Mon, 12 Apr 2021 22:00:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kkCAqzKZGsFL for ; Mon, 12 Apr 2021 22:00:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by smtp4.osuosl.org (Postfix) with ESMTPS id 06711405CB for ; Mon, 12 Apr 2021 22:00:50 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 7F40DE0003; Mon, 12 Apr 2021 22:00:48 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Apr 2021 00:00:20 +0200 Message-Id: <20210412220020.2286954-10-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210412220020.2286954-1-i.maximets@ovn.org> References: <20210412220020.2286954-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 9/9] docs: Add a topic about record/replay with ovsdb-server. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Also added a NEWS entry. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- Documentation/automake.mk | 1 + Documentation/topics/index.rst | 1 + Documentation/topics/record-replay.rst | 138 +++++++++++++++++++++++++ NEWS | 4 + 4 files changed, 144 insertions(+) create mode 100644 Documentation/topics/record-replay.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index ea3475f35..0fd7fdf7e 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -54,6 +54,7 @@ DOC_SOURCE = \ Documentation/topics/ovs-extensions.rst \ Documentation/topics/ovsdb-replication.rst \ Documentation/topics/porting.rst \ + Documentation/topics/record-replay.rst \ Documentation/topics/tracing.rst \ Documentation/topics/userspace-tso.rst \ Documentation/topics/windows.rst \ diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst index 08af3a24d..0036567eb 100644 --- a/Documentation/topics/index.rst +++ b/Documentation/topics/index.rst @@ -48,6 +48,7 @@ OVS dpdk/index windows language-bindings + record-replay testing tracing userspace-tso diff --git a/Documentation/topics/record-replay.rst b/Documentation/topics/record-replay.rst new file mode 100644 index 000000000..5fd672f1a --- /dev/null +++ b/Documentation/topics/record-replay.rst @@ -0,0 +1,138 @@ +.. + Copyright 2021, Red Hat, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +============================ +Debugging with Record/Replay +============================ + +The ``ovs-replay`` library provides a set of internal functions for recording +certain events for later replay. This library is integrated into the +``stream`` and some other modules to record all incoming data across all +streams (ssl, tcp, unixctl) of applications baed on Open vSwitch libraries +and play these streams later for debugging or performance testing purposes. + +Support for this feature is currently integrated into the ``ovsdb-server`` and +``ovsdb-client`` applications. As a result, this allows to record lifecycle +of the ``ovsdb-server`` process in large OVN deployments. Later, by using only +the recorded data, the user can replay transactions and connections that +occurred in a large deployment on their local PC. At the same time it is +possible to tweak various log levels, run a process under debugger or tracer, +measure performance with ``perf``, and so on. + + .. note:: + + The current version of record/replay engine does not work correctly with + internal time-based events that leats to communications with other + processes. For this reason it can not be used with clustered databases + (RAFT implementation is heavily time dependent). + In addition, recording automatically disables inactivity probes on + JSONRPC connections and updates for the Manager status in a _Server + database. + +High-level feature overview was presented on Open vSwitch and OVN 2020 Fall +Conference: `Debugging OVSDB with stream record/replay`__ + +__ https://www.openvswitch.org/support/ovscon2020/slides/Debugging-OVSDB-with-stream-record_replay.pdf + +Recording ovsdb-server events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To start recording events for the ``ovsdb-server`` process, there is a special +command line argument ``--replay-record``. Before starting the database +server, make sure that you have a copy of a database file, so you can use it +for replay later. Here are the general steps to take: + +1. Create a directory where the replay files will be stored:: + + $ mkdir replay-dir + $ REPLAY_DIR=$(pwd)/replay-dir + +2. Copy the current database file for later use:: + + $ cp my_database $REPLAY_DIR/ + +3. Run ``ovsdb-server`` with recording enabled:: + + $ ovsdb-server --replay-record=$REPLAY_DIR my_database + +4. Work with the database as usual. + +5. Stop the ``ovsdb-server`` process at the end (it is important to send an + ``exit`` command so that during replay the process will exit in the end + too):: + + $ ovs-appctl -t ovsdb-server exit + +After that ``$REPLAY_DIR`` should contain replay files with recorded data. + +Replay of recorded session +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +During replay, the ``ovsdb-server`` will receive all the same connections, +transactions and commands as it had at the time of recording, but it will not +create any actual network/socket connections and will not communicate with +any other process. Everything will be read from the replay files. + +Since there is no need to wait for IPC, all events will be received one by one +without any delays, so the application will process them as quickly as +possible. This can be used as a performance test where the user can measure how +quickly the ``ovsdb-server`` can handle some workload recorded in a real +deployment. + +The command line argument to start a replay session is ``--replay``. The steps +will look like this: + +1. Restore the database file from a previous copy:: + + $ cp $REPLAY_DIR/my_database my_database + +2. Start ``ovsdb-server`` with the same set of arguments as in the recording + stage, except for ``--replay-record``:: + + $ ovsdb-server --replay=$REPLAY_DIR my_database + +3. The process should exit in the end when the ``exit`` command is replayed. + +On step 2 it is possible to add extra logging arguments to debug some recorded +issue, or run the process under debugger. It's also possible to replay with +a different version of ``ovsdb-server`` binary as long as this does not affect +the data that goes in and out of the process, e.g. pure performance +optimizations. + +~~~~~~~~~~~ +Limitations +~~~~~~~~~~~ + +The record/replay engine has the following limitations: + +1. Record/Replay of clustered databases is not supported. + +2. Inactivity probes on JSONRPC connections are suppressed. + +3. Manager status updates suppressed in ``ovsdb-server``. + +To remove above limitations, it is necessary to implement correct handling of +internally generated time-based events. (possibly by recording of time and +subsequent time warping). diff --git a/NEWS b/NEWS index 95cf922aa..28b558162 100644 --- a/NEWS +++ b/NEWS @@ -1,5 +1,9 @@ Post-v2.15.0 --------------------- + - OVSDB: + * New command line options --replay-record/--replay for ovsdb-server + and ovsdb-client to record and replay all the incoming transactions, + monitors, etc. More datails in Documentation/topics/record-replay.rst. - In ovs-vsctl and vtep-ctl, the "find" command now accept new operators {in} and {not-in}. - Userspace datapath: