From patchwork Wed Oct 30 13:50:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004217 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpS25mbWz1xwF for ; Thu, 31 Oct 2024 00:50:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 95CDD40622; Wed, 30 Oct 2024 13:50:56 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 31aZZDihu-Cg; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 74FD94031B Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 74FD94031B; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5028DC08A6; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2DCFBC08A6 for ; Wed, 30 Oct 2024 13:50:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id ED829408F1 for ; Wed, 30 Oct 2024 13:50:52 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 8lb2eDMG8xLn for ; Wed, 30 Oct 2024 13:50:51 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.66; helo=mail-ed1-f66.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org DE1AD400FF Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org DE1AD400FF Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by smtp2.osuosl.org (Postfix) with ESMTPS id DE1AD400FF for ; Wed, 30 Oct 2024 13:50:50 +0000 (UTC) Received: by mail-ed1-f66.google.com with SMTP id 4fb4d7f45d1cf-5c40aea5c40so1698321a12.0 for ; Wed, 30 Oct 2024 06:50:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296249; x=1730901049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lLDdpauerpjrpU4eaEtk+kRQXaPhC+ZZ0Abl7Uwn984=; b=gdi49/WGxzii7GSOSbu28zZS3quEO69IOtnWOTgfZou84wLMEHkhXh10uzb1mzQwmO zK72l8qlhOeE9Y06nrIIMyfijmlU2v14SewCmhxiPd4/I+j4Fps0emXP0PGQQPNr1nyB A5xvKSoB9zjq0pF3OSrSaP6SVFlqFgQTLr85FQHvEXRHPmp5QwrBhTollFQKc9+2/fbk wrZx6SrYOFXoYhnEE6Xt9n/zAJEYbjsZf1adjqquzpltOvRTwPpPK4ikSVDw59c0WZqs XRwBCaI3BXdmORJblOi3JV3D5HPiIn2UFP+7bZbA2woFfKBa7uH6oxq7QQr9fWAtMPX8 tVgQ== X-Gm-Message-State: AOJu0Yzl19UVFjRRGtgybdeR3noK7LemCkXrt6cpUxk2RSp706O2gUbp 3hvmUDTtzfIZ4y3GXfty7WXJ2BRN1qQyrigrXckmf47aAUMjvTsGESrlgYMJ X-Google-Smtp-Source: AGHT+IEI8100NgsPfoy3NpsrTPDL+lGgZg21pMJYR/xkjBCSV7Yao4xzKc+MI3GZsptVfrdTtYebgQ== X-Received: by 2002:a05:6402:524b:b0:5c9:88d3:b1bd with SMTP id 4fb4d7f45d1cf-5cd564afb89mr2110853a12.12.1730296248564; Wed, 30 Oct 2024 06:50:48 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:50:48 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:30 +0100 Message-ID: <20241030135043.3139987-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 1/9] ipsec: Add a helper function to run commands from the monitor. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Until now, functions that needed to call external programs like openssl or ipsec commands were using subprocess commands directly. Most of these calls had no failure checks or any logging making it hard to understand what is happening inside the daemon when something doesn't work as intended. Some commands also had a chance to not read the command output in full. That might sound like not a big problem, but in practice it causes ovs-monitor-ipsec to deadlock pluto and itself with certain versions of Libreswan (mainly Libreswan 5+). The order of events is following: 1. ovs-monitor-ipsec calls ipsec status redirecting the output to a pipe. 2. ipsec status calls ipsec whack. 3. ipsec whack connects to pluto and asks for status. 4. ovs-monitor-ipsec doesn't read the pipe in full. 5. ipsec whack blocks on write to the other side of the pipe when it runs out of buffer space. 6. pluto blocks on sendmsg to ipsec whack for the same reason. 7. ovs-monitor-ipsec calls another ipsec command and blocks on connection to pluto. In this scenario the running process is at the mercy of garbage collector and it doesn't run because we're blocked on calling another ipsec command. All the processes are completely blocked and will not do any work until ipsec whack is killed. With this change we're introducing a new function that will be used for all the external process execution commands and will read the full output before returning, avoiding the deadlock. It will also log all the failures as warnings and the commands themselves at the debug level. We'll be adding more logic into this function in later commits as well, so it will not stay that simple. Signed-off-by: Ilya Maximets Acked-by: Roi Dayan --- ipsec/ovs-monitor-ipsec.in | 290 +++++++++++++++++-------------------- 1 file changed, 131 insertions(+), 159 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 37c509ac6..4885e048f 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -84,6 +84,28 @@ monitor = None xfrm = None +def run_command(args, description=None): + """ This function runs the process args[0] with args[1:] arguments + and returns a tuple: return-code, stdout, stderr. """ + + if not description: + description = args[-1] + + vlog.dbg("Running %s" % args) + proc = subprocess.Popen(args, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + pout, perr = proc.communicate() + + if proc.returncode or perr: + vlog.warn("Failed to %s; exit code: %d" + % (description, proc.returncode)) + vlog.warn("cmdline: %s" % proc.args) + vlog.warn("stderr: %s" % perr) + vlog.warn("stdout: %s" % pout) + + return proc.returncode, pout.decode(), perr.decode() + + class XFRM(object): """This class is a simple wrapper around ip-xfrm (8) command line utility. We are using this class only for informational purposes @@ -99,13 +121,14 @@ class XFRM(object): where is destination IPv4 address and is SELECTOR of the IPsec policy.""" policies = {} - proc = subprocess.Popen([self.IP, 'xfrm', 'policy'], - stdout=subprocess.PIPE) - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - a = line.split(" ") + + ret, pout, perr = run_command([self.IP, 'xfrm', 'policy'], + "get XFRM policies") + if ret: + return policies + + for line in pout.splitlines(): + a = line.strip().split(" ") if len(a) >= 4 and a[0] == "src" and a[2] == "dst": dst = (a[3].split("/"))[0] if dst not in policies: @@ -122,13 +145,14 @@ class XFRM(object): in a dictionary where is destination IPv4 address and is SELECTOR.""" securities = {} - proc = subprocess.Popen([self.IP, 'xfrm', 'state'], - stdout=subprocess.PIPE) - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - a = line.split(" ") + + ret, pout, perr = run_command([self.IP, 'xfrm', 'state'], + "get XFRM state") + if ret: + return securities + + for line in pout.splitlines(): + a = line.strip().split(" ") if len(a) >= 4 and a[0] == "sel" \ and a[1] == "src" and a[3] == "dst": remote_ip = a[4].rstrip().split("/")[0] @@ -242,7 +266,7 @@ conn prevent_unencrypted_vxlan f.close() vlog.info("Restarting StrongSwan") - subprocess.call([self.IPSEC, "restart"]) + run_command([self.IPSEC, "restart"], "restart StrongSwan") def get_active_conns(self): """This function parses output from 'ipsec status' command. @@ -252,13 +276,13 @@ conn prevent_unencrypted_vxlan sample line from the parsed outpus as . """ conns = {} - proc = subprocess.Popen([self.IPSEC, 'status'], stdout=subprocess.PIPE) + ret, pout, perr = run_command([self.IPSEC, 'status'], + "get active connections") + if ret: + return conns - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - tunnel_name = line.split(":") + for line in pout.splitlines(): + tunnel_name = line.strip().split(":") if len(tunnel_name) < 2: continue m = re.match(r"(.*)(-in-\d+|-out-\d+|-\d+).*", tunnel_name[0]) @@ -341,15 +365,11 @@ conn prevent_unencrypted_vxlan Once strongSwan vici bindings will be distributed with major Linux distributions this function could be simplified.""" vlog.info("Refreshing StrongSwan configuration") - proc = subprocess.Popen([self.IPSEC, "update"], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - outs, errs = proc.communicate() - if proc.returncode != 0: - vlog.err("StrongSwan failed to update configuration:\n" - "%s \n %s" % (str(outs), str(errs))) - - subprocess.call([self.IPSEC, "rereadsecrets"]) + + run_command([self.IPSEC, "update"], + "update StrongSwan's configuration") + run_command([self.IPSEC, "rereadsecrets"], "re-read secrets") + # "ipsec update" command does not remove those tunnels that were # updated or that disappeared from the ipsec.conf file. So, we have # to manually remove them by calling "ipsec stroke down-nb " @@ -382,7 +402,8 @@ conn prevent_unencrypted_vxlan if not tunnel or tunnel.version != ver: vlog.info("%s is outdated %u" % (conn, ver)) - subprocess.call([self.IPSEC, "stroke", "down-nb", conn]) + run_command([self.IPSEC, "stroke", "down-nb", conn], + "stroke the outdated %s" % conn) class LibreSwanHelper(object): @@ -460,13 +481,11 @@ conn prevent_unencrypted_vxlan # Collect version infromation self.IPSEC = libreswan_root_prefix + "/usr/sbin/ipsec" self.IPSEC_AUTO = [self.IPSEC] - proc = subprocess.Popen([self.IPSEC, "--version"], - stdout=subprocess.PIPE, - encoding="latin1") - pout, perr = proc.communicate() - v = re.match("^Libreswan v?(.*)$", pout) + ret, pout, perr = run_command([self.IPSEC, "--version"], + "get Libreswan's version") try: + v = re.match("^Libreswan v?(.*)$", pout.strip()) version = int(v.group(1).split(".")[0]) except: version = 0 @@ -513,7 +532,7 @@ conn prevent_unencrypted_vxlan f.close() vlog.info("Restarting LibreSwan") - subprocess.call([self.IPSEC, "restart"]) + run_command([self.IPSEC, "restart"], "restart Libreswan") def config_init(self): self.conf_file = open(self.IPSEC_CONF, "w") @@ -599,8 +618,10 @@ conn prevent_unencrypted_vxlan def refresh(self, monitor): vlog.info("Refreshing LibreSwan configuration") - subprocess.call(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, - "--config", self.IPSEC_CONF, "--rereadsecrets"]) + run_command(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--rereadsecrets"], + "re-read secrets") tunnels = set(monitor.tunnels.keys()) # Delete old connections @@ -627,9 +648,10 @@ conn prevent_unencrypted_vxlan if not tunnel or tunnel.version != ver: vlog.info("%s is outdated %u" % (conn, ver)) - subprocess.call(self.IPSEC_AUTO + ["--ctlsocket", - self.IPSEC_CTL, "--config", - self.IPSEC_CONF, "--delete", conn]) + run_command(self.IPSEC_AUTO + + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--delete", conn], "delete %s" % conn) elif ifname in tunnels: tunnels.remove(ifname) @@ -649,43 +671,43 @@ conn prevent_unencrypted_vxlan # Update shunt policy if changed if monitor.conf_in_use["skb_mark"] != monitor.conf["skb_mark"]: if monitor.conf["skb_mark"]: - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_gre"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_geneve"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_stt"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_vxlan"]) else: - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_gre"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_geneve"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_stt"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", @@ -700,14 +722,13 @@ conn prevent_unencrypted_vxlan sample line from the parsed outpus as . """ conns = {} - proc = subprocess.Popen([self.IPSEC, 'status', '--ctlsocket', - self.IPSEC_CTL], stdout=subprocess.PIPE) - - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break + ret, pout, perr = run_command([self.IPSEC, 'status', + '--ctlsocket', self.IPSEC_CTL], + "get active connections") + if ret: + return conns + for line in pout.splitlines(): m = re.search(r"#\d+: \"(.*)\".*", line) if not m: continue @@ -732,15 +753,12 @@ conn prevent_unencrypted_vxlan # the "ipsec auto --start" command is lost. Just retry to make sure # the command is received by LibreSwan. while True: - proc = subprocess.Popen(self.IPSEC_AUTO + - ["--config", self.IPSEC_CONF, - "--ctlsocket", self.IPSEC_CTL, - "--start", - "--asynchronous", conn], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - perr = str(proc.stderr.read()) - pout = str(proc.stdout.read()) + ret, pout, perr = run_command(self.IPSEC_AUTO + + ["--config", self.IPSEC_CONF, + "--ctlsocket", self.IPSEC_CTL, + "--start", + "--asynchronous", conn], + "start %s" % conn) if not re.match(r".*Connection refused.*", perr) and \ not re.match(r".*need --listen.*", pout): break @@ -748,101 +766,59 @@ conn prevent_unencrypted_vxlan if re.match(r".*[F|f]ailed to initiate connection.*", pout): vlog.err('Failed to initiate connection through' ' Interface %s.\n' % (conn.split('-')[0])) - vlog.err(pout) + vlog.err("stdout: %s" % pout) def _nss_clear_database(self): """Remove all OVS IPsec related state from the NSS database""" - try: - proc = subprocess.Popen(['certutil', '-L', '-d', - self.IPSEC_D], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - universal_newlines=True) - lines = proc.stdout.readlines() - - for line in lines: - s = line.strip().split() - if len(s) < 1: - continue - name = s[0] - if name.startswith(self.CERT_PREFIX): - self._nss_delete_cert(name) - elif name.startswith(self.CERTKEY_PREFIX): - self._nss_delete_cert_and_key(name) + ret, pout, perr = run_command(['certutil', '-L', '-d', self.IPSEC_D], + "clear NSS database") + if ret: + return - except Exception as e: - vlog.err("Failed to clear NSS database.\n" + str(e)) + for line in pout.splitlines(): + s = line.strip().split() + if len(s) < 1: + continue + name = s[0] + if name.startswith(self.CERT_PREFIX): + self._nss_delete_cert(name) + elif name.startswith(self.CERTKEY_PREFIX): + self._nss_delete_cert_and_key(name) def _nss_import_cert(self, cert, name, cert_type): """Cert_type is 'CT,,' for the CA certificate and 'P,P,P' for the normal certificate.""" - try: - proc = subprocess.Popen(['certutil', '-A', '-a', '-i', cert, - '-d', self.IPSEC_D, '-n', - name, '-t', cert_type], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - except Exception as e: - vlog.err("Failed to import certificate into NSS.\n" + str(e)) + run_command(['certutil', '-A', '-a', '-i', cert, '-d', self.IPSEC_D, + '-n', name, '-t', cert_type], + "import certificate %s into NSS" % name) def _nss_delete_cert(self, name): - try: - proc = subprocess.Popen(['certutil', '-D', '-d', - self.IPSEC_D, '-n', name], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - except Exception as e: - vlog.err("Failed to delete certificate from NSS.\n" + str(e)) + run_command(['certutil', '-D', '-d', self.IPSEC_D, '-n', name], + "delete certificate %s from NSS" % name) def _nss_import_cert_and_key(self, cert, key, name): - try: - # Avoid deleting other files - path = os.path.abspath('/tmp/%s.p12' % name) - if not path.startswith('/tmp/'): - raise Exception("Illegal certificate name!") - - # Create p12 file from pem files - proc = subprocess.Popen(['openssl', 'pkcs12', '-export', - '-in', cert, '-inkey', key, '-out', - path, '-name', name, '-passout', 'pass:'], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - # Load p12 file to the database - proc = subprocess.Popen(['pk12util', '-i', path, '-d', - self.IPSEC_D, '-W', ''], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - except Exception as e: - vlog.err("Import cert and key failed.\n" + str(e)) + # Avoid deleting other files + path = os.path.abspath('/tmp/%s.p12' % name) + if not path.startswith('/tmp/'): + vlog.err("Illegal certificate name '%s'!" % name) + return + + if run_command(['openssl', 'pkcs12', '-export', + '-in', cert, '-inkey', key, + '-out', path, '-name', name, + '-passout', 'pass:'], + "create p12 file from pem files")[0]: + return + + # Load p12 file to the database + run_command(['pk12util', '-i', path, '-d', self.IPSEC_D, '-W', ''], + "load p12 file to the NSS database") os.remove(path) def _nss_delete_cert_and_key(self, name): - try: - # Delete certificate and private key - proc = subprocess.Popen(['certutil', '-F', '-d', - self.IPSEC_D, '-n', name], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - except Exception as e: - vlog.err("Delete cert and key failed.\n" + str(e)) + # Delete certificate and private key + run_command(['certutil', '-F', '-d', self.IPSEC_D, '-n', name], + "delete certificate and private key for %s" % name) class IPsecTunnel(object): @@ -1220,19 +1196,15 @@ class IPsecMonitor(object): self.ike_helper.refresh(self) def _get_cn_from_cert(self, cert): - try: - proc = subprocess.Popen(['openssl', 'x509', '-noout', '-subject', - '-nameopt', 'RFC2253', '-in', cert], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - m = re.search(r"CN=(.+?),", proc.stdout.readline().decode()) - if not m: - raise Exception("No CN in the certificate subject.") - except Exception as e: - vlog.warn(str(e)) + ret, pout, perr = run_command(['openssl', 'x509', '-noout', '-subject', + '-nameopt', 'RFC2253', '-in', cert], + "get certificate %s options" % cert) + if ret: + return None + + m = re.search(r"CN=(.+?),", pout.strip()) + if not m: + vlog.warn("No CN in the certificate subject (%s)." % cert) return None return m.group(1) From patchwork Wed Oct 30 13:50:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004218 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpS54KSfz1xwF for ; Thu, 31 Oct 2024 00:51:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 098134065F; Wed, 30 Oct 2024 13:50:59 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id SCl1GX5sMa5d; Wed, 30 Oct 2024 13:50:57 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org ED49F404D4 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id ED49F404D4; Wed, 30 Oct 2024 13:50:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D58DFC08BC; Wed, 30 Oct 2024 13:50:55 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 26709C08A6 for ; Wed, 30 Oct 2024 13:50:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id EFEFC408F1 for ; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id U0tjQzbpTFwp for ; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.193; helo=mail-lj1-f193.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 474CE408C7 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 474CE408C7 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by smtp2.osuosl.org (Postfix) with ESMTPS id 474CE408C7 for ; Wed, 30 Oct 2024 13:50:54 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id 38308e7fff4ca-2fb51f39394so66709691fa.2 for ; Wed, 30 Oct 2024 06:50:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296252; x=1730901052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=518M/HQKu788GvjeiQ1ch4kff4ZvhrIlYhefxr5KRlU=; b=iUCpY+uUlAT7rspYi04CcDr2o9jIMVUL/8FWRA5DfKgZsIpXKwPBKtzPfKgqlRjKKK vJ1L4NT4HsZ5ABmf1e9QwSGUsuIMSDFvL3jxBSjOvep6C0eOEcWOkoYiaicOyqivf2dH 67BNP6741GAxNcaI7nWvxwyM/DhNZUR0D/TPAHrJx1O/tZQQDaz/sE0veI0f4IHKqhlM C8uFgrBwKsYhkoIBUu41634HAY2VC+9uftQ5f8iRvIZ1KyLGgJWX7aA+tbtcbt1/AFSH KLVfTtDQvrlwGZ1/KlNsbgWv/0ZYvGqhJuhyR7iHIdw7fj6hG6YrG648lrt2UdsgqsRU vedQ== X-Gm-Message-State: AOJu0YyErtkDyTvasFhiccnf5eHPrJALJz38ruLElmHwB0wIWitgrHMJ u97SAQ2TRuiz6+EdIIEtGiQyX2+gRJKG4LT/5STko0Q//gU5AxIwfe5GzgYg X-Google-Smtp-Source: AGHT+IEBHFP5cebigTWr7Ibo/rsDjrK05Pc2ASM1XeMVuN6bgJO+M/CJZWMnvHRfEn5FQtviJWpnag== X-Received: by 2002:a2e:e02:0:b0:2fa:d25e:39e4 with SMTP id 38308e7fff4ca-2fcbe09a5aamr57091821fa.36.1730296251772; Wed, 30 Oct 2024 06:50:51 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:50:50 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:31 +0100 Message-ID: <20241030135043.3139987-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 2/9] ipsec: libreswan: Fix regexp for connections waiting on child SA. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" These should be considered active, because pluto is waiting for the other side to react. We should not remove them or try to repair. Such connections have an extra text between the SA number and the name of the connection. Ideally, we would like not to parse the output of ipsec status, since it's very error prone, but there is, unfortunately, no other interface. Signed-off-by: Ilya Maximets Acked-by: Roi Dayan Acked-by: Eelco Chaudron --- ipsec/ovs-monitor-ipsec.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 4885e048f..14004f924 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -729,7 +729,7 @@ conn prevent_unencrypted_vxlan return conns for line in pout.splitlines(): - m = re.search(r"#\d+: \"(.*)\".*", line) + m = re.search(r"#\d+: .*\"(.*)\".*", line) if not m: continue From patchwork Wed Oct 30 13:50:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004219 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpS85Xw6z1xwF for ; Thu, 31 Oct 2024 00:51:04 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id F138F80EAD; Wed, 30 Oct 2024 13:51:02 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Jm5HtxbGidLy; Wed, 30 Oct 2024 13:51:01 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 599FF80EB7 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 599FF80EB7; Wed, 30 Oct 2024 13:51:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 47DAFC08A6; Wed, 30 Oct 2024 13:51:01 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 28A45C08A6 for ; Wed, 30 Oct 2024 13:51:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 08CB040BA6 for ; Wed, 30 Oct 2024 13:50:59 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id PKkugsT4W5gw for ; Wed, 30 Oct 2024 13:50:57 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.67; helo=mail-ed1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 4804C40B39 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 4804C40B39 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by smtp2.osuosl.org (Postfix) with ESMTPS id 4804C40B39 for ; Wed, 30 Oct 2024 13:50:57 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id 4fb4d7f45d1cf-5c9693dc739so8945985a12.3 for ; Wed, 30 Oct 2024 06:50:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296255; x=1730901055; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gvyG8o0Yf0XwVIShCcefdcIcRduNfIn/+ZcmJni7WF0=; b=wMRueVUwUoXF9zf4FpARH4HD5k5ksc1ED/nZ3fhBqqFCuah6UP8pz2WDVhVqN66o5t wKvONUCcVDjeLR0ravUtbrCxRIQIlIFW6iWC1A34NCqas8nO0Mha37jyIaejtY6nPghz uanjHZXGkrx+P5SaqinKjZGdClfJRFd/ZQp9I2VK38VPxGvKBRegr7Hzt9VO9oYNXw1Y ncd1Fh1ypLmQoKRqql6/I7gnH5Yssc8EF0w0HKjzxwp3/i/7lDsJ8tdJA0yBmfGj47Mn pk87YliOPkwktiwC31SU8JmBH3+h7zOj4Cat45zPV8JjcxdZuqre2mqaPIK4dyjVqAAh KQKA== X-Gm-Message-State: AOJu0YyNSfUj2XRka2tDAh5+Tx1rHAB528wJjM0Rev1hDsA2b4Jszn5u gGwTiDqcX9MlImOU1OiVvppZVOUW3tUyueVNaQUg4Z+mF1CYZIlxxhUNYmH5 X-Google-Smtp-Source: AGHT+IGE5pst7KIxBdijA/Gw/b5VsipuLZ5yoQ46/OJ638HX/+pUJRmdpWtPDrd/NOIKIlsE2Ui9wA== X-Received: by 2002:a05:6402:354a:b0:5cb:ae1b:4bd1 with SMTP id 4fb4d7f45d1cf-5cbbfa6a9c1mr11965397a12.36.1730296254817; Wed, 30 Oct 2024 06:50:54 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:50:53 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:32 +0100 Message-ID: <20241030135043.3139987-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 3/9] ipsec: libreswan: Reconcile missing connections periodically. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" There are cases where ipsec commands may fail to add new connections or remove the old ones. Unfortunately, this means that those connections may actually never be added or removed, since ovs-monitor-ipsec will not re-visit them, unless something else changes. Wake up the monitor periodically to check if something changed in the system or if some connections still need loading. This addresses two main use cases: 1. Connection failed to start for some reason and was not added to pluto or properly started. The logic will go over all the desired, loaded and active connections and make sure that any undesired connections are removed, non-loaded connections are loaded and non-active connections are brought UP. 2. If pluto re-starts it loads all the connections, but doesn't bring them up, because we're using route (ondemand) activation strategy. This change in this commit will notice all the loaded but not active connections and will bring them up. This helps avoiding packet drops on first packets until the connection activates. Choosing 15 seconds as an interval to wake up to give pluto some breathing room, i.e. a chance to activate the connections properly before we start poking them. And also if pluto is down, 15 second interval will create less spam in the logs. StrongSwan doesn't need such a logic, because it supports a single command 'ipsec update' that re-loads the config as a whole and figures out what configuration changes are needed. But since we're starting all the connections separately with Libreswan, we have to keep track and reconcile manually. Some more details of the logic are in the comments in the code. Signed-off-by: Ilya Maximets Acked-by: Roi Dayan Acked-by: Eelco Chaudron --- ipsec/ovs-monitor-ipsec.in | 185 ++++++++++++++++++++++++------------- 1 file changed, 123 insertions(+), 62 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 14004f924..0ac6297bb 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -20,6 +20,7 @@ import os import re import subprocess import sys +import time from string import Template import ovs.daemon @@ -82,6 +83,7 @@ vlog = ovs.vlog.Vlog("ovs-monitor-ipsec") exiting = False monitor = None xfrm = None +RECONCILIATION_INTERVAL = 15 # seconds def run_command(args, description=None): @@ -295,6 +297,9 @@ conn prevent_unencrypted_vxlan return conns + def need_to_reconcile(self, monitor): + return False + def config_init(self): self.conf_file = open(self.IPSEC_CONF, "w") self.secrets_file = open(self.IPSEC_SECRETS, "w") @@ -511,6 +516,7 @@ conn prevent_unencrypted_vxlan self.IPSEC_D = "sql:" + libreswan_root_prefix + ipsec_d self.IPSEC_CTL = libreswan_root_prefix + ipsec_ctl self.conf_file = None + self.last_refresh = time.time() self.secrets_file = None vlog.dbg("Using: " + self.IPSEC) vlog.dbg("Configuration file: " + self.IPSEC_CONF) @@ -622,51 +628,50 @@ conn prevent_unencrypted_vxlan "--config", self.IPSEC_CONF, "--rereadsecrets"], "re-read secrets") - tunnels = set(monitor.tunnels.keys()) - - # Delete old connections - conns_dict = self.get_active_conns() - for ifname, conns in conns_dict.items(): - tunnel = monitor.tunnels.get(ifname) - - for conn in conns: - # IPsec "connection" names must start with Interface name - if not conn.startswith(ifname): - vlog.err("%s does not start with %s" % (conn, ifname)) - continue - - # version number should be the first integer after - # interface name in IPsec "connection" - try: - ver = int(re.findall(r'\d+', conn[len(ifname):])[0]) - except ValueError: - vlog.err("%s does not contain version number") - continue - except IndexError: - vlog.err("%s does not contain version number") - continue - if not tunnel or tunnel.version != ver: - vlog.info("%s is outdated %u" % (conn, ver)) - run_command(self.IPSEC_AUTO + - ["--ctlsocket", self.IPSEC_CTL, - "--config", self.IPSEC_CONF, - "--delete", conn], "delete %s" % conn) - elif ifname in tunnels: - tunnels.remove(ifname) - - # Activate new connections - for name in tunnels: - ver = monitor.tunnels[name].version - - if monitor.tunnels[name].conf["tunnel_type"] == "gre": - conn = "%s-%s" % (name, ver) - self._start_ipsec_connection(conn) + loaded_conns = self.get_loaded_conns() + active_conns = self.get_active_conns() + + all_names = set(monitor.tunnels.keys()) | \ + set(loaded_conns.keys()) | \ + set(active_conns.keys()) + + for name in all_names: + desired = set(self.get_conn_names(monitor, name)) + loaded = set(loaded_conns.get(name, dict()).keys()) + active = set(active_conns.get(name, dict()).keys()) + + # Remove all the loaded or active but not desired connections. + for conn in loaded | active: + if conn not in desired: + self._delete_ipsec_connection(conn, "is outdated") + loaded.discard(conn) + active.discard(conn) + + # If not all desired are loaded, remove all the loaded and + # active for this tunnel and re-load only the desired ones. + # Need to do that, because connections for the same tunnel + # may share SAs. If one is loaded and the other is not, + # it means the second one failed, so the shared SA may be in + # a broken state. + if desired != loaded: + for conn in loaded | active: + self._delete_ipsec_connection(conn, "is half-loaded") + loaded.discard(conn) + active.discard(conn) + + for conn in desired: + vlog.info("Starting ipsec connection %s" % conn) + self._start_ipsec_connection(conn, "start") else: - conn_in = "%s-in-%s" % (name, ver) - conn_out = "%s-out-%s" % (name, ver) - self._start_ipsec_connection(conn_in) - self._start_ipsec_connection(conn_out) + # Ask pluto to bring UP connections that are loaded, + # but not active for some reason. + # + # desired == loaded and desired >= loaded + active, + # so loaded >= active + for conn in loaded - active: + vlog.info("Bringing up ipsec connection %s" % conn) + self._start_ipsec_connection(conn, "up") # Update shunt policy if changed if monitor.conf_in_use["skb_mark"] != monitor.conf["skb_mark"]: @@ -713,23 +718,27 @@ conn prevent_unencrypted_vxlan "--delete", "--asynchronous", "prevent_unencrypted_vxlan"]) monitor.conf_in_use["skb_mark"] = monitor.conf["skb_mark"] + self.last_refresh = time.time() + vlog.info("Refreshing is done.") - def get_active_conns(self): + def get_conns_from_status(self, pattern): """This function parses output from 'ipsec status' command. It returns dictionary where is interface name (as in OVSDB) and is another dictionary. This another dictionary uses LibreSwan connection name as and more detailed - sample line from the parsed outpus as . """ + sample line from the parsed outpus as . 'pattern' should + be a regular expression that parses out the connection name. + Only the lines that match the pattern will be parsed. """ conns = {} ret, pout, perr = run_command([self.IPSEC, 'status', '--ctlsocket', self.IPSEC_CTL], - "get active connections") + "get ipsec status") if ret: return conns for line in pout.splitlines(): - m = re.search(r"#\d+: .*\"(.*)\".*", line) + m = re.search(pattern, line) if not m: continue @@ -748,25 +757,76 @@ conn prevent_unencrypted_vxlan return conns - def _start_ipsec_connection(self, conn): - # In a corner case, LibreSwan daemon restarts for some reason and - # the "ipsec auto --start" command is lost. Just retry to make sure - # the command is received by LibreSwan. - while True: - ret, pout, perr = run_command(self.IPSEC_AUTO + - ["--config", self.IPSEC_CONF, - "--ctlsocket", self.IPSEC_CTL, - "--start", - "--asynchronous", conn], - "start %s" % conn) - if not re.match(r".*Connection refused.*", perr) and \ - not re.match(r".*need --listen.*", pout): - break + def get_active_conns(self): + return self.get_conns_from_status(r"#\d+: .*\"(.*)\".*") + + def get_loaded_conns(self): + return self.get_conns_from_status(r"\"(.*)\": \d+.*(===|\.\.\.).*") + + def get_conn_names(self, monitor, ifname): + conns = [] + if ifname not in monitor.tunnels: + return conns + + tunnel = monitor.tunnels.get(ifname) + ver = tunnel.version + + if tunnel.conf["tunnel_type"] == "gre": + conns.append("%s-%s" % (ifname, ver)) + else: + conns.append("%s-in-%s" % (ifname, ver)) + conns.append("%s-out-%s" % (ifname, ver)) + + return conns + + def need_to_reconcile(self, monitor): + if time.time() - self.last_refresh < RECONCILIATION_INTERVAL: + return False + + conns_dict = self.get_active_conns() + for ifname, tunnel in monitor.tunnels.items(): + if ifname not in conns_dict: + vlog.info("Connection for port %s is not active, " + "need to reconcile" % ifname) + return True + + existing_conns = conns_dict.get(ifname) + desired_conns = self.get_conn_names(monitor, ifname) + + if set(existing_conns.keys()) != set(desired_conns): + vlog.info("Active connections for port %s %s do not match " + "desired %s, need to reconcile" + % (ifname, list(existing_conns.keys()), + desired_conns)) + return True + + return False + + def _delete_ipsec_connection(self, conn, reason): + vlog.info("%s %s, removing" % (conn, reason)) + run_command(self.IPSEC_AUTO + + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--delete", conn], "delete %s" % conn) + + def _start_ipsec_connection(self, conn, action): + ret, pout, perr = run_command(self.IPSEC_AUTO + + ["--config", self.IPSEC_CONF, + "--ctlsocket", self.IPSEC_CTL, + "--" + action, + "--asynchronous", conn], + "%s %s" % (action, conn)) if re.match(r".*[F|f]ailed to initiate connection.*", pout): vlog.err('Failed to initiate connection through' ' Interface %s.\n' % (conn.split('-')[0])) vlog.err("stdout: %s" % pout) + ret = 1 + + if ret: + # We don't know in which state the connection was left on + # failure. Try to clean it up. + self._delete_ipsec_connection(conn, "--%s failed" % action) def _nss_clear_database(self): """Remove all OVS IPsec related state from the NSS database""" @@ -1192,7 +1252,7 @@ class IPsecMonitor(object): self.ike_helper.clear_tunnel_state(self.tunnels[name]) del self.tunnels[name] - if needs_refresh: + if needs_refresh or self.ike_helper.need_to_reconcile(self): self.ike_helper.refresh(self) def _get_cn_from_cert(self, cert): @@ -1365,6 +1425,7 @@ def main(): poller = ovs.poller.Poller() unixctl_server.wait(poller) idl.wait(poller) + poller.timer_wait(RECONCILIATION_INTERVAL * 1000) poller.block() unixctl_server.close() From patchwork Wed Oct 30 13:50:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004220 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpSF3MJxz1xxd for ; Thu, 31 Oct 2024 00:51:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D61BD40BC7; Wed, 30 Oct 2024 13:51:06 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id i4lEffhK8KSc; Wed, 30 Oct 2024 13:51:04 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 1201540B82 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1201540B82; Wed, 30 Oct 2024 13:51:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C90D0C08A8; Wed, 30 Oct 2024 13:51:03 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 127D4C08A9 for ; Wed, 30 Oct 2024 13:51:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 9ADC6608C7 for ; Wed, 30 Oct 2024 13:51:00 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id JGx_F2j0GwKt for ; Wed, 30 Oct 2024 13:51:00 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.67; helo=mail-ed1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 9A5EE608C3 Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 9A5EE608C3 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9A5EE608C3 for ; Wed, 30 Oct 2024 13:50:59 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id 4fb4d7f45d1cf-5cbb719839eso7213036a12.2 for ; Wed, 30 Oct 2024 06:50:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296257; x=1730901057; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7PoLJfVY4XRs2iOz6Z64+VoUHBEALjgpZ0v3Dy/7vFE=; b=NET/khT52rFbJXKyXu9zK1ljIGkOexK87+f2c2ILE6L9PzUFWODfcMC4M8WlUHdQSb iTJeoFbxU9g4xdPWyz/P0wO+iAJeBYBoeUyn8IFYVJt7w9rPhGAs2kVaSHxgBAR1D6CA xUCVOpVPFy+dpGlC7YaeF3xUFd9HjsCsR7cfbq8brgpubFkBiH79h3evkF5uxmgAecZE +k11JWEpu7vLPAcedmBtcnNs4hZJB2WKreZo3Do50tkvggVv7oWroFzVTIYOY/poH52z JqRvMPfuI/PzcdW5wmVOo5Nmc1UvI2fZ407mCYSQ9lRMqls//iCuq17oAPRVA7JQfISX uKTQ== X-Gm-Message-State: AOJu0YyOScoFoeNtwoTZMJiVcXeTicgxCjPv3XROa9a9rl5CjOKeoJzY G2QKUgOrVtFje8pEEm6y8ogyg4Rn9Ci3/46QtyRdMubp1ymwEHMWc4hXke+C X-Google-Smtp-Source: AGHT+IE7TD1c3kYNfIKSUZ1u/EReDJUMyKXLC1wF8smJkx07rnWG0MulMvrX1OHqejmTccXN/pmMlQ== X-Received: by 2002:a05:6402:2343:b0:5ca:152c:5b5b with SMTP id 4fb4d7f45d1cf-5cbbf8e8291mr11909498a12.21.1730296256904; Wed, 30 Oct 2024 06:50:56 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:50:56 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:33 +0100 Message-ID: <20241030135043.3139987-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 4/9] ipsec: libreswan: Try to bring non-active connections up. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Sometimes connections are getting loaded, but do not become active for some reason on a first try. We can try and bring them up manually. However, if they are still not active after that, it's better to just remove the connection and try to add them from scratch, as there must be some internal issue in libreswan that doesn't allow these connections to actually become active. Note: Once the "defunct" connection is removed, the second connection for the same tunnel will also be removed as "half-loaded". This ensures that all the shared SAs will also be cleaned up, so we can truly start from scratch. Signed-off-by: Ilya Maximets Acked-by: Eelco Chaudron Acked-by: Roi Dayan --- ipsec/ovs-monitor-ipsec.in | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 0ac6297bb..5d4b77bd2 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -516,6 +516,7 @@ conn prevent_unencrypted_vxlan self.IPSEC_D = "sql:" + libreswan_root_prefix + ipsec_d self.IPSEC_CTL = libreswan_root_prefix + ipsec_ctl self.conf_file = None + self.conns_not_active = set() self.last_refresh = time.time() self.secrets_file = None vlog.dbg("Using: " + self.IPSEC) @@ -641,6 +642,14 @@ conn prevent_unencrypted_vxlan loaded = set(loaded_conns.get(name, dict()).keys()) active = set(active_conns.get(name, dict()).keys()) + # Untrack connections that became active. + self.conns_not_active.difference_update(active) + # Remove connections that didn't become active after --start + # and another explicit --up. + for conn in self.conns_not_active & loaded: + self._delete_ipsec_connection(conn, "is defunct") + loaded.remove(conn) + # Remove all the loaded or active but not desired connections. for conn in loaded | active: if conn not in desired: @@ -671,6 +680,8 @@ conn prevent_unencrypted_vxlan # so loaded >= active for conn in loaded - active: vlog.info("Bringing up ipsec connection %s" % conn) + # On failure to --up it will be removed from the set. + self.conns_not_active.add(conn) self._start_ipsec_connection(conn, "up") # Update shunt policy if changed @@ -804,6 +815,7 @@ conn prevent_unencrypted_vxlan def _delete_ipsec_connection(self, conn, reason): vlog.info("%s %s, removing" % (conn, reason)) + self.conns_not_active.discard(conn) run_command(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, "--config", self.IPSEC_CONF, From patchwork Wed Oct 30 13:50:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004221 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpST5Jh4z1xwF for ; Thu, 31 Oct 2024 00:51:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1491760A4C; Wed, 30 Oct 2024 13:51:20 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id NJDlCq9yyYyd; Wed, 30 Oct 2024 13:51:08 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org E75E2608C4 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id E75E2608C4; Wed, 30 Oct 2024 13:51:07 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9CF23C08A6; Wed, 30 Oct 2024 13:51:07 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2F465C08A8 for ; Wed, 30 Oct 2024 13:51:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7FD24608A6 for ; Wed, 30 Oct 2024 13:51:02 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id L-T2UydnboW6 for ; Wed, 30 Oct 2024 13:51:01 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.66; helo=mail-ed1-f66.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 6A0AE608C4 Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 6A0AE608C4 Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6A0AE608C4 for ; Wed, 30 Oct 2024 13:51:01 +0000 (UTC) Received: by mail-ed1-f66.google.com with SMTP id 4fb4d7f45d1cf-5c95a962c2bso8113841a12.2 for ; Wed, 30 Oct 2024 06:51:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296259; x=1730901059; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GjO11DyERGeWjshV4aNCl16qc9GSEAgEgats8DE1iAg=; b=J6MsVRHgiw31BmagQ6fWI0NR7XK9l9n8rutFtk/SEwUconAn1DzqrAaqx+7Gtk3n/e 4refBUlhpG0j8UXPK4TU+pZ0DWX0nqWNHkcHEO+7qtKAHhUrkUNae/8dL3UAA7fLcV+5 GwO9c7R9uzc+pK6StZFNYIqzkvvQUAxEjv+2g7r06FogpiBDJQVE/CDF+R549tfI02DC Hrj0ttdu55HCl+ADMP6p2kbUQat6m1M3QKklaWYmA1nnYxQ9LHK0b4d/1bbHa535oaOD 7ijuyiodjq59grizE1Z45/lwnFBr1tfTioybP4uyCSFa0yqtB0GoBEynqItVvQk7g/KK 4pQQ== X-Gm-Message-State: AOJu0YxkXi+Xir+ibun2kzBz4N853+PSo3rZGzU+3LQDz7pr+I12X1XK jqTXLE5C6WuIM3dSQMWVYntVQjE7kMwlhPaoYTuSbky4ewrC89WBwobd3I9N X-Google-Smtp-Source: AGHT+IF1G5HgvYYkKP0HjKBCzj413+p8XOhPZixl+Mm6sLr/WukkJx+pbyPzcu9Ft6oMMMZkcGpcEA== X-Received: by 2002:a05:6402:d09:b0:5c9:7cd8:7aef with SMTP id 4fb4d7f45d1cf-5cbbf8b171dmr10650214a12.9.1730296259036; Wed, 30 Oct 2024 06:50:59 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:50:58 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:34 +0100 Message-ID: <20241030135043.3139987-6-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 5/9] ipsec: libreswan: Avoid monitor hanging on stuck ipsec commands. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Multiple versions of Libreswan have an issue where ipsec --start command may get stuck forever. This issue affects many popular versions of Libreswan from 4.5 to 4.15, which are shipped in most modern distributions. When ipsec --start gets stuck, ovs-monitor-ipsec hangs and can't do anything else, so not olny this one but all other tunnels are also not being started. Add a timeout to the subprocess call, so we do not wait forever. Just introduced reconciliation process will clean things up and will try to re-add this connection later. Pluto may take a lot of time to process the --start request. Notably, the time depends on the retransmission timeout, which is 60 seconds by default. However, even at high scale, it doesn't take much more than that in tests. So, 120 second timeout should be a reasonable default value. Note: it is observed in practice that the process doesn't actually terminate for a long time, so we can't afford waiting for it. That's the main reason why we're not using the subprocess.run() with a timeout option here (it would wait). But also, because we'd had to catch the exception anyway. Reported-at: https://issues.redhat.com/browse/FDP-846 Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 5d4b77bd2..dba855af5 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -84,6 +84,7 @@ exiting = False monitor = None xfrm = None RECONCILIATION_INTERVAL = 15 # seconds +TIEMOUT_EXPIRED = 37 def run_command(args, description=None): @@ -96,7 +97,16 @@ def run_command(args, description=None): vlog.dbg("Running %s" % args) proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - pout, perr = proc.communicate() + try: + pout, perr = proc.communicate(timeout=120) + ret = proc.returncode + except subprocess.TimeoutExpired: + vlog.warn("Command timed out trying to %s." % description) + pout, perr = b'', b'' + # Just kill the process here. We can't afford waiting for it, + # as it may be stuck and may not actually be terminated. + proc.kill() + ret = TIEMOUT_EXPIRED if proc.returncode or perr: vlog.warn("Failed to %s; exit code: %d" @@ -105,7 +115,7 @@ def run_command(args, description=None): vlog.warn("stderr: %s" % perr) vlog.warn("stdout: %s" % pout) - return proc.returncode, pout.decode(), perr.decode() + return ret, pout.decode(), perr.decode() class XFRM(object): From patchwork Wed Oct 30 13:50:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004222 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpST6ywHz1xxd for ; Thu, 31 Oct 2024 00:51:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 43A114055D; Wed, 30 Oct 2024 13:51:20 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id i57QSA9k8Z-e; Wed, 30 Oct 2024 13:51:19 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 1673B4002A Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 1673B4002A; Wed, 30 Oct 2024 13:51:19 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C5F71C08A6; Wed, 30 Oct 2024 13:51:18 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 195E2C08A6 for ; Wed, 30 Oct 2024 13:51:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E030340BB8 for ; Wed, 30 Oct 2024 13:51:04 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id YZ5Ty2eH7xAh for ; Wed, 30 Oct 2024 13:51:04 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.65; helo=mail-ed1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 7C71940BB2 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 7C71940BB2 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7C71940BB2 for ; Wed, 30 Oct 2024 13:51:03 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id 4fb4d7f45d1cf-5c97c7852e8so9347971a12.1 for ; Wed, 30 Oct 2024 06:51:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296261; x=1730901061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A7eKS93wpE4btmPbivOpuTVJdPbLil/DPvUJ5TKvuPY=; b=hJ8e7C6O6CaaiYDlT3b0IK09aFD6NXw/biK6oJ490FrqmSeOxbqvX9s3Um9j5IH7ur rGiZFRIFdNs7Kjj9j+9DJNHun2aRJ4ksqdKvhPG3Uda2JdA36CLK2kFnh1rR+BucCjUJ KJnNVG6uETaxf/gFaO/j3ROfQmWlzbJgbiUspaBWwk55NVdL/XXjkGwbyHsQPhSblwFB JxpQjBrV+oM10ntcIJIN3A1YGm77lpxDw09L5mJvY3Vv2JUgJvV/11JdSTT8sp8zoJAN 3IGhqZ8SxhdQam04z8T3AYCgU3wvUq/LcH7kfX5KYT2vqLCgSsuCT5OuTbevKnzBJIYa +bTw== X-Gm-Message-State: AOJu0Yyw0EOHJJm0UPzvg1i4RRR38X6q38Za/xkhXZGvvQIQkjO1mbDK JcRcxftq5VUBLnTNxkLAj/LGN808G2ifAt4VaCxhFJ6MQ92mq1Dw+tRjt7kc X-Google-Smtp-Source: AGHT+IFGLLN6Umjjwoi5jKBYVA/fRMoRpuPrIRqcpq/+pKCbD9hR9RmIsNJ5tr6VyC7J1xV7TEEoYA== X-Received: by 2002:a05:6402:3581:b0:5c9:45f2:b26a with SMTP id 4fb4d7f45d1cf-5cbbf73bbfdmr10921149a12.0.1730296261174; Wed, 30 Oct 2024 06:51:01 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:51:00 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:35 +0100 Message-ID: <20241030135043.3139987-7-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 6/9] ipsec: Make command timeout configurable. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new command line option --command-timeout that controls the command timeout. It is important to have this configurable, because the retransmit-timeout is configurable in Libreswan. Also, users may prefer the monitor to be more responsive. ovs-monitor-ipsec options are not documented anywhere, so not trying to address that here. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index dba855af5..bf7b9c6ad 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -83,6 +83,7 @@ vlog = ovs.vlog.Vlog("ovs-monitor-ipsec") exiting = False monitor = None xfrm = None +command_timeout = None RECONCILIATION_INTERVAL = 15 # seconds TIEMOUT_EXPIRED = 37 @@ -98,7 +99,7 @@ def run_command(args, description=None): proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: - pout, perr = proc.communicate(timeout=120) + pout, perr = proc.communicate(timeout=command_timeout) ret = proc.returncode except subprocess.TimeoutExpired: vlog.warn("Command timed out trying to %s." % description) @@ -1387,6 +1388,10 @@ def main(): parser.add_argument("--ipsec-ctl", metavar="IPSEC-CTL", help="Use DIR/IPSEC-CTL as location for " " pluto ctl socket (libreswan only).") + parser.add_argument("--command-timeout", metavar="TIMEOUT", + type=int, default=120, + help="Timeout for external commands called by the " + "ovs-monitor-ipsec daemon, e.g. ipsec --start.") ovs.vlog.add_args(parser) ovs.daemon.add_args(parser) @@ -1396,11 +1401,13 @@ def main(): global monitor global xfrm + global command_timeout root_prefix = args.root_prefix if args.root_prefix else "" xfrm = XFRM(root_prefix) monitor = IPsecMonitor(root_prefix, args.ike_daemon, not args.no_restart_ike_daemon, args) + command_timeout = args.command_timeout remote = args.database schema_helper = ovs.db.idl.SchemaHelper() From patchwork Wed Oct 30 13:50:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004223 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpSX4nxRz1xwF for ; Thu, 31 Oct 2024 00:51:24 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4E45340574; Wed, 30 Oct 2024 13:51:22 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 3tRb0yfc3ngL; Wed, 30 Oct 2024 13:51:21 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 19E0B40564 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 19E0B40564; Wed, 30 Oct 2024 13:51:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CBE45C08AA; Wed, 30 Oct 2024 13:51:20 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3EE19C08A9 for ; Wed, 30 Oct 2024 13:51:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A64C481022 for ; Wed, 30 Oct 2024 13:51:07 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id INXfaZu0k2qI for ; Wed, 30 Oct 2024 13:51:06 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.65; helo=mail-ed1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org BA0DD80FAF Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org BA0DD80FAF Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by smtp1.osuosl.org (Postfix) with ESMTPS id BA0DD80FAF for ; Wed, 30 Oct 2024 13:51:05 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id 4fb4d7f45d1cf-5cb72918bddso8750819a12.3 for ; Wed, 30 Oct 2024 06:51:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296264; x=1730901064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OZebmDeZp5nc1MQzZgRIPjTSpmq5vUKcToSoeiDMkOY=; b=W12OqTLN4vtnnZNvK5iRmLVFDlPi2ICJ2R0GBZUI+mlBAFsWCsfHofFXLqjr6fyT4R 3cYonRGcPzrIDH8elIjhyxuBW0UgS1o8AUsHc31FVyG6V76nV8q88AbkKXEruKvipggC MevNKnQbjLhuyBydWAqj9zUXjb4gcLntOlrWwbU1m7DvR9mpDWCzpYoxhFyW7lpUfJBo Prxe0A22BD3STdh14R2PsThBT/cglxxgww+vcnD9THAJjDpNNVvy4aTaNC/DavxfIBGu xokYxlXTd/89j+KOglh4MWf3B94mHQdh9uqSV5Dkk5trWBL4NYCf3SXPrgwk+/eDHwl5 NsqA== X-Gm-Message-State: AOJu0YzY5Ei0XqlTUm04XPJhLq+OGcjfJwYiXRDT+oscy/PA3GHNfoum EqkZiujL5uygVJnGY3kWPRb9YwGkb/vd33Q9v0o2Q3hsrF8PO/qUpY4RXNYV X-Google-Smtp-Source: AGHT+IE1vSavYZQb5ic1fAMGrRcSI3iWg1KLt2Z5zQDXBw24Mhh4JhOwmFIVGODlQKFonqoPeVGaKg== X-Received: by 2002:a05:6402:5241:b0:5ca:971:badb with SMTP id 4fb4d7f45d1cf-5cbbf8a14d0mr10872816a12.9.1730296263623; Wed, 30 Oct 2024 06:51:03 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.51.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:51:03 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:36 +0100 Message-ID: <20241030135043.3139987-8-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 7/9] system-tests: Verbose cleanup of ports and namespaces. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Removal of ports and network namespaces can take a significant amount of time, and it is not clear if the test is stuck or actually doing something during that time. Add some logging to cleanup commands to see what is going on. Signed-off-by: Ilya Maximets Acked-by: Eelco Chaudron --- tests/system-common-macros.at | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/tests/system-common-macros.at b/tests/system-common-macros.at index e9be021f3..ff86d15cd 100644 --- a/tests/system-common-macros.at +++ b/tests/system-common-macros.at @@ -2,10 +2,7 @@ # # Delete namespaces from the running OS m4_define([DEL_NAMESPACES], - [m4_foreach([ns], [$@], - [ip netns del ns -]) - ] + [m4_foreach([ns], [$@], [echo removing namespace ns; ip netns del ns])] ) # ADD_NAMESPACES(ns [, ns ... ]) @@ -72,7 +69,7 @@ m4_define([ADD_INT], # m4_define([ADD_VETH], [ AT_CHECK([ip link add $1 type veth peer name ovs-$1 || return 77]) - on_exit 'ip link del ovs-$1' + on_exit 'echo removing interface ovs-$1; ip link del ovs-$1' CONFIGURE_VETH_OFFLOADS([$1]) AT_CHECK([ip link set $1 netns $2]) AT_CHECK([ip link set dev ovs-$1 up]) From patchwork Wed Oct 30 13:50:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004224 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpSb67hKz1xwF for ; Thu, 31 Oct 2024 00:51:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2E02060A39; Wed, 30 Oct 2024 13:51:26 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Y7ic0kEgaphZ; Wed, 30 Oct 2024 13:51:24 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org AF9BC60A54 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id AF9BC60A54; Wed, 30 Oct 2024 13:51:24 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8D6E5C08A6; Wed, 30 Oct 2024 13:51:24 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 02974C08A3 for ; Wed, 30 Oct 2024 13:51:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 159C060908 for ; Wed, 30 Oct 2024 13:51:10 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id xVBITwo7Sw8L for ; Wed, 30 Oct 2024 13:51:08 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.65; helo=mail-lf1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org C1E1A608D7 Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org C1E1A608D7 Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id C1E1A608D7 for ; Wed, 30 Oct 2024 13:51:07 +0000 (UTC) Received: by mail-lf1-f65.google.com with SMTP id 2adb3069b0e04-53c779ef19cso446585e87.3 for ; Wed, 30 Oct 2024 06:51:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296265; x=1730901065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aMv6EX/13NHj3ghKGl1c6pEPIgB4kshkcQ2LLD18xPo=; b=WkxZXJFGOvLhkt0/EDvek6BnUN+ErOyHgpe4P8Wmc71pyINzm6vmON8ajxcMvhctP2 gqTlQj8iq/DE2BfWyEK+fH+TzWCw2XDvbaBdWHNGg+TTis9IHOec4mf6GtovB7apDdjw DDbR1felVWIQTxakwF+B8pPi4eokKmTdEoPywXMjP9H6m/7zhvbx7mhFOXjD0LmLJo0F KydFIvEWhfZQisJ0v880GTy2T0uTwEdXrzz5h+tojPiRFPJ1Jnz5MlQx4fPlOYXjWGFg B/Dj/z/6zQ2T9+zp2dVRPvm6vol8e5a05++wCHWv/bRMo3UT5Ti1C8rz5a+B2iDVkPLS Lt6Q== X-Gm-Message-State: AOJu0YxyF6BLwa8YSX3fph4sWEIRRX2qig04nTWVtjK/67TR1/YTjo0R Ulq6V+Cl5cPCsOQxd5IngTs8z/Q/180hRfRmOM7UgXX0QUAhTC9QsPvalLcG X-Google-Smtp-Source: AGHT+IHUqQQbS5yku5VWLHAGaQlr6ZpJ2AhR++y3XHaASgZfJQQxEST62+WxzdG2R6n7RF+hFVNtIg== X-Received: by 2002:a05:6512:b84:b0:539:e0e6:cf44 with SMTP id 2adb3069b0e04-53b347c0d70mr8796311e87.4.1730296264979; Wed, 30 Oct 2024 06:51:04 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.51.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:51:04 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:37 +0100 Message-ID: <20241030135043.3139987-9-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 8/9] tests: ipsec: Add NxN + reconciliation test. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a test to check establishment of IPsec connections among multiple nodes and check the reconciliation logic along the way. The test: - Creates 20 network namespaces. - Starts Libreswan, OVS and ovs-monitor-ipsec in each of them. - Adds a geneve tunnel from each namespace to every other namespace. - Checks that each namespace has all the IPsec connections loaded. - Removes a few connections manually. - Checks that these connections are added back. Unfortunately, many widely used versions of Libreswan have issues of pluto crashing frequently. For that reason the test is trying to bring pluto back online once it finds a dead one. Also, since retransmit-timeout is 60 seconds and our command timeout is 120, we can't actually use the OVS_WAIT_UNTIL macro most of the time, so the checks are done in the custom loop that waits up to 300 seconds. Signed-off-by: Ilya Maximets Acked-by: Eelco Chaudron --- tests/system-ipsec.at | 138 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 125 insertions(+), 13 deletions(-) diff --git a/tests/system-ipsec.at b/tests/system-ipsec.at index 1e155fece..5aa67bf1d 100644 --- a/tests/system-ipsec.at +++ b/tests/system-ipsec.at @@ -8,6 +8,18 @@ m4_define([IPSEC_SETUP_UNDERLAY], dnl Set up the underlay switch AT_CHECK([ovs-ofctl add-flow br0 "actions=normal"])]) +m4_define([START_PLUTO], [ + rm -f $ovs_base/$1/pluto.pid + mkdir -p $ovs_base/$1/ipsec.d + touch $ovs_base/$1/ipsec.conf + touch $ovs_base/$1/secrets + ipsec initnss --nssdir $ovs_base/$1/ipsec.d + NS_CHECK_EXEC([$1], [ipsec pluto --config $ovs_base/$1/ipsec.conf \ + --ipsecdir $ovs_base/$1 --nssdir $ovs_base/$1/ipsec.d \ + --logfile $ovs_base/$1/pluto.log --secretsfile $ovs_base/$1/secrets \ + --rundir $ovs_base/$1], [0], [], [stderr]) +]) + dnl IPSEC_ADD_NODE([namespace], [device], [address], [peer address])) dnl dnl Creates a dummy host that acts as an IPsec endpoint. Creates host in @@ -45,15 +57,8 @@ m4_define([IPSEC_ADD_NODE], on_exit "kill_ovs_vswitchd `cat $ovs_base/$1/vswitchd.pid`" dnl Start pluto - mkdir -p $ovs_base/$1/ipsec.d - touch $ovs_base/$1/ipsec.conf - touch $ovs_base/$1/secrets - ipsec initnss --nssdir $ovs_base/$1/ipsec.d - NS_CHECK_EXEC([$1], [ipsec pluto --config $ovs_base/$1/ipsec.conf \ - --ipsecdir $ovs_base/$1 --nssdir $ovs_base/$1/ipsec.d \ - --logfile $ovs_base/$1/pluto.log --secretsfile $ovs_base/$1/secrets \ - --rundir $ovs_base/$1], [0], [], [stderr]) - on_exit "kill `cat $ovs_base/$1/pluto.pid`" + START_PLUTO([$1]) + on_exit 'kill $(cat $ovs_base/$1/pluto.pid)' dnl Start ovs-monitor-ipsec NS_CHECK_EXEC([$1], [ovs-monitor-ipsec unix:${OVS_RUNDIR}/$1/db.sock\ @@ -110,16 +115,18 @@ m4_define([CHECK_LIBRESWAN], dnl IPSEC_STATUS_LOADED([]) dnl dnl Get number of loaded connections from ipsec status -m4_define([IPSEC_STATUS_LOADED], [ipsec --rundir $ovs_base/$1 status | \ +m4_define([IPSEC_STATUS_LOADED], [ + ipsec --rundir $ovs_base/$1 status | \ grep "Total IPsec connections" | \ - sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-2]]\), active \([[0-2]]\).*/\1/m']) + sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-9]]*\), active \([[0-9]]*\).*/\1/m']) dnl IPSEC_STATUS_ACTIVE([]) dnl dnl Get number of active connections from ipsec status -m4_define([IPSEC_STATUS_ACTIVE], [ipsec --rundir $ovs_base/$1 status | \ +m4_define([IPSEC_STATUS_ACTIVE], [ + ipsec --rundir $ovs_base/$1 status | \ grep "Total IPsec connections" | \ - sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-2]]\), active \([[0-2]]\).*/\2/m']) + sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-9]]*\), active \([[0-9]]*\).*/\2/m']) dnl CHECK_ESP_TRAFFIC() dnl @@ -401,3 +408,108 @@ CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP + +AT_SETUP([IPsec -- Libreswan NxN geneve tunnels + reconciliation]) +AT_KEYWORDS([ipsec libreswan scale reconciliation]) +dnl Note: Geneve test may not work on older kernels due to CVE-2020-25645 +dnl https://bugzilla.redhat.com/show_bug.cgi?id=1883988 + +CHECK_LIBRESWAN() +OVS_TRAFFIC_VSWITCHD_START() +IPSEC_SETUP_UNDERLAY() + +m4_define([NODES], [20]) + +dnl Set up fake hosts. +m4_for([id], [1], NODES, [1], [ + IPSEC_ADD_NODE([node-id], [p-id], 10.1.1.id, 10.1.1.254) + AT_CHECK([ovs-pki -b -d ${ovs_base} -l ${ovs_base}/ovs-pki.log \ + req -u node-id], [0], [stdout]) + AT_CHECK([ovs-pki -b -d ${ovs_base} -l ${ovs_base}/ovs-pki.log \ + self-sign node-id], [0], [stdout]) + AT_CHECK(OVS_VSCTL([node-id], set Open_vSwitch . \ + other_config:certificate=${ovs_base}/node-id-cert.pem \ + other_config:private_key=${ovs_base}/node-id-privkey.pem), + [0], [ignore], [ignore]) + on_exit "ipsec --rundir $ovs_base/node-id status > $ovs_base/node-id/status" +]) + +dnl Create a full mesh of tunnels. +m4_for([LEFT], [1], NODES, [1], [ + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + AT_CHECK(OVS_VSCTL(node-LEFT, add-port br-ipsec tun-RIGHT \ + -- set Interface tun-RIGHT type=geneve options:remote_ip=10.1.1.RIGHT \ + options:remote_cert=${ovs_base}/node-RIGHT-cert.pem), + [0], [ignore], [ignore]) + fi +])]) + +m4_define([WAIT_FOR_LOADED_CONNS], [ + m4_for([id], [1], NODES, [1], [ + echo "================== node-id =========================" + iterations=0 + loaded=0 + dnl Using a custom loop instead of OVS_WAIT_UNTIL, because it may take + dnl much longer than a default timeout. The default retransmit timeout + dnl for pluto is 60 seconds. Also, we need to make sure pluto didn't + dnl crash in the process and revive it if it did, unfortunately. + while true; do + date + AT_CHECK([ipsec --rundir $ovs_base/node-id status 2>&1 \ + | grep -E "whack|Total"], [ignore], [stdout]) + if grep -E 'is Pluto running?|refused' stdout; then + echo "node-id: Pluto died, restarting..." + START_PLUTO([node-id]) + else + loaded=$(IPSEC_STATUS_LOADED(node-id)) + fi + if test "$loaded" -ne $(( (NODES - 1) * 2 )); then + sleep 3 + else + break + fi + let iterations=$iterations+1 + AT_CHECK([test $iterations -lt 100]) + done + ]) +]) + +dnl Wait for all the connections to be loaded to pluto. Not waiting for +dnl them to become active, because if pluto is down on one of the nodes, +dnl some connections may not become active until we revive it. Some +dnl connections may also never become active due to bugs in libreswan 4.x. +WAIT_FOR_LOADED_CONNS() + +AT_CHECK([ipsec auto --help], [ignore], [ignore], [stderr]) +auto=auto +if test -s stderr; then + auto= +fi + +dnl Remove connections for two tunnels. One fully and one partially. +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-5-out-1], [0], [stdout]) +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-2-in-1], [0], [stdout]) +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-2-out-1], [0], [stdout]) + +dnl Wait for the monitor to notice the missing connections. +OVS_WAIT_UNTIL([grep -q 'tun-2.*need to reconcile' \ + $ovs_base/node-1/ovs-monitor-ipsec.log]) + +dnl Wait for all the connections to be loaded back. +WAIT_FOR_LOADED_CONNS() + +dnl These are not necessary, but nice to have in the test log in +dnl order to spot pluto failures during the test. +grep -E 'timed out|outdated|half-loaded|defunct' \ + $ovs_base/node-*/ovs-monitor-ipsec.log +grep -E 'ABORT|ERROR' $ovs_base/node-*/pluto.log + +OVS_TRAFFIC_VSWITCHD_STOP() +AT_CLEANUP From patchwork Wed Oct 30 13:50:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2004225 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XdpTN2x5Bz1xwF for ; Thu, 31 Oct 2024 00:52:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C9DE580FA3; Wed, 30 Oct 2024 13:52:06 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id D9wpZFT5ekIa; Wed, 30 Oct 2024 13:52:05 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A69C88101F Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id A69C88101F; Wed, 30 Oct 2024 13:52:05 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7AC63C08A8; Wed, 30 Oct 2024 13:52:05 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 32F42C08A6 for ; Wed, 30 Oct 2024 13:52:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 397C180E86 for ; Wed, 30 Oct 2024 13:51:19 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ey5tmTmdCsA6 for ; Wed, 30 Oct 2024 13:51:12 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.67; helo=mail-ed1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 015F78107F Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 015F78107F Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by smtp1.osuosl.org (Postfix) with ESMTPS id 015F78107F for ; Wed, 30 Oct 2024 13:51:09 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id 4fb4d7f45d1cf-5c9850ae22eso8554042a12.3 for ; Wed, 30 Oct 2024 06:51:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730296267; x=1730901067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zWHUm9ysTzTJhgBwXFhagDwhdJjg6m2+sLgEtwgWy1I=; b=azdlvl8Gi9wtcPeHj+/j/MJgnM9MJvoa60hguAhsH0PRGhIwEjAnBA1+/VI9wVJomq IKJzO440HMymqaIOA90tCeK5iygpOJR76X7SMcqqXS8MPV4o5oodztRtsSMWtEXnu2dV N684Bbp/zJoXrEhXUC+fBcSjqxTTbqkSpc1uyDc4E1yH/RojeG7XJdgxFztLeiEpXNEL IL4sVZb7TzXshhtnacSGWBYp5Fqcu+SWFdIX1kbgmIsUCxF24I/S0nr0bBwao894a1zb AbeemicbFiFq7AnH+nzAz5fdke/ZpATj8XAaphgpcH9mieD+ZDdnZzbtgJ17Buaq+ZPt xaMA== X-Gm-Message-State: AOJu0YySVI87UsNyhZo33unZ9+2Qm+W6jmgas2QCNLm8T6e+bhbUWXbF etTCbscHXPeEw0kHHOLDczoaYULxDwpdkLnhladNbr/h5M7TROKCItuOe9fx X-Google-Smtp-Source: AGHT+IGjKUSiU853UlEnv51TjZtz65aTf13uwYPogbbVipZ9wAQrCPIJNpzpPKGkXKyMY/xGVxL+LQ== X-Received: by 2002:a05:6402:3595:b0:5cb:def2:be0a with SMTP id 4fb4d7f45d1cf-5cbdef2bf78mr7968684a12.21.1730296266918; Wed, 30 Oct 2024 06:51:06 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb6255be3sm4763479a12.3.2024.10.30.06.51.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 06:51:06 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 30 Oct 2024 14:50:38 +0100 Message-ID: <20241030135043.3139987-10-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241030135043.3139987-1-i.maximets@ovn.org> References: <20241030135043.3139987-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2 9/9] tests: ipsec: Check that nodes can ping each other in the NxN test. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expand the NxN test with the network connectivity check between all the nodes. Unfortunately, we can't really run this test with Libreswan 4.x, since, due to internal issues in these versions, we are getting into states where everything is loaded and active, but no traffic can pass. This is an internal issue in Libreswan that we can't workaround from the outside. So, the fix is required in Libreswan itself. 4.5 and earlier versions seem to not be affected by this problem, at least not severely affected, but it's easier to just cut off all the 4.x versions from the test. 3.32 version from Ubuntu 22.04 and Libreswna 5.1 work just fine with this test. Test is relatively long, but it is very valuable, IMO. Besides stressing ovs-monitor-ipsec with various failure and asynchronous connection establishment conditions, which are important for OVS, it also was used to reproduce and fix several bugs in Libreswan 4.x. Unfortunately, not all the issues are understood and fixed yet. Signed-off-by: Ilya Maximets --- tests/system-ipsec.at | 82 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 8 deletions(-) diff --git a/tests/system-ipsec.at b/tests/system-ipsec.at index 5aa67bf1d..8e6733371 100644 --- a/tests/system-ipsec.at +++ b/tests/system-ipsec.at @@ -71,7 +71,9 @@ m4_define([IPSEC_ADD_NODE], on_exit "kill `cat $ovs_base/$1/ovs-monitor-ipsec.pid`" dnl Set up OVS bridge - NS_EXEC([$1], [ovs-vsctl --db unix:$ovs_base/$1/db.sock add-br br-ipsec])] + NS_CHECK_EXEC([$1], + [ovs-vsctl --db unix:$ovs_base/$1/db.sock add-br br-ipsec \ + -- set-controller br-ipsec punix:$ovs_base/br-ipsec.$1.mgmt])] ) m4_define([IPSEC_ADD_NODE_LEFT], [IPSEC_ADD_NODE(left, p0, $1, $2)]) m4_define([IPSEC_ADD_NODE_RIGHT], [IPSEC_ADD_NODE(right, p1, $1, $2)]) @@ -429,7 +431,8 @@ m4_for([id], [1], NODES, [1], [ self-sign node-id], [0], [stdout]) AT_CHECK(OVS_VSCTL([node-id], set Open_vSwitch . \ other_config:certificate=${ovs_base}/node-id-cert.pem \ - other_config:private_key=${ovs_base}/node-id-privkey.pem), + other_config:private_key=${ovs_base}/node-id-privkey.pem \ + -- set bridge br-ipsec other-config:hwaddr=f2:ff:00:00:00:id), [0], [ignore], [ignore]) on_exit "ipsec --rundir $ovs_base/node-id status > $ovs_base/node-id/status" ]) @@ -445,11 +448,18 @@ m4_for([LEFT], [1], NODES, [1], [ fi ])]) +dnl These are not necessary, but nice to have in the test log in +dnl order to spot pluto failures during the test. +on_exit "grep -E 'timed out|outdated|half-loaded|defunct' \ + $ovs_base/node-*/ovs-monitor-ipsec.log" +on_exit "grep -E 'ABORT|ERROR' $ovs_base/node-*/pluto.log" + m4_define([WAIT_FOR_LOADED_CONNS], [ m4_for([id], [1], NODES, [1], [ echo "================== node-id =========================" iterations=0 loaded=0 + active=0 dnl Using a custom loop instead of OVS_WAIT_UNTIL, because it may take dnl much longer than a default timeout. The default retransmit timeout dnl for pluto is 60 seconds. Also, we need to make sure pluto didn't @@ -463,8 +473,11 @@ m4_define([WAIT_FOR_LOADED_CONNS], [ START_PLUTO([node-id]) else loaded=$(IPSEC_STATUS_LOADED(node-id)) + m4_if([$1], [active], + [active=$(IPSEC_STATUS_ACTIVE(node-id))], [active=$loaded]) fi - if test "$loaded" -ne $(( (NODES - 1) * 2 )); then + if test "$loaded" -ne "$(( (NODES - 1) * 2 ))" -o \ + "$loaded" -ne "$active"; then sleep 3 else break @@ -505,11 +518,64 @@ OVS_WAIT_UNTIL([grep -q 'tun-2.*need to reconcile' \ dnl Wait for all the connections to be loaded back. WAIT_FOR_LOADED_CONNS() -dnl These are not necessary, but nice to have in the test log in -dnl order to spot pluto failures during the test. -grep -E 'timed out|outdated|half-loaded|defunct' \ - $ovs_base/node-*/ovs-monitor-ipsec.log -grep -E 'ABORT|ERROR' $ovs_base/node-*/pluto.log +dnl Next section will check connectivity between all the nodes. +dnl Different versions of Libreswan 4.x have issues where connections +dnl are not being correctly established or never become active in a +dnl way that can not be mitigated from ovs-monitor-ipsec or the test. +dnl So, only checking connectivity for Libreswan 3- or 5+. +if ! (ipsec --version 2>&1 | grep -q 'Libreswan 4\.'); then + dnl Turn off IPv6 and add static ARP entries for all namespaces to avoid + dnl any broadcast / multicast traffic that would otherwise be multiplied + dnl by each node creating a traffic storm. Add specific OpenFlow rules + dnl to forward traffic to exact destinations without any MAC learning. + m4_for([LEFT], [1], NODES, [1], [ + NS_CHECK_EXEC([node-LEFT], [sysctl -w net.ipv6.conf.all.disable_ipv6=1], + [0], [ignore]) + AT_CHECK([ovs-ofctl del-flows unix:$ovs_base/br-ipsec.node-LEFT.mgmt]) + AT_CHECK([ovs-ofctl add-flow unix:$ovs_base/br-ipsec.node-LEFT.mgmt \ + "dl_dst=f2:ff:00:00:00:LEFT actions=LOCAL"]) + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + NS_CHECK_EXEC([node-LEFT], + [ip neigh add 192.0.0.RIGHT lladdr f2:ff:00:00:00:RIGHT dev br-ipsec]) + AT_CHECK([ovs-ofctl add-flow unix:$ovs_base/br-ipsec.node-LEFT.mgmt \ + "dl_dst=f2:ff:00:00:00:RIGHT actions=tun-RIGHT"]) + fi + ]) + ]) + + dnl Bring up and add IP addresses for br-ipsec interface. + m4_for([id], [1], NODES, [1], [ + echo "================== node-id =========================" + NS_CHECK_EXEC([node-id], [ip addr add 192.0.0.id/24 dev br-ipsec]) + NS_CHECK_EXEC([node-id], [ip link set dev br-ipsec up]) + ]) + + dnl Wait for all the connections to be loaded and active. In case one of + dnl the pluto processes crashed some of the connections may never become + dnl active. But we did run this loop with a pluto reviving logic twice + dnl already, so the chances for pluto to be down here are much lower. + WAIT_FOR_LOADED_CONNS([active]) + + dnl Check the full mesh ping. + m4_for([LEFT], [1], NODES, [1], [ + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + echo "====== ping: node-LEFT --> node-RIGHT ==========" + dnl Ping without checking in case connection will recover after the + dnl first packet. + NS_CHECK_EXEC([node-LEFT], + [ping -q -c 1 -W 2 192.0.0.RIGHT | FORMAT_PING], + [ignore], [stdout]) + dnl Now check. If this one fails, there is no actual connectivity. + NS_CHECK_EXEC([node-LEFT], + [ping -q -c 3 -i 0.1 -W 2 192.0.0.RIGHT | FORMAT_PING], + [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + fi + ])]) +fi OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP