From patchwork Tue Apr 13 17:06:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gray X-Patchwork-Id: 1465912 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bRYk0SKm; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKX8N6b9nz9sRf for ; Wed, 14 Apr 2021 03:07:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2F51E405E4; Tue, 13 Apr 2021 17:06:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vrUX3QmlBeI8; Tue, 13 Apr 2021 17:06:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id 3F2EE402B8; Tue, 13 Apr 2021 17:06:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0E4D5C000C; Tue, 13 Apr 2021 17:06:56 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 47E57C000A for ; Tue, 13 Apr 2021 17:06:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1C84940E51 for ; Tue, 13 Apr 2021 17:06:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id otabObbGL7i9 for ; Tue, 13 Apr 2021 17:06:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 825E240E4D for ; Tue, 13 Apr 2021 17:06:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618333612; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=bTRjw/LcuIn661RVMbnu244JQomfFoXtO7Mp6eFxocE=; b=bRYk0SKmT/0J3egHCgk0IEQ853bZ+0qRhRC9I6wzsZt1i/Wg7bvCqDpE4q0WwIL3eTSC9c gc755xy4AgmiCuDz1bgGO1cULzQsUfokpiUnhaRuTWs9CfgcH8PlD8jSahUeQzmb5jHJpY nBVggJnQsm6I5USf+qPPyRGCslEsUf8= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-449-d9kWkuMwOG22pCvcQkUA_Q-1; Tue, 13 Apr 2021 13:06:49 -0400 X-MC-Unique: d9kWkuMwOG22pCvcQkUA_Q-1 Received: by mail-pl1-f199.google.com with SMTP id q12-20020a170902edccb02900e6b86717d2so5452445plk.18 for ; Tue, 13 Apr 2021 10:06:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=bTRjw/LcuIn661RVMbnu244JQomfFoXtO7Mp6eFxocE=; b=i8jNnekNCrKNXUwY2Jjt+DQhZqj0lGHKTwWbjKxlg7fPt3IgAfCqNlwAMC902yZF/M ihCOx5I5T+44XB1UE0+nnRza+9e5Autp0TXr96jsfdg8ydv+pIhe2Frwl9uo3vrHK0HN 8ZbPSHLdc2YZaR5FLXA4oO2lmkN27zqlwzjJGwkVpLtw7kSfjhSbp6w35MSsO2vVwAJa HZps+LFH4OBby12EbqKPyWk0XLN/FPUENCrnFJGSG5QoCaCcN7sQxH9KdEiq91ZZGB4j k+gaWLb7G312z6nJR5oNHYveonK047wvKB9IfEAR0Z3px7gTOuMUiPIQDA1NnzCbbahI o2ew== X-Gm-Message-State: AOAM532/pBQwcaGmPzjchKA5Dnxt7reksAr0SF2+GYR6QBgIxWbosi4d zLK6/M2IFdf13d1U4A9fjiUvHN3fZk6oC1CIp/XHGAgjc66deOoQlZRbHsWE2RdAiV4VIHirVa5 6cObtyaWfcyuXzd2JOe/TEQnMZ7XT+yKoYNvKgXqSlajAnb/whT8BEdvlnfpXj/GO/c+u X-Received: by 2002:a17:90a:1e63:: with SMTP id w90mr997231pjw.115.1618333608392; Tue, 13 Apr 2021 10:06:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx0HrowKkOpdH/G5wk8IngUnBE7gKILDSapXBrpIrglkEruasQS8/5Y+ZdqmjSJzuzjAjeHVg== X-Received: by 2002:a17:90a:1e63:: with SMTP id w90mr997198pjw.115.1618333608002; Tue, 13 Apr 2021 10:06:48 -0700 (PDT) Received: from wsfd-netdev91.ntdv.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id e12sm9426824pfm.29.2021.04.13.10.06.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 10:06:47 -0700 (PDT) From: Mark Gray To: dev@openvswitch.org Date: Tue, 13 Apr 2021 13:06:40 -0400 Message-Id: <20210413170640.56394-1-mark.d.gray@redhat.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mark.d.gray@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org Subject: [ovs-dev] [PATCH] ipsec: Fix race in system tests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch fixes an issue where, depending on timing fluctuations, each node has not fully loaded all connections before the other node begins to establish a connection. In this failure case, the "ovs-monitor-ipsec" instance on the "left" node may `ipsec auto --start` a connection which then gets rejected by the "right" side. Almost, simulaneously, the "right" side may initiate a connection that gets rejected by the "left" side. This can happen as, for all tunnels except for GRE, each node has two connections (an "in" connection and an "out" connection) that get added one after the other. If the "in" connection "starts" on both sides, the "out" connection from the other node may not be available causing the connection to fail. At this point, "Libreswan" will wait to retry the connection. In the interim, the OVS system test times out. This race manifests itself more frequently in a virtualized environment. This patch resolves this issue by waiting for the "left" node to load all connections before starting the "right" side. This will cause the "left" side to fail to establish a connection with the "right" side (as the "right" side connections have not been loaded) but will cause the "right" side to succeed to establish a connection as all connections will have been loaded on the "left" side. Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2021-April/381857.html Fixes: 8fc62df8b135 ("ipsec: Introduce IPsec system tests for Libreswan.") Signed-off-by: Mark Gray Tested-by: Flavio Leitner Acked-by: Flavio Leitner --- tests/system-ipsec.at | 131 +++++++++++++++++++++--------------------- 1 file changed, 65 insertions(+), 66 deletions(-) diff --git a/tests/system-ipsec.at b/tests/system-ipsec.at index 2cd0469f5d33..f45a153eddce 100644 --- a/tests/system-ipsec.at +++ b/tests/system-ipsec.at @@ -79,6 +79,20 @@ m4_define([OVS_VSCTL], m4_define([OVS_VSCTL_LEFT], [OVS_VSCTL(left, $1)]) m4_define([OVS_VSCTL_RIGHT], [OVS_VSCTL(right, $1)]) +dnl IPSEC_ADD_TUNNEL([namespace], [type], [options]]) +dnl +dnl Creates a tunnel of type 'type' in namespace 'namespace' using 'options' +m4_define([IPSEC_ADD_TUNNEL], + [OVS_VSCTL([$1], [add-port br-ipsec tun -- set Interface tun type=$2 $3]) + dnl Wait for all expected connections to be loaded into Libreswan. + dnl GRE creates 1 connection, all others create 2. + m4_if($2, [gre], + [OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED($1)` -eq 1])], + [OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED($1)` -eq 2])]) + ]) +m4_define([IPSEC_ADD_TUNNEL_LEFT], [IPSEC_ADD_TUNNEL(left, $1, $2)]) +m4_define([IPSEC_ADD_TUNNEL_RIGHT], [IPSEC_ADD_TUNNEL(right, $1, $2)]) + dnl CHECK_LIBRESWAN() dnl dnl Check if necessary Libreswan dependencies are available on the test machine @@ -124,13 +138,6 @@ m4_define([CHECK_ESP_TRAFFIC], tcpdump -l -nn -i ovs-p1 esp > $ovs_base/right/tcpdump.log & on_exit "kill $!" - dnl GRE creates 1 connection, all others create 2 - m4_if($1, [gre], - [OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(left)` -eq 1]) - OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(right)` -eq 1])], - [OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(left)` -eq 2]) - OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(right)` -eq 2])]) - dnl Wait for all loaded connections to be active OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(left)` -eq `IPSEC_STATUS_ACTIVE(left)`]) OVS_WAIT_UNTIL([test `IPSEC_STATUS_LOADED(right)` -eq `IPSEC_STATUS_ACTIVE(right)`]) @@ -163,15 +170,13 @@ IPSEC_ADD_NODE_LEFT(10.1.1.1, 10.1.1.2) IPSEC_ADD_NODE_RIGHT(10.1.1.2, 10.1.1.1) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.2 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([geneve], + [options:remote_ip=10.1.1.2 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host - -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.1 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(geneve) +IPSEC_ADD_TUNNEL_RIGHT([geneve], + [options:remote_ip=10.1.1.1 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -190,15 +195,15 @@ IPSEC_ADD_NODE_LEFT(10.1.1.1, 10.1.1.2) IPSEC_ADD_NODE_RIGHT(10.1.1.2, 10.1.1.1) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.2 options:local_ip=10.1.1.1 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([geneve], + [options:remote_ip=10.1.1.2 \ + options:local_ip=10.1.1.1 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host - -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.1 options:local_ip=10.1.1.2 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(geneve) +IPSEC_ADD_TUNNEL_RIGHT([geneve], + [options:remote_ip=10.1.1.1 \ + options:local_ip=10.1.1.2 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -229,15 +234,15 @@ OVS_VSCTL_RIGHT(set Open_vSwitch . \ other_config:private_key=${ovs_base}/right-privkey.pem) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.2 options:remote_cert=${ovs_base}/right-cert.pem) +IPSEC_ADD_TUNNEL_LEFT([geneve], + [options:remote_ip=10.1.1.2 \ + options:remote_cert=${ovs_base}/right-cert.pem]) dnl Set up IPsec tunnel on 'right' host - -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.1 options:remote_cert=${ovs_base}/left-cert.pem) - -CHECK_ESP_TRAFFIC(geneve) +IPSEC_ADD_TUNNEL_RIGHT([geneve], + [options:remote_ip=10.1.1.1 \ + options:remote_cert=${ovs_base}/left-cert.pem]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -269,14 +274,13 @@ OVS_VSCTL_RIGHT(set Open_vSwitch . \ other_config:private_key=${ovs_base}/right-privkey.pem) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.2 options:remote_name=right) +IPSEC_ADD_TUNNEL_LEFT([geneve], + [options:remote_ip=10.1.1.2 options:remote_name=right]) dnl Set up IPsec tunnel on 'right' host -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=10.1.1.1 options:remote_name=left) - -CHECK_ESP_TRAFFIC(geneve) +IPSEC_ADD_TUNNEL_RIGHT([geneve], + [options:remote_ip=10.1.1.1 options:remote_name=left]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -293,15 +297,13 @@ IPSEC_ADD_NODE_LEFT(10.1.1.1, 10.1.1.2) IPSEC_ADD_NODE_RIGHT(10.1.1.2, 10.1.1.1) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=gre \ - options:remote_ip=10.1.1.2 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([gre], + [options:remote_ip=10.1.1.2 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host - -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=gre \ - options:remote_ip=10.1.1.1 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(gre) +IPSEC_ADD_TUNNEL_RIGHT([gre], + [options:remote_ip=10.1.1.1 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -318,15 +320,13 @@ IPSEC_ADD_NODE_LEFT(10.1.1.1, 10.1.1.2) IPSEC_ADD_NODE_RIGHT(10.1.1.2, 10.1.1.1) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=10.1.1.2 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([vxlan], + [options:remote_ip=10.1.1.2 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host - -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=10.1.1.1 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(vxlan) +IPSEC_ADD_TUNNEL_RIGHT([vxlan], + [options:remote_ip=10.1.1.1 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -343,14 +343,13 @@ IPSEC_ADD_NODE_LEFT(fd01::101, fd01::102) IPSEC_ADD_NODE_RIGHT(fd01::102, fd01::101) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=fd01::102 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([vxlan], + [options:remote_ip=fd01::102 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=fd01::101 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(vxlan) +IPSEC_ADD_TUNNEL_RIGHT([vxlan], + [options:remote_ip=fd01::101 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -367,14 +366,15 @@ IPSEC_ADD_NODE_LEFT(fd01::101, fd01::102) IPSEC_ADD_NODE_RIGHT(fd01::102, fd01::101) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=fd01::102 options:local_ip=fd01::101 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([vxlan], + [options:remote_ip=fd01::102 \ + options:local_ip=fd01::101 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=vxlan \ - options:remote_ip=fd01::101 options:local_ip=fd01::102 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(vxlan) +IPSEC_ADD_TUNNEL_RIGHT([vxlan], + [options:remote_ip=fd01::101 \ + options:local_ip=fd01::102 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP @@ -393,14 +393,13 @@ IPSEC_ADD_NODE_LEFT(fd01::101, fd01::102) IPSEC_ADD_NODE_RIGHT(fd01::102, fd01::101) dnl Set up IPsec tunnel on 'left' host -OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=fd01::102 options:psk=swordfish) +IPSEC_ADD_TUNNEL_LEFT([geneve], + [options:remote_ip=fd01::102 options:psk=swordfish]) dnl Set up IPsec tunnel on 'right' host -OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ - options:remote_ip=fd01::101 options:psk=swordfish) - -CHECK_ESP_TRAFFIC(geneve) +IPSEC_ADD_TUNNEL_RIGHT([geneve], + [options:remote_ip=fd01::101 options:psk=swordfish]) +CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP