From patchwork Mon Jun 27 09:30:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xavier Simonart X-Patchwork-Id: 1648814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FpEfRM+Q; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LWjBY4h1pz9sGm for ; Mon, 27 Jun 2022 19:30:29 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 85D8861088; Mon, 27 Jun 2022 09:30:26 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 85D8861088 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FpEfRM+Q X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kDoApyb4Fc_F; Mon, 27 Jun 2022 09:30:25 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4B0F66107F; Mon, 27 Jun 2022 09:30:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 4B0F66107F Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 16556C0039; Mon, 27 Jun 2022 09:30:24 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id CA3F6C002D for ; Mon, 27 Jun 2022 09:30:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id ACEC7831F1 for ; Mon, 27 Jun 2022 09:30:22 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org ACEC7831F1 Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FpEfRM+Q X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YCMQLm-OBL-u for ; Mon, 27 Jun 2022 09:30:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 707B8831EF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 707B8831EF for ; Mon, 27 Jun 2022 09:30:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1656322219; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B1Xxc9T8t34TJvk/aBDagfKV2GP3E3GxFNu/unmnLcc=; b=FpEfRM+QhxKFHVNjimVAsv8rE0n2mSUWMsKFfHCXqJ7hkN9RKmtKLSCDrHNf5fSwZ3qK+D YMKotciedyeKLg+vAimqpv9X5KgenEZ+lbnDpjmXuwRZqBxvxWGSmAhQsxvn0Bfe/r6too iWKgGvlXanMy3SQnNGpz6XWpNT7n2Bw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-408-NxLYndePNIyyHBHiQCO0nQ-1; Mon, 27 Jun 2022 05:30:18 -0400 X-MC-Unique: NxLYndePNIyyHBHiQCO0nQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1141E1C08980 for ; Mon, 27 Jun 2022 09:30:18 +0000 (UTC) Received: from wsfd-netdev90.ntdv.lab.eng.bos.redhat.com (wsfd-netdev90.ntdv.lab.eng.bos.redhat.com [10.19.188.196]) by smtp.corp.redhat.com (Postfix) with ESMTP id E3C0E404E4C8; Mon, 27 Jun 2022 09:30:17 +0000 (UTC) From: Xavier Simonart To: xsimonar@redhat.com, dev@openvswitch.org Date: Mon, 27 Jun 2022 05:30:17 -0400 Message-Id: <20220627093017.2839914-1-xsimonar@redhat.com> In-Reply-To: <20220622150232.3987202-1-xsimonar@redhat.com> References: <20220622150232.3987202-1-xsimonar@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=xsimonar@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2] tests: fixed multiple flaky tests (not waiting for patch flows) X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The following test cases were sometimes failing, (mainly) for the same reason i.e. packet lost as being sent before patch ports were installed. - 2 HVs, 2 LS, switching between multiple localnet ports with same tags - VLAN transparency, passthru=true, ARP responder disabled - send gratuitous arp for l3gateway only on selected chassis - 4 HV, 3 LS, 2 LR, packet test with HA distributed router gateway port - 1 LR with distributed router gateway port - send gratuitous arp for NAT rules on distributed router - send gratuitous ARP for NAT rules on HA distributed router - localnet connectivity with multiple requested-chassis - 2 HVs, 2 lports/HV, localnet ports, DVR chassis mac - 2 HVs, 4 lports/HV, localnet ports - 2 HVs, 2 LS, routing works for multiple collocated segments attached to different switches - 2 HVs, 1 LS, no switching between multiple localnet ports with different tags Signed-off-by: Xavier Simonart --- v2: - handled Mark's comments. rebased on origin/main --- tests/ovn.at | 81 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 54 insertions(+), 27 deletions(-) diff --git a/tests/ovn.at b/tests/ovn.at index bfaa41962..b855653ac 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -76,19 +76,22 @@ m4_divert_text([PREPARE_TESTS], } ovn_wait_patch_port_flows () { - patch_port=$1 - hv=$2 - echo "$3: waiting for flows for $patch_port on $hv" - # Patch port might be created after ports are reported up - OVS_WAIT_UNTIL([ - test 1 = $(as $hv ovs-vsctl show | grep "Port $patch_port" | wc -l) - ]) - # Wait for a flow outputing to patch port - OVS_WAIT_UNTIL([ - hv_patch_ofport=$(as $hv ovs-vsctl --bare --columns ofport find Interface name=$patch_port) - echo "$patch_port=$hv_patch_ofport" - test 1 = $(as $hv ovs-ofctl dump-flows br-int | grep -c "output:$hv_patch_ofport") - ]) + for localnet in $1; do + patch_port="patch-br-int-to-$localnet" + for hv in $2; do + echo "$3: waiting for flows for $patch_port on $hv" + # Patch port might be created after ports are reported up + OVS_WAIT_UNTIL([ + test 1 = $(as $hv ovs-vsctl show | grep "Port \b$patch_port\b" | wc -l) + ]) + # Wait for a flow outputing to patch port + OVS_WAIT_UNTIL([ + hv_patch_ofport=$(as $hv ovs-vsctl --bare --columns ofport find Interface name=$patch_port) + echo "$patch_port=$hv_patch_ofport" + test 1 -le $(as $hv ovs-ofctl dump-flows br-int | grep -c "output:\b$hv_patch_ofport\b") + ]) + done + done } ovn_wait_remote_output_flows () { @@ -2952,6 +2955,8 @@ for i in 1 2; do done done wait_for_ports_up +OVN_WAIT_PATCH_PORT_FLOWS(["ln1" "ln2" "ln3"], ["hv1"] ["hv2"]) + ovn-nbctl --wait=sb sync ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -3122,6 +3127,7 @@ done wait_for_ports_up ovn-nbctl --wait=sb sync +OVN_WAIT_PATCH_PORT_FLOWS(["ln-11" "ln-21"], ["hv-1" "hv-2"]) ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -3394,6 +3400,7 @@ for i in 1 2; do ovn-nbctl lsp-set-port-security $lsp_name f0:00:00:00:00:0$i OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up $lsp_name` = xup]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln-$i-20"], ["hv-$i"]) done wait_for_ports_up @@ -3501,7 +3508,7 @@ done wait_for_ports_up -# Remote output flows are setup whe pb of remote is received +# Remote output flows are setup when pb of remote is received # Hence they can be setup after both ports have been reported up. OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv-1"],["hv-2"]) OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv-2"],["hv-1"]) @@ -3641,7 +3648,7 @@ for i in 1 2; do # Patch port might be created after ports are reported up # Wait for a flow outputing to patch port - OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-100"], ["hv-$i"]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln-100"], ["hv-$i"]) done test_packet() { @@ -3710,7 +3717,7 @@ for i in 1 2; do # Patch port might be created after ports are reported up # Wait for a flow outputing to patch port - OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-100"], ["hv-$i"]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln-100"], ["hv-$i"]) done # create taps on fabric to check vlan encapsulation there for i in 1 2; do @@ -3796,7 +3803,7 @@ for i in 1 2; do # Patch port might be created after ports are reported up # Wait for a flow outputing to patch port - OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln"], ["hv-$i"]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln"], ["hv-$i"]) done for i in 1 2; do @@ -4021,6 +4028,7 @@ for tag in 10 20; do ovn-nbctl lsp-set-port-security $lsp_name f0:00:00:00:0$i:$tag OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up $lsp_name` = xup]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln-$tag"], ["hv-$tag-$i"]) done done wait_for_ports_up @@ -8856,14 +8864,14 @@ AT_CAPTURE_FILE([sbflows]) # Wait until the patch ports are created in hv1 and hv2 to connect br-int to br-eth0 AT_CHECK([ovn-nbctl set logical_router lr0 options:chassis=hv1]) -OVS_WAIT_UNTIL([test 1 = `as hv1 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln_port"], ["hv1"]) AT_CHECK([ovn-nbctl set logical_router lr0 options:chassis=hv2]) -OVS_WAIT_UNTIL([test 1 = `as hv2 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln_port"], ["hv2"]) # Temporarily remove lr0 chassis -AT_CHECK([ovn-nbctl remove logical_router lr0 options chassis]) +# Wait for hv confirmation to make sure chassis is removed before we reset pcap +# Otherwise a garp might be sent after pcap have been reset but before chassis is removed +AT_CHECK([ovn-nbctl --wait=hv remove logical_router lr0 options chassis]) reset_pcap_file() { local hv=$1 @@ -11324,6 +11332,8 @@ check as ext1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys wait_for_ports_up check ovn-nbctl --wait=sb sync +OVN_WAIT_PATCH_PORT_FLOWS(["ln-alice"], ["gw1"] ["gw2"]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln-outside"], ["ext1"]) ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -11540,6 +11550,7 @@ check as hv3 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys dnl Allow some time for ovn-northd and ovn-controller to catch up. wait_for_ports_up ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["ln-outside"], ["hv3"]) (echo "---------NB dump-----" ovn-nbctl show @@ -11654,7 +11665,7 @@ OVN_CHECK_PACKETS_UNIQ([hv3/vif1-tx.pcap], [hv3-vif1.expected]) as hv2 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys # Wait until the patch ports are created in hv2 to connect br-int to br-phys -OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-alice"], ["hv2"]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln-alice"], ["hv2"]) dnl Allow some time for ovn-controller to catch up. wait_for_ports_up @@ -11733,6 +11744,7 @@ ovn-nbctl lsp-add foo foo2 \ # Allow some time for ovn-northd and ovn-controller to catch up. wait_for_ports_up check ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["ln-alice"], ["hv1"]) : > hv1-vif2.expected @@ -11831,8 +11843,7 @@ OVN_CHECK_PACKETS([hv1/snoopvif-tx.pcap], [packets]) AT_CHECK([as hv2 ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=physnet1:br-phys]) # Wait until the patch ports are created in hv2 to connect br-int to br-phys -OVS_WAIT_UNTIL([test 1 = `as hv2 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln_port"], ["hv2"]) # Wait for packets to be received. OVS_WAIT_UNTIL([test `wc -c < "hv1/snoopvif-tx.pcap"` -ge 100]) @@ -11875,7 +11886,7 @@ ovs-vsctl -- add-port br-int hv3-vif2 -- \ AT_CHECK([as hv3 ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=physnet1:br-phys]) # Wait until the patch ports are created in hv3 to connect br-int to br-phys -OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln_port"], ["hv3"]) +OVN_WAIT_PATCH_PORT_FLOWS(["ln_port"], ["hv3"]) # Re-add nat-addresses option ovn-nbctl lsp-set-options lrp0-rp router-port=lrp0 nat-addresses="router" @@ -13550,6 +13561,7 @@ AT_CHECK([ovn-nbctl lsp-set-options ln_port network_name=physnet1]) # wait for earlier changes to take effect wait_for_ports_up check ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["ln_port"], ["hv2"] ["hv3"]) reset_pcap_file() { local iface=$1 @@ -14762,6 +14774,13 @@ reset_env # Start port migration hv1 -> hv2: both hypervisors are now bound check ovn-nbctl lsp-set-options migrator requested-chassis=hv1,hv2 wait_for_ports_up +OVN_WAIT_PATCH_PORT_FLOWS(["public"], ["hv1"]) +OVN_WAIT_PATCH_PORT_FLOWS(["public"], ["hv2"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv1"],["hv2"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv2"],["hv1"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv3"],[hv1"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv3"],[hv2"]) + wait_column "$hv1_uuid" Port_Binding chassis logical_port=migrator wait_column "$hv1_uuid" Port_Binding requested_chassis logical_port=migrator wait_column "$hv2_uuid" Port_Binding additional_chassis logical_port=migrator @@ -20079,6 +20098,11 @@ ovn-nbctl lsp-add ls2 ls2-to-router -- set Logical_Switch_Port ls2-to-router typ wait_for_ports_up ovn-nbctl --wait=sb sync #ovn-sbctl dump-flows +for i in 1 2; do + for j in 1 2; do + OVN_WAIT_PATCH_PORT_FLOWS(["ln$j"], ["hv$i"]) + done +done ovn-nbctl show ovn-sbctl show @@ -22484,6 +22508,8 @@ m4_define([DVR_N_S_ARP_HANDLING], ovn-nbctl lrp-set-gateway-chassis router-to-underlay hv3 wait_for_ports_up ovn-nbctl --wait=sb sync + OVN_WAIT_PATCH_PORT_FLOWS(["ln1"] ["ln2"], ["hv1"] ["hv2"] ["hv3"]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln4"], ["hv4"]) wait_row_count Port_Binding 1 logical_port=cr-router-to-underlay @@ -22704,7 +22730,8 @@ m4_define([DVR_N_S_PING], wait_for_ports_up ovn-nbctl --wait=sb sync - + OVN_WAIT_PATCH_PORT_FLOWS(["ln1"] ["ln2"], ["hv1"] ["hv2"] ["hv3"]) + OVN_WAIT_PATCH_PORT_FLOWS(["ln4"], ["hv4"]) OVN_POPULATE_ARP