From patchwork Wed Jun 22 15:02:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xavier Simonart X-Patchwork-Id: 1646583 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LMnBmm8C; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LSmpF0bGtz9s5V for ; Thu, 23 Jun 2022 01:02:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2987781435; Wed, 22 Jun 2022 15:02:41 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 2987781435 Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LMnBmm8C X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 55BeKOz4_C2M; Wed, 22 Jun 2022 15:02:40 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 019A8813F6; Wed, 22 Jun 2022 15:02:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 019A8813F6 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BD075C0039; Wed, 22 Jun 2022 15:02:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id BBA4BC002D for ; Wed, 22 Jun 2022 15:02:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 87A45813D5 for ; Wed, 22 Jun 2022 15:02:37 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 87A45813D5 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BCIl-1_dmByk for ; Wed, 22 Jun 2022 15:02:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 88D7C813C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 88D7C813C7 for ; Wed, 22 Jun 2022 15:02:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655910155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=GvqABShAcCW8wTBNRzc5K2Fr67Wg2lD29OUbu8wbM3M=; b=LMnBmm8CRV9JdiTJIBSgmCw6ETUHfIcPDrKSiuTj/5bxR8DjUvHBRqjdzSBX+DkvWXxAmX ji1d27Zh68EhaBv54EKj25d2oI0UQ841JSqjWVZAmehE4MRj5AjpSF1uJ7OoSSSgDZwh3X HH7+BFNntbR+jPePbRL0D7IBdjtZW2Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-298-CzXgCBCNNRykneJAK89PUQ-1; Wed, 22 Jun 2022 11:02:34 -0400 X-MC-Unique: CzXgCBCNNRykneJAK89PUQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EE5D9801E80 for ; Wed, 22 Jun 2022 15:02:33 +0000 (UTC) Received: from wsfd-netdev90.ntdv.lab.eng.bos.redhat.com (wsfd-netdev90.ntdv.lab.eng.bos.redhat.com [10.19.188.196]) by smtp.corp.redhat.com (Postfix) with ESMTP id D584B40334F; Wed, 22 Jun 2022 15:02:33 +0000 (UTC) From: Xavier Simonart To: xsimonar@redhat.com, dev@openvswitch.org Date: Wed, 22 Jun 2022 11:02:32 -0400 Message-Id: <20220622150232.3987202-1-xsimonar@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=xsimonar@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn] tests: fixed multiple flaky tests (not waiting for patch flows) X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The following test cases were sometimes failing, (mainly) for the same reason i.e. packet lost as being sent before patch ports were installed. - 2 HVs, 2 LS, switching between multiple localnet ports with same tags - VLAN transparency, passthru=true, ARP responder disabled - send gratuitous arp for l3gateway only on selected chassis - 4 HV, 3 LS, 2 LR, packet test with HA distributed router gateway port - 1 LR with distributed router gateway port - send gratuitous arp for NAT rules on distributed router - send gratuitous ARP for NAT rules on HA distributed router - localnet connectivity with multiple requested-chassis - 2 HVs, 2 lports/HV, localnet ports, DVR chassis mac - 2 HVs, 4 lports/HV, localnet ports - 2 HVs, 2 LS, routing works for multiple collocated segments attached to different switches - 2 HVs, 1 LS, no switching between multiple localnet ports with different tags Signed-off-by: Xavier Simonart --- tests/ovn.at | 70 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 48 insertions(+), 22 deletions(-) diff --git a/tests/ovn.at b/tests/ovn.at index bfaa41962..5913537bc 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -76,19 +76,21 @@ m4_divert_text([PREPARE_TESTS], } ovn_wait_patch_port_flows () { - patch_port=$1 - hv=$2 - echo "$3: waiting for flows for $patch_port on $hv" - # Patch port might be created after ports are reported up - OVS_WAIT_UNTIL([ - test 1 = $(as $hv ovs-vsctl show | grep "Port $patch_port" | wc -l) - ]) - # Wait for a flow outputing to patch port - OVS_WAIT_UNTIL([ - hv_patch_ofport=$(as $hv ovs-vsctl --bare --columns ofport find Interface name=$patch_port) - echo "$patch_port=$hv_patch_ofport" - test 1 = $(as $hv ovs-ofctl dump-flows br-int | grep -c "output:$hv_patch_ofport") - ]) + for patch_port in $1; do + for hv in $2; do + echo "$3: waiting for flows for $patch_port on $hv" + # Patch port might be created after ports are reported up + OVS_WAIT_UNTIL([ + test 1 = $(as $hv ovs-vsctl show | grep "Port \b$patch_port\b" | wc -l) + ]) + # Wait for a flow outputing to patch port + OVS_WAIT_UNTIL([ + hv_patch_ofport=$(as $hv ovs-vsctl --bare --columns ofport find Interface name=$patch_port) + echo "$patch_port=$hv_patch_ofport" + test 1 -le $(as $hv ovs-ofctl dump-flows br-int | grep -c "output:\b$hv_patch_ofport\b") + ]) + done + done } ovn_wait_remote_output_flows () { @@ -2952,6 +2954,8 @@ for i in 1 2; do done done wait_for_ports_up +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln1" "patch-br-int-to-ln2" "patch-br-int-to-ln3"], ["hv1"] ["hv2"]) + ovn-nbctl --wait=sb sync ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -3122,6 +3126,7 @@ done wait_for_ports_up ovn-nbctl --wait=sb sync +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-11" "patch-br-int-to-ln-21"], ["hv-1" "hv-2"]) ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -3394,6 +3399,7 @@ for i in 1 2; do ovn-nbctl lsp-set-port-security $lsp_name f0:00:00:00:00:0$i OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up $lsp_name` = xup]) + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-$i-20"], ["hv-$i"]) done wait_for_ports_up @@ -3501,7 +3507,7 @@ done wait_for_ports_up -# Remote output flows are setup whe pb of remote is received +# Remote output flows are setup when pb of remote is received # Hence they can be setup after both ports have been reported up. OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv-1"],["hv-2"]) OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv-2"],["hv-1"]) @@ -4021,6 +4027,7 @@ for tag in 10 20; do ovn-nbctl lsp-set-port-security $lsp_name f0:00:00:00:0$i:$tag OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up $lsp_name` = xup]) + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-$tag"], ["hv-$tag-$i"]) done done wait_for_ports_up @@ -8856,14 +8863,14 @@ AT_CAPTURE_FILE([sbflows]) # Wait until the patch ports are created in hv1 and hv2 to connect br-int to br-eth0 AT_CHECK([ovn-nbctl set logical_router lr0 options:chassis=hv1]) -OVS_WAIT_UNTIL([test 1 = `as hv1 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln_port"], ["hv1"]) AT_CHECK([ovn-nbctl set logical_router lr0 options:chassis=hv2]) -OVS_WAIT_UNTIL([test 1 = `as hv2 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln_port"], ["hv2"]) # Temporarily remove lr0 chassis -AT_CHECK([ovn-nbctl remove logical_router lr0 options chassis]) +# Wait for hv confirmation to make sure chassis is removed before we reset pcap +# Otherwise a garp might be sent after pcap have been reset but before chassis is removed +AT_CHECK([ovn-nbctl --wait=hv remove logical_router lr0 options chassis]) reset_pcap_file() { local hv=$1 @@ -11324,6 +11331,8 @@ check as ext1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys wait_for_ports_up check ovn-nbctl --wait=sb sync +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-alice"], ["gw1"] ["gw2"]) +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-outside"], ["ext1"]) ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) @@ -11540,6 +11549,7 @@ check as hv3 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys dnl Allow some time for ovn-northd and ovn-controller to catch up. wait_for_ports_up ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-outside"], ["hv3"]) (echo "---------NB dump-----" ovn-nbctl show @@ -11733,6 +11743,7 @@ ovn-nbctl lsp-add foo foo2 \ # Allow some time for ovn-northd and ovn-controller to catch up. wait_for_ports_up check ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln-alice"], ["hv1"]) : > hv1-vif2.expected @@ -11831,8 +11842,7 @@ OVN_CHECK_PACKETS([hv1/snoopvif-tx.pcap], [packets]) AT_CHECK([as hv2 ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=physnet1:br-phys]) # Wait until the patch ports are created in hv2 to connect br-int to br-phys -OVS_WAIT_UNTIL([test 1 = `as hv2 ovs-vsctl show | \ -grep "Port patch-br-int-to-ln_port" | wc -l`]) +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln_port"], ["hv2"]) # Wait for packets to be received. OVS_WAIT_UNTIL([test `wc -c < "hv1/snoopvif-tx.pcap"` -ge 100]) @@ -13550,6 +13560,7 @@ AT_CHECK([ovn-nbctl lsp-set-options ln_port network_name=physnet1]) # wait for earlier changes to take effect wait_for_ports_up check ovn-nbctl --wait=hv sync +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln_port"], ["hv2"] ["hv3"]) reset_pcap_file() { local iface=$1 @@ -14762,6 +14773,13 @@ reset_env # Start port migration hv1 -> hv2: both hypervisors are now bound check ovn-nbctl lsp-set-options migrator requested-chassis=hv1,hv2 wait_for_ports_up +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-public"], ["hv1"]) +OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-public"], ["hv2"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv1"],["hv2"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv2"],["hv1"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv3"],[hv1"]) +OVN_WAIT_REMOTE_OUTPUT_FLOWS(["hv3"],[hv2"]) + wait_column "$hv1_uuid" Port_Binding chassis logical_port=migrator wait_column "$hv1_uuid" Port_Binding requested_chassis logical_port=migrator wait_column "$hv2_uuid" Port_Binding additional_chassis logical_port=migrator @@ -20079,6 +20097,11 @@ ovn-nbctl lsp-add ls2 ls2-to-router -- set Logical_Switch_Port ls2-to-router typ wait_for_ports_up ovn-nbctl --wait=sb sync #ovn-sbctl dump-flows +for i in 1 2; do + for j in 1 2; do + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln$j"], ["hv$i"]) + done +done ovn-nbctl show ovn-sbctl show @@ -22484,6 +22507,8 @@ m4_define([DVR_N_S_ARP_HANDLING], ovn-nbctl lrp-set-gateway-chassis router-to-underlay hv3 wait_for_ports_up ovn-nbctl --wait=sb sync + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln1"] ["patch-br-int-to-ln2"], ["hv1"] ["hv2"] ["hv3"]) + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln4"], ["hv4"]) wait_row_count Port_Binding 1 logical_port=cr-router-to-underlay @@ -22704,7 +22729,8 @@ m4_define([DVR_N_S_PING], wait_for_ports_up ovn-nbctl --wait=sb sync - + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln1"] ["patch-br-int-to-ln2"], ["hv1"] ["hv2"] ["hv3"]) + OVN_WAIT_PATCH_PORT_FLOWS(["patch-br-int-to-ln4"], ["hv4"]) OVN_POPULATE_ARP