| Message ID | 20231023100404.3861299-1-xsimonar@redhat.com |
|---|---|
| State | Accepted |
| Delegated to: | Dumitru Ceara |
| Headers | show |
| Series | [ovs-dev] tests: fixed another set of flaky ovn-ic tests | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | success | github build: passed |
| ovsrobot/github-robot-_ovn-kubernetes | fail | github build: failed |
On Mon, Oct 23, 2023 at 12:05 PM Xavier Simonart <xsimonar@redhat.com> wrote: > Some ic tests were failing as not waiting, or not waiting properly > for some information to be available in the az: > - Adding a port to ts1 before ts1 being created in that az. > - Setting options for ports before the port got created in the az. > - Cleaning up an hv/gw and expecting that it would be immediately > visible in different az. > - Expecting routes updates between az to happen immediately. > > Reported-at: https://issues.redhat.com/browse/FDP-98 > Signed-off-by: Xavier Simonart <xsimonar@redhat.com> > --- > tests/ovn-ic.at | 44 ++++++++++++++++++++++++++------------------ > 1 file changed, 26 insertions(+), 18 deletions(-) > > diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at > index 2ae8dd12d..d4c436f84 100644 > --- a/tests/ovn-ic.at > +++ b/tests/ovn-ic.at > @@ -92,6 +92,7 @@ check ovn-nbctl lr-add lr1 > check ovn-nbctl lrp-add lr1 lrp1 00:00:00:00:00:01 10.0.0.1/24 > check ovn-nbctl lrp-set-gateway-chassis lrp1 gw-az1 > > +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > check ovn-nbctl lsp-add ts1 lsp1 -- \ > lsp-set-addresses lsp1 router -- \ > lsp-set-type lsp1 router -- \ > @@ -156,6 +157,7 @@ create_ic_infra() { > ovn_as $az > > check ovn-ic-nbctl ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > check ovn-nbctl lr-add $lr > check ovn-nbctl lrp-add $lr $lrp 00:00:00:00:00:0$az_id > 10.0.$az_id.1/24 > check ovn-nbctl lrp-set-gateway-chassis $lrp gw-$az > @@ -227,6 +229,7 @@ for i in 1 2; do > check ovn-nbctl lrp-add lr1 lrp$i 00:00:00:00:0$i:01 10.0.$i.1/24 > check ovn-nbctl lrp-set-gateway-chassis lrp$i gw-az$i > > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > check ovn-nbctl lsp-add ts1 lsp$i -- \ > lsp-set-addresses lsp$i router -- \ > lsp-set-type lsp$i router -- \ > @@ -290,7 +293,7 @@ ovs-vsctl set open . external-ids:ovn-is-interconn=true > OVS_WAIT_UNTIL([ovn_as az2 ovn-sbctl show | grep gw1]) > > OVN_CLEANUP_SBOX(gw1) > -AT_CHECK([ovn_as az2 ovn-sbctl show], [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl show], [0], [dnl > ]) > > # Test encap change > @@ -335,17 +338,17 @@ ovn-nbctl lsp-add ts1 lsp-ts1-lr1 > ovn-nbctl lsp-set-addresses lsp-ts1-lr1 router > ovn-nbctl lsp-set-type lsp-ts1-lr1 router > ovn-nbctl --wait=hv lsp-set-options lsp-ts1-lr1 router-port=lrp-lr1-ts1 > +OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep lsp-ts1-lr1]) > ovn_as az2 ovn-nbctl lsp-set-options lsp-ts1-lr1 requested-chassis=gw1 > > -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep "aa:aa:aa:aa:aa:01 > 169.254.100.1/24"]) > -AT_CHECK([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl > switch <0> (ts1) > port lsp-ts1-lr1 > type: remote > addresses: [["aa:aa:aa:aa:aa:01 169.254.100.1/24"]] > ]) > > -AT_CHECK([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings --columns > logical_port,type list port_binding], [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings > --columns logical_port,type list port_binding], [0], [dnl > lsp-ts1-lr1,remote > ]) > > @@ -530,6 +533,7 @@ for i in 1 2; do > for j in 1 2; do > ts=ts$j$j > ovn-ic-nbctl --may-exist ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > > # Create LRP and connect to TS > lr=lr$j$i > @@ -588,6 +592,7 @@ for i in 1 2; do > # Create LRP and connect to TS > ovn-nbctl lr-add lr$i > ovn-nbctl lrp-add lr$i lrp-lr$i-ts1 aa:aa:aa:aa:aa:0$i > 169.254.100.$i/24 > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > ovn-nbctl lsp-add ts1 lsp-ts1-lr$i \ > -- lsp-set-addresses lsp-ts1-lr$i router \ > -- lsp-set-type lsp-ts1-lr$i router \ > @@ -913,6 +918,7 @@ for i in 1 2; do > for j in 1 2 3; do > ts=ts1$j > ovn-ic-nbctl --may-exist ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > > lrp=lrp-$lr-$ts > lsp=lsp-$ts-$lr > @@ -938,6 +944,7 @@ for i in 1 2; do > for j in 1 2; do > ts=ts2$j > ovn-ic-nbctl --may-exist ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > > lrp=lrp-$lr-$ts > lsp=lsp-$ts-$lr > @@ -961,7 +968,7 @@ ovn_as az2 ovn-nbctl --route-table=rtb3 lr-route-add > lr12 10.10.10.0/24 192.168. > ovn_as az2 ovn-nbctl --wait=sb lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bb:01 " > 192.168.0.1/24" > > # Test direct routes from lr12 were learned to lr11 > -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep > 192.168 | > grep learned | awk '{print $1, $2, $5}' | sort ], [0], [dnl > 192.168.0.0/24 169.254.101.2 ecmp > 192.168.0.0/24 169.254.102.2 ecmp > @@ -969,24 +976,24 @@ AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | > grep 192.168 | > ]) > > # Test static routes from lr12 rtbs rtb1,rtb2,rtb3 were learned to lr11 > -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb1 lr-route-list lr11], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb1 > lr-route-list lr11], [0], [dnl > IPv4 Routes > Route Table rtb1: > 10.10.10.0/24 169.254.101.2 dst-ip (learned) > ]) > -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb2 lr-route-list lr11], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb2 > lr-route-list lr11], [0], [dnl > IPv4 Routes > Route Table rtb2: > 10.10.10.0/24 169.254.102.2 dst-ip (learned) > ]) > -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb3 lr-route-list lr11], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb3 > lr-route-list lr11], [0], [dnl > IPv4 Routes > Route Table rtb3: > 10.10.10.0/24 169.254.103.2 dst-ip (learned) > ]) > > # Test routes from lr12 didn't leak as learned to lr21 > -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 192.168 | sort], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep > 192.168 | sort], [0], [dnl > 192.168.0.0/24 169.254.101.2 dst-ip (learned) ecmp > 192.168.0.0/24 169.254.102.2 dst-ip (learned) ecmp > ]) > @@ -1039,6 +1046,7 @@ for i in 1 2; do > for j in 1 2 3; do > ts=ts1$j > ovn-ic-nbctl --may-exist ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > > lrp=lrp-$lr-$ts > lsp=lsp-$ts-$lr > @@ -1064,6 +1072,7 @@ for i in 1 2; do > for j in 1 2; do > ts=ts2$j > ovn-ic-nbctl --may-exist ts-add $ts > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) > > lrp=lrp-$lr-$ts > lsp=lsp-$ts-$lr > @@ -1116,6 +1125,7 @@ Route Table rtb3: > ]) > > # Test routes from lr12 didn't leak as learned to lr21 > +OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep > "2001:db8:2::2" | grep learned]) > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 2001 | sort], > [0], [dnl > 2001:db8:200::/64 2001:db8:1::2 dst-ip (learned) ecmp > 2001:db8:200::/64 2001:db8:2::2 dst-ip (learned) ecmp > @@ -1135,6 +1145,7 @@ ovn-ic-nbctl ts-add ts1 > for i in 1 2; do > ovn_start az$i > ovn_as az$i > + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > > # Enable route learning at AZ level > ovn-nbctl set nb_global . options:ic-route-learn=true > @@ -1160,17 +1171,13 @@ for i in 1 2; do > check ovn-nbctl --wait=sb lr-route-add $lr 0.0.0.0/0 192.168.$i.11 > done > > -OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep learned]) > - > -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep dst-ip | sort], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep > dst-ip | sort] , [0], [dnl > 0.0.0.0/0 192.168.1.11 dst-ip > 10.0.0.0/24 192.168.1.10 dst-ip > 192.168.2.0/24 169.254.100.2 dst-ip (learned) > ]) > > -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep learned]) > - > -AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep dst-ip | sort], > [0], [dnl > +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep > dst-ip | sort], [0], [dnl > 0.0.0.0/0 192.168.2.11 dst-ip > 10.0.0.0/24 192.168.2.10 dst-ip > 192.168.1.0/24 169.254.100.1 dst-ip (learned) > @@ -1203,10 +1210,10 @@ done > # each LR has one connected subnet except TS port > > > -# create lr11, lr21, lr22, ts1 and connect them > -ovn-ic-nbctl ts-add ts1 > +# create lr11, lr21, lr22 and connect them > > ovn_as az1 > +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > > lr=lr11 > ovn-nbctl lr-add $lr > @@ -1221,6 +1228,7 @@ ovn-nbctl lsp-add ts1 $lsp \ > -- lsp-set-options $lsp router-port=$lrp > > ovn_as az2 > +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) > for i in 1 2; do > lr=lr2$i > ovn-nbctl lr-add $lr > @@ -1242,7 +1250,7 @@ ovn_as az2 ovn-nbctl lrp-add lr21 lrp-lr21 > aa:aa:aa:aa:bc:01 "192.168.1.1/24" > ovn_as az2 ovn-nbctl lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bc:02 " > 192.168.2.1/24" > > # Test direct routes from lr21 and lr22 were learned to lr11 > -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | > +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep > 192.168 | > grep learned | awk '{print $1, $2}' | sort ], [0], [dnl > 192.168.1.0/24 169.254.10.21 > 192.168.2.0/24 169.254.10.22 > -- > 2.31.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > Looks good to me, thanks. Acked-by: Ales Musil <amusil@redhat.com>
On 11/3/23 10:26, Ales Musil wrote: > On Mon, Oct 23, 2023 at 12:05 PM Xavier Simonart <xsimonar@redhat.com> > wrote: > >> Some ic tests were failing as not waiting, or not waiting properly >> for some information to be available in the az: >> - Adding a port to ts1 before ts1 being created in that az. >> - Setting options for ports before the port got created in the az. >> - Cleaning up an hv/gw and expecting that it would be immediately >> visible in different az. >> - Expecting routes updates between az to happen immediately. >> >> Reported-at: https://issues.redhat.com/browse/FDP-98 >> Signed-off-by: Xavier Simonart <xsimonar@redhat.com> >> --- >> tests/ovn-ic.at | 44 ++++++++++++++++++++++++++------------------ >> 1 file changed, 26 insertions(+), 18 deletions(-) >> >> diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at >> index 2ae8dd12d..d4c436f84 100644 >> --- a/tests/ovn-ic.at >> +++ b/tests/ovn-ic.at >> @@ -92,6 +92,7 @@ check ovn-nbctl lr-add lr1 >> check ovn-nbctl lrp-add lr1 lrp1 00:00:00:00:00:01 10.0.0.1/24 >> check ovn-nbctl lrp-set-gateway-chassis lrp1 gw-az1 >> >> +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> check ovn-nbctl lsp-add ts1 lsp1 -- \ >> lsp-set-addresses lsp1 router -- \ >> lsp-set-type lsp1 router -- \ >> @@ -156,6 +157,7 @@ create_ic_infra() { >> ovn_as $az >> >> check ovn-ic-nbctl ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> check ovn-nbctl lr-add $lr >> check ovn-nbctl lrp-add $lr $lrp 00:00:00:00:00:0$az_id >> 10.0.$az_id.1/24 >> check ovn-nbctl lrp-set-gateway-chassis $lrp gw-$az >> @@ -227,6 +229,7 @@ for i in 1 2; do >> check ovn-nbctl lrp-add lr1 lrp$i 00:00:00:00:0$i:01 10.0.$i.1/24 >> check ovn-nbctl lrp-set-gateway-chassis lrp$i gw-az$i >> >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> check ovn-nbctl lsp-add ts1 lsp$i -- \ >> lsp-set-addresses lsp$i router -- \ >> lsp-set-type lsp$i router -- \ >> @@ -290,7 +293,7 @@ ovs-vsctl set open . external-ids:ovn-is-interconn=true >> OVS_WAIT_UNTIL([ovn_as az2 ovn-sbctl show | grep gw1]) >> >> OVN_CLEANUP_SBOX(gw1) >> -AT_CHECK([ovn_as az2 ovn-sbctl show], [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl show], [0], [dnl >> ]) >> >> # Test encap change >> @@ -335,17 +338,17 @@ ovn-nbctl lsp-add ts1 lsp-ts1-lr1 >> ovn-nbctl lsp-set-addresses lsp-ts1-lr1 router >> ovn-nbctl lsp-set-type lsp-ts1-lr1 router >> ovn-nbctl --wait=hv lsp-set-options lsp-ts1-lr1 router-port=lrp-lr1-ts1 >> +OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep lsp-ts1-lr1]) >> ovn_as az2 ovn-nbctl lsp-set-options lsp-ts1-lr1 requested-chassis=gw1 >> >> -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep "aa:aa:aa:aa:aa:01 >> 169.254.100.1/24"]) >> -AT_CHECK([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl >> switch <0> (ts1) >> port lsp-ts1-lr1 >> type: remote >> addresses: [["aa:aa:aa:aa:aa:01 169.254.100.1/24"]] >> ]) >> >> -AT_CHECK([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings --columns >> logical_port,type list port_binding], [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings >> --columns logical_port,type list port_binding], [0], [dnl >> lsp-ts1-lr1,remote >> ]) >> >> @@ -530,6 +533,7 @@ for i in 1 2; do >> for j in 1 2; do >> ts=ts$j$j >> ovn-ic-nbctl --may-exist ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> >> # Create LRP and connect to TS >> lr=lr$j$i >> @@ -588,6 +592,7 @@ for i in 1 2; do >> # Create LRP and connect to TS >> ovn-nbctl lr-add lr$i >> ovn-nbctl lrp-add lr$i lrp-lr$i-ts1 aa:aa:aa:aa:aa:0$i >> 169.254.100.$i/24 >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> ovn-nbctl lsp-add ts1 lsp-ts1-lr$i \ >> -- lsp-set-addresses lsp-ts1-lr$i router \ >> -- lsp-set-type lsp-ts1-lr$i router \ >> @@ -913,6 +918,7 @@ for i in 1 2; do >> for j in 1 2 3; do >> ts=ts1$j >> ovn-ic-nbctl --may-exist ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> >> lrp=lrp-$lr-$ts >> lsp=lsp-$ts-$lr >> @@ -938,6 +944,7 @@ for i in 1 2; do >> for j in 1 2; do >> ts=ts2$j >> ovn-ic-nbctl --may-exist ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> >> lrp=lrp-$lr-$ts >> lsp=lsp-$ts-$lr >> @@ -961,7 +968,7 @@ ovn_as az2 ovn-nbctl --route-table=rtb3 lr-route-add >> lr12 10.10.10.0/24 192.168. >> ovn_as az2 ovn-nbctl --wait=sb lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bb:01 " >> 192.168.0.1/24" >> >> # Test direct routes from lr12 were learned to lr11 >> -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep >> 192.168 | >> grep learned | awk '{print $1, $2, $5}' | sort ], [0], [dnl >> 192.168.0.0/24 169.254.101.2 ecmp >> 192.168.0.0/24 169.254.102.2 ecmp >> @@ -969,24 +976,24 @@ AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | >> grep 192.168 | >> ]) >> >> # Test static routes from lr12 rtbs rtb1,rtb2,rtb3 were learned to lr11 >> -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb1 lr-route-list lr11], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb1 >> lr-route-list lr11], [0], [dnl >> IPv4 Routes >> Route Table rtb1: >> 10.10.10.0/24 169.254.101.2 dst-ip (learned) >> ]) >> -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb2 lr-route-list lr11], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb2 >> lr-route-list lr11], [0], [dnl >> IPv4 Routes >> Route Table rtb2: >> 10.10.10.0/24 169.254.102.2 dst-ip (learned) >> ]) >> -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb3 lr-route-list lr11], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb3 >> lr-route-list lr11], [0], [dnl >> IPv4 Routes >> Route Table rtb3: >> 10.10.10.0/24 169.254.103.2 dst-ip (learned) >> ]) >> >> # Test routes from lr12 didn't leak as learned to lr21 >> -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 192.168 | sort], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep >> 192.168 | sort], [0], [dnl >> 192.168.0.0/24 169.254.101.2 dst-ip (learned) ecmp >> 192.168.0.0/24 169.254.102.2 dst-ip (learned) ecmp >> ]) >> @@ -1039,6 +1046,7 @@ for i in 1 2; do >> for j in 1 2 3; do >> ts=ts1$j >> ovn-ic-nbctl --may-exist ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> >> lrp=lrp-$lr-$ts >> lsp=lsp-$ts-$lr >> @@ -1064,6 +1072,7 @@ for i in 1 2; do >> for j in 1 2; do >> ts=ts2$j >> ovn-ic-nbctl --may-exist ts-add $ts >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) >> >> lrp=lrp-$lr-$ts >> lsp=lsp-$ts-$lr >> @@ -1116,6 +1125,7 @@ Route Table rtb3: >> ]) >> >> # Test routes from lr12 didn't leak as learned to lr21 >> +OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep >> "2001:db8:2::2" | grep learned]) >> AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 2001 | sort], >> [0], [dnl >> 2001:db8:200::/64 2001:db8:1::2 dst-ip (learned) ecmp >> 2001:db8:200::/64 2001:db8:2::2 dst-ip (learned) ecmp >> @@ -1135,6 +1145,7 @@ ovn-ic-nbctl ts-add ts1 >> for i in 1 2; do >> ovn_start az$i >> ovn_as az$i >> + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> >> # Enable route learning at AZ level >> ovn-nbctl set nb_global . options:ic-route-learn=true >> @@ -1160,17 +1171,13 @@ for i in 1 2; do >> check ovn-nbctl --wait=sb lr-route-add $lr 0.0.0.0/0 192.168.$i.11 >> done >> >> -OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep learned]) >> - >> -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep dst-ip | sort], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep >> dst-ip | sort] , [0], [dnl >> 0.0.0.0/0 192.168.1.11 dst-ip >> 10.0.0.0/24 192.168.1.10 dst-ip >> 192.168.2.0/24 169.254.100.2 dst-ip (learned) >> ]) >> >> -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep learned]) >> - >> -AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep dst-ip | sort], >> [0], [dnl >> +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep >> dst-ip | sort], [0], [dnl >> 0.0.0.0/0 192.168.2.11 dst-ip >> 10.0.0.0/24 192.168.2.10 dst-ip >> 192.168.1.0/24 169.254.100.1 dst-ip (learned) >> @@ -1203,10 +1210,10 @@ done >> # each LR has one connected subnet except TS port >> >> >> -# create lr11, lr21, lr22, ts1 and connect them >> -ovn-ic-nbctl ts-add ts1 >> +# create lr11, lr21, lr22 and connect them >> >> ovn_as az1 >> +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> >> lr=lr11 >> ovn-nbctl lr-add $lr >> @@ -1221,6 +1228,7 @@ ovn-nbctl lsp-add ts1 $lsp \ >> -- lsp-set-options $lsp router-port=$lrp >> >> ovn_as az2 >> +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) >> for i in 1 2; do >> lr=lr2$i >> ovn-nbctl lr-add $lr >> @@ -1242,7 +1250,7 @@ ovn_as az2 ovn-nbctl lrp-add lr21 lrp-lr21 >> aa:aa:aa:aa:bc:01 "192.168.1.1/24" >> ovn_as az2 ovn-nbctl lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bc:02 " >> 192.168.2.1/24" >> >> # Test direct routes from lr21 and lr22 were learned to lr11 >> -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | >> +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep >> 192.168 | >> grep learned | awk '{print $1, $2}' | sort ], [0], [dnl >> 192.168.1.0/24 169.254.10.21 >> 192.168.2.0/24 169.254.10.22 >> -- >> 2.31.1 >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> >> > > Looks good to me, thanks. > > Acked-by: Ales Musil <amusil@redhat.com> > Thanks, Xavier and Ales! Applied to main and backported down to 22.03. Regards, Dumitru
diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at index 2ae8dd12d..d4c436f84 100644 --- a/tests/ovn-ic.at +++ b/tests/ovn-ic.at @@ -92,6 +92,7 @@ check ovn-nbctl lr-add lr1 check ovn-nbctl lrp-add lr1 lrp1 00:00:00:00:00:01 10.0.0.1/24 check ovn-nbctl lrp-set-gateway-chassis lrp1 gw-az1 +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) check ovn-nbctl lsp-add ts1 lsp1 -- \ lsp-set-addresses lsp1 router -- \ lsp-set-type lsp1 router -- \ @@ -156,6 +157,7 @@ create_ic_infra() { ovn_as $az check ovn-ic-nbctl ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) check ovn-nbctl lr-add $lr check ovn-nbctl lrp-add $lr $lrp 00:00:00:00:00:0$az_id 10.0.$az_id.1/24 check ovn-nbctl lrp-set-gateway-chassis $lrp gw-$az @@ -227,6 +229,7 @@ for i in 1 2; do check ovn-nbctl lrp-add lr1 lrp$i 00:00:00:00:0$i:01 10.0.$i.1/24 check ovn-nbctl lrp-set-gateway-chassis lrp$i gw-az$i + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) check ovn-nbctl lsp-add ts1 lsp$i -- \ lsp-set-addresses lsp$i router -- \ lsp-set-type lsp$i router -- \ @@ -290,7 +293,7 @@ ovs-vsctl set open . external-ids:ovn-is-interconn=true OVS_WAIT_UNTIL([ovn_as az2 ovn-sbctl show | grep gw1]) OVN_CLEANUP_SBOX(gw1) -AT_CHECK([ovn_as az2 ovn-sbctl show], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl show], [0], [dnl ]) # Test encap change @@ -335,17 +338,17 @@ ovn-nbctl lsp-add ts1 lsp-ts1-lr1 ovn-nbctl lsp-set-addresses lsp-ts1-lr1 router ovn-nbctl lsp-set-type lsp-ts1-lr1 router ovn-nbctl --wait=hv lsp-set-options lsp-ts1-lr1 router-port=lrp-lr1-ts1 +OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep lsp-ts1-lr1]) ovn_as az2 ovn-nbctl lsp-set-options lsp-ts1-lr1 requested-chassis=gw1 -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl show | grep "aa:aa:aa:aa:aa:01 169.254.100.1/24"]) -AT_CHECK([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl show | uuidfilt], [0], [dnl switch <0> (ts1) port lsp-ts1-lr1 type: remote addresses: [["aa:aa:aa:aa:aa:01 169.254.100.1/24"]] ]) -AT_CHECK([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings --columns logical_port,type list port_binding], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-sbctl -f csv -d bare --no-headings --columns logical_port,type list port_binding], [0], [dnl lsp-ts1-lr1,remote ]) @@ -530,6 +533,7 @@ for i in 1 2; do for j in 1 2; do ts=ts$j$j ovn-ic-nbctl --may-exist ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) # Create LRP and connect to TS lr=lr$j$i @@ -588,6 +592,7 @@ for i in 1 2; do # Create LRP and connect to TS ovn-nbctl lr-add lr$i ovn-nbctl lrp-add lr$i lrp-lr$i-ts1 aa:aa:aa:aa:aa:0$i 169.254.100.$i/24 + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) ovn-nbctl lsp-add ts1 lsp-ts1-lr$i \ -- lsp-set-addresses lsp-ts1-lr$i router \ -- lsp-set-type lsp-ts1-lr$i router \ @@ -913,6 +918,7 @@ for i in 1 2; do for j in 1 2 3; do ts=ts1$j ovn-ic-nbctl --may-exist ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) lrp=lrp-$lr-$ts lsp=lsp-$ts-$lr @@ -938,6 +944,7 @@ for i in 1 2; do for j in 1 2; do ts=ts2$j ovn-ic-nbctl --may-exist ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) lrp=lrp-$lr-$ts lsp=lsp-$ts-$lr @@ -961,7 +968,7 @@ ovn_as az2 ovn-nbctl --route-table=rtb3 lr-route-add lr12 10.10.10.0/24 192.168. ovn_as az2 ovn-nbctl --wait=sb lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bb:01 "192.168.0.1/24" # Test direct routes from lr12 were learned to lr11 -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | grep learned | awk '{print $1, $2, $5}' | sort ], [0], [dnl 192.168.0.0/24 169.254.101.2 ecmp 192.168.0.0/24 169.254.102.2 ecmp @@ -969,24 +976,24 @@ AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | ]) # Test static routes from lr12 rtbs rtb1,rtb2,rtb3 were learned to lr11 -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb1 lr-route-list lr11], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb1 lr-route-list lr11], [0], [dnl IPv4 Routes Route Table rtb1: 10.10.10.0/24 169.254.101.2 dst-ip (learned) ]) -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb2 lr-route-list lr11], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb2 lr-route-list lr11], [0], [dnl IPv4 Routes Route Table rtb2: 10.10.10.0/24 169.254.102.2 dst-ip (learned) ]) -AT_CHECK([ovn_as az1 ovn-nbctl --route-table=rtb3 lr-route-list lr11], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl --route-table=rtb3 lr-route-list lr11], [0], [dnl IPv4 Routes Route Table rtb3: 10.10.10.0/24 169.254.103.2 dst-ip (learned) ]) # Test routes from lr12 didn't leak as learned to lr21 -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 192.168 | sort], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 192.168 | sort], [0], [dnl 192.168.0.0/24 169.254.101.2 dst-ip (learned) ecmp 192.168.0.0/24 169.254.102.2 dst-ip (learned) ecmp ]) @@ -1039,6 +1046,7 @@ for i in 1 2; do for j in 1 2 3; do ts=ts1$j ovn-ic-nbctl --may-exist ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) lrp=lrp-$lr-$ts lsp=lsp-$ts-$lr @@ -1064,6 +1072,7 @@ for i in 1 2; do for j in 1 2; do ts=ts2$j ovn-ic-nbctl --may-exist ts-add $ts + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep $ts]) lrp=lrp-$lr-$ts lsp=lsp-$ts-$lr @@ -1116,6 +1125,7 @@ Route Table rtb3: ]) # Test routes from lr12 didn't leak as learned to lr21 +OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep "2001:db8:2::2" | grep learned]) AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr21 | grep 2001 | sort], [0], [dnl 2001:db8:200::/64 2001:db8:1::2 dst-ip (learned) ecmp 2001:db8:200::/64 2001:db8:2::2 dst-ip (learned) ecmp @@ -1135,6 +1145,7 @@ ovn-ic-nbctl ts-add ts1 for i in 1 2; do ovn_start az$i ovn_as az$i + OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) # Enable route learning at AZ level ovn-nbctl set nb_global . options:ic-route-learn=true @@ -1160,17 +1171,13 @@ for i in 1 2; do check ovn-nbctl --wait=sb lr-route-add $lr 0.0.0.0/0 192.168.$i.11 done -OVS_WAIT_UNTIL([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep learned]) - -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep dst-ip | sort], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep dst-ip | sort] , [0], [dnl 0.0.0.0/0 192.168.1.11 dst-ip 10.0.0.0/24 192.168.1.10 dst-ip 192.168.2.0/24 169.254.100.2 dst-ip (learned) ]) -OVS_WAIT_UNTIL([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep learned]) - -AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep dst-ip | sort], [0], [dnl +OVS_WAIT_FOR_OUTPUT([ovn_as az2 ovn-nbctl lr-route-list lr12 | grep dst-ip | sort], [0], [dnl 0.0.0.0/0 192.168.2.11 dst-ip 10.0.0.0/24 192.168.2.10 dst-ip 192.168.1.0/24 169.254.100.1 dst-ip (learned) @@ -1203,10 +1210,10 @@ done # each LR has one connected subnet except TS port -# create lr11, lr21, lr22, ts1 and connect them -ovn-ic-nbctl ts-add ts1 +# create lr11, lr21, lr22 and connect them ovn_as az1 +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) lr=lr11 ovn-nbctl lr-add $lr @@ -1221,6 +1228,7 @@ ovn-nbctl lsp-add ts1 $lsp \ -- lsp-set-options $lsp router-port=$lrp ovn_as az2 +OVS_WAIT_UNTIL([ovn-nbctl show | grep switch | grep ts1]) for i in 1 2; do lr=lr2$i ovn-nbctl lr-add $lr @@ -1242,7 +1250,7 @@ ovn_as az2 ovn-nbctl lrp-add lr21 lrp-lr21 aa:aa:aa:aa:bc:01 "192.168.1.1/24" ovn_as az2 ovn-nbctl lrp-add lr22 lrp-lr22 aa:aa:aa:aa:bc:02 "192.168.2.1/24" # Test direct routes from lr21 and lr22 were learned to lr11 -AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | +OVS_WAIT_FOR_OUTPUT([ovn_as az1 ovn-nbctl lr-route-list lr11 | grep 192.168 | grep learned | awk '{print $1, $2}' | sort ], [0], [dnl 192.168.1.0/24 169.254.10.21 192.168.2.0/24 169.254.10.22
Some ic tests were failing as not waiting, or not waiting properly for some information to be available in the az: - Adding a port to ts1 before ts1 being created in that az. - Setting options for ports before the port got created in the az. - Cleaning up an hv/gw and expecting that it would be immediately visible in different az. - Expecting routes updates between az to happen immediately. Reported-at: https://issues.redhat.com/browse/FDP-98 Signed-off-by: Xavier Simonart <xsimonar@redhat.com> --- tests/ovn-ic.at | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-)