| Message ID | 1607681529-2217-1-git-send-email-dceara@redhat.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [ovs-dev] lflow: Relax the condition that detects Load_Balancer hairpin. | expand |
On Fri, Dec 11, 2020 at 3:46 PM Dumitru Ceara <dceara@redhat.com> wrote: > > Commit fc219d84b667 [0] optimized Load Balancer hairpin support from an > OVN Southbound perspective by moving generation of flows to detect > hairpin traffic in ovn-controller. > > However, this can be further optimized by relaxing the condition for > flows that detect hairpin traffic in the "original" direction. > > It is safe to exclude the match on datapath.tunnel_key in the original > direction if we make sure that we only match on traffic that was > successfully load balanced (i.e., ct.natted = 1). At this point it > would be enough to check that ip.src == ip.dst in order to determine > that traffic needs to be hairpinned. Because OVS doesn't allow such > checks between fields we instead check against the known load balancer > backend IPs: ip.src == backend-ip && ip.dst == backend-ip. > > This change improves scalability by reducing the number of hairpin OF > flows by ~50%. For example, on a setup with: > - N Load Balancer VIPs. > - M Backends per VIP. > - All N VIPs applied to S logical switches. > > Before this change the number of hairpin OF flows was: > - N x M x S (for original direction) + N x M x S (for reply direction) > > With this change the number of hairpin OF flows is: > - N x M (for original direction ) + N x M x S (for reply direction) > > That means that on a setup with N=100 VIPs, M=10 backends, S=100 > switches the number of hairpin OF flows will be 101K vs 200K before this > change. > > [0] fc219d84b667 ("actions: Add new actions chk_lb_hairpin, chk_lb_hairpin_reply and ct_snat_to_vip.") > > Signed-off-by: Dumitru Ceara <dceara@redhat.com> Thanks Dumitru for addressing this scale concern and reducing the OF flows by half. Could we also do the same even for OFTABLE_CHK_LB_HAIRPIN_REPLY ? Did you explore that too ? I applied this patch to master and also to branch-20.12 as it seems an important fix. There was one nit which I fixed before applying --- diff --git a/controller/lflow.c b/controller/lflow.c index f63a6f56a5..c02585b1eb 100644 --- a/controller/lflow.c +++ b/controller/lflow.c @@ -1226,7 +1226,7 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb, * to not include the datapath tunnel_key in the match when determining * that a packet needs to be hairpinned because the rest of the match is * restrictive enough: - * - traffic must have already been load balancedu + * - traffic must have already been load balanced. * - packets must have ip.src == ip.dst at this point. * - the destination protocol and port must be of a valid backend that * has the same IP as ip.dst. Thanks Numan > --- > controller/lflow.c | 25 ++++++--- > include/ovn/logical-fields.h | 20 ++++++++ > lib/actions.c | 4 +- > lib/logical-fields.c | 10 +++- > tests/ovn.at | 117 +++++++++++++++++++++---------------------- > 5 files changed, 108 insertions(+), 68 deletions(-) > > diff --git a/controller/lflow.c b/controller/lflow.c > index fc58a16..f63a6f5 100644 > --- a/controller/lflow.c > +++ b/controller/lflow.c > @@ -1221,16 +1221,29 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb, > match_set_tp_src(&hairpin_reply_match, htons(lb_backend->port)); > } > > + /* In the original direction, only match on traffic that was already > + * load balanced, i.e., "ct.natted == 1". Also, it's good enough > + * to not include the datapath tunnel_key in the match when determining > + * that a packet needs to be hairpinned because the rest of the match is > + * restrictive enough: > + * - traffic must have already been load balancedu > + * - packets must have ip.src == ip.dst at this point. > + * - the destination protocol and port must be of a valid backend that > + * has the same IP as ip.dst. > + */ > + ovs_u128 lb_ct_label = { > + .u64.lo = OVN_CT_NATTED, > + }; > + match_set_ct_label_masked(&hairpin_match, lb_ct_label, lb_ct_label); > + > + ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN, 100, > + lb->slb->header_.uuid.parts[0], &hairpin_match, > + &ofpacts, &lb->slb->header_.uuid); > + > for (size_t i = 0; i < lb->slb->n_datapaths; i++) { > - match_set_metadata(&hairpin_match, > - htonll(lb->slb->datapaths[i]->tunnel_key)); > match_set_metadata(&hairpin_reply_match, > htonll(lb->slb->datapaths[i]->tunnel_key)); > > - ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN, 100, > - lb->slb->header_.uuid.parts[0], &hairpin_match, > - &ofpacts, &lb->slb->header_.uuid); > - > ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN_REPLY, 100, > lb->slb->header_.uuid.parts[0], > &hairpin_reply_match, > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h > index 0fe5bc3..aee4748 100644 > --- a/include/ovn/logical-fields.h > +++ b/include/ovn/logical-fields.h > @@ -17,6 +17,7 @@ > #define OVN_LOGICAL_FIELDS_H 1 > > #include "openvswitch/meta-flow.h" > +#include "openvswitch/util.h" > > struct shash; > > @@ -140,4 +141,23 @@ ovn_field_from_id(enum ovn_field_id id) > const char *event_to_string(enum ovn_controller_event event); > int string_to_event(const char *s); > const struct ovn_field *ovn_field_from_name(const char *name); > + > +/* OVN CT label values > + * =================== > + * These are specific ct.label bit values OVN uses to track different types > + * of traffic. > + */ > + > +#define OVN_CT_BLOCKED_BIT 0 > +#define OVN_CT_NATTED_BIT 1 > + > +#define OVN_CT_BLOCKED 1 > +#define OVN_CT_NATTED 2 > + > +#define OVN_CT_STR(LABEL_VALUE) OVS_STRINGIZE(LABEL_VALUE) > +#define OVN_CT_MASKED_STR(LABEL_VALUE) \ > + OVS_STRINGIZE(LABEL_VALUE) "/" OVS_STRINGIZE(LABEL_VALUE) > + > +#define OVN_CT_LABEL_STR(LABEL_VALUE) "ct_label[" OVN_CT_STR(LABEL_VALUE) "]" > + > #endif /* ovn/lib/logical-fields.h */ > diff --git a/lib/actions.c b/lib/actions.c > index 0705e4e..fbaeb34 100644 > --- a/lib/actions.c > +++ b/lib/actions.c > @@ -1224,7 +1224,9 @@ encode_CT_LB(const struct ovnact_ct_lb *cl, > ds_put_format(&ds, ":%"PRIu16, dst->port); > } > ds_put_format(&ds, "),commit,table=%d,zone=NXM_NX_REG%d[0..15]," > - "exec(set_field:2/2->ct_label))", > + "exec(set_field:" > + OVN_CT_MASKED_STR(OVN_CT_NATTED) > + "->ct_label))", > recirc_table, zone_reg); > } > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c > index bf61df7..9d08b44 100644 > --- a/lib/logical-fields.c > +++ b/lib/logical-fields.c > @@ -129,9 +129,15 @@ ovn_init_symtab(struct shash *symtab) > expr_symtab_add_field_scoped(symtab, "ct_label", MFF_CT_LABEL, NULL, > false, WR_CT_COMMIT); > expr_symtab_add_subfield_scoped(symtab, "ct_label.blocked", NULL, > - "ct_label[0]", WR_CT_COMMIT); > + "ct_label[" > + OVN_CT_STR(OVN_CT_BLOCKED_BIT) > + "]", > + WR_CT_COMMIT); > expr_symtab_add_subfield_scoped(symtab, "ct_label.natted", NULL, > - "ct_label[1]", WR_CT_COMMIT); > + "ct_label[" > + OVN_CT_STR(OVN_CT_NATTED_BIT) > + "]", > + WR_CT_COMMIT); > expr_symtab_add_subfield_scoped(symtab, "ct_label.ecmp_reply_eth", NULL, > "ct_label[32..79]", WR_CT_COMMIT); > expr_symtab_add_subfield_scoped(symtab, "ct_label.ecmp_reply_port", NULL, > diff --git a/tests/ovn.at b/tests/ovn.at > index f222fd8..310eff9 100644 > --- a/tests/ovn.at > +++ b/tests/ovn.at > @@ -22935,7 +22935,7 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8-], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8-], [0], [dnl > @@ -22965,9 +22965,9 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23002,9 +23002,9 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23029,10 +23029,10 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23048,10 +23048,10 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23077,11 +23077,11 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23099,11 +23099,11 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23131,12 +23131,12 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23155,12 +23155,12 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23180,22 +23180,23 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct > > check ovn-nbctl --wait=hv ls-lb-add sw1 lb-ipv6-udp > > +# Number of hairpin flows shouldn't change as it doesn't depend on how many > +# datapaths the LB is applied. > OVS_WAIT_UNTIL( > - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] > + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] > ) > > OVS_WAIT_UNTIL( > - [test $(as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] > + [test $(as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23216,13 +23217,12 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > @@ -23261,13 +23261,13 @@ OVS_WAIT_UNTIL( > ) > > OVS_WAIT_UNTIL( > - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] > + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] > ) > > check ovn-nbctl --wait=hv lb-del lb-ipv4-tcp > > OVS_WAIT_UNTIL( > - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 4] > + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 3] > ) > > OVS_WAIT_UNTIL( > @@ -23275,10 +23275,9 @@ OVS_WAIT_UNTIL( > ) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] > ]) > > AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl > -- > 1.8.3.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
On 12/14/20 10:34 AM, Numan Siddique wrote: > On Fri, Dec 11, 2020 at 3:46 PM Dumitru Ceara <dceara@redhat.com> wrote: >> >> Commit fc219d84b667 [0] optimized Load Balancer hairpin support from an >> OVN Southbound perspective by moving generation of flows to detect >> hairpin traffic in ovn-controller. >> >> However, this can be further optimized by relaxing the condition for >> flows that detect hairpin traffic in the "original" direction. >> >> It is safe to exclude the match on datapath.tunnel_key in the original >> direction if we make sure that we only match on traffic that was >> successfully load balanced (i.e., ct.natted = 1). At this point it >> would be enough to check that ip.src == ip.dst in order to determine >> that traffic needs to be hairpinned. Because OVS doesn't allow such >> checks between fields we instead check against the known load balancer >> backend IPs: ip.src == backend-ip && ip.dst == backend-ip. >> >> This change improves scalability by reducing the number of hairpin OF >> flows by ~50%. For example, on a setup with: >> - N Load Balancer VIPs. >> - M Backends per VIP. >> - All N VIPs applied to S logical switches. >> >> Before this change the number of hairpin OF flows was: >> - N x M x S (for original direction) + N x M x S (for reply direction) >> >> With this change the number of hairpin OF flows is: >> - N x M (for original direction ) + N x M x S (for reply direction) >> >> That means that on a setup with N=100 VIPs, M=10 backends, S=100 >> switches the number of hairpin OF flows will be 101K vs 200K before this >> change. >> >> [0] fc219d84b667 ("actions: Add new actions chk_lb_hairpin, chk_lb_hairpin_reply and ct_snat_to_vip.") >> >> Signed-off-by: Dumitru Ceara <dceara@redhat.com> > > Thanks Dumitru for addressing this scale concern and reducing the OF > flows by half. > > Could we also do the same even for OFTABLE_CHK_LB_HAIRPIN_REPLY ? Did > you explore that too ? > Hi Numan, Not really, at least not exactly in the same way. The problem is that reply to hairpinned traffic has ip.src == lb_backend.ip && ip.dst == lb.vip. So we can't just remove the match on metadata (like we did for packets in the original direction) because a load balancer might not be applied on all datapaths. One option is to send all traffic with ip.dst == lb.vip to conntrack to see if it matches the SNAT session we created for the hairpinned packet in the original direction. However, this additional recirculation has implications on throughput and latency for all non-hairpinned load balancer traffic. I don't have a better solution yet but I'll give it more thought. > I applied this patch to master and also to branch-20.12 as it seems an > important fix. Thanks. > > There was one nit which I fixed before applying > > --- > diff --git a/controller/lflow.c b/controller/lflow.c > index f63a6f56a5..c02585b1eb 100644 > --- a/controller/lflow.c > +++ b/controller/lflow.c > @@ -1226,7 +1226,7 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb, > * to not include the datapath tunnel_key in the match when determining > * that a packet needs to be hairpinned because the rest of the match is > * restrictive enough: > - * - traffic must have already been load balancedu > + * - traffic must have already been load balanced. > * - packets must have ip.src == ip.dst at this point. > * - the destination protocol and port must be of a valid backend that > * has the same IP as ip.dst. This looks good to me, thanks for catching the typo! Thanks, Dumitru
diff --git a/controller/lflow.c b/controller/lflow.c index fc58a16..f63a6f5 100644 --- a/controller/lflow.c +++ b/controller/lflow.c @@ -1221,16 +1221,29 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb, match_set_tp_src(&hairpin_reply_match, htons(lb_backend->port)); } + /* In the original direction, only match on traffic that was already + * load balanced, i.e., "ct.natted == 1". Also, it's good enough + * to not include the datapath tunnel_key in the match when determining + * that a packet needs to be hairpinned because the rest of the match is + * restrictive enough: + * - traffic must have already been load balancedu + * - packets must have ip.src == ip.dst at this point. + * - the destination protocol and port must be of a valid backend that + * has the same IP as ip.dst. + */ + ovs_u128 lb_ct_label = { + .u64.lo = OVN_CT_NATTED, + }; + match_set_ct_label_masked(&hairpin_match, lb_ct_label, lb_ct_label); + + ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN, 100, + lb->slb->header_.uuid.parts[0], &hairpin_match, + &ofpacts, &lb->slb->header_.uuid); + for (size_t i = 0; i < lb->slb->n_datapaths; i++) { - match_set_metadata(&hairpin_match, - htonll(lb->slb->datapaths[i]->tunnel_key)); match_set_metadata(&hairpin_reply_match, htonll(lb->slb->datapaths[i]->tunnel_key)); - ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN, 100, - lb->slb->header_.uuid.parts[0], &hairpin_match, - &ofpacts, &lb->slb->header_.uuid); - ofctrl_add_flow(flow_table, OFTABLE_CHK_LB_HAIRPIN_REPLY, 100, lb->slb->header_.uuid.parts[0], &hairpin_reply_match, diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h index 0fe5bc3..aee4748 100644 --- a/include/ovn/logical-fields.h +++ b/include/ovn/logical-fields.h @@ -17,6 +17,7 @@ #define OVN_LOGICAL_FIELDS_H 1 #include "openvswitch/meta-flow.h" +#include "openvswitch/util.h" struct shash; @@ -140,4 +141,23 @@ ovn_field_from_id(enum ovn_field_id id) const char *event_to_string(enum ovn_controller_event event); int string_to_event(const char *s); const struct ovn_field *ovn_field_from_name(const char *name); + +/* OVN CT label values + * =================== + * These are specific ct.label bit values OVN uses to track different types + * of traffic. + */ + +#define OVN_CT_BLOCKED_BIT 0 +#define OVN_CT_NATTED_BIT 1 + +#define OVN_CT_BLOCKED 1 +#define OVN_CT_NATTED 2 + +#define OVN_CT_STR(LABEL_VALUE) OVS_STRINGIZE(LABEL_VALUE) +#define OVN_CT_MASKED_STR(LABEL_VALUE) \ + OVS_STRINGIZE(LABEL_VALUE) "/" OVS_STRINGIZE(LABEL_VALUE) + +#define OVN_CT_LABEL_STR(LABEL_VALUE) "ct_label[" OVN_CT_STR(LABEL_VALUE) "]" + #endif /* ovn/lib/logical-fields.h */ diff --git a/lib/actions.c b/lib/actions.c index 0705e4e..fbaeb34 100644 --- a/lib/actions.c +++ b/lib/actions.c @@ -1224,7 +1224,9 @@ encode_CT_LB(const struct ovnact_ct_lb *cl, ds_put_format(&ds, ":%"PRIu16, dst->port); } ds_put_format(&ds, "),commit,table=%d,zone=NXM_NX_REG%d[0..15]," - "exec(set_field:2/2->ct_label))", + "exec(set_field:" + OVN_CT_MASKED_STR(OVN_CT_NATTED) + "->ct_label))", recirc_table, zone_reg); } diff --git a/lib/logical-fields.c b/lib/logical-fields.c index bf61df7..9d08b44 100644 --- a/lib/logical-fields.c +++ b/lib/logical-fields.c @@ -129,9 +129,15 @@ ovn_init_symtab(struct shash *symtab) expr_symtab_add_field_scoped(symtab, "ct_label", MFF_CT_LABEL, NULL, false, WR_CT_COMMIT); expr_symtab_add_subfield_scoped(symtab, "ct_label.blocked", NULL, - "ct_label[0]", WR_CT_COMMIT); + "ct_label[" + OVN_CT_STR(OVN_CT_BLOCKED_BIT) + "]", + WR_CT_COMMIT); expr_symtab_add_subfield_scoped(symtab, "ct_label.natted", NULL, - "ct_label[1]", WR_CT_COMMIT); + "ct_label[" + OVN_CT_STR(OVN_CT_NATTED_BIT) + "]", + WR_CT_COMMIT); expr_symtab_add_subfield_scoped(symtab, "ct_label.ecmp_reply_eth", NULL, "ct_label[32..79]", WR_CT_COMMIT); expr_symtab_add_subfield_scoped(symtab, "ct_label.ecmp_reply_port", NULL, diff --git a/tests/ovn.at b/tests/ovn.at index f222fd8..310eff9 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -22935,7 +22935,7 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8-], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8-], [0], [dnl @@ -22965,9 +22965,9 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23002,9 +23002,9 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23029,10 +23029,10 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23048,10 +23048,10 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23077,11 +23077,11 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23099,11 +23099,11 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23131,12 +23131,12 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23155,12 +23155,12 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23180,22 +23180,23 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct check ovn-nbctl --wait=hv ls-lb-add sw1 lb-ipv6-udp +# Number of hairpin flows shouldn't change as it doesn't depend on how many +# datapaths the LB is applied. OVS_WAIT_UNTIL( - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] ) OVS_WAIT_UNTIL( - [test $(as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] + [test $(as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23216,13 +23217,12 @@ priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ip,metadata=0x1 actions=ct ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp,metadata=0x1,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=42.42.42.42,nw_dst=42.42.42.42,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp,nw_src=52.52.52.52,nw_dst=52.52.52.52,tp_dst=4042 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl @@ -23261,13 +23261,13 @@ OVS_WAIT_UNTIL( ) OVS_WAIT_UNTIL( - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 7] + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 6] ) check ovn-nbctl --wait=hv lb-del lb-ipv4-tcp OVS_WAIT_UNTIL( - [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 4] + [test $(as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | wc -l) -eq 3] ) OVS_WAIT_UNTIL( @@ -23275,10 +23275,9 @@ OVS_WAIT_UNTIL( ) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=68 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl -priority=100,tcp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp,metadata=0x1,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x1,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] -priority=100,udp6,metadata=0x2,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,tcp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=4041 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp,nw_src=42.42.42.1,nw_dst=42.42.42.1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] +priority=100,ct_label=0x2/0x2,udp6,ipv6_src=4200::1,ipv6_dst=4200::1,tp_dst=2021 actions=load:0x1->NXM_NX_REG10[[7]] ]) AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST | cut -d ' ' -f8- | sort], [0], [dnl
Commit fc219d84b667 [0] optimized Load Balancer hairpin support from an OVN Southbound perspective by moving generation of flows to detect hairpin traffic in ovn-controller. However, this can be further optimized by relaxing the condition for flows that detect hairpin traffic in the "original" direction. It is safe to exclude the match on datapath.tunnel_key in the original direction if we make sure that we only match on traffic that was successfully load balanced (i.e., ct.natted = 1). At this point it would be enough to check that ip.src == ip.dst in order to determine that traffic needs to be hairpinned. Because OVS doesn't allow such checks between fields we instead check against the known load balancer backend IPs: ip.src == backend-ip && ip.dst == backend-ip. This change improves scalability by reducing the number of hairpin OF flows by ~50%. For example, on a setup with: - N Load Balancer VIPs. - M Backends per VIP. - All N VIPs applied to S logical switches. Before this change the number of hairpin OF flows was: - N x M x S (for original direction) + N x M x S (for reply direction) With this change the number of hairpin OF flows is: - N x M (for original direction ) + N x M x S (for reply direction) That means that on a setup with N=100 VIPs, M=10 backends, S=100 switches the number of hairpin OF flows will be 101K vs 200K before this change. [0] fc219d84b667 ("actions: Add new actions chk_lb_hairpin, chk_lb_hairpin_reply and ct_snat_to_vip.") Signed-off-by: Dumitru Ceara <dceara@redhat.com> --- controller/lflow.c | 25 ++++++--- include/ovn/logical-fields.h | 20 ++++++++ lib/actions.c | 4 +- lib/logical-fields.c | 10 +++- tests/ovn.at | 117 +++++++++++++++++++++---------------------- 5 files changed, 108 insertions(+), 68 deletions(-)