| Message ID | 1610749328-81289-2-git-send-email-svc.mail.git@nutanix.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [ovs-dev,v2] OVN: Multiple distributed gateway port support | expand |
On Sat, Jan 16, 2021 at 3:53 AM Ankur Sharma <svc.mail.git@nutanix.com> wrote: > > From: Ankur Sharma <ankur.sharma@nutanix.com> > > By default, OVN support only one distributed gateway > port (we will call it l3dgw port for further reference) > per logical router. While a single l3dgw port suffices > for most of the North South connectivity, however there > are requirements where a logical router could be connected > to multiple physical networks and based on routing decision > packet could go to vlan X or vlan Y. Additionally, packet > may or may not get NATed based on the configuration. > > This patch adds flexibility of having multiple l3dgw ports > per logical router. > > Changes can classified as following: > a. Data structure changes to allow multiple l3dgw ports per > ovn_datapath. > > b. Consumption of new data structure in logical flows for > individual features. > > c. Features that require changes are: > i. Regular NS traffic flow. > ii. Network Address Translation. > iii. Load Balancer > iv. Gateway_mtu. > v. reside-on-redirect-chassis > vi. Misc code sections that assumed a single l3dgw port. > > d. ovn-nbctl cli change to allow multiple external ips > for a logical ip for same type. > > e. Except for reside-on-redirect-chassis all the other features > could be extended to multiple l3dgw ports. Reside on redirect > chassis with its current specification could not be extended > and hence should be used only with the logical router that > has a single l3dgw port. > > FUTURE WORK: > CT ZONES are still common for traffic from different physical networks. > This adds a restriction/assumption that same 5 tuple will not come > from different l3dgw ports. > A cleaner approach would be have different ct zones for each l3dgw port. > Changing the CT ZONE assignment is one of the enhancements we are > considering as next step. > > Signed-off-by: Ankur Sharma <ankur.sharma@nutanix.com> > Signed-off-by: Dhathri Purohith <dhathri.purohith@nutanix.com> > Signed-off-by: Ankur Sharma <ankurmnnit2004@gmail.com> > Co-authored-by: Dhathri Purohith <dhathri.purohith@nutanix.com> > Co-authored-by: Ankur Sharma <ankurmnnit2004@gmail.com> Hi Ankur, Thanks for adding this feature. Since you're submitting this patch, I think you will be the primary author. Probably you can remove Co-authored-by tag with your name. This patch needs a rebase. Can you please rebase it. I was the able to resolve the conflicts and apply the patch on my local repo on top of master. I see below compilation issues with sparse configured (./configure --enable-Werror --enable-sparse) ******** Tpo -c -o northd/ovn-northd.o ../northd/ovn-northd.c &&\ mv -f $depbase.Tpo $depbase.Po ../northd/ovn-northd.c:1406:13: error: incorrect type in assignment (different base types) ../northd/ovn-northd.c:1406:13: expected restricted ovs_be32 [addressable] [usertype] ip4 ../northd/ovn-northd.c:1406:13: got unsigned int ../northd/ovn-northd.c:1441:38: error: incorrect type in initializer (different base types) ../northd/ovn-northd.c:1441:38: expected restricted ovs_be32 [usertype] addr ../northd/ovn-northd.c:1441:38: got unsigned int ../northd/ovn-northd.c:1442:41: error: incorrect type in initializer (different base types) ../northd/ovn-northd.c:1442:41: expected restricted ovs_be32 [usertype] network ../northd/ovn-northd.c:1442:41: got unsigned int ../northd/ovn-northd.c:1443:23: error: incorrect type in assignment (different base types) ../northd/ovn-northd.c:1443:23: expected restricted ovs_be32 [addressable] [usertype] mask4 ../northd/ovn-northd.c:1443:23: got unsigned int ../northd/ovn-northd.c:1446:21: error: restricted ovs_be32 degrades to integer ../northd/ovn-northd.c:1446:28: error: restricted ovs_be32 degrades to integer ../northd/ovn-northd.c:1446:39: error: restricted ovs_be32 degrades to integer ../northd/ovn-northd.c:1446:45: error: restricted ovs_be32 degrades to integer make[1]: *** [Makefile:2044: northd/ovn-northd.o] Error 1 ********* This is not a full review, but please see below with some comments. Thanks Numan > --- > northd/ovn-northd.c | 517 +++++++++++++++++++++++++++++++++----------------- > tests/ovn-northd.at | 353 +++++++++++++++++++++++++++++++++- > tests/ovn.at | 310 +++++++++++++++++++++++++++++- > utilities/ovn-nbctl.c | 35 +++- > 4 files changed, 1024 insertions(+), 191 deletions(-) > > diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c > index 969fbe1..c1cef7d 100644 > --- a/northd/ovn-northd.c > +++ b/northd/ovn-northd.c > @@ -573,6 +573,19 @@ ovn_mcast_group_allocate_key(struct mcast_info *mcast_info) > &mcast_info->group_tnlid_hint); > } > > +struct ovn_datapath_l3dgw_port { I'd suggest renaming this to 'struct ovn_l3dgw_port'. > + > + /* OVN northd only needs to know about the logical router gateway port for > + * NAT on a distributed router. This "distributed gateway port" is > + * populated only when there is a gateway chassis specified for one of s/there is a gateway chassis/here is gateway chassis or ha chassis group > + * the ports on the logical router. Otherwise this will be NULL. */ > + struct ovn_port *dgw_port; > + > + /* The "derived" OVN port representing the instance of l3dgw_port on > + * the gateway chassis. */ > + struct ovn_port *redirect_port; > +}; > + > /* The 'key' comes from nbs->header_.uuid or nbr->header_.uuid or > * sb->external_ids:logical-switch. */ > struct ovn_datapath { > @@ -602,14 +615,9 @@ struct ovn_datapath { > /* Multicast data. */ > struct mcast_info mcast_info; > > - /* OVN northd only needs to know about the logical router gateway port for > - * NAT on a distributed router. This "distributed gateway port" is > - * populated only when there is a gateway chassis specified for one of > - * the ports on the logical router. Otherwise this will be NULL. */ > - struct ovn_port *l3dgw_port; > - /* The "derived" OVN port representing the instance of l3dgw_port on > - * the gateway chassis. */ > - struct ovn_port *l3redirect_port; > + /* L3 distributed gateway ports */ > + struct ovn_datapath_l3dgw_port *l3dgw_ports; > + size_t n_l3dgw_ports; > > /* NAT entries configured on the router. */ > struct ovn_nat *nat_entries; > @@ -814,6 +822,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct ovn_datapath *od) > ovn_destroy_tnlids(&od->port_tnlids); > destroy_ipam_info(&od->ipam_info); > free(od->router_ports); > + free(od->l3dgw_ports); > destroy_nat_entries(od); > free(od->nat_entries); > free(od->localnet_ports); > @@ -1353,6 +1362,95 @@ struct ovn_port { > struct ovs_list list; /* In list of similar records. */ > }; > > +/* Get the l3dgw port corresponding to a logical router port.*/ > +static inline struct ovn_datapath_l3dgw_port* > +ovn_get_l3dgw_port_from_lrp(const struct ovn_port *op) > +{ > + struct ovn_datapath *od = op->od; > + > + if (!op || !op->nbrp) { > + return NULL; > + } > + > + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { Since n_l3dgw_ports is of type size_t, please use iter of the same type. > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + &(od->l3dgw_ports[iter]); > + if (op == l3dgw_port->dgw_port) { > + return l3dgw_port; > + } > + } > + > + return NULL; > +} > + > +/* Get the l3dgw port corresponding to a logical router port > + * with input ip */ > +static struct ovn_datapath_l3dgw_port* > +ovn_get_l3dgw_port_from_ip(struct ovn_datapath *od, char *ip_s, bool is_v6) (> +{ The caller of this function - build_lrouter_nat_defrag_and_lb(), has already parsed the nat->external_ip using ip_parse_masked or ipv6_parse_masked. So I think there is no need to parse the same here. You can take "struct in6_addr" as an argument. You can use 'IN6_IS_ADDR_V4MAPPED' to determine if the 'struct in6_addr' has IPv4 or IPv6 address. build_lrouter_nat_defrag_and_lb() before calling this function - ovn_get_l3dgw_port_from_ip(), can use in6_addr_set_mapped_ipv4() to set IPv4 address. > + ovs_be32 ip4, mask4; > + struct in6_addr ip6, mask6; > + > + char *error = NULL; > + > + if (!od || !od->nbr) { > + return NULL; > + } > + > + if (is_v6) { > + error = ipv6_parse_masked(ip_s, &ip6, &mask6); > + } else { > + error = ip_parse_masked(ip_s, &ip4, &mask4); > + ip4 = ntohl(ip4); > + } > + > + if (error) { > + free(error); > + return NULL; > + } > + > + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + &(od->l3dgw_ports[iter]); > + struct ovn_port *op = l3dgw_port->dgw_port; > + struct lport_addresses lrp_networks; > + > + if (!extract_lrp_networks(op->nbrp, &lrp_networks)) { > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > + VLOG_WARN_RL(&rl, "Extract addresses failed."); > + continue; > + } > + > + if (is_v6) { > + for (int iter2 = 0; iter2 < lrp_networks.n_ipv6_addrs; iter2++) { > + struct ipv6_netaddr *lrp6_addr = > + &(lrp_networks.ipv6_addrs[iter2]); > + struct in6_addr ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask, > + &ip6); > + > + if (ipv6_addr_equals(&ip6_mask, &(lrp6_addr->network))) { > + return l3dgw_port; > + } > + } > + } else { > + for (int iter2 = 0; iter2 < lrp_networks.n_ipv4_addrs; iter2++) { > + struct ipv4_netaddr *lrp4_addr = > + &(lrp_networks.ipv4_addrs[iter2]); > + ovs_be32 addr = ntohl(lrp4_addr->addr); > + ovs_be32 network = ntohl(lrp4_addr->network); > + mask4 = ntohl(lrp4_addr->mask); > + ovs_be32 bcast = addr | ~mask4; > + > + if (ip4 >= network && ip4 < bcast) { > + return l3dgw_port; > + } > + } > + } > + } > + > + return NULL; > +} > + > static void > ovn_port_set_nb(struct ovn_port *op, > const struct nbrec_logical_switch_port *nbsp, > @@ -2246,13 +2344,12 @@ join_logical_ports(struct northd_context *ctx, > "on L3 gateway router", nbrp->name); > continue; > } > - if (od->l3dgw_port || od->l3redirect_port) { > + if (od->n_l3dgw_ports) { > static struct vlog_rate_limit rl > = VLOG_RATE_LIMIT_INIT(1, 1); > - VLOG_WARN_RL(&rl, "Bad configuration: multiple " > - "distributed gateway ports on logical " > - "router %s", od->nbr->name); > - continue; > + VLOG_DBG_RL(&rl, "Multiple ports with " > + "redirect-chassis on same " > + "logical router %s", od->nbr->name); > } > > char *redirect_name = > @@ -2274,8 +2371,12 @@ join_logical_ports(struct northd_context *ctx, > > /* Set l3dgw_port and l3redirect_port in od, for later > * use during flow creation. */ > - od->l3dgw_port = op; > - od->l3redirect_port = crp; > + od->l3dgw_ports = xrealloc(od->l3dgw_ports, > + sizeof *od->l3dgw_ports * > + (od->n_l3dgw_ports + 1)); > + (od->l3dgw_ports[od->n_l3dgw_ports]).dgw_port = op; > + (od->l3dgw_ports[od->n_l3dgw_ports]).redirect_port = crp; > + od->n_l3dgw_ports++; > } > } > } > @@ -2431,7 +2532,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n) > > /* Determine whether this NAT rule satisfies the conditions for > * distributed NAT processing. */ > - if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat") > + if (op->od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") > && nat->logical_port && nat->external_mac) { > /* Distributed NAT rule. */ > if (eth_addr_from_string(nat->external_mac, &mac)) { > @@ -2491,11 +2592,13 @@ get_nat_addresses(const struct ovn_port *op, size_t *n) > sset_destroy(&all_ips_v6); > > if (central_ip_address) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > /* Gratuitous ARP for centralized NAT rules on distributed gateway > * ports should be restricted to the gateway chassis. */ > - if (op->od->l3redirect_port) { > + if (l3dgw_port) { > ds_put_format(&c_addresses, " is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > addresses[n_nats++] = ds_steal_cstr(&c_addresses); > @@ -2988,7 +3091,7 @@ ovn_port_update_sbrec(struct northd_context *ctx, > char **nats = NULL; > if (nat_addresses && !strcmp(nat_addresses, "router")) { > if (op->peer && op->peer->od > - && (chassis || op->peer->od->l3redirect_port)) { > + && (chassis || op->peer->od->n_l3dgw_ports)) { > nats = get_nat_addresses(op->peer, &n_nats); > } > /* Only accept manual specification of ethernet address > @@ -3024,11 +3127,11 @@ ovn_port_update_sbrec(struct northd_context *ctx, > * sending the GARPs for the router port IPs. > * */ > bool add_router_port_garp = false; > - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port && > - op->peer->od->l3redirect_port && > + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL; > + if (op->peer && op->peer->nbrp && op->peer->od->n_l3dgw_ports && > (smap_get_bool(&op->peer->nbrp->options, > "reside-on-redirect-chassis", false) || > - op->peer == op->peer->od->l3dgw_port)) { > + (l3dgw_port = ovn_get_l3dgw_port_from_lrp(op->peer)))) { > add_router_port_garp = true; > } else if (chassis && op->od->n_localnet_ports) { > add_router_port_garp = true; > @@ -3043,9 +3146,12 @@ ovn_port_update_sbrec(struct northd_context *ctx, > op->peer->lrp_networks.ipv4_addrs[i].addr_s); > } > > - if (op->peer->od->l3redirect_port) { > + if (op->peer->od->n_l3dgw_ports) { > + if (!l3dgw_port) { > + l3dgw_port = &(op->peer->od->l3dgw_ports[0]); > + } > ds_put_format(&garp_info, " is_chassis_resident(%s)", > - op->peer->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > n_nats++; > @@ -4637,7 +4743,6 @@ build_lswitch_input_port_sec_op( > struct ovn_port *op, struct hmap *lflows, > struct ds *actions, struct ds *match) > { > - > if (!op->nbsp) { > return; > } > @@ -5937,13 +6042,16 @@ build_lrouter_groups__(struct hmap *ports, struct ovn_datapath *od) > { > ovs_assert((od && od->nbr && od->lr_group)); > > - if (od->l3dgw_port && od->l3redirect_port) { > + for (int i = 0; i < od->n_l3dgw_ports; i++) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + &(od->l3dgw_ports[i]); > + > /* It's a logical router with gateway port. If it > * has HA_Chassis_Group associated to it in SB DB, then store the > * ha chassis group name. */ > - if (od->l3redirect_port->sb->ha_chassis_group) { > + if (l3dgw_port->redirect_port->sb->ha_chassis_group) { > sset_add(&od->lr_group->ha_chassis_groups, > - od->l3redirect_port->sb->ha_chassis_group->name); > + l3dgw_port->redirect_port->sb->ha_chassis_group->name); > } > } > > @@ -7182,17 +7290,18 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, > ds_clear(match); > ds_put_format(match, "eth.dst == "ETH_ADDR_FMT, > ETH_ADDR_ARGS(mac)); > - if (op->peer->od->l3dgw_port > - && op->peer->od->l3redirect_port > - && op->od->n_localnet_ports) { > + if (op->peer->od->n_l3dgw_ports && > + op->od->n_localnet_ports) { > bool add_chassis_resident_check = false; > - if (op->peer == op->peer->od->l3dgw_port) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op->peer); > + if (l3dgw_port) { > /* The peer of this port represents a distributed > * gateway port. The destination lookup flow for the > * router's distributed gateway port MAC address should > * only be programmed on the gateway chassis. */ > add_chassis_resident_check = true; > - } else { > + } else if (op->peer->od->n_l3dgw_ports == 1) { > /* Check if the option 'reside-on-redirect-chassis' > * is set to true on the peer port. If set to true > * and if the logical switch has a localnet port, it > @@ -7200,6 +7309,11 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, > * this logical switch should be run on the chassis > * hosting the gateway port. > */ > + > + /* reside-on-redirect-chassis is supported only for > + * logical routers with single l3dgw port. > + */ > + l3dgw_port = &(op->peer->od->l3dgw_ports[0]); > add_chassis_resident_check = smap_get_bool( > &op->peer->nbrp->options, > "reside-on-redirect-chassis", false); > @@ -7207,7 +7321,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, > > if (add_chassis_resident_check) { > ds_put_format(match, " && is_chassis_resident(%s)", > - op->peer->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > } > > @@ -7220,8 +7334,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, > > /* Add ethernet addresses specified in NAT rules on > * distributed logical routers. */ > - if (op->peer->od->l3dgw_port > - && op->peer == op->peer->od->l3dgw_port) { > + if (ovn_get_l3dgw_port_from_lrp(op->peer)) { > for (int j = 0; j < op->peer->od->nbr->n_nat; j++) { > const struct nbrec_nat *nat > = op->peer->od->nbr->nat[j]; > @@ -8357,34 +8470,44 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od, > build_empty_lb_event_flow(od, lflows, lb_vip, lb, S_ROUTER_IN_DNAT, > meter_groups); > > - /* A match and actions for new connections. */ > - char *new_match = xasprintf("ct.new && %s", ds_cstr(match)); > - if (lb_force_snat_ip) { > - char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s", > - ds_cstr(actions)); > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > - new_match, new_actions, &lb->header_); > - free(new_actions); > - } else { > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > - new_match, ds_cstr(actions), &lb->header_); > - } > + for (int i = 0; i < od->n_l3dgw_ports; i++) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[i]); > + > + /* A match and actions for new connections. */ > + char *new_match = xasprintf("ct.new && %s && inport == %s && " > + "is_chassis_resident(%s)", ds_cstr(match), > + l3dgw_port->dgw_port->json_key, > + l3dgw_port->redirect_port->json_key); > + if (lb_force_snat_ip) { > + char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s", > + ds_cstr(actions)); > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > + new_match, new_actions, &lb->header_); > + free(new_actions); > + } else { > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > + new_match, ds_cstr(actions), &lb->header_); > + } > + > + /* A match and actions for established connections. */ > + char *est_match = xasprintf("ct.est && %s && inport == %s && " > + "is_chassis_resident(%s)", ds_cstr(match), > + l3dgw_port->dgw_port->json_key, > + l3dgw_port->redirect_port->json_key); > + if (lb_force_snat_ip) { > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > + est_match, > + "flags.force_snat_for_lb = 1; ct_dnat;", > + &lb->header_); > + } else { > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > + est_match, "ct_dnat;", &lb->header_); > + } > > - /* A match and actions for established connections. */ > - char *est_match = xasprintf("ct.est && %s", ds_cstr(match)); > - if (lb_force_snat_ip) { > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > - est_match, > - "flags.force_snat_for_lb = 1; ct_dnat;", > - &lb->header_); > - } else { > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, > - est_match, "ct_dnat;", &lb->header_); > + free(new_match); > + free(est_match); > } > > - free(new_match); > - free(est_match); > - > const char *ip_match = NULL; > if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) { > ip_match = "ip4"; > @@ -8418,49 +8541,55 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od, > ds_destroy(&unsnat_match); > } > > - if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) { > + if (!od->n_l3dgw_ports || !lb_vip->n_backends) { > return; > } > > - /* Add logical flows to UNDNAT the load balanced reverse traffic in > - * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the logical > - * router has a gateway router port associated. > - */ > - struct ds undnat_match = DS_EMPTY_INITIALIZER; > - ds_put_format(&undnat_match, "%s && (", ip_match); > + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { > + > + /* Add logical flows to UNDNAT the load balanced reverse traffic in > + * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the > + * logical router has a gateway router port associated. > + */ > + struct ds undnat_match = DS_EMPTY_INITIALIZER; > + ds_put_format(&undnat_match, "%s && (", ip_match); > + > + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[iter]); > > - for (size_t i = 0; i < lb_vip->n_backends; i++) { > - struct ovn_lb_backend *backend = &lb_vip->backends[i]; > - ds_put_format(&undnat_match, "(%s.src == %s", ip_match, > - backend->ip_str); > + for (size_t i = 0; i < lb_vip->n_backends; i++) { > + struct ovn_lb_backend *backend = &lb_vip->backends[i]; > + ds_put_format(&undnat_match, "(%s.src == %s", ip_match, > + backend->ip_str); > > - if (backend->port) { > - ds_put_format(&undnat_match, " && %s.src == %d) || ", > - proto, backend->port); > + if (backend->port) { > + ds_put_format(&undnat_match, " && %s.src == %d) || ", > + proto, backend->port); > + } else { > + ds_put_cstr(&undnat_match, ") || "); > + } > + } > + > + ds_chomp(&undnat_match, ' '); > + ds_chomp(&undnat_match, '|'); > + ds_chomp(&undnat_match, '|'); > + ds_chomp(&undnat_match, ' '); > + ds_put_format(&undnat_match, ") && outport == %s && " > + "is_chassis_resident(%s)", > + l3dgw_port->dgw_port->json_key, > + l3dgw_port->redirect_port->json_key); > + if (lb_force_snat_ip) { > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, > + ds_cstr(&undnat_match), > + "flags.force_snat_for_lb = 1; ct_dnat;", > + &lb->header_); > } else { > - ds_put_cstr(&undnat_match, ") || "); > - } > - } > - > - ds_chomp(&undnat_match, ' '); > - ds_chomp(&undnat_match, '|'); > - ds_chomp(&undnat_match, '|'); > - ds_chomp(&undnat_match, ' '); > - ds_put_format(&undnat_match, ") && outport == %s && " > - "is_chassis_resident(%s)", od->l3dgw_port->json_key, > - od->l3redirect_port->json_key); > - if (lb_force_snat_ip) { > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, > - ds_cstr(&undnat_match), > - "flags.force_snat_for_lb = 1; ct_dnat;", > - &lb->header_); > - } else { > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, > - ds_cstr(&undnat_match), "ct_dnat;", > - &lb->header_); > - } > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, > + ds_cstr(&undnat_match), "ct_dnat;", > + &lb->header_); > + } > > - ds_destroy(&undnat_match); > + ds_destroy(&undnat_match); > + } > } > > #define ND_RA_MAX_INTERVAL_MAX 1800 > @@ -8582,7 +8711,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od, > { > struct nbrec_address_set *allowed_ext_ips = nat->allowed_ext_ips; > struct nbrec_address_set *exempted_ext_ips = nat->exempted_ext_ips; > - bool is_gw_router = !od->l3dgw_port; > + bool is_gw_router = !od->n_l3dgw_ports; > > ovs_assert(allowed_ext_ips || exempted_ext_ips); > > @@ -8797,9 +8926,11 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op, > * upstream MAC learning points to the gateway chassis. > * Also need to avoid generation of multiple ARP responses > * from different chassis. */ > - if (op->od->l3redirect_port) { > + if (op->od->n_l3dgw_ports) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > ds_put_format(&match, "is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > } > > @@ -8986,6 +9117,8 @@ build_adm_ctrl_flows_for_lrouter_port( > struct ovn_port *op, struct hmap *lflows, > struct ds *match, struct ds *actions) > { > + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL; > + > if (op->nbrp) { > if (!lrport_is_enabled(op->nbrp)) { > /* Drop packets from disabled logical ports (since logical flow > @@ -9016,12 +9149,12 @@ build_adm_ctrl_flows_for_lrouter_port( > ds_clear(match); > ds_put_format(match, "eth.dst == %s && inport == %s", > op->lrp_networks.ea_s, op->json_key); > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > - && op->od->l3redirect_port) { > + l3dgw_port = ovn_get_l3dgw_port_from_lrp(op); > + if (l3dgw_port) { > /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s > * should only be received on the gateway chassis. */ > ds_put_format(match, " && is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50, > ds_cstr(match), ds_cstr(actions), > @@ -9141,6 +9274,8 @@ build_neigh_learning_flows_for_lrouter_port( > > /* Check if we need to learn mac-binding from ARP requests. */ > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > if (!learn_from_arp_request) { > /* ARP request to this address should always get learned, > * so add a priority-110 flow to set > @@ -9153,10 +9288,9 @@ build_neigh_learning_flows_for_lrouter_port( > op->lrp_networks.ipv4_addrs[i].network_s, > op->lrp_networks.ipv4_addrs[i].plen, > op->lrp_networks.ipv4_addrs[i].addr_s); > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > - && op->od->l3redirect_port) { > + if (l3dgw_port) { > ds_put_format(match, " && is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT > " = lookup_arp(inport, arp.spa, arp.sha); " > @@ -9173,10 +9307,9 @@ build_neigh_learning_flows_for_lrouter_port( > op->json_key, > op->lrp_networks.ipv4_addrs[i].network_s, > op->lrp_networks.ipv4_addrs[i].plen); > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > - && op->od->l3redirect_port) { > + if (l3dgw_port) { > ds_put_format(match, " && is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > ds_clear(actions); > ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT > @@ -9601,7 +9734,10 @@ build_arp_resolve_flows_for_lrouter_port( > } > } > > - if (!op->derived && op->od->l3redirect_port) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > + > + if (!op->derived && l3dgw_port) { > const char *redirect_type = smap_get(&op->nbrp->options, > "redirect-type"); > if (redirect_type && !strcasecmp(redirect_type, "bridged")) { > @@ -9614,7 +9750,7 @@ build_arp_resolve_flows_for_lrouter_port( > ds_clear(match); > ds_put_format(match, "outport == %s && " > "!is_chassis_resident(%s)", op->json_key, > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > ds_clear(actions); > ds_put_format(actions, "eth.dst = %s; next;", > op->lrp_networks.ea_s); > @@ -9925,11 +10061,14 @@ build_check_pkt_len_flows_for_lrouter( > ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1", > "next;"); > > - if (od->l3dgw_port && od->l3redirect_port) { > + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { > int gw_mtu = 0; > - if (od->l3dgw_port->nbrp) { > - gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options, > - "gateway_mtu", 0); > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + &(od->l3dgw_ports[iter]); > + > + if (l3dgw_port->dgw_port->nbrp) { > + gw_mtu = smap_get_int(&(l3dgw_port->dgw_port->nbrp->options), > + "gateway_mtu", 0); > } > /* Add the flows only if gateway_mtu is configured. */ > if (gw_mtu <= 0) { > @@ -9937,20 +10076,20 @@ build_check_pkt_len_flows_for_lrouter( > } > > ds_clear(match); > - ds_put_format(match, "outport == %s", od->l3dgw_port->json_key); > - > + ds_put_format(match, "outport == %s", > + l3dgw_port->dgw_port->json_key); > ds_clear(actions); > ds_put_format(actions, > REGBIT_PKT_LARGER" = check_pkt_larger(%d);" > " next;", gw_mtu + VLAN_ETH_HEADER_LEN); > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50, > ds_cstr(match), ds_cstr(actions), > - &od->l3dgw_port->nbrp->header_); > + &(l3dgw_port->dgw_port->nbrp->header_)); > > for (size_t i = 0; i < od->nbr->n_ports; i++) { > struct ovn_port *rp = ovn_port_find(ports, > od->nbr->ports[i]->name); > - if (!rp || rp == od->l3dgw_port) { > + if (rp == l3dgw_port->dgw_port) { > continue; > } > > @@ -9958,7 +10097,8 @@ build_check_pkt_len_flows_for_lrouter( > ds_clear(match); > ds_put_format(match, "inport == %s && outport == %s" > " && ip4 && "REGBIT_PKT_LARGER, > - rp->json_key, od->l3dgw_port->json_key); > + rp->json_key, > + l3dgw_port->dgw_port->json_key); > > ds_clear(actions); > /* Set icmp4.frag_mtu to gw_mtu */ > @@ -9987,7 +10127,8 @@ build_check_pkt_len_flows_for_lrouter( > ds_clear(match); > ds_put_format(match, "inport == %s && outport == %s" > " && ip6 && "REGBIT_PKT_LARGER, > - rp->json_key, od->l3dgw_port->json_key); > + rp->json_key, l3dgw_port->dgw_port > + ->json_key); > > ds_clear(actions); > /* Set icmp6.frag_mtu to gw_mtu */ > @@ -10029,11 +10170,14 @@ build_gateway_redirect_flows_for_lrouter( > struct ds *match, struct ds *actions) > { > if (od->nbr) { > - if (od->l3dgw_port && od->l3redirect_port) { > + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { > const struct ovsdb_idl_row *stage_hint = NULL; > > - if (od->l3dgw_port->nbrp) { > - stage_hint = &od->l3dgw_port->nbrp->header_; > + struct ovn_port *l3dgw_port = (od->l3dgw_ports[iter]).dgw_port; > + struct ovn_port *l3redirect_port = > + (od->l3dgw_ports[iter]).redirect_port; > + if (l3dgw_port->nbrp) { > + stage_hint = &l3dgw_port->nbrp->header_; > } > > /* For traffic with outport == l3dgw_port, if the > @@ -10041,13 +10185,12 @@ build_gateway_redirect_flows_for_lrouter( > * rule, then the traffic is redirected to the central > * instance of the l3dgw_port. */ > ds_clear(match); > - ds_put_format(match, "outport == %s", > - od->l3dgw_port->json_key); > + ds_put_format(match, "outport == %s", l3dgw_port->json_key); > ds_clear(actions); > ds_put_format(actions, "outport = %s; next;", > - od->l3redirect_port->json_key); > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50, > - ds_cstr(match), ds_cstr(actions), > + l3redirect_port->json_key); > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, > + 50, ds_cstr(match), ds_cstr(actions), > stage_hint); > } > > @@ -10278,16 +10421,17 @@ build_ipv6_input_flows_for_lrouter_port( > /* ND reply. These flows reply to ND solicitations for the > * router's own IP address. */ > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > ds_clear(match); > - if (op->od->l3dgw_port && op == op->od->l3dgw_port > - && op->od->l3redirect_port) { > + if (l3dgw_port) { > /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s > * should only be sent from the gateway chassi, so that > * upstream MAC learning points to the gateway chassis. > * Also need to avoid generation of multiple ND replies > * from different chassis. */ > ds_put_format(match, "is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > build_lrouter_nd_flow(op->od, op, "nd_na_router", > @@ -10299,7 +10443,7 @@ build_ipv6_input_flows_for_lrouter_port( > > /* UDP/TCP/SCTP port unreachable */ > if (!smap_get(&op->od->nbr->options, "chassis") > - && !op->od->l3dgw_port) { > + && !op->od->n_l3dgw_ports) { > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > ds_clear(match); > ds_put_format(match, > @@ -10515,10 +10659,12 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > op->lrp_networks.ipv4_addrs[i].network_s, > op->lrp_networks.ipv4_addrs[i].plen); > > - if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer > + if (op->od->n_l3dgw_ports && op->peer > && op->peer->od->n_localnet_ports) { > bool add_chassis_resident_check = false; > - if (op == op->od->l3dgw_port) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > + if (l3dgw_port) { > /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s > * should only be sent from the gateway chassis, so that > * upstream MAC learning points to the gateway chassis. > @@ -10534,6 +10680,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > * hosting the gateway port and it should reply to the > * ARP requests for the router port IPs. > */ > + l3dgw_port = &(op->od->l3dgw_ports[0]); > add_chassis_resident_check = smap_get_bool( > &op->nbrp->options, > "reside-on-redirect-chassis", false); > @@ -10541,7 +10688,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > > if (add_chassis_resident_check) { > ds_put_format(match, " && is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > } > > @@ -10559,9 +10706,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > const char *ip_address; > SSET_FOR_EACH (ip_address, &all_ips_v4) { > ds_clear(match); > - if (op == op->od->l3dgw_port) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > + if (l3dgw_port) { > ds_put_format(match, "is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > build_lrouter_arp_flow(op->od, op, > @@ -10571,9 +10720,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > > SSET_FOR_EACH (ip_address, &all_ips_v6) { > ds_clear(match); > - if (op == op->od->l3dgw_port) { > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + ovn_get_l3dgw_port_from_lrp(op); > + if (l3dgw_port) { > ds_put_format(match, "is_chassis_resident(%s)", > - op->od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > build_lrouter_nd_flow(op->od, op, "nd_na", > @@ -10585,7 +10736,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > sset_destroy(&all_ips_v6); > > if (!smap_get(&op->od->nbr->options, "chassis") > - && !op->od->l3dgw_port) { > + && !op->od->n_l3dgw_ports) { > /* UDP/TCP/SCTP port unreachable. */ > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > ds_clear(match); > @@ -10662,7 +10813,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, > * exception is on the l3dgw_port where we might need to use a > * different ETH address. > */ > - if (op != op->od->l3dgw_port) { > + if (!ovn_get_l3dgw_port_from_lrp(op)) { > return; > } > > @@ -10727,7 +10878,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > /* NAT rules are only valid on Gateway routers and routers with > * l3dgw_port (router has a port with gateway chassis > * specified). */ > - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) { > + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) { > return; > } > > @@ -10814,11 +10965,27 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > } > } > > + struct ovn_datapath_l3dgw_port *l3dgw_port = > + &((od)->l3dgw_ports[0]); > + if (od->n_l3dgw_ports) { > + /* Get the L3DGW port only for distributed router. */ > + l3dgw_port = ovn_get_l3dgw_port_from_ip(od, nat->external_ip, > + is_v6); > + if (!l3dgw_port) { > + static struct vlog_rate_limit rl = > + VLOG_RATE_LIMIT_INIT(5, 1); > + VLOG_WARN_RL(&rl, "Could not map external ip: %s to a " > + "gateway port "UUID_FMT"", nat->external_ip, > + UUID_ARGS(&od->key)); > + continue; > + } > + } > + > /* For distributed router NAT, determine whether this NAT rule > * satisfies the conditions for distributed NAT processing. */ > bool distributed = false; > struct eth_addr mac; > - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") && > + if (od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") && > nat->logical_port && nat->external_mac) { > if (eth_addr_from_string(nat->external_mac, &mac)) { > distributed = true; > @@ -10842,7 +11009,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > * egress pipeline. */ > if (!strcmp(nat->type, "snat") > || !strcmp(nat->type, "dnat_and_snat")) { > - if (!od->l3dgw_port) { > + if (!od->n_l3dgw_ports) { > /* Gateway router. */ > ds_clear(match); > ds_clear(actions); > @@ -10870,12 +11037,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > " && inport == %s", > is_v6 ? "6" : "4", > nat->external_ip, > - od->l3dgw_port->json_key); > - if (!distributed && od->l3redirect_port) { > + l3dgw_port->dgw_port->json_key); > + if (!distributed && od->n_l3dgw_ports) { > /* Flows for NAT rules that are centralized are only > * programmed on the gateway chassis. */ > ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > > if (!strcmp(nat->type, "dnat_and_snat") && stateless) { > @@ -10897,7 +11064,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > * to a logical IP address. */ > if (!strcmp(nat->type, "dnat") > || !strcmp(nat->type, "dnat_and_snat")) { > - if (!od->l3dgw_port) { > + if (!od->n_l3dgw_ports) { > /* Gateway router. */ > /* Packet when it goes from the initiator to destination. > * We need to set flags.loopback because the router can > @@ -10947,12 +11114,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > " && inport == %s", > is_v6 ? "6" : "4", > nat->external_ip, > - od->l3dgw_port->json_key); > - if (!distributed && od->l3redirect_port) { > + l3dgw_port->dgw_port->json_key); > + if (!distributed && od->n_l3dgw_ports) { > /* Flows for NAT rules that are centralized are only > * programmed on the gateway chassis. */ > ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > ds_clear(actions); > if (allowed_ext_ips || exempted_ext_ips) { > @@ -10979,12 +11146,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > } > > /* ARP resolve for NAT IPs. */ > - if (od->l3dgw_port) { > + if (od->n_l3dgw_ports) { > if (!strcmp(nat->type, "snat")) { > ds_clear(match); > ds_put_format( > match, "inport == %s && %s == %s", > - od->l3dgw_port->json_key, > + l3dgw_port->dgw_port->json_key, > is_v6 ? "ip6.src" : "ip4.src", nat->external_ip); > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT, > 120, ds_cstr(match), "next;", > @@ -10995,14 +11162,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > ds_clear(match); > ds_put_format( > match, "outport == %s && %s == %s", > - od->l3dgw_port->json_key, > + l3dgw_port->dgw_port->json_key, > is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4, > nat->external_ip); > ds_clear(actions); > ds_put_format( > actions, "eth.dst = %s; next;", > distributed ? nat->external_mac : > - od->l3dgw_port->lrp_networks.ea_s); > + l3dgw_port->dgw_port->lrp_networks.ea_s); > ovn_lflow_add_with_hint(lflows, od, > S_ROUTER_IN_ARP_RESOLVE, > 100, ds_cstr(match), > @@ -11025,19 +11192,19 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > * Note that this only applies for NAT on a distributed router. > * Undo DNAT on a gateway router is done in the ingress DNAT > * pipeline stage. */ > - if (od->l3dgw_port && (!strcmp(nat->type, "dnat") > + if (od->n_l3dgw_ports && (!strcmp(nat->type, "dnat") > || !strcmp(nat->type, "dnat_and_snat"))) { > ds_clear(match); > ds_put_format(match, "ip && ip%s.src == %s" > " && outport == %s", > is_v6 ? "6" : "4", > nat->logical_ip, > - od->l3dgw_port->json_key); > - if (!distributed && od->l3redirect_port) { > + l3dgw_port->dgw_port->json_key); > + if (!distributed && od->n_l3dgw_ports) { > /* Flows for NAT rules that are centralized are only > * programmed on the gateway chassis. */ > ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > ds_clear(actions); > if (distributed) { > @@ -11062,7 +11229,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > * address. */ > if (!strcmp(nat->type, "snat") > || !strcmp(nat->type, "dnat_and_snat")) { > - if (!od->l3dgw_port) { > + if (!od->n_l3dgw_ports) { > /* Gateway router. */ > ds_clear(match); > ds_put_format(match, "ip && ip%s.src == %s", > @@ -11105,13 +11272,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > " && outport == %s", > is_v6 ? "6" : "4", > nat->logical_ip, > - od->l3dgw_port->json_key); > - if (!distributed && od->l3redirect_port) { > + l3dgw_port->dgw_port->json_key); > + if (!distributed && od->n_l3dgw_ports) { > /* Flows for NAT rules that are centralized are only > * programmed on the gateway chassis. */ > priority += 128; > ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } > ds_clear(actions); > > @@ -11160,14 +11327,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > */ > ds_clear(actions); > ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;", > - od->l3dgw_port->lrp_networks.ea_s); > + l3dgw_port->dgw_port->lrp_networks.ea_s); > > ds_clear(match); > ds_put_format(match, > "eth.dst == "ETH_ADDR_FMT" && inport == %s" > " && is_chassis_resident(\"%s\")", > ETH_ADDR_ARGS(mac), > - od->l3dgw_port->json_key, > + l3dgw_port->dgw_port->json_key, > nat->logical_port); > ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50, > ds_cstr(match), ds_cstr(actions), > @@ -11190,7 +11357,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > "ip%s.src == %s && outport == %s && " > "is_chassis_resident(\"%s\")", > is_v6 ? "6" : "4", nat->logical_ip, > - od->l3dgw_port->json_key, nat->logical_port); > + l3dgw_port->dgw_port->json_key, > + nat->logical_port); > ds_put_format(actions, "eth.src = %s; %s = %s; next;", > nat->external_mac, > is_v6 ? REG_SRC_IPV6 : REG_SRC_IPV4, > @@ -11205,16 +11373,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > * gateway port have ip.dst matching a NAT external IP, then > * loop a clone of the packet back to the beginning of the > * ingress pipeline with inport = outport. */ > - if (od->l3dgw_port) { > + if (od->n_l3dgw_ports) { > /* Distributed router. */ > ds_clear(match); > ds_put_format(match, "ip%s.dst == %s && outport == %s", > is_v6 ? "6" : "4", > nat->external_ip, > - od->l3dgw_port->json_key); > + l3dgw_port->dgw_port->json_key); > if (!distributed) { > ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > + l3dgw_port->redirect_port->json_key); > } else { > ds_put_format(match, " && is_chassis_resident(\"%s\")", > nat->logical_port); > @@ -11238,7 +11406,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > } > > /* Handle force SNAT options set in the gateway router. */ > - if (!od->l3dgw_port) { > + if (!od->n_l3dgw_ports) { > if (dnat_force_snat_ip) { > if (od->dnat_force_snat_addrs.n_ipv4_addrs) { > build_lrouter_force_snat_flows(lflows, od, "4", > @@ -11277,7 +11445,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > > /* Load balancing and packet defrag are only valid on > * Gateway routers or router with gateway port. */ > - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) { > + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) { > sset_destroy(&nat_entries); > return; > } > @@ -11347,11 +11515,6 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, > prio = 120; > } > > - if (od->l3redirect_port && > - (lb_vip->n_backends || !lb_vip->empty_backend_rej)) { > - ds_put_format(match, " && is_chassis_resident(%s)", > - od->l3redirect_port->json_key); > - } > add_router_lb_flow(lflows, od, match, actions, prio, > lb_force_snat_ip, lb_vip, proto, > nb_lb, meter_groups, &nat_entries); > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > index 91eb9a3..2c01f20 100644 > --- a/tests/ovn-northd.at > +++ b/tests/ovn-northd.at > @@ -711,7 +711,7 @@ ovn_start > ovn-sbctl chassis-add gw1 geneve 127.0.0.1 > > ovn-nbctl lr-add R1 > -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64 > > ovn-nbctl ls-add S1 > ovn-nbctl lsp-add S1 S1-R1 > @@ -752,13 +752,13 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat 172.16.1.1 > > echo > echo "IPv6: stateful" > -ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2 > +ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat 3000::c 1000::3 > check_flow_match_sets 2 2 2 0 0 0 0 > -ovn-nbctl lr-nat-del R1 dnat_and_snat fd01::1 > +ovn-nbctl lr-nat-del R1 dnat_and_snat 3000::c > > echo > echo "IPv6: stateless" > -ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2 > +ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat 3000::c 1000::3 > check_flow_match_sets 2 0 0 0 0 2 2 > > AT_CLEANUP > @@ -769,7 +769,7 @@ ovn_start > ovn-sbctl chassis-add gw1 geneve 127.0.0.1 > > ovn-nbctl lr-add R1 > -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64 > > ovn-nbctl ls-add S1 > ovn-nbctl lsp-add S1 S1-R1 > @@ -2385,3 +2385,346 @@ ovn-nbctl destroy bfd $uuid > check_row_count bfd 2 > > AT_CLEANUP > + > +AT_SETUP([ovn-northd -- lr multiple gw ports]) > +ovn_start > + > +# Logical network: > +# 1 LR, 3 Logical Switches, > +# 1 gateway chassis attached to each corresponding LRP. > +# > +# | S1 (gw1) > +# | > +# ls ---- DR -- S3 (gw3) > +# (20.0.0.0/24) | > +# | S2 (gw2) > +# > +# We will validate basic LR logical flows. > + > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 > + > +ovn-nbctl lr-add DR > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.0/24 > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.0/24 > +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 > + > +ovn-nbctl ls-add S1 > +ovn-nbctl lsp-add S1 S1-DR > +ovn-nbctl lsp-set-type S1-DR router > +ovn-nbctl lsp-set-addresses S1-DR router > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 > + > +ovn-nbctl ls-add S2 > +ovn-nbctl lsp-add S2 S2-DR > +ovn-nbctl lsp-set-type S2-DR router > +ovn-nbctl lsp-set-addresses S2-DR router > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 > + > +ovn-nbctl ls-add S3 > +ovn-nbctl lsp-add S3 S3-DR > +ovn-nbctl lsp-set-type S3-DR router > +ovn-nbctl lsp-set-addresses S3-DR router > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 > + > +ovn-nbctl ls-add ls > +ovn-nbctl lsp-add ls ls-DR > +ovn-nbctl lsp-set-type ls-DR router > +ovn-nbctl lsp-set-addresses ls-DR router > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls > + > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 > + > +ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows DR > + > +# Check the flows in lr_in_lookup_neighbor stage > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR | wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S1 | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S2 | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S3 | wc -l], [0], [1 > +]) > + > +# Check the flows in lr_in_gw_redirect stage > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR | wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S1 | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S2 | wc -l], [0], [1 > +]) > +AT_CLEANUP > + > +AT_SETUP([ovn-northd -- lr multiple gw ports NAT]) > +ovn_start > + > +# Logical network: > +# 1 LR, 3 Logical Switches, > +# 1 gateway chassis attached to each corresponding LRP. > +# > +# | S1 (gw1) > +# | > +# ls ---- DR -- S3 (gw3) > +# (20.0.0.0/24) | > +# | S2 (gw2) > +# > +# We will validate basic SNAT, DNAT and DNAT_AND_SNAT with > +# multiple distributed gateway LRPs. > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 > + > +ovn-nbctl lr-add DR > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24 > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 > +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 > + > +ovn-nbctl ls-add S1 > +ovn-nbctl lsp-add S1 S1-DR > +ovn-nbctl lsp-set-type S1-DR router > +ovn-nbctl lsp-set-addresses S1-DR router > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 > + > +ovn-nbctl ls-add S2 > +ovn-nbctl lsp-add S2 S2-DR > +ovn-nbctl lsp-set-type S2-DR router > +ovn-nbctl lsp-set-addresses S2-DR router > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 > + > +ovn-nbctl ls-add S3 > +ovn-nbctl lsp-add S3 S3-DR > +ovn-nbctl lsp-set-type S3-DR router > +ovn-nbctl lsp-set-addresses S3-DR router > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 > + > +ovn-nbctl ls-add ls > +ovn-nbctl lsp-add ls ls-DR > +ovn-nbctl lsp-set-type ls-DR router > +ovn-nbctl lsp-set-addresses ls-DR router > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls > + > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 > + > +ovn-nbctl --wait=sb sync > + > +# Configure SNAT > +ovn-nbctl lr-nat-add DR snat 172.16.1.1 20.0.0.10 > +ovn-nbctl lr-nat-add DR snat 10.0.0.1 20.0.0.10 > +ovn-nbctl lr-nat-add DR snat 192.168.0.1 20.0.0.10 > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.1" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.1)"| wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.1" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.1)"| wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.1" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.1)"| wc -l], [0], [1 > +]) > + > +ovn-nbctl lr-nat-del DR snat 20.0.0.10 > +ovn-nbctl lr-nat-del DR snat 20.0.0.10 > +ovn-nbctl lr-nat-del DR snat 20.0.0.10 > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0 > +]) > + > +# Configure DNAT > +ovn-nbctl lr-nat-add DR dnat 172.16.1.10 20.0.0.10 > +ovn-nbctl lr-nat-add DR dnat 10.0.0.10 20.0.0.10 > +ovn-nbctl lr-nat-add DR dnat 192.168.0.10 20.0.0.10 > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +ovn-nbctl lr-nat-del DR dnat 172.16.1.10 > +ovn-nbctl lr-nat-del DR dnat 10.0.0.10 > +ovn-nbctl lr-nat-del DR dnat 192.168.0.10 > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0 > +]) > + > +# Configure DNAT_AND_SNAT > +ovn-nbctl lr-nat-add DR dnat_and_snat 172.16.1.10 20.0.0.10 > +ovn-nbctl lr-nat-add DR dnat_and_snat 10.0.0.10 20.0.0.10 > +ovn-nbctl lr-nat-add DR dnat_and_snat 192.168.0.10 20.0.0.10 > + > +ovn-sbctl dump-flows DR > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.10)"| wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.10)"| wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.10)"| wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1 > +]) > + > +ovn-nbctl lr-nat-del DR dnat_and_snat 172.16.1.10 > +ovn-nbctl lr-nat-del DR dnat_and_snat 10.0.0.10 > +ovn-nbctl lr-nat-del DR dnat_and_snat 192.168.0.10 > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0 > +]) > +AT_CLEANUP > + > +AT_SETUP([ovn-northd -- lr multiple gw ports LB]) > +ovn_start > + > +# Logical network: > +# 1 LR, 3 Logical Switches, > +# 1 gateway chassis attached to each corresponding LRP. > +# > +# | S1 (gw1) > +# | > +# ls ---- DR -- S3 (gw3) > +# (20.0.0.0/24) | > +# | S2 (gw2) > +# > +# We will validate LB with multiple distributed gateway LRPs. > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 > + > +ovn-nbctl lr-add DR > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24 > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 > +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 > + > +ovn-nbctl ls-add S1 > +ovn-nbctl lsp-add S1 S1-DR > +ovn-nbctl lsp-set-type S1-DR router > +ovn-nbctl lsp-set-addresses S1-DR router > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 > + > +ovn-nbctl ls-add S2 > +ovn-nbctl lsp-add S2 S2-DR > +ovn-nbctl lsp-set-type S2-DR router > +ovn-nbctl lsp-set-addresses S2-DR router > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 > + > +ovn-nbctl ls-add S3 > +ovn-nbctl lsp-add S3 S3-DR > +ovn-nbctl lsp-set-type S3-DR router > +ovn-nbctl lsp-set-addresses S3-DR router > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 > + > +ovn-nbctl ls-add ls > +ovn-nbctl lsp-add ls ls-DR > +ovn-nbctl lsp-set-type ls-DR router > +ovn-nbctl lsp-set-addresses ls-DR router > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls > + > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 > + > +ovn-nbctl --wait=sb sync > + > +ovn-nbctl lb-add lb0 192.168.0.3:80 10.0.0.2:80,10.0.0.3:80 > +ovn-nbctl lr-lb-add DR lb0 > + > +ovn-sbctl dump-flows DR > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| wc -l], [0], [3 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| wc -l], [0], [3 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S1 | grep cr-DR-S1| wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S1| grep cr-DR-S1 | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S2 | grep cr-DR-S2| wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S2| grep cr-DR-S2 | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S3 | grep cr-DR-S3| wc -l], [0], [1 > +]) > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S3| grep cr-DR-S3 | wc -l], [0], [1 > +]) > + > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep priority=120| wc -l], [0], [3 > +]) > + > +AT_CLEANUP > diff --git a/tests/ovn.at b/tests/ovn.at > index 9bac94b..7c44cd3 100644 > --- a/tests/ovn.at > +++ b/tests/ovn.at > @@ -19691,7 +19691,7 @@ AT_CAPTURE_FILE([sbflows2]) > OVS_WAIT_FOR_OUTPUT( > [ovn-sbctl dump-flows > sbflows2 > ovn-sbctl dump-flows lr0 | grep ct_lb | grep priority=120 | sed 's/table=..//'], 0, > - [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) > + [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) > ]) > > # get the svc monitor mac. > @@ -19732,8 +19732,8 @@ AT_CHECK( > AT_CAPTURE_FILE([sbflows4]) > ovn-sbctl dump-flows lr0 > sbflows4 > AT_CHECK([grep lr_in_dnat sbflows4 | grep priority=120 | sed 's/table=..//' | sort], [0], [dnl > - (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) > - (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(drop;) > + (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) > + (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(drop;) > ]) > > # Delete sw0-p1 > @@ -23715,3 +23715,307 @@ OVS_WAIT_UNTIL([test `ovs-vsctl get Interface lsp2 external_ids:ovn-installed` = > > OVN_CLEANUP([hv1]) > AT_CLEANUP > + > +AT_SETUP([ovn -- lr multiple gw ports]) > +ovn_start > + > +# Logical network: > +# 1 LR, 3 Logical Switches, > +# 1 gateway chassis attached to each corresponding LRP. > +# > +# | S1 (gw1) > +# | > +# ls ---- DR -- S3 (gw3) > +# (20.0.0.0/24) | > +# | S2 (gw2) > +# > +# S1 - VLAN 1000 > +# S2 - VLAN 2000 > +# S3 - VLAN 3000 > +# > +# 5 chassis(s), HV1----HV5 > +# > +# HV1 - VIF11 > +# HV2 - Gateway chassis gw1 > +# HV3 - Gateway chassis gw2 > +# HV4 - Gateway chassis gw3 > +# HV5 - North endpoint > + > +ovn-nbctl lr-add DR > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 > +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24 > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 > +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.0/24 > + > +ovn-nbctl ls-add S1 > +ovn-nbctl lsp-add S1 S1-DR > +ovn-nbctl lsp-set-type S1-DR router > +ovn-nbctl lsp-set-addresses S1-DR router > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 > +ovn-nbctl lsp-add S1 ln1 "" 1000 > +ovn-nbctl lsp-set-addresses ln1 unknown > +ovn-nbctl lsp-set-type ln1 localnet > +ovn-nbctl lsp-set-options ln1 network_name=phys > + > +ovn-nbctl ls-add S2 > +ovn-nbctl lsp-add S2 S2-DR > +ovn-nbctl lsp-set-type S2-DR router > +ovn-nbctl lsp-set-addresses S2-DR router > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 > +ovn-nbctl lsp-add S2 ln2 "" 2000 > +ovn-nbctl lsp-set-addresses ln2 unknown > +ovn-nbctl lsp-set-type ln2 localnet > +ovn-nbctl lsp-set-options ln2 network_name=phys > + > +ovn-nbctl ls-add S3 > +ovn-nbctl lsp-add S3 S3-DR > +ovn-nbctl lsp-set-type S3-DR router > +ovn-nbctl lsp-set-addresses S3-DR router > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 > +ovn-nbctl lsp-add S3 ln3 "" 3000 > +ovn-nbctl lsp-set-addresses ln3 unknown > +ovn-nbctl lsp-set-type ln3 localnet > +ovn-nbctl lsp-set-options ln3 network_name=phys > + > +ovn-nbctl ls-add ls > +ovn-nbctl lsp-add ls ls-DR > +ovn-nbctl lsp-set-type ls-DR router > +ovn-nbctl lsp-set-addresses ls-DR router > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls > + > +# Add the lsp lp11 to ls. This will map to VIF11. > +ovn-nbctl lsp-add ls lp11 > +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10" > +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10 > + > +# Add the Northbound endpoint, lp-north1 > +ovn-nbctl ls-add ls-north1 > +ovn-nbctl lsp-add ls-north1 ln4 "" 1000 > +ovn-nbctl lsp-set-addresses ln4 unknown > +ovn-nbctl lsp-set-type ln4 localnet > +ovn-nbctl lsp-set-options ln4 network_name=phys > + > +ovn-nbctl lsp-add ls-north1 lp-north1 > +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10" > +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11 > + > +# Add the Northbound endpoint, lp-north2 > +ovn-nbctl ls-add ls-north2 > +ovn-nbctl lsp-add ls-north2 ln5 "" 2000 > +ovn-nbctl lsp-set-addresses ln5 unknown > +ovn-nbctl lsp-set-type ln5 localnet > +ovn-nbctl lsp-set-options ln5 network_name=phys > + > +ovn-nbctl lsp-add ls-north2 lp-north2 > +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10" > +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22 > + > +# Add the Northbound endpoint, lp-north3 > +ovn-nbctl ls-add ls-north3 > +ovn-nbctl lsp-add ls-north3 ln6 "" 3000 > +ovn-nbctl lsp-set-addresses ln6 unknown > +ovn-nbctl lsp-set-type ln6 localnet > +ovn-nbctl lsp-set-options ln6 network_name=phys > + > +ovn-nbctl lsp-add ls-north3 lp-north3 > +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10" > +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33 > + > +# Add 5 chassis > +net_add n1 > +for i in 1 2 3 4 5; do > + sim_add hv$i > + as hv$i > + ovs-vsctl add-br br-phys > + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys > + ovn_attach n1 br-phys 192.168.0.$i 24 $encap > +done > + > +# Add a vif on HV1 > +as hv1 ovs-vsctl add-port br-int vif11 -- \ > + set Interface vif11 external-ids:iface-id=lp11 \ > + options:tx_pcap=hv1/vif11-tx.pcap \ > + options:rxq_pcap=hv1/vif11-rx.pcap \ > + ofport-request=11 > +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup]) > + > +as hv5 ovs-vsctl add-port br-int vif-north1 -- \ > + set Interface vif-north1 external-ids:iface-id=lp-north1 \ > + options:tx_pcap=hv5/vif-north1-tx.pcap \ > + options:rxq_pcap=hv5/vif-north1-rx.pcap \ > + ofport-request=44 > + > +as hv5 ovs-vsctl add-port br-int vif-north2 -- \ > + set Interface vif-north2 external-ids:iface-id=lp-north2 \ > + options:tx_pcap=hv5/vif-north2-tx.pcap \ > + options:rxq_pcap=hv5/vif-north2-rx.pcap \ > + ofport-request=45 > + > +as hv5 ovs-vsctl add-port br-int vif-north3 -- \ > + set Interface vif-north3 external-ids:iface-id=lp-north3 \ > + options:tx_pcap=hv5/vif-north3-tx.pcap \ > + options:rxq_pcap=hv5/vif-north3-rx.pcap \ > + ofport-request=46 > + > +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2 > +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3 > +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4 > + > +ovn-nbctl --wait=sb sync > +OVN_POPULATE_ARP > + > +vif_to_ls () { > + case ${1} in dnl ( > + vif?[[11]]) echo ls ;; dnl ( > + vif-north1) echo ls-north1 ;; dnl ( > + vif-north2) echo ls-north2 ;; dnl ( > + vif-north3) echo ls-north3 ;; dnl ( > + *) AT_FAIL_IF([:]) ;; > + esac > +} > + > +vif_to_hv () { > + case ${1} in dnl ( > + vif[[1]]?) echo hv1 ;; dnl ( > + vif-north1) echo hv5 ;; dnl ( > + vif-north2) echo hv5 ;; dnl ( > + vif-north3) echo hv5 ;; dnl ( > + *) AT_FAIL_IF([:]) ;; > + esac > +} > + > +vif_to_lrp () { > + case ${1} in dnl ( > + vif?[[11]]) echo DR-ls ;; dnl ( > + *) AT_FAIL_IF([:]) ;; > + esac > + > +} > + > +ip_to_hex() { > + printf "%02x%02x%02x%02x" "${@}" > +} > + > +# test_arp INPORT SHA SPA TPA > +# > +# Causes a packet to be received on INPORT. The packet is an ARP > +# request with SHA, SPA, and TPA as specified. > +test_arp() { > + local inport=$1 sha=$2 spa=$3 tpa=$4 > + local request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa} > + hv=`vif_to_hv $inport` > + as $hv ovs-appctl netdev-dummy/receive $inport $request > +} > + > + > +test_ip() { > + # This packet has bad checksums but logical L3 routing doesn't check. > + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5} outport=${6} > + local packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000 > + shift; shift; shift; shift; shift > + hv=`vif_to_hv $inport` > + as $hv ovs-appctl netdev-dummy/receive $inport $packet > + in_ls=`vif_to_ls $inport` > + for outport; do > + out_ls=`vif_to_ls $outport` > + if test $in_ls = $out_ls; then > + # Ports on the same logical switch receive exactly the same packet. > + echo $packet > + else > + # Routing decrements TTL and updates source and dest MAC > + # (and checksum). > + # For North-South, packet will come via gateway chassis, i.e hv3 > + if test $inport = vif-north1; then > + echo f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected > + fi > + if test $outport = vif-north1; then > + echo f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected > + fi > + if test $outport = vif-north2; then > + echo f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected > + fi > + if test $outport = vif-north3; then > + echo f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected > + fi > + fi >> $outport.expected > + done > +} > + > +echo "------ OVN dump ------" > +ovn-nbctl show > +ovn-sbctl show > +ovn-sbctl list port_binding > +ovn-sbctl list mac_binding > +ovn-sbctl list datapath_binding > + > +ovn-sbctl dump-flows DR > +ovn-sbctl dump-flows S1 > +ovn-sbctl dump-flows ls > + > +echo "------ hv1 dump ------" > +as hv1 ovs-vsctl show > +as hv1 ovs-vsctl list Open_Vswitch > +as hv1 ovs-ofctl dump-flows br-int > + > +echo "------ hv2 dump ------" > +as hv2 ovs-vsctl show > +as hv2 ovs-vsctl list Open_Vswitch > +as hv2 ovs-ofctl dump-flows br-int > + > +echo "------ hv3 dump ------" > +as hv3 ovs-vsctl show > +as hv3 ovs-vsctl list Open_Vswitch > +as hv3 ovs-ofctl dump-flows br-int > + > +echo "------ hv4 dump ------" > +as hv4 ovs-vsctl show > +as hv4 ovs-vsctl list Open_Vswitch > +as hv5 ovs-ofctl dump-flows br-int > + > +# N-S with lp-north1 > +echo "Send Dummy ARP" > +sip=`ip_to_hex 172 16 1 10` > +tip=`ip_to_hex 172 16 1 50` > +test_arp vif-north1 f0f000000011 $sip $tip > + > +echo "Send traffic North to South" > +sip=`ip_to_hex 172 16 1 10` > +dip=`ip_to_hex 20 0 0 10` > +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11 > +# Confirm that North to south traffic works fine. > +OVN_CHECK_PACKETS([hv1/vif11-tx.pcap], [vif11.expected]) > + > +echo "Send traffic South to North" > +sip=`ip_to_hex 20 0 0 10` > +dip=`ip_to_hex 172 16 1 10` > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1 > +# Confirm that South to North traffic works fine. > +OVN_CHECK_PACKETS([hv5/vif-north1-tx.pcap], [vif-north1.expected]) > + > +# N-S with lp-north2 > +echo "Send Dummy ARP" > +sip=`ip_to_hex 10 0 0 10` > +tip=`ip_to_hex 10 0 0 50` > +test_arp vif-north2 f0f000000022 $sip $tip > + > +echo "Send traffic South to North" > +sip=`ip_to_hex 20 0 0 10` > +dip=`ip_to_hex 10 0 0 10` > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2 > +# Confirm that South to North traffic works fine. > +OVN_CHECK_PACKETS([hv5/vif-north2-tx.pcap], [vif-north2.expected]) > + > +# N-S with lp-north3 > +echo "Send Dummy ARP" > +sip=`ip_to_hex 192 168 0 10` > +tip=`ip_to_hex 192 168 0 50` > +test_arp vif-north3 f0f000000033 $sip $tip > + > +echo "Send traffic South to North" > +sip=`ip_to_hex 20 0 0 10` > +dip=`ip_to_hex 192 168 0 10` > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3 > +# Confirm that South to North traffic works fine. > +OVN_CHECK_PACKETS([hv5/vif-north3-tx.pcap], [vif-north3.expected]) > + > +AT_CLEANUP > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c > index d67f5c4..912ede7 100644 > --- a/utilities/ovn-nbctl.c > +++ b/utilities/ovn-nbctl.c > @@ -4208,6 +4208,27 @@ done: > return ret; > } > > +static bool > +is_nat_rule_conflict(const struct nbrec_logical_router *lr) > +{ > + int num_l3dgw_ports = 0; > + > + /* TODO: Add a proper validation to confirm that multiple > + * external ips for a logical ip do not belong to same router port. */ > + for (size_t i = 0; i < lr->n_ports; i++) { > + const struct nbrec_logical_router_port *lrp = lr->ports[i]; > + if (lrp->n_gateway_chassis) { > + num_l3dgw_ports++; > + } > + } > + > + if (num_l3dgw_ports > 1) { > + return false; > + } > + > + return true; > +} > + > static void > nbctl_lr_nat_add(struct ctl_context *ctx) > { > @@ -4369,12 +4390,14 @@ nbctl_lr_nat_add(struct ctl_context *ctx) > should_return = true; > } > } else { > - ctl_error(ctx, "a NAT with this type (%s) and %s (%s) " > - "already exists", > - nat_type, > - is_snat ? "logical_ip" : "external_ip", > - is_snat ? new_logical_ip : new_external_ip); > - should_return = true; > + if (is_nat_rule_conflict(lr)) { > + ctl_error(ctx, "a NAT with this type (%s) and %s (%s) " > + "already exists", > + nat_type, > + is_snat ? "logical_ip" : "external_ip", > + is_snat ? new_logical_ip : new_external_ip); > + should_return = true; > + } > } > } > } > -- > 1.8.3.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index 969fbe1..c1cef7d 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -573,6 +573,19 @@ ovn_mcast_group_allocate_key(struct mcast_info *mcast_info) &mcast_info->group_tnlid_hint); } +struct ovn_datapath_l3dgw_port { + + /* OVN northd only needs to know about the logical router gateway port for + * NAT on a distributed router. This "distributed gateway port" is + * populated only when there is a gateway chassis specified for one of + * the ports on the logical router. Otherwise this will be NULL. */ + struct ovn_port *dgw_port; + + /* The "derived" OVN port representing the instance of l3dgw_port on + * the gateway chassis. */ + struct ovn_port *redirect_port; +}; + /* The 'key' comes from nbs->header_.uuid or nbr->header_.uuid or * sb->external_ids:logical-switch. */ struct ovn_datapath { @@ -602,14 +615,9 @@ struct ovn_datapath { /* Multicast data. */ struct mcast_info mcast_info; - /* OVN northd only needs to know about the logical router gateway port for - * NAT on a distributed router. This "distributed gateway port" is - * populated only when there is a gateway chassis specified for one of - * the ports on the logical router. Otherwise this will be NULL. */ - struct ovn_port *l3dgw_port; - /* The "derived" OVN port representing the instance of l3dgw_port on - * the gateway chassis. */ - struct ovn_port *l3redirect_port; + /* L3 distributed gateway ports */ + struct ovn_datapath_l3dgw_port *l3dgw_ports; + size_t n_l3dgw_ports; /* NAT entries configured on the router. */ struct ovn_nat *nat_entries; @@ -814,6 +822,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct ovn_datapath *od) ovn_destroy_tnlids(&od->port_tnlids); destroy_ipam_info(&od->ipam_info); free(od->router_ports); + free(od->l3dgw_ports); destroy_nat_entries(od); free(od->nat_entries); free(od->localnet_ports); @@ -1353,6 +1362,95 @@ struct ovn_port { struct ovs_list list; /* In list of similar records. */ }; +/* Get the l3dgw port corresponding to a logical router port.*/ +static inline struct ovn_datapath_l3dgw_port* +ovn_get_l3dgw_port_from_lrp(const struct ovn_port *op) +{ + struct ovn_datapath *od = op->od; + + if (!op || !op->nbrp) { + return NULL; + } + + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + &(od->l3dgw_ports[iter]); + if (op == l3dgw_port->dgw_port) { + return l3dgw_port; + } + } + + return NULL; +} + +/* Get the l3dgw port corresponding to a logical router port + * with input ip */ +static struct ovn_datapath_l3dgw_port* +ovn_get_l3dgw_port_from_ip(struct ovn_datapath *od, char *ip_s, bool is_v6) +{ + ovs_be32 ip4, mask4; + struct in6_addr ip6, mask6; + + char *error = NULL; + + if (!od || !od->nbr) { + return NULL; + } + + if (is_v6) { + error = ipv6_parse_masked(ip_s, &ip6, &mask6); + } else { + error = ip_parse_masked(ip_s, &ip4, &mask4); + ip4 = ntohl(ip4); + } + + if (error) { + free(error); + return NULL; + } + + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + &(od->l3dgw_ports[iter]); + struct ovn_port *op = l3dgw_port->dgw_port; + struct lport_addresses lrp_networks; + + if (!extract_lrp_networks(op->nbrp, &lrp_networks)) { + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); + VLOG_WARN_RL(&rl, "Extract addresses failed."); + continue; + } + + if (is_v6) { + for (int iter2 = 0; iter2 < lrp_networks.n_ipv6_addrs; iter2++) { + struct ipv6_netaddr *lrp6_addr = + &(lrp_networks.ipv6_addrs[iter2]); + struct in6_addr ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask, + &ip6); + + if (ipv6_addr_equals(&ip6_mask, &(lrp6_addr->network))) { + return l3dgw_port; + } + } + } else { + for (int iter2 = 0; iter2 < lrp_networks.n_ipv4_addrs; iter2++) { + struct ipv4_netaddr *lrp4_addr = + &(lrp_networks.ipv4_addrs[iter2]); + ovs_be32 addr = ntohl(lrp4_addr->addr); + ovs_be32 network = ntohl(lrp4_addr->network); + mask4 = ntohl(lrp4_addr->mask); + ovs_be32 bcast = addr | ~mask4; + + if (ip4 >= network && ip4 < bcast) { + return l3dgw_port; + } + } + } + } + + return NULL; +} + static void ovn_port_set_nb(struct ovn_port *op, const struct nbrec_logical_switch_port *nbsp, @@ -2246,13 +2344,12 @@ join_logical_ports(struct northd_context *ctx, "on L3 gateway router", nbrp->name); continue; } - if (od->l3dgw_port || od->l3redirect_port) { + if (od->n_l3dgw_ports) { static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); - VLOG_WARN_RL(&rl, "Bad configuration: multiple " - "distributed gateway ports on logical " - "router %s", od->nbr->name); - continue; + VLOG_DBG_RL(&rl, "Multiple ports with " + "redirect-chassis on same " + "logical router %s", od->nbr->name); } char *redirect_name = @@ -2274,8 +2371,12 @@ join_logical_ports(struct northd_context *ctx, /* Set l3dgw_port and l3redirect_port in od, for later * use during flow creation. */ - od->l3dgw_port = op; - od->l3redirect_port = crp; + od->l3dgw_ports = xrealloc(od->l3dgw_ports, + sizeof *od->l3dgw_ports * + (od->n_l3dgw_ports + 1)); + (od->l3dgw_ports[od->n_l3dgw_ports]).dgw_port = op; + (od->l3dgw_ports[od->n_l3dgw_ports]).redirect_port = crp; + od->n_l3dgw_ports++; } } } @@ -2431,7 +2532,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n) /* Determine whether this NAT rule satisfies the conditions for * distributed NAT processing. */ - if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat") + if (op->od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") && nat->logical_port && nat->external_mac) { /* Distributed NAT rule. */ if (eth_addr_from_string(nat->external_mac, &mac)) { @@ -2491,11 +2592,13 @@ get_nat_addresses(const struct ovn_port *op, size_t *n) sset_destroy(&all_ips_v6); if (central_ip_address) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); /* Gratuitous ARP for centralized NAT rules on distributed gateway * ports should be restricted to the gateway chassis. */ - if (op->od->l3redirect_port) { + if (l3dgw_port) { ds_put_format(&c_addresses, " is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } addresses[n_nats++] = ds_steal_cstr(&c_addresses); @@ -2988,7 +3091,7 @@ ovn_port_update_sbrec(struct northd_context *ctx, char **nats = NULL; if (nat_addresses && !strcmp(nat_addresses, "router")) { if (op->peer && op->peer->od - && (chassis || op->peer->od->l3redirect_port)) { + && (chassis || op->peer->od->n_l3dgw_ports)) { nats = get_nat_addresses(op->peer, &n_nats); } /* Only accept manual specification of ethernet address @@ -3024,11 +3127,11 @@ ovn_port_update_sbrec(struct northd_context *ctx, * sending the GARPs for the router port IPs. * */ bool add_router_port_garp = false; - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port && - op->peer->od->l3redirect_port && + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL; + if (op->peer && op->peer->nbrp && op->peer->od->n_l3dgw_ports && (smap_get_bool(&op->peer->nbrp->options, "reside-on-redirect-chassis", false) || - op->peer == op->peer->od->l3dgw_port)) { + (l3dgw_port = ovn_get_l3dgw_port_from_lrp(op->peer)))) { add_router_port_garp = true; } else if (chassis && op->od->n_localnet_ports) { add_router_port_garp = true; @@ -3043,9 +3146,12 @@ ovn_port_update_sbrec(struct northd_context *ctx, op->peer->lrp_networks.ipv4_addrs[i].addr_s); } - if (op->peer->od->l3redirect_port) { + if (op->peer->od->n_l3dgw_ports) { + if (!l3dgw_port) { + l3dgw_port = &(op->peer->od->l3dgw_ports[0]); + } ds_put_format(&garp_info, " is_chassis_resident(%s)", - op->peer->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } n_nats++; @@ -4637,7 +4743,6 @@ build_lswitch_input_port_sec_op( struct ovn_port *op, struct hmap *lflows, struct ds *actions, struct ds *match) { - if (!op->nbsp) { return; } @@ -5937,13 +6042,16 @@ build_lrouter_groups__(struct hmap *ports, struct ovn_datapath *od) { ovs_assert((od && od->nbr && od->lr_group)); - if (od->l3dgw_port && od->l3redirect_port) { + for (int i = 0; i < od->n_l3dgw_ports; i++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + &(od->l3dgw_ports[i]); + /* It's a logical router with gateway port. If it * has HA_Chassis_Group associated to it in SB DB, then store the * ha chassis group name. */ - if (od->l3redirect_port->sb->ha_chassis_group) { + if (l3dgw_port->redirect_port->sb->ha_chassis_group) { sset_add(&od->lr_group->ha_chassis_groups, - od->l3redirect_port->sb->ha_chassis_group->name); + l3dgw_port->redirect_port->sb->ha_chassis_group->name); } } @@ -7182,17 +7290,18 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, ds_clear(match); ds_put_format(match, "eth.dst == "ETH_ADDR_FMT, ETH_ADDR_ARGS(mac)); - if (op->peer->od->l3dgw_port - && op->peer->od->l3redirect_port - && op->od->n_localnet_ports) { + if (op->peer->od->n_l3dgw_ports && + op->od->n_localnet_ports) { bool add_chassis_resident_check = false; - if (op->peer == op->peer->od->l3dgw_port) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op->peer); + if (l3dgw_port) { /* The peer of this port represents a distributed * gateway port. The destination lookup flow for the * router's distributed gateway port MAC address should * only be programmed on the gateway chassis. */ add_chassis_resident_check = true; - } else { + } else if (op->peer->od->n_l3dgw_ports == 1) { /* Check if the option 'reside-on-redirect-chassis' * is set to true on the peer port. If set to true * and if the logical switch has a localnet port, it @@ -7200,6 +7309,11 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, * this logical switch should be run on the chassis * hosting the gateway port. */ + + /* reside-on-redirect-chassis is supported only for + * logical routers with single l3dgw port. + */ + l3dgw_port = &(op->peer->od->l3dgw_ports[0]); add_chassis_resident_check = smap_get_bool( &op->peer->nbrp->options, "reside-on-redirect-chassis", false); @@ -7207,7 +7321,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, if (add_chassis_resident_check) { ds_put_format(match, " && is_chassis_resident(%s)", - op->peer->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } } @@ -7220,8 +7334,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op, /* Add ethernet addresses specified in NAT rules on * distributed logical routers. */ - if (op->peer->od->l3dgw_port - && op->peer == op->peer->od->l3dgw_port) { + if (ovn_get_l3dgw_port_from_lrp(op->peer)) { for (int j = 0; j < op->peer->od->nbr->n_nat; j++) { const struct nbrec_nat *nat = op->peer->od->nbr->nat[j]; @@ -8357,34 +8470,44 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od, build_empty_lb_event_flow(od, lflows, lb_vip, lb, S_ROUTER_IN_DNAT, meter_groups); - /* A match and actions for new connections. */ - char *new_match = xasprintf("ct.new && %s", ds_cstr(match)); - if (lb_force_snat_ip) { - char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s", - ds_cstr(actions)); - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, - new_match, new_actions, &lb->header_); - free(new_actions); - } else { - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, - new_match, ds_cstr(actions), &lb->header_); - } + for (int i = 0; i < od->n_l3dgw_ports; i++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[i]); + + /* A match and actions for new connections. */ + char *new_match = xasprintf("ct.new && %s && inport == %s && " + "is_chassis_resident(%s)", ds_cstr(match), + l3dgw_port->dgw_port->json_key, + l3dgw_port->redirect_port->json_key); + if (lb_force_snat_ip) { + char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s", + ds_cstr(actions)); + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, + new_match, new_actions, &lb->header_); + free(new_actions); + } else { + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, + new_match, ds_cstr(actions), &lb->header_); + } + + /* A match and actions for established connections. */ + char *est_match = xasprintf("ct.est && %s && inport == %s && " + "is_chassis_resident(%s)", ds_cstr(match), + l3dgw_port->dgw_port->json_key, + l3dgw_port->redirect_port->json_key); + if (lb_force_snat_ip) { + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, + est_match, + "flags.force_snat_for_lb = 1; ct_dnat;", + &lb->header_); + } else { + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, + est_match, "ct_dnat;", &lb->header_); + } - /* A match and actions for established connections. */ - char *est_match = xasprintf("ct.est && %s", ds_cstr(match)); - if (lb_force_snat_ip) { - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, - est_match, - "flags.force_snat_for_lb = 1; ct_dnat;", - &lb->header_); - } else { - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority, - est_match, "ct_dnat;", &lb->header_); + free(new_match); + free(est_match); } - free(new_match); - free(est_match); - const char *ip_match = NULL; if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) { ip_match = "ip4"; @@ -8418,49 +8541,55 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od, ds_destroy(&unsnat_match); } - if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) { + if (!od->n_l3dgw_ports || !lb_vip->n_backends) { return; } - /* Add logical flows to UNDNAT the load balanced reverse traffic in - * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the logical - * router has a gateway router port associated. - */ - struct ds undnat_match = DS_EMPTY_INITIALIZER; - ds_put_format(&undnat_match, "%s && (", ip_match); + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { + + /* Add logical flows to UNDNAT the load balanced reverse traffic in + * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the + * logical router has a gateway router port associated. + */ + struct ds undnat_match = DS_EMPTY_INITIALIZER; + ds_put_format(&undnat_match, "%s && (", ip_match); + + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[iter]); - for (size_t i = 0; i < lb_vip->n_backends; i++) { - struct ovn_lb_backend *backend = &lb_vip->backends[i]; - ds_put_format(&undnat_match, "(%s.src == %s", ip_match, - backend->ip_str); + for (size_t i = 0; i < lb_vip->n_backends; i++) { + struct ovn_lb_backend *backend = &lb_vip->backends[i]; + ds_put_format(&undnat_match, "(%s.src == %s", ip_match, + backend->ip_str); - if (backend->port) { - ds_put_format(&undnat_match, " && %s.src == %d) || ", - proto, backend->port); + if (backend->port) { + ds_put_format(&undnat_match, " && %s.src == %d) || ", + proto, backend->port); + } else { + ds_put_cstr(&undnat_match, ") || "); + } + } + + ds_chomp(&undnat_match, ' '); + ds_chomp(&undnat_match, '|'); + ds_chomp(&undnat_match, '|'); + ds_chomp(&undnat_match, ' '); + ds_put_format(&undnat_match, ") && outport == %s && " + "is_chassis_resident(%s)", + l3dgw_port->dgw_port->json_key, + l3dgw_port->redirect_port->json_key); + if (lb_force_snat_ip) { + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, + ds_cstr(&undnat_match), + "flags.force_snat_for_lb = 1; ct_dnat;", + &lb->header_); } else { - ds_put_cstr(&undnat_match, ") || "); - } - } - - ds_chomp(&undnat_match, ' '); - ds_chomp(&undnat_match, '|'); - ds_chomp(&undnat_match, '|'); - ds_chomp(&undnat_match, ' '); - ds_put_format(&undnat_match, ") && outport == %s && " - "is_chassis_resident(%s)", od->l3dgw_port->json_key, - od->l3redirect_port->json_key); - if (lb_force_snat_ip) { - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, - ds_cstr(&undnat_match), - "flags.force_snat_for_lb = 1; ct_dnat;", - &lb->header_); - } else { - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, - ds_cstr(&undnat_match), "ct_dnat;", - &lb->header_); - } + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, + ds_cstr(&undnat_match), "ct_dnat;", + &lb->header_); + } - ds_destroy(&undnat_match); + ds_destroy(&undnat_match); + } } #define ND_RA_MAX_INTERVAL_MAX 1800 @@ -8582,7 +8711,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od, { struct nbrec_address_set *allowed_ext_ips = nat->allowed_ext_ips; struct nbrec_address_set *exempted_ext_ips = nat->exempted_ext_ips; - bool is_gw_router = !od->l3dgw_port; + bool is_gw_router = !od->n_l3dgw_ports; ovs_assert(allowed_ext_ips || exempted_ext_ips); @@ -8797,9 +8926,11 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op, * upstream MAC learning points to the gateway chassis. * Also need to avoid generation of multiple ARP responses * from different chassis. */ - if (op->od->l3redirect_port) { + if (op->od->n_l3dgw_ports) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); ds_put_format(&match, "is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } } @@ -8986,6 +9117,8 @@ build_adm_ctrl_flows_for_lrouter_port( struct ovn_port *op, struct hmap *lflows, struct ds *match, struct ds *actions) { + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL; + if (op->nbrp) { if (!lrport_is_enabled(op->nbrp)) { /* Drop packets from disabled logical ports (since logical flow @@ -9016,12 +9149,12 @@ build_adm_ctrl_flows_for_lrouter_port( ds_clear(match); ds_put_format(match, "eth.dst == %s && inport == %s", op->lrp_networks.ea_s, op->json_key); - if (op->od->l3dgw_port && op == op->od->l3dgw_port - && op->od->l3redirect_port) { + l3dgw_port = ovn_get_l3dgw_port_from_lrp(op); + if (l3dgw_port) { /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s * should only be received on the gateway chassis. */ ds_put_format(match, " && is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50, ds_cstr(match), ds_cstr(actions), @@ -9141,6 +9274,8 @@ build_neigh_learning_flows_for_lrouter_port( /* Check if we need to learn mac-binding from ARP requests. */ for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); if (!learn_from_arp_request) { /* ARP request to this address should always get learned, * so add a priority-110 flow to set @@ -9153,10 +9288,9 @@ build_neigh_learning_flows_for_lrouter_port( op->lrp_networks.ipv4_addrs[i].network_s, op->lrp_networks.ipv4_addrs[i].plen, op->lrp_networks.ipv4_addrs[i].addr_s); - if (op->od->l3dgw_port && op == op->od->l3dgw_port - && op->od->l3redirect_port) { + if (l3dgw_port) { ds_put_format(match, " && is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT " = lookup_arp(inport, arp.spa, arp.sha); " @@ -9173,10 +9307,9 @@ build_neigh_learning_flows_for_lrouter_port( op->json_key, op->lrp_networks.ipv4_addrs[i].network_s, op->lrp_networks.ipv4_addrs[i].plen); - if (op->od->l3dgw_port && op == op->od->l3dgw_port - && op->od->l3redirect_port) { + if (l3dgw_port) { ds_put_format(match, " && is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } ds_clear(actions); ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT @@ -9601,7 +9734,10 @@ build_arp_resolve_flows_for_lrouter_port( } } - if (!op->derived && op->od->l3redirect_port) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); + + if (!op->derived && l3dgw_port) { const char *redirect_type = smap_get(&op->nbrp->options, "redirect-type"); if (redirect_type && !strcasecmp(redirect_type, "bridged")) { @@ -9614,7 +9750,7 @@ build_arp_resolve_flows_for_lrouter_port( ds_clear(match); ds_put_format(match, "outport == %s && " "!is_chassis_resident(%s)", op->json_key, - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); ds_clear(actions); ds_put_format(actions, "eth.dst = %s; next;", op->lrp_networks.ea_s); @@ -9925,11 +10061,14 @@ build_check_pkt_len_flows_for_lrouter( ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1", "next;"); - if (od->l3dgw_port && od->l3redirect_port) { + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { int gw_mtu = 0; - if (od->l3dgw_port->nbrp) { - gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options, - "gateway_mtu", 0); + struct ovn_datapath_l3dgw_port *l3dgw_port = + &(od->l3dgw_ports[iter]); + + if (l3dgw_port->dgw_port->nbrp) { + gw_mtu = smap_get_int(&(l3dgw_port->dgw_port->nbrp->options), + "gateway_mtu", 0); } /* Add the flows only if gateway_mtu is configured. */ if (gw_mtu <= 0) { @@ -9937,20 +10076,20 @@ build_check_pkt_len_flows_for_lrouter( } ds_clear(match); - ds_put_format(match, "outport == %s", od->l3dgw_port->json_key); - + ds_put_format(match, "outport == %s", + l3dgw_port->dgw_port->json_key); ds_clear(actions); ds_put_format(actions, REGBIT_PKT_LARGER" = check_pkt_larger(%d);" " next;", gw_mtu + VLAN_ETH_HEADER_LEN); ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50, ds_cstr(match), ds_cstr(actions), - &od->l3dgw_port->nbrp->header_); + &(l3dgw_port->dgw_port->nbrp->header_)); for (size_t i = 0; i < od->nbr->n_ports; i++) { struct ovn_port *rp = ovn_port_find(ports, od->nbr->ports[i]->name); - if (!rp || rp == od->l3dgw_port) { + if (rp == l3dgw_port->dgw_port) { continue; } @@ -9958,7 +10097,8 @@ build_check_pkt_len_flows_for_lrouter( ds_clear(match); ds_put_format(match, "inport == %s && outport == %s" " && ip4 && "REGBIT_PKT_LARGER, - rp->json_key, od->l3dgw_port->json_key); + rp->json_key, + l3dgw_port->dgw_port->json_key); ds_clear(actions); /* Set icmp4.frag_mtu to gw_mtu */ @@ -9987,7 +10127,8 @@ build_check_pkt_len_flows_for_lrouter( ds_clear(match); ds_put_format(match, "inport == %s && outport == %s" " && ip6 && "REGBIT_PKT_LARGER, - rp->json_key, od->l3dgw_port->json_key); + rp->json_key, l3dgw_port->dgw_port + ->json_key); ds_clear(actions); /* Set icmp6.frag_mtu to gw_mtu */ @@ -10029,11 +10170,14 @@ build_gateway_redirect_flows_for_lrouter( struct ds *match, struct ds *actions) { if (od->nbr) { - if (od->l3dgw_port && od->l3redirect_port) { + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) { const struct ovsdb_idl_row *stage_hint = NULL; - if (od->l3dgw_port->nbrp) { - stage_hint = &od->l3dgw_port->nbrp->header_; + struct ovn_port *l3dgw_port = (od->l3dgw_ports[iter]).dgw_port; + struct ovn_port *l3redirect_port = + (od->l3dgw_ports[iter]).redirect_port; + if (l3dgw_port->nbrp) { + stage_hint = &l3dgw_port->nbrp->header_; } /* For traffic with outport == l3dgw_port, if the @@ -10041,13 +10185,12 @@ build_gateway_redirect_flows_for_lrouter( * rule, then the traffic is redirected to the central * instance of the l3dgw_port. */ ds_clear(match); - ds_put_format(match, "outport == %s", - od->l3dgw_port->json_key); + ds_put_format(match, "outport == %s", l3dgw_port->json_key); ds_clear(actions); ds_put_format(actions, "outport = %s; next;", - od->l3redirect_port->json_key); - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50, - ds_cstr(match), ds_cstr(actions), + l3redirect_port->json_key); + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, + 50, ds_cstr(match), ds_cstr(actions), stage_hint); } @@ -10278,16 +10421,17 @@ build_ipv6_input_flows_for_lrouter_port( /* ND reply. These flows reply to ND solicitations for the * router's own IP address. */ for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); ds_clear(match); - if (op->od->l3dgw_port && op == op->od->l3dgw_port - && op->od->l3redirect_port) { + if (l3dgw_port) { /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s * should only be sent from the gateway chassi, so that * upstream MAC learning points to the gateway chassis. * Also need to avoid generation of multiple ND replies * from different chassis. */ ds_put_format(match, "is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } build_lrouter_nd_flow(op->od, op, "nd_na_router", @@ -10299,7 +10443,7 @@ build_ipv6_input_flows_for_lrouter_port( /* UDP/TCP/SCTP port unreachable */ if (!smap_get(&op->od->nbr->options, "chassis") - && !op->od->l3dgw_port) { + && !op->od->n_l3dgw_ports) { for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { ds_clear(match); ds_put_format(match, @@ -10515,10 +10659,12 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, op->lrp_networks.ipv4_addrs[i].network_s, op->lrp_networks.ipv4_addrs[i].plen); - if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer + if (op->od->n_l3dgw_ports && op->peer && op->peer->od->n_localnet_ports) { bool add_chassis_resident_check = false; - if (op == op->od->l3dgw_port) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); + if (l3dgw_port) { /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s * should only be sent from the gateway chassis, so that * upstream MAC learning points to the gateway chassis. @@ -10534,6 +10680,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, * hosting the gateway port and it should reply to the * ARP requests for the router port IPs. */ + l3dgw_port = &(op->od->l3dgw_ports[0]); add_chassis_resident_check = smap_get_bool( &op->nbrp->options, "reside-on-redirect-chassis", false); @@ -10541,7 +10688,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, if (add_chassis_resident_check) { ds_put_format(match, " && is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } } @@ -10559,9 +10706,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, const char *ip_address; SSET_FOR_EACH (ip_address, &all_ips_v4) { ds_clear(match); - if (op == op->od->l3dgw_port) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); + if (l3dgw_port) { ds_put_format(match, "is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } build_lrouter_arp_flow(op->od, op, @@ -10571,9 +10720,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, SSET_FOR_EACH (ip_address, &all_ips_v6) { ds_clear(match); - if (op == op->od->l3dgw_port) { + struct ovn_datapath_l3dgw_port *l3dgw_port = + ovn_get_l3dgw_port_from_lrp(op); + if (l3dgw_port) { ds_put_format(match, "is_chassis_resident(%s)", - op->od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } build_lrouter_nd_flow(op->od, op, "nd_na", @@ -10585,7 +10736,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, sset_destroy(&all_ips_v6); if (!smap_get(&op->od->nbr->options, "chassis") - && !op->od->l3dgw_port) { + && !op->od->n_l3dgw_ports) { /* UDP/TCP/SCTP port unreachable. */ for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { ds_clear(match); @@ -10662,7 +10813,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, * exception is on the l3dgw_port where we might need to use a * different ETH address. */ - if (op != op->od->l3dgw_port) { + if (!ovn_get_l3dgw_port_from_lrp(op)) { return; } @@ -10727,7 +10878,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, /* NAT rules are only valid on Gateway routers and routers with * l3dgw_port (router has a port with gateway chassis * specified). */ - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) { + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) { return; } @@ -10814,11 +10965,27 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, } } + struct ovn_datapath_l3dgw_port *l3dgw_port = + &((od)->l3dgw_ports[0]); + if (od->n_l3dgw_ports) { + /* Get the L3DGW port only for distributed router. */ + l3dgw_port = ovn_get_l3dgw_port_from_ip(od, nat->external_ip, + is_v6); + if (!l3dgw_port) { + static struct vlog_rate_limit rl = + VLOG_RATE_LIMIT_INIT(5, 1); + VLOG_WARN_RL(&rl, "Could not map external ip: %s to a " + "gateway port "UUID_FMT"", nat->external_ip, + UUID_ARGS(&od->key)); + continue; + } + } + /* For distributed router NAT, determine whether this NAT rule * satisfies the conditions for distributed NAT processing. */ bool distributed = false; struct eth_addr mac; - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") && + if (od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") && nat->logical_port && nat->external_mac) { if (eth_addr_from_string(nat->external_mac, &mac)) { distributed = true; @@ -10842,7 +11009,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, * egress pipeline. */ if (!strcmp(nat->type, "snat") || !strcmp(nat->type, "dnat_and_snat")) { - if (!od->l3dgw_port) { + if (!od->n_l3dgw_ports) { /* Gateway router. */ ds_clear(match); ds_clear(actions); @@ -10870,12 +11037,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, " && inport == %s", is_v6 ? "6" : "4", nat->external_ip, - od->l3dgw_port->json_key); - if (!distributed && od->l3redirect_port) { + l3dgw_port->dgw_port->json_key); + if (!distributed && od->n_l3dgw_ports) { /* Flows for NAT rules that are centralized are only * programmed on the gateway chassis. */ ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } if (!strcmp(nat->type, "dnat_and_snat") && stateless) { @@ -10897,7 +11064,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, * to a logical IP address. */ if (!strcmp(nat->type, "dnat") || !strcmp(nat->type, "dnat_and_snat")) { - if (!od->l3dgw_port) { + if (!od->n_l3dgw_ports) { /* Gateway router. */ /* Packet when it goes from the initiator to destination. * We need to set flags.loopback because the router can @@ -10947,12 +11114,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, " && inport == %s", is_v6 ? "6" : "4", nat->external_ip, - od->l3dgw_port->json_key); - if (!distributed && od->l3redirect_port) { + l3dgw_port->dgw_port->json_key); + if (!distributed && od->n_l3dgw_ports) { /* Flows for NAT rules that are centralized are only * programmed on the gateway chassis. */ ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } ds_clear(actions); if (allowed_ext_ips || exempted_ext_ips) { @@ -10979,12 +11146,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, } /* ARP resolve for NAT IPs. */ - if (od->l3dgw_port) { + if (od->n_l3dgw_ports) { if (!strcmp(nat->type, "snat")) { ds_clear(match); ds_put_format( match, "inport == %s && %s == %s", - od->l3dgw_port->json_key, + l3dgw_port->dgw_port->json_key, is_v6 ? "ip6.src" : "ip4.src", nat->external_ip); ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT, 120, ds_cstr(match), "next;", @@ -10995,14 +11162,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, ds_clear(match); ds_put_format( match, "outport == %s && %s == %s", - od->l3dgw_port->json_key, + l3dgw_port->dgw_port->json_key, is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4, nat->external_ip); ds_clear(actions); ds_put_format( actions, "eth.dst = %s; next;", distributed ? nat->external_mac : - od->l3dgw_port->lrp_networks.ea_s); + l3dgw_port->dgw_port->lrp_networks.ea_s); ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 100, ds_cstr(match), @@ -11025,19 +11192,19 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, * Note that this only applies for NAT on a distributed router. * Undo DNAT on a gateway router is done in the ingress DNAT * pipeline stage. */ - if (od->l3dgw_port && (!strcmp(nat->type, "dnat") + if (od->n_l3dgw_ports && (!strcmp(nat->type, "dnat") || !strcmp(nat->type, "dnat_and_snat"))) { ds_clear(match); ds_put_format(match, "ip && ip%s.src == %s" " && outport == %s", is_v6 ? "6" : "4", nat->logical_ip, - od->l3dgw_port->json_key); - if (!distributed && od->l3redirect_port) { + l3dgw_port->dgw_port->json_key); + if (!distributed && od->n_l3dgw_ports) { /* Flows for NAT rules that are centralized are only * programmed on the gateway chassis. */ ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } ds_clear(actions); if (distributed) { @@ -11062,7 +11229,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, * address. */ if (!strcmp(nat->type, "snat") || !strcmp(nat->type, "dnat_and_snat")) { - if (!od->l3dgw_port) { + if (!od->n_l3dgw_ports) { /* Gateway router. */ ds_clear(match); ds_put_format(match, "ip && ip%s.src == %s", @@ -11105,13 +11272,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, " && outport == %s", is_v6 ? "6" : "4", nat->logical_ip, - od->l3dgw_port->json_key); - if (!distributed && od->l3redirect_port) { + l3dgw_port->dgw_port->json_key); + if (!distributed && od->n_l3dgw_ports) { /* Flows for NAT rules that are centralized are only * programmed on the gateway chassis. */ priority += 128; ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } ds_clear(actions); @@ -11160,14 +11327,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, */ ds_clear(actions); ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;", - od->l3dgw_port->lrp_networks.ea_s); + l3dgw_port->dgw_port->lrp_networks.ea_s); ds_clear(match); ds_put_format(match, "eth.dst == "ETH_ADDR_FMT" && inport == %s" " && is_chassis_resident(\"%s\")", ETH_ADDR_ARGS(mac), - od->l3dgw_port->json_key, + l3dgw_port->dgw_port->json_key, nat->logical_port); ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50, ds_cstr(match), ds_cstr(actions), @@ -11190,7 +11357,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, "ip%s.src == %s && outport == %s && " "is_chassis_resident(\"%s\")", is_v6 ? "6" : "4", nat->logical_ip, - od->l3dgw_port->json_key, nat->logical_port); + l3dgw_port->dgw_port->json_key, + nat->logical_port); ds_put_format(actions, "eth.src = %s; %s = %s; next;", nat->external_mac, is_v6 ? REG_SRC_IPV6 : REG_SRC_IPV4, @@ -11205,16 +11373,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, * gateway port have ip.dst matching a NAT external IP, then * loop a clone of the packet back to the beginning of the * ingress pipeline with inport = outport. */ - if (od->l3dgw_port) { + if (od->n_l3dgw_ports) { /* Distributed router. */ ds_clear(match); ds_put_format(match, "ip%s.dst == %s && outport == %s", is_v6 ? "6" : "4", nat->external_ip, - od->l3dgw_port->json_key); + l3dgw_port->dgw_port->json_key); if (!distributed) { ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); + l3dgw_port->redirect_port->json_key); } else { ds_put_format(match, " && is_chassis_resident(\"%s\")", nat->logical_port); @@ -11238,7 +11406,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, } /* Handle force SNAT options set in the gateway router. */ - if (!od->l3dgw_port) { + if (!od->n_l3dgw_ports) { if (dnat_force_snat_ip) { if (od->dnat_force_snat_addrs.n_ipv4_addrs) { build_lrouter_force_snat_flows(lflows, od, "4", @@ -11277,7 +11445,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, /* Load balancing and packet defrag are only valid on * Gateway routers or router with gateway port. */ - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) { + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) { sset_destroy(&nat_entries); return; } @@ -11347,11 +11515,6 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, prio = 120; } - if (od->l3redirect_port && - (lb_vip->n_backends || !lb_vip->empty_backend_rej)) { - ds_put_format(match, " && is_chassis_resident(%s)", - od->l3redirect_port->json_key); - } add_router_lb_flow(lflows, od, match, actions, prio, lb_force_snat_ip, lb_vip, proto, nb_lb, meter_groups, &nat_entries); diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at index 91eb9a3..2c01f20 100644 --- a/tests/ovn-northd.at +++ b/tests/ovn-northd.at @@ -711,7 +711,7 @@ ovn_start ovn-sbctl chassis-add gw1 geneve 127.0.0.1 ovn-nbctl lr-add R1 -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64 ovn-nbctl ls-add S1 ovn-nbctl lsp-add S1 S1-R1 @@ -752,13 +752,13 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat 172.16.1.1 echo echo "IPv6: stateful" -ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2 +ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat 3000::c 1000::3 check_flow_match_sets 2 2 2 0 0 0 0 -ovn-nbctl lr-nat-del R1 dnat_and_snat fd01::1 +ovn-nbctl lr-nat-del R1 dnat_and_snat 3000::c echo echo "IPv6: stateless" -ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2 +ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat 3000::c 1000::3 check_flow_match_sets 2 0 0 0 0 2 2 AT_CLEANUP @@ -769,7 +769,7 @@ ovn_start ovn-sbctl chassis-add gw1 geneve 127.0.0.1 ovn-nbctl lr-add R1 -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64 ovn-nbctl ls-add S1 ovn-nbctl lsp-add S1 S1-R1 @@ -2385,3 +2385,346 @@ ovn-nbctl destroy bfd $uuid check_row_count bfd 2 AT_CLEANUP + +AT_SETUP([ovn-northd -- lr multiple gw ports]) +ovn_start + +# Logical network: +# 1 LR, 3 Logical Switches, +# 1 gateway chassis attached to each corresponding LRP. +# +# | S1 (gw1) +# | +# ls ---- DR -- S3 (gw3) +# (20.0.0.0/24) | +# | S2 (gw2) +# +# We will validate basic LR logical flows. + +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 + +ovn-nbctl lr-add DR +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.0/24 +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.0/24 +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 + +ovn-nbctl ls-add S1 +ovn-nbctl lsp-add S1 S1-DR +ovn-nbctl lsp-set-type S1-DR router +ovn-nbctl lsp-set-addresses S1-DR router +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 + +ovn-nbctl ls-add S2 +ovn-nbctl lsp-add S2 S2-DR +ovn-nbctl lsp-set-type S2-DR router +ovn-nbctl lsp-set-addresses S2-DR router +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 + +ovn-nbctl ls-add S3 +ovn-nbctl lsp-add S3 S3-DR +ovn-nbctl lsp-set-type S3-DR router +ovn-nbctl lsp-set-addresses S3-DR router +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 + +ovn-nbctl ls-add ls +ovn-nbctl lsp-add ls ls-DR +ovn-nbctl lsp-set-type ls-DR router +ovn-nbctl lsp-set-addresses ls-DR router +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls + +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 + +ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows DR + +# Check the flows in lr_in_lookup_neighbor stage +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR | wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S1 | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S2 | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S3 | wc -l], [0], [1 +]) + +# Check the flows in lr_in_gw_redirect stage +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR | wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S1 | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S2 | wc -l], [0], [1 +]) +AT_CLEANUP + +AT_SETUP([ovn-northd -- lr multiple gw ports NAT]) +ovn_start + +# Logical network: +# 1 LR, 3 Logical Switches, +# 1 gateway chassis attached to each corresponding LRP. +# +# | S1 (gw1) +# | +# ls ---- DR -- S3 (gw3) +# (20.0.0.0/24) | +# | S2 (gw2) +# +# We will validate basic SNAT, DNAT and DNAT_AND_SNAT with +# multiple distributed gateway LRPs. +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 + +ovn-nbctl lr-add DR +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24 +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 + +ovn-nbctl ls-add S1 +ovn-nbctl lsp-add S1 S1-DR +ovn-nbctl lsp-set-type S1-DR router +ovn-nbctl lsp-set-addresses S1-DR router +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 + +ovn-nbctl ls-add S2 +ovn-nbctl lsp-add S2 S2-DR +ovn-nbctl lsp-set-type S2-DR router +ovn-nbctl lsp-set-addresses S2-DR router +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 + +ovn-nbctl ls-add S3 +ovn-nbctl lsp-add S3 S3-DR +ovn-nbctl lsp-set-type S3-DR router +ovn-nbctl lsp-set-addresses S3-DR router +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 + +ovn-nbctl ls-add ls +ovn-nbctl lsp-add ls ls-DR +ovn-nbctl lsp-set-type ls-DR router +ovn-nbctl lsp-set-addresses ls-DR router +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls + +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 + +ovn-nbctl --wait=sb sync + +# Configure SNAT +ovn-nbctl lr-nat-add DR snat 172.16.1.1 20.0.0.10 +ovn-nbctl lr-nat-add DR snat 10.0.0.1 20.0.0.10 +ovn-nbctl lr-nat-add DR snat 192.168.0.1 20.0.0.10 + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.1" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.1)"| wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.1" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.1)"| wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.1" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.1)"| wc -l], [0], [1 +]) + +ovn-nbctl lr-nat-del DR snat 20.0.0.10 +ovn-nbctl lr-nat-del DR snat 20.0.0.10 +ovn-nbctl lr-nat-del DR snat 20.0.0.10 +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0 +]) + +# Configure DNAT +ovn-nbctl lr-nat-add DR dnat 172.16.1.10 20.0.0.10 +ovn-nbctl lr-nat-add DR dnat 10.0.0.10 20.0.0.10 +ovn-nbctl lr-nat-add DR dnat 192.168.0.10 20.0.0.10 +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1 +]) + +ovn-nbctl lr-nat-del DR dnat 172.16.1.10 +ovn-nbctl lr-nat-del DR dnat 10.0.0.10 +ovn-nbctl lr-nat-del DR dnat 192.168.0.10 + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0 +]) + +# Configure DNAT_AND_SNAT +ovn-nbctl lr-nat-add DR dnat_and_snat 172.16.1.10 20.0.0.10 +ovn-nbctl lr-nat-add DR dnat_and_snat 10.0.0.10 20.0.0.10 +ovn-nbctl lr-nat-add DR dnat_and_snat 192.168.0.10 20.0.0.10 + +ovn-sbctl dump-flows DR + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.10)"| wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.10)"| wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.10)"| wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1 +]) + +ovn-nbctl lr-nat-del DR dnat_and_snat 172.16.1.10 +ovn-nbctl lr-nat-del DR dnat_and_snat 10.0.0.10 +ovn-nbctl lr-nat-del DR dnat_and_snat 192.168.0.10 + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0 +]) +AT_CLEANUP + +AT_SETUP([ovn-northd -- lr multiple gw ports LB]) +ovn_start + +# Logical network: +# 1 LR, 3 Logical Switches, +# 1 gateway chassis attached to each corresponding LRP. +# +# | S1 (gw1) +# | +# ls ---- DR -- S3 (gw3) +# (20.0.0.0/24) | +# | S2 (gw2) +# +# We will validate LB with multiple distributed gateway LRPs. +ovn-sbctl chassis-add gw1 geneve 127.0.0.1 +ovn-sbctl chassis-add gw2 geneve 128.0.0.1 +ovn-sbctl chassis-add gw3 geneve 129.0.0.1 + +ovn-nbctl lr-add DR +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24 +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24 + +ovn-nbctl ls-add S1 +ovn-nbctl lsp-add S1 S1-DR +ovn-nbctl lsp-set-type S1-DR router +ovn-nbctl lsp-set-addresses S1-DR router +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 + +ovn-nbctl ls-add S2 +ovn-nbctl lsp-add S2 S2-DR +ovn-nbctl lsp-set-type S2-DR router +ovn-nbctl lsp-set-addresses S2-DR router +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 + +ovn-nbctl ls-add S3 +ovn-nbctl lsp-add S3 S3-DR +ovn-nbctl lsp-set-type S3-DR router +ovn-nbctl lsp-set-addresses S3-DR router +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 + +ovn-nbctl ls-add ls +ovn-nbctl lsp-add ls ls-DR +ovn-nbctl lsp-set-type ls-DR router +ovn-nbctl lsp-set-addresses ls-DR router +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls + +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1 +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2 +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3 + +ovn-nbctl --wait=sb sync + +ovn-nbctl lb-add lb0 192.168.0.3:80 10.0.0.2:80,10.0.0.3:80 +ovn-nbctl lr-lb-add DR lb0 + +ovn-sbctl dump-flows DR + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| wc -l], [0], [3 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| wc -l], [0], [3 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S1 | grep cr-DR-S1| wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S1| grep cr-DR-S1 | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S2 | grep cr-DR-S2| wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S2| grep cr-DR-S2 | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S3 | grep cr-DR-S3| wc -l], [0], [1 +]) +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S3| grep cr-DR-S3 | wc -l], [0], [1 +]) + +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep priority=120| wc -l], [0], [3 +]) + +AT_CLEANUP diff --git a/tests/ovn.at b/tests/ovn.at index 9bac94b..7c44cd3 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -19691,7 +19691,7 @@ AT_CAPTURE_FILE([sbflows2]) OVS_WAIT_FOR_OUTPUT( [ovn-sbctl dump-flows > sbflows2 ovn-sbctl dump-flows lr0 | grep ct_lb | grep priority=120 | sed 's/table=..//'], 0, - [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) + [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");) ]) # get the svc monitor mac. @@ -19732,8 +19732,8 @@ AT_CHECK( AT_CAPTURE_FILE([sbflows4]) ovn-sbctl dump-flows lr0 > sbflows4 AT_CHECK([grep lr_in_dnat sbflows4 | grep priority=120 | sed 's/table=..//' | sort], [0], [dnl - (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) - (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(drop;) + (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;) + (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(drop;) ]) # Delete sw0-p1 @@ -23715,3 +23715,307 @@ OVS_WAIT_UNTIL([test `ovs-vsctl get Interface lsp2 external_ids:ovn-installed` = OVN_CLEANUP([hv1]) AT_CLEANUP + +AT_SETUP([ovn -- lr multiple gw ports]) +ovn_start + +# Logical network: +# 1 LR, 3 Logical Switches, +# 1 gateway chassis attached to each corresponding LRP. +# +# | S1 (gw1) +# | +# ls ---- DR -- S3 (gw3) +# (20.0.0.0/24) | +# | S2 (gw2) +# +# S1 - VLAN 1000 +# S2 - VLAN 2000 +# S3 - VLAN 3000 +# +# 5 chassis(s), HV1----HV5 +# +# HV1 - VIF11 +# HV2 - Gateway chassis gw1 +# HV3 - Gateway chassis gw2 +# HV4 - Gateway chassis gw3 +# HV5 - North endpoint + +ovn-nbctl lr-add DR +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24 +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24 +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24 +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.0/24 + +ovn-nbctl ls-add S1 +ovn-nbctl lsp-add S1 S1-DR +ovn-nbctl lsp-set-type S1-DR router +ovn-nbctl lsp-set-addresses S1-DR router +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1 +ovn-nbctl lsp-add S1 ln1 "" 1000 +ovn-nbctl lsp-set-addresses ln1 unknown +ovn-nbctl lsp-set-type ln1 localnet +ovn-nbctl lsp-set-options ln1 network_name=phys + +ovn-nbctl ls-add S2 +ovn-nbctl lsp-add S2 S2-DR +ovn-nbctl lsp-set-type S2-DR router +ovn-nbctl lsp-set-addresses S2-DR router +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2 +ovn-nbctl lsp-add S2 ln2 "" 2000 +ovn-nbctl lsp-set-addresses ln2 unknown +ovn-nbctl lsp-set-type ln2 localnet +ovn-nbctl lsp-set-options ln2 network_name=phys + +ovn-nbctl ls-add S3 +ovn-nbctl lsp-add S3 S3-DR +ovn-nbctl lsp-set-type S3-DR router +ovn-nbctl lsp-set-addresses S3-DR router +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3 +ovn-nbctl lsp-add S3 ln3 "" 3000 +ovn-nbctl lsp-set-addresses ln3 unknown +ovn-nbctl lsp-set-type ln3 localnet +ovn-nbctl lsp-set-options ln3 network_name=phys + +ovn-nbctl ls-add ls +ovn-nbctl lsp-add ls ls-DR +ovn-nbctl lsp-set-type ls-DR router +ovn-nbctl lsp-set-addresses ls-DR router +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls + +# Add the lsp lp11 to ls. This will map to VIF11. +ovn-nbctl lsp-add ls lp11 +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10" +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10 + +# Add the Northbound endpoint, lp-north1 +ovn-nbctl ls-add ls-north1 +ovn-nbctl lsp-add ls-north1 ln4 "" 1000 +ovn-nbctl lsp-set-addresses ln4 unknown +ovn-nbctl lsp-set-type ln4 localnet +ovn-nbctl lsp-set-options ln4 network_name=phys + +ovn-nbctl lsp-add ls-north1 lp-north1 +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10" +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11 + +# Add the Northbound endpoint, lp-north2 +ovn-nbctl ls-add ls-north2 +ovn-nbctl lsp-add ls-north2 ln5 "" 2000 +ovn-nbctl lsp-set-addresses ln5 unknown +ovn-nbctl lsp-set-type ln5 localnet +ovn-nbctl lsp-set-options ln5 network_name=phys + +ovn-nbctl lsp-add ls-north2 lp-north2 +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10" +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22 + +# Add the Northbound endpoint, lp-north3 +ovn-nbctl ls-add ls-north3 +ovn-nbctl lsp-add ls-north3 ln6 "" 3000 +ovn-nbctl lsp-set-addresses ln6 unknown +ovn-nbctl lsp-set-type ln6 localnet +ovn-nbctl lsp-set-options ln6 network_name=phys + +ovn-nbctl lsp-add ls-north3 lp-north3 +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10" +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33 + +# Add 5 chassis +net_add n1 +for i in 1 2 3 4 5; do + sim_add hv$i + as hv$i + ovs-vsctl add-br br-phys + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys + ovn_attach n1 br-phys 192.168.0.$i 24 $encap +done + +# Add a vif on HV1 +as hv1 ovs-vsctl add-port br-int vif11 -- \ + set Interface vif11 external-ids:iface-id=lp11 \ + options:tx_pcap=hv1/vif11-tx.pcap \ + options:rxq_pcap=hv1/vif11-rx.pcap \ + ofport-request=11 +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup]) + +as hv5 ovs-vsctl add-port br-int vif-north1 -- \ + set Interface vif-north1 external-ids:iface-id=lp-north1 \ + options:tx_pcap=hv5/vif-north1-tx.pcap \ + options:rxq_pcap=hv5/vif-north1-rx.pcap \ + ofport-request=44 + +as hv5 ovs-vsctl add-port br-int vif-north2 -- \ + set Interface vif-north2 external-ids:iface-id=lp-north2 \ + options:tx_pcap=hv5/vif-north2-tx.pcap \ + options:rxq_pcap=hv5/vif-north2-rx.pcap \ + ofport-request=45 + +as hv5 ovs-vsctl add-port br-int vif-north3 -- \ + set Interface vif-north3 external-ids:iface-id=lp-north3 \ + options:tx_pcap=hv5/vif-north3-tx.pcap \ + options:rxq_pcap=hv5/vif-north3-rx.pcap \ + ofport-request=46 + +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2 +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3 +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4 + +ovn-nbctl --wait=sb sync +OVN_POPULATE_ARP + +vif_to_ls () { + case ${1} in dnl ( + vif?[[11]]) echo ls ;; dnl ( + vif-north1) echo ls-north1 ;; dnl ( + vif-north2) echo ls-north2 ;; dnl ( + vif-north3) echo ls-north3 ;; dnl ( + *) AT_FAIL_IF([:]) ;; + esac +} + +vif_to_hv () { + case ${1} in dnl ( + vif[[1]]?) echo hv1 ;; dnl ( + vif-north1) echo hv5 ;; dnl ( + vif-north2) echo hv5 ;; dnl ( + vif-north3) echo hv5 ;; dnl ( + *) AT_FAIL_IF([:]) ;; + esac +} + +vif_to_lrp () { + case ${1} in dnl ( + vif?[[11]]) echo DR-ls ;; dnl ( + *) AT_FAIL_IF([:]) ;; + esac + +} + +ip_to_hex() { + printf "%02x%02x%02x%02x" "${@}" +} + +# test_arp INPORT SHA SPA TPA +# +# Causes a packet to be received on INPORT. The packet is an ARP +# request with SHA, SPA, and TPA as specified. +test_arp() { + local inport=$1 sha=$2 spa=$3 tpa=$4 + local request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa} + hv=`vif_to_hv $inport` + as $hv ovs-appctl netdev-dummy/receive $inport $request +} + + +test_ip() { + # This packet has bad checksums but logical L3 routing doesn't check. + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5} outport=${6} + local packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000 + shift; shift; shift; shift; shift + hv=`vif_to_hv $inport` + as $hv ovs-appctl netdev-dummy/receive $inport $packet + in_ls=`vif_to_ls $inport` + for outport; do + out_ls=`vif_to_ls $outport` + if test $in_ls = $out_ls; then + # Ports on the same logical switch receive exactly the same packet. + echo $packet + else + # Routing decrements TTL and updates source and dest MAC + # (and checksum). + # For North-South, packet will come via gateway chassis, i.e hv3 + if test $inport = vif-north1; then + echo f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected + fi + if test $outport = vif-north1; then + echo f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected + fi + if test $outport = vif-north2; then + echo f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected + fi + if test $outport = vif-north3; then + echo f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected + fi + fi >> $outport.expected + done +} + +echo "------ OVN dump ------" +ovn-nbctl show +ovn-sbctl show +ovn-sbctl list port_binding +ovn-sbctl list mac_binding +ovn-sbctl list datapath_binding + +ovn-sbctl dump-flows DR +ovn-sbctl dump-flows S1 +ovn-sbctl dump-flows ls + +echo "------ hv1 dump ------" +as hv1 ovs-vsctl show +as hv1 ovs-vsctl list Open_Vswitch +as hv1 ovs-ofctl dump-flows br-int + +echo "------ hv2 dump ------" +as hv2 ovs-vsctl show +as hv2 ovs-vsctl list Open_Vswitch +as hv2 ovs-ofctl dump-flows br-int + +echo "------ hv3 dump ------" +as hv3 ovs-vsctl show +as hv3 ovs-vsctl list Open_Vswitch +as hv3 ovs-ofctl dump-flows br-int + +echo "------ hv4 dump ------" +as hv4 ovs-vsctl show +as hv4 ovs-vsctl list Open_Vswitch +as hv5 ovs-ofctl dump-flows br-int + +# N-S with lp-north1 +echo "Send Dummy ARP" +sip=`ip_to_hex 172 16 1 10` +tip=`ip_to_hex 172 16 1 50` +test_arp vif-north1 f0f000000011 $sip $tip + +echo "Send traffic North to South" +sip=`ip_to_hex 172 16 1 10` +dip=`ip_to_hex 20 0 0 10` +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11 +# Confirm that North to south traffic works fine. +OVN_CHECK_PACKETS([hv1/vif11-tx.pcap], [vif11.expected]) + +echo "Send traffic South to North" +sip=`ip_to_hex 20 0 0 10` +dip=`ip_to_hex 172 16 1 10` +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1 +# Confirm that South to North traffic works fine. +OVN_CHECK_PACKETS([hv5/vif-north1-tx.pcap], [vif-north1.expected]) + +# N-S with lp-north2 +echo "Send Dummy ARP" +sip=`ip_to_hex 10 0 0 10` +tip=`ip_to_hex 10 0 0 50` +test_arp vif-north2 f0f000000022 $sip $tip + +echo "Send traffic South to North" +sip=`ip_to_hex 20 0 0 10` +dip=`ip_to_hex 10 0 0 10` +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2 +# Confirm that South to North traffic works fine. +OVN_CHECK_PACKETS([hv5/vif-north2-tx.pcap], [vif-north2.expected]) + +# N-S with lp-north3 +echo "Send Dummy ARP" +sip=`ip_to_hex 192 168 0 10` +tip=`ip_to_hex 192 168 0 50` +test_arp vif-north3 f0f000000033 $sip $tip + +echo "Send traffic South to North" +sip=`ip_to_hex 20 0 0 10` +dip=`ip_to_hex 192 168 0 10` +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3 +# Confirm that South to North traffic works fine. +OVN_CHECK_PACKETS([hv5/vif-north3-tx.pcap], [vif-north3.expected]) + +AT_CLEANUP diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c index d67f5c4..912ede7 100644 --- a/utilities/ovn-nbctl.c +++ b/utilities/ovn-nbctl.c @@ -4208,6 +4208,27 @@ done: return ret; } +static bool +is_nat_rule_conflict(const struct nbrec_logical_router *lr) +{ + int num_l3dgw_ports = 0; + + /* TODO: Add a proper validation to confirm that multiple + * external ips for a logical ip do not belong to same router port. */ + for (size_t i = 0; i < lr->n_ports; i++) { + const struct nbrec_logical_router_port *lrp = lr->ports[i]; + if (lrp->n_gateway_chassis) { + num_l3dgw_ports++; + } + } + + if (num_l3dgw_ports > 1) { + return false; + } + + return true; +} + static void nbctl_lr_nat_add(struct ctl_context *ctx) { @@ -4369,12 +4390,14 @@ nbctl_lr_nat_add(struct ctl_context *ctx) should_return = true; } } else { - ctl_error(ctx, "a NAT with this type (%s) and %s (%s) " - "already exists", - nat_type, - is_snat ? "logical_ip" : "external_ip", - is_snat ? new_logical_ip : new_external_ip); - should_return = true; + if (is_nat_rule_conflict(lr)) { + ctl_error(ctx, "a NAT with this type (%s) and %s (%s) " + "already exists", + nat_type, + is_snat ? "logical_ip" : "external_ip", + is_snat ? new_logical_ip : new_external_ip); + should_return = true; + } } } }