diff mbox

[ovs-dev] Support multiple logical routing port configuration "redirect-chassis" on a distributed router

Message ID 20170227131257.4852-1-ligs@dtdream.com
State Changes Requested
Headers show

Commit Message

Guoshuai Li Feb. 27, 2017, 1:12 p.m. UTC
The main application scenario of this patch is that the user flow wants to
different destination addresses through different external networks.
This scenario requires a distributed route to be associated with
multiple external network logical switches.

In a distributed router, the NAT logical flow table is generated based on
the external IP lookup distributed router port, otherwise not generated.

When the destination address of the packet is an external IP of the NAT rule,
and the ingress port is not a gateway,
it is necessary to route the actual outgoing port.

Signed-off-by: Guoshuai Li <ligs@dtdream.com>
Co-authored-by: Dong Jun <dongj@dtdream.com>
---
 ovn/northd/ovn-northd.8.xml |  22 ++---
 ovn/northd/ovn-northd.c     | 232 +++++++++++++++++++++++---------------------
 ovn/ovn-nb.xml              |  12 ++-
 3 files changed, 137 insertions(+), 129 deletions(-)

Comments

Mickey Spiegel Feb. 27, 2017, 6:51 p.m. UTC | #1
This is a quick preliminary review. I will review this in more detail
tomorrow afternoon.

On Mon, Feb 27, 2017 at 5:12 AM, Guoshuai Li <ligs@dtdream.com> wrote:

> The main application scenario of this patch is that the user flow wants to
> different destination addresses through different external networks.
> This scenario requires a distributed route to be associated with
> multiple external network logical switches.
>
> In a distributed router, the NAT logical flow table is generated based on
> the external IP lookup distributed router port, otherwise not generated.
>

I see your problem that you need some way to figure out which router
gateway port this NAT rule should be associated with, now that you have
multiple distributed gateway ports on the same logical router.

However, there is currently no restriction that NAT external IP addresses
need to match an existing subnet on a router port. I am uncomfortable with
the addition of such a restriction in this patch, since it will not support
scenarios that are valid in OVN and in OpenStack today.


> When the destination address of the packet is an external IP of the NAT
> rule,
> and the ingress port is not a gateway,
> it is necessary to route the actual outgoing port.
>

Are you suggesting that static routes need to be programmed to NAT external
addresses?
That would significantly complicate what the user needs to do in order to
make NAT on a distributed router work.
Doesn't this break existing tests?
My suggestion would be to set outport when you set REGBIT_NAT_REDIRECT, in
the DNAT and UNSNAT stages.

I would also like to hear back from Russell or others about the "routed
external networks" discussion at the OpenStack PTG in Atlanta.
Does a router still have only one external network?
Does a router still have only one external IPv4 address?
Is it limited to one segment?
If there can be multiple external IPv4 addresses, then there might be some
overlap with this proposal.

Mickey


>
> Signed-off-by: Guoshuai Li <ligs@dtdream.com>
> Co-authored-by: Dong Jun <dongj@dtdream.com>
> ---
>  ovn/northd/ovn-northd.8.xml |  22 ++---
>  ovn/northd/ovn-northd.c     | 232 +++++++++++++++++++++++-------
> --------------
>  ovn/ovn-nb.xml              |  12 ++-
>  3 files changed, 137 insertions(+), 129 deletions(-)
>
Guoshuai Li Feb. 28, 2017, 2:57 a.m. UTC | #2
Hi Mickey, Thanks for review.

> This is a quick preliminary review. I will review this in more detail 
> tomorrow afternoon.
>
> On Mon, Feb 27, 2017 at 5:12 AM, Guoshuai Li <ligs@dtdream.com 
> <mailto:ligs@dtdream.com>> wrote:
>
>     The main application scenario of this patch is that the user flow
>     wants to
>     different destination addresses through different external networks.
>     This scenario requires a distributed route to be associated with
>     multiple external network logical switches.
>
>     In a distributed router, the NAT logical flow table is generated
>     based on
>     the external IP lookup distributed router port, otherwise not
>     generated.
>
>
> I see your problem that you need some way to figure out which router 
> gateway port this NAT rule should be associated with, now that you 
> have multiple distributed gateway ports on the same logical router.
>
> However, there is currently no restriction that NAT external IP 
> addresses need to match an existing subnet on a router port. I am 
> uncomfortable with the addition of such a restriction in this patch, 
> since it will not support scenarios that are valid in OVN and in 
> OpenStack today.
     I think the general usage is NAT external IP addresses in the 
existing subnet segment.
     Do you accept the NAT*l**o**g**i**c**a**l**_**p**o**r**t *field to 
specify the gateway port ?*e**x**t**e**r**n**a**l**_**m**a**c *field may 
used to distinguish between centralized and distributed.
>
>     When the destination address of the packet is an external IP of
>     the NAT rule,
>     and the ingress port is not a gateway,
>     it is necessary to route the actual outgoing port.
>
>
> Are you suggesting that static routes need to be programmed to NAT 
> external addresses?
     This is Base on NAT external addresses in the router port subnet, 
not need static routes, because matching subnet segment routing.
     If break this limit, your idea is good.
> That would significantly complicate what the user needs to do in order 
> to make NAT on a distributed router work.
> Doesn't this break existing tests?
> My suggestion would be to set outport when you set 
> REGBIT_NAT_REDIRECT, in the DNAT and UNSNAT stages.
>
> I would also like to hear back from Russell or others about the 
> "routed external networks" discussion at the OpenStack PTG in Atlanta.
> Does a router still have only one external network?
> Does a router still have only one external IPv4 address?
> Is it limited to one segment?
> If there can be multiple external IPv4 addresses, then there might be 
> some overlap with this proposal.
>
> Mickey
>
>
>     Signed-off-by: Guoshuai Li <ligs@dtdream.com
>     <mailto:ligs@dtdream.com>>
>     Co-authored-by: Dong Jun <dongj@dtdream.com
>     <mailto:dongj@dtdream.com>>
>     ---
>      ovn/northd/ovn-northd.8.xml |  22 ++---
>      ovn/northd/ovn-northd.c     | 232
>     +++++++++++++++++++++++---------------------
>      ovn/ovn-nb.xml              |  12 ++-
>      3 files changed, 137 insertions(+), 129 deletions(-)
>
diff mbox

Patch

diff --git a/ovn/northd/ovn-northd.8.xml b/ovn/northd/ovn-northd.8.xml
index ab8fd88..d9056ef 100644
--- a/ovn/northd/ovn-northd.8.xml
+++ b/ovn/northd/ovn-northd.8.xml
@@ -1549,16 +1549,6 @@  icmp4 {
     <ul>
       <li>
         <p>
-          For distributed logical routers where one of the logical router
-          ports specifies a <code>redirect-chassis</code>, a priority-300
-          logical flow with match <code>REGBIT_NAT_REDIRECT == 1</code> has
-          actions <code>ip.ttl--; next;</code>.  The <code>outport</code>
-          will be set later in the Gateway Redirect table.
-        </p>
-      </li>
-
-      <li>
-        <p>
           IPv4 routing table.  For each route to IPv4 network <var>N</var> with
           netmask <var>M</var>, on router port <var>P</var> with IP address
           <var>A</var> and Ethernet
@@ -1644,12 +1634,12 @@  next;
     <ul>
       <li>
         <p>
-          For distributed logical routers where one of the logical router
-          ports specifies a <code>redirect-chassis</code>, a priority-200
-          logical flow with match <code>REGBIT_NAT_REDIRECT == 1</code> has
-          actions <code>eth.dst = <var>E</var>; next;</code>, where
-          <var>E</var> is the ethernet address of the router's distributed
-          gateway port.
+          For distributed logical routers where router port <var>P</var>
+          specifies a <code>redirect-chassis</code>, a priority-200
+          logical flow with match <code>REGBIT_NAT_REDIRECT == 1</code>
+          and outport == <var>P</var> has actions
+          <code>eth.dst = <var>E</var>; next;</code>, where <var>E</var>
+          is the ethernet address of the router's distributed gateway port.
         </p>
       </li>
 
diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
index 03dc850..8477e23 100644
--- a/ovn/northd/ovn-northd.c
+++ b/ovn/northd/ovn-northd.c
@@ -398,12 +398,10 @@  struct ovn_datapath {
 
     /* OVN northd only needs to know about the logical router gateway port for
      * NAT on a distributed router.  This "distributed gateway port" is
-     * populated only when there is a "redirect-chassis" specified for one of
-     * the ports on the logical router.  Otherwise this will be NULL. */
-    struct ovn_port *l3dgw_port;
-    /* The "derived" OVN port representing the instance of l3dgw_port on
-     * the "redirect-chassis". */
-    struct ovn_port *l3redirect_port;
+     * populated only when there is a "redirect-chassis" specified for the 
+     * ports on the logical router.  Otherwise this will be NULL. */
+    struct ovn_port **l3dgw_ports;
+    size_t n_l3dgw_ports;
 };
 
 struct macam_node {
@@ -683,6 +681,10 @@  struct ovn_port {
     bool derived; /* Indicates whether this is an additional port
                    * derived from nbsp or nbrp. */
 
+    /* The "derived" OVN port representing the instance of l3dgw_port on
+     * the "redirect-chassis". Otherwise this will be NULL. */
+    struct ovn_port *l3redirect_port;
+
     /* The port's peer:
      *
      *     - A switch port S of type "router" has a router port R as a peer,
@@ -1402,16 +1404,11 @@  join_logical_ports(struct northd_context *ctx,
 
                     /* Set l3dgw_port and l3redirect_port in od, for later
                      * use during flow creation. */
-                    if (od->l3dgw_port || od->l3redirect_port) {
-                        static struct vlog_rate_limit rl
-                            = VLOG_RATE_LIMIT_INIT(1, 1);
-                        VLOG_WARN_RL(&rl, "Bad configuration: multiple ports "
-                                     "with redirect-chassis on same logical "
-                                     "router %s", od->nbr->name);
-                        continue;
-                    } else {
-                        od->l3dgw_port = op;
-                        od->l3redirect_port = crp;
+                    if (!op->l3redirect_port) {
+                        op->l3redirect_port = crp;
+                        od->l3dgw_ports = xrealloc(od->l3dgw_ports,
+                            sizeof *od->l3dgw_ports * (od->n_l3dgw_ports + 1));
+                        od->l3dgw_ports[od->n_l3dgw_ports++] = op;
                     }
                 }
             }
@@ -3300,14 +3297,12 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
                 ds_clear(&match);
                 ds_put_format(&match, "eth.dst == "ETH_ADDR_FMT,
                               ETH_ADDR_ARGS(mac));
-                if (op->peer->od->l3dgw_port
-                    && op->peer == op->peer->od->l3dgw_port
-                    && op->peer->od->l3redirect_port) {
+                if (op->peer->l3redirect_port) {
                     /* The destination lookup flow for the router's
                      * distributed gateway port MAC address should only be
                      * programmed on the "redirect-chassis". */
                     ds_put_format(&match, " && is_chassis_resident(%s)",
-                                  op->peer->od->l3redirect_port->json_key);
+                                  op->peer->l3redirect_port->json_key);
                 }
 
                 ds_clear(&actions);
@@ -3317,8 +3312,7 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
 
                 /* Add ethernet addresses specified in NAT rules on
                  * distributed logical routers. */
-                if (op->peer->od->l3dgw_port
-                    && op->peer == op->peer->od->l3dgw_port) {
+                if (op->peer->l3redirect_port) {
                     for (int i = 0; i < op->peer->od->nbr->n_nat; i++) {
                         const struct nbrec_nat *nat
                                                   = op->peer->od->nbr->nat[i];
@@ -3473,6 +3467,22 @@  find_lrp_member_ip(const struct ovn_port *op, const char *ip_s)
     return NULL;
 }
 
+static struct ovn_port *
+find_l3dgw_port(struct ovn_datapath *od, const char *external_ip)
+{
+    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
+        struct ovn_port *l3dgw_port = od->l3dgw_ports[i];
+        if (find_lrp_member_ip(l3dgw_port, external_ip)){
+            return l3dgw_port;
+        }
+    }
+    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
+    VLOG_WARN_RL(&rl, "can not find l3dgw port with redirect-chassis "
+                 "for nat, external ip %s in router "UUID_FMT"",
+                 external_ip, UUID_ARGS(&od->key));
+    return NULL;
+}
+
 static void
 add_route(struct hmap *lflows, const struct ovn_port *op,
           const char *lrp_addr_s, const char *network_s, int plen,
@@ -3788,12 +3798,11 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         ds_clear(&match);
         ds_put_format(&match, "eth.dst == %s && inport == %s",
                       op->lrp_networks.ea_s, op->json_key);
-        if (op->od->l3dgw_port && op == op->od->l3dgw_port
-            && op->od->l3redirect_port) {
+        if (op->l3redirect_port) {
             /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
              * should only be received on the "redirect-chassis". */
             ds_put_format(&match, " && is_chassis_resident(%s)",
-                          op->od->l3redirect_port->json_key);
+                          op->l3redirect_port->json_key);
         }
         ovn_lflow_add(lflows, op->od, S_ROUTER_IN_ADMISSION, 50,
                       ds_cstr(&match), "next;");
@@ -3902,15 +3911,14 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
             ds_put_format(&match,
                           "inport == %s && arp.tpa == %s && arp.op == 1",
                           op->json_key, op->lrp_networks.ipv4_addrs[i].addr_s);
-            if (op->od->l3dgw_port && op == op->od->l3dgw_port
-                && op->od->l3redirect_port) {
+            if (op->l3redirect_port) {
                 /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
                  * should only be sent from the "redirect-chassis", so that
                  * upstream MAC learning points to the "redirect-chassis".
                  * Also need to avoid generation of multiple ARP responses
                  * from different chassis. */
                 ds_put_format(&match, " && is_chassis_resident(%s)",
-                              op->od->l3redirect_port->json_key);
+                              op->l3redirect_port->json_key);
             }
 
             ds_clear(&actions);
@@ -4046,7 +4054,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                 "arp.op = 2; /* ARP reply */ "
                 "arp.tha = arp.sha; ");
 
-            if (op->od->l3dgw_port && op == op->od->l3dgw_port) {
+            if (op->l3redirect_port) {
                 struct eth_addr mac;
                 if (nat->external_mac &&
                     eth_addr_from_string(nat->external_mac, &mac)
@@ -4075,10 +4083,8 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                      * upstream MAC learning points to the "redirect-chassis".
                      * Also need to avoid generation of multiple ARP responses
                      * from different chassis. */
-                    if (op->od->l3redirect_port) {
-                        ds_put_format(&match, " && is_chassis_resident(%s)",
-                                      op->od->l3redirect_port->json_key);
-                    }
+                    ds_put_format(&match, " && is_chassis_resident(%s)",
+                                  op->l3redirect_port->json_key);
                 }
             } else {
                 ds_put_format(&actions,
@@ -4188,15 +4194,14 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                     op->lrp_networks.ipv6_addrs[i].addr_s,
                     op->lrp_networks.ipv6_addrs[i].sn_addr_s,
                     op->lrp_networks.ipv6_addrs[i].addr_s);
-            if (op->od->l3dgw_port && op == op->od->l3dgw_port
-                && op->od->l3redirect_port) {
+            if (op->l3redirect_port) {
                 /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
                  * should only be sent from the "redirect-chassis", so that
                  * upstream MAC learning points to the "redirect-chassis".
                  * Also need to avoid generation of multiple ND replies
                  * from different chassis. */
                 ds_put_format(&match, " && is_chassis_resident(%s)",
-                              op->od->l3redirect_port->json_key);
+                              op->l3redirect_port->json_key);
             }
 
             ds_clear(&actions);
@@ -4237,7 +4242,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         /* NAT rules are only valid on Gateway routers and routers with
          * l3dgw_port (router has a port with "redirect-chassis"
          * specified). */
-        if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
+        if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) {
             continue;
         }
 
@@ -4291,7 +4296,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * satisfies the conditions for distributed NAT processing. */
             bool distributed = false;
             struct eth_addr mac;
-            if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
+            if (od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") &&
                 nat->logical_port && nat->external_mac) {
                 if (eth_addr_from_string(nat->external_mac, &mac)) {
                     distributed = true;
@@ -4304,6 +4309,9 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                 }
             }
 
+            /* find l3dgw port by external ip */
+            struct ovn_port *l3dgw_port = find_l3dgw_port(od, nat->external_ip);
+
             /* Ingress UNSNAT table: It is for already established connections'
              * reverse traffic. i.e., SNAT has already been done in egress
              * pipeline and now the packet has entered the ingress pipeline as
@@ -4315,14 +4323,14 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * egress pipeline. */
             if (!strcmp(nat->type, "snat")
                 || !strcmp(nat->type, "dnat_and_snat")) {
-                if (!od->l3dgw_port) {
+                if (!od->n_l3dgw_ports) {
                     /* Gateway router. */
                     ds_clear(&match);
                     ds_put_format(&match, "ip && ip4.dst == %s",
                                   nat->external_ip);
                     ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 90,
                                   ds_cstr(&match), "ct_snat; next;");
-                } else {
+                } else if (l3dgw_port) {
                     /* Distributed router. */
 
                     /* Traffic received on l3dgw_port is subject to NAT. */
@@ -4330,12 +4338,12 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                     ds_put_format(&match, "ip && ip4.dst == %s"
                                           " && inport == %s",
                                   nat->external_ip,
-                                  od->l3dgw_port->json_key);
-                    if (!distributed && od->l3redirect_port) {
+                                  l3dgw_port->json_key);
+                    if (!distributed && l3dgw_port->l3redirect_port) {
                         /* Flows for NAT rules that are centralized are only
                          * programmed on the "redirect-chassis". */
                         ds_put_format(&match, " && is_chassis_resident(%s)",
-                                      od->l3redirect_port->json_key);
+                                     l3dgw_port->l3redirect_port->json_key);
                     }
                     ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 100,
                                   ds_cstr(&match), "ct_snat;");
@@ -4357,7 +4365,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * to a logical IP address. */
             if (!strcmp(nat->type, "dnat")
                 || !strcmp(nat->type, "dnat_and_snat")) {
-                if (!od->l3dgw_port) {
+                if (!od->n_l3dgw_ports) {
                     /* Gateway router. */
                     /* Packet when it goes from the initiator to destination.
                      * We need to set flags.loopback because the router can
@@ -4377,7 +4385,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                                   nat->logical_ip);
                     ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 100,
                                   ds_cstr(&match), ds_cstr(&actions));
-                } else {
+                } else if (l3dgw_port) {
                     /* Distributed router. */
 
                     /* Traffic received on l3dgw_port is subject to NAT. */
@@ -4385,12 +4393,12 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                     ds_put_format(&match, "ip && ip4.dst == %s"
                                           " && inport == %s",
                                   nat->external_ip,
-                                  od->l3dgw_port->json_key);
-                    if (!distributed && od->l3redirect_port) {
+                                  l3dgw_port->json_key);
+                    if (!distributed && l3dgw_port->l3redirect_port) {
                         /* Flows for NAT rules that are centralized are only
                          * programmed on the "redirect-chassis". */
                         ds_put_format(&match, " && is_chassis_resident(%s)",
-                                      od->l3redirect_port->json_key);
+                                     l3dgw_port->l3redirect_port->json_key);
                     }
                     ds_clear(&actions);
                     ds_put_format(&actions, "ct_dnat(%s);",
@@ -4418,18 +4426,18 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * Note that this only applies for NAT on a distributed router.
              * Undo DNAT on a gateway router is done in the ingress DNAT
              * pipeline stage. */
-            if (od->l3dgw_port && (!strcmp(nat->type, "dnat")
+            if (l3dgw_port && (!strcmp(nat->type, "dnat")
                 || !strcmp(nat->type, "dnat_and_snat"))) {
                 ds_clear(&match);
                 ds_put_format(&match, "ip && ip4.src == %s"
                                       " && outport == %s",
                               nat->logical_ip,
-                              od->l3dgw_port->json_key);
-                if (!distributed && od->l3redirect_port) {
+                              l3dgw_port->json_key);
+                if (!distributed && l3dgw_port->l3redirect_port) {
                     /* Flows for NAT rules that are centralized are only
                      * programmed on the "redirect-chassis". */
                     ds_put_format(&match, " && is_chassis_resident(%s)",
-                                  od->l3redirect_port->json_key);
+                                  l3dgw_port->l3redirect_port->json_key);
                 }
                 ds_clear(&actions);
                 if (distributed) {
@@ -4446,7 +4454,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * address. */
             if (!strcmp(nat->type, "snat")
                 || !strcmp(nat->type, "dnat_and_snat")) {
-                if (!od->l3dgw_port) {
+                if (!od->n_l3dgw_ports) {
                     /* Gateway router. */
                     ds_clear(&match);
                     ds_put_format(&match, "ip && ip4.src == %s",
@@ -4460,18 +4468,18 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                     ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT,
                                   count_1bits(ntohl(mask)) + 1,
                                   ds_cstr(&match), ds_cstr(&actions));
-                } else {
+                } else if (l3dgw_port) {
                     /* Distributed router. */
                     ds_clear(&match);
                     ds_put_format(&match, "ip && ip4.src == %s"
                                           " && outport == %s",
                                   nat->logical_ip,
-                                  od->l3dgw_port->json_key);
-                    if (!distributed && od->l3redirect_port) {
+                                  l3dgw_port->json_key);
+                    if (!distributed && l3dgw_port->l3redirect_port) {
                         /* Flows for NAT rules that are centralized are only
                          * programmed on the "redirect-chassis". */
                         ds_put_format(&match, " && is_chassis_resident(%s)",
-                                      od->l3redirect_port->json_key);
+                                     l3dgw_port->l3redirect_port->json_key);
                     }
                     ds_clear(&actions);
                     if (distributed) {
@@ -4495,15 +4503,18 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * on the l3dgw_port instance where nat->logical_port is
              * resident. */
             if (distributed) {
-                ds_clear(&match);
-                ds_put_format(&match,
-                              "eth.dst == "ETH_ADDR_FMT" && inport == %s"
-                              " && is_chassis_resident(\"%s\")",
-                              ETH_ADDR_ARGS(mac),
-                              od->l3dgw_port->json_key,
-                              nat->logical_port);
-                ovn_lflow_add(lflows, od, S_ROUTER_IN_ADMISSION, 50,
-                              ds_cstr(&match), "next;");
+                for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
+                    struct ovn_port *l3dgw_port = od->l3dgw_ports[i];
+                    ds_clear(&match);
+                    ds_put_format(&match,
+                                  "eth.dst == "ETH_ADDR_FMT" && inport == %s"
+                                  " && is_chassis_resident(\"%s\")",
+                                  ETH_ADDR_ARGS(mac),
+                                  l3dgw_port->json_key,
+                                  nat->logical_port);
+                    ovn_lflow_add(lflows, od, S_ROUTER_IN_ADMISSION, 50,
+                                  ds_cstr(&match), "next;");
+                }
             }
 
             /* Ingress Gateway Redirect Table: For NAT on a distributed
@@ -4511,12 +4522,15 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * flows indicate the presence of an applicable NAT rule that
              * can be applied in a distributed manner. */
             if (distributed) {
-                ds_clear(&match);
-                ds_put_format(&match, "ip4.src == %s && outport == %s",
-                              nat->logical_ip,
-                              od->l3dgw_port->json_key);
-                ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 100,
-                              ds_cstr(&match), "next;");
+                for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
+                    struct ovn_port *l3dgw_port = od->l3dgw_ports[i];
+                    ds_clear(&match);
+                    ds_put_format(&match, "ip4.src == %s && outport == %s",
+                                  nat->logical_ip,
+                                  l3dgw_port->json_key);
+                    ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 100,
+                                  ds_cstr(&match), "next;");
+                }
             }
 
             /* Egress Loopback table: For NAT on a distributed router.
@@ -4524,12 +4538,12 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
              * gateway port have ip.dst matching a NAT external IP, then
              * loop a clone of the packet back to the beginning of the
              * ingress pipeline with inport = outport. */
-            if (od->l3dgw_port) {
+            if (l3dgw_port) {
                 /* Distributed router. */
                 ds_clear(&match);
                 ds_put_format(&match, "ip4.dst == %s && outport == %s",
                               nat->external_ip,
-                              od->l3dgw_port->json_key);
+                              l3dgw_port->json_key);
                 ds_clear(&actions);
                 ds_put_format(&actions,
                               "clone { ct_clear; "
@@ -4546,7 +4560,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         }
 
         /* Handle force SNAT options set in the gateway router. */
-        if (dnat_force_snat_ip && !od->l3dgw_port) {
+        if (dnat_force_snat_ip && !od->n_l3dgw_ports) {
             /* If a packet with destination IP address as that of the
              * gateway router (as set in options:dnat_force_snat_ip) is seen,
              * UNSNAT it. */
@@ -4565,7 +4579,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
             ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 100,
                           ds_cstr(&match), ds_cstr(&actions));
         }
-        if (lb_force_snat_ip && !od->l3dgw_port) {
+        if (lb_force_snat_ip && !od->n_l3dgw_ports) {
             /* If a packet with destination IP address as that of the
              * gateway router (as set in options:lb_force_snat_ip) is seen,
              * UNSNAT it. */
@@ -4584,7 +4598,7 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                           ds_cstr(&match), ds_cstr(&actions));
         }
 
-        if (!od->l3dgw_port) {
+        if (!od->n_l3dgw_ports) {
             /* For gateway router, re-circulate every packet through
             * the DNAT zone.  This helps with two things.
             *
@@ -4602,36 +4616,33 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
             ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 50,
                           "ip", "flags.loopback = 1; ct_dnat;");
         } else {
-            /* For NAT on a distributed router, add flows to Ingress
-             * IP Routing table, Ingress ARP Resolution table, and
-             * Ingress Gateway Redirect Table that are not specific to a
-             * NAT rule. */
-
-            /* The highest priority IN_IP_ROUTING rule matches packets
-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages),
-             * with action "ip.ttl--; next;".  The IN_GW_REDIRECT table
-             * will take care of setting the outport. */
-            ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING, 300,
-                          REGBIT_NAT_REDIRECT" == 1", "ip.ttl--; next;");
-
-            /* The highest priority IN_ARP_RESOLVE rule matches packets
-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages),
-             * then sets eth.dst to the distributed gateway port's
-             * ethernet address. */
-            ds_clear(&actions);
-            ds_put_format(&actions, "eth.dst = %s; next;",
-                          od->l3dgw_port->lrp_networks.ea_s);
-            ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 200,
-                          REGBIT_NAT_REDIRECT" == 1", ds_cstr(&actions));
-
-            /* The highest priority IN_GW_REDIRECT rule redirects packets
-             * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages) to
-             * the central instance of the l3dgw_port for NAT processing. */
-            ds_clear(&actions);
-            ds_put_format(&actions, "outport = %s; next;",
-                          od->l3redirect_port->json_key);
-            ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 200,
-                          REGBIT_NAT_REDIRECT" == 1", ds_cstr(&actions));
+            /* For NAT on a distributed router, add flows to Ingress 
+             * ARP Resolution table, and Ingress Gateway Redirect Table
+             * that are not specific to a NAT rule. */
+            for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
+                struct ovn_port *l3dgw_port = od->l3dgw_ports[i];
+                /* The highest priority IN_ARP_RESOLVE rule matches packets
+                 * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages),
+                 * then sets eth.dst to the distributed gateway port's
+                 * ethernet address. */
+                ds_clear(&match);
+                ds_put_format(&match, REGBIT_NAT_REDIRECT" == 1 && "
+                              "outport == %s", l3dgw_port->json_key);
+                ds_clear(&actions);
+                ds_put_format(&actions, "eth.dst = %s; next;",
+                              l3dgw_port->lrp_networks.ea_s);
+                ovn_lflow_add(lflows, od, S_ROUTER_IN_ARP_RESOLVE, 200,
+                              ds_cstr(&match), ds_cstr(&actions));
+
+                /* The highest priority IN_GW_REDIRECT rule redirects packets
+                 * with REGBIT_NAT_REDIRECT (set in DNAT or UNSNAT stages) to
+                 * the central instance of the l3dgw_port for NAT processing. */
+                ds_clear(&actions);
+                ds_put_format(&actions, "outport = %s; next;",
+                              l3dgw_port->l3redirect_port->json_key);
+                ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 200,
+                              ds_cstr(&match), ds_cstr(&actions));
+            }
         }
 
         /* Load balancing and packet defrag are only valid on
@@ -4959,17 +4970,18 @@  build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         if (!od->nbr) {
             continue;
         }
-        if (od->l3dgw_port && od->l3redirect_port) {
+        for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
+            struct ovn_port *l3dgw_port = od->l3dgw_ports[i];
             /* For traffic with outport == l3dgw_port, if the
              * packet did not match any higher priority redirect
              * rule, then the traffic is redirected to the central
              * instance of the l3dgw_port. */
             ds_clear(&match);
             ds_put_format(&match, "outport == %s",
-                          od->l3dgw_port->json_key);
+                          l3dgw_port->json_key);
             ds_clear(&actions);
             ds_put_format(&actions, "outport = %s; next;",
-                          od->l3redirect_port->json_key);
+                          l3dgw_port->l3redirect_port->json_key);
             ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
                           ds_cstr(&match), ds_cstr(&actions));
 
diff --git a/ovn/ovn-nb.xml b/ovn/ovn-nb.xml
index c5ebbea..1286b1b 100644
--- a/ovn/ovn-nb.xml
+++ b/ovn/ovn-nb.xml
@@ -1112,8 +1112,7 @@ 
         <p>
           If set, this indicates that this logical router port represents
           a distributed gateway port that connects this router to a logical
-          switch with a localnet port.  There may be at most one such
-          logical router port on each logical router.
+          switch with a localnet port.
         </p>
 
         <p>
@@ -1273,7 +1272,14 @@ 
     </column>
 
     <column name="external_ip">
-      An IPv4 address.
+      <p>
+        An IPv4 address.
+      </p>
+
+      <p>
+        On distributed router, This address must be within the subnet of
+        the gateway port instance on the <code>redirect-chassis</code>.
+      </p>
     </column>
 
     <column name="external_mac">