diff mbox series

[ovs-dev,v3,2/2] northd: Use ct_(snat/dnat)_in_czone action for distributed routers.

Message ID 20211119163226.240709-1-numans@ovn.org
State Accepted
Headers show
Series [ovs-dev,v3,1/2] actions: Add new actions - ct_dnat_in_czone and ct_snat_in_czone. | expand

Checks

Context Check Description
ovsrobot/apply-robot success apply and check: success
ovsrobot/github-robot-_Build_and_Test success github build: passed
ovsrobot/github-robot-_ovn-kubernetes fail github build: failed

Commit Message

Numan Siddique Nov. 19, 2021, 4:32 p.m. UTC 
From: Numan Siddique <numans@ovn.org>

Make of use of these new actions for the distributed routers
for NAT.  These new actions ensure that both sNAT and dNAT
happens in the same zone.  This approach solves a couple of
problems:

 - The datapath flows generated for external traffic which requires
   dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.

 - Since there is only one zone, it would avoid multiple recirculations
   (improving the performance).

If the packet needs to be both sNATted and dNATted (for hairpin traffic
with source and destination on the same chassis), then sNAT is done
in a separate zone.  To detect this scenario, this patch adds a few
extra logical flows.  For each dnat_and_snat entry prior to this patch
ovn-northd was generating 9 logical flows and with this patch it now
generates 12 logical flows.

Similar approach can be taken for gateway routers.

Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
Signed-off-by: Numan Siddique <numans@ovn.org>
---

v2 -> v3
------
  * Addressed Mark's comments and updated the documentation.

v1 -> v2
------
  * Rebased and resolved conflicts.


 include/ovn/logical-fields.h |   1 +
 lib/logical-fields.c         |   4 +
 northd/northd.c              | 147 +++++++--
 northd/ovn-northd.8.xml      | 205 ++++++++++---
 tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
 tests/ovn.at                 |   2 +-
 tests/system-ovn.at          |  64 ++--
 7 files changed, 610 insertions(+), 388 deletions(-)

Comments

Mark Michelson Nov. 19, 2021, 6:10 p.m. UTC | #1 
Thanks for the update, Numan.

Acked-by: Mark Michelson <mmichels@redhat.com>

On 11/19/21 11:32, numans@ovn.org wrote:
> From: Numan Siddique <numans@ovn.org>
> 
> Make of use of these new actions for the distributed routers
> for NAT.  These new actions ensure that both sNAT and dNAT
> happens in the same zone.  This approach solves a couple of
> problems:
> 
>   - The datapath flows generated for external traffic which requires
>     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> 
>   - Since there is only one zone, it would avoid multiple recirculations
>     (improving the performance).
> 
> If the packet needs to be both sNATted and dNATted (for hairpin traffic
> with source and destination on the same chassis), then sNAT is done
> in a separate zone.  To detect this scenario, this patch adds a few
> extra logical flows.  For each dnat_and_snat entry prior to this patch
> ovn-northd was generating 9 logical flows and with this patch it now
> generates 12 logical flows.
> 
> Similar approach can be taken for gateway routers.
> 
> Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> Signed-off-by: Numan Siddique <numans@ovn.org>
> ---
> 
> v2 -> v3
> ------
>    * Addressed Mark's comments and updated the documentation.
> 
> v1 -> v2
> ------
>    * Rebased and resolved conflicts.
> 
> 
>   include/ovn/logical-fields.h |   1 +
>   lib/logical-fields.c         |   4 +
>   northd/northd.c              | 147 +++++++--
>   northd/ovn-northd.8.xml      | 205 ++++++++++---
>   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
>   tests/ovn.at                 |   2 +-
>   tests/system-ovn.at          |  64 ++--
>   7 files changed, 610 insertions(+), 388 deletions(-)
> 
> diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> index c9675f81c..2118f7933 100644
> --- a/include/ovn/logical-fields.h
> +++ b/include/ovn/logical-fields.h
> @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
>       MLF_LOOKUP_FDB_BIT = 8,
>       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
>       MLF_LOCALPORT_BIT = 10,
> +    MLF_USE_SNAT_ZONE = 11,
>   };
>   
>   /* MFF_LOG_FLAGS_REG flag assignments */
> diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> index 7b3d431e0..352a48c89 100644
> --- a/lib/logical-fields.c
> +++ b/lib/logical-fields.c
> @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
>                MLF_SKIP_SNAT_FOR_LB_BIT);
>       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
>                                flags_str);
> +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> +             MLF_USE_SNAT_ZONE);
> +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> +                             flags_str);
>   
>       /* Connection tracking state. */
>       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> diff --git a/northd/northd.c b/northd/northd.c
> index 0ff61deec..e4d051a94 100644
> --- a/northd/northd.c
> +++ b/northd/northd.c
> @@ -159,11 +159,14 @@ enum ovn_stage {
>       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
>                                                                         \
>       /* Logical router egress stages. */                               \
> -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> +                   "lr_out_chk_dnat_local")                                  \
> +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
>   
>   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
>       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> @@ -210,6 +213,7 @@ enum ovn_stage {
>   #define REGBIT_PKT_LARGER        "reg9[1]"
>   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
>   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
>   
>   /* Register to store the eth address associated to a router port for packets
>    * received in S_ROUTER_IN_ADMISSION.
> @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
>                                       undnat_match_p, est_actions,
>                                       &lb->nlb->header_);
>           } else {
> -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> -                                    undnat_match_p, "ct_dnat;",
> -                                    &lb->nlb->header_);
> +            ovn_lflow_add_with_hint(
> +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> +                &lb->nlb->header_);
>           }
>           free(undnat_match_p);
>   next:
> @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
>           uint16_t priority;
>   
>           /* Priority of logical flows corresponding to exempted_ext_ips is
> -         * +1 of the corresponding regulr NAT rule.
> +         * +2 of the corresponding regular NAT rule.
>            * For example, if we have following NAT rule and we associate
>            * exempted external ips to it:
>            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
>            * And now we associate exempted external ip address set to it.
>            * Now corresponding to above rule we will have following logical
>            * flows:
> -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
>            *                             action=(next;)
>            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
>            *
>            */
>           if (is_src) {
>               /* S_ROUTER_IN_DNAT uses priority 100 */
> -            priority = 100 + 1;
> +            priority = 100 + 2;
>           } else {
>               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> -            priority = count_1bits(ntohl(mask)) + 2;
> +            priority = count_1bits(ntohl(mask)) + 3;
>   
>               if (!od->is_gw_router) {
>                   priority += 128;
> @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>           /* Traffic received on l3dgw_port is subject to NAT. */
>           ds_clear(match);
>           ds_clear(actions);
> -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> -                      is_v6 ? "6" : "4", nat->external_ip,
> -                      od->l3dgw_ports[0]->json_key);
> +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
>           if (!distributed && od->n_l3dgw_ports) {
>               /* Flows for NAT rules that are centralized are only
>               * programmed on the gateway chassis. */
> @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>               ds_put_format(actions, "ip%s.dst=%s; next;",
>                             is_v6 ? "6" : "4", nat->logical_ip);
>           } else {
> -            ds_put_cstr(actions, "ct_snat;");
> +            ds_put_cstr(actions, "ct_snat_in_czone;");
>           }
>   
>           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
>                                   100, ds_cstr(match), ds_cstr(actions),
>                                   &nat->header_);
> +
> +        if (!stateless) {
> +            ds_clear(match);
> +            ds_clear(actions);
> +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> +                          is_v6 ? "6" : "4", nat->external_ip,
> +                          od->l3dgw_ports[0]->json_key);
> +            if (!distributed && od->n_l3dgw_ports) {
> +                /* Flows for NAT rules that are centralized are only
> +                * programmed on the gateway chassis. */
> +                ds_put_format(match, " && is_chassis_resident(%s)",
> +                            od->l3dgw_ports[0]->cr_port->json_key);
> +            }
> +            ds_put_cstr(actions, "ct_snat;");
> +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> +                                    100, ds_cstr(match), ds_cstr(actions),
> +                                    &nat->header_);
> +        }
>       }
>   }
>   
> @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>                   ds_put_format(actions, "ip%s.dst=%s; next;",
>                                 is_v6 ? "6" : "4", nat->logical_ip);
>               } else {
> -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
>                   if (nat->external_port_range[0]) {
>                       ds_put_format(actions, ",%s", nat->external_port_range);
>                   }
> @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>           ds_put_format(actions, "ip%s.src=%s; next;",
>                         is_v6 ? "6" : "4", nat->external_ip);
>       } else {
> -        ds_put_format(actions, "ct_dnat;");
> +        ds_put_format(actions,
> +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
>       }
>   
>       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>                               &nat->header_);
>   }
>   
> +static void
> +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> +                                const struct nbrec_nat *nat, struct ds *match,
> +                                struct ds *actions, bool distributed,
> +                                bool is_v6)
> +{
> +    /* Note that this only applies for NAT on a distributed router.
> +     */
> +    if (!od->n_l3dgw_ports) {
> +        return;
> +    }
> +
> +    ds_clear(match);
> +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> +                  is_v6 ? "6" : "4", nat->external_ip);
> +    if (distributed) {
> +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> +    } else {
> +        ds_put_format(match, "is_chassis_resident(%s)",
> +                      od->l3dgw_ports[0]->cr_port->json_key);
> +    }
> +
> +    ds_clear(actions);
> +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> +
> +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> +                            50, ds_cstr(match), ds_cstr(actions),
> +                            &nat->header_);
> +}
> +
>   static void
>   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
>                               const struct nbrec_nat *nat, struct ds *match,
> @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
>           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
>                         is_v6 ? "6" : "4", nat->logical_ip,
>                         od->l3dgw_ports[0]->json_key);
> -        if (!distributed && od->n_l3dgw_ports) {
> -            /* Flows for NAT rules that are centralized are only
> -            * programmed on the gateway chassis. */
> -            priority += 128;
> -            ds_put_format(match, " && is_chassis_resident(%s)",
> -                          od->l3dgw_ports[0]->cr_port->json_key);
> -        } else if (distributed) {
> -            priority += 128;
> -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> -                          nat->logical_port);
> +        if (od->n_l3dgw_ports) {
> +            if (distributed) {
> +                ovs_assert(nat->logical_port);
> +                priority += 128;
> +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> +                              nat->logical_port);
> +            } else {
> +                /* Flows for NAT rules that are centralized are only
> +                * programmed on the gateway chassis. */
> +                priority += 128;
> +                ds_put_format(match, " && is_chassis_resident(%s)",
> +                              od->l3dgw_ports[0]->cr_port->json_key);
> +            }
>           }
>           ds_clear(actions);
>   
> @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
>               ds_put_format(actions, "ip%s.src=%s; next;",
>                             is_v6 ? "6" : "4", nat->external_ip);
>           } else {
> -            ds_put_format(actions, "ct_snat(%s",
> +            ds_put_format(actions, "ct_snat_in_czone(%s",
>                           nat->external_ip);
>               if (nat->external_port_range[0]) {
>                   ds_put_format(actions, ",%s", nat->external_port_range);
> @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
>           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
>                                   priority, ds_cstr(match),
>                                   ds_cstr(actions), &nat->header_);
> +
> +        if (!stateless) {
> +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> +            ds_clear(actions);
> +            if (distributed) {
> +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> +                              ETH_ADDR_ARGS(mac));
> +            }
> +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> +                          nat->external_ip);
> +            if (nat->external_port_range[0]) {
> +                ds_put_format(actions, ",%s", nat->external_port_range);
> +            }
> +            ds_put_format(actions, ");");
> +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> +                                    priority + 1, ds_cstr(match),
> +                                    ds_cstr(actions), &nat->header_);
> +        }
>       }
>   }
>   
> @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
>       /* Packets are allowed by default. */
>       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
>       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
>   
> @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
>        * not committed, it would produce ongoing datapath flows with the ct.new
>        * flag set. Some NICs are unable to offload these flows.
>        */
> -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> -        (od->nbr->n_nat || od->has_lb_vip)) {
> +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
>           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
>                         "ip", "flags.loopback = 1; ct_dnat;");
>           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
>               }
>           }
>   
> +        /* S_ROUTER_OUT_DNAT_LOCAL */
> +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> +                                        distributed, is_v6);
> +
>           /* S_ROUTER_OUT_UNDNAT */
>           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
>                                         mac, is_v6);
> @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
>                             "clone { ct_clear; "
>                             "inport = outport; outport = \"\"; "
>                             "eth.dst <-> eth.src; "
> -                          "flags = 0; flags.loopback = 1; ");
> +                          "flags = 0; flags.loopback = 1; "
> +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
>               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
>                   ds_put_format(actions, "reg%d = 0; ", j);
>               }
> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> index 21d83718c..e39e6e805 100644
> --- a/northd/ovn-northd.8.xml
> +++ b/northd/ovn-northd.8.xml
> @@ -2879,23 +2879,65 @@ icmp6 {
>           <p>
>             For each configuration in the OVN Northbound database, that asks
>             to change the source IP address of a packet from <var>A</var> to
> -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> -          <code>ip &amp;&amp;
> -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> -          where <var>GW</var> is the logical router gateway port, with an
> -          action <code>ct_snat;</code>. If the NAT rule is of type
> -          dnat_and_snat and has <code>stateless=true</code> in the
> -          options, then the action would be <code>ip4/6.dst=
> -          (<var>B</var>)</code>.
> +          <var>B</var>, two priority-100 flows are added.
>           </p>
>   
>           <p>
>             If the NAT rule cannot be handled in a distributed manner, then
> -          the priority-100 flow above is only programmed on the
> +          the below priority-100 flows are only programmed on the
>             gateway chassis.
>           </p>
>   
> +        <ul>
> +          <li>
> +            <p>
> +              The first flow matches <code>ip &amp;&amp;
> +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> +              &amp;&amp; flags.loopback == 0</code> or
> +              <code>ip &amp;&amp;
> +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> +              &amp;&amp; flags.loopback == 0</code>
> +              where <var>GW</var> is the logical router gateway port, with an
> +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> +              zone.  If the NAT rule is of type dnat_and_snat and has
> +              <code>stateless=true</code> in the options, then the action
> +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> +            </p>
> +
> +            <p>
> +              If the NAT entry is of type <code>snat</code>, then there is an
> +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> +              </code> where <var>cr-GW</var> is the chassis resident port of
> +              <var>GW</var>.
> +            </p>
> +          </li>
> +
> +          <li>
> +            <p>
> +              The second flow matches <code>ip &amp;&amp;
> +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> +              flags.use_snat_zone == 1</code> or
> +              <code>ip &amp;&amp;
> +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> +              flags.use_snat_zone == 1</code>
> +              where <var>GW</var> is the logical router gateway port, with an
> +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> +              NAT rule is of type dnat_and_snat and has
> +              <code>stateless=true</code> in the options, then the action
> +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> +            </p>
> +
> +            <p>
> +              If the NAT entry is of type <code>snat</code>, then there is an
> +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> +              </code> where <var>cr-GW</var> is the chassis resident port of
> +              <var>GW</var>.
> +            </p>
> +          </li>
> +        </ul>
> +
>           <p>
>             A priority-0 logical flow with match <code>1</code> has actions
>             <code>next;</code>.
> @@ -4031,7 +4073,43 @@ nd_ns {
>         </li>
>       </ul>
>   
> -    <h3>Egress Table 0: UNDNAT</h3>
> +    <h3>Egress Table 0: Check DNAT local </h3>
> +
> +    <p>
> +      This table checks if the packet needs to be DNATed in the router ingress
> +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> +      to the ingress pipeline.  This check is done only for routers configured
> +      with distributed gateway ports and NAT entries.  This check is done
> +      so that SNAT and DNAT is done in different zones instead of a common
> +      zone.
> +    </p>
> +
> +    <ul>
> +      <li>
> +        <p>
> +          For each NAT rule in the OVN Northbound database on a
> +          distributed router, a priority-50 logical flow with match
> +          <code>ip4.dst == <var>E</var> &amp;&amp;
> +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> +          external IP address specified in the NAT rule, <var>GW</var>
> +          is the logical router distributed gateway port. For dnat_and_snat
> +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> +          If <ref column="logical_port"
> +          table="NAT" db="OVN_Northbound"/> column of
> +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> +          <var>P</var> is the <code>chassisredirect port</code> of
> +          <var>GW</var> with the actions:
> +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> +        </p>
> +      </li>
> +
> +      <li>
> +        A priority-0 logical flow with match <code>1</code> has actions
> +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> +      </li>
> +    </ul>
> +
> +    <h3>Egress Table 1: UNDNAT</h3>
>   
>       <p>
>         This is for already established connections' reverse traffic.
> @@ -4040,6 +4118,23 @@ nd_ns {
>         is unDNATed here.
>       </p>
>   
> +    <ul>
> +      <li>
> +        A priority-0 logical flow with match <code>1</code> has actions
> +        <code>next;</code>.
> +      </li>
> +    </ul>
> +
> +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> +
> +    <ul>
> +      <li>
> +        For all IP packets, a priority-50 flow with an action
> +        <code>flags.loopback = 1; ct_dnat;</code>.
> +      </li>
> +    </ul>
> +
> +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
>       <ul>
>         <li>
>           <p>
> @@ -4050,9 +4145,9 @@ nd_ns {
>             gateway chassis that matches
>             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
>             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> -          router gateway port with an action <code>ct_dnat;</code>. If the
> -          backend IPv4 address <var>B</var> is also configured with L4 port
> -          <var>PORT</var> of protocol <var>P</var>, then the
> +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> +          If the backend IPv4 address <var>B</var> is also configured with
> +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
>             match also includes <code>P.src</code> == <var>PORT</var>.  These
>             flows are not added for load balancers with IPv6 <var>VIPs</var>.
>           </p>
> @@ -4072,7 +4167,7 @@ nd_ns {
>             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
>             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
>             is the logical router gateway port, with an action
> -          <code>ct_dnat;</code>. If the NAT rule is of type
> +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
>             dnat_and_snat and has <code>stateless=true</code> in the
>             options, then the action would be <code>ip4/6.src=
>             (<var>B</var>)</code>.
> @@ -4081,7 +4176,7 @@ nd_ns {
>           <p>
>             If the NAT rule cannot be handled in a distributed manner, then
>             the priority-100 flow above is only programmed on the
> -          gateway chassis.
> +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
>           </p>
>   
>           <p>
> @@ -4094,26 +4189,17 @@ nd_ns {
>           </p>
>         </li>
>   
> -      <li>
> -        For all IP packets, a priority-50 flow with an action
> -        <code>flags.loopback = 1; ct_dnat;</code>.
> -      </li>
> -
> -      <li>
> -        A priority-0 logical flow with match <code>1</code> has actions
> -        <code>next;</code>.
> -      </li>
>       </ul>
>   
> -    <h3>Egress Table 1: Post UNDNAT</h3>
> +    <h3>Egress Table 2: Post UNDNAT</h3>
>   
>       <p>
>         <ul>
>           <li>
>             A priority-50 logical flow is added that commits any untracked flows
> -          from the previous table <code>lr_out_undnat</code>. This flow
> -          matches on <code>ct.new &amp;&amp; ip</code> with action
> -          <code>ct_commit { } ; next; </code>.
> +          from the previous table <code>lr_out_undnat</code> for Gateway
> +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> +          with action <code>ct_commit { } ; next; </code>.
>           </li>
>   
>           <li>
> @@ -4124,7 +4210,7 @@ nd_ns {
>         </ul>
>       </p>
>   
> -    <h3>Egress Table 2: SNAT</h3>
> +    <h3>Egress Table 3: SNAT</h3>
>   
>       <p>
>         Packets that are configured to be SNATed get their source IP address
> @@ -4140,7 +4226,7 @@ nd_ns {
>         </li>
>       </ul>
>   
> -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> +    <p>Egress Table 3: SNAT on Gateway Routers</p>
>   
>       <ul>
>         <li>
> @@ -4239,7 +4325,7 @@ nd_ns {
>         </li>
>       </ul>
>   
> -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> +    <p>Egress Table 3: SNAT on Distributed Routers</p>
>   
>       <ul>
>         <li>
> @@ -4247,28 +4333,47 @@ nd_ns {
>             For each configuration in the OVN Northbound database, that asks
>             to change the source IP address of a packet from an IP address of
>             <var>A</var> or to change the source IP address of a packet that
> -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> -          outport == <var>GW</var></code>, where <var>GW</var> is the
> -          logical router gateway port, with an action
> -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> -          is calculated based on the mask of <var>A</var>, with matches
> -          having larger masks getting higher priorities. If the NAT rule
> -          is of type dnat_and_snat and has <code>stateless=true</code>
> -          in the options, then the action would be <code>ip4/6.src=
> -          (<var>B</var>)</code>.
> +          belongs to network <var>A</var> to <var>B</var>, two flows are
> +          added.  The priority <var>P</var> of these flows are calculated
> +          based on the mask of <var>A</var>, with matches having larger
> +          masks getting higher priorities.
>           </p>
>   
>           <p>
>             If the NAT rule cannot be handled in a distributed manner, then
> -          the flow above is only programmed on the
> -          gateway chassis increasing flow priority by 128 in
> -          order to be run first
> +          the below flows are only programmed on the gateway chassis increasing
> +          flow priority by 128 in order to be run first.
>           </p>
>   
> +        <ul>
> +          <li>
> +            The first flow is added with the calculated priority <var>P</var>
> +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> +            outport == <var>GW</var></code>, where <var>GW</var> is the
> +            logical router gateway port, with an action
> +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> +            common zone.  If the NAT rule is of type dnat_and_snat and has
> +            <code>stateless=true</code> in the options, then the action
> +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> +          </li>
> +
> +          <li>
> +            The second flow is added with the calculated priority
> +            <code><var>P</var> + 1 </code> and match
> +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> +            outport == <var>GW</var> &amp;&amp;
> +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> +            logical router gateway port, with an action
> +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> +            If the NAT rule is of type dnat_and_snat and has
> +            <code>stateless=true</code> in the options, then the action would
> +            be <code>ip4/6.src=(<var>B</var>)</code>.
> +          </li>
> +        </ul>
> +
>           <p>
>             If the NAT rule can be handled in a distributed manner, then
> -          there is an additional action
> +          there is an additional action (for both the flows)
>             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
>             is the ethernet address associated with the IP address
>             <var>A</var> in the NAT rule.  This allows upstream MAC
> @@ -4284,7 +4389,8 @@ nd_ns {
>   
>           <p>
>             If the NAT rule has <code>exempted_ext_ips</code> set, then
> -          there is an additional flow configured at the priority + 1 of
> +          there is an additional flow configured at the priority
> +          <code><var>P</var> + 2 </code> of
>             corresponding NAT rule. The flow matches if destination ip
>             is an <code>exempted_ext_ip</code> and the action is <code>next;
>             </code>. This flow is used to bypass the ct_snat action for a flow
> @@ -4299,7 +4405,7 @@ nd_ns {
>         </li>
>       </ul>
>   
> -    <h3>Egress Table 3: Egress Loopback</h3>
> +    <h3>Egress Table 4: Egress Loopback</h3>
>   
>       <p>
>         For distributed logical routers where one of the logical router
> @@ -4344,6 +4450,7 @@ clone {
>       outport = "";
>       flags = 0;
>       flags.loopback = 1;
> +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
>       reg0 = 0;
>       reg1 = 0;
>       ...
> @@ -4368,7 +4475,7 @@ clone {
>         </li>
>       </ul>
>   
> -    <h3>Egress Table 4: Delivery</h3>
> +    <h3>Egress Table 5: Delivery</h3>
>   
>       <p>
>         Packets that reach this table are ready for delivery.  It contains:
> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> index 85b47a18f..70ec5e2e3 100644
> --- a/tests/ovn-northd.at
> +++ b/tests/ovn-northd.at
> @@ -877,25 +877,25 @@ check_flow_match_sets() {
>   echo
>   echo "IPv4: stateful"
>   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> -check_flow_match_sets 2 2 3 0 0 0 0
> +check_flow_match_sets 3 4 2 0 0 0 0
>   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
>   
>   echo
>   echo "IPv4: stateless"
>   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> -check_flow_match_sets 2 0 1 2 2 0 0
> +check_flow_match_sets 2 0 0 2 2 0 0
>   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
>   
>   echo
>   echo "IPv6: stateful"
>   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> -check_flow_match_sets 2 2 3 0 0 0 0
> +check_flow_match_sets 3 4 2 0 0 0 0
>   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
>   
>   echo
>   echo "IPv6: stateless"
>   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> -check_flow_match_sets 2 0 1 0 0 2 2
> +check_flow_match_sets 2 0 0 0 0 2 2
>   
>   AT_CLEANUP
>   ])
> @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
>   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
>   
>   AT_CAPTURE_FILE([sbflows])
> -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
>   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> -  [0], [1
> +  [0], [2
>   1
>   ])
>   
> @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
>   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
>   
>   AT_CAPTURE_FILE([sbflows2])
> -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
>   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> -  [1], [1
> +  [1], [2
>   0
>   ])
>   
> @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
>   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
>   
>   AT_CAPTURE_FILE([sbflows3])
> -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
>   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
>   
>   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
>   ovn-sbctl dump-flows CR > crflows
>   AT_CAPTURE_FILE([crflows])
>   
> -AT_CHECK([
> -  grep -c lr_out_snat drflows
> -  grep -c lr_out_snat crflows
> -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> -3
> -3
> -1
> -1
> +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> +])
> +
> +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
>   ])
>   
> +
>   # SNAT with DISALLOWED_IPs
>   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
>   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
>   ovn-sbctl dump-flows CR > crflows2
>   AT_CAPTURE_FILE([crflows2])
>   
> -AT_CHECK([
> -  grep -c lr_out_snat drflows2
> -  grep -c lr_out_snat crflows2
> -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> -4
> -4
> -1
> -1
> -1
> -1
> +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> +])
> +
> +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
>   ])
>   
>   # Stateful FIP with ALLOWED_IPs
> @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
>   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
>   
>   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
>   
> -ovn-nbctl show DR
> -ovn-sbctl dump-flows DR
> -ovn-nbctl show CR
> -ovn-sbctl dump-flows CR
> -
> -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> -wc -l`])
> -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> -wc -l`])
> +ovn-sbctl dump-flows DR > drflows3
> +AT_CAPTURE_FILE([drflows2])
> +ovn-sbctl dump-flows CR > crflows3
> +AT_CAPTURE_FILE([crflows2])
>   
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
>   ])
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> +
> +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
>   ])
>   
>   # Stateful FIP with DISALLOWED_IPs
> @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
>   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
>   
>   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
>   
> -ovn-nbctl show DR
> -ovn-sbctl dump-flows DR
> -ovn-nbctl show CR
> -ovn-sbctl dump-flows CR
> -
> -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> -wc -l`])
> -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> -wc -l`])
> +ovn-sbctl dump-flows DR > drflows4
> +AT_CAPTURE_FILE([drflows2])
> +ovn-sbctl dump-flows CR > crflows4
> +AT_CAPTURE_FILE([crflows2])
>   
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
>   ])
>   
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
>   ])
>   
>   # Stateless FIP with DISALLOWED_IPs
> @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
>   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
>   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
>   
> -ovn-nbctl show DR
> -ovn-sbctl dump-flows DR
> -
> -ovn-nbctl show CR
> -ovn-sbctl dump-flows CR
> -
> -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> -wc -l`])
> -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> -wc -l`])
> +ovn-sbctl dump-flows DR > drflows5
> +AT_CAPTURE_FILE([drflows2])
> +ovn-sbctl dump-flows CR > crflows5
> +AT_CAPTURE_FILE([crflows2])
>   
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
>   ])
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> +
> +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
>   ])
>   
>   # Stateful FIP with DISALLOWED_IPs
> @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
>   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
>   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
>   
> -ovn-nbctl show DR
> -ovn-sbctl dump-flows DR
> -ovn-nbctl show CR
> -ovn-sbctl dump-flows CR
> +ovn-nbctl --wait=sb sync
>   
> -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> -wc -l`])
> -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> -wc -l`])
> +ovn-sbctl dump-flows DR > drflows6
> +AT_CAPTURE_FILE([drflows2])
> +ovn-sbctl dump-flows CR > crflows6
> +AT_CAPTURE_FILE([crflows2])
>   
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
>   ])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> -])
> -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> +
> +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
>   ])
>   
>   AT_CLEANUP
> @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
>   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
>   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
>   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
>     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
>   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
>   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
>   AT_CLEANUP
> @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
>   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
>   check ovn-nbctl --wait=sb sync
>   
> +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> +
>   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
>   
>   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
>   # Create few dnat_and_snat entries
> @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +])
> +
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
>   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
>   
>   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
>     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
>   ])
>   
>   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
>   
>   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> +])
> +
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
>   ])
>   
>   # Associate load balancer to lr0
> @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
>   
>   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
>     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
>   ])
>   
>   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
>   
>   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
>     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
>     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> +])
> +
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
>   ])
>   
>   # Make the logical router as Gateway router
> @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +])
> +
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
>   ])
>   
>   # Set lb force snat logical router.
> @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> +])
> +
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
>   ])
>   
>   # Add a LB VIP same as router ip.
> @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +])
> +
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
>   ])
>   
>   # Add IPv6 router port and LB.
> @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> +])
> +
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
>   ])
>   
>   check ovn-nbctl lrp-del lr0-sw0
> @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
>     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
>   ])
>   
> -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> +])
> +
> +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
>   ])
>   
> -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
>   ])
>   
> -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
>   ])
>   
>   AT_CLEANUP
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 0d606b42f..ae5744407 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
>   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
>     grep -c "$regex" sbflows;
>   done], [0], [0
> -1
> +0
>   2
>   2
>   ])
> diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> index c9f5771c9..7f6cb32dc 100644
> --- a/tests/system-ovn.at
> +++ b/tests/system-ovn.at
> @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
>   
>   ovn-nbctl list load_balancer
>   ovn-sbctl dump-flows R2
> -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
>   grep 'nat(src=20.0.0.2)'])
>   
>   dnl Test load-balancing that includes L4 ports in NAT.
> @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
>   
>   ovn-nbctl list load_balancer
>   ovn-sbctl dump-flows R2
> -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
>   grep 'nat(src=20.0.0.2)'])
>   
>   rm -f wget*.log
> @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
>   ])
>   
> +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> +
>   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> -# from 172.16.1.4
> +# from fd20::4
>   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
>   [0], [dnl
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> +ovs-appctl dpctl/dump-conntrack | grep icmpv6
>   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> +])
> +
> +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
>   ])
>   
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> +# We verify that the connection is not tracked.
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
>   ])
>   
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> +# We verify that the connection is not tracked.
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
>   ])
>   
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> +# We verify that the connection is not tracked.
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
>   ])
>   
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> +
>   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
>   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
>   [0], [dnl
> @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
>   
>   # Check conntrack entries.  First SNAT of 'foo1' address happens.
>   # Then DNAT of 'bar1' address happens (listed first below).
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
>   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
>   ])
>   
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
>   
>   # Check conntrack entries.  First SNAT of 'foo2' address happens.
>   # Then DNAT of 'bar1' address happens (listed first below).
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
>   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> -])
> -
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
>   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
>   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> -])
> -
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
>   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
>   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
>   3 packets transmitted, 3 received, 0% packet loss, time 0ms
>   ])
>   
> -# We verify that the connection is tracked but not NATted. This is due to the
> -# unDNAT table in the egress router pipeline
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> -])
> -
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
>   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
>   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
>   # Then DNAT of 'bar1' address happens (listed first below).
>   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
>   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
>   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
>   ])
> @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
>   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
>   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
>   ])
>   
>   OVS_APP_EXIT_AND_WAIT([ovn-controller])
>
Numan Siddique Nov. 19, 2021, 11:48 p.m. UTC | #2 
On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <mmichels@redhat.com> wrote:
>
> Thanks for the update, Numan.
>
> Acked-by: Mark Michelson <mmichels@redhat.com>

Thanks.  I applied both the patches to the main branch.

Numan

>
> On 11/19/21 11:32, numans@ovn.org wrote:
> > From: Numan Siddique <numans@ovn.org>
> >
> > Make of use of these new actions for the distributed routers
> > for NAT.  These new actions ensure that both sNAT and dNAT
> > happens in the same zone.  This approach solves a couple of
> > problems:
> >
> >   - The datapath flows generated for external traffic which requires
> >     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> >
> >   - Since there is only one zone, it would avoid multiple recirculations
> >     (improving the performance).
> >
> > If the packet needs to be both sNATted and dNATted (for hairpin traffic
> > with source and destination on the same chassis), then sNAT is done
> > in a separate zone.  To detect this scenario, this patch adds a few
> > extra logical flows.  For each dnat_and_snat entry prior to this patch
> > ovn-northd was generating 9 logical flows and with this patch it now
> > generates 12 logical flows.
> >
> > Similar approach can be taken for gateway routers.
> >
> > Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > Signed-off-by: Numan Siddique <numans@ovn.org>
> > ---
> >
> > v2 -> v3
> > ------
> >    * Addressed Mark's comments and updated the documentation.
> >
> > v1 -> v2
> > ------
> >    * Rebased and resolved conflicts.
> >
> >
> >   include/ovn/logical-fields.h |   1 +
> >   lib/logical-fields.c         |   4 +
> >   northd/northd.c              | 147 +++++++--
> >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> >   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
> >   tests/ovn.at                 |   2 +-
> >   tests/system-ovn.at          |  64 ++--
> >   7 files changed, 610 insertions(+), 388 deletions(-)
> >
> > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> > index c9675f81c..2118f7933 100644
> > --- a/include/ovn/logical-fields.h
> > +++ b/include/ovn/logical-fields.h
> > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> >       MLF_LOOKUP_FDB_BIT = 8,
> >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> >       MLF_LOCALPORT_BIT = 10,
> > +    MLF_USE_SNAT_ZONE = 11,
> >   };
> >
> >   /* MFF_LOG_FLAGS_REG flag assignments */
> > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > index 7b3d431e0..352a48c89 100644
> > --- a/lib/logical-fields.c
> > +++ b/lib/logical-fields.c
> > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
> >                MLF_SKIP_SNAT_FOR_LB_BIT);
> >       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
> >                                flags_str);
> > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > +             MLF_USE_SNAT_ZONE);
> > +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> > +                             flags_str);
> >
> >       /* Connection tracking state. */
> >       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> > diff --git a/northd/northd.c b/northd/northd.c
> > index 0ff61deec..e4d051a94 100644
> > --- a/northd/northd.c
> > +++ b/northd/northd.c
> > @@ -159,11 +159,14 @@ enum ovn_stage {
> >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
> >                                                                         \
> >       /* Logical router egress stages. */                               \
> > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> > +                   "lr_out_chk_dnat_local")                                  \
> > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
> >
> >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
> >       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> > @@ -210,6 +213,7 @@ enum ovn_stage {
> >   #define REGBIT_PKT_LARGER        "reg9[1]"
> >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> >
> >   /* Register to store the eth address associated to a router port for packets
> >    * received in S_ROUTER_IN_ADMISSION.
> > @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> >                                       undnat_match_p, est_actions,
> >                                       &lb->nlb->header_);
> >           } else {
> > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > -                                    undnat_match_p, "ct_dnat;",
> > -                                    &lb->nlb->header_);
> > +            ovn_lflow_add_with_hint(
> > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> > +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > +                &lb->nlb->header_);
> >           }
> >           free(undnat_match_p);
> >   next:
> > @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> >           uint16_t priority;
> >
> >           /* Priority of logical flows corresponding to exempted_ext_ips is
> > -         * +1 of the corresponding regulr NAT rule.
> > +         * +2 of the corresponding regular NAT rule.
> >            * For example, if we have following NAT rule and we associate
> >            * exempted external ips to it:
> >            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> > @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> >            * And now we associate exempted external ip address set to it.
> >            * Now corresponding to above rule we will have following logical
> >            * flows:
> > -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> > +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
> >            *                             action=(next;)
> >            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
> >            *
> >            */
> >           if (is_src) {
> >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > -            priority = 100 + 1;
> > +            priority = 100 + 2;
> >           } else {
> >               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> > -            priority = count_1bits(ntohl(mask)) + 2;
> > +            priority = count_1bits(ntohl(mask)) + 3;
> >
> >               if (!od->is_gw_router) {
> >                   priority += 128;
> > @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >           /* Traffic received on l3dgw_port is subject to NAT. */
> >           ds_clear(match);
> >           ds_clear(actions);
> > -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > -                      is_v6 ? "6" : "4", nat->external_ip,
> > -                      od->l3dgw_ports[0]->json_key);
> > +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> > +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
> >           if (!distributed && od->n_l3dgw_ports) {
> >               /* Flows for NAT rules that are centralized are only
> >               * programmed on the gateway chassis. */
> > @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >               ds_put_format(actions, "ip%s.dst=%s; next;",
> >                             is_v6 ? "6" : "4", nat->logical_ip);
> >           } else {
> > -            ds_put_cstr(actions, "ct_snat;");
> > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> >           }
> >
> >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> >                                   100, ds_cstr(match), ds_cstr(actions),
> >                                   &nat->header_);
> > +
> > +        if (!stateless) {
> > +            ds_clear(match);
> > +            ds_clear(actions);
> > +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> > +                          is_v6 ? "6" : "4", nat->external_ip,
> > +                          od->l3dgw_ports[0]->json_key);
> > +            if (!distributed && od->n_l3dgw_ports) {
> > +                /* Flows for NAT rules that are centralized are only
> > +                * programmed on the gateway chassis. */
> > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > +                            od->l3dgw_ports[0]->cr_port->json_key);
> > +            }
> > +            ds_put_cstr(actions, "ct_snat;");
> > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > +                                    100, ds_cstr(match), ds_cstr(actions),
> > +                                    &nat->header_);
> > +        }
> >       }
> >   }
> >
> > @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >                   ds_put_format(actions, "ip%s.dst=%s; next;",
> >                                 is_v6 ? "6" : "4", nat->logical_ip);
> >               } else {
> > -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> > +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
> >                   if (nat->external_port_range[0]) {
> >                       ds_put_format(actions, ",%s", nat->external_port_range);
> >                   }
> > @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >           ds_put_format(actions, "ip%s.src=%s; next;",
> >                         is_v6 ? "6" : "4", nat->external_ip);
> >       } else {
> > -        ds_put_format(actions, "ct_dnat;");
> > +        ds_put_format(actions,
> > +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> >       }
> >
> >       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >                               &nat->header_);
> >   }
> >
> > +static void
> > +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> > +                                const struct nbrec_nat *nat, struct ds *match,
> > +                                struct ds *actions, bool distributed,
> > +                                bool is_v6)
> > +{
> > +    /* Note that this only applies for NAT on a distributed router.
> > +     */
> > +    if (!od->n_l3dgw_ports) {
> > +        return;
> > +    }
> > +
> > +    ds_clear(match);
> > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > +                  is_v6 ? "6" : "4", nat->external_ip);
> > +    if (distributed) {
> > +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> > +    } else {
> > +        ds_put_format(match, "is_chassis_resident(%s)",
> > +                      od->l3dgw_ports[0]->cr_port->json_key);
> > +    }
> > +
> > +    ds_clear(actions);
> > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> > +
> > +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > +                            50, ds_cstr(match), ds_cstr(actions),
> > +                            &nat->header_);
> > +}
> > +
> >   static void
> >   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >                               const struct nbrec_nat *nat, struct ds *match,
> > @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> >                         is_v6 ? "6" : "4", nat->logical_ip,
> >                         od->l3dgw_ports[0]->json_key);
> > -        if (!distributed && od->n_l3dgw_ports) {
> > -            /* Flows for NAT rules that are centralized are only
> > -            * programmed on the gateway chassis. */
> > -            priority += 128;
> > -            ds_put_format(match, " && is_chassis_resident(%s)",
> > -                          od->l3dgw_ports[0]->cr_port->json_key);
> > -        } else if (distributed) {
> > -            priority += 128;
> > -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > -                          nat->logical_port);
> > +        if (od->n_l3dgw_ports) {
> > +            if (distributed) {
> > +                ovs_assert(nat->logical_port);
> > +                priority += 128;
> > +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > +                              nat->logical_port);
> > +            } else {
> > +                /* Flows for NAT rules that are centralized are only
> > +                * programmed on the gateway chassis. */
> > +                priority += 128;
> > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > +                              od->l3dgw_ports[0]->cr_port->json_key);
> > +            }
> >           }
> >           ds_clear(actions);
> >
> > @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >               ds_put_format(actions, "ip%s.src=%s; next;",
> >                             is_v6 ? "6" : "4", nat->external_ip);
> >           } else {
> > -            ds_put_format(actions, "ct_snat(%s",
> > +            ds_put_format(actions, "ct_snat_in_czone(%s",
> >                           nat->external_ip);
> >               if (nat->external_port_range[0]) {
> >                   ds_put_format(actions, ",%s", nat->external_port_range);
> > @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> >                                   priority, ds_cstr(match),
> >                                   ds_cstr(actions), &nat->header_);
> > +
> > +        if (!stateless) {
> > +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > +            ds_clear(actions);
> > +            if (distributed) {
> > +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> > +                              ETH_ADDR_ARGS(mac));
> > +            }
> > +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > +                          nat->external_ip);
> > +            if (nat->external_port_range[0]) {
> > +                ds_put_format(actions, ",%s", nat->external_port_range);
> > +            }
> > +            ds_put_format(actions, ");");
> > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > +                                    priority + 1, ds_cstr(match),
> > +                                    ds_cstr(actions), &nat->header_);
> > +        }
> >       }
> >   }
> >
> > @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> >       /* Packets are allowed by default. */
> >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
> >       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> >
> > @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> >        * not committed, it would produce ongoing datapath flows with the ct.new
> >        * flag set. Some NICs are unable to offload these flows.
> >        */
> > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> >                         "ip", "flags.loopback = 1; ct_dnat;");
> >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> >               }
> >           }
> >
> > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> > +                                        distributed, is_v6);
> > +
> >           /* S_ROUTER_OUT_UNDNAT */
> >           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
> >                                         mac, is_v6);
> > @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> >                             "clone { ct_clear; "
> >                             "inport = outport; outport = \"\"; "
> >                             "eth.dst <-> eth.src; "
> > -                          "flags = 0; flags.loopback = 1; ");
> > +                          "flags = 0; flags.loopback = 1; "
> > +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
> >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> >                   ds_put_format(actions, "reg%d = 0; ", j);
> >               }
> > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > index 21d83718c..e39e6e805 100644
> > --- a/northd/ovn-northd.8.xml
> > +++ b/northd/ovn-northd.8.xml
> > @@ -2879,23 +2879,65 @@ icmp6 {
> >           <p>
> >             For each configuration in the OVN Northbound database, that asks
> >             to change the source IP address of a packet from <var>A</var> to
> > -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> > -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> > -          <code>ip &amp;&amp;
> > -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> > -          where <var>GW</var> is the logical router gateway port, with an
> > -          action <code>ct_snat;</code>. If the NAT rule is of type
> > -          dnat_and_snat and has <code>stateless=true</code> in the
> > -          options, then the action would be <code>ip4/6.dst=
> > -          (<var>B</var>)</code>.
> > +          <var>B</var>, two priority-100 flows are added.
> >           </p>
> >
> >           <p>
> >             If the NAT rule cannot be handled in a distributed manner, then
> > -          the priority-100 flow above is only programmed on the
> > +          the below priority-100 flows are only programmed on the
> >             gateway chassis.
> >           </p>
> >
> > +        <ul>
> > +          <li>
> > +            <p>
> > +              The first flow matches <code>ip &amp;&amp;
> > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > +              &amp;&amp; flags.loopback == 0</code> or
> > +              <code>ip &amp;&amp;
> > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > +              &amp;&amp; flags.loopback == 0</code>
> > +              where <var>GW</var> is the logical router gateway port, with an
> > +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> > +              zone.  If the NAT rule is of type dnat_and_snat and has
> > +              <code>stateless=true</code> in the options, then the action
> > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > +            </p>
> > +
> > +            <p>
> > +              If the NAT entry is of type <code>snat</code>, then there is an
> > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > +              <var>GW</var>.
> > +            </p>
> > +          </li>
> > +
> > +          <li>
> > +            <p>
> > +              The second flow matches <code>ip &amp;&amp;
> > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > +              flags.use_snat_zone == 1</code> or
> > +              <code>ip &amp;&amp;
> > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > +              flags.use_snat_zone == 1</code>
> > +              where <var>GW</var> is the logical router gateway port, with an
> > +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> > +              NAT rule is of type dnat_and_snat and has
> > +              <code>stateless=true</code> in the options, then the action
> > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > +            </p>
> > +
> > +            <p>
> > +              If the NAT entry is of type <code>snat</code>, then there is an
> > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > +              <var>GW</var>.
> > +            </p>
> > +          </li>
> > +        </ul>
> > +
> >           <p>
> >             A priority-0 logical flow with match <code>1</code> has actions
> >             <code>next;</code>.
> > @@ -4031,7 +4073,43 @@ nd_ns {
> >         </li>
> >       </ul>
> >
> > -    <h3>Egress Table 0: UNDNAT</h3>
> > +    <h3>Egress Table 0: Check DNAT local </h3>
> > +
> > +    <p>
> > +      This table checks if the packet needs to be DNATed in the router ingress
> > +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> > +      to the ingress pipeline.  This check is done only for routers configured
> > +      with distributed gateway ports and NAT entries.  This check is done
> > +      so that SNAT and DNAT is done in different zones instead of a common
> > +      zone.
> > +    </p>
> > +
> > +    <ul>
> > +      <li>
> > +        <p>
> > +          For each NAT rule in the OVN Northbound database on a
> > +          distributed router, a priority-50 logical flow with match
> > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> > +          external IP address specified in the NAT rule, <var>GW</var>
> > +          is the logical router distributed gateway port. For dnat_and_snat
> > +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> > +          If <ref column="logical_port"
> > +          table="NAT" db="OVN_Northbound"/> column of
> > +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> > +          <var>P</var> is the <code>chassisredirect port</code> of
> > +          <var>GW</var> with the actions:
> > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> > +        </p>
> > +      </li>
> > +
> > +      <li>
> > +        A priority-0 logical flow with match <code>1</code> has actions
> > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > +      </li>
> > +    </ul>
> > +
> > +    <h3>Egress Table 1: UNDNAT</h3>
> >
> >       <p>
> >         This is for already established connections' reverse traffic.
> > @@ -4040,6 +4118,23 @@ nd_ns {
> >         is unDNATed here.
> >       </p>
> >
> > +    <ul>
> > +      <li>
> > +        A priority-0 logical flow with match <code>1</code> has actions
> > +        <code>next;</code>.
> > +      </li>
> > +    </ul>
> > +
> > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > +
> > +    <ul>
> > +      <li>
> > +        For all IP packets, a priority-50 flow with an action
> > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > +      </li>
> > +    </ul>
> > +
> > +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
> >       <ul>
> >         <li>
> >           <p>
> > @@ -4050,9 +4145,9 @@ nd_ns {
> >             gateway chassis that matches
> >             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
> >             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> > -          router gateway port with an action <code>ct_dnat;</code>. If the
> > -          backend IPv4 address <var>B</var> is also configured with L4 port
> > -          <var>PORT</var> of protocol <var>P</var>, then the
> > +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> > +          If the backend IPv4 address <var>B</var> is also configured with
> > +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
> >             match also includes <code>P.src</code> == <var>PORT</var>.  These
> >             flows are not added for load balancers with IPv6 <var>VIPs</var>.
> >           </p>
> > @@ -4072,7 +4167,7 @@ nd_ns {
> >             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
> >             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
> >             is the logical router gateway port, with an action
> > -          <code>ct_dnat;</code>. If the NAT rule is of type
> > +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
> >             dnat_and_snat and has <code>stateless=true</code> in the
> >             options, then the action would be <code>ip4/6.src=
> >             (<var>B</var>)</code>.
> > @@ -4081,7 +4176,7 @@ nd_ns {
> >           <p>
> >             If the NAT rule cannot be handled in a distributed manner, then
> >             the priority-100 flow above is only programmed on the
> > -          gateway chassis.
> > +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
> >           </p>
> >
> >           <p>
> > @@ -4094,26 +4189,17 @@ nd_ns {
> >           </p>
> >         </li>
> >
> > -      <li>
> > -        For all IP packets, a priority-50 flow with an action
> > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > -      </li>
> > -
> > -      <li>
> > -        A priority-0 logical flow with match <code>1</code> has actions
> > -        <code>next;</code>.
> > -      </li>
> >       </ul>
> >
> > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > +    <h3>Egress Table 2: Post UNDNAT</h3>
> >
> >       <p>
> >         <ul>
> >           <li>
> >             A priority-50 logical flow is added that commits any untracked flows
> > -          from the previous table <code>lr_out_undnat</code>. This flow
> > -          matches on <code>ct.new &amp;&amp; ip</code> with action
> > -          <code>ct_commit { } ; next; </code>.
> > +          from the previous table <code>lr_out_undnat</code> for Gateway
> > +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> > +          with action <code>ct_commit { } ; next; </code>.
> >           </li>
> >
> >           <li>
> > @@ -4124,7 +4210,7 @@ nd_ns {
> >         </ul>
> >       </p>
> >
> > -    <h3>Egress Table 2: SNAT</h3>
> > +    <h3>Egress Table 3: SNAT</h3>
> >
> >       <p>
> >         Packets that are configured to be SNATed get their source IP address
> > @@ -4140,7 +4226,7 @@ nd_ns {
> >         </li>
> >       </ul>
> >
> > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> >
> >       <ul>
> >         <li>
> > @@ -4239,7 +4325,7 @@ nd_ns {
> >         </li>
> >       </ul>
> >
> > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> >
> >       <ul>
> >         <li>
> > @@ -4247,28 +4333,47 @@ nd_ns {
> >             For each configuration in the OVN Northbound database, that asks
> >             to change the source IP address of a packet from an IP address of
> >             <var>A</var> or to change the source IP address of a packet that
> > -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> > -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > -          outport == <var>GW</var></code>, where <var>GW</var> is the
> > -          logical router gateway port, with an action
> > -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> > -          is calculated based on the mask of <var>A</var>, with matches
> > -          having larger masks getting higher priorities. If the NAT rule
> > -          is of type dnat_and_snat and has <code>stateless=true</code>
> > -          in the options, then the action would be <code>ip4/6.src=
> > -          (<var>B</var>)</code>.
> > +          belongs to network <var>A</var> to <var>B</var>, two flows are
> > +          added.  The priority <var>P</var> of these flows are calculated
> > +          based on the mask of <var>A</var>, with matches having larger
> > +          masks getting higher priorities.
> >           </p>
> >
> >           <p>
> >             If the NAT rule cannot be handled in a distributed manner, then
> > -          the flow above is only programmed on the
> > -          gateway chassis increasing flow priority by 128 in
> > -          order to be run first
> > +          the below flows are only programmed on the gateway chassis increasing
> > +          flow priority by 128 in order to be run first.
> >           </p>
> >
> > +        <ul>
> > +          <li>
> > +            The first flow is added with the calculated priority <var>P</var>
> > +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > +            outport == <var>GW</var></code>, where <var>GW</var> is the
> > +            logical router gateway port, with an action
> > +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > +            common zone.  If the NAT rule is of type dnat_and_snat and has
> > +            <code>stateless=true</code> in the options, then the action
> > +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> > +          </li>
> > +
> > +          <li>
> > +            The second flow is added with the calculated priority
> > +            <code><var>P</var> + 1 </code> and match
> > +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > +            outport == <var>GW</var> &amp;&amp;
> > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> > +            logical router gateway port, with an action
> > +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> > +            If the NAT rule is of type dnat_and_snat and has
> > +            <code>stateless=true</code> in the options, then the action would
> > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > +          </li>
> > +        </ul>
> > +
> >           <p>
> >             If the NAT rule can be handled in a distributed manner, then
> > -          there is an additional action
> > +          there is an additional action (for both the flows)
> >             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
> >             is the ethernet address associated with the IP address
> >             <var>A</var> in the NAT rule.  This allows upstream MAC
> > @@ -4284,7 +4389,8 @@ nd_ns {
> >
> >           <p>
> >             If the NAT rule has <code>exempted_ext_ips</code> set, then
> > -          there is an additional flow configured at the priority + 1 of
> > +          there is an additional flow configured at the priority
> > +          <code><var>P</var> + 2 </code> of
> >             corresponding NAT rule. The flow matches if destination ip
> >             is an <code>exempted_ext_ip</code> and the action is <code>next;
> >             </code>. This flow is used to bypass the ct_snat action for a flow
> > @@ -4299,7 +4405,7 @@ nd_ns {
> >         </li>
> >       </ul>
> >
> > -    <h3>Egress Table 3: Egress Loopback</h3>
> > +    <h3>Egress Table 4: Egress Loopback</h3>
> >
> >       <p>
> >         For distributed logical routers where one of the logical router
> > @@ -4344,6 +4450,7 @@ clone {
> >       outport = "";
> >       flags = 0;
> >       flags.loopback = 1;
> > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> >       reg0 = 0;
> >       reg1 = 0;
> >       ...
> > @@ -4368,7 +4475,7 @@ clone {
> >         </li>
> >       </ul>
> >
> > -    <h3>Egress Table 4: Delivery</h3>
> > +    <h3>Egress Table 5: Delivery</h3>
> >
> >       <p>
> >         Packets that reach this table are ready for delivery.  It contains:
> > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > index 85b47a18f..70ec5e2e3 100644
> > --- a/tests/ovn-northd.at
> > +++ b/tests/ovn-northd.at
> > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> >   echo
> >   echo "IPv4: stateful"
> >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > -check_flow_match_sets 2 2 3 0 0 0 0
> > +check_flow_match_sets 3 4 2 0 0 0 0
> >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> >
> >   echo
> >   echo "IPv4: stateless"
> >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > -check_flow_match_sets 2 0 1 2 2 0 0
> > +check_flow_match_sets 2 0 0 2 2 0 0
> >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> >
> >   echo
> >   echo "IPv6: stateful"
> >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > -check_flow_match_sets 2 2 3 0 0 0 0
> > +check_flow_match_sets 3 4 2 0 0 0 0
> >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> >
> >   echo
> >   echo "IPv6: stateless"
> >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > -check_flow_match_sets 2 0 1 0 0 2 2
> > +check_flow_match_sets 2 0 0 0 0 2 2
> >
> >   AT_CLEANUP
> >   ])
> > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
> >
> >   AT_CAPTURE_FILE([sbflows])
> > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
> >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> > -  [0], [1
> > +  [0], [2
> >   1
> >   ])
> >
> > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
> >
> >   AT_CAPTURE_FILE([sbflows2])
> > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
> >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> > -  [1], [1
> > +  [1], [2
> >   0
> >   ])
> >
> > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
> >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> >
> >   AT_CAPTURE_FILE([sbflows3])
> > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
> >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
> >
> >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> >   ovn-sbctl dump-flows CR > crflows
> >   AT_CAPTURE_FILE([crflows])
> >
> > -AT_CHECK([
> > -  grep -c lr_out_snat drflows
> > -  grep -c lr_out_snat crflows
> > -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > -3
> > -3
> > -1
> > -1
> > +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > +])
> > +
> > +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
> >   ])
> >
> > +
> >   # SNAT with DISALLOWED_IPs
> >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> >   ovn-sbctl dump-flows CR > crflows2
> >   AT_CAPTURE_FILE([crflows2])
> >
> > -AT_CHECK([
> > -  grep -c lr_out_snat drflows2
> > -  grep -c lr_out_snat crflows2
> > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > -4
> > -4
> > -1
> > -1
> > -1
> > -1
> > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > +])
> > +
> > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> >   ])
> >
> >   # Stateful FIP with ALLOWED_IPs
> > @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> >   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> >
> >   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> >
> > -ovn-nbctl show DR
> > -ovn-sbctl dump-flows DR
> > -ovn-nbctl show CR
> > -ovn-sbctl dump-flows CR
> > -
> > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > -wc -l`])
> > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > -wc -l`])
> > +ovn-sbctl dump-flows DR > drflows3
> > +AT_CAPTURE_FILE([drflows2])
> > +ovn-sbctl dump-flows CR > crflows3
> > +AT_CAPTURE_FILE([crflows2])
> >
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> >   ])
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > +
> > +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
> >   ])
> >
> >   # Stateful FIP with DISALLOWED_IPs
> > @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> >   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> >
> >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> >
> > -ovn-nbctl show DR
> > -ovn-sbctl dump-flows DR
> > -ovn-nbctl show CR
> > -ovn-sbctl dump-flows CR
> > -
> > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > -wc -l`])
> > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > -wc -l`])
> > +ovn-sbctl dump-flows DR > drflows4
> > +AT_CAPTURE_FILE([drflows2])
> > +ovn-sbctl dump-flows CR > crflows4
> > +AT_CAPTURE_FILE([crflows2])
> >
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> >   ])
> >
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> >   ])
> >
> >   # Stateless FIP with DISALLOWED_IPs
> > @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> >   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> >   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> >
> > -ovn-nbctl show DR
> > -ovn-sbctl dump-flows DR
> > -
> > -ovn-nbctl show CR
> > -ovn-sbctl dump-flows CR
> > -
> > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > -wc -l`])
> > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > -wc -l`])
> > +ovn-sbctl dump-flows DR > drflows5
> > +AT_CAPTURE_FILE([drflows2])
> > +ovn-sbctl dump-flows CR > crflows5
> > +AT_CAPTURE_FILE([crflows2])
> >
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> >   ])
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > +
> > +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> >   ])
> >
> >   # Stateful FIP with DISALLOWED_IPs
> > @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> >   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> >
> > -ovn-nbctl show DR
> > -ovn-sbctl dump-flows DR
> > -ovn-nbctl show CR
> > -ovn-sbctl dump-flows CR
> > +ovn-nbctl --wait=sb sync
> >
> > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > -wc -l`])
> > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > -wc -l`])
> > +ovn-sbctl dump-flows DR > drflows6
> > +AT_CAPTURE_FILE([drflows2])
> > +ovn-sbctl dump-flows CR > crflows6
> > +AT_CAPTURE_FILE([crflows2])
> >
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> >   ])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > -])
> > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > +
> > +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> >   ])
> >
> >   AT_CLEANUP
> > @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> > @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> > @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> >   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> > @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> >   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> > @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> >   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> > @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> >   AT_CLEANUP
> > @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
> >   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
> >   check ovn-nbctl --wait=sb sync
> >
> > +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> > +
> >   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
> >
> >   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> > @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> >   # Create few dnat_and_snat entries
> > @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> >   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
> >
> >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> >   ])
> >
> >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> >
> >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> >   ])
> >
> >   # Associate load balancer to lr0
> > @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
> >
> >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> >   ])
> >
> >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> >
> >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> >   ])
> >
> >   # Make the logical router as Gateway router
> > @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +])
> > +
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> >   ])
> >
> >   # Set lb force snat logical router.
> > @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> >   ])
> >
> >   # Add a LB VIP same as router ip.
> > @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +])
> > +
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> >   ])
> >
> >   # Add IPv6 router port and LB.
> > @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> >   ])
> >
> >   check ovn-nbctl lrp-del lr0-sw0
> > @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > +])
> > +
> > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> >   ])
> >
> > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> >   ])
> >
> > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> >   ])
> >
> >   AT_CLEANUP
> > diff --git a/tests/ovn.at b/tests/ovn.at
> > index 0d606b42f..ae5744407 100644
> > --- a/tests/ovn.at
> > +++ b/tests/ovn.at
> > @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
> >   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
> >     grep -c "$regex" sbflows;
> >   done], [0], [0
> > -1
> > +0
> >   2
> >   2
> >   ])
> > diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> > index c9f5771c9..7f6cb32dc 100644
> > --- a/tests/system-ovn.at
> > +++ b/tests/system-ovn.at
> > @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> >
> >   ovn-nbctl list load_balancer
> >   ovn-sbctl dump-flows R2
> > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> >   grep 'nat(src=20.0.0.2)'])
> >
> >   dnl Test load-balancing that includes L4 ports in NAT.
> > @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> >
> >   ovn-nbctl list load_balancer
> >   ovn-sbctl dump-flows R2
> > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> >   grep 'nat(src=20.0.0.2)'])
> >
> >   rm -f wget*.log
> > @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> >   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
> >   ])
> >
> > +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > +
> >   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> > -# from 172.16.1.4
> > +# from fd20::4
> >   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
> >   [0], [dnl
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > +ovs-appctl dpctl/dump-conntrack | grep icmpv6
> >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > +])
> > +
> > +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
> >   ])
> >
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > +# We verify that the connection is not tracked.
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> >   ])
> >
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > +# We verify that the connection is not tracked.
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> >   ])
> >
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > +# We verify that the connection is not tracked.
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
> >   ])
> >
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > +
> >   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
> >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> >   [0], [dnl
> > @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> >
> >   # Check conntrack entries.  First SNAT of 'foo1' address happens.
> >   # Then DNAT of 'bar1' address happens (listed first below).
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> >   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> >   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> >   ])
> >
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> >
> >   # Check conntrack entries.  First SNAT of 'foo2' address happens.
> >   # Then DNAT of 'bar1' address happens (listed first below).
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> >   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> >   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > -])
> > -
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> >   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
> >   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > -])
> > -
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> >   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
> >   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> >   ])
> >
> > -# We verify that the connection is tracked but not NATted. This is due to the
> > -# unDNAT table in the egress router pipeline
> > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > -])
> > -
> >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> >   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
> >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> >   # Then DNAT of 'bar1' address happens (listed first below).
> >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> >   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> >   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> >   ])
> > @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
> >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
> >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> >   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> > -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> >   ])
> >
> >   OVS_APP_EXIT_AND_WAIT([ovn-controller])
> >
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Frode Nordahl March 30, 2022, 9:45 a.m. UTC | #3 
Hello Numan,

This patch does unfortunately break gateway routers in some
circumstances, (but not all!), at least for the way OpenStack consumes
them.

Will dig more and try to figure out what is going on, but sending this
e-mail proactively in case you have any ideas from the top of your
head or if anyone else has run into the same issue.

The gateway appears to handle conntrack as it should:
$ sudo conntrack -E --dst 194.169.254.178
    [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
sport=22 dport=60234 zone=52
 [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
dport=60234 zone=52
 [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
sport=22 dport=60234 [ASSURED] zone=52

However as you can see traffic going to the instance suddenly gets its
source address mangled/set to 0.0.0.0, and the connection can never
establish.
$ sudo tcpdump -nevvi tape5c1862d-b4
tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
capture size 262144 bytes
09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
proto TCP (6), length 60)
    10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
(correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
870680827 ecr 0,nop,wscale 7], length 0
09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
(0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
proto TCP (6), length 60)
    10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
(incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
7], length 0
09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
[DF], proto TCP (6), length 93)
    10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
(correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
ecr 3490675961], length 41
09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
(0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
proto TCP (6), length 52)
    10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
(incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
3490676008 ecr 870680922], length 0
09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
proto TCP (6), length 52)
    10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
(correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
ecr 3490675961], length 0
09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
(0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
proto TCP (6), length 52)
    10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
(incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
3490676009 ecr 870680922], length 0
09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
(0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
[DF], proto TCP (6), length 93)
    10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
(incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
val 3490676018 ecr 870680922], length 41
09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
proto TCP (6), length 52)
    0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
870680932 ecr 3490676018], length 0
09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
[DF], proto TCP (6), length 1564)
    0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
-> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
870680933 ecr 3490676018], length 1512
09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
(0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
[DF], proto TCP (6), length 116)
    0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
3490676018], length 64

--
Frode Nordahl

On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <numans@ovn.org> wrote:
>
> On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <mmichels@redhat.com> wrote:
> >
> > Thanks for the update, Numan.
> >
> > Acked-by: Mark Michelson <mmichels@redhat.com>
>
> Thanks.  I applied both the patches to the main branch.
>
> Numan
>
> >
> > On 11/19/21 11:32, numans@ovn.org wrote:
> > > From: Numan Siddique <numans@ovn.org>
> > >
> > > Make of use of these new actions for the distributed routers
> > > for NAT.  These new actions ensure that both sNAT and dNAT
> > > happens in the same zone.  This approach solves a couple of
> > > problems:
> > >
> > >   - The datapath flows generated for external traffic which requires
> > >     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> > >
> > >   - Since there is only one zone, it would avoid multiple recirculations
> > >     (improving the performance).
> > >
> > > If the packet needs to be both sNATted and dNATted (for hairpin traffic
> > > with source and destination on the same chassis), then sNAT is done
> > > in a separate zone.  To detect this scenario, this patch adds a few
> > > extra logical flows.  For each dnat_and_snat entry prior to this patch
> > > ovn-northd was generating 9 logical flows and with this patch it now
> > > generates 12 logical flows.
> > >
> > > Similar approach can be taken for gateway routers.
> > >
> > > Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > ---
> > >
> > > v2 -> v3
> > > ------
> > >    * Addressed Mark's comments and updated the documentation.
> > >
> > > v1 -> v2
> > > ------
> > >    * Rebased and resolved conflicts.
> > >
> > >
> > >   include/ovn/logical-fields.h |   1 +
> > >   lib/logical-fields.c         |   4 +
> > >   northd/northd.c              | 147 +++++++--
> > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > >   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
> > >   tests/ovn.at                 |   2 +-
> > >   tests/system-ovn.at          |  64 ++--
> > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > >
> > > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> > > index c9675f81c..2118f7933 100644
> > > --- a/include/ovn/logical-fields.h
> > > +++ b/include/ovn/logical-fields.h
> > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > >       MLF_LOOKUP_FDB_BIT = 8,
> > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > >       MLF_LOCALPORT_BIT = 10,
> > > +    MLF_USE_SNAT_ZONE = 11,
> > >   };
> > >
> > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > index 7b3d431e0..352a48c89 100644
> > > --- a/lib/logical-fields.c
> > > +++ b/lib/logical-fields.c
> > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
> > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > >       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
> > >                                flags_str);
> > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > +             MLF_USE_SNAT_ZONE);
> > > +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> > > +                             flags_str);
> > >
> > >       /* Connection tracking state. */
> > >       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> > > diff --git a/northd/northd.c b/northd/northd.c
> > > index 0ff61deec..e4d051a94 100644
> > > --- a/northd/northd.c
> > > +++ b/northd/northd.c
> > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
> > >                                                                         \
> > >       /* Logical router egress stages. */                               \
> > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> > > +                   "lr_out_chk_dnat_local")                                  \
> > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
> > >
> > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
> > >       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > >
> > >   /* Register to store the eth address associated to a router port for packets
> > >    * received in S_ROUTER_IN_ADMISSION.
> > > @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > >                                       undnat_match_p, est_actions,
> > >                                       &lb->nlb->header_);
> > >           } else {
> > > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > -                                    undnat_match_p, "ct_dnat;",
> > > -                                    &lb->nlb->header_);
> > > +            ovn_lflow_add_with_hint(
> > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> > > +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > +                &lb->nlb->header_);
> > >           }
> > >           free(undnat_match_p);
> > >   next:
> > > @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > >           uint16_t priority;
> > >
> > >           /* Priority of logical flows corresponding to exempted_ext_ips is
> > > -         * +1 of the corresponding regulr NAT rule.
> > > +         * +2 of the corresponding regular NAT rule.
> > >            * For example, if we have following NAT rule and we associate
> > >            * exempted external ips to it:
> > >            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> > > @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > >            * And now we associate exempted external ip address set to it.
> > >            * Now corresponding to above rule we will have following logical
> > >            * flows:
> > > -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> > > +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
> > >            *                             action=(next;)
> > >            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
> > >            *
> > >            */
> > >           if (is_src) {
> > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > -            priority = 100 + 1;
> > > +            priority = 100 + 2;
> > >           } else {
> > >               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > +            priority = count_1bits(ntohl(mask)) + 3;
> > >
> > >               if (!od->is_gw_router) {
> > >                   priority += 128;
> > > @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >           /* Traffic received on l3dgw_port is subject to NAT. */
> > >           ds_clear(match);
> > >           ds_clear(actions);
> > > -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > > -                      is_v6 ? "6" : "4", nat->external_ip,
> > > -                      od->l3dgw_ports[0]->json_key);
> > > +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> > > +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
> > >           if (!distributed && od->n_l3dgw_ports) {
> > >               /* Flows for NAT rules that are centralized are only
> > >               * programmed on the gateway chassis. */
> > > @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >               ds_put_format(actions, "ip%s.dst=%s; next;",
> > >                             is_v6 ? "6" : "4", nat->logical_ip);
> > >           } else {
> > > -            ds_put_cstr(actions, "ct_snat;");
> > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > >           }
> > >
> > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > >                                   100, ds_cstr(match), ds_cstr(actions),
> > >                                   &nat->header_);
> > > +
> > > +        if (!stateless) {
> > > +            ds_clear(match);
> > > +            ds_clear(actions);
> > > +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> > > +                          is_v6 ? "6" : "4", nat->external_ip,
> > > +                          od->l3dgw_ports[0]->json_key);
> > > +            if (!distributed && od->n_l3dgw_ports) {
> > > +                /* Flows for NAT rules that are centralized are only
> > > +                * programmed on the gateway chassis. */
> > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > +                            od->l3dgw_ports[0]->cr_port->json_key);
> > > +            }
> > > +            ds_put_cstr(actions, "ct_snat;");
> > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > +                                    100, ds_cstr(match), ds_cstr(actions),
> > > +                                    &nat->header_);
> > > +        }
> > >       }
> > >   }
> > >
> > > @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >                   ds_put_format(actions, "ip%s.dst=%s; next;",
> > >                                 is_v6 ? "6" : "4", nat->logical_ip);
> > >               } else {
> > > -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> > > +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
> > >                   if (nat->external_port_range[0]) {
> > >                       ds_put_format(actions, ",%s", nat->external_port_range);
> > >                   }
> > > @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > >                         is_v6 ? "6" : "4", nat->external_ip);
> > >       } else {
> > > -        ds_put_format(actions, "ct_dnat;");
> > > +        ds_put_format(actions,
> > > +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> > >       }
> > >
> > >       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >                               &nat->header_);
> > >   }
> > >
> > > +static void
> > > +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> > > +                                const struct nbrec_nat *nat, struct ds *match,
> > > +                                struct ds *actions, bool distributed,
> > > +                                bool is_v6)
> > > +{
> > > +    /* Note that this only applies for NAT on a distributed router.
> > > +     */
> > > +    if (!od->n_l3dgw_ports) {
> > > +        return;
> > > +    }
> > > +
> > > +    ds_clear(match);
> > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > +    if (distributed) {
> > > +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> > > +    } else {
> > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > +                      od->l3dgw_ports[0]->cr_port->json_key);
> > > +    }
> > > +
> > > +    ds_clear(actions);
> > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> > > +
> > > +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > +                            50, ds_cstr(match), ds_cstr(actions),
> > > +                            &nat->header_);
> > > +}
> > > +
> > >   static void
> > >   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >                               const struct nbrec_nat *nat, struct ds *match,
> > > @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > >                         is_v6 ? "6" : "4", nat->logical_ip,
> > >                         od->l3dgw_ports[0]->json_key);
> > > -        if (!distributed && od->n_l3dgw_ports) {
> > > -            /* Flows for NAT rules that are centralized are only
> > > -            * programmed on the gateway chassis. */
> > > -            priority += 128;
> > > -            ds_put_format(match, " && is_chassis_resident(%s)",
> > > -                          od->l3dgw_ports[0]->cr_port->json_key);
> > > -        } else if (distributed) {
> > > -            priority += 128;
> > > -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > -                          nat->logical_port);
> > > +        if (od->n_l3dgw_ports) {
> > > +            if (distributed) {
> > > +                ovs_assert(nat->logical_port);
> > > +                priority += 128;
> > > +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > +                              nat->logical_port);
> > > +            } else {
> > > +                /* Flows for NAT rules that are centralized are only
> > > +                * programmed on the gateway chassis. */
> > > +                priority += 128;
> > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > +                              od->l3dgw_ports[0]->cr_port->json_key);
> > > +            }
> > >           }
> > >           ds_clear(actions);
> > >
> > > @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >               ds_put_format(actions, "ip%s.src=%s; next;",
> > >                             is_v6 ? "6" : "4", nat->external_ip);
> > >           } else {
> > > -            ds_put_format(actions, "ct_snat(%s",
> > > +            ds_put_format(actions, "ct_snat_in_czone(%s",
> > >                           nat->external_ip);
> > >               if (nat->external_port_range[0]) {
> > >                   ds_put_format(actions, ",%s", nat->external_port_range);
> > > @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > >                                   priority, ds_cstr(match),
> > >                                   ds_cstr(actions), &nat->header_);
> > > +
> > > +        if (!stateless) {
> > > +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > +            ds_clear(actions);
> > > +            if (distributed) {
> > > +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> > > +                              ETH_ADDR_ARGS(mac));
> > > +            }
> > > +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > +                          nat->external_ip);
> > > +            if (nat->external_port_range[0]) {
> > > +                ds_put_format(actions, ",%s", nat->external_port_range);
> > > +            }
> > > +            ds_put_format(actions, ");");
> > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > +                                    priority + 1, ds_cstr(match),
> > > +                                    ds_cstr(actions), &nat->header_);
> > > +        }
> > >       }
> > >   }
> > >
> > > @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > >       /* Packets are allowed by default. */
> > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
> > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > >
> > > @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > >        * not committed, it would produce ongoing datapath flows with the ct.new
> > >        * flag set. Some NICs are unable to offload these flows.
> > >        */
> > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > >                         "ip", "flags.loopback = 1; ct_dnat;");
> > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > >               }
> > >           }
> > >
> > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> > > +                                        distributed, is_v6);
> > > +
> > >           /* S_ROUTER_OUT_UNDNAT */
> > >           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
> > >                                         mac, is_v6);
> > > @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > >                             "clone { ct_clear; "
> > >                             "inport = outport; outport = \"\"; "
> > >                             "eth.dst <-> eth.src; "
> > > -                          "flags = 0; flags.loopback = 1; ");
> > > +                          "flags = 0; flags.loopback = 1; "
> > > +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
> > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > >                   ds_put_format(actions, "reg%d = 0; ", j);
> > >               }
> > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > > index 21d83718c..e39e6e805 100644
> > > --- a/northd/ovn-northd.8.xml
> > > +++ b/northd/ovn-northd.8.xml
> > > @@ -2879,23 +2879,65 @@ icmp6 {
> > >           <p>
> > >             For each configuration in the OVN Northbound database, that asks
> > >             to change the source IP address of a packet from <var>A</var> to
> > > -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> > > -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> > > -          <code>ip &amp;&amp;
> > > -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> > > -          where <var>GW</var> is the logical router gateway port, with an
> > > -          action <code>ct_snat;</code>. If the NAT rule is of type
> > > -          dnat_and_snat and has <code>stateless=true</code> in the
> > > -          options, then the action would be <code>ip4/6.dst=
> > > -          (<var>B</var>)</code>.
> > > +          <var>B</var>, two priority-100 flows are added.
> > >           </p>
> > >
> > >           <p>
> > >             If the NAT rule cannot be handled in a distributed manner, then
> > > -          the priority-100 flow above is only programmed on the
> > > +          the below priority-100 flows are only programmed on the
> > >             gateway chassis.
> > >           </p>
> > >
> > > +        <ul>
> > > +          <li>
> > > +            <p>
> > > +              The first flow matches <code>ip &amp;&amp;
> > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > +              &amp;&amp; flags.loopback == 0</code> or
> > > +              <code>ip &amp;&amp;
> > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > +              &amp;&amp; flags.loopback == 0</code>
> > > +              where <var>GW</var> is the logical router gateway port, with an
> > > +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> > > +              zone.  If the NAT rule is of type dnat_and_snat and has
> > > +              <code>stateless=true</code> in the options, then the action
> > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > +            </p>
> > > +
> > > +            <p>
> > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > +              <var>GW</var>.
> > > +            </p>
> > > +          </li>
> > > +
> > > +          <li>
> > > +            <p>
> > > +              The second flow matches <code>ip &amp;&amp;
> > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > +              flags.use_snat_zone == 1</code> or
> > > +              <code>ip &amp;&amp;
> > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > +              flags.use_snat_zone == 1</code>
> > > +              where <var>GW</var> is the logical router gateway port, with an
> > > +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> > > +              NAT rule is of type dnat_and_snat and has
> > > +              <code>stateless=true</code> in the options, then the action
> > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > +            </p>
> > > +
> > > +            <p>
> > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > +              <var>GW</var>.
> > > +            </p>
> > > +          </li>
> > > +        </ul>
> > > +
> > >           <p>
> > >             A priority-0 logical flow with match <code>1</code> has actions
> > >             <code>next;</code>.
> > > @@ -4031,7 +4073,43 @@ nd_ns {
> > >         </li>
> > >       </ul>
> > >
> > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > +
> > > +    <p>
> > > +      This table checks if the packet needs to be DNATed in the router ingress
> > > +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> > > +      to the ingress pipeline.  This check is done only for routers configured
> > > +      with distributed gateway ports and NAT entries.  This check is done
> > > +      so that SNAT and DNAT is done in different zones instead of a common
> > > +      zone.
> > > +    </p>
> > > +
> > > +    <ul>
> > > +      <li>
> > > +        <p>
> > > +          For each NAT rule in the OVN Northbound database on a
> > > +          distributed router, a priority-50 logical flow with match
> > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> > > +          external IP address specified in the NAT rule, <var>GW</var>
> > > +          is the logical router distributed gateway port. For dnat_and_snat
> > > +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> > > +          If <ref column="logical_port"
> > > +          table="NAT" db="OVN_Northbound"/> column of
> > > +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> > > +          <var>P</var> is the <code>chassisredirect port</code> of
> > > +          <var>GW</var> with the actions:
> > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> > > +        </p>
> > > +      </li>
> > > +
> > > +      <li>
> > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > +      </li>
> > > +    </ul>
> > > +
> > > +    <h3>Egress Table 1: UNDNAT</h3>
> > >
> > >       <p>
> > >         This is for already established connections' reverse traffic.
> > > @@ -4040,6 +4118,23 @@ nd_ns {
> > >         is unDNATed here.
> > >       </p>
> > >
> > > +    <ul>
> > > +      <li>
> > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > +        <code>next;</code>.
> > > +      </li>
> > > +    </ul>
> > > +
> > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > +
> > > +    <ul>
> > > +      <li>
> > > +        For all IP packets, a priority-50 flow with an action
> > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > +      </li>
> > > +    </ul>
> > > +
> > > +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
> > >       <ul>
> > >         <li>
> > >           <p>
> > > @@ -4050,9 +4145,9 @@ nd_ns {
> > >             gateway chassis that matches
> > >             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
> > >             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> > > -          router gateway port with an action <code>ct_dnat;</code>. If the
> > > -          backend IPv4 address <var>B</var> is also configured with L4 port
> > > -          <var>PORT</var> of protocol <var>P</var>, then the
> > > +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> > > +          If the backend IPv4 address <var>B</var> is also configured with
> > > +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
> > >             match also includes <code>P.src</code> == <var>PORT</var>.  These
> > >             flows are not added for load balancers with IPv6 <var>VIPs</var>.
> > >           </p>
> > > @@ -4072,7 +4167,7 @@ nd_ns {
> > >             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
> > >             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
> > >             is the logical router gateway port, with an action
> > > -          <code>ct_dnat;</code>. If the NAT rule is of type
> > > +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
> > >             dnat_and_snat and has <code>stateless=true</code> in the
> > >             options, then the action would be <code>ip4/6.src=
> > >             (<var>B</var>)</code>.
> > > @@ -4081,7 +4176,7 @@ nd_ns {
> > >           <p>
> > >             If the NAT rule cannot be handled in a distributed manner, then
> > >             the priority-100 flow above is only programmed on the
> > > -          gateway chassis.
> > > +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
> > >           </p>
> > >
> > >           <p>
> > > @@ -4094,26 +4189,17 @@ nd_ns {
> > >           </p>
> > >         </li>
> > >
> > > -      <li>
> > > -        For all IP packets, a priority-50 flow with an action
> > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > -      </li>
> > > -
> > > -      <li>
> > > -        A priority-0 logical flow with match <code>1</code> has actions
> > > -        <code>next;</code>.
> > > -      </li>
> > >       </ul>
> > >
> > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > >
> > >       <p>
> > >         <ul>
> > >           <li>
> > >             A priority-50 logical flow is added that commits any untracked flows
> > > -          from the previous table <code>lr_out_undnat</code>. This flow
> > > -          matches on <code>ct.new &amp;&amp; ip</code> with action
> > > -          <code>ct_commit { } ; next; </code>.
> > > +          from the previous table <code>lr_out_undnat</code> for Gateway
> > > +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> > > +          with action <code>ct_commit { } ; next; </code>.
> > >           </li>
> > >
> > >           <li>
> > > @@ -4124,7 +4210,7 @@ nd_ns {
> > >         </ul>
> > >       </p>
> > >
> > > -    <h3>Egress Table 2: SNAT</h3>
> > > +    <h3>Egress Table 3: SNAT</h3>
> > >
> > >       <p>
> > >         Packets that are configured to be SNATed get their source IP address
> > > @@ -4140,7 +4226,7 @@ nd_ns {
> > >         </li>
> > >       </ul>
> > >
> > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > >
> > >       <ul>
> > >         <li>
> > > @@ -4239,7 +4325,7 @@ nd_ns {
> > >         </li>
> > >       </ul>
> > >
> > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > >
> > >       <ul>
> > >         <li>
> > > @@ -4247,28 +4333,47 @@ nd_ns {
> > >             For each configuration in the OVN Northbound database, that asks
> > >             to change the source IP address of a packet from an IP address of
> > >             <var>A</var> or to change the source IP address of a packet that
> > > -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> > > -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > -          outport == <var>GW</var></code>, where <var>GW</var> is the
> > > -          logical router gateway port, with an action
> > > -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> > > -          is calculated based on the mask of <var>A</var>, with matches
> > > -          having larger masks getting higher priorities. If the NAT rule
> > > -          is of type dnat_and_snat and has <code>stateless=true</code>
> > > -          in the options, then the action would be <code>ip4/6.src=
> > > -          (<var>B</var>)</code>.
> > > +          belongs to network <var>A</var> to <var>B</var>, two flows are
> > > +          added.  The priority <var>P</var> of these flows are calculated
> > > +          based on the mask of <var>A</var>, with matches having larger
> > > +          masks getting higher priorities.
> > >           </p>
> > >
> > >           <p>
> > >             If the NAT rule cannot be handled in a distributed manner, then
> > > -          the flow above is only programmed on the
> > > -          gateway chassis increasing flow priority by 128 in
> > > -          order to be run first
> > > +          the below flows are only programmed on the gateway chassis increasing
> > > +          flow priority by 128 in order to be run first.
> > >           </p>
> > >
> > > +        <ul>
> > > +          <li>
> > > +            The first flow is added with the calculated priority <var>P</var>
> > > +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > +            outport == <var>GW</var></code>, where <var>GW</var> is the
> > > +            logical router gateway port, with an action
> > > +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > +            common zone.  If the NAT rule is of type dnat_and_snat and has
> > > +            <code>stateless=true</code> in the options, then the action
> > > +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> > > +          </li>
> > > +
> > > +          <li>
> > > +            The second flow is added with the calculated priority
> > > +            <code><var>P</var> + 1 </code> and match
> > > +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > +            outport == <var>GW</var> &amp;&amp;
> > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> > > +            logical router gateway port, with an action
> > > +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> > > +            If the NAT rule is of type dnat_and_snat and has
> > > +            <code>stateless=true</code> in the options, then the action would
> > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > +          </li>
> > > +        </ul>
> > > +
> > >           <p>
> > >             If the NAT rule can be handled in a distributed manner, then
> > > -          there is an additional action
> > > +          there is an additional action (for both the flows)
> > >             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
> > >             is the ethernet address associated with the IP address
> > >             <var>A</var> in the NAT rule.  This allows upstream MAC
> > > @@ -4284,7 +4389,8 @@ nd_ns {
> > >
> > >           <p>
> > >             If the NAT rule has <code>exempted_ext_ips</code> set, then
> > > -          there is an additional flow configured at the priority + 1 of
> > > +          there is an additional flow configured at the priority
> > > +          <code><var>P</var> + 2 </code> of
> > >             corresponding NAT rule. The flow matches if destination ip
> > >             is an <code>exempted_ext_ip</code> and the action is <code>next;
> > >             </code>. This flow is used to bypass the ct_snat action for a flow
> > > @@ -4299,7 +4405,7 @@ nd_ns {
> > >         </li>
> > >       </ul>
> > >
> > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > >
> > >       <p>
> > >         For distributed logical routers where one of the logical router
> > > @@ -4344,6 +4450,7 @@ clone {
> > >       outport = "";
> > >       flags = 0;
> > >       flags.loopback = 1;
> > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > >       reg0 = 0;
> > >       reg1 = 0;
> > >       ...
> > > @@ -4368,7 +4475,7 @@ clone {
> > >         </li>
> > >       </ul>
> > >
> > > -    <h3>Egress Table 4: Delivery</h3>
> > > +    <h3>Egress Table 5: Delivery</h3>
> > >
> > >       <p>
> > >         Packets that reach this table are ready for delivery.  It contains:
> > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > index 85b47a18f..70ec5e2e3 100644
> > > --- a/tests/ovn-northd.at
> > > +++ b/tests/ovn-northd.at
> > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > >   echo
> > >   echo "IPv4: stateful"
> > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > +check_flow_match_sets 3 4 2 0 0 0 0
> > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > >
> > >   echo
> > >   echo "IPv4: stateless"
> > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > +check_flow_match_sets 2 0 0 2 2 0 0
> > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > >
> > >   echo
> > >   echo "IPv6: stateful"
> > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > +check_flow_match_sets 3 4 2 0 0 0 0
> > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > >
> > >   echo
> > >   echo "IPv6: stateless"
> > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > +check_flow_match_sets 2 0 0 0 0 2 2
> > >
> > >   AT_CLEANUP
> > >   ])
> > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
> > >
> > >   AT_CAPTURE_FILE([sbflows])
> > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
> > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> > > -  [0], [1
> > > +  [0], [2
> > >   1
> > >   ])
> > >
> > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
> > >
> > >   AT_CAPTURE_FILE([sbflows2])
> > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
> > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> > > -  [1], [1
> > > +  [1], [2
> > >   0
> > >   ])
> > >
> > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
> > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > >
> > >   AT_CAPTURE_FILE([sbflows3])
> > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
> > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
> > >
> > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > >   ovn-sbctl dump-flows CR > crflows
> > >   AT_CAPTURE_FILE([crflows])
> > >
> > > -AT_CHECK([
> > > -  grep -c lr_out_snat drflows
> > > -  grep -c lr_out_snat crflows
> > > -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > -3
> > > -3
> > > -1
> > > -1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > +])
> > > +
> > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
> > >   ])
> > >
> > > +
> > >   # SNAT with DISALLOWED_IPs
> > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > >   ovn-sbctl dump-flows CR > crflows2
> > >   AT_CAPTURE_FILE([crflows2])
> > >
> > > -AT_CHECK([
> > > -  grep -c lr_out_snat drflows2
> > > -  grep -c lr_out_snat crflows2
> > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > -4
> > > -4
> > > -1
> > > -1
> > > -1
> > > -1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > +])
> > > +
> > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > >   ])
> > >
> > >   # Stateful FIP with ALLOWED_IPs
> > > @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > >   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > >
> > >   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > >
> > > -ovn-nbctl show DR
> > > -ovn-sbctl dump-flows DR
> > > -ovn-nbctl show CR
> > > -ovn-sbctl dump-flows CR
> > > -
> > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > -wc -l`])
> > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > -wc -l`])
> > > +ovn-sbctl dump-flows DR > drflows3
> > > +AT_CAPTURE_FILE([drflows2])
> > > +ovn-sbctl dump-flows CR > crflows3
> > > +AT_CAPTURE_FILE([crflows2])
> > >
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > >   ])
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > +
> > > +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
> > >   ])
> > >
> > >   # Stateful FIP with DISALLOWED_IPs
> > > @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > >   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > >
> > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > >
> > > -ovn-nbctl show DR
> > > -ovn-sbctl dump-flows DR
> > > -ovn-nbctl show CR
> > > -ovn-sbctl dump-flows CR
> > > -
> > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > -wc -l`])
> > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > -wc -l`])
> > > +ovn-sbctl dump-flows DR > drflows4
> > > +AT_CAPTURE_FILE([drflows2])
> > > +ovn-sbctl dump-flows CR > crflows4
> > > +AT_CAPTURE_FILE([crflows2])
> > >
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > >   ])
> > >
> > >   # Stateless FIP with DISALLOWED_IPs
> > > @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > >   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > >   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > >
> > > -ovn-nbctl show DR
> > > -ovn-sbctl dump-flows DR
> > > -
> > > -ovn-nbctl show CR
> > > -ovn-sbctl dump-flows CR
> > > -
> > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > -wc -l`])
> > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > -wc -l`])
> > > +ovn-sbctl dump-flows DR > drflows5
> > > +AT_CAPTURE_FILE([drflows2])
> > > +ovn-sbctl dump-flows CR > crflows5
> > > +AT_CAPTURE_FILE([crflows2])
> > >
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > >   ])
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > +
> > > +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > >   ])
> > >
> > >   # Stateful FIP with DISALLOWED_IPs
> > > @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > >
> > > -ovn-nbctl show DR
> > > -ovn-sbctl dump-flows DR
> > > -ovn-nbctl show CR
> > > -ovn-sbctl dump-flows CR
> > > +ovn-nbctl --wait=sb sync
> > >
> > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > -wc -l`])
> > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > -wc -l`])
> > > +ovn-sbctl dump-flows DR > drflows6
> > > +AT_CAPTURE_FILE([drflows2])
> > > +ovn-sbctl dump-flows CR > crflows6
> > > +AT_CAPTURE_FILE([crflows2])
> > >
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > >   ])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > -])
> > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > +
> > > +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > >   ])
> > >
> > >   AT_CLEANUP
> > > @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> > > @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> > > @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > >   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> > > @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > >   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> > > @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > >   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> > > @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > >   AT_CLEANUP
> > > @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
> > >   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
> > >   check ovn-nbctl --wait=sb sync
> > >
> > > +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> > > +
> > >   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
> > >
> > >   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> > > @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > >   # Create few dnat_and_snat entries
> > > @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > >   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > > @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
> > >
> > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > >   ])
> > >
> > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > >
> > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   # Associate load balancer to lr0
> > > @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
> > >
> > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > >   ])
> > >
> > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > >
> > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   # Make the logical router as Gateway router
> > > @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   # Set lb force snat logical router.
> > > @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   # Add a LB VIP same as router ip.
> > > @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   # Add IPv6 router port and LB.
> > > @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > >   ])
> > >
> > >   check ovn-nbctl lrp-del lr0-sw0
> > > @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > +])
> > > +
> > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > >   ])
> > >
> > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > >   ])
> > >
> > >   AT_CLEANUP
> > > diff --git a/tests/ovn.at b/tests/ovn.at
> > > index 0d606b42f..ae5744407 100644
> > > --- a/tests/ovn.at
> > > +++ b/tests/ovn.at
> > > @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
> > >   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
> > >     grep -c "$regex" sbflows;
> > >   done], [0], [0
> > > -1
> > > +0
> > >   2
> > >   2
> > >   ])
> > > diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> > > index c9f5771c9..7f6cb32dc 100644
> > > --- a/tests/system-ovn.at
> > > +++ b/tests/system-ovn.at
> > > @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > >
> > >   ovn-nbctl list load_balancer
> > >   ovn-sbctl dump-flows R2
> > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > >   grep 'nat(src=20.0.0.2)'])
> > >
> > >   dnl Test load-balancing that includes L4 ports in NAT.
> > > @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > >
> > >   ovn-nbctl list load_balancer
> > >   ovn-sbctl dump-flows R2
> > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > >   grep 'nat(src=20.0.0.2)'])
> > >
> > >   rm -f wget*.log
> > > @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > >   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > >   ])
> > >
> > > +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > +
> > >   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> > > -# from 172.16.1.4
> > > +# from fd20::4
> > >   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
> > >   [0], [dnl
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > +ovs-appctl dpctl/dump-conntrack | grep icmpv6
> > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > +])
> > > +
> > > +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
> > >   ])
> > >
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > +# We verify that the connection is not tracked.
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   ])
> > >
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > +# We verify that the connection is not tracked.
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   ])
> > >
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > +# We verify that the connection is not tracked.
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   ])
> > >
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > +
> > >   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
> > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > >   [0], [dnl
> > > @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > >
> > >   # Check conntrack entries.  First SNAT of 'foo1' address happens.
> > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > >   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   ])
> > >
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > >
> > >   # Check conntrack entries.  First SNAT of 'foo2' address happens.
> > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > >   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > -])
> > > -
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > >   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
> > >   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > -])
> > > -
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > >   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
> > >   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > >   ])
> > >
> > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > -# unDNAT table in the egress router pipeline
> > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > -])
> > > -
> > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > >   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
> > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > >   # Then DNAT of 'bar1' address happens (listed first below).
> > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > >   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > >   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > >   ])
> > > @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
> > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
> > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > >   ])
> > >
> > >   OVS_APP_EXIT_AND_WAIT([ovn-controller])
> > >
> >
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Frode Nordahl March 30, 2022, 10:20 a.m. UTC | #4 
On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
<frode.nordahl@canonical.com> wrote:
>
> Hello Numan,
>
> This patch does unfortunately break gateway routers in some
> circumstances, (but not all!), at least for the way OpenStack consumes
> them.
>
> Will dig more and try to figure out what is going on, but sending this
> e-mail proactively in case you have any ideas from the top of your
> head or if anyone else has run into the same issue.
>
> The gateway appears to handle conntrack as it should:
> $ sudo conntrack -E --dst 194.169.254.178
>     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> sport=22 dport=60234 zone=52
>  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> dport=60234 zone=52
>  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> sport=22 dport=60234 [ASSURED] zone=52
>
> However as you can see traffic going to the instance suddenly gets its
> source address mangled/set to 0.0.0.0, and the connection can never
> establish.
> $ sudo tcpdump -nevvi tape5c1862d-b4
> tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> capture size 262144 bytes
> 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> proto TCP (6), length 60)
>     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> 870680827 ecr 0,nop,wscale 7], length 0
> 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> proto TCP (6), length 60)
>     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> 7], length 0
> 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> [DF], proto TCP (6), length 93)
>     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> ecr 3490675961], length 41
> 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> proto TCP (6), length 52)
>     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> 3490676008 ecr 870680922], length 0
> 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> proto TCP (6), length 52)
>     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> ecr 3490675961], length 0
> 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> proto TCP (6), length 52)
>     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> 3490676009 ecr 870680922], length 0
> 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> [DF], proto TCP (6), length 93)
>     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> val 3490676018 ecr 870680922], length 41
> 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> proto TCP (6), length 52)
>     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> 870680932 ecr 3490676018], length 0
> 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> [DF], proto TCP (6), length 1564)
>     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> 870680933 ecr 3490676018], length 1512
> 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> [DF], proto TCP (6), length 116)
>     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> 3490676018], length 64

Turning hardware offload all the way off appears to resolve the issue,
so this might as well be driver/firmware related. May I ask what
card/driver/firmware you've been using this with?
Numan Siddique March 30, 2022, 2:54 p.m. UTC | #5 
On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
<frode.nordahl@canonical.com> wrote:
>
> On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> <frode.nordahl@canonical.com> wrote:
> >
> > Hello Numan,
> >
> > This patch does unfortunately break gateway routers in some
> > circumstances, (but not all!), at least for the way OpenStack consumes
> > them.
> >
> > Will dig more and try to figure out what is going on, but sending this
> > e-mail proactively in case you have any ideas from the top of your
> > head or if anyone else has run into the same issue.
> >
> > The gateway appears to handle conntrack as it should:
> > $ sudo conntrack -E --dst 194.169.254.178
> >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > sport=22 dport=60234 zone=52
> >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > dport=60234 zone=52
> >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> > sport=22 dport=60234 [ASSURED] zone=52
> >
> > However as you can see traffic going to the instance suddenly gets its
> > source address mangled/set to 0.0.0.0, and the connection can never
> > establish.
> > $ sudo tcpdump -nevvi tape5c1862d-b4
> > tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> > capture size 262144 bytes
> > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> > proto TCP (6), length 60)
> >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> > 870680827 ecr 0,nop,wscale 7], length 0
> > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> > proto TCP (6), length 60)
> >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> > options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> > 7], length 0
> > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> > [DF], proto TCP (6), length 93)
> >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> > ecr 3490675961], length 41
> > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> > proto TCP (6), length 52)
> >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> > 3490676008 ecr 870680922], length 0
> > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> > proto TCP (6), length 52)
> >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> > ecr 3490675961], length 0
> > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> > proto TCP (6), length 52)
> >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> > 3490676009 ecr 870680922], length 0
> > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> > [DF], proto TCP (6), length 93)
> >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> > val 3490676018 ecr 870680922], length 41
> > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> > proto TCP (6), length 52)
> >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> > 870680932 ecr 3490676018], length 0
> > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> > [DF], proto TCP (6), length 1564)
> >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > 870680933 ecr 3490676018], length 1512
> > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> > [DF], proto TCP (6), length 116)
> >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> > 3490676018], length 64
>
> Turning hardware offload all the way off appears to resolve the issue,
> so this might as well be driver/firmware related. May I ask what
> card/driver/firmware you've been using this with?
>

Hi Frode,

This patch attempted to resolve this BZ -
https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
This BZ is related to HWOL not working.

I didn't test myself but I think other colleagues of mine tested with
NVidia/Mellanox CX5.
I don't think we saw the source address getting mangled in our testing.

Please let me  know if you have any more questions.

Numan

> --
> Frode Nordahl
>
> > --
> > Frode Nordahl
> >
> > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <mmichels@redhat.com> wrote:
> > > >
> > > > Thanks for the update, Numan.
> > > >
> > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > >
> > > Thanks.  I applied both the patches to the main branch.
> > >
> > > Numan
> > >
> > > >
> > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > From: Numan Siddique <numans@ovn.org>
> > > > >
> > > > > Make of use of these new actions for the distributed routers
> > > > > for NAT.  These new actions ensure that both sNAT and dNAT
> > > > > happens in the same zone.  This approach solves a couple of
> > > > > problems:
> > > > >
> > > > >   - The datapath flows generated for external traffic which requires
> > > > >     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> > > > >
> > > > >   - Since there is only one zone, it would avoid multiple recirculations
> > > > >     (improving the performance).
> > > > >
> > > > > If the packet needs to be both sNATted and dNATted (for hairpin traffic
> > > > > with source and destination on the same chassis), then sNAT is done
> > > > > in a separate zone.  To detect this scenario, this patch adds a few
> > > > > extra logical flows.  For each dnat_and_snat entry prior to this patch
> > > > > ovn-northd was generating 9 logical flows and with this patch it now
> > > > > generates 12 logical flows.
> > > > >
> > > > > Similar approach can be taken for gateway routers.
> > > > >
> > > > > Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > ---
> > > > >
> > > > > v2 -> v3
> > > > > ------
> > > > >    * Addressed Mark's comments and updated the documentation.
> > > > >
> > > > > v1 -> v2
> > > > > ------
> > > > >    * Rebased and resolved conflicts.
> > > > >
> > > > >
> > > > >   include/ovn/logical-fields.h |   1 +
> > > > >   lib/logical-fields.c         |   4 +
> > > > >   northd/northd.c              | 147 +++++++--
> > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > >   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
> > > > >   tests/ovn.at                 |   2 +-
> > > > >   tests/system-ovn.at          |  64 ++--
> > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > >
> > > > > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> > > > > index c9675f81c..2118f7933 100644
> > > > > --- a/include/ovn/logical-fields.h
> > > > > +++ b/include/ovn/logical-fields.h
> > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > >   };
> > > > >
> > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > index 7b3d431e0..352a48c89 100644
> > > > > --- a/lib/logical-fields.c
> > > > > +++ b/lib/logical-fields.c
> > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
> > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > >       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
> > > > >                                flags_str);
> > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > +             MLF_USE_SNAT_ZONE);
> > > > > +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> > > > > +                             flags_str);
> > > > >
> > > > >       /* Connection tracking state. */
> > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > index 0ff61deec..e4d051a94 100644
> > > > > --- a/northd/northd.c
> > > > > +++ b/northd/northd.c
> > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
> > > > >                                                                         \
> > > > >       /* Logical router egress stages. */                               \
> > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> > > > > +                   "lr_out_chk_dnat_local")                                  \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
> > > > >
> > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
> > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > >
> > > > >   /* Register to store the eth address associated to a router port for packets
> > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > >                                       undnat_match_p, est_actions,
> > > > >                                       &lb->nlb->header_);
> > > > >           } else {
> > > > > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > > > -                                    undnat_match_p, "ct_dnat;",
> > > > > -                                    &lb->nlb->header_);
> > > > > +            ovn_lflow_add_with_hint(
> > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> > > > > +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > +                &lb->nlb->header_);
> > > > >           }
> > > > >           free(undnat_match_p);
> > > > >   next:
> > > > > @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > >           uint16_t priority;
> > > > >
> > > > >           /* Priority of logical flows corresponding to exempted_ext_ips is
> > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > +         * +2 of the corresponding regular NAT rule.
> > > > >            * For example, if we have following NAT rule and we associate
> > > > >            * exempted external ips to it:
> > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> > > > > @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > >            * And now we associate exempted external ip address set to it.
> > > > >            * Now corresponding to above rule we will have following logical
> > > > >            * flows:
> > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
> > > > >            *                             action=(next;)
> > > > >            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
> > > > >            *
> > > > >            */
> > > > >           if (is_src) {
> > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > -            priority = 100 + 1;
> > > > > +            priority = 100 + 2;
> > > > >           } else {
> > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > >
> > > > >               if (!od->is_gw_router) {
> > > > >                   priority += 128;
> > > > > @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >           /* Traffic received on l3dgw_port is subject to NAT. */
> > > > >           ds_clear(match);
> > > > >           ds_clear(actions);
> > > > > -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > > > > -                      is_v6 ? "6" : "4", nat->external_ip,
> > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> > > > > +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
> > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > >               /* Flows for NAT rules that are centralized are only
> > > > >               * programmed on the gateway chassis. */
> > > > > @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >               ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > >                             is_v6 ? "6" : "4", nat->logical_ip);
> > > > >           } else {
> > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > >           }
> > > > >
> > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > >                                   100, ds_cstr(match), ds_cstr(actions),
> > > > >                                   &nat->header_);
> > > > > +
> > > > > +        if (!stateless) {
> > > > > +            ds_clear(match);
> > > > > +            ds_clear(actions);
> > > > > +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> > > > > +                          is_v6 ? "6" : "4", nat->external_ip,
> > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > +                * programmed on the gateway chassis. */
> > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > +                            od->l3dgw_ports[0]->cr_port->json_key);
> > > > > +            }
> > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > > +                                    100, ds_cstr(match), ds_cstr(actions),
> > > > > +                                    &nat->header_);
> > > > > +        }
> > > > >       }
> > > > >   }
> > > > >
> > > > > @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >                   ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > >                                 is_v6 ? "6" : "4", nat->logical_ip);
> > > > >               } else {
> > > > > -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> > > > > +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > >                   if (nat->external_port_range[0]) {
> > > > >                       ds_put_format(actions, ",%s", nat->external_port_range);
> > > > >                   }
> > > > > @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > >                         is_v6 ? "6" : "4", nat->external_ip);
> > > > >       } else {
> > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > +        ds_put_format(actions,
> > > > > +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> > > > >       }
> > > > >
> > > > >       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >                               &nat->header_);
> > > > >   }
> > > > >
> > > > > +static void
> > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> > > > > +                                const struct nbrec_nat *nat, struct ds *match,
> > > > > +                                struct ds *actions, bool distributed,
> > > > > +                                bool is_v6)
> > > > > +{
> > > > > +    /* Note that this only applies for NAT on a distributed router.
> > > > > +     */
> > > > > +    if (!od->n_l3dgw_ports) {
> > > > > +        return;
> > > > > +    }
> > > > > +
> > > > > +    ds_clear(match);
> > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > +    if (distributed) {
> > > > > +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > +    } else {
> > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > +                      od->l3dgw_ports[0]->cr_port->json_key);
> > > > > +    }
> > > > > +
> > > > > +    ds_clear(actions);
> > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> > > > > +
> > > > > +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > +                            50, ds_cstr(match), ds_cstr(actions),
> > > > > +                            &nat->header_);
> > > > > +}
> > > > > +
> > > > >   static void
> > > > >   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >                               const struct nbrec_nat *nat, struct ds *match,
> > > > > @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > > > >                         is_v6 ? "6" : "4", nat->logical_ip,
> > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > -            /* Flows for NAT rules that are centralized are only
> > > > > -            * programmed on the gateway chassis. */
> > > > > -            priority += 128;
> > > > > -            ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > -                          od->l3dgw_ports[0]->cr_port->json_key);
> > > > > -        } else if (distributed) {
> > > > > -            priority += 128;
> > > > > -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > -                          nat->logical_port);
> > > > > +        if (od->n_l3dgw_ports) {
> > > > > +            if (distributed) {
> > > > > +                ovs_assert(nat->logical_port);
> > > > > +                priority += 128;
> > > > > +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > +                              nat->logical_port);
> > > > > +            } else {
> > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > +                * programmed on the gateway chassis. */
> > > > > +                priority += 128;
> > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > +                              od->l3dgw_ports[0]->cr_port->json_key);
> > > > > +            }
> > > > >           }
> > > > >           ds_clear(actions);
> > > > >
> > > > > @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >               ds_put_format(actions, "ip%s.src=%s; next;",
> > > > >                             is_v6 ? "6" : "4", nat->external_ip);
> > > > >           } else {
> > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > +            ds_put_format(actions, "ct_snat_in_czone(%s",
> > > > >                           nat->external_ip);
> > > > >               if (nat->external_port_range[0]) {
> > > > >                   ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > >                                   priority, ds_cstr(match),
> > > > >                                   ds_cstr(actions), &nat->header_);
> > > > > +
> > > > > +        if (!stateless) {
> > > > > +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > +            ds_clear(actions);
> > > > > +            if (distributed) {
> > > > > +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > +            }
> > > > > +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > +                          nat->external_ip);
> > > > > +            if (nat->external_port_range[0]) {
> > > > > +                ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > +            }
> > > > > +            ds_put_format(actions, ");");
> > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > > +                                    priority + 1, ds_cstr(match),
> > > > > +                                    ds_cstr(actions), &nat->header_);
> > > > > +        }
> > > > >       }
> > > > >   }
> > > > >
> > > > > @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > >       /* Packets are allowed by default. */
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
> > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > >
> > > > > @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > >        * not committed, it would produce ongoing datapath flows with the ct.new
> > > > >        * flag set. Some NICs are unable to offload these flows.
> > > > >        */
> > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > >                         "ip", "flags.loopback = 1; ct_dnat;");
> > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > >               }
> > > > >           }
> > > > >
> > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> > > > > +                                        distributed, is_v6);
> > > > > +
> > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > >           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
> > > > >                                         mac, is_v6);
> > > > > @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > >                             "clone { ct_clear; "
> > > > >                             "inport = outport; outport = \"\"; "
> > > > >                             "eth.dst <-> eth.src; "
> > > > > -                          "flags = 0; flags.loopback = 1; ");
> > > > > +                          "flags = 0; flags.loopback = 1; "
> > > > > +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > >                   ds_put_format(actions, "reg%d = 0; ", j);
> > > > >               }
> > > > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > > > > index 21d83718c..e39e6e805 100644
> > > > > --- a/northd/ovn-northd.8.xml
> > > > > +++ b/northd/ovn-northd.8.xml
> > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > >           <p>
> > > > >             For each configuration in the OVN Northbound database, that asks
> > > > >             to change the source IP address of a packet from <var>A</var> to
> > > > > -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> > > > > -          <code>ip &amp;&amp;
> > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> > > > > -          where <var>GW</var> is the logical router gateway port, with an
> > > > > -          action <code>ct_snat;</code>. If the NAT rule is of type
> > > > > -          dnat_and_snat and has <code>stateless=true</code> in the
> > > > > -          options, then the action would be <code>ip4/6.dst=
> > > > > -          (<var>B</var>)</code>.
> > > > > +          <var>B</var>, two priority-100 flows are added.
> > > > >           </p>
> > > > >
> > > > >           <p>
> > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > -          the priority-100 flow above is only programmed on the
> > > > > +          the below priority-100 flows are only programmed on the
> > > > >             gateway chassis.
> > > > >           </p>
> > > > >
> > > > > +        <ul>
> > > > > +          <li>
> > > > > +            <p>
> > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > +              <code>ip &amp;&amp;
> > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> > > > > +              zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > +            </p>
> > > > > +
> > > > > +            <p>
> > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > +              <var>GW</var>.
> > > > > +            </p>
> > > > > +          </li>
> > > > > +
> > > > > +          <li>
> > > > > +            <p>
> > > > > +              The second flow matches <code>ip &amp;&amp;
> > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > +              flags.use_snat_zone == 1</code> or
> > > > > +              <code>ip &amp;&amp;
> > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > +              flags.use_snat_zone == 1</code>
> > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > +            </p>
> > > > > +
> > > > > +            <p>
> > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > +              <var>GW</var>.
> > > > > +            </p>
> > > > > +          </li>
> > > > > +        </ul>
> > > > > +
> > > > >           <p>
> > > > >             A priority-0 logical flow with match <code>1</code> has actions
> > > > >             <code>next;</code>.
> > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > >         </li>
> > > > >       </ul>
> > > > >
> > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > +
> > > > > +    <p>
> > > > > +      This table checks if the packet needs to be DNATed in the router ingress
> > > > > +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> > > > > +      to the ingress pipeline.  This check is done only for routers configured
> > > > > +      with distributed gateway ports and NAT entries.  This check is done
> > > > > +      so that SNAT and DNAT is done in different zones instead of a common
> > > > > +      zone.
> > > > > +    </p>
> > > > > +
> > > > > +    <ul>
> > > > > +      <li>
> > > > > +        <p>
> > > > > +          For each NAT rule in the OVN Northbound database on a
> > > > > +          distributed router, a priority-50 logical flow with match
> > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> > > > > +          external IP address specified in the NAT rule, <var>GW</var>
> > > > > +          is the logical router distributed gateway port. For dnat_and_snat
> > > > > +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> > > > > +          If <ref column="logical_port"
> > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> > > > > +          <var>P</var> is the <code>chassisredirect port</code> of
> > > > > +          <var>GW</var> with the actions:
> > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> > > > > +        </p>
> > > > > +      </li>
> > > > > +
> > > > > +      <li>
> > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > +      </li>
> > > > > +    </ul>
> > > > > +
> > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > >
> > > > >       <p>
> > > > >         This is for already established connections' reverse traffic.
> > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > >         is unDNATed here.
> > > > >       </p>
> > > > >
> > > > > +    <ul>
> > > > > +      <li>
> > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > +        <code>next;</code>.
> > > > > +      </li>
> > > > > +    </ul>
> > > > > +
> > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > +
> > > > > +    <ul>
> > > > > +      <li>
> > > > > +        For all IP packets, a priority-50 flow with an action
> > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > +      </li>
> > > > > +    </ul>
> > > > > +
> > > > > +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
> > > > >       <ul>
> > > > >         <li>
> > > > >           <p>
> > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > >             gateway chassis that matches
> > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
> > > > >             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> > > > > -          router gateway port with an action <code>ct_dnat;</code>. If the
> > > > > -          backend IPv4 address <var>B</var> is also configured with L4 port
> > > > > -          <var>PORT</var> of protocol <var>P</var>, then the
> > > > > +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> > > > > +          If the backend IPv4 address <var>B</var> is also configured with
> > > > > +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
> > > > >             match also includes <code>P.src</code> == <var>PORT</var>.  These
> > > > >             flows are not added for load balancers with IPv6 <var>VIPs</var>.
> > > > >           </p>
> > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > >             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
> > > > >             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
> > > > >             is the logical router gateway port, with an action
> > > > > -          <code>ct_dnat;</code>. If the NAT rule is of type
> > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
> > > > >             dnat_and_snat and has <code>stateless=true</code> in the
> > > > >             options, then the action would be <code>ip4/6.src=
> > > > >             (<var>B</var>)</code>.
> > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > >           <p>
> > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > >             the priority-100 flow above is only programmed on the
> > > > > -          gateway chassis.
> > > > > +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
> > > > >           </p>
> > > > >
> > > > >           <p>
> > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > >           </p>
> > > > >         </li>
> > > > >
> > > > > -      <li>
> > > > > -        For all IP packets, a priority-50 flow with an action
> > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > -      </li>
> > > > > -
> > > > > -      <li>
> > > > > -        A priority-0 logical flow with match <code>1</code> has actions
> > > > > -        <code>next;</code>.
> > > > > -      </li>
> > > > >       </ul>
> > > > >
> > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > >
> > > > >       <p>
> > > > >         <ul>
> > > > >           <li>
> > > > >             A priority-50 logical flow is added that commits any untracked flows
> > > > > -          from the previous table <code>lr_out_undnat</code>. This flow
> > > > > -          matches on <code>ct.new &amp;&amp; ip</code> with action
> > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > +          from the previous table <code>lr_out_undnat</code> for Gateway
> > > > > +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> > > > > +          with action <code>ct_commit { } ; next; </code>.
> > > > >           </li>
> > > > >
> > > > >           <li>
> > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > >         </ul>
> > > > >       </p>
> > > > >
> > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > >
> > > > >       <p>
> > > > >         Packets that are configured to be SNATed get their source IP address
> > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > >         </li>
> > > > >       </ul>
> > > > >
> > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > >
> > > > >       <ul>
> > > > >         <li>
> > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > >         </li>
> > > > >       </ul>
> > > > >
> > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > >
> > > > >       <ul>
> > > > >         <li>
> > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > >             For each configuration in the OVN Northbound database, that asks
> > > > >             to change the source IP address of a packet from an IP address of
> > > > >             <var>A</var> or to change the source IP address of a packet that
> > > > > -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > -          outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > -          logical router gateway port, with an action
> > > > > -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> > > > > -          is calculated based on the mask of <var>A</var>, with matches
> > > > > -          having larger masks getting higher priorities. If the NAT rule
> > > > > -          is of type dnat_and_snat and has <code>stateless=true</code>
> > > > > -          in the options, then the action would be <code>ip4/6.src=
> > > > > -          (<var>B</var>)</code>.
> > > > > +          belongs to network <var>A</var> to <var>B</var>, two flows are
> > > > > +          added.  The priority <var>P</var> of these flows are calculated
> > > > > +          based on the mask of <var>A</var>, with matches having larger
> > > > > +          masks getting higher priorities.
> > > > >           </p>
> > > > >
> > > > >           <p>
> > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > -          the flow above is only programmed on the
> > > > > -          gateway chassis increasing flow priority by 128 in
> > > > > -          order to be run first
> > > > > +          the below flows are only programmed on the gateway chassis increasing
> > > > > +          flow priority by 128 in order to be run first.
> > > > >           </p>
> > > > >
> > > > > +        <ul>
> > > > > +          <li>
> > > > > +            The first flow is added with the calculated priority <var>P</var>
> > > > > +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > +            outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > +            logical router gateway port, with an action
> > > > > +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > +            common zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > +            <code>stateless=true</code> in the options, then the action
> > > > > +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > +          </li>
> > > > > +
> > > > > +          <li>
> > > > > +            The second flow is added with the calculated priority
> > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> > > > > +            logical router gateway port, with an action
> > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> > > > > +            If the NAT rule is of type dnat_and_snat and has
> > > > > +            <code>stateless=true</code> in the options, then the action would
> > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > +          </li>
> > > > > +        </ul>
> > > > > +
> > > > >           <p>
> > > > >             If the NAT rule can be handled in a distributed manner, then
> > > > > -          there is an additional action
> > > > > +          there is an additional action (for both the flows)
> > > > >             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
> > > > >             is the ethernet address associated with the IP address
> > > > >             <var>A</var> in the NAT rule.  This allows upstream MAC
> > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > >
> > > > >           <p>
> > > > >             If the NAT rule has <code>exempted_ext_ips</code> set, then
> > > > > -          there is an additional flow configured at the priority + 1 of
> > > > > +          there is an additional flow configured at the priority
> > > > > +          <code><var>P</var> + 2 </code> of
> > > > >             corresponding NAT rule. The flow matches if destination ip
> > > > >             is an <code>exempted_ext_ip</code> and the action is <code>next;
> > > > >             </code>. This flow is used to bypass the ct_snat action for a flow
> > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > >         </li>
> > > > >       </ul>
> > > > >
> > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > >
> > > > >       <p>
> > > > >         For distributed logical routers where one of the logical router
> > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > >       outport = "";
> > > > >       flags = 0;
> > > > >       flags.loopback = 1;
> > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > >       reg0 = 0;
> > > > >       reg1 = 0;
> > > > >       ...
> > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > >         </li>
> > > > >       </ul>
> > > > >
> > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > >
> > > > >       <p>
> > > > >         Packets that reach this table are ready for delivery.  It contains:
> > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > --- a/tests/ovn-northd.at
> > > > > +++ b/tests/ovn-northd.at
> > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > >   echo
> > > > >   echo "IPv4: stateful"
> > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > >
> > > > >   echo
> > > > >   echo "IPv4: stateless"
> > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > >
> > > > >   echo
> > > > >   echo "IPv6: stateful"
> > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > >
> > > > >   echo
> > > > >   echo "IPv6: stateless"
> > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > >
> > > > >   AT_CLEANUP
> > > > >   ])
> > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
> > > > >
> > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> > > > > -  [0], [1
> > > > > +  [0], [2
> > > > >   1
> > > > >   ])
> > > > >
> > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
> > > > >
> > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> > > > > -  [1], [1
> > > > > +  [1], [2
> > > > >   0
> > > > >   ])
> > > > >
> > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
> > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > >
> > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
> > > > >
> > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > >   ovn-sbctl dump-flows CR > crflows
> > > > >   AT_CAPTURE_FILE([crflows])
> > > > >
> > > > > -AT_CHECK([
> > > > > -  grep -c lr_out_snat drflows
> > > > > -  grep -c lr_out_snat crflows
> > > > > -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > -3
> > > > > -3
> > > > > -1
> > > > > -1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
> > > > >   ])
> > > > >
> > > > > +
> > > > >   # SNAT with DISALLOWED_IPs
> > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > >   AT_CAPTURE_FILE([crflows2])
> > > > >
> > > > > -AT_CHECK([
> > > > > -  grep -c lr_out_snat drflows2
> > > > > -  grep -c lr_out_snat crflows2
> > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > -4
> > > > > -4
> > > > > -1
> > > > > -1
> > > > > -1
> > > > > -1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > >   ])
> > > > >
> > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >
> > > > >   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > > -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > >
> > > > > -ovn-nbctl show DR
> > > > > -ovn-sbctl dump-flows DR
> > > > > -ovn-nbctl show CR
> > > > > -ovn-sbctl dump-flows CR
> > > > > -
> > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > +ovn-sbctl dump-flows DR > drflows3
> > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > +ovn-sbctl dump-flows CR > crflows3
> > > > > +AT_CAPTURE_FILE([crflows2])
> > > > >
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > >   ])
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > +
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
> > > > >   ])
> > > > >
> > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >
> > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > >
> > > > > -ovn-nbctl show DR
> > > > > -ovn-sbctl dump-flows DR
> > > > > -ovn-nbctl show CR
> > > > > -ovn-sbctl dump-flows CR
> > > > > -
> > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > +ovn-sbctl dump-flows DR > drflows4
> > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > +ovn-sbctl dump-flows CR > crflows4
> > > > > +AT_CAPTURE_FILE([crflows2])
> > > > >
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > >   ])
> > > > >
> > > > >   # Stateless FIP with DISALLOWED_IPs
> > > > > @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > >   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > >
> > > > > -ovn-nbctl show DR
> > > > > -ovn-sbctl dump-flows DR
> > > > > -
> > > > > -ovn-nbctl show CR
> > > > > -ovn-sbctl dump-flows CR
> > > > > -
> > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > +ovn-sbctl dump-flows DR > drflows5
> > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > +ovn-sbctl dump-flows CR > crflows5
> > > > > +AT_CAPTURE_FILE([crflows2])
> > > > >
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > >   ])
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > +
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > >   ])
> > > > >
> > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > >
> > > > > -ovn-nbctl show DR
> > > > > -ovn-sbctl dump-flows DR
> > > > > -ovn-nbctl show CR
> > > > > -ovn-sbctl dump-flows CR
> > > > > +ovn-nbctl --wait=sb sync
> > > > >
> > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > -wc -l`])
> > > > > +ovn-sbctl dump-flows DR > drflows6
> > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > +ovn-sbctl dump-flows CR > crflows6
> > > > > +AT_CAPTURE_FILE([crflows2])
> > > > >
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > >   ])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > -])
> > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > +
> > > > > +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > >   ])
> > > > >
> > > > >   AT_CLEANUP
> > > > > @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> > > > > @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> > > > > @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> > > > > @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> > > > > @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> > > > > @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > >   AT_CLEANUP
> > > > > @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
> > > > >   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
> > > > >   check ovn-nbctl --wait=sb sync
> > > > >
> > > > > +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> > > > > +
> > > > >   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
> > > > >
> > > > >   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> > > > > @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > >   # Create few dnat_and_snat entries
> > > > > @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > >   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > > > > @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > >
> > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > >
> > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   # Associate load balancer to lr0
> > > > > @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > >
> > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > >
> > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > > @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   # Make the logical router as Gateway router
> > > > > @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   # Set lb force snat logical router.
> > > > > @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   # Add a LB VIP same as router ip.
> > > > > @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   # Add IPv6 router port and LB.
> > > > > @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > >   ])
> > > > >
> > > > >   check ovn-nbctl lrp-del lr0-sw0
> > > > > @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > +])
> > > > > +
> > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > >   ])
> > > > >
> > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > >   ])
> > > > >
> > > > >   AT_CLEANUP
> > > > > diff --git a/tests/ovn.at b/tests/ovn.at
> > > > > index 0d606b42f..ae5744407 100644
> > > > > --- a/tests/ovn.at
> > > > > +++ b/tests/ovn.at
> > > > > @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
> > > > >   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
> > > > >     grep -c "$regex" sbflows;
> > > > >   done], [0], [0
> > > > > -1
> > > > > +0
> > > > >   2
> > > > >   2
> > > > >   ])
> > > > > diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> > > > > index c9f5771c9..7f6cb32dc 100644
> > > > > --- a/tests/system-ovn.at
> > > > > +++ b/tests/system-ovn.at
> > > > > @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > >
> > > > >   ovn-nbctl list load_balancer
> > > > >   ovn-sbctl dump-flows R2
> > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > >   grep 'nat(src=20.0.0.2)'])
> > > > >
> > > > >   dnl Test load-balancing that includes L4 ports in NAT.
> > > > > @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > >
> > > > >   ovn-nbctl list load_balancer
> > > > >   ovn-sbctl dump-flows R2
> > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > >   grep 'nat(src=20.0.0.2)'])
> > > > >
> > > > >   rm -f wget*.log
> > > > > @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > >   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > > +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > +
> > > > >   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> > > > > -# from 172.16.1.4
> > > > > +# from fd20::4
> > > > >   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
> > > > >   [0], [dnl
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > +ovs-appctl dpctl/dump-conntrack | grep icmpv6
> > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > +])
> > > > > +
> > > > > +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > +# We verify that the connection is not tracked.
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > +# We verify that the connection is not tracked.
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > +# We verify that the connection is not tracked.
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > +
> > > > >   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
> > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > >   [0], [dnl
> > > > > @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > >
> > > > >   # Check conntrack entries.  First SNAT of 'foo1' address happens.
> > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > >   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > >
> > > > >   # Check conntrack entries.  First SNAT of 'foo2' address happens.
> > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > >   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > -])
> > > > > -
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > >   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
> > > > >   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > -])
> > > > > -
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > >   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
> > > > >   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > > @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > >   ])
> > > > >
> > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > -# unDNAT table in the egress router pipeline
> > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > -])
> > > > > -
> > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > >   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
> > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > > @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > >   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > >   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > >   ])
> > > > > @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
> > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
> > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > >   ])
> > > > >
> > > > >   OVS_APP_EXIT_AND_WAIT([ovn-controller])
> > > > >
> > > >
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev@openvswitch.org
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > >
> > > _______________________________________________
> > > dev mailing list
> > > dev@openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Frode Nordahl April 6, 2022, 8:57 a.m. UTC | #6 
On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org> wrote:
>
> On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> <frode.nordahl@canonical.com> wrote:
> >
> > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > <frode.nordahl@canonical.com> wrote:
> > >
> > > Hello Numan,
> > >
> > > This patch does unfortunately break gateway routers in some
> > > circumstances, (but not all!), at least for the way OpenStack consumes
> > > them.
> > >
> > > Will dig more and try to figure out what is going on, but sending this
> > > e-mail proactively in case you have any ideas from the top of your
> > > head or if anyone else has run into the same issue.
> > >
> > > The gateway appears to handle conntrack as it should:
> > > $ sudo conntrack -E --dst 194.169.254.178
> > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > sport=22 dport=60234 zone=52
> > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > dport=60234 zone=52
> > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> > > sport=22 dport=60234 [ASSURED] zone=52
> > >
> > > However as you can see traffic going to the instance suddenly gets its
> > > source address mangled/set to 0.0.0.0, and the connection can never
> > > establish.
> > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> > > capture size 262144 bytes
> > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> > > proto TCP (6), length 60)
> > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> > > 870680827 ecr 0,nop,wscale 7], length 0
> > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> > > proto TCP (6), length 60)
> > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> > > options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> > > 7], length 0
> > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> > > [DF], proto TCP (6), length 93)
> > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> > > ecr 3490675961], length 41
> > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> > > proto TCP (6), length 52)
> > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > 3490676008 ecr 870680922], length 0
> > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> > > proto TCP (6), length 52)
> > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> > > ecr 3490675961], length 0
> > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> > > proto TCP (6), length 52)
> > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > 3490676009 ecr 870680922], length 0
> > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> > > [DF], proto TCP (6), length 93)
> > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> > > val 3490676018 ecr 870680922], length 41
> > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> > > proto TCP (6), length 52)
> > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> > > 870680932 ecr 3490676018], length 0
> > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> > > [DF], proto TCP (6), length 1564)
> > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > 870680933 ecr 3490676018], length 1512
> > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> > > [DF], proto TCP (6), length 116)
> > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> > > 3490676018], length 64
> >
> > Turning hardware offload all the way off appears to resolve the issue,
> > so this might as well be driver/firmware related. May I ask what
> > card/driver/firmware you've been using this with?
> >
>
> Hi Frode,
>
> This patch attempted to resolve this BZ -
> https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> This BZ is related to HWOL not working.
>
> I didn't test myself but I think other colleagues of mine tested with
> NVidia/Mellanox CX5.
> I don't think we saw the source address getting mangled in our testing.

Thank you for sharing those details. We see the source mangling with
CX6 here but only with distributed FIPs off, we'll park this for now
until we receive information from Nvidia/Mellanox as to what's going
on from a driver/firmware perspective.

> Please let me  know if you have any more questions.

We did unfortunately run into another issue which appears to have its
root in this patch. The symptomps are described in
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856

This rudimentary patch does appear to fix it, but it may require
acquiring and passing in the `distributed` variable from the
`lrouter_check_nat_entry` function two new places, so I wanted to air
this with you before proceeding with the fix:
diff --git a/northd/northd.c b/northd/northd.c
index 2fb0a93c2..5fae010c0 100644
--- a/northd/northd.c
+++ b/northd/northd.c
@@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
                                     undnat_match_p, est_actions,
                                     &lb->nlb->header_);
         } else {
+            /* XXX do we need to check for distributed here? */
             ovn_lflow_add_with_hint(
                 lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
                 od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
@@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
*lflows, struct ovn_datapath *od,
                       is_v6 ? "6" : "4", nat->external_ip);
     } else {
         ds_put_format(actions,
-                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
+                      (od->is_gw_router ||
+                       (!distributed && od->n_l3dgw_ports)) ?
+                          "ct_dnat;" : "ct_dnat_in_czone;");
     }

     ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
@@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
ovn_datapath *od, struct hmap *lflows,
      * not committed, it would produce ongoing datapath flows with the ct.new
      * flag set. Some NICs are unable to offload these flows.
      */
-    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
+    /* XXX we probably need to get the distributed variable passed in here to
+     * XXX retain the proposed optimization */
+    if ((od->is_gw_router || od->n_l3dgw_ports) &&
+        (od->nbr->n_nat || od->has_lb_vip)) {
         ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
                       "ip", "flags.loopback = 1; ct_dnat;");
         ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,

The environment in question is using distributed gateway ports by
associating multiple Gateway_Chassis with a Logical_Router_Port and at
the same time does not use distributed FIPs.
Numan Siddique April 6, 2022, 4:11 p.m. UTC | #7 
On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
<frode.nordahl@canonical.com> wrote:
>
> On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > <frode.nordahl@canonical.com> wrote:
> > >
> > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > <frode.nordahl@canonical.com> wrote:
> > > >
> > > > Hello Numan,
> > > >
> > > > This patch does unfortunately break gateway routers in some
> > > > circumstances, (but not all!), at least for the way OpenStack consumes
> > > > them.
> > > >
> > > > Will dig more and try to figure out what is going on, but sending this
> > > > e-mail proactively in case you have any ideas from the top of your
> > > > head or if anyone else has run into the same issue.
> > > >
> > > > The gateway appears to handle conntrack as it should:
> > > > $ sudo conntrack -E --dst 194.169.254.178
> > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > sport=22 dport=60234 zone=52
> > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > dport=60234 zone=52
> > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> > > > sport=22 dport=60234 [ASSURED] zone=52
> > > >
> > > > However as you can see traffic going to the instance suddenly gets its
> > > > source address mangled/set to 0.0.0.0, and the connection can never
> > > > establish.
> > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> > > > capture size 262144 bytes
> > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> > > > proto TCP (6), length 60)
> > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> > > > proto TCP (6), length 60)
> > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> > > > options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> > > > 7], length 0
> > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> > > > [DF], proto TCP (6), length 93)
> > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > ecr 3490675961], length 41
> > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> > > > proto TCP (6), length 52)
> > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > 3490676008 ecr 870680922], length 0
> > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> > > > proto TCP (6), length 52)
> > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > ecr 3490675961], length 0
> > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> > > > proto TCP (6), length 52)
> > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > 3490676009 ecr 870680922], length 0
> > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> > > > [DF], proto TCP (6), length 93)
> > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> > > > val 3490676018 ecr 870680922], length 41
> > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> > > > proto TCP (6), length 52)
> > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> > > > 870680932 ecr 3490676018], length 0
> > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> > > > [DF], proto TCP (6), length 1564)
> > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > 870680933 ecr 3490676018], length 1512
> > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> > > > [DF], proto TCP (6), length 116)
> > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> > > > 3490676018], length 64
> > >
> > > Turning hardware offload all the way off appears to resolve the issue,
> > > so this might as well be driver/firmware related. May I ask what
> > > card/driver/firmware you've been using this with?
> > >
> >
> > Hi Frode,
> >
> > This patch attempted to resolve this BZ -
> > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > This BZ is related to HWOL not working.
> >
> > I didn't test myself but I think other colleagues of mine tested with
> > NVidia/Mellanox CX5.
> > I don't think we saw the source address getting mangled in our testing.
>
> Thank you for sharing those details. We see the source mangling with
> CX6 here but only with distributed FIPs off, we'll park this for now
> until we receive information from Nvidia/Mellanox as to what's going
> on from a driver/firmware perspective.
>
> > Please let me  know if you have any more questions.
>
> We did unfortunately run into another issue which appears to have its
> root in this patch. The symptomps are described in
> https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
>
> This rudimentary patch does appear to fix it, but it may require
> acquiring and passing in the `distributed` variable from the
> `lrouter_check_nat_entry` function two new places, so I wanted to air

Thanks for finding this issue.  I think we should use separate zones
for hairpin traffic.
And I thought I had addressed it.  To handle this hairpin issue,  the
patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
and if this bit is set, then ct_snat() action is used instead of
ct_snat_in_czone().
Let me test it out and update here.

Thanks
Numan


> this with you before proceeding with the fix:
> diff --git a/northd/northd.c b/northd/northd.c
> index 2fb0a93c2..5fae010c0 100644
> --- a/northd/northd.c
> +++ b/northd/northd.c
> @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
>                                      undnat_match_p, est_actions,
>                                      &lb->nlb->header_);
>          } else {
> +            /* XXX do we need to check for distributed here? */
>              ovn_lflow_add_with_hint(
>                  lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
>                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> *lflows, struct ovn_datapath *od,
>                        is_v6 ? "6" : "4", nat->external_ip);
>      } else {
>          ds_put_format(actions,
> -                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> +                      (od->is_gw_router ||
> +                       (!distributed && od->n_l3dgw_ports)) ?
> +                          "ct_dnat;" : "ct_dnat_in_czone;");
>      }
>
>      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> ovn_datapath *od, struct hmap *lflows,
>       * not committed, it would produce ongoing datapath flows with the ct.new
>       * flag set. Some NICs are unable to offload these flows.
>       */
> -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> +    /* XXX we probably need to get the distributed variable passed in here to
> +     * XXX retain the proposed optimization */
> +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> +        (od->nbr->n_nat || od->has_lb_vip)) {
>          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
>                        "ip", "flags.loopback = 1; ct_dnat;");
>          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
>
> The environment in question is using distributed gateway ports by
> associating multiple Gateway_Chassis with a Logical_Router_Port and at
> the same time does not use distributed FIPs.
>
> --
> Frode Nordahl
>
> > Numan
> >
> > > --
> > > Frode Nordahl
> > >
> > > > --
> > > > Frode Nordahl
> > > >
> > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <numans@ovn.org> wrote:
> > > > >
> > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <mmichels@redhat.com> wrote:
> > > > > >
> > > > > > Thanks for the update, Numan.
> > > > > >
> > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > >
> > > > > Thanks.  I applied both the patches to the main branch.
> > > > >
> > > > > Numan
> > > > >
> > > > > >
> > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > >
> > > > > > > Make of use of these new actions for the distributed routers
> > > > > > > for NAT.  These new actions ensure that both sNAT and dNAT
> > > > > > > happens in the same zone.  This approach solves a couple of
> > > > > > > problems:
> > > > > > >
> > > > > > >   - The datapath flows generated for external traffic which requires
> > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> > > > > > >
> > > > > > >   - Since there is only one zone, it would avoid multiple recirculations
> > > > > > >     (improving the performance).
> > > > > > >
> > > > > > > If the packet needs to be both sNATted and dNATted (for hairpin traffic
> > > > > > > with source and destination on the same chassis), then sNAT is done
> > > > > > > in a separate zone.  To detect this scenario, this patch adds a few
> > > > > > > extra logical flows.  For each dnat_and_snat entry prior to this patch
> > > > > > > ovn-northd was generating 9 logical flows and with this patch it now
> > > > > > > generates 12 logical flows.
> > > > > > >
> > > > > > > Similar approach can be taken for gateway routers.
> > > > > > >
> > > > > > > Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > ---
> > > > > > >
> > > > > > > v2 -> v3
> > > > > > > ------
> > > > > > >    * Addressed Mark's comments and updated the documentation.
> > > > > > >
> > > > > > > v1 -> v2
> > > > > > > ------
> > > > > > >    * Rebased and resolved conflicts.
> > > > > > >
> > > > > > >
> > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > >   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
> > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > >
> > > > > > > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > >   };
> > > > > > >
> > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > --- a/lib/logical-fields.c
> > > > > > > +++ b/lib/logical-fields.c
> > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
> > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > >       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
> > > > > > >                                flags_str);
> > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> > > > > > > +                             flags_str);
> > > > > > >
> > > > > > >       /* Connection tracking state. */
> > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > --- a/northd/northd.c
> > > > > > > +++ b/northd/northd.c
> > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
> > > > > > >                                                                         \
> > > > > > >       /* Logical router egress stages. */                               \
> > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> > > > > > > +                   "lr_out_chk_dnat_local")                                  \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
> > > > > > >
> > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
> > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > >
> > > > > > >   /* Register to store the eth address associated to a router port for packets
> > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > >                                       undnat_match_p, est_actions,
> > > > > > >                                       &lb->nlb->header_);
> > > > > > >           } else {
> > > > > > > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > -                                    undnat_match_p, "ct_dnat;",
> > > > > > > -                                    &lb->nlb->header_);
> > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> > > > > > > +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > > > +                &lb->nlb->header_);
> > > > > > >           }
> > > > > > >           free(undnat_match_p);
> > > > > > >   next:
> > > > > > > @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > >           uint16_t priority;
> > > > > > >
> > > > > > >           /* Priority of logical flows corresponding to exempted_ext_ips is
> > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > >            * For example, if we have following NAT rule and we associate
> > > > > > >            * exempted external ips to it:
> > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> > > > > > > @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > >            * And now we associate exempted external ip address set to it.
> > > > > > >            * Now corresponding to above rule we will have following logical
> > > > > > >            * flows:
> > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
> > > > > > >            *                             action=(next;)
> > > > > > >            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
> > > > > > >            *
> > > > > > >            */
> > > > > > >           if (is_src) {
> > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > -            priority = 100 + 1;
> > > > > > > +            priority = 100 + 2;
> > > > > > >           } else {
> > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > >
> > > > > > >               if (!od->is_gw_router) {
> > > > > > >                   priority += 128;
> > > > > > > @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >           /* Traffic received on l3dgw_port is subject to NAT. */
> > > > > > >           ds_clear(match);
> > > > > > >           ds_clear(actions);
> > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > > > > > > -                      is_v6 ? "6" : "4", nat->external_ip,
> > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > > > +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> > > > > > > +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
> > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > >               /* Flows for NAT rules that are centralized are only
> > > > > > >               * programmed on the gateway chassis. */
> > > > > > > @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >               ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > > > >                             is_v6 ? "6" : "4", nat->logical_ip);
> > > > > > >           } else {
> > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > >           }
> > > > > > >
> > > > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > > > >                                   100, ds_cstr(match), ds_cstr(actions),
> > > > > > >                                   &nat->header_);
> > > > > > > +
> > > > > > > +        if (!stateless) {
> > > > > > > +            ds_clear(match);
> > > > > > > +            ds_clear(actions);
> > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > > > +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> > > > > > > +                          is_v6 ? "6" : "4", nat->external_ip,
> > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > +                            od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > +            }
> > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > > > > +                                    100, ds_cstr(match), ds_cstr(actions),
> > > > > > > +                                    &nat->header_);
> > > > > > > +        }
> > > > > > >       }
> > > > > > >   }
> > > > > > >
> > > > > > > @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >                   ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > > > >                                 is_v6 ? "6" : "4", nat->logical_ip);
> > > > > > >               } else {
> > > > > > > -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> > > > > > > +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > >                       ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > >                   }
> > > > > > > @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > >                         is_v6 ? "6" : "4", nat->external_ip);
> > > > > > >       } else {
> > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > +        ds_put_format(actions,
> > > > > > > +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> > > > > > >       }
> > > > > > >
> > > > > > >       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >                               &nat->header_);
> > > > > > >   }
> > > > > > >
> > > > > > > +static void
> > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > +                                const struct nbrec_nat *nat, struct ds *match,
> > > > > > > +                                struct ds *actions, bool distributed,
> > > > > > > +                                bool is_v6)
> > > > > > > +{
> > > > > > > +    /* Note that this only applies for NAT on a distributed router.
> > > > > > > +     */
> > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > +        return;
> > > > > > > +    }
> > > > > > > +
> > > > > > > +    ds_clear(match);
> > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > +    if (distributed) {
> > > > > > > +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > +    } else {
> > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > +                      od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > +    }
> > > > > > > +
> > > > > > > +    ds_clear(actions);
> > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> > > > > > > +
> > > > > > > +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > +                            50, ds_cstr(match), ds_cstr(actions),
> > > > > > > +                            &nat->header_);
> > > > > > > +}
> > > > > > > +
> > > > > > >   static void
> > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >                               const struct nbrec_nat *nat, struct ds *match,
> > > > > > > @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > > > > > >                         is_v6 ? "6" : "4", nat->logical_ip,
> > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > -            /* Flows for NAT rules that are centralized are only
> > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > -            priority += 128;
> > > > > > > -            ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > -                          od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > -        } else if (distributed) {
> > > > > > > -            priority += 128;
> > > > > > > -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > > > -                          nat->logical_port);
> > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > +            if (distributed) {
> > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > +                priority += 128;
> > > > > > > +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > > > +                              nat->logical_port);
> > > > > > > +            } else {
> > > > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > +                priority += 128;
> > > > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > +                              od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > +            }
> > > > > > >           }
> > > > > > >           ds_clear(actions);
> > > > > > >
> > > > > > > @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >               ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > >                             is_v6 ? "6" : "4", nat->external_ip);
> > > > > > >           } else {
> > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > +            ds_put_format(actions, "ct_snat_in_czone(%s",
> > > > > > >                           nat->external_ip);
> > > > > > >               if (nat->external_port_range[0]) {
> > > > > > >                   ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > > @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > > > >                                   priority, ds_cstr(match),
> > > > > > >                                   ds_cstr(actions), &nat->header_);
> > > > > > > +
> > > > > > > +        if (!stateless) {
> > > > > > > +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > +            ds_clear(actions);
> > > > > > > +            if (distributed) {
> > > > > > > +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > +            }
> > > > > > > +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > +                          nat->external_ip);
> > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > +                ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > > +            }
> > > > > > > +            ds_put_format(actions, ");");
> > > > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > > > > +                                    priority + 1, ds_cstr(match),
> > > > > > > +                                    ds_cstr(actions), &nat->header_);
> > > > > > > +        }
> > > > > > >       }
> > > > > > >   }
> > > > > > >
> > > > > > > @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > >       /* Packets are allowed by default. */
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
> > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > >
> > > > > > > @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > >        * not committed, it would produce ongoing datapath flows with the ct.new
> > > > > > >        * flag set. Some NICs are unable to offload these flows.
> > > > > > >        */
> > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > > > >                         "ip", "flags.loopback = 1; ct_dnat;");
> > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > >               }
> > > > > > >           }
> > > > > > >
> > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> > > > > > > +                                        distributed, is_v6);
> > > > > > > +
> > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
> > > > > > >                                         mac, is_v6);
> > > > > > > @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > >                             "clone { ct_clear; "
> > > > > > >                             "inport = outport; outport = \"\"; "
> > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > -                          "flags = 0; flags.loopback = 1; ");
> > > > > > > +                          "flags = 0; flags.loopback = 1; "
> > > > > > > +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > >                   ds_put_format(actions, "reg%d = 0; ", j);
> > > > > > >               }
> > > > > > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > >           <p>
> > > > > > >             For each configuration in the OVN Northbound database, that asks
> > > > > > >             to change the source IP address of a packet from <var>A</var> to
> > > > > > > -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> > > > > > > -          <code>ip &amp;&amp;
> > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> > > > > > > -          where <var>GW</var> is the logical router gateway port, with an
> > > > > > > -          action <code>ct_snat;</code>. If the NAT rule is of type
> > > > > > > -          dnat_and_snat and has <code>stateless=true</code> in the
> > > > > > > -          options, then the action would be <code>ip4/6.dst=
> > > > > > > -          (<var>B</var>)</code>.
> > > > > > > +          <var>B</var>, two priority-100 flows are added.
> > > > > > >           </p>
> > > > > > >
> > > > > > >           <p>
> > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > > -          the priority-100 flow above is only programmed on the
> > > > > > > +          the below priority-100 flows are only programmed on the
> > > > > > >             gateway chassis.
> > > > > > >           </p>
> > > > > > >
> > > > > > > +        <ul>
> > > > > > > +          <li>
> > > > > > > +            <p>
> > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > +              <code>ip &amp;&amp;
> > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > > > +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> > > > > > > +              zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > +            </p>
> > > > > > > +
> > > > > > > +            <p>
> > > > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > > > +              <var>GW</var>.
> > > > > > > +            </p>
> > > > > > > +          </li>
> > > > > > > +
> > > > > > > +          <li>
> > > > > > > +            <p>
> > > > > > > +              The second flow matches <code>ip &amp;&amp;
> > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > +              <code>ip &amp;&amp;
> > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > > > +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > +            </p>
> > > > > > > +
> > > > > > > +            <p>
> > > > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > > > +              <var>GW</var>.
> > > > > > > +            </p>
> > > > > > > +          </li>
> > > > > > > +        </ul>
> > > > > > > +
> > > > > > >           <p>
> > > > > > >             A priority-0 logical flow with match <code>1</code> has actions
> > > > > > >             <code>next;</code>.
> > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > >         </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > +
> > > > > > > +    <p>
> > > > > > > +      This table checks if the packet needs to be DNATed in the router ingress
> > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> > > > > > > +      to the ingress pipeline.  This check is done only for routers configured
> > > > > > > +      with distributed gateway ports and NAT entries.  This check is done
> > > > > > > +      so that SNAT and DNAT is done in different zones instead of a common
> > > > > > > +      zone.
> > > > > > > +    </p>
> > > > > > > +
> > > > > > > +    <ul>
> > > > > > > +      <li>
> > > > > > > +        <p>
> > > > > > > +          For each NAT rule in the OVN Northbound database on a
> > > > > > > +          distributed router, a priority-50 logical flow with match
> > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> > > > > > > +          external IP address specified in the NAT rule, <var>GW</var>
> > > > > > > +          is the logical router distributed gateway port. For dnat_and_snat
> > > > > > > +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> > > > > > > +          If <ref column="logical_port"
> > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> > > > > > > +          <var>P</var> is the <code>chassisredirect port</code> of
> > > > > > > +          <var>GW</var> with the actions:
> > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> > > > > > > +        </p>
> > > > > > > +      </li>
> > > > > > > +
> > > > > > > +      <li>
> > > > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > +      </li>
> > > > > > > +    </ul>
> > > > > > > +
> > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > >
> > > > > > >       <p>
> > > > > > >         This is for already established connections' reverse traffic.
> > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > >         is unDNATed here.
> > > > > > >       </p>
> > > > > > >
> > > > > > > +    <ul>
> > > > > > > +      <li>
> > > > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > +        <code>next;</code>.
> > > > > > > +      </li>
> > > > > > > +    </ul>
> > > > > > > +
> > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > +
> > > > > > > +    <ul>
> > > > > > > +      <li>
> > > > > > > +        For all IP packets, a priority-50 flow with an action
> > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > +      </li>
> > > > > > > +    </ul>
> > > > > > > +
> > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
> > > > > > >       <ul>
> > > > > > >         <li>
> > > > > > >           <p>
> > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > >             gateway chassis that matches
> > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
> > > > > > >             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> > > > > > > -          router gateway port with an action <code>ct_dnat;</code>. If the
> > > > > > > -          backend IPv4 address <var>B</var> is also configured with L4 port
> > > > > > > -          <var>PORT</var> of protocol <var>P</var>, then the
> > > > > > > +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> > > > > > > +          If the backend IPv4 address <var>B</var> is also configured with
> > > > > > > +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
> > > > > > >             match also includes <code>P.src</code> == <var>PORT</var>.  These
> > > > > > >             flows are not added for load balancers with IPv6 <var>VIPs</var>.
> > > > > > >           </p>
> > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > >             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
> > > > > > >             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
> > > > > > >             is the logical router gateway port, with an action
> > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of type
> > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
> > > > > > >             dnat_and_snat and has <code>stateless=true</code> in the
> > > > > > >             options, then the action would be <code>ip4/6.src=
> > > > > > >             (<var>B</var>)</code>.
> > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > >           <p>
> > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > >             the priority-100 flow above is only programmed on the
> > > > > > > -          gateway chassis.
> > > > > > > +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
> > > > > > >           </p>
> > > > > > >
> > > > > > >           <p>
> > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > >           </p>
> > > > > > >         </li>
> > > > > > >
> > > > > > > -      <li>
> > > > > > > -        For all IP packets, a priority-50 flow with an action
> > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > -      </li>
> > > > > > > -
> > > > > > > -      <li>
> > > > > > > -        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > -        <code>next;</code>.
> > > > > > > -      </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > >
> > > > > > >       <p>
> > > > > > >         <ul>
> > > > > > >           <li>
> > > > > > >             A priority-50 logical flow is added that commits any untracked flows
> > > > > > > -          from the previous table <code>lr_out_undnat</code>. This flow
> > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code> with action
> > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > +          from the previous table <code>lr_out_undnat</code> for Gateway
> > > > > > > +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> > > > > > > +          with action <code>ct_commit { } ; next; </code>.
> > > > > > >           </li>
> > > > > > >
> > > > > > >           <li>
> > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > >         </ul>
> > > > > > >       </p>
> > > > > > >
> > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > >
> > > > > > >       <p>
> > > > > > >         Packets that are configured to be SNATed get their source IP address
> > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > >         </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > >
> > > > > > >       <ul>
> > > > > > >         <li>
> > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > >         </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > >
> > > > > > >       <ul>
> > > > > > >         <li>
> > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > >             For each configuration in the OVN Northbound database, that asks
> > > > > > >             to change the source IP address of a packet from an IP address of
> > > > > > >             <var>A</var> or to change the source IP address of a packet that
> > > > > > > -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > -          outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > > > -          logical router gateway port, with an action
> > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> > > > > > > -          is calculated based on the mask of <var>A</var>, with matches
> > > > > > > -          having larger masks getting higher priorities. If the NAT rule
> > > > > > > -          is of type dnat_and_snat and has <code>stateless=true</code>
> > > > > > > -          in the options, then the action would be <code>ip4/6.src=
> > > > > > > -          (<var>B</var>)</code>.
> > > > > > > +          belongs to network <var>A</var> to <var>B</var>, two flows are
> > > > > > > +          added.  The priority <var>P</var> of these flows are calculated
> > > > > > > +          based on the mask of <var>A</var>, with matches having larger
> > > > > > > +          masks getting higher priorities.
> > > > > > >           </p>
> > > > > > >
> > > > > > >           <p>
> > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > > -          the flow above is only programmed on the
> > > > > > > -          gateway chassis increasing flow priority by 128 in
> > > > > > > -          order to be run first
> > > > > > > +          the below flows are only programmed on the gateway chassis increasing
> > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > >           </p>
> > > > > > >
> > > > > > > +        <ul>
> > > > > > > +          <li>
> > > > > > > +            The first flow is added with the calculated priority <var>P</var>
> > > > > > > +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > +            outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > > > +            logical router gateway port, with an action
> > > > > > > +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > +            common zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > > > +            <code>stateless=true</code> in the options, then the action
> > > > > > > +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > +          </li>
> > > > > > > +
> > > > > > > +          <li>
> > > > > > > +            The second flow is added with the calculated priority
> > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> > > > > > > +            logical router gateway port, with an action
> > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> > > > > > > +            If the NAT rule is of type dnat_and_snat and has
> > > > > > > +            <code>stateless=true</code> in the options, then the action would
> > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > +          </li>
> > > > > > > +        </ul>
> > > > > > > +
> > > > > > >           <p>
> > > > > > >             If the NAT rule can be handled in a distributed manner, then
> > > > > > > -          there is an additional action
> > > > > > > +          there is an additional action (for both the flows)
> > > > > > >             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
> > > > > > >             is the ethernet address associated with the IP address
> > > > > > >             <var>A</var> in the NAT rule.  This allows upstream MAC
> > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > >
> > > > > > >           <p>
> > > > > > >             If the NAT rule has <code>exempted_ext_ips</code> set, then
> > > > > > > -          there is an additional flow configured at the priority + 1 of
> > > > > > > +          there is an additional flow configured at the priority
> > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > >             corresponding NAT rule. The flow matches if destination ip
> > > > > > >             is an <code>exempted_ext_ip</code> and the action is <code>next;
> > > > > > >             </code>. This flow is used to bypass the ct_snat action for a flow
> > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > >         </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > >
> > > > > > >       <p>
> > > > > > >         For distributed logical routers where one of the logical router
> > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > >       outport = "";
> > > > > > >       flags = 0;
> > > > > > >       flags.loopback = 1;
> > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > >       reg0 = 0;
> > > > > > >       reg1 = 0;
> > > > > > >       ...
> > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > >         </li>
> > > > > > >       </ul>
> > > > > > >
> > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > >
> > > > > > >       <p>
> > > > > > >         Packets that reach this table are ready for delivery.  It contains:
> > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > --- a/tests/ovn-northd.at
> > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > >   echo
> > > > > > >   echo "IPv4: stateful"
> > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > >
> > > > > > >   echo
> > > > > > >   echo "IPv4: stateless"
> > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > >
> > > > > > >   echo
> > > > > > >   echo "IPv6: stateful"
> > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > >
> > > > > > >   echo
> > > > > > >   echo "IPv6: stateless"
> > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > >
> > > > > > >   AT_CLEANUP
> > > > > > >   ])
> > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
> > > > > > >
> > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> > > > > > > -  [0], [1
> > > > > > > +  [0], [2
> > > > > > >   1
> > > > > > >   ])
> > > > > > >
> > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
> > > > > > >
> > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> > > > > > > -  [1], [1
> > > > > > > +  [1], [2
> > > > > > >   0
> > > > > > >   ])
> > > > > > >
> > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
> > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > >
> > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
> > > > > > >
> > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > >
> > > > > > > -AT_CHECK([
> > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > -3
> > > > > > > -3
> > > > > > > -1
> > > > > > > -1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
> > > > > > >   ])
> > > > > > >
> > > > > > > +
> > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > >
> > > > > > > -AT_CHECK([
> > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > -4
> > > > > > > -4
> > > > > > > -1
> > > > > > > -1
> > > > > > > -1
> > > > > > > -1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >
> > > > > > >   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > >
> > > > > > > -ovn-nbctl show DR
> > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > -ovn-nbctl show CR
> > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > -
> > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > +ovn-sbctl dump-flows DR > drflows3
> > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > +ovn-sbctl dump-flows CR > crflows3
> > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > >
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > > > >   ])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > +
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > > > @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >
> > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > >
> > > > > > > -ovn-nbctl show DR
> > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > -ovn-nbctl show CR
> > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > -
> > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > +ovn-sbctl dump-flows DR > drflows4
> > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > +ovn-sbctl dump-flows CR > crflows4
> > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > >
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Stateless FIP with DISALLOWED_IPs
> > > > > > > @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > >   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > >
> > > > > > > -ovn-nbctl show DR
> > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > -
> > > > > > > -ovn-nbctl show CR
> > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > -
> > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > +ovn-sbctl dump-flows DR > drflows5
> > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > +ovn-sbctl dump-flows CR > crflows5
> > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > >
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > > > >   ])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > +
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > > > @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > >
> > > > > > > -ovn-nbctl show DR
> > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > -ovn-nbctl show CR
> > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > +ovn-nbctl --wait=sb sync
> > > > > > >
> > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > -wc -l`])
> > > > > > > +ovn-sbctl dump-flows DR > drflows6
> > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > +ovn-sbctl dump-flows CR > crflows6
> > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > >
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > >   ])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > > > -])
> > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > +
> > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CLEANUP
> > > > > > > @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> > > > > > > @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> > > > > > > @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> > > > > > > @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> > > > > > > @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> > > > > > > @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CLEANUP
> > > > > > > @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
> > > > > > >   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
> > > > > > >   check ovn-nbctl --wait=sb sync
> > > > > > >
> > > > > > > +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> > > > > > > +
> > > > > > >   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
> > > > > > >
> > > > > > >   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> > > > > > > @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Create few dnat_and_snat entries
> > > > > > > @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > > > > > > @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Associate load balancer to lr0
> > > > > > > @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > >
> > > > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > > > > @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Make the logical router as Gateway router
> > > > > > > @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Set lb force snat logical router.
> > > > > > > @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Add a LB VIP same as router ip.
> > > > > > > @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   # Add IPv6 router port and LB.
> > > > > > > @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > >   ])
> > > > > > >
> > > > > > >   check ovn-nbctl lrp-del lr0-sw0
> > > > > > > @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > +])
> > > > > > > +
> > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > >   ])
> > > > > > >
> > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CLEANUP
> > > > > > > diff --git a/tests/ovn.at b/tests/ovn.at
> > > > > > > index 0d606b42f..ae5744407 100644
> > > > > > > --- a/tests/ovn.at
> > > > > > > +++ b/tests/ovn.at
> > > > > > > @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
> > > > > > >   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
> > > > > > >     grep -c "$regex" sbflows;
> > > > > > >   done], [0], [0
> > > > > > > -1
> > > > > > > +0
> > > > > > >   2
> > > > > > >   2
> > > > > > >   ])
> > > > > > > diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> > > > > > > index c9f5771c9..7f6cb32dc 100644
> > > > > > > --- a/tests/system-ovn.at
> > > > > > > +++ b/tests/system-ovn.at
> > > > > > > @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > > > >
> > > > > > >   ovn-nbctl list load_balancer
> > > > > > >   ovn-sbctl dump-flows R2
> > > > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > > > >   grep 'nat(src=20.0.0.2)'])
> > > > > > >
> > > > > > >   dnl Test load-balancing that includes L4 ports in NAT.
> > > > > > > @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > > > >
> > > > > > >   ovn-nbctl list load_balancer
> > > > > > >   ovn-sbctl dump-flows R2
> > > > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > > > >   grep 'nat(src=20.0.0.2)'])
> > > > > > >
> > > > > > >   rm -f wget*.log
> > > > > > > @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > >   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > > +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > +
> > > > > > >   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> > > > > > > -# from 172.16.1.4
> > > > > > > +# from fd20::4
> > > > > > >   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
> > > > > > >   [0], [dnl
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > > > +ovs-appctl dpctl/dump-conntrack | grep icmpv6
> > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > +])
> > > > > > > +
> > > > > > > +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > +# We verify that the connection is not tracked.
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > +# We verify that the connection is not tracked.
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > +# We verify that the connection is not tracked.
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > +
> > > > > > >   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
> > > > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > >   [0], [dnl
> > > > > > > @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > >
> > > > > > >   # Check conntrack entries.  First SNAT of 'foo1' address happens.
> > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > >   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > >
> > > > > > >   # Check conntrack entries.  First SNAT of 'foo2' address happens.
> > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > >   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > -])
> > > > > > > -
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > >   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
> > > > > > >   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > > @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > -])
> > > > > > > -
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > >   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
> > > > > > >   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > > > > @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > >   ])
> > > > > > >
> > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > -])
> > > > > > > -
> > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > >   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
> > > > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > > > > @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > >   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > >   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > > @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
> > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
> > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > >   ])
> > > > > > >
> > > > > > >   OVS_APP_EXIT_AND_WAIT([ovn-controller])
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > dev mailing list
> > > > > > dev@openvswitch.org
> > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > > >
> > > > > _______________________________________________
> > > > > dev mailing list
> > > > > dev@openvswitch.org
> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > _______________________________________________
> > > dev mailing list
> > > dev@openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Numan Siddique April 7, 2022, 4:51 p.m. UTC | #8 
On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
>
> On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> <frode.nordahl@canonical.com> wrote:
> >
> > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > <frode.nordahl@canonical.com> wrote:
> > > >
> > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > <frode.nordahl@canonical.com> wrote:
> > > > >
> > > > > Hello Numan,
> > > > >
> > > > > This patch does unfortunately break gateway routers in some
> > > > > circumstances, (but not all!), at least for the way OpenStack consumes
> > > > > them.
> > > > >
> > > > > Will dig more and try to figure out what is going on, but sending this
> > > > > e-mail proactively in case you have any ideas from the top of your
> > > > > head or if anyone else has run into the same issue.
> > > > >
> > > > > The gateway appears to handle conntrack as it should:
> > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > sport=22 dport=60234 zone=52
> > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > dport=60234 zone=52
> > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > >
> > > > > However as you can see traffic going to the instance suddenly gets its
> > > > > source address mangled/set to 0.0.0.0, and the connection can never
> > > > > establish.
> > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> > > > > capture size 262144 bytes
> > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> > > > > proto TCP (6), length 60)
> > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> > > > > proto TCP (6), length 60)
> > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> > > > > options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> > > > > 7], length 0
> > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> > > > > [DF], proto TCP (6), length 93)
> > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > > ecr 3490675961], length 41
> > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> > > > > proto TCP (6), length 52)
> > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > > 3490676008 ecr 870680922], length 0
> > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> > > > > proto TCP (6), length 52)
> > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > > ecr 3490675961], length 0
> > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> > > > > proto TCP (6), length 52)
> > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > > 3490676009 ecr 870680922], length 0
> > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> > > > > [DF], proto TCP (6), length 93)
> > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> > > > > val 3490676018 ecr 870680922], length 41
> > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> > > > > proto TCP (6), length 52)
> > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> > > > > 870680932 ecr 3490676018], length 0
> > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> > > > > [DF], proto TCP (6), length 1564)
> > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > 870680933 ecr 3490676018], length 1512
> > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> > > > > [DF], proto TCP (6), length 116)
> > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> > > > > 3490676018], length 64
> > > >
> > > > Turning hardware offload all the way off appears to resolve the issue,
> > > > so this might as well be driver/firmware related. May I ask what
> > > > card/driver/firmware you've been using this with?
> > > >
> > >
> > > Hi Frode,
> > >
> > > This patch attempted to resolve this BZ -
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > This BZ is related to HWOL not working.
> > >
> > > I didn't test myself but I think other colleagues of mine tested with
> > > NVidia/Mellanox CX5.
> > > I don't think we saw the source address getting mangled in our testing.
> >
> > Thank you for sharing those details. We see the source mangling with
> > CX6 here but only with distributed FIPs off, we'll park this for now
> > until we receive information from Nvidia/Mellanox as to what's going
> > on from a driver/firmware perspective.
> >
> > > Please let me  know if you have any more questions.
> >
> > We did unfortunately run into another issue which appears to have its
> > root in this patch. The symptomps are described in
> > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> >
> > This rudimentary patch does appear to fix it, but it may require
> > acquiring and passing in the `distributed` variable from the
> > `lrouter_check_nat_entry` function two new places, so I wanted to air
>
> Thanks for finding this issue.  I think we should use separate zones
> for hairpin traffic.
> And I thought I had addressed it.  To handle this hairpin issue,  the
> patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> and if this bit is set, then ct_snat() action is used instead of
> ct_snat_in_czone().
> Let me test it out and update here.

I tested it locally and it seems to work for me.

Can you please share your OVN dbs if possible ?  I've left a comment
on the launchpad bug.  You can attach it there too if it's possible.

Numan

>
> Thanks
> Numan
>
>
> > this with you before proceeding with the fix:
> > diff --git a/northd/northd.c b/northd/northd.c
> > index 2fb0a93c2..5fae010c0 100644
> > --- a/northd/northd.c
> > +++ b/northd/northd.c
> > @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> >                                      undnat_match_p, est_actions,
> >                                      &lb->nlb->header_);
> >          } else {
> > +            /* XXX do we need to check for distributed here? */
> >              ovn_lflow_add_with_hint(
> >                  lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> >                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> > *lflows, struct ovn_datapath *od,
> >                        is_v6 ? "6" : "4", nat->external_ip);
> >      } else {
> >          ds_put_format(actions,
> > -                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> > +                      (od->is_gw_router ||
> > +                       (!distributed && od->n_l3dgw_ports)) ?
> > +                          "ct_dnat;" : "ct_dnat_in_czone;");
> >      }
> >
> >      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> > ovn_datapath *od, struct hmap *lflows,
> >       * not committed, it would produce ongoing datapath flows with the ct.new
> >       * flag set. Some NICs are unable to offload these flows.
> >       */
> > -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > +    /* XXX we probably need to get the distributed variable passed in here to
> > +     * XXX retain the proposed optimization */
> > +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > +        (od->nbr->n_nat || od->has_lb_vip)) {
> >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> >                        "ip", "flags.loopback = 1; ct_dnat;");
> >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> >
> > The environment in question is using distributed gateway ports by
> > associating multiple Gateway_Chassis with a Logical_Router_Port and at
> > the same time does not use distributed FIPs.
> >
> > --
> > Frode Nordahl
> >
> > > Numan
> > >
> > > > --
> > > > Frode Nordahl
> > > >
> > > > > --
> > > > > Frode Nordahl
> > > > >
> > > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <numans@ovn.org> wrote:
> > > > > >
> > > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <mmichels@redhat.com> wrote:
> > > > > > >
> > > > > > > Thanks for the update, Numan.
> > > > > > >
> > > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > > >
> > > > > > Thanks.  I applied both the patches to the main branch.
> > > > > >
> > > > > > Numan
> > > > > >
> > > > > > >
> > > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > > >
> > > > > > > > Make of use of these new actions for the distributed routers
> > > > > > > > for NAT.  These new actions ensure that both sNAT and dNAT
> > > > > > > > happens in the same zone.  This approach solves a couple of
> > > > > > > > problems:
> > > > > > > >
> > > > > > > >   - The datapath flows generated for external traffic which requires
> > > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely HWOL'able.
> > > > > > > >
> > > > > > > >   - Since there is only one zone, it would avoid multiple recirculations
> > > > > > > >     (improving the performance).
> > > > > > > >
> > > > > > > > If the packet needs to be both sNATted and dNATted (for hairpin traffic
> > > > > > > > with source and destination on the same chassis), then sNAT is done
> > > > > > > > in a separate zone.  To detect this scenario, this patch adds a few
> > > > > > > > extra logical flows.  For each dnat_and_snat entry prior to this patch
> > > > > > > > ovn-northd was generating 9 logical flows and with this patch it now
> > > > > > > > generates 12 logical flows.
> > > > > > > >
> > > > > > > > Similar approach can be taken for gateway routers.
> > > > > > > >
> > > > > > > > Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > > ---
> > > > > > > >
> > > > > > > > v2 -> v3
> > > > > > > > ------
> > > > > > > >    * Addressed Mark's comments and updated the documentation.
> > > > > > > >
> > > > > > > > v1 -> v2
> > > > > > > > ------
> > > > > > > >    * Rebased and resolved conflicts.
> > > > > > > >
> > > > > > > >
> > > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > > >   tests/ovn-northd.at          | 575 +++++++++++++++++++----------------
> > > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > > >
> > > > > > > > diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
> > > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > > >   };
> > > > > > > >
> > > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > > --- a/lib/logical-fields.c
> > > > > > > > +++ b/lib/logical-fields.c
> > > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash *symtab)
> > > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > > >       expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
> > > > > > > >                                flags_str);
> > > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > > +    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
> > > > > > > > +                             flags_str);
> > > > > > > >
> > > > > > > >       /* Connection tracking state. */
> > > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
> > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > > --- a/northd/northd.c
> > > > > > > > +++ b/northd/northd.c
> > > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
> > > > > > > >                                                                         \
> > > > > > > >       /* Logical router egress stages. */                               \
> > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
> > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
> > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
> > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
> > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
> > > > > > > > +                   "lr_out_chk_dnat_local")                                  \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
> > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
> > > > > > > >
> > > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
> > > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
> > > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > > >
> > > > > > > >   /* Register to store the eth address associated to a router port for packets
> > > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > > @@ -9568,9 +9572,10 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > > >                                       undnat_match_p, est_actions,
> > > > > > > >                                       &lb->nlb->header_);
> > > > > > > >           } else {
> > > > > > > > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > > -                                    undnat_match_p, "ct_dnat;",
> > > > > > > > -                                    &lb->nlb->header_);
> > > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
> > > > > > > > +                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > > > > +                &lb->nlb->header_);
> > > > > > > >           }
> > > > > > > >           free(undnat_match_p);
> > > > > > > >   next:
> > > > > > > > @@ -9865,7 +9870,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > >           uint16_t priority;
> > > > > > > >
> > > > > > > >           /* Priority of logical flows corresponding to exempted_ext_ips is
> > > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > > >            * For example, if we have following NAT rule and we associate
> > > > > > > >            * exempted external ips to it:
> > > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
> > > > > > > > @@ -9873,17 +9878,17 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > >            * And now we associate exempted external ip address set to it.
> > > > > > > >            * Now corresponding to above rule we will have following logical
> > > > > > > >            * flows:
> > > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
> > > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
> > > > > > > >            *                             action=(next;)
> > > > > > > >            * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
> > > > > > > >            *
> > > > > > > >            */
> > > > > > > >           if (is_src) {
> > > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > > -            priority = 100 + 1;
> > > > > > > > +            priority = 100 + 2;
> > > > > > > >           } else {
> > > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> > > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > > >
> > > > > > > >               if (!od->is_gw_router) {
> > > > > > > >                   priority += 128;
> > > > > > > > @@ -12268,9 +12273,9 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >           /* Traffic received on l3dgw_port is subject to NAT. */
> > > > > > > >           ds_clear(match);
> > > > > > > >           ds_clear(actions);
> > > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> > > > > > > > -                      is_v6 ? "6" : "4", nat->external_ip,
> > > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > > > > +                      "flags.loopback == 0", is_v6 ? "6" : "4",
> > > > > > > > +                      nat->external_ip, od->l3dgw_ports[0]->json_key);
> > > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > > >               /* Flows for NAT rules that are centralized are only
> > > > > > > >               * programmed on the gateway chassis. */
> > > > > > > > @@ -12282,12 +12287,31 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >               ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > > > > >                             is_v6 ? "6" : "4", nat->logical_ip);
> > > > > > > >           } else {
> > > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > > >           }
> > > > > > > >
> > > > > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > > > > >                                   100, ds_cstr(match), ds_cstr(actions),
> > > > > > > >                                   &nat->header_);
> > > > > > > > +
> > > > > > > > +        if (!stateless) {
> > > > > > > > +            ds_clear(match);
> > > > > > > > +            ds_clear(actions);
> > > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
> > > > > > > > +                          "flags.loopback == 1 && flags.use_snat_zone == 1",
> > > > > > > > +                          is_v6 ? "6" : "4", nat->external_ip,
> > > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > > +                            od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > +            }
> > > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
> > > > > > > > +                                    100, ds_cstr(match), ds_cstr(actions),
> > > > > > > > +                                    &nat->header_);
> > > > > > > > +        }
> > > > > > > >       }
> > > > > > > >   }
> > > > > > > >
> > > > > > > > @@ -12364,7 +12388,7 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >                   ds_put_format(actions, "ip%s.dst=%s; next;",
> > > > > > > >                                 is_v6 ? "6" : "4", nat->logical_ip);
> > > > > > > >               } else {
> > > > > > > > -                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
> > > > > > > > +                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > > >                       ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > > >                   }
> > > > > > > > @@ -12417,7 +12441,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > >                         is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > >       } else {
> > > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > > +        ds_put_format(actions,
> > > > > > > > +                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
> > > > > > > >       }
> > > > > > > >
> > > > > > > >       ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > @@ -12425,6 +12450,36 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >                               &nat->header_);
> > > > > > > >   }
> > > > > > > >
> > > > > > > > +static void
> > > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > +                                const struct nbrec_nat *nat, struct ds *match,
> > > > > > > > +                                struct ds *actions, bool distributed,
> > > > > > > > +                                bool is_v6)
> > > > > > > > +{
> > > > > > > > +    /* Note that this only applies for NAT on a distributed router.
> > > > > > > > +     */
> > > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > > +        return;
> > > > > > > > +    }
> > > > > > > > +
> > > > > > > > +    ds_clear(match);
> > > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > > +    if (distributed) {
> > > > > > > > +        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > > +    } else {
> > > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > > +                      od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > +    }
> > > > > > > > +
> > > > > > > > +    ds_clear(actions);
> > > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
> > > > > > > > +
> > > > > > > > +    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > > +                            50, ds_cstr(match), ds_cstr(actions),
> > > > > > > > +                            &nat->header_);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > >   static void
> > > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >                               const struct nbrec_nat *nat, struct ds *match,
> > > > > > > > @@ -12478,16 +12533,19 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >           ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> > > > > > > >                         is_v6 ? "6" : "4", nat->logical_ip,
> > > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > -            /* Flows for NAT rules that are centralized are only
> > > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > > -            priority += 128;
> > > > > > > > -            ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > > -                          od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > -        } else if (distributed) {
> > > > > > > > -            priority += 128;
> > > > > > > > -            ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > > > > -                          nat->logical_port);
> > > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > > +            if (distributed) {
> > > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > > +                priority += 128;
> > > > > > > > +                ds_put_format(match, " && is_chassis_resident(\"%s\")",
> > > > > > > > +                              nat->logical_port);
> > > > > > > > +            } else {
> > > > > > > > +                /* Flows for NAT rules that are centralized are only
> > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > +                priority += 128;
> > > > > > > > +                ds_put_format(match, " && is_chassis_resident(%s)",
> > > > > > > > +                              od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > +            }
> > > > > > > >           }
> > > > > > > >           ds_clear(actions);
> > > > > > > >
> > > > > > > > @@ -12505,7 +12563,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >               ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > >                             is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > >           } else {
> > > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > > +            ds_put_format(actions, "ct_snat_in_czone(%s",
> > > > > > > >                           nat->external_ip);
> > > > > > > >               if (nat->external_port_range[0]) {
> > > > > > > >                   ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > > > @@ -12519,6 +12577,24 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > >           ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > > > > >                                   priority, ds_cstr(match),
> > > > > > > >                                   ds_cstr(actions), &nat->header_);
> > > > > > > > +
> > > > > > > > +        if (!stateless) {
> > > > > > > > +            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > > +            ds_clear(actions);
> > > > > > > > +            if (distributed) {
> > > > > > > > +                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
> > > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > > +            }
> > > > > > > > +            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > > +                          nat->external_ip);
> > > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > > +                ds_put_format(actions, ",%s", nat->external_port_range);
> > > > > > > > +            }
> > > > > > > > +            ds_put_format(actions, ");");
> > > > > > > > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
> > > > > > > > +                                    priority + 1, ds_cstr(match),
> > > > > > > > +                                    ds_cstr(actions), &nat->header_);
> > > > > > > > +        }
> > > > > > > >       }
> > > > > > > >   }
> > > > > > > >
> > > > > > > > @@ -12749,10 +12825,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > > >       /* Packets are allowed by default. */
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
> > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
> > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > > >
> > > > > > > > @@ -12765,8 +12844,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > > >        * not committed, it would produce ongoing datapath flows with the ct.new
> > > > > > > >        * flag set. Some NICs are unable to offload these flows.
> > > > > > > >        */
> > > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > > > > >                         "ip", "flags.loopback = 1; ct_dnat;");
> > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > > @@ -12839,6 +12917,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > > >               }
> > > > > > > >           }
> > > > > > > >
> > > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
> > > > > > > > +                                        distributed, is_v6);
> > > > > > > > +
> > > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
> > > > > > > >                                         mac, is_v6);
> > > > > > > > @@ -12912,7 +12994,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> > > > > > > >                             "clone { ct_clear; "
> > > > > > > >                             "inport = outport; outport = \"\"; "
> > > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > > -                          "flags = 0; flags.loopback = 1; ");
> > > > > > > > +                          "flags = 0; flags.loopback = 1; "
> > > > > > > > +                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > > >                   ds_put_format(actions, "reg%d = 0; ", j);
> > > > > > > >               }
> > > > > > > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> > > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > > >           <p>
> > > > > > > >             For each configuration in the OVN Northbound database, that asks
> > > > > > > >             to change the source IP address of a packet from <var>A</var> to
> > > > > > > > -          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
> > > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
> > > > > > > > -          <code>ip &amp;&amp;
> > > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
> > > > > > > > -          where <var>GW</var> is the logical router gateway port, with an
> > > > > > > > -          action <code>ct_snat;</code>. If the NAT rule is of type
> > > > > > > > -          dnat_and_snat and has <code>stateless=true</code> in the
> > > > > > > > -          options, then the action would be <code>ip4/6.dst=
> > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > +          <var>B</var>, two priority-100 flows are added.
> > > > > > > >           </p>
> > > > > > > >
> > > > > > > >           <p>
> > > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > > > -          the priority-100 flow above is only programmed on the
> > > > > > > > +          the below priority-100 flows are only programmed on the
> > > > > > > >             gateway chassis.
> > > > > > > >           </p>
> > > > > > > >
> > > > > > > > +        <ul>
> > > > > > > > +          <li>
> > > > > > > > +            <p>
> > > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > > > > +              action <code>ct_snat_in_czone;</code> to unSNAT in the common
> > > > > > > > +              zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > +            </p>
> > > > > > > > +
> > > > > > > > +            <p>
> > > > > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > > > > +              <var>GW</var>.
> > > > > > > > +            </p>
> > > > > > > > +          </li>
> > > > > > > > +
> > > > > > > > +          <li>
> > > > > > > > +            <p>
> > > > > > > > +              The second flow matches <code>ip &amp;&amp;
> > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
> > > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > > +              where <var>GW</var> is the logical router gateway port, with an
> > > > > > > > +              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
> > > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > > +              <code>stateless=true</code> in the options, then the action
> > > > > > > > +              would be <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > +            </p>
> > > > > > > > +
> > > > > > > > +            <p>
> > > > > > > > +              If the NAT entry is of type <code>snat</code>, then there is an
> > > > > > > > +              additional match <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > +              </code> where <var>cr-GW</var> is the chassis resident port of
> > > > > > > > +              <var>GW</var>.
> > > > > > > > +            </p>
> > > > > > > > +          </li>
> > > > > > > > +        </ul>
> > > > > > > > +
> > > > > > > >           <p>
> > > > > > > >             A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > >             <code>next;</code>.
> > > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > > >         </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > > +
> > > > > > > > +    <p>
> > > > > > > > +      This table checks if the packet needs to be DNATed in the router ingress
> > > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed  and looped back
> > > > > > > > +      to the ingress pipeline.  This check is done only for routers configured
> > > > > > > > +      with distributed gateway ports and NAT entries.  This check is done
> > > > > > > > +      so that SNAT and DNAT is done in different zones instead of a common
> > > > > > > > +      zone.
> > > > > > > > +    </p>
> > > > > > > > +
> > > > > > > > +    <ul>
> > > > > > > > +      <li>
> > > > > > > > +        <p>
> > > > > > > > +          For each NAT rule in the OVN Northbound database on a
> > > > > > > > +          distributed router, a priority-50 logical flow with match
> > > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > > +          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
> > > > > > > > +          external IP address specified in the NAT rule, <var>GW</var>
> > > > > > > > +          is the logical router distributed gateway port. For dnat_and_snat
> > > > > > > > +          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
> > > > > > > > +          If <ref column="logical_port"
> > > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
> > > > > > > > +          <var>P</var> is the <code>chassisredirect port</code> of
> > > > > > > > +          <var>GW</var> with the actions:
> > > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
> > > > > > > > +        </p>
> > > > > > > > +      </li>
> > > > > > > > +
> > > > > > > > +      <li>
> > > > > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > > +      </li>
> > > > > > > > +    </ul>
> > > > > > > > +
> > > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > > >
> > > > > > > >       <p>
> > > > > > > >         This is for already established connections' reverse traffic.
> > > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > > >         is unDNATed here.
> > > > > > > >       </p>
> > > > > > > >
> > > > > > > > +    <ul>
> > > > > > > > +      <li>
> > > > > > > > +        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > > +        <code>next;</code>.
> > > > > > > > +      </li>
> > > > > > > > +    </ul>
> > > > > > > > +
> > > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > > +
> > > > > > > > +    <ul>
> > > > > > > > +      <li>
> > > > > > > > +        For all IP packets, a priority-50 flow with an action
> > > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > +      </li>
> > > > > > > > +    </ul>
> > > > > > > > +
> > > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
> > > > > > > >       <ul>
> > > > > > > >         <li>
> > > > > > > >           <p>
> > > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > > >             gateway chassis that matches
> > > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
> > > > > > > >             outport == <var>GW</var></code>, where <var>GW</var> is the logical
> > > > > > > > -          router gateway port with an action <code>ct_dnat;</code>. If the
> > > > > > > > -          backend IPv4 address <var>B</var> is also configured with L4 port
> > > > > > > > -          <var>PORT</var> of protocol <var>P</var>, then the
> > > > > > > > +          router gateway port with an action <code>ct_dnat_in_czone;</code>.
> > > > > > > > +          If the backend IPv4 address <var>B</var> is also configured with
> > > > > > > > +          L4 port <var>PORT</var> of protocol <var>P</var>, then the
> > > > > > > >             match also includes <code>P.src</code> == <var>PORT</var>.  These
> > > > > > > >             flows are not added for load balancers with IPv6 <var>VIPs</var>.
> > > > > > > >           </p>
> > > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > > >             matches <code>ip &amp;&amp; ip4.src == <var>B</var>
> > > > > > > >             &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
> > > > > > > >             is the logical router gateway port, with an action
> > > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of type
> > > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
> > > > > > > >             dnat_and_snat and has <code>stateless=true</code> in the
> > > > > > > >             options, then the action would be <code>ip4/6.src=
> > > > > > > >             (<var>B</var>)</code>.
> > > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > > >           <p>
> > > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > > >             the priority-100 flow above is only programmed on the
> > > > > > > > -          gateway chassis.
> > > > > > > > +          gateway chassis with the action <code>ct_dnat_in_czone</code>.
> > > > > > > >           </p>
> > > > > > > >
> > > > > > > >           <p>
> > > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > > >           </p>
> > > > > > > >         </li>
> > > > > > > >
> > > > > > > > -      <li>
> > > > > > > > -        For all IP packets, a priority-50 flow with an action
> > > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > -      </li>
> > > > > > > > -
> > > > > > > > -      <li>
> > > > > > > > -        A priority-0 logical flow with match <code>1</code> has actions
> > > > > > > > -        <code>next;</code>.
> > > > > > > > -      </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > > >
> > > > > > > >       <p>
> > > > > > > >         <ul>
> > > > > > > >           <li>
> > > > > > > >             A priority-50 logical flow is added that commits any untracked flows
> > > > > > > > -          from the previous table <code>lr_out_undnat</code>. This flow
> > > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code> with action
> > > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > > +          from the previous table <code>lr_out_undnat</code> for Gateway
> > > > > > > > +          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
> > > > > > > > +          with action <code>ct_commit { } ; next; </code>.
> > > > > > > >           </li>
> > > > > > > >
> > > > > > > >           <li>
> > > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > > >         </ul>
> > > > > > > >       </p>
> > > > > > > >
> > > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > > >
> > > > > > > >       <p>
> > > > > > > >         Packets that are configured to be SNATed get their source IP address
> > > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > > >         </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > > >
> > > > > > > >       <ul>
> > > > > > > >         <li>
> > > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > > >         </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > > >
> > > > > > > >       <ul>
> > > > > > > >         <li>
> > > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > > >             For each configuration in the OVN Northbound database, that asks
> > > > > > > >             to change the source IP address of a packet from an IP address of
> > > > > > > >             <var>A</var> or to change the source IP address of a packet that
> > > > > > > > -          belongs to network <var>A</var> to <var>B</var>, a flow matches
> > > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > > -          outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > > > > -          logical router gateway port, with an action
> > > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
> > > > > > > > -          is calculated based on the mask of <var>A</var>, with matches
> > > > > > > > -          having larger masks getting higher priorities. If the NAT rule
> > > > > > > > -          is of type dnat_and_snat and has <code>stateless=true</code>
> > > > > > > > -          in the options, then the action would be <code>ip4/6.src=
> > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > +          belongs to network <var>A</var> to <var>B</var>, two flows are
> > > > > > > > +          added.  The priority <var>P</var> of these flows are calculated
> > > > > > > > +          based on the mask of <var>A</var>, with matches having larger
> > > > > > > > +          masks getting higher priorities.
> > > > > > > >           </p>
> > > > > > > >
> > > > > > > >           <p>
> > > > > > > >             If the NAT rule cannot be handled in a distributed manner, then
> > > > > > > > -          the flow above is only programmed on the
> > > > > > > > -          gateway chassis increasing flow priority by 128 in
> > > > > > > > -          order to be run first
> > > > > > > > +          the below flows are only programmed on the gateway chassis increasing
> > > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > > >           </p>
> > > > > > > >
> > > > > > > > +        <ul>
> > > > > > > > +          <li>
> > > > > > > > +            The first flow is added with the calculated priority <var>P</var>
> > > > > > > > +            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > > +            outport == <var>GW</var></code>, where <var>GW</var> is the
> > > > > > > > +            logical router gateway port, with an action
> > > > > > > > +            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > > +            common zone.  If the NAT rule is of type dnat_and_snat and has
> > > > > > > > +            <code>stateless=true</code> in the options, then the action
> > > > > > > > +            would be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > +          </li>
> > > > > > > > +
> > > > > > > > +          <li>
> > > > > > > > +            The second flow is added with the calculated priority
> > > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
> > > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
> > > > > > > > +            logical router gateway port, with an action
> > > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
> > > > > > > > +            If the NAT rule is of type dnat_and_snat and has
> > > > > > > > +            <code>stateless=true</code> in the options, then the action would
> > > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > +          </li>
> > > > > > > > +        </ul>
> > > > > > > > +
> > > > > > > >           <p>
> > > > > > > >             If the NAT rule can be handled in a distributed manner, then
> > > > > > > > -          there is an additional action
> > > > > > > > +          there is an additional action (for both the flows)
> > > > > > > >             <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
> > > > > > > >             is the ethernet address associated with the IP address
> > > > > > > >             <var>A</var> in the NAT rule.  This allows upstream MAC
> > > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > > >
> > > > > > > >           <p>
> > > > > > > >             If the NAT rule has <code>exempted_ext_ips</code> set, then
> > > > > > > > -          there is an additional flow configured at the priority + 1 of
> > > > > > > > +          there is an additional flow configured at the priority
> > > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > > >             corresponding NAT rule. The flow matches if destination ip
> > > > > > > >             is an <code>exempted_ext_ip</code> and the action is <code>next;
> > > > > > > >             </code>. This flow is used to bypass the ct_snat action for a flow
> > > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > > >         </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > > >
> > > > > > > >       <p>
> > > > > > > >         For distributed logical routers where one of the logical router
> > > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > > >       outport = "";
> > > > > > > >       flags = 0;
> > > > > > > >       flags.loopback = 1;
> > > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > > >       reg0 = 0;
> > > > > > > >       reg1 = 0;
> > > > > > > >       ...
> > > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > > >         </li>
> > > > > > > >       </ul>
> > > > > > > >
> > > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > > >
> > > > > > > >       <p>
> > > > > > > >         Packets that reach this table are ready for delivery.  It contains:
> > > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > > --- a/tests/ovn-northd.at
> > > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > > >   echo
> > > > > > > >   echo "IPv4: stateful"
> > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > >
> > > > > > > >   echo
> > > > > > > >   echo "IPv4: stateless"
> > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > >
> > > > > > > >   echo
> > > > > > > >   echo "IPv6: stateful"
> > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > > >
> > > > > > > >   echo
> > > > > > > >   echo "IPv6: stateless"
> > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> > > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > > >
> > > > > > > >   AT_CLEANUP
> > > > > > > >   ])
> > > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
> > > > > > > >
> > > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
> > > > > > > > -  [0], [1
> > > > > > > > +  [0], [2
> > > > > > > >   1
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
> > > > > > > >
> > > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
> > > > > > > > -  [1], [1
> > > > > > > > +  [1], [2
> > > > > > > >   0
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat  172.16.1.1
> > > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > > >
> > > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
> > > > > > > >
> > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > > >
> > > > > > > > -AT_CHECK([
> > > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > > -  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > > -  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > > -3
> > > > > > > > -3
> > > > > > > > -1
> > > > > > > > -1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > +
> > > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > > >
> > > > > > > > -AT_CHECK([
> > > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
> > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > > -4
> > > > > > > > -4
> > > > > > > > -1
> > > > > > > > -1
> > > > > > > > -1
> > > > > > > > -1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > > @@ -1059,25 +1062,24 @@ check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >   check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >
> > > > > > > >   check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > > -check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > > +check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > >
> > > > > > > > -ovn-nbctl show DR
> > > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > > -ovn-nbctl show CR
> > > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > > -
> > > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > +ovn-sbctl dump-flows DR > drflows3
> > > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > > +ovn-sbctl dump-flows CR > crflows3
> > > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > > >
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > > > > >   ])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > > > > @@ -1088,26 +1090,26 @@ ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >   ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >
> > > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > > -ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > > +check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > >
> > > > > > > > -ovn-nbctl show DR
> > > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > > -ovn-nbctl show CR
> > > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > > -
> > > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > +ovn-sbctl dump-flows DR > drflows4
> > > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > > +ovn-sbctl dump-flows CR > crflows4
> > > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > > >
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
> > > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Stateless FIP with DISALLOWED_IPs
> > > > > > > > @@ -1120,24 +1122,21 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >   ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > >   ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
> > > > > > > >
> > > > > > > > -ovn-nbctl show DR
> > > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > > -
> > > > > > > > -ovn-nbctl show CR
> > > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > > -
> > > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > -OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > +ovn-sbctl dump-flows DR > drflows5
> > > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > > +ovn-sbctl dump-flows CR > crflows5
> > > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > > >
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > > > > >   ])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Stateful FIP with DISALLOWED_IPs
> > > > > > > > @@ -1150,23 +1149,25 @@ ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
> > > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > >   ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
> > > > > > > >
> > > > > > > > -ovn-nbctl show DR
> > > > > > > > -ovn-sbctl dump-flows DR
> > > > > > > > -ovn-nbctl show CR
> > > > > > > > -ovn-sbctl dump-flows CR
> > > > > > > > +ovn-nbctl --wait=sb sync
> > > > > > > >
> > > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > -OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
> > > > > > > > -wc -l`])
> > > > > > > > +ovn-sbctl dump-flows DR > drflows6
> > > > > > > > +AT_CAPTURE_FILE([drflows2])
> > > > > > > > +ovn-sbctl dump-flows CR > crflows6
> > > > > > > > +AT_CAPTURE_FILE([crflows2])
> > > > > > > >
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > >   ])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
> > > > > > > > -])
> > > > > > > > -AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > +  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
> > > > > > > > +  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CLEANUP
> > > > > > > > @@ -3475,14 +3476,14 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
> > > > > > > > @@ -3511,21 +3512,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
> > > > > > > > @@ -3557,22 +3558,22 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
> > > > > > > > @@ -3584,9 +3585,9 @@ AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl set logical_router lr0 options:chassis=ch1
> > > > > > > > @@ -3617,23 +3618,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
> > > > > > > > @@ -3661,18 +3662,18 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CLEANUP
> > > > > > > > @@ -4176,6 +4177,8 @@ check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
> > > > > > > >   check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
> > > > > > > >   check ovn-nbctl --wait=sb sync
> > > > > > > >
> > > > > > > > +ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
> > > > > > > > +
> > > > > > > >   AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
> > > > > > > >
> > > > > > > >   dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
> > > > > > > > @@ -4761,17 +4764,17 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Create few dnat_and_snat entries
> > > > > > > > @@ -4797,17 +4800,21 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> > > > > > > > @@ -4828,9 +4835,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > > @@ -4839,26 +4849,34 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Associate load balancer to lr0
> > > > > > > > @@ -4879,9 +4897,12 @@ AT_CAPTURE_FILE([lr0flows])
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > -  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
> > > > > > > > +  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > > @@ -4894,7 +4915,7 @@ AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
> > > > > > > >
> > > > > > > >   AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
> > > > > > > > +  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
> > > > > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > > > > >     table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
> > > > > > > > @@ -4905,27 +4926,35 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Make the logical router as Gateway router
> > > > > > > > @@ -4965,22 +4994,26 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Set lb force snat logical router.
> > > > > > > > @@ -5020,24 +5053,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Add a LB VIP same as router ip.
> > > > > > > > @@ -5081,24 +5118,28 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   # Add IPv6 router port and LB.
> > > > > > > > @@ -5155,26 +5196,30 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
> > > > > > > > +  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   check ovn-nbctl lrp-del lr0-sw0
> > > > > > > > @@ -5209,19 +5254,23 @@ AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
> > > > > > > >     table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > > +AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > > +AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
> > > > > > > > -  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > -  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > > +AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
> > > > > > > > +  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
> > > > > > > > +  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CLEANUP
> > > > > > > > diff --git a/tests/ovn.at b/tests/ovn.at
> > > > > > > > index 0d606b42f..ae5744407 100644
> > > > > > > > --- a/tests/ovn.at
> > > > > > > > +++ b/tests/ovn.at
> > > > > > > > @@ -21604,7 +21604,7 @@ AT_CAPTURE_FILE([sbflows])
> > > > > > > >   AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
> > > > > > > >     grep -c "$regex" sbflows;
> > > > > > > >   done], [0], [0
> > > > > > > > -1
> > > > > > > > +0
> > > > > > > >   2
> > > > > > > >   2
> > > > > > > >   ])
> > > > > > > > diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> > > > > > > > index c9f5771c9..7f6cb32dc 100644
> > > > > > > > --- a/tests/system-ovn.at
> > > > > > > > +++ b/tests/system-ovn.at
> > > > > > > > @@ -2224,7 +2224,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > > > > >
> > > > > > > >   ovn-nbctl list load_balancer
> > > > > > > >   ovn-sbctl dump-flows R2
> > > > > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > > > > >   grep 'nat(src=20.0.0.2)'])
> > > > > > > >
> > > > > > > >   dnl Test load-balancing that includes L4 ports in NAT.
> > > > > > > > @@ -2262,7 +2262,7 @@ ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
> > > > > > > >
> > > > > > > >   ovn-nbctl list load_balancer
> > > > > > > >   ovn-sbctl dump-flows R2
> > > > > > > > -OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
> > > > > > > > +OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
> > > > > > > >   grep 'nat(src=20.0.0.2)'])
> > > > > > > >
> > > > > > > >   rm -f wget*.log
> > > > > > > > @@ -3711,17 +3711,24 @@ sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > >   icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > +
> > > > > > > >   # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
> > > > > > > > -# from 172.16.1.4
> > > > > > > > +# from fd20::4
> > > > > > > >   NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
> > > > > > > >   [0], [dnl
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > > > > +ovs-appctl dpctl/dump-conntrack | grep icmpv6
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > > +])
> > > > > > > > +
> > > > > > > > +# We verify that SNAT indeed happened via 'dump-conntrack' command.
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > > > > > +sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > +icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > @@ -3861,11 +3868,9 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > > +# We verify that the connection is not tracked.
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > @@ -3875,11 +3880,9 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > > +# We verify that the connection is not tracked.
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > @@ -3889,14 +3892,13 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
> > > > > > > > +# We verify that the connection is not tracked.
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > +
> > > > > > > >   # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
> > > > > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > > >   [0], [dnl
> > > > > > > > @@ -3905,11 +3907,10 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > > >
> > > > > > > >   # Check conntrack entries.  First SNAT of 'foo1' address happens.
> > > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > >   icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > > @@ -3922,7 +3923,7 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
> > > > > > > >
> > > > > > > >   # Check conntrack entries.  First SNAT of 'foo2' address happens.
> > > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
> > > > > > > > +AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > >   icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > > @@ -4055,13 +4056,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > > -])
> > > > > > > > -
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > >   # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
> > > > > > > >   NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > > > @@ -4069,13 +4063,6 @@ NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > > -])
> > > > > > > > -
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > >   # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
> > > > > > > >   NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > > > > > @@ -4083,13 +4070,6 @@ NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
> > > > > > > >   3 packets transmitted, 3 received, 0% packet loss, time 0ms
> > > > > > > >   ])
> > > > > > > >
> > > > > > > > -# We verify that the connection is tracked but not NATted. This is due to the
> > > > > > > > -# unDNAT table in the egress router pipeline
> > > > > > > > -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
> > > > > > > > -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > > -])
> > > > > > > > -
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
> > > > > > > >   # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
> > > > > > > >   NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > > > > > @@ -4101,7 +4081,6 @@ NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
> > > > > > > >   # Then DNAT of 'bar1' address happens (listed first below).
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > > -icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > >   icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > >   icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > > @@ -6037,7 +6016,6 @@ NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
> > > > > > > >   AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
> > > > > > > >   sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> > > > > > > >   icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > > -icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
> > > > > > > >   ])
> > > > > > > >
> > > > > > > >   OVS_APP_EXIT_AND_WAIT([ovn-controller])
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > dev mailing list
> > > > > > > dev@openvswitch.org
> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > > > >
> > > > > > _______________________________________________
> > > > > > dev mailing list
> > > > > > dev@openvswitch.org
> > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev@openvswitch.org
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > >
> >
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
Frode Nordahl April 7, 2022, 5:40 p.m. UTC | #9 
On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
>
> On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > <frode.nordahl@canonical.com> wrote:
> > >
> > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org> wrote:
> > > >
> > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > <frode.nordahl@canonical.com> wrote:
> > > > >
> > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > <frode.nordahl@canonical.com> wrote:
> > > > > >
> > > > > > Hello Numan,
> > > > > >
> > > > > > This patch does unfortunately break gateway routers in some
> > > > > > circumstances, (but not all!), at least for the way OpenStack consumes
> > > > > > them.
> > > > > >
> > > > > > Will dig more and try to figure out what is going on, but sending this
> > > > > > e-mail proactively in case you have any ideas from the top of your
> > > > > > head or if anyone else has run into the same issue.
> > > > > >
> > > > > > The gateway appears to handle conntrack as it should:
> > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11 dst=194.169.254.178
> > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > sport=22 dport=60234 zone=52
> > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11 dst=194.169.254.178
> > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > dport=60234 zone=52
> > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11
> > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > >
> > > > > > However as you can see traffic going to the instance suddenly gets its
> > > > > > source address mangled/set to 0.0.0.0, and the connection can never
> > > > > > establish.
> > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB (Ethernet),
> > > > > > capture size 262144 bytes
> > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0, flags [DF],
> > > > > > proto TCP (6), length 60)
> > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > (correct), seq 2926328730, win 64240, options [mss 1460,sackOK,TS val
> > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
> > > > > > proto TCP (6), length 60)
> > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win 62230,
> > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr 870680827,nop,wscale
> > > > > > 7], length 0
> > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0, flags
> > > > > > [DF], proto TCP (6), length 93)
> > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > > > ecr 3490675961], length 41
> > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0, flags [DF],
> > > > > > proto TCP (6), length 52)
> > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > > > 3490676008 ecr 870680922], length 0
> > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0, flags [DF],
> > > > > > proto TCP (6), length 52)
> > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 870680922
> > > > > > ecr 3490675961], length 0
> > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0, flags [DF],
> > > > > > proto TCP (6), length 52)
> > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options [nop,nop,TS val
> > > > > > 3490676009 ecr 870680922], length 0
> > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af, ethertype IPv4
> > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0, flags
> > > > > > [DF], proto TCP (6), length 93)
> > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options [nop,nop,TS
> > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0, flags [DF],
> > > > > > proto TCP (6), length 52)
> > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d (correct),
> > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS val
> > > > > > 870680932 ecr 3490676018], length 0
> > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0, flags
> > > > > > [DF], proto TCP (6), length 1564)
> > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a (incorrect
> > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be, ethertype IPv4
> > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0, flags
> > > > > > [DF], proto TCP (6), length 116)
> > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70 (correct),
> > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val 870681111 ecr
> > > > > > 3490676018], length 64
> > > > >
> > > > > Turning hardware offload all the way off appears to resolve the issue,
> > > > > so this might as well be driver/firmware related. May I ask what
> > > > > card/driver/firmware you've been using this with?
> > > > >
> > > >
> > > > Hi Frode,
> > > >
> > > > This patch attempted to resolve this BZ -
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > This BZ is related to HWOL not working.
> > > >
> > > > I didn't test myself but I think other colleagues of mine tested with
> > > > NVidia/Mellanox CX5.
> > > > I don't think we saw the source address getting mangled in our testing.
> > >
> > > Thank you for sharing those details. We see the source mangling with
> > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > until we receive information from Nvidia/Mellanox as to what's going
> > > on from a driver/firmware perspective.
> > >
> > > > Please let me  know if you have any more questions.
> > >
> > > We did unfortunately run into another issue which appears to have its
> > > root in this patch. The symptomps are described in
> > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > >
> > > This rudimentary patch does appear to fix it, but it may require
> > > acquiring and passing in the `distributed` variable from the
> > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> >
> > Thanks for finding this issue.  I think we should use separate zones
> > for hairpin traffic.
> > And I thought I had addressed it.  To handle this hairpin issue,  the
> > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > and if this bit is set, then ct_snat() action is used instead of
> > ct_snat_in_czone().
> > Let me test it out and update here.
>
> I tested it locally and it seems to work for me.
>
> Can you please share your OVN dbs if possible ?  I've left a comment
> on the launchpad bug.  You can attach it there too if it's possible.

Sure thing, I attached them to the LP bug [0].  In this DB the active
gateway chassis is `deep-ferret.maas` and the instance on
`comic-perch.maas` is unable to have two ping sessions to itself using
non-distributed FIP 10.78.95.196.

I'll keep looking as well, thank you for your help so far!

0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
Frode Nordahl April 11, 2022, 4:36 p.m. UTC | #10 
tor. 7. apr. 2022, 19:40 skrev Frode Nordahl <frode.nordahl@canonical.com>:

> On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > > <frode.nordahl@canonical.com> wrote:
> > > >
> > > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org>
> wrote:
> > > > >
> > > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > > <frode.nordahl@canonical.com> wrote:
> > > > > >
> > > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > >
> > > > > > > Hello Numan,
> > > > > > >
> > > > > > > This patch does unfortunately break gateway routers in some
> > > > > > > circumstances, (but not all!), at least for the way OpenStack
> consumes
> > > > > > > them.
> > > > > > >
> > > > > > > Will dig more and try to figure out what is going on, but
> sending this
> > > > > > > e-mail proactively in case you have any ideas from the top of
> your
> > > > > > > head or if anyone else has run into the same issue.
> > > > > > >
> > > > > > > The gateway appears to handle conntrack as it should:
> > > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11
> dst=194.169.254.178
> > > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > > sport=22 dport=60234 zone=52
> > > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11
> dst=194.169.254.178
> > > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > > dport=60234 zone=52
> > > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34
> dst=10.11.2.11
> > > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > > >
> > > > > > > However as you can see traffic going to the instance suddenly
> gets its
> > > > > > > source address mangled/set to 0.0.0.0, and the connection can
> never
> > > > > > > establish.
> > > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB
> (Ethernet),
> > > > > > > capture size 262144 bytes
> > > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0,
> flags [DF],
> > > > > > > proto TCP (6), length 60)
> > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > > (correct), seq 2926328730, win 64240, options [mss
> 1460,sackOK,TS val
> > > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> ethertype IPv4
> > > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags
> [DF],
> > > > > > > proto TCP (6), length 60)
> > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win
> 62230,
> > > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr
> 870680827,nop,wscale
> > > > > > > 7], length 0
> > > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0,
> flags
> > > > > > > [DF], proto TCP (6), length 93)
> > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val
> 870680922
> > > > > > > ecr 3490675961], length 41
> > > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> ethertype IPv4
> > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0,
> flags [DF],
> > > > > > > proto TCP (6), length 52)
> > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options
> [nop,nop,TS val
> > > > > > > 3490676008 ecr 870680922], length 0
> > > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0,
> flags [DF],
> > > > > > > proto TCP (6), length 52)
> > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val
> 870680922
> > > > > > > ecr 3490675961], length 0
> > > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> ethertype IPv4
> > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0,
> flags [DF],
> > > > > > > proto TCP (6), length 52)
> > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options
> [nop,nop,TS val
> > > > > > > 3490676009 ecr 870680922], length 0
> > > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> ethertype IPv4
> > > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0,
> flags
> > > > > > > [DF], proto TCP (6), length 93)
> > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options
> [nop,nop,TS
> > > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0,
> flags [DF],
> > > > > > > proto TCP (6), length 52)
> > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d
> (correct),
> > > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS
> val
> > > > > > > 870680932 ecr 3490676018], length 0
> > > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0,
> flags
> > > > > > > [DF], proto TCP (6), length 1564)
> > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a
> (incorrect
> > > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> ethertype IPv4
> > > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0,
> flags
> > > > > > > [DF], proto TCP (6), length 116)
> > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70
> (correct),
> > > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val
> 870681111 ecr
> > > > > > > 3490676018], length 64
> > > > > >
> > > > > > Turning hardware offload all the way off appears to resolve the
> issue,
> > > > > > so this might as well be driver/firmware related. May I ask what
> > > > > > card/driver/firmware you've been using this with?
> > > > > >
> > > > >
> > > > > Hi Frode,
> > > > >
> > > > > This patch attempted to resolve this BZ -
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > > This BZ is related to HWOL not working.
> > > > >
> > > > > I didn't test myself but I think other colleagues of mine tested
> with
> > > > > NVidia/Mellanox CX5.
> > > > > I don't think we saw the source address getting mangled in our
> testing.
> > > >
> > > > Thank you for sharing those details. We see the source mangling with
> > > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > > until we receive information from Nvidia/Mellanox as to what's going
> > > > on from a driver/firmware perspective.
> > > >
> > > > > Please let me  know if you have any more questions.
> > > >
> > > > We did unfortunately run into another issue which appears to have its
> > > > root in this patch. The symptomps are described in
> > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > >
> > > > This rudimentary patch does appear to fix it, but it may require
> > > > acquiring and passing in the `distributed` variable from the
> > > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> > >
> > > Thanks for finding this issue.  I think we should use separate zones
> > > for hairpin traffic.
> > > And I thought I had addressed it.  To handle this hairpin issue,  the
> > > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > > and if this bit is set, then ct_snat() action is used instead of
> > > ct_snat_in_czone().
> > > Let me test it out and update here.
> >
> > I tested it locally and it seems to work for me.
> >
> > Can you please share your OVN dbs if possible ?  I've left a comment
> > on the launchpad bug.  You can attach it there too if it's possible.
>
> Sure thing, I attached them to the LP bug [0].  In this DB the active
> gateway chassis is `deep-ferret.maas` and the instance on
> `comic-perch.maas` is unable to have two ping sessions to itself using
> non-distributed FIP 10.78.95.196.
>
> I'll keep looking as well, thank you for your help so far!
>
> 0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
>
> Updated OVN to main and it unfortunately made no difference.

The combination of stateless on the NAT rule and the allow-related ACLs
does indeed look strange, but this is how OpenStack sets it up. Have not
looked into whether that makes sense or not yet.

To ensure we're looking at the same thing I made this modification to the
`DNAT LR hairpin IPv4` system test [2]

And executed it like this:

    sudo make check-kernel TESTSUITEFLAGS="337"

It fails consistently here. If I either revert [1] or remove the check for
the second ping from the test it succeeds.

2:
https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch


--

Frode Nordahl

> > Numan
> >
> > >
> > > Thanks
> > > Numan
> > >
> > >
> > > > this with you before proceeding with the fix:
> > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > index 2fb0a93c2..5fae010c0 100644
> > > > --- a/northd/northd.c
> > > > +++ b/northd/northd.c
> > > > @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct
> ovn_lb_vip *lb_vip,
> > > >                                      undnat_match_p, est_actions,
> > > >                                      &lb->nlb->header_);
> > > >          } else {
> > > > +            /* XXX do we need to check for distributed here? */
> > > >              ovn_lflow_add_with_hint(
> > > >                  lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> undnat_match_p,
> > > >                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> > > > *lflows, struct ovn_datapath *od,
> > > >                        is_v6 ? "6" : "4", nat->external_ip);
> > > >      } else {
> > > >          ds_put_format(actions,
> > > > -                      od->is_gw_router ? "ct_dnat;" :
> "ct_dnat_in_czone;");
> > > > +                      (od->is_gw_router ||
> > > > +                       (!distributed && od->n_l3dgw_ports)) ?
> > > > +                          "ct_dnat;" : "ct_dnat_in_czone;");
> > > >      }
> > > >
> > > >      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> > > > ovn_datapath *od, struct hmap *lflows,
> > > >       * not committed, it would produce ongoing datapath flows with
> the ct.new
> > > >       * flag set. Some NICs are unable to offload these flows.
> > > >       */
> > > > -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > +    /* XXX we probably need to get the distributed variable passed
> in here to
> > > > +     * XXX retain the proposed optimization */
> > > > +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > +        (od->nbr->n_nat || od->has_lb_vip)) {
> > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > >                        "ip", "flags.loopback = 1; ct_dnat;");
> > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > >
> > > > The environment in question is using distributed gateway ports by
> > > > associating multiple Gateway_Chassis with a Logical_Router_Port and
> at
> > > > the same time does not use distributed FIPs.
> > > >
> > > > --
> > > > Frode Nordahl
> > > >
> > > > > Numan
> > > > >
> > > > > > --
> > > > > > Frode Nordahl
> > > > > >
> > > > > > > --
> > > > > > > Frode Nordahl
> > > > > > >
> > > > > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <
> numans@ovn.org> wrote:
> > > > > > > >
> > > > > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <
> mmichels@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks for the update, Numan.
> > > > > > > > >
> > > > > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > > > > >
> > > > > > > > Thanks.  I applied both the patches to the main branch.
> > > > > > > >
> > > > > > > > Numan
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > > > > >
> > > > > > > > > > Make of use of these new actions for the distributed
> routers
> > > > > > > > > > for NAT.  These new actions ensure that both sNAT and
> dNAT
> > > > > > > > > > happens in the same zone.  This approach solves a couple
> of
> > > > > > > > > > problems:
> > > > > > > > > >
> > > > > > > > > >   - The datapath flows generated for external traffic
> which requires
> > > > > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely
> HWOL'able.
> > > > > > > > > >
> > > > > > > > > >   - Since there is only one zone, it would avoid
> multiple recirculations
> > > > > > > > > >     (improving the performance).
> > > > > > > > > >
> > > > > > > > > > If the packet needs to be both sNATted and dNATted (for
> hairpin traffic
> > > > > > > > > > with source and destination on the same chassis), then
> sNAT is done
> > > > > > > > > > in a separate zone.  To detect this scenario, this patch
> adds a few
> > > > > > > > > > extra logical flows.  For each dnat_and_snat entry prior
> to this patch
> > > > > > > > > > ovn-northd was generating 9 logical flows and with this
> patch it now
> > > > > > > > > > generates 12 logical flows.
> > > > > > > > > >
> > > > > > > > > > Similar approach can be taken for gateway routers.
> > > > > > > > > >
> > > > > > > > > > Reported-at:
> https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > v2 -> v3
> > > > > > > > > > ------
> > > > > > > > > >    * Addressed Mark's comments and updated the
> documentation.
> > > > > > > > > >
> > > > > > > > > > v1 -> v2
> > > > > > > > > > ------
> > > > > > > > > >    * Rebased and resolved conflicts.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > > > > >   tests/ovn-northd.at          | 575
> +++++++++++++++++++----------------
> > > > > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > > > > >
> > > > > > > > > > diff --git a/include/ovn/logical-fields.h
> b/include/ovn/logical-fields.h
> > > > > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > > > > >   };
> > > > > > > > > >
> > > > > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > > > > --- a/lib/logical-fields.c
> > > > > > > > > > +++ b/lib/logical-fields.c
> > > > > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash
> *symtab)
> > > > > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > > > > >       expr_symtab_add_subfield(symtab,
> "flags.skip_snat_for_lb", NULL,
> > > > > > > > > >                                flags_str);
> > > > > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > > > > +    expr_symtab_add_subfield(symtab,
> "flags.use_snat_zone", NULL,
> > > > > > > > > > +                             flags_str);
> > > > > > > > > >
> > > > > > > > > >       /* Connection tracking state. */
> > > > > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark",
> MFF_CT_MARK, NULL, false,
> > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > > > > --- a/northd/northd.c
> > > > > > > > > > +++ b/northd/northd.c
> > > > > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18,
> "lr_in_arp_request")  \
> > > > > > > > > >
>                \
> > > > > > > > > >       /* Logical router egress stages. */
>                \
> > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0,
> "lr_out_undnat")        \
> > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1,
> "lr_out_post_undnat")   \
> > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2,
> "lr_out_snat")          \
> > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3,
> "lr_out_egr_loop")      \
> > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4,
> "lr_out_delivery")
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,
>                      \
> > > > > > > > > > +                   "lr_out_chk_dnat_local")
>                       \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1,
> "lr_out_undnat")      \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2,
> "lr_out_post_undnat") \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3,
> "lr_out_snat")        \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4,
> "lr_out_post_snat")   \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5,
> "lr_out_egr_loop")    \
> > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6,
> "lr_out_delivery")
> > > > > > > > > >
> > > > > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE,
> TABLE, NAME)   \
> > > > > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE
>           \
> > > > > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > > > > >
> > > > > > > > > >   /* Register to store the eth address associated to a
> router port for packets
> > > > > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > > > > @@ -9568,9 +9572,10 @@
> build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > > > > >                                       undnat_match_p,
> est_actions,
> > > > > > > > > >                                       &lb->nlb->header_);
> > > > > > > > > >           } else {
> > > > > > > > > > -            ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > > > > -                                    undnat_match_p,
> "ct_dnat;",
> > > > > > > > > > -                                    &lb->nlb->header_);
> > > > > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> undnat_match_p,
> > > > > > > > > > +                od->is_gw_router ? "ct_dnat;" :
> "ct_dnat_in_czone;",
> > > > > > > > > > +                &lb->nlb->header_);
> > > > > > > > > >           }
> > > > > > > > > >           free(undnat_match_p);
> > > > > > > > > >   next:
> > > > > > > > > > @@ -9865,7 +9870,7 @@
> lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > >           uint16_t priority;
> > > > > > > > > >
> > > > > > > > > >           /* Priority of logical flows corresponding to
> exempted_ext_ips is
> > > > > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > > > > >            * For example, if we have following NAT rule
> and we associate
> > > > > > > > > >            * exempted external ips to it:
> > > > > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat
> 10.15.24.139 50.0.0.11"
> > > > > > > > > > @@ -9873,17 +9878,17 @@
> lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > >            * And now we associate exempted external ip
> address set to it.
> > > > > > > > > >            * Now corresponding to above rule we will
> have following logical
> > > > > > > > > >            * flows:
> > > > > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst
> == $exempt_range),
> > > > > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst
> == $exempt_range),
> > > > > > > > > >            *                             action=(next;)
> > > > > > > > > >            * lr_out_snat...priority=161, match=(..),
> action=(ct_snat(....);)
> > > > > > > > > >            *
> > > > > > > > > >            */
> > > > > > > > > >           if (is_src) {
> > > > > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > > > > -            priority = 100 + 1;
> > > > > > > > > > +            priority = 100 + 2;
> > > > > > > > > >           } else {
> > > > > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask +
> 1 + 128 + 1) */
> > > > > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > > > > >
> > > > > > > > > >               if (!od->is_gw_router) {
> > > > > > > > > >                   priority += 128;
> > > > > > > > > > @@ -12268,9 +12273,9 @@
> build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >           /* Traffic received on l3dgw_port is subject
> to NAT. */
> > > > > > > > > >           ds_clear(match);
> > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s &&
> inport == %s",
> > > > > > > > > > -                      is_v6 ? "6" : "4",
> nat->external_ip,
> > > > > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s &&
> inport == %s && "
> > > > > > > > > > +                      "flags.loopback == 0", is_v6 ?
> "6" : "4",
> > > > > > > > > > +                      nat->external_ip,
> od->l3dgw_ports[0]->json_key);
> > > > > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > >               /* Flows for NAT rules that are
> centralized are only
> > > > > > > > > >               * programmed on the gateway chassis. */
> > > > > > > > > > @@ -12282,12 +12287,31 @@
> build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >               ds_put_format(actions, "ip%s.dst=%s;
> next;",
> > > > > > > > > >                             is_v6 ? "6" : "4",
> nat->logical_ip);
> > > > > > > > > >           } else {
> > > > > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > > > > >           }
> > > > > > > > > >
> > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_UNSNAT,
> > > > > > > > > >                                   100, ds_cstr(match),
> ds_cstr(actions),
> > > > > > > > > >                                   &nat->header_);
> > > > > > > > > > +
> > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > +            ds_clear(match);
> > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s
> && inport == %s && "
> > > > > > > > > > +                          "flags.loopback == 1 &&
> flags.use_snat_zone == 1",
> > > > > > > > > > +                          is_v6 ? "6" : "4",
> nat->external_ip,
> > > > > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > +                /* Flows for NAT rules that are
> centralized are only
> > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > +                ds_put_format(match, " &&
> is_chassis_resident(%s)",
> > > > > > > > > > +
> od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > +            }
> > > > > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_UNSNAT,
> > > > > > > > > > +                                    100,
> ds_cstr(match), ds_cstr(actions),
> > > > > > > > > > +                                    &nat->header_);
> > > > > > > > > > +        }
> > > > > > > > > >       }
> > > > > > > > > >   }
> > > > > > > > > >
> > > > > > > > > > @@ -12364,7 +12388,7 @@
> build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >                   ds_put_format(actions, "ip%s.dst=%s;
> next;",
> > > > > > > > > >                                 is_v6 ? "6" : "4",
> nat->logical_ip);
> > > > > > > > > >               } else {
> > > > > > > > > > -                ds_put_format(actions, "ct_dnat(%s",
> nat->logical_ip);
> > > > > > > > > > +                ds_put_format(actions,
> "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > > > > >                       ds_put_format(actions, ",%s",
> nat->external_port_range);
> > > > > > > > > >                   }
> > > > > > > > > > @@ -12417,7 +12441,8 @@
> build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > > > >                         is_v6 ? "6" : "4",
> nat->external_ip);
> > > > > > > > > >       } else {
> > > > > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > > > > +        ds_put_format(actions,
> > > > > > > > > > +                      od->is_gw_router ? "ct_dnat;" :
> "ct_dnat_in_czone;");
> > > > > > > > > >       }
> > > > > > > > > >
> > > > > > > > > >       ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > > > @@ -12425,6 +12450,36 @@
> build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >                               &nat->header_);
> > > > > > > > > >   }
> > > > > > > > > >
> > > > > > > > > > +static void
> > > > > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows,
> struct ovn_datapath *od,
> > > > > > > > > > +                                const struct nbrec_nat
> *nat, struct ds *match,
> > > > > > > > > > +                                struct ds *actions,
> bool distributed,
> > > > > > > > > > +                                bool is_v6)
> > > > > > > > > > +{
> > > > > > > > > > +    /* Note that this only applies for NAT on a
> distributed router.
> > > > > > > > > > +     */
> > > > > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > > > > +        return;
> > > > > > > > > > +    }
> > > > > > > > > > +
> > > > > > > > > > +    ds_clear(match);
> > > > > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > > > > +    if (distributed) {
> > > > > > > > > > +        ds_put_format(match,
> "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > > > > +    } else {
> > > > > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > > > > +
> od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > +    }
> > > > > > > > > > +
> > > > > > > > > > +    ds_clear(actions);
> > > > > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1;
> next;");
> > > > > > > > > > +
> > > > > > > > > > +    ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > > > > +                            50, ds_cstr(match),
> ds_cstr(actions),
> > > > > > > > > > +                            &nat->header_);
> > > > > > > > > > +}
> > > > > > > > > > +
> > > > > > > > > >   static void
> > > > > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows,
> struct ovn_datapath *od,
> > > > > > > > > >                               const struct nbrec_nat
> *nat, struct ds *match,
> > > > > > > > > > @@ -12478,16 +12533,19 @@
> build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >           ds_put_format(match, "ip && ip%s.src == %s &&
> outport == %s",
> > > > > > > > > >                         is_v6 ? "6" : "4",
> nat->logical_ip,
> > > > > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > -            /* Flows for NAT rules that are centralized
> are only
> > > > > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > > > > -            priority += 128;
> > > > > > > > > > -            ds_put_format(match, " &&
> is_chassis_resident(%s)",
> > > > > > > > > > -
> od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > -        } else if (distributed) {
> > > > > > > > > > -            priority += 128;
> > > > > > > > > > -            ds_put_format(match, " &&
> is_chassis_resident(\"%s\")",
> > > > > > > > > > -                          nat->logical_port);
> > > > > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > > > > +            if (distributed) {
> > > > > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > > > > +                priority += 128;
> > > > > > > > > > +                ds_put_format(match, " &&
> is_chassis_resident(\"%s\")",
> > > > > > > > > > +                              nat->logical_port);
> > > > > > > > > > +            } else {
> > > > > > > > > > +                /* Flows for NAT rules that are
> centralized are only
> > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > +                priority += 128;
> > > > > > > > > > +                ds_put_format(match, " &&
> is_chassis_resident(%s)",
> > > > > > > > > > +
> od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > +            }
> > > > > > > > > >           }
> > > > > > > > > >           ds_clear(actions);
> > > > > > > > > >
> > > > > > > > > > @@ -12505,7 +12563,7 @@
> build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >               ds_put_format(actions, "ip%s.src=%s;
> next;",
> > > > > > > > > >                             is_v6 ? "6" : "4",
> nat->external_ip);
> > > > > > > > > >           } else {
> > > > > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > > > > +            ds_put_format(actions,
> "ct_snat_in_czone(%s",
> > > > > > > > > >                           nat->external_ip);
> > > > > > > > > >               if (nat->external_port_range[0]) {
> > > > > > > > > >                   ds_put_format(actions, ",%s",
> nat->external_port_range);
> > > > > > > > > > @@ -12519,6 +12577,24 @@
> build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_OUT_SNAT,
> > > > > > > > > >                                   priority,
> ds_cstr(match),
> > > > > > > > > >                                   ds_cstr(actions),
> &nat->header_);
> > > > > > > > > > +
> > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > +            ds_put_cstr(match, " &&
> "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > +            if (distributed) {
> > > > > > > > > > +                ds_put_format(actions, "eth.src =
> "ETH_ADDR_FMT"; ",
> > > > > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > > > > +            }
> > > > > > > > > > +            ds_put_format(actions,
> REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > > > > +                          nat->external_ip);
> > > > > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > > > > +                ds_put_format(actions, ",%s",
> nat->external_port_range);
> > > > > > > > > > +            }
> > > > > > > > > > +            ds_put_format(actions, ");");
> > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_OUT_SNAT,
> > > > > > > > > > +                                    priority + 1,
> ds_cstr(match),
> > > > > > > > > > +                                    ds_cstr(actions),
> &nat->header_);
> > > > > > > > > > +        }
> > > > > > > > > >       }
> > > > > > > > > >   }
> > > > > > > > > >
> > > > > > > > > > @@ -12749,10 +12825,13 @@
> build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> *lflows,
> > > > > > > > > >       /* Packets are allowed by default. */
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0,
> "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0,
> "1", "next;");
> > > > > > > > > > +    ovn_lflow_add(lflows, od,
> S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0,
> "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0,
> "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0,
> "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od,
> S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT,
> 0, "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP,
> 0, "1", "next;");
> > > > > > > > > >       ovn_lflow_add(lflows, od,
> S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > > > > >
> > > > > > > > > > @@ -12765,8 +12844,7 @@
> build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> *lflows,
> > > > > > > > > >        * not committed, it would produce ongoing
> datapath flows with the ct.new
> > > > > > > > > >        * flag set. Some NICs are unable to offload these
> flows.
> > > > > > > > > >        */
> > > > > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat ||
> od->has_lb_vip)) {
> > > > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT,
> 50,
> > > > > > > > > >                         "ip", "flags.loopback = 1;
> ct_dnat;");
> > > > > > > > > >           ovn_lflow_add(lflows, od,
> S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > > > > @@ -12839,6 +12917,10 @@
> build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> *lflows,
> > > > > > > > > >               }
> > > > > > > > > >           }
> > > > > > > > > >
> > > > > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od,
> nat, match, actions,
> > > > > > > > > > +                                        distributed,
> is_v6);
> > > > > > > > > > +
> > > > > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat,
> match, actions, distributed,
> > > > > > > > > >                                         mac, is_v6);
> > > > > > > > > > @@ -12912,7 +12994,8 @@
> build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> *lflows,
> > > > > > > > > >                             "clone { ct_clear; "
> > > > > > > > > >                             "inport = outport; outport =
> \"\"; "
> > > > > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > > > > -                          "flags = 0; flags.loopback =
> 1; ");
> > > > > > > > > > +                          "flags = 0; flags.loopback =
> 1; "
> > > > > > > > > > +                          "flags.use_snat_zone =
> "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > > > > >                   ds_put_format(actions, "reg%d = 0; ",
> j);
> > > > > > > > > >               }
> > > > > > > > > > diff --git a/northd/ovn-northd.8.xml
> b/northd/ovn-northd.8.xml
> > > > > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > > > > >           <p>
> > > > > > > > > >             For each configuration in the OVN Northbound
> database, that asks
> > > > > > > > > >             to change the source IP address of a packet
> from <var>A</var> to
> > > > > > > > > > -          <var>B</var>, a priority-100 flow matches
> <code>ip &amp;&amp;
> > > > > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport ==
> <var>GW</var></code> or
> > > > > > > > > > -          <code>ip &amp;&amp;
> > > > > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport ==
> <var>GW</var></code>
> > > > > > > > > > -          where <var>GW</var> is the logical router
> gateway port, with an
> > > > > > > > > > -          action <code>ct_snat;</code>. If the NAT rule
> is of type
> > > > > > > > > > -          dnat_and_snat and has
> <code>stateless=true</code> in the
> > > > > > > > > > -          options, then the action would be
> <code>ip4/6.dst=
> > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > +          <var>B</var>, two priority-100 flows are
> added.
> > > > > > > > > >           </p>
> > > > > > > > > >
> > > > > > > > > >           <p>
> > > > > > > > > >             If the NAT rule cannot be handled in a
> distributed manner, then
> > > > > > > > > > -          the priority-100 flow above is only
> programmed on the
> > > > > > > > > > +          the below priority-100 flows are only
> programmed on the
> > > > > > > > > >             gateway chassis.
> > > > > > > > > >           </p>
> > > > > > > > > >
> > > > > > > > > > +        <ul>
> > > > > > > > > > +          <li>
> > > > > > > > > > +            <p>
> > > > > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> == <var>GW</var>
> > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> == <var>GW</var>
> > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > > > > +              where <var>GW</var> is the logical router
> gateway port, with an
> > > > > > > > > > +              action <code>ct_snat_in_czone;</code> to
> unSNAT in the common
> > > > > > > > > > +              zone.  If the NAT rule is of type
> dnat_and_snat and has
> > > > > > > > > > +              <code>stateless=true</code> in the
> options, then the action
> > > > > > > > > > +              would be
> <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > +            </p>
> > > > > > > > > > +
> > > > > > > > > > +            <p>
> > > > > > > > > > +              If the NAT entry is of type
> <code>snat</code>, then there is an
> > > > > > > > > > +              additional match
> <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> chassis resident port of
> > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > +            </p>
> > > > > > > > > > +          </li>
> > > > > > > > > > +
> > > > > > > > > > +          <li>
> > > > > > > > > > +            <p>
> > > > > > > > > > +              The second flow matches <code>ip
> &amp;&amp;
> > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> == <var>GW</var>
> > > > > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> == <var>GW</var>
> > > > > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > > > > +              where <var>GW</var> is the logical router
> gateway port, with an
> > > > > > > > > > +              action <code>ct_snat;</code> to unSNAT in
> the snat zone. If the
> > > > > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > > > > +              <code>stateless=true</code> in the
> options, then the action
> > > > > > > > > > +              would be
> <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > +            </p>
> > > > > > > > > > +
> > > > > > > > > > +            <p>
> > > > > > > > > > +              If the NAT entry is of type
> <code>snat</code>, then there is an
> > > > > > > > > > +              additional match
> <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> chassis resident port of
> > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > +            </p>
> > > > > > > > > > +          </li>
> > > > > > > > > > +        </ul>
> > > > > > > > > > +
> > > > > > > > > >           <p>
> > > > > > > > > >             A priority-0 logical flow with match
> <code>1</code> has actions
> > > > > > > > > >             <code>next;</code>.
> > > > > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > > > > >         </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > > > > +
> > > > > > > > > > +    <p>
> > > > > > > > > > +      This table checks if the packet needs to be
> DNATed in the router ingress
> > > > > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed
> and looped back
> > > > > > > > > > +      to the ingress pipeline.  This check is done only
> for routers configured
> > > > > > > > > > +      with distributed gateway ports and NAT entries.
> This check is done
> > > > > > > > > > +      so that SNAT and DNAT is done in different zones
> instead of a common
> > > > > > > > > > +      zone.
> > > > > > > > > > +    </p>
> > > > > > > > > > +
> > > > > > > > > > +    <ul>
> > > > > > > > > > +      <li>
> > > > > > > > > > +        <p>
> > > > > > > > > > +          For each NAT rule in the OVN Northbound
> database on a
> > > > > > > > > > +          distributed router, a priority-50 logical
> flow with match
> > > > > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > > > > +          is_chassis_resident(<var>P</var>)</code>,
> where <var>E</var> is the
> > > > > > > > > > +          external IP address specified in the NAT
> rule, <var>GW</var>
> > > > > > > > > > +          is the logical router distributed gateway
> port. For dnat_and_snat
> > > > > > > > > > +          NAT rule, <var>P</var> is the logical port
> specified in the NAT rule.
> > > > > > > > > > +          If <ref column="logical_port"
> > > > > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table
> is NOT set, then
> > > > > > > > > > +          <var>P</var> is the <code>chassisredirect
> port</code> of
> > > > > > > > > > +          <var>GW</var> with the actions:
> > > > > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next;
> </code>
> > > > > > > > > > +        </p>
> > > > > > > > > > +      </li>
> > > > > > > > > > +
> > > > > > > > > > +      <li>
> > > > > > > > > > +        A priority-0 logical flow with match
> <code>1</code> has actions
> > > > > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > > > > +      </li>
> > > > > > > > > > +    </ul>
> > > > > > > > > > +
> > > > > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > > > > >
> > > > > > > > > >       <p>
> > > > > > > > > >         This is for already established connections'
> reverse traffic.
> > > > > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > > > > >         is unDNATed here.
> > > > > > > > > >       </p>
> > > > > > > > > >
> > > > > > > > > > +    <ul>
> > > > > > > > > > +      <li>
> > > > > > > > > > +        A priority-0 logical flow with match
> <code>1</code> has actions
> > > > > > > > > > +        <code>next;</code>.
> > > > > > > > > > +      </li>
> > > > > > > > > > +    </ul>
> > > > > > > > > > +
> > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > > > > +
> > > > > > > > > > +    <ul>
> > > > > > > > > > +      <li>
> > > > > > > > > > +        For all IP packets, a priority-50 flow with an
> action
> > > > > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > +      </li>
> > > > > > > > > > +    </ul>
> > > > > > > > > > +
> > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed
> Routers</h3>
> > > > > > > > > >       <ul>
> > > > > > > > > >         <li>
> > > > > > > > > >           <p>
> > > > > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > > > > >             gateway chassis that matches
> > > > > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var>
> &amp;&amp;
> > > > > > > > > >             outport == <var>GW</var></code>, where
> <var>GW</var> is the logical
> > > > > > > > > > -          router gateway port with an action
> <code>ct_dnat;</code>. If the
> > > > > > > > > > -          backend IPv4 address <var>B</var> is also
> configured with L4 port
> > > > > > > > > > -          <var>PORT</var> of protocol <var>P</var>,
> then the
> > > > > > > > > > +          router gateway port with an action
> <code>ct_dnat_in_czone;</code>.
> > > > > > > > > > +          If the backend IPv4 address <var>B</var> is
> also configured with
> > > > > > > > > > +          L4 port <var>PORT</var> of protocol
> <var>P</var>, then the
> > > > > > > > > >             match also includes <code>P.src</code> ==
> <var>PORT</var>.  These
> > > > > > > > > >             flows are not added for load balancers with
> IPv6 <var>VIPs</var>.
> > > > > > > > > >           </p>
> > > > > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > > > > >             matches <code>ip &amp;&amp; ip4.src ==
> <var>B</var>
> > > > > > > > > >             &amp;&amp; outport == <var>GW</var></code>,
> where <var>GW</var>
> > > > > > > > > >             is the logical router gateway port, with an
> action
> > > > > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of
> type
> > > > > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT
> rule is of type
> > > > > > > > > >             dnat_and_snat and has
> <code>stateless=true</code> in the
> > > > > > > > > >             options, then the action would be
> <code>ip4/6.src=
> > > > > > > > > >             (<var>B</var>)</code>.
> > > > > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > > > > >           <p>
> > > > > > > > > >             If the NAT rule cannot be handled in a
> distributed manner, then
> > > > > > > > > >             the priority-100 flow above is only
> programmed on the
> > > > > > > > > > -          gateway chassis.
> > > > > > > > > > +          gateway chassis with the action
> <code>ct_dnat_in_czone</code>.
> > > > > > > > > >           </p>
> > > > > > > > > >
> > > > > > > > > >           <p>
> > > > > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > > > > >           </p>
> > > > > > > > > >         </li>
> > > > > > > > > >
> > > > > > > > > > -      <li>
> > > > > > > > > > -        For all IP packets, a priority-50 flow with an
> action
> > > > > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > -      </li>
> > > > > > > > > > -
> > > > > > > > > > -      <li>
> > > > > > > > > > -        A priority-0 logical flow with match
> <code>1</code> has actions
> > > > > > > > > > -        <code>next;</code>.
> > > > > > > > > > -      </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > > > > >
> > > > > > > > > >       <p>
> > > > > > > > > >         <ul>
> > > > > > > > > >           <li>
> > > > > > > > > >             A priority-50 logical flow is added that
> commits any untracked flows
> > > > > > > > > > -          from the previous table
> <code>lr_out_undnat</code>. This flow
> > > > > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code>
> with action
> > > > > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > > > > +          from the previous table
> <code>lr_out_undnat</code> for Gateway
> > > > > > > > > > +          routers.  This flow matches on <code>ct.new
> &amp;&amp; ip</code>
> > > > > > > > > > +          with action <code>ct_commit { } ; next;
> </code>.
> > > > > > > > > >           </li>
> > > > > > > > > >
> > > > > > > > > >           <li>
> > > > > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > > > > >         </ul>
> > > > > > > > > >       </p>
> > > > > > > > > >
> > > > > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > > > > >
> > > > > > > > > >       <p>
> > > > > > > > > >         Packets that are configured to be SNATed get
> their source IP address
> > > > > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > > > > >         </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > > > > >
> > > > > > > > > >       <ul>
> > > > > > > > > >         <li>
> > > > > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > > > > >         </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > > > > >
> > > > > > > > > >       <ul>
> > > > > > > > > >         <li>
> > > > > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > > > > >             For each configuration in the OVN Northbound
> database, that asks
> > > > > > > > > >             to change the source IP address of a packet
> from an IP address of
> > > > > > > > > >             <var>A</var> or to change the source IP
> address of a packet that
> > > > > > > > > > -          belongs to network <var>A</var> to
> <var>B</var>, a flow matches
> > > > > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var>
> &amp;&amp;
> > > > > > > > > > -          outport == <var>GW</var></code>, where
> <var>GW</var> is the
> > > > > > > > > > -          logical router gateway port, with an action
> > > > > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The
> priority of the flow
> > > > > > > > > > -          is calculated based on the mask of
> <var>A</var>, with matches
> > > > > > > > > > -          having larger masks getting higher
> priorities. If the NAT rule
> > > > > > > > > > -          is of type dnat_and_snat and has
> <code>stateless=true</code>
> > > > > > > > > > -          in the options, then the action would be
> <code>ip4/6.src=
> > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > +          belongs to network <var>A</var> to
> <var>B</var>, two flows are
> > > > > > > > > > +          added.  The priority <var>P</var> of these
> flows are calculated
> > > > > > > > > > +          based on the mask of <var>A</var>, with
> matches having larger
> > > > > > > > > > +          masks getting higher priorities.
> > > > > > > > > >           </p>
> > > > > > > > > >
> > > > > > > > > >           <p>
> > > > > > > > > >             If the NAT rule cannot be handled in a
> distributed manner, then
> > > > > > > > > > -          the flow above is only programmed on the
> > > > > > > > > > -          gateway chassis increasing flow priority by
> 128 in
> > > > > > > > > > -          order to be run first
> > > > > > > > > > +          the below flows are only programmed on the
> gateway chassis increasing
> > > > > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > > > > >           </p>
> > > > > > > > > >
> > > > > > > > > > +        <ul>
> > > > > > > > > > +          <li>
> > > > > > > > > > +            The first flow is added with the calculated
> priority <var>P</var>
> > > > > > > > > > +            and match <code>ip &amp;&amp; ip4.src ==
> <var>A</var> &amp;&amp;
> > > > > > > > > > +            outport == <var>GW</var></code>, where
> <var>GW</var> is the
> > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > +
> <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > > > > +            common zone.  If the NAT rule is of type
> dnat_and_snat and has
> > > > > > > > > > +            <code>stateless=true</code> in the options,
> then the action
> > > > > > > > > > +            would be
> <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > +          </li>
> > > > > > > > > > +
> > > > > > > > > > +          <li>
> > > > > > > > > > +            The second flow is added with the
> calculated priority
> > > > > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var>
> &amp;&amp;
> > > > > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where
> <var>GW</var> is the
> > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT
> in the snat zone.
> > > > > > > > > > +            If the NAT rule is of type dnat_and_snat
> and has
> > > > > > > > > > +            <code>stateless=true</code> in the options,
> then the action would
> > > > > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > +          </li>
> > > > > > > > > > +        </ul>
> > > > > > > > > > +
> > > > > > > > > >           <p>
> > > > > > > > > >             If the NAT rule can be handled in a
> distributed manner, then
> > > > > > > > > > -          there is an additional action
> > > > > > > > > > +          there is an additional action (for both the
> flows)
> > > > > > > > > >             <code>eth.src = <var>EA</var>;</code>, where
> <var>EA</var>
> > > > > > > > > >             is the ethernet address associated with the
> IP address
> > > > > > > > > >             <var>A</var> in the NAT rule.  This allows
> upstream MAC
> > > > > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > > > > >
> > > > > > > > > >           <p>
> > > > > > > > > >             If the NAT rule has
> <code>exempted_ext_ips</code> set, then
> > > > > > > > > > -          there is an additional flow configured at the
> priority + 1 of
> > > > > > > > > > +          there is an additional flow configured at the
> priority
> > > > > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > > > > >             corresponding NAT rule. The flow matches if
> destination ip
> > > > > > > > > >             is an <code>exempted_ext_ip</code> and the
> action is <code>next;
> > > > > > > > > >             </code>. This flow is used to bypass the
> ct_snat action for a flow
> > > > > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > > > > >         </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > > > > >
> > > > > > > > > >       <p>
> > > > > > > > > >         For distributed logical routers where one of the
> logical router
> > > > > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > > > > >       outport = "";
> > > > > > > > > >       flags = 0;
> > > > > > > > > >       flags.loopback = 1;
> > > > > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > > > > >       reg0 = 0;
> > > > > > > > > >       reg1 = 0;
> > > > > > > > > >       ...
> > > > > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > > > > >         </li>
> > > > > > > > > >       </ul>
> > > > > > > > > >
> > > > > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > > > > >
> > > > > > > > > >       <p>
> > > > > > > > > >         Packets that reach this table are ready for
> delivery.  It contains:
> > > > > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > > > > --- a/tests/ovn-northd.at
> > > > > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > > > > >   echo
> > > > > > > > > >   echo "IPv4: stateful"
> > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> 172.16.1.1 50.0.0.11
> > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > >
> > > > > > > > > >   echo
> > > > > > > > > >   echo "IPv4: stateless"
> > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > >
> > > > > > > > > >   echo
> > > > > > > > > >   echo "IPv6: stateful"
> > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> fd01::1 fd11::2
> > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > > > > >
> > > > > > > > > >   echo
> > > > > > > > > >   echo "IPv6: stateless"
> > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> dnat_and_snat fd01::1 fd11::2
> > > > > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > > > > >
> > > > > > > > > >   AT_CLEANUP
> > > > > > > > > >   ])
> > > > > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat
> 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > >
> > > > > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c
> 'ct_dnat.*3000' sbflows],
> > > > > > > > > > -  [0], [1
> > > > > > > > > > +  [0], [2
> > > > > > > > > >   1
> > > > > > > > > >   ])
> > > > > > > > > >
> > > > > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1
> dnat_and_snat  172.16.1.1
> > > > > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat
> 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > >
> > > > > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c
> 'ct_dnat.*3000' sbflows2],
> > > > > > > > > > -  [1], [1
> > > > > > > > > > +  [1], [2
> > > > > > > > > >   0
> > > > > > > > > >   ])
> > > > > > > > > >
> > > > > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat
> 172.16.1.1
> > > > > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add
> R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > > > > >
> > > > > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000'
> sbflows3], [1])
> > > > > > > > > >
> > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > > > > >
> > > > > > > > > > -AT_CHECK([
> > > > > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > > > > -  grep lr_out_snat drflows | grep "ip4.src ==
> 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > > > > -  grep lr_out_snat crflows | grep "ip4.src ==
> 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > > > > -3
> > > > > > > > > > -3
> > > > > > > > > > -1
> > > > > > > > > > -1
> > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed
> 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> match=(1), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> match=(nd_ns), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range),
> action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]]
> == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > > > +])
> > > > > > > > > > +
> > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed
> 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> match=(1), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> match=(nd_ns), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range),
> action=(ct_snat(172.16.1.1);)
> > > > > > > > > >   ])
> > > > > > > > > >
> > > > > > > > > > +
> > > > > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > > > > >
> > > > > > > > > > -AT_CHECK([
> > > > > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> 50.0.0.11" | grep -c "priority=161"
> > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > > > > -4
> > > > > > > > > > -4
> > > > > > > > > > -1
> > > > > > > > > > -1
> > > > > > > > > > -1
> > > > > > > > > > -1
> > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed
> 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> match=(1), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> match=(nd_ns), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0;
> ct_snat(172.16.1.1);)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=163  ,
> match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range),
> action=(next;)
> > > > > > > > > > +])
> > > > > > > > > > +
> > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed
> 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> match=(1), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> match=(nd_ns), action=(next;)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > > > > +  table=??(lr_out_snat        ), priority=35   ,
> match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range),
> action=(next;)
> > > > > > > > > >   ])
> > > > > > > > > >
> > > > > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > > > > @@ -1059,2
Numan Siddique April 12, 2022, 3:01 p.m. UTC | #11 
On Mon, Apr 11, 2022 at 12:37 PM Frode Nordahl
<frode.nordahl@canonical.com> wrote:
>
> tor. 7. apr. 2022, 19:40 skrev Frode Nordahl <frode.nordahl@canonical.com>:
>
> > On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> > > >
> > > > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > > > <frode.nordahl@canonical.com> wrote:
> > > > >
> > > > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org>
> > wrote:
> > > > > >
> > > > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > >
> > > > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > >
> > > > > > > > Hello Numan,
> > > > > > > >
> > > > > > > > This patch does unfortunately break gateway routers in some
> > > > > > > > circumstances, (but not all!), at least for the way OpenStack
> > consumes
> > > > > > > > them.
> > > > > > > >
> > > > > > > > Will dig more and try to figure out what is going on, but
> > sending this
> > > > > > > > e-mail proactively in case you have any ideas from the top of
> > your
> > > > > > > > head or if anyone else has run into the same issue.
> > > > > > > >
> > > > > > > > The gateway appears to handle conntrack as it should:
> > > > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11
> > dst=194.169.254.178
> > > > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > > > sport=22 dport=60234 zone=52
> > > > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11
> > dst=194.169.254.178
> > > > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > > > dport=60234 zone=52
> > > > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34
> > dst=10.11.2.11
> > > > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > > > >
> > > > > > > > However as you can see traffic going to the instance suddenly
> > gets its
> > > > > > > > source address mangled/set to 0.0.0.0, and the connection can
> > never
> > > > > > > > establish.
> > > > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB
> > (Ethernet),
> > > > > > > > capture size 262144 bytes
> > > > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0,
> > flags [DF],
> > > > > > > > proto TCP (6), length 60)
> > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > > > (correct), seq 2926328730, win 64240, options [mss
> > 1460,sackOK,TS val
> > > > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > ethertype IPv4
> > > > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags
> > [DF],
> > > > > > > > proto TCP (6), length 60)
> > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win
> > 62230,
> > > > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr
> > 870680827,nop,wscale
> > > > > > > > 7], length 0
> > > > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0,
> > flags
> > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val
> > 870680922
> > > > > > > > ecr 3490675961], length 41
> > > > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > ethertype IPv4
> > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0,
> > flags [DF],
> > > > > > > > proto TCP (6), length 52)
> > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options
> > [nop,nop,TS val
> > > > > > > > 3490676008 ecr 870680922], length 0
> > > > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0,
> > flags [DF],
> > > > > > > > proto TCP (6), length 52)
> > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val
> > 870680922
> > > > > > > > ecr 3490675961], length 0
> > > > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > ethertype IPv4
> > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0,
> > flags [DF],
> > > > > > > > proto TCP (6), length 52)
> > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options
> > [nop,nop,TS val
> > > > > > > > 3490676009 ecr 870680922], length 0
> > > > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > ethertype IPv4
> > > > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0,
> > flags
> > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options
> > [nop,nop,TS
> > > > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0,
> > flags [DF],
> > > > > > > > proto TCP (6), length 52)
> > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d
> > (correct),
> > > > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS
> > val
> > > > > > > > 870680932 ecr 3490676018], length 0
> > > > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0,
> > flags
> > > > > > > > [DF], proto TCP (6), length 1564)
> > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a
> > (incorrect
> > > > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > ethertype IPv4
> > > > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0,
> > flags
> > > > > > > > [DF], proto TCP (6), length 116)
> > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70
> > (correct),
> > > > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val
> > 870681111 ecr
> > > > > > > > 3490676018], length 64
> > > > > > >
> > > > > > > Turning hardware offload all the way off appears to resolve the
> > issue,
> > > > > > > so this might as well be driver/firmware related. May I ask what
> > > > > > > card/driver/firmware you've been using this with?
> > > > > > >
> > > > > >
> > > > > > Hi Frode,
> > > > > >
> > > > > > This patch attempted to resolve this BZ -
> > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > > > This BZ is related to HWOL not working.
> > > > > >
> > > > > > I didn't test myself but I think other colleagues of mine tested
> > with
> > > > > > NVidia/Mellanox CX5.
> > > > > > I don't think we saw the source address getting mangled in our
> > testing.
> > > > >
> > > > > Thank you for sharing those details. We see the source mangling with
> > > > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > > > until we receive information from Nvidia/Mellanox as to what's going
> > > > > on from a driver/firmware perspective.
> > > > >
> > > > > > Please let me  know if you have any more questions.
> > > > >
> > > > > We did unfortunately run into another issue which appears to have its
> > > > > root in this patch. The symptomps are described in
> > > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > > >
> > > > > This rudimentary patch does appear to fix it, but it may require
> > > > > acquiring and passing in the `distributed` variable from the
> > > > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> > > >
> > > > Thanks for finding this issue.  I think we should use separate zones
> > > > for hairpin traffic.
> > > > And I thought I had addressed it.  To handle this hairpin issue,  the
> > > > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > > > and if this bit is set, then ct_snat() action is used instead of
> > > > ct_snat_in_czone().
> > > > Let me test it out and update here.
> > >
> > > I tested it locally and it seems to work for me.
> > >
> > > Can you please share your OVN dbs if possible ?  I've left a comment
> > > on the launchpad bug.  You can attach it there too if it's possible.
> >
> > Sure thing, I attached them to the LP bug [0].  In this DB the active
> > gateway chassis is `deep-ferret.maas` and the instance on
> > `comic-perch.maas` is unable to have two ping sessions to itself using
> > non-distributed FIP 10.78.95.196.
> >
> > I'll keep looking as well, thank you for your help so far!
> >
> > 0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> >
> > Updated OVN to main and it unfortunately made no difference.
>
> The combination of stateless on the NAT rule and the allow-related ACLs
> does indeed look strange, but this is how OpenStack sets it up. Have not
> looked into whether that makes sense or not yet.
>
> To ensure we're looking at the same thing I made this modification to the
> `DNAT LR hairpin IPv4` system test [2]
>
> And executed it like this:
>
>     sudo make check-kernel TESTSUITEFLAGS="337"
>
> It fails consistently here. If I either revert [1] or remove the check for
> the second ping from the test it succeeds.
>
> 2:
> https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch

Thanks for this reproducer.  The test case is failing for me too.
I'll debug further and get back to you.

Numan

>
>
> --
>
> Frode Nordahl
>
> > > Numan
> > >
> > > >
> > > > Thanks
> > > > Numan
> > > >
> > > >
> > > > > this with you before proceeding with the fix:
> > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > index 2fb0a93c2..5fae010c0 100644
> > > > > --- a/northd/northd.c
> > > > > +++ b/northd/northd.c
> > > > > @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct
> > ovn_lb_vip *lb_vip,
> > > > >                                      undnat_match_p, est_actions,
> > > > >                                      &lb->nlb->header_);
> > > > >          } else {
> > > > > +            /* XXX do we need to check for distributed here? */
> > > > >              ovn_lflow_add_with_hint(
> > > > >                  lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > undnat_match_p,
> > > > >                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> > > > > *lflows, struct ovn_datapath *od,
> > > > >                        is_v6 ? "6" : "4", nat->external_ip);
> > > > >      } else {
> > > > >          ds_put_format(actions,
> > > > > -                      od->is_gw_router ? "ct_dnat;" :
> > "ct_dnat_in_czone;");
> > > > > +                      (od->is_gw_router ||
> > > > > +                       (!distributed && od->n_l3dgw_ports)) ?
> > > > > +                          "ct_dnat;" : "ct_dnat_in_czone;");
> > > > >      }
> > > > >
> > > > >      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> > > > > ovn_datapath *od, struct hmap *lflows,
> > > > >       * not committed, it would produce ongoing datapath flows with
> > the ct.new
> > > > >       * flag set. Some NICs are unable to offload these flows.
> > > > >       */
> > > > > -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > +    /* XXX we probably need to get the distributed variable passed
> > in here to
> > > > > +     * XXX retain the proposed optimization */
> > > > > +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > +        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > >                        "ip", "flags.loopback = 1; ct_dnat;");
> > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > >
> > > > > The environment in question is using distributed gateway ports by
> > > > > associating multiple Gateway_Chassis with a Logical_Router_Port and
> > at
> > > > > the same time does not use distributed FIPs.
> > > > >
> > > > > --
> > > > > Frode Nordahl
> > > > >
> > > > > > Numan
> > > > > >
> > > > > > > --
> > > > > > > Frode Nordahl
> > > > > > >
> > > > > > > > --
> > > > > > > > Frode Nordahl
> > > > > > > >
> > > > > > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <
> > numans@ovn.org> wrote:
> > > > > > > > >
> > > > > > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <
> > mmichels@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks for the update, Numan.
> > > > > > > > > >
> > > > > > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > > > > > >
> > > > > > > > > Thanks.  I applied both the patches to the main branch.
> > > > > > > > >
> > > > > > > > > Numan
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > > > > > >
> > > > > > > > > > > Make of use of these new actions for the distributed
> > routers
> > > > > > > > > > > for NAT.  These new actions ensure that both sNAT and
> > dNAT
> > > > > > > > > > > happens in the same zone.  This approach solves a couple
> > of
> > > > > > > > > > > problems:
> > > > > > > > > > >
> > > > > > > > > > >   - The datapath flows generated for external traffic
> > which requires
> > > > > > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely
> > HWOL'able.
> > > > > > > > > > >
> > > > > > > > > > >   - Since there is only one zone, it would avoid
> > multiple recirculations
> > > > > > > > > > >     (improving the performance).
> > > > > > > > > > >
> > > > > > > > > > > If the packet needs to be both sNATted and dNATted (for
> > hairpin traffic
> > > > > > > > > > > with source and destination on the same chassis), then
> > sNAT is done
> > > > > > > > > > > in a separate zone.  To detect this scenario, this patch
> > adds a few
> > > > > > > > > > > extra logical flows.  For each dnat_and_snat entry prior
> > to this patch
> > > > > > > > > > > ovn-northd was generating 9 logical flows and with this
> > patch it now
> > > > > > > > > > > generates 12 logical flows.
> > > > > > > > > > >
> > > > > > > > > > > Similar approach can be taken for gateway routers.
> > > > > > > > > > >
> > > > > > > > > > > Reported-at:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > > > > > ---
> > > > > > > > > > >
> > > > > > > > > > > v2 -> v3
> > > > > > > > > > > ------
> > > > > > > > > > >    * Addressed Mark's comments and updated the
> > documentation.
> > > > > > > > > > >
> > > > > > > > > > > v1 -> v2
> > > > > > > > > > > ------
> > > > > > > > > > >    * Rebased and resolved conflicts.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > > > > > >   tests/ovn-northd.at          | 575
> > +++++++++++++++++++----------------
> > > > > > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/include/ovn/logical-fields.h
> > b/include/ovn/logical-fields.h
> > > > > > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > > > > > >   };
> > > > > > > > > > >
> > > > > > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > > > > > --- a/lib/logical-fields.c
> > > > > > > > > > > +++ b/lib/logical-fields.c
> > > > > > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash
> > *symtab)
> > > > > > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > > > > > >       expr_symtab_add_subfield(symtab,
> > "flags.skip_snat_for_lb", NULL,
> > > > > > > > > > >                                flags_str);
> > > > > > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > > > > > +    expr_symtab_add_subfield(symtab,
> > "flags.use_snat_zone", NULL,
> > > > > > > > > > > +                             flags_str);
> > > > > > > > > > >
> > > > > > > > > > >       /* Connection tracking state. */
> > > > > > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark",
> > MFF_CT_MARK, NULL, false,
> > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > > > > > --- a/northd/northd.c
> > > > > > > > > > > +++ b/northd/northd.c
> > > > > > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18,
> > "lr_in_arp_request")  \
> > > > > > > > > > >
> >                \
> > > > > > > > > > >       /* Logical router egress stages. */
> >                \
> > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0,
> > "lr_out_undnat")        \
> > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1,
> > "lr_out_post_undnat")   \
> > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2,
> > "lr_out_snat")          \
> > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3,
> > "lr_out_egr_loop")      \
> > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4,
> > "lr_out_delivery")
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,
> >                      \
> > > > > > > > > > > +                   "lr_out_chk_dnat_local")
> >                       \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1,
> > "lr_out_undnat")      \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2,
> > "lr_out_post_undnat") \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3,
> > "lr_out_snat")        \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4,
> > "lr_out_post_snat")   \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5,
> > "lr_out_egr_loop")    \
> > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6,
> > "lr_out_delivery")
> > > > > > > > > > >
> > > > > > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE,
> > TABLE, NAME)   \
> > > > > > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE
> >           \
> > > > > > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > > > > > >
> > > > > > > > > > >   /* Register to store the eth address associated to a
> > router port for packets
> > > > > > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > > > > > @@ -9568,9 +9572,10 @@
> > build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > > > > > >                                       undnat_match_p,
> > est_actions,
> > > > > > > > > > >                                       &lb->nlb->header_);
> > > > > > > > > > >           } else {
> > > > > > > > > > > -            ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > > > > > -                                    undnat_match_p,
> > "ct_dnat;",
> > > > > > > > > > > -                                    &lb->nlb->header_);
> > > > > > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > undnat_match_p,
> > > > > > > > > > > +                od->is_gw_router ? "ct_dnat;" :
> > "ct_dnat_in_czone;",
> > > > > > > > > > > +                &lb->nlb->header_);
> > > > > > > > > > >           }
> > > > > > > > > > >           free(undnat_match_p);
> > > > > > > > > > >   next:
> > > > > > > > > > > @@ -9865,7 +9870,7 @@
> > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > >           uint16_t priority;
> > > > > > > > > > >
> > > > > > > > > > >           /* Priority of logical flows corresponding to
> > exempted_ext_ips is
> > > > > > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > > > > > >            * For example, if we have following NAT rule
> > and we associate
> > > > > > > > > > >            * exempted external ips to it:
> > > > > > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat
> > 10.15.24.139 50.0.0.11"
> > > > > > > > > > > @@ -9873,17 +9878,17 @@
> > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > >            * And now we associate exempted external ip
> > address set to it.
> > > > > > > > > > >            * Now corresponding to above rule we will
> > have following logical
> > > > > > > > > > >            * flows:
> > > > > > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst
> > == $exempt_range),
> > > > > > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst
> > == $exempt_range),
> > > > > > > > > > >            *                             action=(next;)
> > > > > > > > > > >            * lr_out_snat...priority=161, match=(..),
> > action=(ct_snat(....);)
> > > > > > > > > > >            *
> > > > > > > > > > >            */
> > > > > > > > > > >           if (is_src) {
> > > > > > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > > > > > -            priority = 100 + 1;
> > > > > > > > > > > +            priority = 100 + 2;
> > > > > > > > > > >           } else {
> > > > > > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask +
> > 1 + 128 + 1) */
> > > > > > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > > > > > >
> > > > > > > > > > >               if (!od->is_gw_router) {
> > > > > > > > > > >                   priority += 128;
> > > > > > > > > > > @@ -12268,9 +12273,9 @@
> > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >           /* Traffic received on l3dgw_port is subject
> > to NAT. */
> > > > > > > > > > >           ds_clear(match);
> > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s &&
> > inport == %s",
> > > > > > > > > > > -                      is_v6 ? "6" : "4",
> > nat->external_ip,
> > > > > > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s &&
> > inport == %s && "
> > > > > > > > > > > +                      "flags.loopback == 0", is_v6 ?
> > "6" : "4",
> > > > > > > > > > > +                      nat->external_ip,
> > od->l3dgw_ports[0]->json_key);
> > > > > > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > >               /* Flows for NAT rules that are
> > centralized are only
> > > > > > > > > > >               * programmed on the gateway chassis. */
> > > > > > > > > > > @@ -12282,12 +12287,31 @@
> > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >               ds_put_format(actions, "ip%s.dst=%s;
> > next;",
> > > > > > > > > > >                             is_v6 ? "6" : "4",
> > nat->logical_ip);
> > > > > > > > > > >           } else {
> > > > > > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > > > > > >           }
> > > > > > > > > > >
> > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > >                                   100, ds_cstr(match),
> > ds_cstr(actions),
> > > > > > > > > > >                                   &nat->header_);
> > > > > > > > > > > +
> > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > +            ds_clear(match);
> > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s
> > && inport == %s && "
> > > > > > > > > > > +                          "flags.loopback == 1 &&
> > flags.use_snat_zone == 1",
> > > > > > > > > > > +                          is_v6 ? "6" : "4",
> > nat->external_ip,
> > > > > > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > +                /* Flows for NAT rules that are
> > centralized are only
> > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > +                ds_put_format(match, " &&
> > is_chassis_resident(%s)",
> > > > > > > > > > > +
> > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > +            }
> > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > > +                                    100,
> > ds_cstr(match), ds_cstr(actions),
> > > > > > > > > > > +                                    &nat->header_);
> > > > > > > > > > > +        }
> > > > > > > > > > >       }
> > > > > > > > > > >   }
> > > > > > > > > > >
> > > > > > > > > > > @@ -12364,7 +12388,7 @@
> > build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >                   ds_put_format(actions, "ip%s.dst=%s;
> > next;",
> > > > > > > > > > >                                 is_v6 ? "6" : "4",
> > nat->logical_ip);
> > > > > > > > > > >               } else {
> > > > > > > > > > > -                ds_put_format(actions, "ct_dnat(%s",
> > nat->logical_ip);
> > > > > > > > > > > +                ds_put_format(actions,
> > "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > > > > > >                       ds_put_format(actions, ",%s",
> > nat->external_port_range);
> > > > > > > > > > >                   }
> > > > > > > > > > > @@ -12417,7 +12441,8 @@
> > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > > > > >                         is_v6 ? "6" : "4",
> > nat->external_ip);
> > > > > > > > > > >       } else {
> > > > > > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > > > > > +        ds_put_format(actions,
> > > > > > > > > > > +                      od->is_gw_router ? "ct_dnat;" :
> > "ct_dnat_in_czone;");
> > > > > > > > > > >       }
> > > > > > > > > > >
> > > > > > > > > > >       ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > > > > @@ -12425,6 +12450,36 @@
> > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >                               &nat->header_);
> > > > > > > > > > >   }
> > > > > > > > > > >
> > > > > > > > > > > +static void
> > > > > > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows,
> > struct ovn_datapath *od,
> > > > > > > > > > > +                                const struct nbrec_nat
> > *nat, struct ds *match,
> > > > > > > > > > > +                                struct ds *actions,
> > bool distributed,
> > > > > > > > > > > +                                bool is_v6)
> > > > > > > > > > > +{
> > > > > > > > > > > +    /* Note that this only applies for NAT on a
> > distributed router.
> > > > > > > > > > > +     */
> > > > > > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > > > > > +        return;
> > > > > > > > > > > +    }
> > > > > > > > > > > +
> > > > > > > > > > > +    ds_clear(match);
> > > > > > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > > > > > +    if (distributed) {
> > > > > > > > > > > +        ds_put_format(match,
> > "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > > > > > +    } else {
> > > > > > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > > > > > +
> > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > +    }
> > > > > > > > > > > +
> > > > > > > > > > > +    ds_clear(actions);
> > > > > > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1;
> > next;");
> > > > > > > > > > > +
> > > > > > > > > > > +    ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > > > > > +                            50, ds_cstr(match),
> > ds_cstr(actions),
> > > > > > > > > > > +                            &nat->header_);
> > > > > > > > > > > +}
> > > > > > > > > > > +
> > > > > > > > > > >   static void
> > > > > > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows,
> > struct ovn_datapath *od,
> > > > > > > > > > >                               const struct nbrec_nat
> > *nat, struct ds *match,
> > > > > > > > > > > @@ -12478,16 +12533,19 @@
> > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >           ds_put_format(match, "ip && ip%s.src == %s &&
> > outport == %s",
> > > > > > > > > > >                         is_v6 ? "6" : "4",
> > nat->logical_ip,
> > > > > > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > -            /* Flows for NAT rules that are centralized
> > are only
> > > > > > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > -            ds_put_format(match, " &&
> > is_chassis_resident(%s)",
> > > > > > > > > > > -
> > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > -        } else if (distributed) {
> > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > -            ds_put_format(match, " &&
> > is_chassis_resident(\"%s\")",
> > > > > > > > > > > -                          nat->logical_port);
> > > > > > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > +                ds_put_format(match, " &&
> > is_chassis_resident(\"%s\")",
> > > > > > > > > > > +                              nat->logical_port);
> > > > > > > > > > > +            } else {
> > > > > > > > > > > +                /* Flows for NAT rules that are
> > centralized are only
> > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > +                ds_put_format(match, " &&
> > is_chassis_resident(%s)",
> > > > > > > > > > > +
> > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > +            }
> > > > > > > > > > >           }
> > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > >
> > > > > > > > > > > @@ -12505,7 +12563,7 @@
> > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >               ds_put_format(actions, "ip%s.src=%s;
> > next;",
> > > > > > > > > > >                             is_v6 ? "6" : "4",
> > nat->external_ip);
> > > > > > > > > > >           } else {
> > > > > > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > > > > > +            ds_put_format(actions,
> > "ct_snat_in_czone(%s",
> > > > > > > > > > >                           nat->external_ip);
> > > > > > > > > > >               if (nat->external_port_range[0]) {
> > > > > > > > > > >                   ds_put_format(actions, ",%s",
> > nat->external_port_range);
> > > > > > > > > > > @@ -12519,6 +12577,24 @@
> > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_OUT_SNAT,
> > > > > > > > > > >                                   priority,
> > ds_cstr(match),
> > > > > > > > > > >                                   ds_cstr(actions),
> > &nat->header_);
> > > > > > > > > > > +
> > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > +            ds_put_cstr(match, " &&
> > "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > +                ds_put_format(actions, "eth.src =
> > "ETH_ADDR_FMT"; ",
> > > > > > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > > > > > +            }
> > > > > > > > > > > +            ds_put_format(actions,
> > REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > > > > > +                          nat->external_ip);
> > > > > > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > > > > > +                ds_put_format(actions, ",%s",
> > nat->external_port_range);
> > > > > > > > > > > +            }
> > > > > > > > > > > +            ds_put_format(actions, ");");
> > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > S_ROUTER_OUT_SNAT,
> > > > > > > > > > > +                                    priority + 1,
> > ds_cstr(match),
> > > > > > > > > > > +                                    ds_cstr(actions),
> > &nat->header_);
> > > > > > > > > > > +        }
> > > > > > > > > > >       }
> > > > > > > > > > >   }
> > > > > > > > > > >
> > > > > > > > > > > @@ -12749,10 +12825,13 @@
> > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > *lflows,
> > > > > > > > > > >       /* Packets are allowed by default. */
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0,
> > "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0,
> > "1", "next;");
> > > > > > > > > > > +    ovn_lflow_add(lflows, od,
> > S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0,
> > "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0,
> > "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0,
> > "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT,
> > 0, "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP,
> > 0, "1", "next;");
> > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > > > > > >
> > > > > > > > > > > @@ -12765,8 +12844,7 @@
> > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > *lflows,
> > > > > > > > > > >        * not committed, it would produce ongoing
> > datapath flows with the ct.new
> > > > > > > > > > >        * flag set. Some NICs are unable to offload these
> > flows.
> > > > > > > > > > >        */
> > > > > > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat ||
> > od->has_lb_vip)) {
> > > > > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT,
> > 50,
> > > > > > > > > > >                         "ip", "flags.loopback = 1;
> > ct_dnat;");
> > > > > > > > > > >           ovn_lflow_add(lflows, od,
> > S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > > > > > @@ -12839,6 +12917,10 @@
> > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > *lflows,
> > > > > > > > > > >               }
> > > > > > > > > > >           }
> > > > > > > > > > >
> > > > > > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od,
> > nat, match, actions,
> > > > > > > > > > > +                                        distributed,
> > is_v6);
> > > > > > > > > > > +
> > > > > > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat,
> > match, actions, distributed,
> > > > > > > > > > >                                         mac, is_v6);
> > > > > > > > > > > @@ -12912,7 +12994,8 @@
> > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > *lflows,
> > > > > > > > > > >                             "clone { ct_clear; "
> > > > > > > > > > >                             "inport = outport; outport =
> > \"\"; "
> > > > > > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > > > > > -                          "flags = 0; flags.loopback =
> > 1; ");
> > > > > > > > > > > +                          "flags = 0; flags.loopback =
> > 1; "
> > > > > > > > > > > +                          "flags.use_snat_zone =
> > "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > > > > > >                   ds_put_format(actions, "reg%d = 0; ",
> > j);
> > > > > > > > > > >               }
> > > > > > > > > > > diff --git a/northd/ovn-northd.8.xml
> > b/northd/ovn-northd.8.xml
> > > > > > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > > > > > >           <p>
> > > > > > > > > > >             For each configuration in the OVN Northbound
> > database, that asks
> > > > > > > > > > >             to change the source IP address of a packet
> > from <var>A</var> to
> > > > > > > > > > > -          <var>B</var>, a priority-100 flow matches
> > <code>ip &amp;&amp;
> > > > > > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport ==
> > <var>GW</var></code> or
> > > > > > > > > > > -          <code>ip &amp;&amp;
> > > > > > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport ==
> > <var>GW</var></code>
> > > > > > > > > > > -          where <var>GW</var> is the logical router
> > gateway port, with an
> > > > > > > > > > > -          action <code>ct_snat;</code>. If the NAT rule
> > is of type
> > > > > > > > > > > -          dnat_and_snat and has
> > <code>stateless=true</code> in the
> > > > > > > > > > > -          options, then the action would be
> > <code>ip4/6.dst=
> > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > +          <var>B</var>, two priority-100 flows are
> > added.
> > > > > > > > > > >           </p>
> > > > > > > > > > >
> > > > > > > > > > >           <p>
> > > > > > > > > > >             If the NAT rule cannot be handled in a
> > distributed manner, then
> > > > > > > > > > > -          the priority-100 flow above is only
> > programmed on the
> > > > > > > > > > > +          the below priority-100 flows are only
> > programmed on the
> > > > > > > > > > >             gateway chassis.
> > > > > > > > > > >           </p>
> > > > > > > > > > >
> > > > > > > > > > > +        <ul>
> > > > > > > > > > > +          <li>
> > > > > > > > > > > +            <p>
> > > > > > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > == <var>GW</var>
> > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > == <var>GW</var>
> > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > > > > > +              where <var>GW</var> is the logical router
> > gateway port, with an
> > > > > > > > > > > +              action <code>ct_snat_in_czone;</code> to
> > unSNAT in the common
> > > > > > > > > > > +              zone.  If the NAT rule is of type
> > dnat_and_snat and has
> > > > > > > > > > > +              <code>stateless=true</code> in the
> > options, then the action
> > > > > > > > > > > +              would be
> > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > +            </p>
> > > > > > > > > > > +
> > > > > > > > > > > +            <p>
> > > > > > > > > > > +              If the NAT entry is of type
> > <code>snat</code>, then there is an
> > > > > > > > > > > +              additional match
> > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > chassis resident port of
> > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > +            </p>
> > > > > > > > > > > +          </li>
> > > > > > > > > > > +
> > > > > > > > > > > +          <li>
> > > > > > > > > > > +            <p>
> > > > > > > > > > > +              The second flow matches <code>ip
> > &amp;&amp;
> > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > == <var>GW</var>
> > > > > > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > == <var>GW</var>
> > > > > > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > > > > > +              where <var>GW</var> is the logical router
> > gateway port, with an
> > > > > > > > > > > +              action <code>ct_snat;</code> to unSNAT in
> > the snat zone. If the
> > > > > > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > > > > > +              <code>stateless=true</code> in the
> > options, then the action
> > > > > > > > > > > +              would be
> > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > +            </p>
> > > > > > > > > > > +
> > > > > > > > > > > +            <p>
> > > > > > > > > > > +              If the NAT entry is of type
> > <code>snat</code>, then there is an
> > > > > > > > > > > +              additional match
> > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > chassis resident port of
> > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > +            </p>
> > > > > > > > > > > +          </li>
> > > > > > > > > > > +        </ul>
> > > > > > > > > > > +
> > > > > > > > > > >           <p>
> > > > > > > > > > >             A priority-0 logical flow with match
> > <code>1</code> has actions
> > > > > > > > > > >             <code>next;</code>.
> > > > > > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > > > > > >         </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > > > > > +
> > > > > > > > > > > +    <p>
> > > > > > > > > > > +      This table checks if the packet needs to be
> > DNATed in the router ingress
> > > > > > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed
> > and looped back
> > > > > > > > > > > +      to the ingress pipeline.  This check is done only
> > for routers configured
> > > > > > > > > > > +      with distributed gateway ports and NAT entries.
> > This check is done
> > > > > > > > > > > +      so that SNAT and DNAT is done in different zones
> > instead of a common
> > > > > > > > > > > +      zone.
> > > > > > > > > > > +    </p>
> > > > > > > > > > > +
> > > > > > > > > > > +    <ul>
> > > > > > > > > > > +      <li>
> > > > > > > > > > > +        <p>
> > > > > > > > > > > +          For each NAT rule in the OVN Northbound
> > database on a
> > > > > > > > > > > +          distributed router, a priority-50 logical
> > flow with match
> > > > > > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > > > > > +          is_chassis_resident(<var>P</var>)</code>,
> > where <var>E</var> is the
> > > > > > > > > > > +          external IP address specified in the NAT
> > rule, <var>GW</var>
> > > > > > > > > > > +          is the logical router distributed gateway
> > port. For dnat_and_snat
> > > > > > > > > > > +          NAT rule, <var>P</var> is the logical port
> > specified in the NAT rule.
> > > > > > > > > > > +          If <ref column="logical_port"
> > > > > > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table
> > is NOT set, then
> > > > > > > > > > > +          <var>P</var> is the <code>chassisredirect
> > port</code> of
> > > > > > > > > > > +          <var>GW</var> with the actions:
> > > > > > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next;
> > </code>
> > > > > > > > > > > +        </p>
> > > > > > > > > > > +      </li>
> > > > > > > > > > > +
> > > > > > > > > > > +      <li>
> > > > > > > > > > > +        A priority-0 logical flow with match
> > <code>1</code> has actions
> > > > > > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > > > > > +      </li>
> > > > > > > > > > > +    </ul>
> > > > > > > > > > > +
> > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > > > > > >
> > > > > > > > > > >       <p>
> > > > > > > > > > >         This is for already established connections'
> > reverse traffic.
> > > > > > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > > > > > >         is unDNATed here.
> > > > > > > > > > >       </p>
> > > > > > > > > > >
> > > > > > > > > > > +    <ul>
> > > > > > > > > > > +      <li>
> > > > > > > > > > > +        A priority-0 logical flow with match
> > <code>1</code> has actions
> > > > > > > > > > > +        <code>next;</code>.
> > > > > > > > > > > +      </li>
> > > > > > > > > > > +    </ul>
> > > > > > > > > > > +
> > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > > > > > +
> > > > > > > > > > > +    <ul>
> > > > > > > > > > > +      <li>
> > > > > > > > > > > +        For all IP packets, a priority-50 flow with an
> > action
> > > > > > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > +      </li>
> > > > > > > > > > > +    </ul>
> > > > > > > > > > > +
> > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed
> > Routers</h3>
> > > > > > > > > > >       <ul>
> > > > > > > > > > >         <li>
> > > > > > > > > > >           <p>
> > > > > > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > > > > > >             gateway chassis that matches
> > > > > > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var>
> > &amp;&amp;
> > > > > > > > > > >             outport == <var>GW</var></code>, where
> > <var>GW</var> is the logical
> > > > > > > > > > > -          router gateway port with an action
> > <code>ct_dnat;</code>. If the
> > > > > > > > > > > -          backend IPv4 address <var>B</var> is also
> > configured with L4 port
> > > > > > > > > > > -          <var>PORT</var> of protocol <var>P</var>,
> > then the
> > > > > > > > > > > +          router gateway port with an action
> > <code>ct_dnat_in_czone;</code>.
> > > > > > > > > > > +          If the backend IPv4 address <var>B</var> is
> > also configured with
> > > > > > > > > > > +          L4 port <var>PORT</var> of protocol
> > <var>P</var>, then the
> > > > > > > > > > >             match also includes <code>P.src</code> ==
> > <var>PORT</var>.  These
> > > > > > > > > > >             flows are not added for load balancers with
> > IPv6 <var>VIPs</var>.
> > > > > > > > > > >           </p>
> > > > > > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > > > > > >             matches <code>ip &amp;&amp; ip4.src ==
> > <var>B</var>
> > > > > > > > > > >             &amp;&amp; outport == <var>GW</var></code>,
> > where <var>GW</var>
> > > > > > > > > > >             is the logical router gateway port, with an
> > action
> > > > > > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of
> > type
> > > > > > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT
> > rule is of type
> > > > > > > > > > >             dnat_and_snat and has
> > <code>stateless=true</code> in the
> > > > > > > > > > >             options, then the action would be
> > <code>ip4/6.src=
> > > > > > > > > > >             (<var>B</var>)</code>.
> > > > > > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > > > > > >           <p>
> > > > > > > > > > >             If the NAT rule cannot be handled in a
> > distributed manner, then
> > > > > > > > > > >             the priority-100 flow above is only
> > programmed on the
> > > > > > > > > > > -          gateway chassis.
> > > > > > > > > > > +          gateway chassis with the action
> > <code>ct_dnat_in_czone</code>.
> > > > > > > > > > >           </p>
> > > > > > > > > > >
> > > > > > > > > > >           <p>
> > > > > > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > > > > > >           </p>
> > > > > > > > > > >         </li>
> > > > > > > > > > >
> > > > > > > > > > > -      <li>
> > > > > > > > > > > -        For all IP packets, a priority-50 flow with an
> > action
> > > > > > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > -      </li>
> > > > > > > > > > > -
> > > > > > > > > > > -      <li>
> > > > > > > > > > > -        A priority-0 logical flow with match
> > <code>1</code> has actions
> > > > > > > > > > > -        <code>next;</code>.
> > > > > > > > > > > -      </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > > > > > >
> > > > > > > > > > >       <p>
> > > > > > > > > > >         <ul>
> > > > > > > > > > >           <li>
> > > > > > > > > > >             A priority-50 logical flow is added that
> > commits any untracked flows
> > > > > > > > > > > -          from the previous table
> > <code>lr_out_undnat</code>. This flow
> > > > > > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code>
> > with action
> > > > > > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > > > > > +          from the previous table
> > <code>lr_out_undnat</code> for Gateway
> > > > > > > > > > > +          routers.  This flow matches on <code>ct.new
> > &amp;&amp; ip</code>
> > > > > > > > > > > +          with action <code>ct_commit { } ; next;
> > </code>.
> > > > > > > > > > >           </li>
> > > > > > > > > > >
> > > > > > > > > > >           <li>
> > > > > > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > > > > > >         </ul>
> > > > > > > > > > >       </p>
> > > > > > > > > > >
> > > > > > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > > > > > >
> > > > > > > > > > >       <p>
> > > > > > > > > > >         Packets that are configured to be SNATed get
> > their source IP address
> > > > > > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > > > > > >         </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > > > > > >
> > > > > > > > > > >       <ul>
> > > > > > > > > > >         <li>
> > > > > > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > > > > > >         </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > > > > > >
> > > > > > > > > > >       <ul>
> > > > > > > > > > >         <li>
> > > > > > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > > > > > >             For each configuration in the OVN Northbound
> > database, that asks
> > > > > > > > > > >             to change the source IP address of a packet
> > from an IP address of
> > > > > > > > > > >             <var>A</var> or to change the source IP
> > address of a packet that
> > > > > > > > > > > -          belongs to network <var>A</var> to
> > <var>B</var>, a flow matches
> > > > > > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var>
> > &amp;&amp;
> > > > > > > > > > > -          outport == <var>GW</var></code>, where
> > <var>GW</var> is the
> > > > > > > > > > > -          logical router gateway port, with an action
> > > > > > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The
> > priority of the flow
> > > > > > > > > > > -          is calculated based on the mask of
> > <var>A</var>, with matches
> > > > > > > > > > > -          having larger masks getting higher
> > priorities. If the NAT rule
> > > > > > > > > > > -          is of type dnat_and_snat and has
> > <code>stateless=true</code>
> > > > > > > > > > > -          in the options, then the action would be
> > <code>ip4/6.src=
> > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > +          belongs to network <var>A</var> to
> > <var>B</var>, two flows are
> > > > > > > > > > > +          added.  The priority <var>P</var> of these
> > flows are calculated
> > > > > > > > > > > +          based on the mask of <var>A</var>, with
> > matches having larger
> > > > > > > > > > > +          masks getting higher priorities.
> > > > > > > > > > >           </p>
> > > > > > > > > > >
> > > > > > > > > > >           <p>
> > > > > > > > > > >             If the NAT rule cannot be handled in a
> > distributed manner, then
> > > > > > > > > > > -          the flow above is only programmed on the
> > > > > > > > > > > -          gateway chassis increasing flow priority by
> > 128 in
> > > > > > > > > > > -          order to be run first
> > > > > > > > > > > +          the below flows are only programmed on the
> > gateway chassis increasing
> > > > > > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > > > > > >           </p>
> > > > > > > > > > >
> > > > > > > > > > > +        <ul>
> > > > > > > > > > > +          <li>
> > > > > > > > > > > +            The first flow is added with the calculated
> > priority <var>P</var>
> > > > > > > > > > > +            and match <code>ip &amp;&amp; ip4.src ==
> > <var>A</var> &amp;&amp;
> > > > > > > > > > > +            outport == <var>GW</var></code>, where
> > <var>GW</var> is the
> > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > +
> > <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > > > > > +            common zone.  If the NAT rule is of type
> > dnat_and_snat and has
> > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > then the action
> > > > > > > > > > > +            would be
> > <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > +          </li>
> > > > > > > > > > > +
> > > > > > > > > > > +          <li>
> > > > > > > > > > > +            The second flow is added with the
> > calculated priority
> > > > > > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var>
> > &amp;&amp;
> > > > > > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where
> > <var>GW</var> is the
> > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT
> > in the snat zone.
> > > > > > > > > > > +            If the NAT rule is of type dnat_and_snat
> > and has
> > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > then the action would
> > > > > > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > +          </li>
> > > > > > > > > > > +        </ul>
> > > > > > > > > > > +
> > > > > > > > > > >           <p>
> > > > > > > > > > >             If the NAT rule can be handled in a
> > distributed manner, then
> > > > > > > > > > > -          there is an additional action
> > > > > > > > > > > +          there is an additional action (for both the
> > flows)
> > > > > > > > > > >             <code>eth.src = <var>EA</var>;</code>, where
> > <var>EA</var>
> > > > > > > > > > >             is the ethernet address associated with the
> > IP address
> > > > > > > > > > >             <var>A</var> in the NAT rule.  This allows
> > upstream MAC
> > > > > > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > > > > > >
> > > > > > > > > > >           <p>
> > > > > > > > > > >             If the NAT rule has
> > <code>exempted_ext_ips</code> set, then
> > > > > > > > > > > -          there is an additional flow configured at the
> > priority + 1 of
> > > > > > > > > > > +          there is an additional flow configured at the
> > priority
> > > > > > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > > > > > >             corresponding NAT rule. The flow matches if
> > destination ip
> > > > > > > > > > >             is an <code>exempted_ext_ip</code> and the
> > action is <code>next;
> > > > > > > > > > >             </code>. This flow is used to bypass the
> > ct_snat action for a flow
> > > > > > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > > > > > >         </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > > > > > >
> > > > > > > > > > >       <p>
> > > > > > > > > > >         For distributed logical routers where one of the
> > logical router
> > > > > > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > > > > > >       outport = "";
> > > > > > > > > > >       flags = 0;
> > > > > > > > > > >       flags.loopback = 1;
> > > > > > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > > > > > >       reg0 = 0;
> > > > > > > > > > >       reg1 = 0;
> > > > > > > > > > >       ...
> > > > > > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > > > > > >         </li>
> > > > > > > > > > >       </ul>
> > > > > > > > > > >
> > > > > > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > > > > > >
> > > > > > > > > > >       <p>
> > > > > > > > > > >         Packets that reach this table are ready for
> > delivery.  It contains:
> > > > > > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > > > > > --- a/tests/ovn-northd.at
> > > > > > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > > > > > >   echo
> > > > > > > > > > >   echo "IPv4: stateful"
> > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > 172.16.1.1 50.0.0.11
> > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > >
> > > > > > > > > > >   echo
> > > > > > > > > > >   echo "IPv4: stateless"
> > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > >
> > > > > > > > > > >   echo
> > > > > > > > > > >   echo "IPv6: stateful"
> > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > fd01::1 fd11::2
> > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > > > > > >
> > > > > > > > > > >   echo
> > > > > > > > > > >   echo "IPv6: stateless"
> > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > dnat_and_snat fd01::1 fd11::2
> > > > > > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > > > > > >
> > > > > > > > > > >   AT_CLEANUP
> > > > > > > > > > >   ])
> > > > > > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat
> > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > >
> > > > > > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c
> > 'ct_dnat.*3000' sbflows],
> > > > > > > > > > > -  [0], [1
> > > > > > > > > > > +  [0], [2
> > > > > > > > > > >   1
> > > > > > > > > > >   ])
> > > > > > > > > > >
> > > > > > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1
> > dnat_and_snat  172.16.1.1
> > > > > > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat
> > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > >
> > > > > > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c
> > 'ct_dnat.*3000' sbflows2],
> > > > > > > > > > > -  [1], [1
> > > > > > > > > > > +  [1], [2
> > > > > > > > > > >   0
> > > > > > > > > > >   ])
> > > > > > > > > > >
> > > > > > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat
> > 172.16.1.1
> > > > > > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add
> > R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > > > > > >
> > > > > > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000'
> > sbflows3], [1])
> > > > > > > > > > >
> > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > > > > > >
> > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > > > > > -  grep lr_out_snat drflows | grep "ip4.src ==
> > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > > > > > -  grep lr_out_snat crflows | grep "ip4.src ==
> > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > > > > > -3
> > > > > > > > > > > -3
> > > > > > > > > > > -1
> > > > > > > > > > > -1
> > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed
> > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > match=(1), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > match=(nd_ns), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range),
> > action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]]
> > == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > > > > +])
> > > > > > > > > > > +
> > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed
> > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > match=(1), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > match=(nd_ns), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range),
> > action=(ct_snat(172.16.1.1);)
> > > > > > > > > > >   ])
> > > > > > > > > > >
> > > > > > > > > > > +
> > > > > > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > > > > > >
> > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > 50.0.0.11" | grep -c "priority=161"
> > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > > > > > -4
> > > > > > > > > > > -4
> > > > > > > > > > > -1
> > > > > > > > > > > -1
> > > > > > > > > > > -1
> > > > > > > > > > > -1
> > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed
> > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > match=(1), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > match=(nd_ns), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0;
> > ct_snat(172.16.1.1);)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=163  ,
> > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range),
> > action=(next;)
> > > > > > > > > > > +])
> > > > > > > > > > > +
> > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed
> > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > match=(1), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > match=(nd_ns), action=(next;)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > > > > > +  table=??(lr_out_snat        ), priority=35   ,
> > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range),
> > action=(next;)
> > > > > > > > > > >   ])
> > > > > > > > > > >
> > > > > > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > > > > > @@ -1059,2
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Numan Siddique April 20, 2022, 8:10 p.m. UTC | #12 
On Tue, Apr 12, 2022 at 11:01 AM Numan Siddique <numans@ovn.org> wrote:
>
> On Mon, Apr 11, 2022 at 12:37 PM Frode Nordahl
> <frode.nordahl@canonical.com> wrote:
> >
> > tor. 7. apr. 2022, 19:40 skrev Frode Nordahl <frode.nordahl@canonical.com>:
> >
> > > On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
> > > >
> > > > On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> > > > >
> > > > > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > > > > <frode.nordahl@canonical.com> wrote:
> > > > > >
> > > > > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org>
> > > wrote:
> > > > > > >
> > > > > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > > >
> > > > > > > > > Hello Numan,
> > > > > > > > >
> > > > > > > > > This patch does unfortunately break gateway routers in some
> > > > > > > > > circumstances, (but not all!), at least for the way OpenStack
> > > consumes
> > > > > > > > > them.
> > > > > > > > >
> > > > > > > > > Will dig more and try to figure out what is going on, but
> > > sending this
> > > > > > > > > e-mail proactively in case you have any ideas from the top of
> > > your
> > > > > > > > > head or if anyone else has run into the same issue.
> > > > > > > > >
> > > > > > > > > The gateway appears to handle conntrack as it should:
> > > > > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11
> > > dst=194.169.254.178
> > > > > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > > > > sport=22 dport=60234 zone=52
> > > > > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11
> > > dst=194.169.254.178
> > > > > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > > > > dport=60234 zone=52
> > > > > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34
> > > dst=10.11.2.11
> > > > > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > > > > >
> > > > > > > > > However as you can see traffic going to the instance suddenly
> > > gets its
> > > > > > > > > source address mangled/set to 0.0.0.0, and the connection can
> > > never
> > > > > > > > > establish.
> > > > > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB
> > > (Ethernet),
> > > > > > > > > capture size 262144 bytes
> > > > > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0,
> > > flags [DF],
> > > > > > > > > proto TCP (6), length 60)
> > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > > > > (correct), seq 2926328730, win 64240, options [mss
> > > 1460,sackOK,TS val
> > > > > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags
> > > [DF],
> > > > > > > > > proto TCP (6), length 60)
> > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win
> > > 62230,
> > > > > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr
> > > 870680827,nop,wscale
> > > > > > > > > 7], length 0
> > > > > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0,
> > > flags
> > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val
> > > 870680922
> > > > > > > > > ecr 3490675961], length 41
> > > > > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0,
> > > flags [DF],
> > > > > > > > > proto TCP (6), length 52)
> > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options
> > > [nop,nop,TS val
> > > > > > > > > 3490676008 ecr 870680922], length 0
> > > > > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0,
> > > flags [DF],
> > > > > > > > > proto TCP (6), length 52)
> > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val
> > > 870680922
> > > > > > > > > ecr 3490675961], length 0
> > > > > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0,
> > > flags [DF],
> > > > > > > > > proto TCP (6), length 52)
> > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options
> > > [nop,nop,TS val
> > > > > > > > > 3490676009 ecr 870680922], length 0
> > > > > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0,
> > > flags
> > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options
> > > [nop,nop,TS
> > > > > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0,
> > > flags [DF],
> > > > > > > > > proto TCP (6), length 52)
> > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d
> > > (correct),
> > > > > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS
> > > val
> > > > > > > > > 870680932 ecr 3490676018], length 0
> > > > > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0,
> > > flags
> > > > > > > > > [DF], proto TCP (6), length 1564)
> > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a
> > > (incorrect
> > > > > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > ethertype IPv4
> > > > > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0,
> > > flags
> > > > > > > > > [DF], proto TCP (6), length 116)
> > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70
> > > (correct),
> > > > > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val
> > > 870681111 ecr
> > > > > > > > > 3490676018], length 64
> > > > > > > >
> > > > > > > > Turning hardware offload all the way off appears to resolve the
> > > issue,
> > > > > > > > so this might as well be driver/firmware related. May I ask what
> > > > > > > > card/driver/firmware you've been using this with?
> > > > > > > >
> > > > > > >
> > > > > > > Hi Frode,
> > > > > > >
> > > > > > > This patch attempted to resolve this BZ -
> > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > > > > This BZ is related to HWOL not working.
> > > > > > >
> > > > > > > I didn't test myself but I think other colleagues of mine tested
> > > with
> > > > > > > NVidia/Mellanox CX5.
> > > > > > > I don't think we saw the source address getting mangled in our
> > > testing.
> > > > > >
> > > > > > Thank you for sharing those details. We see the source mangling with
> > > > > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > > > > until we receive information from Nvidia/Mellanox as to what's going
> > > > > > on from a driver/firmware perspective.
> > > > > >
> > > > > > > Please let me  know if you have any more questions.
> > > > > >
> > > > > > We did unfortunately run into another issue which appears to have its
> > > > > > root in this patch. The symptomps are described in
> > > > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > > > >
> > > > > > This rudimentary patch does appear to fix it, but it may require
> > > > > > acquiring and passing in the `distributed` variable from the
> > > > > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> > > > >
> > > > > Thanks for finding this issue.  I think we should use separate zones
> > > > > for hairpin traffic.
> > > > > And I thought I had addressed it.  To handle this hairpin issue,  the
> > > > > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > > > > and if this bit is set, then ct_snat() action is used instead of
> > > > > ct_snat_in_czone().
> > > > > Let me test it out and update here.
> > > >
> > > > I tested it locally and it seems to work for me.
> > > >
> > > > Can you please share your OVN dbs if possible ?  I've left a comment
> > > > on the launchpad bug.  You can attach it there too if it's possible.
> > >
> > > Sure thing, I attached them to the LP bug [0].  In this DB the active
> > > gateway chassis is `deep-ferret.maas` and the instance on
> > > `comic-perch.maas` is unable to have two ping sessions to itself using
> > > non-distributed FIP 10.78.95.196.
> > >
> > > I'll keep looking as well, thank you for your help so far!
> > >
> > > 0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > >
> > > Updated OVN to main and it unfortunately made no difference.
> >
> > The combination of stateless on the NAT rule and the allow-related ACLs
> > does indeed look strange, but this is how OpenStack sets it up. Have not
> > looked into whether that makes sense or not yet.
> >
> > To ensure we're looking at the same thing I made this modification to the
> > `DNAT LR hairpin IPv4` system test [2]
> >
> > And executed it like this:
> >
> >     sudo make check-kernel TESTSUITEFLAGS="337"
> >
> > It fails consistently here. If I either revert [1] or remove the check for
> > the second ping from the test it succeeds.
> >
> > 2:
> > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch
>
> Thanks for this reproducer.  The test case is failing for me too.
> I'll debug further and get back to you.
>
> Numan


I debugged a bit.  I can reproduce the issue 100% of the time.  The
fact that the first ping works and the subsequent one fails indicates
that it is not an OVN issue.
From what I could see so far, it looks like conntrack issue to me.  I
need to debug this further to root cause it.

Numan

>
> >
> >
> > --
> >
> > Frode Nordahl
> >
> > > > Numan
> > > >
> > > > >
> > > > > Thanks
> > > > > Numan
> > > > >
> > > > >
> > > > > > this with you before proceeding with the fix:
> > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > index 2fb0a93c2..5fae010c0 100644
> > > > > > --- a/northd/northd.c
> > > > > > +++ b/northd/northd.c
> > > > > > @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct
> > > ovn_lb_vip *lb_vip,
> > > > > >                                      undnat_match_p, est_actions,
> > > > > >                                      &lb->nlb->header_);
> > > > > >          } else {
> > > > > > +            /* XXX do we need to check for distributed here? */
> > > > > >              ovn_lflow_add_with_hint(
> > > > > >                  lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > undnat_match_p,
> > > > > >                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > > @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> > > > > > *lflows, struct ovn_datapath *od,
> > > > > >                        is_v6 ? "6" : "4", nat->external_ip);
> > > > > >      } else {
> > > > > >          ds_put_format(actions,
> > > > > > -                      od->is_gw_router ? "ct_dnat;" :
> > > "ct_dnat_in_czone;");
> > > > > > +                      (od->is_gw_router ||
> > > > > > +                       (!distributed && od->n_l3dgw_ports)) ?
> > > > > > +                          "ct_dnat;" : "ct_dnat_in_czone;");
> > > > > >      }
> > > > > >
> > > > > >      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > > @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> > > > > > ovn_datapath *od, struct hmap *lflows,
> > > > > >       * not committed, it would produce ongoing datapath flows with
> > > the ct.new
> > > > > >       * flag set. Some NICs are unable to offload these flows.
> > > > > >       */
> > > > > > -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > +    /* XXX we probably need to get the distributed variable passed
> > > in here to
> > > > > > +     * XXX retain the proposed optimization */
> > > > > > +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > +        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > > >                        "ip", "flags.loopback = 1; ct_dnat;");
> > > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > >
> > > > > > The environment in question is using distributed gateway ports by
> > > > > > associating multiple Gateway_Chassis with a Logical_Router_Port and
> > > at
> > > > > > the same time does not use distributed FIPs.
> > > > > >
> > > > > > --
> > > > > > Frode Nordahl
> > > > > >
> > > > > > > Numan
> > > > > > >
> > > > > > > > --
> > > > > > > > Frode Nordahl
> > > > > > > >
> > > > > > > > > --
> > > > > > > > > Frode Nordahl
> > > > > > > > >
> > > > > > > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <
> > > numans@ovn.org> wrote:
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <
> > > mmichels@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the update, Numan.
> > > > > > > > > > >
> > > > > > > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > > > > > > >
> > > > > > > > > > Thanks.  I applied both the patches to the main branch.
> > > > > > > > > >
> > > > > > > > > > Numan
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > > > > > > >
> > > > > > > > > > > > Make of use of these new actions for the distributed
> > > routers
> > > > > > > > > > > > for NAT.  These new actions ensure that both sNAT and
> > > dNAT
> > > > > > > > > > > > happens in the same zone.  This approach solves a couple
> > > of
> > > > > > > > > > > > problems:
> > > > > > > > > > > >
> > > > > > > > > > > >   - The datapath flows generated for external traffic
> > > which requires
> > > > > > > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely
> > > HWOL'able.
> > > > > > > > > > > >
> > > > > > > > > > > >   - Since there is only one zone, it would avoid
> > > multiple recirculations
> > > > > > > > > > > >     (improving the performance).
> > > > > > > > > > > >
> > > > > > > > > > > > If the packet needs to be both sNATted and dNATted (for
> > > hairpin traffic
> > > > > > > > > > > > with source and destination on the same chassis), then
> > > sNAT is done
> > > > > > > > > > > > in a separate zone.  To detect this scenario, this patch
> > > adds a few
> > > > > > > > > > > > extra logical flows.  For each dnat_and_snat entry prior
> > > to this patch
> > > > > > > > > > > > ovn-northd was generating 9 logical flows and with this
> > > patch it now
> > > > > > > > > > > > generates 12 logical flows.
> > > > > > > > > > > >
> > > > > > > > > > > > Similar approach can be taken for gateway routers.
> > > > > > > > > > > >
> > > > > > > > > > > > Reported-at:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > > > > > > ---
> > > > > > > > > > > >
> > > > > > > > > > > > v2 -> v3
> > > > > > > > > > > > ------
> > > > > > > > > > > >    * Addressed Mark's comments and updated the
> > > documentation.
> > > > > > > > > > > >
> > > > > > > > > > > > v1 -> v2
> > > > > > > > > > > > ------
> > > > > > > > > > > >    * Rebased and resolved conflicts.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > > > > > > >   tests/ovn-northd.at          | 575
> > > +++++++++++++++++++----------------
> > > > > > > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/include/ovn/logical-fields.h
> > > b/include/ovn/logical-fields.h
> > > > > > > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > > > > > > >   };
> > > > > > > > > > > >
> > > > > > > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > > > > > > --- a/lib/logical-fields.c
> > > > > > > > > > > > +++ b/lib/logical-fields.c
> > > > > > > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash
> > > *symtab)
> > > > > > > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > > > > > > >       expr_symtab_add_subfield(symtab,
> > > "flags.skip_snat_for_lb", NULL,
> > > > > > > > > > > >                                flags_str);
> > > > > > > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > > > > > > +    expr_symtab_add_subfield(symtab,
> > > "flags.use_snat_zone", NULL,
> > > > > > > > > > > > +                             flags_str);
> > > > > > > > > > > >
> > > > > > > > > > > >       /* Connection tracking state. */
> > > > > > > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark",
> > > MFF_CT_MARK, NULL, false,
> > > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > > > > > > --- a/northd/northd.c
> > > > > > > > > > > > +++ b/northd/northd.c
> > > > > > > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18,
> > > "lr_in_arp_request")  \
> > > > > > > > > > > >
> > >                \
> > > > > > > > > > > >       /* Logical router egress stages. */
> > >                \
> > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0,
> > > "lr_out_undnat")        \
> > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1,
> > > "lr_out_post_undnat")   \
> > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2,
> > > "lr_out_snat")          \
> > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3,
> > > "lr_out_egr_loop")      \
> > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4,
> > > "lr_out_delivery")
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,
> > >                      \
> > > > > > > > > > > > +                   "lr_out_chk_dnat_local")
> > >                       \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1,
> > > "lr_out_undnat")      \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2,
> > > "lr_out_post_undnat") \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3,
> > > "lr_out_snat")        \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4,
> > > "lr_out_post_snat")   \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5,
> > > "lr_out_egr_loop")    \
> > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6,
> > > "lr_out_delivery")
> > > > > > > > > > > >
> > > > > > > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE,
> > > TABLE, NAME)   \
> > > > > > > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE
> > >           \
> > > > > > > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > > > > > > >
> > > > > > > > > > > >   /* Register to store the eth address associated to a
> > > router port for packets
> > > > > > > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > > > > > > @@ -9568,9 +9572,10 @@
> > > build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > > > > > > >                                       undnat_match_p,
> > > est_actions,
> > > > > > > > > > > >                                       &lb->nlb->header_);
> > > > > > > > > > > >           } else {
> > > > > > > > > > > > -            ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > > > > > > -                                    undnat_match_p,
> > > "ct_dnat;",
> > > > > > > > > > > > -                                    &lb->nlb->header_);
> > > > > > > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > undnat_match_p,
> > > > > > > > > > > > +                od->is_gw_router ? "ct_dnat;" :
> > > "ct_dnat_in_czone;",
> > > > > > > > > > > > +                &lb->nlb->header_);
> > > > > > > > > > > >           }
> > > > > > > > > > > >           free(undnat_match_p);
> > > > > > > > > > > >   next:
> > > > > > > > > > > > @@ -9865,7 +9870,7 @@
> > > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > > >           uint16_t priority;
> > > > > > > > > > > >
> > > > > > > > > > > >           /* Priority of logical flows corresponding to
> > > exempted_ext_ips is
> > > > > > > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > > > > > > >            * For example, if we have following NAT rule
> > > and we associate
> > > > > > > > > > > >            * exempted external ips to it:
> > > > > > > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat
> > > 10.15.24.139 50.0.0.11"
> > > > > > > > > > > > @@ -9873,17 +9878,17 @@
> > > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > > >            * And now we associate exempted external ip
> > > address set to it.
> > > > > > > > > > > >            * Now corresponding to above rule we will
> > > have following logical
> > > > > > > > > > > >            * flows:
> > > > > > > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst
> > > == $exempt_range),
> > > > > > > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst
> > > == $exempt_range),
> > > > > > > > > > > >            *                             action=(next;)
> > > > > > > > > > > >            * lr_out_snat...priority=161, match=(..),
> > > action=(ct_snat(....);)
> > > > > > > > > > > >            *
> > > > > > > > > > > >            */
> > > > > > > > > > > >           if (is_src) {
> > > > > > > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > > > > > > -            priority = 100 + 1;
> > > > > > > > > > > > +            priority = 100 + 2;
> > > > > > > > > > > >           } else {
> > > > > > > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask +
> > > 1 + 128 + 1) */
> > > > > > > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > > > > > > >
> > > > > > > > > > > >               if (!od->is_gw_router) {
> > > > > > > > > > > >                   priority += 128;
> > > > > > > > > > > > @@ -12268,9 +12273,9 @@
> > > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >           /* Traffic received on l3dgw_port is subject
> > > to NAT. */
> > > > > > > > > > > >           ds_clear(match);
> > > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s &&
> > > inport == %s",
> > > > > > > > > > > > -                      is_v6 ? "6" : "4",
> > > nat->external_ip,
> > > > > > > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s &&
> > > inport == %s && "
> > > > > > > > > > > > +                      "flags.loopback == 0", is_v6 ?
> > > "6" : "4",
> > > > > > > > > > > > +                      nat->external_ip,
> > > od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > >               /* Flows for NAT rules that are
> > > centralized are only
> > > > > > > > > > > >               * programmed on the gateway chassis. */
> > > > > > > > > > > > @@ -12282,12 +12287,31 @@
> > > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >               ds_put_format(actions, "ip%s.dst=%s;
> > > next;",
> > > > > > > > > > > >                             is_v6 ? "6" : "4",
> > > nat->logical_ip);
> > > > > > > > > > > >           } else {
> > > > > > > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > > > > > > >           }
> > > > > > > > > > > >
> > > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > > >                                   100, ds_cstr(match),
> > > ds_cstr(actions),
> > > > > > > > > > > >                                   &nat->header_);
> > > > > > > > > > > > +
> > > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > > +            ds_clear(match);
> > > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s
> > > && inport == %s && "
> > > > > > > > > > > > +                          "flags.loopback == 1 &&
> > > flags.use_snat_zone == 1",
> > > > > > > > > > > > +                          is_v6 ? "6" : "4",
> > > nat->external_ip,
> > > > > > > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > > +                /* Flows for NAT rules that are
> > > centralized are only
> > > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > > +                ds_put_format(match, " &&
> > > is_chassis_resident(%s)",
> > > > > > > > > > > > +
> > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > +            }
> > > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > > > +                                    100,
> > > ds_cstr(match), ds_cstr(actions),
> > > > > > > > > > > > +                                    &nat->header_);
> > > > > > > > > > > > +        }
> > > > > > > > > > > >       }
> > > > > > > > > > > >   }
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -12364,7 +12388,7 @@
> > > build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >                   ds_put_format(actions, "ip%s.dst=%s;
> > > next;",
> > > > > > > > > > > >                                 is_v6 ? "6" : "4",
> > > nat->logical_ip);
> > > > > > > > > > > >               } else {
> > > > > > > > > > > > -                ds_put_format(actions, "ct_dnat(%s",
> > > nat->logical_ip);
> > > > > > > > > > > > +                ds_put_format(actions,
> > > "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > > > > > > >                       ds_put_format(actions, ",%s",
> > > nat->external_port_range);
> > > > > > > > > > > >                   }
> > > > > > > > > > > > @@ -12417,7 +12441,8 @@
> > > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > > > > > >                         is_v6 ? "6" : "4",
> > > nat->external_ip);
> > > > > > > > > > > >       } else {
> > > > > > > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > > > > > > +        ds_put_format(actions,
> > > > > > > > > > > > +                      od->is_gw_router ? "ct_dnat;" :
> > > "ct_dnat_in_czone;");
> > > > > > > > > > > >       }
> > > > > > > > > > > >
> > > > > > > > > > > >       ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > > > > > @@ -12425,6 +12450,36 @@
> > > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >                               &nat->header_);
> > > > > > > > > > > >   }
> > > > > > > > > > > >
> > > > > > > > > > > > +static void
> > > > > > > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows,
> > > struct ovn_datapath *od,
> > > > > > > > > > > > +                                const struct nbrec_nat
> > > *nat, struct ds *match,
> > > > > > > > > > > > +                                struct ds *actions,
> > > bool distributed,
> > > > > > > > > > > > +                                bool is_v6)
> > > > > > > > > > > > +{
> > > > > > > > > > > > +    /* Note that this only applies for NAT on a
> > > distributed router.
> > > > > > > > > > > > +     */
> > > > > > > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > > > > > > +        return;
> > > > > > > > > > > > +    }
> > > > > > > > > > > > +
> > > > > > > > > > > > +    ds_clear(match);
> > > > > > > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > > > > > > +    if (distributed) {
> > > > > > > > > > > > +        ds_put_format(match,
> > > "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > > > > > > +    } else {
> > > > > > > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > > > > > > +
> > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > +    }
> > > > > > > > > > > > +
> > > > > > > > > > > > +    ds_clear(actions);
> > > > > > > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1;
> > > next;");
> > > > > > > > > > > > +
> > > > > > > > > > > > +    ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > > > > > > +                            50, ds_cstr(match),
> > > ds_cstr(actions),
> > > > > > > > > > > > +                            &nat->header_);
> > > > > > > > > > > > +}
> > > > > > > > > > > > +
> > > > > > > > > > > >   static void
> > > > > > > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows,
> > > struct ovn_datapath *od,
> > > > > > > > > > > >                               const struct nbrec_nat
> > > *nat, struct ds *match,
> > > > > > > > > > > > @@ -12478,16 +12533,19 @@
> > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >           ds_put_format(match, "ip && ip%s.src == %s &&
> > > outport == %s",
> > > > > > > > > > > >                         is_v6 ? "6" : "4",
> > > nat->logical_ip,
> > > > > > > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > > -            /* Flows for NAT rules that are centralized
> > > are only
> > > > > > > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > > -            ds_put_format(match, " &&
> > > is_chassis_resident(%s)",
> > > > > > > > > > > > -
> > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > -        } else if (distributed) {
> > > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > > -            ds_put_format(match, " &&
> > > is_chassis_resident(\"%s\")",
> > > > > > > > > > > > -                          nat->logical_port);
> > > > > > > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > > +                ds_put_format(match, " &&
> > > is_chassis_resident(\"%s\")",
> > > > > > > > > > > > +                              nat->logical_port);
> > > > > > > > > > > > +            } else {
> > > > > > > > > > > > +                /* Flows for NAT rules that are
> > > centralized are only
> > > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > > +                ds_put_format(match, " &&
> > > is_chassis_resident(%s)",
> > > > > > > > > > > > +
> > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > +            }
> > > > > > > > > > > >           }
> > > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -12505,7 +12563,7 @@
> > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >               ds_put_format(actions, "ip%s.src=%s;
> > > next;",
> > > > > > > > > > > >                             is_v6 ? "6" : "4",
> > > nat->external_ip);
> > > > > > > > > > > >           } else {
> > > > > > > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > > > > > > +            ds_put_format(actions,
> > > "ct_snat_in_czone(%s",
> > > > > > > > > > > >                           nat->external_ip);
> > > > > > > > > > > >               if (nat->external_port_range[0]) {
> > > > > > > > > > > >                   ds_put_format(actions, ",%s",
> > > nat->external_port_range);
> > > > > > > > > > > > @@ -12519,6 +12577,24 @@
> > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_OUT_SNAT,
> > > > > > > > > > > >                                   priority,
> > > ds_cstr(match),
> > > > > > > > > > > >                                   ds_cstr(actions),
> > > &nat->header_);
> > > > > > > > > > > > +
> > > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > > +            ds_put_cstr(match, " &&
> > > "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > > +                ds_put_format(actions, "eth.src =
> > > "ETH_ADDR_FMT"; ",
> > > > > > > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > > > > > > +            }
> > > > > > > > > > > > +            ds_put_format(actions,
> > > REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > > > > > > +                          nat->external_ip);
> > > > > > > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > > > > > > +                ds_put_format(actions, ",%s",
> > > nat->external_port_range);
> > > > > > > > > > > > +            }
> > > > > > > > > > > > +            ds_put_format(actions, ");");
> > > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > > S_ROUTER_OUT_SNAT,
> > > > > > > > > > > > +                                    priority + 1,
> > > ds_cstr(match),
> > > > > > > > > > > > +                                    ds_cstr(actions),
> > > &nat->header_);
> > > > > > > > > > > > +        }
> > > > > > > > > > > >       }
> > > > > > > > > > > >   }
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -12749,10 +12825,13 @@
> > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > *lflows,
> > > > > > > > > > > >       /* Packets are allowed by default. */
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0,
> > > "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0,
> > > "1", "next;");
> > > > > > > > > > > > +    ovn_lflow_add(lflows, od,
> > > S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0,
> > > "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0,
> > > "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0,
> > > "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > > S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT,
> > > 0, "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP,
> > > 0, "1", "next;");
> > > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > > S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -12765,8 +12844,7 @@
> > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > *lflows,
> > > > > > > > > > > >        * not committed, it would produce ongoing
> > > datapath flows with the ct.new
> > > > > > > > > > > >        * flag set. Some NICs are unable to offload these
> > > flows.
> > > > > > > > > > > >        */
> > > > > > > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat ||
> > > od->has_lb_vip)) {
> > > > > > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT,
> > > 50,
> > > > > > > > > > > >                         "ip", "flags.loopback = 1;
> > > ct_dnat;");
> > > > > > > > > > > >           ovn_lflow_add(lflows, od,
> > > S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > > > > > > @@ -12839,6 +12917,10 @@
> > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > *lflows,
> > > > > > > > > > > >               }
> > > > > > > > > > > >           }
> > > > > > > > > > > >
> > > > > > > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od,
> > > nat, match, actions,
> > > > > > > > > > > > +                                        distributed,
> > > is_v6);
> > > > > > > > > > > > +
> > > > > > > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat,
> > > match, actions, distributed,
> > > > > > > > > > > >                                         mac, is_v6);
> > > > > > > > > > > > @@ -12912,7 +12994,8 @@
> > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > *lflows,
> > > > > > > > > > > >                             "clone { ct_clear; "
> > > > > > > > > > > >                             "inport = outport; outport =
> > > \"\"; "
> > > > > > > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > > > > > > -                          "flags = 0; flags.loopback =
> > > 1; ");
> > > > > > > > > > > > +                          "flags = 0; flags.loopback =
> > > 1; "
> > > > > > > > > > > > +                          "flags.use_snat_zone =
> > > "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > > > > > > >                   ds_put_format(actions, "reg%d = 0; ",
> > > j);
> > > > > > > > > > > >               }
> > > > > > > > > > > > diff --git a/northd/ovn-northd.8.xml
> > > b/northd/ovn-northd.8.xml
> > > > > > > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             For each configuration in the OVN Northbound
> > > database, that asks
> > > > > > > > > > > >             to change the source IP address of a packet
> > > from <var>A</var> to
> > > > > > > > > > > > -          <var>B</var>, a priority-100 flow matches
> > > <code>ip &amp;&amp;
> > > > > > > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport ==
> > > <var>GW</var></code> or
> > > > > > > > > > > > -          <code>ip &amp;&amp;
> > > > > > > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport ==
> > > <var>GW</var></code>
> > > > > > > > > > > > -          where <var>GW</var> is the logical router
> > > gateway port, with an
> > > > > > > > > > > > -          action <code>ct_snat;</code>. If the NAT rule
> > > is of type
> > > > > > > > > > > > -          dnat_and_snat and has
> > > <code>stateless=true</code> in the
> > > > > > > > > > > > -          options, then the action would be
> > > <code>ip4/6.dst=
> > > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > > +          <var>B</var>, two priority-100 flows are
> > > added.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > distributed manner, then
> > > > > > > > > > > > -          the priority-100 flow above is only
> > > programmed on the
> > > > > > > > > > > > +          the below priority-100 flows are only
> > > programmed on the
> > > > > > > > > > > >             gateway chassis.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >
> > > > > > > > > > > > +        <ul>
> > > > > > > > > > > > +          <li>
> > > > > > > > > > > > +            <p>
> > > > > > > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > > == <var>GW</var>
> > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > > == <var>GW</var>
> > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > > > > > > +              where <var>GW</var> is the logical router
> > > gateway port, with an
> > > > > > > > > > > > +              action <code>ct_snat_in_czone;</code> to
> > > unSNAT in the common
> > > > > > > > > > > > +              zone.  If the NAT rule is of type
> > > dnat_and_snat and has
> > > > > > > > > > > > +              <code>stateless=true</code> in the
> > > options, then the action
> > > > > > > > > > > > +              would be
> > > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > > +            </p>
> > > > > > > > > > > > +
> > > > > > > > > > > > +            <p>
> > > > > > > > > > > > +              If the NAT entry is of type
> > > <code>snat</code>, then there is an
> > > > > > > > > > > > +              additional match
> > > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > > chassis resident port of
> > > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > > +            </p>
> > > > > > > > > > > > +          </li>
> > > > > > > > > > > > +
> > > > > > > > > > > > +          <li>
> > > > > > > > > > > > +            <p>
> > > > > > > > > > > > +              The second flow matches <code>ip
> > > &amp;&amp;
> > > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > > == <var>GW</var>
> > > > > > > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > > == <var>GW</var>
> > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > > > > > > +              where <var>GW</var> is the logical router
> > > gateway port, with an
> > > > > > > > > > > > +              action <code>ct_snat;</code> to unSNAT in
> > > the snat zone. If the
> > > > > > > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > > > > > > +              <code>stateless=true</code> in the
> > > options, then the action
> > > > > > > > > > > > +              would be
> > > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > > +            </p>
> > > > > > > > > > > > +
> > > > > > > > > > > > +            <p>
> > > > > > > > > > > > +              If the NAT entry is of type
> > > <code>snat</code>, then there is an
> > > > > > > > > > > > +              additional match
> > > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > > chassis resident port of
> > > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > > +            </p>
> > > > > > > > > > > > +          </li>
> > > > > > > > > > > > +        </ul>
> > > > > > > > > > > > +
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             A priority-0 logical flow with match
> > > <code>1</code> has actions
> > > > > > > > > > > >             <code>next;</code>.
> > > > > > > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <p>
> > > > > > > > > > > > +      This table checks if the packet needs to be
> > > DNATed in the router ingress
> > > > > > > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed
> > > and looped back
> > > > > > > > > > > > +      to the ingress pipeline.  This check is done only
> > > for routers configured
> > > > > > > > > > > > +      with distributed gateway ports and NAT entries.
> > > This check is done
> > > > > > > > > > > > +      so that SNAT and DNAT is done in different zones
> > > instead of a common
> > > > > > > > > > > > +      zone.
> > > > > > > > > > > > +    </p>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > +      <li>
> > > > > > > > > > > > +        <p>
> > > > > > > > > > > > +          For each NAT rule in the OVN Northbound
> > > database on a
> > > > > > > > > > > > +          distributed router, a priority-50 logical
> > > flow with match
> > > > > > > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > > > > > > +          is_chassis_resident(<var>P</var>)</code>,
> > > where <var>E</var> is the
> > > > > > > > > > > > +          external IP address specified in the NAT
> > > rule, <var>GW</var>
> > > > > > > > > > > > +          is the logical router distributed gateway
> > > port. For dnat_and_snat
> > > > > > > > > > > > +          NAT rule, <var>P</var> is the logical port
> > > specified in the NAT rule.
> > > > > > > > > > > > +          If <ref column="logical_port"
> > > > > > > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table
> > > is NOT set, then
> > > > > > > > > > > > +          <var>P</var> is the <code>chassisredirect
> > > port</code> of
> > > > > > > > > > > > +          <var>GW</var> with the actions:
> > > > > > > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next;
> > > </code>
> > > > > > > > > > > > +        </p>
> > > > > > > > > > > > +      </li>
> > > > > > > > > > > > +
> > > > > > > > > > > > +      <li>
> > > > > > > > > > > > +        A priority-0 logical flow with match
> > > <code>1</code> has actions
> > > > > > > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > > > > > > +      </li>
> > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > > > > > > >
> > > > > > > > > > > >       <p>
> > > > > > > > > > > >         This is for already established connections'
> > > reverse traffic.
> > > > > > > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > > > > > > >         is unDNATed here.
> > > > > > > > > > > >       </p>
> > > > > > > > > > > >
> > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > +      <li>
> > > > > > > > > > > > +        A priority-0 logical flow with match
> > > <code>1</code> has actions
> > > > > > > > > > > > +        <code>next;</code>.
> > > > > > > > > > > > +      </li>
> > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > +      <li>
> > > > > > > > > > > > +        For all IP packets, a priority-50 flow with an
> > > action
> > > > > > > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > > +      </li>
> > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > +
> > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed
> > > Routers</h3>
> > > > > > > > > > > >       <ul>
> > > > > > > > > > > >         <li>
> > > > > > > > > > > >           <p>
> > > > > > > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > > > > > > >             gateway chassis that matches
> > > > > > > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var>
> > > &amp;&amp;
> > > > > > > > > > > >             outport == <var>GW</var></code>, where
> > > <var>GW</var> is the logical
> > > > > > > > > > > > -          router gateway port with an action
> > > <code>ct_dnat;</code>. If the
> > > > > > > > > > > > -          backend IPv4 address <var>B</var> is also
> > > configured with L4 port
> > > > > > > > > > > > -          <var>PORT</var> of protocol <var>P</var>,
> > > then the
> > > > > > > > > > > > +          router gateway port with an action
> > > <code>ct_dnat_in_czone;</code>.
> > > > > > > > > > > > +          If the backend IPv4 address <var>B</var> is
> > > also configured with
> > > > > > > > > > > > +          L4 port <var>PORT</var> of protocol
> > > <var>P</var>, then the
> > > > > > > > > > > >             match also includes <code>P.src</code> ==
> > > <var>PORT</var>.  These
> > > > > > > > > > > >             flows are not added for load balancers with
> > > IPv6 <var>VIPs</var>.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > > > > > > >             matches <code>ip &amp;&amp; ip4.src ==
> > > <var>B</var>
> > > > > > > > > > > >             &amp;&amp; outport == <var>GW</var></code>,
> > > where <var>GW</var>
> > > > > > > > > > > >             is the logical router gateway port, with an
> > > action
> > > > > > > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of
> > > type
> > > > > > > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT
> > > rule is of type
> > > > > > > > > > > >             dnat_and_snat and has
> > > <code>stateless=true</code> in the
> > > > > > > > > > > >             options, then the action would be
> > > <code>ip4/6.src=
> > > > > > > > > > > >             (<var>B</var>)</code>.
> > > > > > > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > distributed manner, then
> > > > > > > > > > > >             the priority-100 flow above is only
> > > programmed on the
> > > > > > > > > > > > -          gateway chassis.
> > > > > > > > > > > > +          gateway chassis with the action
> > > <code>ct_dnat_in_czone</code>.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >
> > > > > > > > > > > >           <p>
> > > > > > > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >
> > > > > > > > > > > > -      <li>
> > > > > > > > > > > > -        For all IP packets, a priority-50 flow with an
> > > action
> > > > > > > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > > -      </li>
> > > > > > > > > > > > -
> > > > > > > > > > > > -      <li>
> > > > > > > > > > > > -        A priority-0 logical flow with match
> > > <code>1</code> has actions
> > > > > > > > > > > > -        <code>next;</code>.
> > > > > > > > > > > > -      </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > > > > > > >
> > > > > > > > > > > >       <p>
> > > > > > > > > > > >         <ul>
> > > > > > > > > > > >           <li>
> > > > > > > > > > > >             A priority-50 logical flow is added that
> > > commits any untracked flows
> > > > > > > > > > > > -          from the previous table
> > > <code>lr_out_undnat</code>. This flow
> > > > > > > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code>
> > > with action
> > > > > > > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > > > > > > +          from the previous table
> > > <code>lr_out_undnat</code> for Gateway
> > > > > > > > > > > > +          routers.  This flow matches on <code>ct.new
> > > &amp;&amp; ip</code>
> > > > > > > > > > > > +          with action <code>ct_commit { } ; next;
> > > </code>.
> > > > > > > > > > > >           </li>
> > > > > > > > > > > >
> > > > > > > > > > > >           <li>
> > > > > > > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > > > > > > >         </ul>
> > > > > > > > > > > >       </p>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > > > > > > >
> > > > > > > > > > > >       <p>
> > > > > > > > > > > >         Packets that are configured to be SNATed get
> > > their source IP address
> > > > > > > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > > > > > > >
> > > > > > > > > > > >       <ul>
> > > > > > > > > > > >         <li>
> > > > > > > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > > > > > > >
> > > > > > > > > > > >       <ul>
> > > > > > > > > > > >         <li>
> > > > > > > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > > > > > > >             For each configuration in the OVN Northbound
> > > database, that asks
> > > > > > > > > > > >             to change the source IP address of a packet
> > > from an IP address of
> > > > > > > > > > > >             <var>A</var> or to change the source IP
> > > address of a packet that
> > > > > > > > > > > > -          belongs to network <var>A</var> to
> > > <var>B</var>, a flow matches
> > > > > > > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var>
> > > &amp;&amp;
> > > > > > > > > > > > -          outport == <var>GW</var></code>, where
> > > <var>GW</var> is the
> > > > > > > > > > > > -          logical router gateway port, with an action
> > > > > > > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The
> > > priority of the flow
> > > > > > > > > > > > -          is calculated based on the mask of
> > > <var>A</var>, with matches
> > > > > > > > > > > > -          having larger masks getting higher
> > > priorities. If the NAT rule
> > > > > > > > > > > > -          is of type dnat_and_snat and has
> > > <code>stateless=true</code>
> > > > > > > > > > > > -          in the options, then the action would be
> > > <code>ip4/6.src=
> > > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > > +          belongs to network <var>A</var> to
> > > <var>B</var>, two flows are
> > > > > > > > > > > > +          added.  The priority <var>P</var> of these
> > > flows are calculated
> > > > > > > > > > > > +          based on the mask of <var>A</var>, with
> > > matches having larger
> > > > > > > > > > > > +          masks getting higher priorities.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > distributed manner, then
> > > > > > > > > > > > -          the flow above is only programmed on the
> > > > > > > > > > > > -          gateway chassis increasing flow priority by
> > > 128 in
> > > > > > > > > > > > -          order to be run first
> > > > > > > > > > > > +          the below flows are only programmed on the
> > > gateway chassis increasing
> > > > > > > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > > > > > > >           </p>
> > > > > > > > > > > >
> > > > > > > > > > > > +        <ul>
> > > > > > > > > > > > +          <li>
> > > > > > > > > > > > +            The first flow is added with the calculated
> > > priority <var>P</var>
> > > > > > > > > > > > +            and match <code>ip &amp;&amp; ip4.src ==
> > > <var>A</var> &amp;&amp;
> > > > > > > > > > > > +            outport == <var>GW</var></code>, where
> > > <var>GW</var> is the
> > > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > > +
> > > <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > > > > > > +            common zone.  If the NAT rule is of type
> > > dnat_and_snat and has
> > > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > > then the action
> > > > > > > > > > > > +            would be
> > > <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > > +          </li>
> > > > > > > > > > > > +
> > > > > > > > > > > > +          <li>
> > > > > > > > > > > > +            The second flow is added with the
> > > calculated priority
> > > > > > > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var>
> > > &amp;&amp;
> > > > > > > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where
> > > <var>GW</var> is the
> > > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT
> > > in the snat zone.
> > > > > > > > > > > > +            If the NAT rule is of type dnat_and_snat
> > > and has
> > > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > > then the action would
> > > > > > > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > > +          </li>
> > > > > > > > > > > > +        </ul>
> > > > > > > > > > > > +
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             If the NAT rule can be handled in a
> > > distributed manner, then
> > > > > > > > > > > > -          there is an additional action
> > > > > > > > > > > > +          there is an additional action (for both the
> > > flows)
> > > > > > > > > > > >             <code>eth.src = <var>EA</var>;</code>, where
> > > <var>EA</var>
> > > > > > > > > > > >             is the ethernet address associated with the
> > > IP address
> > > > > > > > > > > >             <var>A</var> in the NAT rule.  This allows
> > > upstream MAC
> > > > > > > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > > > > > > >
> > > > > > > > > > > >           <p>
> > > > > > > > > > > >             If the NAT rule has
> > > <code>exempted_ext_ips</code> set, then
> > > > > > > > > > > > -          there is an additional flow configured at the
> > > priority + 1 of
> > > > > > > > > > > > +          there is an additional flow configured at the
> > > priority
> > > > > > > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > > > > > > >             corresponding NAT rule. The flow matches if
> > > destination ip
> > > > > > > > > > > >             is an <code>exempted_ext_ip</code> and the
> > > action is <code>next;
> > > > > > > > > > > >             </code>. This flow is used to bypass the
> > > ct_snat action for a flow
> > > > > > > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > > > > > > >
> > > > > > > > > > > >       <p>
> > > > > > > > > > > >         For distributed logical routers where one of the
> > > logical router
> > > > > > > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > > > > > > >       outport = "";
> > > > > > > > > > > >       flags = 0;
> > > > > > > > > > > >       flags.loopback = 1;
> > > > > > > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > > > > > > >       reg0 = 0;
> > > > > > > > > > > >       reg1 = 0;
> > > > > > > > > > > >       ...
> > > > > > > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > > > > > > >         </li>
> > > > > > > > > > > >       </ul>
> > > > > > > > > > > >
> > > > > > > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > > > > > > >
> > > > > > > > > > > >       <p>
> > > > > > > > > > > >         Packets that reach this table are ready for
> > > delivery.  It contains:
> > > > > > > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > > > > > > --- a/tests/ovn-northd.at
> > > > > > > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > > > > > > >   echo
> > > > > > > > > > > >   echo "IPv4: stateful"
> > > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > > 172.16.1.1 50.0.0.11
> > > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > >
> > > > > > > > > > > >   echo
> > > > > > > > > > > >   echo "IPv4: stateless"
> > > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > > dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > >
> > > > > > > > > > > >   echo
> > > > > > > > > > > >   echo "IPv6: stateful"
> > > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > > fd01::1 fd11::2
> > > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > > > > > > >
> > > > > > > > > > > >   echo
> > > > > > > > > > > >   echo "IPv6: stateless"
> > > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > > dnat_and_snat fd01::1 fd11::2
> > > > > > > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > > > > > > >
> > > > > > > > > > > >   AT_CLEANUP
> > > > > > > > > > > >   ])
> > > > > > > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat
> > > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > > >
> > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c
> > > 'ct_dnat.*3000' sbflows],
> > > > > > > > > > > > -  [0], [1
> > > > > > > > > > > > +  [0], [2
> > > > > > > > > > > >   1
> > > > > > > > > > > >   ])
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1
> > > dnat_and_snat  172.16.1.1
> > > > > > > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat
> > > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > > >
> > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c
> > > 'ct_dnat.*3000' sbflows2],
> > > > > > > > > > > > -  [1], [1
> > > > > > > > > > > > +  [1], [2
> > > > > > > > > > > >   0
> > > > > > > > > > > >   ])
> > > > > > > > > > > >
> > > > > > > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat
> > > 172.16.1.1
> > > > > > > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add
> > > R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > > > > > > >
> > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > > test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > > test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000'
> > > sbflows3], [1])
> > > > > > > > > > > >
> > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > > > > > > >
> > > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > > > > > > -  grep lr_out_snat drflows | grep "ip4.src ==
> > > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > > > > > > -  grep lr_out_snat crflows | grep "ip4.src ==
> > > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > > > > > > -3
> > > > > > > > > > > > -3
> > > > > > > > > > > > -1
> > > > > > > > > > > > -1
> > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed
> > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > match=(1), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range),
> > > action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]]
> > > == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > > > > > +])
> > > > > > > > > > > > +
> > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed
> > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > match=(1), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range),
> > > action=(ct_snat(172.16.1.1);)
> > > > > > > > > > > >   ])
> > > > > > > > > > > >
> > > > > > > > > > > > +
> > > > > > > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > > > > > > >
> > > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > > 50.0.0.11" | grep -c "priority=161"
> > > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > > 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > > > > > > -4
> > > > > > > > > > > > -4
> > > > > > > > > > > > -1
> > > > > > > > > > > > -1
> > > > > > > > > > > > -1
> > > > > > > > > > > > -1
> > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed
> > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > match=(1), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0;
> > > ct_snat(172.16.1.1);)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=163  ,
> > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range),
> > > action=(next;)
> > > > > > > > > > > > +])
> > > > > > > > > > > > +
> > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed
> > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > match=(1), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > > match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=35   ,
> > > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range),
> > > action=(next;)
> > > > > > > > > > > >   ])
> > > > > > > > > > > >
> > > > > > > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > > > > > > @@ -1059,2
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
Frode Nordahl April 21, 2022, 9:01 a.m. UTC | #13 
On Wed, Apr 20, 2022 at 10:10 PM Numan Siddique <numans@ovn.org> wrote:
>
> On Tue, Apr 12, 2022 at 11:01 AM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Mon, Apr 11, 2022 at 12:37 PM Frode Nordahl
> > <frode.nordahl@canonical.com> wrote:
> > >
> > > tor. 7. apr. 2022, 19:40 skrev Frode Nordahl <frode.nordahl@canonical.com>:
> > >
> > > > On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
> > > > >
> > > > > On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> > > > > >
> > > > > > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > >
> > > > > > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hello Numan,
> > > > > > > > > >
> > > > > > > > > > This patch does unfortunately break gateway routers in some
> > > > > > > > > > circumstances, (but not all!), at least for the way OpenStack
> > > > consumes
> > > > > > > > > > them.
> > > > > > > > > >
> > > > > > > > > > Will dig more and try to figure out what is going on, but
> > > > sending this
> > > > > > > > > > e-mail proactively in case you have any ideas from the top of
> > > > your
> > > > > > > > > > head or if anyone else has run into the same issue.
> > > > > > > > > >
> > > > > > > > > > The gateway appears to handle conntrack as it should:
> > > > > > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11
> > > > dst=194.169.254.178
> > > > > > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > > > > > sport=22 dport=60234 zone=52
> > > > > > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11
> > > > dst=194.169.254.178
> > > > > > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > > > > > dport=60234 zone=52
> > > > > > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34
> > > > dst=10.11.2.11
> > > > > > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > > > > > >
> > > > > > > > > > However as you can see traffic going to the instance suddenly
> > > > gets its
> > > > > > > > > > source address mangled/set to 0.0.0.0, and the connection can
> > > > never
> > > > > > > > > > establish.
> > > > > > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB
> > > > (Ethernet),
> > > > > > > > > > capture size 262144 bytes
> > > > > > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0,
> > > > flags [DF],
> > > > > > > > > > proto TCP (6), length 60)
> > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > > > > > (correct), seq 2926328730, win 64240, options [mss
> > > > 1460,sackOK,TS val
> > > > > > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags
> > > > [DF],
> > > > > > > > > > proto TCP (6), length 60)
> > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win
> > > > 62230,
> > > > > > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr
> > > > 870680827,nop,wscale
> > > > > > > > > > 7], length 0
> > > > > > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0,
> > > > flags
> > > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val
> > > > 870680922
> > > > > > > > > > ecr 3490675961], length 41
> > > > > > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0,
> > > > flags [DF],
> > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options
> > > > [nop,nop,TS val
> > > > > > > > > > 3490676008 ecr 870680922], length 0
> > > > > > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0,
> > > > flags [DF],
> > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val
> > > > 870680922
> > > > > > > > > > ecr 3490675961], length 0
> > > > > > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0,
> > > > flags [DF],
> > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options
> > > > [nop,nop,TS val
> > > > > > > > > > 3490676009 ecr 870680922], length 0
> > > > > > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0,
> > > > flags
> > > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options
> > > > [nop,nop,TS
> > > > > > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0,
> > > > flags [DF],
> > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d
> > > > (correct),
> > > > > > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS
> > > > val
> > > > > > > > > > 870680932 ecr 3490676018], length 0
> > > > > > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0,
> > > > flags
> > > > > > > > > > [DF], proto TCP (6), length 1564)
> > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a
> > > > (incorrect
> > > > > > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > ethertype IPv4
> > > > > > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0,
> > > > flags
> > > > > > > > > > [DF], proto TCP (6), length 116)
> > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70
> > > > (correct),
> > > > > > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val
> > > > 870681111 ecr
> > > > > > > > > > 3490676018], length 64
> > > > > > > > >
> > > > > > > > > Turning hardware offload all the way off appears to resolve the
> > > > issue,
> > > > > > > > > so this might as well be driver/firmware related. May I ask what
> > > > > > > > > card/driver/firmware you've been using this with?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Frode,
> > > > > > > >
> > > > > > > > This patch attempted to resolve this BZ -
> > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > > > > > This BZ is related to HWOL not working.
> > > > > > > >
> > > > > > > > I didn't test myself but I think other colleagues of mine tested
> > > > with
> > > > > > > > NVidia/Mellanox CX5.
> > > > > > > > I don't think we saw the source address getting mangled in our
> > > > testing.
> > > > > > >
> > > > > > > Thank you for sharing those details. We see the source mangling with
> > > > > > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > > > > > until we receive information from Nvidia/Mellanox as to what's going
> > > > > > > on from a driver/firmware perspective.
> > > > > > >
> > > > > > > > Please let me  know if you have any more questions.
> > > > > > >
> > > > > > > We did unfortunately run into another issue which appears to have its
> > > > > > > root in this patch. The symptomps are described in
> > > > > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > > > > >
> > > > > > > This rudimentary patch does appear to fix it, but it may require
> > > > > > > acquiring and passing in the `distributed` variable from the
> > > > > > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> > > > > >
> > > > > > Thanks for finding this issue.  I think we should use separate zones
> > > > > > for hairpin traffic.
> > > > > > And I thought I had addressed it.  To handle this hairpin issue,  the
> > > > > > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > > > > > and if this bit is set, then ct_snat() action is used instead of
> > > > > > ct_snat_in_czone().
> > > > > > Let me test it out and update here.
> > > > >
> > > > > I tested it locally and it seems to work for me.
> > > > >
> > > > > Can you please share your OVN dbs if possible ?  I've left a comment
> > > > > on the launchpad bug.  You can attach it there too if it's possible.
> > > >
> > > > Sure thing, I attached them to the LP bug [0].  In this DB the active
> > > > gateway chassis is `deep-ferret.maas` and the instance on
> > > > `comic-perch.maas` is unable to have two ping sessions to itself using
> > > > non-distributed FIP 10.78.95.196.
> > > >
> > > > I'll keep looking as well, thank you for your help so far!
> > > >
> > > > 0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > >
> > > > Updated OVN to main and it unfortunately made no difference.
> > >
> > > The combination of stateless on the NAT rule and the allow-related ACLs
> > > does indeed look strange, but this is how OpenStack sets it up. Have not
> > > looked into whether that makes sense or not yet.
> > >
> > > To ensure we're looking at the same thing I made this modification to the
> > > `DNAT LR hairpin IPv4` system test [2]
> > >
> > > And executed it like this:
> > >
> > >     sudo make check-kernel TESTSUITEFLAGS="337"
> > >
> > > It fails consistently here. If I either revert [1] or remove the check for
> > > the second ping from the test it succeeds.
> > >
> > > 2:
> > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch
> >
> > Thanks for this reproducer.  The test case is failing for me too.
> > I'll debug further and get back to you.
> >
> > Numan
>
>
> I debugged a bit.  I can reproduce the issue 100% of the time.  The
> fact that the first ping works and the subsequent one fails indicates
> that it is not an OVN issue.
> From what I could see so far, it looks like conntrack issue to me.  I
> need to debug this further to root cause it.

Thank you for the update. If we're sure the change of how OVN programs
the flows is correct and should work that sounds reasonable to me.

Looking at conntrack events or dumping the conntrack table on the
first and second attempt (the synthetic test currently does this) does
indeed indicate that CT responds differently to the two sessions, but
it was unclear to me whether the root of that is the change in OVN or
not.

So essentially our current line of thought should be that the change
in OVN has uncovered a CT-related bug either in OVS, the OVS kernel
datapath or kernel CT in general?
Numan Siddique April 21, 2022, 3:09 p.m. UTC | #14 
On Thu, Apr 21, 2022 at 5:02 AM Frode Nordahl
<frode.nordahl@canonical.com> wrote:
>
> On Wed, Apr 20, 2022 at 10:10 PM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Tue, Apr 12, 2022 at 11:01 AM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Mon, Apr 11, 2022 at 12:37 PM Frode Nordahl
> > > <frode.nordahl@canonical.com> wrote:
> > > >
> > > > tor. 7. apr. 2022, 19:40 skrev Frode Nordahl <frode.nordahl@canonical.com>:
> > > >
> > > > > On Thu, Apr 7, 2022 at 6:51 PM Numan Siddique <numans@ovn.org> wrote:
> > > > > >
> > > > > > On Wed, Apr 6, 2022 at 12:11 PM Numan Siddique <numans@ovn.org> wrote:
> > > > > > >
> > > > > > > On Wed, Apr 6, 2022 at 4:58 AM Frode Nordahl
> > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Mar 30, 2022 at 4:54 PM Numan Siddique <numans@ovn.org>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Mar 30, 2022 at 6:21 AM Frode Nordahl
> > > > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, Mar 30, 2022 at 11:45 AM Frode Nordahl
> > > > > > > > > > <frode.nordahl@canonical.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hello Numan,
> > > > > > > > > > >
> > > > > > > > > > > This patch does unfortunately break gateway routers in some
> > > > > > > > > > > circumstances, (but not all!), at least for the way OpenStack
> > > > > consumes
> > > > > > > > > > > them.
> > > > > > > > > > >
> > > > > > > > > > > Will dig more and try to figure out what is going on, but
> > > > > sending this
> > > > > > > > > > > e-mail proactively in case you have any ideas from the top of
> > > > > your
> > > > > > > > > > > head or if anyone else has run into the same issue.
> > > > > > > > > > >
> > > > > > > > > > > The gateway appears to handle conntrack as it should:
> > > > > > > > > > > $ sudo conntrack -E --dst 194.169.254.178
> > > > > > > > > > >     [NEW] tcp      6 120 SYN_SENT src=10.11.2.11
> > > > > dst=194.169.254.178
> > > > > > > > > > > sport=60234 dport=22 [UNREPLIED] src=10.42.3.34 dst=10.11.2.11
> > > > > > > > > > > sport=22 dport=60234 zone=52
> > > > > > > > > > >  [UPDATE] tcp      6 60 SYN_RECV src=10.11.2.11
> > > > > dst=194.169.254.178
> > > > > > > > > > > sport=60234 dport=22 src=10.42.3.34 dst=10.11.2.11 sport=22
> > > > > > > > > > > dport=60234 zone=52
> > > > > > > > > > >  [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.2.11
> > > > > > > > > > > dst=194.169.254.178 sport=60234 dport=22 src=10.42.3.34
> > > > > dst=10.11.2.11
> > > > > > > > > > > sport=22 dport=60234 [ASSURED] zone=52
> > > > > > > > > > >
> > > > > > > > > > > However as you can see traffic going to the instance suddenly
> > > > > gets its
> > > > > > > > > > > source address mangled/set to 0.0.0.0, and the connection can
> > > > > never
> > > > > > > > > > > establish.
> > > > > > > > > > > $ sudo tcpdump -nevvi tape5c1862d-b4
> > > > > > > > > > > tcpdump: listening on tape5c1862d-b4, link-type EN10MB
> > > > > (Ethernet),
> > > > > > > > > > > capture size 262144 bytes
> > > > > > > > > > > 09:40:34.920187 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 62, id 24824, offset 0,
> > > > > flags [DF],
> > > > > > > > > > > proto TCP (6), length 60)
> > > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [S], cksum 0x9f1b
> > > > > > > > > > > (correct), seq 2926328730, win 64240, options [mss
> > > > > 1460,sackOK,TS val
> > > > > > > > > > > 870680827 ecr 0,nop,wscale 7], length 0
> > > > > > > > > > > 09:40:34.920537 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags
> > > > > [DF],
> > > > > > > > > > > proto TCP (6), length 60)
> > > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [S.], cksum 0x1990
> > > > > > > > > > > (incorrect -> 0x284c), seq 1596962125, ack 2926328731, win
> > > > > 62230,
> > > > > > > > > > > options [mss 8902,sackOK,TS val 3490675961 ecr
> > > > > 870680827,nop,wscale
> > > > > > > > > > > 7], length 0
> > > > > > > > > > > 09:40:34.968033 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 62, id 24826, offset 0,
> > > > > flags
> > > > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [P.], cksum 0x4293
> > > > > > > > > > > (correct), seq 1:42, ack 1, win 502, options [nop,nop,TS val
> > > > > 870680922
> > > > > > > > > > > ecr 3490675961], length 41
> > > > > > > > > > > 09:40:34.968243 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30244, offset 0,
> > > > > flags [DF],
> > > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > > > (incorrect -> 0x64a4), seq 1, ack 42, win 486, options
> > > > > [nop,nop,TS val
> > > > > > > > > > > 3490676008 ecr 870680922], length 0
> > > > > > > > > > > 09:40:34.968857 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24825, offset 0,
> > > > > flags [DF],
> > > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > > >     10.11.2.11.60820 > 10.42.3.34.22: Flags [.], cksum 0x64ec
> > > > > > > > > > > (correct), seq 1, ack 1, win 502, options [nop,nop,TS val
> > > > > 870680922
> > > > > > > > > > > ecr 3490675961], length 0
> > > > > > > > > > > 09:40:34.968932 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 64, id 30245, offset 0,
> > > > > flags [DF],
> > > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [.], cksum 0x1988
> > > > > > > > > > > (incorrect -> 0x64a3), seq 1, ack 42, win 486, options
> > > > > [nop,nop,TS val
> > > > > > > > > > > 3490676009 ecr 870680922], length 0
> > > > > > > > > > > 09:40:34.977991 fa:16:3e:fc:82:be > fa:16:3e:c8:19:af,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 107: (tos 0x0, ttl 64, id 30246, offset 0,
> > > > > flags
> > > > > > > > > > > [DF], proto TCP (6), length 93)
> > > > > > > > > > >     10.42.3.34.22 > 10.11.2.11.60820: Flags [P.], cksum 0x19b1
> > > > > > > > > > > (incorrect -> 0x4241), seq 1:42, ack 42, win 486, options
> > > > > [nop,nop,TS
> > > > > > > > > > > val 3490676018 ecr 870680922], length 41
> > > > > > > > > > > 09:40:34.978323 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 66: (tos 0x0, ttl 62, id 24827, offset 0,
> > > > > flags [DF],
> > > > > > > > > > > proto TCP (6), length 52)
> > > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [.], cksum 0x706d
> > > > > (correct),
> > > > > > > > > > > seq 2926328772, ack 1596962167, win 502, options [nop,nop,TS
> > > > > val
> > > > > > > > > > > 870680932 ecr 3490676018], length 0
> > > > > > > > > > > 09:40:34.979089 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 1578: (tos 0x0, ttl 62, id 24828, offset 0,
> > > > > flags
> > > > > > > > > > > [DF], proto TCP (6), length 1564)
> > > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x135a
> > > > > (incorrect
> > > > > > > > > > > -> 0xd379), seq 0:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > > > > > > > 870680933 ecr 3490676018], length 1512
> > > > > > > > > > > 09:40:35.157196 fa:16:3e:c8:19:af > fa:16:3e:fc:82:be,
> > > > > ethertype IPv4
> > > > > > > > > > > (0x0800), length 130: (tos 0x0, ttl 62, id 24830, offset 0,
> > > > > flags
> > > > > > > > > > > [DF], proto TCP (6), length 116)
> > > > > > > > > > >     0.0.0.0.60820 > 10.42.3.34.22: Flags [P.], cksum 0x5f70
> > > > > (correct),
> > > > > > > > > > > seq 1448:1512, ack 1, win 502, options [nop,nop,TS val
> > > > > 870681111 ecr
> > > > > > > > > > > 3490676018], length 64
> > > > > > > > > >
> > > > > > > > > > Turning hardware offload all the way off appears to resolve the
> > > > > issue,
> > > > > > > > > > so this might as well be driver/firmware related. May I ask what
> > > > > > > > > > card/driver/firmware you've been using this with?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Frode,
> > > > > > > > >
> > > > > > > > > This patch attempted to resolve this BZ -
> > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953.
> > > > > > > > > This BZ is related to HWOL not working.
> > > > > > > > >
> > > > > > > > > I didn't test myself but I think other colleagues of mine tested
> > > > > with
> > > > > > > > > NVidia/Mellanox CX5.
> > > > > > > > > I don't think we saw the source address getting mangled in our
> > > > > testing.
> > > > > > > >
> > > > > > > > Thank you for sharing those details. We see the source mangling with
> > > > > > > > CX6 here but only with distributed FIPs off, we'll park this for now
> > > > > > > > until we receive information from Nvidia/Mellanox as to what's going
> > > > > > > > on from a driver/firmware perspective.
> > > > > > > >
> > > > > > > > > Please let me  know if you have any more questions.
> > > > > > > >
> > > > > > > > We did unfortunately run into another issue which appears to have its
> > > > > > > > root in this patch. The symptomps are described in
> > > > > > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > > > > > >
> > > > > > > > This rudimentary patch does appear to fix it, but it may require
> > > > > > > > acquiring and passing in the `distributed` variable from the
> > > > > > > > `lrouter_check_nat_entry` function two new places, so I wanted to air
> > > > > > >
> > > > > > > Thanks for finding this issue.  I think we should use separate zones
> > > > > > > for hairpin traffic.
> > > > > > > And I thought I had addressed it.  To handle this hairpin issue,  the
> > > > > > > patch sets the register bit  REGBIT_DST_NAT_IP_LOCAL
> > > > > > > and if this bit is set, then ct_snat() action is used instead of
> > > > > > > ct_snat_in_czone().
> > > > > > > Let me test it out and update here.
> > > > > >
> > > > > > I tested it locally and it seems to work for me.
> > > > > >
> > > > > > Can you please share your OVN dbs if possible ?  I've left a comment
> > > > > > on the launchpad bug.  You can attach it there too if it's possible.
> > > > >
> > > > > Sure thing, I attached them to the LP bug [0].  In this DB the active
> > > > > gateway chassis is `deep-ferret.maas` and the instance on
> > > > > `comic-perch.maas` is unable to have two ping sessions to itself using
> > > > > non-distributed FIP 10.78.95.196.
> > > > >
> > > > > I'll keep looking as well, thank you for your help so far!
> > > > >
> > > > > 0: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856
> > > > >
> > > > > Updated OVN to main and it unfortunately made no difference.
> > > >
> > > > The combination of stateless on the NAT rule and the allow-related ACLs
> > > > does indeed look strange, but this is how OpenStack sets it up. Have not
> > > > looked into whether that makes sense or not yet.
> > > >
> > > > To ensure we're looking at the same thing I made this modification to the
> > > > `DNAT LR hairpin IPv4` system test [2]
> > > >
> > > > And executed it like this:
> > > >
> > > >     sudo make check-kernel TESTSUITEFLAGS="337"
> > > >
> > > > It fails consistently here. If I either revert [1] or remove the check for
> > > > the second ping from the test it succeeds.
> > > >
> > > > 2:
> > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch
> > >
> > > Thanks for this reproducer.  The test case is failing for me too.
> > > I'll debug further and get back to you.
> > >
> > > Numan
> >
> >
> > I debugged a bit.  I can reproduce the issue 100% of the time.  The
> > fact that the first ping works and the subsequent one fails indicates
> > that it is not an OVN issue.
> > From what I could see so far, it looks like conntrack issue to me.  I
> > need to debug this further to root cause it.
>
> Thank you for the update. If we're sure the change of how OVN programs
> the flows is correct and should work that sounds reasonable to me.
>
> Looking at conntrack events or dumping the conntrack table on the
> first and second attempt (the synthetic test currently does this) does
> indeed indicate that CT responds differently to the two sessions, but
> it was unclear to me whether the root of that is the change in OVN or
> not.
>
> So essentially our current line of thought should be that the change
> in OVN has uncovered a CT-related bug either in OVS, the OVS kernel
> datapath or kernel CT in general?

Yes.  That's my understanding too.  This needs to be debugged further.
We probably didn't see this issue earlier,  since the packet was going
into 2 zones.

Thanks
Numan

>
> --
> Frode Nordahl
>
> > Numan
> >
> > >
> > > >
> > > >
> > > > --
> > > >
> > > > Frode Nordahl
> > > >
> > > > > > Numan
> > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Numan
> > > > > > >
> > > > > > >
> > > > > > > > this with you before proceeding with the fix:
> > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > index 2fb0a93c2..5fae010c0 100644
> > > > > > > > --- a/northd/northd.c
> > > > > > > > +++ b/northd/northd.c
> > > > > > > > @@ -9891,6 +9891,7 @@ build_lrouter_nat_flows_for_lb(struct
> > > > > ovn_lb_vip *lb_vip,
> > > > > > > >                                      undnat_match_p, est_actions,
> > > > > > > >                                      &lb->nlb->header_);
> > > > > > > >          } else {
> > > > > > > > +            /* XXX do we need to check for distributed here? */
> > > > > > > >              ovn_lflow_add_with_hint(
> > > > > > > >                  lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > > > undnat_match_p,
> > > > > > > >                  od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
> > > > > > > > @@ -12851,7 +12852,9 @@ build_lrouter_out_undnat_flow(struct hmap
> > > > > > > > *lflows, struct ovn_datapath *od,
> > > > > > > >                        is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > >      } else {
> > > > > > > >          ds_put_format(actions,
> > > > > > > > -                      od->is_gw_router ? "ct_dnat;" :
> > > > > "ct_dnat_in_czone;");
> > > > > > > > +                      (od->is_gw_router ||
> > > > > > > > +                       (!distributed && od->n_l3dgw_ports)) ?
> > > > > > > > +                          "ct_dnat;" : "ct_dnat_in_czone;");
> > > > > > > >      }
> > > > > > > >
> > > > > > > >      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > @@ -13250,7 +13253,10 @@ build_lrouter_nat_defrag_and_lb(struct
> > > > > > > > ovn_datapath *od, struct hmap *lflows,
> > > > > > > >       * not committed, it would produce ongoing datapath flows with
> > > > > the ct.new
> > > > > > > >       * flag set. Some NICs are unable to offload these flows.
> > > > > > > >       */
> > > > > > > > -    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > +    /* XXX we probably need to get the distributed variable passed
> > > > > in here to
> > > > > > > > +     * XXX retain the proposed optimization */
> > > > > > > > +    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > +        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> > > > > > > >                        "ip", "flags.loopback = 1; ct_dnat;");
> > > > > > > >          ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > >
> > > > > > > > The environment in question is using distributed gateway ports by
> > > > > > > > associating multiple Gateway_Chassis with a Logical_Router_Port and
> > > > > at
> > > > > > > > the same time does not use distributed FIPs.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Frode Nordahl
> > > > > > > >
> > > > > > > > > Numan
> > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Frode Nordahl
> > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Frode Nordahl
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Nov 20, 2021 at 12:48 AM Numan Siddique <
> > > > > numans@ovn.org> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Nov 19, 2021 at 1:11 PM Mark Michelson <
> > > > > mmichels@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the update, Numan.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Acked-by: Mark Michelson <mmichels@redhat.com>
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.  I applied both the patches to the main branch.
> > > > > > > > > > > >
> > > > > > > > > > > > Numan
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 11/19/21 11:32, numans@ovn.org wrote:
> > > > > > > > > > > > > > From: Numan Siddique <numans@ovn.org>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Make of use of these new actions for the distributed
> > > > > routers
> > > > > > > > > > > > > > for NAT.  These new actions ensure that both sNAT and
> > > > > dNAT
> > > > > > > > > > > > > > happens in the same zone.  This approach solves a couple
> > > > > of
> > > > > > > > > > > > > > problems:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   - The datapath flows generated for external traffic
> > > > > which requires
> > > > > > > > > > > > > >     dNAT (N -> S) or sNAT (S -> N) are completely
> > > > > HWOL'able.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   - Since there is only one zone, it would avoid
> > > > > multiple recirculations
> > > > > > > > > > > > > >     (improving the performance).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If the packet needs to be both sNATted and dNATted (for
> > > > > hairpin traffic
> > > > > > > > > > > > > > with source and destination on the same chassis), then
> > > > > sNAT is done
> > > > > > > > > > > > > > in a separate zone.  To detect this scenario, this patch
> > > > > adds a few
> > > > > > > > > > > > > > extra logical flows.  For each dnat_and_snat entry prior
> > > > > to this patch
> > > > > > > > > > > > > > ovn-northd was generating 9 logical flows and with this
> > > > > patch it now
> > > > > > > > > > > > > > generates 12 logical flows.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Similar approach can be taken for gateway routers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Reported-at:
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984953
> > > > > > > > > > > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > v2 -> v3
> > > > > > > > > > > > > > ------
> > > > > > > > > > > > > >    * Addressed Mark's comments and updated the
> > > > > documentation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > v1 -> v2
> > > > > > > > > > > > > > ------
> > > > > > > > > > > > > >    * Rebased and resolved conflicts.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   include/ovn/logical-fields.h |   1 +
> > > > > > > > > > > > > >   lib/logical-fields.c         |   4 +
> > > > > > > > > > > > > >   northd/northd.c              | 147 +++++++--
> > > > > > > > > > > > > >   northd/ovn-northd.8.xml      | 205 ++++++++++---
> > > > > > > > > > > > > >   tests/ovn-northd.at          | 575
> > > > > +++++++++++++++++++----------------
> > > > > > > > > > > > > >   tests/ovn.at                 |   2 +-
> > > > > > > > > > > > > >   tests/system-ovn.at          |  64 ++--
> > > > > > > > > > > > > >   7 files changed, 610 insertions(+), 388 deletions(-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > diff --git a/include/ovn/logical-fields.h
> > > > > b/include/ovn/logical-fields.h
> > > > > > > > > > > > > > index c9675f81c..2118f7933 100644
> > > > > > > > > > > > > > --- a/include/ovn/logical-fields.h
> > > > > > > > > > > > > > +++ b/include/ovn/logical-fields.h
> > > > > > > > > > > > > > @@ -70,6 +70,7 @@ enum mff_log_flags_bits {
> > > > > > > > > > > > > >       MLF_LOOKUP_FDB_BIT = 8,
> > > > > > > > > > > > > >       MLF_SKIP_SNAT_FOR_LB_BIT = 9,
> > > > > > > > > > > > > >       MLF_LOCALPORT_BIT = 10,
> > > > > > > > > > > > > > +    MLF_USE_SNAT_ZONE = 11,
> > > > > > > > > > > > > >   };
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   /* MFF_LOG_FLAGS_REG flag assignments */
> > > > > > > > > > > > > > diff --git a/lib/logical-fields.c b/lib/logical-fields.c
> > > > > > > > > > > > > > index 7b3d431e0..352a48c89 100644
> > > > > > > > > > > > > > --- a/lib/logical-fields.c
> > > > > > > > > > > > > > +++ b/lib/logical-fields.c
> > > > > > > > > > > > > > @@ -125,6 +125,10 @@ ovn_init_symtab(struct shash
> > > > > *symtab)
> > > > > > > > > > > > > >                MLF_SKIP_SNAT_FOR_LB_BIT);
> > > > > > > > > > > > > >       expr_symtab_add_subfield(symtab,
> > > > > "flags.skip_snat_for_lb", NULL,
> > > > > > > > > > > > > >                                flags_str);
> > > > > > > > > > > > > > +    snprintf(flags_str, sizeof flags_str, "flags[%d]",
> > > > > > > > > > > > > > +             MLF_USE_SNAT_ZONE);
> > > > > > > > > > > > > > +    expr_symtab_add_subfield(symtab,
> > > > > "flags.use_snat_zone", NULL,
> > > > > > > > > > > > > > +                             flags_str);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       /* Connection tracking state. */
> > > > > > > > > > > > > >       expr_symtab_add_field_scoped(symtab, "ct_mark",
> > > > > MFF_CT_MARK, NULL, false,
> > > > > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c
> > > > > > > > > > > > > > index 0ff61deec..e4d051a94 100644
> > > > > > > > > > > > > > --- a/northd/northd.c
> > > > > > > > > > > > > > +++ b/northd/northd.c
> > > > > > > > > > > > > > @@ -159,11 +159,14 @@ enum ovn_stage {
> > > > > > > > > > > > > >       PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18,
> > > > > "lr_in_arp_request")  \
> > > > > > > > > > > > > >
> > > > >                \
> > > > > > > > > > > > > >       /* Logical router egress stages. */
> > > > >                \
> > > > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0,
> > > > > "lr_out_undnat")        \
> > > > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1,
> > > > > "lr_out_post_undnat")   \
> > > > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2,
> > > > > "lr_out_snat")          \
> > > > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3,
> > > > > "lr_out_egr_loop")      \
> > > > > > > > > > > > > > -    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4,
> > > > > "lr_out_delivery")
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,
> > > > >                      \
> > > > > > > > > > > > > > +                   "lr_out_chk_dnat_local")
> > > > >                       \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1,
> > > > > "lr_out_undnat")      \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2,
> > > > > "lr_out_post_undnat") \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3,
> > > > > "lr_out_snat")        \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4,
> > > > > "lr_out_post_snat")   \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5,
> > > > > "lr_out_egr_loop")    \
> > > > > > > > > > > > > > +    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6,
> > > > > "lr_out_delivery")
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE,
> > > > > TABLE, NAME)   \
> > > > > > > > > > > > > >       S_##DP_TYPE##_##PIPELINE##_##STAGE
> > > > >           \
> > > > > > > > > > > > > > @@ -210,6 +213,7 @@ enum ovn_stage {
> > > > > > > > > > > > > >   #define REGBIT_PKT_LARGER        "reg9[1]"
> > > > > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
> > > > > > > > > > > > > >   #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
> > > > > > > > > > > > > > +#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   /* Register to store the eth address associated to a
> > > > > router port for packets
> > > > > > > > > > > > > >    * received in S_ROUTER_IN_ADMISSION.
> > > > > > > > > > > > > > @@ -9568,9 +9572,10 @@
> > > > > build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> > > > > > > > > > > > > >                                       undnat_match_p,
> > > > > est_actions,
> > > > > > > > > > > > > >                                       &lb->nlb->header_);
> > > > > > > > > > > > > >           } else {
> > > > > > > > > > > > > > -            ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_OUT_UNDNAT, 120,
> > > > > > > > > > > > > > -                                    undnat_match_p,
> > > > > "ct_dnat;",
> > > > > > > > > > > > > > -                                    &lb->nlb->header_);
> > > > > > > > > > > > > > +            ovn_lflow_add_with_hint(
> > > > > > > > > > > > > > +                lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> > > > > undnat_match_p,
> > > > > > > > > > > > > > +                od->is_gw_router ? "ct_dnat;" :
> > > > > "ct_dnat_in_czone;",
> > > > > > > > > > > > > > +                &lb->nlb->header_);
> > > > > > > > > > > > > >           }
> > > > > > > > > > > > > >           free(undnat_match_p);
> > > > > > > > > > > > > >   next:
> > > > > > > > > > > > > > @@ -9865,7 +9870,7 @@
> > > > > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > > > > >           uint16_t priority;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           /* Priority of logical flows corresponding to
> > > > > exempted_ext_ips is
> > > > > > > > > > > > > > -         * +1 of the corresponding regulr NAT rule.
> > > > > > > > > > > > > > +         * +2 of the corresponding regular NAT rule.
> > > > > > > > > > > > > >            * For example, if we have following NAT rule
> > > > > and we associate
> > > > > > > > > > > > > >            * exempted external ips to it:
> > > > > > > > > > > > > >            * "ovn-nbctl lr-nat-add router dnat_and_snat
> > > > > 10.15.24.139 50.0.0.11"
> > > > > > > > > > > > > > @@ -9873,17 +9878,17 @@
> > > > > lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> > > > > > > > > > > > > >            * And now we associate exempted external ip
> > > > > address set to it.
> > > > > > > > > > > > > >            * Now corresponding to above rule we will
> > > > > have following logical
> > > > > > > > > > > > > >            * flows:
> > > > > > > > > > > > > > -         * lr_out_snat...priority=162, match=(..ip4.dst
> > > > > == $exempt_range),
> > > > > > > > > > > > > > +         * lr_out_snat...priority=163, match=(..ip4.dst
> > > > > == $exempt_range),
> > > > > > > > > > > > > >            *                             action=(next;)
> > > > > > > > > > > > > >            * lr_out_snat...priority=161, match=(..),
> > > > > action=(ct_snat(....);)
> > > > > > > > > > > > > >            *
> > > > > > > > > > > > > >            */
> > > > > > > > > > > > > >           if (is_src) {
> > > > > > > > > > > > > >               /* S_ROUTER_IN_DNAT uses priority 100 */
> > > > > > > > > > > > > > -            priority = 100 + 1;
> > > > > > > > > > > > > > +            priority = 100 + 2;
> > > > > > > > > > > > > >           } else {
> > > > > > > > > > > > > >               /* S_ROUTER_OUT_SNAT uses priority (mask +
> > > > > 1 + 128 + 1) */
> > > > > > > > > > > > > > -            priority = count_1bits(ntohl(mask)) + 2;
> > > > > > > > > > > > > > +            priority = count_1bits(ntohl(mask)) + 3;
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >               if (!od->is_gw_router) {
> > > > > > > > > > > > > >                   priority += 128;
> > > > > > > > > > > > > > @@ -12268,9 +12273,9 @@
> > > > > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >           /* Traffic received on l3dgw_port is subject
> > > > > to NAT. */
> > > > > > > > > > > > > >           ds_clear(match);
> > > > > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > > > > > -        ds_put_format(match, "ip && ip%s.dst == %s &&
> > > > > inport == %s",
> > > > > > > > > > > > > > -                      is_v6 ? "6" : "4",
> > > > > nat->external_ip,
> > > > > > > > > > > > > > -                      od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > > > +        ds_put_format(match, "ip && ip%s.dst == %s &&
> > > > > inport == %s && "
> > > > > > > > > > > > > > +                      "flags.loopback == 0", is_v6 ?
> > > > > "6" : "4",
> > > > > > > > > > > > > > +                      nat->external_ip,
> > > > > od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > > >           if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > > > >               /* Flows for NAT rules that are
> > > > > centralized are only
> > > > > > > > > > > > > >               * programmed on the gateway chassis. */
> > > > > > > > > > > > > > @@ -12282,12 +12287,31 @@
> > > > > build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >               ds_put_format(actions, "ip%s.dst=%s;
> > > > > next;",
> > > > > > > > > > > > > >                             is_v6 ? "6" : "4",
> > > > > nat->logical_ip);
> > > > > > > > > > > > > >           } else {
> > > > > > > > > > > > > > -            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat_in_czone;");
> > > > > > > > > > > > > >           }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > > > > >                                   100, ds_cstr(match),
> > > > > ds_cstr(actions),
> > > > > > > > > > > > > >                                   &nat->header_);
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > > > > +            ds_clear(match);
> > > > > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > > > > +            ds_put_format(match, "ip && ip%s.dst == %s
> > > > > && inport == %s && "
> > > > > > > > > > > > > > +                          "flags.loopback == 1 &&
> > > > > flags.use_snat_zone == 1",
> > > > > > > > > > > > > > +                          is_v6 ? "6" : "4",
> > > > > nat->external_ip,
> > > > > > > > > > > > > > +                          od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > > > +            if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > > > > +                /* Flows for NAT rules that are
> > > > > centralized are only
> > > > > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > > > > +                ds_put_format(match, " &&
> > > > > is_chassis_resident(%s)",
> > > > > > > > > > > > > > +
> > > > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > > > +            }
> > > > > > > > > > > > > > +            ds_put_cstr(actions, "ct_snat;");
> > > > > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_IN_UNSNAT,
> > > > > > > > > > > > > > +                                    100,
> > > > > ds_cstr(match), ds_cstr(actions),
> > > > > > > > > > > > > > +                                    &nat->header_);
> > > > > > > > > > > > > > +        }
> > > > > > > > > > > > > >       }
> > > > > > > > > > > > > >   }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -12364,7 +12388,7 @@
> > > > > build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >                   ds_put_format(actions, "ip%s.dst=%s;
> > > > > next;",
> > > > > > > > > > > > > >                                 is_v6 ? "6" : "4",
> > > > > nat->logical_ip);
> > > > > > > > > > > > > >               } else {
> > > > > > > > > > > > > > -                ds_put_format(actions, "ct_dnat(%s",
> > > > > nat->logical_ip);
> > > > > > > > > > > > > > +                ds_put_format(actions,
> > > > > "ct_dnat_in_czone(%s", nat->logical_ip);
> > > > > > > > > > > > > >                   if (nat->external_port_range[0]) {
> > > > > > > > > > > > > >                       ds_put_format(actions, ",%s",
> > > > > nat->external_port_range);
> > > > > > > > > > > > > >                   }
> > > > > > > > > > > > > > @@ -12417,7 +12441,8 @@
> > > > > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >           ds_put_format(actions, "ip%s.src=%s; next;",
> > > > > > > > > > > > > >                         is_v6 ? "6" : "4",
> > > > > nat->external_ip);
> > > > > > > > > > > > > >       } else {
> > > > > > > > > > > > > > -        ds_put_format(actions, "ct_dnat;");
> > > > > > > > > > > > > > +        ds_put_format(actions,
> > > > > > > > > > > > > > +                      od->is_gw_router ? "ct_dnat;" :
> > > > > "ct_dnat_in_czone;");
> > > > > > > > > > > > > >       }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_OUT_UNDNAT, 100,
> > > > > > > > > > > > > > @@ -12425,6 +12450,36 @@
> > > > > build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >                               &nat->header_);
> > > > > > > > > > > > > >   }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +static void
> > > > > > > > > > > > > > +build_lrouter_out_is_dnat_local(struct hmap *lflows,
> > > > > struct ovn_datapath *od,
> > > > > > > > > > > > > > +                                const struct nbrec_nat
> > > > > *nat, struct ds *match,
> > > > > > > > > > > > > > +                                struct ds *actions,
> > > > > bool distributed,
> > > > > > > > > > > > > > +                                bool is_v6)
> > > > > > > > > > > > > > +{
> > > > > > > > > > > > > > +    /* Note that this only applies for NAT on a
> > > > > distributed router.
> > > > > > > > > > > > > > +     */
> > > > > > > > > > > > > > +    if (!od->n_l3dgw_ports) {
> > > > > > > > > > > > > > +        return;
> > > > > > > > > > > > > > +    }
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    ds_clear(match);
> > > > > > > > > > > > > > +    ds_put_format(match, "ip && ip%s.dst == %s && ",
> > > > > > > > > > > > > > +                  is_v6 ? "6" : "4", nat->external_ip);
> > > > > > > > > > > > > > +    if (distributed) {
> > > > > > > > > > > > > > +        ds_put_format(match,
> > > > > "is_chassis_resident(\"%s\")", nat->logical_port);
> > > > > > > > > > > > > > +    } else {
> > > > > > > > > > > > > > +        ds_put_format(match, "is_chassis_resident(%s)",
> > > > > > > > > > > > > > +
> > > > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > > > +    }
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    ds_clear(actions);
> > > > > > > > > > > > > > +    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1;
> > > > > next;");
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_OUT_CHECK_DNAT_LOCAL,
> > > > > > > > > > > > > > +                            50, ds_cstr(match),
> > > > > ds_cstr(actions),
> > > > > > > > > > > > > > +                            &nat->header_);
> > > > > > > > > > > > > > +}
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > >   static void
> > > > > > > > > > > > > >   build_lrouter_out_snat_flow(struct hmap *lflows,
> > > > > struct ovn_datapath *od,
> > > > > > > > > > > > > >                               const struct nbrec_nat
> > > > > *nat, struct ds *match,
> > > > > > > > > > > > > > @@ -12478,16 +12533,19 @@
> > > > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >           ds_put_format(match, "ip && ip%s.src == %s &&
> > > > > outport == %s",
> > > > > > > > > > > > > >                         is_v6 ? "6" : "4",
> > > > > nat->logical_ip,
> > > > > > > > > > > > > >                         od->l3dgw_ports[0]->json_key);
> > > > > > > > > > > > > > -        if (!distributed && od->n_l3dgw_ports) {
> > > > > > > > > > > > > > -            /* Flows for NAT rules that are centralized
> > > > > are only
> > > > > > > > > > > > > > -            * programmed on the gateway chassis. */
> > > > > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > > > > -            ds_put_format(match, " &&
> > > > > is_chassis_resident(%s)",
> > > > > > > > > > > > > > -
> > > > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > > > -        } else if (distributed) {
> > > > > > > > > > > > > > -            priority += 128;
> > > > > > > > > > > > > > -            ds_put_format(match, " &&
> > > > > is_chassis_resident(\"%s\")",
> > > > > > > > > > > > > > -                          nat->logical_port);
> > > > > > > > > > > > > > +        if (od->n_l3dgw_ports) {
> > > > > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > > > > +                ovs_assert(nat->logical_port);
> > > > > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > > > > +                ds_put_format(match, " &&
> > > > > is_chassis_resident(\"%s\")",
> > > > > > > > > > > > > > +                              nat->logical_port);
> > > > > > > > > > > > > > +            } else {
> > > > > > > > > > > > > > +                /* Flows for NAT rules that are
> > > > > centralized are only
> > > > > > > > > > > > > > +                * programmed on the gateway chassis. */
> > > > > > > > > > > > > > +                priority += 128;
> > > > > > > > > > > > > > +                ds_put_format(match, " &&
> > > > > is_chassis_resident(%s)",
> > > > > > > > > > > > > > +
> > > > > od->l3dgw_ports[0]->cr_port->json_key);
> > > > > > > > > > > > > > +            }
> > > > > > > > > > > > > >           }
> > > > > > > > > > > > > >           ds_clear(actions);
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -12505,7 +12563,7 @@
> > > > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >               ds_put_format(actions, "ip%s.src=%s;
> > > > > next;",
> > > > > > > > > > > > > >                             is_v6 ? "6" : "4",
> > > > > nat->external_ip);
> > > > > > > > > > > > > >           } else {
> > > > > > > > > > > > > > -            ds_put_format(actions, "ct_snat(%s",
> > > > > > > > > > > > > > +            ds_put_format(actions,
> > > > > "ct_snat_in_czone(%s",
> > > > > > > > > > > > > >                           nat->external_ip);
> > > > > > > > > > > > > >               if (nat->external_port_range[0]) {
> > > > > > > > > > > > > >                   ds_put_format(actions, ",%s",
> > > > > nat->external_port_range);
> > > > > > > > > > > > > > @@ -12519,6 +12577,24 @@
> > > > > build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> > > > > > > > > > > > > >           ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_OUT_SNAT,
> > > > > > > > > > > > > >                                   priority,
> > > > > ds_cstr(match),
> > > > > > > > > > > > > >                                   ds_cstr(actions),
> > > > > &nat->header_);
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        if (!stateless) {
> > > > > > > > > > > > > > +            ds_put_cstr(match, " &&
> > > > > "REGBIT_DST_NAT_IP_LOCAL" == 1");
> > > > > > > > > > > > > > +            ds_clear(actions);
> > > > > > > > > > > > > > +            if (distributed) {
> > > > > > > > > > > > > > +                ds_put_format(actions, "eth.src =
> > > > > "ETH_ADDR_FMT"; ",
> > > > > > > > > > > > > > +                              ETH_ADDR_ARGS(mac));
> > > > > > > > > > > > > > +            }
> > > > > > > > > > > > > > +            ds_put_format(actions,
> > > > > REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
> > > > > > > > > > > > > > +                          nat->external_ip);
> > > > > > > > > > > > > > +            if (nat->external_port_range[0]) {
> > > > > > > > > > > > > > +                ds_put_format(actions, ",%s",
> > > > > nat->external_port_range);
> > > > > > > > > > > > > > +            }
> > > > > > > > > > > > > > +            ds_put_format(actions, ");");
> > > > > > > > > > > > > > +            ovn_lflow_add_with_hint(lflows, od,
> > > > > S_ROUTER_OUT_SNAT,
> > > > > > > > > > > > > > +                                    priority + 1,
> > > > > ds_cstr(match),
> > > > > > > > > > > > > > +                                    ds_cstr(actions),
> > > > > &nat->header_);
> > > > > > > > > > > > > > +        }
> > > > > > > > > > > > > >       }
> > > > > > > > > > > > > >   }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -12749,10 +12825,13 @@
> > > > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > > > *lflows,
> > > > > > > > > > > > > >       /* Packets are allowed by default. */
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0,
> > > > > "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0,
> > > > > "1", "next;");
> > > > > > > > > > > > > > +    ovn_lflow_add(lflows, od,
> > > > > S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
> > > > > > > > > > > > > > +                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0,
> > > > > "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0,
> > > > > "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0,
> > > > > "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > > > > S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
> > > > > > > > > > > > > > +    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT,
> > > > > 0, "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP,
> > > > > 0, "1", "next;");
> > > > > > > > > > > > > >       ovn_lflow_add(lflows, od,
> > > > > S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -12765,8 +12844,7 @@
> > > > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > > > *lflows,
> > > > > > > > > > > > > >        * not committed, it would produce ongoing
> > > > > datapath flows with the ct.new
> > > > > > > > > > > > > >        * flag set. Some NICs are unable to offload these
> > > > > flows.
> > > > > > > > > > > > > >        */
> > > > > > > > > > > > > > -    if ((od->is_gw_router || od->n_l3dgw_ports) &&
> > > > > > > > > > > > > > -        (od->nbr->n_nat || od->has_lb_vip)) {
> > > > > > > > > > > > > > +    if (od->is_gw_router && (od->nbr->n_nat ||
> > > > > od->has_lb_vip)) {
> > > > > > > > > > > > > >           ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT,
> > > > > 50,
> > > > > > > > > > > > > >                         "ip", "flags.loopback = 1;
> > > > > ct_dnat;");
> > > > > > > > > > > > > >           ovn_lflow_add(lflows, od,
> > > > > S_ROUTER_OUT_POST_UNDNAT, 50,
> > > > > > > > > > > > > > @@ -12839,6 +12917,10 @@
> > > > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > > > *lflows,
> > > > > > > > > > > > > >               }
> > > > > > > > > > > > > >           }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +        /* S_ROUTER_OUT_DNAT_LOCAL */
> > > > > > > > > > > > > > +        build_lrouter_out_is_dnat_local(lflows, od,
> > > > > nat, match, actions,
> > > > > > > > > > > > > > +                                        distributed,
> > > > > is_v6);
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > >           /* S_ROUTER_OUT_UNDNAT */
> > > > > > > > > > > > > >           build_lrouter_out_undnat_flow(lflows, od, nat,
> > > > > match, actions, distributed,
> > > > > > > > > > > > > >                                         mac, is_v6);
> > > > > > > > > > > > > > @@ -12912,7 +12994,8 @@
> > > > > build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap
> > > > > *lflows,
> > > > > > > > > > > > > >                             "clone { ct_clear; "
> > > > > > > > > > > > > >                             "inport = outport; outport =
> > > > > \"\"; "
> > > > > > > > > > > > > >                             "eth.dst <-> eth.src; "
> > > > > > > > > > > > > > -                          "flags = 0; flags.loopback =
> > > > > 1; ");
> > > > > > > > > > > > > > +                          "flags = 0; flags.loopback =
> > > > > 1; "
> > > > > > > > > > > > > > +                          "flags.use_snat_zone =
> > > > > "REGBIT_DST_NAT_IP_LOCAL"; ");
> > > > > > > > > > > > > >               for (int j = 0; j < MFF_N_LOG_REGS; j++) {
> > > > > > > > > > > > > >                   ds_put_format(actions, "reg%d = 0; ",
> > > > > j);
> > > > > > > > > > > > > >               }
> > > > > > > > > > > > > > diff --git a/northd/ovn-northd.8.xml
> > > > > b/northd/ovn-northd.8.xml
> > > > > > > > > > > > > > index 21d83718c..e39e6e805 100644
> > > > > > > > > > > > > > --- a/northd/ovn-northd.8.xml
> > > > > > > > > > > > > > +++ b/northd/ovn-northd.8.xml
> > > > > > > > > > > > > > @@ -2879,23 +2879,65 @@ icmp6 {
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             For each configuration in the OVN Northbound
> > > > > database, that asks
> > > > > > > > > > > > > >             to change the source IP address of a packet
> > > > > from <var>A</var> to
> > > > > > > > > > > > > > -          <var>B</var>, a priority-100 flow matches
> > > > > <code>ip &amp;&amp;
> > > > > > > > > > > > > > -          ip4.dst == <var>B</var> &amp;&amp; inport ==
> > > > > <var>GW</var></code> or
> > > > > > > > > > > > > > -          <code>ip &amp;&amp;
> > > > > > > > > > > > > > -          ip6.dst == <var>B</var> &amp;&amp; inport ==
> > > > > <var>GW</var></code>
> > > > > > > > > > > > > > -          where <var>GW</var> is the logical router
> > > > > gateway port, with an
> > > > > > > > > > > > > > -          action <code>ct_snat;</code>. If the NAT rule
> > > > > is of type
> > > > > > > > > > > > > > -          dnat_and_snat and has
> > > > > <code>stateless=true</code> in the
> > > > > > > > > > > > > > -          options, then the action would be
> > > > > <code>ip4/6.dst=
> > > > > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > > > > +          <var>B</var>, two priority-100 flows are
> > > > > added.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > > > distributed manner, then
> > > > > > > > > > > > > > -          the priority-100 flow above is only
> > > > > programmed on the
> > > > > > > > > > > > > > +          the below priority-100 flows are only
> > > > > programmed on the
> > > > > > > > > > > > > >             gateway chassis.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +        <ul>
> > > > > > > > > > > > > > +          <li>
> > > > > > > > > > > > > > +            <p>
> > > > > > > > > > > > > > +              The first flow matches <code>ip &amp;&amp;
> > > > > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > > > > == <var>GW</var>
> > > > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code> or
> > > > > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > > > > == <var>GW</var>
> > > > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0</code>
> > > > > > > > > > > > > > +              where <var>GW</var> is the logical router
> > > > > gateway port, with an
> > > > > > > > > > > > > > +              action <code>ct_snat_in_czone;</code> to
> > > > > unSNAT in the common
> > > > > > > > > > > > > > +              zone.  If the NAT rule is of type
> > > > > dnat_and_snat and has
> > > > > > > > > > > > > > +              <code>stateless=true</code> in the
> > > > > options, then the action
> > > > > > > > > > > > > > +              would be
> > > > > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > > > > +            </p>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +            <p>
> > > > > > > > > > > > > > +              If the NAT entry is of type
> > > > > <code>snat</code>, then there is an
> > > > > > > > > > > > > > +              additional match
> > > > > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > > > > chassis resident port of
> > > > > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > > > > +            </p>
> > > > > > > > > > > > > > +          </li>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +          <li>
> > > > > > > > > > > > > > +            <p>
> > > > > > > > > > > > > > +              The second flow matches <code>ip
> > > > > &amp;&amp;
> > > > > > > > > > > > > > +              ip4.dst == <var>B</var> &amp;&amp; inport
> > > > > == <var>GW</var>
> > > > > > > > > > > > > > +              &amp;&amp; flags.loopback == 1 &amp;&amp;
> > > > > > > > > > > > > > +              flags.use_snat_zone == 1</code> or
> > > > > > > > > > > > > > +              <code>ip &amp;&amp;
> > > > > > > > > > > > > > +              ip6.dst == <var>B</var> &amp;&amp; inport
> > > > > == <var>GW</var>
> > > > > > > > > > > > > > +              &amp;&amp; flags.loopback == 0 &amp;&amp;
> > > > > > > > > > > > > > +              flags.use_snat_zone == 1</code>
> > > > > > > > > > > > > > +              where <var>GW</var> is the logical router
> > > > > gateway port, with an
> > > > > > > > > > > > > > +              action <code>ct_snat;</code> to unSNAT in
> > > > > the snat zone. If the
> > > > > > > > > > > > > > +              NAT rule is of type dnat_and_snat and has
> > > > > > > > > > > > > > +              <code>stateless=true</code> in the
> > > > > options, then the action
> > > > > > > > > > > > > > +              would be
> > > > > <code>ip4/6.dst=(<var>B</var>)</code>.
> > > > > > > > > > > > > > +            </p>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +            <p>
> > > > > > > > > > > > > > +              If the NAT entry is of type
> > > > > <code>snat</code>, then there is an
> > > > > > > > > > > > > > +              additional match
> > > > > <code>is_chassis_resident(<var>cr-GW</var>)
> > > > > > > > > > > > > > +              </code> where <var>cr-GW</var> is the
> > > > > chassis resident port of
> > > > > > > > > > > > > > +              <var>GW</var>.
> > > > > > > > > > > > > > +            </p>
> > > > > > > > > > > > > > +          </li>
> > > > > > > > > > > > > > +        </ul>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             A priority-0 logical flow with match
> > > > > <code>1</code> has actions
> > > > > > > > > > > > > >             <code>next;</code>.
> > > > > > > > > > > > > > @@ -4031,7 +4073,43 @@ nd_ns {
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <h3>Egress Table 0: UNDNAT</h3>
> > > > > > > > > > > > > > +    <h3>Egress Table 0: Check DNAT local </h3>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <p>
> > > > > > > > > > > > > > +      This table checks if the packet needs to be
> > > > > DNATed in the router ingress
> > > > > > > > > > > > > > +      table <code>lr_in_dnat</code> after it is SNATed
> > > > > and looped back
> > > > > > > > > > > > > > +      to the ingress pipeline.  This check is done only
> > > > > for routers configured
> > > > > > > > > > > > > > +      with distributed gateway ports and NAT entries.
> > > > > This check is done
> > > > > > > > > > > > > > +      so that SNAT and DNAT is done in different zones
> > > > > instead of a common
> > > > > > > > > > > > > > +      zone.
> > > > > > > > > > > > > > +    </p>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > > > +      <li>
> > > > > > > > > > > > > > +        <p>
> > > > > > > > > > > > > > +          For each NAT rule in the OVN Northbound
> > > > > database on a
> > > > > > > > > > > > > > +          distributed router, a priority-50 logical
> > > > > flow with match
> > > > > > > > > > > > > > +          <code>ip4.dst == <var>E</var> &amp;&amp;
> > > > > > > > > > > > > > +          is_chassis_resident(<var>P</var>)</code>,
> > > > > where <var>E</var> is the
> > > > > > > > > > > > > > +          external IP address specified in the NAT
> > > > > rule, <var>GW</var>
> > > > > > > > > > > > > > +          is the logical router distributed gateway
> > > > > port. For dnat_and_snat
> > > > > > > > > > > > > > +          NAT rule, <var>P</var> is the logical port
> > > > > specified in the NAT rule.
> > > > > > > > > > > > > > +          If <ref column="logical_port"
> > > > > > > > > > > > > > +          table="NAT" db="OVN_Northbound"/> column of
> > > > > > > > > > > > > > +          <ref table="NAT" db="OVN_Northbound"/> table
> > > > > is NOT set, then
> > > > > > > > > > > > > > +          <var>P</var> is the <code>chassisredirect
> > > > > port</code> of
> > > > > > > > > > > > > > +          <var>GW</var> with the actions:
> > > > > > > > > > > > > > +          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next;
> > > > > </code>
> > > > > > > > > > > > > > +        </p>
> > > > > > > > > > > > > > +      </li>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +      <li>
> > > > > > > > > > > > > > +        A priority-0 logical flow with match
> > > > > <code>1</code> has actions
> > > > > > > > > > > > > > +        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
> > > > > > > > > > > > > > +      </li>
> > > > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT</h3>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <p>
> > > > > > > > > > > > > >         This is for already established connections'
> > > > > reverse traffic.
> > > > > > > > > > > > > > @@ -4040,6 +4118,23 @@ nd_ns {
> > > > > > > > > > > > > >         is unDNATed here.
> > > > > > > > > > > > > >       </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > > > +      <li>
> > > > > > > > > > > > > > +        A priority-0 logical flow with match
> > > > > <code>1</code> has actions
> > > > > > > > > > > > > > +        <code>next;</code>.
> > > > > > > > > > > > > > +      </li>
> > > > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <ul>
> > > > > > > > > > > > > > +      <li>
> > > > > > > > > > > > > > +        For all IP packets, a priority-50 flow with an
> > > > > action
> > > > > > > > > > > > > > +        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > > > > +      </li>
> > > > > > > > > > > > > > +    </ul>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    <h3>Egress Table 1: UNDNAT on Distributed
> > > > > Routers</h3>
> > > > > > > > > > > > > >       <ul>
> > > > > > > > > > > > > >         <li>
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > > @@ -4050,9 +4145,9 @@ nd_ns {
> > > > > > > > > > > > > >             gateway chassis that matches
> > > > > > > > > > > > > >             <code>ip &amp;&amp; ip4.src == <var>B</var>
> > > > > &amp;&amp;
> > > > > > > > > > > > > >             outport == <var>GW</var></code>, where
> > > > > <var>GW</var> is the logical
> > > > > > > > > > > > > > -          router gateway port with an action
> > > > > <code>ct_dnat;</code>. If the
> > > > > > > > > > > > > > -          backend IPv4 address <var>B</var> is also
> > > > > configured with L4 port
> > > > > > > > > > > > > > -          <var>PORT</var> of protocol <var>P</var>,
> > > > > then the
> > > > > > > > > > > > > > +          router gateway port with an action
> > > > > <code>ct_dnat_in_czone;</code>.
> > > > > > > > > > > > > > +          If the backend IPv4 address <var>B</var> is
> > > > > also configured with
> > > > > > > > > > > > > > +          L4 port <var>PORT</var> of protocol
> > > > > <var>P</var>, then the
> > > > > > > > > > > > > >             match also includes <code>P.src</code> ==
> > > > > <var>PORT</var>.  These
> > > > > > > > > > > > > >             flows are not added for load balancers with
> > > > > IPv6 <var>VIPs</var>.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > > @@ -4072,7 +4167,7 @@ nd_ns {
> > > > > > > > > > > > > >             matches <code>ip &amp;&amp; ip4.src ==
> > > > > <var>B</var>
> > > > > > > > > > > > > >             &amp;&amp; outport == <var>GW</var></code>,
> > > > > where <var>GW</var>
> > > > > > > > > > > > > >             is the logical router gateway port, with an
> > > > > action
> > > > > > > > > > > > > > -          <code>ct_dnat;</code>. If the NAT rule is of
> > > > > type
> > > > > > > > > > > > > > +          <code>ct_dnat_in_czone;</code>. If the NAT
> > > > > rule is of type
> > > > > > > > > > > > > >             dnat_and_snat and has
> > > > > <code>stateless=true</code> in the
> > > > > > > > > > > > > >             options, then the action would be
> > > > > <code>ip4/6.src=
> > > > > > > > > > > > > >             (<var>B</var>)</code>.
> > > > > > > > > > > > > > @@ -4081,7 +4176,7 @@ nd_ns {
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > > > distributed manner, then
> > > > > > > > > > > > > >             the priority-100 flow above is only
> > > > > programmed on the
> > > > > > > > > > > > > > -          gateway chassis.
> > > > > > > > > > > > > > +          gateway chassis with the action
> > > > > <code>ct_dnat_in_czone</code>.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > > @@ -4094,26 +4189,17 @@ nd_ns {
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -      <li>
> > > > > > > > > > > > > > -        For all IP packets, a priority-50 flow with an
> > > > > action
> > > > > > > > > > > > > > -        <code>flags.loopback = 1; ct_dnat;</code>.
> > > > > > > > > > > > > > -      </li>
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > > -      <li>
> > > > > > > > > > > > > > -        A priority-0 logical flow with match
> > > > > <code>1</code> has actions
> > > > > > > > > > > > > > -        <code>next;</code>.
> > > > > > > > > > > > > > -      </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <h3>Egress Table 1: Post UNDNAT</h3>
> > > > > > > > > > > > > > +    <h3>Egress Table 2: Post UNDNAT</h3>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <p>
> > > > > > > > > > > > > >         <ul>
> > > > > > > > > > > > > >           <li>
> > > > > > > > > > > > > >             A priority-50 logical flow is added that
> > > > > commits any untracked flows
> > > > > > > > > > > > > > -          from the previous table
> > > > > <code>lr_out_undnat</code>. This flow
> > > > > > > > > > > > > > -          matches on <code>ct.new &amp;&amp; ip</code>
> > > > > with action
> > > > > > > > > > > > > > -          <code>ct_commit { } ; next; </code>.
> > > > > > > > > > > > > > +          from the previous table
> > > > > <code>lr_out_undnat</code> for Gateway
> > > > > > > > > > > > > > +          routers.  This flow matches on <code>ct.new
> > > > > &amp;&amp; ip</code>
> > > > > > > > > > > > > > +          with action <code>ct_commit { } ; next;
> > > > > </code>.
> > > > > > > > > > > > > >           </li>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           <li>
> > > > > > > > > > > > > > @@ -4124,7 +4210,7 @@ nd_ns {
> > > > > > > > > > > > > >         </ul>
> > > > > > > > > > > > > >       </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <h3>Egress Table 2: SNAT</h3>
> > > > > > > > > > > > > > +    <h3>Egress Table 3: SNAT</h3>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <p>
> > > > > > > > > > > > > >         Packets that are configured to be SNATed get
> > > > > their source IP address
> > > > > > > > > > > > > > @@ -4140,7 +4226,7 @@ nd_ns {
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <p>Egress Table 2: SNAT on Gateway Routers</p>
> > > > > > > > > > > > > > +    <p>Egress Table 3: SNAT on Gateway Routers</p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <ul>
> > > > > > > > > > > > > >         <li>
> > > > > > > > > > > > > > @@ -4239,7 +4325,7 @@ nd_ns {
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <p>Egress Table 2: SNAT on Distributed Routers</p>
> > > > > > > > > > > > > > +    <p>Egress Table 3: SNAT on Distributed Routers</p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <ul>
> > > > > > > > > > > > > >         <li>
> > > > > > > > > > > > > > @@ -4247,28 +4333,47 @@ nd_ns {
> > > > > > > > > > > > > >             For each configuration in the OVN Northbound
> > > > > database, that asks
> > > > > > > > > > > > > >             to change the source IP address of a packet
> > > > > from an IP address of
> > > > > > > > > > > > > >             <var>A</var> or to change the source IP
> > > > > address of a packet that
> > > > > > > > > > > > > > -          belongs to network <var>A</var> to
> > > > > <var>B</var>, a flow matches
> > > > > > > > > > > > > > -          <code>ip &amp;&amp; ip4.src == <var>A</var>
> > > > > &amp;&amp;
> > > > > > > > > > > > > > -          outport == <var>GW</var></code>, where
> > > > > <var>GW</var> is the
> > > > > > > > > > > > > > -          logical router gateway port, with an action
> > > > > > > > > > > > > > -          <code>ct_snat(<var>B</var>);</code>.  The
> > > > > priority of the flow
> > > > > > > > > > > > > > -          is calculated based on the mask of
> > > > > <var>A</var>, with matches
> > > > > > > > > > > > > > -          having larger masks getting higher
> > > > > priorities. If the NAT rule
> > > > > > > > > > > > > > -          is of type dnat_and_snat and has
> > > > > <code>stateless=true</code>
> > > > > > > > > > > > > > -          in the options, then the action would be
> > > > > <code>ip4/6.src=
> > > > > > > > > > > > > > -          (<var>B</var>)</code>.
> > > > > > > > > > > > > > +          belongs to network <var>A</var> to
> > > > > <var>B</var>, two flows are
> > > > > > > > > > > > > > +          added.  The priority <var>P</var> of these
> > > > > flows are calculated
> > > > > > > > > > > > > > +          based on the mask of <var>A</var>, with
> > > > > matches having larger
> > > > > > > > > > > > > > +          masks getting higher priorities.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             If the NAT rule cannot be handled in a
> > > > > distributed manner, then
> > > > > > > > > > > > > > -          the flow above is only programmed on the
> > > > > > > > > > > > > > -          gateway chassis increasing flow priority by
> > > > > 128 in
> > > > > > > > > > > > > > -          order to be run first
> > > > > > > > > > > > > > +          the below flows are only programmed on the
> > > > > gateway chassis increasing
> > > > > > > > > > > > > > +          flow priority by 128 in order to be run first.
> > > > > > > > > > > > > >           </p>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +        <ul>
> > > > > > > > > > > > > > +          <li>
> > > > > > > > > > > > > > +            The first flow is added with the calculated
> > > > > priority <var>P</var>
> > > > > > > > > > > > > > +            and match <code>ip &amp;&amp; ip4.src ==
> > > > > <var>A</var> &amp;&amp;
> > > > > > > > > > > > > > +            outport == <var>GW</var></code>, where
> > > > > <var>GW</var> is the
> > > > > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > > > > +
> > > > > <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
> > > > > > > > > > > > > > +            common zone.  If the NAT rule is of type
> > > > > dnat_and_snat and has
> > > > > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > > > > then the action
> > > > > > > > > > > > > > +            would be
> > > > > <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > > > > +          </li>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +          <li>
> > > > > > > > > > > > > > +            The second flow is added with the
> > > > > calculated priority
> > > > > > > > > > > > > > +            <code><var>P</var> + 1 </code> and match
> > > > > > > > > > > > > > +            <code>ip &amp;&amp; ip4.src == <var>A</var>
> > > > > &amp;&amp;
> > > > > > > > > > > > > > +            outport == <var>GW</var> &amp;&amp;
> > > > > > > > > > > > > > +            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where
> > > > > <var>GW</var> is the
> > > > > > > > > > > > > > +            logical router gateway port, with an action
> > > > > > > > > > > > > > +            <code>ct_snat(<var>B</var>);</code> to SNAT
> > > > > in the snat zone.
> > > > > > > > > > > > > > +            If the NAT rule is of type dnat_and_snat
> > > > > and has
> > > > > > > > > > > > > > +            <code>stateless=true</code> in the options,
> > > > > then the action would
> > > > > > > > > > > > > > +            be <code>ip4/6.src=(<var>B</var>)</code>.
> > > > > > > > > > > > > > +          </li>
> > > > > > > > > > > > > > +        </ul>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             If the NAT rule can be handled in a
> > > > > distributed manner, then
> > > > > > > > > > > > > > -          there is an additional action
> > > > > > > > > > > > > > +          there is an additional action (for both the
> > > > > flows)
> > > > > > > > > > > > > >             <code>eth.src = <var>EA</var>;</code>, where
> > > > > <var>EA</var>
> > > > > > > > > > > > > >             is the ethernet address associated with the
> > > > > IP address
> > > > > > > > > > > > > >             <var>A</var> in the NAT rule.  This allows
> > > > > upstream MAC
> > > > > > > > > > > > > > @@ -4284,7 +4389,8 @@ nd_ns {
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >           <p>
> > > > > > > > > > > > > >             If the NAT rule has
> > > > > <code>exempted_ext_ips</code> set, then
> > > > > > > > > > > > > > -          there is an additional flow configured at the
> > > > > priority + 1 of
> > > > > > > > > > > > > > +          there is an additional flow configured at the
> > > > > priority
> > > > > > > > > > > > > > +          <code><var>P</var> + 2 </code> of
> > > > > > > > > > > > > >             corresponding NAT rule. The flow matches if
> > > > > destination ip
> > > > > > > > > > > > > >             is an <code>exempted_ext_ip</code> and the
> > > > > action is <code>next;
> > > > > > > > > > > > > >             </code>. This flow is used to bypass the
> > > > > ct_snat action for a flow
> > > > > > > > > > > > > > @@ -4299,7 +4405,7 @@ nd_ns {
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <h3>Egress Table 3: Egress Loopback</h3>
> > > > > > > > > > > > > > +    <h3>Egress Table 4: Egress Loopback</h3>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <p>
> > > > > > > > > > > > > >         For distributed logical routers where one of the
> > > > > logical router
> > > > > > > > > > > > > > @@ -4344,6 +4450,7 @@ clone {
> > > > > > > > > > > > > >       outport = "";
> > > > > > > > > > > > > >       flags = 0;
> > > > > > > > > > > > > >       flags.loopback = 1;
> > > > > > > > > > > > > > +    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
> > > > > > > > > > > > > >       reg0 = 0;
> > > > > > > > > > > > > >       reg1 = 0;
> > > > > > > > > > > > > >       ...
> > > > > > > > > > > > > > @@ -4368,7 +4475,7 @@ clone {
> > > > > > > > > > > > > >         </li>
> > > > > > > > > > > > > >       </ul>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -    <h3>Egress Table 4: Delivery</h3>
> > > > > > > > > > > > > > +    <h3>Egress Table 5: Delivery</h3>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >       <p>
> > > > > > > > > > > > > >         Packets that reach this table are ready for
> > > > > delivery.  It contains:
> > > > > > > > > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> > > > > > > > > > > > > > index 85b47a18f..70ec5e2e3 100644
> > > > > > > > > > > > > > --- a/tests/ovn-northd.at
> > > > > > > > > > > > > > +++ b/tests/ovn-northd.at
> > > > > > > > > > > > > > @@ -877,25 +877,25 @@ check_flow_match_sets() {
> > > > > > > > > > > > > >   echo
> > > > > > > > > > > > > >   echo "IPv4: stateful"
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > > > > 172.16.1.1 50.0.0.11
> > > > > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   echo
> > > > > > > > > > > > > >   echo "IPv4: stateless"
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > > > > dnat_and_snat  172.16.1.1 50.0.0.11
> > > > > > > > > > > > > > -check_flow_match_sets 2 0 1 2 2 0 0
> > > > > > > > > > > > > > +check_flow_match_sets 2 0 0 2 2 0 0
> > > > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   echo
> > > > > > > > > > > > > >   echo "IPv6: stateful"
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat
> > > > > fd01::1 fd11::2
> > > > > > > > > > > > > > -check_flow_match_sets 2 2 3 0 0 0 0
> > > > > > > > > > > > > > +check_flow_match_sets 3 4 2 0 0 0 0
> > > > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   echo
> > > > > > > > > > > > > >   echo "IPv6: stateless"
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb --stateless lr-nat-add R1
> > > > > dnat_and_snat fd01::1 fd11::2
> > > > > > > > > > > > > > -check_flow_match_sets 2 0 1 0 0 2 2
> > > > > > > > > > > > > > +check_flow_match_sets 2 0 0 0 0 2 2
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   AT_CLEANUP
> > > > > > > > > > > > > >   ])
> > > > > > > > > > > > > > @@ -924,9 +924,9 @@ echo "CR-LRP UUID is: " $uuid
> > > > > > > > > > > > > >   ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat
> > > > > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows])
> > > > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > > > > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows &&
> > > > > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c
> > > > > 'ct_dnat.*3000' sbflows],
> > > > > > > > > > > > > > -  [0], [1
> > > > > > > > > > > > > > +  [0], [2
> > > > > > > > > > > > > >   1
> > > > > > > > > > > > > >   ])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -934,9 +934,9 @@ ovn-nbctl lr-nat-del R1
> > > > > dnat_and_snat  172.16.1.1
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat
> > > > > 172.16.1.1 50.0.0.11 1-3000
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows2])
> > > > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > > > > test 2 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 &&
> > > > > test 3 = `grep -c lr_in_unsnat sbflows`])
> > > > > > > > > > > > > >   AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c
> > > > > 'ct_dnat.*3000' sbflows2],
> > > > > > > > > > > > > > -  [1], [1
> > > > > > > > > > > > > > +  [1], [2
> > > > > > > > > > > > > >   0
> > > > > > > > > > > > > >   ])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @@ -944,7 +944,7 @@ ovn-nbctl lr-nat-del R1 snat
> > > > > 172.16.1.1
> > > > > > > > > > > > > >   ovn-nbctl --wait=sb --portrange --stateless lr-nat-add
> > > > > R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   AT_CAPTURE_FILE([sbflows3])
> > > > > > > > > > > > > > -OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > > > > test 3 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > > > > > +OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 &&
> > > > > test 4 = `grep -c lr_in_unsnat sbflows3`])
> > > > > > > > > > > > > >   AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000'
> > > > > sbflows3], [1])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
> > > > > > > > > > > > > > @@ -1008,17 +1008,20 @@ AT_CAPTURE_FILE([drflows])
> > > > > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows
> > > > > > > > > > > > > >   AT_CAPTURE_FILE([crflows])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > > > > -  grep -c lr_out_snat drflows
> > > > > > > > > > > > > > -  grep -c lr_out_snat crflows
> > > > > > > > > > > > > > -  grep lr_out_snat drflows | grep "ip4.src ==
> > > > > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
> > > > > > > > > > > > > > -  grep lr_out_snat crflows | grep "ip4.src ==
> > > > > 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
> > > > > > > > > > > > > > -3
> > > > > > > > > > > > > > -3
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows | sed
> > > > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > > > match=(1), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > > > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range),
> > > > > action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > > > is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]]
> > > > > == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
> > > > > > > > > > > > > > +])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows | sed
> > > > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > > > match=(1), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range),
> > > > > action=(ct_snat(172.16.1.1);)
> > > > > > > > > > > > > >   ])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > >   # SNAT with DISALLOWED_IPs
> > > > > > > > > > > > > >   check ovn-nbctl lr-nat-del DR snat  50.0.0.11
> > > > > > > > > > > > > >   check ovn-nbctl lr-nat-del CR snat  50.0.0.11
> > > > > > > > > > > > > > @@ -1036,19 +1039,19 @@ AT_CAPTURE_FILE([drflows2])
> > > > > > > > > > > > > >   ovn-sbctl dump-flows CR > crflows2
> > > > > > > > > > > > > >   AT_CAPTURE_FILE([crflows2])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -AT_CHECK([
> > > > > > > > > > > > > > -  grep -c lr_out_snat drflows2
> > > > > > > > > > > > > > -  grep -c lr_out_snat crflows2
> > > > > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > > > > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
> > > > > > > > > > > > > > -  grep lr_out_snat drflows2 | grep "ip4.src ==
> > > > > 50.0.0.11" | grep -c "priority=161"
> > > > > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > > > > 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
> > > > > > > > > > > > > > -  grep lr_out_snat crflows2 | grep "ip4.src ==
> > > > > 50.0.0.11" | grep -c "priority=33"], [0], [dnl
> > > > > > > > > > > > > > -4
> > > > > > > > > > > > > > -4
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > -1
> > > > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" drflows2 | sed
> > > > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > > > match=(1), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=161  ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > > > is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=162  ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > > > is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0;
> > > > > ct_snat(172.16.1.1);)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=163  ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" &&
> > > > > is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range),
> > > > > action=(next;)
> > > > > > > > > > > > > > +])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +AT_CHECK([grep -e "lr_out_snat" crflows2 | sed
> > > > > 's/table=../table=??/' | sort], [0], [dnl
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=0    ,
> > > > > match=(1), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=120  ,
> > > > > match=(nd_ns), action=(next;)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=33   ,
> > > > > match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
> > > > > > > > > > > > > > +  table=??(lr_out_snat        ), priority=35   ,
> > > > > match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range),
> > > > > action=(next;)
> > > > > > > > > > > > > >   ])
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   # Stateful FIP with ALLOWED_IPs
> > > > > > > > > > > > > > @@ -1059,2
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev@openvswitch.org
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > >
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
diff mbox series

Patch

diff --git a/include/ovn/logical-fields.h b/include/ovn/logical-fields.h
index c9675f81c..2118f7933 100644
--- a/include/ovn/logical-fields.h
+++ b/include/ovn/logical-fields.h
@@ -70,6 +70,7 @@  enum mff_log_flags_bits {
     MLF_LOOKUP_FDB_BIT = 8,
     MLF_SKIP_SNAT_FOR_LB_BIT = 9,
     MLF_LOCALPORT_BIT = 10,
+    MLF_USE_SNAT_ZONE = 11,
 };
 
 /* MFF_LOG_FLAGS_REG flag assignments */
diff --git a/lib/logical-fields.c b/lib/logical-fields.c
index 7b3d431e0..352a48c89 100644
--- a/lib/logical-fields.c
+++ b/lib/logical-fields.c
@@ -125,6 +125,10 @@  ovn_init_symtab(struct shash *symtab)
              MLF_SKIP_SNAT_FOR_LB_BIT);
     expr_symtab_add_subfield(symtab, "flags.skip_snat_for_lb", NULL,
                              flags_str);
+    snprintf(flags_str, sizeof flags_str, "flags[%d]",
+             MLF_USE_SNAT_ZONE);
+    expr_symtab_add_subfield(symtab, "flags.use_snat_zone", NULL,
+                             flags_str);
 
     /* Connection tracking state. */
     expr_symtab_add_field_scoped(symtab, "ct_mark", MFF_CT_MARK, NULL, false,
diff --git a/northd/northd.c b/northd/northd.c
index 0ff61deec..e4d051a94 100644
--- a/northd/northd.c
+++ b/northd/northd.c
@@ -159,11 +159,14 @@  enum ovn_stage {
     PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,     18, "lr_in_arp_request")  \
                                                                       \
     /* Logical router egress stages. */                               \
-    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,      0, "lr_out_undnat")        \
-    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT, 1, "lr_out_post_undnat")   \
-    PIPELINE_STAGE(ROUTER, OUT, SNAT,        2, "lr_out_snat")          \
-    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,    3, "lr_out_egr_loop")      \
-    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,    4, "lr_out_delivery")
+    PIPELINE_STAGE(ROUTER, OUT, CHECK_DNAT_LOCAL,   0,                       \
+                   "lr_out_chk_dnat_local")                                  \
+    PIPELINE_STAGE(ROUTER, OUT, UNDNAT,             1, "lr_out_undnat")      \
+    PIPELINE_STAGE(ROUTER, OUT, POST_UNDNAT,        2, "lr_out_post_undnat") \
+    PIPELINE_STAGE(ROUTER, OUT, SNAT,               3, "lr_out_snat")        \
+    PIPELINE_STAGE(ROUTER, OUT, POST_SNAT,          4, "lr_out_post_snat")   \
+    PIPELINE_STAGE(ROUTER, OUT, EGR_LOOP,           5, "lr_out_egr_loop")    \
+    PIPELINE_STAGE(ROUTER, OUT, DELIVERY,           6, "lr_out_delivery")
 
 #define PIPELINE_STAGE(DP_TYPE, PIPELINE, STAGE, TABLE, NAME)   \
     S_##DP_TYPE##_##PIPELINE##_##STAGE                          \
@@ -210,6 +213,7 @@  enum ovn_stage {
 #define REGBIT_PKT_LARGER        "reg9[1]"
 #define REGBIT_LOOKUP_NEIGHBOR_RESULT "reg9[2]"
 #define REGBIT_LOOKUP_NEIGHBOR_IP_RESULT "reg9[3]"
+#define REGBIT_DST_NAT_IP_LOCAL "reg9[4]"
 
 /* Register to store the eth address associated to a router port for packets
  * received in S_ROUTER_IN_ADMISSION.
@@ -9568,9 +9572,10 @@  build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
                                     undnat_match_p, est_actions,
                                     &lb->nlb->header_);
         } else {
-            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
-                                    undnat_match_p, "ct_dnat;",
-                                    &lb->nlb->header_);
+            ovn_lflow_add_with_hint(
+                lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p,
+                od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;",
+                &lb->nlb->header_);
         }
         free(undnat_match_p);
 next:
@@ -9865,7 +9870,7 @@  lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
         uint16_t priority;
 
         /* Priority of logical flows corresponding to exempted_ext_ips is
-         * +1 of the corresponding regulr NAT rule.
+         * +2 of the corresponding regular NAT rule.
          * For example, if we have following NAT rule and we associate
          * exempted external ips to it:
          * "ovn-nbctl lr-nat-add router dnat_and_snat 10.15.24.139 50.0.0.11"
@@ -9873,17 +9878,17 @@  lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
          * And now we associate exempted external ip address set to it.
          * Now corresponding to above rule we will have following logical
          * flows:
-         * lr_out_snat...priority=162, match=(..ip4.dst == $exempt_range),
+         * lr_out_snat...priority=163, match=(..ip4.dst == $exempt_range),
          *                             action=(next;)
          * lr_out_snat...priority=161, match=(..), action=(ct_snat(....);)
          *
          */
         if (is_src) {
             /* S_ROUTER_IN_DNAT uses priority 100 */
-            priority = 100 + 1;
+            priority = 100 + 2;
         } else {
             /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
-            priority = count_1bits(ntohl(mask)) + 2;
+            priority = count_1bits(ntohl(mask)) + 3;
 
             if (!od->is_gw_router) {
                 priority += 128;
@@ -12268,9 +12273,9 @@  build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
         /* Traffic received on l3dgw_port is subject to NAT. */
         ds_clear(match);
         ds_clear(actions);
-        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
-                      is_v6 ? "6" : "4", nat->external_ip,
-                      od->l3dgw_ports[0]->json_key);
+        ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
+                      "flags.loopback == 0", is_v6 ? "6" : "4",
+                      nat->external_ip, od->l3dgw_ports[0]->json_key);
         if (!distributed && od->n_l3dgw_ports) {
             /* Flows for NAT rules that are centralized are only
             * programmed on the gateway chassis. */
@@ -12282,12 +12287,31 @@  build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
             ds_put_format(actions, "ip%s.dst=%s; next;",
                           is_v6 ? "6" : "4", nat->logical_ip);
         } else {
-            ds_put_cstr(actions, "ct_snat;");
+            ds_put_cstr(actions, "ct_snat_in_czone;");
         }
 
         ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
                                 100, ds_cstr(match), ds_cstr(actions),
                                 &nat->header_);
+
+        if (!stateless) {
+            ds_clear(match);
+            ds_clear(actions);
+            ds_put_format(match, "ip && ip%s.dst == %s && inport == %s && "
+                          "flags.loopback == 1 && flags.use_snat_zone == 1",
+                          is_v6 ? "6" : "4", nat->external_ip,
+                          od->l3dgw_ports[0]->json_key);
+            if (!distributed && od->n_l3dgw_ports) {
+                /* Flows for NAT rules that are centralized are only
+                * programmed on the gateway chassis. */
+                ds_put_format(match, " && is_chassis_resident(%s)",
+                            od->l3dgw_ports[0]->cr_port->json_key);
+            }
+            ds_put_cstr(actions, "ct_snat;");
+            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_UNSNAT,
+                                    100, ds_cstr(match), ds_cstr(actions),
+                                    &nat->header_);
+        }
     }
 }
 
@@ -12364,7 +12388,7 @@  build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
                 ds_put_format(actions, "ip%s.dst=%s; next;",
                               is_v6 ? "6" : "4", nat->logical_ip);
             } else {
-                ds_put_format(actions, "ct_dnat(%s", nat->logical_ip);
+                ds_put_format(actions, "ct_dnat_in_czone(%s", nat->logical_ip);
                 if (nat->external_port_range[0]) {
                     ds_put_format(actions, ",%s", nat->external_port_range);
                 }
@@ -12417,7 +12441,8 @@  build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
         ds_put_format(actions, "ip%s.src=%s; next;",
                       is_v6 ? "6" : "4", nat->external_ip);
     } else {
-        ds_put_format(actions, "ct_dnat;");
+        ds_put_format(actions,
+                      od->is_gw_router ? "ct_dnat;" : "ct_dnat_in_czone;");
     }
 
     ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 100,
@@ -12425,6 +12450,36 @@  build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
                             &nat->header_);
 }
 
+static void
+build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
+                                const struct nbrec_nat *nat, struct ds *match,
+                                struct ds *actions, bool distributed,
+                                bool is_v6)
+{
+    /* Note that this only applies for NAT on a distributed router.
+     */
+    if (!od->n_l3dgw_ports) {
+        return;
+    }
+
+    ds_clear(match);
+    ds_put_format(match, "ip && ip%s.dst == %s && ",
+                  is_v6 ? "6" : "4", nat->external_ip);
+    if (distributed) {
+        ds_put_format(match, "is_chassis_resident(\"%s\")", nat->logical_port);
+    } else {
+        ds_put_format(match, "is_chassis_resident(%s)",
+                      od->l3dgw_ports[0]->cr_port->json_key);
+    }
+
+    ds_clear(actions);
+    ds_put_cstr(actions, REGBIT_DST_NAT_IP_LOCAL" = 1; next;");
+
+    ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL,
+                            50, ds_cstr(match), ds_cstr(actions),
+                            &nat->header_);
+}
+
 static void
 build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
                             const struct nbrec_nat *nat, struct ds *match,
@@ -12478,16 +12533,19 @@  build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
         ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
                       is_v6 ? "6" : "4", nat->logical_ip,
                       od->l3dgw_ports[0]->json_key);
-        if (!distributed && od->n_l3dgw_ports) {
-            /* Flows for NAT rules that are centralized are only
-            * programmed on the gateway chassis. */
-            priority += 128;
-            ds_put_format(match, " && is_chassis_resident(%s)",
-                          od->l3dgw_ports[0]->cr_port->json_key);
-        } else if (distributed) {
-            priority += 128;
-            ds_put_format(match, " && is_chassis_resident(\"%s\")",
-                          nat->logical_port);
+        if (od->n_l3dgw_ports) {
+            if (distributed) {
+                ovs_assert(nat->logical_port);
+                priority += 128;
+                ds_put_format(match, " && is_chassis_resident(\"%s\")",
+                              nat->logical_port);
+            } else {
+                /* Flows for NAT rules that are centralized are only
+                * programmed on the gateway chassis. */
+                priority += 128;
+                ds_put_format(match, " && is_chassis_resident(%s)",
+                              od->l3dgw_ports[0]->cr_port->json_key);
+            }
         }
         ds_clear(actions);
 
@@ -12505,7 +12563,7 @@  build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
             ds_put_format(actions, "ip%s.src=%s; next;",
                           is_v6 ? "6" : "4", nat->external_ip);
         } else {
-            ds_put_format(actions, "ct_snat(%s",
+            ds_put_format(actions, "ct_snat_in_czone(%s",
                         nat->external_ip);
             if (nat->external_port_range[0]) {
                 ds_put_format(actions, ",%s", nat->external_port_range);
@@ -12519,6 +12577,24 @@  build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
         ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
                                 priority, ds_cstr(match),
                                 ds_cstr(actions), &nat->header_);
+
+        if (!stateless) {
+            ds_put_cstr(match, " && "REGBIT_DST_NAT_IP_LOCAL" == 1");
+            ds_clear(actions);
+            if (distributed) {
+                ds_put_format(actions, "eth.src = "ETH_ADDR_FMT"; ",
+                              ETH_ADDR_ARGS(mac));
+            }
+            ds_put_format(actions,  REGBIT_DST_NAT_IP_LOCAL" = 0; ct_snat(%s",
+                          nat->external_ip);
+            if (nat->external_port_range[0]) {
+                ds_put_format(actions, ",%s", nat->external_port_range);
+            }
+            ds_put_format(actions, ");");
+            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_SNAT,
+                                    priority + 1, ds_cstr(match),
+                                    ds_cstr(actions), &nat->header_);
+        }
     }
 }
 
@@ -12749,10 +12825,13 @@  build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
     /* Packets are allowed by default. */
     ovn_lflow_add(lflows, od, S_ROUTER_IN_DEFRAG, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 0, "1", "next;");
+    ovn_lflow_add(lflows, od, S_ROUTER_OUT_CHECK_DNAT_LOCAL, 0, "1",
+                  REGBIT_DST_NAT_IP_LOCAL" = 0; next;");
     ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 0, "1", "next;");
+    ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_SNAT, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_OUT_EGR_LOOP, 0, "1", "next;");
     ovn_lflow_add(lflows, od, S_ROUTER_IN_ECMP_STATEFUL, 0, "1", "next;");
 
@@ -12765,8 +12844,7 @@  build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
      * not committed, it would produce ongoing datapath flows with the ct.new
      * flag set. Some NICs are unable to offload these flows.
      */
-    if ((od->is_gw_router || od->n_l3dgw_ports) &&
-        (od->nbr->n_nat || od->has_lb_vip)) {
+    if (od->is_gw_router && (od->nbr->n_nat || od->has_lb_vip)) {
         ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
                       "ip", "flags.loopback = 1; ct_dnat;");
         ovn_lflow_add(lflows, od, S_ROUTER_OUT_POST_UNDNAT, 50,
@@ -12839,6 +12917,10 @@  build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
             }
         }
 
+        /* S_ROUTER_OUT_DNAT_LOCAL */
+        build_lrouter_out_is_dnat_local(lflows, od, nat, match, actions,
+                                        distributed, is_v6);
+
         /* S_ROUTER_OUT_UNDNAT */
         build_lrouter_out_undnat_flow(lflows, od, nat, match, actions, distributed,
                                       mac, is_v6);
@@ -12912,7 +12994,8 @@  build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
                           "clone { ct_clear; "
                           "inport = outport; outport = \"\"; "
                           "eth.dst <-> eth.src; "
-                          "flags = 0; flags.loopback = 1; ");
+                          "flags = 0; flags.loopback = 1; "
+                          "flags.use_snat_zone = "REGBIT_DST_NAT_IP_LOCAL"; ");
             for (int j = 0; j < MFF_N_LOG_REGS; j++) {
                 ds_put_format(actions, "reg%d = 0; ", j);
             }
diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
index 21d83718c..e39e6e805 100644
--- a/northd/ovn-northd.8.xml
+++ b/northd/ovn-northd.8.xml
@@ -2879,23 +2879,65 @@  icmp6 {
         <p>
           For each configuration in the OVN Northbound database, that asks
           to change the source IP address of a packet from <var>A</var> to
-          <var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
-          ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code> or
-          <code>ip &amp;&amp;
-          ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var></code>
-          where <var>GW</var> is the logical router gateway port, with an
-          action <code>ct_snat;</code>. If the NAT rule is of type
-          dnat_and_snat and has <code>stateless=true</code> in the
-          options, then the action would be <code>ip4/6.dst=
-          (<var>B</var>)</code>.
+          <var>B</var>, two priority-100 flows are added.
         </p>
 
         <p>
           If the NAT rule cannot be handled in a distributed manner, then
-          the priority-100 flow above is only programmed on the
+          the below priority-100 flows are only programmed on the
           gateway chassis.
         </p>
 
+        <ul>
+          <li>
+            <p>
+              The first flow matches <code>ip &amp;&amp;
+              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
+              &amp;&amp; flags.loopback == 0</code> or
+              <code>ip &amp;&amp;
+              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
+              &amp;&amp; flags.loopback == 0</code>
+              where <var>GW</var> is the logical router gateway port, with an
+              action <code>ct_snat_in_czone;</code> to unSNAT in the common
+              zone.  If the NAT rule is of type dnat_and_snat and has
+              <code>stateless=true</code> in the options, then the action
+              would be <code>ip4/6.dst=(<var>B</var>)</code>.
+            </p>
+
+            <p>
+              If the NAT entry is of type <code>snat</code>, then there is an
+              additional match <code>is_chassis_resident(<var>cr-GW</var>)
+              </code> where <var>cr-GW</var> is the chassis resident port of
+              <var>GW</var>.
+            </p>
+          </li>
+
+          <li>
+            <p>
+              The second flow matches <code>ip &amp;&amp;
+              ip4.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
+              &amp;&amp; flags.loopback == 1 &amp;&amp;
+              flags.use_snat_zone == 1</code> or
+              <code>ip &amp;&amp;
+              ip6.dst == <var>B</var> &amp;&amp; inport == <var>GW</var>
+              &amp;&amp; flags.loopback == 0 &amp;&amp;
+              flags.use_snat_zone == 1</code>
+              where <var>GW</var> is the logical router gateway port, with an
+              action <code>ct_snat;</code> to unSNAT in the snat zone. If the
+              NAT rule is of type dnat_and_snat and has
+              <code>stateless=true</code> in the options, then the action
+              would be <code>ip4/6.dst=(<var>B</var>)</code>.
+            </p>
+
+            <p>
+              If the NAT entry is of type <code>snat</code>, then there is an
+              additional match <code>is_chassis_resident(<var>cr-GW</var>)
+              </code> where <var>cr-GW</var> is the chassis resident port of
+              <var>GW</var>.
+            </p>
+          </li>
+        </ul>
+
         <p>
           A priority-0 logical flow with match <code>1</code> has actions
           <code>next;</code>.
@@ -4031,7 +4073,43 @@  nd_ns {
       </li>
     </ul>
 
-    <h3>Egress Table 0: UNDNAT</h3>
+    <h3>Egress Table 0: Check DNAT local </h3>
+
+    <p>
+      This table checks if the packet needs to be DNATed in the router ingress
+      table <code>lr_in_dnat</code> after it is SNATed  and looped back
+      to the ingress pipeline.  This check is done only for routers configured
+      with distributed gateway ports and NAT entries.  This check is done
+      so that SNAT and DNAT is done in different zones instead of a common
+      zone.
+    </p>
+
+    <ul>
+      <li>
+        <p>
+          For each NAT rule in the OVN Northbound database on a
+          distributed router, a priority-50 logical flow with match
+          <code>ip4.dst == <var>E</var> &amp;&amp;
+          is_chassis_resident(<var>P</var>)</code>, where <var>E</var> is the
+          external IP address specified in the NAT rule, <var>GW</var>
+          is the logical router distributed gateway port. For dnat_and_snat
+          NAT rule, <var>P</var> is the logical port specified in the NAT rule.
+          If <ref column="logical_port"
+          table="NAT" db="OVN_Northbound"/> column of
+          <ref table="NAT" db="OVN_Northbound"/> table is NOT set, then
+          <var>P</var> is the <code>chassisredirect port</code> of
+          <var>GW</var> with the actions:
+          <code>REGBIT_DST_NAT_IP_LOCAL = 1; next; </code>
+        </p>
+      </li>
+
+      <li>
+        A priority-0 logical flow with match <code>1</code> has actions
+        <code>REGBIT_DST_NAT_IP_LOCAL = 0; next;</code>.
+      </li>
+    </ul>
+
+    <h3>Egress Table 1: UNDNAT</h3>
 
     <p>
       This is for already established connections' reverse traffic.
@@ -4040,6 +4118,23 @@  nd_ns {
       is unDNATed here.
     </p>
 
+    <ul>
+      <li>
+        A priority-0 logical flow with match <code>1</code> has actions
+        <code>next;</code>.
+      </li>
+    </ul>
+
+    <h3>Egress Table 1: UNDNAT on Gateway Routers</h3>
+
+    <ul>
+      <li>
+        For all IP packets, a priority-50 flow with an action
+        <code>flags.loopback = 1; ct_dnat;</code>.
+      </li>
+    </ul>
+
+    <h3>Egress Table 1: UNDNAT on Distributed Routers</h3>
     <ul>
       <li>
         <p>
@@ -4050,9 +4145,9 @@  nd_ns {
           gateway chassis that matches
           <code>ip &amp;&amp; ip4.src == <var>B</var> &amp;&amp;
           outport == <var>GW</var></code>, where <var>GW</var> is the logical
-          router gateway port with an action <code>ct_dnat;</code>. If the
-          backend IPv4 address <var>B</var> is also configured with L4 port
-          <var>PORT</var> of protocol <var>P</var>, then the
+          router gateway port with an action <code>ct_dnat_in_czone;</code>.
+          If the backend IPv4 address <var>B</var> is also configured with
+          L4 port <var>PORT</var> of protocol <var>P</var>, then the
           match also includes <code>P.src</code> == <var>PORT</var>.  These
           flows are not added for load balancers with IPv6 <var>VIPs</var>.
         </p>
@@ -4072,7 +4167,7 @@  nd_ns {
           matches <code>ip &amp;&amp; ip4.src == <var>B</var>
           &amp;&amp; outport == <var>GW</var></code>, where <var>GW</var>
           is the logical router gateway port, with an action
-          <code>ct_dnat;</code>. If the NAT rule is of type
+          <code>ct_dnat_in_czone;</code>. If the NAT rule is of type
           dnat_and_snat and has <code>stateless=true</code> in the
           options, then the action would be <code>ip4/6.src=
           (<var>B</var>)</code>.
@@ -4081,7 +4176,7 @@  nd_ns {
         <p>
           If the NAT rule cannot be handled in a distributed manner, then
           the priority-100 flow above is only programmed on the
-          gateway chassis.
+          gateway chassis with the action <code>ct_dnat_in_czone</code>.
         </p>
 
         <p>
@@ -4094,26 +4189,17 @@  nd_ns {
         </p>
       </li>
 
-      <li>
-        For all IP packets, a priority-50 flow with an action
-        <code>flags.loopback = 1; ct_dnat;</code>.
-      </li>
-
-      <li>
-        A priority-0 logical flow with match <code>1</code> has actions
-        <code>next;</code>.
-      </li>
     </ul>
 
-    <h3>Egress Table 1: Post UNDNAT</h3>
+    <h3>Egress Table 2: Post UNDNAT</h3>
 
     <p>
       <ul>
         <li>
           A priority-50 logical flow is added that commits any untracked flows
-          from the previous table <code>lr_out_undnat</code>. This flow
-          matches on <code>ct.new &amp;&amp; ip</code> with action
-          <code>ct_commit { } ; next; </code>.
+          from the previous table <code>lr_out_undnat</code> for Gateway
+          routers.  This flow matches on <code>ct.new &amp;&amp; ip</code>
+          with action <code>ct_commit { } ; next; </code>.
         </li>
 
         <li>
@@ -4124,7 +4210,7 @@  nd_ns {
       </ul>
     </p>
 
-    <h3>Egress Table 2: SNAT</h3>
+    <h3>Egress Table 3: SNAT</h3>
 
     <p>
       Packets that are configured to be SNATed get their source IP address
@@ -4140,7 +4226,7 @@  nd_ns {
       </li>
     </ul>
 
-    <p>Egress Table 2: SNAT on Gateway Routers</p>
+    <p>Egress Table 3: SNAT on Gateway Routers</p>
 
     <ul>
       <li>
@@ -4239,7 +4325,7 @@  nd_ns {
       </li>
     </ul>
 
-    <p>Egress Table 2: SNAT on Distributed Routers</p>
+    <p>Egress Table 3: SNAT on Distributed Routers</p>
 
     <ul>
       <li>
@@ -4247,28 +4333,47 @@  nd_ns {
           For each configuration in the OVN Northbound database, that asks
           to change the source IP address of a packet from an IP address of
           <var>A</var> or to change the source IP address of a packet that
-          belongs to network <var>A</var> to <var>B</var>, a flow matches
-          <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
-          outport == <var>GW</var></code>, where <var>GW</var> is the
-          logical router gateway port, with an action
-          <code>ct_snat(<var>B</var>);</code>.  The priority of the flow
-          is calculated based on the mask of <var>A</var>, with matches
-          having larger masks getting higher priorities. If the NAT rule
-          is of type dnat_and_snat and has <code>stateless=true</code>
-          in the options, then the action would be <code>ip4/6.src=
-          (<var>B</var>)</code>.
+          belongs to network <var>A</var> to <var>B</var>, two flows are
+          added.  The priority <var>P</var> of these flows are calculated
+          based on the mask of <var>A</var>, with matches having larger
+          masks getting higher priorities.
         </p>
 
         <p>
           If the NAT rule cannot be handled in a distributed manner, then
-          the flow above is only programmed on the
-          gateway chassis increasing flow priority by 128 in
-          order to be run first
+          the below flows are only programmed on the gateway chassis increasing
+          flow priority by 128 in order to be run first.
         </p>
 
+        <ul>
+          <li>
+            The first flow is added with the calculated priority <var>P</var>
+            and match <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
+            outport == <var>GW</var></code>, where <var>GW</var> is the
+            logical router gateway port, with an action
+            <code>ct_snat_in_czone(<var>B</var>);</code> to SNATed in the
+            common zone.  If the NAT rule is of type dnat_and_snat and has
+            <code>stateless=true</code> in the options, then the action
+            would be <code>ip4/6.src=(<var>B</var>)</code>.
+          </li>
+
+          <li>
+            The second flow is added with the calculated priority
+            <code><var>P</var> + 1 </code> and match
+            <code>ip &amp;&amp; ip4.src == <var>A</var> &amp;&amp;
+            outport == <var>GW</var> &amp;&amp;
+            REGBIT_DST_NAT_IP_LOCAL == 0</code>, where <var>GW</var> is the
+            logical router gateway port, with an action
+            <code>ct_snat(<var>B</var>);</code> to SNAT in the snat zone.
+            If the NAT rule is of type dnat_and_snat and has
+            <code>stateless=true</code> in the options, then the action would
+            be <code>ip4/6.src=(<var>B</var>)</code>.
+          </li>
+        </ul>
+
         <p>
           If the NAT rule can be handled in a distributed manner, then
-          there is an additional action
+          there is an additional action (for both the flows)
           <code>eth.src = <var>EA</var>;</code>, where <var>EA</var>
           is the ethernet address associated with the IP address
           <var>A</var> in the NAT rule.  This allows upstream MAC
@@ -4284,7 +4389,8 @@  nd_ns {
 
         <p>
           If the NAT rule has <code>exempted_ext_ips</code> set, then
-          there is an additional flow configured at the priority + 1 of
+          there is an additional flow configured at the priority
+          <code><var>P</var> + 2 </code> of
           corresponding NAT rule. The flow matches if destination ip
           is an <code>exempted_ext_ip</code> and the action is <code>next;
           </code>. This flow is used to bypass the ct_snat action for a flow
@@ -4299,7 +4405,7 @@  nd_ns {
       </li>
     </ul>
 
-    <h3>Egress Table 3: Egress Loopback</h3>
+    <h3>Egress Table 4: Egress Loopback</h3>
 
     <p>
       For distributed logical routers where one of the logical router
@@ -4344,6 +4450,7 @@  clone {
     outport = "";
     flags = 0;
     flags.loopback = 1;
+    flags.use_snat_zone = REGBIT_DST_NAT_IP_LOCAL;
     reg0 = 0;
     reg1 = 0;
     ...
@@ -4368,7 +4475,7 @@  clone {
       </li>
     </ul>
 
-    <h3>Egress Table 4: Delivery</h3>
+    <h3>Egress Table 5: Delivery</h3>
 
     <p>
       Packets that reach this table are ready for delivery.  It contains:
diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
index 85b47a18f..70ec5e2e3 100644
--- a/tests/ovn-northd.at
+++ b/tests/ovn-northd.at
@@ -877,25 +877,25 @@  check_flow_match_sets() {
 echo
 echo "IPv4: stateful"
 ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
-check_flow_match_sets 2 2 3 0 0 0 0
+check_flow_match_sets 3 4 2 0 0 0 0
 ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
 
 echo
 echo "IPv4: stateless"
 ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11
-check_flow_match_sets 2 0 1 2 2 0 0
+check_flow_match_sets 2 0 0 2 2 0 0
 ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
 
 echo
 echo "IPv6: stateful"
 ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
-check_flow_match_sets 2 2 3 0 0 0 0
+check_flow_match_sets 3 4 2 0 0 0 0
 ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
 
 echo
 echo "IPv6: stateless"
 ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
-check_flow_match_sets 2 0 1 0 0 2 2
+check_flow_match_sets 2 0 0 0 0 2 2
 
 AT_CLEANUP
 ])
@@ -924,9 +924,9 @@  echo "CR-LRP UUID is: " $uuid
 ovn-nbctl --portrange lr-nat-add R1 dnat_and_snat  172.16.1.1 50.0.0.11 1-3000
 
 AT_CAPTURE_FILE([sbflows])
-OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 2 = `grep -c lr_in_unsnat sbflows`])
+OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows && test 3 = `grep -c lr_in_unsnat sbflows`])
 AT_CHECK([grep -c 'ct_snat.*3000' sbflows && grep -c 'ct_dnat.*3000' sbflows],
-  [0], [1
+  [0], [2
 1
 ])
 
@@ -934,9 +934,9 @@  ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
 ovn-nbctl --wait=sb --portrange lr-nat-add R1 snat  172.16.1.1 50.0.0.11 1-3000
 
 AT_CAPTURE_FILE([sbflows2])
-OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 2 = `grep -c lr_in_unsnat sbflows`])
+OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows2 && test 3 = `grep -c lr_in_unsnat sbflows`])
 AT_CHECK([grep -c 'ct_snat.*3000' sbflows2 && grep -c 'ct_dnat.*3000' sbflows2],
-  [1], [1
+  [1], [2
 0
 ])
 
@@ -944,7 +944,7 @@  ovn-nbctl lr-nat-del R1 snat  172.16.1.1
 ovn-nbctl --wait=sb --portrange --stateless lr-nat-add R1 dnat_and_snat  172.16.1.2 50.0.0.12 1-3000
 
 AT_CAPTURE_FILE([sbflows3])
-OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 3 = `grep -c lr_in_unsnat sbflows3`])
+OVS_WAIT_UNTIL([ovn-sbctl dump-flows R1 > sbflows3 && test 4 = `grep -c lr_in_unsnat sbflows3`])
 AT_CHECK([grep 'ct_[s]dnat.*172\.16\.1\.2.*3000' sbflows3], [1])
 
 ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
@@ -1008,17 +1008,20 @@  AT_CAPTURE_FILE([drflows])
 ovn-sbctl dump-flows CR > crflows
 AT_CAPTURE_FILE([crflows])
 
-AT_CHECK([
-  grep -c lr_out_snat drflows
-  grep -c lr_out_snat crflows
-  grep lr_out_snat drflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"
-  grep lr_out_snat crflows | grep "ip4.src == 50.0.0.11" | grep -c "ip4.dst == $allowed_range"], [0], [dnl
-3
-3
-1
-1
+AT_CHECK([grep -e "lr_out_snat" drflows | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.1);)
+  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
+])
+
+AT_CHECK([grep -e "lr_out_snat" crflows | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.1);)
 ])
 
+
 # SNAT with DISALLOWED_IPs
 check ovn-nbctl lr-nat-del DR snat  50.0.0.11
 check ovn-nbctl lr-nat-del CR snat  50.0.0.11
@@ -1036,19 +1039,19 @@  AT_CAPTURE_FILE([drflows2])
 ovn-sbctl dump-flows CR > crflows2
 AT_CAPTURE_FILE([crflows2])
 
-AT_CHECK([
-  grep -c lr_out_snat drflows2
-  grep -c lr_out_snat crflows2
-  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=162"
-  grep lr_out_snat drflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=161"
-  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep -c "priority=34"
-  grep lr_out_snat crflows2 | grep "ip4.src == 50.0.0.11" | grep -c "priority=33"], [0], [dnl
-4
-4
-1
-1
-1
-1
+AT_CHECK([grep -e "lr_out_snat" drflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.1);)
+  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.1);)
+  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
+])
+
+AT_CHECK([grep -e "lr_out_snat" crflows2 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.1);)
+  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
 ])
 
 # Stateful FIP with ALLOWED_IPs
@@ -1059,25 +1062,24 @@  check ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
 check ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
 
 check ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
-check ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
+check ovn-nbctl --wait=sb lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
 
-ovn-nbctl show DR
-ovn-sbctl dump-flows DR
-ovn-nbctl show CR
-ovn-sbctl dump-flows CR
-
-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
-wc -l`])
-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
-wc -l`])
+ovn-sbctl dump-flows DR > drflows3
+AT_CAPTURE_FILE([drflows2])
+ovn-sbctl dump-flows CR > crflows3
+AT_CAPTURE_FILE([crflows2])
 
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
+AT_CHECK([grep -e "lr_out_snat" drflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ct_snat_in_czone(172.16.1.2);)
+  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
 ])
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
+
+AT_CHECK([grep -e "lr_out_snat" crflows3 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ct_snat(172.16.1.2);)
 ])
 
 # Stateful FIP with DISALLOWED_IPs
@@ -1088,26 +1090,26 @@  ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.2 50.0.0.11
 ovn-nbctl lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
 
 ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
-ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
+check ovn-nbctl --wait=sb --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
 
-ovn-nbctl show DR
-ovn-sbctl dump-flows DR
-ovn-nbctl show CR
-ovn-sbctl dump-flows CR
-
-OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
-wc -l`])
-OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
-wc -l`])
+ovn-sbctl dump-flows DR > drflows4
+AT_CAPTURE_FILE([drflows2])
+ovn-sbctl dump-flows CR > crflows4
+AT_CAPTURE_FILE([crflows2])
 
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
+AT_CHECK([grep -e "lr_out_snat" drflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ct_snat_in_czone(172.16.1.2);)
+  table=??(lr_out_snat        ), priority=162  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.16.1.2);)
+  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
 ])
 
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
+AT_CHECK([grep -e "lr_out_snat" crflows4 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ct_snat(172.16.1.2);)
+  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
 ])
 
 # Stateless FIP with DISALLOWED_IPs
@@ -1120,24 +1122,21 @@  ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
 ovn-nbctl lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 allowed_range
 ovn-nbctl lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 allowed_range
 
-ovn-nbctl show DR
-ovn-sbctl dump-flows DR
-
-ovn-nbctl show CR
-ovn-sbctl dump-flows CR
-
-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
-wc -l`])
-OVS_WAIT_UNTIL([test 3 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
-wc -l`])
+ovn-sbctl dump-flows DR > drflows5
+AT_CAPTURE_FILE([drflows2])
+ovn-sbctl dump-flows CR > crflows5
+AT_CAPTURE_FILE([crflows2])
 
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
+AT_CHECK([grep -e "lr_out_snat" drflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
 ])
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $allowed_range" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $allowed_range" | wc -l], [0], [1
+
+AT_CHECK([grep -e "lr_out_snat" crflows5 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $allowed_range), action=(ip4.src=172.16.1.2; next;)
 ])
 
 # Stateful FIP with DISALLOWED_IPs
@@ -1150,23 +1149,25 @@  ovn-nbctl --stateless lr-nat-add CR dnat_and_snat  172.16.1.2 50.0.0.11
 ovn-nbctl --is-exempted lr-nat-update-ext-ip DR dnat_and_snat 172.16.1.2 disallowed_range
 ovn-nbctl --is-exempted lr-nat-update-ext-ip CR dnat_and_snat 172.16.1.2 disallowed_range
 
-ovn-nbctl show DR
-ovn-sbctl dump-flows DR
-ovn-nbctl show CR
-ovn-sbctl dump-flows CR
+ovn-nbctl --wait=sb sync
 
-OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows DR | grep lr_out_snat | \
-wc -l`])
-OVS_WAIT_UNTIL([test 4 = `ovn-sbctl dump-flows CR | grep lr_out_snat | \
-wc -l`])
+ovn-sbctl dump-flows DR > drflows6
+AT_CAPTURE_FILE([drflows2])
+ovn-sbctl dump-flows CR > crflows6
+AT_CAPTURE_FILE([crflows2])
 
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=162" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
+AT_CHECK([grep -e "lr_out_snat" drflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=161  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(ip4.src=172.16.1.2; next;)
+  table=??(lr_out_snat        ), priority=163  , match=(ip && ip4.src == 50.0.0.11 && outport == "DR-S1" && is_chassis_resident("cr-DR-S1") && ip4.dst == $disallowed_range), action=(next;)
 ])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_out_snat | grep "ip4.src == 50.0.0.11" | grep "ip4.dst == $disallowed_range" | grep "priority=34" | wc -l], [0], [1
-])
-AT_CHECK([ovn-sbctl dump-flows CR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.2" | grep "ip4.src == $disallowed_range" | grep "priority=101" | wc -l], [0], [1
+
+AT_CHECK([grep -e "lr_out_snat" crflows6 | sed 's/table=../table=??/' | sort], [0], [dnl
+  table=??(lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=??(lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=??(lr_out_snat        ), priority=33   , match=(ip && ip4.src == 50.0.0.11), action=(ip4.src=172.16.1.2; next;)
+  table=??(lr_out_snat        ), priority=35   , match=(ip && ip4.src == 50.0.0.11 && ip4.dst == $disallowed_range), action=(next;)
 ])
 
 AT_CLEANUP
@@ -3475,14 +3476,14 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(ct_lb(backends=10.0.0.40:8080);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
 check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="20.0.0.4 aef0::4"
@@ -3511,21 +3512,21 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
-  table=2 (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip4), action=(ct_snat(20.0.0.4);)
+  table=? (lr_out_snat        ), priority=100  , match=(flags.force_snat_for_lb == 1 && ip6), action=(ct_snat(aef0::4);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
 check ovn-nbctl --wait=sb set logical_router lr0 options:lb_force_snat_ip="router_ip"
@@ -3557,22 +3558,22 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
 check ovn-nbctl --wait=sb remove logical_router lr0 options chassis
@@ -3584,9 +3585,9 @@  AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
   table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
 check ovn-nbctl set logical_router lr0 options:chassis=ch1
@@ -3617,23 +3618,23 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.100 && tcp && reg9[[16..31]] == 80), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.100);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw1"), action=(ct_snat(20.0.0.1);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw1"), action=(ct_snat(bef0::1);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
 check ovn-nbctl --wait=sb lb-add lb2 10.0.0.20:80 10.0.0.40:8080
@@ -3661,18 +3662,18 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.0.0.20 && tcp && reg9[[16..31]] == 80), action=(flags.skip_snat_for_lb = 1; ct_lb(backends=10.0.0.40:8080);)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | grep skip_snat_for_lb | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=120  , match=(flags.skip_snat_for_lb == 1 && ip), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
 AT_CLEANUP
@@ -4176,6 +4177,8 @@  check ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1
 check ovn-nbctl lr-nat-add lr0 dnat 42.42.42.42 192.168.0.2
 check ovn-nbctl --wait=sb sync
 
+ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64'
+
 AT_CHECK([ovn-trace --minimal 'inport == "sw1-port1" && eth.src == 50:54:00:00:00:03 && eth.dst == 00:00:00:00:ff:02 && ip4.dst == 42.42.42.42 && ip4.src == 11.0.0.2 && ip.ttl == 64' | grep "output(\"sw0-port1\")"], [0], [ignore])
 
 dnl If we remove the DNAT entry we will be unable to trace to the DNAT address
@@ -4761,17 +4764,17 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
 # Create few dnat_and_snat entries
@@ -4797,17 +4800,21 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+])
+
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
 ovn-sbctl chassis-add gw1 geneve 127.0.0.1
@@ -4828,9 +4835,12 @@  AT_CAPTURE_FILE([lr0flows])
 
 AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
   table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
 ])
 
 AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
@@ -4839,26 +4849,34 @@  AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
 
 AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
-  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
+  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
+])
+
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
+  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
 ])
 
 # Associate load balancer to lr0
@@ -4879,9 +4897,12 @@  AT_CAPTURE_FILE([lr0flows])
 
 AT_CHECK([grep "lr_in_unsnat" lr0flows | sort], [0], [dnl
   table=4 (lr_in_unsnat       ), priority=0    , match=(1), action=(next;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
-  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.10 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 0 && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone;)
+  table=4 (lr_in_unsnat       ), priority=100  , match=(ip && ip4.dst == 172.168.0.30 && inport == "lr0-public" && flags.loopback == 1 && flags.use_snat_zone == 1 && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
 ])
 
 AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
@@ -4894,7 +4915,7 @@  AT_CHECK([grep "lr_in_defrag" lr0flows | sort], [0], [dnl
 
 AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=0    , match=(1), action=(next;)
-  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat(10.0.0.3);)
+  table=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 172.168.0.20 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone(10.0.0.3);)
   table=6 (lr_in_dnat         ), priority=110  , match=(ct.est && ip4 && reg0 == 172.168.0.200 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
   table=6 (lr_in_dnat         ), priority=110  , match=(ct.new && ip4 && reg0 == 172.168.0.200 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.80,10.0.0.81);)
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.0.0.10 && tcp && reg9[[16..31]] == 80 && ct_label.natted == 1 && is_chassis_resident("cr-lr0-public")), action=(next;)
@@ -4905,27 +4926,35 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.10 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.20 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
+  table=? (lr_out_chk_dnat_local), priority=50   , match=(ip && ip4.dst == 172.168.0.30 && is_chassis_resident("cr-lr0-public")), action=(reg9[[4]] = 1; next;)
+])
+
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=100  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
+  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.4 && tcp.src == 8080)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
+  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && tcp.src == 82) || (ip4.src == 10.0.0.60 && tcp.src == 82)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
+  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.50 && udp.src == 6062) || (ip4.src == 10.0.0.60 && udp.src == 6062)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
+  table=? (lr_out_undnat      ), priority=120  , match=(ip4 && ((ip4.src == 10.0.0.80) || (ip4.src == 10.0.0.81)) && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat_in_czone;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=153  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=154  , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=161  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat_in_czone(172.168.0.20);)
+  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.10 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=162  , match=(ip && ip4.src == 10.0.0.3 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public") && reg9[[4]] == 1), action=(reg9[[4]] = 0; ct_snat(172.168.0.20);)
 ])
 
 # Make the logical router as Gateway router
@@ -4965,22 +4994,26 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+])
+
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
 ])
 
 # Set lb force snat logical router.
@@ -5020,24 +5053,28 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
+])
+
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
 ])
 
 # Add a LB VIP same as router ip.
@@ -5081,24 +5118,28 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+])
+
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
 ])
 
 # Add IPv6 router port and LB.
@@ -5155,26 +5196,30 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip6 && xxreg0 == def0::2 && tcp && reg9[[16..31]] == 8000), action=(flags.force_snat_for_lb = 1; ct_lb(backends=[[aef0::2]]:80,[[aef0::3]]:80);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
+])
+
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
-  table=2 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
-  table=2 (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
-  table=2 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-public"), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "lr0-sw0"), action=(ct_snat(10.0.0.1);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-public"), action=(ct_snat(def0::10);)
+  table=? (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip6 && outport == "lr0-sw0"), action=(ct_snat(aef0::1);)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+  table=? (lr_out_snat        ), priority=25   , match=(ip && ip4.src == 10.0.0.0/24), action=(ct_snat(172.168.0.10);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.10), action=(ct_snat(172.168.0.30);)
+  table=? (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 10.0.0.3), action=(ct_snat(172.168.0.20);)
 ])
 
 check ovn-nbctl lrp-del lr0-sw0
@@ -5209,19 +5254,23 @@  AT_CHECK([grep "lr_in_dnat" lr0flows | sort], [0], [dnl
   table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 172.168.0.210 && udp && reg9[[16..31]] == 60), action=(flags.force_snat_for_lb = 1; ct_lb(backends=10.0.0.50:6062,10.0.0.60:6062);)
 ])
 
-AT_CHECK([grep "lr_out_undnat" lr0flows | sort], [0], [dnl
-  table=0 (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
-  table=0 (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
+AT_CHECK([grep "lr_out_chk_dnat_local" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_chk_dnat_local), priority=0    , match=(1), action=(reg9[[4]] = 0; next;)
+])
+
+AT_CHECK([grep "lr_out_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_undnat      ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_undnat      ), priority=50   , match=(ip), action=(flags.loopback = 1; ct_dnat;)
 ])
 
-AT_CHECK([grep "lr_out_post_undnat" lr0flows | sort], [0], [dnl
-  table=1 (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
-  table=1 (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
+AT_CHECK([grep "lr_out_post_undnat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_post_undnat ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_post_undnat ), priority=50   , match=(ip && ct.new), action=(ct_commit { } ; next; )
 ])
 
-AT_CHECK([grep "lr_out_snat" lr0flows | sort], [0], [dnl
-  table=2 (lr_out_snat        ), priority=0    , match=(1), action=(next;)
-  table=2 (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
+AT_CHECK([grep "lr_out_snat" lr0flows | sed 's/table=./table=?/' | sort], [0], [dnl
+  table=? (lr_out_snat        ), priority=0    , match=(1), action=(next;)
+  table=? (lr_out_snat        ), priority=120  , match=(nd_ns), action=(next;)
 ])
 
 AT_CLEANUP
diff --git a/tests/ovn.at b/tests/ovn.at
index 0d606b42f..ae5744407 100644
--- a/tests/ovn.at
+++ b/tests/ovn.at
@@ -21604,7 +21604,7 @@  AT_CAPTURE_FILE([sbflows])
 AT_CHECK([for regex in ct_snat ct_dnat ip4.dst= ip4.src=; do
   grep -c "$regex" sbflows;
 done], [0], [0
-1
+0
 2
 2
 ])
diff --git a/tests/system-ovn.at b/tests/system-ovn.at
index c9f5771c9..7f6cb32dc 100644
--- a/tests/system-ovn.at
+++ b/tests/system-ovn.at
@@ -2224,7 +2224,7 @@  ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
 
 ovn-nbctl list load_balancer
 ovn-sbctl dump-flows R2
-OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
+OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
 grep 'nat(src=20.0.0.2)'])
 
 dnl Test load-balancing that includes L4 ports in NAT.
@@ -2262,7 +2262,7 @@  ovn-nbctl set load_balancer $uuid vips:'"30.0.0.2:8000"'='"192.168.1.2:80,192.16
 
 ovn-nbctl list load_balancer
 ovn-sbctl dump-flows R2
-OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=42 | \
+OVS_WAIT_UNTIL([ovs-ofctl -O OpenFlow13 dump-flows br-int table=43 | \
 grep 'nat(src=20.0.0.2)'])
 
 rm -f wget*.log
@@ -3711,17 +3711,24 @@  sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
 icmpv6,orig=(src=fd20::2,dst=fd20::3,id=<cleared>,type=128,code=0),reply=(src=fd11::2,dst=fd20::2,id=<cleared>,type=129,code=0),zone=<cleared>
 ])
 
+AT_CHECK([ovs-appctl dpctl/flush-conntrack])
+
 # South-North SNAT: 'foo2' pings 'alice1'. But 'alice1' receives traffic
-# from 172.16.1.4
+# from fd20::4
 NS_CHECK_EXEC([foo2], [ping6 -q -c 3 -i 0.3 -w 2 fd20::2 | FORMAT_PING], \
 [0], [dnl
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that SNAT indeed happened via 'dump-conntrack' command.
+ovs-appctl dpctl/dump-conntrack | grep icmpv6
 AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd11::3) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
+])
+
+# We verify that SNAT indeed happened via 'dump-conntrack' command.
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
+sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
+icmpv6,orig=(src=fd11::3,dst=fd20::2,id=<cleared>,type=128,code=0),reply=(src=fd20::2,dst=fd20::4,id=<cleared>,type=129,code=0),zone=<cleared>
 ])
 
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
@@ -3861,11 +3868,9 @@  NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
+# We verify that the connection is not tracked.
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmp,orig=(src=192.168.1.2,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
 ])
 
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
@@ -3875,11 +3880,9 @@  NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 192.168.2.2 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
+# We verify that the connection is not tracked.
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmp,orig=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
 ])
 
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
@@ -3889,14 +3892,13 @@  NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 192.168.1.3 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(192.168.2.2) | \
+# We verify that the connection is not tracked.
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(192.168.2.2) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmp,orig=(src=192.168.2.2,dst=192.168.1.3,id=<cleared>,type=8,code=0),reply=(src=192.168.1.3,dst=192.168.2.2,id=<cleared>,type=0,code=0),zone=<cleared>
 ])
 
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
+
 # East-West NAT: 'foo1' pings 'bar1' using 172.16.1.4.
 NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
 [0], [dnl
@@ -3905,11 +3907,10 @@  NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
 
 # Check conntrack entries.  First SNAT of 'foo1' address happens.
 # Then DNAT of 'bar1' address happens (listed first below).
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.4) | \
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.4) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
 icmp,orig=(src=172.16.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
 icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.3,id=<cleared>,type=0,code=0),zone=<cleared>
-icmp,orig=(src=192.168.1.2,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
 ])
 
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
@@ -3922,7 +3923,7 @@  NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 172.16.1.4 | FORMAT_PING], \
 
 # Check conntrack entries.  First SNAT of 'foo2' address happens.
 # Then DNAT of 'bar1' address happens (listed first below).
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.1) | \
+AT_CHECK([ovs-appctl dpctl/dump-conntrack | grep icmp | FORMAT_CT(172.16.1.1) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
 icmp,orig=(src=172.16.1.1,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=192.168.2.2,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
 icmp,orig=(src=192.168.1.3,dst=172.16.1.4,id=<cleared>,type=8,code=0),reply=(src=172.16.1.4,dst=172.16.1.1,id=<cleared>,type=0,code=0),zone=<cleared>
@@ -4055,13 +4056,6 @@  NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
-sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmpv6,orig=(src=fd11::2,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
-])
-
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
 # East-West No NAT: 'foo2' pings 'bar1' using fd12::2.
 NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
@@ -4069,13 +4063,6 @@  NS_CHECK_EXEC([foo2], [ping -q -c 3 -i 0.3 -w 2 fd12::2 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
-sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmpv6,orig=(src=fd11::3,dst=fd12::2,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd11::3,id=<cleared>,type=129,code=0),zone=<cleared>
-])
-
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
 # East-West No NAT: 'bar1' pings 'foo2' using fd11::3.
 NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
@@ -4083,13 +4070,6 @@  NS_CHECK_EXEC([bar1], [ping -q -c 3 -i 0.3 -w 2 fd11::3 | FORMAT_PING], \
 3 packets transmitted, 3 received, 0% packet loss, time 0ms
 ])
 
-# We verify that the connection is tracked but not NATted. This is due to the
-# unDNAT table in the egress router pipeline
-AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd12::2) | \
-sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmpv6,orig=(src=fd12::2,dst=fd11::3,id=<cleared>,type=128,code=0),reply=(src=fd11::3,dst=fd12::2,id=<cleared>,type=129,code=0),zone=<cleared>
-])
-
 AT_CHECK([ovs-appctl dpctl/flush-conntrack])
 # East-West NAT: 'foo1' pings 'bar1' using fd20::4.
 NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
@@ -4101,7 +4081,6 @@  NS_CHECK_EXEC([foo1], [ping -q -c 3 -i 0.3 -w 2 fd20::4 | FORMAT_PING], \
 # Then DNAT of 'bar1' address happens (listed first below).
 AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(fd20::4) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
-icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd11::2,id=<cleared>,type=129,code=0),zone=<cleared>
 icmpv6,orig=(src=fd11::2,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd20::4,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
 icmpv6,orig=(src=fd20::3,dst=fd20::4,id=<cleared>,type=128,code=0),reply=(src=fd12::2,dst=fd20::3,id=<cleared>,type=129,code=0),zone=<cleared>
 ])
@@ -6037,7 +6016,6 @@  NS_CHECK_EXEC([sw01-x], [ping -q -c 3 -i 0.3 -w 2 172.16.1.100 | FORMAT_PING], \
 AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(172.16.1.100) | \
 sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
 icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=172.16.1.20,id=<cleared>,type=0,code=0),zone=<cleared>
-icmp,orig=(src=192.168.1.2,dst=172.16.1.100,id=<cleared>,type=8,code=0),reply=(src=172.16.1.100,dst=192.168.1.2,id=<cleared>,type=0,code=0),zone=<cleared>
 ])
 
 OVS_APP_EXIT_AND_WAIT([ovn-controller])