diff mbox series

[ovs-dev,1/2] lflow: Refactor OpenFlow hairpin flows

Message ID 20210805130907.1715667-1-mark.d.gray@redhat.com
State Superseded, archived
Headers show
Series [ovs-dev,1/2] lflow: Refactor OpenFlow hairpin flows | expand

Checks

Context Check Description
ovsrobot/apply-robot success apply and check: success
ovsrobot/github-robot-_Build_and_Test success github build: passed
ovsrobot/github-robot-_ovn-kubernetes fail github build: failed

Commit Message

Mark Gray Aug. 5, 2021, 1:09 p.m. UTC 
Currently, ovn-controller generates N x V OpenFlow
hairpin flows where:

N = number of datapaths
V = number of LB VIPs

In a scale setup with 120 nodes, 15k VIPs, and 3 Protocols,
this can generate 5.4M OpenFlows in the OFTABLE_CT_SNAT_FOR_VIP
table with the following form:

table=70, priority=100,udp,reg1=0x4001149,metadata=0x2f actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))

These flows can be optimized by splitting them across two tables.
For example,

table=70, priority=100,metadata=0x2f actions=resubmit(,73)
table=73, priority=100,udp,reg1=0x4001149, actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))

This results in N flows in table 70 and V flows in table 73. In
the example above this equates to 45120 flows a 120x reduction.

In an example sandbox environment with ~15k VIPs, and 120 nodes,
this results in:

* A reduction in ovn-controller recompute time for
  logical flows: 16 -> 11.8s
* A reduction in total ovs-vswitchd OpenFlows: 7.7M -> 2.1M
* A reduction in ovs-vswitchd RSS: 9.9G -> 2.7G

This patch updates these flows and associated tests.

Signed-off-by: Mark Gray <mark.d.gray@redhat.com>
---
 controller/lflow.c |  72 +++++++++++++-----
 controller/lflow.h |   3 +-
 tests/ovn.at       | 181 +++++++++++++++++++++++++++++++--------------
 tests/test-ovn.c   |   2 +-
 4 files changed, 179 insertions(+), 79 deletions(-)

Comments

Numan Siddique Aug. 5, 2021, 8:43 p.m. UTC | #1 
On Thu, Aug 5, 2021 at 9:09 AM Mark Gray <mark.d.gray@redhat.com> wrote:
>
> Currently, ovn-controller generates N x V OpenFlow
> hairpin flows where:
>
> N = number of datapaths
> V = number of LB VIPs
>
> In a scale setup with 120 nodes, 15k VIPs, and 3 Protocols,
> this can generate 5.4M OpenFlows in the OFTABLE_CT_SNAT_FOR_VIP
> table with the following form:
>
> table=70, priority=100,udp,reg1=0x4001149,metadata=0x2f actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))
>
> These flows can be optimized by splitting them across two tables.
> For example,
>
> table=70, priority=100,metadata=0x2f actions=resubmit(,73)
> table=73, priority=100,udp,reg1=0x4001149, actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))
>
> This results in N flows in table 70 and V flows in table 73. In
> the example above this equates to 45120 flows a 120x reduction.
>
> In an example sandbox environment with ~15k VIPs, and 120 nodes,
> this results in:
>
> * A reduction in ovn-controller recompute time for
>   logical flows: 16 -> 11.8s
> * A reduction in total ovs-vswitchd OpenFlows: 7.7M -> 2.1M
> * A reduction in ovs-vswitchd RSS: 9.9G -> 2.7G
>
> This patch updates these flows and associated tests.
>
> Signed-off-by: Mark Gray <mark.d.gray@redhat.com>

Hi Mark,

Thanks for the patch.  This is a great improvement.

The OVN action 'ct_snat_to_vip' translates to "resubmit(,70 (or
OFTABLE_CT_SNAT_FOR_VIP)".

And the corresponding logical flow would look like below :

 table=14(ls_in_nat_hairpin  ), priority=100  , match=(ip && ct.new &&
ct.trk && reg0[6] == 1), action=(ct_snat_to_vip; next;)

Given that the lflows in "ls_in_nat_hairpin" stage would resubmit the
packet to table 70 only for valid cases,  how about we
just remove the check for metadata field in the existing table 70 flows ?

In the proposed patch, the OF flows in table 70 seem redundant to me
and instead we can just have the flows of table 73 in table 70
(i,e without matching the metadata)

What do you think ?

Thanks
Numan


> ---
>  controller/lflow.c |  72 +++++++++++++-----
>  controller/lflow.h |   3 +-
>  tests/ovn.at       | 181 +++++++++++++++++++++++++++++++--------------
>  tests/test-ovn.c   |   2 +-
>  4 files changed, 179 insertions(+), 79 deletions(-)
>
> diff --git a/controller/lflow.c b/controller/lflow.c
> index 2e08261c6aed..bbf94c677c75 100644
> --- a/controller/lflow.c
> +++ b/controller/lflow.c
> @@ -644,7 +644,7 @@ add_matches_to_flow_table(const struct sbrec_logical_flow *lflow,
>          .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
>          .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
>          .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
> -        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
> +        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
>          .fdb_ptable = OFTABLE_GET_FDB,
>          .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
>          .ctrl_meter_id = ctrl_meter_id,
> @@ -1420,17 +1420,37 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb,
>      ofpbuf_uninit(&ofpacts);
>  }
>
> -/* Adds flows to perform SNAT for hairpin sessions.
> - *
> - * For backwards compatibilty with older ovn-northd versions, uses
> - * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
> - * original destination tuple stored by ovn-northd.
> - */
>  static void
> -add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
> -                         struct ovn_lb_vip *lb_vip,
> -                         uint8_t lb_proto,
> -                         struct ovn_desired_flow_table *flow_table)
> +add_lb_ct_snat_hairpin_dp_flows(struct ovn_controller_lb *lb,
> +                                struct ovn_desired_flow_table *flow_table)
> +{
> +    uint64_t stub2[1024 / 8];
> +    struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub2);
> +
> +    struct ofpact_resubmit *resubmit = ofpact_put_RESUBMIT(&ofpacts);
> +    resubmit->in_port = OFPP_IN_PORT;
> +    resubmit->table_id = OFTABLE_CT_SNAT_HAIRPIN_VIP;
> +
> +    struct match match = MATCH_CATCHALL_INITIALIZER;
> +
> +    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
> +        match_set_metadata(&match,
> +                           htonll(lb->slb->datapaths[i]->tunnel_key));
> +
> +        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_DP, 100,
> +                        lb->slb->header_.uuid.parts[0],
> +                        &match, &ofpacts, &lb->slb->header_.uuid);
> +    }
> +
> +    ofpbuf_uninit(&ofpacts);
> +}
> +
> +
> +static void
> +add_lb_ct_snat_hairpin_vip_flows(struct ovn_controller_lb *lb,
> +                                 struct ovn_lb_vip *lb_vip,
> +                                 uint8_t lb_proto,
> +                                struct ovn_desired_flow_table *flow_table)
>  {
>      uint64_t stub[1024 / 8];
>      struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub);
> @@ -1513,16 +1533,28 @@ add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
>          }
>      }
>
> -    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
> -        match_set_metadata(&match,
> -                           htonll(lb->slb->datapaths[i]->tunnel_key));
> -
> -        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_FOR_VIP, 100,
> -                        lb->slb->header_.uuid.parts[0],
> -                        &match, &ofpacts, &lb->slb->header_.uuid);
> -    }
> +    ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_VIP, 100,
> +                lb->slb->header_.uuid.parts[0],
> +                &match, &ofpacts, &lb->slb->header_.uuid);
>
>      ofpbuf_uninit(&ofpacts);
> +
> +}
> +
> +/* Adds flows to perform SNAT for hairpin sessions.
> + *
> + * For backwards compatibilty with older ovn-northd versions, uses
> + * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
> + * original destination tuple stored by ovn-northd.
> + */
> +static void
> +add_lb_ct_snat_hairpin_flows(struct ovn_controller_lb *lb,
> +                             struct ovn_lb_vip *lb_vip,
> +                             uint8_t lb_proto,
> +                             struct ovn_desired_flow_table *flow_table)
> +{
> +    add_lb_ct_snat_hairpin_dp_flows(lb, flow_table);
> +    add_lb_ct_snat_hairpin_vip_flows(lb, lb_vip, lb_proto, flow_table);
>  }
>
>  static void
> @@ -1565,7 +1597,7 @@ consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
>                                       flow_table);
>          }
>
> -        add_lb_ct_snat_vip_flows(lb, lb_vip, lb_proto, flow_table);
> +        add_lb_ct_snat_hairpin_flows(lb, lb_vip, lb_proto, flow_table);
>      }
>
>      ovn_controller_lb_destroy(lb);
> diff --git a/controller/lflow.h b/controller/lflow.h
> index 6fcda5355cdf..b02eadad0423 100644
> --- a/controller/lflow.h
> +++ b/controller/lflow.h
> @@ -72,9 +72,10 @@ struct uuid;
>  #define OFTABLE_MAC_LOOKUP           67
>  #define OFTABLE_CHK_LB_HAIRPIN       68
>  #define OFTABLE_CHK_LB_HAIRPIN_REPLY 69
> -#define OFTABLE_CT_SNAT_FOR_VIP      70
> +#define OFTABLE_CT_SNAT_HAIRPIN_DP   70
>  #define OFTABLE_GET_FDB              71
>  #define OFTABLE_LOOKUP_FDB           72
> +#define OFTABLE_CT_SNAT_HAIRPIN_VIP  73
>
>  enum ref_type {
>      REF_TYPE_ADDRSET,
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 7ae136ad9afe..4a85a089b017 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -25186,6 +25186,9 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>  ])
>
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
> +])
> +
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
>  ])
>
> @@ -25195,6 +25198,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>  ])
>
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
> +])
> +
>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-tcp
>
>  OVS_WAIT_UNTIL(
> @@ -25210,7 +25216,11 @@ NXST_FLOW reply (xid=0x8):
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
> @@ -25222,6 +25232,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>  ])
>
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
> +])
> +
>  check ovn-nbctl lb-add lb-ipv4-tcp 88.88.88.90:8080 42.42.42.42:4041,52.52.52.52:4042 tcp
>
>  OVS_WAIT_UNTIL(
> @@ -25239,8 +25252,12 @@ NXST_FLOW reply (xid=0x8):
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
> @@ -25252,6 +25269,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>  ])
>
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
> +])
> +
>  check ovn-nbctl lsp-add sw0 sw0-p2
>  # hv2 should bind sw0-p2 and it should install the LB hairpin flows.
>  OVS_WAIT_UNTIL([test x$(ovn-nbctl lsp-get-up sw0-p2) = xup])
> @@ -25270,8 +25290,12 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>  ])
>
>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-udp
> @@ -25295,9 +25319,13 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
> @@ -25311,9 +25339,13 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>  ])
>
>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-tcp
> @@ -25338,10 +25370,14 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
> @@ -25356,10 +25392,14 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>  ])
>
>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-udp
> @@ -25385,11 +25425,15 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
> @@ -25405,11 +25449,15 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>  ])
>
>  check ovn-nbctl --wait=hv ls-lb-add sw1 lb-ipv6-udp
> @@ -25437,12 +25485,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
> @@ -25458,12 +25510,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>  ])
>
>  # Check backwards compatibility with ovn-northd versions that don't store the
> @@ -25492,12 +25548,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
> +])
> +
> +OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>  ])
>
>  OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
> @@ -25513,14 +25573,17 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>  ])
>
>  OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>  ])
>
> +OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
> +])
>  # Resume ovn-northd.
>  as northd ovn-appctl -t NORTHD_TYPE resume
>  as northd-backup ovn-appctl -t NORTHD_TYPE resume
> @@ -25569,10 +25632,14 @@ NXST_FLOW reply (xid=0x8):
>  ])
>
>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
> +])
> +
> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>  ])
>
>  check ovn-nbctl --wait=hv ls-del sw0
> diff --git a/tests/test-ovn.c b/tests/test-ovn.c
> index 680cf06d6bf1..bf60274de3b2 100644
> --- a/tests/test-ovn.c
> +++ b/tests/test-ovn.c
> @@ -1348,7 +1348,7 @@ test_parse_actions(struct ovs_cmdl_context *ctx OVS_UNUSED)
>                  .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
>                  .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
>                  .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
> -                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
> +                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
>                  .fdb_ptable = OFTABLE_GET_FDB,
>                  .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
>              };
> --
> 2.27.0
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Mark Gray Aug. 11, 2021, 8:01 p.m. UTC | #2 
On 05/08/2021 21:43, Numan Siddique wrote:
> On Thu, Aug 5, 2021 at 9:09 AM Mark Gray <mark.d.gray@redhat.com> wrote:
>>
>> Currently, ovn-controller generates N x V OpenFlow
>> hairpin flows where:
>>
>> N = number of datapaths
>> V = number of LB VIPs
>>
>> In a scale setup with 120 nodes, 15k VIPs, and 3 Protocols,
>> this can generate 5.4M OpenFlows in the OFTABLE_CT_SNAT_FOR_VIP
>> table with the following form:
>>
>> table=70, priority=100,udp,reg1=0x4001149,metadata=0x2f actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))
>>
>> These flows can be optimized by splitting them across two tables.
>> For example,
>>
>> table=70, priority=100,metadata=0x2f actions=resubmit(,73)
>> table=73, priority=100,udp,reg1=0x4001149, actions=ct(commit,zone=NXM_NX_REG12[0..15],nat(src=4.0.17.73))
>>
>> This results in N flows in table 70 and V flows in table 73. In
>> the example above this equates to 45120 flows a 120x reduction.
>>
>> In an example sandbox environment with ~15k VIPs, and 120 nodes,
>> this results in:
>>
>> * A reduction in ovn-controller recompute time for
>>   logical flows: 16 -> 11.8s
>> * A reduction in total ovs-vswitchd OpenFlows: 7.7M -> 2.1M
>> * A reduction in ovs-vswitchd RSS: 9.9G -> 2.7G
>>
>> This patch updates these flows and associated tests.
>>
>> Signed-off-by: Mark Gray <mark.d.gray@redhat.com>
> 
> Hi Mark,
> 
> Thanks for the patch.  This is a great improvement.

Thanks, I have an updated patch at:
https://mail.openvswitch.org/pipermail/ovs-dev/2021-August/386657.html
> 
> The OVN action 'ct_snat_to_vip' translates to "resubmit(,70 (or
> OFTABLE_CT_SNAT_FOR_VIP)".
> 
> And the corresponding logical flow would look like below :
> 
>  table=14(ls_in_nat_hairpin  ), priority=100  , match=(ip && ct.new &&
> ct.trk && reg0[6] == 1), action=(ct_snat_to_vip; next;)
> 
> Given that the lflows in "ls_in_nat_hairpin" stage would resubmit the
> packet to table 70 only for valid cases,  how about we
> just remove the check for metadata field in the existing table 70 flows ?

I did some tests and prototyped this. It does work BUT only in the case
we do not use "hairpin_snat_ip" in this case we need to specify the
metadata field.

You can see why this is the case if you consider two LBs with same VIPs
added to two datapaths .. and then set "hairpin_snat_ip" on one of them.

This corner case complicates things but we can use conjunctive flows to
reduce the flow explosion when there are a large number of VIPs and
datapaths in the "hairpin_snat_ip" case.

> 
> In the proposed patch, the OF flows in table 70 seem redundant to me
> and instead we can just have the flows of table 73 in table 70
> (i,e without matching the metadata)

Yes, I managed to remove the additional table
> 
> What do you think ?

Thanks for the review.
> 
> Thanks
> Numan
> 
> 
>> ---
>>  controller/lflow.c |  72 +++++++++++++-----
>>  controller/lflow.h |   3 +-
>>  tests/ovn.at       | 181 +++++++++++++++++++++++++++++++--------------
>>  tests/test-ovn.c   |   2 +-
>>  4 files changed, 179 insertions(+), 79 deletions(-)
>>
>> diff --git a/controller/lflow.c b/controller/lflow.c
>> index 2e08261c6aed..bbf94c677c75 100644
>> --- a/controller/lflow.c
>> +++ b/controller/lflow.c
>> @@ -644,7 +644,7 @@ add_matches_to_flow_table(const struct sbrec_logical_flow *lflow,
>>          .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
>>          .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
>>          .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
>> -        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
>> +        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
>>          .fdb_ptable = OFTABLE_GET_FDB,
>>          .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
>>          .ctrl_meter_id = ctrl_meter_id,
>> @@ -1420,17 +1420,37 @@ add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb,
>>      ofpbuf_uninit(&ofpacts);
>>  }
>>
>> -/* Adds flows to perform SNAT for hairpin sessions.
>> - *
>> - * For backwards compatibilty with older ovn-northd versions, uses
>> - * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
>> - * original destination tuple stored by ovn-northd.
>> - */
>>  static void
>> -add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
>> -                         struct ovn_lb_vip *lb_vip,
>> -                         uint8_t lb_proto,
>> -                         struct ovn_desired_flow_table *flow_table)
>> +add_lb_ct_snat_hairpin_dp_flows(struct ovn_controller_lb *lb,
>> +                                struct ovn_desired_flow_table *flow_table)
>> +{
>> +    uint64_t stub2[1024 / 8];
>> +    struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub2);
>> +
>> +    struct ofpact_resubmit *resubmit = ofpact_put_RESUBMIT(&ofpacts);
>> +    resubmit->in_port = OFPP_IN_PORT;
>> +    resubmit->table_id = OFTABLE_CT_SNAT_HAIRPIN_VIP;
>> +
>> +    struct match match = MATCH_CATCHALL_INITIALIZER;
>> +
>> +    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
>> +        match_set_metadata(&match,
>> +                           htonll(lb->slb->datapaths[i]->tunnel_key));
>> +
>> +        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_DP, 100,
>> +                        lb->slb->header_.uuid.parts[0],
>> +                        &match, &ofpacts, &lb->slb->header_.uuid);
>> +    }
>> +
>> +    ofpbuf_uninit(&ofpacts);
>> +}
>> +
>> +
>> +static void
>> +add_lb_ct_snat_hairpin_vip_flows(struct ovn_controller_lb *lb,
>> +                                 struct ovn_lb_vip *lb_vip,
>> +                                 uint8_t lb_proto,
>> +                                struct ovn_desired_flow_table *flow_table)
>>  {
>>      uint64_t stub[1024 / 8];
>>      struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub);
>> @@ -1513,16 +1533,28 @@ add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
>>          }
>>      }
>>
>> -    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
>> -        match_set_metadata(&match,
>> -                           htonll(lb->slb->datapaths[i]->tunnel_key));
>> -
>> -        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_FOR_VIP, 100,
>> -                        lb->slb->header_.uuid.parts[0],
>> -                        &match, &ofpacts, &lb->slb->header_.uuid);
>> -    }
>> +    ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_VIP, 100,
>> +                lb->slb->header_.uuid.parts[0],
>> +                &match, &ofpacts, &lb->slb->header_.uuid);
>>
>>      ofpbuf_uninit(&ofpacts);
>> +
>> +}
>> +
>> +/* Adds flows to perform SNAT for hairpin sessions.
>> + *
>> + * For backwards compatibilty with older ovn-northd versions, uses
>> + * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
>> + * original destination tuple stored by ovn-northd.
>> + */
>> +static void
>> +add_lb_ct_snat_hairpin_flows(struct ovn_controller_lb *lb,
>> +                             struct ovn_lb_vip *lb_vip,
>> +                             uint8_t lb_proto,
>> +                             struct ovn_desired_flow_table *flow_table)
>> +{
>> +    add_lb_ct_snat_hairpin_dp_flows(lb, flow_table);
>> +    add_lb_ct_snat_hairpin_vip_flows(lb, lb_vip, lb_proto, flow_table);
>>  }
>>
>>  static void
>> @@ -1565,7 +1597,7 @@ consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
>>                                       flow_table);
>>          }
>>
>> -        add_lb_ct_snat_vip_flows(lb, lb_vip, lb_proto, flow_table);
>> +        add_lb_ct_snat_hairpin_flows(lb, lb_vip, lb_proto, flow_table);
>>      }
>>
>>      ovn_controller_lb_destroy(lb);
>> diff --git a/controller/lflow.h b/controller/lflow.h
>> index 6fcda5355cdf..b02eadad0423 100644
>> --- a/controller/lflow.h
>> +++ b/controller/lflow.h
>> @@ -72,9 +72,10 @@ struct uuid;
>>  #define OFTABLE_MAC_LOOKUP           67
>>  #define OFTABLE_CHK_LB_HAIRPIN       68
>>  #define OFTABLE_CHK_LB_HAIRPIN_REPLY 69
>> -#define OFTABLE_CT_SNAT_FOR_VIP      70
>> +#define OFTABLE_CT_SNAT_HAIRPIN_DP   70
>>  #define OFTABLE_GET_FDB              71
>>  #define OFTABLE_LOOKUP_FDB           72
>> +#define OFTABLE_CT_SNAT_HAIRPIN_VIP  73
>>
>>  enum ref_type {
>>      REF_TYPE_ADDRSET,
>> diff --git a/tests/ovn.at b/tests/ovn.at
>> index 7ae136ad9afe..4a85a089b017 100644
>> --- a/tests/ovn.at
>> +++ b/tests/ovn.at
>> @@ -25186,6 +25186,9 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>>  ])
>>
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
>> +])
>> +
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
>>  ])
>>
>> @@ -25195,6 +25198,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>>  ])
>>
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
>> +])
>> +
>>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-tcp
>>
>>  OVS_WAIT_UNTIL(
>> @@ -25210,7 +25216,11 @@ NXST_FLOW reply (xid=0x8):
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
>> @@ -25222,6 +25232,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>>  ])
>>
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
>> +])
>> +
>>  check ovn-nbctl lb-add lb-ipv4-tcp 88.88.88.90:8080 42.42.42.42:4041,52.52.52.52:4042 tcp
>>
>>  OVS_WAIT_UNTIL(
>> @@ -25239,8 +25252,12 @@ NXST_FLOW reply (xid=0x8):
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
>> @@ -25252,6 +25269,9 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
>>  ])
>>
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
>> +])
>> +
>>  check ovn-nbctl lsp-add sw0 sw0-p2
>>  # hv2 should bind sw0-p2 and it should install the LB hairpin flows.
>>  OVS_WAIT_UNTIL([test x$(ovn-nbctl lsp-get-up sw0-p2) = xup])
>> @@ -25270,8 +25290,12 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>>  ])
>>
>>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-udp
>> @@ -25295,9 +25319,13 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> @@ -25311,9 +25339,13 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>>  ])
>>
>>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-tcp
>> @@ -25338,10 +25370,14 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> @@ -25356,10 +25392,14 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>>  ])
>>
>>  check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-udp
>> @@ -25385,11 +25425,15 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> @@ -25405,11 +25449,15 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>>  ])
>>
>>  check ovn-nbctl --wait=hv ls-lb-add sw1 lb-ipv6-udp
>> @@ -25437,12 +25485,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> @@ -25458,12 +25510,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>>  ])
>>
>>  # Check backwards compatibility with ovn-northd versions that don't store the
>> @@ -25492,12 +25548,16 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>> +])
>> +
>> +OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>>  ])
>>
>>  OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> @@ -25513,14 +25573,17 @@ AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
>>  ])
>>
>>  OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>>  ])
>>
>> +OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
>> +])
>>  # Resume ovn-northd.
>>  as northd ovn-appctl -t NORTHD_TYPE resume
>>  as northd-backup ovn-appctl -t NORTHD_TYPE resume
>> @@ -25569,10 +25632,14 @@ NXST_FLOW reply (xid=0x8):
>>  ])
>>
>>  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> - table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> - table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=70, priority=100,metadata=0x1 actions=resubmit(,73)
>> + table=70, priority=100,metadata=0x2 actions=resubmit(,73)
>> +])
>> +
>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
>> + table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>> + table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
>> + table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
>>  ])
>>
>>  check ovn-nbctl --wait=hv ls-del sw0
>> diff --git a/tests/test-ovn.c b/tests/test-ovn.c
>> index 680cf06d6bf1..bf60274de3b2 100644
>> --- a/tests/test-ovn.c
>> +++ b/tests/test-ovn.c
>> @@ -1348,7 +1348,7 @@ test_parse_actions(struct ovs_cmdl_context *ctx OVS_UNUSED)
>>                  .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
>>                  .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
>>                  .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
>> -                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
>> +                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
>>                  .fdb_ptable = OFTABLE_GET_FDB,
>>                  .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
>>              };
>> --
>> 2.27.0
>>
>> _______________________________________________
>> dev mailing list
>> dev@openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>
>
diff mbox series

Patch

diff --git a/controller/lflow.c b/controller/lflow.c
index 2e08261c6aed..bbf94c677c75 100644
--- a/controller/lflow.c
+++ b/controller/lflow.c
@@ -644,7 +644,7 @@  add_matches_to_flow_table(const struct sbrec_logical_flow *lflow,
         .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
         .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
         .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
-        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
+        .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
         .fdb_ptable = OFTABLE_GET_FDB,
         .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
         .ctrl_meter_id = ctrl_meter_id,
@@ -1420,17 +1420,37 @@  add_lb_vip_hairpin_flows(struct ovn_controller_lb *lb,
     ofpbuf_uninit(&ofpacts);
 }
 
-/* Adds flows to perform SNAT for hairpin sessions.
- *
- * For backwards compatibilty with older ovn-northd versions, uses
- * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
- * original destination tuple stored by ovn-northd.
- */
 static void
-add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
-                         struct ovn_lb_vip *lb_vip,
-                         uint8_t lb_proto,
-                         struct ovn_desired_flow_table *flow_table)
+add_lb_ct_snat_hairpin_dp_flows(struct ovn_controller_lb *lb,
+                                struct ovn_desired_flow_table *flow_table)
+{
+    uint64_t stub2[1024 / 8];
+    struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub2);
+
+    struct ofpact_resubmit *resubmit = ofpact_put_RESUBMIT(&ofpacts);
+    resubmit->in_port = OFPP_IN_PORT;
+    resubmit->table_id = OFTABLE_CT_SNAT_HAIRPIN_VIP;
+
+    struct match match = MATCH_CATCHALL_INITIALIZER;
+
+    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
+        match_set_metadata(&match,
+                           htonll(lb->slb->datapaths[i]->tunnel_key));
+
+        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_DP, 100,
+                        lb->slb->header_.uuid.parts[0],
+                        &match, &ofpacts, &lb->slb->header_.uuid);
+    }
+
+    ofpbuf_uninit(&ofpacts);
+}
+
+
+static void
+add_lb_ct_snat_hairpin_vip_flows(struct ovn_controller_lb *lb,
+                                 struct ovn_lb_vip *lb_vip,
+                                 uint8_t lb_proto,
+                                struct ovn_desired_flow_table *flow_table)
 {
     uint64_t stub[1024 / 8];
     struct ofpbuf ofpacts = OFPBUF_STUB_INITIALIZER(stub);
@@ -1513,16 +1533,28 @@  add_lb_ct_snat_vip_flows(struct ovn_controller_lb *lb,
         }
     }
 
-    for (size_t i = 0; i < lb->slb->n_datapaths; i++) {
-        match_set_metadata(&match,
-                           htonll(lb->slb->datapaths[i]->tunnel_key));
-
-        ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_FOR_VIP, 100,
-                        lb->slb->header_.uuid.parts[0],
-                        &match, &ofpacts, &lb->slb->header_.uuid);
-    }
+    ofctrl_add_flow(flow_table, OFTABLE_CT_SNAT_HAIRPIN_VIP, 100,
+                lb->slb->header_.uuid.parts[0],
+                &match, &ofpacts, &lb->slb->header_.uuid);
 
     ofpbuf_uninit(&ofpacts);
+
+}
+
+/* Adds flows to perform SNAT for hairpin sessions.
+ *
+ * For backwards compatibilty with older ovn-northd versions, uses
+ * ct_nw_dst(), ct_ipv6_dst(), ct_tp_dst(), otherwise uses the
+ * original destination tuple stored by ovn-northd.
+ */
+static void
+add_lb_ct_snat_hairpin_flows(struct ovn_controller_lb *lb,
+                             struct ovn_lb_vip *lb_vip,
+                             uint8_t lb_proto,
+                             struct ovn_desired_flow_table *flow_table)
+{
+    add_lb_ct_snat_hairpin_dp_flows(lb, flow_table);
+    add_lb_ct_snat_hairpin_vip_flows(lb, lb_vip, lb_proto, flow_table);
 }
 
 static void
@@ -1565,7 +1597,7 @@  consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
                                      flow_table);
         }
 
-        add_lb_ct_snat_vip_flows(lb, lb_vip, lb_proto, flow_table);
+        add_lb_ct_snat_hairpin_flows(lb, lb_vip, lb_proto, flow_table);
     }
 
     ovn_controller_lb_destroy(lb);
diff --git a/controller/lflow.h b/controller/lflow.h
index 6fcda5355cdf..b02eadad0423 100644
--- a/controller/lflow.h
+++ b/controller/lflow.h
@@ -72,9 +72,10 @@  struct uuid;
 #define OFTABLE_MAC_LOOKUP           67
 #define OFTABLE_CHK_LB_HAIRPIN       68
 #define OFTABLE_CHK_LB_HAIRPIN_REPLY 69
-#define OFTABLE_CT_SNAT_FOR_VIP      70
+#define OFTABLE_CT_SNAT_HAIRPIN_DP   70
 #define OFTABLE_GET_FDB              71
 #define OFTABLE_LOOKUP_FDB           72
+#define OFTABLE_CT_SNAT_HAIRPIN_VIP  73
 
 enum ref_type {
     REF_TYPE_ADDRSET,
diff --git a/tests/ovn.at b/tests/ovn.at
index 7ae136ad9afe..4a85a089b017 100644
--- a/tests/ovn.at
+++ b/tests/ovn.at
@@ -25186,6 +25186,9 @@  AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
 ])
 
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
+])
+
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
 ])
 
@@ -25195,6 +25198,9 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
 ])
 
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
+])
+
 check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-tcp
 
 OVS_WAIT_UNTIL(
@@ -25210,7 +25216,11 @@  NXST_FLOW reply (xid=0x8):
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
@@ -25222,6 +25232,9 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
 ])
 
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
+])
+
 check ovn-nbctl lb-add lb-ipv4-tcp 88.88.88.90:8080 42.42.42.42:4041,52.52.52.52:4042 tcp
 
 OVS_WAIT_UNTIL(
@@ -25239,8 +25252,12 @@  NXST_FLOW reply (xid=0x8):
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | grep -v NXST], [1], [dnl
@@ -25252,6 +25269,9 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | grep -v NXST], [1], [dnl
 ])
 
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | grep -v NXST], [1], [dnl
+])
+
 check ovn-nbctl lsp-add sw0 sw0-p2
 # hv2 should bind sw0-p2 and it should install the LB hairpin flows.
 OVS_WAIT_UNTIL([test x$(ovn-nbctl lsp-get-up sw0-p2) = xup])
@@ -25270,8 +25290,12 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
 ])
 
 check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv4-udp
@@ -25295,9 +25319,13 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
@@ -25311,9 +25339,13 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
 ])
 
 check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-tcp
@@ -25338,10 +25370,14 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
@@ -25356,10 +25392,14 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
 ])
 
 check ovn-nbctl --wait=hv ls-lb-add sw0 lb-ipv6-udp
@@ -25385,11 +25425,15 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
@@ -25405,11 +25449,15 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
 ])
 
 check ovn-nbctl --wait=hv ls-lb-add sw1 lb-ipv6-udp
@@ -25437,12 +25485,16 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+ table=70, priority=100,metadata=0x2 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
@@ -25458,12 +25510,16 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+ table=70, priority=100,metadata=0x2 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp,reg1=0x58585858,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,tcp,reg1=0x5858585a,reg2=0x1f90/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
 ])
 
 # Check backwards compatibility with ovn-northd versions that don't store the
@@ -25492,12 +25548,16 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+ table=70, priority=100,metadata=0x2 actions=resubmit(,73)
+])
+
+OVS_WAIT_FOR_OUTPUT([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
 ])
 
 OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=68 | ofctl_strip_all | grep -v NXST], [0], [dnl
@@ -25513,14 +25573,17 @@  AT_CHECK([as hv2 ovs-ofctl dump-flows br-int table=69 | grep -v NXST], [1], [dnl
 ])
 
 OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+ table=70, priority=100,metadata=0x2 actions=resubmit(,73)
 ])
 
+OVS_WAIT_FOR_OUTPUT([as hv2 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=17,ct_tp_dst=4040,udp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_ipv6_dst=8800::88,ct_nw_proto=6,ct_tp_dst=8080,tcp6 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=17,ct_tp_dst=4040,udp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.88,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,ct_state=+trk+dnat,ct_nw_dst=88.88.88.90,ct_nw_proto=6,ct_tp_dst=8080,tcp actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.90))
+])
 # Resume ovn-northd.
 as northd ovn-appctl -t NORTHD_TYPE resume
 as northd-backup ovn-appctl -t NORTHD_TYPE resume
@@ -25569,10 +25632,14 @@  NXST_FLOW reply (xid=0x8):
 ])
 
 AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=70 | ofctl_strip_all | grep -v NXST], [0], [dnl
- table=70, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x1 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
- table=70, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88,metadata=0x2 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=70, priority=100,metadata=0x1 actions=resubmit(,73)
+ table=70, priority=100,metadata=0x2 actions=resubmit(,73)
+])
+
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int table=73 | ofctl_strip_all | grep -v NXST], [0], [dnl
+ table=73, priority=100,tcp6,reg2=0x1f90/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
+ table=73, priority=100,udp,reg1=0x58585858,reg2=0xfc8/0xffff actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=88.88.88.88))
+ table=73, priority=100,udp6,reg2=0xfc8/0xffff,reg4=0x88000000,reg5=0,reg6=0,reg7=0x88 actions=ct(commit,zone=NXM_NX_REG12[[0..15]],nat(src=8800::88))
 ])
 
 check ovn-nbctl --wait=hv ls-del sw0
diff --git a/tests/test-ovn.c b/tests/test-ovn.c
index 680cf06d6bf1..bf60274de3b2 100644
--- a/tests/test-ovn.c
+++ b/tests/test-ovn.c
@@ -1348,7 +1348,7 @@  test_parse_actions(struct ovs_cmdl_context *ctx OVS_UNUSED)
                 .mac_lookup_ptable = OFTABLE_MAC_LOOKUP,
                 .lb_hairpin_ptable = OFTABLE_CHK_LB_HAIRPIN,
                 .lb_hairpin_reply_ptable = OFTABLE_CHK_LB_HAIRPIN_REPLY,
-                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_FOR_VIP,
+                .ct_snat_vip_ptable = OFTABLE_CT_SNAT_HAIRPIN_DP,
                 .fdb_ptable = OFTABLE_GET_FDB,
                 .fdb_lookup_ptable = OFTABLE_LOOKUP_FDB,
             };