[ovs-dev,v3,4/5] lb: Support using templates.

Allow the CMS to configure template LBs.  The following configurations are
supported:
- VIPs of the form: ^vip_variable[:^port_variable|:port]
- Backends of the form:
  ^backendip_variable1[:^port_variable1|:port],^backendip_variable2[:^port_variable2|:port]
  OR
  ^backends_variable1,^backends_variable2

The CMS needs to provide a bit more information than with non-template load
balancers and must explicitly specify the address family to be used.

There is currently no support for template load balancers with
options:add_route=true set.  That is because ovn-northd does not
instantiate template variables.  While this is a limitation in a way, its
impact is not huge.  The load balancer 'add_route' option was added as a
way to make the CMS life easier and to avoid having to explicitly add a
route for the VIP.  The CMS can still achieve the same logical topology by
explicitly adding the VIP route.

Template load balancers don't support the "reachable" neighbor-responder
mode.  Instead the CMS can explicitly configure the responder mode to
either "all" or "none".

To properly handle template updates in ovn-controller we also add a
Chassis_Template_Var <- LB reference in ovn-controller.  This way, when
a Chassis_Template_Var changes value all load balancers that refer to
it will also get updated.

Signed-off-by: Dumitru Ceara <dceara@redhat.com>
---
V3:
- Addressed Mark's comments:
  - Added TODO items about potential future template LB improvements.
  - Removed n_backends arg from ovn_lb_backends_init_explicit() and
    ovn_lb_backends_init_template().
  - Fixed ovn_northd_lb_create() to first get the template option before
    using its value.
  - Fixed up comments and man pages.
  - Hardenned setting of address family in ovn-nbctl.

V2:
- Fix GCC build due to missing explicit return.
- Fix ls_in_pre_stateful flows due to using wrong lb field.
- Use new lexer_parse_template_string().
- Changed lb_handle_changed_ref() signature to return bool.
- Update documentation with info about responder mode=none, LB template
  supported formats, lb explicit address family requirements.
- Squashed the template LB patches into a single one
- Added more tests.
- Squashed the system tests patch into this one.
---
 TODO.rst                    |    7 +
 controller/lflow.c          |  115 +++++++++--
 controller/lflow.h          |    7 +
 controller/ovn-controller.c |   67 +++++-
 lib/lb.c                    |  452 ++++++++++++++++++++++++++++++++++++++-----
 lib/lb.h                    |   40 +++-
 lib/ovn-util.c              |    3 
 northd/northd.c             |   89 ++++----
 ovn-nb.xml                  |   62 ++++++
 tests/ovn-nbctl.at          |   23 +-
 tests/ovn-northd.at         |    7 +
 tests/ovn.at                |  131 ++++++++++++
 tests/system-ovn.at         |  183 +++++++++++++++++
 utilities/ovn-nbctl.c       |  120 ++++++-----
 14 files changed, 1078 insertions(+), 228 deletions(-)

On Tue, Nov 22, 2022 at 6:15 AM Dumitru Ceara <dceara@redhat.com> wrote:
>
> Allow the CMS to configure template LBs.  The following configurations are
> supported:
> - VIPs of the form: ^vip_variable[:^port_variable|:port]
> - Backends of the form:
>
^backendip_variable1[:^port_variable1|:port],^backendip_variable2[:^port_variable2|:port]
>   OR
>   ^backends_variable1,^backends_variable2

Sorry if I missed, but I didn't see any tests that test the form
"^backends_variable1,^backends_variable2". I only see tests with a single
backend variable with a single IP in it. Better to test:
1. Multiple backends variables
2. Multiple IPs in a single variable (I saw this in the tutorial in patch
5, but better to be covered here, too)



>
> The CMS needs to provide a bit more information than with non-template
load
> balancers and must explicitly specify the address family to be used.
>
> There is currently no support for template load balancers with
> options:add_route=true set.  That is because ovn-northd does not
> instantiate template variables.  While this is a limitation in a way, its
> impact is not huge.  The load balancer 'add_route' option was added as a
> way to make the CMS life easier and to avoid having to explicitly add a
> route for the VIP.  The CMS can still achieve the same logical topology by
> explicitly adding the VIP route.
>
> Template load balancers don't support the "reachable" neighbor-responder
> mode.  Instead the CMS can explicitly configure the responder mode to
> either "all" or "none".
>
> To properly handle template updates in ovn-controller we also add a
> Chassis_Template_Var <- LB reference in ovn-controller.  This way, when
> a Chassis_Template_Var changes value all load balancers that refer to
> it will also get updated.
>
> Signed-off-by: Dumitru Ceara <dceara@redhat.com>
> ---
> V3:
> - Addressed Mark's comments:
>   - Added TODO items about potential future template LB improvements.
>   - Removed n_backends arg from ovn_lb_backends_init_explicit() and
>     ovn_lb_backends_init_template().
>   - Fixed ovn_northd_lb_create() to first get the template option before
>     using its value.
>   - Fixed up comments and man pages.
>   - Hardenned setting of address family in ovn-nbctl.
>
> V2:
> - Fix GCC build due to missing explicit return.
> - Fix ls_in_pre_stateful flows due to using wrong lb field.
> - Use new lexer_parse_template_string().
> - Changed lb_handle_changed_ref() signature to return bool.
> - Update documentation with info about responder mode=none, LB template
>   supported formats, lb explicit address family requirements.
> - Squashed the template LB patches into a single one
> - Added more tests.
> - Squashed the system tests patch into this one.
> ---
>  TODO.rst                    |    7 +
>  controller/lflow.c          |  115 +++++++++--
>  controller/lflow.h          |    7 +
>  controller/ovn-controller.c |   67 +++++-
>  lib/lb.c                    |  452
++++++++++++++++++++++++++++++++++++++-----
>  lib/lb.h                    |   40 +++-
>  lib/ovn-util.c              |    3
>  northd/northd.c             |   89 ++++----
>  ovn-nb.xml                  |   62 ++++++
>  tests/ovn-nbctl.at          |   23 +-
>  tests/ovn-northd.at         |    7 +
>  tests/ovn.at                |  131 ++++++++++++
>  tests/system-ovn.at         |  183 +++++++++++++++++
>  utilities/ovn-nbctl.c       |  120 ++++++-----
>  14 files changed, 1078 insertions(+), 228 deletions(-)
>
> diff --git a/TODO.rst b/TODO.rst
> index fe5f9a2f30..53cf2870b2 100644
> --- a/TODO.rst
> +++ b/TODO.rst
> @@ -183,3 +183,10 @@ OVN To-do List
>  * Chassis_Template_Var
>
>    * Support template variables when tracing packets with ovn-trace.
> +
> +* Load Balancer templates
> +
> +  * Support combining the VIP (or backend) IP and port into a single
> +    template variable.

Is it still a TODO for backends? At least the tutorial test (in patch 5) is
already doing something like this:
backends0="42.0.0.1:1,42.1.0.1:1,42.2.0.1:1,42.3.0.1:1,42.4.0.1:1"

> +
> +  * Support combining all backends into a single template variable.

What does it mean here? Isn't the tutorial test already combining multiple
backends into a single variable?

> diff --git a/controller/lflow.c b/controller/lflow.c
> index 84625fb3f1..f6ac639541 100644
> --- a/controller/lflow.c
> +++ b/controller/lflow.c
> @@ -97,6 +97,15 @@ consider_logical_flow(const struct sbrec_logical_flow
*lflow,
>                        struct lflow_ctx_in *l_ctx_in,
>                        struct lflow_ctx_out *l_ctx_out);
>
> +static void
> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
> +                          const struct sbrec_load_balancer *sbrec_lb,
> +                          const struct hmap *local_datapaths,
> +                          const struct smap *template_vars,
> +                          bool use_ct_mark,
> +                          struct ovn_desired_flow_table *flow_table,
> +                          struct simap *ids);
> +
>  static void add_port_sec_flows(const struct shash *binding_lports,
>                                 const struct sbrec_chassis *,
>                                 struct ovn_desired_flow_table *);
> @@ -223,7 +232,7 @@ lflow_handle_changed_flows(struct lflow_ctx_in
*l_ctx_in,
>          UUIDSET_INITIALIZER(&flood_remove_nodes);
>      SBREC_LOGICAL_FLOW_TABLE_FOR_EACH_TRACKED (lflow,
>
l_ctx_in->logical_flow_table) {
> -        if (uuidset_find(l_ctx_out->lflows_processed,
&lflow->header_.uuid)) {
> +        if (uuidset_find(l_ctx_out->objs_processed,
&lflow->header_.uuid)) {
>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
>                       UUID_ARGS(&lflow->header_.uuid));
>              continue;
> @@ -253,14 +262,14 @@ lflow_handle_changed_flows(struct lflow_ctx_in
*l_ctx_in,
>                       UUID_ARGS(&lflow->header_.uuid));
>
>              /* For the extra lflows that need to be reprocessed because
of the
> -             * flood remove, remove it from lflows_processed. */
> +             * flood remove, remove it from objs_processed. */
>              struct uuidset_node *unode =
> -                uuidset_find(l_ctx_out->lflows_processed,
> +                uuidset_find(l_ctx_out->objs_processed,
>                               &lflow->header_.uuid);
>              if (unode) {
>                  VLOG_DBG("lflow "UUID_FMT"has been processed, now
reprocess.",
>                           UUID_ARGS(&lflow->header_.uuid));
> -                uuidset_delete(l_ctx_out->lflows_processed, unode);
> +                uuidset_delete(l_ctx_out->objs_processed, unode);
>              }
>
>              consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
> @@ -687,7 +696,7 @@ lflow_handle_addr_set_update(const char *as_name,
>      struct object_to_resources_list_node *resource_list_node;
>      RESOURCE_FOR_EACH_OBJ (resource_list_node, resource_node) {
>          const struct uuid *obj_uuid = &resource_list_node->obj_uuid;
> -        if (uuidset_find(l_ctx_out->lflows_processed, obj_uuid)) {
> +        if (uuidset_find(l_ctx_out->objs_processed, obj_uuid)) {
>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
>                       UUID_ARGS(obj_uuid));
>              continue;
> @@ -777,13 +786,13 @@ lflow_handle_changed_ref(enum objdep_type type,
const char *res_name,
>          }
>
>          /* For the extra lflows that need to be reprocessed because of
the
> -         * flood remove, remove it from lflows_processed. */
> +         * flood remove, remove it from objs_processed. */
>          struct uuidset_node *unode =
> -            uuidset_find(l_ctx_out->lflows_processed,
&lflow->header_.uuid);
> +            uuidset_find(l_ctx_out->objs_processed,
&lflow->header_.uuid);
>          if (unode) {
>              VLOG_DBG("lflow "UUID_FMT"has been processed, now
reprocess.",
>                       UUID_ARGS(&lflow->header_.uuid));
> -            uuidset_delete(l_ctx_out->lflows_processed, unode);
> +            uuidset_delete(l_ctx_out->objs_processed, unode);
>          }
>
>          consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
> @@ -792,6 +801,43 @@ lflow_handle_changed_ref(enum objdep_type type,
const char *res_name,
>      return true;
>  }
>
> +bool
> +lb_handle_changed_ref(enum objdep_type type, const char *res_name,
> +                      struct ovs_list *objs_todo,
> +                      const void *in_arg, void *out_arg)
> +{
> +    struct lflow_ctx_in *l_ctx_in = CONST_CAST(struct lflow_ctx_in *,
in_arg);
> +    struct lflow_ctx_out *l_ctx_out = out_arg;
> +
> +    struct object_to_resources_list_node *resource_lb_uuid;
> +    LIST_FOR_EACH_POP (resource_lb_uuid, list_node, objs_todo) {
> +        VLOG_DBG("Reprocess LB "UUID_FMT" for resource type: %s, name:
%s",
> +                 UUID_ARGS(&resource_lb_uuid->obj_uuid),
> +                 objdep_type_name(type), res_name);
> +
> +        const struct sbrec_load_balancer *lb =
> +            sbrec_load_balancer_table_get_for_uuid(
> +                l_ctx_in->lb_table, &resource_lb_uuid->obj_uuid);
> +        if (!lb) {
> +            VLOG_DBG("Failed to find LB "UUID_FMT" referred by: %s",

nit: I think it should be: Failed to find LB ... that refers ...


> +                     UUID_ARGS(&resource_lb_uuid->obj_uuid), res_name);
> +        } else {
> +            ofctrl_remove_flows(l_ctx_out->flow_table,
> +                                &resource_lb_uuid->obj_uuid);
> +
> +            consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
> +                                      l_ctx_in->local_datapaths,
> +                                      l_ctx_in->template_vars,
> +                                      l_ctx_in->lb_hairpin_use_ct_mark,
> +                                      l_ctx_out->flow_table,
> +                                      l_ctx_out->hairpin_lb_ids);
> +        }
> +
> +        free(resource_lb_uuid);
> +    }
> +    return true;
> +}
> +
>  static void
>  lflow_parse_ctrl_meter(const struct sbrec_logical_flow *lflow,
>                         struct ovn_extend_table *meter_table,
> @@ -1259,9 +1305,9 @@ consider_logical_flow(const struct
sbrec_logical_flow *lflow,
>
>      COVERAGE_INC(consider_logical_flow);
>      if (!is_recompute) {
> -        ovs_assert(!uuidset_find(l_ctx_out->lflows_processed,
> +        ovs_assert(!uuidset_find(l_ctx_out->objs_processed,
>                                   &lflow->header_.uuid));
> -        uuidset_insert(l_ctx_out->lflows_processed,
&lflow->header_.uuid);
> +        uuidset_insert(l_ctx_out->objs_processed, &lflow->header_.uuid);
>      }
>
>      if (dp) {
> @@ -2001,8 +2047,10 @@ add_lb_ct_snat_hairpin_flows(struct
ovn_controller_lb *lb,
>  }
>
>  static void
> -consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
> +                          const struct sbrec_load_balancer *sbrec_lb,
>                            const struct hmap *local_datapaths,
> +                          const struct smap *template_vars,
>                            bool use_ct_mark,
>                            struct ovn_desired_flow_table *flow_table,
>                            struct simap *ids)
> @@ -2039,7 +2087,9 @@ consider_lb_hairpin_flows(const struct
sbrec_load_balancer *sbrec_lb,
>          return;
>      }
>
> -    struct ovn_controller_lb *lb = ovn_controller_lb_create(sbrec_lb);
> +    struct sset template_vars_ref = SSET_INITIALIZER(&template_vars_ref);
> +    struct ovn_controller_lb *lb =
> +        ovn_controller_lb_create(sbrec_lb, template_vars,
&template_vars_ref);
>      uint8_t lb_proto = IPPROTO_TCP;
>      if (lb->slb->protocol && lb->slb->protocol[0]) {
>          if (!strcmp(lb->slb->protocol, "udp")) {
> @@ -2049,6 +2099,11 @@ consider_lb_hairpin_flows(const struct
sbrec_load_balancer *sbrec_lb,
>          }
>      }
>
> +    const char *tv_name;
> +    SSET_FOR_EACH (tv_name, &template_vars_ref) {
> +        objdep_mgr_add(mgr, OBJDEP_TYPE_TEMPLATE, tv_name,
> +                       &sbrec_lb->header_.uuid);
> +    }
>      for (i = 0; i < lb->n_vips; i++) {
>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>
> @@ -2063,13 +2118,17 @@ consider_lb_hairpin_flows(const struct
sbrec_load_balancer *sbrec_lb,
>      add_lb_ct_snat_hairpin_flows(lb, id, lb_proto, flow_table);
>
>      ovn_controller_lb_destroy(lb);
> +    sset_destroy(&template_vars_ref);
>  }
>
>  /* Adds OpenFlow flows to flow tables for each Load balancer VIPs and
>   * backends to handle the load balanced hairpin traffic. */
>  static void
> -add_lb_hairpin_flows(const struct sbrec_load_balancer_table *lb_table,
> -                     const struct hmap *local_datapaths, bool
use_ct_mark,
> +add_lb_hairpin_flows(struct objdep_mgr *mgr,
> +                     const struct sbrec_load_balancer_table *lb_table,
> +                     const struct hmap *local_datapaths,
> +                     const struct smap *template_vars,
> +                     bool use_ct_mark,
>                       struct ovn_desired_flow_table *flow_table,
>                       struct simap *ids,
>                       struct id_pool *pool)
> @@ -2092,8 +2151,8 @@ add_lb_hairpin_flows(const struct
sbrec_load_balancer_table *lb_table,
>              ovs_assert(id_pool_alloc_id(pool, &id));
>              simap_put(ids, lb->name, id);
>          }
> -        consider_lb_hairpin_flows(lb, local_datapaths, use_ct_mark,
> -                                  flow_table, ids);
> +        consider_lb_hairpin_flows(mgr, lb, local_datapaths,
template_vars,
> +                                  use_ct_mark, flow_table, ids);
>      }
>  }
>
> @@ -2229,7 +2288,9 @@ lflow_run(struct lflow_ctx_in *l_ctx_in, struct
lflow_ctx_out *l_ctx_out)
>                         l_ctx_in->static_mac_binding_table,
>                         l_ctx_in->local_datapaths,
>                         l_ctx_out->flow_table);
> -    add_lb_hairpin_flows(l_ctx_in->lb_table, l_ctx_in->local_datapaths,
> +    add_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, l_ctx_in->lb_table,
> +                         l_ctx_in->local_datapaths,
> +                         l_ctx_in->template_vars,
>                           l_ctx_in->lb_hairpin_use_ct_mark,
>                           l_ctx_out->flow_table,
>                           l_ctx_out->hairpin_lb_ids,
> @@ -2280,10 +2341,10 @@ lflow_add_flows_for_datapath(const struct
sbrec_datapath_binding *dp,
>      const struct sbrec_logical_flow *lflow;
>      SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
>          lflow, lf_row, l_ctx_in->sbrec_logical_flow_by_logical_datapath)
{
> -        if (uuidset_find(l_ctx_out->lflows_processed,
&lflow->header_.uuid)) {
> +        if (uuidset_find(l_ctx_out->objs_processed,
&lflow->header_.uuid)) {
>              continue;
>          }
> -        uuidset_insert(l_ctx_out->lflows_processed,
&lflow->header_.uuid);
> +        uuidset_insert(l_ctx_out->objs_processed, &lflow->header_.uuid);
>          consider_logical_flow__(lflow, dp, l_ctx_in, l_ctx_out);
>      }
>      sbrec_logical_flow_index_destroy_row(lf_row);
> @@ -2308,7 +2369,7 @@ lflow_add_flows_for_datapath(const struct
sbrec_datapath_binding *dp,
>          sbrec_logical_flow_index_set_logical_dp_group(lf_row, ldpg);
>          SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
>              lflow, lf_row,
l_ctx_in->sbrec_logical_flow_by_logical_dp_group) {
> -            if (uuidset_find(l_ctx_out->lflows_processed,
> +            if (uuidset_find(l_ctx_out->objs_processed,
>                               &lflow->header_.uuid)) {
>                  continue;
>              }
> @@ -2360,7 +2421,9 @@ lflow_add_flows_for_datapath(const struct
sbrec_datapath_binding *dp,
>      /* Add load balancer hairpin flows if the datapath has any load
balancers
>       * associated. */
>      for (size_t i = 0; i < n_dp_lbs; i++) {
> -        consider_lb_hairpin_flows(dp_lbs[i], l_ctx_in->local_datapaths,
> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, dp_lbs[i],
> +                                  l_ctx_in->local_datapaths,
> +                                  l_ctx_in->template_vars,
>                                    l_ctx_in->lb_hairpin_use_ct_mark,
>                                    l_ctx_out->flow_table,
>                                    l_ctx_out->hairpin_lb_ids);
> @@ -2382,7 +2445,7 @@ lflow_handle_flows_for_lport(const struct
sbrec_port_binding *pb,
>                                    OBJDEP_TYPE_PORTBINDING,
>                                    pb->logical_port,
>                                    lflow_handle_changed_ref,
> -                                  l_ctx_out->lflows_processed,
> +                                  l_ctx_out->objs_processed,
>                                    l_ctx_in, l_ctx_out, &changed)) {
>          return false;
>      }
> @@ -2421,7 +2484,7 @@ lflow_handle_changed_port_bindings(struct
lflow_ctx_in *l_ctx_in,
>                                        OBJDEP_TYPE_PORTBINDING,
>                                        pb->logical_port,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out->lflows_processed,
> +                                      l_ctx_out->objs_processed,
>                                        l_ctx_in, l_ctx_out, &changed)) {
>              ret = false;
>              break;
> @@ -2448,7 +2511,7 @@ lflow_handle_changed_mc_groups(struct lflow_ctx_in
*l_ctx_in,
>          if (!objdep_mgr_handle_change(l_ctx_out->lflow_deps_mgr,
>                                        OBJDEP_TYPE_MC_GROUP,
ds_cstr(&mg_key),
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out->lflows_processed,
> +                                      l_ctx_out->objs_processed,
>                                        l_ctx_in, l_ctx_out, &changed)) {
>              ret = false;
>              break;
> @@ -2502,7 +2565,9 @@ lflow_handle_changed_lbs(struct lflow_ctx_in
*l_ctx_in,
>
>          VLOG_DBG("Add load balancer hairpin flows for "UUID_FMT,
>                   UUID_ARGS(&lb->header_.uuid));
> -        consider_lb_hairpin_flows(lb, l_ctx_in->local_datapaths,
> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
> +                                  l_ctx_in->local_datapaths,
> +                                  l_ctx_in->template_vars,
>                                    l_ctx_in->lb_hairpin_use_ct_mark,
>                                    l_ctx_out->flow_table,
>                                    l_ctx_out->hairpin_lb_ids);
> diff --git a/controller/lflow.h b/controller/lflow.h
> index d95fd41142..9e8f9afd33 100644
> --- a/controller/lflow.h
> +++ b/controller/lflow.h
> @@ -122,9 +122,10 @@ struct lflow_ctx_out {
>      struct ovn_extend_table *group_table;
>      struct ovn_extend_table *meter_table;
>      struct objdep_mgr *lflow_deps_mgr;
> +    struct objdep_mgr *lb_deps_mgr;
>      struct lflow_cache *lflow_cache;
>      struct conj_ids *conj_ids;
> -    struct uuidset *lflows_processed;
> +    struct uuidset *objs_processed;
>      struct simap *hairpin_lb_ids;
>      struct id_pool *hairpin_id_pool;
>  };
> @@ -174,4 +175,8 @@ bool lflow_handle_changed_mc_groups(struct
lflow_ctx_in *,
>                                      struct lflow_ctx_out *);
>  bool lflow_handle_changed_port_bindings(struct lflow_ctx_in *,
>                                          struct lflow_ctx_out *);
> +
> +bool lb_handle_changed_ref(enum objdep_type type, const char *res_name,
> +                           struct ovs_list *objs_todo,
> +                           const void *in_arg, void *out_arg);
>  #endif /* controller/lflow.h */
> diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c
> index f9ed0e3855..9807ecd8eb 100644
> --- a/controller/ovn-controller.c
> +++ b/controller/ovn-controller.c
> @@ -2791,13 +2791,15 @@ struct ed_type_lflow_output {
>      struct ovn_extend_table meter_table;
>      /* lflow <-> resource cross reference */
>      struct objdep_mgr lflow_deps_mgr;;
> +    /* load balancer <-> resource cross reference */
> +    struct objdep_mgr lb_deps_mgr;
>      /* conjunciton ID usage information of lflows */
>      struct conj_ids conj_ids;
>
> -    /* lflows processed in the current engine execution.
> +    /* objects (lflows and lbs) processed in the current engine
execution.
>       * Cleared by en_lflow_output_clear_tracked_data before each engine
>       * execution. */
> -    struct uuidset lflows_processed;
> +    struct uuidset objs_processed;
>
>      /* Data which is persistent and not cleared during
>       * full recompute. */
> @@ -2954,8 +2956,9 @@ init_lflow_ctx(struct engine_node *node,
>      l_ctx_out->group_table = &fo->group_table;
>      l_ctx_out->meter_table = &fo->meter_table;
>      l_ctx_out->lflow_deps_mgr = &fo->lflow_deps_mgr;
> +    l_ctx_out->lb_deps_mgr = &fo->lb_deps_mgr;
>      l_ctx_out->conj_ids = &fo->conj_ids;
> -    l_ctx_out->lflows_processed = &fo->lflows_processed;
> +    l_ctx_out->objs_processed = &fo->objs_processed;
>      l_ctx_out->lflow_cache = fo->pd.lflow_cache;
>      l_ctx_out->hairpin_id_pool = fo->hd.pool;
>      l_ctx_out->hairpin_lb_ids = &fo->hd.ids;
> @@ -2970,8 +2973,9 @@ en_lflow_output_init(struct engine_node *node
OVS_UNUSED,
>      ovn_extend_table_init(&data->group_table);
>      ovn_extend_table_init(&data->meter_table);
>      objdep_mgr_init(&data->lflow_deps_mgr);
> +    objdep_mgr_init(&data->lb_deps_mgr);
>      lflow_conj_ids_init(&data->conj_ids);
> -    uuidset_init(&data->lflows_processed);
> +    uuidset_init(&data->objs_processed);
>      simap_init(&data->hd.ids);
>      data->hd.pool = id_pool_create(1, UINT32_MAX - 1);
>      nd_ra_opts_init(&data->nd_ra_opts);
> @@ -2983,7 +2987,7 @@ static void
>  en_lflow_output_clear_tracked_data(void *data)
>  {
>      struct ed_type_lflow_output *flow_output_data = data;
> -    uuidset_clear(&flow_output_data->lflows_processed);
> +    uuidset_clear(&flow_output_data->objs_processed);
>  }
>
>  static void
> @@ -2994,8 +2998,9 @@ en_lflow_output_cleanup(void *data)
>      ovn_extend_table_destroy(&flow_output_data->group_table);
>      ovn_extend_table_destroy(&flow_output_data->meter_table);
>      objdep_mgr_destroy(&flow_output_data->lflow_deps_mgr);
> +    objdep_mgr_destroy(&flow_output_data->lb_deps_mgr);
>      lflow_conj_ids_destroy(&flow_output_data->conj_ids);
> -    uuidset_destroy(&flow_output_data->lflows_processed);
> +    uuidset_destroy(&flow_output_data->objs_processed);
>      lflow_cache_destroy(flow_output_data->pd.lflow_cache);
>      simap_destroy(&flow_output_data->hd.ids);
>      id_pool_destroy(flow_output_data->hd.pool);
> @@ -3030,6 +3035,7 @@ en_lflow_output_run(struct engine_node *node, void
*data)
>      struct ovn_extend_table *group_table = &fo->group_table;
>      struct ovn_extend_table *meter_table = &fo->meter_table;
>      struct objdep_mgr *lflow_deps_mgr = &fo->lflow_deps_mgr;
> +    struct objdep_mgr *lb_deps_mgr = &fo->lb_deps_mgr;
>
>      static bool first_run = true;
>      if (first_run) {
> @@ -3039,6 +3045,7 @@ en_lflow_output_run(struct engine_node *node, void
*data)
>          ovn_extend_table_clear(group_table, false /* desired */);
>          ovn_extend_table_clear(meter_table, false /* desired */);
>          objdep_mgr_clear(lflow_deps_mgr);
> +        objdep_mgr_clear(lb_deps_mgr);
>          lflow_conj_ids_clear(&fo->conj_ids);
>      }
>
> @@ -3172,7 +3179,7 @@ lflow_output_addr_sets_handler(struct engine_node
*node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_ADDRSET, ref_name,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3191,7 +3198,7 @@ lflow_output_addr_sets_handler(struct engine_node
*node, void *data)
>                                            OBJDEP_TYPE_ADDRSET,
>                                            shash_node->name,
>                                            lflow_handle_changed_ref,
> -                                          l_ctx_out.lflows_processed,
> +                                          l_ctx_out.objs_processed,
>                                            &l_ctx_in, &l_ctx_out,
&changed)) {
>                  return false;
>              }
> @@ -3204,7 +3211,7 @@ lflow_output_addr_sets_handler(struct engine_node
*node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_ADDRSET, ref_name,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3239,7 +3246,7 @@ lflow_output_port_groups_handler(struct engine_node
*node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3251,7 +3258,7 @@ lflow_output_port_groups_handler(struct engine_node
*node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3263,7 +3270,7 @@ lflow_output_port_groups_handler(struct engine_node
*node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>                                        lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3297,7 +3304,17 @@ lflow_output_template_vars_handler(struct
engine_node *node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_TEMPLATE,
>                                        res_name, lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
> +            return false;
> +        }
> +        if (changed) {
> +            engine_set_node_state(node, EN_UPDATED);
> +        }
> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> +                                      OBJDEP_TYPE_TEMPLATE,
> +                                      res_name, lb_handle_changed_ref,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3309,7 +3326,17 @@ lflow_output_template_vars_handler(struct
engine_node *node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_TEMPLATE,
>                                        res_name, lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
> +            return false;
> +        }
> +        if (changed) {
> +            engine_set_node_state(node, EN_UPDATED);
> +        }
> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> +                                      OBJDEP_TYPE_TEMPLATE,
> +                                      res_name, lb_handle_changed_ref,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> @@ -3321,7 +3348,17 @@ lflow_output_template_vars_handler(struct
engine_node *node, void *data)
>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>                                        OBJDEP_TYPE_TEMPLATE,
>                                        res_name, lflow_handle_changed_ref,
> -                                      l_ctx_out.lflows_processed,
> +                                      l_ctx_out.objs_processed,
> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
> +            return false;
> +        }
> +        if (changed) {
> +            engine_set_node_state(node, EN_UPDATED);
> +        }
> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> +                                      OBJDEP_TYPE_TEMPLATE,
> +                                      res_name, lb_handle_changed_ref,
> +                                      l_ctx_out.objs_processed,
>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>              return false;
>          }
> diff --git a/lib/lb.c b/lib/lb.c
> index c08ccceda1..43628bba77 100644
> --- a/lib/lb.c
> +++ b/lib/lb.c
> @@ -19,6 +19,7 @@
>  #include "lib/ovn-nb-idl.h"
>  #include "lib/ovn-sb-idl.h"
>  #include "lib/ovn-util.h"
> +#include "ovn/lex.h"
>
>  /* OpenvSwitch lib includes. */
>  #include "openvswitch/vlog.h"
> @@ -26,6 +27,16 @@
>
>  VLOG_DEFINE_THIS_MODULE(lb);
>
> +static const char *lb_neighbor_responder_mode_names[] = {
> +    [LB_NEIGH_RESPOND_REACHABLE] = "reachable",
> +    [LB_NEIGH_RESPOND_ALL] = "all",
> +    [LB_NEIGH_RESPOND_NONE] = "none",
> +};
> +
> +static struct nbrec_load_balancer_health_check *
> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
> +                        const char *vip_port_str, bool template);
> +
>  struct ovn_lb_ip_set *
>  ovn_lb_ip_set_create(void)
>  {
> @@ -71,94 +82,293 @@ ovn_lb_ip_set_clone(struct ovn_lb_ip_set *lb_ip_set)
>      return clone;
>  }
>
> -static
> -bool ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> -                     const char *lb_value)
> +/* Format for backend ips: "IP1:port1,IP2:port2,...". */
> +static char *
> +ovn_lb_backends_init_explicit(struct ovn_lb_vip *lb_vip, const char
*value)
>  {
> -    int addr_family;
> -
> -    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
> -                                         &lb_vip->vip, &lb_vip->vip_port,
> -                                         &addr_family)) {
> -        return false;
> -    }
> -
> -    /* Format for backend ips: "IP1:port1,IP2:port2,...". */
> -    size_t n_backends = 0;
> +    struct ds errors = DS_EMPTY_INITIALIZER;
>      size_t n_allocated_backends = 0;
> -    char *tokstr = xstrdup(lb_value);
> +    char *tokstr = xstrdup(value);
>      char *save_ptr = NULL;
> +    lb_vip->n_backends = 0;
> +
>      for (char *token = strtok_r(tokstr, ",", &save_ptr);
>          token != NULL;
>          token = strtok_r(NULL, ",", &save_ptr)) {
>
> -        if (n_backends == n_allocated_backends) {
> +        if (lb_vip->n_backends == n_allocated_backends) {
>              lb_vip->backends = x2nrealloc(lb_vip->backends,
>                                            &n_allocated_backends,
>                                            sizeof *lb_vip->backends);
>          }
>
> -        struct ovn_lb_backend *backend = &lb_vip->backends[n_backends];
> +        struct ovn_lb_backend *backend =
&lb_vip->backends[lb_vip->n_backends];
>          int backend_addr_family;
>          if (!ip_address_and_port_from_lb_key(token, &backend->ip_str,
>                                               &backend->ip,
&backend->port,
>                                               &backend_addr_family)) {
> +            if (lb_vip->port_str) {
> +                ds_put_format(&errors, "%s: should be an IP address and
a "
> +                                       "port number with : as a
separator, ",
> +                              token);
> +            } else {
> +                ds_put_format(&errors, "%s: should be an IP address, ",
token);
> +            }
>              continue;
>          }
>
> -        if (addr_family != backend_addr_family) {
> +        if (lb_vip->address_family != backend_addr_family) {
>              free(backend->ip_str);
> +            ds_put_format(&errors, "%s: IP address family is different
from "
> +                                   "VIP %s, ",
> +                          token, lb_vip->vip_str);
>              continue;
>          }
>
> -        n_backends++;
> +        if (lb_vip->port_str) {
> +            if (!backend->port) {
> +                free(backend->ip_str);
> +                ds_put_format(&errors, "%s: should be an IP address and "
> +                                       "a port number with : as a
separator, ",
> +                              token);
> +                continue;
> +            }
> +        } else {
> +            if (backend->port) {
> +                free(backend->ip_str);
> +                ds_put_format(&errors, "%s: should be an IP address, ",
token);
> +                continue;
> +            }
> +        }
> +
> +        backend->port_str =
> +            backend->port ? xasprintf("%"PRIu16, backend->port) : NULL;
> +        lb_vip->n_backends++;
>      }
>      free(tokstr);
> -    lb_vip->n_backends = n_backends;
> -    return true;
> +
> +    if (ds_last(&errors) != EOF) {
> +        ds_chomp(&errors, ' ');
> +        ds_chomp(&errors, ',');
> +        ds_put_char(&errors, '.');
> +        return ds_steal_cstr(&errors);
> +    }
> +    return NULL;
>  }
>
>  static
> -void ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
> +char *ovn_lb_vip_init_explicit(struct ovn_lb_vip *lb_vip, const char
*lb_key,
> +                               const char *lb_value)
> +{
> +    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
> +                                         &lb_vip->vip, &lb_vip->vip_port,
> +                                         &lb_vip->address_family)) {
> +        return xasprintf("%s: should be an IP address (or an IP address "
> +                         "and a port number with : as a separator).",
lb_key);
> +    }
> +
> +    lb_vip->port_str = lb_vip->vip_port
> +                       ? xasprintf("%"PRIu16, lb_vip->vip_port)
> +                       : NULL;
> +
> +    return ovn_lb_backends_init_explicit(lb_vip, lb_value);
> +}
> +
> +/* Parses backends of a templated LB VIP.
> + * For now only the following template forms are supported:
> + * A.
> + *   ^backendip_variable1[:^port_variable1|:port],
> + *   ^backendip_variable2[:^port_variable2|:port]
> + *
> + * B.
> + *   ^backends_variable1,^backends_variable2 is also a thing
> + *      where 'backends_variable1' may expand to IP1_1:PORT1_1 on
chassis-1
> + *                                               IP1_2:PORT1_2 on
chassis-2
> + *        and 'backends_variable2' may expand to IP2_1:PORT2_1 on
chassis-1
> + *                                               IP2_2:PORT2_2 on
chassis-2
> + */
> +static char *
> +ovn_lb_backends_init_template(struct ovn_lb_vip *lb_vip, const char
*value_)
> +{
> +    struct ds errors = DS_EMPTY_INITIALIZER;
> +    char *value = xstrdup(value_);
> +    char *save_ptr = NULL;
> +    size_t n_allocated_backends = 0;
> +    lb_vip->n_backends = 0;
> +
> +    for (char *backend = strtok_r(value, ",", &save_ptr); backend;
> +         backend = strtok_r(NULL, ",", &save_ptr)) {
> +
> +        char *atom = xstrdup(backend);
> +        char *save_ptr2 = NULL;
> +        bool success = false;
> +        char *backend_ip = NULL;
> +        char *backend_port = NULL;
> +
> +        for (char *subatom = strtok_r(atom, ":", &save_ptr2); subatom;
> +             subatom = strtok_r(NULL, ":", &save_ptr2)) {
> +            if (backend_ip && backend_port) {
> +                success = false;
> +                break;
> +            }
> +            success = true;
> +            if (!backend_ip) {
> +                backend_ip = xstrdup(subatom);
> +            } else {
> +                backend_port = xstrdup(subatom);
> +            }
> +        }
> +
> +        if (success) {
> +            if (lb_vip->n_backends == n_allocated_backends) {
> +                lb_vip->backends = x2nrealloc(lb_vip->backends,
> +                                              &n_allocated_backends,
> +                                              sizeof *lb_vip->backends);
> +            }
> +
> +            struct ovn_lb_backend *lb_backend =
> +                &lb_vip->backends[lb_vip->n_backends];
> +            lb_backend->ip_str = backend_ip;
> +            lb_backend->port_str = backend_port;
> +            lb_backend->port = 0;
> +            lb_vip->n_backends++;
> +        } else {
> +            ds_put_format(&errors, "%s: should be a template of the
form: "
> +
 "'^backendip_variable1[:^port_variable1|:port]', ",
> +                          atom);
> +        }
> +        free(atom);
> +    }
> +
> +    free(value);
> +    if (ds_last(&errors) != EOF) {
> +        ds_chomp(&errors, ' ');
> +        ds_chomp(&errors, ',');
> +        ds_put_char(&errors, '.');
> +        return ds_steal_cstr(&errors);
> +    }
> +    return NULL;
> +}
> +
> +/* Parses a VIP of a templated LB.
> + * For now only the following template forms are supported:
> + *   ^vip_variable[:^port_variable|:port]
> + */
> +static char *
> +ovn_lb_vip_init_template(struct ovn_lb_vip *lb_vip, const char *lb_key_,
> +                         const char *lb_value, int address_family)
> +{
> +    char *save_ptr = NULL;
> +    char *lb_key = xstrdup(lb_key_);
> +    bool success = false;
> +
> +    for (char *atom = strtok_r(lb_key, ":", &save_ptr); atom;
> +         atom = strtok_r(NULL, ":", &save_ptr)) {
> +        if (lb_vip->vip_str && lb_vip->port_str) {
> +            success = false;
> +            break;
> +        }
> +        success = true;
> +        if (!lb_vip->vip_str) {
> +            lb_vip->vip_str = xstrdup(atom);
> +        } else {
> +            lb_vip->port_str = xstrdup(atom);
> +        }
> +    }
> +    free(lb_key);
> +
> +    if (!success) {
> +        return xasprintf("%s: should be a template of the form: "
> +                         "'^vip_variable[:^port_variable|:port]'.",
> +                         lb_key_);
> +    }
> +
> +    lb_vip->address_family = address_family;
> +    return ovn_lb_backends_init_template(lb_vip, lb_value);
> +}
> +
> +/* Returns NULL on success, an error string on failure.  The caller is
> + * responsible for destroying 'lb_vip' in all cases.
> + */
> +char *
> +ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> +                const char *lb_value, bool template, int address_family)
> +{
> +    memset(lb_vip, 0, sizeof *lb_vip);
> +
> +    return !template
> +           ?  ovn_lb_vip_init_explicit(lb_vip, lb_key, lb_value)
> +           :  ovn_lb_vip_init_template(lb_vip, lb_key, lb_value,
> +                                       address_family);
> +}
> +
> +void
> +ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
>  {
>      free(vip->vip_str);
> +    free(vip->port_str);
>      for (size_t i = 0; i < vip->n_backends; i++) {
>          free(vip->backends[i].ip_str);
> +        free(vip->backends[i].port_str);
>      }
>      free(vip->backends);
>  }
>
> +void
> +ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s, bool
template)
> +{
> +    bool needs_brackets = vip->address_family == AF_INET6 &&
vip->port_str
> +                          && !template;
> +    if (needs_brackets) {
> +        ds_put_char(s, '[');
> +    }
> +    ds_put_cstr(s, vip->vip_str);
> +    if (needs_brackets) {
> +        ds_put_char(s, ']');
> +    }
> +    if (vip->port_str) {
> +        ds_put_format(s, ":%s", vip->port_str);
> +    }
> +}
> +
> +void
> +ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct ds *s,
> +                           bool template)
> +{
> +    bool needs_brackets = vip->address_family == AF_INET6 &&
vip->port_str
> +                          && !template;
> +    for (size_t i = 0; i < vip->n_backends; i++) {
> +        struct ovn_lb_backend *backend = &vip->backends[i];
> +
> +        if (needs_brackets) {
> +            ds_put_char(s, '[');
> +        }
> +        ds_put_cstr(s, backend->ip_str);
> +        if (needs_brackets) {
> +            ds_put_char(s, ']');
> +        }
> +        if (backend->port_str) {
> +            ds_put_format(s, ":%s", backend->port_str);
> +        }
> +        if (i != vip->n_backends - 1) {
> +            ds_put_char(s, ',');
> +        }
> +    }
> +}
> +
>  static
>  void ovn_northd_lb_vip_init(struct ovn_northd_lb_vip *lb_vip_nb,
>                              const struct ovn_lb_vip *lb_vip,
>                              const struct nbrec_load_balancer *nbrec_lb,
> -                            const char *vip_port_str, const char
*backend_ips)
> +                            const char *vip_port_str, const char
*backend_ips,
> +                            bool template)
>  {
>      lb_vip_nb->backend_ips = xstrdup(backend_ips);
>      lb_vip_nb->n_backends = lb_vip->n_backends;
>      lb_vip_nb->backends_nb = xcalloc(lb_vip_nb->n_backends,
>                                       sizeof *lb_vip_nb->backends_nb);
> -
> -    struct nbrec_load_balancer_health_check *lb_health_check = NULL;
> -    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
> -        if (nbrec_lb->n_health_check > 0) {
> -            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1,
1);
> -            VLOG_WARN_RL(&rl,
> -                         "SCTP load balancers do not currently support "
> -                         "health checks. Not creating health checks for "
> -                         "load balancer " UUID_FMT,
> -                         UUID_ARGS(&nbrec_lb->header_.uuid));
> -        }
> -    } else {
> -        for (size_t j = 0; j < nbrec_lb->n_health_check; j++) {
> -            if (!strcmp(nbrec_lb->health_check[j]->vip, vip_port_str)) {
> -                lb_health_check = nbrec_lb->health_check[j];
> -                break;
> -            }
> -        }
> -    }
> -
> -    lb_vip_nb->lb_health_check = lb_health_check;
> +    lb_vip_nb->lb_health_check =
> +        ovn_lb_get_health_check(nbrec_lb, vip_port_str, template);
>  }
>
>  static
> @@ -189,12 +399,113 @@ ovn_lb_get_hairpin_snat_ip(const struct uuid
*lb_uuid,
>      }
>  }
>
> +static bool
> +ovn_lb_get_routable_mode(const struct nbrec_load_balancer *nbrec_lb,
> +                         bool routable, bool template)
> +{
> +    if (template && routable) {
> +        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> +        VLOG_WARN_RL(&rl, "Template load balancer "UUID_FMT" does not
suport "
> +                           "option 'add_route'.  Forcing it to
disabled.",
> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> +        return false;
> +    }
> +    return routable;
> +}
> +
> +static bool
> +ovn_lb_neigh_mode_is_valid(enum lb_neighbor_responder_mode mode, bool
template)
> +{
> +    if (!template) {
> +        return true;
> +    }
> +
> +    switch (mode) {
> +    case LB_NEIGH_RESPOND_REACHABLE:
> +        return false;
> +    case LB_NEIGH_RESPOND_ALL:
> +    case LB_NEIGH_RESPOND_NONE:
> +        return true;
> +    }
> +    return false;
> +}
> +
> +static enum lb_neighbor_responder_mode
> +ovn_lb_get_neigh_mode(const struct nbrec_load_balancer *nbrec_lb,
> +                      const char *mode, bool template)
> +{
> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> +    enum lb_neighbor_responder_mode default_mode =
> +        template ? LB_NEIGH_RESPOND_NONE : LB_NEIGH_RESPOND_REACHABLE;
> +
> +    if (!mode) {
> +        mode = lb_neighbor_responder_mode_names[default_mode];
> +    }
> +
> +    for (size_t i = 0; i < ARRAY_SIZE(lb_neighbor_responder_mode_names);
i++) {
> +        if (!strcmp(mode, lb_neighbor_responder_mode_names[i])) {
> +            if (ovn_lb_neigh_mode_is_valid(i, template)) {
> +                return i;
> +            }
> +            break;
> +        }
> +    }
> +
> +    VLOG_WARN_RL(&rl, "Invalid neighbor responder mode %s for load
balancer "
> +                       UUID_FMT", forcing it to %s",
> +                 mode, UUID_ARGS(&nbrec_lb->header_.uuid),
> +                 lb_neighbor_responder_mode_names[default_mode]);
> +    return default_mode;
> +}
> +
> +static struct nbrec_load_balancer_health_check *
> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
> +                        const char *vip_port_str, bool template)
> +{
> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> +
> +    if (!nbrec_lb->n_health_check) {
> +        return NULL;
> +    }
> +
> +    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
> +        VLOG_WARN_RL(&rl,
> +                     "SCTP load balancers do not currently support "
> +                     "health checks. Not creating health checks for "
> +                     "load balancer " UUID_FMT,
> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> +        return NULL;
> +    }
> +
> +    if (template) {
> +        VLOG_WARN_RL(&rl,
> +                     "Template load balancers do not currently support "
> +                     "health checks. Not creating health checks for "
> +                     "load balancer " UUID_FMT,
> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> +        return NULL;
> +    }
> +
> +    for (size_t i = 0; i < nbrec_lb->n_health_check; i++) {
> +        if (!strcmp(nbrec_lb->health_check[i]->vip, vip_port_str)) {
> +            return nbrec_lb->health_check[i];
> +        }
> +    }
> +    return NULL;
> +}
> +
>  struct ovn_northd_lb *
>  ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb)
>  {
> +    bool template = smap_get_bool(&nbrec_lb->options, "template", false);
>      bool is_udp = nullable_string_is_equal(nbrec_lb->protocol, "udp");
>      bool is_sctp = nullable_string_is_equal(nbrec_lb->protocol, "sctp");
>      struct ovn_northd_lb *lb = xzalloc(sizeof *lb);
> +    int address_family = !strcmp(smap_get_def(&nbrec_lb->options,
> +                                              "address-family", "ipv4"),
> +                                 "ipv4")
> +                         ? AF_INET
> +                         : AF_INET6;
>
>      lb->nlb = nbrec_lb;
>      lb->proto = is_udp ? "udp" : is_sctp ? "sctp" : "tcp";
> @@ -202,12 +513,16 @@ ovn_northd_lb_create(const struct
nbrec_load_balancer *nbrec_lb)
>      lb->vips = xcalloc(lb->n_vips, sizeof *lb->vips);
>      lb->vips_nb = xcalloc(lb->n_vips, sizeof *lb->vips_nb);
>      lb->controller_event = smap_get_bool(&nbrec_lb->options, "event",
false);
> -    lb->routable = smap_get_bool(&nbrec_lb->options, "add_route", false);
> +
> +    bool routable = smap_get_bool(&nbrec_lb->options, "add_route",
false);
> +    lb->routable = ovn_lb_get_routable_mode(nbrec_lb, routable,
template);
> +
>      lb->skip_snat = smap_get_bool(&nbrec_lb->options, "skip_snat",
false);
> -    const char *mode =
> -        smap_get_def(&nbrec_lb->options, "neighbor_responder",
"reachable");
> -    lb->neigh_mode = strcmp(mode, "all") ? LB_NEIGH_RESPOND_REACHABLE
> -                                         : LB_NEIGH_RESPOND_ALL;
> +    lb->template = template;
> +
> +    const char *mode = smap_get(&nbrec_lb->options,
"neighbor_responder");
> +    lb->neigh_mode = ovn_lb_get_neigh_mode(nbrec_lb, mode, template);
> +
>      uint32_t affinity_timeout =
>          smap_get_uint(&nbrec_lb->options, "affinity_timeout", 0);
>      if (affinity_timeout > UINT16_MAX) {
> @@ -227,13 +542,19 @@ ovn_northd_lb_create(const struct
nbrec_load_balancer *nbrec_lb)
>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[n_vips];
>
> -        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
> -                                                  "reject", false);
> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
> +        char *error = ovn_lb_vip_init(lb_vip, node->key, node->value,
> +                                      template, address_family);
> +        if (error) {
> +            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5,
1);
> +            VLOG_WARN_RL(&rl, "Failed to initialize LB VIP: %s", error);
> +            ovn_lb_vip_destroy(lb_vip);
> +            free(error);
>              continue;
>          }
> +        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
> +                                                  "reject", false);
>          ovn_northd_lb_vip_init(lb_vip_nb, lb_vip, nbrec_lb,
> -                               node->key, node->value);
> +                               node->key, node->value, template);
>          if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>              sset_add(&lb->ips_v4, lb_vip->vip_str);
>          } else {
> @@ -381,9 +702,12 @@ ovn_lb_group_find(const struct hmap *lb_groups,
const struct uuid *uuid)
>  }
>
>  struct ovn_controller_lb *
> -ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb)
> +ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb,
> +                         const struct smap *template_vars,
> +                         struct sset *template_vars_ref)
>  {
>      struct ovn_controller_lb *lb = xzalloc(sizeof *lb);
> +    bool template = smap_get_bool(&sbrec_lb->options, "template", false);
>
>      lb->slb = sbrec_lb;
>      lb->n_vips = smap_count(&sbrec_lb->vips);
> @@ -395,10 +719,26 @@ ovn_controller_lb_create(const struct
sbrec_load_balancer *sbrec_lb)
>      SMAP_FOR_EACH (node, &sbrec_lb->vips) {
>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
>
> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
> -            continue;
> +        struct lex_str key_s = template
> +                               ? lexer_parse_template_string(node->key,
> +
template_vars,
> +
template_vars_ref)
> +                               : lex_str_use(node->key);
> +        struct lex_str value_s = template
> +                               ? lexer_parse_template_string(node->value,
> +
template_vars,
> +
template_vars_ref)
> +                               : lex_str_use(node->value);
> +        char *error = ovn_lb_vip_init_explicit(lb_vip,
> +                                               lex_str_get(&key_s),
> +                                               lex_str_get(&value_s));
> +        if (error) {
> +            free(error);
> +        } else {
> +            n_vips++;
>          }
> -        n_vips++;
> +        lex_str_free(&key_s);
> +        lex_str_free(&value_s);
>      }
>
>      /* It's possible that parsing VIPs fails.  Update the lb->n_vips to
the
> diff --git a/lib/lb.h b/lib/lb.h
> index 62843e4716..55a41ae0bc 100644
> --- a/lib/lb.h
> +++ b/lib/lb.h
> @@ -35,6 +35,7 @@ struct uuid;
>  enum lb_neighbor_responder_mode {
>      LB_NEIGH_RESPOND_REACHABLE,
>      LB_NEIGH_RESPOND_ALL,
> +    LB_NEIGH_RESPOND_NONE,
>  };
>
>  /* The "routable" ssets are subsets of the load balancer IPs for which IP
> @@ -67,6 +68,7 @@ struct ovn_northd_lb {
>      bool controller_event;
>      bool routable;
>      bool skip_snat;
> +    bool template;
>      uint16_t affinity_timeout;
>
>      struct sset ips_v4;
> @@ -82,19 +84,31 @@ struct ovn_northd_lb {
>  };
>
>  struct ovn_lb_vip {
> -    struct in6_addr vip;
> -    char *vip_str;
> -    uint16_t vip_port;
> -
> +    struct in6_addr vip; /* Only used in ovn-controller. */
> +    char *vip_str;       /* Actual VIP string representation (without
port).
> +                          * To be used in ovn-northd.
> +                          */
> +    uint16_t vip_port;   /* Only used in ovn-controller. */
> +    char *port_str;      /* Actual port string representation.  To be
used
> +                          * in ovn-northd.
> +                          */
>      struct ovn_lb_backend *backends;
>      size_t n_backends;
>      bool empty_backend_rej;
> +    int address_family;
>  };
>
>  struct ovn_lb_backend {
> -    struct in6_addr ip;
> -    char *ip_str;
> -    uint16_t port;
> +    struct in6_addr ip;  /* Only used in ovn-controller. */
> +    char *ip_str;        /* Actual IP string representation. To be used
in
> +                          * ovn-northd.
> +                          */
> +    uint16_t port;       /* Mostly used in ovn-controller but also for
> +                          * healthcheck in ovn-northd.
> +                          */
> +    char *port_str;      /* Actual port string representation. To be used
> +                          * in ovn-northd.
> +                          */
>  };
>
>  /* ovn-northd specific backend information. */
> @@ -174,7 +188,17 @@ struct ovn_controller_lb {
>  };
>
>  struct ovn_controller_lb *ovn_controller_lb_create(
> -    const struct sbrec_load_balancer *);
> +    const struct sbrec_load_balancer *,
> +    const struct smap *template_vars,
> +    struct sset *template_vars_ref);
>  void ovn_controller_lb_destroy(struct ovn_controller_lb *);
>
> +char *ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> +                      const char *lb_value, bool template, int
address_family);
> +void ovn_lb_vip_destroy(struct ovn_lb_vip *vip);
> +void ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s,
> +                       bool template);
> +void ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct ds
*s,
> +                                bool template);
> +
>  #endif /* OVN_LIB_LB_H 1 */
> diff --git a/lib/ovn-util.c b/lib/ovn-util.c
> index 597625a291..1f8d0b8add 100644
> --- a/lib/ovn-util.c
> +++ b/lib/ovn-util.c
> @@ -793,9 +793,6 @@ ip_address_and_port_from_lb_key(const char *key, char
**ip_address,
>  {
>      struct sockaddr_storage ss;
>      if (!inet_parse_active(key, 0, &ss, false, NULL)) {
> -        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
> -        VLOG_WARN_RL(&rl, "bad ip address or port for load balancer key
%s",
> -                     key);
>          *ip_address = NULL;
>          memset(ip, 0, sizeof(*ip));
>          *port = 0;
> diff --git a/northd/northd.c b/northd/northd.c
> index 123127e9c1..c590c14818 100644
> --- a/northd/northd.c
> +++ b/northd/northd.c
> @@ -3740,6 +3740,10 @@ static void
>  ovn_lb_svc_create(struct ovsdb_idl_txn *ovnsb_txn, struct ovn_northd_lb
*lb,
>                    struct hmap *monitor_map, struct hmap *ports)
>  {
> +    if (lb->template) {
> +        return;
> +    }
> +
>      for (size_t i = 0; i < lb->n_vips; i++) {
>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
> @@ -4056,12 +4060,19 @@ static void
>  build_lrouter_lb_reachable_ips(struct ovn_datapath *od,
>                                 const struct ovn_northd_lb *lb)
>  {
> +    /* If configured to not reply to any neighbor requests for all VIPs
> +     * return early.
> +     */
> +    if (lb->neigh_mode == LB_NEIGH_RESPOND_NONE) {
> +        return;
> +    }
> +
>      /* If configured to reply to neighbor requests for all VIPs force
them
>       * all to be considered "reachable".
>       */
>      if (lb->neigh_mode == LB_NEIGH_RESPOND_ALL) {
>          for (size_t i = 0; i < lb->n_vips; i++) {
> -            if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
> +            if (lb->vips[i].address_family == AF_INET) {
>                  sset_add(&od->lb_ips->ips_v4_reachable,
lb->vips[i].vip_str);
>              } else {
>                  sset_add(&od->lb_ips->ips_v6_reachable,
lb->vips[i].vip_str);
> @@ -4073,8 +4084,9 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath
*od,
>      /* Otherwise, a VIP is reachable if there's at least one router
>       * subnet that includes it.
>       */
> +    ovs_assert(lb->neigh_mode == LB_NEIGH_RESPOND_REACHABLE);
>      for (size_t i = 0; i < lb->n_vips; i++) {
> -        if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
> +        if (lb->vips[i].address_family == AF_INET) {
>              ovs_be32 vip_ip4 =
in6_addr_get_mapped_ipv4(&lb->vips[i].vip);
>              struct ovn_port *op;
>
> @@ -5834,16 +5846,16 @@ build_empty_lb_event_flow(struct ovn_lb_vip
*lb_vip,
>      ds_clear(action);
>      ds_clear(match);
>
> -    bool ipv4 = IN6_IS_ADDR_V4MAPPED(&lb_vip->vip);
> +    bool ipv4 = lb_vip->address_family == AF_INET;
>
>      ds_put_format(match, "ip%s.dst == %s && %s",
>                    ipv4 ? "4": "6", lb_vip->vip_str, lb->proto);
>
>      char *vip = lb_vip->vip_str;
> -    if (lb_vip->vip_port) {
> -        ds_put_format(match, " && %s.dst == %u", lb->proto,
lb_vip->vip_port);
> -        vip = xasprintf("%s%s%s:%u", ipv4 ? "" : "[", lb_vip->vip_str,
> -                        ipv4 ? "" : "]", lb_vip->vip_port);
> +    if (lb_vip->port_str) {
> +        ds_put_format(match, " && %s.dst == %s", lb->proto,
lb_vip->port_str);
> +        vip = xasprintf("%s%s%s:%s", ipv4 ? "" : "[", lb_vip->vip_str,
> +                        ipv4 ? "" : "]", lb_vip->port_str);
>      }
>
>      ds_put_format(action,
> @@ -5854,7 +5866,7 @@ build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip,
>                    event_to_string(OVN_EVENT_EMPTY_LB_BACKENDS),
>                    vip, lb->proto,
>                    UUID_ARGS(&lb->nlb->header_.uuid));
> -    if (lb_vip->vip_port) {
> +    if (lb_vip->port_str) {
>          free(vip);
>      }
>      return true;
> @@ -6910,7 +6922,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
struct ovn_northd_lb *lb,
>          /* Store the original destination IP to be used when generating
>           * hairpin flows.
>           */
> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> +        if (lb->vips[i].address_family == AF_INET) {
>              ip_match = "ip4";
>              ds_put_format(action, REG_ORIG_DIP_IPV4 " = %s; ",
>                            lb_vip->vip_str);
> @@ -6921,7 +6933,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
struct ovn_northd_lb *lb,
>          }
>
>          const char *proto = NULL;
> -        if (lb_vip->vip_port) {
> +        if (lb_vip->port_str) {
>              proto = "tcp";
>              if (lb->nlb->protocol) {
>                  if (!strcmp(lb->nlb->protocol, "udp")) {
> @@ -6934,14 +6946,14 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
struct ovn_northd_lb *lb,
>              /* Store the original destination port to be used when
generating
>               * hairpin flows.
>               */
> -            ds_put_format(action, REG_ORIG_TP_DPORT " = %"PRIu16"; ",
> -                          lb_vip->vip_port);
> +            ds_put_format(action, REG_ORIG_TP_DPORT " = %s; ",
> +                          lb_vip->port_str);
>          }
>          ds_put_format(action, "%s;", ct_lb_mark ? "ct_lb_mark" :
"ct_lb");
>
>          ds_put_format(match, "%s.dst == %s", ip_match, lb_vip->vip_str);
> -        if (lb_vip->vip_port) {
> -            ds_put_format(match, " && %s.dst == %d", proto,
lb_vip->vip_port);
> +        if (lb_vip->port_str) {
> +            ds_put_format(match, " && %s.dst == %s", proto,
lb_vip->port_str);
>          }
>
>          struct ovn_lflow *lflow_ref = NULL;
> @@ -7192,24 +7204,12 @@ build_lb_rules(struct hmap *lflows, struct
ovn_northd_lb *lb, bool ct_lb_mark,
>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
>          const char *ip_match = NULL;
> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> +        if (lb_vip->address_family == AF_INET) {
>              ip_match = "ip4";
>          } else {
>              ip_match = "ip6";
>          }
>
> -        const char *proto = NULL;
> -        if (lb_vip->vip_port) {
> -            proto = "tcp";
> -            if (lb->nlb->protocol) {
> -                if (!strcmp(lb->nlb->protocol, "udp")) {
> -                    proto = "udp";
> -                } else if (!strcmp(lb->nlb->protocol, "sctp")) {
> -                    proto = "sctp";
> -                }
> -            }
> -        }
> -
>          ds_clear(action);
>          ds_clear(match);
>
> @@ -7227,8 +7227,9 @@ build_lb_rules(struct hmap *lflows, struct
ovn_northd_lb *lb, bool ct_lb_mark,
>          ds_put_format(match, "ct.new && %s.dst == %s", ip_match,
>                        lb_vip->vip_str);
>          int priority = 110;
> -        if (lb_vip->vip_port) {
> -            ds_put_format(match, " && %s.dst == %d", proto,
lb_vip->vip_port);
> +        if (lb_vip->port_str) {
> +            ds_put_format(match, " && %s.dst == %s", lb->proto,
> +                          lb_vip->port_str);
>              priority = 120;
>          }
>
> @@ -10231,7 +10232,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
*lb_vip,
>       * of "ct_lb_mark($targets);". The other flow is for ct.est with
>       * an action of "next;".
>       */
> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> +    if (lb_vip->address_family == AF_INET) {
>          ds_put_format(match, "ip4 && "REG_NEXT_HOP_IPV4" == %s",
>                        lb_vip->vip_str);
>      } else {
> @@ -10247,14 +10248,14 @@ build_lrouter_nat_flows_for_lb(struct
ovn_lb_vip *lb_vip,
>      }
>
>      int prio = 110;
> -    if (lb_vip->vip_port) {
> +    if (lb_vip->port_str) {
>          prio = 120;
>          new_match = xasprintf("ct.new && %s && %s && "
> -                              REG_ORIG_TP_DPORT_ROUTER" == %d",
> -                              ds_cstr(match), lb->proto,
lb_vip->vip_port);
> +                              REG_ORIG_TP_DPORT_ROUTER" == %s",
> +                              ds_cstr(match), lb->proto,
lb_vip->port_str);
>          est_match = xasprintf("ct.est && %s && %s && "
> -                              REG_ORIG_TP_DPORT_ROUTER" == %d && %s ==
1",
> -                              ds_cstr(match), lb->proto,
lb_vip->vip_port,
> +                              REG_ORIG_TP_DPORT_ROUTER" == %s && %s ==
1",
> +                              ds_cstr(match), lb->proto,
lb_vip->port_str,
>                                ct_natted);
>      } else {
>          new_match = xasprintf("ct.new && %s", ds_cstr(match));
> @@ -10263,7 +10264,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
*lb_vip,
>      }
>
>      const char *ip_match = NULL;
> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> +    if (lb_vip->address_family == AF_INET) {
>          ip_match = "ip4";
>      } else {
>          ip_match = "ip6";
> @@ -10281,9 +10282,9 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
*lb_vip,
>          ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
>                        backend->ip_str);
>
> -        if (backend->port) {
> -            ds_put_format(&undnat_match, " && %s.src == %d) || ",
> -                          lb->proto, backend->port);
> +        if (backend->port_str) {
> +            ds_put_format(&undnat_match, " && %s.src == %s) || ",
> +                          lb->proto, backend->port_str);
>          } else {
>              ds_put_cstr(&undnat_match, ") || ");
>          }
> @@ -10296,9 +10297,9 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
*lb_vip,
>      struct ds unsnat_match = DS_EMPTY_INITIALIZER;
>      ds_put_format(&unsnat_match, "%s && %s.dst == %s && %s",
>                    ip_match, ip_match, lb_vip->vip_str, lb->proto);
> -    if (lb_vip->vip_port) {
> -        ds_put_format(&unsnat_match, " && %s.dst == %d", lb->proto,
> -                      lb_vip->vip_port);
> +    if (lb_vip->port_str) {
> +        ds_put_format(&unsnat_match, " && %s.dst == %s", lb->proto,
> +                      lb_vip->port_str);
>      }
>
>      struct ovn_datapath **gw_router_skip_snat =
> @@ -10571,7 +10572,7 @@ build_lrouter_defrag_flows_for_lb(struct
ovn_northd_lb *lb,
>          ds_clear(&defrag_actions);
>          ds_clear(match);
>
> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> +        if (lb_vip->address_family == AF_INET) {
>              ds_put_format(match, "ip && ip4.dst == %s", lb_vip->vip_str);
>              ds_put_format(&defrag_actions, REG_NEXT_HOP_IPV4" = %s; ",
>                            lb_vip->vip_str);
> @@ -10581,7 +10582,7 @@ build_lrouter_defrag_flows_for_lb(struct
ovn_northd_lb *lb,
>                            lb_vip->vip_str);
>          }
>
> -        if (lb_vip->vip_port) {
> +        if (lb_vip->port_str) {
>              ds_put_format(match, " && %s", lb->proto);
>              prio = 110;
>
> diff --git a/ovn-nb.xml b/ovn-nb.xml
> index 553c0e48c3..8cd2427e8f 100644
> --- a/ovn-nb.xml
> +++ b/ovn-nb.xml
> @@ -1905,8 +1905,66 @@
>          is applied reply to ARP/neighbor discovery requests for all VIPs
>          of the load balancer.  If set to <code>reachable</code>, then
routers
>          on which the load balancer is applied reply to ARP/neighbor
discovery
> -        requests only for VIPs that are part of a router's subnet.  The
default
> -        value of this option, if not specified, is
<code>reachable</code>.
> +        requests only for VIPs that are part of a router's subnet.  If
set to
> +        <code>none</code>, then routers on which the load balancer is
applied
> +        never reply to ARP/neighbor discovery requests for any of the
load
> +        balancer VIPs. Load balancers with
<code>options:template=true</code>
> +        do not support <code>reachable</code> as a valid mode.  The
default
> +        value of this option, if not specified, is
<code>reachable</code> for
> +        regular load balancers and <code>none</code> for template load
> +        balancers.
> +      </column>
> +
> +      <column name="options" key="template">
> +        <p>
> +          Option to be set to <code>true</code>, if the load balancer is
a
> +          template.  The load balancer VIPs and backends must be using
> +          <ref table="Chassis_Template_Var"/> in their definitions.
> +        </p>
> +
> +        <p>
> +          Load balancer template VIP supported formats are:
> +        </p>
> +        <pre>
> +^VIP_VAR[:^PORT_VAR|:port]
> +        </pre>
> +
> +        <p>
> +          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are names
of
> +        <ref table="Chassis_Template_Var"/> records.

In this version, the vars are not names but keys of the "variables" column.

> +        </p>
> +
> +        <p>
> +          Note: The VIP and PORT cannot be combined into a single
template
> +          variable. For example, a <ref table="Chassis_Template_Var"/>
> +          variable expanding to <code>10.0.0.1:8080</code> is not valid
> +          if used as VIP.
> +        </p>
> +
> +        <p>
> +          Load balancer template backend supported formats are:
> +        </p>
> +        <pre>
> +^BACKEND_VAR1[:^PORT_VAR1|:port],^BACKEND_VAR2[:^PORT_VAR2|:port]
> +
> +or
> +
> +^BACKENDS_VAR1,^BACKENDS_VAR2

I think here each var means a single backend IP, right? So,
s/BACKENDS/BACKEND/g

> +        </pre>
> +        <p>
> +          where <code>BACKEND_VAR1</code>, <code>PORT_VAR1</code>,
> +          <code>BACKEND_VAR2</code>, <code>PORT_VAR2</code>,
> +          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
names

Same here, and they are keys instead of "names".

Acked-by: Han Zhou <hzhou@ovn.org>

> +          of <ref table="Chassis_Template_Var"/> records.
> +        </p>
> +      </column>
> +
> +      <column name="options" key="address-family">
> +        Address family used by the load balancer.  Supported values are
> +        <code>ipv4</code> and <code>ipv6</code>.  The address-family is
> +        only used for load balancers with
<code>options:template=true</code>.
> +        For explicit load balancers, setting the address-family has no
> +        effect.
>        </column>
>
>        <column name="options" key="affinity_timeout">
> diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at
> index 4d480e3573..9da7c26b31 100644
> --- a/tests/ovn-nbctl.at
> +++ b/tests/ovn-nbctl.at
> @@ -857,23 +857,19 @@ AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0
30.0.0.10 192.168.10.10:a80], [
>  [ovn-nbctl: 192.168.10.10:a80: should be an IP address.
>  ])
>
> -AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 30.0.0.10 192.168.10.10:],
[1], [],
> -[ovn-nbctl: 192.168.10.10:: should be an IP address.
> -])
> -
>  AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 30.0.0.10
192.168.10.1a], [1], [],
>  [ovn-nbctl: 192.168.10.1a: should be an IP address.
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10: 192.168.10.10:80,
192.168.10.20:80 tcp], [1], [],
> -[ovn-nbctl: Protocol is unnecessary when no port of vip is given.
> +[ovn-nbctl: 192.168.10.10:80: should be an IP address, 192.168.10.20:80:
should be an IP address.
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10 192.168.10.10 tcp], [1], [],
>  [ovn-nbctl: Protocol is unnecessary when no port of vip is given.
>  ])
>
> -AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10 192.168.10.10:900 tcp], [1], [],
> +AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10 192.168.10.10 tcp], [1], [],
>  [ovn-nbctl: Protocol is unnecessary when no port of vip is given.
>  ])
>
> @@ -1111,7 +1107,7 @@ AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0
ae0f::10fff [[fd0f::10]]:80,fd0
>
>
>  AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 ae0f::10
[[fd0f::10]]:80,[[fd0f::20]]:80], [1], [],
> -[ovn-nbctl: [[fd0f::10]]:80: should be an IP address.
> +[ovn-nbctl: [[fd0f::10]]:80: should be an IP address, [[fd0f::20]]:80:
should be an IP address.
>  ])
>
>
> @@ -1125,18 +1121,13 @@ AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0
ae0f::10 [[fd0f::10]]:a80], [1]
>  ])
>
>
> -AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 ae0f::10
[[fd0f::10]]:], [1], [],
> -[ovn-nbctl: [[fd0f::10]]:: should be an IP address.
> -])
> -
> -
>  AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 ae0f::10 fd0f::1001a],
[1], [],
>  [ovn-nbctl: fd0f::1001a: should be an IP address.
>  ])
>
>
>  AT_CHECK([ovn-nbctl -vsocket_util:off lb-add lb0 [[ae0f::10]]:
[[fd0f::10]]:80,[[fd0f::20]]:80 tcp], [1], [],
> -[ovn-nbctl: Protocol is unnecessary when no port of vip is given.
> +[ovn-nbctl: [[fd0f::10]]:80: should be an IP address, [[fd0f::20]]:80:
should be an IP address.
>  ])
>
>
> @@ -1146,7 +1137,7 @@ AT_CHECK([ovn-nbctl lb-add lb0 ae0f::10 fd0f::10
tcp], [1], [],
>
>
>  AT_CHECK([ovn-nbctl lb-add lb0 ae0f::10 [[fd0f::10]]:900 tcp], [1], [],
> -[ovn-nbctl: Protocol is unnecessary when no port of vip is given.
> +[ovn-nbctl: [[fd0f::10]]:900: should be an IP address.
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 ae0f::10 192.168.10.10], [1], [],
> @@ -1158,7 +1149,7 @@ AT_CHECK([ovn-nbctl lb-add lb0 ae0f::10
192.168.10.10], [1], [],
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 [[ae0f::10]]:80 192.168.10.10:80], [1],
[],
> -[ovn-nbctl: 192.168.10.10:80: IP address family is different from VIP
[[ae0f::10]]:80.
> +[ovn-nbctl: 192.168.10.10:80: IP address family is different from VIP
ae0f::10.
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10 ae0f::10], [1], [],
> @@ -1166,7 +1157,7 @@ AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10 ae0f::10],
[1], [],
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 30.0.0.10:80 [[ae0f::10]]:80], [1], [],
> -[ovn-nbctl: [[ae0f::10]]:80: IP address family is different from VIP
30.0.0.10:80.
> +[ovn-nbctl: [[ae0f::10]]:80: IP address family is different from VIP
30.0.0.10.
>  ])
>
>  AT_CHECK([ovn-nbctl lb-add lb0 ae0f::10 fd0f::10])
> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> index 86ab376fd6..770ccbdcda 100644
> --- a/tests/ovn-northd.at
> +++ b/tests/ovn-northd.at
> @@ -1828,6 +1828,11 @@ ovn-nbctl set Load_Balancer lb8
options:neighbor_responder=all
>  ovn-nbctl lb-add lb9 "[[4444::4444]]:8080" "[[10::10]]:8080" udp
>  ovn-nbctl set Load_Balancer lb9 options:neighbor_responder=all
>
> +ovn-nbctl lb-add lb10 "55.55.55.55:8080" "10.0.0.8:8080" udp
> +ovn-nbctl set Load_Balancer lb10 options:neighbor_responder=none
> +ovn-nbctl lb-add lb11 "[[5555::5555]]:8080" "[[10::10]]:8080" udp
> +ovn-nbctl set Load_Balancer lb11 options:neighbor_responder=none
> +
>  ovn-nbctl lr-lb-add lr lb1
>  ovn-nbctl lr-lb-add lr lb2
>  ovn-nbctl lr-lb-add lr lb3
> @@ -1837,6 +1842,8 @@ ovn-nbctl lr-lb-add lr lb6
>  ovn-nbctl lr-lb-add lr lb7
>  ovn-nbctl lr-lb-add lr lb8
>  ovn-nbctl lr-lb-add lr lb9
> +ovn-nbctl lr-lb-add lr lb10
> +ovn-nbctl lr-lb-add lr lb11
>
>  ovn-nbctl --wait=sb sync
>  lr_key=$(fetch_column sb:datapath_binding tunnel_key
external_ids:name=lr)
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 68e788e78f..7c1f2acd04 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -33145,3 +33145,134 @@ AT_CHECK([ovs-ofctl dump-flows br-int | grep
'42\.42\.42\.42'], [1], [])
>  OVN_CLEANUP([hv1])
>  AT_CLEANUP
>  ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([Load balancers with Chassis_Template_Var references])
> +AT_KEYWORDS([templates])
> +ovn_start
> +net_add n1
> +
> +sim_add hv1
> +as hv1
> +ovs-vsctl add-br br-phys
> +ovn_attach n1 br-phys 192.168.0.1
> +
> +check ovn-nbctl ls-add sw
> +
> +dnl Use --wait=sb to ensure lsp1 getting a tunnel_key before lsp2.
> +check ovn-nbctl --wait=sb lsp-add sw lsp1
> +check ovn-nbctl --wait=sb lsp-add sw lsp2
> +
> +AT_CHECK([ovn-nbctl create Chassis_Template_Var chassis=hv1], [0],
[ignore])
> +
> +dnl Create a few LBs that use "uninstantiated" templates.
> +check ovn-nbctl --template lb-add lb-test1 "^VIP1:^VPORT1" "^BACKENDS1"
tcp
> +check ovn-nbctl --template lb-add lb-test2 "^VIP2:^VPORT2"
"^BACKENDS21,^BACKENDS22" tcp
> +check ovn-nbctl --template lb-add lb-test3 "^VIP3:^VPORT3"
"^BACKENDS31:^BPORT1,^BACKENDS32:^BPORT2" tcp
> +check ovn-nbctl ls-lb-add sw lb-test1
> +check ovn-nbctl ls-lb-add sw lb-test2
> +check ovn-nbctl ls-lb-add sw lb-test3
> +
> +check ovs-vsctl add-port br-int p1 -- set interface p1
external_ids:iface-id=lsp1
> +check ovs-vsctl add-port br-int p2 -- set interface p2
external_ids:iface-id=lsp2
> +
> +wait_for_ports_up
> +ovn-nbctl --wait=hv sync
> +
> +dnl Ensure the LBs are not translated to OpenFlow.
> +as hv1
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat'], [1], [])
> +
> +dnl Create Chassis_Template_Var mappings.
> +check ovn-nbctl --wait=hv set Chassis_Template_Var hv1 \
> +    variables:VIP1='43.43.43.1' variables:VPORT1='4301' \
> +    variables:BACKENDS1='85.85.85.1:8501' \
> +    variables:VIP2='43.43.43.2' variables:VPORT2='4302' \
> +    variables:BACKENDS21='85.85.85.21:8502' \
> +    variables:BACKENDS22='85.85.85.22:8502' \
> +    variables:VIP3='43.43.43.3' variables:VPORT3='4303' \
> +    variables:BACKENDS31='85.85.85.31' \
> +    variables:BACKENDS32='85.85.85.32' \
> +    variables:BPORT1='8503' variables:BPORT2='8503'
> +
> +dnl Ensure the LBs are translated to OpenFlow.
> +as hv1
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=85.85.85.1:8501)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=85.85.85.21:8502)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=85.85.85.22:8502)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=85.85.85.31:8503)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=85.85.85.32:8503)'
-c], [0], [dnl
> +1
> +])
> +
> +dnl Ensure hairpin flows are correct.
> +as hv1
> +AT_CHECK([ovs-ofctl dump-flows br-int | grep table=68 |
ofctl_strip_all], [0], [dnl
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b01,reg2=0x10cd/0xffff,nw_src=85.85.85.1,nw_dst=85.85.85.1,tp_dst=8501
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.1,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b02,reg2=0x10ce/0xffff,nw_src=85.85.85.21,nw_dst=85.85.85.21,tp_dst=8502
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b02,reg2=0x10ce/0xffff,nw_src=85.85.85.22,nw_dst=85.85.85.22,tp_dst=8502
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.31,nw_dst=85.85.85.31,tp_dst=8503
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.32,nw_dst=85.85.85.32,tp_dst=8503
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> +])
> +
> +dnl Change Chassis_Template_Var mappings
> +check ovn-nbctl --wait=hv set Chassis_Template_Var hv1 \
> +    variables:VIP1='42.42.42.1' variables:VPORT1='4201' \
> +    variables:BACKENDS1='84.84.84.1:8401' \
> +    variables:VIP2='42.42.42.2' variables:VPORT2='4202' \
> +    variables:BACKENDS21='84.84.84.21:8402' \
> +    variables:BACKENDS22='84.84.84.22:8402' \
> +    variables:VIP3='42.42.42.3' variables:VPORT3='4203' \
> +    variables:BACKENDS31='84.84.84.31' \
> +    variables:BACKENDS32='84.84.84.32' \
> +    variables:BPORT1='8403' variables:BPORT2='8403'
> +
> +dnl Ensure the LBs are translated to OpenFlow.
> +as hv1
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=84.84.84.1:8401)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=84.84.84.21:8402)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=84.84.84.22:8402)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=84.84.84.31:8403)'
-c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat(dst=84.84.84.32:8403)'
-c], [0], [dnl
> +1
> +])
> +
> +dnl Ensure hairpin flows are correct.
> +as hv1
> +AT_CHECK([ovs-ofctl dump-flows br-int | grep table=68 |
ofctl_strip_all], [0], [dnl
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a01,reg2=0x1069/0xffff,nw_src=84.84.84.1,nw_dst=84.84.84.1,tp_dst=8401
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.1,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a02,reg2=0x106a/0xffff,nw_src=84.84.84.21,nw_dst=84.84.84.21,tp_dst=8402
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a02,reg2=0x106a/0xffff,nw_src=84.84.84.22,nw_dst=84.84.84.22,tp_dst=8402
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.31,nw_dst=84.84.84.31,tp_dst=8403
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.32,nw_dst=84.84.84.32,tp_dst=8403
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> +])
> +
> +dnl Remove Chassis_Template_Variables and check that everything is
> +dnl removed from OpenFlow.
> +check ovn-nbctl --wait=hv clear Chassis_Template_Var hv1 variables
> +
> +as hv1
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep 'nat'], [1], [])
> +
> +as hv1
> +AT_CHECK([ovs-ofctl dump-flows br-int | grep table=68 |
ofctl_strip_all], [0], [])
> +
> +OVN_CLEANUP([hv1])
> +AT_CLEANUP
> +])
> diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> index cb34127174..a15e332de6 100644
> --- a/tests/system-ovn.at
> +++ b/tests/system-ovn.at
> @@ -9138,3 +9138,186 @@ OVS_TRAFFIC_VSWITCHD_STOP(["/failed to query port
patch-.*/d
>  /connection dropped.*/d"])
>  AT_CLEANUP
>  ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([load-balancer template IPv4])
> +AT_SKIP_IF([test $HAVE_NC = no])
> +AT_KEYWORDS([ovnlb templates])
> +
> +CHECK_CONNTRACK()
> +CHECK_CONNTRACK_NAT()
> +ovn_start
> +OVS_TRAFFIC_VSWITCHD_START()
> +OVS_CHECK_CT_ZERO_SNAT()
> +ADD_BR([br-int])
> +
> +# Set external-ids in br-int needed for ovn-controller
> +ovs-vsctl \
> +        -- set Open_vSwitch . external-ids:system-id=hv1 \
> +        -- set Open_vSwitch .
external-ids:ovn-remote=unix:$ovs_base/ovn-sb/ovn-sb.sock \
> +        -- set Open_vSwitch . external-ids:ovn-encap-type=geneve \
> +        -- set Open_vSwitch . external-ids:ovn-encap-ip=169.0.0.1 \
> +        -- set bridge br-int fail-mode=secure
other-config:disable-in-band=true
> +
> +# Start ovn-controller
> +start_daemon ovn-controller
> +
> +# Logical network:
> +# VM1 -- LS1 -- GW-Router -- LS2 -- VM3
> +#         |
> +# VM2 ----+
> +#
> +# A templated load balancer applied on LS1 and GW-Router with
> +# VM1 as backend.  The VIP should be accessible from both VM2 and VM3.
> +
> +check ovn-nbctl                                                   \
> +    -- lr-add rtr                                                 \
> +    -- set Logical_Router rtr options:chassis=hv1                 \
> +    -- lrp-add rtr rtr-ls1 00:00:00:00:01:00 42.42.42.1/24        \
> +    -- lrp-add rtr rtr-ls2 00:00:00:00:02:00 43.43.43.1/24        \
> +    -- ls-add ls1                                                 \
> +    -- lsp-add ls1 ls1-rtr                                        \
> +    -- lsp-set-addresses ls1-rtr 00:00:00:00:01:00                \
> +    -- lsp-set-type ls1-rtr router                                \
> +    -- lsp-set-options ls1-rtr router-port=rtr-ls1                \
> +    -- lsp-add ls1 vm1 -- lsp-set-addresses vm1 00:00:00:00:00:01 \
> +    -- lsp-add ls1 vm2 -- lsp-set-addresses vm2 00:00:00:00:00:02 \
> +    -- ls-add ls2                                                 \
> +    -- lsp-add ls2 ls2-rtr                                        \
> +    -- lsp-set-addresses ls2-rtr 00:00:00:00:02:00                \
> +    -- lsp-set-type ls2-rtr router                                \
> +    -- lsp-set-options ls2-rtr router-port=rtr-ls2                \
> +    -- lsp-add ls2 vm3 -- lsp-set-addresses vm3 00:00:00:00:00:03
> +
> +# Add a template LB that eventually expands to:
> +# VIP=66.66.66.66:666 backends=42.42.42.2:4242 proto=tcp
> +
> +AT_CHECK([ovn-nbctl -- create chassis_template_var chassis="hv1"
variables="{vip=66.66.66.66,vport=666,backends=\"42.42.42.2:4242\"}"],
> +         [0], [ignore])
> +
> +check ovn-nbctl --template lb-add lb-test "^vip:^vport" "^backends" tcp \
> +    -- ls-lb-add ls1 lb-test                                            \
> +    -- lr-lb-add rtr lb-test
> +
> +ADD_NAMESPACES(vm1)
> +ADD_VETH(vm1, vm1, br-int, "42.42.42.2/24", "00:00:00:00:00:01",
"42.42.42.1")
> +
> +ADD_NAMESPACES(vm2)
> +ADD_VETH(vm2, vm2, br-int, "42.42.42.3/24", "00:00:00:00:00:02",
"42.42.42.1")
> +
> +ADD_NAMESPACES(vm3)
> +ADD_VETH(vm3, vm3, br-int, "43.43.43.2/24", "00:00:00:00:00:03",
"43.43.43.1")
> +
> +# Wait for ovn-controller to catch up.
> +wait_for_ports_up
> +check ovn-nbctl --wait=hv sync
> +
> +AT_CHECK([ovn-appctl -t ovn-controller debug/dump-local-template-vars |
sort], [0], [dnl
> +Local template vars:
> +name: 'backends' value: '42.42.42.2:4242'
> +name: 'vip' value: '66.66.66.66'
> +name: 'vport' value: '666'
> +])
> +
> +# Start IPv4 TCP server on vm1.
> +NETNS_DAEMONIZE([vm1], [nc -k -l 42.42.42.2 4242], [nc-vm1.pid])
> +
> +# Make sure connecting to the VIP works.
> +NS_CHECK_EXEC([vm2], [nc 66.66.66.66 666 -z], [0], [ignore], [ignore])
> +NS_CHECK_EXEC([vm3], [nc 66.66.66.66 666 -z], [0], [ignore], [ignore])
> +
> +AT_CLEANUP
> +])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([load-balancer template IPv6])
> +AT_SKIP_IF([test $HAVE_NC = no])
> +AT_KEYWORDS([ovnlb templates])
> +
> +CHECK_CONNTRACK()
> +CHECK_CONNTRACK_NAT()
> +ovn_start
> +OVS_TRAFFIC_VSWITCHD_START()
> +OVS_CHECK_CT_ZERO_SNAT()
> +ADD_BR([br-int])
> +
> +# Set external-ids in br-int needed for ovn-controller
> +ovs-vsctl \
> +        -- set Open_vSwitch . external-ids:system-id=hv1 \
> +        -- set Open_vSwitch .
external-ids:ovn-remote=unix:$ovs_base/ovn-sb/ovn-sb.sock \
> +        -- set Open_vSwitch . external-ids:ovn-encap-type=geneve \
> +        -- set Open_vSwitch . external-ids:ovn-encap-ip=169.0.0.1 \
> +        -- set bridge br-int fail-mode=secure
other-config:disable-in-band=true
> +
> +# Start ovn-controller
> +start_daemon ovn-controller
> +
> +# Logical network:
> +# VM1 -- LS1 -- GW-Router -- LS2 -- VM3
> +#         |
> +# VM2 ----+
> +#
> +# A templated load balancer applied on LS1 and GW-Router with
> +# VM1 as backend.  The VIP should be accessible from both VM2 and VM3.
> +
> +check ovn-nbctl                                                   \
> +    -- lr-add rtr                                                 \
> +    -- set Logical_Router rtr options:chassis=hv1                 \
> +    -- lrp-add rtr rtr-ls1 00:00:00:00:01:00 4242::1/64           \
> +    -- lrp-add rtr rtr-ls2 00:00:00:00:02:00 4343::1/64           \
> +    -- ls-add ls1                                                 \
> +    -- lsp-add ls1 ls1-rtr                                        \
> +    -- lsp-set-addresses ls1-rtr 00:00:00:00:01:00                \
> +    -- lsp-set-type ls1-rtr router                                \
> +    -- lsp-set-options ls1-rtr router-port=rtr-ls1                \
> +    -- lsp-add ls1 vm1 -- lsp-set-addresses vm1 00:00:00:00:00:01 \
> +    -- lsp-add ls1 vm2 -- lsp-set-addresses vm2 00:00:00:00:00:02 \
> +    -- ls-add ls2                                                 \
> +    -- lsp-add ls2 ls2-rtr                                        \
> +    -- lsp-set-addresses ls2-rtr 00:00:00:00:02:00                \
> +    -- lsp-set-type ls2-rtr router                                \
> +    -- lsp-set-options ls2-rtr router-port=rtr-ls2                \
> +    -- lsp-add ls2 vm3 -- lsp-set-addresses vm3 00:00:00:00:00:03
> +
> +# Add a template LB that eventually expands to:
> +# VIP=6666::1 backends=[4242::2]:4242 proto=tcp
> +
> +AT_CHECK([ovn-nbctl -- create chassis_template_var chassis="hv1"
variables="{vip=\"6666::1\",vport=666,backends=\"[[4242::2]]:4242\"}"],
> +         [0], [ignore])
> +
> +check ovn-nbctl --template lb-add lb-test "^vip:^vport" "^backends" tcp
ipv6 \
> +    -- ls-lb-add ls1 lb-test
    \
> +    -- lr-lb-add rtr lb-test
> +
> +ADD_NAMESPACES(vm1)
> +ADD_VETH(vm1, vm1, br-int, "4242::2/64", "00:00:00:00:00:01", "4242::1")
> +OVS_WAIT_UNTIL([test "$(ip netns exec vm1 ip a | grep 4242::2 | grep
tentative)" = ""])
> +
> +ADD_NAMESPACES(vm2)
> +ADD_VETH(vm2, vm2, br-int, "4242::3/64", "00:00:00:00:00:02", "4242::1")
> +OVS_WAIT_UNTIL([test "$(ip netns exec vm2 ip a | grep 4242::3 | grep
tentative)" = ""])
> +
> +ADD_NAMESPACES(vm3)
> +ADD_VETH(vm3, vm3, br-int, "4343::2/64", "00:00:00:00:00:03", "4343::1")
> +OVS_WAIT_UNTIL([test "$(ip netns exec vm3 ip a | grep 4343::2 | grep
tentative)" = ""])
> +
> +# Wait for ovn-controller to catch up.
> +wait_for_ports_up
> +check ovn-nbctl --wait=hv sync
> +
> +AT_CHECK([ovn-appctl -t ovn-controller debug/dump-local-template-vars |
sort], [0], [dnl
> +Local template vars:
> +name: 'backends' value: '[[4242::2]]:4242'
> +name: 'vip' value: '6666::1'
> +name: 'vport' value: '666'
> +])
> +
> +# Start IPv6 TCP server on vm1.
> +NETNS_DAEMONIZE([vm1], [nc -k -l 4242::2 4242], [nc-vm1.pid])
> +
> +# Make sure connecting to the VIP works.
> +NS_CHECK_EXEC([vm2], [nc 6666::1 666 -z], [0], [ignore], [ignore])
> +NS_CHECK_EXEC([vm3], [nc 6666::1 666 -z], [0], [ignore], [ignore])
> +
> +AT_CLEANUP
> +])
> diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c
> index d2dee6b31c..9e9b83ef1f 100644
> --- a/utilities/ovn-nbctl.c
> +++ b/utilities/ovn-nbctl.c
> @@ -28,6 +28,7 @@
>  #include "openvswitch/json.h"
>  #include "lib/acl-log.h"
>  #include "lib/copp.h"
> +#include "lib/lb.h"
>  #include "lib/ovn-nb-idl.h"
>  #include "lib/ovn-util.h"
>  #include "memory.h"
> @@ -2837,6 +2838,7 @@ nbctl_lb_add(struct ctl_context *ctx)
>      bool empty_backend_rej = shash_find(&ctx->options, "--reject") !=
NULL;
>      bool empty_backend_event = shash_find(&ctx->options, "--event") !=
NULL;
>      bool add_route = shash_find(&ctx->options, "--add-route") != NULL;
> +    bool template = shash_find(&ctx->options, "--template") != NULL;
>
>      if (empty_backend_event && empty_backend_rej) {
>              ctl_error(ctx,
> @@ -2844,10 +2846,11 @@ nbctl_lb_add(struct ctl_context *ctx)
>              return;
>      }
>
> +    int lb_address_family = AF_INET;
>      const char *lb_proto;
>      bool is_update_proto = false;
>
> -    if (ctx->argc == 4) {
> +    if (ctx->argc <= 4) {
>          /* Default protocol. */
>          lb_proto = "tcp";
>      } else {
> @@ -2863,79 +2866,59 @@ nbctl_lb_add(struct ctl_context *ctx)
>          }
>      }
>
> -    struct sockaddr_storage ss_vip;
> -    if (!inet_parse_active(lb_vip, 0, &ss_vip, false, NULL)) {
> -        ctl_error(ctx, "%s: should be an IP address (or an IP address "
> -                  "and a port number with : as a separator).", lb_vip);
> -        return;
> -    }
> -
> -    struct ds lb_vip_normalized_ds = DS_EMPTY_INITIALIZER;
> -    uint16_t lb_vip_port = ss_get_port(&ss_vip);
> -    if (lb_vip_port) {
> -        ss_format_address(&ss_vip, &lb_vip_normalized_ds);
> -        ds_put_format(&lb_vip_normalized_ds, ":%d", lb_vip_port);
> -    } else {
> -        ss_format_address_nobracks(&ss_vip, &lb_vip_normalized_ds);
> -    }
> -    const char *lb_vip_normalized = ds_cstr(&lb_vip_normalized_ds);
> +    if (ctx->argc > 5) {
> +        lb_address_family = !strcmp(ctx->argv[5], "ipv4") ? AF_INET :
AF_INET6;
>
> -    if (!lb_vip_port && is_update_proto) {
> -        ds_destroy(&lb_vip_normalized_ds);
> -        ctl_error(ctx, "Protocol is unnecessary when no port of vip "
> -                  "is given.");
> -        return;
>      }
>
> -    char *token = NULL, *save_ptr = NULL;
> +    struct ds lb_vip_normalized = DS_EMPTY_INITIALIZER;
>      struct ds lb_ips_new = DS_EMPTY_INITIALIZER;
> -    for (token = strtok_r(lb_ips, ",", &save_ptr);
> -            token != NULL; token = strtok_r(NULL, ",", &save_ptr)) {
> -        struct sockaddr_storage ss_dst;
> +    struct ovn_lb_vip lb_vip_parsed;
>
> -        if (lb_vip_port) {
> -            if (!inet_parse_active(token, -1, &ss_dst, false, NULL)) {
> -                ctl_error(ctx, "%s: should be an IP address and a port "
> -                          "number with : as a separator.", token);
> -                goto out;
> -            }
> -        } else {
> -            if (!inet_parse_address(token, &ss_dst)) {
> -                ctl_error(ctx, "%s: should be an IP address.", token);
> -                goto out;
> -            }
> -        }
> +    char *error = ovn_lb_vip_init(&lb_vip_parsed, lb_vip, lb_ips,
template,
> +                                  lb_address_family);
> +    if (error) {
> +        ctl_error(ctx, "%s", error);
> +        ovn_lb_vip_destroy(&lb_vip_parsed);
> +        free(error);
> +        return;
> +    }
>
> -        if (ss_vip.ss_family != ss_dst.ss_family) {
> -            ctl_error(ctx, "%s: IP address family is different from VIP
%s.",
> -                      token, lb_vip_normalized);
> -            goto out;
> -        }
> -        ds_put_format(&lb_ips_new, "%s%s",
> -                lb_ips_new.length ? "," : "", token);
> +    if (is_update_proto && !lb_vip_parsed.port_str) {
> +        ctl_error(ctx, "Protocol is unnecessary when no port of vip is "
> +                       "given.");
> +        ovn_lb_vip_destroy(&lb_vip_parsed);
> +        return;
>      }
>
> +    ovn_lb_vip_format(&lb_vip_parsed, &lb_vip_normalized, template);
> +    ovn_lb_vip_backends_format(&lb_vip_parsed, &lb_ips_new, template);
> +    ovn_lb_vip_destroy(&lb_vip_parsed);
> +
>      const struct nbrec_load_balancer *lb = NULL;
>      if (!add_duplicate) {
> -        char *error = lb_by_name_or_uuid(ctx, lb_name, false, &lb);
> +        error = lb_by_name_or_uuid(ctx, lb_name, false, &lb);
>          if (error) {
>              ctx->error = error;
>              goto out;
>          }
>          if (lb) {
> -            if (smap_get(&lb->vips, lb_vip_normalized)) {
> +            if (smap_get(&lb->vips, ds_cstr(&lb_vip_normalized))) {
>                  if (!may_exist) {
>                      ctl_error(ctx, "%s: a load balancer with this vip
(%s) "
> -                              "already exists", lb_name,
lb_vip_normalized);
> +                              "already exists", lb_name,
> +                              ds_cstr(&lb_vip_normalized));
>                      goto out;
>                  }
>                  /* Update the vips. */
>                  smap_replace(CONST_CAST(struct smap *, &lb->vips),
> -                        lb_vip_normalized, ds_cstr(&lb_ips_new));
> +                             ds_cstr(&lb_vip_normalized),
> +                             ds_cstr(&lb_ips_new));
>              } else {
>                  /* Add the new vips. */
>                  smap_add(CONST_CAST(struct smap *, &lb->vips),
> -                        lb_vip_normalized, ds_cstr(&lb_ips_new));
> +                         ds_cstr(&lb_vip_normalized),
> +                         ds_cstr(&lb_ips_new));
>              }
>
>              /* Update the load balancer. */
> @@ -2954,7 +2937,7 @@ nbctl_lb_add(struct ctl_context *ctx)
>      nbrec_load_balancer_set_name(lb, lb_name);
>      nbrec_load_balancer_set_protocol(lb, lb_proto);
>      smap_add(CONST_CAST(struct smap *, &lb->vips),
> -            lb_vip_normalized, ds_cstr(&lb_ips_new));
> +             ds_cstr(&lb_vip_normalized), ds_cstr(&lb_ips_new));
>      nbrec_load_balancer_set_vips(lb, &lb->vips);
>      struct smap options = SMAP_INITIALIZER(&options);
>      if (empty_backend_rej) {
> @@ -2966,12 +2949,17 @@ nbctl_lb_add(struct ctl_context *ctx)
>      if (add_route) {
>          smap_add(&options, "add_route", "true");
>      }
> +    if (template) {
> +        smap_add(&options, "template", "true");
> +        smap_add(&options, "address-family",
> +                 lb_address_family == AF_INET ? "ipv4" : "ipv6");
> +    }
>      nbrec_load_balancer_set_options(lb, &options);
>      smap_destroy(&options);
>  out:
>      ds_destroy(&lb_ips_new);
>
> -    ds_destroy(&lb_vip_normalized_ds);
> +    ds_destroy(&lb_vip_normalized);
>  }
>
>  static void
> @@ -3025,6 +3013,7 @@ static void
>  nbctl_pre_lb_list(struct ctl_context *ctx)
>  {
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_name);
> +    ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_options);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_protocol);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_vips);
>  }
> @@ -3033,6 +3022,7 @@ static void
>  lb_info_add_smap(const struct nbrec_load_balancer *lb,
>                   struct smap *lbs, int vip_width)
>  {
> +    bool template = smap_get_bool(&lb->options, "template", false);
>      const struct smap_node **nodes = smap_sort(&lb->vips);
>      if (!nodes) {
>          return;
> @@ -3041,13 +3031,24 @@ lb_info_add_smap(const struct nbrec_load_balancer
*lb,
>      struct ds val = DS_EMPTY_INITIALIZER;
>      for (size_t i = 0; i < smap_count(&lb->vips); i++) {
>          const struct smap_node *node = nodes[i];
> +        const char *protocol = lb->protocol;
>
> -        struct sockaddr_storage ss;
> -        if (!inet_parse_active(node->key, 0, &ss, false, NULL)) {
> -            continue;
> +        if (!template) {
> +            struct sockaddr_storage ss;
> +            if (!inet_parse_active(node->key, 0, &ss, false, NULL)) {
> +                continue;
> +            }
> +            protocol = ss_get_port(&ss) ? lb->protocol : "";
> +        } else {
> +            if (!lb->protocol) {
> +                VLOG_WARN("Load Balancer "UUID_FMT" (%s) is a template
and "
> +                          "misses protocol",
UUID_ARGS(&lb->header_.uuid),
> +                          lb->name);
> +                continue;
> +            }
> +            protocol = lb->protocol;
>          }
>
> -        char *protocol = ss_get_port(&ss) ? lb->protocol : "";
>          if (i == 0) {
>              ds_put_format(&val, UUID_FMT "    %-20.16s%-11.7s%-*.*s%s",
>                            UUID_ARGS(&lb->header_.uuid),
> @@ -3239,6 +3240,7 @@ nbctl_pre_lr_lb_list(struct ctl_context *ctx)
>                           &nbrec_logical_router_col_load_balancer_group);
>
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_name);
> +    ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_options);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_protocol);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_vips);
>
> @@ -3402,6 +3404,7 @@ nbctl_pre_ls_lb_list(struct ctl_context *ctx)
>                           &nbrec_logical_switch_col_load_balancer_group);
>
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_name);
> +    ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_options);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_protocol);
>      ovsdb_idl_add_column(ctx->idl, &nbrec_load_balancer_col_vips);
>
> @@ -7472,9 +7475,10 @@ static const struct ctl_command_syntax
nbctl_commands[] = {
>        nbctl_pre_lr_nat_set_ext_ips, nbctl_lr_nat_set_ext_ips,
>        NULL, "--is-exempted", RW},
>      /* load balancer commands. */
> -    { "lb-add", 3, 4, "LB VIP[:PORT] IP[:PORT]... [PROTOCOL]",
> +    { "lb-add", 3, 5, "LB VIP[:PORT] IP[:PORT]... [PROTOCOL]
[ADDRESS_FAMILY]",
>        nbctl_pre_lb_add, nbctl_lb_add, NULL,
> -      "--may-exist,--add-duplicate,--reject,--event,--add-route", RW },
> +
 "--may-exist,--add-duplicate,--reject,--event,--add-route,--template",
> +      RW },
>      { "lb-del", 1, 2, "LB [VIP]", nbctl_pre_lb_del, nbctl_lb_del, NULL,
>          "--if-exists", RW },
>      { "lb-list", 0, 1, "[LB]", nbctl_pre_lb_list, nbctl_lb_list, NULL,
"", RO },
>

On 11/29/22 09:14, Han Zhou wrote:
> On Tue, Nov 22, 2022 at 6:15 AM Dumitru Ceara <dceara@redhat.com> wrote:
>>
>> Allow the CMS to configure template LBs.  The following configurations are
>> supported:
>> - VIPs of the form: ^vip_variable[:^port_variable|:port]
>> - Backends of the form:
>>
> ^backendip_variable1[:^port_variable1|:port],^backendip_variable2[:^port_variable2|:port]
>>   OR
>>   ^backends_variable1,^backends_variable2
> 
> Sorry if I missed, but I didn't see any tests that test the form
> "^backends_variable1,^backends_variable2". I only see tests with a single
> backend variable with a single IP in it. Better to test:
> 1. Multiple backends variables

This should be covered.  I have this in the test:

check ovn-nbctl --template lb-add lb-test1 "^VIP1:^VPORT1" "^BACKENDS1" tcp
check ovn-nbctl --template lb-add lb-test2 "^VIP2:^VPORT2"
"^BACKENDS21,^BACKENDS22" tcp
check ovn-nbctl --template lb-add lb-test3 "^VIP3:^VPORT3"
"^BACKENDS31:^BPORT1,^BACKENDS32:^BPORT2" tcp

> 2. Multiple IPs in a single variable (I saw this in the tutorial in patch
> 5, but better to be covered here, too)
> 

You're right, I added a LB with such a variable instantiation to the
test.

> 
> 
>>
>> The CMS needs to provide a bit more information than with non-template
> load
>> balancers and must explicitly specify the address family to be used.
>>
>> There is currently no support for template load balancers with
>> options:add_route=true set.  That is because ovn-northd does not
>> instantiate template variables.  While this is a limitation in a way, its
>> impact is not huge.  The load balancer 'add_route' option was added as a
>> way to make the CMS life easier and to avoid having to explicitly add a
>> route for the VIP.  The CMS can still achieve the same logical topology by
>> explicitly adding the VIP route.
>>
>> Template load balancers don't support the "reachable" neighbor-responder
>> mode.  Instead the CMS can explicitly configure the responder mode to
>> either "all" or "none".
>>
>> To properly handle template updates in ovn-controller we also add a
>> Chassis_Template_Var <- LB reference in ovn-controller.  This way, when
>> a Chassis_Template_Var changes value all load balancers that refer to
>> it will also get updated.
>>
>> Signed-off-by: Dumitru Ceara <dceara@redhat.com>
>> ---
>> V3:
>> - Addressed Mark's comments:
>>   - Added TODO items about potential future template LB improvements.
>>   - Removed n_backends arg from ovn_lb_backends_init_explicit() and
>>     ovn_lb_backends_init_template().
>>   - Fixed ovn_northd_lb_create() to first get the template option before
>>     using its value.
>>   - Fixed up comments and man pages.
>>   - Hardenned setting of address family in ovn-nbctl.
>>
>> V2:
>> - Fix GCC build due to missing explicit return.
>> - Fix ls_in_pre_stateful flows due to using wrong lb field.
>> - Use new lexer_parse_template_string().
>> - Changed lb_handle_changed_ref() signature to return bool.
>> - Update documentation with info about responder mode=none, LB template
>>   supported formats, lb explicit address family requirements.
>> - Squashed the template LB patches into a single one
>> - Added more tests.
>> - Squashed the system tests patch into this one.
>> ---
>>  TODO.rst                    |    7 +
>>  controller/lflow.c          |  115 +++++++++--
>>  controller/lflow.h          |    7 +
>>  controller/ovn-controller.c |   67 +++++-
>>  lib/lb.c                    |  452
> ++++++++++++++++++++++++++++++++++++++-----
>>  lib/lb.h                    |   40 +++-
>>  lib/ovn-util.c              |    3
>>  northd/northd.c             |   89 ++++----
>>  ovn-nb.xml                  |   62 ++++++
>>  tests/ovn-nbctl.at          |   23 +-
>>  tests/ovn-northd.at         |    7 +
>>  tests/ovn.at                |  131 ++++++++++++
>>  tests/system-ovn.at         |  183 +++++++++++++++++
>>  utilities/ovn-nbctl.c       |  120 ++++++-----
>>  14 files changed, 1078 insertions(+), 228 deletions(-)
>>
>> diff --git a/TODO.rst b/TODO.rst
>> index fe5f9a2f30..53cf2870b2 100644
>> --- a/TODO.rst
>> +++ b/TODO.rst
>> @@ -183,3 +183,10 @@ OVN To-do List
>>  * Chassis_Template_Var
>>
>>    * Support template variables when tracing packets with ovn-trace.
>> +
>> +* Load Balancer templates
>> +
>> +  * Support combining the VIP (or backend) IP and port into a single
>> +    template variable.
> 
> Is it still a TODO for backends? At least the tutorial test (in patch 5) is
> already doing something like this:
> backends0="42.0.0.1:1,42.1.0.1:1,42.2.0.1:1,42.3.0.1:1,42.4.0.1:1"
> 

You're right, we can combine the backends and it works fine.  I removed
the "(or backend)" part.

>> +
>> +  * Support combining all backends into a single template variable.
> 
> What does it mean here? Isn't the tutorial test already combining multiple
> backends into a single variable?
> 

Yes, it is, and it's working fine.  I'm not sure anymore why I had
added this TODO item here.  I removed it now.

>> diff --git a/controller/lflow.c b/controller/lflow.c
>> index 84625fb3f1..f6ac639541 100644
>> --- a/controller/lflow.c
>> +++ b/controller/lflow.c
>> @@ -97,6 +97,15 @@ consider_logical_flow(const struct sbrec_logical_flow
> *lflow,
>>                        struct lflow_ctx_in *l_ctx_in,
>>                        struct lflow_ctx_out *l_ctx_out);
>>
>> +static void
>> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
>> +                          const struct sbrec_load_balancer *sbrec_lb,
>> +                          const struct hmap *local_datapaths,
>> +                          const struct smap *template_vars,
>> +                          bool use_ct_mark,
>> +                          struct ovn_desired_flow_table *flow_table,
>> +                          struct simap *ids);
>> +
>>  static void add_port_sec_flows(const struct shash *binding_lports,
>>                                 const struct sbrec_chassis *,
>>                                 struct ovn_desired_flow_table *);
>> @@ -223,7 +232,7 @@ lflow_handle_changed_flows(struct lflow_ctx_in
> *l_ctx_in,
>>          UUIDSET_INITIALIZER(&flood_remove_nodes);
>>      SBREC_LOGICAL_FLOW_TABLE_FOR_EACH_TRACKED (lflow,
>>
> l_ctx_in->logical_flow_table) {
>> -        if (uuidset_find(l_ctx_out->lflows_processed,
> &lflow->header_.uuid)) {
>> +        if (uuidset_find(l_ctx_out->objs_processed,
> &lflow->header_.uuid)) {
>>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
>>                       UUID_ARGS(&lflow->header_.uuid));
>>              continue;
>> @@ -253,14 +262,14 @@ lflow_handle_changed_flows(struct lflow_ctx_in
> *l_ctx_in,
>>                       UUID_ARGS(&lflow->header_.uuid));
>>
>>              /* For the extra lflows that need to be reprocessed because
> of the
>> -             * flood remove, remove it from lflows_processed. */
>> +             * flood remove, remove it from objs_processed. */
>>              struct uuidset_node *unode =
>> -                uuidset_find(l_ctx_out->lflows_processed,
>> +                uuidset_find(l_ctx_out->objs_processed,
>>                               &lflow->header_.uuid);
>>              if (unode) {
>>                  VLOG_DBG("lflow "UUID_FMT"has been processed, now
> reprocess.",
>>                           UUID_ARGS(&lflow->header_.uuid));
>> -                uuidset_delete(l_ctx_out->lflows_processed, unode);
>> +                uuidset_delete(l_ctx_out->objs_processed, unode);
>>              }
>>
>>              consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
>> @@ -687,7 +696,7 @@ lflow_handle_addr_set_update(const char *as_name,
>>      struct object_to_resources_list_node *resource_list_node;
>>      RESOURCE_FOR_EACH_OBJ (resource_list_node, resource_node) {
>>          const struct uuid *obj_uuid = &resource_list_node->obj_uuid;
>> -        if (uuidset_find(l_ctx_out->lflows_processed, obj_uuid)) {
>> +        if (uuidset_find(l_ctx_out->objs_processed, obj_uuid)) {
>>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
>>                       UUID_ARGS(obj_uuid));
>>              continue;
>> @@ -777,13 +786,13 @@ lflow_handle_changed_ref(enum objdep_type type,
> const char *res_name,
>>          }
>>
>>          /* For the extra lflows that need to be reprocessed because of
> the
>> -         * flood remove, remove it from lflows_processed. */
>> +         * flood remove, remove it from objs_processed. */
>>          struct uuidset_node *unode =
>> -            uuidset_find(l_ctx_out->lflows_processed,
> &lflow->header_.uuid);
>> +            uuidset_find(l_ctx_out->objs_processed,
> &lflow->header_.uuid);
>>          if (unode) {
>>              VLOG_DBG("lflow "UUID_FMT"has been processed, now
> reprocess.",
>>                       UUID_ARGS(&lflow->header_.uuid));
>> -            uuidset_delete(l_ctx_out->lflows_processed, unode);
>> +            uuidset_delete(l_ctx_out->objs_processed, unode);
>>          }
>>
>>          consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
>> @@ -792,6 +801,43 @@ lflow_handle_changed_ref(enum objdep_type type,
> const char *res_name,
>>      return true;
>>  }
>>
>> +bool
>> +lb_handle_changed_ref(enum objdep_type type, const char *res_name,
>> +                      struct ovs_list *objs_todo,
>> +                      const void *in_arg, void *out_arg)
>> +{
>> +    struct lflow_ctx_in *l_ctx_in = CONST_CAST(struct lflow_ctx_in *,
> in_arg);
>> +    struct lflow_ctx_out *l_ctx_out = out_arg;
>> +
>> +    struct object_to_resources_list_node *resource_lb_uuid;
>> +    LIST_FOR_EACH_POP (resource_lb_uuid, list_node, objs_todo) {
>> +        VLOG_DBG("Reprocess LB "UUID_FMT" for resource type: %s, name:
> %s",
>> +                 UUID_ARGS(&resource_lb_uuid->obj_uuid),
>> +                 objdep_type_name(type), res_name);
>> +
>> +        const struct sbrec_load_balancer *lb =
>> +            sbrec_load_balancer_table_get_for_uuid(
>> +                l_ctx_in->lb_table, &resource_lb_uuid->obj_uuid);
>> +        if (!lb) {
>> +            VLOG_DBG("Failed to find LB "UUID_FMT" referred by: %s",
> 
> nit: I think it should be: Failed to find LB ... that refers ...
> 

Yes, fixed.

> 
>> +                     UUID_ARGS(&resource_lb_uuid->obj_uuid), res_name);
>> +        } else {
>> +            ofctrl_remove_flows(l_ctx_out->flow_table,
>> +                                &resource_lb_uuid->obj_uuid);
>> +
>> +            consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
>> +                                      l_ctx_in->local_datapaths,
>> +                                      l_ctx_in->template_vars,
>> +                                      l_ctx_in->lb_hairpin_use_ct_mark,
>> +                                      l_ctx_out->flow_table,
>> +                                      l_ctx_out->hairpin_lb_ids);
>> +        }
>> +
>> +        free(resource_lb_uuid);
>> +    }
>> +    return true;
>> +}
>> +
>>  static void
>>  lflow_parse_ctrl_meter(const struct sbrec_logical_flow *lflow,
>>                         struct ovn_extend_table *meter_table,
>> @@ -1259,9 +1305,9 @@ consider_logical_flow(const struct
> sbrec_logical_flow *lflow,
>>
>>      COVERAGE_INC(consider_logical_flow);
>>      if (!is_recompute) {
>> -        ovs_assert(!uuidset_find(l_ctx_out->lflows_processed,
>> +        ovs_assert(!uuidset_find(l_ctx_out->objs_processed,
>>                                   &lflow->header_.uuid));
>> -        uuidset_insert(l_ctx_out->lflows_processed,
> &lflow->header_.uuid);
>> +        uuidset_insert(l_ctx_out->objs_processed, &lflow->header_.uuid);
>>      }
>>
>>      if (dp) {
>> @@ -2001,8 +2047,10 @@ add_lb_ct_snat_hairpin_flows(struct
> ovn_controller_lb *lb,
>>  }
>>
>>  static void
>> -consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
>> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
>> +                          const struct sbrec_load_balancer *sbrec_lb,
>>                            const struct hmap *local_datapaths,
>> +                          const struct smap *template_vars,
>>                            bool use_ct_mark,
>>                            struct ovn_desired_flow_table *flow_table,
>>                            struct simap *ids)
>> @@ -2039,7 +2087,9 @@ consider_lb_hairpin_flows(const struct
> sbrec_load_balancer *sbrec_lb,
>>          return;
>>      }
>>
>> -    struct ovn_controller_lb *lb = ovn_controller_lb_create(sbrec_lb);
>> +    struct sset template_vars_ref = SSET_INITIALIZER(&template_vars_ref);
>> +    struct ovn_controller_lb *lb =
>> +        ovn_controller_lb_create(sbrec_lb, template_vars,
> &template_vars_ref);
>>      uint8_t lb_proto = IPPROTO_TCP;
>>      if (lb->slb->protocol && lb->slb->protocol[0]) {
>>          if (!strcmp(lb->slb->protocol, "udp")) {
>> @@ -2049,6 +2099,11 @@ consider_lb_hairpin_flows(const struct
> sbrec_load_balancer *sbrec_lb,
>>          }
>>      }
>>
>> +    const char *tv_name;
>> +    SSET_FOR_EACH (tv_name, &template_vars_ref) {
>> +        objdep_mgr_add(mgr, OBJDEP_TYPE_TEMPLATE, tv_name,
>> +                       &sbrec_lb->header_.uuid);
>> +    }
>>      for (i = 0; i < lb->n_vips; i++) {
>>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>>
>> @@ -2063,13 +2118,17 @@ consider_lb_hairpin_flows(const struct
> sbrec_load_balancer *sbrec_lb,
>>      add_lb_ct_snat_hairpin_flows(lb, id, lb_proto, flow_table);
>>
>>      ovn_controller_lb_destroy(lb);
>> +    sset_destroy(&template_vars_ref);
>>  }
>>
>>  /* Adds OpenFlow flows to flow tables for each Load balancer VIPs and
>>   * backends to handle the load balanced hairpin traffic. */
>>  static void
>> -add_lb_hairpin_flows(const struct sbrec_load_balancer_table *lb_table,
>> -                     const struct hmap *local_datapaths, bool
> use_ct_mark,
>> +add_lb_hairpin_flows(struct objdep_mgr *mgr,
>> +                     const struct sbrec_load_balancer_table *lb_table,
>> +                     const struct hmap *local_datapaths,
>> +                     const struct smap *template_vars,
>> +                     bool use_ct_mark,
>>                       struct ovn_desired_flow_table *flow_table,
>>                       struct simap *ids,
>>                       struct id_pool *pool)
>> @@ -2092,8 +2151,8 @@ add_lb_hairpin_flows(const struct
> sbrec_load_balancer_table *lb_table,
>>              ovs_assert(id_pool_alloc_id(pool, &id));
>>              simap_put(ids, lb->name, id);
>>          }
>> -        consider_lb_hairpin_flows(lb, local_datapaths, use_ct_mark,
>> -                                  flow_table, ids);
>> +        consider_lb_hairpin_flows(mgr, lb, local_datapaths,
> template_vars,
>> +                                  use_ct_mark, flow_table, ids);
>>      }
>>  }
>>
>> @@ -2229,7 +2288,9 @@ lflow_run(struct lflow_ctx_in *l_ctx_in, struct
> lflow_ctx_out *l_ctx_out)
>>                         l_ctx_in->static_mac_binding_table,
>>                         l_ctx_in->local_datapaths,
>>                         l_ctx_out->flow_table);
>> -    add_lb_hairpin_flows(l_ctx_in->lb_table, l_ctx_in->local_datapaths,
>> +    add_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, l_ctx_in->lb_table,
>> +                         l_ctx_in->local_datapaths,
>> +                         l_ctx_in->template_vars,
>>                           l_ctx_in->lb_hairpin_use_ct_mark,
>>                           l_ctx_out->flow_table,
>>                           l_ctx_out->hairpin_lb_ids,
>> @@ -2280,10 +2341,10 @@ lflow_add_flows_for_datapath(const struct
> sbrec_datapath_binding *dp,
>>      const struct sbrec_logical_flow *lflow;
>>      SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
>>          lflow, lf_row, l_ctx_in->sbrec_logical_flow_by_logical_datapath)
> {
>> -        if (uuidset_find(l_ctx_out->lflows_processed,
> &lflow->header_.uuid)) {
>> +        if (uuidset_find(l_ctx_out->objs_processed,
> &lflow->header_.uuid)) {
>>              continue;
>>          }
>> -        uuidset_insert(l_ctx_out->lflows_processed,
> &lflow->header_.uuid);
>> +        uuidset_insert(l_ctx_out->objs_processed, &lflow->header_.uuid);
>>          consider_logical_flow__(lflow, dp, l_ctx_in, l_ctx_out);
>>      }
>>      sbrec_logical_flow_index_destroy_row(lf_row);
>> @@ -2308,7 +2369,7 @@ lflow_add_flows_for_datapath(const struct
> sbrec_datapath_binding *dp,
>>          sbrec_logical_flow_index_set_logical_dp_group(lf_row, ldpg);
>>          SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
>>              lflow, lf_row,
> l_ctx_in->sbrec_logical_flow_by_logical_dp_group) {
>> -            if (uuidset_find(l_ctx_out->lflows_processed,
>> +            if (uuidset_find(l_ctx_out->objs_processed,
>>                               &lflow->header_.uuid)) {
>>                  continue;
>>              }
>> @@ -2360,7 +2421,9 @@ lflow_add_flows_for_datapath(const struct
> sbrec_datapath_binding *dp,
>>      /* Add load balancer hairpin flows if the datapath has any load
> balancers
>>       * associated. */
>>      for (size_t i = 0; i < n_dp_lbs; i++) {
>> -        consider_lb_hairpin_flows(dp_lbs[i], l_ctx_in->local_datapaths,
>> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, dp_lbs[i],
>> +                                  l_ctx_in->local_datapaths,
>> +                                  l_ctx_in->template_vars,
>>                                    l_ctx_in->lb_hairpin_use_ct_mark,
>>                                    l_ctx_out->flow_table,
>>                                    l_ctx_out->hairpin_lb_ids);
>> @@ -2382,7 +2445,7 @@ lflow_handle_flows_for_lport(const struct
> sbrec_port_binding *pb,
>>                                    OBJDEP_TYPE_PORTBINDING,
>>                                    pb->logical_port,
>>                                    lflow_handle_changed_ref,
>> -                                  l_ctx_out->lflows_processed,
>> +                                  l_ctx_out->objs_processed,
>>                                    l_ctx_in, l_ctx_out, &changed)) {
>>          return false;
>>      }
>> @@ -2421,7 +2484,7 @@ lflow_handle_changed_port_bindings(struct
> lflow_ctx_in *l_ctx_in,
>>                                        OBJDEP_TYPE_PORTBINDING,
>>                                        pb->logical_port,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out->lflows_processed,
>> +                                      l_ctx_out->objs_processed,
>>                                        l_ctx_in, l_ctx_out, &changed)) {
>>              ret = false;
>>              break;
>> @@ -2448,7 +2511,7 @@ lflow_handle_changed_mc_groups(struct lflow_ctx_in
> *l_ctx_in,
>>          if (!objdep_mgr_handle_change(l_ctx_out->lflow_deps_mgr,
>>                                        OBJDEP_TYPE_MC_GROUP,
> ds_cstr(&mg_key),
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out->lflows_processed,
>> +                                      l_ctx_out->objs_processed,
>>                                        l_ctx_in, l_ctx_out, &changed)) {
>>              ret = false;
>>              break;
>> @@ -2502,7 +2565,9 @@ lflow_handle_changed_lbs(struct lflow_ctx_in
> *l_ctx_in,
>>
>>          VLOG_DBG("Add load balancer hairpin flows for "UUID_FMT,
>>                   UUID_ARGS(&lb->header_.uuid));
>> -        consider_lb_hairpin_flows(lb, l_ctx_in->local_datapaths,
>> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
>> +                                  l_ctx_in->local_datapaths,
>> +                                  l_ctx_in->template_vars,
>>                                    l_ctx_in->lb_hairpin_use_ct_mark,
>>                                    l_ctx_out->flow_table,
>>                                    l_ctx_out->hairpin_lb_ids);
>> diff --git a/controller/lflow.h b/controller/lflow.h
>> index d95fd41142..9e8f9afd33 100644
>> --- a/controller/lflow.h
>> +++ b/controller/lflow.h
>> @@ -122,9 +122,10 @@ struct lflow_ctx_out {
>>      struct ovn_extend_table *group_table;
>>      struct ovn_extend_table *meter_table;
>>      struct objdep_mgr *lflow_deps_mgr;
>> +    struct objdep_mgr *lb_deps_mgr;
>>      struct lflow_cache *lflow_cache;
>>      struct conj_ids *conj_ids;
>> -    struct uuidset *lflows_processed;
>> +    struct uuidset *objs_processed;
>>      struct simap *hairpin_lb_ids;
>>      struct id_pool *hairpin_id_pool;
>>  };
>> @@ -174,4 +175,8 @@ bool lflow_handle_changed_mc_groups(struct
> lflow_ctx_in *,
>>                                      struct lflow_ctx_out *);
>>  bool lflow_handle_changed_port_bindings(struct lflow_ctx_in *,
>>                                          struct lflow_ctx_out *);
>> +
>> +bool lb_handle_changed_ref(enum objdep_type type, const char *res_name,
>> +                           struct ovs_list *objs_todo,
>> +                           const void *in_arg, void *out_arg);
>>  #endif /* controller/lflow.h */
>> diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c
>> index f9ed0e3855..9807ecd8eb 100644
>> --- a/controller/ovn-controller.c
>> +++ b/controller/ovn-controller.c
>> @@ -2791,13 +2791,15 @@ struct ed_type_lflow_output {
>>      struct ovn_extend_table meter_table;
>>      /* lflow <-> resource cross reference */
>>      struct objdep_mgr lflow_deps_mgr;;
>> +    /* load balancer <-> resource cross reference */
>> +    struct objdep_mgr lb_deps_mgr;
>>      /* conjunciton ID usage information of lflows */
>>      struct conj_ids conj_ids;
>>
>> -    /* lflows processed in the current engine execution.
>> +    /* objects (lflows and lbs) processed in the current engine
> execution.
>>       * Cleared by en_lflow_output_clear_tracked_data before each engine
>>       * execution. */
>> -    struct uuidset lflows_processed;
>> +    struct uuidset objs_processed;
>>
>>      /* Data which is persistent and not cleared during
>>       * full recompute. */
>> @@ -2954,8 +2956,9 @@ init_lflow_ctx(struct engine_node *node,
>>      l_ctx_out->group_table = &fo->group_table;
>>      l_ctx_out->meter_table = &fo->meter_table;
>>      l_ctx_out->lflow_deps_mgr = &fo->lflow_deps_mgr;
>> +    l_ctx_out->lb_deps_mgr = &fo->lb_deps_mgr;
>>      l_ctx_out->conj_ids = &fo->conj_ids;
>> -    l_ctx_out->lflows_processed = &fo->lflows_processed;
>> +    l_ctx_out->objs_processed = &fo->objs_processed;
>>      l_ctx_out->lflow_cache = fo->pd.lflow_cache;
>>      l_ctx_out->hairpin_id_pool = fo->hd.pool;
>>      l_ctx_out->hairpin_lb_ids = &fo->hd.ids;
>> @@ -2970,8 +2973,9 @@ en_lflow_output_init(struct engine_node *node
> OVS_UNUSED,
>>      ovn_extend_table_init(&data->group_table);
>>      ovn_extend_table_init(&data->meter_table);
>>      objdep_mgr_init(&data->lflow_deps_mgr);
>> +    objdep_mgr_init(&data->lb_deps_mgr);
>>      lflow_conj_ids_init(&data->conj_ids);
>> -    uuidset_init(&data->lflows_processed);
>> +    uuidset_init(&data->objs_processed);
>>      simap_init(&data->hd.ids);
>>      data->hd.pool = id_pool_create(1, UINT32_MAX - 1);
>>      nd_ra_opts_init(&data->nd_ra_opts);
>> @@ -2983,7 +2987,7 @@ static void
>>  en_lflow_output_clear_tracked_data(void *data)
>>  {
>>      struct ed_type_lflow_output *flow_output_data = data;
>> -    uuidset_clear(&flow_output_data->lflows_processed);
>> +    uuidset_clear(&flow_output_data->objs_processed);
>>  }
>>
>>  static void
>> @@ -2994,8 +2998,9 @@ en_lflow_output_cleanup(void *data)
>>      ovn_extend_table_destroy(&flow_output_data->group_table);
>>      ovn_extend_table_destroy(&flow_output_data->meter_table);
>>      objdep_mgr_destroy(&flow_output_data->lflow_deps_mgr);
>> +    objdep_mgr_destroy(&flow_output_data->lb_deps_mgr);
>>      lflow_conj_ids_destroy(&flow_output_data->conj_ids);
>> -    uuidset_destroy(&flow_output_data->lflows_processed);
>> +    uuidset_destroy(&flow_output_data->objs_processed);
>>      lflow_cache_destroy(flow_output_data->pd.lflow_cache);
>>      simap_destroy(&flow_output_data->hd.ids);
>>      id_pool_destroy(flow_output_data->hd.pool);
>> @@ -3030,6 +3035,7 @@ en_lflow_output_run(struct engine_node *node, void
> *data)
>>      struct ovn_extend_table *group_table = &fo->group_table;
>>      struct ovn_extend_table *meter_table = &fo->meter_table;
>>      struct objdep_mgr *lflow_deps_mgr = &fo->lflow_deps_mgr;
>> +    struct objdep_mgr *lb_deps_mgr = &fo->lb_deps_mgr;
>>
>>      static bool first_run = true;
>>      if (first_run) {
>> @@ -3039,6 +3045,7 @@ en_lflow_output_run(struct engine_node *node, void
> *data)
>>          ovn_extend_table_clear(group_table, false /* desired */);
>>          ovn_extend_table_clear(meter_table, false /* desired */);
>>          objdep_mgr_clear(lflow_deps_mgr);
>> +        objdep_mgr_clear(lb_deps_mgr);
>>          lflow_conj_ids_clear(&fo->conj_ids);
>>      }
>>
>> @@ -3172,7 +3179,7 @@ lflow_output_addr_sets_handler(struct engine_node
> *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_ADDRSET, ref_name,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3191,7 +3198,7 @@ lflow_output_addr_sets_handler(struct engine_node
> *node, void *data)
>>                                            OBJDEP_TYPE_ADDRSET,
>>                                            shash_node->name,
>>                                            lflow_handle_changed_ref,
>> -                                          l_ctx_out.lflows_processed,
>> +                                          l_ctx_out.objs_processed,
>>                                            &l_ctx_in, &l_ctx_out,
> &changed)) {
>>                  return false;
>>              }
>> @@ -3204,7 +3211,7 @@ lflow_output_addr_sets_handler(struct engine_node
> *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_ADDRSET, ref_name,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3239,7 +3246,7 @@ lflow_output_port_groups_handler(struct engine_node
> *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3251,7 +3258,7 @@ lflow_output_port_groups_handler(struct engine_node
> *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3263,7 +3270,7 @@ lflow_output_port_groups_handler(struct engine_node
> *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
>>                                        lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3297,7 +3304,17 @@ lflow_output_template_vars_handler(struct
> engine_node *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_TEMPLATE,
>>                                        res_name, lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
>> +            return false;
>> +        }
>> +        if (changed) {
>> +            engine_set_node_state(node, EN_UPDATED);
>> +        }
>> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
>> +                                      OBJDEP_TYPE_TEMPLATE,
>> +                                      res_name, lb_handle_changed_ref,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3309,7 +3326,17 @@ lflow_output_template_vars_handler(struct
> engine_node *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_TEMPLATE,
>>                                        res_name, lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
>> +            return false;
>> +        }
>> +        if (changed) {
>> +            engine_set_node_state(node, EN_UPDATED);
>> +        }
>> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
>> +                                      OBJDEP_TYPE_TEMPLATE,
>> +                                      res_name, lb_handle_changed_ref,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> @@ -3321,7 +3348,17 @@ lflow_output_template_vars_handler(struct
> engine_node *node, void *data)
>>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
>>                                        OBJDEP_TYPE_TEMPLATE,
>>                                        res_name, lflow_handle_changed_ref,
>> -                                      l_ctx_out.lflows_processed,
>> +                                      l_ctx_out.objs_processed,
>> +                                      &l_ctx_in, &l_ctx_out, &changed)) {
>> +            return false;
>> +        }
>> +        if (changed) {
>> +            engine_set_node_state(node, EN_UPDATED);
>> +        }
>> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
>> +                                      OBJDEP_TYPE_TEMPLATE,
>> +                                      res_name, lb_handle_changed_ref,
>> +                                      l_ctx_out.objs_processed,
>>                                        &l_ctx_in, &l_ctx_out, &changed)) {
>>              return false;
>>          }
>> diff --git a/lib/lb.c b/lib/lb.c
>> index c08ccceda1..43628bba77 100644
>> --- a/lib/lb.c
>> +++ b/lib/lb.c
>> @@ -19,6 +19,7 @@
>>  #include "lib/ovn-nb-idl.h"
>>  #include "lib/ovn-sb-idl.h"
>>  #include "lib/ovn-util.h"
>> +#include "ovn/lex.h"
>>
>>  /* OpenvSwitch lib includes. */
>>  #include "openvswitch/vlog.h"
>> @@ -26,6 +27,16 @@
>>
>>  VLOG_DEFINE_THIS_MODULE(lb);
>>
>> +static const char *lb_neighbor_responder_mode_names[] = {
>> +    [LB_NEIGH_RESPOND_REACHABLE] = "reachable",
>> +    [LB_NEIGH_RESPOND_ALL] = "all",
>> +    [LB_NEIGH_RESPOND_NONE] = "none",
>> +};
>> +
>> +static struct nbrec_load_balancer_health_check *
>> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
>> +                        const char *vip_port_str, bool template);
>> +
>>  struct ovn_lb_ip_set *
>>  ovn_lb_ip_set_create(void)
>>  {
>> @@ -71,94 +82,293 @@ ovn_lb_ip_set_clone(struct ovn_lb_ip_set *lb_ip_set)
>>      return clone;
>>  }
>>
>> -static
>> -bool ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
>> -                     const char *lb_value)
>> +/* Format for backend ips: "IP1:port1,IP2:port2,...". */
>> +static char *
>> +ovn_lb_backends_init_explicit(struct ovn_lb_vip *lb_vip, const char
> *value)
>>  {
>> -    int addr_family;
>> -
>> -    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
>> -                                         &lb_vip->vip, &lb_vip->vip_port,
>> -                                         &addr_family)) {
>> -        return false;
>> -    }
>> -
>> -    /* Format for backend ips: "IP1:port1,IP2:port2,...". */
>> -    size_t n_backends = 0;
>> +    struct ds errors = DS_EMPTY_INITIALIZER;
>>      size_t n_allocated_backends = 0;
>> -    char *tokstr = xstrdup(lb_value);
>> +    char *tokstr = xstrdup(value);
>>      char *save_ptr = NULL;
>> +    lb_vip->n_backends = 0;
>> +
>>      for (char *token = strtok_r(tokstr, ",", &save_ptr);
>>          token != NULL;
>>          token = strtok_r(NULL, ",", &save_ptr)) {
>>
>> -        if (n_backends == n_allocated_backends) {
>> +        if (lb_vip->n_backends == n_allocated_backends) {
>>              lb_vip->backends = x2nrealloc(lb_vip->backends,
>>                                            &n_allocated_backends,
>>                                            sizeof *lb_vip->backends);
>>          }
>>
>> -        struct ovn_lb_backend *backend = &lb_vip->backends[n_backends];
>> +        struct ovn_lb_backend *backend =
> &lb_vip->backends[lb_vip->n_backends];
>>          int backend_addr_family;
>>          if (!ip_address_and_port_from_lb_key(token, &backend->ip_str,
>>                                               &backend->ip,
> &backend->port,
>>                                               &backend_addr_family)) {
>> +            if (lb_vip->port_str) {
>> +                ds_put_format(&errors, "%s: should be an IP address and
> a "
>> +                                       "port number with : as a
> separator, ",
>> +                              token);
>> +            } else {
>> +                ds_put_format(&errors, "%s: should be an IP address, ",
> token);
>> +            }
>>              continue;
>>          }
>>
>> -        if (addr_family != backend_addr_family) {
>> +        if (lb_vip->address_family != backend_addr_family) {
>>              free(backend->ip_str);
>> +            ds_put_format(&errors, "%s: IP address family is different
> from "
>> +                                   "VIP %s, ",
>> +                          token, lb_vip->vip_str);
>>              continue;
>>          }
>>
>> -        n_backends++;
>> +        if (lb_vip->port_str) {
>> +            if (!backend->port) {
>> +                free(backend->ip_str);
>> +                ds_put_format(&errors, "%s: should be an IP address and "
>> +                                       "a port number with : as a
> separator, ",
>> +                              token);
>> +                continue;
>> +            }
>> +        } else {
>> +            if (backend->port) {
>> +                free(backend->ip_str);
>> +                ds_put_format(&errors, "%s: should be an IP address, ",
> token);
>> +                continue;
>> +            }
>> +        }
>> +
>> +        backend->port_str =
>> +            backend->port ? xasprintf("%"PRIu16, backend->port) : NULL;
>> +        lb_vip->n_backends++;
>>      }
>>      free(tokstr);
>> -    lb_vip->n_backends = n_backends;
>> -    return true;
>> +
>> +    if (ds_last(&errors) != EOF) {
>> +        ds_chomp(&errors, ' ');
>> +        ds_chomp(&errors, ',');
>> +        ds_put_char(&errors, '.');
>> +        return ds_steal_cstr(&errors);
>> +    }
>> +    return NULL;
>>  }
>>
>>  static
>> -void ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
>> +char *ovn_lb_vip_init_explicit(struct ovn_lb_vip *lb_vip, const char
> *lb_key,
>> +                               const char *lb_value)
>> +{
>> +    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
>> +                                         &lb_vip->vip, &lb_vip->vip_port,
>> +                                         &lb_vip->address_family)) {
>> +        return xasprintf("%s: should be an IP address (or an IP address "
>> +                         "and a port number with : as a separator).",
> lb_key);
>> +    }
>> +
>> +    lb_vip->port_str = lb_vip->vip_port
>> +                       ? xasprintf("%"PRIu16, lb_vip->vip_port)
>> +                       : NULL;
>> +
>> +    return ovn_lb_backends_init_explicit(lb_vip, lb_value);
>> +}
>> +
>> +/* Parses backends of a templated LB VIP.
>> + * For now only the following template forms are supported:
>> + * A.
>> + *   ^backendip_variable1[:^port_variable1|:port],
>> + *   ^backendip_variable2[:^port_variable2|:port]
>> + *
>> + * B.
>> + *   ^backends_variable1,^backends_variable2 is also a thing
>> + *      where 'backends_variable1' may expand to IP1_1:PORT1_1 on
> chassis-1
>> + *                                               IP1_2:PORT1_2 on
> chassis-2
>> + *        and 'backends_variable2' may expand to IP2_1:PORT2_1 on
> chassis-1
>> + *                                               IP2_2:PORT2_2 on
> chassis-2
>> + */
>> +static char *
>> +ovn_lb_backends_init_template(struct ovn_lb_vip *lb_vip, const char
> *value_)
>> +{
>> +    struct ds errors = DS_EMPTY_INITIALIZER;
>> +    char *value = xstrdup(value_);
>> +    char *save_ptr = NULL;
>> +    size_t n_allocated_backends = 0;
>> +    lb_vip->n_backends = 0;
>> +
>> +    for (char *backend = strtok_r(value, ",", &save_ptr); backend;
>> +         backend = strtok_r(NULL, ",", &save_ptr)) {
>> +
>> +        char *atom = xstrdup(backend);
>> +        char *save_ptr2 = NULL;
>> +        bool success = false;
>> +        char *backend_ip = NULL;
>> +        char *backend_port = NULL;
>> +
>> +        for (char *subatom = strtok_r(atom, ":", &save_ptr2); subatom;
>> +             subatom = strtok_r(NULL, ":", &save_ptr2)) {
>> +            if (backend_ip && backend_port) {
>> +                success = false;
>> +                break;
>> +            }
>> +            success = true;
>> +            if (!backend_ip) {
>> +                backend_ip = xstrdup(subatom);
>> +            } else {
>> +                backend_port = xstrdup(subatom);
>> +            }
>> +        }
>> +
>> +        if (success) {
>> +            if (lb_vip->n_backends == n_allocated_backends) {
>> +                lb_vip->backends = x2nrealloc(lb_vip->backends,
>> +                                              &n_allocated_backends,
>> +                                              sizeof *lb_vip->backends);
>> +            }
>> +
>> +            struct ovn_lb_backend *lb_backend =
>> +                &lb_vip->backends[lb_vip->n_backends];
>> +            lb_backend->ip_str = backend_ip;
>> +            lb_backend->port_str = backend_port;
>> +            lb_backend->port = 0;
>> +            lb_vip->n_backends++;
>> +        } else {
>> +            ds_put_format(&errors, "%s: should be a template of the
> form: "
>> +
>  "'^backendip_variable1[:^port_variable1|:port]', ",
>> +                          atom);
>> +        }
>> +        free(atom);
>> +    }
>> +
>> +    free(value);
>> +    if (ds_last(&errors) != EOF) {
>> +        ds_chomp(&errors, ' ');
>> +        ds_chomp(&errors, ',');
>> +        ds_put_char(&errors, '.');
>> +        return ds_steal_cstr(&errors);
>> +    }
>> +    return NULL;
>> +}
>> +
>> +/* Parses a VIP of a templated LB.
>> + * For now only the following template forms are supported:
>> + *   ^vip_variable[:^port_variable|:port]
>> + */
>> +static char *
>> +ovn_lb_vip_init_template(struct ovn_lb_vip *lb_vip, const char *lb_key_,
>> +                         const char *lb_value, int address_family)
>> +{
>> +    char *save_ptr = NULL;
>> +    char *lb_key = xstrdup(lb_key_);
>> +    bool success = false;
>> +
>> +    for (char *atom = strtok_r(lb_key, ":", &save_ptr); atom;
>> +         atom = strtok_r(NULL, ":", &save_ptr)) {
>> +        if (lb_vip->vip_str && lb_vip->port_str) {
>> +            success = false;
>> +            break;
>> +        }
>> +        success = true;
>> +        if (!lb_vip->vip_str) {
>> +            lb_vip->vip_str = xstrdup(atom);
>> +        } else {
>> +            lb_vip->port_str = xstrdup(atom);
>> +        }
>> +    }
>> +    free(lb_key);
>> +
>> +    if (!success) {
>> +        return xasprintf("%s: should be a template of the form: "
>> +                         "'^vip_variable[:^port_variable|:port]'.",
>> +                         lb_key_);
>> +    }
>> +
>> +    lb_vip->address_family = address_family;
>> +    return ovn_lb_backends_init_template(lb_vip, lb_value);
>> +}
>> +
>> +/* Returns NULL on success, an error string on failure.  The caller is
>> + * responsible for destroying 'lb_vip' in all cases.
>> + */
>> +char *
>> +ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
>> +                const char *lb_value, bool template, int address_family)
>> +{
>> +    memset(lb_vip, 0, sizeof *lb_vip);
>> +
>> +    return !template
>> +           ?  ovn_lb_vip_init_explicit(lb_vip, lb_key, lb_value)
>> +           :  ovn_lb_vip_init_template(lb_vip, lb_key, lb_value,
>> +                                       address_family);
>> +}
>> +
>> +void
>> +ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
>>  {
>>      free(vip->vip_str);
>> +    free(vip->port_str);
>>      for (size_t i = 0; i < vip->n_backends; i++) {
>>          free(vip->backends[i].ip_str);
>> +        free(vip->backends[i].port_str);
>>      }
>>      free(vip->backends);
>>  }
>>
>> +void
>> +ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s, bool
> template)
>> +{
>> +    bool needs_brackets = vip->address_family == AF_INET6 &&
> vip->port_str
>> +                          && !template;
>> +    if (needs_brackets) {
>> +        ds_put_char(s, '[');
>> +    }
>> +    ds_put_cstr(s, vip->vip_str);
>> +    if (needs_brackets) {
>> +        ds_put_char(s, ']');
>> +    }
>> +    if (vip->port_str) {
>> +        ds_put_format(s, ":%s", vip->port_str);
>> +    }
>> +}
>> +
>> +void
>> +ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct ds *s,
>> +                           bool template)
>> +{
>> +    bool needs_brackets = vip->address_family == AF_INET6 &&
> vip->port_str
>> +                          && !template;
>> +    for (size_t i = 0; i < vip->n_backends; i++) {
>> +        struct ovn_lb_backend *backend = &vip->backends[i];
>> +
>> +        if (needs_brackets) {
>> +            ds_put_char(s, '[');
>> +        }
>> +        ds_put_cstr(s, backend->ip_str);
>> +        if (needs_brackets) {
>> +            ds_put_char(s, ']');
>> +        }
>> +        if (backend->port_str) {
>> +            ds_put_format(s, ":%s", backend->port_str);
>> +        }
>> +        if (i != vip->n_backends - 1) {
>> +            ds_put_char(s, ',');
>> +        }
>> +    }
>> +}
>> +
>>  static
>>  void ovn_northd_lb_vip_init(struct ovn_northd_lb_vip *lb_vip_nb,
>>                              const struct ovn_lb_vip *lb_vip,
>>                              const struct nbrec_load_balancer *nbrec_lb,
>> -                            const char *vip_port_str, const char
> *backend_ips)
>> +                            const char *vip_port_str, const char
> *backend_ips,
>> +                            bool template)
>>  {
>>      lb_vip_nb->backend_ips = xstrdup(backend_ips);
>>      lb_vip_nb->n_backends = lb_vip->n_backends;
>>      lb_vip_nb->backends_nb = xcalloc(lb_vip_nb->n_backends,
>>                                       sizeof *lb_vip_nb->backends_nb);
>> -
>> -    struct nbrec_load_balancer_health_check *lb_health_check = NULL;
>> -    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
>> -        if (nbrec_lb->n_health_check > 0) {
>> -            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1,
> 1);
>> -            VLOG_WARN_RL(&rl,
>> -                         "SCTP load balancers do not currently support "
>> -                         "health checks. Not creating health checks for "
>> -                         "load balancer " UUID_FMT,
>> -                         UUID_ARGS(&nbrec_lb->header_.uuid));
>> -        }
>> -    } else {
>> -        for (size_t j = 0; j < nbrec_lb->n_health_check; j++) {
>> -            if (!strcmp(nbrec_lb->health_check[j]->vip, vip_port_str)) {
>> -                lb_health_check = nbrec_lb->health_check[j];
>> -                break;
>> -            }
>> -        }
>> -    }
>> -
>> -    lb_vip_nb->lb_health_check = lb_health_check;
>> +    lb_vip_nb->lb_health_check =
>> +        ovn_lb_get_health_check(nbrec_lb, vip_port_str, template);
>>  }
>>
>>  static
>> @@ -189,12 +399,113 @@ ovn_lb_get_hairpin_snat_ip(const struct uuid
> *lb_uuid,
>>      }
>>  }
>>
>> +static bool
>> +ovn_lb_get_routable_mode(const struct nbrec_load_balancer *nbrec_lb,
>> +                         bool routable, bool template)
>> +{
>> +    if (template && routable) {
>> +        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
>> +        VLOG_WARN_RL(&rl, "Template load balancer "UUID_FMT" does not
> suport "
>> +                           "option 'add_route'.  Forcing it to
> disabled.",
>> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
>> +        return false;
>> +    }
>> +    return routable;
>> +}
>> +
>> +static bool
>> +ovn_lb_neigh_mode_is_valid(enum lb_neighbor_responder_mode mode, bool
> template)
>> +{
>> +    if (!template) {
>> +        return true;
>> +    }
>> +
>> +    switch (mode) {
>> +    case LB_NEIGH_RESPOND_REACHABLE:
>> +        return false;
>> +    case LB_NEIGH_RESPOND_ALL:
>> +    case LB_NEIGH_RESPOND_NONE:
>> +        return true;
>> +    }
>> +    return false;
>> +}
>> +
>> +static enum lb_neighbor_responder_mode
>> +ovn_lb_get_neigh_mode(const struct nbrec_load_balancer *nbrec_lb,
>> +                      const char *mode, bool template)
>> +{
>> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
>> +    enum lb_neighbor_responder_mode default_mode =
>> +        template ? LB_NEIGH_RESPOND_NONE : LB_NEIGH_RESPOND_REACHABLE;
>> +
>> +    if (!mode) {
>> +        mode = lb_neighbor_responder_mode_names[default_mode];
>> +    }
>> +
>> +    for (size_t i = 0; i < ARRAY_SIZE(lb_neighbor_responder_mode_names);
> i++) {
>> +        if (!strcmp(mode, lb_neighbor_responder_mode_names[i])) {
>> +            if (ovn_lb_neigh_mode_is_valid(i, template)) {
>> +                return i;
>> +            }
>> +            break;
>> +        }
>> +    }
>> +
>> +    VLOG_WARN_RL(&rl, "Invalid neighbor responder mode %s for load
> balancer "
>> +                       UUID_FMT", forcing it to %s",
>> +                 mode, UUID_ARGS(&nbrec_lb->header_.uuid),
>> +                 lb_neighbor_responder_mode_names[default_mode]);
>> +    return default_mode;
>> +}
>> +
>> +static struct nbrec_load_balancer_health_check *
>> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
>> +                        const char *vip_port_str, bool template)
>> +{
>> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
>> +
>> +    if (!nbrec_lb->n_health_check) {
>> +        return NULL;
>> +    }
>> +
>> +    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
>> +        VLOG_WARN_RL(&rl,
>> +                     "SCTP load balancers do not currently support "
>> +                     "health checks. Not creating health checks for "
>> +                     "load balancer " UUID_FMT,
>> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
>> +        return NULL;
>> +    }
>> +
>> +    if (template) {
>> +        VLOG_WARN_RL(&rl,
>> +                     "Template load balancers do not currently support "
>> +                     "health checks. Not creating health checks for "
>> +                     "load balancer " UUID_FMT,
>> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
>> +        return NULL;
>> +    }
>> +
>> +    for (size_t i = 0; i < nbrec_lb->n_health_check; i++) {
>> +        if (!strcmp(nbrec_lb->health_check[i]->vip, vip_port_str)) {
>> +            return nbrec_lb->health_check[i];
>> +        }
>> +    }
>> +    return NULL;
>> +}
>> +
>>  struct ovn_northd_lb *
>>  ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb)
>>  {
>> +    bool template = smap_get_bool(&nbrec_lb->options, "template", false);
>>      bool is_udp = nullable_string_is_equal(nbrec_lb->protocol, "udp");
>>      bool is_sctp = nullable_string_is_equal(nbrec_lb->protocol, "sctp");
>>      struct ovn_northd_lb *lb = xzalloc(sizeof *lb);
>> +    int address_family = !strcmp(smap_get_def(&nbrec_lb->options,
>> +                                              "address-family", "ipv4"),
>> +                                 "ipv4")
>> +                         ? AF_INET
>> +                         : AF_INET6;
>>
>>      lb->nlb = nbrec_lb;
>>      lb->proto = is_udp ? "udp" : is_sctp ? "sctp" : "tcp";
>> @@ -202,12 +513,16 @@ ovn_northd_lb_create(const struct
> nbrec_load_balancer *nbrec_lb)
>>      lb->vips = xcalloc(lb->n_vips, sizeof *lb->vips);
>>      lb->vips_nb = xcalloc(lb->n_vips, sizeof *lb->vips_nb);
>>      lb->controller_event = smap_get_bool(&nbrec_lb->options, "event",
> false);
>> -    lb->routable = smap_get_bool(&nbrec_lb->options, "add_route", false);
>> +
>> +    bool routable = smap_get_bool(&nbrec_lb->options, "add_route",
> false);
>> +    lb->routable = ovn_lb_get_routable_mode(nbrec_lb, routable,
> template);
>> +
>>      lb->skip_snat = smap_get_bool(&nbrec_lb->options, "skip_snat",
> false);
>> -    const char *mode =
>> -        smap_get_def(&nbrec_lb->options, "neighbor_responder",
> "reachable");
>> -    lb->neigh_mode = strcmp(mode, "all") ? LB_NEIGH_RESPOND_REACHABLE
>> -                                         : LB_NEIGH_RESPOND_ALL;
>> +    lb->template = template;
>> +
>> +    const char *mode = smap_get(&nbrec_lb->options,
> "neighbor_responder");
>> +    lb->neigh_mode = ovn_lb_get_neigh_mode(nbrec_lb, mode, template);
>> +
>>      uint32_t affinity_timeout =
>>          smap_get_uint(&nbrec_lb->options, "affinity_timeout", 0);
>>      if (affinity_timeout > UINT16_MAX) {
>> @@ -227,13 +542,19 @@ ovn_northd_lb_create(const struct
> nbrec_load_balancer *nbrec_lb)
>>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
>>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[n_vips];
>>
>> -        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
>> -                                                  "reject", false);
>> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
>> +        char *error = ovn_lb_vip_init(lb_vip, node->key, node->value,
>> +                                      template, address_family);
>> +        if (error) {
>> +            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5,
> 1);
>> +            VLOG_WARN_RL(&rl, "Failed to initialize LB VIP: %s", error);
>> +            ovn_lb_vip_destroy(lb_vip);
>> +            free(error);
>>              continue;
>>          }
>> +        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
>> +                                                  "reject", false);
>>          ovn_northd_lb_vip_init(lb_vip_nb, lb_vip, nbrec_lb,
>> -                               node->key, node->value);
>> +                               node->key, node->value, template);
>>          if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>>              sset_add(&lb->ips_v4, lb_vip->vip_str);
>>          } else {
>> @@ -381,9 +702,12 @@ ovn_lb_group_find(const struct hmap *lb_groups,
> const struct uuid *uuid)
>>  }
>>
>>  struct ovn_controller_lb *
>> -ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb)
>> +ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb,
>> +                         const struct smap *template_vars,
>> +                         struct sset *template_vars_ref)
>>  {
>>      struct ovn_controller_lb *lb = xzalloc(sizeof *lb);
>> +    bool template = smap_get_bool(&sbrec_lb->options, "template", false);
>>
>>      lb->slb = sbrec_lb;
>>      lb->n_vips = smap_count(&sbrec_lb->vips);
>> @@ -395,10 +719,26 @@ ovn_controller_lb_create(const struct
> sbrec_load_balancer *sbrec_lb)
>>      SMAP_FOR_EACH (node, &sbrec_lb->vips) {
>>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
>>
>> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
>> -            continue;
>> +        struct lex_str key_s = template
>> +                               ? lexer_parse_template_string(node->key,
>> +
> template_vars,
>> +
> template_vars_ref)
>> +                               : lex_str_use(node->key);
>> +        struct lex_str value_s = template
>> +                               ? lexer_parse_template_string(node->value,
>> +
> template_vars,
>> +
> template_vars_ref)
>> +                               : lex_str_use(node->value);
>> +        char *error = ovn_lb_vip_init_explicit(lb_vip,
>> +                                               lex_str_get(&key_s),
>> +                                               lex_str_get(&value_s));
>> +        if (error) {
>> +            free(error);
>> +        } else {
>> +            n_vips++;
>>          }
>> -        n_vips++;
>> +        lex_str_free(&key_s);
>> +        lex_str_free(&value_s);
>>      }
>>
>>      /* It's possible that parsing VIPs fails.  Update the lb->n_vips to
> the
>> diff --git a/lib/lb.h b/lib/lb.h
>> index 62843e4716..55a41ae0bc 100644
>> --- a/lib/lb.h
>> +++ b/lib/lb.h
>> @@ -35,6 +35,7 @@ struct uuid;
>>  enum lb_neighbor_responder_mode {
>>      LB_NEIGH_RESPOND_REACHABLE,
>>      LB_NEIGH_RESPOND_ALL,
>> +    LB_NEIGH_RESPOND_NONE,
>>  };
>>
>>  /* The "routable" ssets are subsets of the load balancer IPs for which IP
>> @@ -67,6 +68,7 @@ struct ovn_northd_lb {
>>      bool controller_event;
>>      bool routable;
>>      bool skip_snat;
>> +    bool template;
>>      uint16_t affinity_timeout;
>>
>>      struct sset ips_v4;
>> @@ -82,19 +84,31 @@ struct ovn_northd_lb {
>>  };
>>
>>  struct ovn_lb_vip {
>> -    struct in6_addr vip;
>> -    char *vip_str;
>> -    uint16_t vip_port;
>> -
>> +    struct in6_addr vip; /* Only used in ovn-controller. */
>> +    char *vip_str;       /* Actual VIP string representation (without
> port).
>> +                          * To be used in ovn-northd.
>> +                          */
>> +    uint16_t vip_port;   /* Only used in ovn-controller. */
>> +    char *port_str;      /* Actual port string representation.  To be
> used
>> +                          * in ovn-northd.
>> +                          */
>>      struct ovn_lb_backend *backends;
>>      size_t n_backends;
>>      bool empty_backend_rej;
>> +    int address_family;
>>  };
>>
>>  struct ovn_lb_backend {
>> -    struct in6_addr ip;
>> -    char *ip_str;
>> -    uint16_t port;
>> +    struct in6_addr ip;  /* Only used in ovn-controller. */
>> +    char *ip_str;        /* Actual IP string representation. To be used
> in
>> +                          * ovn-northd.
>> +                          */
>> +    uint16_t port;       /* Mostly used in ovn-controller but also for
>> +                          * healthcheck in ovn-northd.
>> +                          */
>> +    char *port_str;      /* Actual port string representation. To be used
>> +                          * in ovn-northd.
>> +                          */
>>  };
>>
>>  /* ovn-northd specific backend information. */
>> @@ -174,7 +188,17 @@ struct ovn_controller_lb {
>>  };
>>
>>  struct ovn_controller_lb *ovn_controller_lb_create(
>> -    const struct sbrec_load_balancer *);
>> +    const struct sbrec_load_balancer *,
>> +    const struct smap *template_vars,
>> +    struct sset *template_vars_ref);
>>  void ovn_controller_lb_destroy(struct ovn_controller_lb *);
>>
>> +char *ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
>> +                      const char *lb_value, bool template, int
> address_family);
>> +void ovn_lb_vip_destroy(struct ovn_lb_vip *vip);
>> +void ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s,
>> +                       bool template);
>> +void ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct ds
> *s,
>> +                                bool template);
>> +
>>  #endif /* OVN_LIB_LB_H 1 */
>> diff --git a/lib/ovn-util.c b/lib/ovn-util.c
>> index 597625a291..1f8d0b8add 100644
>> --- a/lib/ovn-util.c
>> +++ b/lib/ovn-util.c
>> @@ -793,9 +793,6 @@ ip_address_and_port_from_lb_key(const char *key, char
> **ip_address,
>>  {
>>      struct sockaddr_storage ss;
>>      if (!inet_parse_active(key, 0, &ss, false, NULL)) {
>> -        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
>> -        VLOG_WARN_RL(&rl, "bad ip address or port for load balancer key
> %s",
>> -                     key);
>>          *ip_address = NULL;
>>          memset(ip, 0, sizeof(*ip));
>>          *port = 0;
>> diff --git a/northd/northd.c b/northd/northd.c
>> index 123127e9c1..c590c14818 100644
>> --- a/northd/northd.c
>> +++ b/northd/northd.c
>> @@ -3740,6 +3740,10 @@ static void
>>  ovn_lb_svc_create(struct ovsdb_idl_txn *ovnsb_txn, struct ovn_northd_lb
> *lb,
>>                    struct hmap *monitor_map, struct hmap *ports)
>>  {
>> +    if (lb->template) {
>> +        return;
>> +    }
>> +
>>      for (size_t i = 0; i < lb->n_vips; i++) {
>>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
>> @@ -4056,12 +4060,19 @@ static void
>>  build_lrouter_lb_reachable_ips(struct ovn_datapath *od,
>>                                 const struct ovn_northd_lb *lb)
>>  {
>> +    /* If configured to not reply to any neighbor requests for all VIPs
>> +     * return early.
>> +     */
>> +    if (lb->neigh_mode == LB_NEIGH_RESPOND_NONE) {
>> +        return;
>> +    }
>> +
>>      /* If configured to reply to neighbor requests for all VIPs force
> them
>>       * all to be considered "reachable".
>>       */
>>      if (lb->neigh_mode == LB_NEIGH_RESPOND_ALL) {
>>          for (size_t i = 0; i < lb->n_vips; i++) {
>> -            if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
>> +            if (lb->vips[i].address_family == AF_INET) {
>>                  sset_add(&od->lb_ips->ips_v4_reachable,
> lb->vips[i].vip_str);
>>              } else {
>>                  sset_add(&od->lb_ips->ips_v6_reachable,
> lb->vips[i].vip_str);
>> @@ -4073,8 +4084,9 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath
> *od,
>>      /* Otherwise, a VIP is reachable if there's at least one router
>>       * subnet that includes it.
>>       */
>> +    ovs_assert(lb->neigh_mode == LB_NEIGH_RESPOND_REACHABLE);
>>      for (size_t i = 0; i < lb->n_vips; i++) {
>> -        if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
>> +        if (lb->vips[i].address_family == AF_INET) {
>>              ovs_be32 vip_ip4 =
> in6_addr_get_mapped_ipv4(&lb->vips[i].vip);
>>              struct ovn_port *op;
>>
>> @@ -5834,16 +5846,16 @@ build_empty_lb_event_flow(struct ovn_lb_vip
> *lb_vip,
>>      ds_clear(action);
>>      ds_clear(match);
>>
>> -    bool ipv4 = IN6_IS_ADDR_V4MAPPED(&lb_vip->vip);
>> +    bool ipv4 = lb_vip->address_family == AF_INET;
>>
>>      ds_put_format(match, "ip%s.dst == %s && %s",
>>                    ipv4 ? "4": "6", lb_vip->vip_str, lb->proto);
>>
>>      char *vip = lb_vip->vip_str;
>> -    if (lb_vip->vip_port) {
>> -        ds_put_format(match, " && %s.dst == %u", lb->proto,
> lb_vip->vip_port);
>> -        vip = xasprintf("%s%s%s:%u", ipv4 ? "" : "[", lb_vip->vip_str,
>> -                        ipv4 ? "" : "]", lb_vip->vip_port);
>> +    if (lb_vip->port_str) {
>> +        ds_put_format(match, " && %s.dst == %s", lb->proto,
> lb_vip->port_str);
>> +        vip = xasprintf("%s%s%s:%s", ipv4 ? "" : "[", lb_vip->vip_str,
>> +                        ipv4 ? "" : "]", lb_vip->port_str);
>>      }
>>
>>      ds_put_format(action,
>> @@ -5854,7 +5866,7 @@ build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip,
>>                    event_to_string(OVN_EVENT_EMPTY_LB_BACKENDS),
>>                    vip, lb->proto,
>>                    UUID_ARGS(&lb->nlb->header_.uuid));
>> -    if (lb_vip->vip_port) {
>> +    if (lb_vip->port_str) {
>>          free(vip);
>>      }
>>      return true;
>> @@ -6910,7 +6922,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
> struct ovn_northd_lb *lb,
>>          /* Store the original destination IP to be used when generating
>>           * hairpin flows.
>>           */
>> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>> +        if (lb->vips[i].address_family == AF_INET) {
>>              ip_match = "ip4";
>>              ds_put_format(action, REG_ORIG_DIP_IPV4 " = %s; ",
>>                            lb_vip->vip_str);
>> @@ -6921,7 +6933,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
> struct ovn_northd_lb *lb,
>>          }
>>
>>          const char *proto = NULL;
>> -        if (lb_vip->vip_port) {
>> +        if (lb_vip->port_str) {
>>              proto = "tcp";
>>              if (lb->nlb->protocol) {
>>                  if (!strcmp(lb->nlb->protocol, "udp")) {
>> @@ -6934,14 +6946,14 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
> struct ovn_northd_lb *lb,
>>              /* Store the original destination port to be used when
> generating
>>               * hairpin flows.
>>               */
>> -            ds_put_format(action, REG_ORIG_TP_DPORT " = %"PRIu16"; ",
>> -                          lb_vip->vip_port);
>> +            ds_put_format(action, REG_ORIG_TP_DPORT " = %s; ",
>> +                          lb_vip->port_str);
>>          }
>>          ds_put_format(action, "%s;", ct_lb_mark ? "ct_lb_mark" :
> "ct_lb");
>>
>>          ds_put_format(match, "%s.dst == %s", ip_match, lb_vip->vip_str);
>> -        if (lb_vip->vip_port) {
>> -            ds_put_format(match, " && %s.dst == %d", proto,
> lb_vip->vip_port);
>> +        if (lb_vip->port_str) {
>> +            ds_put_format(match, " && %s.dst == %s", proto,
> lb_vip->port_str);
>>          }
>>
>>          struct ovn_lflow *lflow_ref = NULL;
>> @@ -7192,24 +7204,12 @@ build_lb_rules(struct hmap *lflows, struct
> ovn_northd_lb *lb, bool ct_lb_mark,
>>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
>>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
>>          const char *ip_match = NULL;
>> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>> +        if (lb_vip->address_family == AF_INET) {
>>              ip_match = "ip4";
>>          } else {
>>              ip_match = "ip6";
>>          }
>>
>> -        const char *proto = NULL;
>> -        if (lb_vip->vip_port) {
>> -            proto = "tcp";
>> -            if (lb->nlb->protocol) {
>> -                if (!strcmp(lb->nlb->protocol, "udp")) {
>> -                    proto = "udp";
>> -                } else if (!strcmp(lb->nlb->protocol, "sctp")) {
>> -                    proto = "sctp";
>> -                }
>> -            }
>> -        }
>> -
>>          ds_clear(action);
>>          ds_clear(match);
>>
>> @@ -7227,8 +7227,9 @@ build_lb_rules(struct hmap *lflows, struct
> ovn_northd_lb *lb, bool ct_lb_mark,
>>          ds_put_format(match, "ct.new && %s.dst == %s", ip_match,
>>                        lb_vip->vip_str);
>>          int priority = 110;
>> -        if (lb_vip->vip_port) {
>> -            ds_put_format(match, " && %s.dst == %d", proto,
> lb_vip->vip_port);
>> +        if (lb_vip->port_str) {
>> +            ds_put_format(match, " && %s.dst == %s", lb->proto,
>> +                          lb_vip->port_str);
>>              priority = 120;
>>          }
>>
>> @@ -10231,7 +10232,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
>>       * of "ct_lb_mark($targets);". The other flow is for ct.est with
>>       * an action of "next;".
>>       */
>> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>> +    if (lb_vip->address_family == AF_INET) {
>>          ds_put_format(match, "ip4 && "REG_NEXT_HOP_IPV4" == %s",
>>                        lb_vip->vip_str);
>>      } else {
>> @@ -10247,14 +10248,14 @@ build_lrouter_nat_flows_for_lb(struct
> ovn_lb_vip *lb_vip,
>>      }
>>
>>      int prio = 110;
>> -    if (lb_vip->vip_port) {
>> +    if (lb_vip->port_str) {
>>          prio = 120;
>>          new_match = xasprintf("ct.new && %s && %s && "
>> -                              REG_ORIG_TP_DPORT_ROUTER" == %d",
>> -                              ds_cstr(match), lb->proto,
> lb_vip->vip_port);
>> +                              REG_ORIG_TP_DPORT_ROUTER" == %s",
>> +                              ds_cstr(match), lb->proto,
> lb_vip->port_str);
>>          est_match = xasprintf("ct.est && %s && %s && "
>> -                              REG_ORIG_TP_DPORT_ROUTER" == %d && %s ==
> 1",
>> -                              ds_cstr(match), lb->proto,
> lb_vip->vip_port,
>> +                              REG_ORIG_TP_DPORT_ROUTER" == %s && %s ==
> 1",
>> +                              ds_cstr(match), lb->proto,
> lb_vip->port_str,
>>                                ct_natted);
>>      } else {
>>          new_match = xasprintf("ct.new && %s", ds_cstr(match));
>> @@ -10263,7 +10264,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
>>      }
>>
>>      const char *ip_match = NULL;
>> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>> +    if (lb_vip->address_family == AF_INET) {
>>          ip_match = "ip4";
>>      } else {
>>          ip_match = "ip6";
>> @@ -10281,9 +10282,9 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
>>          ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
>>                        backend->ip_str);
>>
>> -        if (backend->port) {
>> -            ds_put_format(&undnat_match, " && %s.src == %d) || ",
>> -                          lb->proto, backend->port);
>> +        if (backend->port_str) {
>> +            ds_put_format(&undnat_match, " && %s.src == %s) || ",
>> +                          lb->proto, backend->port_str);
>>          } else {
>>              ds_put_cstr(&undnat_match, ") || ");
>>          }
>> @@ -10296,9 +10297,9 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip
> *lb_vip,
>>      struct ds unsnat_match = DS_EMPTY_INITIALIZER;
>>      ds_put_format(&unsnat_match, "%s && %s.dst == %s && %s",
>>                    ip_match, ip_match, lb_vip->vip_str, lb->proto);
>> -    if (lb_vip->vip_port) {
>> -        ds_put_format(&unsnat_match, " && %s.dst == %d", lb->proto,
>> -                      lb_vip->vip_port);
>> +    if (lb_vip->port_str) {
>> +        ds_put_format(&unsnat_match, " && %s.dst == %s", lb->proto,
>> +                      lb_vip->port_str);
>>      }
>>
>>      struct ovn_datapath **gw_router_skip_snat =
>> @@ -10571,7 +10572,7 @@ build_lrouter_defrag_flows_for_lb(struct
> ovn_northd_lb *lb,
>>          ds_clear(&defrag_actions);
>>          ds_clear(match);
>>
>> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
>> +        if (lb_vip->address_family == AF_INET) {
>>              ds_put_format(match, "ip && ip4.dst == %s", lb_vip->vip_str);
>>              ds_put_format(&defrag_actions, REG_NEXT_HOP_IPV4" = %s; ",
>>                            lb_vip->vip_str);
>> @@ -10581,7 +10582,7 @@ build_lrouter_defrag_flows_for_lb(struct
> ovn_northd_lb *lb,
>>                            lb_vip->vip_str);
>>          }
>>
>> -        if (lb_vip->vip_port) {
>> +        if (lb_vip->port_str) {
>>              ds_put_format(match, " && %s", lb->proto);
>>              prio = 110;
>>
>> diff --git a/ovn-nb.xml b/ovn-nb.xml
>> index 553c0e48c3..8cd2427e8f 100644
>> --- a/ovn-nb.xml
>> +++ b/ovn-nb.xml
>> @@ -1905,8 +1905,66 @@
>>          is applied reply to ARP/neighbor discovery requests for all VIPs
>>          of the load balancer.  If set to <code>reachable</code>, then
> routers
>>          on which the load balancer is applied reply to ARP/neighbor
> discovery
>> -        requests only for VIPs that are part of a router's subnet.  The
> default
>> -        value of this option, if not specified, is
> <code>reachable</code>.
>> +        requests only for VIPs that are part of a router's subnet.  If
> set to
>> +        <code>none</code>, then routers on which the load balancer is
> applied
>> +        never reply to ARP/neighbor discovery requests for any of the
> load
>> +        balancer VIPs. Load balancers with
> <code>options:template=true</code>
>> +        do not support <code>reachable</code> as a valid mode.  The
> default
>> +        value of this option, if not specified, is
> <code>reachable</code> for
>> +        regular load balancers and <code>none</code> for template load
>> +        balancers.
>> +      </column>
>> +
>> +      <column name="options" key="template">
>> +        <p>
>> +          Option to be set to <code>true</code>, if the load balancer is
> a
>> +          template.  The load balancer VIPs and backends must be using
>> +          <ref table="Chassis_Template_Var"/> in their definitions.
>> +        </p>
>> +
>> +        <p>
>> +          Load balancer template VIP supported formats are:
>> +        </p>
>> +        <pre>
>> +^VIP_VAR[:^PORT_VAR|:port]
>> +        </pre>
>> +
>> +        <p>
>> +          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are names
> of
>> +        <ref table="Chassis_Template_Var"/> records.
> 
> In this version, the vars are not names but keys of the "variables" column.
> 

True, I fixed it.

>> +        </p>
>> +
>> +        <p>
>> +          Note: The VIP and PORT cannot be combined into a single
> template
>> +          variable. For example, a <ref table="Chassis_Template_Var"/>
>> +          variable expanding to <code>10.0.0.1:8080</code> is not valid
>> +          if used as VIP.
>> +        </p>
>> +
>> +        <p>
>> +          Load balancer template backend supported formats are:
>> +        </p>
>> +        <pre>
>> +^BACKEND_VAR1[:^PORT_VAR1|:port],^BACKEND_VAR2[:^PORT_VAR2|:port]
>> +
>> +or
>> +
>> +^BACKENDS_VAR1,^BACKENDS_VAR2
> 
> I think here each var means a single backend IP, right? So,
> s/BACKENDS/BACKEND/g
> 

Not necessarily.  There's actually no restriction.

>> +        </pre>
>> +        <p>
>> +          where <code>BACKEND_VAR1</code>, <code>PORT_VAR1</code>,
>> +          <code>BACKEND_VAR2</code>, <code>PORT_VAR2</code>,
>> +          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
> names
> 
> Same here, and they are keys instead of "names".
> 
> Acked-by: Han Zhou <hzhou@ovn.org>
> 

Thanks, Han!  I ended up with the following incremental.  Let me
know if it looks ok to you and I can fold it in.

Regards,
Dumitru

---
diff --git a/TODO.rst b/TODO.rst
index 53cf2870b2..15fd131d39 100644
--- a/TODO.rst
+++ b/TODO.rst
@@ -186,7 +186,4 @@ OVN To-do List
  * Load Balancer templates
 -  * Support combining the VIP (or backend) IP and port into a single
-    template variable.
-
-  * Support combining all backends into a single template variable.
+  * Support combining the VIP IP and port into a single template variable.
diff --git a/ovn-nb.xml b/ovn-nb.xml
index 7ecf2047e7..0edc3da96c 100644
--- a/ovn-nb.xml
+++ b/ovn-nb.xml
@@ -1958,8 +1958,9 @@
         </pre>
          <p>
-          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are names of
-        <ref table="Chassis_Template_Var"/> records.
+          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are keys of
+          the <ref table="Chassis_Template_Var"/> <ref column="variables"/>
+          records.
         </p>
          <p>
@@ -1982,8 +1983,9 @@ or
         <p>
           where <code>BACKEND_VAR1</code>, <code>PORT_VAR1</code>,
           <code>BACKEND_VAR2</code>, <code>PORT_VAR2</code>,
-          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
names
-          of <ref table="Chassis_Template_Var"/> records.
+          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
keys
+          of the <ref table="Chassis_Template_Var"/> <ref
column="variables"/>
+          records.
         </p>
       </column>
 diff --git a/tests/ovn.at b/tests/ovn.at
index bc3a7adfba..f3bd532423 100644
--- a/tests/ovn.at
+++ b/tests/ovn.at
@@ -33525,9 +33525,11 @@ dnl Create a few LBs that use "uninstantiated"
templates.
 check ovn-nbctl --template lb-add lb-test1 "^VIP1:^VPORT1" "^BACKENDS1" tcp
 check ovn-nbctl --template lb-add lb-test2 "^VIP2:^VPORT2"
"^BACKENDS21,^BACKENDS22" tcp
 check ovn-nbctl --template lb-add lb-test3 "^VIP3:^VPORT3"
"^BACKENDS31:^BPORT1,^BACKENDS32:^BPORT2" tcp
+check ovn-nbctl --template lb-add lb-test4 "^VIP4:^VPORT4"
"^BACKENDS41,^BACKENDS42" tcp
 check ovn-nbctl ls-lb-add sw lb-test1
 check ovn-nbctl ls-lb-add sw lb-test2
 check ovn-nbctl ls-lb-add sw lb-test3
+check ovn-nbctl ls-lb-add sw lb-test4
  check ovs-vsctl add-port br-int p1 -- set interface p1
external_ids:iface-id=lsp1
 check ovs-vsctl add-port br-int p2 -- set interface p2
external_ids:iface-id=lsp2
@@ -33549,7 +33551,10 @@ check ovn-nbctl --wait=hv set
Chassis_Template_Var hv1 \
     variables:VIP3='43.43.43.3' variables:VPORT3='4303' \
     variables:BACKENDS31='85.85.85.31' \
     variables:BACKENDS32='85.85.85.32' \
-    variables:BPORT1='8503' variables:BPORT2='8503'
+    variables:BPORT1='8503' variables:BPORT2='8503' \
+    variables:VIP4='43.43.43.4' variables:VPORT4='4304' \
+    variables:BACKENDS41='85.85.85.41:8504,85.85.85.42:8504' \
+    variables:BACKENDS42='85.85.85.43:8504,85.85.85.44:8504'
  dnl Ensure the LBs are translated to OpenFlow.
 as hv1
@@ -33568,6 +33573,18 @@ AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.31:8503)' -c], [
 AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.32:8503)' -c], [0], [dnl
 1
 ])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.41:8504)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.42:8504)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.43:8504)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=85.85.85.44:8504)' -c], [0], [dnl
+1
+])
  dnl Ensure hairpin flows are correct.
 as hv1
@@ -33577,6 +33594,10 @@ AT_CHECK([ovs-ofctl dump-flows br-int | grep
table=68 | ofctl_strip_all], [0], [
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b02,reg2=0x10ce/0xffff,nw_src=85.85.85.22,nw_dst=85.85.85.22,tp_dst=8502
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.31,nw_dst=85.85.85.31,tp_dst=8503
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.32,nw_dst=85.85.85.32,tp_dst=8503
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.41,nw_dst=85.85.85.41,tp_dst=8504
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.42,nw_dst=85.85.85.42,tp_dst=8504
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.43,nw_dst=85.85.85.43,tp_dst=8504
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.44,nw_dst=85.85.85.44,tp_dst=8504
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
 ])
  dnl Change Chassis_Template_Var mappings
@@ -33589,7 +33610,10 @@ check ovn-nbctl --wait=hv set
Chassis_Template_Var hv1 \
     variables:VIP3='42.42.42.3' variables:VPORT3='4203' \
     variables:BACKENDS31='84.84.84.31' \
     variables:BACKENDS32='84.84.84.32' \
-    variables:BPORT1='8403' variables:BPORT2='8403'
+    variables:BPORT1='8403' variables:BPORT2='8403' \
+    variables:VIP4='42.42.42.4' variables:VPORT4='4204' \
+    variables:BACKENDS41='84.84.84.41:8404,84.84.84.42:8404' \
+    variables:BACKENDS42='84.84.84.43:8404,84.84.84.44:8404'
  dnl Ensure the LBs are translated to OpenFlow.
 as hv1
@@ -33608,6 +33632,18 @@ AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.31:8403)' -c], [
 AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.32:8403)' -c], [0], [dnl
 1
 ])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.41:8404)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.42:8404)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.43:8404)' -c], [0], [dnl
+1
+])
+AT_CHECK([ovs-ofctl dump-groups br-int | grep
'nat(dst=84.84.84.44:8404)' -c], [0], [dnl
+1
+])
  dnl Ensure hairpin flows are correct.
 as hv1
@@ -33617,6 +33653,10 @@ AT_CHECK([ovs-ofctl dump-flows br-int | grep
table=68 | ofctl_strip_all], [0], [
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a02,reg2=0x106a/0xffff,nw_src=84.84.84.22,nw_dst=84.84.84.22,tp_dst=8402
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.31,nw_dst=84.84.84.31,tp_dst=8403
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
  table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.32,nw_dst=84.84.84.32,tp_dst=8403
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.41,nw_dst=84.84.84.41,tp_dst=8404
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.42,nw_dst=84.84.84.42,tp_dst=8404
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.43,nw_dst=84.84.84.43,tp_dst=8404
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
+ table=68,
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.44,nw_dst=84.84.84.44,tp_dst=8404
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
 ])
  dnl Remove Chassis_Template_Variables and check that everything is
---

On Tue, Nov 29, 2022 at 5:16 AM Dumitru Ceara <dceara@redhat.com> wrote:
>
> On 11/29/22 09:14, Han Zhou wrote:
> > On Tue, Nov 22, 2022 at 6:15 AM Dumitru Ceara <dceara@redhat.com> wrote:
> >>
> >> Allow the CMS to configure template LBs.  The following configurations
are
> >> supported:
> >> - VIPs of the form: ^vip_variable[:^port_variable|:port]
> >> - Backends of the form:
> >>
> >
^backendip_variable1[:^port_variable1|:port],^backendip_variable2[:^port_variable2|:port]
> >>   OR
> >>   ^backends_variable1,^backends_variable2
> >
> > Sorry if I missed, but I didn't see any tests that test the form
> > "^backends_variable1,^backends_variable2". I only see tests with a
single
> > backend variable with a single IP in it. Better to test:
> > 1. Multiple backends variables
>
> This should be covered.  I have this in the test:
>
> check ovn-nbctl --template lb-add lb-test1 "^VIP1:^VPORT1" "^BACKENDS1"
tcp
> check ovn-nbctl --template lb-add lb-test2 "^VIP2:^VPORT2"
> "^BACKENDS21,^BACKENDS22" tcp
> check ovn-nbctl --template lb-add lb-test3 "^VIP3:^VPORT3"
> "^BACKENDS31:^BPORT1,^BACKENDS32:^BPORT2" tcp
>
> > 2. Multiple IPs in a single variable (I saw this in the tutorial in
patch
> > 5, but better to be covered here, too)
> >
>
> You're right, I added a LB with such a variable instantiation to the
> test.
>
> >
> >
> >>
> >> The CMS needs to provide a bit more information than with non-template
> > load
> >> balancers and must explicitly specify the address family to be used.
> >>
> >> There is currently no support for template load balancers with
> >> options:add_route=true set.  That is because ovn-northd does not
> >> instantiate template variables.  While this is a limitation in a way,
its
> >> impact is not huge.  The load balancer 'add_route' option was added as
a
> >> way to make the CMS life easier and to avoid having to explicitly add a
> >> route for the VIP.  The CMS can still achieve the same logical
topology by
> >> explicitly adding the VIP route.
> >>
> >> Template load balancers don't support the "reachable"
neighbor-responder
> >> mode.  Instead the CMS can explicitly configure the responder mode to
> >> either "all" or "none".
> >>
> >> To properly handle template updates in ovn-controller we also add a
> >> Chassis_Template_Var <- LB reference in ovn-controller.  This way, when
> >> a Chassis_Template_Var changes value all load balancers that refer to
> >> it will also get updated.
> >>
> >> Signed-off-by: Dumitru Ceara <dceara@redhat.com>
> >> ---
> >> V3:
> >> - Addressed Mark's comments:
> >>   - Added TODO items about potential future template LB improvements.
> >>   - Removed n_backends arg from ovn_lb_backends_init_explicit() and
> >>     ovn_lb_backends_init_template().
> >>   - Fixed ovn_northd_lb_create() to first get the template option
before
> >>     using its value.
> >>   - Fixed up comments and man pages.
> >>   - Hardenned setting of address family in ovn-nbctl.
> >>
> >> V2:
> >> - Fix GCC build due to missing explicit return.
> >> - Fix ls_in_pre_stateful flows due to using wrong lb field.
> >> - Use new lexer_parse_template_string().
> >> - Changed lb_handle_changed_ref() signature to return bool.
> >> - Update documentation with info about responder mode=none, LB template
> >>   supported formats, lb explicit address family requirements.
> >> - Squashed the template LB patches into a single one
> >> - Added more tests.
> >> - Squashed the system tests patch into this one.
> >> ---
> >>  TODO.rst                    |    7 +
> >>  controller/lflow.c          |  115 +++++++++--
> >>  controller/lflow.h          |    7 +
> >>  controller/ovn-controller.c |   67 +++++-
> >>  lib/lb.c                    |  452
> > ++++++++++++++++++++++++++++++++++++++-----
> >>  lib/lb.h                    |   40 +++-
> >>  lib/ovn-util.c              |    3
> >>  northd/northd.c             |   89 ++++----
> >>  ovn-nb.xml                  |   62 ++++++
> >>  tests/ovn-nbctl.at          |   23 +-
> >>  tests/ovn-northd.at         |    7 +
> >>  tests/ovn.at                |  131 ++++++++++++
> >>  tests/system-ovn.at         |  183 +++++++++++++++++
> >>  utilities/ovn-nbctl.c       |  120 ++++++-----
> >>  14 files changed, 1078 insertions(+), 228 deletions(-)
> >>
> >> diff --git a/TODO.rst b/TODO.rst
> >> index fe5f9a2f30..53cf2870b2 100644
> >> --- a/TODO.rst
> >> +++ b/TODO.rst
> >> @@ -183,3 +183,10 @@ OVN To-do List
> >>  * Chassis_Template_Var
> >>
> >>    * Support template variables when tracing packets with ovn-trace.
> >> +
> >> +* Load Balancer templates
> >> +
> >> +  * Support combining the VIP (or backend) IP and port into a single
> >> +    template variable.
> >
> > Is it still a TODO for backends? At least the tutorial test (in patch
5) is
> > already doing something like this:
> > backends0="42.0.0.1:1,42.1.0.1:1,42.2.0.1:1,42.3.0.1:1,42.4.0.1:1"
> >
>
> You're right, we can combine the backends and it works fine.  I removed
> the "(or backend)" part.
>
> >> +
> >> +  * Support combining all backends into a single template variable.
> >
> > What does it mean here? Isn't the tutorial test already combining
multiple
> > backends into a single variable?
> >
>
> Yes, it is, and it's working fine.  I'm not sure anymore why I had
> added this TODO item here.  I removed it now.
>
> >> diff --git a/controller/lflow.c b/controller/lflow.c
> >> index 84625fb3f1..f6ac639541 100644
> >> --- a/controller/lflow.c
> >> +++ b/controller/lflow.c
> >> @@ -97,6 +97,15 @@ consider_logical_flow(const struct
sbrec_logical_flow
> > *lflow,
> >>                        struct lflow_ctx_in *l_ctx_in,
> >>                        struct lflow_ctx_out *l_ctx_out);
> >>
> >> +static void
> >> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
> >> +                          const struct sbrec_load_balancer *sbrec_lb,
> >> +                          const struct hmap *local_datapaths,
> >> +                          const struct smap *template_vars,
> >> +                          bool use_ct_mark,
> >> +                          struct ovn_desired_flow_table *flow_table,
> >> +                          struct simap *ids);
> >> +
> >>  static void add_port_sec_flows(const struct shash *binding_lports,
> >>                                 const struct sbrec_chassis *,
> >>                                 struct ovn_desired_flow_table *);
> >> @@ -223,7 +232,7 @@ lflow_handle_changed_flows(struct lflow_ctx_in
> > *l_ctx_in,
> >>          UUIDSET_INITIALIZER(&flood_remove_nodes);
> >>      SBREC_LOGICAL_FLOW_TABLE_FOR_EACH_TRACKED (lflow,
> >>
> > l_ctx_in->logical_flow_table) {
> >> -        if (uuidset_find(l_ctx_out->lflows_processed,
> > &lflow->header_.uuid)) {
> >> +        if (uuidset_find(l_ctx_out->objs_processed,
> > &lflow->header_.uuid)) {
> >>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
> >>                       UUID_ARGS(&lflow->header_.uuid));
> >>              continue;
> >> @@ -253,14 +262,14 @@ lflow_handle_changed_flows(struct lflow_ctx_in
> > *l_ctx_in,
> >>                       UUID_ARGS(&lflow->header_.uuid));
> >>
> >>              /* For the extra lflows that need to be reprocessed
because
> > of the
> >> -             * flood remove, remove it from lflows_processed. */
> >> +             * flood remove, remove it from objs_processed. */
> >>              struct uuidset_node *unode =
> >> -                uuidset_find(l_ctx_out->lflows_processed,
> >> +                uuidset_find(l_ctx_out->objs_processed,
> >>                               &lflow->header_.uuid);
> >>              if (unode) {
> >>                  VLOG_DBG("lflow "UUID_FMT"has been processed, now
> > reprocess.",
> >>                           UUID_ARGS(&lflow->header_.uuid));
> >> -                uuidset_delete(l_ctx_out->lflows_processed, unode);
> >> +                uuidset_delete(l_ctx_out->objs_processed, unode);
> >>              }
> >>
> >>              consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
> >> @@ -687,7 +696,7 @@ lflow_handle_addr_set_update(const char *as_name,
> >>      struct object_to_resources_list_node *resource_list_node;
> >>      RESOURCE_FOR_EACH_OBJ (resource_list_node, resource_node) {
> >>          const struct uuid *obj_uuid = &resource_list_node->obj_uuid;
> >> -        if (uuidset_find(l_ctx_out->lflows_processed, obj_uuid)) {
> >> +        if (uuidset_find(l_ctx_out->objs_processed, obj_uuid)) {
> >>              VLOG_DBG("lflow "UUID_FMT"has been processed, skip.",
> >>                       UUID_ARGS(obj_uuid));
> >>              continue;
> >> @@ -777,13 +786,13 @@ lflow_handle_changed_ref(enum objdep_type type,
> > const char *res_name,
> >>          }
> >>
> >>          /* For the extra lflows that need to be reprocessed because of
> > the
> >> -         * flood remove, remove it from lflows_processed. */
> >> +         * flood remove, remove it from objs_processed. */
> >>          struct uuidset_node *unode =
> >> -            uuidset_find(l_ctx_out->lflows_processed,
> > &lflow->header_.uuid);
> >> +            uuidset_find(l_ctx_out->objs_processed,
> > &lflow->header_.uuid);
> >>          if (unode) {
> >>              VLOG_DBG("lflow "UUID_FMT"has been processed, now
> > reprocess.",
> >>                       UUID_ARGS(&lflow->header_.uuid));
> >> -            uuidset_delete(l_ctx_out->lflows_processed, unode);
> >> +            uuidset_delete(l_ctx_out->objs_processed, unode);
> >>          }
> >>
> >>          consider_logical_flow(lflow, false, l_ctx_in, l_ctx_out);
> >> @@ -792,6 +801,43 @@ lflow_handle_changed_ref(enum objdep_type type,
> > const char *res_name,
> >>      return true;
> >>  }
> >>
> >> +bool
> >> +lb_handle_changed_ref(enum objdep_type type, const char *res_name,
> >> +                      struct ovs_list *objs_todo,
> >> +                      const void *in_arg, void *out_arg)
> >> +{
> >> +    struct lflow_ctx_in *l_ctx_in = CONST_CAST(struct lflow_ctx_in *,
> > in_arg);
> >> +    struct lflow_ctx_out *l_ctx_out = out_arg;
> >> +
> >> +    struct object_to_resources_list_node *resource_lb_uuid;
> >> +    LIST_FOR_EACH_POP (resource_lb_uuid, list_node, objs_todo) {
> >> +        VLOG_DBG("Reprocess LB "UUID_FMT" for resource type: %s, name:
> > %s",
> >> +                 UUID_ARGS(&resource_lb_uuid->obj_uuid),
> >> +                 objdep_type_name(type), res_name);
> >> +
> >> +        const struct sbrec_load_balancer *lb =
> >> +            sbrec_load_balancer_table_get_for_uuid(
> >> +                l_ctx_in->lb_table, &resource_lb_uuid->obj_uuid);
> >> +        if (!lb) {
> >> +            VLOG_DBG("Failed to find LB "UUID_FMT" referred by: %s",
> >
> > nit: I think it should be: Failed to find LB ... that refers ...
> >
>
> Yes, fixed.
>
> >
> >> +                     UUID_ARGS(&resource_lb_uuid->obj_uuid),
res_name);
> >> +        } else {
> >> +            ofctrl_remove_flows(l_ctx_out->flow_table,
> >> +                                &resource_lb_uuid->obj_uuid);
> >> +
> >> +            consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
> >> +                                      l_ctx_in->local_datapaths,
> >> +                                      l_ctx_in->template_vars,
> >> +
 l_ctx_in->lb_hairpin_use_ct_mark,
> >> +                                      l_ctx_out->flow_table,
> >> +                                      l_ctx_out->hairpin_lb_ids);
> >> +        }
> >> +
> >> +        free(resource_lb_uuid);
> >> +    }
> >> +    return true;
> >> +}
> >> +
> >>  static void
> >>  lflow_parse_ctrl_meter(const struct sbrec_logical_flow *lflow,
> >>                         struct ovn_extend_table *meter_table,
> >> @@ -1259,9 +1305,9 @@ consider_logical_flow(const struct
> > sbrec_logical_flow *lflow,
> >>
> >>      COVERAGE_INC(consider_logical_flow);
> >>      if (!is_recompute) {
> >> -        ovs_assert(!uuidset_find(l_ctx_out->lflows_processed,
> >> +        ovs_assert(!uuidset_find(l_ctx_out->objs_processed,
> >>                                   &lflow->header_.uuid));
> >> -        uuidset_insert(l_ctx_out->lflows_processed,
> > &lflow->header_.uuid);
> >> +        uuidset_insert(l_ctx_out->objs_processed,
&lflow->header_.uuid);
> >>      }
> >>
> >>      if (dp) {
> >> @@ -2001,8 +2047,10 @@ add_lb_ct_snat_hairpin_flows(struct
> > ovn_controller_lb *lb,
> >>  }
> >>
> >>  static void
> >> -consider_lb_hairpin_flows(const struct sbrec_load_balancer *sbrec_lb,
> >> +consider_lb_hairpin_flows(struct objdep_mgr *mgr,
> >> +                          const struct sbrec_load_balancer *sbrec_lb,
> >>                            const struct hmap *local_datapaths,
> >> +                          const struct smap *template_vars,
> >>                            bool use_ct_mark,
> >>                            struct ovn_desired_flow_table *flow_table,
> >>                            struct simap *ids)
> >> @@ -2039,7 +2087,9 @@ consider_lb_hairpin_flows(const struct
> > sbrec_load_balancer *sbrec_lb,
> >>          return;
> >>      }
> >>
> >> -    struct ovn_controller_lb *lb = ovn_controller_lb_create(sbrec_lb);
> >> +    struct sset template_vars_ref =
SSET_INITIALIZER(&template_vars_ref);
> >> +    struct ovn_controller_lb *lb =
> >> +        ovn_controller_lb_create(sbrec_lb, template_vars,
> > &template_vars_ref);
> >>      uint8_t lb_proto = IPPROTO_TCP;
> >>      if (lb->slb->protocol && lb->slb->protocol[0]) {
> >>          if (!strcmp(lb->slb->protocol, "udp")) {
> >> @@ -2049,6 +2099,11 @@ consider_lb_hairpin_flows(const struct
> > sbrec_load_balancer *sbrec_lb,
> >>          }
> >>      }
> >>
> >> +    const char *tv_name;
> >> +    SSET_FOR_EACH (tv_name, &template_vars_ref) {
> >> +        objdep_mgr_add(mgr, OBJDEP_TYPE_TEMPLATE, tv_name,
> >> +                       &sbrec_lb->header_.uuid);
> >> +    }
> >>      for (i = 0; i < lb->n_vips; i++) {
> >>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
> >>
> >> @@ -2063,13 +2118,17 @@ consider_lb_hairpin_flows(const struct
> > sbrec_load_balancer *sbrec_lb,
> >>      add_lb_ct_snat_hairpin_flows(lb, id, lb_proto, flow_table);
> >>
> >>      ovn_controller_lb_destroy(lb);
> >> +    sset_destroy(&template_vars_ref);
> >>  }
> >>
> >>  /* Adds OpenFlow flows to flow tables for each Load balancer VIPs and
> >>   * backends to handle the load balanced hairpin traffic. */
> >>  static void
> >> -add_lb_hairpin_flows(const struct sbrec_load_balancer_table *lb_table,
> >> -                     const struct hmap *local_datapaths, bool
> > use_ct_mark,
> >> +add_lb_hairpin_flows(struct objdep_mgr *mgr,
> >> +                     const struct sbrec_load_balancer_table *lb_table,
> >> +                     const struct hmap *local_datapaths,
> >> +                     const struct smap *template_vars,
> >> +                     bool use_ct_mark,
> >>                       struct ovn_desired_flow_table *flow_table,
> >>                       struct simap *ids,
> >>                       struct id_pool *pool)
> >> @@ -2092,8 +2151,8 @@ add_lb_hairpin_flows(const struct
> > sbrec_load_balancer_table *lb_table,
> >>              ovs_assert(id_pool_alloc_id(pool, &id));
> >>              simap_put(ids, lb->name, id);
> >>          }
> >> -        consider_lb_hairpin_flows(lb, local_datapaths, use_ct_mark,
> >> -                                  flow_table, ids);
> >> +        consider_lb_hairpin_flows(mgr, lb, local_datapaths,
> > template_vars,
> >> +                                  use_ct_mark, flow_table, ids);
> >>      }
> >>  }
> >>
> >> @@ -2229,7 +2288,9 @@ lflow_run(struct lflow_ctx_in *l_ctx_in, struct
> > lflow_ctx_out *l_ctx_out)
> >>                         l_ctx_in->static_mac_binding_table,
> >>                         l_ctx_in->local_datapaths,
> >>                         l_ctx_out->flow_table);
> >> -    add_lb_hairpin_flows(l_ctx_in->lb_table,
l_ctx_in->local_datapaths,
> >> +    add_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, l_ctx_in->lb_table,
> >> +                         l_ctx_in->local_datapaths,
> >> +                         l_ctx_in->template_vars,
> >>                           l_ctx_in->lb_hairpin_use_ct_mark,
> >>                           l_ctx_out->flow_table,
> >>                           l_ctx_out->hairpin_lb_ids,
> >> @@ -2280,10 +2341,10 @@ lflow_add_flows_for_datapath(const struct
> > sbrec_datapath_binding *dp,
> >>      const struct sbrec_logical_flow *lflow;
> >>      SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
> >>          lflow, lf_row,
l_ctx_in->sbrec_logical_flow_by_logical_datapath)
> > {
> >> -        if (uuidset_find(l_ctx_out->lflows_processed,
> > &lflow->header_.uuid)) {
> >> +        if (uuidset_find(l_ctx_out->objs_processed,
> > &lflow->header_.uuid)) {
> >>              continue;
> >>          }
> >> -        uuidset_insert(l_ctx_out->lflows_processed,
> > &lflow->header_.uuid);
> >> +        uuidset_insert(l_ctx_out->objs_processed,
&lflow->header_.uuid);
> >>          consider_logical_flow__(lflow, dp, l_ctx_in, l_ctx_out);
> >>      }
> >>      sbrec_logical_flow_index_destroy_row(lf_row);
> >> @@ -2308,7 +2369,7 @@ lflow_add_flows_for_datapath(const struct
> > sbrec_datapath_binding *dp,
> >>          sbrec_logical_flow_index_set_logical_dp_group(lf_row, ldpg);
> >>          SBREC_LOGICAL_FLOW_FOR_EACH_EQUAL (
> >>              lflow, lf_row,
> > l_ctx_in->sbrec_logical_flow_by_logical_dp_group) {
> >> -            if (uuidset_find(l_ctx_out->lflows_processed,
> >> +            if (uuidset_find(l_ctx_out->objs_processed,
> >>                               &lflow->header_.uuid)) {
> >>                  continue;
> >>              }
> >> @@ -2360,7 +2421,9 @@ lflow_add_flows_for_datapath(const struct
> > sbrec_datapath_binding *dp,
> >>      /* Add load balancer hairpin flows if the datapath has any load
> > balancers
> >>       * associated. */
> >>      for (size_t i = 0; i < n_dp_lbs; i++) {
> >> -        consider_lb_hairpin_flows(dp_lbs[i],
l_ctx_in->local_datapaths,
> >> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, dp_lbs[i],
> >> +                                  l_ctx_in->local_datapaths,
> >> +                                  l_ctx_in->template_vars,
> >>                                    l_ctx_in->lb_hairpin_use_ct_mark,
> >>                                    l_ctx_out->flow_table,
> >>                                    l_ctx_out->hairpin_lb_ids);
> >> @@ -2382,7 +2445,7 @@ lflow_handle_flows_for_lport(const struct
> > sbrec_port_binding *pb,
> >>                                    OBJDEP_TYPE_PORTBINDING,
> >>                                    pb->logical_port,
> >>                                    lflow_handle_changed_ref,
> >> -                                  l_ctx_out->lflows_processed,
> >> +                                  l_ctx_out->objs_processed,
> >>                                    l_ctx_in, l_ctx_out, &changed)) {
> >>          return false;
> >>      }
> >> @@ -2421,7 +2484,7 @@ lflow_handle_changed_port_bindings(struct
> > lflow_ctx_in *l_ctx_in,
> >>                                        OBJDEP_TYPE_PORTBINDING,
> >>                                        pb->logical_port,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out->lflows_processed,
> >> +                                      l_ctx_out->objs_processed,
> >>                                        l_ctx_in, l_ctx_out, &changed))
{
> >>              ret = false;
> >>              break;
> >> @@ -2448,7 +2511,7 @@ lflow_handle_changed_mc_groups(struct
lflow_ctx_in
> > *l_ctx_in,
> >>          if (!objdep_mgr_handle_change(l_ctx_out->lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_MC_GROUP,
> > ds_cstr(&mg_key),
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out->lflows_processed,
> >> +                                      l_ctx_out->objs_processed,
> >>                                        l_ctx_in, l_ctx_out, &changed))
{
> >>              ret = false;
> >>              break;
> >> @@ -2502,7 +2565,9 @@ lflow_handle_changed_lbs(struct lflow_ctx_in
> > *l_ctx_in,
> >>
> >>          VLOG_DBG("Add load balancer hairpin flows for "UUID_FMT,
> >>                   UUID_ARGS(&lb->header_.uuid));
> >> -        consider_lb_hairpin_flows(lb, l_ctx_in->local_datapaths,
> >> +        consider_lb_hairpin_flows(l_ctx_out->lb_deps_mgr, lb,
> >> +                                  l_ctx_in->local_datapaths,
> >> +                                  l_ctx_in->template_vars,
> >>                                    l_ctx_in->lb_hairpin_use_ct_mark,
> >>                                    l_ctx_out->flow_table,
> >>                                    l_ctx_out->hairpin_lb_ids);
> >> diff --git a/controller/lflow.h b/controller/lflow.h
> >> index d95fd41142..9e8f9afd33 100644
> >> --- a/controller/lflow.h
> >> +++ b/controller/lflow.h
> >> @@ -122,9 +122,10 @@ struct lflow_ctx_out {
> >>      struct ovn_extend_table *group_table;
> >>      struct ovn_extend_table *meter_table;
> >>      struct objdep_mgr *lflow_deps_mgr;
> >> +    struct objdep_mgr *lb_deps_mgr;
> >>      struct lflow_cache *lflow_cache;
> >>      struct conj_ids *conj_ids;
> >> -    struct uuidset *lflows_processed;
> >> +    struct uuidset *objs_processed;
> >>      struct simap *hairpin_lb_ids;
> >>      struct id_pool *hairpin_id_pool;
> >>  };
> >> @@ -174,4 +175,8 @@ bool lflow_handle_changed_mc_groups(struct
> > lflow_ctx_in *,
> >>                                      struct lflow_ctx_out *);
> >>  bool lflow_handle_changed_port_bindings(struct lflow_ctx_in *,
> >>                                          struct lflow_ctx_out *);
> >> +
> >> +bool lb_handle_changed_ref(enum objdep_type type, const char
*res_name,
> >> +                           struct ovs_list *objs_todo,
> >> +                           const void *in_arg, void *out_arg);
> >>  #endif /* controller/lflow.h */
> >> diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c
> >> index f9ed0e3855..9807ecd8eb 100644
> >> --- a/controller/ovn-controller.c
> >> +++ b/controller/ovn-controller.c
> >> @@ -2791,13 +2791,15 @@ struct ed_type_lflow_output {
> >>      struct ovn_extend_table meter_table;
> >>      /* lflow <-> resource cross reference */
> >>      struct objdep_mgr lflow_deps_mgr;;
> >> +    /* load balancer <-> resource cross reference */
> >> +    struct objdep_mgr lb_deps_mgr;
> >>      /* conjunciton ID usage information of lflows */
> >>      struct conj_ids conj_ids;
> >>
> >> -    /* lflows processed in the current engine execution.
> >> +    /* objects (lflows and lbs) processed in the current engine
> > execution.
> >>       * Cleared by en_lflow_output_clear_tracked_data before each
engine
> >>       * execution. */
> >> -    struct uuidset lflows_processed;
> >> +    struct uuidset objs_processed;
> >>
> >>      /* Data which is persistent and not cleared during
> >>       * full recompute. */
> >> @@ -2954,8 +2956,9 @@ init_lflow_ctx(struct engine_node *node,
> >>      l_ctx_out->group_table = &fo->group_table;
> >>      l_ctx_out->meter_table = &fo->meter_table;
> >>      l_ctx_out->lflow_deps_mgr = &fo->lflow_deps_mgr;
> >> +    l_ctx_out->lb_deps_mgr = &fo->lb_deps_mgr;
> >>      l_ctx_out->conj_ids = &fo->conj_ids;
> >> -    l_ctx_out->lflows_processed = &fo->lflows_processed;
> >> +    l_ctx_out->objs_processed = &fo->objs_processed;
> >>      l_ctx_out->lflow_cache = fo->pd.lflow_cache;
> >>      l_ctx_out->hairpin_id_pool = fo->hd.pool;
> >>      l_ctx_out->hairpin_lb_ids = &fo->hd.ids;
> >> @@ -2970,8 +2973,9 @@ en_lflow_output_init(struct engine_node *node
> > OVS_UNUSED,
> >>      ovn_extend_table_init(&data->group_table);
> >>      ovn_extend_table_init(&data->meter_table);
> >>      objdep_mgr_init(&data->lflow_deps_mgr);
> >> +    objdep_mgr_init(&data->lb_deps_mgr);
> >>      lflow_conj_ids_init(&data->conj_ids);
> >> -    uuidset_init(&data->lflows_processed);
> >> +    uuidset_init(&data->objs_processed);
> >>      simap_init(&data->hd.ids);
> >>      data->hd.pool = id_pool_create(1, UINT32_MAX - 1);
> >>      nd_ra_opts_init(&data->nd_ra_opts);
> >> @@ -2983,7 +2987,7 @@ static void
> >>  en_lflow_output_clear_tracked_data(void *data)
> >>  {
> >>      struct ed_type_lflow_output *flow_output_data = data;
> >> -    uuidset_clear(&flow_output_data->lflows_processed);
> >> +    uuidset_clear(&flow_output_data->objs_processed);
> >>  }
> >>
> >>  static void
> >> @@ -2994,8 +2998,9 @@ en_lflow_output_cleanup(void *data)
> >>      ovn_extend_table_destroy(&flow_output_data->group_table);
> >>      ovn_extend_table_destroy(&flow_output_data->meter_table);
> >>      objdep_mgr_destroy(&flow_output_data->lflow_deps_mgr);
> >> +    objdep_mgr_destroy(&flow_output_data->lb_deps_mgr);
> >>      lflow_conj_ids_destroy(&flow_output_data->conj_ids);
> >> -    uuidset_destroy(&flow_output_data->lflows_processed);
> >> +    uuidset_destroy(&flow_output_data->objs_processed);
> >>      lflow_cache_destroy(flow_output_data->pd.lflow_cache);
> >>      simap_destroy(&flow_output_data->hd.ids);
> >>      id_pool_destroy(flow_output_data->hd.pool);
> >> @@ -3030,6 +3035,7 @@ en_lflow_output_run(struct engine_node *node,
void
> > *data)
> >>      struct ovn_extend_table *group_table = &fo->group_table;
> >>      struct ovn_extend_table *meter_table = &fo->meter_table;
> >>      struct objdep_mgr *lflow_deps_mgr = &fo->lflow_deps_mgr;
> >> +    struct objdep_mgr *lb_deps_mgr = &fo->lb_deps_mgr;
> >>
> >>      static bool first_run = true;
> >>      if (first_run) {
> >> @@ -3039,6 +3045,7 @@ en_lflow_output_run(struct engine_node *node,
void
> > *data)
> >>          ovn_extend_table_clear(group_table, false /* desired */);
> >>          ovn_extend_table_clear(meter_table, false /* desired */);
> >>          objdep_mgr_clear(lflow_deps_mgr);
> >> +        objdep_mgr_clear(lb_deps_mgr);
> >>          lflow_conj_ids_clear(&fo->conj_ids);
> >>      }
> >>
> >> @@ -3172,7 +3179,7 @@ lflow_output_addr_sets_handler(struct engine_node
> > *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_ADDRSET, ref_name,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3191,7 +3198,7 @@ lflow_output_addr_sets_handler(struct engine_node
> > *node, void *data)
> >>                                            OBJDEP_TYPE_ADDRSET,
> >>                                            shash_node->name,
> >>                                            lflow_handle_changed_ref,
> >> -                                          l_ctx_out.lflows_processed,
> >> +                                          l_ctx_out.objs_processed,
> >>                                            &l_ctx_in, &l_ctx_out,
> > &changed)) {
> >>                  return false;
> >>              }
> >> @@ -3204,7 +3211,7 @@ lflow_output_addr_sets_handler(struct engine_node
> > *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_ADDRSET, ref_name,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3239,7 +3246,7 @@ lflow_output_port_groups_handler(struct
engine_node
> > *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3251,7 +3258,7 @@ lflow_output_port_groups_handler(struct
engine_node
> > *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3263,7 +3270,7 @@ lflow_output_port_groups_handler(struct
engine_node
> > *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_PORTGROUP, ref_name,
> >>                                        lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3297,7 +3304,17 @@ lflow_output_template_vars_handler(struct
> > engine_node *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_TEMPLATE,
> >>                                        res_name,
lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >> +                                      &l_ctx_in, &l_ctx_out,
&changed)) {
> >> +            return false;
> >> +        }
> >> +        if (changed) {
> >> +            engine_set_node_state(node, EN_UPDATED);
> >> +        }
> >> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> >> +                                      OBJDEP_TYPE_TEMPLATE,
> >> +                                      res_name, lb_handle_changed_ref,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3309,7 +3326,17 @@ lflow_output_template_vars_handler(struct
> > engine_node *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_TEMPLATE,
> >>                                        res_name,
lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >> +                                      &l_ctx_in, &l_ctx_out,
&changed)) {
> >> +            return false;
> >> +        }
> >> +        if (changed) {
> >> +            engine_set_node_state(node, EN_UPDATED);
> >> +        }
> >> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> >> +                                      OBJDEP_TYPE_TEMPLATE,
> >> +                                      res_name, lb_handle_changed_ref,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> @@ -3321,7 +3348,17 @@ lflow_output_template_vars_handler(struct
> > engine_node *node, void *data)
> >>          if (!objdep_mgr_handle_change(l_ctx_out.lflow_deps_mgr,
> >>                                        OBJDEP_TYPE_TEMPLATE,
> >>                                        res_name,
lflow_handle_changed_ref,
> >> -                                      l_ctx_out.lflows_processed,
> >> +                                      l_ctx_out.objs_processed,
> >> +                                      &l_ctx_in, &l_ctx_out,
&changed)) {
> >> +            return false;
> >> +        }
> >> +        if (changed) {
> >> +            engine_set_node_state(node, EN_UPDATED);
> >> +        }
> >> +        if (!objdep_mgr_handle_change(l_ctx_out.lb_deps_mgr,
> >> +                                      OBJDEP_TYPE_TEMPLATE,
> >> +                                      res_name, lb_handle_changed_ref,
> >> +                                      l_ctx_out.objs_processed,
> >>                                        &l_ctx_in, &l_ctx_out,
&changed)) {
> >>              return false;
> >>          }
> >> diff --git a/lib/lb.c b/lib/lb.c
> >> index c08ccceda1..43628bba77 100644
> >> --- a/lib/lb.c
> >> +++ b/lib/lb.c
> >> @@ -19,6 +19,7 @@
> >>  #include "lib/ovn-nb-idl.h"
> >>  #include "lib/ovn-sb-idl.h"
> >>  #include "lib/ovn-util.h"
> >> +#include "ovn/lex.h"
> >>
> >>  /* OpenvSwitch lib includes. */
> >>  #include "openvswitch/vlog.h"
> >> @@ -26,6 +27,16 @@
> >>
> >>  VLOG_DEFINE_THIS_MODULE(lb);
> >>
> >> +static const char *lb_neighbor_responder_mode_names[] = {
> >> +    [LB_NEIGH_RESPOND_REACHABLE] = "reachable",
> >> +    [LB_NEIGH_RESPOND_ALL] = "all",
> >> +    [LB_NEIGH_RESPOND_NONE] = "none",
> >> +};
> >> +
> >> +static struct nbrec_load_balancer_health_check *
> >> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
> >> +                        const char *vip_port_str, bool template);
> >> +
> >>  struct ovn_lb_ip_set *
> >>  ovn_lb_ip_set_create(void)
> >>  {
> >> @@ -71,94 +82,293 @@ ovn_lb_ip_set_clone(struct ovn_lb_ip_set
*lb_ip_set)
> >>      return clone;
> >>  }
> >>
> >> -static
> >> -bool ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> >> -                     const char *lb_value)
> >> +/* Format for backend ips: "IP1:port1,IP2:port2,...". */
> >> +static char *
> >> +ovn_lb_backends_init_explicit(struct ovn_lb_vip *lb_vip, const char
> > *value)
> >>  {
> >> -    int addr_family;
> >> -
> >> -    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
> >> -                                         &lb_vip->vip,
&lb_vip->vip_port,
> >> -                                         &addr_family)) {
> >> -        return false;
> >> -    }
> >> -
> >> -    /* Format for backend ips: "IP1:port1,IP2:port2,...". */
> >> -    size_t n_backends = 0;
> >> +    struct ds errors = DS_EMPTY_INITIALIZER;
> >>      size_t n_allocated_backends = 0;
> >> -    char *tokstr = xstrdup(lb_value);
> >> +    char *tokstr = xstrdup(value);
> >>      char *save_ptr = NULL;
> >> +    lb_vip->n_backends = 0;
> >> +
> >>      for (char *token = strtok_r(tokstr, ",", &save_ptr);
> >>          token != NULL;
> >>          token = strtok_r(NULL, ",", &save_ptr)) {
> >>
> >> -        if (n_backends == n_allocated_backends) {
> >> +        if (lb_vip->n_backends == n_allocated_backends) {
> >>              lb_vip->backends = x2nrealloc(lb_vip->backends,
> >>                                            &n_allocated_backends,
> >>                                            sizeof *lb_vip->backends);
> >>          }
> >>
> >> -        struct ovn_lb_backend *backend =
&lb_vip->backends[n_backends];
> >> +        struct ovn_lb_backend *backend =
> > &lb_vip->backends[lb_vip->n_backends];
> >>          int backend_addr_family;
> >>          if (!ip_address_and_port_from_lb_key(token, &backend->ip_str,
> >>                                               &backend->ip,
> > &backend->port,
> >>                                               &backend_addr_family)) {
> >> +            if (lb_vip->port_str) {
> >> +                ds_put_format(&errors, "%s: should be an IP address
and
> > a "
> >> +                                       "port number with : as a
> > separator, ",
> >> +                              token);
> >> +            } else {
> >> +                ds_put_format(&errors, "%s: should be an IP address,
",
> > token);
> >> +            }
> >>              continue;
> >>          }
> >>
> >> -        if (addr_family != backend_addr_family) {
> >> +        if (lb_vip->address_family != backend_addr_family) {
> >>              free(backend->ip_str);
> >> +            ds_put_format(&errors, "%s: IP address family is different
> > from "
> >> +                                   "VIP %s, ",
> >> +                          token, lb_vip->vip_str);
> >>              continue;
> >>          }
> >>
> >> -        n_backends++;
> >> +        if (lb_vip->port_str) {
> >> +            if (!backend->port) {
> >> +                free(backend->ip_str);
> >> +                ds_put_format(&errors, "%s: should be an IP address
and "
> >> +                                       "a port number with : as a
> > separator, ",
> >> +                              token);
> >> +                continue;
> >> +            }
> >> +        } else {
> >> +            if (backend->port) {
> >> +                free(backend->ip_str);
> >> +                ds_put_format(&errors, "%s: should be an IP address,
",
> > token);
> >> +                continue;
> >> +            }
> >> +        }
> >> +
> >> +        backend->port_str =
> >> +            backend->port ? xasprintf("%"PRIu16, backend->port) :
NULL;
> >> +        lb_vip->n_backends++;
> >>      }
> >>      free(tokstr);
> >> -    lb_vip->n_backends = n_backends;
> >> -    return true;
> >> +
> >> +    if (ds_last(&errors) != EOF) {
> >> +        ds_chomp(&errors, ' ');
> >> +        ds_chomp(&errors, ',');
> >> +        ds_put_char(&errors, '.');
> >> +        return ds_steal_cstr(&errors);
> >> +    }
> >> +    return NULL;
> >>  }
> >>
> >>  static
> >> -void ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
> >> +char *ovn_lb_vip_init_explicit(struct ovn_lb_vip *lb_vip, const char
> > *lb_key,
> >> +                               const char *lb_value)
> >> +{
> >> +    if (!ip_address_and_port_from_lb_key(lb_key, &lb_vip->vip_str,
> >> +                                         &lb_vip->vip,
&lb_vip->vip_port,
> >> +                                         &lb_vip->address_family)) {
> >> +        return xasprintf("%s: should be an IP address (or an IP
address "
> >> +                         "and a port number with : as a separator).",
> > lb_key);
> >> +    }
> >> +
> >> +    lb_vip->port_str = lb_vip->vip_port
> >> +                       ? xasprintf("%"PRIu16, lb_vip->vip_port)
> >> +                       : NULL;
> >> +
> >> +    return ovn_lb_backends_init_explicit(lb_vip, lb_value);
> >> +}
> >> +
> >> +/* Parses backends of a templated LB VIP.
> >> + * For now only the following template forms are supported:
> >> + * A.
> >> + *   ^backendip_variable1[:^port_variable1|:port],
> >> + *   ^backendip_variable2[:^port_variable2|:port]
> >> + *
> >> + * B.
> >> + *   ^backends_variable1,^backends_variable2 is also a thing
> >> + *      where 'backends_variable1' may expand to IP1_1:PORT1_1 on
> > chassis-1
> >> + *                                               IP1_2:PORT1_2 on
> > chassis-2
> >> + *        and 'backends_variable2' may expand to IP2_1:PORT2_1 on
> > chassis-1
> >> + *                                               IP2_2:PORT2_2 on
> > chassis-2
> >> + */
> >> +static char *
> >> +ovn_lb_backends_init_template(struct ovn_lb_vip *lb_vip, const char
> > *value_)
> >> +{
> >> +    struct ds errors = DS_EMPTY_INITIALIZER;
> >> +    char *value = xstrdup(value_);
> >> +    char *save_ptr = NULL;
> >> +    size_t n_allocated_backends = 0;
> >> +    lb_vip->n_backends = 0;
> >> +
> >> +    for (char *backend = strtok_r(value, ",", &save_ptr); backend;
> >> +         backend = strtok_r(NULL, ",", &save_ptr)) {
> >> +
> >> +        char *atom = xstrdup(backend);
> >> +        char *save_ptr2 = NULL;
> >> +        bool success = false;
> >> +        char *backend_ip = NULL;
> >> +        char *backend_port = NULL;
> >> +
> >> +        for (char *subatom = strtok_r(atom, ":", &save_ptr2); subatom;
> >> +             subatom = strtok_r(NULL, ":", &save_ptr2)) {
> >> +            if (backend_ip && backend_port) {
> >> +                success = false;
> >> +                break;
> >> +            }
> >> +            success = true;
> >> +            if (!backend_ip) {
> >> +                backend_ip = xstrdup(subatom);
> >> +            } else {
> >> +                backend_port = xstrdup(subatom);
> >> +            }
> >> +        }
> >> +
> >> +        if (success) {
> >> +            if (lb_vip->n_backends == n_allocated_backends) {
> >> +                lb_vip->backends = x2nrealloc(lb_vip->backends,
> >> +                                              &n_allocated_backends,
> >> +                                              sizeof
*lb_vip->backends);
> >> +            }
> >> +
> >> +            struct ovn_lb_backend *lb_backend =
> >> +                &lb_vip->backends[lb_vip->n_backends];
> >> +            lb_backend->ip_str = backend_ip;
> >> +            lb_backend->port_str = backend_port;
> >> +            lb_backend->port = 0;
> >> +            lb_vip->n_backends++;
> >> +        } else {
> >> +            ds_put_format(&errors, "%s: should be a template of the
> > form: "
> >> +
> >  "'^backendip_variable1[:^port_variable1|:port]', ",
> >> +                          atom);
> >> +        }
> >> +        free(atom);
> >> +    }
> >> +
> >> +    free(value);
> >> +    if (ds_last(&errors) != EOF) {
> >> +        ds_chomp(&errors, ' ');
> >> +        ds_chomp(&errors, ',');
> >> +        ds_put_char(&errors, '.');
> >> +        return ds_steal_cstr(&errors);
> >> +    }
> >> +    return NULL;
> >> +}
> >> +
> >> +/* Parses a VIP of a templated LB.
> >> + * For now only the following template forms are supported:
> >> + *   ^vip_variable[:^port_variable|:port]
> >> + */
> >> +static char *
> >> +ovn_lb_vip_init_template(struct ovn_lb_vip *lb_vip, const char
*lb_key_,
> >> +                         const char *lb_value, int address_family)
> >> +{
> >> +    char *save_ptr = NULL;
> >> +    char *lb_key = xstrdup(lb_key_);
> >> +    bool success = false;
> >> +
> >> +    for (char *atom = strtok_r(lb_key, ":", &save_ptr); atom;
> >> +         atom = strtok_r(NULL, ":", &save_ptr)) {
> >> +        if (lb_vip->vip_str && lb_vip->port_str) {
> >> +            success = false;
> >> +            break;
> >> +        }
> >> +        success = true;
> >> +        if (!lb_vip->vip_str) {
> >> +            lb_vip->vip_str = xstrdup(atom);
> >> +        } else {
> >> +            lb_vip->port_str = xstrdup(atom);
> >> +        }
> >> +    }
> >> +    free(lb_key);
> >> +
> >> +    if (!success) {
> >> +        return xasprintf("%s: should be a template of the form: "
> >> +                         "'^vip_variable[:^port_variable|:port]'.",
> >> +                         lb_key_);
> >> +    }
> >> +
> >> +    lb_vip->address_family = address_family;
> >> +    return ovn_lb_backends_init_template(lb_vip, lb_value);
> >> +}
> >> +
> >> +/* Returns NULL on success, an error string on failure.  The caller is
> >> + * responsible for destroying 'lb_vip' in all cases.
> >> + */
> >> +char *
> >> +ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> >> +                const char *lb_value, bool template, int
address_family)
> >> +{
> >> +    memset(lb_vip, 0, sizeof *lb_vip);
> >> +
> >> +    return !template
> >> +           ?  ovn_lb_vip_init_explicit(lb_vip, lb_key, lb_value)
> >> +           :  ovn_lb_vip_init_template(lb_vip, lb_key, lb_value,
> >> +                                       address_family);
> >> +}
> >> +
> >> +void
> >> +ovn_lb_vip_destroy(struct ovn_lb_vip *vip)
> >>  {
> >>      free(vip->vip_str);
> >> +    free(vip->port_str);
> >>      for (size_t i = 0; i < vip->n_backends; i++) {
> >>          free(vip->backends[i].ip_str);
> >> +        free(vip->backends[i].port_str);
> >>      }
> >>      free(vip->backends);
> >>  }
> >>
> >> +void
> >> +ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s, bool
> > template)
> >> +{
> >> +    bool needs_brackets = vip->address_family == AF_INET6 &&
> > vip->port_str
> >> +                          && !template;
> >> +    if (needs_brackets) {
> >> +        ds_put_char(s, '[');
> >> +    }
> >> +    ds_put_cstr(s, vip->vip_str);
> >> +    if (needs_brackets) {
> >> +        ds_put_char(s, ']');
> >> +    }
> >> +    if (vip->port_str) {
> >> +        ds_put_format(s, ":%s", vip->port_str);
> >> +    }
> >> +}
> >> +
> >> +void
> >> +ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct ds *s,
> >> +                           bool template)
> >> +{
> >> +    bool needs_brackets = vip->address_family == AF_INET6 &&
> > vip->port_str
> >> +                          && !template;
> >> +    for (size_t i = 0; i < vip->n_backends; i++) {
> >> +        struct ovn_lb_backend *backend = &vip->backends[i];
> >> +
> >> +        if (needs_brackets) {
> >> +            ds_put_char(s, '[');
> >> +        }
> >> +        ds_put_cstr(s, backend->ip_str);
> >> +        if (needs_brackets) {
> >> +            ds_put_char(s, ']');
> >> +        }
> >> +        if (backend->port_str) {
> >> +            ds_put_format(s, ":%s", backend->port_str);
> >> +        }
> >> +        if (i != vip->n_backends - 1) {
> >> +            ds_put_char(s, ',');
> >> +        }
> >> +    }
> >> +}
> >> +
> >>  static
> >>  void ovn_northd_lb_vip_init(struct ovn_northd_lb_vip *lb_vip_nb,
> >>                              const struct ovn_lb_vip *lb_vip,
> >>                              const struct nbrec_load_balancer
*nbrec_lb,
> >> -                            const char *vip_port_str, const char
> > *backend_ips)
> >> +                            const char *vip_port_str, const char
> > *backend_ips,
> >> +                            bool template)
> >>  {
> >>      lb_vip_nb->backend_ips = xstrdup(backend_ips);
> >>      lb_vip_nb->n_backends = lb_vip->n_backends;
> >>      lb_vip_nb->backends_nb = xcalloc(lb_vip_nb->n_backends,
> >>                                       sizeof *lb_vip_nb->backends_nb);
> >> -
> >> -    struct nbrec_load_balancer_health_check *lb_health_check = NULL;
> >> -    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
> >> -        if (nbrec_lb->n_health_check > 0) {
> >> -            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1,
> > 1);
> >> -            VLOG_WARN_RL(&rl,
> >> -                         "SCTP load balancers do not currently
support "
> >> -                         "health checks. Not creating health checks
for "
> >> -                         "load balancer " UUID_FMT,
> >> -                         UUID_ARGS(&nbrec_lb->header_.uuid));
> >> -        }
> >> -    } else {
> >> -        for (size_t j = 0; j < nbrec_lb->n_health_check; j++) {
> >> -            if (!strcmp(nbrec_lb->health_check[j]->vip,
vip_port_str)) {
> >> -                lb_health_check = nbrec_lb->health_check[j];
> >> -                break;
> >> -            }
> >> -        }
> >> -    }
> >> -
> >> -    lb_vip_nb->lb_health_check = lb_health_check;
> >> +    lb_vip_nb->lb_health_check =
> >> +        ovn_lb_get_health_check(nbrec_lb, vip_port_str, template);
> >>  }
> >>
> >>  static
> >> @@ -189,12 +399,113 @@ ovn_lb_get_hairpin_snat_ip(const struct uuid
> > *lb_uuid,
> >>      }
> >>  }
> >>
> >> +static bool
> >> +ovn_lb_get_routable_mode(const struct nbrec_load_balancer *nbrec_lb,
> >> +                         bool routable, bool template)
> >> +{
> >> +    if (template && routable) {
> >> +        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> >> +        VLOG_WARN_RL(&rl, "Template load balancer "UUID_FMT" does not
> > suport "
> >> +                           "option 'add_route'.  Forcing it to
> > disabled.",
> >> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> >> +        return false;
> >> +    }
> >> +    return routable;
> >> +}
> >> +
> >> +static bool
> >> +ovn_lb_neigh_mode_is_valid(enum lb_neighbor_responder_mode mode, bool
> > template)
> >> +{
> >> +    if (!template) {
> >> +        return true;
> >> +    }
> >> +
> >> +    switch (mode) {
> >> +    case LB_NEIGH_RESPOND_REACHABLE:
> >> +        return false;
> >> +    case LB_NEIGH_RESPOND_ALL:
> >> +    case LB_NEIGH_RESPOND_NONE:
> >> +        return true;
> >> +    }
> >> +    return false;
> >> +}
> >> +
> >> +static enum lb_neighbor_responder_mode
> >> +ovn_lb_get_neigh_mode(const struct nbrec_load_balancer *nbrec_lb,
> >> +                      const char *mode, bool template)
> >> +{
> >> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> >> +    enum lb_neighbor_responder_mode default_mode =
> >> +        template ? LB_NEIGH_RESPOND_NONE : LB_NEIGH_RESPOND_REACHABLE;
> >> +
> >> +    if (!mode) {
> >> +        mode = lb_neighbor_responder_mode_names[default_mode];
> >> +    }
> >> +
> >> +    for (size_t i = 0; i <
ARRAY_SIZE(lb_neighbor_responder_mode_names);
> > i++) {
> >> +        if (!strcmp(mode, lb_neighbor_responder_mode_names[i])) {
> >> +            if (ovn_lb_neigh_mode_is_valid(i, template)) {
> >> +                return i;
> >> +            }
> >> +            break;
> >> +        }
> >> +    }
> >> +
> >> +    VLOG_WARN_RL(&rl, "Invalid neighbor responder mode %s for load
> > balancer "
> >> +                       UUID_FMT", forcing it to %s",
> >> +                 mode, UUID_ARGS(&nbrec_lb->header_.uuid),
> >> +                 lb_neighbor_responder_mode_names[default_mode]);
> >> +    return default_mode;
> >> +}
> >> +
> >> +static struct nbrec_load_balancer_health_check *
> >> +ovn_lb_get_health_check(const struct nbrec_load_balancer *nbrec_lb,
> >> +                        const char *vip_port_str, bool template)
> >> +{
> >> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> >> +
> >> +    if (!nbrec_lb->n_health_check) {
> >> +        return NULL;
> >> +    }
> >> +
> >> +    if (nbrec_lb->protocol && !strcmp(nbrec_lb->protocol, "sctp")) {
> >> +        VLOG_WARN_RL(&rl,
> >> +                     "SCTP load balancers do not currently support "
> >> +                     "health checks. Not creating health checks for "
> >> +                     "load balancer " UUID_FMT,
> >> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> >> +        return NULL;
> >> +    }
> >> +
> >> +    if (template) {
> >> +        VLOG_WARN_RL(&rl,
> >> +                     "Template load balancers do not currently
support "
> >> +                     "health checks. Not creating health checks for "
> >> +                     "load balancer " UUID_FMT,
> >> +                     UUID_ARGS(&nbrec_lb->header_.uuid));
> >> +        return NULL;
> >> +    }
> >> +
> >> +    for (size_t i = 0; i < nbrec_lb->n_health_check; i++) {
> >> +        if (!strcmp(nbrec_lb->health_check[i]->vip, vip_port_str)) {
> >> +            return nbrec_lb->health_check[i];
> >> +        }
> >> +    }
> >> +    return NULL;
> >> +}
> >> +
> >>  struct ovn_northd_lb *
> >>  ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb)
> >>  {
> >> +    bool template = smap_get_bool(&nbrec_lb->options, "template",
false);
> >>      bool is_udp = nullable_string_is_equal(nbrec_lb->protocol, "udp");
> >>      bool is_sctp = nullable_string_is_equal(nbrec_lb->protocol,
"sctp");
> >>      struct ovn_northd_lb *lb = xzalloc(sizeof *lb);
> >> +    int address_family = !strcmp(smap_get_def(&nbrec_lb->options,
> >> +                                              "address-family",
"ipv4"),
> >> +                                 "ipv4")
> >> +                         ? AF_INET
> >> +                         : AF_INET6;
> >>
> >>      lb->nlb = nbrec_lb;
> >>      lb->proto = is_udp ? "udp" : is_sctp ? "sctp" : "tcp";
> >> @@ -202,12 +513,16 @@ ovn_northd_lb_create(const struct
> > nbrec_load_balancer *nbrec_lb)
> >>      lb->vips = xcalloc(lb->n_vips, sizeof *lb->vips);
> >>      lb->vips_nb = xcalloc(lb->n_vips, sizeof *lb->vips_nb);
> >>      lb->controller_event = smap_get_bool(&nbrec_lb->options, "event",
> > false);
> >> -    lb->routable = smap_get_bool(&nbrec_lb->options, "add_route",
false);
> >> +
> >> +    bool routable = smap_get_bool(&nbrec_lb->options, "add_route",
> > false);
> >> +    lb->routable = ovn_lb_get_routable_mode(nbrec_lb, routable,
> > template);
> >> +
> >>      lb->skip_snat = smap_get_bool(&nbrec_lb->options, "skip_snat",
> > false);
> >> -    const char *mode =
> >> -        smap_get_def(&nbrec_lb->options, "neighbor_responder",
> > "reachable");
> >> -    lb->neigh_mode = strcmp(mode, "all") ? LB_NEIGH_RESPOND_REACHABLE
> >> -                                         : LB_NEIGH_RESPOND_ALL;
> >> +    lb->template = template;
> >> +
> >> +    const char *mode = smap_get(&nbrec_lb->options,
> > "neighbor_responder");
> >> +    lb->neigh_mode = ovn_lb_get_neigh_mode(nbrec_lb, mode, template);
> >> +
> >>      uint32_t affinity_timeout =
> >>          smap_get_uint(&nbrec_lb->options, "affinity_timeout", 0);
> >>      if (affinity_timeout > UINT16_MAX) {
> >> @@ -227,13 +542,19 @@ ovn_northd_lb_create(const struct
> > nbrec_load_balancer *nbrec_lb)
> >>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
> >>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[n_vips];
> >>
> >> -        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
> >> -                                                  "reject", false);
> >> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
> >> +        char *error = ovn_lb_vip_init(lb_vip, node->key, node->value,
> >> +                                      template, address_family);
> >> +        if (error) {
> >> +            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5,
> > 1);
> >> +            VLOG_WARN_RL(&rl, "Failed to initialize LB VIP: %s",
error);
> >> +            ovn_lb_vip_destroy(lb_vip);
> >> +            free(error);
> >>              continue;
> >>          }
> >> +        lb_vip->empty_backend_rej = smap_get_bool(&nbrec_lb->options,
> >> +                                                  "reject", false);
> >>          ovn_northd_lb_vip_init(lb_vip_nb, lb_vip, nbrec_lb,
> >> -                               node->key, node->value);
> >> +                               node->key, node->value, template);
> >>          if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >>              sset_add(&lb->ips_v4, lb_vip->vip_str);
> >>          } else {
> >> @@ -381,9 +702,12 @@ ovn_lb_group_find(const struct hmap *lb_groups,
> > const struct uuid *uuid)
> >>  }
> >>
> >>  struct ovn_controller_lb *
> >> -ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb)
> >> +ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb,
> >> +                         const struct smap *template_vars,
> >> +                         struct sset *template_vars_ref)
> >>  {
> >>      struct ovn_controller_lb *lb = xzalloc(sizeof *lb);
> >> +    bool template = smap_get_bool(&sbrec_lb->options, "template",
false);
> >>
> >>      lb->slb = sbrec_lb;
> >>      lb->n_vips = smap_count(&sbrec_lb->vips);
> >> @@ -395,10 +719,26 @@ ovn_controller_lb_create(const struct
> > sbrec_load_balancer *sbrec_lb)
> >>      SMAP_FOR_EACH (node, &sbrec_lb->vips) {
> >>          struct ovn_lb_vip *lb_vip = &lb->vips[n_vips];
> >>
> >> -        if (!ovn_lb_vip_init(lb_vip, node->key, node->value)) {
> >> -            continue;
> >> +        struct lex_str key_s = template
> >> +                               ?
lexer_parse_template_string(node->key,
> >> +
> > template_vars,
> >> +
> > template_vars_ref)
> >> +                               : lex_str_use(node->key);
> >> +        struct lex_str value_s = template
> >> +                               ?
lexer_parse_template_string(node->value,
> >> +
> > template_vars,
> >> +
> > template_vars_ref)
> >> +                               : lex_str_use(node->value);
> >> +        char *error = ovn_lb_vip_init_explicit(lb_vip,
> >> +                                               lex_str_get(&key_s),
> >> +                                               lex_str_get(&value_s));
> >> +        if (error) {
> >> +            free(error);
> >> +        } else {
> >> +            n_vips++;
> >>          }
> >> -        n_vips++;
> >> +        lex_str_free(&key_s);
> >> +        lex_str_free(&value_s);
> >>      }
> >>
> >>      /* It's possible that parsing VIPs fails.  Update the lb->n_vips
to
> > the
> >> diff --git a/lib/lb.h b/lib/lb.h
> >> index 62843e4716..55a41ae0bc 100644
> >> --- a/lib/lb.h
> >> +++ b/lib/lb.h
> >> @@ -35,6 +35,7 @@ struct uuid;
> >>  enum lb_neighbor_responder_mode {
> >>      LB_NEIGH_RESPOND_REACHABLE,
> >>      LB_NEIGH_RESPOND_ALL,
> >> +    LB_NEIGH_RESPOND_NONE,
> >>  };
> >>
> >>  /* The "routable" ssets are subsets of the load balancer IPs for
which IP
> >> @@ -67,6 +68,7 @@ struct ovn_northd_lb {
> >>      bool controller_event;
> >>      bool routable;
> >>      bool skip_snat;
> >> +    bool template;
> >>      uint16_t affinity_timeout;
> >>
> >>      struct sset ips_v4;
> >> @@ -82,19 +84,31 @@ struct ovn_northd_lb {
> >>  };
> >>
> >>  struct ovn_lb_vip {
> >> -    struct in6_addr vip;
> >> -    char *vip_str;
> >> -    uint16_t vip_port;
> >> -
> >> +    struct in6_addr vip; /* Only used in ovn-controller. */
> >> +    char *vip_str;       /* Actual VIP string representation (without
> > port).
> >> +                          * To be used in ovn-northd.
> >> +                          */
> >> +    uint16_t vip_port;   /* Only used in ovn-controller. */
> >> +    char *port_str;      /* Actual port string representation.  To be
> > used
> >> +                          * in ovn-northd.
> >> +                          */
> >>      struct ovn_lb_backend *backends;
> >>      size_t n_backends;
> >>      bool empty_backend_rej;
> >> +    int address_family;
> >>  };
> >>
> >>  struct ovn_lb_backend {
> >> -    struct in6_addr ip;
> >> -    char *ip_str;
> >> -    uint16_t port;
> >> +    struct in6_addr ip;  /* Only used in ovn-controller. */
> >> +    char *ip_str;        /* Actual IP string representation. To be
used
> > in
> >> +                          * ovn-northd.
> >> +                          */
> >> +    uint16_t port;       /* Mostly used in ovn-controller but also for
> >> +                          * healthcheck in ovn-northd.
> >> +                          */
> >> +    char *port_str;      /* Actual port string representation. To be
used
> >> +                          * in ovn-northd.
> >> +                          */
> >>  };
> >>
> >>  /* ovn-northd specific backend information. */
> >> @@ -174,7 +188,17 @@ struct ovn_controller_lb {
> >>  };
> >>
> >>  struct ovn_controller_lb *ovn_controller_lb_create(
> >> -    const struct sbrec_load_balancer *);
> >> +    const struct sbrec_load_balancer *,
> >> +    const struct smap *template_vars,
> >> +    struct sset *template_vars_ref);
> >>  void ovn_controller_lb_destroy(struct ovn_controller_lb *);
> >>
> >> +char *ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key,
> >> +                      const char *lb_value, bool template, int
> > address_family);
> >> +void ovn_lb_vip_destroy(struct ovn_lb_vip *vip);
> >> +void ovn_lb_vip_format(const struct ovn_lb_vip *vip, struct ds *s,
> >> +                       bool template);
> >> +void ovn_lb_vip_backends_format(const struct ovn_lb_vip *vip, struct
ds
> > *s,
> >> +                                bool template);
> >> +
> >>  #endif /* OVN_LIB_LB_H 1 */
> >> diff --git a/lib/ovn-util.c b/lib/ovn-util.c
> >> index 597625a291..1f8d0b8add 100644
> >> --- a/lib/ovn-util.c
> >> +++ b/lib/ovn-util.c
> >> @@ -793,9 +793,6 @@ ip_address_and_port_from_lb_key(const char *key,
char
> > **ip_address,
> >>  {
> >>      struct sockaddr_storage ss;
> >>      if (!inet_parse_active(key, 0, &ss, false, NULL)) {
> >> -        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
> >> -        VLOG_WARN_RL(&rl, "bad ip address or port for load balancer
key
> > %s",
> >> -                     key);
> >>          *ip_address = NULL;
> >>          memset(ip, 0, sizeof(*ip));
> >>          *port = 0;
> >> diff --git a/northd/northd.c b/northd/northd.c
> >> index 123127e9c1..c590c14818 100644
> >> --- a/northd/northd.c
> >> +++ b/northd/northd.c
> >> @@ -3740,6 +3740,10 @@ static void
> >>  ovn_lb_svc_create(struct ovsdb_idl_txn *ovnsb_txn, struct
ovn_northd_lb
> > *lb,
> >>                    struct hmap *monitor_map, struct hmap *ports)
> >>  {
> >> +    if (lb->template) {
> >> +        return;
> >> +    }
> >> +
> >>      for (size_t i = 0; i < lb->n_vips; i++) {
> >>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
> >>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
> >> @@ -4056,12 +4060,19 @@ static void
> >>  build_lrouter_lb_reachable_ips(struct ovn_datapath *od,
> >>                                 const struct ovn_northd_lb *lb)
> >>  {
> >> +    /* If configured to not reply to any neighbor requests for all
VIPs
> >> +     * return early.
> >> +     */
> >> +    if (lb->neigh_mode == LB_NEIGH_RESPOND_NONE) {
> >> +        return;
> >> +    }
> >> +
> >>      /* If configured to reply to neighbor requests for all VIPs force
> > them
> >>       * all to be considered "reachable".
> >>       */
> >>      if (lb->neigh_mode == LB_NEIGH_RESPOND_ALL) {
> >>          for (size_t i = 0; i < lb->n_vips; i++) {
> >> -            if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
> >> +            if (lb->vips[i].address_family == AF_INET) {
> >>                  sset_add(&od->lb_ips->ips_v4_reachable,
> > lb->vips[i].vip_str);
> >>              } else {
> >>                  sset_add(&od->lb_ips->ips_v6_reachable,
> > lb->vips[i].vip_str);
> >> @@ -4073,8 +4084,9 @@ build_lrouter_lb_reachable_ips(struct
ovn_datapath
> > *od,
> >>      /* Otherwise, a VIP is reachable if there's at least one router
> >>       * subnet that includes it.
> >>       */
> >> +    ovs_assert(lb->neigh_mode == LB_NEIGH_RESPOND_REACHABLE);
> >>      for (size_t i = 0; i < lb->n_vips; i++) {
> >> -        if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) {
> >> +        if (lb->vips[i].address_family == AF_INET) {
> >>              ovs_be32 vip_ip4 =
> > in6_addr_get_mapped_ipv4(&lb->vips[i].vip);
> >>              struct ovn_port *op;
> >>
> >> @@ -5834,16 +5846,16 @@ build_empty_lb_event_flow(struct ovn_lb_vip
> > *lb_vip,
> >>      ds_clear(action);
> >>      ds_clear(match);
> >>
> >> -    bool ipv4 = IN6_IS_ADDR_V4MAPPED(&lb_vip->vip);
> >> +    bool ipv4 = lb_vip->address_family == AF_INET;
> >>
> >>      ds_put_format(match, "ip%s.dst == %s && %s",
> >>                    ipv4 ? "4": "6", lb_vip->vip_str, lb->proto);
> >>
> >>      char *vip = lb_vip->vip_str;
> >> -    if (lb_vip->vip_port) {
> >> -        ds_put_format(match, " && %s.dst == %u", lb->proto,
> > lb_vip->vip_port);
> >> -        vip = xasprintf("%s%s%s:%u", ipv4 ? "" : "[", lb_vip->vip_str,
> >> -                        ipv4 ? "" : "]", lb_vip->vip_port);
> >> +    if (lb_vip->port_str) {
> >> +        ds_put_format(match, " && %s.dst == %s", lb->proto,
> > lb_vip->port_str);
> >> +        vip = xasprintf("%s%s%s:%s", ipv4 ? "" : "[", lb_vip->vip_str,
> >> +                        ipv4 ? "" : "]", lb_vip->port_str);
> >>      }
> >>
> >>      ds_put_format(action,
> >> @@ -5854,7 +5866,7 @@ build_empty_lb_event_flow(struct ovn_lb_vip
*lb_vip,
> >>                    event_to_string(OVN_EVENT_EMPTY_LB_BACKENDS),
> >>                    vip, lb->proto,
> >>                    UUID_ARGS(&lb->nlb->header_.uuid));
> >> -    if (lb_vip->vip_port) {
> >> +    if (lb_vip->port_str) {
> >>          free(vip);
> >>      }
> >>      return true;
> >> @@ -6910,7 +6922,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
> > struct ovn_northd_lb *lb,
> >>          /* Store the original destination IP to be used when
generating
> >>           * hairpin flows.
> >>           */
> >> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >> +        if (lb->vips[i].address_family == AF_INET) {
> >>              ip_match = "ip4";
> >>              ds_put_format(action, REG_ORIG_DIP_IPV4 " = %s; ",
> >>                            lb_vip->vip_str);
> >> @@ -6921,7 +6933,7 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
> > struct ovn_northd_lb *lb,
> >>          }
> >>
> >>          const char *proto = NULL;
> >> -        if (lb_vip->vip_port) {
> >> +        if (lb_vip->port_str) {
> >>              proto = "tcp";
> >>              if (lb->nlb->protocol) {
> >>                  if (!strcmp(lb->nlb->protocol, "udp")) {
> >> @@ -6934,14 +6946,14 @@ build_lb_rules_pre_stateful(struct hmap
*lflows,
> > struct ovn_northd_lb *lb,
> >>              /* Store the original destination port to be used when
> > generating
> >>               * hairpin flows.
> >>               */
> >> -            ds_put_format(action, REG_ORIG_TP_DPORT " = %"PRIu16"; ",
> >> -                          lb_vip->vip_port);
> >> +            ds_put_format(action, REG_ORIG_TP_DPORT " = %s; ",
> >> +                          lb_vip->port_str);
> >>          }
> >>          ds_put_format(action, "%s;", ct_lb_mark ? "ct_lb_mark" :
> > "ct_lb");
> >>
> >>          ds_put_format(match, "%s.dst == %s", ip_match,
lb_vip->vip_str);
> >> -        if (lb_vip->vip_port) {
> >> -            ds_put_format(match, " && %s.dst == %d", proto,
> > lb_vip->vip_port);
> >> +        if (lb_vip->port_str) {
> >> +            ds_put_format(match, " && %s.dst == %s", proto,
> > lb_vip->port_str);
> >>          }
> >>
> >>          struct ovn_lflow *lflow_ref = NULL;
> >> @@ -7192,24 +7204,12 @@ build_lb_rules(struct hmap *lflows, struct
> > ovn_northd_lb *lb, bool ct_lb_mark,
> >>          struct ovn_lb_vip *lb_vip = &lb->vips[i];
> >>          struct ovn_northd_lb_vip *lb_vip_nb = &lb->vips_nb[i];
> >>          const char *ip_match = NULL;
> >> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >> +        if (lb_vip->address_family == AF_INET) {
> >>              ip_match = "ip4";
> >>          } else {
> >>              ip_match = "ip6";
> >>          }
> >>
> >> -        const char *proto = NULL;
> >> -        if (lb_vip->vip_port) {
> >> -            proto = "tcp";
> >> -            if (lb->nlb->protocol) {
> >> -                if (!strcmp(lb->nlb->protocol, "udp")) {
> >> -                    proto = "udp";
> >> -                } else if (!strcmp(lb->nlb->protocol, "sctp")) {
> >> -                    proto = "sctp";
> >> -                }
> >> -            }
> >> -        }
> >> -
> >>          ds_clear(action);
> >>          ds_clear(match);
> >>
> >> @@ -7227,8 +7227,9 @@ build_lb_rules(struct hmap *lflows, struct
> > ovn_northd_lb *lb, bool ct_lb_mark,
> >>          ds_put_format(match, "ct.new && %s.dst == %s", ip_match,
> >>                        lb_vip->vip_str);
> >>          int priority = 110;
> >> -        if (lb_vip->vip_port) {
> >> -            ds_put_format(match, " && %s.dst == %d", proto,
> > lb_vip->vip_port);
> >> +        if (lb_vip->port_str) {
> >> +            ds_put_format(match, " && %s.dst == %s", lb->proto,
> >> +                          lb_vip->port_str);
> >>              priority = 120;
> >>          }
> >>
> >> @@ -10231,7 +10232,7 @@ build_lrouter_nat_flows_for_lb(struct
ovn_lb_vip
> > *lb_vip,
> >>       * of "ct_lb_mark($targets);". The other flow is for ct.est with
> >>       * an action of "next;".
> >>       */
> >> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >> +    if (lb_vip->address_family == AF_INET) {
> >>          ds_put_format(match, "ip4 && "REG_NEXT_HOP_IPV4" == %s",
> >>                        lb_vip->vip_str);
> >>      } else {
> >> @@ -10247,14 +10248,14 @@ build_lrouter_nat_flows_for_lb(struct
> > ovn_lb_vip *lb_vip,
> >>      }
> >>
> >>      int prio = 110;
> >> -    if (lb_vip->vip_port) {
> >> +    if (lb_vip->port_str) {
> >>          prio = 120;
> >>          new_match = xasprintf("ct.new && %s && %s && "
> >> -                              REG_ORIG_TP_DPORT_ROUTER" == %d",
> >> -                              ds_cstr(match), lb->proto,
> > lb_vip->vip_port);
> >> +                              REG_ORIG_TP_DPORT_ROUTER" == %s",
> >> +                              ds_cstr(match), lb->proto,
> > lb_vip->port_str);
> >>          est_match = xasprintf("ct.est && %s && %s && "
> >> -                              REG_ORIG_TP_DPORT_ROUTER" == %d && %s ==
> > 1",
> >> -                              ds_cstr(match), lb->proto,
> > lb_vip->vip_port,
> >> +                              REG_ORIG_TP_DPORT_ROUTER" == %s && %s ==
> > 1",
> >> +                              ds_cstr(match), lb->proto,
> > lb_vip->port_str,
> >>                                ct_natted);
> >>      } else {
> >>          new_match = xasprintf("ct.new && %s", ds_cstr(match));
> >> @@ -10263,7 +10264,7 @@ build_lrouter_nat_flows_for_lb(struct
ovn_lb_vip
> > *lb_vip,
> >>      }
> >>
> >>      const char *ip_match = NULL;
> >> -    if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >> +    if (lb_vip->address_family == AF_INET) {
> >>          ip_match = "ip4";
> >>      } else {
> >>          ip_match = "ip6";
> >> @@ -10281,9 +10282,9 @@ build_lrouter_nat_flows_for_lb(struct
ovn_lb_vip
> > *lb_vip,
> >>          ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
> >>                        backend->ip_str);
> >>
> >> -        if (backend->port) {
> >> -            ds_put_format(&undnat_match, " && %s.src == %d) || ",
> >> -                          lb->proto, backend->port);
> >> +        if (backend->port_str) {
> >> +            ds_put_format(&undnat_match, " && %s.src == %s) || ",
> >> +                          lb->proto, backend->port_str);
> >>          } else {
> >>              ds_put_cstr(&undnat_match, ") || ");
> >>          }
> >> @@ -10296,9 +10297,9 @@ build_lrouter_nat_flows_for_lb(struct
ovn_lb_vip
> > *lb_vip,
> >>      struct ds unsnat_match = DS_EMPTY_INITIALIZER;
> >>      ds_put_format(&unsnat_match, "%s && %s.dst == %s && %s",
> >>                    ip_match, ip_match, lb_vip->vip_str, lb->proto);
> >> -    if (lb_vip->vip_port) {
> >> -        ds_put_format(&unsnat_match, " && %s.dst == %d", lb->proto,
> >> -                      lb_vip->vip_port);
> >> +    if (lb_vip->port_str) {
> >> +        ds_put_format(&unsnat_match, " && %s.dst == %s", lb->proto,
> >> +                      lb_vip->port_str);
> >>      }
> >>
> >>      struct ovn_datapath **gw_router_skip_snat =
> >> @@ -10571,7 +10572,7 @@ build_lrouter_defrag_flows_for_lb(struct
> > ovn_northd_lb *lb,
> >>          ds_clear(&defrag_actions);
> >>          ds_clear(match);
> >>
> >> -        if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> >> +        if (lb_vip->address_family == AF_INET) {
> >>              ds_put_format(match, "ip && ip4.dst == %s",
lb_vip->vip_str);
> >>              ds_put_format(&defrag_actions, REG_NEXT_HOP_IPV4" = %s; ",
> >>                            lb_vip->vip_str);
> >> @@ -10581,7 +10582,7 @@ build_lrouter_defrag_flows_for_lb(struct
> > ovn_northd_lb *lb,
> >>                            lb_vip->vip_str);
> >>          }
> >>
> >> -        if (lb_vip->vip_port) {
> >> +        if (lb_vip->port_str) {
> >>              ds_put_format(match, " && %s", lb->proto);
> >>              prio = 110;
> >>
> >> diff --git a/ovn-nb.xml b/ovn-nb.xml
> >> index 553c0e48c3..8cd2427e8f 100644
> >> --- a/ovn-nb.xml
> >> +++ b/ovn-nb.xml
> >> @@ -1905,8 +1905,66 @@
> >>          is applied reply to ARP/neighbor discovery requests for all
VIPs
> >>          of the load balancer.  If set to <code>reachable</code>, then
> > routers
> >>          on which the load balancer is applied reply to ARP/neighbor
> > discovery
> >> -        requests only for VIPs that are part of a router's subnet.
The
> > default
> >> -        value of this option, if not specified, is
> > <code>reachable</code>.
> >> +        requests only for VIPs that are part of a router's subnet.  If
> > set to
> >> +        <code>none</code>, then routers on which the load balancer is
> > applied
> >> +        never reply to ARP/neighbor discovery requests for any of the
> > load
> >> +        balancer VIPs. Load balancers with
> > <code>options:template=true</code>
> >> +        do not support <code>reachable</code> as a valid mode.  The
> > default
> >> +        value of this option, if not specified, is
> > <code>reachable</code> for
> >> +        regular load balancers and <code>none</code> for template load
> >> +        balancers.
> >> +      </column>
> >> +
> >> +      <column name="options" key="template">
> >> +        <p>
> >> +          Option to be set to <code>true</code>, if the load balancer
is
> > a
> >> +          template.  The load balancer VIPs and backends must be using
> >> +          <ref table="Chassis_Template_Var"/> in their definitions.
> >> +        </p>
> >> +
> >> +        <p>
> >> +          Load balancer template VIP supported formats are:
> >> +        </p>
> >> +        <pre>
> >> +^VIP_VAR[:^PORT_VAR|:port]
> >> +        </pre>
> >> +
> >> +        <p>
> >> +          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are
names
> > of
> >> +        <ref table="Chassis_Template_Var"/> records.
> >
> > In this version, the vars are not names but keys of the "variables"
column.
> >
>
> True, I fixed it.
>
> >> +        </p>
> >> +
> >> +        <p>
> >> +          Note: The VIP and PORT cannot be combined into a single
> > template
> >> +          variable. For example, a <ref table="Chassis_Template_Var"/>
> >> +          variable expanding to <code>10.0.0.1:8080</code> is not
valid
> >> +          if used as VIP.
> >> +        </p>
> >> +
> >> +        <p>
> >> +          Load balancer template backend supported formats are:
> >> +        </p>
> >> +        <pre>
> >> +^BACKEND_VAR1[:^PORT_VAR1|:port],^BACKEND_VAR2[:^PORT_VAR2|:port]
> >> +
> >> +or
> >> +
> >> +^BACKENDS_VAR1,^BACKENDS_VAR2
> >
> > I think here each var means a single backend IP, right? So,
> > s/BACKENDS/BACKEND/g
> >
>
> Not necessarily.  There's actually no restriction.

Sorry I was intended to remove this comment but forgot to do that before
sending.

>
> >> +        </pre>
> >> +        <p>
> >> +          where <code>BACKEND_VAR1</code>, <code>PORT_VAR1</code>,
> >> +          <code>BACKEND_VAR2</code>, <code>PORT_VAR2</code>,
> >> +          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code>
are
> > names
> >
> > Same here, and they are keys instead of "names".
> >
> > Acked-by: Han Zhou <hzhou@ovn.org>
> >
>
> Thanks, Han!  I ended up with the following incremental.  Let me
> know if it looks ok to you and I can fold it in.
>

Yes, looks good to me. Thanks!

Han

> Regards,
> Dumitru
>
> ---
> diff --git a/TODO.rst b/TODO.rst
> index 53cf2870b2..15fd131d39 100644
> --- a/TODO.rst
> +++ b/TODO.rst
> @@ -186,7 +186,4 @@ OVN To-do List
>   * Load Balancer templates
>  -  * Support combining the VIP (or backend) IP and port into a single
> -    template variable.
> -
> -  * Support combining all backends into a single template variable.
> +  * Support combining the VIP IP and port into a single template
variable.
> diff --git a/ovn-nb.xml b/ovn-nb.xml
> index 7ecf2047e7..0edc3da96c 100644
> --- a/ovn-nb.xml
> +++ b/ovn-nb.xml
> @@ -1958,8 +1958,9 @@
>          </pre>
>           <p>
> -          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are names
of
> -        <ref table="Chassis_Template_Var"/> records.
> +          where <code>VIP_VAR</code> and <code>PORT_VAR</code> are keys
of
> +          the <ref table="Chassis_Template_Var"/> <ref
column="variables"/>
> +          records.
>          </p>
>           <p>
> @@ -1982,8 +1983,9 @@ or
>          <p>
>            where <code>BACKEND_VAR1</code>, <code>PORT_VAR1</code>,
>            <code>BACKEND_VAR2</code>, <code>PORT_VAR2</code>,
> -          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
> names
> -          of <ref table="Chassis_Template_Var"/> records.
> +          <code>BACKENDS_VAR1</code> and <code>BACKENDS_VAR2</code> are
> keys
> +          of the <ref table="Chassis_Template_Var"/> <ref
> column="variables"/>
> +          records.
>          </p>
>        </column>
>  diff --git a/tests/ovn.at b/tests/ovn.at
> index bc3a7adfba..f3bd532423 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -33525,9 +33525,11 @@ dnl Create a few LBs that use "uninstantiated"
> templates.
>  check ovn-nbctl --template lb-add lb-test1 "^VIP1:^VPORT1" "^BACKENDS1"
tcp
>  check ovn-nbctl --template lb-add lb-test2 "^VIP2:^VPORT2"
> "^BACKENDS21,^BACKENDS22" tcp
>  check ovn-nbctl --template lb-add lb-test3 "^VIP3:^VPORT3"
> "^BACKENDS31:^BPORT1,^BACKENDS32:^BPORT2" tcp
> +check ovn-nbctl --template lb-add lb-test4 "^VIP4:^VPORT4"
> "^BACKENDS41,^BACKENDS42" tcp
>  check ovn-nbctl ls-lb-add sw lb-test1
>  check ovn-nbctl ls-lb-add sw lb-test2
>  check ovn-nbctl ls-lb-add sw lb-test3
> +check ovn-nbctl ls-lb-add sw lb-test4
>   check ovs-vsctl add-port br-int p1 -- set interface p1
> external_ids:iface-id=lsp1
>  check ovs-vsctl add-port br-int p2 -- set interface p2
> external_ids:iface-id=lsp2
> @@ -33549,7 +33551,10 @@ check ovn-nbctl --wait=hv set
> Chassis_Template_Var hv1 \
>      variables:VIP3='43.43.43.3' variables:VPORT3='4303' \
>      variables:BACKENDS31='85.85.85.31' \
>      variables:BACKENDS32='85.85.85.32' \
> -    variables:BPORT1='8503' variables:BPORT2='8503'
> +    variables:BPORT1='8503' variables:BPORT2='8503' \
> +    variables:VIP4='43.43.43.4' variables:VPORT4='4304' \
> +    variables:BACKENDS41='85.85.85.41:8504,85.85.85.42:8504' \
> +    variables:BACKENDS42='85.85.85.43:8504,85.85.85.44:8504'
>   dnl Ensure the LBs are translated to OpenFlow.
>  as hv1
> @@ -33568,6 +33573,18 @@ AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.31:8503)' -c], [
>  AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.32:8503)' -c], [0], [dnl
>  1
>  ])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.41:8504)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.42:8504)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.43:8504)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=85.85.85.44:8504)' -c], [0], [dnl
> +1
> +])
>   dnl Ensure hairpin flows are correct.
>  as hv1
> @@ -33577,6 +33594,10 @@ AT_CHECK([ovs-ofctl dump-flows br-int | grep
> table=68 | ofctl_strip_all], [0], [
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b02,reg2=0x10ce/0xffff,nw_src=85.85.85.22,nw_dst=85.85.85.22,tp_dst=8502
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.31,nw_dst=85.85.85.31,tp_dst=8503
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b03,reg2=0x10cf/0xffff,nw_src=85.85.85.32,nw_dst=85.85.85.32,tp_dst=8503
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.41,nw_dst=85.85.85.41,tp_dst=8504
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.42,nw_dst=85.85.85.42,tp_dst=8504
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.43,nw_dst=85.85.85.43,tp_dst=8504
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2b2b2b04,reg2=0x10d0/0xffff,nw_src=85.85.85.44,nw_dst=85.85.85.44,tp_dst=8504
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=43.43.43.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>  ])
>   dnl Change Chassis_Template_Var mappings
> @@ -33589,7 +33610,10 @@ check ovn-nbctl --wait=hv set
> Chassis_Template_Var hv1 \
>      variables:VIP3='42.42.42.3' variables:VPORT3='4203' \
>      variables:BACKENDS31='84.84.84.31' \
>      variables:BACKENDS32='84.84.84.32' \
> -    variables:BPORT1='8403' variables:BPORT2='8403'
> +    variables:BPORT1='8403' variables:BPORT2='8403' \
> +    variables:VIP4='42.42.42.4' variables:VPORT4='4204' \
> +    variables:BACKENDS41='84.84.84.41:8404,84.84.84.42:8404' \
> +    variables:BACKENDS42='84.84.84.43:8404,84.84.84.44:8404'
>   dnl Ensure the LBs are translated to OpenFlow.
>  as hv1
> @@ -33608,6 +33632,18 @@ AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.31:8403)' -c], [
>  AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.32:8403)' -c], [0], [dnl
>  1
>  ])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.41:8404)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.42:8404)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.43:8404)' -c], [0], [dnl
> +1
> +])
> +AT_CHECK([ovs-ofctl dump-groups br-int | grep
> 'nat(dst=84.84.84.44:8404)' -c], [0], [dnl
> +1
> +])
>   dnl Ensure hairpin flows are correct.
>  as hv1
> @@ -33617,6 +33653,10 @@ AT_CHECK([ovs-ofctl dump-flows br-int | grep
> table=68 | ofctl_strip_all], [0], [
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a02,reg2=0x106a/0xffff,nw_src=84.84.84.22,nw_dst=84.84.84.22,tp_dst=8402
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.2,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.31,nw_dst=84.84.84.31,tp_dst=8403
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>   table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a03,reg2=0x106b/0xffff,nw_src=84.84.84.32,nw_dst=84.84.84.32,tp_dst=8403
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.3,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.41,nw_dst=84.84.84.41,tp_dst=8404
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.42,nw_dst=84.84.84.42,tp_dst=8404
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.43,nw_dst=84.84.84.43,tp_dst=8404
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
> + table=68,
>
priority=100,ct_mark=0x2/0x2,tcp,reg1=0x2a2a2a04,reg2=0x106c/0xffff,nw_src=84.84.84.44,nw_dst=84.84.84.44,tp_dst=8404
>
actions=load:0x1->NXM_NX_REG10[[7]],learn(table=69,delete_learned,OXM_OF_METADATA[[]],eth_type=0x800,NXM_OF_IP_SRC[[]],ip_dst=42.42.42.4,nw_proto=6,NXM_OF_TCP_SRC[[]]=NXM_OF_TCP_DST[[]],load:0x1->NXM_NX_REG10[[7]])
>  ])
>   dnl Remove Chassis_Template_Variables and check that everything is
> ---
>