diff mbox series

[ovs-dev] ovn: Support a new Logical_Switch_Port.type - 'external'

Message ID 20180910181906.2627-1-nusiddiq@redhat.com
State Changes Requested
Headers show
Series [ovs-dev] ovn: Support a new Logical_Switch_Port.type - 'external' | expand

Commit Message

Numan Siddique Sept. 10, 2018, 6:19 p.m. UTC
From: Numan Siddique <nusiddiq@redhat.com>

In the case of OpenStack + OVN, when the VMs are booted on
hypervisors supporting SR-IOV nics, there are no OVS ports
for these VMs. When these VMs sends DHCPv4, DHPCv6 or IPv6
Router Solicitation requests, the local ovn-controller cannot
reply to these packets. OpenStack Neutron dhcp agent service
needs to be run to serve these requests.

With the new logical port type - 'external', OVN itself can
handle these requests avoiding the need to deploy any external
services like neutron dhcp agent.

To make use of this feature, CMS has to
 - create a logical port for such VMs
 - set the type to 'external'
 - set requested-chassis="<chassis-name>" in the options column.
 - create a localnet port for the logical switch
 - configure the ovn-bridge-mappings option in the OVS db.

When the ovn-controller running in that 'chassis', detects the
Port_Binding row, it adds the necessary DHCPv4/v6 OF flows. Since
the packet enters the logical switch pipeline via the localnet port,
the inport register (reg14) is set to the tunnel key of localnet
port in the match conditions.

In case the chassis goes down for some reason, it is the responsibility
of CMS to change the 'requested-chassis' option to some other active
chassis, so that it can serve these requests.

Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
---
 ovn/controller/binding.c        |  13 +-
 ovn/controller/lflow.c          |  40 ++-
 ovn/controller/lflow.h          |   2 +
 ovn/controller/lport.c          |  26 ++
 ovn/controller/lport.h          |   5 +
 ovn/controller/ovn-controller.c |   6 +
 ovn/lib/ovn-util.c              |   1 +
 ovn/northd/ovn-northd.c         |  23 +-
 ovn/ovn-nb.xml                  |  10 +
 tests/ovn.at                    | 465 ++++++++++++++++++++++++++++++++
 10 files changed, 581 insertions(+), 10 deletions(-)

Comments

Ben Pfaff Sept. 18, 2018, 8:19 a.m. UTC | #1
On Mon, Sep 10, 2018 at 11:49:06PM +0530, nusiddiq@redhat.com wrote:
> From: Numan Siddique <nusiddiq@redhat.com>
> 
> In the case of OpenStack + OVN, when the VMs are booted on
> hypervisors supporting SR-IOV nics, there are no OVS ports
> for these VMs. When these VMs sends DHCPv4, DHPCv6 or IPv6
> Router Solicitation requests, the local ovn-controller cannot
> reply to these packets. OpenStack Neutron dhcp agent service
> needs to be run to serve these requests.
> 
> With the new logical port type - 'external', OVN itself can
> handle these requests avoiding the need to deploy any external
> services like neutron dhcp agent.
> 
> To make use of this feature, CMS has to
>  - create a logical port for such VMs
>  - set the type to 'external'
>  - set requested-chassis="<chassis-name>" in the options column.
>  - create a localnet port for the logical switch
>  - configure the ovn-bridge-mappings option in the OVS db.
> 
> When the ovn-controller running in that 'chassis', detects the
> Port_Binding row, it adds the necessary DHCPv4/v6 OF flows. Since
> the packet enters the logical switch pipeline via the localnet port,
> the inport register (reg14) is set to the tunnel key of localnet
> port in the match conditions.
> 
> In case the chassis goes down for some reason, it is the responsibility
> of CMS to change the 'requested-chassis' option to some other active
> chassis, so that it can serve these requests.
> 
> Signed-off-by: Numan Siddique <nusiddiq@redhat.com>

Thank you for working on OVN.

This is a specialized use case, so documentation is important.  The
commit message above gives a lot of context for the feature.  The
explanation that it's related to SR-IOV and the list of steps are both
valuable.  I think that it would be helpful to put that information into
the OVN documentation, too.  It might be appropriate to put something
about SR-IOV into ovn-architecture.

I don't yet understand the connection to localnet.  Can you explain how
traffic between the SR-IOV NIC gets to and from the localnet port?

Thanks,

Ben.
Numan Siddique Sept. 18, 2018, 6:19 p.m. UTC | #2
On Tue, Sep 18, 2018 at 1:49 PM Ben Pfaff <blp@ovn.org> wrote:

> On Mon, Sep 10, 2018 at 11:49:06PM +0530, nusiddiq@redhat.com wrote:
> > From: Numan Siddique <nusiddiq@redhat.com>
> >
> > In the case of OpenStack + OVN, when the VMs are booted on
> > hypervisors supporting SR-IOV nics, there are no OVS ports
> > for these VMs. When these VMs sends DHCPv4, DHPCv6 or IPv6
> > Router Solicitation requests, the local ovn-controller cannot
> > reply to these packets. OpenStack Neutron dhcp agent service
> > needs to be run to serve these requests.
> >
> > With the new logical port type - 'external', OVN itself can
> > handle these requests avoiding the need to deploy any external
> > services like neutron dhcp agent.
> >
> > To make use of this feature, CMS has to
> >  - create a logical port for such VMs
> >  - set the type to 'external'
> >  - set requested-chassis="<chassis-name>" in the options column.
> >  - create a localnet port for the logical switch
> >  - configure the ovn-bridge-mappings option in the OVS db.
> >
> > When the ovn-controller running in that 'chassis', detects the
> > Port_Binding row, it adds the necessary DHCPv4/v6 OF flows. Since
> > the packet enters the logical switch pipeline via the localnet port,
> > the inport register (reg14) is set to the tunnel key of localnet
> > port in the match conditions.
> >
> > In case the chassis goes down for some reason, it is the responsibility
> > of CMS to change the 'requested-chassis' option to some other active
> > chassis, so that it can serve these requests.
> >
> > Signed-off-by: Numan Siddique <nusiddiq@redhat.com>
>
> Thank you for working on OVN.
>
> This is a specialized use case, so documentation is important.  The
> commit message above gives a lot of context for the feature.  The
> explanation that it's related to SR-IOV and the list of steps are both
> valuable.  I think that it would be helpful to put that information into
> the OVN documentation, too.  It might be appropriate to put something
> about SR-IOV into ovn-architecture.
>
>
Thanks for the review . Sure. I will update the documentation.

I don't yet understand the connection to localnet.  Can you explain how
> traffic between the SR-IOV NIC gets to and from the localnet port?
>
>
Lets say we have two hosts chassis C1 and C2 and C1 and C2 are
connected to  the same switch. Suppose C1 has SRIOV configured. When
a VM (booted on this chassis), sends a DHCP packet, it  doesn't go via the
host kernel stack. Instead it is handled by the SRIOV nic directly. So the
packet doesn't go out of localnet port on C1. DHCP packet will be received
on C2. If C2 is configured with ovn-bridge-mappings properly (lets say
public:br-ex and br-ex has the physical nic added to it), the DHCP packet
will come from br-ex to br-int via the localnet patch port.
With this propose solution, the ovn-controller will respond to this DHCP
request and replies to back via the same localnet port. And the packet will
reach the C1's SRIOV nic and it will deliver the response packet to the VM.

Since the SRIOV VM traffic is not handled by host, it is expected that
provider networks (vlan or flat) are used. In the case of OpenStack + OVN,
networking-ovn creates a localnet port for the provider networks.

In the case of OpenStack with neutron reference implementation,
neutron dhcp agent will be running on controller nodes which handles
these DHCP requests. So with Openstack + OVN, most likely the
ovn-controller's running on these controller nodes will be handling these
DHCP requests.

Along with SRIOV, there is another usecase. In the case of OpenStack,
there is an ironic service which is used to provision a baremetal server
for tenants (instead of VMs). When a baremetal server sends a DHCP
request we want to handle these requests by one of the ovn-controller.

I will document both these use cases.

Thanks
Numan



> Thanks,
>
> Ben.
>
diff mbox series

Patch

diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index 021ecddcf..13f5ca691 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -471,13 +471,24 @@  consider_local_datapath(struct ovsdb_idl_txn *ovnsb_idl_txn,
          * for them. */
         sset_add(local_lports, binding_rec->logical_port);
         our_chassis = false;
+    } else if (!strcmp(binding_rec->type, "external")) {
+        const char *chassis_id = smap_get(&binding_rec->options,
+                                          "requested-chassis");
+        our_chassis = chassis_id && !strcmp(chassis_id, chassis_rec->name);
+        if (our_chassis) {
+            add_local_datapath(sbrec_datapath_binding_by_key,
+                               sbrec_port_binding_by_datapath,
+                               sbrec_port_binding_by_name,
+                               binding_rec->datapath, true, local_datapaths);
+        }
     }
 
     if (our_chassis
         || !strcmp(binding_rec->type, "patch")
         || !strcmp(binding_rec->type, "localport")
         || !strcmp(binding_rec->type, "vtep")
-        || !strcmp(binding_rec->type, "localnet")) {
+        || !strcmp(binding_rec->type, "localnet")
+        || !strcmp(binding_rec->type, "external")) {
         update_local_lport_ids(local_lport_ids, binding_rec);
     }
 
diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index 8db81927e..96b5f7e05 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -52,7 +52,10 @@  lflow_init(void)
 struct lookup_port_aux {
     struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath;
     struct ovsdb_idl_index *sbrec_port_binding_by_name;
+    struct ovsdb_idl_index *sbrec_port_binding_by_type;
+    struct ovsdb_idl_index *sbrec_datapath_binding_by_key;
     const struct sbrec_datapath_binding *dp;
+    const struct sbrec_chassis *chassis;
 };
 
 struct condition_aux {
@@ -66,6 +69,8 @@  static void consider_logical_flow(
     struct ovsdb_idl_index *sbrec_chassis_by_name,
     struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath,
     struct ovsdb_idl_index *sbrec_port_binding_by_name,
+    struct ovsdb_idl_index *sbrec_port_binding_by_type,
+    struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
     const struct sbrec_logical_flow *,
     const struct hmap *local_datapaths,
     const struct sbrec_chassis *,
@@ -89,8 +94,23 @@  lookup_port_cb(const void *aux_, const char *port_name, unsigned int *portp)
     const struct sbrec_port_binding *pb
         = lport_lookup_by_name(aux->sbrec_port_binding_by_name, port_name);
     if (pb && pb->datapath == aux->dp) {
-        *portp = pb->tunnel_key;
-        return true;
+        if (strcmp(pb->type, "external")) {
+            *portp = pb->tunnel_key;
+            return true;
+        }
+        const char *chassis_id = smap_get(&pb->options,
+                                          "requested-chassis");
+        if (chassis_id && !strcmp(chassis_id, aux->chassis->name)) {
+            const struct sbrec_port_binding *localnet_pb
+                = lport_lookup_by_type(aux->sbrec_datapath_binding_by_key,
+                                       aux->sbrec_port_binding_by_type,
+                                       aux->dp->tunnel_key, "localnet");
+            if (localnet_pb) {
+                *portp = localnet_pb->tunnel_key;
+                return true;
+            }
+        }
+        return false;
     }
 
     const struct sbrec_multicast_group *mg = mcgroup_lookup_by_dp_name(
@@ -144,6 +164,8 @@  add_logical_flows(
     struct ovsdb_idl_index *sbrec_chassis_by_name,
     struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath,
     struct ovsdb_idl_index *sbrec_port_binding_by_name,
+    struct ovsdb_idl_index *sbrec_port_binding_by_type,
+    struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
     const struct sbrec_dhcp_options_table *dhcp_options_table,
     const struct sbrec_dhcpv6_options_table *dhcpv6_options_table,
     const struct sbrec_logical_flow_table *logical_flow_table,
@@ -183,6 +205,8 @@  add_logical_flows(
         consider_logical_flow(sbrec_chassis_by_name,
                               sbrec_multicast_group_by_name_datapath,
                               sbrec_port_binding_by_name,
+                              sbrec_port_binding_by_type,
+                              sbrec_datapath_binding_by_key,
                               lflow, local_datapaths,
                               chassis, &dhcp_opts, &dhcpv6_opts, &nd_ra_opts,
                               addr_sets, port_groups, active_tunnels,
@@ -200,6 +224,8 @@  consider_logical_flow(
     struct ovsdb_idl_index *sbrec_chassis_by_name,
     struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath,
     struct ovsdb_idl_index *sbrec_port_binding_by_name,
+    struct ovsdb_idl_index *sbrec_port_binding_by_type,
+    struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
     const struct sbrec_logical_flow *lflow,
     const struct hmap *local_datapaths,
     const struct sbrec_chassis *chassis,
@@ -292,7 +318,10 @@  consider_logical_flow(
         .sbrec_multicast_group_by_name_datapath
             = sbrec_multicast_group_by_name_datapath,
         .sbrec_port_binding_by_name = sbrec_port_binding_by_name,
-        .dp = lflow->logical_datapath
+        .sbrec_port_binding_by_type = sbrec_port_binding_by_type,
+        .sbrec_datapath_binding_by_key = sbrec_datapath_binding_by_key,
+        .dp = lflow->logical_datapath,
+        .chassis = chassis
     };
     struct condition_aux cond_aux = {
         .sbrec_chassis_by_name = sbrec_chassis_by_name,
@@ -463,6 +492,8 @@  void
 lflow_run(struct ovsdb_idl_index *sbrec_chassis_by_name,
           struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath,
           struct ovsdb_idl_index *sbrec_port_binding_by_name,
+          struct ovsdb_idl_index *sbrec_port_binding_by_type,
+          struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
           const struct sbrec_dhcp_options_table *dhcp_options_table,
           const struct sbrec_dhcpv6_options_table *dhcpv6_options_table,
           const struct sbrec_logical_flow_table *logical_flow_table,
@@ -481,7 +512,8 @@  lflow_run(struct ovsdb_idl_index *sbrec_chassis_by_name,
 
     add_logical_flows(sbrec_chassis_by_name,
                       sbrec_multicast_group_by_name_datapath,
-                      sbrec_port_binding_by_name, dhcp_options_table,
+                      sbrec_port_binding_by_name, sbrec_port_binding_by_type,
+                      sbrec_datapath_binding_by_key, dhcp_options_table,
                       dhcpv6_options_table, logical_flow_table,
                       local_datapaths, chassis, addr_sets, port_groups,
                       active_tunnels, local_lport_ids, flow_table, group_table,
diff --git a/ovn/controller/lflow.h b/ovn/controller/lflow.h
index d19338140..b2911e0eb 100644
--- a/ovn/controller/lflow.h
+++ b/ovn/controller/lflow.h
@@ -68,6 +68,8 @@  void lflow_init(void);
 void lflow_run(struct ovsdb_idl_index *sbrec_chassis_by_name,
                struct ovsdb_idl_index *sbrec_multicast_group_by_name_datapath,
                struct ovsdb_idl_index *sbrec_port_binding_by_name,
+               struct ovsdb_idl_index *sbrec_port_binding_by_type,
+               struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
                const struct sbrec_dhcp_options_table *,
                const struct sbrec_dhcpv6_options_table *,
                const struct sbrec_logical_flow_table *,
diff --git a/ovn/controller/lport.c b/ovn/controller/lport.c
index cc5c5fbb2..9c827d9b0 100644
--- a/ovn/controller/lport.c
+++ b/ovn/controller/lport.c
@@ -64,6 +64,32 @@  lport_lookup_by_key(struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
     return retval;
 }
 
+const struct sbrec_port_binding *
+lport_lookup_by_type(struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
+                     struct ovsdb_idl_index *sbrec_port_binding_by_type,
+                     uint64_t dp_key, const char *port_type)
+{
+    /* Lookup datapath corresponding to dp_key. */
+    const struct sbrec_datapath_binding *db = datapath_lookup_by_key(
+        sbrec_datapath_binding_by_key, dp_key);
+    if (!db) {
+        return NULL;
+    }
+
+    /* Build key for an indexed lookup. */
+    struct sbrec_port_binding *pb = sbrec_port_binding_index_init_row(
+            sbrec_port_binding_by_type);
+    sbrec_port_binding_index_set_datapath(pb, db);
+    sbrec_port_binding_index_set_type(pb, port_type);
+
+    const struct sbrec_port_binding *retval = sbrec_port_binding_index_find(
+            sbrec_port_binding_by_type, pb);
+
+    sbrec_port_binding_index_destroy_row(pb);
+
+    return retval;
+}
+
 const struct sbrec_datapath_binding *
 datapath_lookup_by_key(struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
                        uint64_t dp_key)
diff --git a/ovn/controller/lport.h b/ovn/controller/lport.h
index 7dcd5bee0..2d49792f6 100644
--- a/ovn/controller/lport.h
+++ b/ovn/controller/lport.h
@@ -42,6 +42,11 @@  const struct sbrec_port_binding *lport_lookup_by_key(
     struct ovsdb_idl_index *sbrec_port_binding_by_key,
     uint64_t dp_key, uint64_t port_key);
 
+const struct sbrec_port_binding *lport_lookup_by_type(
+    struct ovsdb_idl_index *sbrec_datapath_binding_by_key,
+    struct ovsdb_idl_index *sbrec_port_binding_by_type,
+    uint64_t dp_key, const char *port_type);
+
 const struct sbrec_datapath_binding *datapath_lookup_by_key(
     struct ovsdb_idl_index *sbrec_datapath_binding_by_key, uint64_t dp_key);
 
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 85921a03a..bdbb32448 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -148,6 +148,7 @@  update_sb_monitors(struct ovsdb_idl *ovnsb_idl,
      * ports that have a Gateway_Chassis that point's to our own
      * chassis */
     sbrec_port_binding_add_clause_type(&pb, OVSDB_F_EQ, "chassisredirect");
+    sbrec_port_binding_add_clause_type(&pb, OVSDB_F_EQ, "external");
     if (chassis) {
         /* This should be mostly redundant with the other clauses for port
          * bindings, but it allows us to catch any ports that are assigned to
@@ -622,6 +623,9 @@  main(int argc, char *argv[])
     struct ovsdb_idl_index *sbrec_port_binding_by_datapath
         = ovsdb_idl_index_create1(ovnsb_idl_loop.idl,
                                   &sbrec_port_binding_col_datapath);
+    struct ovsdb_idl_index *sbrec_port_binding_by_type
+        = ovsdb_idl_index_create1(ovnsb_idl_loop.idl,
+                                  &sbrec_port_binding_col_type);
     struct ovsdb_idl_index *sbrec_datapath_binding_by_key
         = ovsdb_idl_index_create1(ovnsb_idl_loop.idl,
                                   &sbrec_datapath_binding_col_tunnel_key);
@@ -755,6 +759,8 @@  main(int argc, char *argv[])
                     lflow_run(sbrec_chassis_by_name,
                               sbrec_multicast_group_by_name_datapath,
                               sbrec_port_binding_by_name,
+                              sbrec_port_binding_by_type,
+                              sbrec_datapath_binding_by_key,
                               sbrec_dhcp_options_table_get(ovnsb_idl_loop.idl),
                               sbrec_dhcpv6_options_table_get(ovnsb_idl_loop.idl),
                               sbrec_logical_flow_table_get(ovnsb_idl_loop.idl),
diff --git a/ovn/lib/ovn-util.c b/ovn/lib/ovn-util.c
index e9464e926..0e4439c5d 100644
--- a/ovn/lib/ovn-util.c
+++ b/ovn/lib/ovn-util.c
@@ -311,6 +311,7 @@  static const char *OVN_NB_LSP_TYPES[] = {
     "localport",
     "router",
     "vtep",
+    "external",
 };
 
 bool
diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
index 31ea5f410..90b887954 100644
--- a/ovn/northd/ovn-northd.c
+++ b/ovn/northd/ovn-northd.c
@@ -2870,6 +2870,15 @@  lsp_is_up(const struct nbrec_logical_switch_port *lsp)
     return !lsp->up || *lsp->up;
 }
 
+static bool
+lsp_is_external(const struct nbrec_logical_switch_port *lsp)
+{
+    if (lsp->type && lsp->type[0] && !strcmp(lsp->type, "external")) {
+        return true;
+    }
+    return false;
+}
+
 static bool
 build_dhcpv4_action(struct ovn_port *op, ovs_be32 offer_ip,
                     struct ds *options_action, struct ds *response_action,
@@ -4045,9 +4054,9 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
             continue;
         }
 
-        if (!lsp_is_enabled(op->nbsp)) {
+        if (!lsp_is_enabled(op->nbsp) || lsp_is_external(op->nbsp)) {
             /* Drop packets from disabled logical ports (since logical flow
-             * tables are default-drop). */
+             * tables are default-drop) or from 'external' ports. */
             continue;
         }
 
@@ -4113,7 +4122,7 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
          *  - port type is localport
          */
         if (!lsp_is_up(op->nbsp) && strcmp(op->nbsp->type, "router") &&
-            strcmp(op->nbsp->type, "localport")) {
+            strcmp(op->nbsp->type, "localport") && lsp_is_external(op->nbsp)) {
             continue;
         }
 
@@ -4363,7 +4372,7 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
             continue;
         }
 
-        if (lsp_is_enabled(op->nbsp)) {
+        if (lsp_is_enabled(op->nbsp) && !lsp_is_external(op->nbsp)) {
             ovn_multicast_add(mcgroups, &mc_flood, op);
         }
     }
@@ -4378,7 +4387,7 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
 
     /* Ingress table 16: Destination lookup, unicast handling (priority 50), */
     HMAP_FOR_EACH (op, key_node, ports) {
-        if (!op->nbsp) {
+        if (!op->nbsp || lsp_is_external(op->nbsp)) {
             continue;
         }
 
@@ -4515,6 +4524,10 @@  build_lswitch_flows(struct hmap *datapaths, struct hmap *ports,
             continue;
         }
 
+        if (lsp_is_external(op->nbsp)) {
+            continue;
+        }
+
         ds_clear(&match);
         ds_put_format(&match, "outport == %s", op->json_key);
         if (lsp_is_enabled(op->nbsp)) {
diff --git a/ovn/ovn-nb.xml b/ovn/ovn-nb.xml
index 441a2deae..56cbba7e6 100644
--- a/ovn/ovn-nb.xml
+++ b/ovn/ovn-nb.xml
@@ -302,6 +302,16 @@ 
           <dd>
             A port to a logical switch on a VTEP gateway.
           </dd>
+
+          <dt><code>external</code></dt>
+          <dd>
+            Represents a logical port which is external and not bound
+            by <code>ovn-controller</code>. <code>OVN</code> will never receive
+            any traffic from this port or send any traffic to this port.
+            This can be used to support <code>OVN's</code> native services -
+            like DHCP for these external ports provided the logical switch has
+            a localnet port.
+          </dd>
         </dl>
       </column>
     </group>
diff --git a/tests/ovn.at b/tests/ovn.at
index 769e09f81..6a607c750 100644
--- a/tests/ovn.at
+++ b/tests/ovn.at
@@ -11103,6 +11103,471 @@  as hv2 start_daemon ovn-controller
 OVN_CLEANUP([hv1],[hv2])
 AT_CLEANUP
 
+AT_SETUP([ovn -- external logical port])
+AT_SKIP_IF([test $HAVE_PYTHON = no])
+ovn_start
+
+net_add n1
+sim_add hv1
+sim_add hv2
+
+ovn-nbctl ls-add ls1
+ovn-nbctl lsp-add ls1 ls1-lp1 \
+-- lsp-set-addresses ls1-lp1 "f0:00:00:00:00:01 10.0.0.4 ae70::4"
+
+# Add a couple of external logical port
+ovn-nbctl lsp-add ls1 ls1-lp_ext1 \
+-- lsp-set-addresses ls1-lp_ext1 "f0:00:00:00:00:03 10.0.0.6 ae70::6"
+ovn-nbctl lsp-set-port-security ls1-lp_ext1 \
+"f0:00:00:00:00:03 10.0.0.6 ae70::6"
+ovn-nbctl lsp-set-type ls1-lp_ext1 external
+
+ovn-nbctl lsp-add ls1 ls1-lp_ext2 \
+-- lsp-set-addresses ls1-lp_ext2 "f0:00:00:00:00:04 10.0.0.7 ae70::7"
+ovn-nbctl lsp-set-port-security ls1-lp_ext2 \
+"f0:00:00:00:00:04 10.0.0.7 ae70::8"
+ovn-nbctl lsp-set-type ls1-lp_ext2 external
+
+d1="$(ovn-nbctl create DHCP_Options cidr=10.0.0.0/24 \
+options="\"server_id\"=\"10.0.0.1\" \"server_mac\"=\"ff:10:00:00:00:01\" \
+\"lease_time\"=\"3600\" \"router\"=\"10.0.0.1\"")"
+
+d2="$(ovn-nbctl create DHCP_Options cidr="ae70\:\:/64" \
+options="\"server_id\"=\"00:00:00:10:00:01\"")"
+
+ovn-nbctl lsp-set-dhcpv4-options ls1-lp1 ${d1}
+ovn-nbctl lsp-set-dhcpv4-options ls1-lp_ext1 ${d1}
+ovn-nbctl lsp-set-dhcpv4-options ls1-lp_ext2 ${d1}
+
+ovn-nbctl lsp-set-dhcpv6-options ls1-lp1 ${d2}
+ovn-nbctl lsp-set-dhcpv6-options ls1-lp_ext1 ${d2}
+ovn-nbctl lsp-set-dhcpv6-options ls1-lp_ext2 ${d2}
+
+as hv1
+ovs-vsctl add-br br-phys
+ovn_attach n1 br-phys 192.168.0.1
+ovs-vsctl -- add-port br-int hv1-vif1 -- \
+    set interface hv1-vif1 external-ids:iface-id=ls1-lp1 \
+    options:tx_pcap=hv1/vif1-tx.pcap \
+    options:rxq_pcap=hv1/vif1-rx.pcap \
+    ofport-request=1
+ovs-vsctl -- add-port br-phys hv1-ext1 -- \
+    set interface hv1-ext1 options:tx_pcap=hv1/ext1-tx.pcap \
+    options:rxq_pcap=hv1/ext1-rx.pcap \
+    ofport-request=2
+ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
+
+as hv2
+ovs-vsctl add-br br-phys
+ovn_attach n1 br-phys 192.168.0.2
+ovs-vsctl -- add-port br-phys hv2-ext -- \
+    set interface hv2-ext options:tx_pcap=hv2/ext2-tx.pcap \
+    options:rxq_pcap=hv2/ext2-rx.pcap \
+    ofport-request=2
+ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
+
+ovn-sbctl dump-flows ls1
+AT_CHECK([ovn-sbctl dump-flows ls1 | grep "offerip = 10.0.0.6" | \
+wc -l], [0], [2
+])
+
+# No DHCPv4/v6 flows for the external port - ls1-lp_ext1 - 10.0.0.6 in hv1 and
+# hv2 as requested-chassis option is not set and no localnet port added to ls1.
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | wc -l], [0], [0
+])
+
+hv1_uuid=$(ovn-sbctl list chassis hv1 | grep uuid | awk '{print $3}')
+
+# The port_binding row for ls1-lp_ext1 should have empty chassis
+chassis=$(ovn-sbctl list port_binding ls1-lp_ext1 | grep -v gateway | \
+grep -v requested | grep chassis | awk '{print $3}')
+
+AT_CHECK([test $chassis == "[[]]"], [0], [])
+
+# Set the requested-chassis option for ls1-lp_ext1
+ovn-nbctl --wait=hv lsp-set-options ls1-lp_ext1 requested-chassis=hv1
+
+# No DHCPv4/v6 flows for the external port - ls1-lp_ext1 - 10.0.0.6 in hv1 and hv2
+# as no localnet port added to ls1 yet.
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | wc -l], [0], [0
+])
+
+# Add the localnet port to the logical switch ls1
+ovn-nbctl lsp-add ls1 ln-public
+ovn-nbctl lsp-set-addresses ln-public unknown
+ovn-nbctl lsp-set-type ln-public localnet
+ovn-nbctl --wait=hv lsp-set-options ln-public network_name=phys
+
+ln_public_key=$(ovn-sbctl list port_binding ln-public | grep  tunnel_key | \
+awk '{print $3}')
+
+# The ls1-lp_ext1 should be bound to hv1
+chassis=$(ovn-sbctl list port_binding ls1-lp_ext1 | grep -v gateway | \
+grep -v requested | grep chassis | awk '{print $3}')
+AT_CHECK([test $chassis == "$hv1_uuid"], [0], [])
+
+# There should be DHCPv4/v6 OF flows for the ls1-lp_ext1 port in hv1
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | grep reg14=0x$ln_public_key | \
+wc -l], [0], [3
+])
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | \
+grep reg14=0x$ln_public_key | wc -l], [0], [1
+])
+
+# There should ne no DHCPv4/v6 flows for ls1-lp_ext1 on hv2
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | wc -l], [0], [0
+])
+
+# No DHCPv4/v6 flows for the external port - ls1-lp_ext2 - 10.0.0.7 in hv1 and
+# hv2 as requested-chassis option is not set.
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.07" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.07" | wc -l], [0], [0
+])
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.07" | wc -l], [0], [0
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.07" | wc -l], [0], [0
+])
+
+as hv1
+ovs-vsctl show
+
+# This shell function sends a DHCP request packet
+# test_dhcp INPORT SRC_MAC DHCP_TYPE OFFER_IP ...
+test_dhcp() {
+    local inport=$1 src_mac=$2 dhcp_type=$3 offer_ip=$4 use_ip=$5
+    shift; shift; shift; shift; shift;
+    if test $use_ip != 0; then
+        src_ip=$1
+        dst_ip=$2
+        shift; shift;
+    else
+        src_ip=`ip_to_hex 0 0 0 0`
+        dst_ip=`ip_to_hex 255 255 255 255`
+    fi
+    local request=ffffffffffff${src_mac}0800451001100000000080110000${src_ip}${dst_ip}
+    # udp header and dhcp header
+    request=${request}0044004300fc0000
+    request=${request}010106006359aa760000000000000000000000000000000000000000${src_mac}
+    # client hardware padding
+    request=${request}00000000000000000000
+    # server hostname
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    # boot file name
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    request=${request}0000000000000000000000000000000000000000000000000000000000000000
+    # dhcp magic cookie
+    request=${request}63825363
+    # dhcp message type
+    request=${request}3501${dhcp_type}ff
+
+    local srv_mac=$1 srv_ip=$2 expected_dhcp_opts=$3
+    local req_pkt_in_expected=$4
+    # total IP length will be the IP length of the request packet
+    # (which is 272 in our case) + 8 (padding bytes) + (expected_dhcp_opts / 2)
+    ip_len=`expr 280 + ${#expected_dhcp_opts} / 2`
+    udp_len=`expr $ip_len - 20`
+    ip_len=$(printf "%x" $ip_len)
+    udp_len=$(printf "%x" $udp_len)
+    # $ip_len var will be in 3 digits i.e 134. So adding a '0' before $ip_len
+    local reply=${src_mac}${srv_mac}080045100${ip_len}000000008011XXXX${srv_ip}${offer_ip}
+    # udp header and dhcp header.
+    # $udp_len var will be in 3 digits. So adding a '0' before $udp_len
+    reply=${reply}004300440${udp_len}0000020106006359aa760000000000000000
+    # your ip address
+    reply=${reply}${offer_ip}
+    # next server ip address, relay agent ip address, client mac address
+    reply=${reply}0000000000000000${src_mac}
+    # client hardware padding
+    reply=${reply}00000000000000000000
+    # server hostname
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    # boot file name
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    reply=${reply}0000000000000000000000000000000000000000000000000000000000000000
+    # dhcp magic cookie
+    reply=${reply}63825363
+    # dhcp message type
+    local dhcp_reply_type=02
+    if test $dhcp_type = 03; then
+        dhcp_reply_type=05
+    fi
+    reply=${reply}3501${dhcp_reply_type}${expected_dhcp_opts}00000000ff00000000
+    if test $req_pkt_in_expected = 1; then
+        echo $request > ext${inport}_v4.expected
+    fi
+    echo $reply >> ext1_v4.expected
+
+    as hv1 ovs-appctl netdev-dummy/receive hv${inport}-ext${inport} $request
+}
+
+
+trim_zeros() {
+    sed 's/\(00\)\{1,\}$//'
+}
+
+# This shell function sends a DHCPv6 request packet
+# test_dhcpv6 INPORT SRC_MAC SRC_LLA DHCPv6_MSG_TYPE OFFER_IP OUTPORT...
+# The OUTPORTs (zero or more) list the VIFs on which the original DHCPv6
+# packet should be received twice (one from ovn-controller and the other
+# from the "ovs-ofctl monitor br-int resume"
+test_dhcpv6() {
+    local inport=$1 src_mac=$2 src_lla=$3 msg_code=$4 offer_ip=$5
+    local req_pkt_in_expected=$6
+    local request=ffffffffffff${src_mac}86dd00000000002a1101${src_lla}
+    # dst ip ff02::1:2
+    request=${request}ff020000000000000000000000010002
+    # udp header and dhcpv6 header
+    request=${request}02220223002affff${msg_code}010203
+    # Client identifier
+    request=${request}0001000a00030001${src_mac}
+    # IA-NA (Identity Association for Non Temporary Address)
+    request=${request}0003000c0102030400000e1000001518
+    shift; shift; shift; shift; shift;
+
+    local server_mac=000000100001
+    local server_lla=fe80000000000000020000fffe100001
+    local reply_code=07
+    if test $msg_code = 01; then
+        reply_code=02
+    fi
+    local msg_len=54
+    if test $offer_ip = 1; then
+        msg_len=28
+    fi
+    local reply=${src_mac}${server_mac}86dd0000000000${msg_len}1101
+    reply=${reply}${server_lla}${src_lla}
+
+    # udp header and dhcpv6 header
+    reply=${reply}0223022200${msg_len}ffff${reply_code}010203
+    # Client identifier
+    reply=${reply}0001000a00030001${src_mac}
+    # IA-NA
+    if test $offer_ip != 1; then
+        reply=${reply}0003002801020304ffffffffffffffff00050018${offer_ip}
+        reply=${reply}ffffffffffffffff
+    fi
+    # Server identifier
+    reply=${reply}0002000a00030001${server_mac}
+
+    echo $reply | trim_zeros >> ext${inport}_v6.expected
+    # The inport also receives the request packet since it is connected
+    # to the br-phys.
+    echo $request >> ext${inport}_v6.expected
+
+    as hv1 ovs-appctl netdev-dummy/receive hv${inport}-ext${inport} $request
+}
+
+reset_pcap_file() {
+    local iface=$1
+    local pcap_file=$2
+    ovs-vsctl -- set Interface $iface options:tx_pcap=dummy-tx.pcap \
+options:rxq_pcap=dummy-rx.pcap
+    rm -f ${pcap_file}*.pcap
+    ovs-vsctl -- set Interface $iface options:tx_pcap=${pcap_file}-tx.pcap \
+options:rxq_pcap=${pcap_file}-rx.pcap
+}
+
+ip_to_hex() {
+    printf "%02x%02x%02x%02x" "$@"
+}
+
+AT_CAPTURE_FILE([ofctl_monitor0_hv1.log])
+as hv1 ovs-ofctl monitor br-int resume --detach --no-chdir \
+--pidfile=ovs-ofctl0.pid 2> ofctl_monitor0_hv1.log
+
+AT_CAPTURE_FILE([ofctl_monitor0_hv2.log])
+as hv2 ovs-ofctl monitor br-int resume --detach --no-chdir \
+--pidfile=ovs-ofctl0.pid 2> ofctl_monitor0_hv2.log
+
+# Send DHCPDISCOVER.
+offer_ip=`ip_to_hex 10 0 0 6`
+server_ip=`ip_to_hex 10 0 0 1`
+server_mac=ff1000000001
+expected_dhcp_opts=330400000e100104ffffff0003040a00000136040a000001
+test_dhcp 1 f00000000003 01 $offer_ip 0 $server_mac $server_ip \
+$expected_dhcp_opts
+
+# NXT_RESUMEs should be 1 in hv1.
+OVS_WAIT_UNTIL([test 1 = `cat ofctl_monitor0_hv1.log | grep -c NXT_RESUME`])
+
+# NXT_RESUMEs should be 0 in hv2.
+OVS_WAIT_UNTIL([test 0 = `cat ofctl_monitor0_hv2.log | grep -c NXT_RESUME`])
+
+$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv1/ext1-tx.pcap > ext1_v4.packets
+cat ext1_v4.expected | cut -c -48 > expout
+AT_CHECK([cat ext1_v4.packets | cut -c -48], [0], [expout])
+# Skipping the IPv4 checksum.
+cat ext1_v4.expected | cut -c 53- > expout
+AT_CHECK([cat ext1_v4.packets | cut -c 53-], [0], [expout])
+
+# ovs-ofctl also resumes the packets and this causes other ports to receive
+# the DHCP request packet. So reset the pcap files so that its easier to test.
+reset_pcap_file hv1-vif1 hv1/ext1
+rm -f ext1_v4.expected
+rm -f ext1_v4.packets
+
+# Send DHCPv6 request
+src_mac=f00000000003
+src_lla=fe80000000000000f20000fffe000003
+offer_ip=ae700000000000000000000000000006
+test_dhcpv6 1 $src_mac $src_lla 01 $offer_ip
+
+# NXT_RESUMEs should be 2 in hv1.
+OVS_WAIT_UNTIL([test 2 = `cat ofctl_monitor0_hv1.log | grep -c NXT_RESUME`])
+
+# NXT_RESUMEs should be 0 in hv2.
+OVS_WAIT_UNTIL([test 0 = `cat ofctl_monitor0_hv2.log | grep -c NXT_RESUME`])
+
+$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv1/ext1-tx.pcap | \
+sort > ext1_v6.packets
+cat ext1_v6.expected | cut -c -120 > expout
+AT_CHECK([cat ext1_v6.packets | cut -c -120], [0], [expout])
+# Skipping the UDP checksum
+cat ext1_v6.expected | cut -c 125- > expout
+AT_CHECK([cat ext1_v6.packets | cut -c 125-], [0], [expout])
+
+rm -f ext1_v6.expected
+rm -f ext1_v6.packets
+reset_pcap_file hv1-vif1 hv1/ext1
+
+# Change the requested-chassis option for ls1-lp_ext1 from hv1 to hv2
+ovn-nbctl --wait=hv lsp-set-options ls1-lp_ext1 requested-chassis=hv2
+
+hv2_uuid=$(ovn-sbctl list chassis hv2 | grep uuid | awk '{print $3}')
+
+# The ls1-lp_ext1 should be bound to hv2
+chassis=$(ovn-sbctl list port_binding ls1-lp_ext1 | grep -v gateway | \
+grep -v requested | grep chassis | awk '{print $3}')
+AT_CHECK([test $chassis == "$hv2_uuid"], [0], [])
+
+# There should be OF flows for DHCP4/v6 for the ls1-lp_ext1 port in hv2
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | grep reg14=0x$ln_public_key | \
+wc -l], [0], [3
+])
+AT_CHECK([as hv2 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | \
+grep reg14=0x$ln_public_key | wc -l], [0], [1
+])
+
+# There should ne no DHCPv4/v6 flows for ls1-lp_ext1 on hv1
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep "0a.00.00.06" | wc -l], [0], [0
+])
+AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep table=20 | \
+grep controller | grep tp_src=546 | grep \
+"ae.70.00.00.00.00.00.00.00.00.00.00.00.00.00.06" | \
+grep reg14=0x$ln_public_key | wc -l], [0], [0
+])
+
+# Send DHCPDISCOVER again for hv1/ext1. The DHCP response should come from
+# hv2 ovn-controller. Due to the test setup, the port hv1/ext1 is also
+# receiving the expected packet.
+offer_ip=`ip_to_hex 10 0 0 6`
+server_ip=`ip_to_hex 10 0 0 1`
+server_mac=ff1000000001
+expected_dhcp_opts=330400000e100104ffffff0003040a00000136040a000001
+test_dhcp 1 f00000000003 01 $offer_ip 0 $server_mac $server_ip \
+$expected_dhcp_opts 1
+
+# NXT_RESUMEs should be 2 in hv1.
+OVS_WAIT_UNTIL([test 2 = `cat ofctl_monitor0_hv1.log | grep -c NXT_RESUME`])
+
+# NXT_RESUMEs should be 1 in hv2.
+OVS_WAIT_UNTIL([test 1 = `cat ofctl_monitor0_hv2.log | grep -c NXT_RESUME`])
+
+$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv1/ext1-tx.pcap > ext1_v4.packets
+cat ext1_v4.expected | cut -c -48 > expout
+AT_CHECK([cat ext1_v4.packets | cut -c -48], [0], [expout])
+# Skipping the IPv4 checksum.
+cat ext1_v4.expected | cut -c 53- > expout
+AT_CHECK([cat ext1_v4.packets | cut -c 53-], [0], [expout])
+
+# ovs-ofctl also resumes the packets and this causes other ports to receive
+# the DHCP request packet. So reset the pcap files so that its easier to test.
+reset_pcap_file hv1-vif1 hv1/ext1
+rm -f ext1_v4.expected
+
+# Send DHCPv6 request again
+src_mac=f00000000003
+src_lla=fe80000000000000f20000fffe000003
+offer_ip=ae700000000000000000000000000006
+test_dhcpv6 1 $src_mac $src_lla 01 $offer_ip 1
+
+# NXT_RESUMEs should be 2 in hv1.
+OVS_WAIT_UNTIL([test 2 = `cat ofctl_monitor0_hv1.log | grep -c NXT_RESUME`])
+
+# NXT_RESUMEs should be 2 in hv2.
+OVS_WAIT_UNTIL([test 2 = `cat ofctl_monitor0_hv2.log | grep -c NXT_RESUME`])
+
+as hv1
+ovs-vsctl show
+ovs-ofctl dump-flows br-int
+
+as hv2
+ovs-vsctl show
+ovs-ofctl dump-flows br-int
+
+$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv1/ext1-tx.pcap | \
+sort > ext1_v6.packets
+cat ext1_v6.expected | cut -c -120 > expout
+AT_CHECK([cat ext1_v6.packets | cut -c -120], [0], [expout])
+# Skipping the UDP checksum
+cat ext1_v6.expected | cut -c 125- > expout
+AT_CHECK([cat ext1_v6.packets | cut -c 125-], [0], [expout])
+
+rm -f ext1_v6.expected
+rm -f ext1_v6.packets
+
+OVN_CLEANUP([hv1],[hv2])
+AT_CLEANUP
+
 AT_SETUP([ovn -- ovn-controller restart])
 AT_SKIP_IF([test $HAVE_PYTHON = no])
 ovn_start