diff mbox

[ovs-dev,2/5] ovn: Introduce l3 gateway router.

Message ID 1462935081-18291-2-git-send-email-guru@ovn.org
State Not Applicable
Headers show

Commit Message

Gurucharan Shetty May 11, 2016, 2:51 a.m. UTC 
Currently OVN has distributed switches and routers. When a packet
exits a container or a VM, the entire lifecycle of the packet
through multiple switches and routers are calculated in source
chassis itself. When the destination endpoint resides on a different
chassis, the packet is sent to the other chassis and it only goes
through the egress pipeline of that chassis once and eventually to
the real destination.

When the packet returns back, the same thing happens. The return
packet leaves the VM/container on the chassis where it resides.
The packet goes through all the switches and routers in the logical
pipleline on that chassis and then sent to the eventual destination
over the tunnel.

The above makes the logical pipeline very flexible and easy. But,
creates a problem for cases where you need to add stateful services
(via conntrack) on switches and routers.

For l3 gateways, we plan to leverage DNAT and SNAT functionality
and we want to apply DNAT and SNAT rules on a router. So we ideally need
the packet to go through that router in both directions in the same
chassis. To achieve this, this commit introduces a new gateway router which is
static and can be connected to your distributed router via a switch.

To make minimal changes in OVN's logical pipeline, this commit
tries to make the switch port connected to a l3 gateway router look like
a container/VM endpoint for every other chassis except the chassis
on which the l3 gateway router resides. On the chassis where the
gateway router resides, the connection looks just like a patch port.

This is achieved by the doing the following:
Introduces a new type of port_binding record called 'gateway'.
On the chassis where the gateway router resides, this port behaves just
like the port of type 'patch'. The ovn-controller on that chassis
populates the "chassis" column for this record as an indication for
other ovn-controllers of its physical location. Other ovn-controllers
treat this port as they would treat a VM/Container port on a different
chassis.

Signed-off-by: Gurucharan Shetty <guru@ovn.org>
---
v1-v2:
No change.
---
 ovn/controller/binding.c        |    3 +-
 ovn/controller/ovn-controller.c |    3 +-
 ovn/controller/patch.c          |   29 +++++-
 ovn/controller/patch.h          |    3 +-
 ovn/northd/ovn-northd.c         |   42 +++++++--
 ovn/ovn-nb.ovsschema            |    9 +-
 ovn/ovn-nb.xml                  |   15 ++++
 ovn/ovn-sb.xml                  |   35 +++++++-
 tests/ovn.at                    |  184 +++++++++++++++++++++++++++++++++++++++
 9 files changed, 308 insertions(+), 15 deletions(-)

Comments

Mickey Spiegel May 12, 2016, 10:22 p.m. UTC | #1 
See comments inline.

>To: dev@openvswitch.org
>From: Gurucharan Shetty 
>Sent by: "dev" 
>Date: 05/10/2016 08:10PM
>Cc: Gurucharan Shetty <guru@ovn.org>
>Subject: [ovs-dev] [PATCH 2/5] ovn: Introduce l3 gateway router.
>
>Currently OVN has distributed switches and routers. When a packet
>exits a container or a VM, the entire lifecycle of the packet
>through multiple switches and routers are calculated in source
>chassis itself. When the destination endpoint resides on a different
>chassis, the packet is sent to the other chassis and it only goes
>through the egress pipeline of that chassis once and eventually to
>the real destination.
>
>When the packet returns back, the same thing happens. The return
>packet leaves the VM/container on the chassis where it resides.
>The packet goes through all the switches and routers in the logical
>pipleline on that chassis and then sent to the eventual destination
>over the tunnel.
>
>The above makes the logical pipeline very flexible and easy. But,
>creates a problem for cases where you need to add stateful services
>(via conntrack) on switches and routers.

Completely agree up to this point.

>For l3 gateways, we plan to leverage DNAT and SNAT functionality
>and we want to apply DNAT and SNAT rules on a router. So we ideally need
>the packet to go through that router in both directions in the same
>chassis. To achieve this, this commit introduces a new gateway router which is
>static and can be connected to your distributed router via a switch.

Completely agree that you need to go through a common point in both directions
in the same chassis.

Why does this require a separate gateway router?
Why can't it just be a centralized gateway router port on an otherwise distributed
router?

Looking at the logic for ports on remote chassis in physical.c, I see no reason why
that logic cannot work for logical router datapaths just like it works for logical
switch datapaths. On logical switches, some ports are distributed and run
everywhere, e.g. localnet, and other ports run on a specific chassis, e.g. vif and
your proposed "gateway" port.
Am I missing something that prevents a mix of centralized and distributed ports
on a logical router datapath?

We have not tried it yet, but it seems like this would simplify things a lot:
1. Only one router needs to be provisioned rather than a distributed router and a
centralized gateway router
2. No need for static routes between the distributed and centralized gateway routers
3. No need for transit logical switches, transit subnets, or transit flows
4. Less passes through datapaths, improving performance

You can then pin DNAT and SNAT logic to the centralized gateway port, for traffic to
physical networks. East/west traffic to floating IPs still requires additional logic on
other ports, as proposed in Chandra's floating IP patch.

We want to get to a point where SNAT traffic goes through a centralized gateway
port, but DNAT traffic goes through a distributed patch port. This would achieve
parity with the OpenStack ML2 OVS DVR reference implementation, in terms of traffic
subsets that are centralized versus distributed.

>To make minimal changes in OVN's logical pipeline, this commit
>tries to make the switch port connected to a l3 gateway router look like
>a container/VM endpoint for every other chassis except the chassis
>on which the l3 gateway router resides. On the chassis where the
>gateway router resides, the connection looks just like a patch port.

Completely agree that this is the right way to go.

>This is achieved by the doing the following:
>Introduces a new type of port_binding record called 'gateway'.
>On the chassis where the gateway router resides, this port behaves just
>like the port of type 'patch'. The ovn-controller on that chassis
>populates the "chassis" column for this record as an indication for
>other ovn-controllers of its physical location. Other ovn-controllers
>treat this port as they would treat a VM/Container port on a different
>chassis.

I like this logic. My only concern is whether the logical switch port for this
functionality should really be called 'gateway', since this may get confused
with L2 gateway. Some possibilities: 'patch2l3gateway', 'localpatch',
'chassispatch'.

Holding off on more specific comments until we resolve the big picture stuff.

Mickey
Gurucharan Shetty May 12, 2016, 11:20 p.m. UTC | #2 
>
>
>
> Completely agree that you need to go through a common point in both
> directions
> in the same chassis.
>

> Why does this require a separate gateway router?
>
The primary reason to choose a separate gateway router was to support
multiple physical gateways for k8s to which you can loadbalance your
traffic from external world. i.e you will have a router in each physical
gateway with its own floating IP per service. From external world, you can
loadbalance traffic to your gateways. The floating IP is further
loadbalanced to an internal workload.



> Why can't it just be a centralized gateway router port on an otherwise
> distributed
> router?
>
It is indeed one of the solutions for my problem statement (provided you
can support multiple physical gateway chassis.). I haven't spent too much
time thinking on how to do this for multiple physical gateways.


>
> Looking at the logic for ports on remote chassis in physical.c, I see no
> reason why
> that logic cannot work for logical router datapaths just like it works for
> logical
> switch datapaths. On logical switches, some ports are distributed and run
> everywhere, e.g. localnet, and other ports run on a specific chassis, e.g.
> vif and
> your proposed "gateway" port.
> Am I missing something that prevents a mix of centralized and distributed
> ports
> on a logical router datapath?
>

You will have to give me some more details (I am currently unable to
visualize your solution). May be start with a simple topology of one DR
connected to two LS. Simple packet walkthrough (in english) for north-south
(external to internal via floating IPs) and its return traffic (going
through conntrack), south-north traffic (and its return traffic) and
east-west (via central gateway).  My thinking is this:

If we want to do NAT in a router, then we need to have a ingress pipeline
as well as an egress pipeline. A router has multiple ports. When a packet
comes in any router port, I want to be able to do DNAT (and reverse its
effect) and when packet exits any port, I want to be able to do SNAT. I
also should be able to do both DNAT and SNAT on a single packet (to handle
north-south loadbalancing). So the entire router should be there at a
single location.






>
> We have not tried it yet, but it seems like this would simplify things a
> lot:
> 1. Only one router needs to be provisioned rather than a distributed
> router and a
> centralized gateway router
> 2. No need for static routes between the distributed and centralized
> gateway routers
> 3. No need for transit logical switches, transit subnets, or transit flows
> 4. Less passes through datapaths, improving performance
>

The above is ideal.


>
> You can then pin DNAT and SNAT logic to the centralized gateway port, for
> traffic to
> physical networks. East/west traffic to floating IPs still requires
> additional logic on
> other ports, as proposed in Chandra's floating IP patch.
>
> We want to get to a point where SNAT traffic goes through a centralized
> gateway
> port, but DNAT traffic goes through a distributed patch port.

Please tell me what does DNAT mean and what does SNAT mean for you. I may
be talking the opposite thing than you.

dev@openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
>
diff mbox

Patch

diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index 32fcb85..76c86bf 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -205,7 +205,8 @@  binding_run(struct controller_ctx *ctx, const struct ovsrec_bridge *br_int,
                 }
                 sbrec_port_binding_set_chassis(binding_rec, chassis_rec);
             }
-        } else if (binding_rec->chassis == chassis_rec) {
+        } else if (binding_rec->chassis == chassis_rec
+                   && strcmp(binding_rec->type, "gateway")) {
             if (ctx->ovnsb_idl_txn) {
                 VLOG_INFO("Releasing lport %s from this chassis.",
                           binding_rec->logical_port);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index f68f842..9d134e1 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -364,7 +364,8 @@  main(int argc, char *argv[])
         }
 
         if (br_int) {
-            patch_run(&ctx, br_int, &local_datapaths, &patched_datapaths);
+            patch_run(&ctx, br_int, chassis_id, &local_datapaths,
+                      &patched_datapaths);
 
             struct lport_index lports;
             struct mcgroup_index mcgroups;
diff --git a/ovn/controller/patch.c b/ovn/controller/patch.c
index 4808146..a88f7aa 100644
--- a/ovn/controller/patch.c
+++ b/ovn/controller/patch.c
@@ -267,12 +267,28 @@  add_patched_datapath(struct hmap *patched_datapaths,
 static void
 add_logical_patch_ports(struct controller_ctx *ctx,
                         const struct ovsrec_bridge *br_int,
+                        const char *local_chassis_id,
                         struct shash *existing_ports,
                         struct hmap *patched_datapaths)
 {
+    const struct sbrec_chassis *chassis_rec;
+    chassis_rec = get_chassis(ctx->ovnsb_idl, local_chassis_id);
+    if (!chassis_rec) {
+        return;
+    }
+
     const struct sbrec_port_binding *binding;
     SBREC_PORT_BINDING_FOR_EACH (binding, ctx->ovnsb_idl) {
-        if (!strcmp(binding->type, "patch")) {
+        bool local_port = false;
+        if (!strcmp(binding->type, "gateway")) {
+            const char *chassis = smap_get(&binding->options,
+                                           "gateway-chassis");
+            if (!strcmp(local_chassis_id, chassis)) {
+                local_port = true;
+            }
+        }
+
+        if (!strcmp(binding->type, "patch") || local_port) {
             const char *local = binding->logical_port;
             const char *peer = smap_get(&binding->options, "peer");
             if (!peer) {
@@ -287,13 +303,19 @@  add_logical_patch_ports(struct controller_ctx *ctx,
             free(dst_name);
             free(src_name);
             add_patched_datapath(patched_datapaths, binding);
+            if (local_port) {
+                if (binding->chassis != chassis_rec) {
+                    sbrec_port_binding_set_chassis(binding, chassis_rec);
+                }
+            }
         }
     }
 }
 
 void
 patch_run(struct controller_ctx *ctx, const struct ovsrec_bridge *br_int,
-          struct hmap *local_datapaths, struct hmap *patched_datapaths)
+          const char *chassis_id, struct hmap *local_datapaths,
+          struct hmap *patched_datapaths)
 {
     if (!ctx->ovs_idl_txn) {
         return;
@@ -313,7 +335,8 @@  patch_run(struct controller_ctx *ctx, const struct ovsrec_bridge *br_int,
      * 'existing_ports' any patch ports that do exist in the database and
      * should be there. */
     add_bridge_mappings(ctx, br_int, &existing_ports, local_datapaths);
-    add_logical_patch_ports(ctx, br_int, &existing_ports, patched_datapaths);
+    add_logical_patch_ports(ctx, br_int, chassis_id, &existing_ports,
+                            patched_datapaths);
 
     /* Now 'existing_ports' only still contains patch ports that exist in the
      * database but shouldn't.  Delete them from the database. */
diff --git a/ovn/controller/patch.h b/ovn/controller/patch.h
index d5d842e..7920a48 100644
--- a/ovn/controller/patch.h
+++ b/ovn/controller/patch.h
@@ -27,6 +27,7 @@  struct hmap;
 struct ovsrec_bridge;
 
 void patch_run(struct controller_ctx *, const struct ovsrec_bridge *br_int,
-               struct hmap *local_datapaths, struct hmap *patched_datapaths);
+               const char *chassis_id, struct hmap *local_datapaths,
+               struct hmap *patched_datapaths);
 
 #endif /* ovn/patch.h */
diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
index 3a29770..2cfaf95 100644
--- a/ovn/northd/ovn-northd.c
+++ b/ovn/northd/ovn-northd.c
@@ -689,11 +689,24 @@  ovn_port_update_sbrec(const struct ovn_port *op)
 {
     sbrec_port_binding_set_datapath(op->sb, op->od->sb);
     if (op->nbr) {
-        sbrec_port_binding_set_type(op->sb, "patch");
+        /* If the router is for l3 gateway, it resides on a chassis
+         * and its port type is "gateway". */
+        const char *chassis = smap_get(&op->od->nbr->options, "chassis");
+        if (chassis) {
+            sbrec_port_binding_set_type(op->sb, "gateway");
+        } else {
+            sbrec_port_binding_set_type(op->sb, "patch");
+        }
 
         const char *peer = op->peer ? op->peer->key : "<error>";
-        const struct smap ids = SMAP_CONST1(&ids, "peer", peer);
-        sbrec_port_binding_set_options(op->sb, &ids);
+        struct smap new;
+        smap_init(&new);
+        smap_add(&new, "peer", peer);
+        if (chassis) {
+            smap_add(&new, "gateway-chassis", chassis);
+        }
+        sbrec_port_binding_set_options(op->sb, &new);
+        smap_destroy(&new);
 
         sbrec_port_binding_set_parent_port(op->sb, NULL);
         sbrec_port_binding_set_tag(op->sb, NULL, 0);
@@ -703,15 +716,32 @@  ovn_port_update_sbrec(const struct ovn_port *op)
             sbrec_port_binding_set_type(op->sb, op->nbs->type);
             sbrec_port_binding_set_options(op->sb, &op->nbs->options);
         } else {
-            sbrec_port_binding_set_type(op->sb, "patch");
+            const char *chassis = NULL;
+            if (op->peer && op->peer->od && op->peer->od->nbr) {
+                chassis = smap_get(&op->peer->od->nbr->options, "chassis");
+            }
+
+            /* A switch port connected to a gateway router is also of
+             * type "gateway". */
+            if (chassis) {
+                sbrec_port_binding_set_type(op->sb, "gateway");
+            } else {
+                sbrec_port_binding_set_type(op->sb, "patch");
+            }
 
             const char *router_port = smap_get(&op->nbs->options,
                                                "router-port");
             if (!router_port) {
                 router_port = "<error>";
             }
-            const struct smap ids = SMAP_CONST1(&ids, "peer", router_port);
-            sbrec_port_binding_set_options(op->sb, &ids);
+            struct smap new;
+            smap_init(&new);
+            smap_add(&new, "peer", router_port);
+            if (chassis) {
+                smap_add(&new, "gateway-chassis", chassis);
+            }
+            sbrec_port_binding_set_options(op->sb, &new);
+            smap_destroy(&new);
         }
         sbrec_port_binding_set_parent_port(op->sb, op->nbs->parent_name);
         sbrec_port_binding_set_tag(op->sb, op->nbs->tag, op->nbs->n_tag);
diff --git a/ovn/ovn-nb.ovsschema b/ovn/ovn-nb.ovsschema
index 8163f6a..fa21b30 100644
--- a/ovn/ovn-nb.ovsschema
+++ b/ovn/ovn-nb.ovsschema
@@ -1,7 +1,7 @@ 
 {
     "name": "OVN_Northbound",
-    "version": "2.1.1",
-    "cksum": "2615511875 5108",
+    "version": "2.1.2",
+    "cksum": "429668869 5325",
     "tables": {
         "Logical_Switch": {
             "columns": {
@@ -78,6 +78,11 @@ 
                                    "max": "unlimited"}},
                 "default_gw": {"type": {"key": "string", "min": 0, "max": 1}},
                 "enabled": {"type": {"key": "boolean", "min": 0, "max": 1}},
+                "options": {
+                     "type": {"key": "string",
+                              "value": "string",
+                              "min": 0,
+                              "max": "unlimited"}},
                 "external_ids": {
                     "type": {"key": "string", "value": "string",
                              "min": 0, "max": "unlimited"}}},
diff --git a/ovn/ovn-nb.xml b/ovn/ovn-nb.xml
index d7fd595..d239499 100644
--- a/ovn/ovn-nb.xml
+++ b/ovn/ovn-nb.xml
@@ -630,6 +630,21 @@ 
       column is set to <code>false</code>, the router is disabled.  A disabled
       router has all ingress and egress traffic dropped.
     </column>
+
+    <group title="Options">
+      <p>
+        Additional options for the logical router.
+      </p>
+
+      <column name="options" key="chassis">
+        If set, indicates that the logical router in question is
+        non-distributed and resides in the set chassis. The same
+        value is also used by <code>ovn-controller</code> to
+        uniquely identify the chassis in the OVN deployment and
+        comes from <code>external_ids:system-id</code> in the
+        <code>Open_vSwitch</code> table of Open_vSwitch database.
+      </column>
+    </group>
     
     <group title="Common Columns">
       <column name="external_ids">
diff --git a/ovn/ovn-sb.xml b/ovn/ovn-sb.xml
index efd2f9a..741228c 100644
--- a/ovn/ovn-sb.xml
+++ b/ovn/ovn-sb.xml
@@ -1220,7 +1220,12 @@  tcp.flags = RST;
       which <code>ovn-controller</code>/<code>ovn-controller-vtep</code> in
       turn finds out by monitoring the local hypervisor's Open_vSwitch
       database, which identifies logical ports via the conventions described
-      in <code>IntegrationGuide.md</code>.
+      in <code>IntegrationGuide.md</code>. (The exceptions are for
+      <code>Port_Binding</code> records of <code>type</code> 'gateway',
+      whose locations are identified by <code>ovn-northd</code> via
+      the <code>options:gateway-chassis</code> column in this table.
+      <code>ovn-controller</code> is still responsible to populate the
+      <code>chassis</code> column.)
     </p>
 
     <p>
@@ -1298,6 +1303,14 @@  tcp.flags = RST;
             a logical router to a logical switch or to another logical router.
           </dd>
 
+          <dt><code>gateway</code></dt>
+          <dd>
+            One of a pair of logical ports that act as if connected by a patch
+            cable across multiple chassis.  Useful for connecting a logical
+            switch with a gateway router (which is only resident on a
+            particular chassis).
+          </dd>
+
           <dt><code>localnet</code></dt>
           <dd>
             A connection to a locally accessible network from each
@@ -1336,6 +1349,26 @@  tcp.flags = RST;
       </column>
     </group>
 
+    <group title="Gateway Options">
+      <p>
+        These options apply to logical ports with <ref column="type"/> of
+        <code>gateway</code>.
+      </p>
+
+      <column name="options" key="peer">
+        The <ref column="logical_port"/> in the <ref table="Port_Binding"/>
+        record for the other side of the 'gateway' port.  The named <ref
+        column="logical_port"/> must specify this <ref column="logical_port"/>
+        in its own <code>peer</code> option.  That is, the two 'gateway'
+        logical ports must have reversed <ref column="logical_port"/> and
+        <code>peer</code> values.
+      </column>
+
+      <column name="options" key="gateway-chassis">
+        The <code>chassis</code> in which the port resides.
+      </column>
+    </group>
+
     <group title="Localnet Options">
       <p>
         These options apply to logical ports with <ref column="type"/> of
diff --git a/tests/ovn.at b/tests/ovn.at
index 6961be0..b3c6e8b 100644
--- a/tests/ovn.at
+++ b/tests/ovn.at
@@ -2781,3 +2781,187 @@  OVS_APP_EXIT_AND_WAIT([ovs-vswitchd])
 OVS_APP_EXIT_AND_WAIT([ovsdb-server])
 
 AT_CLEANUP
+
+
+AT_SETUP([ovn -- 2 HVs, 2 LRs connected via LS, gateway router])
+AT_KEYWORDS([ovngatewayrouter])
+AT_SKIP_IF([test $HAVE_PYTHON = no])
+ovn_start
+
+# Logical network:
+# Two LRs - R1 and R2 that are connected to each other via LS "join"
+# in 20.0.0.0/24 network. R1 has switchess foo (192.168.1.0/24)
+# connected to it. R2 has alice (172.16.1.0/24) connected to it.
+# R2 is a gateway router.
+
+
+
+# Create two hypervisor and create OVS ports corresponding to logical ports.
+net_add n1
+
+sim_add hv1
+as hv1
+ovs-vsctl add-br br-phys
+ovn_attach n1 br-phys 192.168.0.1
+ovs-vsctl -- add-port br-int hv1-vif1 -- \
+    set interface hv1-vif1 external-ids:iface-id=foo1 \
+    options:tx_pcap=hv1/vif1-tx.pcap \
+    options:rxq_pcap=hv1/vif1-rx.pcap \
+    ofport-request=1
+
+
+sim_add hv2
+as hv2
+ovs-vsctl add-br br-phys
+ovn_attach n1 br-phys 192.168.0.2
+ovs-vsctl -- add-port br-int hv2-vif1 -- \
+    set interface hv2-vif1 external-ids:iface-id=alice1 \
+    options:tx_pcap=hv2/vif1-tx.pcap \
+    options:rxq_pcap=hv2/vif1-rx.pcap \
+    ofport-request=1
+
+# Pre-populate the hypervisors' ARP tables so that we don't lose any
+# packets for ARP resolution (native tunneling doesn't queue packets
+# for ARP resolution).
+ovn_populate_arp
+
+ovn-nbctl create Logical_Router name=R1
+ovn-nbctl create Logical_Router name=R2 options:chassis="hv2"
+
+ovn-nbctl lswitch-add foo
+ovn-nbctl lswitch-add alice
+ovn-nbctl lswitch-add join
+
+# Connect foo to R1
+ovn-nbctl -- --id=@lrp create Logical_Router_port name=foo \
+network=192.168.1.1/24 mac=\"00:00:01:01:02:03\" -- add Logical_Router R1 \
+ports @lrp -- lport-add foo rp-foo
+
+ovn-nbctl set Logical_port rp-foo type=router options:router-port=foo \
+addresses=\"00:00:01:01:02:03\"
+
+# Connect alice to R2
+ovn-nbctl -- --id=@lrp create Logical_Router_port name=alice \
+network=172.16.1.1/24 mac=\"00:00:02:01:02:03\" -- add Logical_Router R2 \
+ports @lrp -- lport-add alice rp-alice
+
+ovn-nbctl set Logical_port rp-alice type=router options:router-port=alice \
+addresses=\"00:00:02:01:02:03\"
+
+
+# Connect R1 to join
+ovn-nbctl -- --id=@lrp create Logical_Router_port name=R1_join \
+network=20.0.0.1/24 mac=\"00:00:04:01:02:03\" -- add Logical_Router R1 \
+ports @lrp -- lport-add join r1-join
+
+ovn-nbctl set Logical_port r1-join type=router options:router-port=R1_join \
+addresses='"00:00:04:01:02:03"'
+
+# Connect R2 to join
+ovn-nbctl -- --id=@lrp create Logical_Router_port name=R2_join \
+network=20.0.0.2/24 mac=\"00:00:04:01:02:04\" -- add Logical_Router R2 \
+ports @lrp -- lport-add join r2-join
+
+ovn-nbctl set Logical_port r2-join type=router options:router-port=R2_join \
+addresses='"00:00:04:01:02:04"'
+
+
+#install static routes
+ovn-nbctl -- --id=@lrt create Logical_Router_Static_Route \
+ip_prefix=172.16.1.0/24 nexthop=20.0.0.2 -- add Logical_Router \
+R1 static_routes @lrt
+
+ovn-nbctl -- --id=@lrt create Logical_Router_Static_Route \
+ip_prefix=192.168.1.0/24 nexthop=20.0.0.1 -- add Logical_Router \
+R2 static_routes @lrt
+
+# Create logical port foo1 in foo
+ovn-nbctl lport-add foo foo1 \
+-- lport-set-addresses foo1 "f0:00:00:01:02:03 192.168.1.2"
+
+# Create logical port alice1 in alice
+ovn-nbctl lport-add alice alice1 \
+-- lport-set-addresses alice1 "f0:00:00:01:02:04 172.16.1.2"
+
+
+# Allow some time for ovn-northd and ovn-controller to catch up.
+# XXX This should be more systematic.
+sleep 2
+
+ip_to_hex() {
+    printf "%02x%02x%02x%02x" "$@"
+}
+trim_zeros() {
+    sed 's/\(00\)\{1,\}$//'
+}
+
+# Send ip packets between foo1 and alice1
+src_mac="f00000010203"
+dst_mac="000001010203"
+src_ip=`ip_to_hex 192 168 1 2`
+dst_ip=`ip_to_hex 172 16 1 2`
+packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
+
+echo "---------NB dump-----"
+ovn-nbctl show
+echo "---------------------"
+ovn-nbctl list logical_router
+echo "---------------------"
+ovn-nbctl list logical_router_port
+echo "---------------------"
+
+echo "---------SB dump-----"
+ovn-sbctl list datapath_binding
+echo "---------------------"
+ovn-sbctl list port_binding
+echo "---------------------"
+ovn-sbctl dump-flows
+echo "---------------------"
+ovn-sbctl list chassis
+ovn-sbctl list encap
+echo "---------------------"
+
+echo "------ hv1 dump ----------"
+as hv1 ovs-ofctl show br-int
+as hv1 ovs-ofctl dump-flows br-int
+echo "------ hv2 dump ----------"
+as hv2 ovs-ofctl show br-int
+as hv2 ovs-ofctl dump-flows br-int
+echo "----------------------------"
+
+# Packet to Expect at alice1
+src_mac="000002010203"
+dst_mac="f00000010204"
+src_ip=`ip_to_hex 192 168 1 2`
+dst_ip=`ip_to_hex 172 16 1 2`
+expected=${dst_mac}${src_mac}08004500001c000000003e110200${src_ip}${dst_ip}0035111100080000
+
+
+as hv1 ovs-appctl netdev-dummy/receive hv1-vif1 $packet
+as hv1 ovs-appctl ofproto/trace br-int in_port=1 $packet
+
+$PYTHON "$top_srcdir/utilities/ovs-pcap.in" hv2/vif1-tx.pcap | trim_zeros > received1.packets
+echo $expected | trim_zeros > expout
+AT_CHECK([cat received1.packets], [0], [expout])
+
+for sim in hv1 hv2; do
+    as $sim
+    OVS_APP_EXIT_AND_WAIT([ovn-controller])
+    OVS_APP_EXIT_AND_WAIT([ovs-vswitchd])
+    OVS_APP_EXIT_AND_WAIT([ovsdb-server])
+done
+
+as ovn-sb
+OVS_APP_EXIT_AND_WAIT([ovsdb-server])
+
+as ovn-nb
+OVS_APP_EXIT_AND_WAIT([ovsdb-server])
+
+as northd
+OVS_APP_EXIT_AND_WAIT([ovn-northd])
+
+as main
+OVS_APP_EXIT_AND_WAIT([ovs-vswitchd])
+OVS_APP_EXIT_AND_WAIT([ovsdb-server])
+
+AT_CLEANUP