[ovs-dev,no-slow,v2,7/8] ofproto-dpif: Don't slow-path controller actions.

Message ID 1515623246-3820-7-git-send-email-jpettit@ovn.org
State Accepted
Headers show
Series
  • [ovs-dev,no-slow,v2,1/8] ofproto-dpif: Use a fixed size userspace cookie.
Related show

Commit Message

Justin Pettit Jan. 10, 2018, 10:27 p.m.
Controller actions have become more commonly used for purposes other
than just making forwarding decisions (e.g., packet logging).  A packet
that needs to be copied to the controller and forwarded would always be
sent to ovs-vswitchd to be handled, which could negatively affect
performance and cause heavier CPU utilization in ovs-vswitchd.

This commit changes the behavior so that OpenFlow controller actions
become userspace datapath actions while continuing to let packet
forwarding and manipulation continue to be handled by the datapath
directly.

This patch still slow-paths controller actions with the "pause" flag
set.  A future patch will stop slow-pathing these pause actions as
well.

Signed-off-by: Justin Pettit <jpettit@ovn.org>
---
v1->v2: Changes suggested by Ben.
---
 Documentation/tutorials/faucet.rst |  46 ++----
 NEWS                               |   3 +
 lib/odp-util.c                     |  54 ++++++-
 lib/odp-util.h                     |  16 +-
 ofproto/ofproto-dpif-rid.c         |  10 +-
 ofproto/ofproto-dpif-rid.h         |   4 +
 ofproto/ofproto-dpif-upcall.c      |  95 +++++++++---
 ofproto/ofproto-dpif-xlate.c       | 297 +++++++++++--------------------------
 ofproto/ofproto-unixctl.man        |   6 +-
 tests/odp.at                       |   2 +
 tests/ofproto-dpif.at              |  65 +++++---
 tests/pmd.at                       |   4 +-
 tests/tunnel-push-pop-ipv6.at      |   2 +-
 tests/tunnel-push-pop.at           |   2 +-
 14 files changed, 307 insertions(+), 299 deletions(-)

Comments

Ben Pfaff Jan. 11, 2018, 10:20 a.m. | #1
On Wed, Jan 10, 2018 at 02:27:25PM -0800, Justin Pettit wrote:
> Controller actions have become more commonly used for purposes other
> than just making forwarding decisions (e.g., packet logging).  A packet
> that needs to be copied to the controller and forwarded would always be
> sent to ovs-vswitchd to be handled, which could negatively affect
> performance and cause heavier CPU utilization in ovs-vswitchd.
> 
> This commit changes the behavior so that OpenFlow controller actions
> become userspace datapath actions while continuing to let packet
> forwarding and manipulation continue to be handled by the datapath
> directly.
> 
> This patch still slow-paths controller actions with the "pause" flag
> set.  A future patch will stop slow-pathing these pause actions as
> well.
> 
> Signed-off-by: Justin Pettit <jpettit@ovn.org>
> ---
> v1->v2: Changes suggested by Ben.

Looking at the updates in v2:
Acked-by: Ben Pfaff <blp@ovn.org>

Patch

diff --git a/Documentation/tutorials/faucet.rst b/Documentation/tutorials/faucet.rst
index 2f7ac62343ba..2672791908e6 100644
--- a/Documentation/tutorials/faucet.rst
+++ b/Documentation/tutorials/faucet.rst
@@ -620,9 +620,7 @@  trivial example::
 
   Final flow: unchanged
   Megaflow: recirc_id=0,eth,in_port=1,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000
-  Datapath actions: push_vlan(vid=100,pcp=0),pop_vlan,2,3
-  This flow is handled by the userspace slow path because it:
-	  - Sends "packet-in" messages to the OpenFlow controller.
+  Datapath actions: push_vlan(vid=100,pcp=0),userspace(pid=0,controller(reason=1,flags=1,recirc_id=1,rule_cookie=0x5adc15c0,controller_id=0,max_len=96)),pop_vlan,2,3
 
 The first line of output, beginning with ``Flow:``, just repeats our
 request in a more verbose form, including the L2 fields that were
@@ -640,18 +638,7 @@  Summary information follows the numbered tables.  The packet hasn't
 been changed (overall, even though a VLAN was pushed and then popped
 back off) since ingress, hence ``Final flow: unchanged``.  We'll look
 at the ``Megaflow`` information later.  The ``Datapath actions``
-summarize what would actually happen to such a packet.  Finally, the
-note at the end gives a hint that this flow would not perform well for
-large volumes of traffic, because it has to be handled in the switch's
-slow path since it sends OpenFlow messages to the controller.
-
-.. note::
-
-  This performance limitation is probably not problematic in this case
-  because it is only used for MAC learning, so that most packets won't
-  encounter it.  However, the Open vSwitch 2.9 release (which is
-  upcoming as of this writing) will likely remove this performance
-  limitation anyway.
+summarize what would actually happen to such a packet.
 
 Triggering MAC Learning
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -753,18 +740,15 @@  without any further MAC learning, e.g.::
 Performance
 ~~~~~~~~~~~
 
-We've already seen one factor that can be important for performance:
-Open vSwitch forces any flow that sends a packet to an OpenFlow
-controller into its "slow path", which means that processing packets
-in the flow will be orders of magnitude slower than otherwise.  This
-distinction between "slow path" and "fast path" is the key to making
-sure that Open vSwitch performs as fast as possible.
-
-In addition to sending packets to a controller, some other factors can
-force a flow or a packet to take the slow path.  As one example, all
-CFM, BFD, LACP, STP, and LLDP processing takes place in the slow path,
-in the cases where Open vSwitch processes these protocols itself
-instead of delegating to controller-written flows.  As a second
+Open vSwitch has a concept of a "fast path" and a "slow path"; ideally
+all packets stay in the fast path.  This distinction between slow path
+and fast path is the key to making sure that Open vSwitch performs as
+fast as possible.
+
+Some factors can force a flow or a packet to take the slow path.  As one
+example, all CFM, BFD, LACP, STP, and LLDP processing takes place in the
+slow path, in the cases where Open vSwitch processes these protocols
+itself instead of delegating to controller-written flows.  As a second
 example, any flow that modifies ARP fields is processed in the slow
 path.  These are corner cases that are unlikely to cause performance
 problems in practice because these protocols send packets at a
@@ -1142,9 +1126,7 @@  this::
 
   Final flow: udp,in_port=1,dl_vlan=100,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0
   Megaflow: recirc_id=0,eth,ip,in_port=1,vlan_tci=0x0000/0x1fff,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_dst=10.200.0.0/25,nw_frag=no
-  Datapath actions: push_vlan(vid=100,pcp=0)
-  This flow is handled by the userspace slow path because it:
-	  - Sends "packet-in" messages to the OpenFlow controller.
+  Datapath actions: push_vlan(vid=100,pcp=0),userspace(pid=0,controller(reason=1,flags=0,recirc_id=6,rule_cookie=0x5adc15c0,controller_id=0,max_len=128))
 
 Observe that the packet gets recognized as destined to the router, in
 table 3, and then as properly destined to the 10.200.0.0/24 network,
@@ -1201,9 +1183,7 @@  reply::
 
   Final flow: arp,in_port=4,dl_vlan=200,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:10:20:30:40:50,dl_dst=0e:00:00:00:00:01,arp_spa=10.200.0.1,arp_tpa=10.200.0.254,arp_op=2,arp_sha=00:10:20:30:40:50,arp_tha=0e:00:00:00:00:01
   Megaflow: recirc_id=0,eth,arp,in_port=4,vlan_tci=0x0000/0x1fff,dl_dst=0e:00:00:00:00:01,arp_tpa=10.200.0.254
-  Datapath actions: push_vlan(vid=200,pcp=0)
-  This flow is handled by the userspace slow path because it:
-	  - Sends "packet-in" messages to the OpenFlow controller.
+  Datapath actions: push_vlan(vid=200,pcp=0),userspace(pid=0,controller(reason=1,flags=0,recirc_id=7,rule_cookie=0x5adc15c0,controller_id=0,max_len=128))
 
 It shows up in ``inst/faucet.log``::
 
diff --git a/NEWS b/NEWS
index fd0e2d84e507..11c086518dfc 100644
--- a/NEWS
+++ b/NEWS
@@ -10,6 +10,9 @@  Post-v2.8.0
      * ovsdb-tool: New "db-name" and "schema-name" commands.
    - ovs-vsctl and other commands that display data in tables now support a
      --max-column-width option to limit column width.
+   - No longer slow-path traffic that sends to a controller.  Applications,
+     such as OVN ACL logging, want to send a copy of a packet to a
+     controller while leaving the actual packet forwarding in the datapath.
    - OVN:
      * The "requested-chassis" option for a logical switch port now accepts a
        chassis "hostname" in addition to a chassis "name".
diff --git a/lib/odp-util.c b/lib/odp-util.c
index a2a8d615f708..1b4b847e1285 100644
--- a/lib/odp-util.c
+++ b/lib/odp-util.c
@@ -478,6 +478,21 @@  format_odp_userspace_action(struct ds *ds, const struct nlattr *attr,
                 odp_portno_name_format(portno_names,
                                        cookie.ipfix.output_odp_port, ds);
                 ds_put_char(ds, ')');
+            } else if (cookie.type == USER_ACTION_COOKIE_CONTROLLER) {
+                ds_put_format(ds, ",controller(reason=%"PRIu16
+                              ",dont_send=%"PRIu8
+                              ",recirc_id=%"PRIu32
+                              ",rule_cookie=%#"PRIx64
+                              ",controller_id=%"PRIu16
+                              ",max_len=%"PRIu16,
+                              cookie.controller.reason,
+                              cookie.controller.dont_send ? 1 : 0,
+                              cookie.controller.recirc_id,
+                              ntohll(get_32aligned_be64(
+                                         &cookie.controller.rule_cookie)),
+                              cookie.controller.controller_id,
+                              cookie.controller.max_len);
+                ds_put_char(ds, ')');
             } else {
                 userdata_unspec = true;
             }
@@ -1128,6 +1143,15 @@  parse_odp_userspace_action(const char *s, struct ofpbuf *actions)
         uint32_t collector_set_id;
         uint32_t obs_domain_id;
         uint32_t obs_point_id;
+
+        /* USER_ACTION_COOKIE_CONTROLLER. */
+        uint8_t dont_send;
+        uint16_t reason;
+        uint32_t recirc_id;
+        ovs_be64 rule_cookie;
+        uint16_t controller_id;
+        uint16_t max_len;
+
         int vid, pcp;
         int n1 = -1;
         if (ovs_scan(&s[n], ",sFlow(vid=%i,"
@@ -1201,8 +1225,26 @@  parse_odp_userspace_action(const char *s, struct ofpbuf *actions)
             cookie.ofp_in_port = OFPP_NONE;
             cookie.ofproto_uuid = UUID_ZERO;
             cookie.ipfix.output_odp_port = u32_to_odp(output);
-        } else if (ovs_scan(&s[n], ",userdata(%n",
-                            &n1)) {
+        } else if (ovs_scan(&s[n], ",controller(reason=%"SCNu16
+                              ",dont_send=%"SCNu8
+                              ",recirc_id=%"SCNu32
+                              ",rule_cookie=%"SCNx64
+                              ",controller_id=%"SCNu16
+                              ",max_len=%"SCNu16")%n",
+                              &reason, &dont_send, &recirc_id, &rule_cookie,
+                              &controller_id, &max_len, &n1)) {
+            n += n1;
+            cookie.type = USER_ACTION_COOKIE_CONTROLLER;
+            cookie.ofp_in_port = OFPP_NONE;
+            cookie.ofproto_uuid = UUID_ZERO;
+            cookie.controller.dont_send = dont_send ? true : false;
+            cookie.controller.reason = reason;
+            cookie.controller.recirc_id = recirc_id;
+            put_32aligned_be64(&cookie.controller.rule_cookie,
+                               htonll(rule_cookie));
+            cookie.controller.controller_id = controller_id;
+            cookie.controller.max_len = max_len;
+       } else if (ovs_scan(&s[n], ",userdata(%n", &n1)) {
             char *end;
 
             n += n1;
@@ -3266,7 +3308,7 @@  format_odp_tun_attr(const struct nlattr *attr, const struct nlattr *mask_attr,
         case OVS_TUNNEL_KEY_ATTR_ID:
             format_be64(ds, "tun_id", nl_attr_get_be64(a),
                         ma ? nl_attr_get(ma) : NULL, verbose);
-	    flags |= FLOW_TNL_F_KEY;
+            flags |= FLOW_TNL_F_KEY;
             if (ma) {
                 mask_flags |= FLOW_TNL_F_KEY;
             }
@@ -3302,10 +3344,10 @@  format_odp_tun_attr(const struct nlattr *attr, const struct nlattr *mask_attr,
                        ma ? nl_attr_get(ma) : NULL, verbose);
             break;
         case OVS_TUNNEL_KEY_ATTR_DONT_FRAGMENT:
-	    flags |= FLOW_TNL_F_DONT_FRAGMENT;
+            flags |= FLOW_TNL_F_DONT_FRAGMENT;
             break;
         case OVS_TUNNEL_KEY_ATTR_CSUM:
-	    flags |= FLOW_TNL_F_CSUM;
+            flags |= FLOW_TNL_F_CSUM;
             break;
         case OVS_TUNNEL_KEY_ATTR_TP_SRC:
             format_be16(ds, "tp_src", nl_attr_get_be16(a),
@@ -3316,7 +3358,7 @@  format_odp_tun_attr(const struct nlattr *attr, const struct nlattr *mask_attr,
                         ma ? nl_attr_get(ma) : NULL, verbose);
             break;
         case OVS_TUNNEL_KEY_ATTR_OAM:
-	    flags |= FLOW_TNL_F_OAM;
+            flags |= FLOW_TNL_F_OAM;
             break;
         case OVS_TUNNEL_KEY_ATTR_VXLAN_OPTS:
             ds_put_cstr(ds, "vxlan(");
diff --git a/lib/odp-util.h b/lib/odp-util.h
index 2a4b3d13812b..981dcd8f2a23 100644
--- a/lib/odp-util.h
+++ b/lib/odp-util.h
@@ -42,9 +42,8 @@  struct pkt_metadata;
     SPR(SLOW_BFD,        "bfd",        "Consists of BFD packets")       \
     SPR(SLOW_LACP,       "lacp",       "Consists of LACP packets")      \
     SPR(SLOW_STP,        "stp",        "Consists of STP packets")       \
-    SPR(SLOW_LLDP,       "lldp",       "Consists of LLDP packets")    \
-    SPR(SLOW_CONTROLLER, "controller",                                  \
-        "Sends \"packet-in\" messages to the OpenFlow controller")      \
+    SPR(SLOW_LLDP,       "lldp",       "Consists of LLDP packets")      \
+    SPR(SLOW_PAUSE,      "pause",      "Controller action with pause")  \
     SPR(SLOW_ACTION,     "action",                                      \
         "Uses action(s) not supported by datapath")
 
@@ -298,6 +297,7 @@  enum user_action_cookie_type {
     USER_ACTION_COOKIE_SLOW_PATH,    /* Userspace must process this flow. */
     USER_ACTION_COOKIE_FLOW_SAMPLE,  /* Packet for per-flow sampling. */
     USER_ACTION_COOKIE_IPFIX,        /* Packet for per-bridge IPFIX sampling. */
+    USER_ACTION_COOKIE_CONTROLLER,   /* Forward packet to controller. */
 };
 
 /* user_action_cookie is passed as argument to OVS_ACTION_ATTR_USERSPACE. */
@@ -333,6 +333,16 @@  struct user_action_cookie {
             /* USER_ACTION_COOKIE_IPFIX. */
             odp_port_t output_odp_port; /* The output odp port. */
         } ipfix;
+
+        struct {
+            /* USER_ACTION_COOKIE_CONTROLLER. */
+            bool dont_send;         /* Don't send the packet to controller. */
+            uint16_t reason;
+            uint32_t recirc_id;
+            ovs_32aligned_be64 rule_cookie;
+            uint16_t controller_id;
+            uint16_t max_len;
+        } controller;
     };
 };
 BUILD_ASSERT_DECL(sizeof(struct user_action_cookie) == 48);
diff --git a/ofproto/ofproto-dpif-rid.c b/ofproto/ofproto-dpif-rid.c
index fc570048954c..83278d82bbef 100644
--- a/ofproto/ofproto-dpif-rid.c
+++ b/ofproto/ofproto-dpif-rid.c
@@ -142,6 +142,9 @@  frozen_state_hash(const struct frozen_state *state)
         hash = hash_bytes64(ALIGNED_CAST(const uint64_t *, state->ofpacts),
                             state->ofpacts_len, hash);
     }
+    if (state->userdata && state->userdata_len) {
+        hash = hash_bytes(state->userdata, state->userdata_len, hash);
+    }
     return hash;
 }
 
@@ -158,7 +161,8 @@  frozen_state_equal(const struct frozen_state *a, const struct frozen_state *b)
             && ofpacts_equal(a->ofpacts, a->ofpacts_len,
                              b->ofpacts, b->ofpacts_len)
             && ofpacts_equal(a->action_set, a->action_set_len,
-                             b->action_set, b->action_set_len));
+                             b->action_set, b->action_set_len)
+            && !memcmp(a->userdata, b->userdata, a->userdata_len));
 }
 
 /* Lockless RCU protected lookup.  If node is needed accross RCU quiescent
@@ -203,6 +207,9 @@  frozen_state_clone(struct frozen_state *new, const struct frozen_state *old)
     new->action_set = (new->action_set_len
                        ? xmemdup(new->action_set, new->action_set_len)
                        : NULL);
+    new->userdata = (new->userdata_len
+                     ? xmemdup(new->userdata, new->userdata_len)
+                     : NULL);
 }
 
 static void
@@ -211,6 +218,7 @@  frozen_state_free(struct frozen_state *state)
     free(state->stack);
     free(state->ofpacts);
     free(state->action_set);
+    free(state->userdata);
 }
 
 /* Allocate a unique recirculation id for the given set of flow metadata.
diff --git a/ofproto/ofproto-dpif-rid.h b/ofproto/ofproto-dpif-rid.h
index 19fc27c7ca0c..858b02e39302 100644
--- a/ofproto/ofproto-dpif-rid.h
+++ b/ofproto/ofproto-dpif-rid.h
@@ -149,6 +149,10 @@  struct frozen_state {
     size_t ofpacts_len;           /* Size of 'ofpacts', in bytes. */
     struct ofpact *action_set;
     size_t action_set_len;        /* Size of 'action_set', in bytes. */
+
+    /* User data for controller userspace cookie. */
+    uint8_t *userdata;
+    size_t userdata_len;
 };
 
 /* This maps a recirculation ID to saved state that flow translation can
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index bd7dc9e7b718..c7bfa472f24a 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -187,7 +187,8 @@  enum upcall_type {
     SLOW_PATH_UPCALL,           /* Slow path upcall.  */
     SFLOW_UPCALL,               /* sFlow sample. */
     FLOW_SAMPLE_UPCALL,         /* Per-flow sampling. */
-    IPFIX_UPCALL                /* Per-bridge sampling. */
+    IPFIX_UPCALL,               /* Per-bridge sampling. */
+    CONTROLLER_UPCALL           /* Destined for the controller. */
 };
 
 enum reval_result {
@@ -1011,6 +1012,8 @@  classify_upcall(enum dpif_upcall_type type, const struct nlattr *userdata,
         return FLOW_SAMPLE_UPCALL;
     } else if (cookie->type == USER_ACTION_COOKIE_IPFIX) {
         return IPFIX_UPCALL;
+    } else if (cookie->type == USER_ACTION_COOKIE_CONTROLLER) {
+        return CONTROLLER_UPCALL;
     } else {
         VLOG_WARN_RL(&rl, "invalid user cookie of type %"PRIu16
                      " and size %"PRIuSIZE, cookie->type, userdata_len);
@@ -1024,8 +1027,8 @@  static void
 compose_slow_path(struct udpif *udpif, struct xlate_out *xout,
                   const struct flow *flow,
                   odp_port_t odp_in_port, ofp_port_t ofp_in_port,
-                  struct ofpbuf *buf, uint32_t slowpath_meter_id,
-                  uint32_t controller_meter_id, struct uuid *ofproto_uuid)
+                  struct ofpbuf *buf, uint32_t meter_id,
+                  struct uuid *ofproto_uuid)
 {
     struct user_action_cookie cookie;
     odp_port_t port;
@@ -1043,9 +1046,6 @@  compose_slow_path(struct udpif *udpif, struct xlate_out *xout,
 
     size_t offset;
     size_t ac_offset;
-    uint32_t meter_id = xout->slow & SLOW_CONTROLLER ? controller_meter_id
-                                                     : slowpath_meter_id;
-
     if (meter_id != UINT32_MAX) {
         /* If slowpath meter is configured, generate clone(meter, userspace)
          * action. */
@@ -1178,12 +1178,11 @@  upcall_xlate(struct udpif *udpif, struct upcall *upcall,
         ofpbuf_use_const(&upcall->put_actions,
                          odp_actions->data, odp_actions->size);
     } else {
-        uint32_t smid = upcall->ofproto->up.slowpath_meter_id;
-        uint32_t cmid = upcall->ofproto->up.controller_meter_id;
         /* upcall->put_actions already initialized by upcall_receive(). */
         compose_slow_path(udpif, &upcall->xout, upcall->flow,
                           upcall->flow->in_port.odp_port, upcall->ofp_in_port,
-                          &upcall->put_actions, smid, cmid,
+                          &upcall->put_actions,
+                          upcall->ofproto->up.slowpath_meter_id,
                           &upcall->ofproto->uuid);
     }
 
@@ -1340,6 +1339,7 @@  dpif_read_actions(struct udpif *udpif, struct upcall *upcall,
     case BAD_UPCALL:
     case MISS_UPCALL:
     case SLOW_PATH_UPCALL:
+    case CONTROLLER_UPCALL:
     default:
         break;
     }
@@ -1409,6 +1409,68 @@  process_upcall(struct udpif *udpif, struct upcall *upcall,
         }
         break;
 
+    case CONTROLLER_UPCALL:
+        {
+            struct user_action_cookie *cookie = &upcall->cookie;
+
+            if (cookie->controller.dont_send) {
+                return 0;
+            }
+
+            uint32_t recirc_id = cookie->controller.recirc_id;
+            if (!recirc_id) {
+                break;
+            }
+
+            const struct recirc_id_node *recirc_node
+                                = recirc_id_node_find(recirc_id);
+            if (!recirc_node) {
+                break;
+            }
+
+            struct ofproto_async_msg *am = xmalloc(sizeof *am);
+            *am = (struct ofproto_async_msg) {
+                .controller_id = cookie->controller.controller_id,
+                .oam = OAM_PACKET_IN,
+                .pin = {
+                    .up = {
+                        .base = {
+                            .packet = xmemdup(dp_packet_data(packet),
+                                              dp_packet_size(packet)),
+                            .packet_len = dp_packet_size(packet),
+                            .reason = cookie->controller.reason,
+                            .table_id = recirc_node->state.table_id,
+                            .cookie = get_32aligned_be64(
+                                         &cookie->controller.rule_cookie),
+                            .userdata = (recirc_node->state.userdata_len
+                                     ? xmemdup(recirc_node->state.userdata,
+                                               recirc_node->state.userdata_len)
+                                      : NULL),
+                            .userdata_len = recirc_node->state.userdata_len,
+                        },
+                    },
+                    .max_len = cookie->controller.max_len,
+                },
+            };
+
+            /* We don't want to use the upcall 'flow', since it may be
+             * more specific than the point at which the "controller"
+             * action was specified. */
+            struct flow frozen_flow;
+
+            frozen_flow = *flow;
+            if (!recirc_node->state.conntracked) {
+                flow_clear_conntrack(&frozen_flow);
+            }
+
+            frozen_metadata_to_flow(&recirc_node->state.metadata,
+                                    &frozen_flow);
+            flow_get_metadata(&frozen_flow, &am->pin.up.base.flow_metadata);
+
+            ofproto_dpif_send_async_msg(upcall->ofproto, am);
+        }
+        break;
+
     case BAD_UPCALL:
         break;
     }
@@ -1430,11 +1492,11 @@  handle_upcalls(struct udpif *udpif, struct upcall *upcalls,
      *     translation is what processes received packets for these
      *     protocols.
      *
-     *   - For SLOW_CONTROLLER, translation sends the packet to the OpenFlow
-     *     controller.
-     *
      *   - For SLOW_ACTION, translation executes the actions directly.
      *
+     *   - For SLOW_PAUSE, translation needs to handle a pause request
+     *     from the controller.
+     *
      * The loop fills 'ops' with an array of operations to execute in the
      * datapath. */
     n_ops = 0;
@@ -2042,15 +2104,12 @@  revalidate_ukey__(struct udpif *udpif, const struct udpif_key *ukey,
         struct ofproto_dpif *ofproto;
         ofp_port_t ofp_in_port;
 
-        ofproto = xlate_lookup_ofproto(udpif->backer, &ctx.flow,
-                                       &ofp_in_port);
-        uint32_t smid = ofproto ? ofproto->up.slowpath_meter_id : UINT32_MAX;
-        uint32_t cmid = ofproto ? ofproto->up.controller_meter_id : UINT32_MAX;
+        ofproto = xlate_lookup_ofproto(udpif->backer, &ctx.flow, &ofp_in_port);
 
         ofpbuf_clear(odp_actions);
         compose_slow_path(udpif, xoutp, &ctx.flow, ctx.flow.in_port.odp_port,
-                          ofp_in_port, odp_actions, smid, cmid,
-                          &ofproto->uuid);
+                          ofp_in_port, odp_actions,
+                          ofproto->up.slowpath_meter_id, &ofproto->uuid);
     }
 
     if (odp_flow_key_to_mask(ukey->mask, ukey->mask_len, &dp_mask, &ctx.flow)
diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
index 2b1cbddae4e2..a2520cf487a3 100644
--- a/ofproto/ofproto-dpif-xlate.c
+++ b/ofproto/ofproto-dpif-xlate.c
@@ -4355,174 +4355,13 @@  flood_packets(struct xlate_ctx *ctx, bool all, bool is_last_action)
     ctx->nf_output_iface = NF_OUT_FLOOD;
 }
 
-/* Copy and reformat a partially xlated odp actions to a new
- * odp actions list in 'b', so that the new actions list
- * can be executed by odp_execute_actions.
- *
- * When xlate using nested odp actions, such as sample and clone,
- * the nested action created by nl_msg_start_nested() may not
- * have been properly closed yet, thus can not be executed
- * directly.
- *
- * Since unclosed nested action has to be last action, it can be
- * fixed by skipping the outer header, and treating the actions within
- * as if they are outside the nested attribute since the effect
- * of executing them on packet is the same.
- *
- * As an optimization, a fully closed 'sample' or 'clone' action
- * is skipped since their execution has no effect to the packet.
- *
- * Returns true if success. 'b' contains the new actions list.
- * The caller is responsible for disposing 'b'.
- *
- * Returns false if error, 'b' has been freed already.  */
-static bool
-xlate_fixup_actions(struct ofpbuf *b, const struct nlattr *actions,
-                    size_t actions_len)
-{
-    const struct nlattr *a;
-    unsigned int left;
-
-    NL_ATTR_FOR_EACH_UNSAFE (a, left, actions, actions_len) {
-        int type = nl_attr_type(a);
-
-        switch ((enum ovs_action_attr) type) {
-        case OVS_ACTION_ATTR_HASH:
-        case OVS_ACTION_ATTR_PUSH_VLAN:
-        case OVS_ACTION_ATTR_POP_VLAN:
-        case OVS_ACTION_ATTR_PUSH_MPLS:
-        case OVS_ACTION_ATTR_POP_MPLS:
-        case OVS_ACTION_ATTR_SET:
-        case OVS_ACTION_ATTR_SET_MASKED:
-        case OVS_ACTION_ATTR_TRUNC:
-        case OVS_ACTION_ATTR_OUTPUT:
-        case OVS_ACTION_ATTR_TUNNEL_PUSH:
-        case OVS_ACTION_ATTR_TUNNEL_POP:
-        case OVS_ACTION_ATTR_USERSPACE:
-        case OVS_ACTION_ATTR_RECIRC:
-        case OVS_ACTION_ATTR_CT:
-        case OVS_ACTION_ATTR_PUSH_ETH:
-        case OVS_ACTION_ATTR_POP_ETH:
-        case OVS_ACTION_ATTR_PUSH_NSH:
-        case OVS_ACTION_ATTR_POP_NSH:
-        case OVS_ACTION_ATTR_METER:
-            ofpbuf_put(b, a, nl_attr_len_pad(a, left));
-            break;
-
-        case OVS_ACTION_ATTR_CLONE:
-            /* If the clone action has been fully xlated, it can
-             * be skipped, since any actions executed within clone
-             * do not affect the current packet.
-             *
-             * When xlating actions within clone, the clone action,
-             * because it is an nested netlink attribute, do not have
-             * a valid 'nla_len'; it will be zero instead.  Skip
-             * the clone header to find the start of the actions
-             * enclosed. Treat those actions as if they are written
-             * outside of clone.   */
-            if (!a->nla_len) {
-                bool ok;
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-
-                ok = xlate_fixup_actions(b, nl_attr_get_unspec(a, 0),
-                                         left - NLA_HDRLEN);
-                if (!ok) {
-                    goto error;
-                }
-            }
-            break;
-
-        case OVS_ACTION_ATTR_SAMPLE:
-            if (!a->nla_len) {
-                bool ok;
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-                const struct nlattr *attr = nl_attr_get_unspec(a, 0);
-                left -= NLA_HDRLEN;
-
-                while (left > 0 &&
-                       nl_attr_type(attr) != OVS_SAMPLE_ATTR_ACTIONS) {
-                    /* Only OVS_SAMPLE_ATTR_ACTIONS can have unclosed
-                     * nested netlink attribute.  */
-                    if (!attr->nla_len) {
-                        goto error;
-                    }
-
-                    left -= NLA_ALIGN(attr->nla_len);
-                    attr = nl_attr_next(attr);
-                }
-
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-
-                ok = xlate_fixup_actions(b, nl_attr_get_unspec(attr, 0),
-                                         left - NLA_HDRLEN);
-                if (!ok) {
-                    goto error;
-                }
-            }
-            break;
-
-        case OVS_ACTION_ATTR_UNSPEC:
-        case __OVS_ACTION_ATTR_MAX:
-            OVS_NOT_REACHED();
-        }
-    }
-
-    return true;
-
-error:
-    ofpbuf_delete(b);
-    return false;
-}
-
-static bool
-xlate_execute_odp_actions(struct dp_packet *packet,
-                          const struct nlattr *actions, int actions_len)
-{
-    struct dp_packet_batch batch;
-    struct ofpbuf *b = ofpbuf_new(actions_len);
-
-    if (!xlate_fixup_actions(b, actions, actions_len)) {
-        return false;
-    }
-
-    dp_packet_batch_init_packet(&batch, packet);
-    odp_execute_actions(NULL, &batch, false, b->data, b->size, NULL);
-    ofpbuf_delete(b);
-
-    return true;
-}
-
 static void
-execute_controller_action(struct xlate_ctx *ctx, int len,
-                          enum ofp_packet_in_reason reason,
-                          uint16_t controller_id,
-                          const uint8_t *userdata, size_t userdata_len)
+xlate_controller_action(struct xlate_ctx *ctx, int len,
+                        enum ofp_packet_in_reason reason,
+                        uint16_t controller_id,
+                        const uint8_t *userdata, size_t userdata_len)
 {
-    struct dp_packet *packet;
-
-    ctx->xout->slow |= SLOW_CONTROLLER;
     xlate_commit_actions(ctx);
-    if (!ctx->xin->packet) {
-        return;
-    }
-
-    if (!ctx->xin->allow_side_effects && !ctx->xin->xcache) {
-        return;
-    }
-
-    packet = dp_packet_clone(ctx->xin->packet);
-    if (!xlate_execute_odp_actions(packet, ctx->odp_actions->data,
-                                  ctx->odp_actions->size)) {
-        xlate_report_error(ctx, "Failed to execute controller action");
-        dp_packet_delete(packet);
-        return;
-    }
 
     /* A packet sent by an action in a table-miss rule is considered an
      * explicit table miss.  OpenFlow before 1.3 doesn't have that concept so
@@ -4532,44 +4371,74 @@  execute_controller_action(struct xlate_ctx *ctx, int len,
         reason = OFPR_EXPLICIT_MISS;
     }
 
-    size_t packet_len = dp_packet_size(packet);
-
-    struct ofproto_async_msg *am = xmalloc(sizeof *am);
-    *am = (struct ofproto_async_msg) {
-        .controller_id = controller_id,
-        .oam = OAM_PACKET_IN,
-        .pin = {
-            .up = {
-                .base = {
-                    .packet = dp_packet_steal_data(packet),
-                    .packet_len = packet_len,
-                    .reason = reason,
-                    .table_id = ctx->table_id,
-                    .cookie = ctx->rule_cookie,
-                    .userdata = (userdata_len
-                                 ? xmemdup(userdata, userdata_len)
-                                 : NULL),
-                    .userdata_len = userdata_len,
-                }
-            },
-            .max_len = len,
-        },
+    struct frozen_state state = {
+        .table_id = ctx->table_id,
+        .ofproto_uuid = ctx->xbridge->ofproto->uuid,
+        .stack = ctx->stack.data,
+        .stack_size = ctx->stack.size,
+        .mirrors = ctx->mirrors,
+        .conntracked = ctx->conntracked,
+        .ofpacts = NULL,
+        .ofpacts_len = 0,
+        .action_set = NULL,
+        .action_set_len = 0,
+        .userdata = CONST_CAST(uint8_t *, userdata),
+        .userdata_len = userdata_len,
     };
-    flow_get_metadata(&ctx->xin->flow, &am->pin.up.base.flow_metadata);
+    frozen_metadata_from_flow(&state.metadata, &ctx->xin->flow);
 
-    /* Async messages are only sent once, so if we send one now, no
-     * xlate cache entry is created.  */
-    if (ctx->xin->allow_side_effects) {
-        ofproto_dpif_send_async_msg(ctx->xbridge->ofproto, am);
-    } else /* xcache */ {
-        struct xc_entry *entry;
+    uint32_t recirc_id = recirc_alloc_id_ctx(&state);
+    if (!recirc_id) {
+        xlate_report_error(ctx, "Failed to allocate recirculation id");
+        ctx->error = XLATE_NO_RECIRCULATION_CONTEXT;
+        return;
+    }
+    recirc_refs_add(&ctx->xout->recircs, recirc_id);
 
-        entry = xlate_cache_add_entry(ctx->xin->xcache, XC_CONTROLLER);
-        entry->controller.ofproto = ctx->xbridge->ofproto;
-        entry->controller.am = am;
+    size_t offset;
+    size_t ac_offset;
+    uint32_t meter_id = ctx->xbridge->ofproto->up.controller_meter_id;
+    if (meter_id != UINT32_MAX) {
+        /* If controller meter is configured, generate clone(meter, userspace)
+         * action. */
+        offset = nl_msg_start_nested(ctx->odp_actions, OVS_ACTION_ATTR_SAMPLE);
+        nl_msg_put_u32(ctx->odp_actions, OVS_SAMPLE_ATTR_PROBABILITY,
+                       UINT32_MAX);
+        ac_offset = nl_msg_start_nested(ctx->odp_actions,
+                                        OVS_SAMPLE_ATTR_ACTIONS);
+        nl_msg_put_u32(ctx->odp_actions, OVS_ACTION_ATTR_METER, meter_id);
     }
 
-    dp_packet_delete(packet);
+    struct user_action_cookie cookie;
+
+    memset(&cookie, 0, sizeof cookie);
+    cookie.type = USER_ACTION_COOKIE_CONTROLLER;
+    cookie.ofp_in_port = OFPP_NONE;
+    cookie.ofproto_uuid = ctx->xbridge->ofproto->uuid;
+    cookie.controller.dont_send = false;
+    cookie.controller.reason = reason;
+    cookie.controller.recirc_id = recirc_id;
+    put_32aligned_be64(&cookie.controller.rule_cookie, ctx->rule_cookie);
+    cookie.controller.max_len = len;
+    cookie.controller.controller_id = controller_id;
+
+    /* Generate the datapath flows even if we don't send the packet-in
+     * so that debugging more closely represents normal state. */
+    if (!ctx->xin->allow_side_effects && !ctx->xin->xcache) {
+        cookie.controller.dont_send = true;
+    }
+
+    odp_port_t odp_port = ofp_port_to_odp_port(
+        ctx->xbridge, ctx->xin->flow.in_port.ofp_port);
+    uint32_t pid = dpif_port_get_pid(ctx->xbridge->dpif, odp_port,
+                                     flow_hash_5tuple(&ctx->xin->flow, 0));
+    odp_put_userspace_action(pid, &cookie, sizeof cookie, ODPP_NONE,
+                             false, ctx->odp_actions);
+
+    if (meter_id != UINT32_MAX) {
+        nl_msg_end_nested(ctx->odp_actions, ac_offset);
+        nl_msg_end_nested(ctx->odp_actions, offset);
+    }
 }
 
 static void
@@ -4782,8 +4651,8 @@  compose_dec_ttl(struct xlate_ctx *ctx, struct ofpact_cnt_ids *ids)
         size_t i;
 
         for (i = 0; i < ids->n_controllers; i++) {
-            execute_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL,
-                                      ids->cnt_ids[i], NULL, 0);
+            xlate_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL,
+                                    ids->cnt_ids[i], NULL, 0);
         }
 
         /* Stop processing for current table. */
@@ -4834,8 +4703,8 @@  compose_dec_mpls_ttl_action(struct xlate_ctx *ctx)
             set_mpls_lse_ttl(&flow->mpls_lse[0], ttl);
             return false;
         } else {
-            execute_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL, 0,
-                                      NULL, 0);
+            xlate_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL, 0,
+                                    NULL, 0);
         }
     }
 
@@ -4888,12 +4757,12 @@  xlate_output_action(struct xlate_ctx *ctx, ofp_port_t port,
         flood_packets(ctx, true, is_last_action);
         break;
     case OFPP_CONTROLLER:
-        execute_controller_action(ctx, controller_len,
-                                  (ctx->in_packet_out ? OFPR_PACKET_OUT
-                                   : ctx->in_group ? OFPR_GROUP
-                                   : ctx->in_action_set ? OFPR_ACTION_SET
-                                   : OFPR_ACTION),
-                                  0, NULL, 0);
+        xlate_controller_action(ctx, controller_len,
+                                (ctx->in_packet_out ? OFPR_PACKET_OUT
+                                 : ctx->in_group ? OFPR_GROUP
+                                 : ctx->in_action_set ? OFPR_ACTION_SET
+                                 : OFPR_ACTION),
+                                0, NULL, 0);
         break;
     case OFPP_NONE:
         break;
@@ -6251,16 +6120,16 @@  do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             controller = ofpact_get_CONTROLLER(a);
             if (controller->pause) {
                 ctx->pause = controller;
-                ctx->xout->slow |= SLOW_CONTROLLER;
+                ctx->xout->slow |= SLOW_PAUSE;
                 *ctx->paused_flow = ctx->xin->flow;
                 ctx_trigger_freeze(ctx);
                 a = ofpact_next(a);
             } else {
-                execute_controller_action(ctx, controller->max_len,
-                                          controller->reason,
-                                          controller->controller_id,
-                                          controller->userdata,
-                                          controller->userdata_len);
+                xlate_controller_action(ctx, controller->max_len,
+                                        controller->reason,
+                                        controller->controller_id,
+                                        controller->userdata,
+                                        controller->userdata_len);
             }
             break;
 
diff --git a/ofproto/ofproto-unixctl.man b/ofproto/ofproto-unixctl.man
index f511c392b548..ee1f81fceaec 100644
--- a/ofproto/ofproto-unixctl.man
+++ b/ofproto/ofproto-unixctl.man
@@ -107,9 +107,9 @@  effects when a packet is specified.  If you want side effects to take
 place, then you must supply a packet.
 .
 .IP
-(Output actions are obviously side effects too, but
-the trace commands never execute them, even when one specifies a
-packet.)
+(Side effects when tracing do not have external consequences.  Even if a
+packet is specified, a trace will not output a packet or generate sFlow,
+NetFlow or controller events.)
 .
 .IP "Incomplete information."
 Most of the time, Open vSwitch can figure out everything about the
diff --git a/tests/odp.at b/tests/odp.at
index 4891653eb81a..1f8080aed1ed 100644
--- a/tests/odp.at
+++ b/tests/odp.at
@@ -260,6 +260,8 @@  userspace(pid=6633,flow_sample(probability=123,collector_set_id=1234,obs_domain_
 userspace(pid=6633,flow_sample(probability=123,collector_set_id=1234,obs_domain_id=2345,obs_point_id=3456,output_port=10,egress),tunnel_out_port=10)
 userspace(pid=6633,ipfix(output_port=10))
 userspace(pid=6633,ipfix(output_port=10),tunnel_out_port=10)
+userspace(pid=6633,controller(reason=1,dont_send=0,recirc_id=4444,rule_cookie=0x5555,controller_id=0,max_len=65535))
+userspace(pid=6633,controller(reason=1,dont_send=1,recirc_id=4444,rule_cookie=0x5555,controller_id=0,max_len=65535))
 set(in_port(2))
 set(eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15))
 set(eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15/ff:ff:ff:00:00:00))
diff --git a/tests/ofproto-dpif.at b/tests/ofproto-dpif.at
index d9dab9ba159a..4c092bc2e3d5 100644
--- a/tests/ofproto-dpif.at
+++ b/tests/ofproto-dpif.at
@@ -781,11 +781,10 @@  table=1 in_port=1 action=dec_ttl,output:3
 ])
 AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
 AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)' -generate], [0], [stdout])
-AT_CHECK([tail -4 stdout], [0],
-  [Megaflow: recirc_id=0,eth,ip,in_port=1,nw_ttl=2,nw_frag=no
-Datapath actions: set(ipv4(ttl=1)),2,4
-This flow is handled by the userspace slow path because it:
-	- Sends "packet-in" messages to the OpenFlow controller.
+AT_CHECK([tail -4 stdout], [0], [
+Final flow: ip,in_port=1,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_proto=111,nw_tos=0,nw_ecn=0,nw_ttl=1
+Megaflow: recirc_id=0,eth,ip,in_port=1,nw_ttl=2,nw_frag=no
+Datapath actions: set(ipv4(ttl=1)),2,userspace(pid=0,controller(reason=2,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535)),4
 ])
 AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=3,frag=no)'], [0], [stdout])
 AT_CHECK([tail -2 stdout], [0],
@@ -800,7 +799,9 @@  Datapath actions: set(ipv6(hlimit=127)),2,set(ipv6(hlimit=126)),3,4
 
 AT_CAPTURE_FILE([ofctl_monitor.log])
 AT_CHECK([ovs-ofctl -P nxt_packet_in monitor br0 65534 invalid_ttl --detach --no-chdir --pidfile 2> ofctl_monitor.log])
-AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)' -generate], [0], [stdout])
+
+ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)'
+
 OVS_APP_EXIT_AND_WAIT([ovs-ofctl])
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
 NXT_PACKET_IN (xid=0x0): table_id=1 cookie=0x0 total_len=34 in_port=1 (via invalid_ttl) data_len=34 (unbuffered)
@@ -1628,6 +1629,37 @@  NXST_FLOW reply:
 OVS_VSWITCHD_STOP
 AT_CLEANUP
 
+
+AT_SETUP([ofproto-dpif - controller with slow-path action])
+OVS_VSWITCHD_START
+add_of_ports br0 1 2
+
+AT_CHECK([ovs-ofctl add-flow br0 "in_port=1,actions=debug_slow,controller"])
+
+AT_CHECK([ovs-ofctl monitor -P standard br0 65534 --detach --no-chdir --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3 ; do
+    ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=6,tos=0,ttl=64,frag=no),tcp(src=8,dst=9),tcp_flags(0x010)'
+done
+
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 3])
+OVS_APP_EXIT_AND_WAIT(ovs-ofctl)
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+])
+
+OVS_VSWITCHD_STOP
+AT_CLEANUP
+
+
 AT_SETUP([ofproto-dpif - controller action without megaflows])
 OVS_VSWITCHD_START
 add_of_ports br0 1
@@ -1649,7 +1681,7 @@  OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
 AT_CHECK([ovs-appctl dpctl/dump-flows | sed 's/.*\(packets:\)/\1/' | sed 's/used:[[0-9]].[[0-9]]*s/used:0.001s/'], [0], [dnl
 flow-dump from non-dpdk interfaces:
-packets:1, bytes:14, used:0.001s, actions:userspace(pid=0,slow_path(controller))
+packets:1, bytes:14, used:0.001s, actions:userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
@@ -1663,7 +1695,7 @@  AT_CHECK([ovs-appctl revalidator/purge])
 AT_CHECK([ovs-ofctl monitor br0 65534 invalid_ttl -P nxt_packet_in --detach --no-chdir --pidfile 2> ofctl_monitor.log])
 
 dnl Add a controller meter.
-AT_CHECK([ovs-ofctl -O OpenFlow13 add-meter br0 'meter=controller pktps stats bands=type=drop rate=1'])
+AT_CHECK([ovs-ofctl -O OpenFlow13 add-meter br0 'meter=controller pktps stats bands=type=drop rate=2'])
 
 dnl Advance time by 1 second.
 AT_CHECK([ovs-appctl time/warp 1000], [0], [ignore])
@@ -1674,15 +1706,14 @@  done
 
 AT_CHECK([ovs-appctl dpctl/dump-flows | sed 's/.*\(packets:\)/\1/' | sed 's/used:[[0-9]].[[0-9]]*s/used:0.001s/'], [0], [dnl
 flow-dump from non-dpdk interfaces:
-packets:7, bytes:98, used:0.001s, actions:sample(sample=100.0%,actions(meter(0),userspace(pid=0,slow_path(controller))))
+packets:7, bytes:98, used:0.001s, actions:sample(sample=100.0%,actions(meter(0),userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=2,rule_cookie=0,controller_id=0,max_len=65535))))
 ])
 
 AT_CHECK([ovs-appctl time/warp 1], [0], [ignore])
 OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
-dnl Out of 8 packets we sent, one executes the controller action via
-dnl miss upcall. Another one got passed the rate limiter.
-dnl The rest of packets are blocked by the rate limiter.
+dnl Out of 8 packets we sent, two were passed by the rate limiter, and
+dnl the rest of packets were blocked.
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
 NXT_PACKET_IN (xid=0x0): cookie=0x0 total_len=14 in_port=1 (via action) data_len=14 (unbuffered)
 vlan_tci=0x0000,dl_src=50:54:00:00:00:09,dl_dst=50:54:00:00:00:0a,dl_type=0x4321
@@ -1693,7 +1724,7 @@  dnl Check meter stats to make it gives the same picture;
 dnl 7 packets hit the meter, but 6 packets are dropped by band0.
 AT_CHECK([ovs-ofctl -O OpenFlow13 meter-stats br0 | strip_timers], [0], [dnl
 OFPST_METER reply (OF1.3) (xid=0x2):
-meter:controller flow_count:0 packet_in_count:7 byte_in_count:98 duration:0.0s bands:
+meter:controller flow_count:0 packet_in_count:8 byte_in_count:112 duration:0.0s bands:
 0: packet_count:6 byte_count:84
 ])
 
@@ -7398,8 +7429,8 @@  for dl_src in 00 01; do
 done
 sleep 1  # wait for the datapath flow installed
 AT_CHECK_UNQUOTED([strip_ufid < ovs-vswitchd.log | filter_flow_install | strip_used], [0], [dnl
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:userspace(pid=0,slow_path(controller))
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:userspace(pid=0,slow_path(controller))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:push_mpls(label=20,tc=0,ttl=32,bos=0,eth_type=0x8847),userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:pop_mpls(eth_type=0x8847),userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 OVS_VSWITCHD_STOP
@@ -7437,8 +7468,8 @@  for dl_src in 00 01; do
 done
 sleep 1  # wait for the datapath flow installed
 AT_CHECK_UNQUOTED([strip_ufid < ovs-vswitchd.log | filter_flow_install | strip_used], [0], [dnl
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:userspace(pid=0,slow_path(controller))
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:userspace(pid=0,slow_path(controller))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:push_mpls(label=20,tc=0,ttl=32,bos=0,eth_type=0x8847),userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:pop_mpls(eth_type=0x8847),userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 OVS_VSWITCHD_STOP
diff --git a/tests/pmd.at b/tests/pmd.at
index e39a23a43386..adfdb4c01e78 100644
--- a/tests/pmd.at
+++ b/tests/pmd.at
@@ -303,8 +303,8 @@  OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
 dnl Make sure that both flows have been installed
 AT_CHECK([ovs-appctl dpctl/dump-flows | flow_dump_prepend_pmd], [0], [dnl
-0 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
-1 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+0 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
+1 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
diff --git a/tests/tunnel-push-pop-ipv6.at b/tests/tunnel-push-pop-ipv6.at
index 8f51c067ebd4..4d609b2f1944 100644
--- a/tests/tunnel-push-pop-ipv6.at
+++ b/tests/tunnel-push-pop-ipv6.at
@@ -164,7 +164,7 @@  AT_CHECK([ovs-ofctl dump-ports int-br | grep 'port  5'], [0], [dnl
   port  5: rx pkts=1, bytes=98, drop=?, errs=?, frame=?, over=?, crc=?
 ])
 AT_CHECK([ovs-appctl dpif/dump-flows int-br | grep 'in_port(6081)'], [0], [dnl
-tunnel(tun_id=0x7b,ipv6_src=2001:cafe::92,ipv6_dst=2001:cafe::88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+tunnel(tun_id=0x7b,ipv6_src=2001:cafe::92,ipv6_dst=2001:cafe::88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=3,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 ovs-appctl time/warp 10000
diff --git a/tests/tunnel-push-pop.at b/tests/tunnel-push-pop.at
index 5f2f172a2f7e..cdfbb36a47d9 100644
--- a/tests/tunnel-push-pop.at
+++ b/tests/tunnel-push-pop.at
@@ -215,7 +215,7 @@  AT_CHECK([ovs-ofctl dump-ports int-br | grep 'port  5'], [0], [dnl
   port  5: rx pkts=1, bytes=98, drop=?, errs=?, frame=?, over=?, crc=?
 ])
 AT_CHECK([ovs-appctl dpif/dump-flows int-br | grep 'in_port(6081)'], [0], [dnl
-tunnel(tun_id=0x7b,src=1.1.2.92,dst=1.1.2.88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+tunnel(tun_id=0x7b,src=1.1.2.92,dst=1.1.2.88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,dont_send=0,recirc_id=3,rule_cookie=0,controller_id=0,max_len=65535))
 ])
 
 ovs-appctl time/warp 10000