[ovs-dev,no-slow,5/6] ofproto-dpif: Don't slow-path controller actions.

Message ID 1513895115-35879-5-git-send-email-jpettit@ovn.org
State New
Headers show
Series
  • [ovs-dev,no-slow,1/6] ofproto-dpif: Add ability to look up an ofproto by UUID.
Related show

Commit Message

Justin Pettit Dec. 21, 2017, 10:25 p.m.
Controller actions have become more commonly used for purposes other
than just making forwarding decisions (e.g., packet logging).  A packet
that needs to be copied to the controller and forwarded would always be
sent to ovs-vswitchd to be handled, which could negatively affect
performance and cause heavier CPU utilization in ovs-vswitchd.

This commit changes the behavior so that OpenFlow controller actions
become userspace datapath actions while continuing to let packet
forwarding and manipulation continue to be handled by the datapath
directly.

This patch still slow-paths controller actions with the "pause" flag
set.  A future patch will stop slow-pathing these pause actions as
well.

Signed-off-by: Justin Pettit <jpettit@ovn.org>
---
 NEWS                          |   1 +
 lib/odp-util.c                |  27 ++++
 lib/odp-util.h                |  24 +++-
 ofproto/ofproto-dpif-upcall.c |  99 +++++++++++---
 ofproto/ofproto-dpif-xlate.c  | 298 ++++++++++++------------------------------
 ofproto/ofproto-unixctl.man   |   6 +-
 tests/ofproto-dpif.at         |  65 ++++++---
 tests/pmd.at                  |   4 +-
 tests/tunnel-push-pop-ipv6.at |   2 +-
 tests/tunnel-push-pop.at      |   2 +-
 10 files changed, 268 insertions(+), 260 deletions(-)

Comments

Ben Pfaff Jan. 2, 2018, 6:36 p.m. | #1
On Thu, Dec 21, 2017 at 02:25:14PM -0800, Justin Pettit wrote:
> Controller actions have become more commonly used for purposes other
> than just making forwarding decisions (e.g., packet logging).  A packet
> that needs to be copied to the controller and forwarded would always be
> sent to ovs-vswitchd to be handled, which could negatively affect
> performance and cause heavier CPU utilization in ovs-vswitchd.
> 
> This commit changes the behavior so that OpenFlow controller actions
> become userspace datapath actions while continuing to let packet
> forwarding and manipulation continue to be handled by the datapath
> directly.
> 
> This patch still slow-paths controller actions with the "pause" flag
> set.  A future patch will stop slow-pathing these pause actions as
> well.
> 
> Signed-off-by: Justin Pettit <jpettit@ovn.org>

checkpatch says:

    WARNING: Line length is >79-characters long
    #22 FILE: lib/odp-util.c:482:
                           && cookie.header.type == USER_ACTION_COOKIE_CONTROLLER) {

This is missing code from parse_odp_userspace_action() that should be
able to parse the formatted actions.

I would add a test to "OVS datapath actions parsing and formatting -
valid forms" in tests/odp.at for parsing and formatting the new kind of
userspace action.

This is a nice simplification!  I thought it would make the code bigger
and more complicated, but it shrank it (at least in the worst areas) and
made it simpler.

Acked-by: Ben Pfaff <blp@ovn.org>
Ben Pfaff Jan. 4, 2018, 12:57 a.m. | #2
On Thu, Dec 21, 2017 at 02:25:14PM -0800, Justin Pettit wrote:
> Controller actions have become more commonly used for purposes other
> than just making forwarding decisions (e.g., packet logging).  A packet
> that needs to be copied to the controller and forwarded would always be
> sent to ovs-vswitchd to be handled, which could negatively affect
> performance and cause heavier CPU utilization in ovs-vswitchd.
> 
> This commit changes the behavior so that OpenFlow controller actions
> become userspace datapath actions while continuing to let packet
> forwarding and manipulation continue to be handled by the datapath
> directly.
> 
> This patch still slow-paths controller actions with the "pause" flag
> set.  A future patch will stop slow-pathing these pause actions as
> well.
> 
> Signed-off-by: Justin Pettit <jpettit@ovn.org>

I took a second look at this patch, by request.

The NEWS item doesn't explain what this feature means to users.

It wasn't clear to me whether UACF_DONT_SEND re-implements existing
behavior or introduces a behavioral change.  If it introduces a change,
maybe one of the tutorials (like the Faucet tutorial?) relies on it, so
we might need to change them.

I don't know how many flags you expect.  If it's only one or two, maybe
they should just be "bool"s.  (There's no ABI to freeze here, so they
could be converted to a flag word if they got bigger.)

In xlate_controller_action(), a comment refers to the slowpath meter,
but I think that it should refer to the controller meter.

I wonder whether some of the data in the user_action_cookie for
controller actions should actually be put into the frozen_state.  I
think that all of it other than the recirc_id could actually go in the
frozen_state, but the userdata in particular can have an arbitrary size,
which in extremis might make the datapath action too large and which we
don't really need to copy between the kernel and userspace repeatedly.

The comment on struct user_action_cookie here seems inappropriate, since
I believe that recirc_id will always be nonzero:
        uint32_t recirc_id;     /* Non-zero if recirc id allocated. */

Thanks,

Ben.
Justin Pettit Jan. 10, 2018, 6:44 a.m. | #3
> On Jan 2, 2018, at 10:36 AM, Ben Pfaff <blp@ovn.org> wrote:
> 
> On Thu, Dec 21, 2017 at 02:25:14PM -0800, Justin Pettit wrote:
>> Controller actions have become more commonly used for purposes other
>> than just making forwarding decisions (e.g., packet logging).  A packet
>> that needs to be copied to the controller and forwarded would always be
>> sent to ovs-vswitchd to be handled, which could negatively affect
>> performance and cause heavier CPU utilization in ovs-vswitchd.
>> 
>> This commit changes the behavior so that OpenFlow controller actions
>> become userspace datapath actions while continuing to let packet
>> forwarding and manipulation continue to be handled by the datapath
>> directly.
>> 
>> This patch still slow-paths controller actions with the "pause" flag
>> set.  A future patch will stop slow-pathing these pause actions as
>> well.
>> 
>> Signed-off-by: Justin Pettit <jpettit@ovn.org>
> 
> checkpatch says:
> 
>    WARNING: Line length is >79-characters long
>    #22 FILE: lib/odp-util.c:482:
>                           && cookie.header.type == USER_ACTION_COOKIE_CONTROLLER) {

Fixed.

> This is missing code from parse_odp_userspace_action() that should be
> able to parse the formatted actions.
> 
> I would add a test to "OVS datapath actions parsing and formatting -
> valid forms" in tests/odp.at for parsing and formatting the new kind of
> userspace action.

Good catch.  I've added those to v2.

> This is a nice simplification!  I thought it would make the code bigger
> and more complicated, but it shrank it (at least in the worst areas) and
> made it simpler.

Thanks!

--Justin
Justin Pettit Jan. 10, 2018, 10:28 p.m. | #4
> On Jan 3, 2018, at 4:57 PM, Ben Pfaff <blp@ovn.org> wrote:
> 
> On Thu, Dec 21, 2017 at 02:25:14PM -0800, Justin Pettit wrote:
>> Controller actions have become more commonly used for purposes other
>> than just making forwarding decisions (e.g., packet logging).  A packet
>> that needs to be copied to the controller and forwarded would always be
>> sent to ovs-vswitchd to be handled, which could negatively affect
>> performance and cause heavier CPU utilization in ovs-vswitchd.
>> 
>> This commit changes the behavior so that OpenFlow controller actions
>> become userspace datapath actions while continuing to let packet
>> forwarding and manipulation continue to be handled by the datapath
>> directly.
>> 
>> This patch still slow-paths controller actions with the "pause" flag
>> set.  A future patch will stop slow-pathing these pause actions as
>> well.
>> 
>> Signed-off-by: Justin Pettit <jpettit@ovn.org>
> 
> I took a second look at this patch, by request.
> 
> The NEWS item doesn't explain what this feature means to users.

Done.

> It wasn't clear to me whether UACF_DONT_SEND re-implements existing
> behavior or introduces a behavioral change.  If it introduces a change,
> maybe one of the tutorials (like the Faucet tutorial?) relies on it, so
> we might need to change them.

This should only be visible when looking at the datapath flows and tracing traffic.  I did go through the Faucet tutorial, though, and made some changes to the ofproto/trace output.

> I don't know how many flags you expect.  If it's only one or two, maybe
> they should just be "bool"s.  (There's no ABI to freeze here, so they
> could be converted to a flag word if they got bigger.)

Okay, I went ahead and did that.

> In xlate_controller_action(), a comment refers to the slowpath meter,
> but I think that it should refer to the controller meter.

Fixed.

> I wonder whether some of the data in the user_action_cookie for
> controller actions should actually be put into the frozen_state.  I
> think that all of it other than the recirc_id could actually go in the
> frozen_state, but the userdata in particular can have an arbitrary size,
> which in extremis might make the datapath action too large and which we
> don't really need to copy between the kernel and userspace repeatedly.

Good point.  I reworked the series to use a fixed length userspace cookie and moved the userdata into the frozen state.

> The comment on struct user_action_cookie here seems inappropriate, since
> I believe that recirc_id will always be nonzero:
>        uint32_t recirc_id;     /* Non-zero if recirc id allocated. */

Agreed.  I dropped it.

I'll send out v2 in just a moment.

Thanks for the reviews!

--Justin

Patch

diff --git a/NEWS b/NEWS
index 89bb7a7dee2a..41e1b8fbed23 100644
--- a/NEWS
+++ b/NEWS
@@ -86,6 +86,7 @@  v2.8.0 - 31 Aug 2017
      * New support for role-based access control (see ovsdb-server(1)).
    - New commands 'stp/show' and 'rstp/show' (see ovs-vswitchd(8)).
    - OpenFlow:
+     * No longer slow-path traffic that sends to a controller.
      * All features required by OpenFlow 1.4 are now implemented, so
        ovs-vswitchd now enables OpenFlow 1.4 by default (in addition to
        OpenFlow 1.0 to 1.3).
diff --git a/lib/odp-util.c b/lib/odp-util.c
index 173a868e0728..1b567e55e6f7 100644
--- a/lib/odp-util.c
+++ b/lib/odp-util.c
@@ -477,6 +477,33 @@  format_odp_userspace_action(struct ds *ds, const struct nlattr *attr,
                 odp_portno_name_format(portno_names,
                                        cookie.ipfix.output_odp_port, ds);
                 ds_put_char(ds, ')');
+            } else if (userdata_len >= sizeof cookie.controller
+                       && cookie.header.type == USER_ACTION_COOKIE_CONTROLLER) {
+                ds_put_format(ds, ",controller(reason=%"PRIu16
+                              ",flags=%"PRIu16
+                              ",recirc_id=%"PRIu16
+                              ",rule_cookie=%"PRIx64
+                              ",controller_id=%"PRIu16
+                              ",max_len=%"PRIu16
+                              ",userdata_len=%"PRIu16,
+                              cookie.controller.reason,
+                              cookie.controller.flags,
+                              cookie.controller.recirc_id,
+                              ntohll(get_32aligned_be64(
+                                         &cookie.controller.rule_cookie)),
+                              cookie.controller.controller_id,
+                              cookie.controller.max_len,
+                              cookie.controller.userdata_len);
+                if (cookie.controller.userdata_len) {
+                    size_t i;
+                    ds_put_format(ds, ",userdata(");
+                    for (i = 0; i < cookie.controller.userdata_len; i++) {
+                        ds_put_format(ds, "%02x",
+                                      cookie.controller.userdata[i]);
+                    }
+                    ds_put_char(ds, ')');
+                }
+                ds_put_char(ds, ')');
             } else {
                 userdata_unspec = true;
             }
diff --git a/lib/odp-util.h b/lib/odp-util.h
index 040334695835..ff9ecf00e3c2 100644
--- a/lib/odp-util.h
+++ b/lib/odp-util.h
@@ -42,9 +42,8 @@  struct pkt_metadata;
     SPR(SLOW_BFD,        "bfd",        "Consists of BFD packets")       \
     SPR(SLOW_LACP,       "lacp",       "Consists of LACP packets")      \
     SPR(SLOW_STP,        "stp",        "Consists of STP packets")       \
-    SPR(SLOW_LLDP,       "lldp",       "Consists of LLDP packets")    \
-    SPR(SLOW_CONTROLLER, "controller",                                  \
-        "Sends \"packet-in\" messages to the OpenFlow controller")      \
+    SPR(SLOW_LLDP,       "lldp",       "Consists of LLDP packets")      \
+    SPR(SLOW_PAUSE,      "pause",      "Controller action with pause")  \
     SPR(SLOW_ACTION,     "action",                                      \
         "Uses action(s) not supported by datapath")
 
@@ -294,6 +293,12 @@  enum user_action_cookie_type {
     USER_ACTION_COOKIE_SLOW_PATH,    /* Userspace must process this flow. */
     USER_ACTION_COOKIE_FLOW_SAMPLE,  /* Packet for per-flow sampling. */
     USER_ACTION_COOKIE_IPFIX,        /* Packet for per-bridge IPFIX sampling. */
+    USER_ACTION_COOKIE_CONTROLLER,   /* Forward packet to controller. */
+};
+
+/* Flags for controller cookies. */
+enum user_action_controller_flags {
+    UACF_DONT_SEND    = 1 << 0,      /* Don't send the packet to controller. */
 };
 
 struct user_action_cookie_header {
@@ -335,6 +340,19 @@  union user_action_cookie {
         struct user_action_cookie_header header;
         odp_port_t output_odp_port; /* The output odp port. */
     } ipfix;
+
+    struct {
+        /* USER_ACTION_COOKIE_CONTROLLER. */
+        struct user_action_cookie_header header;
+        uint16_t flags;         /* UACF_* */
+        uint16_t reason;
+        uint32_t recirc_id;     /* Non-zero if recirc id allocated. */
+        ovs_32aligned_be64 rule_cookie;
+        uint16_t controller_id;
+        uint16_t max_len;
+        uint16_t userdata_len;
+        uint8_t userdata[0];
+    } controller;
 };
 BUILD_ASSERT_DECL(sizeof(union user_action_cookie) == 48);
 
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 88501803b5b8..405a3f74368e 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -187,7 +187,8 @@  enum upcall_type {
     SLOW_PATH_UPCALL,           /* Slow path upcall.  */
     SFLOW_UPCALL,               /* sFlow sample. */
     FLOW_SAMPLE_UPCALL,         /* Per-flow sampling. */
-    IPFIX_UPCALL                /* Per-bridge sampling. */
+    IPFIX_UPCALL,               /* Per-bridge sampling. */
+    CONTROLLER_UPCALL           /* Destined for the controller. */
 };
 
 enum reval_result {
@@ -998,9 +999,9 @@  classify_upcall(enum dpif_upcall_type type, const struct nlattr *userdata,
         VLOG_WARN_RL(&rl, "action upcall missing cookie");
         return BAD_UPCALL;
     }
+
     userdata_len = nl_attr_get_size(userdata);
-    if (userdata_len < sizeof cookie->header.type
-        || userdata_len > sizeof *cookie) {
+    if (userdata_len < sizeof cookie->header.type) {
         VLOG_WARN_RL(&rl, "action upcall cookie has unexpected size %"PRIuSIZE,
                      userdata_len);
         return BAD_UPCALL;
@@ -1020,6 +1021,9 @@  classify_upcall(enum dpif_upcall_type type, const struct nlattr *userdata,
     } else if (userdata_len == sizeof cookie->ipfix
                && cookie->header.type == USER_ACTION_COOKIE_IPFIX) {
         return IPFIX_UPCALL;
+    } else if (userdata_len >= sizeof cookie->controller
+               && cookie->header.type == USER_ACTION_COOKIE_CONTROLLER) {
+        return CONTROLLER_UPCALL;
     } else {
         VLOG_WARN_RL(&rl, "invalid user cookie of type %"PRIu16
                      " and size %"PRIuSIZE, cookie->header.type, userdata_len);
@@ -1033,8 +1037,8 @@  static void
 compose_slow_path(struct udpif *udpif, struct xlate_out *xout,
                   const struct flow *flow,
                   odp_port_t odp_in_port, ofp_port_t ofp_in_port,
-                  struct ofpbuf *buf, uint32_t slowpath_meter_id,
-                  uint32_t controller_meter_id, struct uuid *ofproto_uuid)
+                  struct ofpbuf *buf, uint32_t meter_id,
+                  struct uuid *ofproto_uuid)
 {
     union user_action_cookie cookie;
     odp_port_t port;
@@ -1052,9 +1056,6 @@  compose_slow_path(struct udpif *udpif, struct xlate_out *xout,
 
     size_t offset;
     size_t ac_offset;
-    uint32_t meter_id = xout->slow & SLOW_CONTROLLER ? controller_meter_id
-                                                     : slowpath_meter_id;
-
     if (meter_id != UINT32_MAX) {
         /* If slowpath meter is configured, generate clone(meter, userspace)
          * action. */
@@ -1187,12 +1188,11 @@  upcall_xlate(struct udpif *udpif, struct upcall *upcall,
         ofpbuf_use_const(&upcall->put_actions,
                          odp_actions->data, odp_actions->size);
     } else {
-        uint32_t smid = upcall->ofproto->up.slowpath_meter_id;
-        uint32_t cmid = upcall->ofproto->up.controller_meter_id;
         /* upcall->put_actions already initialized by upcall_receive(). */
         compose_slow_path(udpif, &upcall->xout, upcall->flow,
                           upcall->flow->in_port.odp_port, upcall->ofp_in_port,
-                          &upcall->put_actions, smid, cmid,
+                          &upcall->put_actions,
+                          upcall->ofproto->up.slowpath_meter_id,
                           &upcall->ofproto->uuid);
     }
 
@@ -1349,6 +1349,7 @@  dpif_read_actions(struct udpif *udpif, struct upcall *upcall,
     case BAD_UPCALL:
     case MISS_UPCALL:
     case SLOW_PATH_UPCALL:
+    case CONTROLLER_UPCALL:
     default:
         break;
     }
@@ -1429,6 +1430,68 @@  process_upcall(struct udpif *udpif, struct upcall *upcall,
         }
         break;
 
+    case CONTROLLER_UPCALL:
+        {
+            union user_action_cookie *cookie = &upcall->cookie;
+
+            if (cookie->controller.flags & UACF_DONT_SEND) {
+                return 0;
+            }
+
+            uint32_t recirc_id = cookie->controller.recirc_id;
+            if (!recirc_id) {
+                break;
+            }
+
+            const struct recirc_id_node *recirc_node
+                                = recirc_id_node_find(recirc_id);
+            if (!recirc_node) {
+                break;
+            }
+
+            struct ofproto_async_msg *am = xmalloc(sizeof *am);
+            *am = (struct ofproto_async_msg) {
+                .controller_id = cookie->controller.controller_id,
+                .oam = OAM_PACKET_IN,
+                .pin = {
+                    .up = {
+                        .base = {
+                            .packet = xmemdup(dp_packet_data(packet),
+                                              dp_packet_size(packet)),
+                            .packet_len = dp_packet_size(packet),
+                            .reason = cookie->controller.reason,
+                            .table_id = recirc_node->state.table_id,
+                            .cookie = get_32aligned_be64(
+                                         &cookie->controller.rule_cookie),
+                            .userdata = (cookie->controller.userdata_len
+                                     ? xmemdup(cookie->controller.userdata,
+                                               cookie->controller.userdata_len)
+                                      : NULL),
+                            .userdata_len = cookie->controller.userdata_len,
+                        },
+                    },
+                    .max_len = cookie->controller.max_len,
+                },
+            };
+
+            /* We don't want to use the upcall 'flow', since it may be
+             * more specific than the point at which the "controller"
+             * action was specified. */
+            struct flow frozen_flow;
+
+            frozen_flow = *flow;
+            if (!recirc_node->state.conntracked) {
+                flow_clear_conntrack(&frozen_flow);
+            }
+
+            frozen_metadata_to_flow(&recirc_node->state.metadata,
+                                    &frozen_flow);
+            flow_get_metadata(&frozen_flow, &am->pin.up.base.flow_metadata);
+
+            ofproto_dpif_send_async_msg(upcall->ofproto, am);
+        }
+        break;
+
     case BAD_UPCALL:
         break;
     }
@@ -1450,11 +1513,11 @@  handle_upcalls(struct udpif *udpif, struct upcall *upcalls,
      *     translation is what processes received packets for these
      *     protocols.
      *
-     *   - For SLOW_CONTROLLER, translation sends the packet to the OpenFlow
-     *     controller.
-     *
      *   - For SLOW_ACTION, translation executes the actions directly.
      *
+     *   - For SLOW_PAUSE, translation needs to handle a pause request
+     *     from the controller.
+     *
      * The loop fills 'ops' with an array of operations to execute in the
      * datapath. */
     n_ops = 0;
@@ -2062,14 +2125,12 @@  revalidate_ukey__(struct udpif *udpif, const struct udpif_key *ukey,
         struct ofproto_dpif *ofproto;
         ofp_port_t ofp_in_port;
 
-        ofproto = xlate_lookup_ofproto(udpif->backer, &ctx.flow,
-                                       &ofp_in_port);
-        uint32_t smid = ofproto ? ofproto->up.slowpath_meter_id : UINT32_MAX;
-        uint32_t cmid = ofproto ? ofproto->up.controller_meter_id : UINT32_MAX;
+        ofproto = xlate_lookup_ofproto(udpif->backer, &ctx.flow, &ofp_in_port);
 
         ofpbuf_clear(odp_actions);
         compose_slow_path(udpif, xoutp, &ctx.flow, ctx.flow.in_port.odp_port,
-                          ofp_in_port, odp_actions, smid, cmid, &ofproto->uuid);
+                          ofp_in_port, odp_actions,
+                          ofproto->up.slowpath_meter_id, &ofproto->uuid);
     }
 
     if (odp_flow_key_to_mask(ukey->mask, ukey->mask_len, &dp_mask, &ctx.flow)
diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
index ec92c6a730bf..9ec8d54be5ed 100644
--- a/ofproto/ofproto-dpif-xlate.c
+++ b/ofproto/ofproto-dpif-xlate.c
@@ -4345,174 +4345,13 @@  flood_packets(struct xlate_ctx *ctx, bool all, bool is_last_action)
     ctx->nf_output_iface = NF_OUT_FLOOD;
 }
 
-/* Copy and reformat a partially xlated odp actions to a new
- * odp actions list in 'b', so that the new actions list
- * can be executed by odp_execute_actions.
- *
- * When xlate using nested odp actions, such as sample and clone,
- * the nested action created by nl_msg_start_nested() may not
- * have been properly closed yet, thus can not be executed
- * directly.
- *
- * Since unclosed nested action has to be last action, it can be
- * fixed by skipping the outer header, and treating the actions within
- * as if they are outside the nested attribute since the effect
- * of executing them on packet is the same.
- *
- * As an optimization, a fully closed 'sample' or 'clone' action
- * is skipped since their execution has no effect to the packet.
- *
- * Returns true if success. 'b' contains the new actions list.
- * The caller is responsible for disposing 'b'.
- *
- * Returns false if error, 'b' has been freed already.  */
-static bool
-xlate_fixup_actions(struct ofpbuf *b, const struct nlattr *actions,
-                    size_t actions_len)
-{
-    const struct nlattr *a;
-    unsigned int left;
-
-    NL_ATTR_FOR_EACH_UNSAFE (a, left, actions, actions_len) {
-        int type = nl_attr_type(a);
-
-        switch ((enum ovs_action_attr) type) {
-        case OVS_ACTION_ATTR_HASH:
-        case OVS_ACTION_ATTR_PUSH_VLAN:
-        case OVS_ACTION_ATTR_POP_VLAN:
-        case OVS_ACTION_ATTR_PUSH_MPLS:
-        case OVS_ACTION_ATTR_POP_MPLS:
-        case OVS_ACTION_ATTR_SET:
-        case OVS_ACTION_ATTR_SET_MASKED:
-        case OVS_ACTION_ATTR_TRUNC:
-        case OVS_ACTION_ATTR_OUTPUT:
-        case OVS_ACTION_ATTR_TUNNEL_PUSH:
-        case OVS_ACTION_ATTR_TUNNEL_POP:
-        case OVS_ACTION_ATTR_USERSPACE:
-        case OVS_ACTION_ATTR_RECIRC:
-        case OVS_ACTION_ATTR_CT:
-        case OVS_ACTION_ATTR_PUSH_ETH:
-        case OVS_ACTION_ATTR_POP_ETH:
-        case OVS_ACTION_ATTR_ENCAP_NSH:
-        case OVS_ACTION_ATTR_DECAP_NSH:
-        case OVS_ACTION_ATTR_METER:
-            ofpbuf_put(b, a, nl_attr_len_pad(a, left));
-            break;
-
-        case OVS_ACTION_ATTR_CLONE:
-            /* If the clone action has been fully xlated, it can
-             * be skipped, since any actions executed within clone
-             * do not affect the current packet.
-             *
-             * When xlating actions within clone, the clone action,
-             * because it is an nested netlink attribute, do not have
-             * a valid 'nla_len'; it will be zero instead.  Skip
-             * the clone header to find the start of the actions
-             * enclosed. Treat those actions as if they are written
-             * outside of clone.   */
-            if (!a->nla_len) {
-                bool ok;
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-
-                ok = xlate_fixup_actions(b, nl_attr_get_unspec(a, 0),
-                                         left - NLA_HDRLEN);
-                if (!ok) {
-                    goto error;
-                }
-            }
-            break;
-
-        case OVS_ACTION_ATTR_SAMPLE:
-            if (!a->nla_len) {
-                bool ok;
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-                const struct nlattr *attr = nl_attr_get_unspec(a, 0);
-                left -= NLA_HDRLEN;
-
-                while (left > 0 &&
-                       nl_attr_type(attr) != OVS_SAMPLE_ATTR_ACTIONS) {
-                    /* Only OVS_SAMPLE_ATTR_ACTIONS can have unclosed
-                     * nested netlink attribute.  */
-                    if (!attr->nla_len) {
-                        goto error;
-                    }
-
-                    left -= NLA_ALIGN(attr->nla_len);
-                    attr = nl_attr_next(attr);
-                }
-
-                if (left < NLA_HDRLEN) {
-                    goto error;
-                }
-
-                ok = xlate_fixup_actions(b, nl_attr_get_unspec(attr, 0),
-                                         left - NLA_HDRLEN);
-                if (!ok) {
-                    goto error;
-                }
-            }
-            break;
-
-        case OVS_ACTION_ATTR_UNSPEC:
-        case __OVS_ACTION_ATTR_MAX:
-            OVS_NOT_REACHED();
-        }
-    }
-
-    return true;
-
-error:
-    ofpbuf_delete(b);
-    return false;
-}
-
-static bool
-xlate_execute_odp_actions(struct dp_packet *packet,
-                          const struct nlattr *actions, int actions_len)
-{
-    struct dp_packet_batch batch;
-    struct ofpbuf *b = ofpbuf_new(actions_len);
-
-    if (!xlate_fixup_actions(b, actions, actions_len)) {
-        return false;
-    }
-
-    dp_packet_batch_init_packet(&batch, packet);
-    odp_execute_actions(NULL, &batch, false, b->data, b->size, NULL);
-    ofpbuf_delete(b);
-
-    return true;
-}
-
 static void
-execute_controller_action(struct xlate_ctx *ctx, int len,
-                          enum ofp_packet_in_reason reason,
-                          uint16_t controller_id,
-                          const uint8_t *userdata, size_t userdata_len)
+xlate_controller_action(struct xlate_ctx *ctx, int len,
+                        enum ofp_packet_in_reason reason,
+                        uint16_t controller_id,
+                        const uint8_t *userdata, size_t userdata_len)
 {
-    struct dp_packet *packet;
-
-    ctx->xout->slow |= SLOW_CONTROLLER;
     xlate_commit_actions(ctx);
-    if (!ctx->xin->packet) {
-        return;
-    }
-
-    if (!ctx->xin->allow_side_effects && !ctx->xin->xcache) {
-        return;
-    }
-
-    packet = dp_packet_clone(ctx->xin->packet);
-    if (!xlate_execute_odp_actions(packet, ctx->odp_actions->data,
-                                  ctx->odp_actions->size)) {
-        xlate_report_error(ctx, "Failed to execute controller action");
-        dp_packet_delete(packet);
-        return;
-    }
 
     /* A packet sent by an action in a table-miss rule is considered an
      * explicit table miss.  OpenFlow before 1.3 doesn't have that concept so
@@ -4522,44 +4361,75 @@  execute_controller_action(struct xlate_ctx *ctx, int len,
         reason = OFPR_EXPLICIT_MISS;
     }
 
-    size_t packet_len = dp_packet_size(packet);
-
-    struct ofproto_async_msg *am = xmalloc(sizeof *am);
-    *am = (struct ofproto_async_msg) {
-        .controller_id = controller_id,
-        .oam = OAM_PACKET_IN,
-        .pin = {
-            .up = {
-                .base = {
-                    .packet = dp_packet_steal_data(packet),
-                    .packet_len = packet_len,
-                    .reason = reason,
-                    .table_id = ctx->table_id,
-                    .cookie = ctx->rule_cookie,
-                    .userdata = (userdata_len
-                                 ? xmemdup(userdata, userdata_len)
-                                 : NULL),
-                    .userdata_len = userdata_len,
-                }
-            },
-            .max_len = len,
-        },
+    struct frozen_state state = {
+        .table_id = ctx->table_id,
+        .ofproto_uuid = ctx->xbridge->ofproto->uuid,
+        .stack = ctx->stack.data,
+        .stack_size = ctx->stack.size,
+        .mirrors = ctx->mirrors,
+        .conntracked = ctx->conntracked,
+        .ofpacts = NULL,
+        .ofpacts_len = 0,
+        .action_set = NULL,
+        .action_set_len = 0
     };
-    flow_get_metadata(&ctx->xin->flow, &am->pin.up.base.flow_metadata);
+    frozen_metadata_from_flow(&state.metadata, &ctx->xin->flow);
 
-    /* Async messages are only sent once, so if we send one now, no
-     * xlate cache entry is created.  */
-    if (ctx->xin->allow_side_effects) {
-        ofproto_dpif_send_async_msg(ctx->xbridge->ofproto, am);
-    } else /* xcache */ {
-        struct xc_entry *entry;
+    uint32_t recirc_id = recirc_alloc_id_ctx(&state);
+    if (!recirc_id) {
+        xlate_report_error(ctx, "Failed to allocate recirculation id");
+        ctx->error = XLATE_NO_RECIRCULATION_CONTEXT;
+        return;
+    }
+    recirc_refs_add(&ctx->xout->recircs, recirc_id);
 
-        entry = xlate_cache_add_entry(ctx->xin->xcache, XC_CONTROLLER);
-        entry->controller.ofproto = ctx->xbridge->ofproto;
-        entry->controller.am = am;
+    size_t offset;
+    size_t ac_offset;
+    uint32_t meter_id = ctx->xbridge->ofproto->up.controller_meter_id;
+    if (meter_id != UINT32_MAX) {
+        /* If slowpath meter is configured, generate clone(meter, userspace)
+         * action. */
+        offset = nl_msg_start_nested(ctx->odp_actions, OVS_ACTION_ATTR_SAMPLE);
+        nl_msg_put_u32(ctx->odp_actions, OVS_SAMPLE_ATTR_PROBABILITY,
+                       UINT32_MAX);
+        ac_offset = nl_msg_start_nested(ctx->odp_actions,
+                                        OVS_SAMPLE_ATTR_ACTIONS);
+        nl_msg_put_u32(ctx->odp_actions, OVS_ACTION_ATTR_METER, meter_id);
+    }
+
+    union user_action_cookie *cookie;
+    cookie = xmalloc(sizeof cookie->controller + userdata_len);
+
+    memset(&cookie->controller, 0, sizeof cookie->controller);
+    cookie->header.type = USER_ACTION_COOKIE_CONTROLLER,
+    cookie->header.ofproto_uuid = ctx->xbridge->ofproto->uuid,
+    cookie->controller.recirc_id = recirc_id,
+    cookie->controller.max_len = len,
+    cookie->controller.controller_id = controller_id,
+    cookie->controller.reason = reason,
+    cookie->controller.userdata_len = userdata_len,
+    put_32aligned_be64(&cookie->controller.rule_cookie, ctx->rule_cookie);
+    memcpy(cookie->controller.userdata, userdata, userdata_len);
+
+    /* Generate the datapath flows even if we don't send the packet-in
+     * so that debugging more closely represents normal state. */
+    if (!ctx->xin->allow_side_effects && !ctx->xin->xcache) {
+        cookie->controller.flags |= UACF_DONT_SEND;
     }
 
-    dp_packet_delete(packet);
+    odp_port_t odp_port = ofp_port_to_odp_port(
+        ctx->xbridge, ctx->xin->flow.in_port.ofp_port);
+    uint32_t pid = dpif_port_get_pid(ctx->xbridge->dpif, odp_port,
+                                     flow_hash_5tuple(&ctx->xin->flow, 0));
+    odp_put_userspace_action(pid, cookie,
+                             sizeof cookie->controller + userdata_len,
+                             ODPP_NONE, false, ctx->odp_actions);
+    free(cookie);
+
+    if (meter_id != UINT32_MAX) {
+        nl_msg_end_nested(ctx->odp_actions, ac_offset);
+        nl_msg_end_nested(ctx->odp_actions, offset);
+    }
 }
 
 static void
@@ -4772,8 +4642,8 @@  compose_dec_ttl(struct xlate_ctx *ctx, struct ofpact_cnt_ids *ids)
         size_t i;
 
         for (i = 0; i < ids->n_controllers; i++) {
-            execute_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL,
-                                      ids->cnt_ids[i], NULL, 0);
+            xlate_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL,
+                                    ids->cnt_ids[i], NULL, 0);
         }
 
         /* Stop processing for current table. */
@@ -4824,8 +4694,8 @@  compose_dec_mpls_ttl_action(struct xlate_ctx *ctx)
             set_mpls_lse_ttl(&flow->mpls_lse[0], ttl);
             return false;
         } else {
-            execute_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL, 0,
-                                      NULL, 0);
+            xlate_controller_action(ctx, UINT16_MAX, OFPR_INVALID_TTL, 0,
+                                    NULL, 0);
         }
     }
 
@@ -4878,12 +4748,12 @@  xlate_output_action(struct xlate_ctx *ctx, ofp_port_t port,
         flood_packets(ctx, true, is_last_action);
         break;
     case OFPP_CONTROLLER:
-        execute_controller_action(ctx, controller_len,
-                                  (ctx->in_packet_out ? OFPR_PACKET_OUT
-                                   : ctx->in_group ? OFPR_GROUP
-                                   : ctx->in_action_set ? OFPR_ACTION_SET
-                                   : OFPR_ACTION),
-                                  0, NULL, 0);
+        xlate_controller_action(ctx, controller_len,
+                                (ctx->in_packet_out ? OFPR_PACKET_OUT
+                                 : ctx->in_group ? OFPR_GROUP
+                                 : ctx->in_action_set ? OFPR_ACTION_SET
+                                 : OFPR_ACTION),
+                                0, NULL, 0);
         break;
     case OFPP_NONE:
         break;
@@ -6249,16 +6119,16 @@  do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             controller = ofpact_get_CONTROLLER(a);
             if (controller->pause) {
                 ctx->pause = controller;
-                ctx->xout->slow |= SLOW_CONTROLLER;
+                ctx->xout->slow |= SLOW_PAUSE;
                 *ctx->paused_flow = ctx->xin->flow;
                 ctx_trigger_freeze(ctx);
                 a = ofpact_next(a);
             } else {
-                execute_controller_action(ctx, controller->max_len,
-                                          controller->reason,
-                                          controller->controller_id,
-                                          controller->userdata,
-                                          controller->userdata_len);
+                xlate_controller_action(ctx, controller->max_len,
+                                        controller->reason,
+                                        controller->controller_id,
+                                        controller->userdata,
+                                        controller->userdata_len);
             }
             break;
 
diff --git a/ofproto/ofproto-unixctl.man b/ofproto/ofproto-unixctl.man
index f511c392b548..ee1f81fceaec 100644
--- a/ofproto/ofproto-unixctl.man
+++ b/ofproto/ofproto-unixctl.man
@@ -107,9 +107,9 @@  effects when a packet is specified.  If you want side effects to take
 place, then you must supply a packet.
 .
 .IP
-(Output actions are obviously side effects too, but
-the trace commands never execute them, even when one specifies a
-packet.)
+(Side effects when tracing do not have external consequences.  Even if a
+packet is specified, a trace will not output a packet or generate sFlow,
+NetFlow or controller events.)
 .
 .IP "Incomplete information."
 Most of the time, Open vSwitch can figure out everything about the
diff --git a/tests/ofproto-dpif.at b/tests/ofproto-dpif.at
index d9dab9ba159a..c396889f7ffb 100644
--- a/tests/ofproto-dpif.at
+++ b/tests/ofproto-dpif.at
@@ -781,11 +781,10 @@  table=1 in_port=1 action=dec_ttl,output:3
 ])
 AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
 AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)' -generate], [0], [stdout])
-AT_CHECK([tail -4 stdout], [0],
-  [Megaflow: recirc_id=0,eth,ip,in_port=1,nw_ttl=2,nw_frag=no
-Datapath actions: set(ipv4(ttl=1)),2,4
-This flow is handled by the userspace slow path because it:
-	- Sends "packet-in" messages to the OpenFlow controller.
+AT_CHECK([tail -4 stdout], [0], [
+Final flow: ip,in_port=1,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_proto=111,nw_tos=0,nw_ecn=0,nw_ttl=1
+Megaflow: recirc_id=0,eth,ip,in_port=1,nw_ttl=2,nw_frag=no
+Datapath actions: set(ipv4(ttl=1)),2,userspace(pid=0,controller(reason=2,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0)),4
 ])
 AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=3,frag=no)'], [0], [stdout])
 AT_CHECK([tail -2 stdout], [0],
@@ -800,7 +799,9 @@  Datapath actions: set(ipv6(hlimit=127)),2,set(ipv6(hlimit=126)),3,4
 
 AT_CAPTURE_FILE([ofctl_monitor.log])
 AT_CHECK([ovs-ofctl -P nxt_packet_in monitor br0 65534 invalid_ttl --detach --no-chdir --pidfile 2> ofctl_monitor.log])
-AT_CHECK([ovs-appctl ofproto/trace ovs-dummy 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)' -generate], [0], [stdout])
+
+ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=111,tos=0,ttl=2,frag=no)'
+
 OVS_APP_EXIT_AND_WAIT([ovs-ofctl])
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
 NXT_PACKET_IN (xid=0x0): table_id=1 cookie=0x0 total_len=34 in_port=1 (via invalid_ttl) data_len=34 (unbuffered)
@@ -1628,6 +1629,37 @@  NXST_FLOW reply:
 OVS_VSWITCHD_STOP
 AT_CLEANUP
 
+
+AT_SETUP([ofproto-dpif - controller with slow-path action])
+OVS_VSWITCHD_START
+add_of_ports br0 1 2
+
+AT_CHECK([ovs-ofctl add-flow br0 "in_port=1,actions=debug_slow,controller"])
+
+AT_CHECK([ovs-ofctl monitor -P standard br0 65534 --detach --no-chdir --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3 ; do
+    ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=50:54:00:00:00:05,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=6,tos=0,ttl=64,frag=no),tcp(src=8,dst=9),tcp_flags(0x010)'
+done
+
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 3])
+OVS_APP_EXIT_AND_WAIT(ovs-ofctl)
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+dnl
+OFPT_PACKET_IN (xid=0x0): total_len=54 in_port=1 (via action) data_len=54 (unbuffered)
+tcp,vlan_tci=0x0000,dl_src=50:54:00:00:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=8,tp_dst=9,tcp_flags=ack tcp_csum:2e70
+])
+
+OVS_VSWITCHD_STOP
+AT_CLEANUP
+
+
 AT_SETUP([ofproto-dpif - controller action without megaflows])
 OVS_VSWITCHD_START
 add_of_ports br0 1
@@ -1649,7 +1681,7 @@  OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
 AT_CHECK([ovs-appctl dpctl/dump-flows | sed 's/.*\(packets:\)/\1/' | sed 's/used:[[0-9]].[[0-9]]*s/used:0.001s/'], [0], [dnl
 flow-dump from non-dpdk interfaces:
-packets:1, bytes:14, used:0.001s, actions:userspace(pid=0,slow_path(controller))
+packets:1, bytes:14, used:0.001s, actions:userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
@@ -1663,7 +1695,7 @@  AT_CHECK([ovs-appctl revalidator/purge])
 AT_CHECK([ovs-ofctl monitor br0 65534 invalid_ttl -P nxt_packet_in --detach --no-chdir --pidfile 2> ofctl_monitor.log])
 
 dnl Add a controller meter.
-AT_CHECK([ovs-ofctl -O OpenFlow13 add-meter br0 'meter=controller pktps stats bands=type=drop rate=1'])
+AT_CHECK([ovs-ofctl -O OpenFlow13 add-meter br0 'meter=controller pktps stats bands=type=drop rate=2'])
 
 dnl Advance time by 1 second.
 AT_CHECK([ovs-appctl time/warp 1000], [0], [ignore])
@@ -1674,15 +1706,14 @@  done
 
 AT_CHECK([ovs-appctl dpctl/dump-flows | sed 's/.*\(packets:\)/\1/' | sed 's/used:[[0-9]].[[0-9]]*s/used:0.001s/'], [0], [dnl
 flow-dump from non-dpdk interfaces:
-packets:7, bytes:98, used:0.001s, actions:sample(sample=100.0%,actions(meter(0),userspace(pid=0,slow_path(controller))))
+packets:7, bytes:98, used:0.001s, actions:sample(sample=100.0%,actions(meter(0),userspace(pid=0,controller(reason=1,flags=0,recirc_id=2,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))))
 ])
 
 AT_CHECK([ovs-appctl time/warp 1], [0], [ignore])
 OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
-dnl Out of 8 packets we sent, one executes the controller action via
-dnl miss upcall. Another one got passed the rate limiter.
-dnl The rest of packets are blocked by the rate limiter.
+dnl Out of 8 packets we sent, two were passed by the rate limiter, and
+dnl the rest of packets were blocked.
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
 NXT_PACKET_IN (xid=0x0): cookie=0x0 total_len=14 in_port=1 (via action) data_len=14 (unbuffered)
 vlan_tci=0x0000,dl_src=50:54:00:00:00:09,dl_dst=50:54:00:00:00:0a,dl_type=0x4321
@@ -1693,7 +1724,7 @@  dnl Check meter stats to make it gives the same picture;
 dnl 7 packets hit the meter, but 6 packets are dropped by band0.
 AT_CHECK([ovs-ofctl -O OpenFlow13 meter-stats br0 | strip_timers], [0], [dnl
 OFPST_METER reply (OF1.3) (xid=0x2):
-meter:controller flow_count:0 packet_in_count:7 byte_in_count:98 duration:0.0s bands:
+meter:controller flow_count:0 packet_in_count:8 byte_in_count:112 duration:0.0s bands:
 0: packet_count:6 byte_count:84
 ])
 
@@ -7398,8 +7429,8 @@  for dl_src in 00 01; do
 done
 sleep 1  # wait for the datapath flow installed
 AT_CHECK_UNQUOTED([strip_ufid < ovs-vswitchd.log | filter_flow_install | strip_used], [0], [dnl
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:userspace(pid=0,slow_path(controller))
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:userspace(pid=0,slow_path(controller))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:push_mpls(label=20,tc=0,ttl=32,bos=0,eth_type=0x8847),userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:pop_mpls(eth_type=0x8847),userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 OVS_VSWITCHD_STOP
@@ -7437,8 +7468,8 @@  for dl_src in 00 01; do
 done
 sleep 1  # wait for the datapath flow installed
 AT_CHECK_UNQUOTED([strip_ufid < ovs-vswitchd.log | filter_flow_install | strip_used], [0], [dnl
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:userspace(pid=0,slow_path(controller))
-recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:userspace(pid=0,slow_path(controller))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:00),eth_type(0x8847),mpls(label=20,tc=0,ttl=32,bos=0,label=20,tc=0,ttl=32,bos=1), actions:push_mpls(label=20,tc=0,ttl=32,bos=0,eth_type=0x8847),userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth(src=60:66:66:66:66:01),eth_type(0x8847),mpls(label=20/0x0,tc=0/0,ttl=32/0x0,bos=0/1,label=20/0xfffff,tc=0/7,ttl=32/0xff,bos=1/1), actions:pop_mpls(eth_type=0x8847),userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 OVS_VSWITCHD_STOP
diff --git a/tests/pmd.at b/tests/pmd.at
index e39a23a43386..848571c8206c 100644
--- a/tests/pmd.at
+++ b/tests/pmd.at
@@ -303,8 +303,8 @@  OVS_WAIT_UNTIL([ovs-appctl -t ovs-ofctl exit])
 
 dnl Make sure that both flows have been installed
 AT_CHECK([ovs-appctl dpctl/dump-flows | flow_dump_prepend_pmd], [0], [dnl
-0 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
-1 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+0 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
+1 recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,flags=0,recirc_id=1,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 AT_CHECK([cat ofctl_monitor.log], [0], [dnl
diff --git a/tests/tunnel-push-pop-ipv6.at b/tests/tunnel-push-pop-ipv6.at
index 8f51c067ebd4..571039275c91 100644
--- a/tests/tunnel-push-pop-ipv6.at
+++ b/tests/tunnel-push-pop-ipv6.at
@@ -164,7 +164,7 @@  AT_CHECK([ovs-ofctl dump-ports int-br | grep 'port  5'], [0], [dnl
   port  5: rx pkts=1, bytes=98, drop=?, errs=?, frame=?, over=?, crc=?
 ])
 AT_CHECK([ovs-appctl dpif/dump-flows int-br | grep 'in_port(6081)'], [0], [dnl
-tunnel(tun_id=0x7b,ipv6_src=2001:cafe::92,ipv6_dst=2001:cafe::88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+tunnel(tun_id=0x7b,ipv6_src=2001:cafe::92,ipv6_dst=2001:cafe::88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,flags=0,recirc_id=3,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 ovs-appctl time/warp 10000
diff --git a/tests/tunnel-push-pop.at b/tests/tunnel-push-pop.at
index 5f2f172a2f7e..fff8e5c3cfb6 100644
--- a/tests/tunnel-push-pop.at
+++ b/tests/tunnel-push-pop.at
@@ -215,7 +215,7 @@  AT_CHECK([ovs-ofctl dump-ports int-br | grep 'port  5'], [0], [dnl
   port  5: rx pkts=1, bytes=98, drop=?, errs=?, frame=?, over=?, crc=?
 ])
 AT_CHECK([ovs-appctl dpif/dump-flows int-br | grep 'in_port(6081)'], [0], [dnl
-tunnel(tun_id=0x7b,src=1.1.2.92,dst=1.1.2.88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(controller))
+tunnel(tun_id=0x7b,src=1.1.2.92,dst=1.1.2.88,geneve({class=0xffff,type=0x80,len=4,0xa/0xf}{class=0xffff,type=0,len=4}),flags(-df-csum+key)),recirc_id(0),in_port(6081),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:userspace(pid=0,controller(reason=1,flags=0,recirc_id=3,rule_cookie=0,controller_id=0,max_len=65535,userdata_len=0))
 ])
 
 ovs-appctl time/warp 10000