From patchwork Fri Jul 6 10:40:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Doyle via dev X-Patchwork-Id: 940384 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="FI36y+c6"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41MWHY2mmZz9s4b for ; Fri, 6 Jul 2018 20:31:53 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id BF7E7CBE; Fri, 6 Jul 2018 10:31:19 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id EECE7C9A for ; Fri, 6 Jul 2018 10:31:17 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 2C55276E for ; Fri, 6 Jul 2018 10:31:17 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id s11-v6so3681603wra.13 for ; Fri, 06 Jul 2018 03:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=h5mBogxX/X/fTP8ln/tuPPBIkfrNG8zWnFRwvoCjhoY=; b=FI36y+c6LlJWUk7gTI133j9+Dc+62NxHJSRR1Z5CbU/235EEbZh+zKbso5dyIvc8oI nSXzCrYGAZ9wNkHzTswzfomw9bPMNU7PU1vCNhqBna1M/0pfkXlvuAztBrgh/j4ogwMp GYb6ZVpDVrrpA2KiWIMYtNBywzQAbYI2OaV0A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=h5mBogxX/X/fTP8ln/tuPPBIkfrNG8zWnFRwvoCjhoY=; b=BL4s5mMdNRPMH7z/vwipiKSyZNMddEQR11jPQEJQDliy0oNnT+jj6IQi04pwwGCvN4 IgKZaOb/TZDobZtsJeIJyDVHlB3V4D65ukyAheFZ3VC5xm8w5k9lWn/TO6DJ8SdOLQg9 KAv68IpfBFY/9bliO0TvFAWPYZRpXPMnLl085BSXYsxHGgPUlxGfm74TxMTpnmog9WwO wLAurKt7QV6VvqxkQsm8dW0fLNzR1BN1yotakJfCBw2iApJAUsDX+DNggMZqbKOJhK2i Ez/5hSsXj8+4p1G/m5Y4J5rEQcYTn6weFh70WLgcdQoitM0iFGrfQyM46hDpOKc88hVf H5rA== X-Gm-Message-State: APt69E3VTRxU6CQ70/28YLzyMlcHtEio2K5lbempOoURLRMUvlFqtOA7 kkalGiwhtsNjRLv71/NxzdULyNkISj8= X-Google-Smtp-Source: AAOMgpc2vLKKEfcb0IDzjD9l4Qt+u/iiwkvMdkbbCgs+uDD1FpqMyf1Lp/tH0GRcV8DzL7Be7PG6wA== X-Received: by 2002:adf:a9f0:: with SMTP id b103-v6mr6880955wrd.271.1530873075388; Fri, 06 Jul 2018 03:31:15 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d102-v6sm14903866wma.10.2018.07.06.03.31.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jul 2018 03:31:14 -0700 (PDT) To: dev@openvswitch.org Date: Fri, 6 Jul 2018 16:10:47 +0530 Message-Id: <20180706104050.24938-2-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> References: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 1/4] ovs-dev: Detect Out-Of-Resource condition on a netdev X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: Brendan Doyle via dev Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the first patch in the patch-set to support dynamic rebalancing of offloaded flows. The patch detects OOR condition on a netdev port when ENOSPC error is returned by TC-Flower while adding a flow rule. A new structure is added to the netdev called "netdev_hw_info", to store OOR related information required to perform dynamic offload-rebalancing. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-netlink.c | 23 ++++++++++++++++++----- lib/flow.c | 27 +++++++++++++++++++++++++++ lib/flow.h | 1 + lib/netdev-provider.h | 7 +++++++ lib/netdev.c | 2 ++ 5 files changed, 55 insertions(+), 5 deletions(-) diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index aa9bbd946..e20096116 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -1004,7 +1004,6 @@ dpif_netlink_rtnl_port_create_and_add(struct dpif_netlink *dpif, return error; } -static int dpif_netlink_port_add(struct dpif *dpif_, struct netdev *netdev, odp_port_t *port_nop) { @@ -2097,11 +2096,16 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) { static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20); const struct dpif_class *dpif_class = dpif->dpif.dpif_class; + struct netdev_hw_info *hw_info; struct match match; odp_port_t in_port; + odp_port_t out_port; const struct nlattr *nla; size_t left; struct netdev *dev; + struct netdev *outdev = NULL; + struct netdev *tunnel_netdev = NULL; + struct netdev *oor_netdev = NULL; struct offload_info info; ovs_be16 dst_port = 0; int err; @@ -2131,8 +2135,6 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) NL_ATTR_FOR_EACH(nla, left, put->actions, put->actions_len) { if (nl_attr_type(nla) == OVS_ACTION_ATTR_OUTPUT) { const struct netdev_tunnel_config *tnl_cfg; - struct netdev *outdev; - odp_port_t out_port; out_port = nl_attr_get_odp_port(nla); outdev = netdev_ports_get(out_port, dpif_class); @@ -2144,7 +2146,6 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) if (tnl_cfg && tnl_cfg->dst_port != 0) { dst_port = tnl_cfg->dst_port; } - netdev_close(outdev); } } @@ -2175,7 +2176,18 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) VLOG_DBG("added flow"); } else if (err != EEXIST) { - VLOG_ERR_RL(&rl, "failed to offload flow: %s", ovs_strerror(err)); + if (outdev && dev && (err == ENOSPC)) { + tunnel_netdev = flow_get_tunnel_netdev(&match.flow.tunnel); + if (tunnel_netdev) { + oor_netdev = tunnel_netdev; + } else { + oor_netdev = dev; + } + hw_info = &oor_netdev->hw_info; + hw_info->oor = true; + } + VLOG_ERR_RL(&rl, "failed to offload flow: %s: %s", ovs_strerror(err), + (oor_netdev ? oor_netdev->name : dev->name)); } out: @@ -2196,6 +2208,7 @@ out: } } + netdev_close(outdev); netdev_close(dev); return err; diff --git a/lib/flow.c b/lib/flow.c index 75ca45672..912afc99d 100644 --- a/lib/flow.c +++ b/lib/flow.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -41,6 +42,8 @@ #include "unaligned.h" #include "util.h" #include "openvswitch/nsh.h" +#include "ovs-router.h" +#include "lib/netdev-provider.h" COVERAGE_DEFINE(flow_extract); COVERAGE_DEFINE(miniflow_malloc); @@ -3299,3 +3302,27 @@ flow_limit_vlans(int vlan_limit) flow_vlan_limit = MIN(vlan_limit, FLOW_MAX_VLAN_HEADERS); } } + +struct netdev * +flow_get_tunnel_netdev(struct flow_tnl *tunnel) +{ + struct netdev *tunnel_netdev; + char iface[IFNAMSIZ]; + struct in6_addr ip6; + struct in6_addr gw; + + if (tunnel->ip_src) { + in6_addr_set_mapped_ipv4(&ip6, tunnel->ip_src); + } else if (ipv6_addr_is_set(&tunnel->ipv6_src)) { + ip6 = tunnel->ipv6_src; + } else { + return NULL; + } + + if (!ovs_router_lookup(0, &ip6, iface, NULL, &gw)) { + return (NULL); + } + + tunnel_netdev = netdev_from_name(iface); + return tunnel_netdev; +} diff --git a/lib/flow.h b/lib/flow.h index 5b6585f11..a67abc9c9 100644 --- a/lib/flow.h +++ b/lib/flow.h @@ -73,6 +73,7 @@ void flow_extract(struct dp_packet *, struct flow *); void flow_zero_wildcards(struct flow *, const struct flow_wildcards *); void flow_unwildcard_tp_ports(const struct flow *, struct flow_wildcards *); void flow_get_metadata(const struct flow *, struct match *flow_metadata); +struct netdev *flow_get_tunnel_netdev(struct flow_tnl *tunnel); const char *ct_state_to_string(uint32_t state); uint32_t ct_state_from_string(const char *); diff --git a/lib/netdev-provider.h b/lib/netdev-provider.h index 1a572f5b8..62e05619e 100644 --- a/lib/netdev-provider.h +++ b/lib/netdev-provider.h @@ -35,6 +35,11 @@ extern "C" { struct netdev_tnl_build_header_params; #define NETDEV_NUMA_UNSPEC OVS_NUMA_UNSPEC +/* Offload-capable (HW) netdev information */ +struct netdev_hw_info { + bool oor; /* Out of Offload Resources ? */ +}; + /* A network device (e.g. an Ethernet device). * * Network device implementations may read these members but should not modify @@ -80,6 +85,8 @@ struct netdev { int n_rxq; struct shash_node *node; /* Pointer to element in global map. */ struct ovs_list saved_flags_list; /* Contains "struct netdev_saved_flags". */ + + struct netdev_hw_info hw_info; /* offload-capable netdev info */ }; static inline void diff --git a/lib/netdev.c b/lib/netdev.c index 82ffeb901..b17f0563f 100644 --- a/lib/netdev.c +++ b/lib/netdev.c @@ -415,6 +415,7 @@ netdev_open(const char *name, const char *type, struct netdev **netdevp) netdev->reconfigure_seq = seq_create(); netdev->last_reconfigure_seq = seq_read(netdev->reconfigure_seq); + netdev->hw_info.oor = false; netdev->node = shash_add(&netdev_shash, name, netdev); /* By default enable one tx and rx queue per netdev. */ @@ -2472,6 +2473,7 @@ netdev_free_custom_stats_counters(struct netdev_custom_stats *custom_stats) } #ifdef __linux__ + static void netdev_ports_flow_init(void) { From patchwork Fri Jul 6 10:40:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Doyle via dev X-Patchwork-Id: 940386 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="RinMJ90q"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41MWJH4YZYz9s29 for ; Fri, 6 Jul 2018 20:32:31 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 813D9CBA; Fri, 6 Jul 2018 10:31:22 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 47AC2CAF for ; Fri, 6 Jul 2018 10:31:21 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 50DC376E for ; Fri, 6 Jul 2018 10:31:20 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id h10-v6so3689781wrq.8 for ; Fri, 06 Jul 2018 03:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PUQI9icz2HgX9Ix8u6ic3CF+ft5hlGdn7A+4fHZZbpI=; b=RinMJ90qjzYinatIOsCg8MT4h6MQB1JclfnXk5T5zXMr5GycUZGl1Lnt4EYKLnfmva 5A8tGbaqWRRqfZxPAKbbNPBZJjPQzd7TUlYRxU7KJinDfyR2s/ynoDIeS8I0kUqgZtMZ L9RwXSFG3hzmke/p6lMfTmY9aEeSd8dq3b7h8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PUQI9icz2HgX9Ix8u6ic3CF+ft5hlGdn7A+4fHZZbpI=; b=W1Fstcf+3XVSItLyoEm8OBZ+OS/5UNkl+LKK39IsK1N6Y1GNOY+wSHYPRxlTj2fDJI EX4f1MGfblkw9veR7yNxIVoxP2I2RL6fibtko2DnUdkFMJUAK3IN6DfiylAev3lLmO8q F0rrDuDnqy3AeOUjwHW2t0uBzddxUeuVrgQnzIrRCL8c0TFiVV92/1cHT39iRq4rQ/lu OpJ6WcT8cIhbsKN9zOrrkst2NlV/yVspfQxmuxq3gl9QZEkSSyPzxmh8XPox23z4SGni /R6P5SQ7MxUWQXf5sX++8HR4/kl6nI57A47tZvMC3uWe9ceSLQfpnT38fDOG6vQnhWqU cfKA== X-Gm-Message-State: APt69E169CC27kxYMciZzj2v1STFiaDzZzK5+sKj6/x6PVCF6w2cJLcd vMwRHICGtNTnMicSKDL45fsA99DbJy4= X-Google-Smtp-Source: AAOMgpedu8e49yuOh/w4ary4sdiovekCdQGdy8QG7Bvkjse50pFSi/X0XaLcDJqyoZ9Jcb0slr7ZWA== X-Received: by 2002:adf:9d81:: with SMTP id p1-v6mr7286923wre.12.1530873078586; Fri, 06 Jul 2018 03:31:18 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d102-v6sm14903866wma.10.2018.07.06.03.31.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jul 2018 03:31:17 -0700 (PDT) To: dev@openvswitch.org Date: Fri, 6 Jul 2018 16:10:48 +0530 Message-Id: <20180706104050.24938-3-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> References: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 2/4] ovs-dev: Gather packets-per-second rate of flows X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: Brendan Doyle via dev Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the second patch in the patch-set to support dynamic rebalancing of offloaded flows. The packets-per-second (pps) rate for each flow is computed in the context of revalidator threads when the flow stats are retrieved. The pps-rate is computed only after a flow is revalidated and is not scheduled for deletion. The parameters used to compute pps and the pps itself are saved in udpif_key since they need to be persisted across iterations of rebalancing. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-provider.h | 1 + ofproto/ofproto-dpif-upcall.c | 157 ++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+) diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index 62b3598ac..cc6571b28 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -39,6 +39,7 @@ struct dpif { char *full_name; uint8_t netflow_engine_type; uint8_t netflow_engine_id; + long long int current_ms; }; void dpif_init(struct dpif *, const struct dpif_class *, const char *name, diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 85f579251..6f4101f36 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -42,6 +42,7 @@ #include "tunnel.h" #include "unixctl.h" #include "openvswitch/vlog.h" +#include "lib/dpif-provider.h" #define MAX_QUEUE_LENGTH 512 #define UPCALL_MAX_BATCH 64 @@ -304,6 +305,16 @@ struct udpif_key { uint32_t key_recirc_id; /* Non-zero if reference is held by the ukey. */ struct recirc_refs recircs; /* Action recirc IDs with references held. */ + +#define OFFL_REBAL_INTVL_MSEC 3000 /* dynamic offload rebalance freq */ + struct netdev *in_netdev; /* in_odp_port's netdev */ + struct netdev *out_netdev; /* out_odp_port's netdev */ + struct netdev *tunnel_netdev; /* tunnel netdev */ + bool offloaded; /* True if flow is offloaded */ + float flow_pps_rate; /* Packets-Per-Second rate */ + long long int flow_time; /* last pps update time */ + uint64_t flow_packets; /* #pkts seen in interval */ + uint64_t flow_backlog_packets; /* prev-mode #pkts (offl or kernel) */ }; /* Datapath operation with optional ukey attached. */ @@ -1667,6 +1678,11 @@ ukey_create__(const struct nlattr *key, size_t key_len, ukey->stats.used = used; ukey->xcache = NULL; + ukey->offloaded = false; + ukey->flow_time = 0; + ukey->in_netdev = ukey->out_netdev = ukey->tunnel_netdev = NULL; + ukey->flow_packets = ukey->flow_backlog_packets = 0; + ukey->key_recirc_id = key_recirc_id; recirc_refs_init(&ukey->recircs); if (xout) { @@ -2442,6 +2458,143 @@ reval_op_init(struct ukey_op *op, enum reval_result result, } } +/* + * Given a dpif_flow, get its input and output ports (netdevs) by parsing + * the flow keys and actions. Save them in udpif_key. + */ +static void +dpif_flow_to_netdevs(struct dpif *dpif, struct udpif_key *ukey, + struct dpif_flow *f) +{ + const struct dpif_class *dpif_class = dpif->dpif_class; + odp_port_t out_port = ODPP_NONE; + odp_port_t in_port = ODPP_NONE; + const struct nlattr *a; + const struct nlattr *k; + bool forward = false; + unsigned int left; + + ukey->in_netdev = NULL; + ukey->out_netdev = NULL; + + /* Capture the output port */ + NL_ATTR_FOR_EACH (a, left, f->actions, f->actions_len) { + enum ovs_action_attr type = nl_attr_type(a); + if (type == OVS_ACTION_ATTR_OUTPUT) { + /* Only unicast is supported */ + if (forward) { + ukey->out_netdev = NULL; + ukey->in_netdev = NULL; + return; + } + + forward = true; + out_port = nl_attr_get_odp_port(a); + ukey->out_netdev = netdev_ports_get(out_port, dpif_class); + break; + } + } + + /* Now find the input port */ + NL_ATTR_FOR_EACH (k, left, f->key, f->key_len) { + unsigned int type; + struct flow_tnl tnl; + enum odp_key_fitness res; + + type = nl_attr_type(k); + + switch (type) { + case OVS_KEY_ATTR_IN_PORT: + in_port = *(odp_port_t *)nl_attr_get(k); + ukey->in_netdev = netdev_ports_get(in_port, dpif_class); + VLOG_DBG("%s: in_netdev: %s\n", __func__, ukey->in_netdev->name); + break; + case OVS_KEY_ATTR_TUNNEL: + res = odp_tun_key_from_attr(k, &tnl); + if (res == ODP_FIT_ERROR) { + ukey->tunnel_netdev = NULL; + break; + } + ukey->tunnel_netdev = flow_get_tunnel_netdev(&tnl); + VLOG_DBG("%s: tunnel_netdev: %s\n", __func__, + ukey->tunnel_netdev->name); + break; + default: + break; + } + } +} + +/* + * Given a udpif_key and a corresponding dpif_flow, get its input and output + * ports (netdevs). The flow may not contain flow attributes if is a terse + * dump; read its attributes from the ukey and then parse the flow to get + * the port info. + */ +static void +udpif_key_to_flow_netdevs(struct udpif *udpif, struct udpif_key *ukey, + struct dpif_flow *f) +{ + const struct nlattr *actions; + size_t actions_len; + + /* Read flow keys and masks from ukey */ + f->key = ukey->key; + f->key_len = ukey->key_len; + f->mask = ukey->mask; + f->mask_len = ukey->mask_len; + + /* Read actions from ukey */ + ukey_get_actions(ukey, &actions, &actions_len); + f->actions = actions; + f->actions_len = actions_len; + + /* Get netdev info for the flow now */ + dpif_flow_to_netdevs(udpif->dpif, ukey, f); +} + +static float +udpif_flow_packet_delta(struct udpif_key *ukey, const struct dpif_flow *f) +{ + return ((float)((f->stats.n_packets + ukey->flow_backlog_packets) - + ukey->flow_packets)); +} + +static long long int +udpif_flow_time_delta(struct udpif *udpif, struct udpif_key *ukey) +{ + return (udpif->dpif->current_ms - ukey->flow_time) / 1000; +} + +/* Gather pps-rate for the given dpif_flow and save it in its ukey */ +static void +udpif_update_flow_pps(struct udpif *udpif, struct udpif_key *ukey, + const struct dpif_flow *f) +{ + struct dpif_flow tmp_flow; + float pps; + + ukey->offloaded = f->attrs.offloaded; + udpif_key_to_flow_netdevs(udpif, ukey, &tmp_flow); + + if ((udpif->dpif->current_ms - ukey->flow_time) <= OFFL_REBAL_INTVL_MSEC) { + return; + } + if ((f->stats.n_packets + ukey->flow_backlog_packets) < + ukey->flow_packets) { + VLOG_DBG("%s: Current pktcnt: %lu < Saved pktcnt: %lu \n", + __func__, f->stats.n_packets + ukey->flow_backlog_packets, + ukey->flow_packets); + return; + } + + pps = udpif_flow_packet_delta(ukey, f) / + udpif_flow_time_delta(udpif, ukey); + ukey->flow_pps_rate = pps; + ukey->flow_packets = ukey->flow_backlog_packets + f->stats.n_packets; + ukey->flow_time = udpif->dpif->current_ms; +} + static void revalidate(struct revalidator *revalidator) { @@ -2550,6 +2703,10 @@ revalidate(struct revalidator *revalidator) } ukey->dump_seq = dump_seq; + if (result != UKEY_DELETE) { + udpif_update_flow_pps(udpif, ukey, f); + } + if (result != UKEY_KEEP) { /* Takes ownership of 'recircs'. */ reval_op_init(&ops[n_ops++], result, udpif, ukey, &recircs, From patchwork Fri Jul 6 10:40:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Doyle via dev X-Patchwork-Id: 940387 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="KPohBJZD"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41MWJl2nbDz9s29 for ; Fri, 6 Jul 2018 20:32:55 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 6E5B1CC2; Fri, 6 Jul 2018 10:31:27 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 7BB5FCAA for ; Fri, 6 Jul 2018 10:31:26 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 87728772 for ; Fri, 6 Jul 2018 10:31:24 +0000 (UTC) Received: by mail-wm0-f66.google.com with SMTP id z6-v6so5029002wma.0 for ; Fri, 06 Jul 2018 03:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WW9+8fRuJSSIm0w8jKGUyLR9JZ3IJLja+56YIOLVUC8=; b=KPohBJZDaNe7ArPcNW+36OWkQVSj0Xlchfs0ZSlmAkueFNVscWxo26J/A3ekLwXP3g /RxoBkAoyxmL0u4nQPrWTf8Fr9vUnXpoOfB/Ii4pXXavaP92XyfTih4TYVYwiHG6unJ1 EdlZwVxvy4EAGG7+pqPAxLFUFFz+NkCULMlRw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WW9+8fRuJSSIm0w8jKGUyLR9JZ3IJLja+56YIOLVUC8=; b=WFSc6viPbD5rzbgjuRIRpi5gaKK1v/6pzsgMGRlinl8PkkBtYjd53W1dljoEdHx82v A99yymM7GJMrE9DcMsg40g/9eGYLw07YM0WCJIJLhfIm/4Cfc2ZkFr7dwdc7vBFVnQQs yDIOa6O0LK32E6zuabdMO/dtGqNmiwmPuyPpqd8YHb+E5DezxFl2IP7yKTNI28Enr/0i AyJMs3/7C20w8QvRBSBcgtCb2lGvUqfP2iTsvB3D4lwA4BLTkgTefZVxZQ5TVQANSYTe m5fmuXX8ZX0TcZDiuRQ2gfbXPxCTR6p2PPkRQVkSzviLdRL5qWTuSDaYKree8NMwnRj6 uguA== X-Gm-Message-State: APt69E2xmaomFn3DhXsgbI3WfLPtmrPsJB7Y6/aFYuAnupzDh7LIfQnJ TLOdux6+e5JdN1NdbSTZEwTS1STRiac= X-Google-Smtp-Source: AAOMgpfy4ib75vVyleE1EfotBy07DBNCcJKSfUeVTFwOLzVR3rkiGdqtY/kLPjUyot6+AGcwjd8w6A== X-Received: by 2002:a1c:7eca:: with SMTP id z193-v6mr6089232wmc.139.1530873082436; Fri, 06 Jul 2018 03:31:22 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d102-v6sm14903866wma.10.2018.07.06.03.31.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jul 2018 03:31:21 -0700 (PDT) To: dev@openvswitch.org Date: Fri, 6 Jul 2018 16:10:49 +0530 Message-Id: <20180706104050.24938-4-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> References: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 3/4] ovs-dev: Rebalance offloaded flows based on the pps rate X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: Brendan Doyle via dev Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the third patch in the patch-set to support dynamic rebalancing of offloaded flows. The dynamic rebalancing functionality is implemented in this patch. The ukeys that are not scheduled for deletion are obtained and passed as input to the rebalancing routine. The rebalancing is done in the context of revalidation leader thread, after all other revalidator threads are done with gathering rebalancing data for flows. For each netdev that is in OOR state, a list of flows - both offloaded and non-offloaded (pending) - is obtained using the ukeys. For each netdev that is in OOR state, the flows are grouped and sorted into offloaded and pending flows. The offloaded flows are sorted in descending order of pps-rate, while pending flows are sorted in ascending order of pps-rate. The rebalancing is done in two phases. In the first phase, we try to offload all pending flows and if that succeeds, the OOR state on the device is cleared. If some (or none) of the pending flows could not be offloaded, then we start replacing an offloaded flow that has a lower pps-rate than a pending flow, until there are no more pending flows with a higher rate than an offloaded flow. The flows that are replaced from the device are added into kernel datapath. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-netdev.c | 3 +- lib/dpif-netlink.c | 15 +- lib/dpif-provider.h | 7 +- lib/dpif.c | 20 +- lib/dpif.h | 20 +- lib/netdev-provider.h | 3 +- ofproto/ofproto-dpif-upcall.c | 435 +++++++++++++++++++++++++++++++++- 7 files changed, 482 insertions(+), 21 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 9390fff68..ad5aac62b 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -3052,7 +3052,8 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute) } static void -dpif_netdev_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops) +dpif_netdev_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, + enum dpif_op_skip_type skip_flag OVS_UNUSED) { size_t i; diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index e20096116..3c4b9b69f 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -1004,6 +1004,7 @@ dpif_netlink_rtnl_port_create_and_add(struct dpif_netlink *dpif, return error; } +static int dpif_netlink_port_add(struct dpif *dpif_, struct netdev *netdev, odp_port_t *port_nop) { @@ -2275,7 +2276,8 @@ dpif_netlink_operate_chunks(struct dpif_netlink *dpif, struct dpif_op **ops, } static void -dpif_netlink_operate(struct dpif *dpif_, struct dpif_op **ops, size_t n_ops) +dpif_netlink_operate(struct dpif *dpif_, struct dpif_op **ops, size_t n_ops, + enum dpif_op_skip_type skip_flag) { struct dpif_netlink *dpif = dpif_netlink_cast(dpif_); struct dpif_op *new_ops[OPERATE_MAX_OPS]; @@ -2283,7 +2285,7 @@ dpif_netlink_operate(struct dpif *dpif_, struct dpif_op **ops, size_t n_ops) int i = 0; int err = 0; - if (netdev_is_flow_api_enabled()) { + if (skip_flag != DPIF_OP_SKIP_OFFLOAD && netdev_is_flow_api_enabled()) { while (n_ops > 0) { count = 0; @@ -2292,6 +2294,10 @@ dpif_netlink_operate(struct dpif *dpif_, struct dpif_op **ops, size_t n_ops) err = try_send_to_netdev(dpif, op); if (err && err != EEXIST) { + if (skip_flag == DPIF_OP_SKIP_DP) { + op->error = err; + return; + } new_ops[count++] = op; } else { op->error = err; @@ -2302,8 +2308,11 @@ dpif_netlink_operate(struct dpif *dpif_, struct dpif_op **ops, size_t n_ops) dpif_netlink_operate_chunks(dpif, new_ops, count); } - } else { + } else if (skip_flag != DPIF_OP_SKIP_DP) { dpif_netlink_operate_chunks(dpif, ops, n_ops); + } else { + VLOG_ERR("%s: Invalid skip_flag: %d while flow api is disabled\n", + __func__, skip_flag); } } diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index cc6571b28..2bbfcf7f6 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -296,12 +296,13 @@ struct dpif_class { int (*flow_dump_next)(struct dpif_flow_dump_thread *thread, struct dpif_flow *flows, int max_flows); - /* Executes each of the 'n_ops' operations in 'ops' on 'dpif', in the order * in which they are specified, placing each operation's results in the * "output" members documented in comments and the 'error' member of each - * dpif_op. */ - void (*operate)(struct dpif *dpif, struct dpif_op **ops, size_t n_ops); + * dpif_op. The skip_flag argument tells the provider if 'ops' should be + * offloaded to a netdev or to the kernel datapath or to both. */ + void (*operate)(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, + enum dpif_op_skip_type skip_flag); /* Enables or disables receiving packets with dpif_recv() for 'dpif'. * Turning packet receive off and then back on is allowed to change Netlink diff --git a/lib/dpif.c b/lib/dpif.c index f6a7f6a72..d3eec1124 100644 --- a/lib/dpif.c +++ b/lib/dpif.c @@ -49,6 +49,7 @@ #include "valgrind.h" #include "openvswitch/ofp-errors.h" #include "openvswitch/vlog.h" +#include "lib/netdev-provider.h" VLOG_DEFINE_THIS_MODULE(dpif); @@ -1010,7 +1011,7 @@ dpif_flow_get(struct dpif *dpif, op.flow_get.flow->key_len = key_len; opp = &op; - dpif_operate(dpif, &opp, 1); + dpif_operate(dpif, &opp, 1, DPIF_OP_SKIP_NONE); return op.error; } @@ -1040,7 +1041,7 @@ dpif_flow_put(struct dpif *dpif, enum dpif_flow_put_flags flags, op.flow_put.stats = stats; opp = &op; - dpif_operate(dpif, &opp, 1); + dpif_operate(dpif, &opp, 1, DPIF_OP_SKIP_NONE); return op.error; } @@ -1063,7 +1064,7 @@ dpif_flow_del(struct dpif *dpif, op.flow_del.terse = false; opp = &op; - dpif_operate(dpif, &opp, 1); + dpif_operate(dpif, &opp, 1, DPIF_OP_SKIP_NONE); return op.error; } @@ -1320,7 +1321,7 @@ dpif_execute(struct dpif *dpif, struct dpif_execute *execute) op.execute = *execute; opp = &op; - dpif_operate(dpif, &opp, 1); + dpif_operate(dpif, &opp, 1, DPIF_OP_SKIP_NONE); return op.error; } else { @@ -1331,9 +1332,12 @@ dpif_execute(struct dpif *dpif, struct dpif_execute *execute) /* Executes each of the 'n_ops' operations in 'ops' on 'dpif', in the order in * which they are specified. Places each operation's results in the "output" * members documented in comments, and 0 in the 'error' member on success or a - * positive errno on failure. */ + * positive errno on failure. The skip_flag argument tells the provider if + * 'ops' should be offloaded to a netdev or to the kernel datapath or to both. + */ void -dpif_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops) +dpif_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, + enum dpif_op_skip_type skip_flag) { while (n_ops > 0) { size_t chunk; @@ -1355,7 +1359,7 @@ dpif_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops) * handle itself, without help. */ size_t i; - dpif->dpif_class->operate(dpif, ops, chunk); + dpif->dpif_class->operate(dpif, ops, chunk, skip_flag); for (i = 0; i < chunk; i++) { struct dpif_op *op = ops[i]; @@ -1652,7 +1656,7 @@ dpif_queue_to_priority(const struct dpif *dpif, uint32_t queue_id, log_operation(dpif, "queue_to_priority", error); return error; } - + void dpif_init(struct dpif *dpif, const struct dpif_class *dpif_class, const char *name, diff --git a/lib/dpif.h b/lib/dpif.h index 33d2d0bec..cc3ba343b 100644 --- a/lib/dpif.h +++ b/lib/dpif.h @@ -614,6 +614,23 @@ enum dpif_op_type { DPIF_OP_FLOW_GET, }; +/* skip_flag argument types to (*operate) interface */ +enum dpif_op_skip_type { + DPIF_OP_SKIP_NONE = 1, /* If Offload enabled: + * Offload first; if it + * fails, go to datapath. + * If Offload disabled: Go to datapath. + */ + + DPIF_OP_SKIP_OFFLOAD, /* Go to datapath only; do not offload even if + * Offload enabled + */ + + DPIF_OP_SKIP_DP /* Do not go to datapath, even if Offload + * enabled and fail to offload. + */ +}; + /* Add or modify a flow. * * The flow is specified by the Netlink attributes with types OVS_KEY_ATTR_* in @@ -768,7 +785,8 @@ struct dpif_op { }; }; -void dpif_operate(struct dpif *, struct dpif_op **ops, size_t n_ops); +void dpif_operate(struct dpif *, struct dpif_op **ops, size_t n_ops, + enum dpif_op_skip_type skip_flag); /* Upcalls. */ diff --git a/lib/netdev-provider.h b/lib/netdev-provider.h index 62e05619e..5c919f4a1 100644 --- a/lib/netdev-provider.h +++ b/lib/netdev-provider.h @@ -38,6 +38,8 @@ struct netdev_tnl_build_header_params; /* Offload-capable (HW) netdev information */ struct netdev_hw_info { bool oor; /* Out of Offload Resources ? */ + int offload_count; + int pending_count; }; /* A network device (e.g. an Ethernet device). @@ -85,7 +87,6 @@ struct netdev { int n_rxq; struct shash_node *node; /* Pointer to element in global map. */ struct ovs_list saved_flags_list; /* Contains "struct netdev_saved_flags". */ - struct netdev_hw_info hw_info; /* offload-capable netdev info */ }; diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 6f4101f36..28596baea 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -22,6 +22,7 @@ #include "connmgr.h" #include "coverage.h" #include "cmap.h" +#include "lib/dpif-provider.h" #include "dpif.h" #include "openvswitch/dynamic-string.h" #include "fail-open.h" @@ -42,7 +43,7 @@ #include "tunnel.h" #include "unixctl.h" #include "openvswitch/vlog.h" -#include "lib/dpif-provider.h" +#include "lib/netdev-provider.h" #define MAX_QUEUE_LENGTH 512 #define UPCALL_MAX_BATCH 64 @@ -398,6 +399,18 @@ static int upcall_receive(struct upcall *, const struct dpif_backer *, const ovs_u128 *ufid, const unsigned pmd_id); static void upcall_uninit(struct upcall *); +static void udpif_flow_rebalance_prepare(struct udpif *udpif, + struct udpif_key *** + active_flows, + int *num_active_flows); +static void udpif_flow_rebalance(struct udpif *udpif, + struct udpif_key **active_flows, + int num_active_flows); +static int udpif_flow_program(struct udpif *udpif, struct udpif_key *ukey, + enum dpif_op_skip_type skip_flag); +static int udpif_flow_unprogram(struct udpif *udpif, struct udpif_key *ukey, + enum dpif_op_skip_type skip_flag); + static upcall_callback upcall_cb; static dp_purge_callback dp_purge_cb; @@ -861,6 +874,25 @@ free_dupcall: return n_upcalls; } +static void +udpif_run_flow_rebalance(struct udpif *udpif) +{ + struct udpif_key **active_flows = NULL; + static long long int time = 0; + static long long int now = 0; + int num_active_flows; + + /* Don't rebalance if OFFL_REBAL_INTVL_MSEC have not elapsed */ + now = time_msec(); + if (now < time + OFFL_REBAL_INTVL_MSEC) { + return; + } + time = now; + + udpif_flow_rebalance_prepare(udpif, &active_flows, &num_active_flows); + udpif_flow_rebalance(udpif, active_flows, num_active_flows); +} + static void * udpif_revalidator(void *arg) { @@ -935,6 +967,7 @@ udpif_revalidator(void *arg) dpif_flow_dump_destroy(udpif->dump); seq_change(udpif->dump_seq); + udpif_run_flow_rebalance(udpif); duration = MAX(time_msec() - start_time, 1); udpif->dump_duration = duration; @@ -1581,7 +1614,7 @@ handle_upcalls(struct udpif *udpif, struct upcall *upcalls, for (i = 0; i < n_ops; i++) { opsp[n_opsp++] = &ops[i].dop; } - dpif_operate(udpif->dpif, opsp, n_opsp); + dpif_operate(udpif->dpif, opsp, n_opsp, DPIF_OP_SKIP_NONE); for (i = 0; i < n_ops; i++) { struct udpif_key *ukey = ops[i].ukey; @@ -1673,7 +1706,7 @@ ukey_create__(const struct nlattr *key, size_t key_len, ukey->state = UKEY_CREATED; ukey->state_thread = ovsthread_id_self(); ukey->state_where = OVS_SOURCE_LOCATOR; - ukey->created = time_msec(); + ukey->created = ukey->flow_time = time_msec(); memset(&ukey->stats, 0, sizeof ukey->stats); ukey->stats.used = used; ukey->xcache = NULL; @@ -2332,7 +2365,7 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) for (i = 0; i < n_ops; i++) { opsp[i] = &ops[i].dop; } - dpif_operate(udpif->dpif, opsp, n_ops); + dpif_operate(udpif->dpif, opsp, n_ops, DPIF_OP_SKIP_NONE); for (i = 0; i < n_ops; i++) { struct ukey_op *op = &ops[i]; @@ -2566,6 +2599,16 @@ udpif_flow_time_delta(struct udpif *udpif, struct udpif_key *ukey) return (udpif->dpif->current_ms - ukey->flow_time) / 1000; } +/* + * Save backlog packet count while switching modes + * between offloaded and kernel datapaths. + */ +static void +udpif_set_ukey_backlog_packets(struct udpif_key *ukey) +{ + ukey->flow_backlog_packets = ukey->flow_packets; +} + /* Gather pps-rate for the given dpif_flow and save it in its ukey */ static void udpif_update_flow_pps(struct udpif *udpif, struct udpif_key *ukey, @@ -2646,6 +2689,7 @@ revalidate(struct revalidator *revalidator) kill_them_all = n_dp_flows > flow_limit * 2; max_idle = n_dp_flows > flow_limit ? 100 : ofproto_max_idle; + udpif->dpif->current_ms = time_msec(); for (f = flows; f < &flows[n_dumped]; f++) { long long int used = f->stats.used; struct recirc_refs recircs = RECIRC_REFS_EMPTY_INITIALIZER; @@ -3021,3 +3065,386 @@ upcall_unixctl_purge(struct unixctl_conn *conn, int argc OVS_UNUSED, } unixctl_command_reply(conn, ""); } + +/* + * Walk umaps and build an array of pointers to ukeys. That + * serves as the input to udpif_flow_rebalance() routine. + */ +static void +udpif_flow_rebalance_prepare(struct udpif *udpif, struct udpif_key + ***active_flows, int *num_active_flows) +{ + struct udpif_key **f = NULL; + size_t n = 0; + size_t i; + size_t j; + + if (!active_flows || !num_active_flows) { + return; + } + + for (i = 0; i < N_UMAPS; i++) { + struct umap *umap = &udpif->ukeys[i]; + n += cmap_count(&umap->cmap); + } + + f = xzalloc(n * sizeof(*f)); + + for (i = 0, j = 0; i < N_UMAPS; i++) { + struct udpif_key *ukey; + struct umap *umap = &udpif->ukeys[i]; + + CMAP_FOR_EACH (ukey, cmap_node, &umap->cmap) { + f[j++] = ukey; + if (j >= n) { + goto done; + } + } + } + +done: + *active_flows = f; + *num_active_flows = n; +} + +/* Flows are sorted in the following order: + * netdev name, flow state (offloaded/kernel path), flow_pps_rate. + */ +static int +flow_compare_rebalance(const void *elem1, const void *elem2) +{ + const struct udpif_key *f1 = + *(struct udpif_key **)elem1; + const struct udpif_key *f2 = + *(struct udpif_key **)elem2; + struct netdev *netdev1; + struct netdev *netdev2; + int rv; + + netdev1 = f1->tunnel_netdev ? f1->tunnel_netdev : f1->in_netdev; + netdev2 = f2->tunnel_netdev ? f2->tunnel_netdev : f2->in_netdev; + + rv = strcmp(netdev1->name, netdev2->name); + if (rv) { + return (rv); + } + + if (f1->offloaded != f2->offloaded) { + return (f2->offloaded - f1->offloaded); + } + + float diff = (f1->offloaded == true) ? + f1->flow_pps_rate - f2->flow_pps_rate : + f2->flow_pps_rate - f1->flow_pps_rate; + + return (diff < 0) ? -1 : 1; +} + +/* Insert flows from pending array during rebalancing */ +static int +rebalance_insert_pending(struct udpif *udpif, struct udpif_key **pending_flows, + int pending_count, int insert_count, + float rate_threshold) +{ + int i, err = 0; + struct udpif_key *flow; + int count = 0; + + for (i = 0; i < pending_count; i++) { + flow = pending_flows[i]; + + /* Stop offloading pending flows if the insert count is + * reached and the flow rate is less than the threshold + */ + if ((count >= insert_count) && + (flow->flow_pps_rate < rate_threshold)) { + break; + } + + /* Offload the flow to netdev */ + err = udpif_flow_program(udpif, flow, DPIF_OP_SKIP_DP); + + if (err == ENOSPC) { + /* Stop if we are out of resources */ + break; + } + + if (err) { + continue; + } + + /* Offload succeeded; delete it from the kernel datapath */ + udpif_flow_unprogram(udpif, flow, DPIF_OP_SKIP_OFFLOAD); + + /* Change the state of the flow, adjust dpif counters */ + flow->offloaded = true; + + udpif_set_ukey_backlog_packets(flow); + count++; + } + + return count; +} + +/* Remove flows from offloaded array during rebalancing */ +static void +rebalance_remove_offloaded(struct udpif *udpif, + struct udpif_key **offloaded_flows, + int offload_count) +{ + int i; + struct udpif_key *flow; + + for (i = 0; i < offload_count; i++) { + flow = offloaded_flows[i]; + + /* Remove offloaded flow from netdev */ + udpif_flow_unprogram(udpif, flow, DPIF_OP_SKIP_DP); + + /* Install the flow into kernel path */ + udpif_flow_program(udpif, flow, DPIF_OP_SKIP_OFFLOAD); + flow->offloaded = false; + + udpif_set_ukey_backlog_packets(flow); + } +} + +/* + * Rebalance offloaded flows on a netdev that's in OOR state. + * + * The rebalancing is done in two phases. In the first phase, we check if + * the pending flows can be offloaded (if some resources became available + * in the meantime) by trying to offload each pending flow. If all pending + * flows get successfully offloaded, the OOR state is cleared on the netdev + * and there's nothing to rebalance. + * + * If some of the pending flows could not be offloaded, i.e, we still see + * the OOR error, then we move to the second phase of rebalancing. In this + * phase, the rebalancer compares pps-rate of an offloaded flow with the + * least pps-rate with that of a pending flow with the highest pps-rate from + * their respective sorted arrays. If pps-rate of the offloaded flow is less + * than the pps-rate of the pending flow, then it deletes the offloaded flow + * from the HW/netdev and adds it to kernel datapath and then offloads pending + * to HW/netdev. This process is repeated for every pair of offloaded and + * pending flows in the ordered list. The process stops when we encounter an + * offloaded flow that has a higher pps-rate than the corresponding pending + * flow. The entire rebalancing process is repeated in the next iteration. + */ +static bool +rebalance_device(struct udpif *udpif, struct udpif_key **offloaded_flows, + int offload_count, struct udpif_key **pending_flows, + int pending_count) +{ + + /* Phase 1 */ + int num_inserted = rebalance_insert_pending(udpif, pending_flows, + pending_count, pending_count, + 0.0); + if (num_inserted) { + VLOG_DBG("rebalance_device phase1 inserted %d pending flows\n", + num_inserted); + } + + /* Adjust pending array */ + pending_flows = &pending_flows[num_inserted]; + pending_count -= num_inserted; + + if (!pending_count) { + /* + * Successfully offloaded all pending flows. The device + * is no longer in OOR state; done rebalancing this device. + */ + return false; + } + + /* + * Phase 2; determine how many offloaded flows to churn. + */ + int churn_count = 0; + while ((churn_count < offload_count) && + (churn_count < pending_count)) { + struct ds ds1 = DS_EMPTY_INITIALIZER; + struct ds ds2 = DS_EMPTY_INITIALIZER; + + odp_format_ufid(&(pending_flows[churn_count]->ufid), &ds1); + odp_format_ufid(&(offloaded_flows[churn_count]->ufid), &ds2); + + VLOG_DBG("Index: %d pend-flow: id: %s pps: %f offl-flow: id: %s " + "pps: %f\n", churn_count, + ds_cstr(&ds1), pending_flows[churn_count]->flow_pps_rate, + ds_cstr(&ds2), offloaded_flows[churn_count]->flow_pps_rate); + + if (pending_flows[churn_count]->flow_pps_rate <= + offloaded_flows[churn_count]->flow_pps_rate) + break; + + churn_count++; + } + + if (churn_count) { + VLOG_DBG("rebalance_device phase2 removed %d offloaded flows\n", + churn_count); + } + + /* Bail early if nothing to churn */ + if (!churn_count) { + VLOG_DBG("rebalance_device phase2 churn_count is zero\n"); + return true; + } + + /* Remove offloaded flows */ + rebalance_remove_offloaded(udpif, offloaded_flows, churn_count); + + /* Adjust offloaded array */ + offloaded_flows = &offloaded_flows[churn_count]; + offload_count -= churn_count; + + /* Replace offloaded flows with pending flows */ + num_inserted = rebalance_insert_pending(udpif, pending_flows, + pending_count, churn_count, + offload_count ? + offloaded_flows[0]->flow_pps_rate : + 0.0); + if (num_inserted) { + VLOG_DBG("rebalance_device phase2 inserted %d pending flows\n", + num_inserted); + } + + return true; +} + +/* + * Given all ukeys that are active in the datapath, rebalance + * offloaded flows on HW netdevs that are in OOR state. + */ +static void +udpif_flow_rebalance(struct udpif *udpif, + struct udpif_key **active_flows, + int num_active_flows) +{ + struct udpif_key **sort_flows = NULL; + int total_offload_count = 0; + int total_pending_count = 0; + int total_flow_count = 0; + int oor_netdev_count = 0; + struct udpif_key *ukey; + struct netdev *netdev; + int offload_index = 0; + int pending_index; + bool oor; + int i; + + if (!num_active_flows) { + return; + } + + sort_flows = xzalloc(sizeof(*sort_flows) * num_active_flows); + + /* Populate sort_flows[] initially with flows that + * have an 'OOR' device as their input or output port + */ + for (i = 0; i < num_active_flows; i++) { + ukey = active_flows[i]; + + /* Both input and output netdevs must be available for the flow */ + if (!ukey->in_netdev || !ukey->out_netdev) { + continue; + } + + /* Get input netdev for the flow */ + netdev = ukey->tunnel_netdev ? ukey->tunnel_netdev : ukey->in_netdev; + + /* Is the in-netdev for this flow in OOR state ? */ + if (netdev->hw_info.oor) { + /* Add the flow to sort_flows[] */ + sort_flows[total_flow_count++] = active_flows[i]; + + if (ukey->offloaded) { + total_offload_count++; + if (netdev->hw_info.offload_count++ == 0) { + oor_netdev_count++; + } + } else { + total_pending_count++; + netdev->hw_info.pending_count++; + } + } + } + + VLOG_DBG("%s: Total: offload_count: %d pending_count %d\n", + __func__, total_offload_count, total_pending_count); + qsort(sort_flows, total_flow_count, sizeof(struct udpif_key *), + flow_compare_rebalance); + + /* + * We now have a count of offloaded and pending flows on each of the + * HW netdevs that are in OOR state. Now loop through all netdevs in + * the datapath and run rebalance only on HW netdevs that are in OOR + * state. + */ + while (oor_netdev_count) { + netdev = sort_flows[offload_index]->tunnel_netdev ? + sort_flows[offload_index]->tunnel_netdev : + sort_flows[offload_index]->in_netdev; + if (!netdev->hw_info.oor) { + continue; + } + + VLOG_DBG("%s: netdev: %s is OOR\n", __func__, netdev->name); + pending_index = offload_index + netdev->hw_info.offload_count; + oor = rebalance_device(udpif, + &sort_flows[offload_index], + netdev->hw_info.offload_count, + &sort_flows[pending_index], + netdev->hw_info.pending_count); + + netdev->hw_info.oor = oor; + offload_index = pending_index + netdev->hw_info.pending_count; + netdev->hw_info.offload_count = netdev->hw_info.pending_count = 0; + oor_netdev_count--; + } + + free(sort_flows); + free(active_flows); +} + +static void +ukey_to_ukey_op(struct udpif *udpif, struct ukey_op *uop, + struct udpif_key *ukey, bool put) +{ + if (put) { + put_op_init(uop, ukey, DPIF_FP_CREATE); + } else { + delete_op_init(udpif, uop, ukey); + } +} + +#define FLOW_PGM_NUM_OPS 1 + +static int +udpif_flow_program(struct udpif *udpif, struct udpif_key *ukey, + enum dpif_op_skip_type skip_flag) +{ + struct dpif_op *opsp; + struct ukey_op uop; + + opsp = &uop.dop; + ukey_to_ukey_op(udpif, &uop, ukey, true); + dpif_operate(udpif->dpif, &opsp, FLOW_PGM_NUM_OPS, skip_flag); + + return opsp->error; +} + +static int +udpif_flow_unprogram(struct udpif *udpif, struct udpif_key *ukey, + enum dpif_op_skip_type skip_flag) +{ + struct dpif_op *opsp; + struct ukey_op uop; + + opsp = &uop.dop; + ukey_to_ukey_op(udpif, &uop, ukey, false); + dpif_operate(udpif->dpif, &opsp, FLOW_PGM_NUM_OPS, skip_flag); + + return opsp->error; +} From patchwork Fri Jul 6 10:40:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Doyle via dev X-Patchwork-Id: 940389 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="HBiKjRcm"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41MWKM3xkjz9s29 for ; Fri, 6 Jul 2018 20:33:27 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 960EFCDB; Fri, 6 Jul 2018 10:31:29 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 5306FCC0 for ; Fri, 6 Jul 2018 10:31:28 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 6CAD676E for ; Fri, 6 Jul 2018 10:31:27 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id k7-v6so3686751wrq.0 for ; Fri, 06 Jul 2018 03:31:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2JwcClPR4pHML9LdxX5CUYgiTkaYZp/qW1M11jJ2OBw=; b=HBiKjRcm1TKBwxlXQZD02xjFxAToiznNPWb1esA2NjeLBYtkisJCCCYYyP4pYVm50n +AVxTet7vi6B82a1UUgnZok2lEidnsguVc+eWn4CP1/7XJNn4nDq7MWlUUr0Xi/3Jf7K HmtemXQnO7BkCzDnf5cnI+8h/zHM77MC2VAkA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2JwcClPR4pHML9LdxX5CUYgiTkaYZp/qW1M11jJ2OBw=; b=CX0UhNRQvp0gQRC6d5Rga8xNjO9gEQ0mwgVfqjct9nw0J+TKK/APphP8rtHLsxSg/B 5z1QALxmpV6IWavFhNT0Td1wP0TjCFs/LcjDgFZ/5UosLR0EpO0IKeT01bZM49O6jsiy 1pDL5dtnhRvb5mYHFKN/DOxtaUy3ygSdc9L3ay8MSd3P43XyOYKzQnI0lbE636ykDJH+ 2P33F/bNm+eFfui3LadQwdCiqpv7hPxKrsIjWoXZbOeZR1yqvbaCfMkrS6BA0hg+326A q010VFFItGupbS8Da0UurrDKNJRSTPNVWsfAIJx2tEvu7jPzpiu3aKhEXSgcoGltEVsC 2PNw== X-Gm-Message-State: APt69E121pZyFjOEt9bOKKvlVeYygL1FHjzhA6+E9dViRFWOS9q3dD4K HsKoXtMXs3HXGQwgXhC9qoxaqaC376c= X-Google-Smtp-Source: AAOMgpeTp3M91hqvH1foGfyc17LCgkbNaeVASZz3PYaeKgz7GRbKifJjPj2M9s4B+mtBzdiAQZqQKg== X-Received: by 2002:adf:8712:: with SMTP id a18-v6mr7674200wra.178.1530873085690; Fri, 06 Jul 2018 03:31:25 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id d102-v6sm14903866wma.10.2018.07.06.03.31.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Jul 2018 03:31:24 -0700 (PDT) To: dev@openvswitch.org Date: Fri, 6 Jul 2018 16:10:50 +0530 Message-Id: <20180706104050.24938-5-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> References: <20180706104050.24938-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 4/4] ovs-dev: Add an openvswitch option to enable dynamic rebalancing of flows X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: Brendan Doyle via dev Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the fourth patch in the patch-set to support dynamic rebalancing of offloaded flows. A new OVS configuration parameter "offload-rebalance", is added to ovsdb. The default value of this is "disable". To enable this feature, set the value of this parameter to "pps-rate", which provides packets-per-second rate based policy to dynamically offload and un-offload flows. Note: This option can be enabled only when 'hw-offload' policy is enabled. It also requires 'tc-policy' to be set to 'skip_sw'; otherwise, flow offload errors (specifically ENOSPC error this feature depends on) reported by an offloaded device are supressed by TC-Flower kernel module. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-netlink.c | 3 ++- lib/netdev.c | 43 +++++++++++++++++++++++++++++++++++ lib/netdev.h | 2 ++ ofproto/ofproto-dpif-upcall.c | 7 ++++-- vswitchd/vswitch.xml | 22 ++++++++++++++++++ 5 files changed, 74 insertions(+), 3 deletions(-) diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 3c4b9b69f..dfe21971d 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -2177,7 +2177,8 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) VLOG_DBG("added flow"); } else if (err != EEXIST) { - if (outdev && dev && (err == ENOSPC)) { + if (netdev_is_offload_rebalance_policy_enabled() && outdev && + dev && (err == ENOSPC)) { tunnel_netdev = flow_get_tunnel_netdev(&match.flow.tunnel); if (tunnel_netdev) { oor_netdev = tunnel_netdev; diff --git a/lib/netdev.c b/lib/netdev.c index b17f0563f..9992b846c 100644 --- a/lib/netdev.c +++ b/lib/netdev.c @@ -2474,6 +2474,43 @@ netdev_free_custom_stats_counters(struct netdev_custom_stats *custom_stats) #ifdef __linux__ +enum offload_rebalance_policy { + OFFLOAD_REBALANCE_POLICY_DISABLE, + OFFLOAD_REBALANCE_POLICY_PPS_RATE +}; + +static enum +offload_rebalance_policy netdev_offload_rebalance_policy = + OFFLOAD_REBALANCE_POLICY_DISABLE; + +static void +netdev_offload_rebalance_set_policy(const char *policy) +{ + if (!policy) { + return; + } + + if (!strcmp(policy, "pps-rate")) { + netdev_offload_rebalance_policy = OFFLOAD_REBALANCE_POLICY_PPS_RATE; + } else if (!strcmp(policy, "disable")) { + netdev_offload_rebalance_policy = OFFLOAD_REBALANCE_POLICY_DISABLE; + } else { + VLOG_WARN("netdev: Invalid policy '%s'", policy); + return; + } + + VLOG_INFO("netdev: Using policy '%s'", policy); +} + +bool +netdev_is_offload_rebalance_policy_enabled(void) +{ + if (netdev_offload_rebalance_policy == OFFLOAD_REBALANCE_POLICY_PPS_RATE) { + return true; + } + return false; +} + static void netdev_ports_flow_init(void) { @@ -2494,12 +2531,18 @@ netdev_set_flow_api_enabled(const struct smap *ovs_other_config) if (ovsthread_once_start(&once)) { netdev_flow_api_enabled = true; + const char *offl_rebal_policy = NULL; VLOG_INFO("netdev: Flow API Enabled"); tc_set_policy(smap_get_def(ovs_other_config, "tc-policy", TC_POLICY_DEFAULT)); + offl_rebal_policy = smap_get_def(ovs_other_config, + "offload-rebalance", + OFFLOAD_REBALANCE_POLICY_DEFAULT); + netdev_offload_rebalance_set_policy(offl_rebal_policy); + netdev_ports_flow_init(); ovsthread_once_done(&once); diff --git a/lib/netdev.h b/lib/netdev.h index c941f1e1e..0ac2774c9 100644 --- a/lib/netdev.h +++ b/lib/netdev.h @@ -223,6 +223,8 @@ int netdev_init_flow_api(struct netdev *); uint32_t netdev_get_block_id(struct netdev *); bool netdev_is_flow_api_enabled(void); void netdev_set_flow_api_enabled(const struct smap *ovs_other_config); +bool netdev_is_offload_rebalance_policy_enabled(void); +#define OFFLOAD_REBALANCE_POLICY_DEFAULT "disable" struct dpif_port; int netdev_ports_insert(struct netdev *, const struct dpif_class *, diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 28596baea..49922089e 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -967,7 +967,9 @@ udpif_revalidator(void *arg) dpif_flow_dump_destroy(udpif->dump); seq_change(udpif->dump_seq); - udpif_run_flow_rebalance(udpif); + if (netdev_is_offload_rebalance_policy_enabled()) { + udpif_run_flow_rebalance(udpif); + } duration = MAX(time_msec() - start_time, 1); udpif->dump_duration = duration; @@ -2747,7 +2749,8 @@ revalidate(struct revalidator *revalidator) } ukey->dump_seq = dump_seq; - if (result != UKEY_DELETE) { + if (netdev_is_offload_rebalance_policy_enabled() && + result != UKEY_DELETE) { udpif_update_flow_pps(udpif, ukey, f); } diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 76094852d..a0aad0204 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -475,6 +475,28 @@

+ +

+ Configures HW offload rebalancing, that allows to dynamically + offload and un-offload flows while an offload-device is out of + resources (OOR). The value specifies the criteria used to + select flows for offloading. The default value of 'disable', + disables this policy. To enable packets-per-second rate based + offload policy, set the value to 'pps-rate'. + Options: + disable - No offload rebalancing + pps-rate - Packets-per-second rate based policy +

+

+ This is only relevant if HW offloading is enabled (hw-offload). + When this policy is enabled, it also requires 'tc-policy' to + be set to 'skip_sw'. +

+

+ The default value is disable. +

+