From patchwork Tue Jul 10 06:29:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li,Rongqing via dev" X-Patchwork-Id: 941812 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="dc22PcSm"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41PsYC3MxCz9s37 for ; Tue, 10 Jul 2018 16:21:51 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 83B05D14; Tue, 10 Jul 2018 06:20:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 16357D12 for ; Tue, 10 Jul 2018 06:20:09 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-qk0-f196.google.com (mail-qk0-f196.google.com [209.85.220.196]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 5BA7667E for ; Tue, 10 Jul 2018 06:20:07 +0000 (UTC) Received: by mail-qk0-f196.google.com with SMTP id c131-v6so10985291qkb.0 for ; Mon, 09 Jul 2018 23:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BbfNuHaGyISvRuMTeT1BpUu+DtBMaSJ1/Sj8uiI77Jw=; b=dc22PcSmbrcV0GRgnDkCgSuOO21XejaC21alY0Xd0L42VmJW93yhYENL/6Ppu0+UZ8 elZXx5pWtyxa08t9CPU/UoUbAVjBvJfkY4PkwFMP04dy2S1WLs3s9Z3wQutgBgHMrrtX PGcjnonKxWRHcnOnyBw24xMZl5QNUe5D88xsI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BbfNuHaGyISvRuMTeT1BpUu+DtBMaSJ1/Sj8uiI77Jw=; b=Ngp3aevGJMj5sikpauq6LX+2soe37z7vf2+9gnwwIAMrJ6SH+Nokdx5QzDUPbrlSHj FU1ge9z2B2qBZPzmz192vg/ffzLICUQZorQybqClxByjdBsqzjUMHAMcA06SRWxy38Ps wI5j+jd8q/5a4J5TtFJzDH+4+SoAvo8X46gg/lqI3HfEii/gHWVIFc42pSyxYKn3YEsT RswLGm5Cgl/JahUakQ5Sp2Ja0r6vW8goBsXgwYFy+Kde+JwggsSnDVDIf9rJCX8sO0VE 9k32GsNOoHr4arzAtf2EnnIJw8uZ4eyIIffGdKr0CswKBU/Gb5dKISmFdYoK9+sawjLE fcXQ== X-Gm-Message-State: APt69E2eWb4SiZY9I6Q/4XcZDr7EZP/NlWGV7/KrKEQs/W/K1ISd+zud NJBtbKrV8qhST7IaDMMAgHqHOeLDAzY= X-Google-Smtp-Source: AAOMgpcdcib27LRsUPpCFWePRkSSBG0tcWTTaesdxljzUB2qy27F9tngsFpuVtLG2dJLO9ttZhJSbA== X-Received: by 2002:a37:1adb:: with SMTP id l88-v6mr20251705qkh.413.1531203606181; Mon, 09 Jul 2018 23:20:06 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id m51-v6sm2592203qtc.85.2018.07.09.23.20.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Jul 2018 23:20:05 -0700 (PDT) To: dev@openvswitch.org Date: Tue, 10 Jul 2018 11:59:38 +0530 Message-Id: <20180710062938.18939-5-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180710062938.18939-1-sriharsha.basavapatna@broadcom.com> References: <20180710062938.18939-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v3 4/4] netdev: Add a configuration option to enable dynamic rebalancing of flows X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: "Li,Rongqing via dev" Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the fourth patch in the patch-set to support dynamic rebalancing of offloaded flows. A new OVS configuration parameter "offload-rebalance", is added to ovsdb. The default value of this is "disable". To enable this feature, set the value of this parameter to "pps-rate", which provides packets-per-second rate based policy to dynamically offload and un-offload flows. Note: This option can be enabled only when 'hw-offload' policy is enabled. It also requires 'tc-policy' to be set to 'skip_sw'; otherwise, flow offload errors (specifically ENOSPC error this feature depends on) reported by an offloaded device are supressed by TC-Flower kernel module. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-netlink.c | 3 ++- lib/netdev.c | 43 +++++++++++++++++++++++++++++++++++ lib/netdev.h | 2 ++ ofproto/ofproto-dpif-upcall.c | 7 ++++-- vswitchd/vswitch.xml | 22 ++++++++++++++++++ 5 files changed, 74 insertions(+), 3 deletions(-) diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 3c4b9b69f..dfe21971d 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -2177,7 +2177,8 @@ parse_flow_put(struct dpif_netlink *dpif, struct dpif_flow_put *put) VLOG_DBG("added flow"); } else if (err != EEXIST) { - if (outdev && dev && (err == ENOSPC)) { + if (netdev_is_offload_rebalance_policy_enabled() && outdev && + dev && (err == ENOSPC)) { tunnel_netdev = flow_get_tunnel_netdev(&match.flow.tunnel); if (tunnel_netdev) { oor_netdev = tunnel_netdev; diff --git a/lib/netdev.c b/lib/netdev.c index b17f0563f..9992b846c 100644 --- a/lib/netdev.c +++ b/lib/netdev.c @@ -2474,6 +2474,43 @@ netdev_free_custom_stats_counters(struct netdev_custom_stats *custom_stats) #ifdef __linux__ +enum offload_rebalance_policy { + OFFLOAD_REBALANCE_POLICY_DISABLE, + OFFLOAD_REBALANCE_POLICY_PPS_RATE +}; + +static enum +offload_rebalance_policy netdev_offload_rebalance_policy = + OFFLOAD_REBALANCE_POLICY_DISABLE; + +static void +netdev_offload_rebalance_set_policy(const char *policy) +{ + if (!policy) { + return; + } + + if (!strcmp(policy, "pps-rate")) { + netdev_offload_rebalance_policy = OFFLOAD_REBALANCE_POLICY_PPS_RATE; + } else if (!strcmp(policy, "disable")) { + netdev_offload_rebalance_policy = OFFLOAD_REBALANCE_POLICY_DISABLE; + } else { + VLOG_WARN("netdev: Invalid policy '%s'", policy); + return; + } + + VLOG_INFO("netdev: Using policy '%s'", policy); +} + +bool +netdev_is_offload_rebalance_policy_enabled(void) +{ + if (netdev_offload_rebalance_policy == OFFLOAD_REBALANCE_POLICY_PPS_RATE) { + return true; + } + return false; +} + static void netdev_ports_flow_init(void) { @@ -2494,12 +2531,18 @@ netdev_set_flow_api_enabled(const struct smap *ovs_other_config) if (ovsthread_once_start(&once)) { netdev_flow_api_enabled = true; + const char *offl_rebal_policy = NULL; VLOG_INFO("netdev: Flow API Enabled"); tc_set_policy(smap_get_def(ovs_other_config, "tc-policy", TC_POLICY_DEFAULT)); + offl_rebal_policy = smap_get_def(ovs_other_config, + "offload-rebalance", + OFFLOAD_REBALANCE_POLICY_DEFAULT); + netdev_offload_rebalance_set_policy(offl_rebal_policy); + netdev_ports_flow_init(); ovsthread_once_done(&once); diff --git a/lib/netdev.h b/lib/netdev.h index c941f1e1e..0ac2774c9 100644 --- a/lib/netdev.h +++ b/lib/netdev.h @@ -223,6 +223,8 @@ int netdev_init_flow_api(struct netdev *); uint32_t netdev_get_block_id(struct netdev *); bool netdev_is_flow_api_enabled(void); void netdev_set_flow_api_enabled(const struct smap *ovs_other_config); +bool netdev_is_offload_rebalance_policy_enabled(void); +#define OFFLOAD_REBALANCE_POLICY_DEFAULT "disable" struct dpif_port; int netdev_ports_insert(struct netdev *, const struct dpif_class *, diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index cf1524615..0910b66b1 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -967,7 +967,9 @@ udpif_revalidator(void *arg) dpif_flow_dump_destroy(udpif->dump); seq_change(udpif->dump_seq); - udpif_run_flow_rebalance(udpif); + if (netdev_is_offload_rebalance_policy_enabled()) { + udpif_run_flow_rebalance(udpif); + } duration = MAX(time_msec() - start_time, 1); udpif->dump_duration = duration; @@ -2741,7 +2743,8 @@ revalidate(struct revalidator *revalidator) } ukey->dump_seq = dump_seq; - if (result != UKEY_DELETE) { + if (netdev_is_offload_rebalance_policy_enabled() && + result != UKEY_DELETE) { udpif_update_flow_pps(udpif, ukey, f); } diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 76094852d..a0aad0204 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -475,6 +475,28 @@

+ +

+ Configures HW offload rebalancing, that allows to dynamically + offload and un-offload flows while an offload-device is out of + resources (OOR). The value specifies the criteria used to + select flows for offloading. The default value of 'disable', + disables this policy. To enable packets-per-second rate based + offload policy, set the value to 'pps-rate'. + Options: + disable - No offload rebalancing + pps-rate - Packets-per-second rate based policy +

+

+ This is only relevant if HW offloading is enabled (hw-offload). + When this policy is enabled, it also requires 'tc-policy' to + be set to 'skip_sw'. +

+

+ The default value is disable. +

+