From patchwork Wed Sep 12 14:27:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li,Rongqing via dev" X-Patchwork-Id: 969050 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=openvswitch.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=broadcom.com header.i=@broadcom.com header.b="NSqy4q3e"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 429PDk0pb6z9sCD for ; Thu, 13 Sep 2018 00:24:38 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 0D5E3CB2; Wed, 12 Sep 2018 14:17:06 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 6E76DCAB for ; Wed, 12 Sep 2018 14:17:05 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id A35B7782 for ; Wed, 12 Sep 2018 14:17:04 +0000 (UTC) Received: by mail-wm0-f68.google.com with SMTP id y2-v6so2610214wma.1 for ; Wed, 12 Sep 2018 07:17:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9/fb3HKoGt37lupDwmoFdYqmOng9HKTrjWhATE04N5o=; b=NSqy4q3e5Tx0IrBTltToFhkxLv35fP5kGsa7cngHEsWhIRp+vV3I66Hqw3Pza3UO18 2tRSgUdCY5xX7YfC+6NDwYbuDsdLYNxT+QTNIdyFhHz2eR4+/9UfZrHFMzdzXlpEPPzv d9uB5jN+QHnJ0GIg5tD0L6DhK8rxQyaB+JBms= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9/fb3HKoGt37lupDwmoFdYqmOng9HKTrjWhATE04N5o=; b=R88XKdUkJsQJ3jisBBVX3T8hV1Pbb8jocq+MTxHanibkZG+H0IoSyqgH6oMvKy2mXU UHgbIaKE8iTStr9WdI/sJ3ptRe01jiFS5eLsScZECxkqgG0b9u4Zngq93XvMi2DZk+SR OH01Tu1b2oU9y2f/kp7DmbvTv579zYk/x4mF/0BW3ZJDf0nr6mSOIVVeh4tpdfBTbzU1 UI1kYtNJjeDxbndOQKQR7A7I1j7KISW+hXNEoXpYir5+1RJmPUeaAncmhAfNTejGyNo5 NcgjpN0tO6SHZfheTZcPAABgZUwYOZfD1jD3T8tXTHUOtLMS6gPgJHhzryx/yHgibo5E 7I+g== X-Gm-Message-State: APzg51DqwA2qq2ZgM/YrerRZjk5K5e/6xVk1dVggTqhORADwbgQQKwal R7/kFp4SC/VcI5CFt/wpKpJGaqfFaCk= X-Google-Smtp-Source: ANB0VdapeXN3QSjkWJriPJEhP0fs6cfMyNmAECYIj01aW0anSXsh1g0UgXRMKFvdrgdJLsVx0D01/w== X-Received: by 2002:a1c:6386:: with SMTP id x128-v6mr1787540wmb.104.1536761822670; Wed, 12 Sep 2018 07:17:02 -0700 (PDT) Received: from host-10-123-153-36.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id v6-v6sm843498wmc.43.2018.09.12.07.16.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Sep 2018 07:17:02 -0700 (PDT) To: dev@openvswitch.org Date: Wed, 12 Sep 2018 19:57:05 +0530 Message-Id: <20180912142706.19969-3-sriharsha.basavapatna@broadcom.com> X-Mailer: git-send-email 2.18.0.rc1.1.g6f333ff In-Reply-To: <20180912142706.19969-1-sriharsha.basavapatna@broadcom.com> References: <20180912142706.19969-1-sriharsha.basavapatna@broadcom.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v5 2/3] revalidator: Gather packets-per-second rate of flows X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Sriharsha Basavapatna via dev From: "Li,Rongqing via dev" Reply-To: Sriharsha Basavapatna MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is the second patch in the patch-set to support dynamic rebalancing of offloaded flows. The packets-per-second (pps) rate for each flow is computed in the context of revalidator threads when the flow stats are retrieved. The pps-rate is computed only after a flow is revalidated and is not scheduled for deletion. The parameters used to compute pps and the pps itself are saved in udpif_key since they need to be persisted across iterations of rebalancing. Signed-off-by: Sriharsha Basavapatna Co-authored-by: Venkat Duvvuru Signed-off-by: Venkat Duvvuru Reviewed-by: Sathya Perla --- lib/dpif-provider.h | 1 + ofproto/ofproto-dpif-upcall.c | 129 ++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+) diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index 873b6e3d4..7a71f5c0a 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -39,6 +39,7 @@ struct dpif { char *full_name; uint8_t netflow_engine_type; uint8_t netflow_engine_id; + long long int current_ms; }; void dpif_init(struct dpif *, const struct dpif_class *, const char *name, diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 62222079f..ef64b9c48 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -42,6 +42,8 @@ #include "tunnel.h" #include "unixctl.h" #include "openvswitch/vlog.h" +#include "lib/dpif-provider.h" +#include "lib/netdev-provider.h" #define MAX_QUEUE_LENGTH 512 #define UPCALL_MAX_BATCH 64 @@ -304,6 +306,15 @@ struct udpif_key { uint32_t key_recirc_id; /* Non-zero if reference is held by the ukey. */ struct recirc_refs recircs; /* Action recirc IDs with references held. */ + +#define OFFL_REBAL_INTVL_MSEC 3000 /* dynamic offload rebalance freq */ + struct netdev *in_netdev; /* in_odp_port's netdev */ + struct netdev *out_netdev; /* out_odp_port's netdev */ + bool offloaded; /* True if flow is offloaded */ + uint64_t flow_pps_rate; /* Packets-Per-Second rate */ + long long int flow_time; /* last pps update time */ + uint64_t flow_packets; /* #pkts seen in interval */ + uint64_t flow_backlog_packets; /* prev-mode #pkts (offl or kernel) */ }; /* Datapath operation with optional ukey attached. */ @@ -1667,6 +1678,11 @@ ukey_create__(const struct nlattr *key, size_t key_len, ukey->stats.used = used; ukey->xcache = NULL; + ukey->offloaded = false; + ukey->flow_time = 0; + ukey->in_netdev = ukey->out_netdev = NULL; + ukey->flow_packets = ukey->flow_backlog_packets = 0; + ukey->key_recirc_id = key_recirc_id; recirc_refs_init(&ukey->recircs); if (xout) { @@ -2442,6 +2458,115 @@ reval_op_init(struct ukey_op *op, enum reval_result result, } } +static void +ukey_netdev_unref(struct udpif_key *ukey) +{ + if (ukey->in_netdev) { + netdev_close(ukey->in_netdev); + ukey->in_netdev = NULL; + } + if (ukey->out_netdev) { + netdev_close(ukey->out_netdev); + ukey->out_netdev = NULL; + } +} + +/* + * Given a udpif_key, get its input and output ports (netdevs) by parsing + * the flow keys and actions. The flow may not contain flow attributes if + * it is a terse dump; read its attributes from the ukey and then parse + * the flow to get the port info. Save them in udpif_key. + */ +static void +ukey_to_flow_netdevs(struct udpif *udpif, struct udpif_key *ukey) +{ + const struct dpif *dpif = udpif->dpif; + const struct dpif_class *dpif_class = dpif->dpif_class; + const struct nlattr *actions; + const struct nlattr *a; + const struct nlattr *k; + size_t actions_len; + unsigned int left; + + /* Remove existing references to netdevs */ + ukey_netdev_unref(ukey); + + /* Read actions from ukey */ + ukey_get_actions(ukey, &actions, &actions_len); + + /* Capture the output port and get a reference to its netdev; we are + * only interested that the flow has an output port, so we just save the + * first port if there are multiple output actions associated with it. + */ + NL_ATTR_FOR_EACH (a, left, actions, actions_len) { + enum ovs_action_attr type = nl_attr_type(a); + if (type == OVS_ACTION_ATTR_OUTPUT) { + ukey->out_netdev = netdev_ports_get(nl_attr_get_odp_port(a), + dpif_class); + break; + } + } + + /* Now find the input port and get a reference to its netdev */ + NL_ATTR_FOR_EACH (k, left, ukey->key, ukey->key_len) { + enum ovs_key_attr type = nl_attr_type(k); + + if (type == OVS_KEY_ATTR_IN_PORT) { + ukey->in_netdev = netdev_ports_get(nl_attr_get_odp_port(k), + dpif_class); + } else if (type == OVS_KEY_ATTR_TUNNEL) { + struct flow_tnl tnl; + enum odp_key_fitness res; + + if (ukey->in_netdev) { + netdev_close(ukey->in_netdev); + ukey->in_netdev = NULL; + } + res = odp_tun_key_from_attr(k, &tnl); + if (res != ODP_FIT_ERROR) { + ukey->in_netdev = flow_get_tunnel_netdev(&tnl); + break; + } + } + } +} + +static uint64_t +udpif_flow_packet_delta(struct udpif_key *ukey, const struct dpif_flow *f) +{ + return f->stats.n_packets + ukey->flow_backlog_packets - + ukey->flow_packets; +} + +static long long int +udpif_flow_time_delta(struct udpif *udpif, struct udpif_key *ukey) +{ + return (udpif->dpif->current_ms - ukey->flow_time) / 1000; +} + +/* Gather pps-rate for the given dpif_flow and save it in its ukey */ +static void +udpif_update_flow_pps(struct udpif *udpif, struct udpif_key *ukey, + const struct dpif_flow *f) +{ + uint64_t pps; + + /* Update pps-rate only when we are close to rebalance interval */ + if (udpif->dpif->current_ms - ukey->flow_time < OFFL_REBAL_INTVL_MSEC) { + return; + } + + ovs_assert(f->stats.n_packets + ukey->flow_backlog_packets >= + ukey->flow_packets); + + ukey->offloaded = f->attrs.offloaded; + pps = udpif_flow_packet_delta(ukey, f) / + udpif_flow_time_delta(udpif, ukey); + ukey->flow_pps_rate = pps; + ukey->flow_packets = ukey->flow_backlog_packets + f->stats.n_packets; + ukey->flow_time = udpif->dpif->current_ms; +} + static void revalidate(struct revalidator *revalidator) { @@ -2550,6 +2675,10 @@ revalidate(struct revalidator *revalidator) } ukey->dump_seq = dump_seq; + if (result != UKEY_DELETE) { + udpif_update_flow_pps(udpif, ukey, f); + } + if (result != UKEY_KEEP) { /* Takes ownership of 'recircs'. */ reval_op_init(&ops[n_ops++], result, udpif, ukey, &recircs,