From patchwork Wed Jun 9 13:09:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=LLzM58Ib; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=D8fjFZd3; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SC736nCz9sT6 for ; Wed, 9 Jun 2021 23:10:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B2A1560B01; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Io0d_HBeswpt; Wed, 9 Jun 2021 13:10:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2680E60A4B; Wed, 9 Jun 2021 13:10:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CBD96C000D; Wed, 9 Jun 2021 13:10:13 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 09E7AC000B for ; Wed, 9 Jun 2021 13:10:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D6708405F6 for ; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="LLzM58Ib"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="D8fjFZd3" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GkSabXzdCv1h for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2BC7C40560 for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 9805D2664; Wed, 9 Jun 2021 09:10:00 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=VvFaWPqls6Qd5 RnxckPORK0/ZbeCpqf2hLip2IBzXUk=; b=LLzM58Ib9Aziyx4lF5V9sg4rh0k8R n1R+z7iR5SxhseT537KL2DJy558RHzcQ8i9KnZeGB70yZRQ7U5CI3E/kdX5iovkh bgCgR9+E3TfGC1+I3BzXcgcPAhlycvHzD+6A6mUN5Vb1JeNWsjOqcBhIKt4EUGWa G37pigtQW3mqFgroA5hEHP9N7FyIByjP2U67HWAqD5+lqosDzgkruOHRJTW6zLQI 5YKIkPwZ6OjjB5TdCv4ZXfXQdse526A6BAzaUd6fqRb+LFm7Duz2p5NlSBIJ140N Hhg/wmwHmrWO9tgqloMROIZIbK7D5glyzERkzgJ4Ez1Wf3rTEzV+ltcGA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=VvFaWPqls6Qd5RnxckPORK0/ZbeCpqf2hLip2IBzXUk=; b=D8fjFZd3 fXXEA0+EyFv+/gHT8wiCcjJh2bt1ASospMKBzIP//C0qX+2wwPaGsmAEeaNlKh8p +HqkiMtY40YmGYyqNIsRRZdoVv8YfY4C6QhuNpwONF29fKVfTf9jdePDE/1TW4DC ZgMJWH8Qx21BtkQd7B5MLu/ylOszIpve7x9gse5nsr1x7bEqe5eQT/GhDaNeO5LM H8JovItMkagGbmZ4GegGj4OyZdFnjcOU6i0Tfs3Quff3csqX6yOINOl53HjgVIlV EJ9CUVdCO+Ly85xchV4RNt79rbRdDg/EfVG6RmlpfCVlnqdR2f2jDKRoUapXBgPb yUnYLo13fVLV/w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:59 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:18 +0200 Message-Id: <49a53d7be3f11f399733b922a7549c81c281a664.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 10/27] dpif-netdev: Implement hardware offloads stats query X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In the netdev datapath, keep track of the enqueued offloads between the PMDs and the offload thread. Additionally, query each netdev for their hardware offload counters. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index b666bc405..a20eeda4d 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -51,6 +51,7 @@ #include "hmapx.h" #include "id-pool.h" #include "ipf.h" +#include "mov-avg.h" #include "netdev.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -431,6 +432,7 @@ struct dp_offload_thread_item { struct match match; struct nlattr *actions; size_t actions_len; + long long int timestamp; struct ovs_list node; }; @@ -438,12 +440,18 @@ struct dp_offload_thread_item { struct dp_offload_thread { struct ovs_mutex mutex; struct ovs_list list; + uint64_t enqueued_item; + struct mov_avg_cma cma; + struct mov_avg_ema ema; pthread_cond_t cond; }; static struct dp_offload_thread dp_offload_thread = { .mutex = OVS_MUTEX_INITIALIZER, .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), + .enqueued_item = 0, + .cma = MOV_AVG_CMA_INITIALIZER, + .ema = MOV_AVG_EMA_INITIALIZER(100), }; static struct ovsthread_once offload_thread_once @@ -2632,6 +2640,7 @@ dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { ovs_mutex_lock(&dp_offload_thread.mutex); ovs_list_push_back(&dp_offload_thread.list, &offload->node); + dp_offload_thread.enqueued_item++; xpthread_cond_signal(&dp_offload_thread.cond); ovs_mutex_unlock(&dp_offload_thread.mutex); } @@ -2736,6 +2745,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; struct ovs_list *list; + long long int latency_us; const char *op; int ret; @@ -2748,6 +2758,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ovsrcu_quiesce_end(); } list = ovs_list_pop_front(&dp_offload_thread.list); + dp_offload_thread.enqueued_item--; offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); ovs_mutex_unlock(&dp_offload_thread.mutex); @@ -2768,6 +2779,10 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) OVS_NOT_REACHED(); } + latency_us = time_usec() - offload->timestamp; + mov_avg_cma_update(&dp_offload_thread.cma, latency_us); + mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + VLOG_DBG("%s to %s netdev flow "UUID_FMT, ret == 0 ? "succeed" : "failed", op, UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); @@ -2792,6 +2807,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -2824,6 +2840,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, memcpy(offload->actions, actions, actions_len); offload->actions_len = actions_len; + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -4209,6 +4226,77 @@ dpif_netdev_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, } } +static int +dpif_netdev_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats) +{ + enum { + DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED, + DP_NETDEV_HW_OFFLOADS_STATS_INSERTED, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV, + }; + const char *names[] = { + [DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED] = + " Enqueued offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = + " Inserted offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = + " Cumulative Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = + " Cumulative Latency stddev (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = + " Exponential Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = + " Exponential Latency stddev (us)", + }; + struct dp_netdev *dp = get_dp_netdev(dpif); + struct dp_netdev_port *port; + uint64_t nb_offloads; + size_t i; + + if (!netdev_is_flow_api_enabled()) { + return EINVAL; + } + + stats->size = ARRAY_SIZE(names); + stats->counters = xcalloc(stats->size, sizeof *stats->counters); + + nb_offloads = 0; + + ovs_mutex_lock(&dp->port_mutex); + HMAP_FOR_EACH (port, node, &dp->ports) { + uint64_t port_nb_offloads = 0; + + /* Do not abort on read error from a port, just report 0. */ + if (!netdev_flow_get_n_flows(port->netdev, &port_nb_offloads)) { + nb_offloads += port_nb_offloads; + } + } + ovs_mutex_unlock(&dp->port_mutex); + + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value = + dp_offload_thread.enqueued_item; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = + mov_avg_cma(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].value = + mov_avg_cma_std_dev(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].value = + mov_avg_ema(&dp_offload_thread.ema); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].value = + mov_avg_ema_std_dev(&dp_offload_thread.ema); + + for (i = 0; i < ARRAY_SIZE(names); i++) { + snprintf(stats->counters[i].name, sizeof(stats->counters[i].name), + "%s", names[i]); + } + + return 0; +} + /* Enable or Disable PMD auto load balancing. */ static void set_pmd_auto_lb(struct dp_netdev *dp, bool always_log) @@ -8483,7 +8571,7 @@ const struct dpif_class dpif_netdev_class = { dpif_netdev_flow_dump_thread_destroy, dpif_netdev_flow_dump_next, dpif_netdev_operate, - NULL, /* offload_stats_get */ + dpif_netdev_offload_stats_get, NULL, /* recv_set */ NULL, /* handlers_set */ dpif_netdev_set_config,