From patchwork Fri Oct 7 11:16:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1687358 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=jFoOLw1k; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MkQk92Bs1z20Zg for ; Fri, 7 Oct 2022 22:16:49 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 7083F41C26; Fri, 7 Oct 2022 11:16:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 7083F41C26 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=jFoOLw1k X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lUdw0iDZWebO; Fri, 7 Oct 2022 11:16:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id F0D2741BE1; Fri, 7 Oct 2022 11:16:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org F0D2741BE1 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BF04AC0033; Fri, 7 Oct 2022 11:16:44 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0F722C0033 for ; Fri, 7 Oct 2022 11:16:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4C92D41BE7 for ; Fri, 7 Oct 2022 11:16:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4C92D41BE7 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K_kcw_jG4QSl for ; Fri, 7 Oct 2022 11:16:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4207A41BA2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4207A41BA2 for ; Fri, 7 Oct 2022 11:16:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665141395; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qSqK7dLW5puUHukuOs9CHP57maA5BIh/Kq7dfwsq/QI=; b=jFoOLw1kK03DsTTp99+aqp/DQlO1Bo7U06E/tqzSNI1f2RlOkQnziRAOXzNzRpYYC3i5zP f5P0r2as1gntRQApWetK3bK/lEfW4h8BkC3CnmOvEkChl4GO5dqPDkeDqqLuafCzeq+XUl Rrg8uTnm2r6JS70ZNUkeaeSFgD5z/Mc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-656-8Cgtf7dmMM-y3OMmvL0Epw-1; Fri, 07 Oct 2022 07:16:32 -0400 X-MC-Unique: 8Cgtf7dmMM-y3OMmvL0Epw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8A53095E3A2; Fri, 7 Oct 2022 11:16:31 +0000 (UTC) Received: from localhost.localdomain (unknown [10.40.193.115]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69948C15BA4; Fri, 7 Oct 2022 11:16:30 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Fri, 7 Oct 2022 13:16:13 +0200 Message-Id: <20221007111613.1695524-4-david.marchand@redhat.com> In-Reply-To: <20221007111613.1695524-1-david.marchand@redhat.com> References: <20221007111613.1695524-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: maxime.coquelin@redhat.com Subject: [ovs-dev] [RFC dpdk-latest v2 3/3] netdev-dpdk: Add per virtqueue statistics. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Request per virtqueue statistics from the vhost library and expose them as per port OVS custom stats. Note: - the vhost stats API is experimental at the moment, this patch is sent as a demonstrator, - we may drop maintaining rx stats in OVS itself and instead aggregate the per vq stats, this is something to investigate, - a unit test is missing, Signed-off-by: David Marchand --- lib/netdev-dpdk.c | 203 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 194 insertions(+), 9 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 132ebb2843..3db5944977 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -532,11 +532,20 @@ struct netdev_dpdk { ); PADDED_MEMBERS(CACHE_LINE_SIZE, - /* Names of all XSTATS counters */ - struct rte_eth_xstat_name *rte_xstats_names; - int rte_xstats_names_size; - int rte_xstats_ids_size; - uint64_t *rte_xstats_ids; + union { + struct { + /* Names of all XSTATS counters */ + struct rte_eth_xstat_name *rte_xstats_names; + int rte_xstats_names_size; + int rte_xstats_ids_size; + uint64_t *rte_xstats_ids; + }; + struct { + /* Names of all vhost stats */ + struct rte_vhost_stat_name *vhost_stat_names; + int vhost_stat_size; + }; + }; ); }; @@ -552,6 +561,7 @@ static int netdev_dpdk_get_sw_custom_stats(const struct netdev *, struct netdev_custom_stats *); static void netdev_dpdk_configure_xstats(struct netdev_dpdk *dev); static void netdev_dpdk_clear_xstats(struct netdev_dpdk *dev); +static void netdev_dpdk_vhost_clear_stats(struct netdev_dpdk *dev); int netdev_dpdk_get_vid(const struct netdev_dpdk *dev); @@ -1586,6 +1596,7 @@ netdev_dpdk_vhost_destruct(struct netdev *netdev) dev->vhost_id = NULL; rte_free(dev->vhost_rxq_enabled); + netdev_dpdk_vhost_clear_stats(dev); common_destruct(dev); ovs_mutex_unlock(&dpdk_mutex); @@ -3039,6 +3050,80 @@ netdev_dpdk_vhost_get_stats(const struct netdev *netdev, return 0; } +static int +netdev_dpdk_vhost_get_custom_stats(const struct netdev *netdev, + struct netdev_custom_stats *custom_stats) +{ + netdev_dpdk_get_sw_custom_stats(netdev, custom_stats); + +#ifdef ALLOW_EXPERIMENTAL_API + struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + + ovs_mutex_lock(&dev->mutex); + + int vid = netdev_dpdk_get_vid(dev); + + if (vid >= 0 && dev->vhost_stat_size > 0) { + struct rte_vhost_stat *vhost_stats; + int stat_offset; + int sw_stats_size; + + vhost_stats = xcalloc(dev->vhost_stat_size, sizeof *vhost_stats); + + stat_offset = 0; + + for (int q = 0; q < dev->up.n_rxq; q++) { + int qid = q * VIRTIO_QNUM + VIRTIO_TXQ; + int err; + + err = rte_vhost_vring_stats_get(vid, qid, + &vhost_stats[stat_offset], + dev->vhost_stat_size + - stat_offset); + if (err < 0 || stat_offset + err > dev->vhost_stat_size) { + goto fail; + } + stat_offset += err; + } + + for (int q = 0; q < dev->up.n_txq; q++) { + int qid = q * VIRTIO_QNUM; + int err; + + err = rte_vhost_vring_stats_get(vid, qid, + &vhost_stats[stat_offset], + dev->vhost_stat_size + - stat_offset); + if (err < 0 || stat_offset + err > dev->vhost_stat_size) { + goto fail; + } + stat_offset += err; + } + + sw_stats_size = custom_stats->size; + custom_stats->size += dev->vhost_stat_size; + custom_stats->counters = xrealloc(custom_stats->counters, + custom_stats->size * + sizeof *custom_stats->counters); + + for (int i = 0; i < stat_offset; i++) { + ovs_strlcpy(custom_stats->counters[sw_stats_size + i].name, + dev->vhost_stat_names[i].name, + NETDEV_CUSTOM_STATS_NAME_SIZE); + custom_stats->counters[sw_stats_size + i].value = + vhost_stats[i].value; + } + +fail: + free(vhost_stats); + } + + ovs_mutex_unlock(&dev->mutex); +#endif + + return 0; +} + static void netdev_dpdk_convert_xstats(struct netdev_stats *stats, const struct rte_eth_xstat *xstats, @@ -5014,12 +5099,107 @@ out: return err; } +static void +netdev_dpdk_vhost_clear_stats(struct netdev_dpdk *dev) + OVS_REQUIRES(dev->mutex) +{ + free(dev->vhost_stat_names); + dev->vhost_stat_names = NULL; + dev->vhost_stat_size = 0; +}; + +static void +netdev_dpdk_vhost_configure_stats(struct netdev_dpdk *dev OVS_UNUSED) + OVS_REQUIRES(dev->mutex) +{ +#ifdef ALLOW_EXPERIMENTAL_API + struct rte_vhost_stat_name *vhost_stat_names = NULL; + int vhost_stat_size; + int stat_offset; + int vid; + + vid = netdev_dpdk_get_vid(dev); + if (vid < 0) { + goto fail; + } + + vhost_stat_size = 0; + + for (int q = 0; q < dev->up.n_rxq; q++) { + int qid = q * VIRTIO_QNUM + VIRTIO_TXQ; + int err; + + err = rte_vhost_vring_stats_get_names(vid, qid, NULL, 0); + if (err < 0) { + goto fail; + } + vhost_stat_size += err; + } + + for (int q = 0; q < dev->up.n_txq; q++) { + int qid = q * VIRTIO_QNUM; + int err; + + err = rte_vhost_vring_stats_get_names(vid, qid, NULL, 0); + if (err < 0) { + goto fail; + } + vhost_stat_size += err; + } + + vhost_stat_names = xcalloc(vhost_stat_size, sizeof *vhost_stat_names); + + stat_offset = 0; + + for (int q = 0; q < dev->up.n_rxq; q++) { + int qid = q * VIRTIO_QNUM + VIRTIO_TXQ; + int err; + + err = rte_vhost_vring_stats_get_names(vid, qid, + &vhost_stat_names[stat_offset], + vhost_stat_size - stat_offset); + if (err < 0 || stat_offset + err > vhost_stat_size) { + goto fail; + } + stat_offset += err; + } + + for (int q = 0; q < dev->up.n_txq; q++) { + int qid = q * VIRTIO_QNUM; + int err; + + err = rte_vhost_vring_stats_get_names(vid, qid, + &vhost_stat_names[stat_offset], + vhost_stat_size - stat_offset); + if (err < 0 || stat_offset + err > vhost_stat_size) { + goto fail; + } + stat_offset += err; + } + + dev->vhost_stat_names = vhost_stat_names; + vhost_stat_names = NULL; + dev->vhost_stat_size = vhost_stat_size; + +fail: + free(vhost_stat_names); +#endif +} + static int dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) OVS_REQUIRES(dev->mutex) { - dev->up.n_txq = dev->requested_n_txq; - dev->up.n_rxq = dev->requested_n_rxq; + if (dev->up.n_rxq != dev->requested_n_rxq + || dev->up.n_txq != dev->requested_n_txq + || dev->vhost_stat_size <= 0) { + + dev->up.n_txq = dev->requested_n_txq; + dev->up.n_rxq = dev->requested_n_rxq; + + netdev_dpdk_vhost_clear_stats(dev); + netdev_dpdk_vhost_configure_stats(dev); + } /* Always keep RX queue 0 enabled for implementations that won't * report vring states. */ @@ -5090,6 +5270,11 @@ netdev_dpdk_vhost_client_reconfigure(struct netdev *netdev) /* Register client-mode device. */ vhost_flags |= RTE_VHOST_USER_CLIENT; +#ifdef ALLOW_EXPERIMENTAL_API + /* Extended per vq statistics. */ + vhost_flags |= RTE_VHOST_USER_NET_STATS_ENABLE; +#endif + /* There is no support for multi-segments buffers. */ vhost_flags |= RTE_VHOST_USER_LINEARBUF_SUPPORT; @@ -5489,7 +5674,7 @@ static const struct netdev_class dpdk_vhost_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv, @@ -5505,7 +5690,7 @@ static const struct netdev_class dpdk_vhost_client_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_client_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv,