From patchwork Wed Nov 24 21:23:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1559371 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=iGMjUdRb; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HzvCk44Nhz9sRN for ; Thu, 25 Nov 2021 08:24:34 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2F02A4092C; Wed, 24 Nov 2021 21:24:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kv2JG_8AmPlG; Wed, 24 Nov 2021 21:24:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 3895640922; Wed, 24 Nov 2021 21:24:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F1E36C002F; Wed, 24 Nov 2021 21:24:29 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 06D4AC002F for ; Wed, 24 Nov 2021 21:24:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E956260617 for ; Wed, 24 Nov 2021 21:24:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HN6gGQE039NO for ; Wed, 24 Nov 2021 21:24:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id E918461BBD for ; Wed, 24 Nov 2021 21:24:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637789067; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sLiiHLDwvKrPKC+y8kUEBvw3DuaFF/dY9/RgUmcBVcE=; b=iGMjUdRb+MJK8bGfcK/fHR24ZjQ7byL0BbkWZ3qafUIWjGxFTB6OJuDCwJ8UrpEt32lqJC 6BTqHMHQA9WJHqVbUgFXw8ST4qrlDpCpN0fgplUP6O1qo118CXc80eL3XabpANxE7vw0G6 1lFFr8bgCwZoIbgSG//4DBLyu77N30I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-595-tFUGbmOlN0mEL1uK9qFnWg-1; Wed, 24 Nov 2021 16:24:21 -0500 X-MC-Unique: tFUGbmOlN0mEL1uK9qFnWg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A7396100CFAF; Wed, 24 Nov 2021 21:24:20 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id B880A5D9CA; Wed, 24 Nov 2021 21:24:18 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Wed, 24 Nov 2021 22:23:57 +0100 Message-Id: <20211124212400.70613-2-maxime.coquelin@redhat.com> In-Reply-To: <20211124212400.70613-1-maxime.coquelin@redhat.com> References: <20211124212400.70613-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH 1/4] netdev-dpdk: Introduce per rxq/txq Vhost-user statistics X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" HXPS feature will enable steering Tx packets on transmist queues based on their hashes. In order to test the feature, it is needed to be able to get the per-queue statistics for Vhost-user ports. This patch introduces "bytes", "packets" and "error" per-queue custom statistics for Vhost-user ports. Suggested-by David Marchand Signed-off-by: Maxime Coquelin --- lib/netdev-dpdk.c | 143 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 135 insertions(+), 8 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index ca92c947a..e80d5b4ab 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -192,6 +192,13 @@ static const struct vhost_device_ops virtio_net_device_ops = .guest_notified = vhost_guest_notified, }; +/* Custome software per-queue stats for Vhost ports */ +struct netdev_dpdk_vhost_q_stats { + uint64_t bytes; + uint64_t packets; + uint64_t errors; +}; + /* Custom software stats for dpdk ports */ struct netdev_dpdk_sw_stats { /* No. of retries when unable to transmit. */ @@ -206,6 +213,10 @@ struct netdev_dpdk_sw_stats { uint64_t rx_qos_drops; /* Packet drops in HWOL processing. */ uint64_t tx_invalid_hwol_drops; + /* Per-queue Vhost Tx stats */ + struct netdev_dpdk_vhost_q_stats *txq; + /* Per-queue Vhost Rx stats */ + struct netdev_dpdk_vhost_q_stats *rxq; }; enum dpdk_dev_type { @@ -1276,6 +1287,13 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->sw_stats = xzalloc(sizeof *dev->sw_stats); dev->sw_stats->tx_retries = (dev->type == DPDK_DEV_VHOST) ? 0 : UINT64_MAX; + if (dev->type == DPDK_DEV_VHOST) { + dev->sw_stats->txq = xcalloc(netdev->n_txq, + sizeof *dev->sw_stats->txq); + dev->sw_stats->rxq = xcalloc(netdev->n_rxq, + sizeof *dev->sw_stats->rxq); + } + return 0; } @@ -2353,17 +2371,21 @@ netdev_dpdk_vhost_update_rx_size_counters(struct netdev_stats *stats, } static inline void -netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, +netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, int qid, struct dp_packet **packets, int count, int qos_drops) { struct netdev_stats *stats = &dev->stats; + struct netdev_dpdk_vhost_q_stats *q_stats = &dev->sw_stats->rxq[qid]; struct dp_packet *packet; unsigned int packet_size; int i; stats->rx_packets += count; + q_stats->packets += count; stats->rx_dropped += qos_drops; + q_stats->errors += qos_drops; + for (i = 0; i < count; i++) { packet = packets[i]; packet_size = dp_packet_size(packet); @@ -2374,6 +2396,7 @@ netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, * further processing. */ stats->rx_errors++; stats->rx_length_errors++; + q_stats->errors++; continue; } @@ -2385,6 +2408,7 @@ netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, } stats->rx_bytes += packet_size; + q_stats->bytes += packet_size; } if (OVS_UNLIKELY(qos_drops)) { @@ -2437,7 +2461,7 @@ netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq, } rte_spinlock_lock(&dev->stats_lock); - netdev_dpdk_vhost_update_rx_counters(dev, batch->packets, + netdev_dpdk_vhost_update_rx_counters(dev, qid, batch->packets, nb_rx, qos_drops); rte_spinlock_unlock(&dev->stats_lock); @@ -2551,7 +2575,7 @@ netdev_dpdk_filter_packet_len(struct netdev_dpdk *dev, struct rte_mbuf **pkts, } static inline void -netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, +netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, int qid, struct dp_packet **packets, int attempted, struct netdev_dpdk_sw_stats *sw_stats_add) @@ -2561,14 +2585,20 @@ netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, sw_stats_add->tx_failure_drops + sw_stats_add->tx_invalid_hwol_drops; struct netdev_stats *stats = &dev->stats; + struct netdev_dpdk_vhost_q_stats *q_stats = &dev->sw_stats->txq[qid]; int sent = attempted - dropped; int i; stats->tx_packets += sent; + q_stats->packets += sent; stats->tx_dropped += dropped; + q_stats->errors += dropped; for (i = 0; i < sent; i++) { - stats->tx_bytes += dp_packet_size(packets[i]); + uint64_t bytes = dp_packet_size(packets[i]); + + stats->tx_bytes += bytes; + q_stats->bytes += bytes; } if (OVS_UNLIKELY(dropped || sw_stats_add->tx_retries)) { @@ -2656,7 +2686,7 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, sw_stats_add.tx_retries = MIN(retries, max_retries); rte_spinlock_lock(&dev->stats_lock); - netdev_dpdk_vhost_update_tx_counters(dev, pkts, total_packets, + netdev_dpdk_vhost_update_tx_counters(dev, qid, pkts, total_packets, &sw_stats_add); rte_spinlock_unlock(&dev->stats_lock); @@ -3286,6 +3316,72 @@ netdev_dpdk_get_sw_custom_stats(const struct netdev *netdev, return 0; } +static int +netdev_dpdk_vhost_get_custom_stats(const struct netdev *netdev, + struct netdev_custom_stats *custom_stats) +{ + struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + int sw_stats_size, i, j; + + netdev_dpdk_get_sw_custom_stats(netdev, custom_stats); + + ovs_mutex_lock(&dev->mutex); + + sw_stats_size = custom_stats->size; + custom_stats->size += netdev->n_rxq * sizeof(*dev->sw_stats->rxq) / + sizeof(uint64_t); + custom_stats->size += netdev->n_txq * sizeof(*dev->sw_stats->txq) / + sizeof(uint64_t); + + custom_stats->counters = xrealloc(custom_stats->counters, + custom_stats->size * + sizeof *custom_stats->counters); + + j = 0; + for (i = 0; i < netdev->n_rxq; i++) { + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "rx_q%d_bytes", i); + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "rx_q%d_packets", i); + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "rx_q%d_errors", i); + } + + for (i = 0; i < netdev->n_txq; i++) { + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "tx_q%d_bytes", i); + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "tx_q%d_packets", i); + snprintf(custom_stats->counters[sw_stats_size + j++].name, + NETDEV_CUSTOM_STATS_NAME_SIZE, "tx_q%d_errors", i); + } + + rte_spinlock_lock(&dev->stats_lock); + + j = 0; + for (i = 0; i < netdev->n_rxq; i++) { + struct netdev_dpdk_vhost_q_stats *rxq_stats = &dev->sw_stats->rxq[i]; + + custom_stats->counters[sw_stats_size + j++].value = rxq_stats->bytes; + custom_stats->counters[sw_stats_size + j++].value = rxq_stats->packets; + custom_stats->counters[sw_stats_size + j++].value = rxq_stats->errors; + } + + for (i = 0; i < netdev->n_txq; i++) { + struct netdev_dpdk_vhost_q_stats *txq_stats = &dev->sw_stats->txq[i]; + + custom_stats->counters[sw_stats_size + j++].value = txq_stats->bytes; + custom_stats->counters[sw_stats_size + j++].value = txq_stats->packets; + custom_stats->counters[sw_stats_size + j++].value = txq_stats->errors; + } + + rte_spinlock_unlock(&dev->stats_lock); + + ovs_mutex_unlock(&dev->mutex); + + return 0; +} + static int netdev_dpdk_get_features(const struct netdev *netdev, enum netdev_features *current, @@ -5043,9 +5139,12 @@ static int dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) OVS_REQUIRES(dev->mutex) { + int old_n_txq = dev->up.n_txq; + int old_n_rxq = dev->up.n_rxq; + int err; + dev->up.n_txq = dev->requested_n_txq; dev->up.n_rxq = dev->requested_n_rxq; - int err; /* Always keep RX queue 0 enabled for implementations that won't * report vring states. */ @@ -5063,6 +5162,34 @@ dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) netdev_dpdk_remap_txqs(dev); + if (dev->up.n_txq != old_n_txq) { + struct netdev_dpdk_vhost_q_stats *old_txq_stats, *new_txq_stats; + + new_txq_stats = xcalloc(dev->up.n_txq, sizeof *dev->sw_stats->txq); + rte_spinlock_lock(&dev->stats_lock); + old_txq_stats = dev->sw_stats->txq; + memcpy(new_txq_stats, old_txq_stats, + MIN(dev->up.n_txq, old_n_txq) * sizeof *dev->sw_stats->txq); + dev->sw_stats->txq = new_txq_stats; + rte_spinlock_unlock(&dev->stats_lock); + free(old_txq_stats); + + } + + if (dev->up.n_rxq != old_n_rxq) { + struct netdev_dpdk_vhost_q_stats *old_rxq_stats, *new_rxq_stats; + + new_rxq_stats = xcalloc(dev->up.n_rxq, sizeof *dev->sw_stats->rxq); + rte_spinlock_lock(&dev->stats_lock); + old_rxq_stats = dev->sw_stats->rxq; + memcpy(new_rxq_stats, old_rxq_stats, + MIN(dev->up.n_rxq, old_n_rxq) * sizeof *dev->sw_stats->rxq); + dev->sw_stats->rxq = new_rxq_stats; + rte_spinlock_unlock(&dev->stats_lock); + free(old_rxq_stats); + + } + err = netdev_dpdk_mempool_configure(dev); if (!err) { /* A new mempool was created or re-used. */ @@ -5467,7 +5594,7 @@ static const struct netdev_class dpdk_vhost_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv, @@ -5483,7 +5610,7 @@ static const struct netdev_class dpdk_vhost_client_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_client_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv, From patchwork Wed Nov 24 21:23:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1559372 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=e9zzZ828; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HzvCn72nMz9sRN for ; Thu, 25 Nov 2021 08:24:37 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 78AA5405A5; Wed, 24 Nov 2021 21:24:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SQbbFqGleGCd; Wed, 24 Nov 2021 21:24:34 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id D4D2A40554; Wed, 24 Nov 2021 21:24:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4381BC002F; Wed, 24 Nov 2021 21:24:31 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 80454C000A for ; Wed, 24 Nov 2021 21:24:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5663D40529 for ; Wed, 24 Nov 2021 21:24:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZYYDYd58Xs26 for ; Wed, 24 Nov 2021 21:24:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 0A7AD40531 for ; Wed, 24 Nov 2021 21:24:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637789066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xRFRNIky83xCUBuueeADWYXaJYiE4jcz2PdP+Dv42gU=; b=e9zzZ828vR4QDvXJ19mgeMEeERjgzuwHowae2XIZcKt3xtPehGylRYb5BpwbL28t07rQg4 5BLMlUOaSIKbFo0TG3I2ttHLbII8gGr5FERRdDG4zZ0k6RUb0AUJED5JD7Q6zNbXtsLsct 5OVZlvvFV9lcvXzNFy9GhEpz+1W4fXk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-286-RREWwRWWNluFpRtx_PIfig-1; Wed, 24 Nov 2021 16:24:23 -0500 X-MC-Unique: RREWwRWWNluFpRtx_PIfig-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A59FE100CFAF; Wed, 24 Nov 2021 21:24:22 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 011535DF21; Wed, 24 Nov 2021 21:24:20 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Wed, 24 Nov 2021 22:23:58 +0100 Message-Id: <20211124212400.70613-3-maxime.coquelin@redhat.com> In-Reply-To: <20211124212400.70613-1-maxime.coquelin@redhat.com> References: <20211124212400.70613-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH 2/4] dpif-netdev: Introduce Tx queue mode X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A boolean is currently used to differenciate between the static and XPS Tx queue modes. Since we are going to introduce a new RSS mode, replace this boolean with an enum. This patch does not introduce functionnal changes. Signed-off-by: Maxime Coquelin --- lib/dpif-netdev.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 69d7ec26e..0407f30b3 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -385,10 +385,15 @@ struct dp_netdev_rxq { atomic_ullong cycles_intrvl[PMD_INTERVAL_MAX]; }; +enum txq_mode { + TXQ_MODE_STATIC, + TXQ_MODE_XPS, +}; + /* A port in a netdev-based datapath. */ struct dp_netdev_port { odp_port_t port_no; - bool dynamic_txqs; /* If true XPS will be used. */ + enum txq_mode txq_mode; /* static, XPS */ bool need_reconfigure; /* True if we should reconfigure netdev. */ struct netdev *netdev; struct hmap_node node; /* Node in dp_netdev's 'ports'. */ @@ -4591,24 +4596,25 @@ dp_netdev_pmd_flush_output_on_port(struct dp_netdev_pmd_thread *pmd, int i; int tx_qid; int output_cnt; - bool dynamic_txqs; + bool concurrent_txqs; struct cycle_timer timer; uint64_t cycles; uint32_t tx_flush_interval; cycle_timer_start(&pmd->perf_stats, &timer); - dynamic_txqs = p->port->dynamic_txqs; - if (dynamic_txqs) { + if (p->port->txq_mode == TXQ_MODE_XPS) { tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p); + concurrent_txqs = true; } else { tx_qid = pmd->static_tx_qid; + concurrent_txqs = false; } output_cnt = dp_packet_batch_size(&p->output_pkts); ovs_assert(output_cnt > 0); - netdev_send(p->port->netdev, tx_qid, &p->output_pkts, dynamic_txqs); + netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); dp_packet_batch_init(&p->output_pkts); /* Update time of the next flush. */ @@ -5761,14 +5767,14 @@ reconfigure_datapath(struct dp_netdev *dp) * 'port->need_reconfigure', because netdev_is_reconf_required() can * change at any time. * Also mark for reconfiguration all ports which will likely change their - * 'dynamic_txqs' parameter. It's required to stop using them before + * 'txq_mode' parameter. It's required to stop using them before * changing this setting and it's simpler to mark ports here and allow * 'pmd_remove_stale_ports' to remove them from threads. There will be * no actual reconfiguration in 'port_reconfigure' because it's * unnecessary. */ HMAP_FOR_EACH (port, node, &dp->ports) { if (netdev_is_reconf_required(port->netdev) - || (port->dynamic_txqs + || ((port->txq_mode == TXQ_MODE_XPS) != (netdev_n_txq(port->netdev) < wanted_txqs))) { port->need_reconfigure = true; } @@ -5804,7 +5810,8 @@ reconfigure_datapath(struct dp_netdev *dp) seq_change(dp->port_seq); port_destroy(port); } else { - port->dynamic_txqs = netdev_n_txq(port->netdev) < wanted_txqs; + port->txq_mode = (netdev_n_txq(port->netdev) < wanted_txqs) ? + TXQ_MODE_XPS : TXQ_MODE_STATIC; } } @@ -7818,7 +7825,7 @@ dpif_netdev_xps_revalidate_pmd(const struct dp_netdev_pmd_thread *pmd, long long interval; HMAP_FOR_EACH (tx, node, &pmd->send_port_cache) { - if (!tx->port->dynamic_txqs) { + if (tx->port->txq_mode != TXQ_MODE_XPS) { continue; } interval = pmd->ctx.now - tx->last_used; From patchwork Wed Nov 24 21:23:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1559373 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Y15bI1ud; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HzvCs6hDhz9sRN for ; Thu, 25 Nov 2021 08:24:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 202E982564; Wed, 24 Nov 2021 21:24:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JOXYQ-EkCY6G; Wed, 24 Nov 2021 21:24:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6076782425; Wed, 24 Nov 2021 21:24:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 95980C0041; Wed, 24 Nov 2021 21:24:33 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1EA28C0044 for ; Wed, 24 Nov 2021 21:24:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 8302C4053A for ; Wed, 24 Nov 2021 21:24:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uRx6sJIl5Sax for ; Wed, 24 Nov 2021 21:24:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 13A6940531 for ; Wed, 24 Nov 2021 21:24:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637789069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jkBH+Bs4BI4C8GnjJyEqauxKycoEC6AhfOSj0shJ2vc=; b=Y15bI1udJSYJxeGmgJSSgJhBDNdhUrIfRGEhH2hoOBHPxaXBRMR2gsCfJhbL6XKMEoqnqU YKDoT1xzpxWV6YUX8CtTRTU8JEczLTXZ6ZCAG1t9hj4xGNPaCVVuudZfWKIZpRKzo9tEZ6 eY+vUj3B84cGsPoO7Xct39c0IlMwjD4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-296-CDDUd5DOMSaMJiaipqTLDg-1; Wed, 24 Nov 2021 16:24:25 -0500 X-MC-Unique: CDDUd5DOMSaMJiaipqTLDg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A301C801B10; Wed, 24 Nov 2021 21:24:24 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 011BF5D9CA; Wed, 24 Nov 2021 21:24:22 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Wed, 24 Nov 2021 22:23:59 +0100 Message-Id: <20211124212400.70613-4-maxime.coquelin@redhat.com> In-Reply-To: <20211124212400.70613-1-maxime.coquelin@redhat.com> References: <20211124212400.70613-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH 3/4] dpif-netdev: Add HXPS Tx queue mode X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch adds a new HXPS Tx mode that distributes the traffic on all the Tx queues, whatever the number of PMD threads. It would be useful for guests expecting traffic to be distributed on all the vCPUs. The idea here is to re-use the 5-tuple hash of the packets, already computed to build the flows batches (and so it does not provide flexibility on which fields are part of the hash). There are also no user-configurable indirection table, given the feature is transparent to the guest. The queue selection is just a modulo operation between the packet hash and the number of Tx queues. There are no (at least intentionnally) functionnal changes for the existing XPS and static modes. There should not be noticeable performance changes for these modes (only one more branch in the hot path). For the HXPS mode, performance could be impacted due to locking when multiple PMD threads are in use (same as XPS mode) and also because of the second level of batching. Regarding the batching, the existing Tx port output_pkts is not modified. It means that at maximum, NETDEV_MAX_BURST can be batched for all the Tx queues. A second level of batching is done in dp_netdev_pmd_flush_output_on_port(), only for this HXPS mode. Signed-off-by: Maxime Coquelin --- Documentation/automake.mk | 1 + Documentation/topics/dpdk/hxps.rst | 51 +++++++++++++++++++ Documentation/topics/dpdk/index.rst | 1 + lib/dpif-netdev.c | 78 +++++++++++++++++++++++------ 4 files changed, 117 insertions(+), 14 deletions(-) create mode 100644 Documentation/topics/dpdk/hxps.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index 137cc57c5..c982207d5 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -33,6 +33,7 @@ DOC_SOURCE = \ Documentation/topics/datapath.rst \ Documentation/topics/design.rst \ Documentation/topics/dpdk/index.rst \ + Documentation/topics/dpdk/hxps.rst \ Documentation/topics/dpdk/bridge.rst \ Documentation/topics/dpdk/jumbo-frames.rst \ Documentation/topics/dpdk/memory.rst \ diff --git a/Documentation/topics/dpdk/hxps.rst b/Documentation/topics/dpdk/hxps.rst new file mode 100644 index 000000000..1395f88c3 --- /dev/null +++ b/Documentation/topics/dpdk/hxps.rst @@ -0,0 +1,51 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +============================= +Hash-based Tx packet steering +============================= + +HXPS mode distributes the traffic on all the port transmit queues, whatever the +number of PMD threads. Queue selection is based on the 5-tuples hash already +computed to build the flows batches, the selected queue being the modulo +between the hash and the number of Tx queues of the port. + +HXPS may be used for example with Vhost-user ports, when the number of vCPUs +and queues of the guest are greater than the number of PMD threads, aq without +HXPS, the Tx queues used would be limited to the number of PMD. + +Hash-based Tx packet steering may have an impact on the performance, given the +Tx lock acquisition is required and a second level of batching is performed. + +This feature is disabled by default. + +Usage +~~~~~ + +To enable HXPS:: + + $ ovs-vsctl set Interface other_config:hxps=true + +To disable HXPS:: + + $ ovs-vsctl set Interface other_config:hxps=false diff --git a/Documentation/topics/dpdk/index.rst b/Documentation/topics/dpdk/index.rst index a5be5e344..ab6132357 100644 --- a/Documentation/topics/dpdk/index.rst +++ b/Documentation/topics/dpdk/index.rst @@ -39,3 +39,4 @@ DPDK Support /topics/dpdk/qos /topics/dpdk/jumbo-frames /topics/dpdk/memory + /topics/dpdk/hxps diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 0407f30b3..966741751 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -388,12 +388,13 @@ struct dp_netdev_rxq { enum txq_mode { TXQ_MODE_STATIC, TXQ_MODE_XPS, + TXQ_MODE_HXPS, }; /* A port in a netdev-based datapath. */ struct dp_netdev_port { odp_port_t port_no; - enum txq_mode txq_mode; /* static, XPS */ + enum txq_mode txq_mode; /* static, XPS, HXPS */ bool need_reconfigure; /* True if we should reconfigure netdev. */ struct netdev *netdev; struct hmap_node node; /* Node in dp_netdev's 'ports'. */ @@ -405,6 +406,7 @@ struct dp_netdev_port { bool emc_enabled; /* If true EMC will be used. */ char *type; /* Port type as requested by user. */ char *rxq_affinity_list; /* Requested affinity of rx queues. */ + bool txq_hxps; /* Tx HXPS mode will be used */ }; static bool dp_netdev_flow_ref(struct dp_netdev_flow *); @@ -440,6 +442,7 @@ struct tx_port { struct hmap_node node; long long flush_time; struct dp_packet_batch output_pkts; + struct dp_packet_batch *txq_pkts; /* Only for HXPS mode */ struct dp_netdev_rxq *output_pkts_rxqs[NETDEV_MAX_BURST]; }; @@ -4435,6 +4438,7 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, int error = 0; const char *affinity_list = smap_get(cfg, "pmd-rxq-affinity"); bool emc_enabled = smap_get_bool(cfg, "emc-enable", true); + bool txq_hxps = smap_get_bool(cfg, "hxps", false); ovs_mutex_lock(&dp->port_mutex); error = get_port_by_number(dp, port_no, &port); @@ -4476,19 +4480,28 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, } /* Checking for RXq affinity changes. */ - if (!netdev_is_pmd(port->netdev) - || nullable_string_is_equal(affinity_list, port->rxq_affinity_list)) { - goto unlock; + if (netdev_is_pmd(port->netdev) + && !nullable_string_is_equal(affinity_list, port->rxq_affinity_list)) { + + error = dpif_netdev_port_set_rxq_affinity(port, affinity_list); + if (error) { + goto unlock; + } + free(port->rxq_affinity_list); + port->rxq_affinity_list = nullable_xstrdup(affinity_list); + + dp_netdev_request_reconfigure(dp); } - error = dpif_netdev_port_set_rxq_affinity(port, affinity_list); - if (error) { - goto unlock; + if (txq_hxps != port->txq_hxps) { + port->txq_hxps = txq_hxps; + VLOG_INFO("%s: Txq HXPS mode has been %s.", + netdev_get_name(port->netdev), + (txq_hxps) ? "enabled" : "disabled"); + dp_netdev_request_reconfigure(dp); } - free(port->rxq_affinity_list); - port->rxq_affinity_list = nullable_xstrdup(affinity_list); - dp_netdev_request_reconfigure(dp); + unlock: ovs_mutex_unlock(&dp->port_mutex); return error; @@ -4614,7 +4627,28 @@ dp_netdev_pmd_flush_output_on_port(struct dp_netdev_pmd_thread *pmd, output_cnt = dp_packet_batch_size(&p->output_pkts); ovs_assert(output_cnt > 0); - netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); + if (p->port->txq_mode == TXQ_MODE_HXPS) { + int n_txq = netdev_n_txq(p->port->netdev); + + /* Re-batch per txq based on packet hash */ + for (i = 0; i < output_cnt; i++) { + struct dp_packet *packet = p->output_pkts.packets[i]; + + tx_qid = dp_packet_get_rss_hash(packet) % n_txq; + dp_packet_batch_add(&p->txq_pkts[tx_qid], packet); + } + + /* Flush batches of each Tx queues */ + for (i = 0; i < n_txq; i++) { + if (dp_packet_batch_is_empty(&p->txq_pkts[i])) { + continue; + } + netdev_send(p->port->netdev, i, &p->txq_pkts[i], true); + dp_packet_batch_init(&p->txq_pkts[i]); + } + } else { + netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); + } dp_packet_batch_init(&p->output_pkts); /* Update time of the next flush. */ @@ -5775,7 +5809,9 @@ reconfigure_datapath(struct dp_netdev *dp) HMAP_FOR_EACH (port, node, &dp->ports) { if (netdev_is_reconf_required(port->netdev) || ((port->txq_mode == TXQ_MODE_XPS) - != (netdev_n_txq(port->netdev) < wanted_txqs))) { + != (netdev_n_txq(port->netdev) < wanted_txqs)) + || ((port->txq_mode == TXQ_MODE_HXPS) + != port->txq_hxps)) { port->need_reconfigure = true; } } @@ -5810,8 +5846,13 @@ reconfigure_datapath(struct dp_netdev *dp) seq_change(dp->port_seq); port_destroy(port); } else { - port->txq_mode = (netdev_n_txq(port->netdev) < wanted_txqs) ? - TXQ_MODE_XPS : TXQ_MODE_STATIC; + if (port->txq_hxps && netdev_n_txq(port->netdev) > 1) { + port->txq_mode = TXQ_MODE_HXPS; + } else if (netdev_n_txq(port->netdev) < wanted_txqs) { + port->txq_mode = TXQ_MODE_XPS; + } else { + port->txq_mode = TXQ_MODE_STATIC; + } } } @@ -6981,6 +7022,14 @@ dp_netdev_add_port_tx_to_pmd(struct dp_netdev_pmd_thread *pmd, tx->flush_time = 0LL; dp_packet_batch_init(&tx->output_pkts); + if (tx->port->txq_mode == TXQ_MODE_HXPS) { + int i, n_txq = netdev_n_txq(tx->port->netdev); + tx->txq_pkts = xzalloc(n_txq * sizeof *tx->txq_pkts); + for (i = 0; i < n_txq; i++) { + dp_packet_batch_init(&tx->txq_pkts[i]); + } + } + hmap_insert(&pmd->tx_ports, &tx->node, hash_port_no(tx->port->port_no)); pmd->need_reload = true; } @@ -6993,6 +7042,7 @@ dp_netdev_del_port_tx_from_pmd(struct dp_netdev_pmd_thread *pmd, OVS_REQUIRES(pmd->port_mutex) { hmap_remove(&pmd->tx_ports, &tx->node); + free(tx->txq_pkts); free(tx); pmd->need_reload = true; } From patchwork Wed Nov 24 21:24:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1559374 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AQ1OrE3u; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HzvDd19txz9sRN for ; Thu, 25 Nov 2021 08:25:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EDE6C61BE5; Wed, 24 Nov 2021 21:25:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UA5VlhUgUfB0; Wed, 24 Nov 2021 21:25:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1090160BC5; Wed, 24 Nov 2021 21:25:17 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DC02CC0039; Wed, 24 Nov 2021 21:25:16 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id E40A6C000A for ; Wed, 24 Nov 2021 21:25:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 038EE409A3 for ; Wed, 24 Nov 2021 21:24:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8JTx_LO3MLhc for ; Wed, 24 Nov 2021 21:24:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 162934093E for ; Wed, 24 Nov 2021 21:24:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637789082; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LZVicWzdfn2RYQk1Lz27l66vjyi89zqoKAqCq8gHyXk=; b=AQ1OrE3udWBT8UO86D5CcTKVwW1lAJXrFsHQzIrWwHfTXZnw+vVVl9bahOL5WrsCaB618T Ss2EciDBHpf0rL5iYi7zPFkHpkgIwZDgqiYeRfr4Mi04wuEy2xdLwvqZnPFZ/Zb0o/4PHx OizJPE+wDNJ2iJK/TxVY8EcZrtt6PsE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-166-14m5ARH8N1qDSRO1KNkYYQ-1; Wed, 24 Nov 2021 16:24:38 -0500 X-MC-Unique: 14m5ARH8N1qDSRO1KNkYYQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D683FA40C1; Wed, 24 Nov 2021 21:24:37 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0715CADD6; Wed, 24 Nov 2021 21:24:24 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Wed, 24 Nov 2021 22:24:00 +0100 Message-Id: <20211124212400.70613-5-maxime.coquelin@redhat.com> In-Reply-To: <20211124212400.70613-1-maxime.coquelin@redhat.com> References: <20211124212400.70613-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH 4/4] system-dpdk: Add tests for HXPS X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch adds two tests for Vhost-user client ports with multiqueue. First test is with HXPS disabled, where the expectation is that a single queue Vhost-user queue receives the 256 packets injected on the dummy port. Second test is with HXPS enabled, where the expectation is that the 256 packets injected on the dummy port are are spread on the two vhsot-user queues. Signed-off-by: Maxime Coquelin --- tests/system-dpdk.at | 135 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 135 insertions(+) diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at index 894e01316..2cb81729c 100644 --- a/tests/system-dpdk.at +++ b/tests/system-dpdk.at @@ -223,6 +223,141 @@ OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [ AT_CLEANUP dnl -------------------------------------------------------------------------- +dnl -------------------------------------------------------------------------- +dnl Multiqueue, HXPS disabled (default) with vhost-user-client port +AT_SETUP([OVS-DPDK - MQ, HXPS disabled with vhost-user-client port]) +AT_KEYWORDS([dpdk]) +OVS_DPDK_PRE_CHECK() +AT_SKIP_IF([! which dpdk-testpmd >/dev/null 2>/dev/null]) +AT_SKIP_IF([! $PYTHON3 -c "import scapy"], [], []) +OVS_DPDK_START() + +dnl Find number of sockets +AT_CHECK([lscpu], [], [stdout]) +AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) + +dnl Add userspace bridge and attach it to OVS +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) +AT_CHECK([ovs-vsctl add-port br10 dummy0 -- set interface dummy0 type=dummy-pmd]) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \ + dpdkvhostuserclient0 \ + type=dpdkvhostuserclient \ + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Parse log file +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created" ovs-vswitchd.log], [], [stdout]) +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout]) +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0: reconnecting..." ovs-vswitchd.log], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +tail -f /dev/null | dpdk-testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1,queues=2" \ + --file-prefix page0 --single-file-segments -- \ + --rxq=2 --txq=2 --nb-cores=1 --forward-mode=rxonly -a \ + >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & + +OVS_WAIT_UNTIL([grep "assigned port 'dpdkvhostuserclient0' rx queue 1" ovs-vswitchd.log]) + +on_exit "pkill -f -x -9 '$PYTHON3 $srcdir/mfex_pkts.py 256 fuzz'" +($PYTHON3 $srcdir/genpkts.py 256 fuzz | while read pkt; do + ovs-appctl netdev-dummy/receive dummy0 "$pkt" || break + done) + +OVS_WAIT_UNTIL([test `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_packets` -eq 256]) +AT_CHECK([ovs-vsctl get Interface dpdkvhostuserclient0 statistics], [], [stdout]) +AT_CHECK([[[[ `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q0_packets` -eq 0 && \ + `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q1_packets` -eq 256 ]]] || \ + [[[ `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q0_packets` -eq 256 && \ + `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q1_packets` -eq 0 ]]]]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr]) +AT_CHECK([ovs-vsctl del-port br10 dummy0], [], [stdout], [stderr]) +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [ +\@Failed to enable flow control@d +\@VHOST_CONFIG: recvmsg failed@d +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No such file or directory@d +\@dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports.@d +\@failed to enumerate system datapaths: No such file or directory@d +])") +AT_CLEANUP +dnl -------------------------------------------------------------------------- + +dnl -------------------------------------------------------------------------- +dnl Multiqueue, HXPS enabled with vhost-user-client port +AT_SETUP([OVS-DPDK - MQ, HXPS enabled with vhost-user-client port]) +AT_KEYWORDS([dpdk]) +OVS_DPDK_PRE_CHECK() +AT_SKIP_IF([! which dpdk-testpmd >/dev/null 2>/dev/null]) +AT_SKIP_IF([! $PYTHON3 -c "import scapy"], [], []) +OVS_DPDK_START() + +dnl Find number of sockets +AT_CHECK([lscpu], [], [stdout]) +AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) + +dnl Add userspace bridge and attach it to OVS +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) +AT_CHECK([ovs-vsctl add-port br10 dummy0 -- set interface dummy0 type=dummy-pmd]) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \ + dpdkvhostuserclient0 \ + type=dpdkvhostuserclient \ + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Parse log file +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created" ovs-vswitchd.log], [], [stdout]) +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout]) +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0: reconnecting..." ovs-vswitchd.log], [], [stdout]) + +dnl Enable HXPS +AT_CHECK([ovs-vsctl set Interface dpdkvhostuserclient0 other_config:hxps=true], + [], [stdout], [stderr]) +AT_CHECK([grep "dpdkvhostuserclient0: Txq HXPS mode has been enabled" ovs-vswitchd.log], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +tail -f /dev/null | dpdk-testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1,queues=2" \ + --file-prefix page0 --single-file-segments -- \ + --rxq=2 --txq=2 --nb-cores=1 --forward-mode=rxonly -a \ + >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & + +OVS_WAIT_UNTIL([grep "assigned port 'dpdkvhostuserclient0' rx queue 1" ovs-vswitchd.log]) + +on_exit "pkill -f -x -9 '$PYTHON3 $srcdir/mfex_pkts.py 256 fuzz'" +($PYTHON3 $srcdir/genpkts.py 256 fuzz | while read pkt; do + ovs-appctl netdev-dummy/receive dummy0 "$pkt" || break + done) + +OVS_WAIT_UNTIL([test `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_packets` -eq 256]) +AT_CHECK([ovs-vsctl get Interface dpdkvhostuserclient0 statistics], [], [stdout]) +AT_CHECK([[[[ `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q0_packets` -ge 0 && \ + `ovs-vsctl get Interface dpdkvhostuserclient0 statistics:tx_q1_packets` -ge 0 ]]]]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr]) +AT_CHECK([ovs-vsctl del-port br10 dummy0], [], [stdout], [stderr]) +OVS_VSWITCHD_STOP("m4_join([], [SYSTEM_DPDK_ALLOWED_LOGS], [ +\@Failed to enable flow control@d +\@VHOST_CONFIG: recvmsg failed@d +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No such file or directory@d +\@dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports.@d +\@failed to enumerate system datapaths: No such file or directory@d +])") +AT_CLEANUP +dnl -------------------------------------------------------------------------- + dnl -------------------------------------------------------------------------- dnl Add standard DPDK PHY port AT_SETUP([OVS-DPDK - MFEX Autovalidator])