From patchwork Fri Dec 17 15:10:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1570107 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=C3yg82Dm; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JFsqg6lZwz9t55 for ; Sat, 18 Dec 2021 02:10:39 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2B9C640BE9; Fri, 17 Dec 2021 15:10:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yxW-IDdS2RtK; Fri, 17 Dec 2021 15:10:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id AEEDE40BCD; Fri, 17 Dec 2021 15:10:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DAFCEC006E; Fri, 17 Dec 2021 15:10:31 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11BDEC006E for ; Fri, 17 Dec 2021 15:10:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 00DCE41EAC for ; Fri, 17 Dec 2021 15:10:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eLgc_-WyuMOe for ; Fri, 17 Dec 2021 15:10:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 1462641E80 for ; Fri, 17 Dec 2021 15:10:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639753829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L0XwpFQvmvOdu7Z3Db33SJVAZFveiUEmheddvWfnCcA=; b=C3yg82DmfQiBXQVDfeAD4WnW3/NeOLbCyrSoPKgmOLbkSaWI8lrI2OCSD4SLjJvN8hFV27 MvAUhAq9ZC5gGBM6AjLOV8/YhG6JzT8sjsOADs2Od4VDjdnECUvcqiB/UeHBy6GfRmJwtA 50KvRInGobnm242bd/V7yL5z9dIxhoo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-164-jsOMC4vbNtCbE7Swt1x0HA-1; Fri, 17 Dec 2021 10:10:25 -0500 X-MC-Unique: jsOMC4vbNtCbE7Swt1x0HA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DE4DA801AAB; Fri, 17 Dec 2021 15:10:24 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0F27A5E26D; Fri, 17 Dec 2021 15:10:22 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Fri, 17 Dec 2021 16:10:14 +0100 Message-Id: <20211217151018.436874-2-maxime.coquelin@redhat.com> In-Reply-To: <20211217151018.436874-1-maxime.coquelin@redhat.com> References: <20211217151018.436874-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v4 1/5] netdev-dpdk: Introduce per rxq/txq Vhost-user statistics. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Hash-based Tx steering feature will enable steering Tx packets on transmit queues based on their hashes. In order to test the feature, it is needed to be able to get the per-queue statistics for Vhost-user ports. This patch introduces "bytes", "packets" and "error" per-queue custom statistics for Vhost-user ports. Suggested-by David Marchand Signed-off-by: Maxime Coquelin Reviewed-by: David Marchand --- lib/netdev-dpdk.c | 149 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 140 insertions(+), 9 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 6782d3e8f..d824a2eda 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -192,6 +192,13 @@ static const struct rte_vhost_device_ops virtio_net_device_ops = .guest_notified = vhost_guest_notified, }; +/* Custom software per-queue stats for vhost ports */ +struct netdev_dpdk_vhost_q_stats { + uint64_t bytes; + uint64_t packets; + uint64_t errors; +}; + /* Custom software stats for dpdk ports */ struct netdev_dpdk_sw_stats { /* No. of retries when unable to transmit. */ @@ -479,9 +486,12 @@ struct netdev_dpdk { PADDED_MEMBERS(CACHE_LINE_SIZE, struct netdev_stats stats; struct netdev_dpdk_sw_stats *sw_stats; + /* Per-queue vhost Tx stats */ + struct netdev_dpdk_vhost_q_stats *vhost_txq_stats; + /* Per-queue vhost Rx stats */ + struct netdev_dpdk_vhost_q_stats *vhost_rxq_stats; /* Protects stats */ rte_spinlock_t stats_lock; - /* 36 pad bytes here. */ ); PADDED_MEMBERS(CACHE_LINE_SIZE, @@ -1276,6 +1286,13 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, dev->sw_stats = xzalloc(sizeof *dev->sw_stats); dev->sw_stats->tx_retries = (dev->type == DPDK_DEV_VHOST) ? 0 : UINT64_MAX; + if (dev->type == DPDK_DEV_VHOST) { + dev->vhost_txq_stats = xcalloc(netdev->n_txq, + sizeof *dev->vhost_txq_stats); + dev->vhost_rxq_stats = xcalloc(netdev->n_rxq, + sizeof *dev->vhost_rxq_stats); + } + return 0; } @@ -2354,17 +2371,21 @@ netdev_dpdk_vhost_update_rx_size_counters(struct netdev_stats *stats, } static inline void -netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, +netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, int qid, struct dp_packet **packets, int count, int qos_drops) { + struct netdev_dpdk_vhost_q_stats *q_stats = &dev->vhost_rxq_stats[qid]; struct netdev_stats *stats = &dev->stats; struct dp_packet *packet; unsigned int packet_size; int i; stats->rx_packets += count; + q_stats->packets += count; stats->rx_dropped += qos_drops; + q_stats->errors += qos_drops; + for (i = 0; i < count; i++) { packet = packets[i]; packet_size = dp_packet_size(packet); @@ -2375,6 +2396,7 @@ netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, * further processing. */ stats->rx_errors++; stats->rx_length_errors++; + q_stats->errors++; continue; } @@ -2386,6 +2408,7 @@ netdev_dpdk_vhost_update_rx_counters(struct netdev_dpdk *dev, } stats->rx_bytes += packet_size; + q_stats->bytes += packet_size; } if (OVS_UNLIKELY(qos_drops)) { @@ -2438,7 +2461,7 @@ netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq, } rte_spinlock_lock(&dev->stats_lock); - netdev_dpdk_vhost_update_rx_counters(dev, batch->packets, + netdev_dpdk_vhost_update_rx_counters(dev, rxq->queue_id, batch->packets, nb_rx, qos_drops); rte_spinlock_unlock(&dev->stats_lock); @@ -2552,11 +2575,12 @@ netdev_dpdk_filter_packet_len(struct netdev_dpdk *dev, struct rte_mbuf **pkts, } static inline void -netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, +netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, int qid, struct dp_packet **packets, int attempted, struct netdev_dpdk_sw_stats *sw_stats_add) { + struct netdev_dpdk_vhost_q_stats *q_stats = &dev->vhost_txq_stats[qid]; int dropped = sw_stats_add->tx_mtu_exceeded_drops + sw_stats_add->tx_qos_drops + sw_stats_add->tx_failure_drops + @@ -2566,10 +2590,15 @@ netdev_dpdk_vhost_update_tx_counters(struct netdev_dpdk *dev, int i; stats->tx_packets += sent; + q_stats->packets += sent; stats->tx_dropped += dropped; + q_stats->errors += dropped; for (i = 0; i < sent; i++) { - stats->tx_bytes += dp_packet_size(packets[i]); + uint64_t bytes = dp_packet_size(packets[i]); + + stats->tx_bytes += bytes; + q_stats->bytes += bytes; } if (OVS_UNLIKELY(dropped || sw_stats_add->tx_retries)) { @@ -2657,7 +2686,7 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, sw_stats_add.tx_retries = MIN(retries, max_retries); rte_spinlock_lock(&dev->stats_lock); - netdev_dpdk_vhost_update_tx_counters(dev, pkts, total_packets, + netdev_dpdk_vhost_update_tx_counters(dev, qid, pkts, total_packets, &sw_stats_add); rte_spinlock_unlock(&dev->stats_lock); @@ -3287,6 +3316,76 @@ netdev_dpdk_get_sw_custom_stats(const struct netdev *netdev, return 0; } +static int +netdev_dpdk_vhost_get_custom_stats(const struct netdev *netdev, + struct netdev_custom_stats *custom_stats) +{ + struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + int sw_stats_size, i, j; + + netdev_dpdk_get_sw_custom_stats(netdev, custom_stats); + + ovs_mutex_lock(&dev->mutex); + +#define VHOST_Q_STATS \ + VHOST_Q_STAT(bytes) \ + VHOST_Q_STAT(packets) \ + VHOST_Q_STAT(errors) + + sw_stats_size = custom_stats->size; +#define VHOST_Q_STAT(NAME) + netdev->n_rxq + custom_stats->size += VHOST_Q_STATS; +#undef VHOST_Q_STAT +#define VHOST_Q_STAT(NAME) + netdev->n_txq + custom_stats->size += VHOST_Q_STATS; +#undef VHOST_Q_STAT + custom_stats->counters = xrealloc(custom_stats->counters, + custom_stats->size * + sizeof *custom_stats->counters); + + j = 0; + for (i = 0; i < netdev->n_rxq; i++) { +#define VHOST_Q_STAT(NAME) \ + snprintf(custom_stats->counters[sw_stats_size + j++].name, \ + NETDEV_CUSTOM_STATS_NAME_SIZE, "rx_q%d_"#NAME, i); + VHOST_Q_STATS +#undef VHOST_Q_STAT + } + + for (i = 0; i < netdev->n_txq; i++) { +#define VHOST_Q_STAT(NAME) \ + snprintf(custom_stats->counters[sw_stats_size + j++].name, \ + NETDEV_CUSTOM_STATS_NAME_SIZE, "tx_q%d_"#NAME, i); + VHOST_Q_STATS +#undef VHOST_Q_STAT + } + + rte_spinlock_lock(&dev->stats_lock); + + j = 0; + for (i = 0; i < netdev->n_rxq; i++) { +#define VHOST_Q_STAT(NAME) \ + custom_stats->counters[sw_stats_size + j++].value = \ + dev->vhost_rxq_stats[i].NAME; + VHOST_Q_STATS +#undef VHOST_Q_STAT + } + + for (i = 0; i < netdev->n_txq; i++) { +#define VHOST_Q_STAT(NAME) \ + custom_stats->counters[sw_stats_size + j++].value = \ + dev->vhost_txq_stats[i].NAME; + VHOST_Q_STATS +#undef VHOST_Q_STAT + } + + rte_spinlock_unlock(&dev->stats_lock); + + ovs_mutex_unlock(&dev->mutex); + + return 0; +} + static int netdev_dpdk_get_features(const struct netdev *netdev, enum netdev_features *current, @@ -3556,6 +3655,11 @@ netdev_dpdk_update_flags__(struct netdev_dpdk *dev, if (NETDEV_UP & on) { rte_spinlock_lock(&dev->stats_lock); memset(&dev->stats, 0, sizeof dev->stats); + memset(dev->sw_stats, 0, sizeof *dev->sw_stats); + memset(dev->vhost_rxq_stats, 0, + dev->up.n_rxq * sizeof *dev->vhost_rxq_stats); + memset(dev->vhost_txq_stats, 0, + dev->up.n_txq * sizeof *dev->vhost_txq_stats); rte_spinlock_unlock(&dev->stats_lock); } } @@ -5048,9 +5152,12 @@ static int dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) OVS_REQUIRES(dev->mutex) { + int old_n_txq = dev->up.n_txq; + int old_n_rxq = dev->up.n_rxq; + int err; + dev->up.n_txq = dev->requested_n_txq; dev->up.n_rxq = dev->requested_n_rxq; - int err; /* Always keep RX queue 0 enabled for implementations that won't * report vring states. */ @@ -5068,6 +5175,30 @@ dpdk_vhost_reconfigure_helper(struct netdev_dpdk *dev) netdev_dpdk_remap_txqs(dev); + /* Reset all stats if number of queues changed. */ + if (dev->up.n_txq != old_n_txq || dev->up.n_rxq != old_n_rxq) { + struct netdev_dpdk_vhost_q_stats *old_txq_stats, *new_txq_stats; + struct netdev_dpdk_vhost_q_stats *old_rxq_stats, *new_rxq_stats; + + new_txq_stats = xcalloc(dev->up.n_txq, sizeof *dev->vhost_txq_stats); + new_rxq_stats = xcalloc(dev->up.n_rxq, sizeof *dev->vhost_rxq_stats); + + rte_spinlock_lock(&dev->stats_lock); + + memset(&dev->stats, 0, sizeof dev->stats); + memset(dev->sw_stats, 0, sizeof *dev->sw_stats); + + old_txq_stats = dev->vhost_txq_stats; + dev->vhost_txq_stats = new_txq_stats; + old_rxq_stats = dev->vhost_rxq_stats; + dev->vhost_rxq_stats = new_rxq_stats; + + rte_spinlock_unlock(&dev->stats_lock); + + free(old_txq_stats); + free(old_rxq_stats); + } + err = netdev_dpdk_mempool_configure(dev); if (!err) { /* A new mempool was created or re-used. */ @@ -5473,7 +5604,7 @@ static const struct netdev_class dpdk_vhost_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv, @@ -5489,7 +5620,7 @@ static const struct netdev_class dpdk_vhost_client_class = { .send = netdev_dpdk_vhost_send, .get_carrier = netdev_dpdk_vhost_get_carrier, .get_stats = netdev_dpdk_vhost_get_stats, - .get_custom_stats = netdev_dpdk_get_sw_custom_stats, + .get_custom_stats = netdev_dpdk_vhost_get_custom_stats, .get_status = netdev_dpdk_vhost_user_get_status, .reconfigure = netdev_dpdk_vhost_client_reconfigure, .rxq_recv = netdev_dpdk_vhost_rxq_recv, From patchwork Fri Dec 17 15:10:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1570109 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UU7t7u3m; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JFsqm4s8Pz9t55 for ; Sat, 18 Dec 2021 02:10:44 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B91DE40BF9; Fri, 17 Dec 2021 15:10:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jYqvw9Vyq43U; Fri, 17 Dec 2021 15:10:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 526AD40BCC; Fri, 17 Dec 2021 15:10:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 07551C006E; Fri, 17 Dec 2021 15:10:34 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8292CC0030 for ; Fri, 17 Dec 2021 15:10:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2D9ED41E80 for ; Fri, 17 Dec 2021 15:10:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wTMm_V7HPtDd for ; Fri, 17 Dec 2021 15:10:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 5ACFE41E84 for ; Fri, 17 Dec 2021 15:10:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639753829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LPUnN3cxtA3F2l2P9HnYt5ic2M4tE21qFofPsxWHfUo=; b=UU7t7u3mUVtURf2TdSCJ1jgXHX/kfIq8xfAFbnEdPyjbNoPJ2aCsUqIL/XKr4nFL2SYBFp eMD0Td/yX00GTVYsMJVACLrPOYa74e0VMuidftARBdNk2Vs5oHqlK05IyHv1M8/UiO6b7d ruk9R7LEcA0MOhF9XgDR/bK9n4U7h64= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-199-5DbPryvKOZWJ2m_vPZeanA-1; Fri, 17 Dec 2021 10:10:28 -0500 X-MC-Unique: 5DbPryvKOZWJ2m_vPZeanA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EF1B1874987; Fri, 17 Dec 2021 15:10:26 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BABA5ED48; Fri, 17 Dec 2021 15:10:25 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Fri, 17 Dec 2021 16:10:15 +0100 Message-Id: <20211217151018.436874-3-maxime.coquelin@redhat.com> In-Reply-To: <20211217151018.436874-1-maxime.coquelin@redhat.com> References: <20211217151018.436874-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v4 2/5] netdev-dummy: Introduce per rxq/txq statistics. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch adds Rx and Tx per-queue statistics. It will be used to test hash-based Tx packet steering. Only "bytes", and "packets" per-queue custom statistics are added, as there are no global "errors" counters in netdev-dummy. Signed-off-by: Maxime Coquelin Reviewed-by: David Marchand --- lib/netdev-dummy.c | 82 ++++++++++++++++++++++++++++++++++++++++++---- tests/ofproto.at | 3 +- 2 files changed, 77 insertions(+), 8 deletions(-) diff --git a/lib/netdev-dummy.c b/lib/netdev-dummy.c index 1f386b81b..59a9efefd 100644 --- a/lib/netdev-dummy.c +++ b/lib/netdev-dummy.c @@ -101,6 +101,11 @@ struct offloaded_flow { uint32_t mark; }; +struct netdev_dummy_q_stats { + uint64_t bytes; + uint64_t packets; +}; + /* Protects 'dummy_list'. */ static struct ovs_mutex dummy_list_mutex = OVS_MUTEX_INITIALIZER; @@ -121,6 +126,8 @@ struct netdev_dummy { int mtu OVS_GUARDED; struct netdev_stats stats OVS_GUARDED; struct netdev_custom_counter custom_stats[C_STATS_SIZE] OVS_GUARDED; + struct netdev_dummy_q_stats *rxq_stats OVS_GUARDED; + struct netdev_dummy_q_stats *txq_stats OVS_GUARDED; enum netdev_flags flags OVS_GUARDED; int ifindex OVS_GUARDED; int numa_id OVS_GUARDED; @@ -707,6 +714,9 @@ netdev_dummy_construct(struct netdev *netdev_) ovs_strlcpy(netdev->custom_stats[1].name, "rx_custom_packets_2", NETDEV_CUSTOM_STATS_NAME_SIZE); + netdev->rxq_stats = xcalloc(netdev->up.n_rxq, sizeof *netdev->rxq_stats); + netdev->txq_stats = xcalloc(netdev->up.n_rxq, sizeof *netdev->txq_stats); + dummy_packet_conn_init(&netdev->conn); ovs_list_init(&netdev->rxes); @@ -731,6 +741,8 @@ netdev_dummy_destruct(struct netdev *netdev_) ovs_mutex_unlock(&dummy_list_mutex); ovs_mutex_lock(&netdev->mutex); + free(netdev->rxq_stats); + free(netdev->txq_stats); if (netdev->rxq_pcap) { ovs_pcap_close(netdev->rxq_pcap); } @@ -954,6 +966,8 @@ static int netdev_dummy_reconfigure(struct netdev *netdev_) { struct netdev_dummy *netdev = netdev_dummy_cast(netdev_); + int old_n_txq = netdev_->n_txq; + int old_n_rxq = netdev_->n_rxq; ovs_mutex_lock(&netdev->mutex); @@ -961,6 +975,24 @@ netdev_dummy_reconfigure(struct netdev *netdev_) netdev_->n_rxq = netdev->requested_n_rxq; netdev->numa_id = netdev->requested_numa_id; + if (netdev_->n_txq != old_n_txq || netdev_->n_rxq != old_n_rxq) { + /* Resize the per queue stats arrays */ + netdev->txq_stats = xrealloc(netdev->txq_stats, + netdev_->n_txq * sizeof *netdev->txq_stats); + netdev->rxq_stats = xrealloc(netdev->rxq_stats, + netdev_->n_rxq * sizeof *netdev->rxq_stats); + + /* Reset all stats for consistency between per-queue and global + * counters */ + memset(&netdev->stats, 0, sizeof netdev->stats); + netdev->custom_stats[0].value = 0; + netdev->custom_stats[1].value = 0; + memset(netdev->txq_stats, 0, + netdev_->n_txq * sizeof *netdev->txq_stats); + memset(netdev->rxq_stats, 0, + netdev_->n_rxq * sizeof *netdev->rxq_stats); + } + ovs_mutex_unlock(&netdev->mutex); return 0; } @@ -1049,7 +1081,9 @@ netdev_dummy_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet_batch *batch, } ovs_mutex_lock(&netdev->mutex); netdev->stats.rx_packets++; + netdev->rxq_stats[rxq_->queue_id].packets++; netdev->stats.rx_bytes += dp_packet_size(packet); + netdev->rxq_stats[rxq_->queue_id].bytes += dp_packet_size(packet); netdev->custom_stats[0].value++; netdev->custom_stats[1].value++; ovs_mutex_unlock(&netdev->mutex); @@ -1096,7 +1130,7 @@ netdev_dummy_rxq_drain(struct netdev_rxq *rxq_) } static int -netdev_dummy_send(struct netdev *netdev, int qid OVS_UNUSED, +netdev_dummy_send(struct netdev *netdev, int qid, struct dp_packet_batch *batch, bool concurrent_txq OVS_UNUSED) { @@ -1135,7 +1169,9 @@ netdev_dummy_send(struct netdev *netdev, int qid OVS_UNUSED, ovs_mutex_lock(&dev->mutex); dev->stats.tx_packets++; + dev->txq_stats[qid].packets++; dev->stats.tx_bytes += size; + dev->txq_stats[qid].bytes += size; dummy_packet_conn_send(&dev->conn, buffer, size); @@ -1252,22 +1288,54 @@ static int netdev_dummy_get_custom_stats(const struct netdev *netdev, struct netdev_custom_stats *custom_stats) { - int i; + int i,j; struct netdev_dummy *dev = netdev_dummy_cast(netdev); - custom_stats->size = 2; + ovs_mutex_lock(&dev->mutex); + +#define DUMMY_Q_STATS \ + DUMMY_Q_STAT(bytes) \ + DUMMY_Q_STAT(packets) + + custom_stats->size = C_STATS_SIZE; +#define DUMMY_Q_STAT(NAME) + netdev->n_rxq + custom_stats->size += DUMMY_Q_STATS; +#undef DUMMY_Q_STAT +#define DUMMY_Q_STAT(NAME) + netdev->n_txq + custom_stats->size += DUMMY_Q_STATS; +#undef DUMMY_Q_STAT + custom_stats->counters = - (struct netdev_custom_counter *) xcalloc(C_STATS_SIZE, + (struct netdev_custom_counter *) xcalloc(custom_stats->size, sizeof(struct netdev_custom_counter)); - ovs_mutex_lock(&dev->mutex); + j = 0; for (i = 0 ; i < C_STATS_SIZE ; i++) { - custom_stats->counters[i].value = dev->custom_stats[i].value; - ovs_strlcpy(custom_stats->counters[i].name, + custom_stats->counters[j].value = dev->custom_stats[i].value; + ovs_strlcpy(custom_stats->counters[j++].name, dev->custom_stats[i].name, NETDEV_CUSTOM_STATS_NAME_SIZE); } + + for (i = 0; i < netdev->n_rxq; i++) { +#define DUMMY_Q_STAT(NAME) \ + snprintf(custom_stats->counters[j].name, \ + NETDEV_CUSTOM_STATS_NAME_SIZE, "rx_q%d_"#NAME, i); \ + custom_stats->counters[j++].value = dev->rxq_stats[i].NAME; + DUMMY_Q_STATS +#undef DUMMY_Q_STAT + } + + for (i = 0; i < netdev->n_txq; i++) { +#define DUMMY_Q_STAT(NAME) \ + snprintf(custom_stats->counters[j].name, \ + NETDEV_CUSTOM_STATS_NAME_SIZE, "tx_q%d_"#NAME, i); \ + custom_stats->counters[j++].value = dev->txq_stats[i].NAME; + DUMMY_Q_STATS +#undef DUMMY_Q_STAT + } + ovs_mutex_unlock(&dev->mutex); return 0; diff --git a/tests/ofproto.at b/tests/ofproto.at index 08c0a20b6..156d3e058 100644 --- a/tests/ofproto.at +++ b/tests/ofproto.at @@ -112,7 +112,8 @@ OFPST_PORT reply (OF1.4): 1 ports tx pkts=0, bytes=0, drop=?, errs=?, coll=? duration=?s CUSTOM Statistics - rx_custom_packets_1=0, rx_custom_packets_2=0, + rx_custom_packets_1=0, rx_custom_packets_2=0, rx_q0_bytes=0, + rx_q0_packets=0, tx_q0_bytes=0, tx_q0_packets=0, ]) OVS_VSWITCHD_STOP AT_CLEANUP From patchwork Fri Dec 17 15:10:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1570108 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EXhp3BiH; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JFsqj3Ffpz9t55 for ; Sat, 18 Dec 2021 02:10:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id ED83141F3F; Fri, 17 Dec 2021 15:10:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f-S3x9FWK_lS; Fri, 17 Dec 2021 15:10:38 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 15C8241EF4; Fri, 17 Dec 2021 15:10:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DE9F4C0039; Fri, 17 Dec 2021 15:10:36 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id D8655C001E for ; Fri, 17 Dec 2021 15:10:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A89BB40BEF for ; Fri, 17 Dec 2021 15:10:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jOCdAvDgnNA2 for ; Fri, 17 Dec 2021 15:10:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 638B440BEE for ; Fri, 17 Dec 2021 15:10:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639753833; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UAqcQcVqWjolbJmi7Bors0OofMWAU85ws/q3gpnj0FU=; b=EXhp3BiHtF9ESkE7THWlqzijGJIBgsMIwN5jXYGmXtzkYtc/kDWQEJW3OxZCtqigzo2DGR PLLTJxzVNxoK51kaDbUKq+LVVfBN7iDvvHm1JWrzmyrpo9WEBTP++6I3KqbcGd/O4cIvn6 R9kX3oqziWXmRVyAxUX2IOoUmUm76UA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-231-oPdu5pzWMB-Y_ac7fWRzMw-1; Fri, 17 Dec 2021 10:10:30 -0500 X-MC-Unique: oPdu5pzWMB-Y_ac7fWRzMw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E414C874984; Fri, 17 Dec 2021 15:10:28 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E7905ED50; Fri, 17 Dec 2021 15:10:27 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Fri, 17 Dec 2021 16:10:16 +0100 Message-Id: <20211217151018.436874-4-maxime.coquelin@redhat.com> In-Reply-To: <20211217151018.436874-1-maxime.coquelin@redhat.com> References: <20211217151018.436874-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v4 3/5] dpif-netdev: Introduce Tx queue mode. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A boolean is currently used to differenciate between the static and XPS Tx queue modes. Since we are going to introduce a new steering mode, replace this boolean with an enum. This patch does not introduce functionnal changes. Signed-off-by: Maxime Coquelin Reviewed-by: David Marchand --- lib/dpif-netdev.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index a790df5fd..ba823a482 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -385,10 +385,15 @@ struct dp_netdev_rxq { atomic_ullong cycles_intrvl[PMD_INTERVAL_MAX]; }; +enum txq_mode { + TXQ_MODE_STATIC, + TXQ_MODE_XPS, +}; + /* A port in a netdev-based datapath. */ struct dp_netdev_port { odp_port_t port_no; - bool dynamic_txqs; /* If true XPS will be used. */ + enum txq_mode txq_mode; /* static, XPS */ bool need_reconfigure; /* True if we should reconfigure netdev. */ struct netdev *netdev; struct hmap_node node; /* Node in dp_netdev's 'ports'. */ @@ -4591,24 +4596,25 @@ dp_netdev_pmd_flush_output_on_port(struct dp_netdev_pmd_thread *pmd, int i; int tx_qid; int output_cnt; - bool dynamic_txqs; + bool concurrent_txqs; struct cycle_timer timer; uint64_t cycles; uint32_t tx_flush_interval; cycle_timer_start(&pmd->perf_stats, &timer); - dynamic_txqs = p->port->dynamic_txqs; - if (dynamic_txqs) { + if (p->port->txq_mode == TXQ_MODE_XPS) { tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p); + concurrent_txqs = true; } else { tx_qid = pmd->static_tx_qid; + concurrent_txqs = false; } output_cnt = dp_packet_batch_size(&p->output_pkts); ovs_assert(output_cnt > 0); - netdev_send(p->port->netdev, tx_qid, &p->output_pkts, dynamic_txqs); + netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); dp_packet_batch_init(&p->output_pkts); /* Update time of the next flush. */ @@ -5761,14 +5767,14 @@ reconfigure_datapath(struct dp_netdev *dp) * 'port->need_reconfigure', because netdev_is_reconf_required() can * change at any time. * Also mark for reconfiguration all ports which will likely change their - * 'dynamic_txqs' parameter. It's required to stop using them before + * 'txq_mode' parameter. It's required to stop using them before * changing this setting and it's simpler to mark ports here and allow * 'pmd_remove_stale_ports' to remove them from threads. There will be * no actual reconfiguration in 'port_reconfigure' because it's * unnecessary. */ HMAP_FOR_EACH (port, node, &dp->ports) { if (netdev_is_reconf_required(port->netdev) - || (port->dynamic_txqs + || ((port->txq_mode == TXQ_MODE_XPS) != (netdev_n_txq(port->netdev) < wanted_txqs))) { port->need_reconfigure = true; } @@ -5804,7 +5810,8 @@ reconfigure_datapath(struct dp_netdev *dp) seq_change(dp->port_seq); port_destroy(port); } else { - port->dynamic_txqs = netdev_n_txq(port->netdev) < wanted_txqs; + port->txq_mode = (netdev_n_txq(port->netdev) < wanted_txqs) ? + TXQ_MODE_XPS : TXQ_MODE_STATIC; } } @@ -7812,7 +7819,7 @@ dpif_netdev_xps_revalidate_pmd(const struct dp_netdev_pmd_thread *pmd, long long interval; HMAP_FOR_EACH (tx, node, &pmd->send_port_cache) { - if (!tx->port->dynamic_txqs) { + if (tx->port->txq_mode != TXQ_MODE_XPS) { continue; } interval = pmd->ctx.now - tx->last_used; From patchwork Fri Dec 17 15:10:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1570110 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=M4xIEv4R; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JFsqq70hsz9t55 for ; Sat, 18 Dec 2021 02:10:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2AF4041F14; Fri, 17 Dec 2021 15:10:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yhs61s1ORzs7; Fri, 17 Dec 2021 15:10:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id F068941ECF; Fri, 17 Dec 2021 15:10:42 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BFC2AC001E; Fri, 17 Dec 2021 15:10:42 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 17D87C0039 for ; Fri, 17 Dec 2021 15:10:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9672E40BE1 for ; Fri, 17 Dec 2021 15:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qa4t1HcgUPL7 for ; Fri, 17 Dec 2021 15:10:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 9D83040BFB for ; Fri, 17 Dec 2021 15:10:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639753836; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SMjrmm5kaDRf1X7Ukck6h79QyR0j5Tg9O2S/w7g/SRg=; b=M4xIEv4RF4wJsft/i/hKKTmhySdBwfDy14UDsV+9cLMKscl28pYeDHp6gU0O1TtfU0Tx/w NE/C5/mk80K94yu+vGXKDkIBtk8ZfPP6cZUfWiSE32QfYDj/r7EUgBbzL/REszbK026IRu YPZBCBNNXwaXC8beavomcQ0WNdkcnmo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-369-QWbGbf_TPyyluI9lzJjZXQ-1; Fri, 17 Dec 2021 10:10:33 -0500 X-MC-Unique: QWbGbf_TPyyluI9lzJjZXQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 20F81192FDA2; Fri, 17 Dec 2021 15:10:31 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4A8E45ED54; Fri, 17 Dec 2021 15:10:29 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Fri, 17 Dec 2021 16:10:17 +0100 Message-Id: <20211217151018.436874-5-maxime.coquelin@redhat.com> In-Reply-To: <20211217151018.436874-1-maxime.coquelin@redhat.com> References: <20211217151018.436874-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v4 4/5] dpif-netdev: Introduce hash-based Tx packet steering mode. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch adds a new hash Tx steering mode that distributes the traffic on all the Tx queues, whatever the number of PMD threads. It would be useful for guests expecting traffic to be distributed on all the vCPUs. The idea here is to re-use the 5-tuple hash of the packets, already computed to build the flows batches (and so it does not provide flexibility on which fields are part of the hash). There are also no user-configurable indirection table, given the feature is transparent to the guest. The queue selection is just a modulo operation between the packet hash and the number of Tx queues. There are no (at least intentionnally) functionnal changes for the existing XPS and static modes. There should not be noticeable performance changes for these modes (only one more branch in the hot path). For the hash mode, performance could be impacted due to locking when multiple PMD threads are in use (same as XPS mode) and also because of the second level of batching. Regarding the batching, the existing Tx port output_pkts is not modified. It means that at maximum, NETDEV_MAX_BURST can be batched for all the Tx queues. A second level of batching is done in dp_netdev_pmd_flush_output_on_port(), only for this hash mode. Signed-off-by: Maxime Coquelin Reviewed-by: David Marchand --- Documentation/automake.mk | 1 + Documentation/topics/index.rst | 1 + .../topics/userspace-tx-steering.rst | 75 ++++++++++ lib/dpif-netdev.c | 131 +++++++++++++++--- 4 files changed, 185 insertions(+), 23 deletions(-) create mode 100644 Documentation/topics/userspace-tx-steering.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index 137cc57c5..01e3c4f9e 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -58,6 +58,7 @@ DOC_SOURCE = \ Documentation/topics/record-replay.rst \ Documentation/topics/tracing.rst \ Documentation/topics/userspace-tso.rst \ + Documentation/topics/userspace-tx-steering.rst \ Documentation/topics/windows.rst \ Documentation/howto/index.rst \ Documentation/howto/dpdk.rst \ diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst index d8ccbd757..3699fd5c4 100644 --- a/Documentation/topics/index.rst +++ b/Documentation/topics/index.rst @@ -55,3 +55,4 @@ OVS userspace-tso idl-compound-indexes ovs-extensions + userspace-tx-steering diff --git a/Documentation/topics/userspace-tx-steering.rst b/Documentation/topics/userspace-tx-steering.rst new file mode 100644 index 000000000..a4a9aaa72 --- /dev/null +++ b/Documentation/topics/userspace-tx-steering.rst @@ -0,0 +1,75 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +============================ +Userspace Tx packet steering +============================ + +The userspace datapath supports two transmit packets steering modes. + +Thread mode +~~~~~~~~~~~ + +This mode is automatically selected when port's ``tx-steering`` option is set +to ``thread`` or unset. + +Depending on port's number of Tx queues being greater or equal than the number +of PMD threads, static txq mapping or XPS will be used. + +This the recommended mode for performance reasons if the number of Tx queues +is greater or equal to the number of PMD threads, because the Tx lock is not +acquired. + +If the number of Tx queues is greater than the number of PMDs, the +remaining Tx queues will not be used. + +This mode is enabled by default. + +Hash mode +~~~~~~~~~ + +Hash-based Tx packets steering mode distributes the traffic on all the port's +transmit queues, whatever the number of PMD threads. Queue selection is based +on the 5-tuples hash already computed to build the flows batches, the selected +queue being the modulo between the hash and the number of Tx queues of the +port. + +Hash mode may be used for example with Vhost-user ports, when the number of +vCPUs and queues of thevguest are greater than the number of PMD threads. +Without hash mode, the Tx queues used would bevlimited to the number of PMD. + +Hash-based Tx packet steering may have an impact on the performance, given the +Tx lock acquisition is required and a second level of batching is performed. + +This mode is disabled by default. + +Usage +~~~~~ + +To enable hash mode:: + + $ ovs-vsctl set Interface other_config:tx-steering=hash + +To disable hash mode:: + + $ ovs-vsctl set Interface other_config:tx-steering=thread diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index ba823a482..30e1eb543 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -385,15 +385,21 @@ struct dp_netdev_rxq { atomic_ullong cycles_intrvl[PMD_INTERVAL_MAX]; }; +enum txq_req_mode { + TXQ_REQ_MODE_THREAD, + TXQ_REQ_MODE_HASH, +}; + enum txq_mode { TXQ_MODE_STATIC, TXQ_MODE_XPS, + TXQ_MODE_HASH, }; /* A port in a netdev-based datapath. */ struct dp_netdev_port { odp_port_t port_no; - enum txq_mode txq_mode; /* static, XPS */ + enum txq_mode txq_mode; /* static, XPS, HASH. */ bool need_reconfigure; /* True if we should reconfigure netdev. */ struct netdev *netdev; struct hmap_node node; /* Node in dp_netdev's 'ports'. */ @@ -405,6 +411,7 @@ struct dp_netdev_port { bool emc_enabled; /* If true EMC will be used. */ char *type; /* Port type as requested by user. */ char *rxq_affinity_list; /* Requested affinity of rx queues. */ + enum txq_req_mode txq_requested_mode; }; static bool dp_netdev_flow_ref(struct dp_netdev_flow *); @@ -440,6 +447,7 @@ struct tx_port { struct hmap_node node; long long flush_time; struct dp_packet_batch output_pkts; + struct dp_packet_batch *txq_pkts; /* Only for hash mode */ struct dp_netdev_rxq *output_pkts_rxqs[NETDEV_MAX_BURST]; }; @@ -4435,6 +4443,8 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, int error = 0; const char *affinity_list = smap_get(cfg, "pmd-rxq-affinity"); bool emc_enabled = smap_get_bool(cfg, "emc-enable", true); + const char *tx_steering_mode = smap_get(cfg, "tx-steering"); + enum txq_req_mode txq_mode; ovs_mutex_lock(&dp->port_mutex); error = get_port_by_number(dp, port_no, &port); @@ -4476,19 +4486,33 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, } /* Checking for RXq affinity changes. */ - if (!netdev_is_pmd(port->netdev) - || nullable_string_is_equal(affinity_list, port->rxq_affinity_list)) { - goto unlock; + if (netdev_is_pmd(port->netdev) + && !nullable_string_is_equal(affinity_list, port->rxq_affinity_list)) { + + error = dpif_netdev_port_set_rxq_affinity(port, affinity_list); + if (error) { + goto unlock; + } + free(port->rxq_affinity_list); + port->rxq_affinity_list = nullable_xstrdup(affinity_list); + + dp_netdev_request_reconfigure(dp); } - error = dpif_netdev_port_set_rxq_affinity(port, affinity_list); - if (error) { - goto unlock; + if (nullable_string_is_equal(tx_steering_mode, "hash")) { + txq_mode = TXQ_REQ_MODE_HASH; + } else { + txq_mode = TXQ_REQ_MODE_THREAD; + } + + if (txq_mode != port->txq_requested_mode) { + port->txq_requested_mode = txq_mode; + VLOG_INFO("%s: Txq mode has been set to %s.", + netdev_get_name(port->netdev), + (txq_mode == TXQ_REQ_MODE_THREAD) ? "thread" : "hash"); + dp_netdev_request_reconfigure(dp); } - free(port->rxq_affinity_list); - port->rxq_affinity_list = nullable_xstrdup(affinity_list); - dp_netdev_request_reconfigure(dp); unlock: ovs_mutex_unlock(&dp->port_mutex); return error; @@ -4603,18 +4627,45 @@ dp_netdev_pmd_flush_output_on_port(struct dp_netdev_pmd_thread *pmd, cycle_timer_start(&pmd->perf_stats, &timer); - if (p->port->txq_mode == TXQ_MODE_XPS) { - tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p); - concurrent_txqs = true; - } else { - tx_qid = pmd->static_tx_qid; - concurrent_txqs = false; - } - output_cnt = dp_packet_batch_size(&p->output_pkts); ovs_assert(output_cnt > 0); - netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); + if (p->port->txq_mode == TXQ_MODE_HASH) { + int n_txq = netdev_n_txq(p->port->netdev); + + /* Re-batch per txq based on packet hash. */ + for (i = 0; i < output_cnt; i++) { + struct dp_packet *packet = p->output_pkts.packets[i]; + struct flow flow; + uint32_t hash; + + if (OVS_LIKELY(dp_packet_rss_valid(packet))) { + hash = dp_packet_get_rss_hash(packet); + } else { + flow_extract(packet, &flow); + hash = flow_hash_5tuple(&flow, 0); + } + dp_packet_batch_add(&p->txq_pkts[hash % n_txq], packet); + } + + /* Flush batches of each Tx queues. */ + for (i = 0; i < n_txq; i++) { + if (dp_packet_batch_is_empty(&p->txq_pkts[i])) { + continue; + } + netdev_send(p->port->netdev, i, &p->txq_pkts[i], true); + dp_packet_batch_init(&p->txq_pkts[i]); + } + } else { + if (p->port->txq_mode == TXQ_MODE_XPS) { + tx_qid = dpif_netdev_xps_get_tx_qid(pmd, p); + concurrent_txqs = true; + } else { + tx_qid = pmd->static_tx_qid; + concurrent_txqs = false; + } + netdev_send(p->port->netdev, tx_qid, &p->output_pkts, concurrent_txqs); + } dp_packet_batch_init(&p->output_pkts); /* Update time of the next flush. */ @@ -5775,7 +5826,9 @@ reconfigure_datapath(struct dp_netdev *dp) HMAP_FOR_EACH (port, node, &dp->ports) { if (netdev_is_reconf_required(port->netdev) || ((port->txq_mode == TXQ_MODE_XPS) - != (netdev_n_txq(port->netdev) < wanted_txqs))) { + != (netdev_n_txq(port->netdev) < wanted_txqs)) + || ((port->txq_mode == TXQ_MODE_HASH) + != (port->txq_requested_mode == TXQ_REQ_MODE_HASH))) { port->need_reconfigure = true; } } @@ -5810,8 +5863,15 @@ reconfigure_datapath(struct dp_netdev *dp) seq_change(dp->port_seq); port_destroy(port); } else { - port->txq_mode = (netdev_n_txq(port->netdev) < wanted_txqs) ? - TXQ_MODE_XPS : TXQ_MODE_STATIC; + /* With a single queue, there is no point in using hash mode. */ + if (port->txq_requested_mode == TXQ_REQ_MODE_HASH && + netdev_n_txq(port->netdev) > 1) { + port->txq_mode = TXQ_MODE_HASH; + } else if (netdev_n_txq(port->netdev) < wanted_txqs) { + port->txq_mode = TXQ_MODE_XPS; + } else { + port->txq_mode = TXQ_MODE_STATIC; + } } } @@ -6100,9 +6160,11 @@ pmd_free_cached_ports(struct dp_netdev_pmd_thread *pmd) dpif_netdev_xps_revalidate_pmd(pmd, true); HMAP_FOR_EACH_POP (tx_port_cached, node, &pmd->tnl_port_cache) { + free(tx_port_cached->txq_pkts); free(tx_port_cached); } HMAP_FOR_EACH_POP (tx_port_cached, node, &pmd->send_port_cache) { + free(tx_port_cached->txq_pkts); free(tx_port_cached); } } @@ -6122,14 +6184,27 @@ pmd_load_cached_ports(struct dp_netdev_pmd_thread *pmd) hmap_shrink(&pmd->tnl_port_cache); HMAP_FOR_EACH (tx_port, node, &pmd->tx_ports) { + int n_txq = netdev_n_txq(tx_port->port->netdev); + struct dp_packet_batch *txq_pkts_cached; + if (netdev_has_tunnel_push_pop(tx_port->port->netdev)) { tx_port_cached = xmemdup(tx_port, sizeof *tx_port_cached); + if (tx_port->txq_pkts) { + txq_pkts_cached = xmemdup(tx_port->txq_pkts, + n_txq * sizeof *tx_port->txq_pkts); + tx_port_cached->txq_pkts = txq_pkts_cached; + } hmap_insert(&pmd->tnl_port_cache, &tx_port_cached->node, hash_port_no(tx_port_cached->port->port_no)); } - if (netdev_n_txq(tx_port->port->netdev)) { + if (n_txq) { tx_port_cached = xmemdup(tx_port, sizeof *tx_port_cached); + if (tx_port->txq_pkts) { + txq_pkts_cached = xmemdup(tx_port->txq_pkts, + n_txq * sizeof *tx_port->txq_pkts); + tx_port_cached->txq_pkts = txq_pkts_cached; + } hmap_insert(&pmd->send_port_cache, &tx_port_cached->node, hash_port_no(tx_port_cached->port->port_no)); } @@ -6911,6 +6986,7 @@ dp_netdev_pmd_clear_ports(struct dp_netdev_pmd_thread *pmd) free(poll); } HMAP_FOR_EACH_POP (port, node, &pmd->tx_ports) { + free(port->txq_pkts); free(port); } ovs_mutex_unlock(&pmd->port_mutex); @@ -6981,6 +7057,14 @@ dp_netdev_add_port_tx_to_pmd(struct dp_netdev_pmd_thread *pmd, tx->flush_time = 0LL; dp_packet_batch_init(&tx->output_pkts); + if (tx->port->txq_mode == TXQ_MODE_HASH) { + int i, n_txq = netdev_n_txq(tx->port->netdev); + tx->txq_pkts = xzalloc(n_txq * sizeof *tx->txq_pkts); + for (i = 0; i < n_txq; i++) { + dp_packet_batch_init(&tx->txq_pkts[i]); + } + } + hmap_insert(&pmd->tx_ports, &tx->node, hash_port_no(tx->port->port_no)); pmd->need_reload = true; } @@ -6993,6 +7077,7 @@ dp_netdev_del_port_tx_from_pmd(struct dp_netdev_pmd_thread *pmd, OVS_REQUIRES(pmd->port_mutex) { hmap_remove(&pmd->tx_ports, &tx->node); + free(tx->txq_pkts); free(tx); pmd->need_reload = true; } From patchwork Fri Dec 17 15:10:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 1570111 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LjzpdT2/; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JFsr75DpVz9t55 for ; Sat, 18 Dec 2021 02:11:03 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E96E784D04; Fri, 17 Dec 2021 15:11:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D5tKtllUoZpf; Fri, 17 Dec 2021 15:10:59 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 33E3084CF4; Fri, 17 Dec 2021 15:10:57 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 04CC6C001E; Fri, 17 Dec 2021 15:10:57 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 136BAC001E for ; Fri, 17 Dec 2021 15:10:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3D9AA40C10 for ; Fri, 17 Dec 2021 15:10:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Vsxv8hrk8gY for ; Fri, 17 Dec 2021 15:10:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2F2A740BFA for ; Fri, 17 Dec 2021 15:10:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639753842; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u1XXUBVggpV1R65SLDvrbGwyiUcx0PCawrbYTsR4g5I=; b=LjzpdT2/DHbA5F9eF5SPcK0t1exw1q/7CAJ2HijhYlHfLZp10JVuysjCPmmC5oHGceN8lj G3uTNc7WUktj8DKo0l3Lzwt7F9eNrMMzfeLnSbcv5WE5O4UnSrsQKlAp4Qs8cdZdrBuINQ 2n7sIRLiQ2L/ywy0NXhcieq3MK9CXKY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-260-JAN0waT0NSCmWxg3FlVAbQ-1; Fri, 17 Dec 2021 10:10:34 -0500 X-MC-Unique: JAN0waT0NSCmWxg3FlVAbQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2A1EE192FDB5; Fri, 17 Dec 2021 15:10:33 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 781485E26D; Fri, 17 Dec 2021 15:10:31 +0000 (UTC) From: Maxime Coquelin To: dev@openvswitch.org Date: Fri, 17 Dec 2021 16:10:18 +0100 Message-Id: <20211217151018.436874-6-maxime.coquelin@redhat.com> In-Reply-To: <20211217151018.436874-1-maxime.coquelin@redhat.com> References: <20211217151018.436874-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: fbl@sysclose.org, Maxime Coquelin , i.maximets@ovn.org, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v4 5/5] dpif-netdev.at: Add test for Tx packets steering. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch introduces a new test for Tx packets steering modes. First test validates the static mode, by checking that all packets are transmitted on a single queue (single PMD thread), then it tests the same with enabling hash based packet steering, ensuring packets are transmitted on both queues. Signed-off-by: Maxime Coquelin Reviewed-by: David Marchand --- tests/dpif-netdev.at | 67 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/tests/dpif-netdev.at b/tests/dpif-netdev.at index 53eee185a..b7f1c3f8b 100644 --- a/tests/dpif-netdev.at +++ b/tests/dpif-netdev.at @@ -635,3 +635,70 @@ OVS_WAIT_UNTIL([grep "flow: in_port is not an exact match" ovs-vswitchd.log]) OVS_VSWITCHD_STOP(["/flow: in_port is not an exact match/d /failed to put/d"]) AT_CLEANUP + +# SEND_UDP_PKTS([p_name], [p_ofport]) +# +# Sends 128 packets to port 'p_name' with different UDP destination ports. +m4_define([SEND_UDP_PKTS], + [ + for i in `seq 1 128`; do + pkt="in_port($2),eth(src=50:54:00:00:00:05,dst=50:54:00:00:01:00),eth_type(0x0800),ipv4(src=10.0.1.1,dst=10.0.0.1,proto=17),udp(src=1000,dst=$i)" + ovs-appctl netdev-dummy/receive $1 $pkt --len 256 + done + ] +) + +AT_SETUP([dpif-netdev - tx packets steering]) +OVS_VSWITCHD_START( + [add-port br0 p1 -- set Interface p1 type=dummy-pmd ofport_request=1], [], [], [--dummy-numa 0]) + +dnl 'thread' mode, packets are expected to be transmitted on a single +dnl queue since there is only one PMD thread. + +AT_CHECK([ovs-vsctl add-port br0 p2 -- set Interface p2 type=dummy-pmd ofport_request=2 options:n_txq=2 other_config:tx-steering=thread]) +AT_CHECK([ovs-ofctl add-flow br0 in_port=1,actions=output:2]) + +AT_CHECK([SEND_UDP_PKTS([p1], [1])]) + +OVS_WAIT_UNTIL([test `ovs-vsctl get Interface p2 statistics:tx_packets` -eq 128]) +AT_CHECK([ovs-vsctl get Interface p2 statistics], [], [stdout]) +AT_CHECK([test `ovs-vsctl get Interface p2 statistics:tx_q0_packets` -eq 0 -a dnl + `ovs-vsctl get Interface p2 statistics:tx_q1_packets` -eq 128 || dnl + test `ovs-vsctl get Interface p2 statistics:tx_q0_packets` -eq 128 -a dnl + `ovs-vsctl get Interface p2 statistics:tx_q1_packets` -eq 0]) + +AT_CHECK([ovs-vsctl del-port p2]) + +dnl 'hash' mode, packets are expected to be transmitted on both +dnl queues, based on their hash value. + +AT_CHECK([ovs-vsctl add-port br0 p2 -- set Interface p2 type=dummy-pmd ofport_request=2 options:n_txq=2 other_config:tx-steering=hash]) +AT_CHECK([ovs-ofctl add-flow br0 in_port=1,actions=output:2]) + +AT_CHECK([SEND_UDP_PKTS([p1], [1])]) + +OVS_WAIT_UNTIL([test `ovs-vsctl get Interface p2 statistics:tx_packets` -eq 128]) +AT_CHECK([ovs-vsctl get Interface p2 statistics], [], [stdout]) + +AT_CHECK([test `ovs-vsctl get Interface p2 statistics:tx_q0_packets` -gt 0 -a dnl + `ovs-vsctl get Interface p2 statistics:tx_q1_packets` -gt 0]) + +AT_CHECK([ovs-vsctl del-port p2]) + +dnl 'hash' mode with hw-offload enabled, packets are expected to be transmitted on both +dnl queues, based on their hash value. + +AT_CHECK([ovs-vsctl set Open_vSwitch . other_config:hw-offload=true]) +AT_CHECK([ovs-vsctl add-port br0 p2 -- set Interface p2 type=dummy-pmd ofport_request=2 options:n_txq=2 other_config:tx-steering=hash]) +AT_CHECK([ovs-ofctl add-flow br0 in_port=1,actions=output:2]) + +AT_CHECK([SEND_UDP_PKTS([p1], [1])]) + +OVS_WAIT_UNTIL([test `ovs-vsctl get Interface p2 statistics:tx_packets` -eq 128]) +AT_CHECK([ovs-vsctl get Interface p2 statistics], [], [stdout]) + +AT_CHECK([test `ovs-vsctl get Interface p2 statistics:tx_q0_packets` -gt 0 -a dnl + `ovs-vsctl get Interface p2 statistics:tx_q1_packets` -gt 0]) + +OVS_VSWITCHD_STOP +AT_CLEANUP