From patchwork Sun Jul 7 16:13:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1128693 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hYhk1ghLz9s7T for ; Mon, 8 Jul 2019 02:20:21 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 8AA921E64; Sun, 7 Jul 2019 16:20:17 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 438BB1E42 for ; Sun, 7 Jul 2019 16:13:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id B1AAB823 for ; Sun, 7 Jul 2019 16:13:34 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 10B6359469; Sun, 7 Jul 2019 16:13:29 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-39.brq.redhat.com [10.40.204.39]) by smtp.corp.redhat.com (Postfix) with ESMTP id 51610BA51; Sun, 7 Jul 2019 16:13:26 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Sun, 7 Jul 2019 18:13:20 +0200 Message-Id: <1562516000-6807-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Sun, 07 Jul 2019 16:13:34 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@samsung.com Subject: [ovs-dev] [PATCH] netdev-dpdk: Add a statistic on vhost tx contention. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Add a statistic to help diagnose contention on the vhost txqs. This is usually seen as dropped packets on the physical ports for rates that are usually handled fine by OVS. Signed-off-by: David Marchand --- This patch applies on top of Kevin "vhost-retry" series [1]. 1: https://patchwork.ozlabs.org/project/openvswitch/list/?submitter=70482 --- Documentation/topics/dpdk/vhost-user.rst | 25 +++++++++++++++++++++++++ lib/netdev-dpdk.c | 20 ++++++++++++++++++-- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst index 724aa62..e276b21 100644 --- a/Documentation/topics/dpdk/vhost-user.rst +++ b/Documentation/topics/dpdk/vhost-user.rst @@ -553,6 +553,31 @@ shown with:: $ ovs-vsctl get Interface dpdkvhostclient0 statistics:tx_retries +vhost tx contention +------------------- + +Contrary to physical ports where the number of Transmit queues is decided at +the configuration of the port and does not change once set up, the number of +Transmit queues for a vhost-user or vhost-user-client interface is negociated +with the guest when its driver enables a virtio interface and OVS has no +control over it. + +Each PMD threads needs a Transmit queue to send packet on such a port. +To accomodate with the cases where the number of PMD threads differs from the +number of Transmit queues enabled by a Virtual Machine, the Transmit queues are +distributed among the PMD threads and their accesses are protected by a +spinlock. + +When contention occurs (because two PMD threads are mapped to the same Transmit +queue), it can be hard to understand this phenomenon. +It usually will be seen as OVS taking more cycles to handle the packet and can +be seen as packets getting dropped on the receive side (rx_dropped statistic +on the physical ports). + +A statistic for the port is available to catch those cases:: + + ovs-vsctl get Interface dpdkvhostclient0 statistics:tx_contentions + vhost-user Dequeue Zero Copy (experimental) ------------------------------------------- diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 0e70e44..844fb3c 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -139,10 +139,11 @@ BUILD_ASSERT_DECL((MAX_NB_MBUF / ROUND_DOWN_POW2(MAX_NB_MBUF / MIN_NB_MBUF)) #define XSTAT_RX_JABBER_ERRORS "rx_jabber_errors" /* Size of vHost custom stats. */ -#define VHOST_CUSTOM_STATS_SIZE 1 +#define VHOST_CUSTOM_STATS_SIZE 2 /* Names of vHost custom stats. */ #define VHOST_STAT_TX_RETRIES "tx_retries" +#define VHOST_STAT_TX_CONTENTIONS "tx_contentions" #define SOCKET0 0 @@ -340,6 +341,8 @@ struct dpdk_tx_queue { * pmd threads (see 'concurrent_txq'). */ int map; /* Mapping of configured vhost-user queues * to enabled by guest. */ + uint64_t tx_contentions; /* Number of times a pmd has been waiting + * for another to xmit on this queue. */ }; /* dpdk has no way to remove dpdk ring ethernet devices @@ -2387,7 +2390,10 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, goto out; } - rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + if (unlikely(!rte_spinlock_trylock(&dev->tx_q[qid].tx_lock))) { + rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + atomic_count_inc64(&dev->tx_q[qid].tx_contentions); + } cnt = netdev_dpdk_filter_packet_len(dev, cur_pkts, cnt); /* Check has QoS has been configured for the netdev */ @@ -2878,6 +2884,16 @@ netdev_dpdk_vhost_get_custom_stats(const struct netdev *netdev, custom_stats->counters[0].value = dev->tx_retries; rte_spinlock_unlock(&dev->stats_lock); + ovs_strlcpy(custom_stats->counters[1].name, VHOST_STAT_TX_CONTENTIONS, + NETDEV_CUSTOM_STATS_NAME_SIZE); + custom_stats->counters[1].value = 0; + for (unsigned int i = 0; i < netdev->n_txq; i++) { + uint64_t tx_contentions; + + atomic_read_relaxed(&dev->tx_q[i].tx_contentions, &tx_contentions); + custom_stats->counters[1].value += tx_contentions; + } + ovs_mutex_unlock(&dev->mutex); return 0;