From patchwork Sun Oct 13 15:55:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1175849 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46rmj74fjmz9sP4 for ; Mon, 14 Oct 2019 03:04:26 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id CE06923BA; Sun, 13 Oct 2019 16:04:22 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 12B9523B1 for ; Sun, 13 Oct 2019 15:55:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 76E536CE for ; Sun, 13 Oct 2019 15:55:23 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6449E30833CB; Sun, 13 Oct 2019 15:55:22 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-29.brq.redhat.com [10.40.204.29]) by smtp.corp.redhat.com (Postfix) with ESMTP id 118C960C05; Sun, 13 Oct 2019 15:55:19 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Sun, 13 Oct 2019 17:55:08 +0200 Message-Id: <1570982108-3671-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Sun, 13 Oct 2019 15:55:22 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2] netdev-dpdk: Track vhost tx contention. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Add a coverage counter to help diagnose contention on the vhost txqs. This is seen as dropped packets on the physical ports for rates that are usually handled fine by OVS. Document how to further debug this contention with perf. Signed-off-by: David Marchand Acked-by: Eelco Chaudron --- Changelog since v1: - added documentation as a bonus: not sure this is the right place, or if it really makes sense to enter into such details. But I still find it useful. Comments? --- Documentation/topics/dpdk/vhost-user.rst | 61 ++++++++++++++++++++++++++++++++ lib/netdev-dpdk.c | 8 ++++- 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst index fab87bd..c7e605e 100644 --- a/Documentation/topics/dpdk/vhost-user.rst +++ b/Documentation/topics/dpdk/vhost-user.rst @@ -623,3 +623,64 @@ Because of this limitation, this feature is considered 'experimental'. Further information can be found in the `DPDK documentation `__ + +Troubleshooting vhost-user tx contention +---------------------------------------- + +Depending on the number of a virtio port Rx queues enabled by a guest and on +the number of PMDs used on OVS side, OVS can end up with contention occuring +on the lock protecting the vhost Tx queue. +This problem can be hard to catch since it is noticeable as an increased cpu +cost for handling the received packets and, usually, as drops in the +statistics of the physical port receiving the packets. + +To identify such a situation, a coverage statistic is available:: + + $ ovs-appctl coverage/read-counter vhost_tx_contention + 59530681 + +If you want to further debug this contention, perf can be used if your OVS +daemon had been compiled with debug symbols. + +First, identify the point in the binary sources where the contention occurs:: + + $ perf probe -x $(which ovs-vswitchd) -L __netdev_dpdk_vhost_send \ + |grep -B 3 -A 3 'COVERAGE_INC(vhost_tx_contention)' + } + + 21 if (unlikely(!rte_spinlock_trylock(&dev->tx_q[qid].tx_lock))) { + 22 COVERAGE_INC(vhost_tx_contention); + 23 rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + } + +Then, place a probe at the line where the lock is taken. +You can add additional context to catch which port and queue are concerned:: + + $ perf probe -x $(which ovs-vswitchd) \ + 'vhost_tx_contention=__netdev_dpdk_vhost_send:23 netdev->name:string qid' + +Finally, gather data and generate a report:: + + $ perf record -e probe_ovs:vhost_tx_contention -aR sleep 10 + [ perf record: Woken up 120 times to write data ] + [ perf record: Captured and wrote 30.151 MB perf.data (356278 samples) ] + + $ perf report -F +pid --stdio + # To display the perf.data header info, please use --header/--header-only options. + # + # + # Total Lost Samples: 0 + # + # Samples: 356K of event 'probe_ovs:vhost_tx_contention' + # Event count (approx.): 356278 + # + # Overhead Pid:Command Trace output + # ........ ..................... ............................ + # + 55.57% 83332:pmd-c01/id:33 (9e9775) name="vhost0" qid=0 + 44.43% 83333:pmd-c15/id:34 (9e9775) name="vhost0" qid=0 + + + # + # (Tip: Treat branches as callchains: perf report --branch-history) + # diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 7f709ff..3525870 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -41,6 +41,7 @@ #include #include "cmap.h" +#include "coverage.h" #include "dirs.h" #include "dp-packet.h" #include "dpdk.h" @@ -72,6 +73,8 @@ enum {VIRTIO_RXQ, VIRTIO_TXQ, VIRTIO_QNUM}; VLOG_DEFINE_THIS_MODULE(netdev_dpdk); static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20); +COVERAGE_DEFINE(vhost_tx_contention); + #define DPDK_PORT_WATCHDOG_INTERVAL 5 #define OVS_CACHE_LINE_SIZE CACHE_LINE_SIZE @@ -2353,7 +2356,10 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, goto out; } - rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + if (unlikely(!rte_spinlock_trylock(&dev->tx_q[qid].tx_lock))) { + COVERAGE_INC(vhost_tx_contention); + rte_spinlock_lock(&dev->tx_q[qid].tx_lock); + } cnt = netdev_dpdk_filter_packet_len(dev, cur_pkts, cnt); /* Check has QoS has been configured for the netdev */