From patchwork Thu Apr 25 15:22:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1090900 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44qgvZ6wLFz9s5c for ; Fri, 26 Apr 2019 01:24:10 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id BA7C41F19; Thu, 25 Apr 2019 15:23:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 0E6981E4A for ; Thu, 25 Apr 2019 15:22:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 97F2982A for ; Thu, 25 Apr 2019 15:22:30 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3037C309266A; Thu, 25 Apr 2019 15:22:30 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.205.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 93593600C0; Thu, 25 Apr 2019 15:22:27 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Thu, 25 Apr 2019 17:22:10 +0200 Message-Id: <1556205730-10204-3-git-send-email-david.marchand@redhat.com> In-Reply-To: <1556205730-10204-1-git-send-email-david.marchand@redhat.com> References: <1556205730-10204-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Thu, 25 Apr 2019 15:22:30 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@samsung.com, maxime.coquelin@redhat.com Subject: [ovs-dev] [PATCH v3 3/3] netdev-dpdk: Reset queue number for vhost devices on vm shutdown. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Rather than poll all disabled queues and waste some memory for vms that have been shutdown, we can reconfigure when receiving a destroy connection notification from the vhost library. $ while true; do ovs-appctl dpif-netdev/pmd-rxq-show |awk ' /port: / { tot++; if ($5 == "(enabled)") { en++; } } END { print "total: " tot ", enabled: " en }' sleep 1 done total: 66, enabled: 66 total: 6, enabled: 2 Co-authored-by: Ilya Maximets Signed-off-by: Ilya Maximets Signed-off-by: David Marchand --- This patch requires a dpdk fix for the vhost library: https://git.dpdk.org/dpdk/commit/?id=2344395df2e9 The best would be to consider applying it when this dpdk fix has been backported in 18.11 stable branch, most likely, for 18.11.2. I can resubmit it later if needed. Changes since v2: - added authorship tags for Ilya - added logs in destroy_connection --- lib/netdev-dpdk.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 3372373..79f3983 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -186,12 +186,15 @@ static const struct rte_eth_conf port_conf = { static int new_device(int vid); static void destroy_device(int vid); static int vring_state_changed(int vid, uint16_t queue_id, int enable); +static void destroy_connection(int vid); static const struct vhost_device_ops virtio_net_device_ops = { .new_device = new_device, .destroy_device = destroy_device, .vring_state_changed = vring_state_changed, - .features_changed = NULL + .features_changed = NULL, + .new_connection = NULL, + .destroy_connection = destroy_connection, }; enum { DPDK_RING_SIZE = 256 }; @@ -3661,6 +3664,48 @@ vring_state_changed(int vid, uint16_t queue_id, int enable) return 0; } +static void +destroy_connection(int vid) +{ + struct netdev_dpdk *dev; + char ifname[IF_NAME_SZ]; + bool exists = false; + + rte_vhost_get_ifname(vid, ifname, sizeof ifname); + + ovs_mutex_lock(&dpdk_mutex); + LIST_FOR_EACH (dev, list_node, &dpdk_list) { + ovs_mutex_lock(&dev->mutex); + if (nullable_string_is_equal(ifname, dev->vhost_id)) { + uint32_t qp_num = NR_QUEUE; + + if (netdev_dpdk_get_vid(dev) >= 0) { + VLOG_ERR("Connection on socket '%s' destroyed while vhost " + "device still attached.", dev->vhost_id); + } + + /* Restore the number of queue pairs to default. */ + if (dev->requested_n_txq != qp_num + || dev->requested_n_rxq != qp_num) { + dev->requested_n_rxq = qp_num; + dev->requested_n_txq = qp_num; + netdev_request_reconfigure(&dev->up); + } + ovs_mutex_unlock(&dev->mutex); + exists = true; + break; + } + ovs_mutex_unlock(&dev->mutex); + } + ovs_mutex_unlock(&dpdk_mutex); + + if (exists) { + VLOG_INFO("vHost Device '%s' connection has been destroyed", ifname); + } else { + VLOG_INFO("vHost Device '%s' not found", ifname); + } +} + /* * Retrieve the DPDK virtio device ID (vid) associated with a vhostuser * or vhostuserclient netdev.