From patchwork Thu Jun 27 09:43:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1123211 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45ZFN0496Gz9s4Y for ; Thu, 27 Jun 2019 19:44:00 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 51082E6E; Thu, 27 Jun 2019 09:43:58 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id C057BE61 for ; Thu, 27 Jun 2019 09:43:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 2307D82D for ; Thu, 27 Jun 2019 09:43:56 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 827A759468; Thu, 27 Jun 2019 09:43:52 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-76.brq.redhat.com [10.40.204.76]) by smtp.corp.redhat.com (Postfix) with ESMTP id 204F460856; Thu, 27 Jun 2019 09:43:44 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Thu, 27 Jun 2019 11:43:36 +0200 Message-Id: <1561628616-10218-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 27 Jun 2019 09:43:52 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: i.maximets@samsung.com, maxime.coquelin@redhat.com Subject: [ovs-dev] [PATCH v4] netdev-dpdk: Reset queue number for vhost devices on vm shutdown. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Rather than poll all disabled queues and waste some memory for vms that have been shutdown, we can reconfigure when receiving a destroy connection notification from the vhost library. $ while true; do ovs-appctl dpif-netdev/pmd-rxq-show |awk ' /port: / { tot++; if ($5 == "(enabled)") { en++; } } END { print "total: " tot ", enabled: " en }' sleep 1 done total: 66, enabled: 66 total: 6, enabled: 2 This change requires a fix in the DPDK vhost library, so bump the minimal required version to 18.11.2. Co-authored-by: Ilya Maximets Signed-off-by: Ilya Maximets Signed-off-by: David Marchand --- Changes since v3: - rebased on master following 18.11.2 support - added a note in NEWS about minimal dpdk version Changes since v2: - added authorship tags for Ilya - added logs in destroy_connection --- NEWS | 4 +++- lib/netdev-dpdk.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/NEWS b/NEWS index c03e287..2f8171f 100644 --- a/NEWS +++ b/NEWS @@ -4,7 +4,9 @@ Post-v2.11.0 * New option 'other_config:dpdk-socket-limit' to limit amount of hugepage memory that can be used by DPDK. * Add support for vHost Post-copy Live Migration (experimental). - * OVS validated with DPDK 18.11.2 which is recommended to be used. + * OVS validated with DPDK 18.11.2 which is the new minimal supported + version. + * DPDK 18.11.1 and lower is no longer supported. - OpenFlow: * All features required by OpenFlow 1.5 are now implemented, so ovs-vswitchd now enables OpenFlow 1.5 by default (in addition to diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index b7f4543..0b9bea4 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -185,12 +185,15 @@ static const struct rte_eth_conf port_conf = { static int new_device(int vid); static void destroy_device(int vid); static int vring_state_changed(int vid, uint16_t queue_id, int enable); +static void destroy_connection(int vid); static const struct vhost_device_ops virtio_net_device_ops = { .new_device = new_device, .destroy_device = destroy_device, .vring_state_changed = vring_state_changed, - .features_changed = NULL + .features_changed = NULL, + .new_connection = NULL, + .destroy_connection = destroy_connection, }; enum { DPDK_RING_SIZE = 256 }; @@ -3663,6 +3666,48 @@ vring_state_changed(int vid, uint16_t queue_id, int enable) return 0; } +static void +destroy_connection(int vid) +{ + struct netdev_dpdk *dev; + char ifname[IF_NAME_SZ]; + bool exists = false; + + rte_vhost_get_ifname(vid, ifname, sizeof ifname); + + ovs_mutex_lock(&dpdk_mutex); + LIST_FOR_EACH (dev, list_node, &dpdk_list) { + ovs_mutex_lock(&dev->mutex); + if (nullable_string_is_equal(ifname, dev->vhost_id)) { + uint32_t qp_num = NR_QUEUE; + + if (netdev_dpdk_get_vid(dev) >= 0) { + VLOG_ERR("Connection on socket '%s' destroyed while vhost " + "device still attached.", dev->vhost_id); + } + + /* Restore the number of queue pairs to default. */ + if (dev->requested_n_txq != qp_num + || dev->requested_n_rxq != qp_num) { + dev->requested_n_rxq = qp_num; + dev->requested_n_txq = qp_num; + netdev_request_reconfigure(&dev->up); + } + ovs_mutex_unlock(&dev->mutex); + exists = true; + break; + } + ovs_mutex_unlock(&dev->mutex); + } + ovs_mutex_unlock(&dpdk_mutex); + + if (exists) { + VLOG_INFO("vHost Device '%s' connection has been destroyed", ifname); + } else { + VLOG_INFO("vHost Device '%s' not found", ifname); + } +} + /* * Retrieve the DPDK virtio device ID (vid) associated with a vhostuser * or vhostuserclient netdev.