From patchwork Fri May 12 15:10:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 761690 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wPYPG0Dy9z9s3w for ; Sat, 13 May 2017 01:12:34 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 8DFE8B8C; Fri, 12 May 2017 15:11:08 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id D2CD7B65 for ; Fri, 12 May 2017 15:11:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id D1DFF1D6 for ; Fri, 12 May 2017 15:11:05 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OPU00J4NI6F3C40@mailout1.w1.samsung.com> for dev@openvswitch.org; Fri, 12 May 2017 16:11:03 +0100 (BST) Received: from eusmges3.samsung.com (unknown [203.254.199.242]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170512151103eucas1p29749bf11ac305c52006df3afedcb0e6a~95HnJTZGA1677416774eucas1p2b; Fri, 12 May 2017 15:11:03 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges3.samsung.com (EUCPMTA) with SMTP id CD.EA.17464.680D5195; Fri, 12 May 2017 16:11:02 +0100 (BST) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170512151101eucas1p2a5c60f31d80db8cfcb6f1789517d8540~95HmHFmng2264422644eucas1p2W; Fri, 12 May 2017 15:11:01 +0000 (GMT) X-AuditID: cbfec7f2-f797e6d000004438-8c-5915d08698ed Received: from eusync3.samsung.com ( [203.254.199.213]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id F7.7B.17452.911D5195; Fri, 12 May 2017 16:13:29 +0100 (BST) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync3.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OPU00L2WI5RWP70@eusync3.samsung.com>; Fri, 12 May 2017 16:11:01 +0100 (BST) From: Ilya Maximets To: dev@openvswitch.org, Daniele Di Proietto , Darrell Ball Date: Fri, 12 May 2017 18:10:27 +0300 Message-id: <1494601828-30999-3-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.7.4 In-reply-to: <1494601828-30999-1-git-send-email-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrCIsWRmVeSWpSXmKPExsWy7djPc7ptF0QjDdbPNbL49eYBu8WryQ2M Fi39M5ktnr9YyGxx9PQeZou7P1YwWUz7fJvd4kr7T3YHDo/Fe14yeTy7+Z/R4/m1HhaP9/uu snn0bVnF6PFu/lu2ALYoLpuU1JzMstQifbsEroz7q5+zF7y3r9j9fwpzA+M8/S5GDg4JAROJ a79Suhg5gUwxiQv31rN1MXJxCAksZZS4PHclM4TzGci5+JkdospEYsnVv+wQiWWMEgs/XmWE cJqZJNa8mM8KUsUmoCNxavURRpAVIgKpEo8WS4KEmQV2MEqsmCgIYgsLuEl8urmNBcRmEVCV uLljIhuIzQsUb926mBlimZzEzXOdYDangLvEm1W7WEF2SQg0s0vMWrOWDeIFWYlNB6DqXSS2 LTkBZQtLvDq+BepoGYnOjoNMUL2MEg2rLjFCOBMYJb40L2eCqLKXOHXzKhPEpXwSk7ZNZ4ZY wCvR0SYEUeIhcb3lNdRQR4nDfxuhnp/FKDF3wTe2CYwyCxgZVjGKpJYW56anFhvrFSfmFpfm pesl5+duYgTG9el/xz/tYPx6wuoQowAHoxIPr8Ja0Ugh1sSy4srcQ4wSHMxKIrw5J4BCvCmJ lVWpRfnxRaU5qcWHGKU5WJTEeblOXYsQEkhPLEnNTk0tSC2CyTJxcEo1MBpfDdX6app1+TjD ln23NjaFCJ/kzGDasjJ52Y/P6z78vFOwzdNvxtemnyYHfql/vRilNvl8r9wnlquzpjxu8DAX cAu5Gcyowvb4YlrDq7QV/QfUd5zZanY6tEhwovh/qzrbS2e36GjOuGY6iSPIeNHDzeU/ZZZu llipkxHOXF5y8WbW12eNN7mUWIozEg21mIuKEwGh56gR5wIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrMLMWRmVeSWpSXmKPExsVy+t/xq7qSF0UjDXbMlrL49eYBu8WryQ2M Fi39M5ktnr9YyGxx9PQeZou7P1YwWUz7fJvd4kr7T3YHDo/Fe14yeTy7+Z/R4/m1HhaP9/uu snn0bVnF6PFu/lu2ALYoN5uM1MSU1CKF1Lzk/JTMvHRbpdAQN10LJYW8xNxUW6UIXd+QICWF ssScUiDPyAANODgHuAcr6dsluGXcX/2cveC9fcXu/1OYGxjn6XcxcnJICJhILLn6lx3CFpO4 cG89WxcjF4eQwBJGifV7e5kgnFYmid53m5hAqtgEdCROrT7C2MXIwSEikCpx7pwISJhZYAej xK0HRSC2sICbxKeb21hAbBYBVYmbOyaygdi8QPHWrYuZIZbJSdw81wlmcwq4S7xZtYsVxBYC qpn6fjr7BEbeBYwMqxhFUkuLc9Nziw31ihNzi0vz0vWS83M3MQIDfNuxn5t3MF7aGHyIUYCD UYmHV2GtaKQQa2JZcWXuIUYJDmYlEd6cE0Ah3pTEyqrUovz4otKc1OJDjKZAR01klhJNzgdG X15JvKGJobmloZGxhYW5kZGSOG/JhyvhQgLpiSWp2ampBalFMH1MHJxSDYxT99u7OWT1/tp2 Jfvj2bX/Wx7N3Cn6Xl7UfOHiWa8rUmq1Uw9bHzDXLBUuWb69KvfbHRn9kpsLfzlq7Fxqt2hT Z+Liijvnsmf+SZP0f1N/9dz9ZitmcV5/Fr6VwiI92vnSv169WzH3+6NXb58oTDdZnFd1SHG3 PMt92wUbE/Xa8l9Vzd4/+2euEktxRqKhFnNRcSIAYII4AIYCAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170512151101eucas1p2a5c60f31d80db8cfcb6f1789517d8540 X-Msg-Generator: CA X-Sender-IP: 182.198.249.179 X-Local-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G+yCvOyEseyghOyekBtMZWFkaW5nIEVuZ2luZWVy?= X-Global-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G1NhbXN1bmcgRWxlY3Ryb25pY3MbTGVhZGluZyBFbmdpbmVlcg==?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170512151101eucas1p2a5c60f31d80db8cfcb6f1789517d8540 X-RootMTR: 20170512151101eucas1p2a5c60f31d80db8cfcb6f1789517d8540 References: <1494601828-30999-1-git-send-email-i.maximets@samsung.com> X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , Heetae Ahn Subject: [ovs-dev] [PATCH v2 2/3] netdev-dpdk: Fix device leak on port deletion. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Currently, once created device in dpdk will exist forever even after del-port operation untill we manually call 'ovs-appctl netdev-dpdk/detach ', where is not the port's name but the name of dpdk eth device or pci address. Few issues with current implementation: 1. Different API for usual (system) and DPDK devices. (We have to call 'ovs-appctl netdev-dpdk/detach' each time after 'del-port' to actually free the device) This is a big issue mostly for virtual DPDK devices. 2. Follows from 1: For DPDK devices 'del-port' leads just to 'rte_eth_dev_stop' and subsequent 'add-port' will just start the already existing device. Such behaviour will not reset the device to initial state as it could be expected. For example: virtual pcap pmd will continue reading input file instead of reading it from the beginning. 3. Follows from 2: After execution of the following commands 'port1' will be configured with the 'old-options' while 'ovs-vsctl show' will show us 'new-options' in dpdk-devargs field: ovs-vsctl add-port port1 -- set interface port1 type=dpdk \ options:dpdk-devargs=, ovs-vsctl del-port port1 ovs-vsctl add-port port1 -- set interface port1 type=dpdk \ options:dpdk-devargs=, 4. Follows from 1: Not detached device consumes 'port_id'. Since we have very limited number of 'port_id's (32 in common case) this may lead to quick exhausting of id pool and inability to add any other port. To avoid above issues we need to detach all the attached devices on port destruction. appctl 'netdev-dpdk/detach' removed because not needed anymore. We need to use internal 'attached' variable to track ports on which rte_eth_dev_attach() was called and returned successfully to avoid closing and detaching devices that do not support hotplug or by any other reason attached using the 'dpdk-extra' cmdline options. CC: Ciara Loftus Fixes: 55e075e65ef9 ("netdev-dpdk: Arbitrary 'dpdk' port naming") Fixes: 69876ed78611 ("netdev-dpdk: Add support for virtual DPDK PMDs (vdevs)") Signed-off-by: Ilya Maximets --- Documentation/howto/dpdk.rst | 5 ++- lib/netdev-dpdk.c | 72 ++++++++++++-------------------------------- 2 files changed, 22 insertions(+), 55 deletions(-) diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index dc63f7d..20d8975 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -342,10 +342,9 @@ Then it can be attached to OVS:: $ ovs-vsctl add-port br0 dpdkx -- set Interface dpdkx type=dpdk \ options:dpdk-devargs=0000:01:00.0 -It is also possible to detach a port from ovs, the user has to remove the -port using the del-port command, then it can be detached using:: +Detaching will be performed while processing del-port command:: - $ ovs-appctl netdev-dpdk/detach 0000:01:00.0 + $ ovs-vsctl del-port dpdkx This feature is not supported with VFIO and does not work with some NICs. For more information please refer to the `DPDK Port Hotplug Framework diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 56b9d25..baac7c3 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -359,6 +359,9 @@ struct netdev_dpdk { /* Device arguments for dpdk ports */ char *devargs; + /* If true, device was attached by rte_eth_dev_attach(). */ + bool attached; + /* In dpdk_list. */ struct ovs_list list_node OVS_GUARDED_BY(dpdk_mutex); @@ -852,6 +855,7 @@ common_construct(struct netdev *netdev, unsigned int port_no, dev->max_packet_len = MTU_TO_FRAME_LEN(dev->mtu); ovsrcu_index_init(&dev->vid, -1); dev->vhost_reconfigured = false; + dev->attached = false; ovsrcu_init(&dev->qos_conf, NULL); @@ -997,10 +1001,21 @@ static void netdev_dpdk_destruct(struct netdev *netdev) { struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + char devname[RTE_ETH_NAME_MAX_LEN]; ovs_mutex_lock(&dpdk_mutex); rte_eth_dev_stop(dev->port_id); + + if (dev->attached) { + rte_eth_dev_close(dev->port_id); + if (rte_eth_dev_detach(dev->port_id, devname) < 0) { + VLOG_ERR("Device '%s' can not be detached", dev->devargs); + } else { + VLOG_INFO("Device '%s' detached", devname); + } + } + free(dev->devargs); common_destruct(dev); @@ -1112,7 +1127,8 @@ netdev_dpdk_lookup_by_port_id(int port_id) } static int -netdev_dpdk_process_devargs(const char *devargs, char **errp) +netdev_dpdk_process_devargs(struct netdev_dpdk *dev, + const char *devargs, char **errp) { /* Get the name up to the first comma. */ char *name = xmemdup0(devargs, strcspn(devargs, ",")); @@ -1124,6 +1140,7 @@ netdev_dpdk_process_devargs(const char *devargs, char **errp) /* Device not found in DPDK, attempt to attach it */ if (!rte_eth_dev_attach(devargs, &new_port_id)) { /* Attach successful */ + dev->attached = true; VLOG_INFO("Device '%s' attached to DPDK", devargs); } else { /* Attach unsuccessful */ @@ -1209,7 +1226,8 @@ netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args, * is valid */ if (!(dev->devargs && !strcmp(dev->devargs, new_devargs) && rte_eth_dev_is_valid_port(dev->port_id))) { - int new_port_id = netdev_dpdk_process_devargs(new_devargs, errp); + int new_port_id = netdev_dpdk_process_devargs(dev, new_devargs, + errp); if (!rte_eth_dev_is_valid_port(new_port_id)) { err = EINVAL; } else if (new_port_id == dev->port_id) { @@ -2432,53 +2450,6 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn, int argc, unixctl_command_reply(conn, "OK"); } -static void -netdev_dpdk_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, - const char *argv[], void *aux OVS_UNUSED) -{ - int ret; - char *response; - uint8_t port_id; - char devname[RTE_ETH_NAME_MAX_LEN]; - struct netdev_dpdk *dev; - - ovs_mutex_lock(&dpdk_mutex); - - if (!rte_eth_dev_count() || rte_eth_dev_get_port_by_name(argv[1], - &port_id)) { - response = xasprintf("Device '%s' not found in DPDK", argv[1]); - goto error; - } - - dev = netdev_dpdk_lookup_by_port_id(port_id); - if (dev) { - response = xasprintf("Device '%s' is being used by interface '%s'. " - "Remove it before detaching", - argv[1], netdev_get_name(&dev->up)); - goto error; - } - - rte_eth_dev_close(port_id); - - ret = rte_eth_dev_detach(port_id, devname); - if (ret < 0) { - response = xasprintf("Device '%s' can not be detached", argv[1]); - goto error; - } - - response = xasprintf("Device '%s' has been detached", argv[1]); - - ovs_mutex_unlock(&dpdk_mutex); - unixctl_command_reply(conn, response); - free(response); - return; - -error: - ovs_mutex_unlock(&dpdk_mutex); - unixctl_command_reply_error(conn, response); - free(response); -} - /* * Set virtqueue flags so that we do not receive interrupts. */ @@ -2754,9 +2725,6 @@ netdev_dpdk_class_init(void) unixctl_command_register("netdev-dpdk/set-admin-state", "[netdev] up|down", 1, 2, netdev_dpdk_set_admin_state, NULL); - unixctl_command_register("netdev-dpdk/detach", - "pci address of device", 1, 1, - netdev_dpdk_detach, NULL); ovsthread_once_done(&once); }