From patchwork Wed May 17 14:40:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 763583 X-Patchwork-Delegate: dlu998@gmail.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wScTq5xxvz9s7f for ; Thu, 18 May 2017 00:42:07 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 3D5BABBE; Wed, 17 May 2017 14:41:01 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id B622CBA2 for ; Wed, 17 May 2017 14:40:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout4.w1.samsung.com (mailout4.w1.samsung.com [210.118.77.14]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 589FD193 for ; Wed, 17 May 2017 14:40:58 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout4.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0OQ300J9TQ48OM80@mailout4.w1.samsung.com> for dev@openvswitch.org; Wed, 17 May 2017 15:40:56 +0100 (BST) Received: from eusmges2.samsung.com (unknown [203.254.199.241]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20170517144056eucas1p1c6b207d734e9f17b3c9d63fb7145bb2c~-a7vu9MT32200222002eucas1p1L; Wed, 17 May 2017 14:40:56 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2.samsung.com (EUCPMTA) with SMTP id 4C.AD.04459.7F06C195; Wed, 17 May 2017 15:40:55 +0100 (BST) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20170517144055eucas1p2bbdf11f385f65bf07c454a15dae7f07c~-a7vFrick2628426284eucas1p2N; Wed, 17 May 2017 14:40:55 +0000 (GMT) X-AuditID: cbfec7f1-f796e6d00000116b-ba-591c60f77fb5 Received: from eusync4.samsung.com ( [203.254.199.214]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 6D.93.17452.E816C195; Wed, 17 May 2017 15:43:26 +0100 (BST) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync4.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0OQ300KCWQ3NUI60@eusync4.samsung.com>; Wed, 17 May 2017 15:40:55 +0100 (BST) From: Ilya Maximets To: dev@openvswitch.org, Daniele Di Proietto , Darrell Ball Date: Wed, 17 May 2017 17:40:26 +0300 Message-id: <1495032027-23517-3-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.7.4 In-reply-to: <1495032027-23517-1-git-send-email-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrEIsWRmVeSWpSXmKPExsWy7djP87rfE2QiDTpuWln8evOA3eLV5AZG i5b+mcwWz18sZLY4enoPs8XdHyuYLKZ9vs1ucaX9J7sDh8fiPS+ZPJ7d/M/o8fxaD4vH+31X 2Tz6tqxi9Hg3/y1bAFsUl01Kak5mWWqRvl0CV8afT59YC97bV1x7NZe1gXGefhcjJ4eEgInE 1tnbmSFsMYkL99azdTFycQgJLGWU+D3zGjOE85lR4uiVSWwwHQv7vrBCJJYxSpw+8IwFwmlm kmidMI0FpIpNQEfi1OojjF2MHBwiAqkSjxZLgoSZBXYwSqyYKAhiCwu4STzvnMgOYrMIqErs X38E7AxeoPiH7SvZIZbJSdw818kMMoZTwF3i3joHkFUSAs3sEjcenGAFiUsIyEpsOsAMYbpI rF6bBtEpLPHq+BaoKTISnR0HmaBaGSUaVl1ihHAmMEp8aV7OBFFlL3Hq5lUmiDv5JCZtmw41 lFeio00IosRD4szha1BrHSV+NBtDfD6LUeLg5KNsExhlFjAyrGIUSS0tzk1PLTbSK07MLS7N S9dLzs/dxAiM6dP/jn/cwfj+hNUhRgEORiUe3g1BMpFCrIllxZW5hxglOJiVRHhz4oBCvCmJ lVWpRfnxRaU5qcWHGKU5WJTEeblOXYsQEkhPLEnNTk0tSC2CyTJxcEo1MFZ112QfPeHRy5gs bxkldXOJvHyQ2F8Lv5Uh5nK19YqXxX/75GQlLV3T8r3qmWHqzU0S/iszGHgWzDbolss9crdf 4sLx0ii23j+bvHaEbZl47VbNrJfPH1ceOiCpYRIQMGE/v1TK78sX/Tm/7ai5z6vdcLMyvdg3 KC2XR+WXd+tyj+AnIt+tlFiKMxINtZiLihMBq61SOOUCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrELMWRmVeSWpSXmKPExsVy+t/xa7p9iTKRBn9fm1n8evOA3eLV5AZG i5b+mcwWz18sZLY4enoPs8XdHyuYLKZ9vs1ucaX9J7sDh8fiPS+ZPJ7d/M/o8fxaD4vH+31X 2Tz6tqxi9Hg3/y1bAFuUm01GamJKapFCal5yfkpmXrqtUmiIm66FkkJeYm6qrVKErm9IkJJC WWJOKZBnZIAGHJwD3IOV9O0S3DL+fPrEWvDevuLaq7msDYzz9LsYOTkkBEwkFvZ9YYWwxSQu 3FvP1sXIxSEksIRR4ujjd+wQTiuTxLLp55hBqtgEdCROrT7C2MXIwSEikCpx7pwISJhZYAej xK0HRSC2sICbxPPOiewgNouAqsT+9UfAWnmB4h+2r2SHWCYncfNcJzPIGE4Bd4l76xxAwkJA JT0HrjJPYORdwMiwilEktbQ4Nz232FCvODG3uDQvXS85P3cTIzC8tx37uXkH46WNwYcYBTgY lXh4IwJkIoVYE8uKK3MPMUpwMCuJ8ObEAYV4UxIrq1KL8uOLSnNSiw8xmgLdNJFZSjQ5Hxh7 eSXxhiaG5paGRsYWFuZGRkrivCUfroQLCaQnlqRmp6YWpBbB9DFxcEo1ME7vfqZ3sX510Kfc KOWKwzobp67bZfTxmUumauXORTP6vqspLHzPduV14w4xReujiXdEl8utvsFhtrhv2QI3zWDW ySeXXDBtWCe3VYfh4tugv0t+FM99bnXvQWql612tFdl8Cb/ubFxjGRKxdcrEbQxG2xqyppxK O8h+hzntLMudQxy7p6TXLrdRYinOSDTUYi4qTgQAMzYSdIUCAAA= X-MTR: 20000000000000000@CPGS X-CMS-MailID: 20170517144055eucas1p2bbdf11f385f65bf07c454a15dae7f07c X-Msg-Generator: CA X-Sender-IP: 182.198.249.179 X-Local-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G+yCvOyEseyghOyekBtMZWFkaW5nIEVuZ2luZWVy?= X-Global-Sender: =?UTF-8?B?SWx5YSBNYXhpbWV0cxtTUlItVmlydHVhbGl6YXRpb24gTGFi?= =?UTF-8?B?G1NhbXN1bmcgRWxlY3Ryb25pY3MbTGVhZGluZyBFbmdpbmVlcg==?= X-Sender-Code: =?UTF-8?B?QzEwG0NJU0hRG0MxMEdEMDFHRDAxMDE1NA==?= CMS-TYPE: 201P X-HopCount: 7 X-CMS-RootMailID: 20170517144055eucas1p2bbdf11f385f65bf07c454a15dae7f07c X-RootMTR: 20170517144055eucas1p2bbdf11f385f65bf07c454a15dae7f07c References: <1495032027-23517-1-git-send-email-i.maximets@samsung.com> X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , Heetae Ahn Subject: [ovs-dev] [PATCH v3 2/3] netdev-dpdk: Fix device leak on port deletion. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Currently, once created device in dpdk will exist forever even after del-port operation untill we manually call 'ovs-appctl netdev-dpdk/detach ', where is not the port's name but the name of dpdk eth device or pci address. Few issues with current implementation: 1. Different API for usual (system) and DPDK devices. (We have to call 'ovs-appctl netdev-dpdk/detach' each time after 'del-port' to actually free the device) This is a big issue mostly for virtual DPDK devices. 2. Follows from 1: For DPDK devices 'del-port' leads just to 'rte_eth_dev_stop' and subsequent 'add-port' will just start the already existing device. Such behaviour will not reset the device to initial state as it could be expected. For example: virtual pcap pmd will continue reading input file instead of reading it from the beginning. 3. Follows from 2: After execution of the following commands 'port1' will be configured with the 'old-options' while 'ovs-vsctl show' will show us 'new-options' in dpdk-devargs field: ovs-vsctl add-port port1 -- set interface port1 type=dpdk \ options:dpdk-devargs=, ovs-vsctl del-port port1 ovs-vsctl add-port port1 -- set interface port1 type=dpdk \ options:dpdk-devargs=, 4. Follows from 1: Not detached device consumes 'port_id'. Since we have very limited number of 'port_id's (32 in common case) this may lead to quick exhausting of id pool and inability to add any other port. To avoid above issues we need to detach all the attached devices on port destruction. appctl 'netdev-dpdk/detach' removed because not needed anymore. We need to use internal 'attached' variable to track ports on which rte_eth_dev_attach() was called and returned successfully to avoid closing and detaching devices that do not support hotplug or by any other reason attached using the 'dpdk-extra' cmdline options. CC: Ciara Loftus Fixes: 55e075e65ef9 ("netdev-dpdk: Arbitrary 'dpdk' port naming") Fixes: 69876ed78611 ("netdev-dpdk: Add support for virtual DPDK PMDs (vdevs)") Signed-off-by: Ilya Maximets --- Documentation/howto/dpdk.rst | 5 ++- lib/netdev-dpdk.c | 72 ++++++++++++-------------------------------- 2 files changed, 22 insertions(+), 55 deletions(-) diff --git a/Documentation/howto/dpdk.rst b/Documentation/howto/dpdk.rst index dc63f7d..20d8975 100644 --- a/Documentation/howto/dpdk.rst +++ b/Documentation/howto/dpdk.rst @@ -342,10 +342,9 @@ Then it can be attached to OVS:: $ ovs-vsctl add-port br0 dpdkx -- set Interface dpdkx type=dpdk \ options:dpdk-devargs=0000:01:00.0 -It is also possible to detach a port from ovs, the user has to remove the -port using the del-port command, then it can be detached using:: +Detaching will be performed while processing del-port command:: - $ ovs-appctl netdev-dpdk/detach 0000:01:00.0 + $ ovs-vsctl del-port dpdkx This feature is not supported with VFIO and does not work with some NICs. For more information please refer to the `DPDK Port Hotplug Framework diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 25d9c9b..ddd3d50 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -359,6 +359,9 @@ struct netdev_dpdk { /* Device arguments for dpdk ports */ char *devargs; + /* If true, device was attached by rte_eth_dev_attach(). */ + bool attached; + /* In dpdk_list. */ struct ovs_list list_node OVS_GUARDED_BY(dpdk_mutex); @@ -852,6 +855,7 @@ common_construct(struct netdev *netdev, unsigned int port_no, dev->max_packet_len = MTU_TO_FRAME_LEN(dev->mtu); ovsrcu_index_init(&dev->vid, -1); dev->vhost_reconfigured = false; + dev->attached = false; ovsrcu_init(&dev->qos_conf, NULL); @@ -997,10 +1001,21 @@ static void netdev_dpdk_destruct(struct netdev *netdev) { struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); + char devname[RTE_ETH_NAME_MAX_LEN]; ovs_mutex_lock(&dpdk_mutex); rte_eth_dev_stop(dev->port_id); + + if (dev->attached) { + rte_eth_dev_close(dev->port_id); + if (rte_eth_dev_detach(dev->port_id, devname) < 0) { + VLOG_ERR("Device '%s' can not be detached", dev->devargs); + } else { + VLOG_INFO("Device '%s' detached", devname); + } + } + free(dev->devargs); common_destruct(dev); @@ -1112,7 +1127,8 @@ netdev_dpdk_lookup_by_port_id(int port_id) } static int -netdev_dpdk_process_devargs(const char *devargs, char **errp) +netdev_dpdk_process_devargs(struct netdev_dpdk *dev, + const char *devargs, char **errp) { /* Get the name up to the first comma. */ char *name = xmemdup0(devargs, strcspn(devargs, ",")); @@ -1124,6 +1140,7 @@ netdev_dpdk_process_devargs(const char *devargs, char **errp) /* Device not found in DPDK, attempt to attach it */ if (!rte_eth_dev_attach(devargs, &new_port_id)) { /* Attach successful */ + dev->attached = true; VLOG_INFO("Device '%s' attached to DPDK", devargs); } else { /* Attach unsuccessful */ @@ -1209,7 +1226,8 @@ netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args, * is valid */ if (!(dev->devargs && !strcmp(dev->devargs, new_devargs) && rte_eth_dev_is_valid_port(dev->port_id))) { - int new_port_id = netdev_dpdk_process_devargs(new_devargs, errp); + int new_port_id = netdev_dpdk_process_devargs(dev, new_devargs, + errp); if (!rte_eth_dev_is_valid_port(new_port_id)) { err = EINVAL; } else if (new_port_id == dev->port_id) { @@ -2432,53 +2450,6 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn, int argc, unixctl_command_reply(conn, "OK"); } -static void -netdev_dpdk_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, - const char *argv[], void *aux OVS_UNUSED) -{ - int ret; - char *response; - uint8_t port_id; - char devname[RTE_ETH_NAME_MAX_LEN]; - struct netdev_dpdk *dev; - - ovs_mutex_lock(&dpdk_mutex); - - if (!rte_eth_dev_count() || rte_eth_dev_get_port_by_name(argv[1], - &port_id)) { - response = xasprintf("Device '%s' not found in DPDK", argv[1]); - goto error; - } - - dev = netdev_dpdk_lookup_by_port_id(port_id); - if (dev) { - response = xasprintf("Device '%s' is being used by interface '%s'. " - "Remove it before detaching", - argv[1], netdev_get_name(&dev->up)); - goto error; - } - - rte_eth_dev_close(port_id); - - ret = rte_eth_dev_detach(port_id, devname); - if (ret < 0) { - response = xasprintf("Device '%s' can not be detached", argv[1]); - goto error; - } - - response = xasprintf("Device '%s' has been detached", argv[1]); - - ovs_mutex_unlock(&dpdk_mutex); - unixctl_command_reply(conn, response); - free(response); - return; - -error: - ovs_mutex_unlock(&dpdk_mutex); - unixctl_command_reply_error(conn, response); - free(response); -} - /* * Set virtqueue flags so that we do not receive interrupts. */ @@ -2754,9 +2725,6 @@ netdev_dpdk_class_init(void) unixctl_command_register("netdev-dpdk/set-admin-state", "[netdev] up|down", 1, 2, netdev_dpdk_set_admin_state, NULL); - unixctl_command_register("netdev-dpdk/detach", - "pci address of device", 1, 1, - netdev_dpdk_detach, NULL); ovsthread_once_done(&once); }