From patchwork Thu May 26 21:28:41 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Mauricio_V=C3=A1squez?= X-Patchwork-Id: 626897 X-Patchwork-Delegate: diproiettod@vmware.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3rG2Mr0b8wz9t5b for ; Fri, 27 May 2016 07:29:12 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=studenti.polito.it header.i=@studenti.polito.it header.b=kcYoyWAI; dkim-atps=neutral Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 63664109C0; Thu, 26 May 2016 14:29:10 -0700 (PDT) X-Original-To: dev@openvswitch.org Delivered-To: dev@openvswitch.org Received: from mx1e4.cudamail.com (mx1.cudamail.com [69.90.118.67]) by archives.nicira.com (Postfix) with ESMTPS id A4D771099E for ; Thu, 26 May 2016 14:29:09 -0700 (PDT) Received: from bar5.cudamail.com (unknown [192.168.21.12]) by mx1e4.cudamail.com (Postfix) with ESMTPS id 1517F1E02B4 for ; Thu, 26 May 2016 15:29:09 -0600 (MDT) X-ASG-Debug-ID: 1464298145-09eadd582268c40001-byXFYA Received: from mx3-pf2.cudamail.com ([192.168.14.1]) by bar5.cudamail.com with ESMTP id 6CI3BdsalskjC2Qq (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 26 May 2016 15:29:05 -0600 (MDT) X-Barracuda-Envelope-From: mauricio.vasquezbernal@studenti.polito.it X-Barracuda-RBL-Trusted-Forwarder: 192.168.14.1 Received: from unknown (HELO compass.polito.it) (130.192.55.110) by mx3-pf2.cudamail.com with ESMTPS (DHE-RSA-AES256-SHA encrypted); 26 May 2016 21:29:04 -0000 Received-SPF: pass (mx3-pf2.cudamail.com: SPF record at studenti.polito.it designates 130.192.55.110 as permitted sender) X-Barracuda-Apparent-Source-IP: 130.192.55.110 X-Barracuda-RBL-IP: 130.192.55.110 Received: from localhost (localhost [127.0.0.1]) by compass.polito.it (Postfix) with ESMTP id D2248100130; Thu, 26 May 2016 23:29:01 +0200 (CEST) Authentication-Results: compass.polito.it (amavisd-new); dkim=pass (1024-bit key) reason="pass (just generated, assumed good)" header.d=studenti.polito.it DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= studenti.polito.it; h=x-mailer:message-id:date:date:subject :subject:cc:to:from:from:received:received; s=y2k10; t= 1464298139; bh=eam8UbudO82mMAlmBJYkonSoG152G2e5QfmmTDx5cjY=; b=k cYoyWAI5WzJo9q9H2vP1sLYY76JYTbypvDinvqByTmNRvPi8jXPyEqSgN4EMwBw6 FmTyE4vEfDZAFw1IjmFA6L0wcU+Ni1IV9YP/6jnzKqRmih2HJWEM/CDiQjTaH8k7 94A7W4+MUZCJ2wWMM2YV1EbAGfzUEOVZQAwhsl8hy8= X-Virus-Scanned: amavisd-new at studenti.polito.it X-Spam-Flag: NO X-Spam-Score: -5.863 X-Spam-Level: X-Spam-Status: No, score=-5.863 tagged_above=-100 required=3.5 tests=[ALL_TRUSTED=-5, AWL=0.637, BAYES_00=-1.5] autolearn=ham Received: from compass.polito.it ([127.0.0.1]) by localhost (compass.polito.it [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7ghYRzR4S8Ep; Thu, 26 May 2016 23:28:59 +0200 (CEST) Received: from mvb-X455LJ.lan (unknown [93.56.97.147]) (using TLSv1.2 with cipher AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: s203403@studenti.polito.it) by compass.polito.it (Postfix) with ESMTPSA id A34F510012B; Thu, 26 May 2016 23:28:51 +0200 (CEST) X-CudaMail-Envelope-Sender: mauricio.vasquezbernal@studenti.polito.it From: Mauricio Vasquez B To: dev@openvswitch.org X-CudaMail-MID: CM-V2-525054963 X-CudaMail-DTE: 052616 X-CudaMail-Originating-IP: 130.192.55.110 Date: Thu, 26 May 2016 23:28:41 +0200 X-ASG-Orig-Subj: [##CM-V2-525054963##][PATCH v6] netdev-dpdk: add hotplug support Message-Id: <1464298121-4959-1-git-send-email-mauricio.vasquezbernal@studenti.polito.it> X-Mailer: git-send-email 1.9.1 X-GBUdb-Analysis: 0, 130.192.55.110, Ugly c=0 p=0 Source New X-MessageSniffer-Rules: 0-0-0-14162-c X-Barracuda-Connect: UNKNOWN[192.168.14.1] X-Barracuda-Start-Time: 1464298145 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://web.cudamail.com:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at cudamail.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 1.85 X-Barracuda-Spam-Status: No, SCORE=1.85 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=4.0 tests=BSF_RULE_7580B, BSF_RULE_7582B, BSF_SC5_MJ1963, DKIM_SIGNED, RDNS_NONE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.29928 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 DKIM_SIGNED Domain Keys Identified Mail: message has a signature 0.50 BSF_RULE_7582B Custom Rule 7582B 0.10 RDNS_NONE Delivered to trusted network by a host with no rDNS 0.75 BSF_RULE_7580B Custom Rule 7580B 0.50 BSF_SC5_MJ1963 Custom Rule MJ1963 Cc: fbl@sysclose.org, kevin.traynor@intel.com Subject: [ovs-dev] [PATCH v6] netdev-dpdk: add hotplug support X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" In order to use dpdk ports in ovs they have to be bound to a DPDK compatible driver before ovs is started. This patch adds the possibility to hotplug (or hot-unplug) a device after ovs has been started. The implementation adds two appctl commands: netdev-dpdk/port-attach and netdev-dpdk/port-detach After the user attaches a new device, it has to be added to a bridge using the add-port command, similarly, before detaching a device, it has to be removed using the del-port command. Signed-off-by: Mauricio Vasquez B Tested-by: ian.stokes@intel.com Acked-by: Flavio Leitner --- v6: - add explicit comment about supporting VFIO - rebase to master v5: - use two appctl commands instead of a single one - rebase to master v4: - fix typo in commit message - remove unnecessary whitespace change in INSTALL.DPDK.md v3: - create dpdk_port_attach and dpdk_port_detach functions - modify mutex locking order v2: - use rte_eth_dev_is_valid_port() to check if a port is valid INSTALL.DPDK.md | 25 ++++++++++++++ NEWS | 1 + lib/netdev-dpdk.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 120 insertions(+), 7 deletions(-) diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 68735cc..948957b 100644 --- a/INSTALL.DPDK.md +++ b/INSTALL.DPDK.md @@ -286,6 +286,30 @@ Using the DPDK with ovs-vswitchd: For more details regarding ingress-policer see the vswitch.xml. +10. Port Hotplug + + Ovs supports port hotplugging, it allows to use ports that were not bound + to DPDK when vswitchd was started. + In order to attach a port, it has to be bound to DPDK using the + dpdk_nic_bind.py script: + + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0` + + Then it can be attached to OVS: + + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0` + + At this point, the user can create a ovs port using the add-port command. + + It is also possible to detach a port from ovs, the user has to remove the + port using the del-port command, then it can be detached using: + + `ovs-appctl netdev-dpdk/port-detach dpdk0` + + This feature is not supported with VFIO and could not work with some NICs, + please refer to the [DPDK Port Hotplug Framework] in order to get more + information. + Performance Tuning: ------------------- @@ -1018,3 +1042,4 @@ Please report problems to bugs@openvswitch.org. [INSTALL.md]:INSTALL.md [DPDK Linux GSG]: http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules [DPDK Docs]: http://dpdk.org/doc +[DPDK Port Hotplug Framework]: http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html diff --git a/NEWS b/NEWS index ba201cf..4018836 100644 --- a/NEWS +++ b/NEWS @@ -33,6 +33,7 @@ Post-v2.5.0 arguments. Additional arguments can be passed via the dpdk-extra entry. * Add ingress policing functionality. + * Port Hotplug is now supported. - ovs-benchmark: This utility has been removed due to lack of use and bitrot. - ovs-appctl: diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 6cae930..f5a8678 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -649,7 +649,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev) OVS_REQUIRES(dpdk_mutex) int diag; int n_rxq, n_txq; - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) { + if (!rte_eth_dev_is_valid_port(dev->port_id)) { return ENODEV; } @@ -2231,6 +2231,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn *conn, int argc, unixctl_command_reply(conn, "OK"); } +static void +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *aux OVS_UNUSED) +{ + int ret; + char response[128]; + uint8_t port_id; + + ovs_mutex_lock(&dpdk_mutex); + + ret = rte_eth_dev_attach(argv[1], &port_id); + if (ret < 0) { + snprintf(response, sizeof(response), + "Error attaching device '%s'", argv[1]); + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply_error(conn, response); + return; + } + + snprintf(response, sizeof(response), + "Device '%s' has been attached as 'dpdk%d'", argv[1], port_id); + + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply(conn, response); +} + +static void +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *aux OVS_UNUSED) +{ + int ret; + char response[128]; + unsigned int parsed_port; + uint8_t port_id; + char devname[RTE_ETH_NAME_MAX_LEN]; + + ovs_mutex_lock(&dpdk_mutex); + + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port); + if (ret) { + snprintf(response, sizeof(response), + "'%s' is not a valid port", argv[1]); + goto error; + } + + port_id = parsed_port; + + struct netdev *netdev = netdev_from_name(argv[1]); + if (netdev) { + netdev_close(netdev); + snprintf(response, sizeof(response), + "Port '%s' is being used. Remove it before detaching", + argv[1]); + goto error; + } + + rte_eth_dev_close(port_id); + + ret = rte_eth_dev_detach(port_id, devname); + if (ret < 0) { + snprintf(response, sizeof(response), + "Port '%s' can not be detached", argv[1]); + goto error; + } + + snprintf(response, sizeof(response), + "Port '%s' has been detached", argv[1]); + + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply(conn, response); + return; + +error: + ovs_mutex_unlock(&dpdk_mutex); + unixctl_command_reply_error(conn, response); +} + /* * Set virtqueue flags so that we do not receive interrupts. */ @@ -2523,6 +2600,15 @@ dpdk_common_init(void) "[netdev] up|down", 1, 2, netdev_dpdk_set_admin_state, NULL); + unixctl_command_register("netdev-dpdk/port-attach", + "pci address of device", 1, 1, + netdev_dpdk_port_attach, NULL); + + unixctl_command_register("netdev-dpdk/port-detach", + "port", 1, 1, + netdev_dpdk_port_detach, NULL); + + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL); } /* Client Rings */ @@ -2533,7 +2619,7 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, { struct dpdk_ring *ivshmem; char ring_name[RTE_RING_NAMESIZE]; - int err; + int err, port_id; ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem); if (ivshmem == NULL) { @@ -2567,19 +2653,20 @@ dpdk_ring_create(const char dev_name[], unsigned int port_no, return ENOMEM; } - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, - &ivshmem->cring_tx, 1, SOCKET0); + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1, + &ivshmem->cring_tx, 1, SOCKET0); - if (err < 0) { + if (port_id < 0) { rte_free(ivshmem); return ENODEV; } ivshmem->user_port_id = port_no; - ivshmem->eth_port_id = rte_eth_dev_count() - 1; + ivshmem->eth_port_id = port_id; + *eth_port_id = port_id; + ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node); - *eth_port_id = ivshmem->eth_port_id; return 0; }