From patchwork Tue Mar 15 22:30:01 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniele Di Proietto X-Patchwork-Id: 597882 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3qPq8k49qmz9sRB for ; Wed, 16 Mar 2016 09:31:18 +1100 (AEDT) Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id DE689108F3; Tue, 15 Mar 2016 15:30:39 -0700 (PDT) X-Original-To: dev@openvswitch.org Delivered-To: dev@openvswitch.org Received: from mx3v3.cudamail.com (mx3.cudamail.com [64.34.241.5]) by archives.nicira.com (Postfix) with ESMTPS id 9A31310873 for ; Tue, 15 Mar 2016 15:30:34 -0700 (PDT) Received: from bar6.cudamail.com (localhost [127.0.0.1]) by mx3v3.cudamail.com (Postfix) with ESMTPS id 2F2D61622F7 for ; Tue, 15 Mar 2016 16:30:34 -0600 (MDT) X-ASG-Debug-ID: 1458081033-0b3237457634560001-byXFYA Received: from mx1-pf1.cudamail.com ([192.168.24.1]) by bar6.cudamail.com with ESMTP id ojYf0YD46Dj7UoRx (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 15 Mar 2016 16:30:33 -0600 (MDT) X-Barracuda-Envelope-From: diproiettod@vmware.com X-Barracuda-RBL-Trusted-Forwarder: 192.168.24.1 Received: from unknown (HELO smtp-outbound-1.vmware.com) (208.91.2.12) by mx1-pf1.cudamail.com with ESMTPS (DHE-RSA-AES256-SHA encrypted); 15 Mar 2016 22:30:33 -0000 Received-SPF: error (mx1-pf1.cudamail.com: error in processing during lookup of vmware.com: DNS problem) X-Barracuda-Apparent-Source-IP: 208.91.2.12 X-Barracuda-RBL-IP: 208.91.2.12 Received: from sc9-mailhost1.vmware.com (sc9-mailhost1.vmware.com [10.113.161.71]) by smtp-outbound-1.vmware.com (Postfix) with ESMTP id 42F0329159; Tue, 15 Mar 2016 15:30:31 -0700 (PDT) Received: from sc9-mailhost2.vmware.com (unknown [10.33.74.30]) by sc9-mailhost1.vmware.com (Postfix) with ESMTP id 27F5518CDA; Tue, 15 Mar 2016 15:30:32 -0700 (PDT) X-CudaMail-Envelope-Sender: diproiettod@vmware.com From: Daniele Di Proietto To: dev@openvswitch.org X-CudaMail-Whitelist-To: dev@openvswitch.org X-CudaMail-MID: CM-E1-314096891 X-CudaMail-DTE: 031516 X-CudaMail-Originating-IP: 208.91.2.12 Date: Tue, 15 Mar 2016 15:30:01 -0700 X-ASG-Orig-Subj: [##CM-E1-314096891##][PATCH v3 09/11] dpif-netdev: Fix reconfigure_pmd_threads(). Message-Id: <1458081003-82542-10-git-send-email-diproiettod@vmware.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1458081003-82542-1-git-send-email-diproiettod@vmware.com> References: <1458081003-82542-1-git-send-email-diproiettod@vmware.com> X-Barracuda-Connect: UNKNOWN[192.168.24.1] X-Barracuda-Start-Time: 1458081033 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://web.cudamail.com:443/cgi-mod/mark.cgi X-ASG-Whitelist: Header =?UTF-8?B?eFwtY3VkYW1haWxcLXdoaXRlbGlzdFwtdG8=?= X-Virus-Scanned: by bsmtpd at cudamail.com X-Barracuda-BRTS-Status: 1 X-ASG-Whitelist: EmailCat (corporate) Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3 09/11] dpif-netdev: Fix reconfigure_pmd_threads(). X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" This commit change reconfigure_pmd_threads() to interact with the ports cmap using RCU semantics (the content of the port structure is not altered while concurrent readers might access it) and to fail more gracefully in case of a set_multiq fail (now we remove the port from the datapath, instead of returning prematurely from the function without restarting the pmd threads). Signed-off-by: Daniele Di Proietto --- lib/dpif-netdev.c | 78 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 54 insertions(+), 24 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 597e1d7..9fbf821 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2591,6 +2591,11 @@ static void reconfigure_pmd_threads(struct dp_netdev *dp) { struct dp_netdev_port *port; + struct hmapx to_reconfigure = HMAPX_INITIALIZER(&to_reconfigure); + struct hmapx_node *node; + bool failed_config = false; + + ovs_mutex_lock(&dp->port_mutex); dp_netdev_destroy_all_pmds(dp); @@ -2599,33 +2604,58 @@ reconfigure_pmd_threads(struct dp_netdev *dp) int requested_n_rxq = netdev_requested_n_rxq(netdev); if (netdev_is_pmd(port->netdev) && port->latest_requested_n_rxq != requested_n_rxq) { - int i, err; + cmap_remove(&dp->ports, &port->node, hash_odp_port(port->port_no)); + hmapx_add(&to_reconfigure, port); + } + } + ovs_mutex_unlock(&dp->port_mutex); - /* Closes the existing 'rxq's. */ - for (i = 0; i < port->n_rxq; i++) { - netdev_rxq_close(port->rxq[i]); - port->rxq[i] = NULL; - } - port->n_rxq = 0; - - /* Sets the new rx queue config. */ - err = netdev_set_multiq(port->netdev, ovs_numa_get_n_cores() + 1, - requested_n_rxq); - if (err && (err != EOPNOTSUPP)) { - VLOG_ERR("Failed to set dpdk interface %s rx_queue to: %u", - netdev_get_name(port->netdev), - requested_n_rxq); - return; - } - port->latest_requested_n_rxq = requested_n_rxq; - /* If the set_multiq() above succeeds, reopens the 'rxq's. */ - port->n_rxq = netdev_n_rxq(port->netdev); - port->rxq = xrealloc(port->rxq, sizeof *port->rxq * port->n_rxq); - for (i = 0; i < port->n_rxq; i++) { - netdev_rxq_open(port->netdev, &port->rxq[i], i); - } + /* Waits for the other threads to see the ports removed from the cmap, + * otherwise we are not allowed to alter them. */ + ovsrcu_synchronize(); + + ovs_mutex_lock(&dp->port_mutex); + HMAPX_FOR_EACH (node, &to_reconfigure) { + int requested_n_rxq, i, err; + + port = node->data; + requested_n_rxq = netdev_requested_n_rxq(port->netdev); + /* Closes the existing 'rxq's. */ + for (i = 0; i < port->n_rxq; i++) { + netdev_rxq_close(port->rxq[i]); + port->rxq[i] = NULL; + } + port->n_rxq = 0; + + /* Sets the new rx queue config. */ + err = netdev_set_multiq(port->netdev, ovs_numa_get_n_cores() + 1, + requested_n_rxq); + if (err && (err != EOPNOTSUPP)) { + VLOG_ERR("Failed to set dpdk interface %s rx_queue to: %u", + netdev_get_name(port->netdev), + requested_n_rxq); + do_destroy_port(port); + failed_config = true; + continue; + } + port->latest_requested_n_rxq = requested_n_rxq; + /* If the netdev_reconfigure() above succeeds, reopens the 'rxq's and + * inserts the port back in the cmap, to allow transmitting packets. */ + port->n_rxq = netdev_n_rxq(port->netdev); + port->rxq = xrealloc(port->rxq, sizeof *port->rxq * port->n_rxq); + for (i = 0; i < port->n_rxq; i++) { + netdev_rxq_open(port->netdev, &port->rxq[i], i); } + cmap_insert(&dp->ports, &port->node, hash_port_no(port->port_no)); } + ovs_mutex_unlock(&dp->port_mutex); + + hmapx_destroy(&to_reconfigure); + + if (failed_config) { + seq_change(dp->port_seq); + } + /* Reconfigures the cpu mask. */ ovs_numa_set_cpu_mask(dp->requested_pmd_cmask); free(dp->pmd_cmask);