From patchwork Wed Jan 15 17:29:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1223746 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ovn.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47yZ8B5GWPz9s4Y for ; Thu, 16 Jan 2020 04:29:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B2D2E87933; Wed, 15 Jan 2020 17:29:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9mbtLv5B2mYG; Wed, 15 Jan 2020 17:29:39 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 4F27F875B2; Wed, 15 Jan 2020 17:29:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3FFC6C1D83; Wed, 15 Jan 2020 17:29:39 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 58ACAC077D for ; Wed, 15 Jan 2020 17:29:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4764884076 for ; Wed, 15 Jan 2020 17:29:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mquKpqB2GsUz for ; Wed, 15 Jan 2020 17:29:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 96C5883604 for ; Wed, 15 Jan 2020 17:29:31 +0000 (UTC) X-Originating-IP: 90.177.210.238 Received: from localhost.localdomain (238.210.broadband10.iol.cz [90.177.210.238]) (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 070D71BF209; Wed, 15 Jan 2020 17:29:27 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org, Ian Stokes Date: Wed, 15 Jan 2020 18:29:19 +0100 Message-Id: <20200115172919.30551-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.17.1 Cc: Eli Britstein , Ilya Maximets Subject: [ovs-dev] [PATCH] dpif-netdev: Make datapath port mutex recursive. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Upcoming HW offloading will request flow statistics from the dpdk offloading module. This operation requires holding datapath port mutex. However, there is a possible scenarion in which flow deletion happens during datapath reconfiguration process and the mutex already acquired: 0 raise () from /lib64/libc.so.6 1 abort () from /lib64/libc.so.6 2 ovs_abort_valist () 3 ovs_abort () 4 ovs_mutex_lock_at () 5 dpif_netdev_get_flow_offload_status () 6 get_dpif_flow_status () 7 flow_del_on_pmd () 8 dpif_netdev_flow_del () 9 dpif_netdev_operate () 10 dpif_operate () 11 push_dp_ops () 12 push_ukey_ops () 13 dp_purge_cb () 14 dp_netdev_del_pmd () 15 reconfigure_pmd_threads () 16 reconfigure_datapath () 17 do_del_port () 18 dpif_netdev_port_del () 19 dpif_port_del () 20 port_del () 21 ofproto_port_del () 22 bridge_delete_or_reconfigure_ports () 23 bridge_reconfigure () 24 bridge_run () 25 main () This happens while removing the last port of a particular PMD thread. Reconfiguration process decides that we need to remove current PMD thread and calls datapath purge callback in order to clean up resources assigned to it. This turns into flow removal and flow_del() tries to request statistics. Turning the dp->mutex into recursive version as a quick fix for this issue. Better solutions might be to avoid statistics request somehow, or fully disassociate offloaded flows from the datapath flows. Signed-off-by: Ilya Maximets --- Ideas for better solution are welcome. lib/dpif-netdev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 079bd1bdf..d4b1ebbff 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -1547,7 +1547,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, ovs_refcount_init(&dp->ref_cnt); atomic_flag_clear(&dp->destroyed); - ovs_mutex_init(&dp->port_mutex); + ovs_mutex_init_recursive(&dp->port_mutex); hmap_init(&dp->ports); dp->port_seq = seq_create(); fat_rwlock_init(&dp->upcall_rwlock);