From patchwork Tue Dec 22 14:26:46 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 560082 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (li376-54.members.linode.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 2F8F1140BC1 for ; Wed, 23 Dec 2015 01:27:06 +1100 (AEDT) Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 5207A10928; Tue, 22 Dec 2015 06:27:05 -0800 (PST) X-Original-To: dev@openvswitch.org Delivered-To: dev@openvswitch.org Received: from mx3v3.cudamail.com (mx3.cudamail.com [64.34.241.5]) by archives.nicira.com (Postfix) with ESMTPS id 545CD10915 for ; Tue, 22 Dec 2015 06:27:04 -0800 (PST) Received: from bar4.cudamail.com (localhost [127.0.0.1]) by mx3v3.cudamail.com (Postfix) with ESMTPS id 901E81614B1 for ; Tue, 22 Dec 2015 07:27:03 -0700 (MST) X-ASG-Debug-ID: 1450794423-03dc213fef4d8700001-byXFYA Received: from mx3-pf3.cudamail.com ([192.168.14.3]) by bar4.cudamail.com with ESMTP id fmevc20wHPMFuk2i (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 22 Dec 2015 07:27:03 -0700 (MST) X-Barracuda-Envelope-From: i.maximets@samsung.com X-Barracuda-RBL-Trusted-Forwarder: 192.168.14.3 Received: from unknown (HELO mailout2.w1.samsung.com) (210.118.77.12) by mx3-pf3.cudamail.com with ESMTPS (DHE-RSA-AES128-SHA encrypted); 22 Dec 2015 14:42:55 -0000 Received-SPF: none (mx3-pf3.cudamail.com: domain at samsung.com does not designate permitted sender hosts) X-Barracuda-Apparent-Source-IP: 210.118.77.12 X-Barracuda-RBL-IP: 210.118.77.12 Received: from eucpsbgm1.samsung.com (unknown [203.254.199.244]) by mailout2.w1.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0NZR00DYZK4ZMJ80@mailout2.w1.samsung.com> for dev@openvswitch.org; Tue, 22 Dec 2015 14:26:59 +0000 (GMT) X-AuditID: cbfec7f4-f79026d00000418a-e1-56795db306c8 Received: from eusync2.samsung.com ( [203.254.199.212]) by eucpsbgm1.samsung.com (EUCPMTA) with SMTP id A7.C2.16778.3BD59765; Tue, 22 Dec 2015 14:26:59 +0000 (GMT) Received: from imaximets.rnd.samsung.ru ([106.109.129.180]) by eusync2.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTPA id <0NZR000NYK4QZC20@eusync2.samsung.com>; Tue, 22 Dec 2015 14:26:59 +0000 (GMT) X-CudaMail-Envelope-Sender: i.maximets@samsung.com From: Ilya Maximets To: dev@openvswitch.org, Ben Pfaff , Daniele Di Proietto X-CudaMail-MID: CM-V3-1221012988 X-CudaMail-DTE: 122215 X-CudaMail-Originating-IP: 210.118.77.12 Date: Tue, 22 Dec 2015 17:26:46 +0300 X-ASG-Orig-Subj: [##CM-V3-1221012988##][PATCH 0/2] Per numa node barriers for pmd threads Message-id: <1450794408-17970-1-git-send-email-i.maximets@samsung.com> X-Mailer: git-send-email 2.1.4 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDJMWRmVeSWpSXmKPExsVy+t/xK7qbYyvDDB7uUbB4NbmB0eLo6T3M Fi+777FbXGn/yW4xebaUA6vHs5v/GT2eX+th8ejbsorR4938t2wBLFFcNimpOZllqUX6dglc Gb+f/WQpWCFRMWfODJYGxlPCXYycHBICJhI9C16zQ9hiEhfurWfrYuTiEBJYyijxZuImZgin lUni+oIWJpAqNgEdiVOrjzCC2CICCRLntrwEs5kFAiS2/OtnAbGFBawlVkxrZO1i5OBgEVCV 2PpaACTMK+AmcXbBJjaIZXIS54//ZJ7AyL2AkWEVo2hqaXJBcVJ6rqFecWJucWleul5yfu4m RkgYfNnBuPiY1SFGAQ5GJR7eCQsqwoRYE8uKK3MPMUpwMCuJ8AYEVYYJ8aYkVlalFuXHF5Xm pBYfYpTmYFES5527632IkEB6YklqdmpqQWoRTJaJg1OqgdGSz6VekEHUwMOqV3mmcIT72jnv sx7/Xdh4cfk/v0Yn+1W+H3lFevUOzdkh9W6BUM3X9uo1a9u2f5Hb7zXnP/sh44+HJOYeevnh n9f72SsdlecvmZ+UOT9oS/Q+VqNfnSlbZsXm/dj1e130znCfK0FlE54s1lhW5vtM0tvikVBa 6sUlqplvLM4qsRRnJBpqMRcVJwIAA+kET/8BAAA= X-GBUdb-Analysis: 0, 210.118.77.12, Ugly c=0 p=0 Source New X-MessageSniffer-Rules: 0-0-0-8188-c X-Barracuda-Connect: UNKNOWN[192.168.14.3] X-Barracuda-Start-Time: 1450794423 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://web.cudamail.com:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at cudamail.com X-Barracuda-BRTS-Status: 1 X-ASG-Whitelist: EmailCat (corporate) Cc: Ilya Maximets , Dyasly Sergey Subject: [ovs-dev] [PATCH 0/2] Per numa node barriers for pmd threads X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" While reloading, two PMD threads, one already reloaded and one not yet reloaded, can poll same queue of the same port. This may be easily reproduced using that patch: --- How To: <-- ovs-vsctl add-port ovs_br0 dpdk0 -- set Interface dpdk0 type=dpdk --> <-- ovs-vsctl add-port ovs_br0 dpdk1 -- set Interface dpdk1 type=dpdk --> 49:58Z|00054|dpif_netdev(pmd99) |INFO|Core 0 processing port 'dpdk0' 49:58Z|00054|dpif_netdev(pmd100)|INFO|Core 1 processing port 'dpdk1' 49:58Z|00075|dpif_netdev|INFO|Created 2 pmd threads on numa node 0 <-- ovs-vsctl del-port ovs_br0 dpdk0 --> 50:20Z|00083|dpif_netdev(pmd99)|INFO|Core 0 processing port 'dpdk1' ovs-vswitchd(pmd100): lib/dpif-netdev.c:2571: assertion ovs_refcount_unref_relaxed(&rxq->ref_cnt) == 2 failed in dp_netdev_process_rxq_port() Program received signal SIGABRT, Aborted. Fix that by introducing per numa node barriers prior to polling. Ilya Maximets (2): poll: Suppress logging for pmd threads. dpif-netdev: Per numa node barriers for pmd threads. lib/dpif-netdev.c | 53 ++++++++++++++++++++++++++++++++++++++++------------- lib/ovs-thread.c | 9 +++++++++ lib/ovs-thread.h | 1 + lib/poll-loop.c | 4 +++- lib/timeval.c | 2 +- 5 files changed, 54 insertions(+), 15 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index ad4a665..8b744d9 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -48,6 +48,7 @@ #include "match.h" #include "netdev.h" #include "netdev-dpdk.h" +#include "netdev-provider.h" #include "netdev-vport.h" #include "netlink.h" #include "odp-execute.h" @@ -2524,7 +2524,9 @@ dp_netdev_process_rxq_port(struct dp_netdev_pmd_thread *pmd, int error, cnt; cycles_count_start(pmd); + ovs_refcount_ref(&rxq->ref_cnt); error = netdev_rxq_recv(rxq, packets, &cnt); + ovs_assert(ovs_refcount_unref_relaxed(&rxq->ref_cnt) == 2); cycles_count_end(pmd, PMD_CYCLES_POLLING); if (!error) { int i; diff --git a/lib/netdev-provider.h b/lib/netdev-provider.h index a33bb3b..d377066 100644 --- a/lib/netdev-provider.h +++ b/lib/netdev-provider.h @@ -22,6 +22,7 @@ #include "connectivity.h" #include "netdev.h" #include "list.h" +#include "ovs-atomic.h" #include "ovs-numa.h" #include "packets.h" #include "seq.h" @@ -88,6 +89,7 @@ struct netdev **netdev_get_vports(size_t *size); struct netdev_rxq { struct netdev *netdev; /* Owns a reference to the netdev. */ int queue_id; + struct ovs_refcount ref_cnt; }; struct netdev *netdev_rxq_get_netdev(const struct netdev_rxq *); diff --git a/lib/netdev.c b/lib/netdev.c index e3b70b1..b01c04c 100644 --- a/lib/netdev.c +++ b/lib/netdev.c @@ -601,6 +601,7 @@ netdev_rxq_open(struct netdev *netdev, struct netdev_rxq **rxp, int id) if (rx) { rx->netdev = netdev; rx->queue_id = id; + ovs_refcount_init(&rx->ref_cnt); error = netdev->netdev_class->rxq_construct(rx); if (!error) { netdev_ref(netdev);