From patchwork Mon Aug 26 14:54:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1153246 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="h+vW+b+Y"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46HFSF6dWRz9s7T for ; Tue, 27 Aug 2019 00:55:57 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id C920617D6; Mon, 26 Aug 2019 14:55:55 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id E36FE17C6 for ; Mon, 26 Aug 2019 14:54:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id EE9DA710 for ; Mon, 26 Aug 2019 14:54:13 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190826145411euoutp018299eb2d37d5b968d042321cb06f837e~_gLjin0IV2590425904euoutp01h for ; Mon, 26 Aug 2019 14:54:11 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190826145411euoutp018299eb2d37d5b968d042321cb06f837e~_gLjin0IV2590425904euoutp01h DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1566831251; bh=KAKaAqlpTJtXCGiwAXmhlfOyjoZoeW0jr05ZWh5o3eE=; h=From:To:Cc:Subject:Date:References:From; b=h+vW+b+Y/zMkC7VdndBUbvCwhm+iMrWNQyCj/qZS/5iVTPkHJE526FUXPqQkh3OaN Ae4oSoF28IgwNVMgYQ2FnKL1BCED+zbsZId+87ylcW8P8jo8sI9k1xkdB0Zp+58E33 V8l+niXGU1JkNe5nJya0VtdwwfToAoIrLPTYqq64= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190826145411eucas1p1e6cc8dbc9795f7da57aff769d0c5d7ae~_gLi7onBP2780827808eucas1p1N; Mon, 26 Aug 2019 14:54:11 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id 99.89.04469.292F36D5; Mon, 26 Aug 2019 15:54:10 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190826145410eucas1p1551ae3467f536413dba12a4ff610125e~_gLhyrczU3243132431eucas1p1y; Mon, 26 Aug 2019 14:54:10 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190826145409eusmtrp25a19225b2e7db437128df1625f7fdb3e~_gLhklhMs1888218882eusmtrp2Z; Mon, 26 Aug 2019 14:54:09 +0000 (GMT) X-AuditID: cbfec7f2-569ff70000001175-a6-5d63f292e621 Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id A2.C1.04166.192F36D5; Mon, 26 Aug 2019 15:54:09 +0100 (BST) Received: from localhost.localdomain (unknown [106.109.129.180]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20190826145409eusmtip22719928c66f8a4f499145b1b8317b98e~_gLg_FK7q1136411364eusmtip2Y; Mon, 26 Aug 2019 14:54:09 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Mon, 26 Aug 2019 17:54:04 +0300 Message-Id: <20190826145404.3111-1-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCIsWRmVeSWpSXmKPExsWy7djP87qTPiXHGjRPsrbYvqKLzeLOlZ9s Flfaf7JbbHx4ltVi7aEP7BZzPz1ndGDzWLznJZPHs5v/GT3e77vK5tG3ZRVjAEsUl01Kak5m WWqRvl0CV8aFY51sBev5Kw4+k2pg/MjVxcjJISFgIvHw8Q7mLkYuDiGBFYwSV3euZgVJCAl8 YZT4Mp0fIvGZUeJCwz8mmI7FfyazQSSWM0psefIJqv0ro8SevX9ZQKrYBHQkTq0+wghiiwhI S7zufcMKUsQscIJR4vfbfrCEsIClxM1VDWwgNouAqsSGS7/AdvMKWEksvz+DHWKdvMTqDQfA NkgInGCTeHdtNlARB5DjIvHhfg5EjbDEq+NboOplJP7vnA91ar3E/ZaXjBC9HYwS0w/B/GAv seX1OXaQOcwCmhLrd+lDhB0letZsZocYzydx460gSJgZyJy0bTozRJhXoqNNCKJaReL3weXM ELaUxM13n6Eu8JDYvPIVGyQUYyXen+tlnsAoNwth1wJGxlWM4qmlxbnpqcWGeanlesWJucWl eel6yfm5mxiBcX/63/FPOxi/Xko6xCjAwajEwytxJjlWiDWxrLgy9xCjBAezkghvjn5irBBv SmJlVWpRfnxRaU5q8SFGaQ4WJXHeaoYH0UIC6YklqdmpqQWpRTBZJg5OYMzvlAh/7PL6420v EV+PmCnFbx2tV7QkhNSnfo6f2coqMOH2p1cFb6fOuj8/tqzvgvKvC8xH1ldkLV+18OPTOU/N /abuuBLWJTnj5vpTMkk3j/S917QsEKgNKX4o8iXxp6ZTr0C0BB+fwc9bmWklPus21a/st/LK 2nc0qdHo35QC7xm/WR87xqkrsRRnJBpqMRcVJwIAmMkPOvcCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrNLMWRmVeSWpSXmKPExsVy+t/xe7oTPyXHGhx5rW2xfUUXm8WdKz/Z LK60/2S32PjwLKvF2kMf2C3mfnrO6MDmsXjPSyaPZzf/M3q833eVzaNvyyrGAJYoPZui/NKS VIWM/OISW6VoQwsjPUNLCz0jE0s9Q2PzWCsjUyV9O5uU1JzMstQifbsEvYwLxzrZCtbzVxx8 JtXA+JGri5GTQ0LARGLxn8lsXYxcHEICSxklDl2aywaRkJL48esCK4QtLPHnWhdYXEjgM6PE gl/mIDabgI7EqdVHGEFsEQFpide9b1hBBjELnGGUuLbhGlhCWMBS4uaqBrBmFgFViQ2XfoEN 5RWwklh+fwY7xAJ5idUbDjBPYORZwMiwilEktbQ4Nz232FCvODG3uDQvXS85P3cTIzDgth37 uXkH46WNwYcYBTgYlXh4Jc4kxwqxJpYVV+YeYpTgYFYS4c3RT4wV4k1JrKxKLcqPLyrNSS0+ xGgKtHwis5Rocj4wGvJK4g1NDc0tLA3Njc2NzSyUxHk7BA7GCAmkJ5akZqemFqQWwfQxcXAC A0t8eeleoW6H72xX52yXY+BO1+i7Inr6plDvj8eLTy9/vvy5OlN9X+XBJJbg1eUpF7Of1Raw qu6ZvDFx3qx9P4pXNG5jFGdnPhRhsuxdvvr3CTuPCB0WXaK7QaAhTHnGM7/ywuwW632LmY98 ffj9VX/xhPJnL5RKX//5q65hL3truX7q8XX7P/gosRRnJBpqMRcVJwIA/e/Odk4CAAA= X-CMS-MailID: 20190826145410eucas1p1551ae3467f536413dba12a4ff610125e X-Msg-Generator: CA X-RootMTR: 20190826145410eucas1p1551ae3467f536413dba12a4ff610125e X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190826145410eucas1p1551ae3467f536413dba12a4ff610125e References: X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH] netdev-dpdk: Fix Tx queue false sharing. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org 'tx_q' array is allocated for each DPDK netdev. 'struct dpdk_tx_queue' is 8 bytes long, so 8 tx queues are sharing the same cache line in case of 64B cacheline size. This causes 'false sharing' issue in mutliqueue case because taking the spinlock implies write to memory i.e. cache invalidation. Signed-off-by: Ilya Maximets Acked-by: Eelco Chaudron --- I didn't really test the performance impact yet, so testing is very welcome. Relevant test case could be PVP with 8 queues in HW NIC and 8 queues in vhost-user inerface and 8 PMD threads. lib/netdev-dpdk.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index bc20d6843..adfe76a63 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -298,14 +298,17 @@ struct dpdk_mp { }; /* There should be one 'struct dpdk_tx_queue' created for - * each cpu core. */ + * each netdev tx queue. */ struct dpdk_tx_queue { - rte_spinlock_t tx_lock; /* Protects the members and the NIC queue - * from concurrent access. It is used only - * if the queue is shared among different - * pmd threads (see 'concurrent_txq'). */ - int map; /* Mapping of configured vhost-user queues - * to enabled by guest. */ + /* Padding to make dpdk_tx_queue exactly one cache line long. */ + PADDED_MEMBERS(CACHE_LINE_SIZE, + /* Protects the members and the NIC queue from concurrent access. + * It is used only if the queue is shared among different pmd threads + * (see 'concurrent_txq'). */ + rte_spinlock_t tx_lock; + /* Mapping of configured vhost-user queue to enabled by guest. */ + int map; + ); }; /* dpdk has no way to remove dpdk ring ethernet devices