From patchwork Mon Sep 2 11:27:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1156592 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="DP8rdTSR"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46MSWD1qr2z9s7T for ; Mon, 2 Sep 2019 21:28:08 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id A651CDA7; Mon, 2 Sep 2019 11:27:25 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id D77A5D9C for ; Mon, 2 Sep 2019 11:27:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 553B77DB for ; Mon, 2 Sep 2019 11:27:24 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190902112722euoutp01f3401ad5d4217e1fb4787eab907936d9~Am3_p4aL82557125571euoutp01v for ; Mon, 2 Sep 2019 11:27:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190902112722euoutp01f3401ad5d4217e1fb4787eab907936d9~Am3_p4aL82557125571euoutp01v DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1567423642; bh=QYIe/k0ftndR1Tr3EFfW2tK1faE8EjrX3kcGZDthFpQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DP8rdTSRz6pm3HS3oTIDJO8G0pn2B4qsyzLaKoABZ65aXi/iwfIzO8Q5tJtYtOK6P h9FNzlxdqw0SlrZupL+goeWVK21/sCNn5ZWZvoyRBepxrQc4/+TUXvQq9tV12Pxjz6 sumdBTPxzTtfaxybQliZVyaluscU4dp/+zI34lcU= Received: from eusmges3new.samsung.com (unknown [203.254.199.245]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190902112722eucas1p1373ecedad2b9f600ebbff0a18368f5cd~Am3_C6TnD1979819798eucas1p1t; Mon, 2 Sep 2019 11:27:22 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges3new.samsung.com (EUCPMTA) with SMTP id 95.76.04374.99CFC6D5; Mon, 2 Sep 2019 12:27:22 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190902112721eucas1p1d554e13c4d7821dfd851400869cb7518~Am39I6yI61512315123eucas1p1f; Mon, 2 Sep 2019 11:27:21 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190902112721eusmtrp2fd2f0cab291165f0a390d12f3fa5a74f~Am3865S5-2554125541eusmtrp2V; Mon, 2 Sep 2019 11:27:21 +0000 (GMT) X-AuditID: cbfec7f5-4ddff70000001116-60-5d6cfc99fede Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 4C.F9.04166.99CFC6D5; Mon, 2 Sep 2019 12:27:21 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20190902112720eusmtip1e97a48b6d67ad77580a53bf429fd1d93~Am38Zl4WB2536925369eusmtip1F; Mon, 2 Sep 2019 11:27:20 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Mon, 2 Sep 2019 14:27:09 +0300 Message-Id: <20190902112711.2919-2-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190902112711.2919-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrGIsWRmVeSWpSXmKPExsWy7djPc7qz/uTEGnRNYrTYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUVw2 Kak5mWWpRfp2CVwZp16tYS+YJFux8fAPlgbGdWJdjJwcEgImEs3Xn7F2MXJxCAmsYJS49HUb O4TzhVHidu9RNgjnM6PE5PVHGGFalp5+AVW1nFFi8/lrbCAJIYEfjBJX7piC2GwCOhKnVkM0 iAhIS7zufQO2g1ngMKPErt6HYA3CAt4S63+tAprEwcEioCqx4Rc3SJhXwEri5N6pzBDL5CVW bzgAZnMKWEt0bFwIdpGEwGs2iWNH7rFAFLlInLx/HMoWlnh1fAs7hC0j8X/nfCYIu17ifstL RojmDkaJ6Yf+QSXsJba8Pgd2BLOApsT6XfoQYUeJX/vnsoCEJQT4JG68FQQJMwOZk7ZNZ4YI 80p0tAlBVKtI/D64HOpkKYmb7z5DXeAhcf1wIzSs+hgldqyewDaBUX4WwrIFjIyrGMVTS4tz 01OLjfNSy/WKE3OLS/PS9ZLzczcxAlPD6X/Hv+5g3Pcn6RCjAAejEg8vx6fsWCHWxLLiytxD jBIczEoivKF7cmKFeFMSK6tSi/Lji0pzUosPMUpzsCiJ81YzPIgWEkhPLEnNTk0tSC2CyTJx cEo1MJrc9OR6ddUwd/PVeSV+d0J6voUUFGkcnLguRPLtzd0L3YLPTX63SVy1OuHu6vNGk2M4 GCouGuc53AhyOX330ATOpt+56/wtXsuLrjEU4W1W8Y2y2RMXqFIY9qWNfaZG27+uvS3BsuJB rVGr+W24kiXPVIrGry6LuB8t2yL8+POj/U2BvDOblFiKMxINtZiLihMBlcvnJgkDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsVy+t/xu7oz/+TEGsx6xGyxfUUXm8WV9p/s FhsfnmW1WHvoA7vF3E/PGS2uX+JxYPPYOesuu8fiPS+ZPJ7d/M/o8X7fVTaPvi2rGANYo/Rs ivJLS1IVMvKLS2yVog0tjPQMLS30jEws9QyNzWOtjEyV9O1sUlJzMstSi/TtEvQyTr1aw14w SbZi4+EfLA2M68S6GDk5JARMJJaefsHexcjFISSwlFHi/KfbrBAJKYkfvy5A2cISf651sUEU fWOUOHLjJliCTUBH4tTqI4wgtoiAtMTr3jesIEXMAscZJVZ17mACSQgLeEus/7UKaAUHB4uA qsSGX9wgYV4BK4mTe6cyQyyQl1i94QCYzSlgLdGxcSEbiC0EVDN98RfGCYx8CxgZVjGKpJYW 56bnFhvqFSfmFpfmpesl5+duYgSG6rZjPzfvYLy0MfgQowAHoxIPL8en7Fgh1sSy4srcQ4wS HMxKIryhe3JihXhTEiurUovy44tKc1KLDzGaAt00kVlKNDkfGEd5JfGGpobmFpaG5sbmxmYW SuK8HQIHY4QE0hNLUrNTUwtSi2D6mDg4pRoYZ16z+X7WOruc5/+1HdI6Elrxjru7vv2NN/7B etJ6XluP+7E//zzDHDeLhcRV6PSmR7PGPn6yp5dJ91T8ba9qlxVX5J/1H9/lLXDqzKLGqOiu lJQTC6JKrc+ZcGi92F1/t4bdc4O86VWmFGvtRqnOPralSzlexR+fdZmfZcUdSdvpQW9VLqUo sRRnJBpqMRcVJwIAclKMymsCAAA= X-CMS-MailID: 20190902112721eucas1p1d554e13c4d7821dfd851400869cb7518 X-Msg-Generator: CA X-RootMTR: 20190902112721eucas1p1d554e13c4d7821dfd851400869cb7518 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190902112721eucas1p1d554e13c4d7821dfd851400869cb7518 References: <20190902112711.2919-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH v2 1/3] ovs-numa: Add dump based thread affinity functions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org New functions to get and set CPU affinity using CPU dumps. This will abstract OS specific implementation details from the cross-platform code. Signed-off-by: Ilya Maximets Reviewed-by: David Marchand --- lib/ovs-numa.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++---- lib/ovs-numa.h | 2 ++ 2 files changed, 69 insertions(+), 5 deletions(-) diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 24edeab2a..eb608edc4 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -532,22 +532,69 @@ ovs_numa_dump_destroy(struct ovs_numa_dump *dump) free(dump); } -int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED) +struct ovs_numa_dump * +ovs_numa_thread_getaffinity_dump(void) { if (dummy_numa) { - /* Nothing to do */ + /* Nothing to do. */ + return NULL; + } + +#ifndef __linux__ + return NULL; +#else + struct ovs_numa_dump *dump; + const struct numa_node *n; + cpu_set_t cpuset; + int err; + + CPU_ZERO(&cpuset); + err = pthread_getaffinity_np(pthread_self(), sizeof cpuset, &cpuset); + if (err) { + VLOG_ERR("Thread getaffinity error: %s", ovs_strerror(err)); + return NULL; + } + + dump = ovs_numa_dump_create(); + + HMAP_FOR_EACH (n, hmap_node, &all_numa_nodes) { + const struct cpu_core *core; + + LIST_FOR_EACH (core, list_node, &n->cores) { + if (CPU_ISSET(core->core_id, &cpuset)) { + ovs_numa_dump_add(dump, core->numa->numa_id, core->core_id); + } + } + } + + if (!ovs_numa_dump_count(dump)) { + ovs_numa_dump_destroy(dump); + return NULL; + } + return dump; +#endif /* __linux__ */ +} + +int +ovs_numa_thread_setaffinity_dump(struct ovs_numa_dump *dump) +{ + if (!dump || dummy_numa) { + /* Nothing to do. */ return 0; } #ifdef __linux__ + struct ovs_numa_info_core *core; cpu_set_t cpuset; int err; CPU_ZERO(&cpuset); - CPU_SET(core_id, &cpuset); - err = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset); + FOR_EACH_CORE_ON_DUMP (core, dump) { + CPU_SET(core->core_id, &cpuset); + } + err = pthread_setaffinity_np(pthread_self(), sizeof cpuset, &cpuset); if (err) { - VLOG_ERR("Thread affinity error %d",err); + VLOG_ERR("Thread setaffinity error: %s", ovs_strerror(err)); return err; } @@ -556,3 +603,18 @@ int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED) return EOPNOTSUPP; #endif /* __linux__ */ } + +int ovs_numa_thread_setaffinity_core(unsigned core_id) +{ + struct ovs_numa_dump *affinity = ovs_numa_dump_create(); + struct cpu_core *core = get_core_by_core_id(core_id); + int ret = EINVAL; + + if (core) { + ovs_numa_dump_add(affinity, core->numa->numa_id, core->core_id); + ret = ovs_numa_thread_setaffinity_dump(affinity); + } + + ovs_numa_dump_destroy(affinity); + return ret; +} diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index 088fcb8c3..88352a93e 100644 --- a/lib/ovs-numa.h +++ b/lib/ovs-numa.h @@ -60,6 +60,8 @@ struct ovs_numa_dump *ovs_numa_dump_n_cores_per_numa(int n); bool ovs_numa_dump_contains_core(const struct ovs_numa_dump *, int numa_id, unsigned core_id); size_t ovs_numa_dump_count(const struct ovs_numa_dump *); +struct ovs_numa_dump * ovs_numa_thread_getaffinity_dump(void); +int ovs_numa_thread_setaffinity_dump(struct ovs_numa_dump *); void ovs_numa_dump_destroy(struct ovs_numa_dump *); int ovs_numa_thread_setaffinity_core(unsigned core_id); From patchwork Mon Sep 2 11:27:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1156594 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="WUBLGBLM"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46MSX3487Lz9s7T for ; Mon, 2 Sep 2019 21:28:51 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 92C23DBC; Mon, 2 Sep 2019 11:27:28 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id B5D74DB5 for ; Mon, 2 Sep 2019 11:27:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id EA96E5D3 for ; Mon, 2 Sep 2019 11:27:26 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190902112725euoutp0180844f4a3aa5efb41e16622ce58a1337~Am4BJJvEh2557125571euoutp01x for ; Mon, 2 Sep 2019 11:27:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190902112725euoutp0180844f4a3aa5efb41e16622ce58a1337~Am4BJJvEh2557125571euoutp01x DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1567423645; bh=At0oXA0yh178485tSIbzmQZcH39iuwmOp2euf5wcTkY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WUBLGBLMATuQnS2fW+cblGdzTSeFABB8SR2GDf9s48rJSDY/H3DVE76egA5BrAW4G 4z/qRkSo1akMe4fT2cJ4kUX/j5VP54ZlmDtA0MVHco/MtRE4clCCeXzr64FlLf527n swaTpx0QYwGD3zRurRsTGODZeQc5WQGVlbpjB8kU= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190902112725eucas1p11eff21fd0db6e6a0c058ced42f417fe4~Am4AofXfm3085430854eucas1p1x; Mon, 2 Sep 2019 11:27:25 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id 20.A5.04469.C9CFC6D5; Mon, 2 Sep 2019 12:27:24 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190902112724eucas1p10be06a9fede425def829b9fe69094872~Am3-14xnj1961219612eucas1p1A; Mon, 2 Sep 2019 11:27:24 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190902112723eusmtrp2366d7cfdccc355fac6c1d7452902b957~Am3-n6A7N2554125541eusmtrp2c; Mon, 2 Sep 2019 11:27:23 +0000 (GMT) X-AuditID: cbfec7f2-569ff70000001175-93-5d6cfc9c34b9 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id DD.F9.04166.B9CFC6D5; Mon, 2 Sep 2019 12:27:23 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20190902112723eusmtip10aa4557b41aacd146cd5e6a3a2034e56~Am3-FP_b92569025690eusmtip1h; Mon, 2 Sep 2019 11:27:23 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Mon, 2 Sep 2019 14:27:10 +0300 Message-Id: <20190902112711.2919-3-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190902112711.2919-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrCIsWRmVeSWpSXmKPExsWy7djP87pz/uTEGqyfpmOxfUUXm8WV9p/s FhsfnmW1WHvoA7vF3E/PGS2uX+JxYPPYOesuu8fiPS+ZPJ7d/M/o8X7fVTaPvi2rGANYo7hs UlJzMstSi/TtErgy3kzfwVww2bbizPmzLA2M9/W6GDk4JARMJK6s9u9i5OIQEljBKLFvzitG COcLo8SsqQuZIJzPjBK/Fx9i7mLkBOtoX/aBHSKxnFFiWstHKOcHo8STtz/YQarYBHQkTq0+ wghiiwhIS7zufcMKUsQscJhRYlfvQzaQhLBAgMSbORtZQWwWAVWJL58vMYMcxStgJXHyFyPE NnmJ1RsOgG3mFLCW6Ni4kA1kjoTAezaJ189uMkM84SJxoScfol5Y4tXxLewQtozE/53zmSDs eon7LS8ZIXo7GCWmH/oHlbCX2PL6HDvIHGYBTYn1u/QhRjpKTN8QBGHySdx4KwhSzAxkTto2 HWopr0RHmxDEDBWJ3weXQ4NHSuLmu89QB3hIvOjeDg2dPkaJtv07WCYwys9C2LWAkXEVo3hq aXFuemqxYV5quV5xYm5xaV66XnJ+7iZGYFo4/e/4px2MXy8lHWIU4GBU4uHl+JQdK8SaWFZc mXuIUYKDWUmEN3RPTqwQb0piZVVqUX58UWlOavEhRmkOFiVx3mqGB9FCAumJJanZqakFqUUw WSYOTqkGRoPiJpmcllMKPGZ/0m7w7PObu2r7Gv3MS0IrbltLbt6p+OPImtp7V9dtkVhYx7Xn bsLpjQETEpdOKEm+bTsz9e89GZ0XW395vNH61+4aZq4kUNT0YZa/X47vXY7JPyYpO+48Wpgs OTPauvLrxBtvq5m22hcumfj4kH2K887J/luWeEvO7V63LkaJpTgj0VCLuag4EQA5RI6iBwMA AA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsVy+t/xu7qz/+TEGlyfZmmxfUUXm8WV9p/s FhsfnmW1WHvoA7vF3E/PGS2uX+JxYPPYOesuu8fiPS+ZPJ7d/M/o8X7fVTaPvi2rGANYo/Rs ivJLS1IVMvKLS2yVog0tjPQMLS30jEws9QyNzWOtjEyV9O1sUlJzMstSi/TtEvQy3kzfwVww 2bbizPmzLA2M9/W6GDk5JARMJNqXfWDvYuTiEBJYyijx9PMpZoiElMSPXxdYIWxhiT/Xutgg ir4xSjT/OMkCkmAT0JE4tfoII4gtIiAt8br3DStIEbPAcUaJVZ07mEASwgJ+EntWLgFrYBFQ lfjy+RLQBg4OXgEriZO/GCEWyEus3nAAbDGngLVEx8aFbCC2EFDJ9MVfGCcw8i1gZFjFKJJa WpybnltsqFecmFtcmpeul5yfu4kRGKrbjv3cvIPx0sbgQ4wCHIxKPLwcn7JjhVgTy4orcw8x SnAwK4nwhu7JiRXiTUmsrEotyo8vKs1JLT7EaAp000RmKdHkfGAc5ZXEG5oamltYGpobmxub WSiJ83YIHIwREkhPLEnNTk0tSC2C6WPi4JRqYIyWaLz0qXXzxkOLH3OulD6bvSPs8uF1t/xl swIYZD0KkoQOyPirWJnvPm23+pHcvhXsk7wWsL7yW8DasXTBluYNlt8+i6dNqHgmv9OaRTcv M55XN+y41eatGe7TV3+yEt1o3hSRyB14Tur4TI/3T15cOrjiD8+ZCXFbntVPNHntlDf7KNOU je5KLMUZiYZazEXFiQBTSwzuawIAAA== X-CMS-MailID: 20190902112724eucas1p10be06a9fede425def829b9fe69094872 X-Msg-Generator: CA X-RootMTR: 20190902112724eucas1p10be06a9fede425def829b9fe69094872 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190902112724eucas1p10be06a9fede425def829b9fe69094872 References: <20190902112711.2919-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH v2 2/3] dpif-netdev-perf: Fix TSC frequency for non-DPDK case. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Unlike 'rte_get_tsc_cycles()' which doesn't need any specific initialization, 'rte_get_tsc_hz()' could be used only after successfull call to 'rte_eal_init()'. 'rte_eal_init()' estimates the TSC frequency for later use by 'rte_get_tsc_hz()'. Fairly said, we're not allowed to use 'rte_get_tsc_cycles()' before initializing DPDK too, but it works this way for now and provides correct results. This patch provides TSC frequency estimation code that will be used in two cases: * DPDK is not compiled in, i.e. DPDK_NETDEV not defined. * DPDK compiled in but not initialized, i.e. other_config:dpdk-init=false This change is mostly useful for AF_XDP netdev support, i.e. allows to use dpif-netdev/pmd-perf-show command and various PMD perf metrics. Signed-off-by: Ilya Maximets Acked-by: William Tu Reviewed-by: David Marchand --- lib/dpdk-stub.c | 6 ++++ lib/dpdk.c | 6 ++++ lib/dpdk.h | 1 + lib/dpif-netdev-perf.c | 75 ++++++++++++++++++++++++++++++++---------- lib/dpif-netdev-perf.h | 2 ++ lib/dpif-netdev.c | 9 +++++ 6 files changed, 81 insertions(+), 18 deletions(-) diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c index e55be5750..c332c217c 100644 --- a/lib/dpdk-stub.c +++ b/lib/dpdk-stub.c @@ -68,6 +68,12 @@ dpdk_per_port_memory(void) return false; } +bool +dpdk_available(void) +{ + return false; +} + void print_dpdk_version(void) { diff --git a/lib/dpdk.c b/lib/dpdk.c index f31e1580c..fc58de55a 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -518,6 +518,12 @@ dpdk_per_port_memory(void) return per_port_memory; } +bool +dpdk_available(void) +{ + return dpdk_initialized; +} + void dpdk_set_lcore_id(unsigned cpu) { diff --git a/lib/dpdk.h b/lib/dpdk.h index 7dab83775..736a64279 100644 --- a/lib/dpdk.h +++ b/lib/dpdk.h @@ -41,6 +41,7 @@ const char *dpdk_get_vhost_sock_dir(void); bool dpdk_vhost_iommu_enabled(void); bool dpdk_vhost_postcopy_enabled(void); bool dpdk_per_port_memory(void); +bool dpdk_available(void); void print_dpdk_version(void); void dpdk_status(const struct ovsrec_open_vswitch *); #endif /* dpdk.h */ diff --git a/lib/dpif-netdev-perf.c b/lib/dpif-netdev-perf.c index e7ed49e7e..baf90b0f4 100644 --- a/lib/dpif-netdev-perf.c +++ b/lib/dpif-netdev-perf.c @@ -17,9 +17,11 @@ #include #include +#include "dpdk.h" #include "dpif-netdev-perf.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/vlog.h" +#include "ovs-numa.h" #include "ovs-thread.h" #include "timeval.h" @@ -43,21 +45,59 @@ uint64_t iter_cycle_threshold; static struct vlog_rate_limit latency_rl = VLOG_RATE_LIMIT_INIT(600, 600); -#ifdef DPDK_NETDEV -static uint64_t -get_tsc_hz(void) -{ - return rte_get_tsc_hz(); -} -#else -/* This function is only invoked from PMD threads which depend on DPDK. - * A dummy function is sufficient when building without DPDK_NETDEV. */ -static uint64_t -get_tsc_hz(void) +static uint64_t tsc_hz = 1; + +void +pmd_perf_estimate_tsc_frequency(void) { - return 1; -} +#ifdef DPDK_NETDEV + if (dpdk_available()) { + tsc_hz = rte_get_tsc_hz(); + } + if (tsc_hz > 1) { + VLOG_INFO("DPDK provided TSC frequency: %"PRIu64" KHz", tsc_hz / 1000); + return; + } #endif + struct ovs_numa_dump *affinity; + struct pmd_perf_stats s; + uint64_t start, stop; + + /* DPDK is not available or returned unreliable value. + * Trying to estimate. */ + affinity = ovs_numa_thread_getaffinity_dump(); + if (affinity) { + const struct ovs_numa_info_core *core; + + FOR_EACH_CORE_ON_DUMP (core, affinity) { + /* Setting affinity to a single core from the affinity mask to + * avoid re-scheduling to another core while sleeping. */ + ovs_numa_thread_setaffinity_core(core->core_id); + break; + } + } + + start = cycles_counter_update(&s); + /* Using xnanosleep as it's interrupt resistant. + * Sleeping only 100 ms to avoid holding the main thread for too long. */ + xnanosleep(1E8); + stop = cycles_counter_update(&s); + + if (affinity) { + /* Restoring previous affinity. */ + ovs_numa_thread_setaffinity_dump(affinity); + ovs_numa_dump_destroy(affinity); + } + + if (stop <= start) { + VLOG_WARN("TSC source is unreliable."); + tsc_hz = 1; + } else { + tsc_hz = (stop - start) * 10; + } + + VLOG_INFO("Estimated TSC frequency: %"PRIu64" KHz", tsc_hz / 1000); +} /* Histogram functions. */ @@ -170,7 +210,6 @@ pmd_perf_format_overall_stats(struct ds *str, struct pmd_perf_stats *s, double duration) { uint64_t stats[PMD_N_STATS]; - uint64_t tsc_hz = get_tsc_hz(); double us_per_cycle = 1000000.0 / tsc_hz; if (duration == 0) { @@ -555,7 +594,7 @@ pmd_perf_end_iteration(struct pmd_perf_stats *s, int rx_packets, cum_ms->timestamp = now; } /* Do the next check after 4 us (10K cycles at 2.5 GHz TSC clock). */ - s->next_check_tsc = cycles_counter_update(s) + get_tsc_hz() / 250000; + s->next_check_tsc = cycles_counter_update(s) + tsc_hz / 250000; } } @@ -585,7 +624,7 @@ pmd_perf_set_log_susp_iteration(struct pmd_perf_stats *s, " duration=%"PRIu64" us\n", s->log_reason, susp->timestamp, - (1000000L * susp->cycles) / get_tsc_hz()); + (1000000L * susp->cycles) / tsc_hz); new_end_it = history_add(s->iterations.idx, log_it_after + 1); new_range = history_sub(new_end_it, s->log_begin_it); @@ -615,7 +654,7 @@ pmd_perf_log_susp_iteration_neighborhood(struct pmd_perf_stats *s) " duration=%"PRIu64" us\n", s->log_reason, susp->timestamp, - (1000000L * susp->cycles) / get_tsc_hz()); + (1000000L * susp->cycles) / tsc_hz); pmd_perf_format_iteration_history(&log, s, range); VLOG_WARN_RL(&latency_rl, @@ -729,7 +768,7 @@ pmd_perf_log_set_cmd(struct unixctl_conn *conn, log_it_after = it_after; log_q_thr = q_thr; log_us_thr = us_thr; - iter_cycle_threshold = (log_us_thr * get_tsc_hz()) / 1000000L; + iter_cycle_threshold = (log_us_thr * tsc_hz) / 1000000L; unixctl_command_reply(conn, ""); } diff --git a/lib/dpif-netdev-perf.h b/lib/dpif-netdev-perf.h index 244813ffe..ce369375b 100644 --- a/lib/dpif-netdev-perf.h +++ b/lib/dpif-netdev-perf.h @@ -233,6 +233,8 @@ cycles_counter_get(struct pmd_perf_stats *s) return s->last_tsc; } +void pmd_perf_estimate_tsc_frequency(void); + /* A nestable timer for measuring execution time in TSC cycles. * * Usage: diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 75d85b2fd..17323696f 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -1517,9 +1517,18 @@ create_dp_netdev(const char *name, const struct dpif_class *class, struct dp_netdev **dpp) OVS_REQUIRES(dp_netdev_mutex) { + static struct ovsthread_once tsc_freq_check = OVSTHREAD_ONCE_INITIALIZER; struct dp_netdev *dp; int error; + /* Avoid estimating TSC frequency for dummy datapath to not slow down + * unit tests. */ + if (!dpif_netdev_class_is_dummy(class) + && ovsthread_once_start(&tsc_freq_check)) { + pmd_perf_estimate_tsc_frequency(); + ovsthread_once_done(&tsc_freq_check); + } + dp = xzalloc(sizeof *dp); shash_add(&dp_netdevs, name, dp); From patchwork Mon Sep 2 11:27:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1156595 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="HlV4dfst"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46MSXh0Ypmz9sN1 for ; Mon, 2 Sep 2019 21:29:23 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 4E230DC6; Mon, 2 Sep 2019 11:27:32 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id C4BB6DA4 for ; Mon, 2 Sep 2019 11:27:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4E9C65D3 for ; Mon, 2 Sep 2019 11:27:30 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190902112728euoutp01bd0efaa98e3b441070b9e3b4e86b9f3d~Am4EOdSDC2495124951euoutp01M for ; Mon, 2 Sep 2019 11:27:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190902112728euoutp01bd0efaa98e3b441070b9e3b4e86b9f3d~Am4EOdSDC2495124951euoutp01M DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1567423648; bh=/UK3KLairZ9QvOcMCm8UIKnuomqQiGtcksnitjBGZOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HlV4dfstB7+M9o4sk+fDJnLT0rmiaCMGhz1sIaQ8nQxrXPxPA8CGvic+5WMZM3vP2 OfAl8LLYooUP3wWfYpRsZWPUA1L1zskSpr4DXkuWyC8nchhGI3PyIBYeS2Gd/g6xeA cdimtchmmrl2wbl1z3yHp2qjVVCljRR/NrMwVt7A= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20190902112728eucas1p2c0cc0dc1f49fc77893c5e0b1ac739d97~Am4DzNLoS1273912739eucas1p26; Mon, 2 Sep 2019 11:27:28 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id 34.A5.04469.0ACFC6D5; Mon, 2 Sep 2019 12:27:28 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20190902112727eucas1p153c3e24e8f6b822d624574f916f423a3~Am4Co3iPk1980919809eucas1p1d; Mon, 2 Sep 2019 11:27:27 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190902112726eusmtrp2fb36ef8b248cbffc54ef27fd9b8e0ac3~Am4Ca0Ie92550325503eusmtrp2B; Mon, 2 Sep 2019 11:27:26 +0000 (GMT) X-AuditID: cbfec7f2-569ff70000001175-a1-5d6cfca01db8 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id EF.F9.04166.E9CFC6D5; Mon, 2 Sep 2019 12:27:26 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20190902112726eusmtip171ef3fa528f0970b2637dff1c97c3fb1~Am4B4EOdT2389623896eusmtip1D; Mon, 2 Sep 2019 11:27:26 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Mon, 2 Sep 2019 14:27:11 +0300 Message-Id: <20190902112711.2919-4-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190902112711.2919-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrBIsWRmVeSWpSXmKPExsWy7djP87oL/uTEGnw5z26xfUUXm8WV9p/s FhsfnmW1WHvoA7vF3E/PGS2uX+JxYPPYOesuu8fiPS+ZPJ7d/M/o8X7fVTaPvi2rGANYo7hs UlJzMstSi/TtErgylh86xVzwSqTi7JRTbA2MM/i6GDk5JARMJH41NTJ2MXJxCAmsYJR4t3Me K4TzhVFi1ufDUM5nRomP5+6zwLQs2zefDSKxnFHi6swDUP0/gJwJG8Cq2AR0JE6tPsIIYosI SEu87n0DNopZ4DCjxK7eh2wgCWGBKInelg4wm0VAVeLJpR9gzbwCVhJzV51khlgnL7F6wwEw m1PAWqJj40Kw1RICr9kkLvS8hCpykTi2tIMRwhaWeHV8CzuELSPxf+d8Jgi7XuJ+y0tGiOYO Ronph/5BJewltrw+B9TAAXSepsT6XfoQYUeJZYfns4CEJQT4JG68FQQJMwOZk7ZNZ4YI80p0 tAlBVKtI/D64HOoaKYmb7z5DXeAhsWPxE3ZIAPUxSuw49YF9AqP8LIRlCxgZVzGKp5YW56an FhvmpZbrFSfmFpfmpesl5+duYgSmh9P/jn/awfj1UtIhRgEORiUeXo5P2bFCrIllxZW5hxgl OJiVRHhD9+TECvGmJFZWpRblxxeV5qQWH2KU5mBREuetZngQLSSQnliSmp2aWpBaBJNl4uCU amDkkPvy4oDw8W+rA+OuX3rCL8gqOef7L/61Em82Ld6x98+FA5O+aa9hZlBct+WClO1b3xN8 yuyGnFxxljobGhYfPJOXyKm4T3D1z8kT9yh1n20vUGL8I950WbUwYUuBTrXoyX1f7/nnbqq9 JmQ+RfloL0PqNGXr+5/jnm13Erk9t5/7s5TeMQ8XJZbijERDLeai4kQAcnDGtgsDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrJLMWRmVeSWpSXmKPExsVy+t/xu7rz/uTEGjxYZmmxfUUXm8WV9p/s FhsfnmW1WHvoA7vF3E/PGS2uX+JxYPPYOesuu8fiPS+ZPJ7d/M/o8X7fVTaPvi2rGANYo/Rs ivJLS1IVMvKLS2yVog0tjPQMLS30jEws9QyNzWOtjEyV9O1sUlJzMstSi/TtEvQylh86xVzw SqTi7JRTbA2MM/i6GDk5JARMJJbtm8/WxcjFISSwlFHi6OH7bBAJKYkfvy6wQtjCEn+udUEV fWOUmPhgIjNIgk1AR+LU6iOMILaIgLTE6943rCBFzALHGSVWde5gAkkIC0RIdK78DmazCKhK PLn0gwXE5hWwkpi76iQzxAZ5idUbDoDZnALWEh0bF4JdIQRUM33xF8YJjHwLGBlWMYqklhbn pucWG+oVJ+YWl+al6yXn525iBAbrtmM/N+9gvLQx+BCjAAejEg8vx6fsWCHWxLLiytxDjBIc zEoivKF7cmKFeFMSK6tSi/Lji0pzUosPMZoCHTWRWUo0OR8YSXkl8YamhuYWlobmxubGZhZK 4rwdAgdjhATSE0tSs1NTC1KLYPqYODilGhhrjLk+VfnU7n1ueLyX69xkDo/Wpqd9u4zlT7Aa SIvaKhxgFzmdfvN2qtPS+ZM1Tlt0Tbn2+dO+8uPPd7ze/CEzYm3ipF9O3Tecf994eXD54nf1 s2a4liX9Pf59wvq4f+dPffNWW7JXc4/1MrmNf1n5HP1fWMiwT3gjcGf2gVf3IxJDJv3U/HXl lRJLcUaioRZzUXEiANJwSk1sAgAA X-CMS-MailID: 20190902112727eucas1p153c3e24e8f6b822d624574f916f423a3 X-Msg-Generator: CA X-RootMTR: 20190902112727eucas1p153c3e24e8f6b822d624574f916f423a3 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190902112727eucas1p153c3e24e8f6b822d624574f916f423a3 References: <20190902112711.2919-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH v2 3/3] dpdk: Use ovs-numa provided functions to manage thread affinity. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This allows to decrease code duplication and avoid using Linux-specific functions (this might be useful in the future if we'll try to allow running OvS+DPDK on FreeBSD). Signed-off-by: Ilya Maximets Acked-by: William Tu Reviewed-by: David Marchand --- lib/dpdk.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/lib/dpdk.c b/lib/dpdk.c index fc58de55a..6f297d918 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -275,7 +275,7 @@ dpdk_init__(const struct smap *ovs_other_config) int result; bool auto_determine = true; int err = 0; - cpu_set_t cpuset; + struct ovs_numa_dump *affinity = NULL; struct svec args = SVEC_EMPTY_INITIALIZER; log_stream = fopencookie(NULL, "w+", dpdk_log_func); @@ -357,22 +357,22 @@ dpdk_init__(const struct smap *ovs_other_config) * lcore for the DPDK Master. */ if (auto_determine) { + const struct ovs_numa_info_core *core; int cpu = 0; /* Get the main thread affinity */ - CPU_ZERO(&cpuset); - err = pthread_getaffinity_np(pthread_self(), sizeof(cpu_set_t), - &cpuset); - if (!err) { - for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { - if (CPU_ISSET(cpu, &cpuset)) { - break; + affinity = ovs_numa_thread_getaffinity_dump(); + if (affinity) { + cpu = INT_MAX; + FOR_EACH_CORE_ON_DUMP (core, affinity) { + if (cpu > core->core_id) { + cpu = core->core_id; } } } else { /* User did not set dpdk-lcore-mask and unable to get current * thread affintity - default to core #0 */ - VLOG_ERR("Thread getaffinity error %d. Using core #0", err); + VLOG_ERR("Thread getaffinity failed. Using core #0"); } svec_add(&args, "-l"); svec_add_nocopy(&args, xasprintf("%d", cpu)); @@ -403,12 +403,9 @@ dpdk_init__(const struct smap *ovs_other_config) svec_destroy(&args); /* Set the main thread affinity back to pre rte_eal_init() value */ - if (auto_determine && !err) { - err = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), - &cpuset); - if (err) { - VLOG_ERR("Thread setaffinity error %d", err); - } + if (affinity) { + ovs_numa_thread_setaffinity_dump(affinity); + ovs_numa_dump_destroy(affinity); } if (result < 0) {