From patchwork Tue Aug 13 15:30:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1146410 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="YPh3g4EY"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 467Grg3Ckgz9sN1 for ; Wed, 14 Aug 2019 01:30:59 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 6DC25CD5; Tue, 13 Aug 2019 15:30:23 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id C4D95CAC for ; Tue, 13 Aug 2019 15:30:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout2.w1.samsung.com (mailout2.w1.samsung.com [210.118.77.12]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 3FEC789D for ; Tue, 13 Aug 2019 15:30:21 +0000 (UTC) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout2.w1.samsung.com (KnoxPortal) with ESMTP id 20190813153019euoutp0238c9611b2ac10ea56ed099e0871c3514~6hSZD3wYL3042130421euoutp02K for ; Tue, 13 Aug 2019 15:30:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.w1.samsung.com 20190813153019euoutp0238c9611b2ac10ea56ed099e0871c3514~6hSZD3wYL3042130421euoutp02K DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1565710219; bh=Eec5ajJuS7iCZndJtulDt/Cbo5i+b+wrK9oEP493L30=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YPh3g4EY+X7UvbT9+70iuo21YaaHHyEbYchENRN/lAUZApFd/l7QYIU/9j80OCp4P KZSu0ldgxYfHuPd7gqJuaYVyhArhBQ29VUMkDlP5RSnMVNpX+LX5ctgVO93UysvzIq wBdHz0yzryQd1rKNYDCevNf4XXgu49kU50YR+NgQ= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20190813153019eucas1p2c2d60e6d292640aaa9804de4fab9abdf~6hSYoZ7HH0292002920eucas1p2i; Tue, 13 Aug 2019 15:30:19 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id 23.40.04309.B87D25D5; Tue, 13 Aug 2019 16:30:19 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20190813153018eucas1p2475c91240eb42350eb2b32a6a255e80e~6hSX3XRUw0048000480eucas1p2L; Tue, 13 Aug 2019 15:30:18 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190813153018eusmtrp272a9dac8188600c8f8911ed0ca6ff70d~6hSXn0wtO0716707167eusmtrp2c; Tue, 13 Aug 2019 15:30:18 +0000 (GMT) X-AuditID: cbfec7f4-ae1ff700000010d5-88-5d52d78b3f37 Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id E5.73.04166.987D25D5; Tue, 13 Aug 2019 16:30:17 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20190813153017eusmtip2f4a6aacb80d76f32e02cbeb6fbb73cdf~6hSW3fYe51343113431eusmtip2m; Tue, 13 Aug 2019 15:30:17 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Aug 2019 18:30:05 +0300 Message-Id: <20190813153007.32305-2-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813153007.32305-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrKIsWRmVeSWpSXmKPExsWy7djP87rd14NiDR51sFpsX9HFZnGl/Se7 xcaHZ1kt1h76wG4x99NzRovrl3gc2Dx2zrrL7rF4z0smj2c3/zN6vN93lc2jb8sqxgDWKC6b lNSczLLUIn27BK6MxSceshe0SVdMOPyJsYHxikgXIyeHhICJxLuLm9m7GLk4hARWMEqc33KI FcL5wiixYO46RpAqIYHPjBLTtgElOMA6bnz3hKhZzijxe/t5ZgjnB6PEltblbCANbAI6EqdW HwFrFhGQlnjd+wZsKrPAYUaJXb0PwYqEBTwkbkycygRiswioSrzr62QD2cArYC2xobcO4jx5 idUbDjCD2JwCNhJrG+4zgcyREHjNJnH5+nd2iCIXiWN/p7NA2MISr45vgYrLSPzfOZ8Jwq6X uN/ykhGiuYNRYvqhf1AJe4ktr8+xgyxmFtCUWL9LHyLsKPGlaQ4bxMd8EjfeCoKEmYHMSdum M0OEeSU62oQgqlUkfh9czgxhS0ncfPcZ6gIPiTlP26AB2s8o0bN3PssERvlZCMsWMDKuYhRP LS3OTU8tNspLLdcrTswtLs1L10vOz93ECEwMp/8d/7KDcdefpEOMAhyMSjy8AVuCYoVYE8uK K3MPMUpwMCuJ8E64CBTiTUmsrEotyo8vKs1JLT7EKM3BoiTOW83wIFpIID2xJDU7NbUgtQgm y8TBKdXAuCbxzFL5JxMVY3ZPOyaeGeObl/Ll1PnjDIrXhN7Ndzfut7RrORp/MlPEYpa8+22L W54zp5RZPPNrf+mSe0O8fvPB6bLdOr8vVylk37unlntgoafrCmuOvdUpnU+Xi3Vf44hw/NHd fdXtTMOWI9OtLwT2bDs6UVdv07sJVo+mrJQXu2i41k20VYmlOCPRUIu5qDgRAI6M7YkIAwAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsVy+t/xe7qd14NiDSbetrLYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUXo2 RfmlJakKGfnFJbZK0YYWRnqGlhZ6RiaWeobG5rFWRqZK+nY2Kak5mWWpRfp2CXoZi088ZC9o k66YcPgTYwPjFZEuRg4OCQETiRvfPbsYOTmEBJYySvxuyAOxJQSkJH78usAKYQtL/LnWxdbF yAVU841RYs70JWwgCTYBHYlTq48wgtgiAtISr3vfsIIUMQscZ5RY1bmDCSQhLOAhcWPiVDCb RUBV4l1fJxvIYl4Ba4kNvXUQC+QlVm84wAxicwrYSKxtuM8EcZC1RMu51UwTGPkWMDKsYhRJ LS3OTc8tNtQrTswtLs1L10vOz93ECAzTbcd+bt7BeGlj8CFGAQ5GJR7egC1BsUKsiWXFlbmH GCU4mJVEeCdcBArxpiRWVqUW5ccXleakFh9iNAW6aSKzlGhyPjCG8kriDU0NzS0sDc2NzY3N LJTEeTsEDsYICaQnlqRmp6YWpBbB9DFxcEo1MDqdm7LHaqd1w1bLtGviXa/T5e6u7ZwZUbnW rTxz3sY1GTWyL88Znfp+IzGMIdvMQdZEvru177j1sytxn9M47pt9PTK5p2r+Tv2XXbverZg3 S1bp14SZueaSbcF9xq3dd1Tfs990c9plf8jL4dfFNRtaExSC3xjn7lqor1/b7bUhZ42gzvHZ CkosxRmJhlrMRcWJAA+5TwppAgAA X-CMS-MailID: 20190813153018eucas1p2475c91240eb42350eb2b32a6a255e80e X-Msg-Generator: CA X-RootMTR: 20190813153018eucas1p2475c91240eb42350eb2b32a6a255e80e X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190813153018eucas1p2475c91240eb42350eb2b32a6a255e80e References: <20190813153007.32305-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH 1/3] ovs-numa: Add dump based thread affinity functions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org New functions to get and set CPU affinity using CPU dumps. This will abstract OS specific implementation details from the cross-platform code. Signed-off-by: Ilya Maximets Acked-by: William Tu --- lib/ovs-numa.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++- lib/ovs-numa.h | 2 ++ 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index 24edeab2a..a417730b8 100644 --- a/lib/ovs-numa.c +++ b/lib/ovs-numa.c @@ -532,6 +532,78 @@ ovs_numa_dump_destroy(struct ovs_numa_dump *dump) free(dump); } +struct ovs_numa_dump * +ovs_numa_thread_getaffinity_dump(void) +{ + if (dummy_numa) { + /* Nothing to do. */ + return NULL; + } + +#ifndef __linux__ + return NULL; +#else + struct ovs_numa_dump *dump; + const struct numa_node *n; + cpu_set_t cpuset; + int err; + + CPU_ZERO(&cpuset); + err = pthread_getaffinity_np(pthread_self(), sizeof cpuset, &cpuset); + if (err) { + VLOG_ERR("Thread getaffinity error: %s", ovs_strerror(err)); + return NULL; + } + + dump = ovs_numa_dump_create(); + + HMAP_FOR_EACH (n, hmap_node, &all_numa_nodes) { + const struct cpu_core *core; + + LIST_FOR_EACH (core, list_node, &n->cores) { + if (CPU_ISSET(core->core_id, &cpuset)) { + ovs_numa_dump_add(dump, core->numa->numa_id, core->core_id); + } + } + } + + if (!ovs_numa_dump_count(dump)) { + ovs_numa_dump_destroy(dump); + return NULL; + } + return dump; +#endif /* __linux__ */ +} + +int +ovs_numa_thread_setaffinity_dump(struct ovs_numa_dump *dump OVS_UNUSED) +{ + if (!dump || dummy_numa) { + /* Nothing to do. */ + return 0; + } + +#ifdef __linux__ + struct ovs_numa_info_core *core; + cpu_set_t cpuset; + int err; + + CPU_ZERO(&cpuset); + FOR_EACH_CORE_ON_DUMP (core, dump) { + CPU_SET(core->core_id, &cpuset); + } + err = pthread_setaffinity_np(pthread_self(), sizeof cpuset, &cpuset); + if (err) { + VLOG_ERR("Thread setaffinity error: %s", ovs_strerror(err)); + return err; + } + + return 0; +#else /* !__linux__ */ + return EOPNOTSUPP; +#endif /* __linux__ */ +} + int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED) { if (dummy_numa) { @@ -547,7 +619,7 @@ int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED) CPU_SET(core_id, &cpuset); err = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset); if (err) { - VLOG_ERR("Thread affinity error %d",err); + VLOG_ERR("Thread setaffinity error: %s", ovs_strerror(err)); return err; } diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index 088fcb8c3..88352a93e 100644 --- a/lib/ovs-numa.h +++ b/lib/ovs-numa.h @@ -60,6 +60,8 @@ struct ovs_numa_dump *ovs_numa_dump_n_cores_per_numa(int n); bool ovs_numa_dump_contains_core(const struct ovs_numa_dump *, int numa_id, unsigned core_id); size_t ovs_numa_dump_count(const struct ovs_numa_dump *); +struct ovs_numa_dump * ovs_numa_thread_getaffinity_dump(void); +int ovs_numa_thread_setaffinity_dump(struct ovs_numa_dump *); void ovs_numa_dump_destroy(struct ovs_numa_dump *); int ovs_numa_thread_setaffinity_core(unsigned core_id); From patchwork Tue Aug 13 15:30:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1146411 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="g9IuJLDO"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 467GsL07dyz9sN1 for ; Wed, 14 Aug 2019 01:31:34 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 1AF8CCDB; Tue, 13 Aug 2019 15:30:25 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 42798CD4 for ; Tue, 13 Aug 2019 15:30:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout2.w1.samsung.com (mailout2.w1.samsung.com [210.118.77.12]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 563058A6 for ; Tue, 13 Aug 2019 15:30:23 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout2.w1.samsung.com (KnoxPortal) with ESMTP id 20190813153021euoutp028dd77a5c245958a2faf1199c6f8c4e1e~6hSbC1P-g2908929089euoutp02C for ; Tue, 13 Aug 2019 15:30:21 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.w1.samsung.com 20190813153021euoutp028dd77a5c245958a2faf1199c6f8c4e1e~6hSbC1P-g2908929089euoutp02C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1565710222; bh=LNAKjQ/WWj7a2lgY3nAFNYcxSDSyzeDYMFatAzJXEvo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g9IuJLDOgyQAGa7CojI25eYpC5K4+uJThAIpZfg5trwNH+wBnU1t4LHSoukpdRK4f B7qI1vhT4kZaMQcpF4eXkmfE9HyU/NNPJo4Qvrpe7O3y8hYtMu+u3ibJ1Xv8MRWqKp Nd6fQuVl475A6eKLu7mXgF3WwRUwM8C1u5vcySQ4= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190813153021eucas1p1bfdc4541aac6e14058f6fa5860d76935~6hSaonghc1373113731eucas1p1O; Tue, 13 Aug 2019 15:30:21 +0000 (GMT) Received: from eucas1p2.samsung.com ( [182.198.249.207]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id A7.0D.04469.D87D25D5; Tue, 13 Aug 2019 16:30:21 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20190813153020eucas1p2f600846d7b2afa2814939ec312b9fb72~6hSZxXKkC0089000890eucas1p2n; Tue, 13 Aug 2019 15:30:20 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190813153020eusmtrp248e0cbdf3060d13dba4d3d3fb64c6487~6hSZh5QLO0716707167eusmtrp2j; Tue, 13 Aug 2019 15:30:20 +0000 (GMT) X-AuditID: cbfec7f2-54fff70000001175-0a-5d52d78d0df6 Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 57.73.04166.C87D25D5; Tue, 13 Aug 2019 16:30:20 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20190813153019eusmtip2edeaf6685e048ee1d44d384aa1442e7b~6hSZFH9ZY1702317023eusmtip2Y; Tue, 13 Aug 2019 15:30:19 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Aug 2019 18:30:06 +0300 Message-Id: <20190813153007.32305-3-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813153007.32305-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrGIsWRmVeSWpSXmKPExsWy7djP87q914NiDQ78ErDYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUVw2 Kak5mWWpRfp2CVwZjVudC17YVJzs6mZvYNyp18XIySEhYCJx8Op/pi5GLg4hgRWMEvs3noBy vjBKzHuyD8r5zCjxfftFRpiWydtusEAkljNKbLu9kxXC+cEo8ejJEmaQKjYBHYlTq4+AdYgI SEu87n0DVsQscJhRYlfvQzaQhLCAt8TlFzvAbBYBVYmp3xaANfMKWEvMW3CXCWKdvMTqDQfA 4pwCNhJrG+6D3SQh8JxNYueD0ywQRS4STS86WSFsYYlXx7ewQ9gyEv93zocaVC9xv+UlI0Rz B6PE9EP/oBL2EltenwNq4AA6T1Ni/S59iLCjROvGDlaQsIQAn8SNt4IgYWYgc9K26cwQYV6J jjYhiGoVid8HlzND2FISN999Zoco8ZDY/kIVEj79jBIX/zxmm8AoPwth1wJGxlWM4qmlxbnp qcWGeanlesWJucWleel6yfm5mxiBqeH0v+OfdjB+vZR0iFGAg1GJhzdgS1CsEGtiWXFl7iFG CQ5mJRHeCReBQrwpiZVVqUX58UWlOanFhxilOViUxHmrGR5ECwmkJ5akZqemFqQWwWSZODil Ghj3HTyW5j0jtNJ4Xaj6Z7E9J//ZnoqycGje78ldNNV4O+/938JzM/eu29VdcePg5doTnUeX rrrRd75Vou/zXoknWgor/CfauOZuuWigtOfTgbapX39pXrx6d/OipK6qy5/YPrwwS7fxCo97 2rvuTd/0i/35rXvziuUmt/xYtlKQq3Vt8uxbTneslFiKMxINtZiLihMBU9cvCwkDAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsVy+t/xe7o914NiDSZtF7HYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUXo2 RfmlJakKGfnFJbZK0YYWRnqGlhZ6RiaWeobG5rFWRqZK+nY2Kak5mWWpRfp2CXoZjVudC17Y VJzs6mZvYNyp18XIySEhYCIxedsNli5GLg4hgaWMEp0LNrFAJKQkfvy6wAphC0v8udbFBlH0 jVFi6dvfbCAJNgEdiVOrjzCC2CIC0hKve9+wghQxCxxnlFjVuYMJJCEs4C1x+cUOsAYWAVWJ qd8WMIPYvALWEvMW3GWC2CAvsXrDAbA4p4CNxNqG+2BxIaCalnOrmSYw8i1gZFjFKJJaWpyb nltsqFecmFtcmpeul5yfu4kRGKrbjv3cvIPx0sbgQ4wCHIxKPLwBW4JihVgTy4orcw8xSnAw K4nwTrgIFOJNSaysSi3Kjy8qzUktPsRoCnTURGYp0eR8YBzllcQbmhqaW1gamhubG5tZKInz dggcjBESSE8sSc1OTS1ILYLpY+LglGpgVP/S8+rjR6ZfWnU8jUevnuO6Y1U62V408dhd3s4s 8XiGZWcq7C+KcpjqXjHSv3Xxhlv+zAuPn3LMMMr0c5rZqyJj1iR2lfGP0eyNQf5leV8mbYhf N1Ui8HD4yaVeV52erIh9No9zXeZa78M5Kz6/kbgRO1nh0/l445qdDiWNoVlztXff3fqpQoml OCPRUIu5qDgRAOCcneNrAgAA X-CMS-MailID: 20190813153020eucas1p2f600846d7b2afa2814939ec312b9fb72 X-Msg-Generator: CA X-RootMTR: 20190813153020eucas1p2f600846d7b2afa2814939ec312b9fb72 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190813153020eucas1p2f600846d7b2afa2814939ec312b9fb72 References: <20190813153007.32305-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH 2/3] dpif-netdev-perf: Fix TSC frequency for non-DPDK case. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Unlike 'rte_get_tsc_cycles()' which doesn't need any specific initialization, 'rte_get_tsc_hz()' could be used only after successfull call to 'rte_eal_init()'. 'rte_eal_init()' estimates the TSC frequency for later use by 'rte_get_tsc_hz()'. Fairly said, we're not allowed to use 'rte_get_tsc_cycles()' before initializing DPDK too, but it works this way for now and provides correct results. This patch provides TSC frequency estimation code that will be used in two cases: * DPDK is not compiled in, i.e. DPDK_NETDEV not defined. * DPDK compiled in but not initialized, i.e. other_config:dpdk-init=false This change is mostly useful for AF_XDP netdev support, i.e. allows to use dpif-netdev/pmd-perf-show command and various PMD perf metrics. Signed-off-by: Ilya Maximets Acked-by: William Tu --- lib/dpdk-stub.c | 6 ++++ lib/dpdk.c | 6 ++++ lib/dpdk.h | 1 + lib/dpif-netdev-perf.c | 75 ++++++++++++++++++++++++++++++++---------- lib/dpif-netdev-perf.h | 2 ++ lib/dpif-netdev.c | 9 +++++ 6 files changed, 81 insertions(+), 18 deletions(-) diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c index e55be5750..c332c217c 100644 --- a/lib/dpdk-stub.c +++ b/lib/dpdk-stub.c @@ -68,6 +68,12 @@ dpdk_per_port_memory(void) return false; } +bool +dpdk_available(void) +{ + return false; +} + void print_dpdk_version(void) { diff --git a/lib/dpdk.c b/lib/dpdk.c index f31e1580c..fc58de55a 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -518,6 +518,12 @@ dpdk_per_port_memory(void) return per_port_memory; } +bool +dpdk_available(void) +{ + return dpdk_initialized; +} + void dpdk_set_lcore_id(unsigned cpu) { diff --git a/lib/dpdk.h b/lib/dpdk.h index 7dab83775..736a64279 100644 --- a/lib/dpdk.h +++ b/lib/dpdk.h @@ -41,6 +41,7 @@ const char *dpdk_get_vhost_sock_dir(void); bool dpdk_vhost_iommu_enabled(void); bool dpdk_vhost_postcopy_enabled(void); bool dpdk_per_port_memory(void); +bool dpdk_available(void); void print_dpdk_version(void); void dpdk_status(const struct ovsrec_open_vswitch *); #endif /* dpdk.h */ diff --git a/lib/dpif-netdev-perf.c b/lib/dpif-netdev-perf.c index e7ed49e7e..baa4d92c6 100644 --- a/lib/dpif-netdev-perf.c +++ b/lib/dpif-netdev-perf.c @@ -17,9 +17,11 @@ #include #include +#include "dpdk.h" #include "dpif-netdev-perf.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/vlog.h" +#include "ovs-numa.h" #include "ovs-thread.h" #include "timeval.h" @@ -43,21 +45,59 @@ uint64_t iter_cycle_threshold; static struct vlog_rate_limit latency_rl = VLOG_RATE_LIMIT_INIT(600, 600); -#ifdef DPDK_NETDEV -static uint64_t -get_tsc_hz(void) -{ - return rte_get_tsc_hz(); -} -#else -/* This function is only invoked from PMD threads which depend on DPDK. - * A dummy function is sufficient when building without DPDK_NETDEV. */ -static uint64_t -get_tsc_hz(void) +static uint64_t tsc_hz = 1; + +void +pmd_perf_estimate_tsc_frequency(void) { - return 1; -} +#ifdef DPDK_NETDEV + if (dpdk_available()) { + tsc_hz = rte_get_tsc_hz(); + } + if (tsc_hz > 1) { + VLOG_INFO("DPDK provided TSC frequency: %"PRIu64" KHz", tsc_hz / 1000); + return; + } #endif + struct ovs_numa_dump *affinity; + struct pmd_perf_stats s; + uint64_t start, stop; + + /* DPDK is not available or returned unreliable value. + * Trying to estimate. */ + affinity = ovs_numa_thread_getaffinity_dump(); + if (affinity) { + const struct ovs_numa_info_core *core; + + FOR_EACH_CORE_ON_DUMP (core, affinity) { + /* Setting affinity to a single core from the affinity mask to + * avoid re-scheduling to another core while sleeping. */ + ovs_numa_thread_setaffinity_core(core->core_id); + break; + } + } + + start = cycles_counter_update(&s); + /* Using xnanosleep as it's interrupt resistant. + * Sleeping only 100 ms to avoid holding the main thread for too long. */ + xnanosleep(1E8); + stop = cycles_counter_update(&s); + + if (affinity) { + /* Restoring previous affinity. */ + ovs_numa_thread_setaffinity_dump(affinity); + ovs_numa_dump_destroy(affinity); + } + + if (stop <= start) { + VLOG_WARN("TSC source is unreliable."); + tsc_hz = 1; + } else { + tsc_hz = (stop - start) * 10; + } + + VLOG_INFO("Estimated TSC frequency: %"PRIu64" KHz", tsc_hz / 1000); +} /* Histogram functions. */ @@ -170,7 +210,6 @@ pmd_perf_format_overall_stats(struct ds *str, struct pmd_perf_stats *s, double duration) { uint64_t stats[PMD_N_STATS]; - uint64_t tsc_hz = get_tsc_hz(); double us_per_cycle = 1000000.0 / tsc_hz; if (duration == 0) { @@ -555,7 +594,7 @@ pmd_perf_end_iteration(struct pmd_perf_stats *s, int rx_packets, cum_ms->timestamp = now; } /* Do the next check after 4 us (10K cycles at 2.5 GHz TSC clock). */ - s->next_check_tsc = cycles_counter_update(s) + get_tsc_hz() / 250000; + s->next_check_tsc = cycles_counter_update(s) + tsc_hz / 250000; } } @@ -585,7 +624,7 @@ pmd_perf_set_log_susp_iteration(struct pmd_perf_stats *s, " duration=%"PRIu64" us\n", s->log_reason, susp->timestamp, - (1000000L * susp->cycles) / get_tsc_hz()); + (1000000L * susp->cycles) / tsc_hz); new_end_it = history_add(s->iterations.idx, log_it_after + 1); new_range = history_sub(new_end_it, s->log_begin_it); @@ -615,7 +654,7 @@ pmd_perf_log_susp_iteration_neighborhood(struct pmd_perf_stats *s) " duration=%"PRIu64" us\n", s->log_reason, susp->timestamp, - (1000000L * susp->cycles) / get_tsc_hz()); + (1000000L * susp->cycles) / tsc_hz); pmd_perf_format_iteration_history(&log, s, range); VLOG_WARN_RL(&latency_rl, @@ -729,7 +768,7 @@ pmd_perf_log_set_cmd(struct unixctl_conn *conn, log_it_after = it_after; log_q_thr = q_thr; log_us_thr = us_thr; - iter_cycle_threshold = (log_us_thr * get_tsc_hz()) / 1000000L; + iter_cycle_threshold = (log_us_thr * tsc_hz) / 1000000L; unixctl_command_reply(conn, ""); } diff --git a/lib/dpif-netdev-perf.h b/lib/dpif-netdev-perf.h index 244813ffe..ce369375b 100644 --- a/lib/dpif-netdev-perf.h +++ b/lib/dpif-netdev-perf.h @@ -233,6 +233,8 @@ cycles_counter_get(struct pmd_perf_stats *s) return s->last_tsc; } +void pmd_perf_estimate_tsc_frequency(void); + /* A nestable timer for measuring execution time in TSC cycles. * * Usage: diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index d0a1c58ad..bc8791016 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -1517,9 +1517,18 @@ create_dp_netdev(const char *name, const struct dpif_class *class, struct dp_netdev **dpp) OVS_REQUIRES(dp_netdev_mutex) { + static struct ovsthread_once tsc_freq_check = OVSTHREAD_ONCE_INITIALIZER; struct dp_netdev *dp; int error; + /* Avoid estimating TSC frequency for dummy datapath to not slow down + * unit tests. */ + if (!dpif_netdev_class_is_dummy(class) + && ovsthread_once_start(&tsc_freq_check)) { + pmd_perf_estimate_tsc_frequency(); + ovsthread_once_done(&tsc_freq_check); + } + dp = xzalloc(sizeof *dp); shash_add(&dp_netdevs, name, dp); From patchwork Tue Aug 13 15:30:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1146412 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="Ou5bAETL"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 467GtC1z72z9sN1 for ; Wed, 14 Aug 2019 01:32:19 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 06DB2D4B; Tue, 13 Aug 2019 15:30:29 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 6F438D4F for ; Tue, 13 Aug 2019 15:30:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id C99FC8AB for ; Tue, 13 Aug 2019 15:30:26 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190813153025euoutp01a3a2daa7bf1562c6ae67ee8263a170b6~6hSd4pIKg0999909999euoutp01d for ; Tue, 13 Aug 2019 15:30:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190813153025euoutp01a3a2daa7bf1562c6ae67ee8263a170b6~6hSd4pIKg0999909999euoutp01d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1565710225; bh=QTHUst9RT8ylBOoy5ricPJOrr/4lK3O1syoSJjIuAxA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ou5bAETLq1TqSKR2gn/3IeyjT2UGmeL2PCW6DKd3A2Ut50hqz+Vu0BPpyyLTSH34i hR3jmqVKzWlL630Jrd9+evvBzAmqTFfgtXe3VZIgPi3kk/S9kP6yMFZKQk56i/xBst U4XQTPmy3u6BeViXXmqyjmfcLHKyNRSgdUSSRDxg= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20190813153024eucas1p11e1603ab04a8ea4afbca098c24e1b561~6hSdbqk3T1487414874eucas1p1V; Tue, 13 Aug 2019 15:30:24 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id 45.40.04309.097D25D5; Tue, 13 Aug 2019 16:30:24 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20190813153023eucas1p27213f4f18327515251b66bea7ec4ee6d~6hScZBq2s0050700507eucas1p2Q; Tue, 13 Aug 2019 15:30:23 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190813153023eusmtrp2b32cace9537d6c31cc1873c1f0b71f1e~6hScI25Qv0716707167eusmtrp2s; Tue, 13 Aug 2019 15:30:23 +0000 (GMT) X-AuditID: cbfec7f4-ae1ff700000010d5-94-5d52d790231f Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 18.73.04166.F87D25D5; Tue, 13 Aug 2019 16:30:23 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20190813153022eusmtip2de4d98fcfed6cfb9e77386b8d470c7a2~6hSbs_4ME1702317023eusmtip2a; Tue, 13 Aug 2019 15:30:22 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 13 Aug 2019 18:30:07 +0300 Message-Id: <20190813153007.32305-4-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813153007.32305-1-i.maximets@samsung.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrGIsWRmVeSWpSXmKPExsWy7djPc7oTrgfFGjRP47PYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUVw2 Kak5mWWpRfp2CVwZl062shfsF6mYOWU3UwPjJ94uRg4OCQETienHS7oYuTiEBFYwSqw585EN wvnCKPH0zGV2COczo8Sr69+AHE6wjoez17JCJJYzSlzbt48RwvnBKLHp+3NGkCo2AR2JU6uP gNkiAtISr3vfgHUwCxxmlNjV+5ANJCEsEC7xpvEGmM0ioCqxblsLWAOvgLXE5sNfmSHWyUus 3nAAzOYUsJFY23CfCWSQhMBzNon+ff3MEF+4SGw+5ANRLyzx6vgWqFNlJP7vnM8EYddL3G95 yQjR28EoMf3QP6iEvcSW1+fYQeYwC2hKrN+lDxF2lNg88SILxHg+iRtvBUHCzEDmpG3Tobby SnS0CUFUq0j8Prgc6mIpiZvvPkNd4CHx69Z0aPj0M0os27WOeQKj/CyEZQsYGVcxiqeWFuem pxYb5aWW6xUn5haX5qXrJefnbmIEpobT/45/2cG460/SIUYBDkYlHt6ALUGxQqyJZcWVuYcY JTiYlUR4J1wECvGmJFZWpRblxxeV5qQWH2KU5mBREuetZngQLSSQnliSmp2aWpBaBJNl4uCU amCUu9of3XAt5tbqGbVn5lya2uLhvXDRM40le7ceSTTd5fpFYL6b27mdrF3Oy73OKl6fpmV+ crcRp6nc+0VuC3/Pve3/4mic1eTehUv2u8x7KtT/yueaZ0CKkqiz4L7PS5qyb06cb7nh9tv9 Bf48k7fZ1LN9i78ya7/K0p+3fLX+R786O93jnvurUiWW4oxEQy3mouJEAFVrwY8JAwAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsVy+t/xe7r914NiDQ5P4bTYvqKLzeJK+092 i40Pz7JarD30gd1i7qfnjBbXL/E4sHnsnHWX3WPxnpdMHs9u/mf0eL/vKptH35ZVjAGsUXo2 RfmlJakKGfnFJbZK0YYWRnqGlhZ6RiaWeobG5rFWRqZK+nY2Kak5mWWpRfp2CXoZl062shfs F6mYOWU3UwPjJ94uRk4OCQETiYez17KC2EICSxkl7j7mg4hLSfz4dYEVwhaW+HOti62LkQuo 5hujxOct/xlBEmwCOhKnVh8Bs0UEpCVe975hBSliFjjOKLGqcwcTSEJYIFRi4twTYDaLgKrE um0tYA28AtYSmw9/ZYbYIC+xesMBMJtTwEZibcN9JoiLrCVazq1mmsDIt4CRYRWjSGppcW56 brGhXnFibnFpXrpecn7uJkZgqG479nPzDsZLG4MPMQpwMCrx8AZsCYoVYk0sK67MPcQowcGs JMI74SJQiDclsbIqtSg/vqg0J7X4EKMp0FETmaVEk/OBcZRXEm9oamhuYWlobmxubGahJM7b IXAwRkggPbEkNTs1tSC1CKaPiYNTqoGx/3OaSODGjWez3vUX5O99Y87n4HTccn6XQtriuyG3 CtQC2kyEKju2BDZKuT2Se71M/iLva8HHfoV3IiVDkxe6Xv340Sh9Rdmt3l+uDGEXHucdyvh+ dKpVyO41R0/7KzZ9YiuczaIfqfks4P3+xqPrFvNZOzyd3XHMt1bpxoPTjVmfbh2oD21XYinO SDTUYi4qTgQAWpulnmsCAAA= X-CMS-MailID: 20190813153023eucas1p27213f4f18327515251b66bea7ec4ee6d X-Msg-Generator: CA X-RootMTR: 20190813153023eucas1p27213f4f18327515251b66bea7ec4ee6d X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190813153023eucas1p27213f4f18327515251b66bea7ec4ee6d References: <20190813153007.32305-1-i.maximets@samsung.com> X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH 3/3] dpdk: Use ovs-numa provided functions to manage thread affinity. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This allows to decrease code duplication and avoid using Linux-specific functions (this might be useful in the future if we'll try to allow running OvS+DPDK on FreeBSD). Signed-off-by: Ilya Maximets Acked-by: William Tu --- lib/dpdk.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/lib/dpdk.c b/lib/dpdk.c index fc58de55a..6f297d918 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -275,7 +275,7 @@ dpdk_init__(const struct smap *ovs_other_config) int result; bool auto_determine = true; int err = 0; - cpu_set_t cpuset; + struct ovs_numa_dump *affinity = NULL; struct svec args = SVEC_EMPTY_INITIALIZER; log_stream = fopencookie(NULL, "w+", dpdk_log_func); @@ -357,22 +357,22 @@ dpdk_init__(const struct smap *ovs_other_config) * lcore for the DPDK Master. */ if (auto_determine) { + const struct ovs_numa_info_core *core; int cpu = 0; /* Get the main thread affinity */ - CPU_ZERO(&cpuset); - err = pthread_getaffinity_np(pthread_self(), sizeof(cpu_set_t), - &cpuset); - if (!err) { - for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { - if (CPU_ISSET(cpu, &cpuset)) { - break; + affinity = ovs_numa_thread_getaffinity_dump(); + if (affinity) { + cpu = INT_MAX; + FOR_EACH_CORE_ON_DUMP (core, affinity) { + if (cpu > core->core_id) { + cpu = core->core_id; } } } else { /* User did not set dpdk-lcore-mask and unable to get current * thread affintity - default to core #0 */ - VLOG_ERR("Thread getaffinity error %d. Using core #0", err); + VLOG_ERR("Thread getaffinity failed. Using core #0"); } svec_add(&args, "-l"); svec_add_nocopy(&args, xasprintf("%d", cpu)); @@ -403,12 +403,9 @@ dpdk_init__(const struct smap *ovs_other_config) svec_destroy(&args); /* Set the main thread affinity back to pre rte_eal_init() value */ - if (auto_determine && !err) { - err = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), - &cpuset); - if (err) { - VLOG_ERR("Thread setaffinity error %d", err); - } + if (affinity) { + ovs_numa_thread_setaffinity_dump(affinity); + ovs_numa_dump_destroy(affinity); } if (result < 0) {