From patchwork Wed Jul 10 11:50:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1130369 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=samsung.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=samsung.com header.i=@samsung.com header.b="pbSysyRr"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45kHp943WGz9s8m for ; Wed, 10 Jul 2019 22:01:03 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id D02DE4433; Wed, 10 Jul 2019 12:00:37 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id A30D94416 for ; Wed, 10 Jul 2019 11:51:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 55D08883 for ; Wed, 10 Jul 2019 11:51:01 +0000 (UTC) Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20190710115059euoutp0181bba019a98f3b1cec871d3b0ce82faa~wCXLAq--K0404804048euoutp01i for ; Wed, 10 Jul 2019 11:50:59 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20190710115059euoutp0181bba019a98f3b1cec871d3b0ce82faa~wCXLAq--K0404804048euoutp01i DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1562759459; bh=bWoKHTVhIvX1B6lqkhOtV6feR0Rvhba9zT3pqJ6t8D0=; h=From:To:Cc:Subject:Date:References:From; b=pbSysyRrA4VGTdrXxfbMDcpayrC7R2qHOdBfunSMSkWkau/X3GoVR9R9lPegtFRLJ 3GrQhwZ3+rOEciUOIAp9j2w8+MedCIwviqQrpYGHiai/dPnptp3OGPzbt0/onALUAb u2iyUnYq4nIG+8DqO9kNp2XMVRbWA5TfocKCY/UA= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20190710115058eucas1p29b34f26f79264fcbf03dd10afb5031ee~wCXKoEjVr2419324193eucas1p2j; Wed, 10 Jul 2019 11:50:58 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id E7.10.04298.221D52D5; Wed, 10 Jul 2019 12:50:58 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20190710115057eucas1p249c2ff40647e62636a73b9eb16487709~wCXJ30EqM2422124221eucas1p2g; Wed, 10 Jul 2019 11:50:57 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20190710115057eusmtrp26de026c515250976b38cc89121f3d665~wCXJpxFcV2010620106eusmtrp26; Wed, 10 Jul 2019 11:50:57 +0000 (GMT) X-AuditID: cbfec7f2-f13ff700000010ca-2d-5d25d1222d4b Received: from eusmtip2.samsung.com ( [203.254.199.222]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 4F.33.04140.121D52D5; Wed, 10 Jul 2019 12:50:57 +0100 (BST) Received: from imaximets.rnd.samsung.ru (unknown [106.109.129.180]) by eusmtip2.samsung.com (KnoxPortal) with ESMTPA id 20190710115057eusmtip22eb32b1bc88e4569101f026d59f7e5ef~wCXJLaWPN0527405274eusmtip2O; Wed, 10 Jul 2019 11:50:57 +0000 (GMT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 10 Jul 2019 14:50:52 +0300 Message-Id: <20190710115052.28451-1-i.maximets@samsung.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKIsWRmVeSWpSXmKPExsWy7djPc7pKF1VjDabdk7XYvqKLzeLOlZ9s Flfaf7JbbHx4ltVi7aEP7BZzPz1ndGDzWLznJZPHs5v/GT3e77vK5tG3ZRVjAEsUl01Kak5m WWqRvl0CV8aidVOYCvZIVmycsZG1gXGLUBcjJ4eEgInEggVtzF2MXBxCAisYJWY8m8sMkhAS +MIosfmnPkTiM6PE6aeTmWA6Ol4dY4dILGeUOLn+NRuE84NR4t/BX6wgVWwCOhKnVh9hBLFF BKQlXve+AYszC5xglNh1xgzEFhawlrg1cS0biM0ioCpxdfZzMJsXKP73zV52iG3yEqs3HAC7 T0LgBJvE8VX3WCESLhLTLs9ghrCFJV4d3wLVICPxf+d8qFPrJe63vGSEaO5glJh+6B9Uwl5i y+tzQA0cQBdpSqzfpQ8RdpTYtG06G0hYQoBP4sZbQYib+SQmbZvODBHmlehogwadisTvg8uh LpCSuPnuM9QFHhIvjtxjgYRirMTWI8sYJzDKzULYtYCRcRWjeGppcW56arFhXmq5XnFibnFp Xrpecn7uJkZg5J/+d/zTDsavl5IOMQpwMCrx8Eq0qMQKsSaWFVfmHmKU4GBWEuHd564cK8Sb klhZlVqUH19UmpNafIhRmoNFSZy3muFBtJBAemJJanZqakFqEUyWiYNTqoHR+rbSLr/sSMHU 191VjnprW+dwibi9deLWYFS05IywqPK+mnj99pq33cKadZZaV5Y9nN72rvjtK2nTJUL/bsQn vCv+yyt3hWFZ8jPRmoDWrVrrtCemsthOe7ciZpvG/l4f9+a5+79OmMj08OmxJBO1bYw5q3nV bEssfOeapIe953/1j8/3whQlluKMREMt5qLiRAAb1HnD+AIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrLLMWRmVeSWpSXmKPExsVy+t/xe7qKF1VjDSauULfYvqKLzeLOlZ9s Flfaf7JbbHx4ltVi7aEP7BZzPz1ndGDzWLznJZPHs5v/GT3e77vK5tG3ZRVjAEuUnk1RfmlJ qkJGfnGJrVK0oYWRnqGlhZ6RiaWeobF5rJWRqZK+nU1Kak5mWWqRvl2CXsaidVOYCvZIVmyc sZG1gXGLUBcjJ4eEgIlEx6tj7F2MXBxCAksZJbbOPsUGkZCS+PHrAiuELSzx51oXG0TRN0aJ H/OeMoMk2AR0JE6tPsIIYosISEu87n3DClLELHCGUeLctdVgk4QFrCVuTVwLZrMIqEpcnf0c zOYFiv99s5cdYoO8xOoNB5gnMPIsYGRYxSiSWlqcm55bbKRXnJhbXJqXrpecn7uJERh02479 3LKDsetd8CFGAQ5GJR5eiRaVWCHWxLLiytxDjBIczEoivPvclWOFeFMSK6tSi/Lji0pzUosP MZoCLZ/ILCWanA+MiLySeENTQ3MLS0NzY3NjMwslcd4OgYMxQgLpiSWp2ampBalFMH1MHJxS DYxTDod+e3P8Yg171qR8roz/jZ+/nbT3sT2Q28V97In9d5tmhZDwff7yWo96X0vd12h31jmv u/CQzOHrc/r0lbpjk+eGvj+77a9I78+ztqc+z7jlIDkx9ptg4ldGjbTvIdlhYefeZPzym3X3 2kLT2OVhq235g+/fvPrQf8+Cu9c/buV9ZvamRGSzEktxRqKhFnNRcSIAZVRFEVACAAA= X-CMS-MailID: 20190710115057eucas1p249c2ff40647e62636a73b9eb16487709 X-Msg-Generator: CA X-RootMTR: 20190710115057eucas1p249c2ff40647e62636a73b9eb16487709 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20190710115057eucas1p249c2ff40647e62636a73b9eb16487709 References: X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ilya Maximets , David Marchand Subject: [ovs-dev] [PATCH] dpif-netdev: Clarify PMD reloading scheme. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It became more complicated, hence needs to be documented. Signed-off-by: Ilya Maximets Reviewed-by: David Marchand --- Applicable on top of "Quicker pmd threads reloads" patch-set: https://patchwork.ozlabs.org/project/openvswitch/list/?series=118588&state=* lib/dpif-netdev.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 647a8ee4b..c5ffc72f5 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -682,10 +682,49 @@ struct dp_netdev_pmd_thread { struct seq *reload_seq; uint64_t last_reload_seq; + + /* These are atomic variables used as a synchronization and configuration + * points for thread reload/exit. + * + * 'reload' atomic is the main one and it's used as a memory + * synchronization point for all other knobs and data. + * + * For a thread that requests PMD reload: + * + * * All changes that should be visible to the PMD thread must be made + * before setting the 'reload'. These changes could use any memory + * ordering model including 'relaxed'. + * * Setting the 'reload' atomic should occur in the same thread where + * all other PMD configuration options updated. + * * Setting the 'reload' atomic should be done with 'release' memory + * ordering model or stricter. This will guarantee that all previous + * changes (including non-atomic and 'relaxed') will be visible to + * the PMD thread. + * * To check that reload is done, thread should poll the 'reload' atomic + * to become 'false'. Polling should be done with 'acquire' memory + * ordering model or stricter. This ensures that PMD thread completed + * the reload process. + * + * For the PMD thread: + * + * * PMD thread should read 'reload' atomic with 'acquire' memory + * ordering model or stricter. This will guarantee that all changes + * made before setting the 'reload' in the requesting thread will be + * visible to the PMD thread. + * * All other configuration data could be read with any memory + * ordering model (including non-atomic and 'relaxed') but *only after* + * reading the 'reload' atomic set to 'true'. + * * When the PMD reload done, PMD should (optionally) set all the below + * knobs except the 'reload' to their default ('false') values and + * (mandatory), as the last step, set the 'reload' to 'false' using + * 'release' memory ordering model or stricter. This will inform the + * requesting thread that PMD has completed a reload cycle. + */ atomic_bool reload; /* Do we need to reload ports? */ atomic_bool wait_for_reload; /* Can we busy wait for the next reload? */ atomic_bool reload_tx_qid; /* Do we need to reload static_tx_qid? */ atomic_bool exit; /* For terminating the pmd thread. */ + pthread_t thread; unsigned core_id; /* CPU core id of this pmd thread. */ int numa_id; /* numa node id of this pmd thread. */