From patchwork Tue Apr 30 20:40:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1093491 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=mellanox.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b="RpNofgVB"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44tth75VJwz9s47 for ; Wed, 1 May 2019 06:40:23 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727300AbfD3UkW (ORCPT ); Tue, 30 Apr 2019 16:40:22 -0400 Received: from mail-eopbgr30081.outbound.protection.outlook.com ([40.107.3.81]:37375 "EHLO EUR03-AM5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727259AbfD3UkS (ORCPT ); Tue, 30 Apr 2019 16:40:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zmBqzyF9pCkfs/DzmH994jOViYWVnJFIwy01Dw2U6EY=; b=RpNofgVBm+tr9LIpFQz4MDkcrI6aGt97hgsSySmsrfYN3R9CLW8AezCW/25W5DGDXA8zQseEWSSIU0F0pUunEeOMBJnmOBMmrtNp5CXYuzeo60MJ/5FiWY0m8sbJWIfYA4hAoXdBhB/iyo9pPlGvm577RgvINo1S4q9dutpttjY= Received: from VI1PR05MB5902.eurprd05.prod.outlook.com (20.178.125.223) by VI1PR05MB6542.eurprd05.prod.outlook.com (20.179.27.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1835.15; Tue, 30 Apr 2019 20:40:14 +0000 Received: from VI1PR05MB5902.eurprd05.prod.outlook.com ([fe80::1d74:be4b:cfe9:59a2]) by VI1PR05MB5902.eurprd05.prod.outlook.com ([fe80::1d74:be4b:cfe9:59a2%5]) with mapi id 15.20.1835.018; Tue, 30 Apr 2019 20:40:14 +0000 From: Saeed Mahameed To: "David S. Miller" CC: "netdev@vger.kernel.org" , Bodong Wang , Parav Pandit , Vu Pham , Saeed Mahameed Subject: [net-next 15/15] net/mlx5: E-Switch, Use atomic rep state to serialize state change Thread-Topic: [net-next 15/15] net/mlx5: E-Switch, Use atomic rep state to serialize state change Thread-Index: AQHU/5TpAazup7hKP0mk4d4d08soQg== Date: Tue, 30 Apr 2019 20:40:14 +0000 Message-ID: <20190430203926.19284-16-saeedm@mellanox.com> References: <20190430203926.19284-1-saeedm@mellanox.com> In-Reply-To: <20190430203926.19284-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.20.1 x-originating-ip: [209.116.155.178] x-clientproxiedby: BYAPR02CA0055.namprd02.prod.outlook.com (2603:10b6:a03:54::32) To VI1PR05MB5902.eurprd05.prod.outlook.com (2603:10a6:803:df::31) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: add7b07e-17d8-4d89-d71d-08d6cdac0bd3 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(4618075)(2017052603328)(7193020); SRVR:VI1PR05MB6542; x-ms-traffictypediagnostic: VI1PR05MB6542: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2150; x-forefront-prvs: 00235A1EEF x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(346002)(376002)(396003)(366004)(39860400002)(189003)(199004)(99286004)(25786009)(102836004)(53936002)(36756003)(386003)(478600001)(186003)(6506007)(7736002)(26005)(2616005)(476003)(446003)(52116002)(6436002)(4326008)(305945005)(5660300002)(76176011)(486006)(66066001)(11346002)(14454004)(107886003)(6512007)(6486002)(68736007)(71200400001)(2906002)(6916009)(81166006)(316002)(1076003)(97736004)(81156014)(66446008)(64756008)(66556008)(50226002)(66476007)(66946007)(73956011)(8936002)(54906003)(256004)(8676002)(3846002)(86362001)(6116002)(71190400001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB6542; H:VI1PR05MB5902.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: PZSuyCg2RLEdYjOivxGU/3kD6fRy0dSo8BumMIOeqHaTku0m4DVSgPJyx54ggPKCNxs4QoQ7oW0f4qFyifa0wstRJWWWqWpUUHV3+wYM2pQegtPNnVpT1WZuXue4rTX8wdqo/mJZOktxo3NHWXc9CpNx8dXhMsGC+dh0elGh6AQo8chPTuFjQEuDolvRBIl05u+RWPFgWWw/nF1S5wl3qZg0OV7gtqtnc7BsFHUDkAqj3AQE6Yl5k7ky5UJDU+E8CyQxwquAGbLXOkg6xOzxEJ56lgU6/4d67XAuWF70f87InkZ4thxdADqu2DOG9/YUo8f3QMJjU09kVwtnRCmfjk+JBzydJbDDv601ufRQXcOWA7DyJ8LLMLAWflKrMXxP+m91BhFNcAjXTb2qu1APTma5/AsZvJHohVLFI6vrUmM= MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: add7b07e-17d8-4d89-d71d-08d6cdac0bd3 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2019 20:40:14.1879 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB6542 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Bodong Wang When the state of rep was introduced, it was also designed to prevent duplicate unloading of the same rep. Considering the following two flows when an eswitch manager is at switchdev mode with n VF reps loaded. +--------------------------------------+--------------------------------+ | cpu-0 | cpu-1 | | -------- | -------- | | mlx5_ib_remove | mlx5_eswitch_disable_sriov | | mlx5_ib_unregister_vport_reps | esw_offloads_cleanup | | mlx5_eswitch_unregister_vport_reps | esw_offloads_unload_all_reps | | __unload_reps_all_vport | __unload_reps_all_vport | +--------------------------------------+--------------------------------+ These two flows will try to unload the same rep. Per original design, once one flow unloads the rep, the state moves to REGISTERED. The 2nd flow will no longer needs to do the unload and bails out. However, as read and write of the state is not atomic, when 1st flow is doing the unload, the state is still LOADED, 2nd flow is able to do the same unload action. Kernel crash will happen. To solve this, driver should do atomic test-and-set for the state. So that only one flow can change the rep state from LOADED to REGISTERED, and proceed to do the actual unloading. Since the state is changing to atomic type, all other read/write should be atomic action as well. Fixes: f121e0ea9586 (net/mlx5: E-Switch, Add state to eswitch vport representors) Signed-off-by: Bodong Wang Reviewed-by: Parav Pandit Reviewed-by: Vu Pham Signed-off-by: Saeed Mahameed --- .../mellanox/mlx5/core/eswitch_offloads.c | 36 +++++++++---------- include/linux/mlx5/eswitch.h | 2 +- 2 files changed, 18 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index e88feaa293f6..e09ae27485ee 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -333,7 +333,7 @@ static int esw_set_global_vlan_pop(struct mlx5_eswitch *esw, u8 val) esw_debug(esw->dev, "%s applying global %s policy\n", __func__, val ? "pop" : "none"); for (vf_vport = 1; vf_vport < esw->enabled_vports; vf_vport++) { rep = &esw->offloads.vport_reps[vf_vport]; - if (rep->rep_if[REP_ETH].state != REP_LOADED) + if (atomic_read(&rep->rep_if[REP_ETH].state) != REP_LOADED) continue; err = __mlx5_eswitch_set_vport_vlan(esw, rep->vport, 0, 0, val); @@ -1277,7 +1277,8 @@ int esw_offloads_init_reps(struct mlx5_eswitch *esw) ether_addr_copy(rep->hw_id, hw_id); for (rep_type = 0; rep_type < NUM_REP_TYPES; rep_type++) - rep->rep_if[rep_type].state = REP_UNREGISTERED; + atomic_set(&rep->rep_if[rep_type].state, + REP_UNREGISTERED); } return 0; @@ -1286,11 +1287,9 @@ int esw_offloads_init_reps(struct mlx5_eswitch *esw) static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep, u8 rep_type) { - if (rep->rep_if[rep_type].state != REP_LOADED) - return; - - rep->rep_if[rep_type].unload(rep); - rep->rep_if[rep_type].state = REP_REGISTERED; + if (atomic_cmpxchg(&rep->rep_if[rep_type].state, + REP_LOADED, REP_REGISTERED) == REP_LOADED) + rep->rep_if[rep_type].unload(rep); } static void __unload_reps_special_vport(struct mlx5_eswitch *esw, u8 rep_type) @@ -1351,16 +1350,15 @@ static int __esw_offloads_load_rep(struct mlx5_eswitch *esw, { int err = 0; - if (rep->rep_if[rep_type].state != REP_REGISTERED) - return 0; - - err = rep->rep_if[rep_type].load(esw->dev, rep); - if (err) - return err; - - rep->rep_if[rep_type].state = REP_LOADED; + if (atomic_cmpxchg(&rep->rep_if[rep_type].state, + REP_REGISTERED, REP_LOADED) == REP_REGISTERED) { + err = rep->rep_if[rep_type].load(esw->dev, rep); + if (err) + atomic_set(&rep->rep_if[rep_type].state, + REP_REGISTERED); + } - return 0; + return err; } static int __load_reps_special_vport(struct mlx5_eswitch *esw, u8 rep_type) @@ -2217,7 +2215,7 @@ void mlx5_eswitch_register_vport_reps(struct mlx5_eswitch *esw, rep_if->get_proto_dev = __rep_if->get_proto_dev; rep_if->priv = __rep_if->priv; - rep_if->state = REP_REGISTERED; + atomic_set(&rep_if->state, REP_REGISTERED); } } EXPORT_SYMBOL(mlx5_eswitch_register_vport_reps); @@ -2232,7 +2230,7 @@ void mlx5_eswitch_unregister_vport_reps(struct mlx5_eswitch *esw, u8 rep_type) __unload_reps_all_vport(esw, max_vf, rep_type); mlx5_esw_for_all_reps(esw, i, rep) - rep->rep_if[rep_type].state = REP_UNREGISTERED; + atomic_set(&rep->rep_if[rep_type].state, REP_UNREGISTERED); } EXPORT_SYMBOL(mlx5_eswitch_unregister_vport_reps); @@ -2252,7 +2250,7 @@ void *mlx5_eswitch_get_proto_dev(struct mlx5_eswitch *esw, rep = mlx5_eswitch_get_rep(esw, vport); - if (rep->rep_if[rep_type].state == REP_LOADED && + if (atomic_read(&rep->rep_if[rep_type].state) == REP_LOADED && rep->rep_if[rep_type].get_proto_dev) return rep->rep_if[rep_type].get_proto_dev(rep); return NULL; diff --git a/include/linux/mlx5/eswitch.h b/include/linux/mlx5/eswitch.h index 96d8435421de..0ca77dd1429c 100644 --- a/include/linux/mlx5/eswitch.h +++ b/include/linux/mlx5/eswitch.h @@ -35,7 +35,7 @@ struct mlx5_eswitch_rep_if { void (*unload)(struct mlx5_eswitch_rep *rep); void *(*get_proto_dev)(struct mlx5_eswitch_rep *rep); void *priv; - u8 state; + atomic_t state; }; struct mlx5_eswitch_rep {