From patchwork Wed Nov 1 14:49:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1857984 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SL90X6Q0Qz1yQn for ; Thu, 2 Nov 2023 01:50:20 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1qyCXo-0001oe-FT; Wed, 01 Nov 2023 14:50:08 +0000 Received: from mail-mw2nam10on2080.outbound.protection.outlook.com ([40.107.94.80] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1qyCXe-0001mr-KB for kernel-team@lists.ubuntu.com; Wed, 01 Nov 2023 14:49:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GiB2lr5yCDFaD0wTVuaq4e/N9JSRlNP/Mff0WoZIg5QsXo1uV6RQD2YUOtAvriSk6Q/2GH0D7iluec8Pj3pH8RYeHMtf02powOn4F47H++D44muULMdqcOT1zM9Is4MhinlUSPUMvZOp5v15gq0kfkXpxZHL7N7+8z8zuOI3estn8UHBfO1zSIsUpLelYZXbfR8qwDjAPVF1jyIjEvsWzPMSqsTGiwn9Dqq0b99bzKar5T6qogDLAwIHehbiLPtknbA4Z+biiHTFgHWs87SX9a5UgItbxqT2UFKJZPS2uXrYw/bVcn2CXq9NPAIJOmNXPqVy/UiBUPrJRGsb56UKUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gsu+2GZRajRvTsBB5CNUBS4bA3JfuO02au4exMdvmcY=; b=oFu/1oVRtOg1CKWXqO3CN6Zxb//S7kjUA3ZfH2hQez8WUPuodQhadHD7SDHmcR5/7NEgl27el5yfrgFTS4N/9+Kg+rVBoq68TKcPS/r4ZBodGr6QGs3h9mL11JW+AkIY/zQ6Q1K7LqjKoS+MbiAzfpMi4nxqpUjEiESWmeaPa7sVcFx/E1jgWdrBZ81v0LeTO9g//Z6HPIPPRp9mQ4NvmobUVlvMN/BxUwkwwthuhx4m88F+y48iZp8iNGWYZYfIN6OtXDGA0+EgjntJUtCYG9tc0CutL8gTk64FwIrBoFrNutCzXDVcqijVN8yEEn5ii4LHeaAdtmARxRGJEHzEaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=lists.ubuntu.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) Received: from CY5PR20CA0008.namprd20.prod.outlook.com (2603:10b6:930:3::14) by SJ1PR12MB6074.namprd12.prod.outlook.com (2603:10b6:a03:45f::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.29; Wed, 1 Nov 2023 14:49:55 +0000 Received: from CY4PEPF0000EE3D.namprd03.prod.outlook.com (2603:10b6:930:3:cafe::b7) by CY5PR20CA0008.outlook.office365.com (2603:10b6:930:3::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.19 via Frontend Transport; Wed, 1 Nov 2023 14:49:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CY4PEPF0000EE3D.mail.protection.outlook.com (10.167.242.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.19 via Frontend Transport; Wed, 1 Nov 2023 14:49:54 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 1 Nov 2023 07:49:53 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 1 Nov 2023 07:49:52 -0700 Received: from witu-mlt.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 1 Nov 2023 07:49:52 -0700 From: William Tu To: Subject: [SRU][J:linux-bluefield][PATCH 1/7] net: devlink: avoid false DEADLOCK warning reported by lockdep Date: Wed, 1 Nov 2023 07:49:45 -0700 Message-ID: <20231101144951.26198-2-witu@nvidia.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20231101144951.26198-1-witu@nvidia.com> References: <20231101144951.26198-1-witu@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3D:EE_|SJ1PR12MB6074:EE_ X-MS-Office365-Filtering-Correlation-Id: 83f8e914-abaa-47ab-6c78-08dbdae9cfb0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PzJ47wuP59QR+DHHGkmOqM05fkmxzUzjBepYhc6z21uVWbsZGkSjSfBasUCQU8XjExQF1t9BMJyNQeoKF5TNgK8UhnlBNZTTGBCpj5oPxlVNLoQRNZx9qsRAuxoaj8BNvs+RSt25ScOaFnT1BOuLr4IVnVpEqm1zWvvpz3RXecl54/2acnfATEW8XDd0cTxMva1CNVCBkGS1nWYRupi9ODmFKkIK4BSd2t9JX/phsrV6xltUcG/VuyobI35ltjtca36F0sdxo1NawnRHcPhIE34c3i8QWFH3db/Z5KGoJ66PVRPopvNFeqecCPCjfL43umsIUJlIBWv0QJIoockl5ZX9yrdSO7csVtrBdyNT+rTlLwXHDsLTCAc5AxVrqeqohmNQQsW+5m5SkRcwxgZOLTQkuw0t5iIjfZktN/xaidYpF2A5wwNc9oMkCc1HtvEilgXvHLfK3tQxIqMUjePZSmXDYCjwhAOTk36u5gw0gbh9IeCTETGvqVceQuRaYHYwjW/QTflhmRI1T2UbnFDK4n91n7ubRRBpD/Z3YLsTsFoTjeTOFD3eYdU+vqqry8OV9B6RkLcsklPZDug20bSnJBdAZ92RwKHWjmpNYTKVHVsoKgNKn4rVbAFGvHL6zCWQ8lIFD2szZ6sAfa4Fge+OfEpLJSIE40fF5R77XKBjUCNSarEqWce7QWaojGB/4/R3/XjwU9GxmDtUHExrj5luoK/S/0Of1oHmmWifNLrWLDoZZ7DR8HQ59PverxvzEEEHB1lYua6AFpm7xwNvHjREwhZ5D9PCvpJx5wwPoV3x+MW1GtlZAyfA0RrmgBQ1FcK3fKBTZFAqMZw50RkOEFVZcAc2Tlrv/yXZL2sVORAMy9Q= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(376002)(346002)(396003)(136003)(39860400002)(230922051799003)(230273577357003)(230173577357003)(1800799009)(186009)(451199024)(64100799003)(82310400011)(46966006)(40470700004)(36840700001)(40460700003)(40480700001)(426003)(336012)(36756003)(1076003)(86362001)(107886003)(2616005)(36860700001)(26005)(6666004)(7696005)(478600001)(356005)(82740400003)(47076005)(966005)(7636003)(83380400001)(2906002)(70206006)(54906003)(316002)(41300700001)(6916009)(70586007)(4326008)(5660300002)(8676002)(8936002)(46800400005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2023 14:49:54.8800 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 83f8e914-abaa-47ab-6c78-08dbdae9cfb0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3D.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6074 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vlad@nvidia.com, bodong@nvidia.com, majd@nvidia.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Moshe Shemesh BugLink: https://bugs.launchpad.net/bugs/2042455 Add a lock_class_key per devlink instance to avoid DEADLOCK warning by lockdep, while locking more than one devlink instance in driver code, for example in opening VFs flow. Kernel log: [ 101.433802] ============================================ [ 101.433803] WARNING: possible recursive locking detected [ 101.433810] 5.19.0-rc1+ #35 Not tainted [ 101.433812] -------------------------------------------- [ 101.433813] bash/892 is trying to acquire lock: [ 101.433815] ffff888127bfc2f8 (&devlink->lock){+.+.}-{3:3}, at: probe_one+0x3c/0x690 [mlx5_core] [ 101.433909] but task is already holding lock: [ 101.433910] ffff888118f4c2f8 (&devlink->lock){+.+.}-{3:3}, at: mlx5_core_sriov_configure+0x62/0x280 [mlx5_core] [ 101.433989] other info that might help us debug this: [ 101.433990] Possible unsafe locking scenario: [ 101.433991] CPU0 [ 101.433991] ---- [ 101.433992] lock(&devlink->lock); [ 101.433993] lock(&devlink->lock); [ 101.433995] *** DEADLOCK *** [ 101.433996] May be due to missing lock nesting notation [ 101.433996] 6 locks held by bash/892: [ 101.433998] #0: ffff88810eb50448 (sb_writers#3){.+.+}-{0:0}, at: ksys_write+0xf3/0x1d0 [ 101.434009] #1: ffff888114777c88 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x20d/0x520 [ 101.434017] #2: ffff888102b58660 (kn->active#231){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x230/0x520 [ 101.434023] #3: ffff888102d70198 (&dev->mutex){....}-{3:3}, at: sriov_numvfs_store+0x132/0x310 [ 101.434031] #4: ffff888118f4c2f8 (&devlink->lock){+.+.}-{3:3}, at: mlx5_core_sriov_configure+0x62/0x280 [mlx5_core] [ 101.434108] #5: ffff88812adce198 (&dev->mutex){....}-{3:3}, at: __device_attach+0x76/0x430 [ 101.434116] stack backtrace: [ 101.434118] CPU: 5 PID: 892 Comm: bash Not tainted 5.19.0-rc1+ #35 [ 101.434120] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014 [ 101.434130] Call Trace: [ 101.434133] [ 101.434135] dump_stack_lvl+0x57/0x7d [ 101.434145] __lock_acquire.cold+0x1df/0x3e7 [ 101.434151] ? register_lock_class+0x1880/0x1880 [ 101.434157] lock_acquire+0x1c1/0x550 [ 101.434160] ? probe_one+0x3c/0x690 [mlx5_core] [ 101.434229] ? lockdep_hardirqs_on_prepare+0x400/0x400 [ 101.434232] ? __xa_alloc+0x1ed/0x2d0 [ 101.434236] ? ksys_write+0xf3/0x1d0 [ 101.434239] __mutex_lock+0x12c/0x14b0 [ 101.434243] ? probe_one+0x3c/0x690 [mlx5_core] [ 101.434312] ? probe_one+0x3c/0x690 [mlx5_core] [ 101.434380] ? devlink_alloc_ns+0x11b/0x910 [ 101.434385] ? mutex_lock_io_nested+0x1320/0x1320 [ 101.434388] ? lockdep_init_map_type+0x21a/0x7d0 [ 101.434391] ? lockdep_init_map_type+0x21a/0x7d0 [ 101.434393] ? __init_swait_queue_head+0x70/0xd0 [ 101.434397] probe_one+0x3c/0x690 [mlx5_core] [ 101.434467] pci_device_probe+0x1b4/0x480 [ 101.434471] really_probe+0x1e0/0xaa0 [ 101.434474] __driver_probe_device+0x219/0x480 [ 101.434478] driver_probe_device+0x49/0x130 [ 101.434481] __device_attach_driver+0x1b8/0x280 [ 101.434484] ? driver_allows_async_probing+0x140/0x140 [ 101.434487] bus_for_each_drv+0x123/0x1a0 [ 101.434489] ? bus_for_each_dev+0x1a0/0x1a0 [ 101.434491] ? lockdep_hardirqs_on_prepare+0x286/0x400 [ 101.434494] ? trace_hardirqs_on+0x2d/0x100 [ 101.434498] __device_attach+0x1a3/0x430 [ 101.434501] ? device_driver_attach+0x1e0/0x1e0 [ 101.434503] ? pci_bridge_d3_possible+0x1e0/0x1e0 [ 101.434506] ? pci_create_resource_files+0xeb/0x190 [ 101.434511] pci_bus_add_device+0x6c/0xa0 [ 101.434514] pci_iov_add_virtfn+0x9e4/0xe00 [ 101.434517] ? trace_hardirqs_on+0x2d/0x100 [ 101.434521] sriov_enable+0x64a/0xca0 [ 101.434524] ? pcibios_sriov_disable+0x10/0x10 [ 101.434528] mlx5_core_sriov_configure+0xab/0x280 [mlx5_core] [ 101.434602] sriov_numvfs_store+0x20a/0x310 [ 101.434605] ? sriov_totalvfs_show+0xc0/0xc0 [ 101.434608] ? sysfs_file_ops+0x170/0x170 [ 101.434611] ? sysfs_file_ops+0x117/0x170 [ 101.434614] ? sysfs_file_ops+0x170/0x170 [ 101.434616] kernfs_fop_write_iter+0x348/0x520 [ 101.434619] new_sync_write+0x2e5/0x520 [ 101.434621] ? new_sync_read+0x520/0x520 [ 101.434624] ? lock_acquire+0x1c1/0x550 [ 101.434626] ? lockdep_hardirqs_on_prepare+0x400/0x400 [ 101.434630] vfs_write+0x5cb/0x8d0 [ 101.434633] ksys_write+0xf3/0x1d0 [ 101.434635] ? __x64_sys_read+0xb0/0xb0 [ 101.434638] ? lockdep_hardirqs_on_prepare+0x286/0x400 [ 101.434640] ? syscall_enter_from_user_mode+0x1d/0x50 [ 101.434643] do_syscall_64+0x3d/0x90 [ 101.434647] entry_SYSCALL_64_after_hwframe+0x46/0xb0 [ 101.434650] RIP: 0033:0x7f5ff536b2f7 [ 101.434658] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 101.434661] RSP: 002b:00007ffd9ea85d58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 101.434664] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f5ff536b2f7 [ 101.434666] RDX: 0000000000000002 RSI: 000055c4c279e230 RDI: 0000000000000001 [ 101.434668] RBP: 000055c4c279e230 R08: 000000000000000a R09: 0000000000000001 [ 101.434669] R10: 000055c4c283cbf0 R11: 0000000000000246 R12: 0000000000000002 [ 101.434670] R13: 00007f5ff543d500 R14: 0000000000000002 R15: 00007f5ff543d700 [ 101.434673] Signed-off-by: Moshe Shemesh Signed-off-by: Jiri Pirko Signed-off-by: Jakub Kicinski (cherry picked from commit e26fde2f5befad0951fe6345403616bf51e901be) Signed-off-by: William Tu --- net/core/devlink.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/net/core/devlink.c b/net/core/devlink.c index 4330fa6f5852..842c203a2a50 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -66,6 +66,7 @@ struct devlink { * port, sb, dpipe, resource, params, region, traps and more. */ struct mutex lock; + struct lock_class_key lock_key; u8 reload_failed:1; refcount_t refcount; struct completion comp; @@ -9733,7 +9734,9 @@ struct devlink *devlink_alloc_ns(const struct devlink_ops *ops, INIT_LIST_HEAD(&devlink->trap_list); INIT_LIST_HEAD(&devlink->trap_group_list); INIT_LIST_HEAD(&devlink->trap_policer_list); + lockdep_register_key(&devlink->lock_key); mutex_init(&devlink->lock); + lockdep_set_class(&devlink->lock, &devlink->lock_key); mutex_init(&devlink->reporters_lock); mutex_init(&devlink->linecards_lock); refcount_set(&devlink->refcount, 1); @@ -9880,6 +9883,7 @@ void devlink_free(struct devlink *devlink) mutex_destroy(&devlink->linecards_lock); mutex_destroy(&devlink->reporters_lock); mutex_destroy(&devlink->lock); + lockdep_unregister_key(&devlink->lock_key); WARN_ON(!list_empty(&devlink->trap_policer_list)); WARN_ON(!list_empty(&devlink->trap_group_list)); WARN_ON(!list_empty(&devlink->trap_list));