From patchwork Tue Oct 19 10:58:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543203 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=oagh23Ey; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW2y1ZFbz9sP7 for ; Tue, 19 Oct 2021 21:59:22 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235211AbhJSLBc (ORCPT ); Tue, 19 Oct 2021 07:01:32 -0400 Received: from mail-dm6nam12on2080.outbound.protection.outlook.com ([40.107.243.80]:18529 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S230117AbhJSLBc (ORCPT ); Tue, 19 Oct 2021 07:01:32 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j9aOe8B9+D5W+4SCEExNL865sEcECrOkZBy2Jg11ryN4p1JrNj/yuMoXjQfJ1J8KLbwOD0GYE6ToQa+rg9hLKAQ2bDiBNMuZPlCdLKWRiHzWe8sIvPPJ0VNxQNHDLF/uV1BZ+OXpFC0iP7CvUoUFO1k49Jn8j6P7ej+RSKRRL01lYLSM+jS+ExRFEKflWpdgWtC/P/hh8Ns+qAwXyGEnn0zz1+xfYpc5b23zesQSQcvHcKt5N4g9JZVTk4edmzi8wJAcnhNC2pebRFO94S3VHXL6ncG6N2/pwRA/SIO4kWmGcnSmbvLpKZoWpHq7w7VVZDBfqKgN3CybvxK9COYaNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4WQC7MbzY4SkKjUkPCgiLuU12kLGzfDWQ2BT0ZKYN6s=; b=gwdenvJQOR+y+CX67q3V3rLfmrv47GkvVYtHaK95f0ITjldOregAT7deq7hMWdb92Gs8PmRXiRcL3Sz29IuX7R2oFs4JgpT8R6eJRy0CU5iYQHYw0dUhLRgGk5RvFOz27e9bEZujheJATxbYetwLbnlkggi4QXWWKRrJ7wOo7IJExnwvtD69FsFXETvG4xXWcKSB0Gq/xFWRj7o+c/ScE6WpCiQmlGHdnmnT+ih5zwIpKUESrbqZvmlLf/1JyJxIGhx7VGVDSC8ZQkkLT0128iriF4llwyMvHKnmZQyHAo5Z8/2sDE5Kn/G/ReMdydq1sE6kWrzc/ffxtQMY/N07VA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4WQC7MbzY4SkKjUkPCgiLuU12kLGzfDWQ2BT0ZKYN6s=; b=oagh23EyK636eF1Ha/Uiws+8fKg/0uFy7tPGUNstujGUrpyv68q3/30vo5814NxtxWvJH/OEAT/7YCYndS7yRAg9UhUu1sN/uWO75QHK6E6cvI0OfQW+VJ0Oxv/5LtTln64ppO4aaHrMRIfuRz/1fbozxFDybBfaZBF+Fvt96agL660BLQhzww/ivQPumUlgxbMQR/nmRhA5oPHAc+oy+E6xJ/qXa4aiuKCs5dpSWC3eDPRvxuEmNbkGtb+UZ6aU3bs6yOU5Htf9UBKKWvVEBLuhgicjK5ABG3hWSNujnzeQBxRC9jOd57aW85G1yAOpc7vVgK0KTNdWsurxsWvkjA== Received: from BN0PR04CA0008.namprd04.prod.outlook.com (2603:10b6:408:ee::13) by DM6PR12MB4562.namprd12.prod.outlook.com (2603:10b6:5:2aa::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.15; Tue, 19 Oct 2021 10:59:18 +0000 Received: from BN8NAM11FT010.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::34) by BN0PR04CA0008.outlook.office365.com (2603:10b6:408:ee::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by BN8NAM11FT010.mail.protection.outlook.com (10.13.177.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:17 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 03:59:13 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:10 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 01/14] PCI/IOV: Add pci_iov_vf_id() to get VF index Date: Tue, 19 Oct 2021 13:58:25 +0300 Message-ID: <20211019105838.227569-2-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ebdae487-bd45-4198-4fd9-08d992ef7f42 X-MS-TrafficTypeDiagnostic: DM6PR12MB4562: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VdmVEh/aFKsKFAOMC6EPfRj4Czc18wRjXpv0iI07nrHHhLWGngDRt4IR7Ffr0wkeTRwicVBN+K7S6XigVVYRqkL7/nSc2rZAfVhQ5Cse+FI3QxratnuG+WbTUGIBemZUfitK8ZZYVS6MeRug3AQDtlLjyDm+DCm493CLrbcJTmA7bGWRzjaYl+KN1vrz1zV0R8CD1NBV1tE7S8MkvLfqhIXp2/DuynBKFO9iELM4iqdqFWaHfltMuPJi4K3uV/OeiY8iWX7Uyh3zX6td/yGsmU69oBW7qTdSZvyC7USb3v7ppB5n44xorNST32vi/qpujgfN1PAb/0W0EG8yaf/xEsWz1VpuRbrIfVN43IxyB1W5ilJZLgsc+ErGWvTbFGJb1ndTdA3YrJgHwA76qKtEhiywH9qdgWS9J/280ZTD5HPOojunD7Jdlm/ZrwZ7KTl0MDoU7pbYdNI42QMqYidwAZr4MpZmhdea5FpMApsNQe2hZvEIhApXWgvXENebKb1nIH5z31+bgqA5qobxjDCara0AYIgaNPIFH02RrRPi+O/aWfZ5hkemIJQuVhH5NWv/HwZsoQbzPXlrsKppvCmwnw2oitUIDoYWgYDjMi2q62UBoKmILOfGdvhBp56Ipv+JVmFLcbWHC2lzek1HSZ61QUe4CSzXcSCYu1yMo6ire7rBhAfkgb4lV9okESWZMCTyZaK+OfONYeqGpn8orCnopQ== X-Forefront-Antispam-Report: CIP:216.228.112.32;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid01.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(36756003)(110136005)(82310400003)(6636002)(86362001)(8936002)(5660300002)(54906003)(336012)(8676002)(2616005)(1076003)(426003)(26005)(107886003)(6666004)(508600001)(186003)(36860700001)(7696005)(4326008)(47076005)(70206006)(70586007)(2906002)(7636003)(316002)(356005)(83380400001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:17.4480 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ebdae487-bd45-4198-4fd9-08d992ef7f42 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT010.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4562 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Jason Gunthorpe The PCI core uses the VF index internally, often called the vf_id, during the setup of the VF, eg pci_iov_add_virtfn(). This index is needed for device drivers that implement live migration for their internal operations that configure/control their VFs. Specifically, mlx5_vfio_pci driver that is introduced in coming patches from this series needs it and not the bus/device/function which is exposed today. Add pci_iov_vf_id() which computes the vf_id by reversing the math that was used to create the bus/device/function. Signed-off-by: Yishai Hadas Signed-off-by: Jason Gunthorpe Acked-by: Bjorn Helgaas Signed-off-by: Leon Romanovsky --- drivers/pci/iov.c | 14 ++++++++++++++ include/linux/pci.h | 8 +++++++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index dafdc652fcd0..e7751fa3fe0b 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -33,6 +33,20 @@ int pci_iov_virtfn_devfn(struct pci_dev *dev, int vf_id) } EXPORT_SYMBOL_GPL(pci_iov_virtfn_devfn); +int pci_iov_vf_id(struct pci_dev *dev) +{ + struct pci_dev *pf; + + if (!dev->is_virtfn) + return -EINVAL; + + pf = pci_physfn(dev); + return (((dev->bus->number << 8) + dev->devfn) - + ((pf->bus->number << 8) + pf->devfn + pf->sriov->offset)) / + pf->sriov->stride; +} +EXPORT_SYMBOL_GPL(pci_iov_vf_id); + /* * Per SR-IOV spec sec 3.3.10 and 3.3.11, First VF Offset and VF Stride may * change when NumVFs changes. diff --git a/include/linux/pci.h b/include/linux/pci.h index cd8aa6fce204..2337512e67f0 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2153,7 +2153,7 @@ void __iomem *pci_ioremap_wc_bar(struct pci_dev *pdev, int bar); #ifdef CONFIG_PCI_IOV int pci_iov_virtfn_bus(struct pci_dev *dev, int id); int pci_iov_virtfn_devfn(struct pci_dev *dev, int id); - +int pci_iov_vf_id(struct pci_dev *dev); int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn); void pci_disable_sriov(struct pci_dev *dev); @@ -2181,6 +2181,12 @@ static inline int pci_iov_virtfn_devfn(struct pci_dev *dev, int id) { return -ENOSYS; } + +static inline int pci_iov_vf_id(struct pci_dev *dev) +{ + return -ENOSYS; +} + static inline int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn) { return -ENODEV; } From patchwork Tue Oct 19 10:58:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543204 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=qofYD30V; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3669XCz9sP7 for ; Tue, 19 Oct 2021 21:59:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235180AbhJSLBl (ORCPT ); Tue, 19 Oct 2021 07:01:41 -0400 Received: from mail-bn8nam08on2080.outbound.protection.outlook.com ([40.107.100.80]:63969 "EHLO NAM04-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235252AbhJSLBe (ORCPT ); Tue, 19 Oct 2021 07:01:34 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SolwTWCne9TDjGsxJJQu1JOlCpTNvkjy4UaSI0TkuSckxjemk0267U+kalvNf9zxN1x7eIXGMjxOOOvYCRIK8G/++jqkrCUqn/KTaflIxYgcdM9IdrDfWXk8pazJCK7hVhJzQQVxCOjzA/mtJv5f3OPe32JvM1pC51e+kmllj+6t72/sLYxf2+Aq5cSJzKkobB9I8QuphJYTNtRu9u28xArWjZP8jfQFAWzeTim6Y1gM6qgpmcb/wnIqNy6H6HHDGPVbyGPAtEoWl4JSY8g4qS8y74HAeiIDLZU8ldsZEmHDMAHQM8WaNNyRIRJ+uwsCCA+bOEKxsmscDxZmqQjrkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eawxF+jFRBAH1P9hWq5BE145QVmYPfYNp1U9MJCuY0E=; b=UuoIsmC52MJmg+MdlxDRGG/Lt1fo3cTCErLU0RYlGs01wYXnZ66SnRxIF1EZ4T5I0/qujMCzqsNgoOd6sRCJoBFuQSN2hDcBQlIgX1A1NwAfO1p80n6Sa04B3w9XIesQOaq8BVj1NMX3Gg3SdH61OBvFwp2lwD2+CmD6aJS0jC+zGoBs0Hpza/sc18Dpm67WN6/PR00AJ7ZKa469nzWM1Pp+hDb/p1dfTTh3T28Kfj4a+0BTvJ7njqfYoYBSMwpq8xs34z6g5CuBjf+JCrvZ3eK6F1zTkpgMtjkUdd9g5pMhi5KQWt00uKDrzXXqIIGA1k0INaSuiu+Jg2Qb3Ab29g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eawxF+jFRBAH1P9hWq5BE145QVmYPfYNp1U9MJCuY0E=; b=qofYD30VHhbrNkCgl0CvGSyXxE6LfQjeT4ePym/WCjKtVRt7oHdNbjjVCV47cvxko8A05j5XrBjcqMjWNpYxj409a9sK9iCpM5qEzFAsX/LPUNbfbv31bAJkn09EphpDUD5e7C38zROJiJlzRj/WhS+hqM21MeuFrSRl226XACYVbzQzIx59JX0I54dOQBSsGLehhRvexad7/tf/aKz/vUkDxBySJvn1kogzlQSp0rzXQC5V1a2HnhUzwCvUXXQAbrlp92o7FazvV/D8hvTvaooY9yrqz6CuG4tREfZVYamCB8gCtteNdo55ivWjCDVKVLhVGbe/PSS9a1YINikNMQ== Received: from BN9PR03CA0389.namprd03.prod.outlook.com (2603:10b6:408:f7::34) by BL0PR12MB4849.namprd12.prod.outlook.com (2603:10b6:208:1c2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:20 +0000 Received: from BN8NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f7:cafe::73) by BN9PR03CA0389.outlook.office365.com (2603:10b6:408:f7::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT018.mail.protection.outlook.com (10.13.176.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:17 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:17 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:14 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 02/14] net/mlx5: Reuse exported virtfn index function call Date: Tue, 19 Oct 2021 13:58:26 +0300 Message-ID: <20211019105838.227569-3-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4ef018d5-4445-494a-741b-08d992ef7f5a X-MS-TrafficTypeDiagnostic: BL0PR12MB4849: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kXCy1exH5yFc0/XSmfRy+6tYDBa5YLOkmjCAUpn5vgTfqk8kwJg/F8PD/r3Qd2Z+rdfSlAPRjP7SDmcBwNDBVP32a65iTOpW1vmtMkOBvkEzy2H6NOeHmtYBItqaHGAOBqaZS+aU8kSJ15+vXjNCF0uM9vQevxBxfjGhsixunbw6JnBwzJp6UXT4u8km46ZhgW0qgSmu3ukQUv0u8zouYwo5HgHC0Jda9sOV0xZcaL55B4P0iB/elCP+veaJRGRXQzEKY6JEKHHT78JMuVKzQIDPjrwKJzdBWY2JHR8aR+HXL2i/QfDYe6pcxfQ/EAIFhPdN2U+bdofz2p0IcJjZnnPltQ1dNT6/H5yBR+t+/Jxwhzj/KvT//VX8HaCMPf7iRZiNpU7jRWtn81j92hMgeODj5bUdReSBOkDRNj48oQ2/Y5qxY8RpSigdtYsXDJScOiu7WEeeoKuhkr+EUF27ulCvnQWqmBq/BFV7k61nDpWWzJyllgvQ7+n2Qk7R5CxfCVhKEj+ze3mloYwp6PQfhr3fqFPhuc8wBsLPEFwbaZa9pPLMbaQT5rAa+aw3jpfCQB2No/3FtuE/C0yNaTAXyg8KvBFHyzueqx7Nq7uijlix5WtXFQmm4YyxtLsojZz75nBzqCLRCd6kY63rx0kS41rntKEuHuPP1YlVbpJE+9Hv+mIV3Rq5TgbhLP0vU3be8KXZ7pqM6z8sjUoB/TckKA== X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(6636002)(70206006)(5660300002)(6666004)(186003)(70586007)(336012)(26005)(110136005)(36756003)(7696005)(47076005)(107886003)(508600001)(8936002)(8676002)(1076003)(426003)(4326008)(7636003)(54906003)(83380400001)(82310400003)(36860700001)(356005)(2906002)(86362001)(316002)(2616005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:17.6501 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4ef018d5-4445-494a-741b-08d992ef7f5a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4849 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Leon Romanovsky Instead open-code iteration to compare virtfn internal index, use newly introduced pci_iov_vf_id() call. Signed-off-by: Leon Romanovsky Signed-off-by: Yishai Hadas --- drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c index e8185b69ac6c..24c4b4f05214 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c @@ -205,19 +205,8 @@ int mlx5_core_sriov_set_msix_vec_count(struct pci_dev *vf, int msix_vec_count) mlx5_get_default_msix_vec_count(dev, pci_num_vf(pf)); sriov = &dev->priv.sriov; - - /* Reversed translation of PCI VF function number to the internal - * function_id, which exists in the name of virtfn symlink. - */ - for (id = 0; id < pci_num_vf(pf); id++) { - if (!sriov->vfs_ctx[id].enabled) - continue; - - if (vf->devfn == pci_iov_virtfn_devfn(pf, id)) - break; - } - - if (id == pci_num_vf(pf) || !sriov->vfs_ctx[id].enabled) + id = pci_iov_vf_id(vf); + if (id < 0 || !sriov->vfs_ctx[id].enabled) return -EINVAL; return mlx5_set_msix_vec_count(dev, id + 1, msix_vec_count); From patchwork Tue Oct 19 10:58:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543207 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=AmMuRecH; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3F6JLMz9sP7 for ; Tue, 19 Oct 2021 21:59:37 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235343AbhJSLBs (ORCPT ); Tue, 19 Oct 2021 07:01:48 -0400 Received: from mail-mw2nam08on2055.outbound.protection.outlook.com ([40.107.101.55]:55872 "EHLO NAM04-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235320AbhJSLBq (ORCPT ); Tue, 19 Oct 2021 07:01:46 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ycez+8bDCi35bBAufWDKHhb6GUvklOLNdCUc2w8txAxbzwP/Nbyznwb3a/nvZYCzOJ6GWj+RWqC5aZtR3UKOPCsS2DlPoV4lZQKtaFWt6q2cl1ecPzfvf7Gp2SUzQWz4QXUQszbj8c2hd3a49xugoYT9dlWdB+mHD846KErcJzwapswQSfsar4K5+6Uoq/r1mQ/vQBYaVp20mr76+5oR7FvwM2X9/P5d/LqdVSsYZTa3/vQ+HzHPLuh6JPJ+S6E3F9op2ceI33+zglsKQxiKSWNdfx+2KfLU4m0M186OUQSkef42b/NftIsVvnPSRpVHQ7krsNNUNGfnnAfkgxz0jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ATQOdGVF4dTjbvMJ1bmmazMNjFwWqMg4M5PKqHqf20k=; b=QJXNyU5qVveAM6lXFMvl5EZm0WBjZ2UgToeK/4594M0VSmDDbGRlCulbh+hQrpWH3bBvvqYQ95Z7V/WJ4/YAEMvx1rgnsvnLktf8IolpPVySjLPNrugVWpsJsBNm4l/1VR934KEqiK2/u99HyC09UDZI4v3pP9W7/uXM5WayjTa8c35SJz0HLs0pZv8hVQ39VdnWEOsHN6uH/nCHqM3iJxHio6mMac1Vx0nud/YD3xmEmbC+PV4/xyoSjP0MF2z0zGYWZCqIayKEIVZlrbqoSKxrDw1+fT4CCdtc/rkciLfdYO30bdTE9wwHz3t5NNx9+4+OvZgkMyXVuQhc5EeMIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ATQOdGVF4dTjbvMJ1bmmazMNjFwWqMg4M5PKqHqf20k=; b=AmMuRecHoexdCSocSjKhnYlJORYzxL7rA7F3v436vfvihI/9SZ6XFMHaN/lM7jSQq6Ev9zNyrwLptOXNDEv4X1z4CqgO+judlRQSJaBbpClUm2Hom08p8/XqbgzB79bXYx5SjHT6K/AGOPdUDuIW7luIy9TtJeVL2ZFKCDqAWOePMyNmPz0hS4Qb+DcJt4FXkMHYSiMKJcTucK1f1kMQ92kNH5qUXUctAcbPLz4HtGIGrAD8D4e7U6AlOScSKVkC7BZ8gy8xetkrgc/LfNLVUtMRFevIVbalMASFpsfCp925/iJjz/DI7buCZiDiSJIy/yiwQOfLI+fFM2Fimvo4lQ== Received: from BN0PR04CA0016.namprd04.prod.outlook.com (2603:10b6:408:ee::21) by MN2PR12MB4256.namprd12.prod.outlook.com (2603:10b6:208:1d2::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Tue, 19 Oct 2021 10:59:31 +0000 Received: from BN8NAM11FT010.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::58) by BN0PR04CA0016.outlook.office365.com (2603:10b6:408:ee::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by BN8NAM11FT010.mail.protection.outlook.com (10.13.177.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:30 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 03:59:20 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:17 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 03/14] net/mlx5: Disable SRIOV before PF removal Date: Tue, 19 Oct 2021 13:58:27 +0300 Message-ID: <20211019105838.227569-4-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bbbe4664-a2ca-4ef6-19e9-08d992ef8732 X-MS-TrafficTypeDiagnostic: MN2PR12MB4256: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: I0xIIyclpzT74vTaRCorz0wK8AtoGFoMEfDr/f2JROKjXHjQoMEoKGJFR+0zWl2Qstha4z+HbLzSNlzbPIpC1wZfYiCT2svHjb7d8tCv9fe1V+n9bA+PobI704CfKjWRsV0N2KqPUXr0DEiLah3wnAIw270Anb4zmk+TX5Cet5BxeegBZUOXGMMBbvNcYVjXLXko1ytgPh1hkg6hD/NPjDpPPZxetgIiigrgkGZvZ2UptRYL7vO/jnCGuwYpAtz+deD57y+37NUZULsAAxaVrF4mixMhdeHhTwCPxn+vWgU75YVo1fF1ojuOsN0MpJBjLbF40vX8bKuqYgagsEihYtbzjgwdxMJCvTPfaSA6JFIMfpvsqcnztiX9ruLSwEE0X7ANT9bmv3rAgcRxpKLZkTUw/zl7ivfzzTc7C/H1qO3rdXfmdHwi/ESjAj+V6v8QWtpKRkGe7uVg9DobAtZO8o1F8vw//bLfSreKeLVRDWlJycu/CtuIW1bngrGU0tnN7TIgu2szsni++TlLsrgMpZgPW3ERBv4HvClkW/eqGyTGKRtNPxa7VGN/hrrBfpj1taGqJ7o3wnwAMb3nA2DHqPeKyIiUAYhwSG8xsFL+ke6lP8TxBZnipk4jd8CTjBfApLGR5jQG0NTTazoTQdXvtK+reCQZ47v5E0BpVaiOXS3i+LWwT5XYEEEJD92oBUoinvXCXdD/BWwfHzK6vumTQA== X-Forefront-Antispam-Report: CIP:216.228.112.32;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid01.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(356005)(70586007)(5660300002)(7696005)(1076003)(6666004)(508600001)(107886003)(2616005)(36860700001)(7636003)(336012)(82310400003)(2906002)(186003)(8936002)(70206006)(426003)(6636002)(86362001)(316002)(47076005)(110136005)(4326008)(36756003)(8676002)(83380400001)(54906003)(26005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:30.7755 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bbbe4664-a2ca-4ef6-19e9-08d992ef8732 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT010.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4256 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Virtual functions depend on physical function for device access (for example firmware host PAGE management), so make sure to disable SRIOV once PF is gone. This will prevent also the below warning if PF has gone before disabling SRIOV. "driver left SR-IOV enabled after remove" Next patch from this series will rely on that when the VF may need to access safely the PF 'driver data'. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- drivers/net/ethernet/mellanox/mlx5/core/main.c | 1 + drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h | 1 + drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 79482824c64f..0b9a911acfc1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1558,6 +1558,7 @@ static void remove_one(struct pci_dev *pdev) struct mlx5_core_dev *dev = pci_get_drvdata(pdev); struct devlink *devlink = priv_to_devlink(dev); + mlx5_sriov_disable(pdev); devlink_reload_disable(devlink); mlx5_crdump_disable(dev); mlx5_drain_health_wq(dev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index 230eab7e3bc9..f21d64416f7f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -140,6 +140,7 @@ void mlx5_sriov_cleanup(struct mlx5_core_dev *dev); int mlx5_sriov_attach(struct mlx5_core_dev *dev); void mlx5_sriov_detach(struct mlx5_core_dev *dev); int mlx5_core_sriov_configure(struct pci_dev *dev, int num_vfs); +void mlx5_sriov_disable(struct pci_dev *pdev); int mlx5_core_sriov_set_msix_vec_count(struct pci_dev *vf, int msix_vec_count); int mlx5_core_enable_hca(struct mlx5_core_dev *dev, u16 func_id); int mlx5_core_disable_hca(struct mlx5_core_dev *dev, u16 func_id); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c index 24c4b4f05214..887ee0f729d1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sriov.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sriov.c @@ -161,7 +161,7 @@ static int mlx5_sriov_enable(struct pci_dev *pdev, int num_vfs) return err; } -static void mlx5_sriov_disable(struct pci_dev *pdev) +void mlx5_sriov_disable(struct pci_dev *pdev) { struct mlx5_core_dev *dev = pci_get_drvdata(pdev); int num_vfs = pci_num_vf(dev->pdev); From patchwork Tue Oct 19 10:58:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543205 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=XfUXOvVn; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3C53YLz9sP7 for ; Tue, 19 Oct 2021 21:59:35 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235319AbhJSLBp (ORCPT ); Tue, 19 Oct 2021 07:01:45 -0400 Received: from mail-bn8nam12on2048.outbound.protection.outlook.com ([40.107.237.48]:12897 "EHLO NAM12-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235313AbhJSLBm (ORCPT ); Tue, 19 Oct 2021 07:01:42 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mI4pWCMtcuZ4//lLN5vovN7rhPFsliPlLI5vArjXOi8ID85GQnabaEX8y2bGSKzuYBPca6wpT3hFdGMbneHccrWjc+S+8qFIjCmQzTrH61ehEjtX8sDLGQU2DnyqBl1fVTuT4f0rZvDOuEKZ6wkUbolq4CC6qOc+G98DuROP7o/r0dK7cBTx15XQRmPGTnUmdOvckF2IpeFgEJXyKwl0NFuNcI2Fdwss3oN4Z4gIbA4IaFmuzgzLXOgt9lxnzac/V83lhSKbYgADwUwnCZ59cdt66ZRTBi9/lqY+sDXAR4W7+mZ5yDYpQy8sZOSDzHtcr4Wp9gEC7Y5d45FxkTpzKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PX8vNibUQ+77tQfzz+YEsNNh/fZeI9f68zyWZiZWxw8=; b=nQYXS2MCo+Pvv/rIKnFDQMUhKSNos+CtZf5YY1cpiSJtlBOWPs+S5+I4n0shKUO0lTfAyJefnbDxO1rtDvxQ6DCr9V3tMWF6EeHR9Upwm+iEg686DukIdJtFzza0HsqsMZ6h8RmU7cTOtMhNqwggPy+4rUDsjBvmkA2T64rVybxWw/3g3XQM8BabDPoJlrKsEWx+n4WKMEchPvFPdWuM/QBCxwQgeoVcziHftqMdep2qBFssdDf/BFjoQTJvh8t92eUTpJjvtw6qlbzPMtnPCO1PfAAor/3MPY4zI/uDG740RAelBSb1uHyogXvOpdVkCMCmuZWh352xWubMsSCFYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PX8vNibUQ+77tQfzz+YEsNNh/fZeI9f68zyWZiZWxw8=; b=XfUXOvVn9KZXBCZbxxMWx1aIE7CrxGh0g138PPFBnUJIKVykZODzmdG74lRKN8DsDTaphYcBq0MJgIFaMi8PzuSPxmM0KNfIphsUwwmvZOo9LcyaHb2bwy2y4mbMLP8RNcpLWdN51LmD9PgvIDs2d2a4hqZS/L/Pqluxm0Ym42d5+Zj4UoPSYoR6e8igGy2CFZeO0DpkMXiG59DszaH9wkvf6514CL40pEJrkLF1wu7nnPBOrBYZRB4Y6Mq+3DdCo3vnmszehVCfnCWJlxvQKYDLZEG8zCAufIhrJSJ6uzK62PpLigO9eMWmGCtG+R/7MeG+cZ4rCazvrqIICJjgDg== Received: from BN8PR15CA0016.namprd15.prod.outlook.com (2603:10b6:408:c0::29) by SN1PR12MB2447.namprd12.prod.outlook.com (2603:10b6:802:27::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Tue, 19 Oct 2021 10:59:28 +0000 Received: from BN8NAM11FT008.eop-nam11.prod.protection.outlook.com (2603:10b6:408:c0:cafe::f0) by BN8PR15CA0016.outlook.office365.com (2603:10b6:408:c0::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT008.mail.protection.outlook.com (10.13.177.95) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:27 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:23 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:20 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 04/14] PCI/IOV: Add pci_iov_get_pf_drvdata() to allow VF reaching the drvdata of a PF Date: Tue, 19 Oct 2021 13:58:28 +0300 Message-ID: <20211019105838.227569-5-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f697d0cb-f59b-4715-fdea-08d992ef8539 X-MS-TrafficTypeDiagnostic: SN1PR12MB2447: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FmAnxwlkMmz0bEjvfhggbbwQjA6QEL+rXYtbH7gGLlz856M2vB9wFhG+K6fsb4eOhJmj3cGG6qYiC268nsEMPOHyocJORl6Hf95yN7OAlhSV7AHoowWqRv5fLWIbxlBiz9LRjzKSON4dKXn2yXNOiFpGwP/GKK/RNUJc+WsdhQckONQRytmSWHY2uZWPo/ye3i5LYkOg50QgdxwAujMAyDVbQnKsy9Fo5G3mxZjVt5BZ43XCj0EscF+beRJ4+XvtLcFfVsgWRFzP1phslDoqqePyRfpLZ4s05bC3JnviYTZAbvYryOSlw2c24JhrkC2vIifviIfb05Rqtp6b/S4GaAxvy6o0Jr2nWLVgOE14/IWHf/MtwGYzyUd4eBPDooEyVJn+h3Csd5RfojFPM9QBxhKCViSVtUI6hw4EsI34tsAUm4Ftzb6vDr2v2yY0EX9HHZodwZDWNehhLFfKlYNd30IEB7rGSLQb1ikQxx6ldX34LzTOl/I7acAyJkk+xNtPQDLuAtzQhZWbNf8ARndlRSZ5bynwWTFO+ZHfbemkM+F1TOKlVUL9VVfvcEJh/QNSJzu5KIwtvA2SuDbpUJSvmztIOI2JIWYfr1ukAX27QXtsQMyqSzD7trszMDMuoaPOHGrgD+2RUVcT78kFbR8u0361SvhuQw+CYofArZ3Bxd+IKvQjsP77aZBKrB8CZL0OnISeZhi5T9n3TrKhOD2zOw== X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(36860700001)(426003)(82310400003)(26005)(356005)(36756003)(186003)(36906005)(70206006)(4326008)(5660300002)(86362001)(7636003)(336012)(508600001)(8936002)(83380400001)(110136005)(2616005)(70586007)(54906003)(6636002)(7696005)(316002)(8676002)(2906002)(107886003)(1076003)(47076005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:27.4198 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f697d0cb-f59b-4715-fdea-08d992ef8539 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT008.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2447 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Jason Gunthorpe There are some cases where a SR-IOV VF driver will need to reach into and interact with the PF driver. This requires accessing the drvdata of the PF. Provide a function pci_iov_get_pf_drvdata() to return this PF drvdata in a safe way. Normally accessing a drvdata of a foreign struct device would be done using the device_lock() to protect against device driver probe()/remove() races. However, due to the design of pci_enable_sriov() this will result in a ABBA deadlock on the device_lock as the PF's device_lock is held during PF sriov_configure() while calling pci_enable_sriov() which in turn holds the VF's device_lock while calling VF probe(), and similarly for remove. This means the VF driver can never obtain the PF's device_lock. Instead use the implicit locking created by pci_enable/disable_sriov(). A VF driver can access its PF drvdata only while its own driver is attached, and the PF driver can control access to its own drvdata based on when it calls pci_enable/disable_sriov(). To use this API the PF driver will setup the PF drvdata in the probe() function. pci_enable_sriov() is only called from sriov_configure() which cannot happen until probe() completes, ensuring no VF races with drvdata setup. For removal, the PF driver must call pci_disable_sriov() in its remove function before destroying any of the drvdata. This ensures that all VF drivers are unbound before returning, fencing concurrent access to the drvdata. The introduction of a new function to do this access makes clear the special locking scheme and the documents the requirements on the PF/VF drivers using this. Signed-off-by: Jason Gunthorpe Acked-by: Bjorn Helgaas Signed-off-by: Leon Romanovsky Signed-off-by: Yishai Hadas --- drivers/pci/iov.c | 29 +++++++++++++++++++++++++++++ include/linux/pci.h | 7 +++++++ 2 files changed, 36 insertions(+) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index e7751fa3fe0b..8c724bc134c7 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -47,6 +47,35 @@ int pci_iov_vf_id(struct pci_dev *dev) } EXPORT_SYMBOL_GPL(pci_iov_vf_id); +/** + * pci_iov_get_pf_drvdata - Return the drvdata of a PF + * @dev - VF pci_dev + * @pf_driver - Device driver required to own the PF + * + * This must be called from a context that ensures that a VF driver is attached. + * The value returned is invalid once the VF driver completes its remove() + * callback. + * + * Locking is achieved by the driver core. A VF driver cannot be probed until + * pci_enable_sriov() is called and pci_disable_sriov() does not return until + * all VF drivers have completed their remove(). + * + * The PF driver must call pci_disable_sriov() before it begins to destroy the + * drvdata. + */ +void *pci_iov_get_pf_drvdata(struct pci_dev *dev, struct pci_driver *pf_driver) +{ + struct pci_dev *pf_dev; + + if (!dev->is_virtfn) + return ERR_PTR(-EINVAL); + pf_dev = dev->physfn; + if (pf_dev->driver != pf_driver) + return ERR_PTR(-EINVAL); + return pci_get_drvdata(pf_dev); +} +EXPORT_SYMBOL_GPL(pci_iov_get_pf_drvdata); + /* * Per SR-IOV spec sec 3.3.10 and 3.3.11, First VF Offset and VF Stride may * change when NumVFs changes. diff --git a/include/linux/pci.h b/include/linux/pci.h index 2337512e67f0..639a0a239774 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2154,6 +2154,7 @@ void __iomem *pci_ioremap_wc_bar(struct pci_dev *pdev, int bar); int pci_iov_virtfn_bus(struct pci_dev *dev, int id); int pci_iov_virtfn_devfn(struct pci_dev *dev, int id); int pci_iov_vf_id(struct pci_dev *dev); +void *pci_iov_get_pf_drvdata(struct pci_dev *dev, struct pci_driver *pf_driver); int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn); void pci_disable_sriov(struct pci_dev *dev); @@ -2187,6 +2188,12 @@ static inline int pci_iov_vf_id(struct pci_dev *dev) return -ENOSYS; } +static inline void *pci_iov_get_pf_drvdata(struct pci_dev *dev, + struct pci_driver *pf_driver) +{ + return ERR_PTR(-EINVAL); +} + static inline int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn) { return -ENODEV; } From patchwork Tue Oct 19 10:58:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543206 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=d9klGhN1; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3D2NZCz9sP7 for ; Tue, 19 Oct 2021 21:59:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235313AbhJSLBr (ORCPT ); Tue, 19 Oct 2021 07:01:47 -0400 Received: from mail-mw2nam10on2060.outbound.protection.outlook.com ([40.107.94.60]:43232 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235315AbhJSLBp (ORCPT ); Tue, 19 Oct 2021 07:01:45 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TO+rlkTIB7I3GipoCT9vQhOdzhOl5bemBSyh4WZEAvSZVuML/iQfQbK33o1MPUSoh/bQijtcGJSZ4c6ZmjJRWUXyihQ+coIoIcvvabtFRJEXwFrenfSRJqixBzCJQuPoaXkhRpmfM37dzwYdCCozQh4vePomdTVmXC0pZXuW7iVg8X+/h25qF3sbbUDeMsouupvI30GW8bQ1R68VTN9pLZEmhGF9m6CzWAauhuz2QhznKBKOLAgAB8y3eWyD5mpS8sQBUqezefuWDZc7oUQTLpXZxidxqTzYlE50o8Ucr6YNoni8W52wlM5dBOg0wneFIb69Ej9D0kvnJCbOC/FidQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0Rg93GuPxk5N0KtlMyEKO/efa5SGHQlpbOGoEZBLbxs=; b=Q8nZhcY4rBZoLVHYCLFF0IWzwJwnMVdvWTCHTZArPEIFI4WlWkz8kLXVjELk84vufok2F1vJHVhA6niUrDjqk0t++XA/lSO00+db03B8t+MXwxArErZjb+d1p+LsYh1A9c/5eM1+BZswebnj13WfVSmlgXqJX8ouAnWrQTIiTtk98QjKw7y5o5QVvOVHQFOm67VaB5GKVl0HJEsqENhdzTl7FL/W4HCeCVCyxBB6cwV0+/9ja7OVw9pS7xkVzA7N1c9HoIYVf4NU9pe2Q/dxn0fE0VZBuzzIWBCo5bKckijZpBP+BfREF7/8Xa9JE601Y/IQRiijRRYgUMJCHdFQvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Rg93GuPxk5N0KtlMyEKO/efa5SGHQlpbOGoEZBLbxs=; b=d9klGhN1FrWQl/+r+D/TPIUOTHgFBmjaruPYrlXhrPZofVqClxTfLIKTix6UJ6Wlvi/mquLJbSm+mkCoLAt5k3X+imUjlaNn3zEfz5FZKTO/uHz6g/Lj4Aj2JBM9+iKTof669UiOI+QNHQhuT8V9lxdYo+MsPs4HLWuG78LO2/xz6HGOFnowsTy8Isrz5Y8ZxMIo1fzd6JUXoU2DM3omPiOwx9ZpWtKoTHVz+8F9DCptgreysJ/hJdHIWlwS403jQ8/5UvWjctvcBRRWUZIyiGnx0gfZdY2+l8KTaJ35itM75sZU3jKvIzKtDlEOWnLT3p55R3RM4XxVkSPWrWvtYQ== Received: from DM5PR04CA0072.namprd04.prod.outlook.com (2603:10b6:3:ef::34) by CY4PR1201MB0197.namprd12.prod.outlook.com (2603:10b6:910:23::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Tue, 19 Oct 2021 10:59:30 +0000 Received: from DM6NAM11FT021.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ef:cafe::f1) by DM5PR04CA0072.outlook.office365.com (2603:10b6:3:ef::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT021.mail.protection.outlook.com (10.13.173.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:30 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:27 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:24 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 05/14] net/mlx5: Expose APIs to get/put the mlx5 core device Date: Tue, 19 Oct 2021 13:58:29 +0300 Message-ID: <20211019105838.227569-6-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5773b9b1-14b4-4608-acd0-08d992ef86ab X-MS-TrafficTypeDiagnostic: CY4PR1201MB0197: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qlA7fcjHq5W7Wpjx13LS95N9NaBtQPqL5WL5xMt8hcVhthIsqkie0GjDPlDqyotMunzQEDuMN3IvEa1WjUZWeNuo23tCAUHM4jFnjZ63knazczQMUoma7ATHoLbqp/E9TDpwu1rv4kFq7tTwiG5Jiq6PEmcxpHxRxvCGtzsfl5bDblA8jvZq7nkyczuanJk8b7qLUzvaO1DaGpS4fF/ilUpqxdry8/M+EogWYlGh12MOYx+Y2FL5AgLYUF7MKEny3SpdwgV2GwZvjUlo6dzAcUreFFBqmrp8pgDsg5F0mhBfreWQk7LRCbbk2m3L0SBujklUxfkgJchFiGGMBZ5Yd+G69j2tAWBcNxhJKIZK5q0Z1tHW7wqOZRY5ehLCZQRSHqfSAtjxfvUTBR2X8HkvowiE3YnNORmPuQHve8dRiBsT3N+wml60HyEy5AtAz4dpsh49B/p66Sb/nfqjRbF7JFEDyeLZDBO/KXVBuoF1kc68i3s2ejOGjPSRvARlx0+/IA5Unp0eKVYGXyyYHw2RCXSrr8pHOYOZSovd+8WWwkVQFb7iYEKA/cQFKzBgLznsAYdwcs+nNwTYw6LAnmpC2gwvgjk4BGEfIuWujcIwaOr+rJEtw5D3OHCZFT/GktcN5x03Kei24RVvZ6J10tdG5zwBbci6ZojVda1LFENzjZ8DNPYYxreHAejgrjapQzQS7sjpUvwWiJWBmW1c63a33A== X-Forefront-Antispam-Report: CIP:216.228.112.35;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid02.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(107886003)(83380400001)(7696005)(8936002)(1076003)(508600001)(26005)(47076005)(86362001)(316002)(82310400003)(70586007)(70206006)(54906003)(6636002)(8676002)(4326008)(7636003)(36860700001)(426003)(110136005)(356005)(336012)(186003)(5660300002)(2906002)(2616005)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:30.2803 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5773b9b1-14b4-4608-acd0-08d992ef86ab X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.35];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT021.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB0197 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Expose an API to get the mlx5 core device from a given VF PCI device if mlx5_core is its driver. Upon the get API we stay with the intf_state_mutex locked to make sure that the device can't be gone/unloaded till the caller will complete its job over the device, this expects to be for a short period of time for any flow that the lock is taken. Upon the put API we unlock the intf_state_mutex. The use case for those APIs is the migration flow of a VF over VFIO PCI. In that case the VF doesn't ride on mlx5_core, because the device is driving *two* different PCI devices, the PF owned by mlx5_core and the VF owned by the vfio driver. The mlx5_core of the PF is accessed only during the narrow window of the VF's ioctl that requires its services. This allows the PF driver to be more independent of the VF driver, so long as it doesn't reset the FW. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- .../net/ethernet/mellanox/mlx5/core/main.c | 43 +++++++++++++++++++ include/linux/mlx5/driver.h | 3 ++ 2 files changed, 46 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 0b9a911acfc1..38e7c692e733 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1796,6 +1796,49 @@ static struct pci_driver mlx5_core_driver = { .sriov_set_msix_vec_count = mlx5_core_sriov_set_msix_vec_count, }; +/** + * mlx5_vf_get_core_dev - Get the mlx5 core device from a given VF PCI device if + * mlx5_core is its driver. + * @pdev: The associated PCI device. + * + * Upon return the interface state lock stay held to let caller uses it safely. + * Caller must ensure to use the returned mlx5 device for a narrow window + * and put it back with mlx5_vf_put_core_dev() immediately once usage was over. + * + * Return: Pointer to the associated mlx5_core_dev or NULL. + */ +struct mlx5_core_dev *mlx5_vf_get_core_dev(struct pci_dev *pdev) + __acquires(&mdev->intf_state_mutex) +{ + struct mlx5_core_dev *mdev; + + mdev = pci_iov_get_pf_drvdata(pdev, &mlx5_core_driver); + if (IS_ERR(mdev)) + return NULL; + + mutex_lock(&mdev->intf_state_mutex); + if (!test_bit(MLX5_INTERFACE_STATE_UP, &mdev->intf_state)) { + mutex_unlock(&mdev->intf_state_mutex); + return NULL; + } + + return mdev; +} +EXPORT_SYMBOL(mlx5_vf_get_core_dev); + +/** + * mlx5_vf_put_core_dev - Put the mlx5 core device back. + * @mdev: The mlx5 core device. + * + * Upon return the interface state lock is unlocked and caller should not + * access the mdev any more. + */ +void mlx5_vf_put_core_dev(struct mlx5_core_dev *mdev) +{ + mutex_unlock(&mdev->intf_state_mutex); +} +EXPORT_SYMBOL(mlx5_vf_put_core_dev); + static void mlx5_core_verify_params(void) { if (prof_sel >= ARRAY_SIZE(profile)) { diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 441a2f8715f8..197a76ea3f0f 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -1138,6 +1138,9 @@ int mlx5_dm_sw_icm_alloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type, int mlx5_dm_sw_icm_dealloc(struct mlx5_core_dev *dev, enum mlx5_sw_icm_type type, u64 length, u16 uid, phys_addr_t addr, u32 obj_id); +struct mlx5_core_dev *mlx5_vf_get_core_dev(struct pci_dev *pdev); +void mlx5_vf_put_core_dev(struct mlx5_core_dev *mdev); + #ifdef CONFIG_MLX5_CORE_IPOIB struct net_device *mlx5_rdma_netdev_alloc(struct mlx5_core_dev *mdev, struct ib_device *ibdev, From patchwork Tue Oct 19 10:58:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543208 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=HQqj4sf+; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3N4tJ8z9sP7 for ; Tue, 19 Oct 2021 21:59:44 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235357AbhJSLB4 (ORCPT ); Tue, 19 Oct 2021 07:01:56 -0400 Received: from mail-dm3nam07on2086.outbound.protection.outlook.com ([40.107.95.86]:27329 "EHLO NAM02-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235315AbhJSLBt (ORCPT ); Tue, 19 Oct 2021 07:01:49 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cTMdLuGwA1qItbUdQZN12oP9I3cgsjc7pNUio+JTLw24VT6S7GhaShcHfiJVnoPehugbXQPtGbL651pqsbAYVvLFloUeMe36ZFZCETFYAu1ZLq3sBbfHhQ+43bNxEsH+4+ABHTIFMtw4x2GmGLqkyEhcMkPcPUr88H5XlCT5mSwp/BkHinEibHx9P4p3Z7TR5NFk/vLyF4xBsznanoiHp3OmzvJXYoM7r1MlMHDQU0+b1ydS4ViuEPmqYo3kE2oi81+CD6OOVq5HbyK2Q6HdGKnpme9RpWivMlF8DjCleJH1NY4/1FDSix0Uqsa2RLSNfk/3+9DdcDrZzLvf8vuVDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=C89Jlk8BDEI17QHQ1FU/2vPIIO2h4K4F/VMusgvFzJI=; b=nW4xlz5Cgj0akhqelWDoWToszzwiargfR44rZMihxnAdBw9ep53v9fBfMTvg3+Fza++VG/u/MV7PqOYE8FZ0WqDvtfxX8VRrGa/r4v81KziCsgRN9qn25iduv0wGMMyz2RU4sTtfeWX7TB9MypUdeoFCX5kY7y1VFu0m8ohxYiriitUdRnsDmzOtTGAg+ZgX/CsF6rXu9syC591sV4QluQVJkyzxZ6beziixyYBHeeXqNGCT7eIUxFkFgwMAK7kXF3KIA3EwIujULkwKo26ov9yDvLiRwqIFQ+7GcphRp7gIW2qyom/1x1dbWaFLjQ+6Dwip/FAZaYHio3StK2Op7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C89Jlk8BDEI17QHQ1FU/2vPIIO2h4K4F/VMusgvFzJI=; b=HQqj4sf+Zff21UJX1eoEsOXq90YCCO02rVzqghXwgdHzr/elYt0yRLIof8dlRuU+dAPXsNSfj/xt2tLGk0WjGfxCu8y3NhQ6dyoEKfHdr1nQ+wk+RY7Drk4ZDXFtTtTSdhd6ne/hNuqzTTmVEctnmbsjYm3Ra7d/gHLykkuiU7jg8RsqSvldjpOiQV3OgGCfW4I6Btmhprmr2G1OLmoRV+q+/Ul/eOUKwati3XbZzCFrfv4cgqSvU5pVdmt2zkOyIKHKmHKJk8oDdf4iZV/T0lLvXgRDnAyM7MK5USd0ZSpJB9xOO01nt5x9lpGSn1klItoh37j1gUGR03Dl+le62Q== Received: from BN0PR04CA0018.namprd04.prod.outlook.com (2603:10b6:408:ee::23) by MWHPR12MB1536.namprd12.prod.outlook.com (2603:10b6:301:6::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:34 +0000 Received: from BN8NAM11FT010.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::3f) by BN0PR04CA0018.outlook.office365.com (2603:10b6:408:ee::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by BN8NAM11FT010.mail.protection.outlook.com (10.13.177.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:33 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 03:59:30 -0700 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:27 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 06/14] vdpa/mlx5: Use mlx5_vf_get_core_dev() to get PF device Date: Tue, 19 Oct 2021 13:58:30 +0300 Message-ID: <20211019105838.227569-7-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2b0d1e82-748a-406c-8d39-08d992ef8890 X-MS-TrafficTypeDiagnostic: MWHPR12MB1536: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3173; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NGvEPwLbZ8L1IWS9S6UoDHV3sbm40hHfPcBlHAG41Q4yIO0RBysLUdQ1clinlB+riugUIBjV39+F/0fVcCxy8dv2ul50nOCdkTPWTBcyq53fDedUpiOx+JYbefEu+3ovuUxFRUM5Cd7xrLTl2MPw8i489V/3/bGHgT7OjDr7BAQUGJQn5m8h3m3xnFleUYwU0nnJ4RK2PMU8aGw/EtVXfsY/5rwrjST5RF48r6XiZ9OC+z8BdI7Mt9cTLVWup18irCsOeuWoZS2cYhbBHm2LWnucVuWD9l9wkN80HdljyhFLiep/0+z29/Up8r8utnjieGpwdaULebV2Tfco0EjjvpCecMhZ70qGjH2qx3yoURFBlKIDhVXSwbvImLuvLb6qV+sJde1FMpxX+PLKqoKIGwx0Wi8nd9qYQqLduCn1D6DH/QZeVF21ufLD2/WirhDWFFdDSUuGQHAyM2YTgzgTzJgOTaiW2EOAVtyNdHwJFYOvFcMF8iv7eqxfMkTVxHilyDRtaT8lMf+xgNPI26Cyo6zoIWb5UTfPFyCFAWg1KsEd0dP7MO1VS83ZmMLncca/i75bja60XUCu8r8r3nQgLAPmhJbTtPQ6g5jaiECB17d/stYWyNvy/JEDpBlO+D4Doc7gNz1s2UsCYDSqh82P/lAxLwm554zcgCIU00Wlbz79TEnt0LaPINA0d/5ytzmdk7qPs5svCozZvGfc3/ckIQ== X-Forefront-Antispam-Report: CIP:216.228.112.32;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid01.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(110136005)(107886003)(54906003)(8936002)(47076005)(7636003)(83380400001)(82310400003)(356005)(1076003)(8676002)(70206006)(36860700001)(26005)(508600001)(6636002)(36756003)(86362001)(2616005)(426003)(6666004)(336012)(7696005)(5660300002)(2906002)(70586007)(4326008)(316002)(186003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:33.0742 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2b0d1e82-748a-406c-8d39-08d992ef8890 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT010.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1536 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Use mlx5_vf_get_core_dev() to get PF device instead of accessing directly the PF data structure from the VF one. The mlx5_vf_get_core_dev() API in its turn uses the generic PCI API (i.e. pci_iov_get_pf_drvdata) to get it. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 5c7d2a953dbd..97b8917bc34d 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1445,7 +1445,10 @@ static virtio_net_ctrl_ack handle_ctrl_mac(struct mlx5_vdpa_dev *mvdev, u8 cmd) size_t read; u8 mac[ETH_ALEN]; - pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); + pfmdev = mlx5_vf_get_core_dev(mvdev->mdev->pdev); + if (!pfmdev) + return status; + switch (cmd) { case VIRTIO_NET_CTRL_MAC_ADDR_SET: read = vringh_iov_pull_iotlb(&cvq->vring, &cvq->riov, (void *)mac, ETH_ALEN); @@ -1479,6 +1482,7 @@ static virtio_net_ctrl_ack handle_ctrl_mac(struct mlx5_vdpa_dev *mvdev, u8 cmd) break; } + mlx5_vf_put_core_dev(pfmdev); return status; } @@ -2261,8 +2265,11 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) free_resources(ndev); mlx5_vdpa_destroy_mr(mvdev); if (!is_zero_ether_addr(ndev->config.mac)) { - pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); - mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); + pfmdev = mlx5_vf_get_core_dev(mvdev->mdev->pdev); + if (pfmdev) { + mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); + mlx5_vf_put_core_dev(pfmdev); + } } mlx5_vdpa_free_resources(&ndev->mvdev); mutex_destroy(&ndev->reslock); @@ -2449,8 +2456,11 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name) goto err_mtu; if (!is_zero_ether_addr(config->mac)) { - pfmdev = pci_get_drvdata(pci_physfn(mdev->pdev)); + pfmdev = mlx5_vf_get_core_dev(mdev->pdev); + if (!pfmdev) + goto err_mtu; err = mlx5_mpfs_add_mac(pfmdev, config->mac); + mlx5_vf_put_core_dev(pfmdev); if (err) goto err_mtu; @@ -2497,8 +2507,13 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name) err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: - if (!is_zero_ether_addr(config->mac)) - mlx5_mpfs_del_mac(pfmdev, config->mac); + if (!is_zero_ether_addr(config->mac)) { + pfmdev = mlx5_vf_get_core_dev(mdev->pdev); + if (pfmdev) { + mlx5_mpfs_del_mac(pfmdev, config->mac); + mlx5_vf_put_core_dev(pfmdev); + } + } err_mtu: mutex_destroy(&ndev->reslock); put_device(&mvdev->vdev.dev); From patchwork Tue Oct 19 10:58:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543209 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=azXqeXBc; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3d4SBmz9sP7 for ; Tue, 19 Oct 2021 21:59:57 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235375AbhJSLCI (ORCPT ); Tue, 19 Oct 2021 07:02:08 -0400 Received: from mail-bn8nam11on2063.outbound.protection.outlook.com ([40.107.236.63]:26817 "EHLO NAM11-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235376AbhJSLBy (ORCPT ); Tue, 19 Oct 2021 07:01:54 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NVVL3YHpW4F7ZXP+fjpC+HMv4xM6A95F7g8RjBG6Qg/rLIQNyed7fCeZOo9DXAd+cAgNC8Ig6oFFa9CiL4J3kgwcXK5rYIVGUeGRl0JIEcq/6mFIQy2wjW8EoQFPTnQV+3/ADngeLO3/oliy+UmE970ZswLBNVFLPGJ5S798Yr0OUrZ/Xun6eOf8p+UoPqP++DBZMyBEEbk6oHWg6S4C2O80nltdRhNC5Q6tNQcOOdcrFEDKzTkBlqA12BKFhfUiRigMoxlqhpqIEejM2g7PaxY7M8WZ+LWOwwier3iNFb3l7drOo0Joh8GCybO07go/rccMsf7fYjAoVmvAk5e9kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6JQIq0W3l53X1SP4Yfz2n4wvIoj4KiKObvdNUB8PnJI=; b=ck3Rs3zUsdF/vfocjFxG9dy39zBv4SYwM7Jp54FmCpYJzFSqy7pPkQI490BAQ/iIPeHdgpIZtTV4kwCxZ/9JhGdCMQLQ++YsT5s02VCFrK+yxzJWGn3FCdxSrhqA8nMNBzIG3laXsZ3OZcITlU1jPJnG99J3oeuDDeB8YTOMh2cf5LKoW3StMOyChK1CVGkAnBQVaNcMFLBL8feQBL+gwY4SgrbKvl5hYuTuKYYrU20aJh0fSacHHSYMQYG3ZHrolO3TcMFru39kzUNn7ZYpxpDc/4VkKz97iXerNgKDmkfeQiGQaNJUxmZwYOdUQ8br+DFQMBPZUlTyh28mb9qRxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6JQIq0W3l53X1SP4Yfz2n4wvIoj4KiKObvdNUB8PnJI=; b=azXqeXBcmA8wF7QsxpNFRq8/ZFDqAWomLJPHNKyh7tn1pYmpDMTrKhFQhQoT35N9+z/KvfcfK4CnslSsE5WeAWXE5KbfYfTW7nqJvAvbvWGGrZXvGDie5anc3jhTOo+gNEvomFb1FVXLNK66dUEBMaXK7r8fOMGHGTWFnURHdxOO95ZU7rzS1vUrT9GdNW2tctP2rkXoASgFFCWKN8WhEZLE9gW7KHi/guldMRFl4fUWnLziWFFloFZWtD4sSkm2YNiZlrLPWn408y8QOYaKSHuasBc2SItKAytCIf/1GphVugdLIqbp/7ondb00FPPTcWLDCu9iM7d+dbR58gJJGA== Received: from BN8PR04CA0015.namprd04.prod.outlook.com (2603:10b6:408:70::28) by MWHPR1201MB0141.namprd12.prod.outlook.com (2603:10b6:301:56::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:40 +0000 Received: from BN8NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::ec) by BN8PR04CA0015.outlook.office365.com (2603:10b6:408:70::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT051.mail.protection.outlook.com (10.13.177.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:39 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:33 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:30 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 07/14] vfio: Fix VFIO_DEVICE_STATE_SET_ERROR macro Date: Tue, 19 Oct 2021 13:58:31 +0300 Message-ID: <20211019105838.227569-8-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aab02568-1df5-40b5-49f5-08d992ef8c28 X-MS-TrafficTypeDiagnostic: MWHPR1201MB0141: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2657; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N70uHExVQweb4ol61fQWXUM4HB14bfc4zvAdi4qNPebGco79QA4obZu2y9/rtJudqTDxrYKkisD91tf56t/JBakMVwlJc30YpzabNbRuzUA9EjNoXzg7lkUJ3UTlCJUnoRXAXc0nf2NpGqrjo0ht8kg5Pk9NSclWD763fhqaLHVzA6588L1EtRTx1BV+spRBZsq/7ru/WoCIRl3f+OUzf9aB5cxf6T3RgUGmn75/0X1gRcbG0bHbLOfqWULr5DH9p/Ag1b0Fpxlo9DKm3K6A5m0sMoiGzYuEN+cVI6Fnso4AfuLGzXKqeRMBKYgUT9P4w4RURaQgAefQ3wTjLeyYyJfog3rDumcYqIuz2CUfpj9IQBSxn7BWSpnoPM8ZyS0x7btQtJPKlhDKnZPlmgC1UA0h6tJ+ji6YIENgiXa+9RXqhRhbGuM+EVfTBvXZsUqB1SbMOziT86kxA0RHjEfpNFO0DP9boY7qmh5X1jJ6K6o0tYNjQrRmBxCfEijs14n+2glLe575BhZ8I04MOEU4U3XLuvD1akfEDCyji/F4IGnVImVP2p8WK8rPLBosGx/7H5OQLVoErCis4enshs9iK0O8QZz8UAp/FDacyouHOSM2eaJgdC9BfyI19nzyeKevdm11V6Ap8e3VhwMeR5gR5hVE5H4DRT8IAb11mevq5kF3Si0yudcaf0KD0n/+B0sHnNmOsQ59D/lJRI9KzVMfpA== X-Forefront-Antispam-Report: CIP:216.228.112.36;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid05.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(110136005)(82310400003)(54906003)(6636002)(5660300002)(7636003)(70206006)(36860700001)(2616005)(4744005)(1076003)(8936002)(107886003)(4326008)(86362001)(70586007)(336012)(426003)(316002)(47076005)(8676002)(2906002)(26005)(7696005)(508600001)(36756003)(186003)(356005)(83380400001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:39.1338 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aab02568-1df5-40b5-49f5-08d992ef8c28 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.36];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0141 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Fixed the non-compiled macro VFIO_DEVICE_STATE_SET_ERROR (i.e. SATE instead of STATE). Fixes: a8a24f3f6e38 ("vfio: UAPI for migration interface for device state") Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- include/uapi/linux/vfio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index ef33ea002b0b..114ffcefe437 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -622,7 +622,7 @@ struct vfio_device_migration_info { VFIO_DEVICE_STATE_RESUMING)) #define VFIO_DEVICE_STATE_SET_ERROR(state) \ - ((state & ~VFIO_DEVICE_STATE_MASK) | VFIO_DEVICE_SATE_SAVING | \ + ((state & ~VFIO_DEVICE_STATE_MASK) | VFIO_DEVICE_STATE_SAVING | \ VFIO_DEVICE_STATE_RESUMING) __u32 reserved; From patchwork Tue Oct 19 10:58:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543210 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=uIPkgSFi; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3f3WFWz9sP7 for ; Tue, 19 Oct 2021 21:59:58 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235380AbhJSLCJ (ORCPT ); Tue, 19 Oct 2021 07:02:09 -0400 Received: from mail-dm6nam12on2051.outbound.protection.outlook.com ([40.107.243.51]:45888 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235381AbhJSLBz (ORCPT ); Tue, 19 Oct 2021 07:01:55 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a+TCeiheYJ5rWLLT0etm9qRvJio3ZwmjKYfrrMRdgwOi+mVuXGEwEOOSg4oYBy8OViWitNcyLakrSpEpezJPDf+/2lzIYvWdK2JoPcuPR2flXZb6NqqiPV0Kt39IJw/NuzcHzMlbmbkha9362QwWDn8AVbSgVJrOw7nfU3F9wnR9QkeFR6czD/s3Hwk7Eed3BZWwHAzDDhtztMA84rye9YeEePxcTpr7raKqV/ne57gM7Es5Mc9Q6p1Qs6MJm+alUyjZ6Es1kR7Wb0ZHz/r7UqyyyVU9OvGCqHGZ3frda6xZQpFYsKo1X4FntZd4Q7tpCQNNxYgTzrigBtowJ84v4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8goAoxyjUAsHv6c1ws9+V6erIF3GK7IY7fbncebHEMs=; b=Ullfd+d+UxxEGN6XnK8j19BK6ychqw2X6GfMx9NCyGP1DU5Y3qgJsgSGszFYHLEBe/naX3OQecj3J3FxJmqlBBRAfuSuyTtJkK2hmyvpj77s9QK3JGEo+d4bOHP2h0321VUjRxeCVSa0qlv4ToU9lDEvinTTtaO5FCaAAmU/60IwEAdSrPP+2cs2luvVHXbM+ybw4TcEBSZYzCXjD6TH9d9nOpbjfwg45kgWuZ0Wpwz0Avhni2oLz6UWfkGFQqFnsqwUV1bWSV7yzgdJGG1ir/dxBZiHjyC2xJSH+WzS0E4ftXKJU6EThM/3YNXjTTivCJxegsECr4EPWQV0POD4XQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8goAoxyjUAsHv6c1ws9+V6erIF3GK7IY7fbncebHEMs=; b=uIPkgSFibttehPVWAx+CCiZ7AxxvJCydrmafg8vu2moarF/SBE1PWqhT7mbIHxlDIVeUUsXrSPaaRvztONGuxd/D191wlvELp0Kj590jNn6QLRm4XJtElHdIKZMFAp/NQTpm6Hxekatl5ACv33cXkzXWd9TiXJoEXtVAI8YEL9uB3IE1ISlSqI+/Ey5OHG2Tq2H9WurEdSdiOMOTu/Ae4k/tW62sMIUkOib5Ia/w0kn65+jND4Afry3xXSVgr4qDkF4UOgjgHiGwfNk/Ki+Wgrh4p8rVBHpEXXmWH+U0ciKt1OeYcIoxkPFAGChfMueC063A5wEaCaKdF5DOXqC4zg== Received: from BN8PR04CA0014.namprd04.prod.outlook.com (2603:10b6:408:70::27) by DM4PR12MB5293.namprd12.prod.outlook.com (2603:10b6:5:39d::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Tue, 19 Oct 2021 10:59:41 +0000 Received: from BN8NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::21) by BN8PR04CA0014.outlook.office365.com (2603:10b6:408:70::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17 via Frontend Transport; Tue, 19 Oct 2021 10:59:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT051.mail.protection.outlook.com (10.13.177.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:40 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:37 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:34 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 08/14] vfio: Add a macro for VFIO_DEVICE_STATE_ERROR Date: Tue, 19 Oct 2021 13:58:32 +0300 Message-ID: <20211019105838.227569-9-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1d54cdcf-763c-48b7-97f3-08d992ef8d1c X-MS-TrafficTypeDiagnostic: DM4PR12MB5293: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3044; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: okj9jkAQ2Svbb95/pb8cAQzm+poaMhUARTMyk5xjBrmb8CKfHcz/CtA41SYkwvUWz42KY1XXlI5XGAOxwCEgAPLzmaOxiUhAAxRqxeHOueuQvqkmdSDMgWD97EyRVDoTfsmG3xZQzwS8GDRNNLon3NXEVDtbrItHhTDPZK+bT10nj+Y91wwRFDxVHpNqdzMNpdSBDlGxGouKwotKcxFYEl5J9ZpGlCD4ZE4cgnk3Y67nncXR1ZPP/GzWmo9LqXMWJw6MZY+nGSVbm00Tw9KGQCYkXsAdppYRuafzyfKl9mydJmO83M25Mrx7Ldp0AbG9IOWmoeNhM00x7lqF4YfuvCd6I8XF/9emQLom5IxUcwIGW+y67ckXUcNIdDukURBuwBorl7ax5cQzIKh0Ku06p8lhC+i7DaXpRUgKA/uyQ5oV/IW1lkTdWiyFn/AiHWb/pxBvEFMydyAVB2sgLQXuA+dYfYgO8nJBeP3ik2/i5dwdGHrC6NnSVhcmuKiardLztkEb3MnwO8xkPhO5FPPhLHy3GIUUteRys54lftOuZeBxJ5hZEnoXpWb7/hW79nlhnADXv4c4IucZBe78DkuWBK0HyLmkUx+EVhE4b4/Pia8vcrEotus9AI8vzSc1fD5aW/yqRiIVhZ8d08TpvZXG3OR2AqAIhXlGqXjTCsU34Wls8QmB8mzhpooVPytReLnoNzPU76QtQUCkhcfQRb8I6w== X-Forefront-Antispam-Report: CIP:216.228.112.36;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid05.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(1076003)(6636002)(36860700001)(26005)(8676002)(107886003)(70206006)(4326008)(8936002)(508600001)(4744005)(7696005)(2906002)(47076005)(5660300002)(7636003)(82310400003)(86362001)(356005)(186003)(336012)(36756003)(2616005)(54906003)(316002)(70586007)(426003)(83380400001)(110136005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:40.6619 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1d54cdcf-763c-48b7-97f3-08d992ef8d1c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.36];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5293 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add a macro for VFIO_DEVICE_STATE_ERROR to be used to set/check an error state. Signed-off-by: Yishai Hadas --- include/uapi/linux/vfio.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 114ffcefe437..6d41a0f011db 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -631,6 +631,8 @@ struct vfio_device_migration_info { __u64 data_size; }; +#define VFIO_DEVICE_STATE_ERROR (VFIO_DEVICE_STATE_SAVING | \ + VFIO_DEVICE_STATE_RESUMING) /* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within From patchwork Tue Oct 19 10:58:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543211 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=MYnsdO30; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3g5Q8kz9sP7 for ; Tue, 19 Oct 2021 21:59:59 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235391AbhJSLCK (ORCPT ); Tue, 19 Oct 2021 07:02:10 -0400 Received: from mail-co1nam11on2080.outbound.protection.outlook.com ([40.107.220.80]:47073 "EHLO NAM11-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235388AbhJSLBz (ORCPT ); Tue, 19 Oct 2021 07:01:55 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E3dgd43EFCnSM+7fM+lxMcWMVmKb0csnqL5hoHRT6so94UXAyczA+Fe1tgbFriUu2vfBXKUdBiZuTOH67hM0QnJhwTCZ4ILTc6QpPruaz1kwuobpjvOSbSz/jlL0g28Yi9bu3upbvZ9NYSfRLpcjfZADwRbszrD3Vds5w2s3RZer18nHiASkIF32BrVOO6TH73d94kAxwPAJuTlQKNpyqffFUbhEGaiZE0ytZI3UcZWEgoma0hft78biaKBLQfSAHscPatppn5Q8yKS/aryWigG8Az8OEgHDZpJu4woi9B6SEPYxj0zDnDwUAfiuM94AfVIokbazXBat0WMM93autA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DOcFU6eSD++XS6aHs3kXEvGIdsHFeir924wVdQSptkg=; b=gK5nPEE8moLltUqWlTvNlhm18fOP3NhxOq0xIo1oPyKhSvk1Wg7jh/AM0ZQ7I9ZE9G8yopUb/RKLsa2KbckSfjIf0ifNC5OXZ5yhzweMNe4ItefB0D18d3i5ls5gUhnlwMraw/uufWa+5z39+u5pkbsqgAr3iqa2Lf+Ybr0UZk9PKV34bltsm+Ud+X9F9yxldwpypMDdpm5J0DSNTbbXJtGXO8meSLv+lE7oz1B+nzYF205pVz+xglYMOg7Kjv5YHYlvLHpI5jkJshzSL9kWButMtIwpDtONHKO+b+tfzd7ki1PRUX/OB/SzvZMmcEbE4jgoa3tcnx6WCCm3aRY0BQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DOcFU6eSD++XS6aHs3kXEvGIdsHFeir924wVdQSptkg=; b=MYnsdO30GLQeqBtmVvOO4FC2VDC+7ej0TN/SGiL+IXpeJbJ4RTN6TmGKcf0Z9mWJX2nJ/XPj3eTKxVXbh/MPyU5O+XPnLH58imtkIVXbzN4klS0+4OpUwGxgv/eGtryVeLA89HwXj5Tz3CCySqer91WWqibo8UxEgf85onNqq9IO8qGo4ocGDJXq2v6h8zHLi3xtBnCnuO89xRWvgK6rNTolleQGafz5SljjVdejATUw0sDFWYiHZ9DjbMEDwOKejAL75x/VxqluO/FiRNFDcQgt5wRn8Qlbx92t0x3UJlX/wrwXYFrVulX4cLQb68riQ2nHNFqOt/UaQ52df1TC2Q== Received: from DM5PR13CA0034.namprd13.prod.outlook.com (2603:10b6:3:7b::20) by BN8PR12MB3187.namprd12.prod.outlook.com (2603:10b6:408:69::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:41 +0000 Received: from DM6NAM11FT062.eop-nam11.prod.protection.outlook.com (2603:10b6:3:7b:cafe::d9) by DM5PR13CA0034.outlook.office365.com (2603:10b6:3:7b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.8 via Frontend Transport; Tue, 19 Oct 2021 10:59:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by DM6NAM11FT062.mail.protection.outlook.com (10.13.173.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:41 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:40 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:37 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 09/14] vfio/pci_core: Make the region->release() function optional Date: Tue, 19 Oct 2021 13:58:33 +0300 Message-ID: <20211019105838.227569-10-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9e174a3d-2f5b-4de7-dd36-08d992ef8d40 X-MS-TrafficTypeDiagnostic: BN8PR12MB3187: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DlLbkfjqLccmtm2ADlDsifX6AESuilpsmVvvGl4S1cqYw7/LsR85d/ot+tFXn0fSvvwgNvPxFJhTv3vnUZt716zvQvoi448PgEtdRzlvCOwnQlnQJbQpLl7TfhFb+Ikn3PV6t98tS6GIb5ikg2wOz8JtC4xdC8BJIQG1O4pG4bpzHJCW0XTg9O0WeWk9FaCQQAdCx/Yz08f9kX0TeKW07zPElUbfdKzrNG7oJs5FQbczRAmxH6vhnQdFQ4HYAO3tXlJZeJKL5Lve4Yjg99MiFEFW85H/xhBFa1/0aRcb6viQ5TyHxkTRxj47Kv7WxDCylmptqJnUv/ONJFZNN39UNGA6Bvlj0bOW4tFoNqj2bDIPDMueOWDRput3YKbA0/nPDxg5A8Tg5iB7PMmcQ2tVoHgMP17TEZmpvOzOB9e2yFjWLrO4BVGuBOBXgge9wzJ/R7BAuGGjDcySodUS29tKh9Wnvn0pyuqYyLTDcBL8mpFIjsCoir37F98TnYgKwyfGXktcZ0yoGv3rqrLLXBuByW0b1q4tRgkpbJDvDpetqiWRWjZ8EnG+2ehBrKUvW89VR5sOPYiW7PvjnJsqPrPUAPcS61qPCTlAAomhDlwvmt/HqD9hE4m6xBhTk+ezYQeS8PlU0aJC3uc9lCaiYq+OZtwODACEXiiz7jZi94WcrlIW84OUtV4OVWN8nvxCtuQcJppjwRBBLjU9tllqawGzzA== X-Forefront-Antispam-Report: CIP:216.228.112.35;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid04.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(4326008)(36906005)(316002)(6666004)(426003)(47076005)(6636002)(110136005)(83380400001)(5660300002)(36860700001)(36756003)(7636003)(356005)(54906003)(2906002)(82310400003)(7696005)(8676002)(186003)(70586007)(70206006)(508600001)(336012)(1076003)(8936002)(26005)(86362001)(2616005)(107886003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:41.3213 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9e174a3d-2f5b-4de7-dd36-08d992ef8d40 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.35];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT062.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3187 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Make the region->release() function optional as in some cases there is nothing to do by driver as part of it. This is needed for coming patch from this series once we add mlx5_vfio_pci driver to support live migration but we don't need a migration release function. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/vfio_pci_core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a03b5a99c2da..e581a327f90d 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -341,7 +341,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) vdev->virq_disabled = false; for (i = 0; i < vdev->num_regions; i++) - vdev->region[i].ops->release(vdev, &vdev->region[i]); + if (vdev->region[i].ops->release) + vdev->region[i].ops->release(vdev, &vdev->region[i]); vdev->num_regions = 0; kfree(vdev->region); From patchwork Tue Oct 19 10:58:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543212 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=fIB/RfTj; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3t4Klyz9sPB for ; Tue, 19 Oct 2021 22:00:10 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235434AbhJSLCU (ORCPT ); Tue, 19 Oct 2021 07:02:20 -0400 Received: from mail-dm6nam11on2083.outbound.protection.outlook.com ([40.107.223.83]:46529 "EHLO NAM11-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235413AbhJSLCD (ORCPT ); Tue, 19 Oct 2021 07:02:03 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Oa9w8xGyD5V7PncRZfVO+buOL7gdNGFpTn0P1fN4sDkAP+b3qlLvp4jH6B2xDz1WaG7qvSLlCKOa1e70nD3zcbjc4YHuPl4a+ULUeAjT16on7E3XrrBMKWLz8Rka1Er9jvFuE4LLGOyO2lToBIHL5wy0QFNWyWHPtS3X121/4HZzB4X1sh7dQeL9tLCldG4SiBUuCLZBJlZrE9mw+m3GmUVVnGp8Q+IQNJ9tpKiCm539hRLhUt5ckT2IRSS9kRBWH2nYfFana7k5piX7hnjBFd0k0eB2ct8iYcFJkkR+8Wcx68Oo+BpFtVikXG2t1BWwKXTPLRkBp6cVE5uDm7tAVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R7DbtBQIcZF4LUA2VaUaiDzMK1m8SVbr7xND56mts4w=; b=KCk1DWrsP10xbdhwM0DXnh4sYZLtqVoCU9PC4ruuN/npGDJmXADo8idIjZUy90pjvvmGlu4BECACU7O0DbKQJABrbhKPR06UrvVbTcVn0SwBM3WW/Zxy4x/ZyjSwZR874hySi1/Sgu0kOwjRNyF8pIY4pEyB34QAeg3pk5MAgz6hdN5zbGA06T0CwHhDs7euBRu7KrOBLPG2euf7AtNZ4Q6FiVSO4xWuP4oTt1q8Tv3fn0y+JuX6GNmX8PnxDrpGPnTVRAQ21/s310zn2z+kZtBTapvvaZ4qMn4AZ37+R+MklWgEDp3WyoTnBgxCCjrYJnOOvofzHFa5/qE4K62MAg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R7DbtBQIcZF4LUA2VaUaiDzMK1m8SVbr7xND56mts4w=; b=fIB/RfTj62nwRMInn3T3upxC4Dr9MJpTdbpPdcoj54/vrrRakOxCegEX5j437Coy0qMiegQ/+py7v53MSg+x4LgafG8z0XCas5ErlHUPszeS8us0Wfq9WcHjXMNOfoC8RGXtYUy8k8Vy2h6XbIav0TzEHPtIaZcOa/zAyCPv/JnCll0SzqRniYarEnVCqm4lIsr0a/PUVaZEECwS2rDBkY2qsW8x4AtsimvKXPtOPMzCRMm2rPOMDg4cxYFHrqUkpS67Oi7v96N0NOZVPktfEY+MhI4RgVrSY47bjp3TD8cDBX4nb78eOV5ZqP/hMSAKCB6dk8oDedo1cFw2PA18SA== Received: from BN8PR04CA0016.namprd04.prod.outlook.com (2603:10b6:408:70::29) by DM6PR12MB4577.namprd12.prod.outlook.com (2603:10b6:5:2aa::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Tue, 19 Oct 2021 10:59:46 +0000 Received: from BN8NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::b1) by BN8PR04CA0016.outlook.office365.com (2603:10b6:408:70::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17 via Frontend Transport; Tue, 19 Oct 2021 10:59:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT051.mail.protection.outlook.com (10.13.177.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:45 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:44 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:43 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:40 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 10/14] net/mlx5: Introduce migration bits and structures Date: Tue, 19 Oct 2021 13:58:34 +0300 Message-ID: <20211019105838.227569-11-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cb2ac369-4f0a-4c91-25c6-08d992ef9045 X-MS-TrafficTypeDiagnostic: DM6PR12MB4577: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nl8iu8j93R+x4kInlvE5DSFsbtpZaIofFenWxmgJk1CsobDS49Fu26hAqmgEbC3iW+R7l9EEwyIWEshiODfmlx9KBg79tVQToARIIt7YRTfP24rVQSwdVyGTmVCcO2QGZ7pp5ecT+KdPrBpSnJLgeRbxjEC2Yq3XAhX6qkAINnBULfz0TodH1YKGVYNaE+bX4V9zOymLUnh/SYkT7HfucC/qAautYe+wglOV66JDzR/VLmDoSz8IAWTYe6TysvUlDj+tqHbJGRIU46UU/mmL5gZa1+UyTxoj5CsIFt0W2Ub6DgLzQL3HL6tvStSYJ3pOLM24818xb8x/ckWCo+//+Pxos4J2cPwObB6glS4G9Rb4MrJYmLmiGWFvCKG4vod3GVhJhQNZtaTy1mbY8dFtZz038eeVQmODoPnVfPKQEPcmk+RNFYE+PY4oGr5YIQB8BfJ+U9fkx/cxEx1G7f72ZlZY6ITtI9j3TtEjzEEHvo7pMd16cupWqhy7JA0UZ+wLgf6FfNy4DUIRRkeVKZSxwfaNk6WKAI5rw7RKjVRy5R3Fj+Ne8jw+P2iYpWRL+1oH8rGTIWEib2KSEZl7iYbfLHQo5Mi2h2vdqVXyr0IgktmvAYy7qMKnF/P7OlFx+a9T3yJv4fOIVNLnAR/+fHCP0j2h48GYtpN4VrsbogxgG1BHy2r2HeFpScd8KqESCx4rHKLim8WqjNZcyNoEuRvpTQ== X-Forefront-Antispam-Report: CIP:216.228.112.36;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid05.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(26005)(356005)(186003)(82310400003)(426003)(2616005)(107886003)(5660300002)(6636002)(508600001)(83380400001)(54906003)(86362001)(316002)(4326008)(336012)(1076003)(110136005)(2906002)(7696005)(70206006)(70586007)(47076005)(36756003)(7636003)(8676002)(36860700001)(8936002)(6666004);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:45.9498 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cb2ac369-4f0a-4c91-25c6-08d992ef9045 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.36];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4577 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Introduce migration IFC related stuff to enable migration commands. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- include/linux/mlx5/mlx5_ifc.h | 145 +++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 399ea52171fe..f7bad4ccc24f 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -126,6 +126,11 @@ enum { MLX5_CMD_OP_QUERY_SF_PARTITION = 0x111, MLX5_CMD_OP_ALLOC_SF = 0x113, MLX5_CMD_OP_DEALLOC_SF = 0x114, + MLX5_CMD_OP_SUSPEND_VHCA = 0x115, + MLX5_CMD_OP_RESUME_VHCA = 0x116, + MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE = 0x117, + MLX5_CMD_OP_SAVE_VHCA_STATE = 0x118, + MLX5_CMD_OP_LOAD_VHCA_STATE = 0x119, MLX5_CMD_OP_CREATE_MKEY = 0x200, MLX5_CMD_OP_QUERY_MKEY = 0x201, MLX5_CMD_OP_DESTROY_MKEY = 0x202, @@ -1719,7 +1724,9 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 reserved_at_682[0x1]; u8 log_max_sf[0x5]; u8 apu[0x1]; - u8 reserved_at_689[0x7]; + u8 reserved_at_689[0x4]; + u8 migration[0x1]; + u8 reserved_at_68e[0x2]; u8 log_min_sf_size[0x8]; u8 max_num_sf_partitions[0x8]; @@ -11146,4 +11153,140 @@ enum { MLX5_MTT_PERM_RW = MLX5_MTT_PERM_READ | MLX5_MTT_PERM_WRITE, }; +enum { + MLX5_SUSPEND_VHCA_IN_OP_MOD_SUSPEND_MASTER = 0x0, + MLX5_SUSPEND_VHCA_IN_OP_MOD_SUSPEND_SLAVE = 0x1, +}; + +struct mlx5_ifc_suspend_vhca_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x10]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_suspend_vhca_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; +}; + +enum { + MLX5_RESUME_VHCA_IN_OP_MOD_RESUME_SLAVE = 0x0, + MLX5_RESUME_VHCA_IN_OP_MOD_RESUME_MASTER = 0x1, +}; + +struct mlx5_ifc_resume_vhca_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x10]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_resume_vhca_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_query_vhca_migration_state_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x10]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_query_vhca_migration_state_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; + + u8 required_umem_size[0x20]; + + u8 reserved_at_a0[0x160]; +}; + +struct mlx5_ifc_save_vhca_state_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x10]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x20]; + + u8 va[0x40]; + + u8 mkey[0x20]; + + u8 size[0x20]; +}; + +struct mlx5_ifc_save_vhca_state_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_load_vhca_state_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x10]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x20]; + + u8 va[0x40]; + + u8 mkey[0x20]; + + u8 size[0x20]; +}; + +struct mlx5_ifc_load_vhca_state_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; +}; + #endif /* MLX5_IFC_H */ From patchwork Tue Oct 19 10:58:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543213 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=TEpyO9c2; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW3z5Vjsz9sPB for ; Tue, 19 Oct 2021 22:00:15 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235487AbhJSLCX (ORCPT ); Tue, 19 Oct 2021 07:02:23 -0400 Received: from mail-bn8nam11on2073.outbound.protection.outlook.com ([40.107.236.73]:23392 "EHLO NAM11-BN8-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235360AbhJSLCH (ORCPT ); Tue, 19 Oct 2021 07:02:07 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gPvXp4dyZBhbt182gVy7YqmvLNB3oouPqFqzhTF6xMuLs0QNAtBcSz8i5Q+nWLhEQ/jIeOFRc+IAWZF2lKP0cIqi+zBadZtua/Y3EciBrAQNdwJy4szhwQP0jh1Uk7noOIybbspykYDlMjswYwm+jBOhKEPdypVGYMDxjtOYzXuwKiS3AKWwDEYxnbKIIb957bik8A9PS1bH5A5BkM//9/GQPexRZevBO0DIzFkmvyGbL8dlAmphdLKhvPFh5LzTjGVDNcQPpsQEbiAJWotXzfPENytTtczkmJN4deJpZPFf6rhcCIupWv0ePbEnB4vBz5LMoT4KHB5aZKDMF6kdOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SDzDerJ8BHTpflGMC1W6xDcWZWOCwERj9bPjORhyA/I=; b=MCgA1fcBOlUOKm2SAqgShEw/0fRzJmH74+CqEQCpvY/0OctGngt/xAH5aGEdmAMIR+p4JQEWnJ2L2XXONQELa61gyM85Jmb4/Z4OlyGjB/DPrKFiVreR8AudJNcZm88cLPYfXiboWIMBPKt4o/Z5CJElx23pDhEi+/zH0FMDq/6vH1qOfCPYwLjjhlDKZLaXrNesw28tAllmY386YgIVoKNWShJxmgvLg2R7TDEIBiOxkti6Fb5Yzo387hdPOCzdGkMRXyjWgRNkZkgvUrBLqHyefzl0E37UNk2VLfxykLf0OkjGWsw1lhx2va1WV6r9rLZmZo4YSv5OjHYng4HJNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SDzDerJ8BHTpflGMC1W6xDcWZWOCwERj9bPjORhyA/I=; b=TEpyO9c2iQPM26zsLlTWKAK/jApfCJGyzQ/I+c5Tfx7muiGsCrvQOigEdvQ+1etFojEZEXS0Ck5TRfRyViAnQdExuXd0TWxTABlHZ1Jd6qwtR8LlfRk7XirjIQ4xpRE3VIhJKdZ5NBnUjK14yBmllqAlKieEXzIHheqvdPU57SRnsd5FkbhF5Ycrnc0km56h1Auo1Mxk7IIK3c5dVxuvsHl16yhF1jH34y4nXEjFfif+FvNpXzBPkfDu3EgYPHILspT56uahFn6SNE8XrbSGeSoK3jNMsqcfb5VvRgXqs0gT7WqQrVKK1yoUIbXcdja7fWqznRkzM8cSP8Z5c/zRRw== Received: from BN1PR10CA0028.namprd10.prod.outlook.com (2603:10b6:408:e0::33) by BN8PR12MB3427.namprd12.prod.outlook.com (2603:10b6:408:62::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:52 +0000 Received: from BN8NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e0:cafe::40) by BN1PR10CA0028.outlook.office365.com (2603:10b6:408:e0::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT011.mail.protection.outlook.com (10.13.176.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:49 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:47 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:44 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 11/14] vfio/mlx5: Expose migration commands over mlx5 device Date: Tue, 19 Oct 2021 13:58:35 +0300 Message-ID: <20211019105838.227569-12-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f0cc8075-a280-4b71-0cff-08d992ef9254 X-MS-TrafficTypeDiagnostic: BN8PR12MB3427: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AGsW2Yvoyydgo/TcnAJ7bF81UF3pZrRoxG7I13S/Yju9PJvi1FLcMRt11POt2CpBB63stYohRxvPR64ENWlKGwRTQca+bIAGxszATdZFjYAJRTtvYN2FNf95lPeQB1KyOQ+0ap0I7uuCt7dkkDviO4+j7CZI/Jj44sttIrh4nHovZy4AMcWIHKvFTMBAK1++O8aydVz+C4ZwLWn6KizRvfkktCDxcdqZV554wKoS0oB02atyv6BmcrHpaAuK5Wdw8okdGJoMiMz18bm4Mzo6Mc10rCb3VNPaycHndpNALw8MPQH4fnAzJmNaKZlZbwAzNR/UI0rV0JdgiTGM28IHJzWjQTRLxDrDxiMNLjqlo+KV9nwlMqM9/u9amnym85xHkeOqF8kLHoKVVVEo746azBQJC+fPTgYOdlQaSn1aD/kKlwEI504cVPyJk+aHimsjefrdck5G0c5GfO9kuO0CLBFW9xod4pHm+aIEwBEeojpF9sEIYYIDxNendf1w8LH/9VsDdfrc/xcqhF+y7J8gGtkHjvEAVCU0safNcv5KGsvAbiRbN5Q+oiCi7D80Vod061APyKhHJiiYXmXt5r21DypqIjxcaD/A2V5cYp3lAeXCUb88mXWd4Jw+o8nTa6suIBn3kZpt2k0ivOThEuNmpymcpoU64+ES7RPFB+WLXV08UwLcbOSIOC/okUmPsEDI+aUR4wFbpq30hH0e4Q5iOQ== X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(82310400003)(47076005)(107886003)(6666004)(7696005)(2616005)(36756003)(5660300002)(70586007)(30864003)(86362001)(2906002)(70206006)(8676002)(110136005)(316002)(186003)(8936002)(7636003)(356005)(54906003)(426003)(336012)(6636002)(26005)(508600001)(83380400001)(4326008)(36860700001)(1076003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:49.6908 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f0cc8075-a280-4b71-0cff-08d992ef9254 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3427 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Expose migration commands over the device, it includes: suspend, resume, get vhca id, query/save/load state. As part of this adds the APIs and data structure that are needed to manage the migration data. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 353 ++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/mlx5/cmd.h | 43 +++++ 2 files changed, 396 insertions(+) create mode 100644 drivers/vfio/pci/mlx5/cmd.c create mode 100644 drivers/vfio/pci/mlx5/cmd.h diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c new file mode 100644 index 000000000000..5b24a7625b8a --- /dev/null +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -0,0 +1,353 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include "cmd.h" + +int mlx5vf_cmd_suspend_vhca(struct pci_dev *pdev, u16 vhca_id, u16 op_mod) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 out[MLX5_ST_SZ_DW(suspend_vhca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(suspend_vhca_in)] = {}; + int ret; + + if (!mdev) + return -ENOTCONN; + + MLX5_SET(suspend_vhca_in, in, opcode, MLX5_CMD_OP_SUSPEND_VHCA); + MLX5_SET(suspend_vhca_in, in, vhca_id, vhca_id); + MLX5_SET(suspend_vhca_in, in, op_mod, op_mod); + + ret = mlx5_cmd_exec_inout(mdev, suspend_vhca, in, out); + mlx5_vf_put_core_dev(mdev); + return ret; +} + +int mlx5vf_cmd_resume_vhca(struct pci_dev *pdev, u16 vhca_id, u16 op_mod) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 out[MLX5_ST_SZ_DW(resume_vhca_out)] = {}; + u32 in[MLX5_ST_SZ_DW(resume_vhca_in)] = {}; + int ret; + + if (!mdev) + return -ENOTCONN; + + MLX5_SET(resume_vhca_in, in, opcode, MLX5_CMD_OP_RESUME_VHCA); + MLX5_SET(resume_vhca_in, in, vhca_id, vhca_id); + MLX5_SET(resume_vhca_in, in, op_mod, op_mod); + + ret = mlx5_cmd_exec_inout(mdev, resume_vhca, in, out); + mlx5_vf_put_core_dev(mdev); + return ret; +} + +int mlx5vf_cmd_query_vhca_migration_state(struct pci_dev *pdev, u16 vhca_id, + u32 *state_size) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 out[MLX5_ST_SZ_DW(query_vhca_migration_state_out)] = {}; + u32 in[MLX5_ST_SZ_DW(query_vhca_migration_state_in)] = {}; + int ret; + + if (!mdev) + return -ENOTCONN; + + MLX5_SET(query_vhca_migration_state_in, in, opcode, + MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE); + MLX5_SET(query_vhca_migration_state_in, in, vhca_id, vhca_id); + MLX5_SET(query_vhca_migration_state_in, in, op_mod, 0); + + ret = mlx5_cmd_exec_inout(mdev, query_vhca_migration_state, in, out); + if (ret) + goto end; + + *state_size = MLX5_GET(query_vhca_migration_state_out, out, + required_umem_size); + +end: + mlx5_vf_put_core_dev(mdev); + return ret; +} + +int mlx5vf_cmd_get_vhca_id(struct pci_dev *pdev, u16 function_id, u16 *vhca_id) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {}; + int out_size; + void *out; + int ret; + + if (!mdev) + return -ENOTCONN; + + out_size = MLX5_ST_SZ_BYTES(query_hca_cap_out); + out = kzalloc(out_size, GFP_KERNEL); + if (!out) { + ret = -ENOMEM; + goto end; + } + + MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + MLX5_SET(query_hca_cap_in, in, other_function, 1); + MLX5_SET(query_hca_cap_in, in, function_id, function_id); + MLX5_SET(query_hca_cap_in, in, op_mod, + MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE << 1 | + HCA_CAP_OPMOD_GET_CUR); + + ret = mlx5_cmd_exec_inout(mdev, query_hca_cap, in, out); + if (ret) + goto err_exec; + + *vhca_id = MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.vhca_id); + +err_exec: + kfree(out); +end: + mlx5_vf_put_core_dev(mdev); + return ret; +} + +static int _create_state_mkey(struct mlx5_core_dev *mdev, u32 pdn, + struct mlx5_vhca_state_data *state, u32 *mkey) +{ + struct sg_dma_page_iter dma_iter; + int err = 0, inlen; + __be64 *mtt; + void *mkc; + u32 *in; + + inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + + sizeof(*mtt) * round_up(state->num_pages, 2); + + in = kvzalloc(inlen, GFP_KERNEL); + if (!in) + return -ENOMEM; + + MLX5_SET(create_mkey_in, in, translations_octword_actual_size, + DIV_ROUND_UP(state->num_pages, 2)); + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt); + + for_each_sgtable_dma_page(&state->mig_data.table.sgt, &dma_iter, 0) + *mtt++ = cpu_to_be64(sg_page_iter_dma_address(&dma_iter)); + + mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); + MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_MTT); + MLX5_SET(mkc, mkc, lr, 1); + MLX5_SET(mkc, mkc, lw, 1); + MLX5_SET(mkc, mkc, pd, pdn); + MLX5_SET(mkc, mkc, bsf_octword_size, 0); + MLX5_SET(mkc, mkc, qpn, 0xffffff); + MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); + MLX5_SET(mkc, mkc, translations_octword_size, + DIV_ROUND_UP(state->num_pages, 2)); + MLX5_SET64(mkc, mkc, len, state->num_pages * PAGE_SIZE); + err = mlx5_core_create_mkey(mdev, mkey, in, inlen); + + kvfree(in); + + return err; +} + +struct page *mlx5vf_get_migration_page(struct migration_data *data, + unsigned long offset) +{ + unsigned long cur_offset = 0; + struct scatterlist *sg; + unsigned int i; + + if (offset < data->last_offset || !data->last_offset_sg) { + data->last_offset = 0; + data->last_offset_sg = data->table.sgt.sgl; + data->sg_last_entry = 0; + } + + cur_offset = data->last_offset; + + for_each_sg(data->last_offset_sg, sg, + data->table.sgt.orig_nents - data->sg_last_entry, i) { + if (offset < sg->length + cur_offset) { + data->last_offset_sg = sg; + data->sg_last_entry += i; + data->last_offset = cur_offset; + return nth_page(sg_page(sg), + (offset - cur_offset) / PAGE_SIZE); + } + cur_offset += sg->length; + } + return NULL; +} + +void mlx5vf_reset_vhca_state(struct mlx5_vhca_state_data *state) +{ + struct migration_data *data = &state->mig_data; + struct sg_page_iter sg_iter; + + if (!data->table.prv) + goto end; + + /* Undo alloc_pages_bulk_array() */ + for_each_sgtable_page(&data->table.sgt, &sg_iter, 0) + __free_page(sg_page_iter_page(&sg_iter)); + sg_free_append_table(&data->table); +end: + memset(state, 0, sizeof(*state)); +} + +int mlx5vf_add_migration_pages(struct mlx5_vhca_state_data *state, + unsigned int npages) +{ + unsigned int to_alloc = npages; + struct page **page_list; + unsigned long filled; + unsigned int to_fill; + int ret = 0; + + to_fill = min_t(unsigned int, npages, PAGE_SIZE / sizeof(*page_list)); + page_list = kvzalloc(to_fill * sizeof(*page_list), GFP_KERNEL); + if (!page_list) + return -ENOMEM; + + do { + filled = alloc_pages_bulk_array(GFP_KERNEL, to_fill, + page_list); + if (!filled) { + ret = -ENOMEM; + goto err; + } + to_alloc -= filled; + ret = sg_alloc_append_table_from_pages( + &state->mig_data.table, page_list, filled, 0, + filled << PAGE_SHIFT, UINT_MAX, SG_MAX_SINGLE_ALLOC, + GFP_KERNEL); + + if (ret) + goto err; + /* clean input for another bulk allocation */ + memset(page_list, 0, filled * sizeof(*page_list)); + to_fill = min_t(unsigned int, to_alloc, + PAGE_SIZE / sizeof(*page_list)); + } while (to_alloc > 0); + + kvfree(page_list); + state->num_pages += npages; + + return 0; + +err: + kvfree(page_list); + return ret; +} + +int mlx5vf_cmd_save_vhca_state(struct pci_dev *pdev, u16 vhca_id, + u64 state_size, + struct mlx5_vhca_state_data *state) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 out[MLX5_ST_SZ_DW(save_vhca_state_out)] = {}; + u32 in[MLX5_ST_SZ_DW(save_vhca_state_in)] = {}; + u32 pdn, mkey; + int err; + + if (!mdev) + return -ENOTCONN; + + err = mlx5_core_alloc_pd(mdev, &pdn); + if (err) + goto end; + + err = mlx5vf_add_migration_pages(state, + DIV_ROUND_UP_ULL(state_size, PAGE_SIZE)); + if (err < 0) + goto err_alloc_pages; + + err = dma_map_sgtable(mdev->device, &state->mig_data.table.sgt, + DMA_FROM_DEVICE, 0); + if (err) + goto err_reg_dma; + + err = _create_state_mkey(mdev, pdn, state, &mkey); + if (err) + goto err_create_mkey; + + MLX5_SET(save_vhca_state_in, in, opcode, + MLX5_CMD_OP_SAVE_VHCA_STATE); + MLX5_SET(save_vhca_state_in, in, op_mod, 0); + MLX5_SET(save_vhca_state_in, in, vhca_id, vhca_id); + MLX5_SET(save_vhca_state_in, in, mkey, mkey); + MLX5_SET(save_vhca_state_in, in, size, state_size); + + err = mlx5_cmd_exec_inout(mdev, save_vhca_state, in, out); + if (err) + goto err_exec; + + state->state_size = state_size; + + mlx5_core_destroy_mkey(mdev, mkey); + mlx5_core_dealloc_pd(mdev, pdn); + dma_unmap_sgtable(mdev->device, &state->mig_data.table.sgt, + DMA_FROM_DEVICE, 0); + mlx5_vf_put_core_dev(mdev); + + return 0; + +err_exec: + mlx5_core_destroy_mkey(mdev, mkey); +err_create_mkey: + dma_unmap_sgtable(mdev->device, &state->mig_data.table.sgt, + DMA_FROM_DEVICE, 0); +err_reg_dma: + mlx5vf_reset_vhca_state(state); +err_alloc_pages: + mlx5_core_dealloc_pd(mdev, pdn); +end: + mlx5_vf_put_core_dev(mdev); + return err; +} + +int mlx5vf_cmd_load_vhca_state(struct pci_dev *pdev, u16 vhca_id, + struct mlx5_vhca_state_data *state) +{ + struct mlx5_core_dev *mdev = mlx5_vf_get_core_dev(pdev); + u32 out[MLX5_ST_SZ_DW(save_vhca_state_out)] = {}; + u32 in[MLX5_ST_SZ_DW(save_vhca_state_in)] = {}; + u32 pdn, mkey; + int err; + + if (!mdev) + return -ENOTCONN; + + err = mlx5_core_alloc_pd(mdev, &pdn); + if (err) + goto end; + + err = dma_map_sgtable(mdev->device, &state->mig_data.table.sgt, + DMA_TO_DEVICE, 0); + if (err) + goto err_reg; + + err = _create_state_mkey(mdev, pdn, state, &mkey); + if (err) + goto err_mkey; + + MLX5_SET(load_vhca_state_in, in, opcode, + MLX5_CMD_OP_LOAD_VHCA_STATE); + MLX5_SET(load_vhca_state_in, in, op_mod, 0); + MLX5_SET(load_vhca_state_in, in, vhca_id, vhca_id); + MLX5_SET(load_vhca_state_in, in, mkey, mkey); + MLX5_SET(load_vhca_state_in, in, size, state->state_size); + + err = mlx5_cmd_exec_inout(mdev, load_vhca_state, in, out); + + mlx5_core_destroy_mkey(mdev, mkey); +err_mkey: + dma_unmap_sgtable(mdev->device, &state->mig_data.table.sgt, + DMA_TO_DEVICE, 0); +err_reg: + mlx5_core_dealloc_pd(mdev, pdn); +end: + mlx5_vf_put_core_dev(mdev); + return err; +} diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h new file mode 100644 index 000000000000..66221df24b19 --- /dev/null +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* + * Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + */ + +#ifndef MLX5_VFIO_CMD_H +#define MLX5_VFIO_CMD_H + +#include +#include + +struct migration_data { + struct sg_append_table table; + + struct scatterlist *last_offset_sg; + unsigned int sg_last_entry; + unsigned long last_offset; +}; + +/* state data of vhca to be used as part of migration flow */ +struct mlx5_vhca_state_data { + u64 state_size; + u64 num_pages; + u32 win_start_offset; + struct migration_data mig_data; +}; + +int mlx5vf_cmd_suspend_vhca(struct pci_dev *pdev, u16 vhca_id, u16 op_mod); +int mlx5vf_cmd_resume_vhca(struct pci_dev *pdev, u16 vhca_id, u16 op_mod); +int mlx5vf_cmd_query_vhca_migration_state(struct pci_dev *pdev, u16 vhca_id, + uint32_t *state_size); +int mlx5vf_cmd_get_vhca_id(struct pci_dev *pdev, u16 function_id, u16 *vhca_id); +int mlx5vf_cmd_save_vhca_state(struct pci_dev *pdev, u16 vhca_id, + u64 state_size, + struct mlx5_vhca_state_data *state); +void mlx5vf_reset_vhca_state(struct mlx5_vhca_state_data *state); +int mlx5vf_cmd_load_vhca_state(struct pci_dev *pdev, u16 vhca_id, + struct mlx5_vhca_state_data *state); +int mlx5vf_add_migration_pages(struct mlx5_vhca_state_data *state, + unsigned int npages); +struct page *mlx5vf_get_migration_page(struct migration_data *data, + unsigned long offset); +#endif /* MLX5_VFIO_CMD_H */ From patchwork Tue Oct 19 10:58:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543214 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=ISxM2+/+; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW451K1hz9sP7 for ; Tue, 19 Oct 2021 22:00:21 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235517AbhJSLCa (ORCPT ); Tue, 19 Oct 2021 07:02:30 -0400 Received: from mail-mw2nam10on2064.outbound.protection.outlook.com ([40.107.94.64]:64256 "EHLO NAM10-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235363AbhJSLCI (ORCPT ); Tue, 19 Oct 2021 07:02:08 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=csEz72a4FabbkPRvX+qLrGzKqmWqPK1tjgUiL7QBZcV1R0+MlTWtxbXlquAxBGWMU+vvjsesZmmvHd8hviPiMURYwCF+rkdJOskaNI7OBvWlgOdmyXVjMOQ6ABT5DE+ELiTx/uYQ2ZtgCmwdcJJPlQjoHfZld/4MrdwT+ox+hKbyMCuQko3UV8f27yF4dUKCmxZjGqZHu6o/v8T8O8DAZvlx03TalNJa/lJJ/uHxXAv2WoAH+Fr6j0xzjHpOHx4qCqhV5TrwKLdUq8Vi6d0cZ6AFIuBIk+nuq2bvyu+dnsgjjCgIir4rtPeMkJpmtKx+2f54VXJtzRjlXsBG5pW24w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lblNMaWWcj0DFsNFjAibivTlin7EtvLAE09+tjeG798=; b=WNZBb6tKDDdfvDCAVPWOISCIqMrSikB2YeK3M2l4M3VXHb2WYwKOch5+aGe5Ls/POPoy6jn6MZqS0m9ceKibN7U0MWqLnKy1Qa6P6tBsZ3R6egR13i/G/crY5tvxP37DnUaKdTe6zPBusmYF0EM1uUM2XUYc+lLYamW4+4OJcZKl3AsPcw6ciZ8j7WhZZvtIyKO8Lkk2UdmpW/crVr0As/fADMpM5+tr4azPXUy+qsb3d5WCJ3UG2dggSOCn1AFQdhOPXv15ZTiV5Z7l92gLUpgHcCtF6y+6mptITC4RJw2M8Dr8M2D8Z1tC7PdguqnWx4d3IrsKpHTC56BLXL1rjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lblNMaWWcj0DFsNFjAibivTlin7EtvLAE09+tjeG798=; b=ISxM2+/+ad2fLrxN1xmCC6tnpU76GoidsrEBlFr9AdY0/PUiz8g3BT2qN9h5BQtrPtP8dldVrJoRAT6TKnymow7t+WADnozts6TJtvc9ttSbp61kb8FVpe9FIVl7QmGvJleSIW8bE5RbGI16GRH2mTBMO7ybPjkZiN1aTbsKb+uuFJzStHa6g1+z8q7ia1UIjwkXEwxJYFEcVyZZTyPoSZ7nNwYFMYPHofK0hqMz+XwzfVudd9fSeZjlDzyMCrY3eOUS85MHx/WNrmKm4GJk7z4jKE85Ehco0o+i2w9IbZ2jMsJNhTPB2fz+z5ajIWXblswqNo1GxLZyryZvKGaRBA== Received: from BN1PR10CA0017.namprd10.prod.outlook.com (2603:10b6:408:e0::22) by BL1PR12MB5045.namprd12.prod.outlook.com (2603:10b6:208:310::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Tue, 19 Oct 2021 10:59:53 +0000 Received: from BN8NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e0:cafe::90) by BN1PR10CA0017.outlook.office365.com (2603:10b6:408:e0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT011.mail.protection.outlook.com (10.13.176.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:53 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:50 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:50 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:47 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 12/14] vfio/mlx5: Implement vfio_pci driver for mlx5 devices Date: Tue, 19 Oct 2021 13:58:36 +0300 Message-ID: <20211019105838.227569-13-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 215f6ace-2fa9-433e-65f0-08d992ef94b2 X-MS-TrafficTypeDiagnostic: BL1PR12MB5045: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zIRufIzEV1YMOiZOcQeexmP/3ni1CsOaBnQE9mGG4wzX1YKjve03K+UmKA+giUNSPtSXwiEuEa4Euqqq5nGtT9QuPFbc2TGEB+U7bd2eOhiz8sblxgPux5bEduGMbHlNY58rmdjKguPLTPrkZVfK+6/gZlGufi6Z0kJ73bsohDe8tEzPPOB0MawiIPNKEqVXq599zhLIPFHSd9j7yt7bkkBQKTtR8KCuqftczWqWmdHZl5DjADujTWIZKMNNyxd+/iGv5AIJRUwS22lZaLy8CbB1YprRebon9rw42qhXtLjtjTUxJM1bmSEFiiw0gPsV9sYMatDJgw9eT6UO6DXdXyaBZbyPN/vS4CXcfsQailg8iqAfJultHDBNu71NjkoSKJ1I6tWzbSVNXbRYOjJgb9vzWPWTKlyByqgeNR/ITF0QUWkc6871RtacSEml7/ukaBY0UPO5ZjW03gzCrruOLTKRW6m7Ttsr16y95LByALvT03bGKDNyTzUI4ayv96IM8wTjkPVuoPFeC+x3rvbCsD7P8EkL6hxak7TEc0QCKF90fhOv2wTVSt85GbT31MeB2E1hA7TMpf982SvHrxrhSHuAqIShxx4oEY82a1Fk0dSmQ97kWoDmBzMl2jZZk8BCTlTzvR8wrfVZLeeoknZFMJJrdhTLCJNswcoZuGInsA9O9AjeEYvUR9o6vOqQdsNwGLNqID2+Lq+n2yAWRNdA/kVq+I6XG70eadDRYNCSqtI= X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(2616005)(70206006)(426003)(70586007)(336012)(36756003)(86362001)(356005)(316002)(54906003)(8936002)(107886003)(508600001)(26005)(1076003)(82310400003)(6666004)(7696005)(6636002)(7636003)(36860700001)(4326008)(110136005)(83380400001)(2906002)(30864003)(47076005)(36906005)(186003)(5660300002)(8676002)(21314003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:53.7445 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 215f6ace-2fa9-433e-65f0-08d992ef94b2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5045 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This patch adds support for vfio_pci driver for mlx5 devices. It uses vfio_pci_core to register to the VFIO subsystem and then implements the mlx5 specific logic in the migration area. The migration implementation follows the definition from uapi/vfio.h and uses the mlx5 VF->PF command channel to achieve it. This patch implements the suspend/resume flows. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- MAINTAINERS | 6 + drivers/vfio/pci/Kconfig | 3 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/mlx5/Kconfig | 10 + drivers/vfio/pci/mlx5/Makefile | 4 + drivers/vfio/pci/mlx5/main.c | 696 +++++++++++++++++++++++++++++++++ 6 files changed, 721 insertions(+) create mode 100644 drivers/vfio/pci/mlx5/Kconfig create mode 100644 drivers/vfio/pci/mlx5/Makefile create mode 100644 drivers/vfio/pci/mlx5/main.c diff --git a/MAINTAINERS b/MAINTAINERS index abdcbcfef73d..e824bfab4a01 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -19699,6 +19699,12 @@ L: kvm@vger.kernel.org S: Maintained F: drivers/vfio/platform/ +VFIO MLX5 PCI DRIVER +M: Yishai Hadas +L: kvm@vger.kernel.org +S: Maintained +F: drivers/vfio/pci/mlx5/ + VGA_SWITCHEROO R: Lukas Wunner S: Maintained diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig index 860424ccda1b..187b9c259944 100644 --- a/drivers/vfio/pci/Kconfig +++ b/drivers/vfio/pci/Kconfig @@ -43,4 +43,7 @@ config VFIO_PCI_IGD To enable Intel IGD assignment through vfio-pci, say Y. endif + +source "drivers/vfio/pci/mlx5/Kconfig" + endif diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index 349d68d242b4..ed9d6f2e0555 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -7,3 +7,5 @@ obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o vfio-pci-y := vfio_pci.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o obj-$(CONFIG_VFIO_PCI) += vfio-pci.o + +obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ diff --git a/drivers/vfio/pci/mlx5/Kconfig b/drivers/vfio/pci/mlx5/Kconfig new file mode 100644 index 000000000000..119712656400 --- /dev/null +++ b/drivers/vfio/pci/mlx5/Kconfig @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +config MLX5_VFIO_PCI + tristate "VFIO support for MLX5 PCI devices" + depends on MLX5_CORE + select VFIO_PCI_CORE + help + This provides a migration support for MLX5 devices using the VFIO + framework. + + If you don't know what to do here, say N. diff --git a/drivers/vfio/pci/mlx5/Makefile b/drivers/vfio/pci/mlx5/Makefile new file mode 100644 index 000000000000..689627da7ff5 --- /dev/null +++ b/drivers/vfio/pci/mlx5/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5-vfio-pci.o +mlx5-vfio-pci-y := main.o cmd.o + diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c new file mode 100644 index 000000000000..621b7fc60544 --- /dev/null +++ b/drivers/vfio/pci/mlx5/main.c @@ -0,0 +1,696 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "cmd.h" + +enum { + MLX5VF_PCI_FREEZED = 1 << 0, +}; + +enum { + MLX5VF_REGION_PENDING_BYTES = 1 << 0, + MLX5VF_REGION_DATA_SIZE = 1 << 1, +}; + +enum { + MLX5VF_SUPPORTED_DEVICE_STATES = VFIO_DEVICE_STATE_RUNNING | + VFIO_DEVICE_STATE_SAVING | + VFIO_DEVICE_STATE_RESUMING, +}; + +#define MLX5VF_MIG_REGION_DATA_SIZE SZ_128K +/* Data section offset from migration region */ +#define MLX5VF_MIG_REGION_DATA_OFFSET \ + (sizeof(struct vfio_device_migration_info)) + +#define VFIO_DEVICE_MIGRATION_OFFSET(x) \ + (offsetof(struct vfio_device_migration_info, x)) + +struct mlx5vf_pci_migration_info { + u32 vfio_dev_state; /* VFIO_DEVICE_STATE_XXX */ + u32 dev_state; /* device migration state */ + u32 region_state; /* Use MLX5VF_REGION_XXX */ + u16 vhca_id; + struct mlx5_vhca_state_data vhca_state_data; +}; + +struct mlx5vf_pci_core_device { + struct vfio_pci_core_device core_device; + u8 migrate_cap:1; + /* protect migartion state */ + struct mutex state_mutex; + struct mlx5vf_pci_migration_info vmig; +}; + +static int mlx5vf_pci_unquiesce_device(struct mlx5vf_pci_core_device *mvdev) +{ + return mlx5vf_cmd_resume_vhca(mvdev->core_device.pdev, + mvdev->vmig.vhca_id, + MLX5_RESUME_VHCA_IN_OP_MOD_RESUME_MASTER); +} + +static int mlx5vf_pci_quiesce_device(struct mlx5vf_pci_core_device *mvdev) +{ + return mlx5vf_cmd_suspend_vhca( + mvdev->core_device.pdev, mvdev->vmig.vhca_id, + MLX5_SUSPEND_VHCA_IN_OP_MOD_SUSPEND_MASTER); +} + +static int mlx5vf_pci_unfreeze_device(struct mlx5vf_pci_core_device *mvdev) +{ + int ret; + + ret = mlx5vf_cmd_resume_vhca(mvdev->core_device.pdev, + mvdev->vmig.vhca_id, + MLX5_RESUME_VHCA_IN_OP_MOD_RESUME_SLAVE); + if (ret) + return ret; + + mvdev->vmig.dev_state &= ~MLX5VF_PCI_FREEZED; + return 0; +} + +static int mlx5vf_pci_freeze_device(struct mlx5vf_pci_core_device *mvdev) +{ + int ret; + + ret = mlx5vf_cmd_suspend_vhca( + mvdev->core_device.pdev, mvdev->vmig.vhca_id, + MLX5_SUSPEND_VHCA_IN_OP_MOD_SUSPEND_SLAVE); + if (ret) + return ret; + + mvdev->vmig.dev_state |= MLX5VF_PCI_FREEZED; + return 0; +} + +static int mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) +{ + u32 state_size = 0; + int ret; + + if (!(mvdev->vmig.dev_state & MLX5VF_PCI_FREEZED)) + return -EFAULT; + + /* If we already read state no reason to re-read */ + if (mvdev->vmig.vhca_state_data.state_size) + return 0; + + ret = mlx5vf_cmd_query_vhca_migration_state( + mvdev->core_device.pdev, mvdev->vmig.vhca_id, &state_size); + if (ret) + return ret; + + return mlx5vf_cmd_save_vhca_state(mvdev->core_device.pdev, + mvdev->vmig.vhca_id, state_size, + &mvdev->vmig.vhca_state_data); +} + +static int mlx5vf_pci_new_write_window(struct mlx5vf_pci_core_device *mvdev) +{ + struct mlx5_vhca_state_data *state_data = &mvdev->vmig.vhca_state_data; + u32 num_pages_needed; + u64 allocated_ready; + u32 bytes_needed; + + /* Check how many bytes are available from previous flows */ + WARN_ON(state_data->num_pages * PAGE_SIZE < + state_data->win_start_offset); + allocated_ready = (state_data->num_pages * PAGE_SIZE) - + state_data->win_start_offset; + WARN_ON(allocated_ready > MLX5VF_MIG_REGION_DATA_SIZE); + + bytes_needed = MLX5VF_MIG_REGION_DATA_SIZE - allocated_ready; + if (!bytes_needed) + return 0; + + num_pages_needed = DIV_ROUND_UP_ULL(bytes_needed, PAGE_SIZE); + return mlx5vf_add_migration_pages(state_data, num_pages_needed); +} + +static ssize_t +mlx5vf_pci_handle_migration_data_size(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, bool iswrite) +{ + struct mlx5vf_pci_migration_info *vmig = &mvdev->vmig; + u64 data_size; + int ret; + + if (iswrite) { + /* data_size is writable only during resuming state */ + if (vmig->vfio_dev_state != VFIO_DEVICE_STATE_RESUMING) + return -EINVAL; + + ret = copy_from_user(&data_size, buf, sizeof(data_size)); + if (ret) + return -EFAULT; + + vmig->vhca_state_data.state_size += data_size; + vmig->vhca_state_data.win_start_offset += data_size; + ret = mlx5vf_pci_new_write_window(mvdev); + if (ret) + return ret; + + } else { + if (vmig->vfio_dev_state != VFIO_DEVICE_STATE_SAVING) + return -EINVAL; + + data_size = min_t(u64, MLX5VF_MIG_REGION_DATA_SIZE, + vmig->vhca_state_data.state_size - + vmig->vhca_state_data.win_start_offset); + ret = copy_to_user(buf, &data_size, sizeof(data_size)); + if (ret) + return -EFAULT; + } + + vmig->region_state |= MLX5VF_REGION_DATA_SIZE; + return sizeof(data_size); +} + +static ssize_t +mlx5vf_pci_handle_migration_data_offset(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, bool iswrite) +{ + static const u64 data_offset = MLX5VF_MIG_REGION_DATA_OFFSET; + int ret; + + /* RO field */ + if (iswrite) + return -EFAULT; + + ret = copy_to_user(buf, &data_offset, sizeof(data_offset)); + if (ret) + return -EFAULT; + + return sizeof(data_offset); +} + +static ssize_t +mlx5vf_pci_handle_migration_pending_bytes(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, bool iswrite) +{ + struct mlx5vf_pci_migration_info *vmig = &mvdev->vmig; + u64 pending_bytes; + int ret; + + /* RO field */ + if (iswrite) + return -EFAULT; + + if (vmig->vfio_dev_state == (VFIO_DEVICE_STATE_SAVING | + VFIO_DEVICE_STATE_RUNNING)) { + /* In pre-copy state we have no data to return for now, + * return 0 pending bytes + */ + pending_bytes = 0; + } else { + if (!vmig->vhca_state_data.state_size) + return 0; + pending_bytes = vmig->vhca_state_data.state_size - + vmig->vhca_state_data.win_start_offset; + } + + ret = copy_to_user(buf, &pending_bytes, sizeof(pending_bytes)); + if (ret) + return -EFAULT; + + /* Window moves forward once data from previous iteration was read */ + if (vmig->region_state & MLX5VF_REGION_DATA_SIZE) + vmig->vhca_state_data.win_start_offset += + min_t(u64, MLX5VF_MIG_REGION_DATA_SIZE, pending_bytes); + + WARN_ON(vmig->vhca_state_data.win_start_offset > + vmig->vhca_state_data.state_size); + + /* New iteration started */ + vmig->region_state = MLX5VF_REGION_PENDING_BYTES; + return sizeof(pending_bytes); +} + +static int mlx5vf_load_state(struct mlx5vf_pci_core_device *mvdev) +{ + if (!mvdev->vmig.vhca_state_data.state_size) + return 0; + + return mlx5vf_cmd_load_vhca_state(mvdev->core_device.pdev, + mvdev->vmig.vhca_id, + &mvdev->vmig.vhca_state_data); +} + +static void mlx5vf_reset_mig_state(struct mlx5vf_pci_core_device *mvdev) +{ + struct mlx5vf_pci_migration_info *vmig = &mvdev->vmig; + + vmig->region_state = 0; + mlx5vf_reset_vhca_state(&vmig->vhca_state_data); +} + +static int mlx5vf_pci_set_device_state(struct mlx5vf_pci_core_device *mvdev, + u32 state) +{ + struct mlx5vf_pci_migration_info *vmig = &mvdev->vmig; + u32 old_state = vmig->vfio_dev_state; + int ret = 0; + + if (old_state == VFIO_DEVICE_STATE_ERROR || + !VFIO_DEVICE_STATE_VALID(state) || + (state & ~MLX5VF_SUPPORTED_DEVICE_STATES)) + return -EINVAL; + + /* Running switches off */ + if (((old_state ^ state) & VFIO_DEVICE_STATE_RUNNING) && + (old_state & VFIO_DEVICE_STATE_RUNNING)) { + ret = mlx5vf_pci_quiesce_device(mvdev); + if (ret) + return ret; + ret = mlx5vf_pci_freeze_device(mvdev); + if (ret) { + vmig->vfio_dev_state = VFIO_DEVICE_STATE_ERROR; + return ret; + } + } + + /* Resuming switches off */ + if (((old_state ^ state) & VFIO_DEVICE_STATE_RESUMING) && + (old_state & VFIO_DEVICE_STATE_RESUMING)) { + /* deserialize state into the device */ + ret = mlx5vf_load_state(mvdev); + if (ret) { + vmig->vfio_dev_state = VFIO_DEVICE_STATE_ERROR; + return ret; + } + } + + /* Resuming switches on */ + if (((old_state ^ state) & VFIO_DEVICE_STATE_RESUMING) && + (state & VFIO_DEVICE_STATE_RESUMING)) { + mlx5vf_reset_mig_state(mvdev); + ret = mlx5vf_pci_new_write_window(mvdev); + if (ret) + return ret; + } + + /* Saving switches on */ + if (((old_state ^ state) & VFIO_DEVICE_STATE_SAVING) && + (state & VFIO_DEVICE_STATE_SAVING)) { + if (!(state & VFIO_DEVICE_STATE_RUNNING)) { + /* serialize post copy */ + ret = mlx5vf_pci_save_device_data(mvdev); + if (ret) + return ret; + } + } + + /* Running switches on */ + if (((old_state ^ state) & VFIO_DEVICE_STATE_RUNNING) && + (state & VFIO_DEVICE_STATE_RUNNING)) { + ret = mlx5vf_pci_unfreeze_device(mvdev); + if (ret) + return ret; + ret = mlx5vf_pci_unquiesce_device(mvdev); + if (ret) { + vmig->vfio_dev_state = VFIO_DEVICE_STATE_ERROR; + return ret; + } + } + + vmig->vfio_dev_state = state; + return 0; +} + +static ssize_t +mlx5vf_pci_handle_migration_device_state(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, bool iswrite) +{ + size_t count = sizeof(mvdev->vmig.vfio_dev_state); + int ret; + + if (iswrite) { + u32 device_state; + + ret = copy_from_user(&device_state, buf, count); + if (ret) + return -EFAULT; + + ret = mlx5vf_pci_set_device_state(mvdev, device_state); + if (ret) + return ret; + } else { + ret = copy_to_user(buf, &mvdev->vmig.vfio_dev_state, count); + if (ret) + return -EFAULT; + } + + return count; +} + +static ssize_t +mlx5vf_pci_copy_user_data_to_device_state(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, size_t count, + u64 offset) +{ + struct mlx5_vhca_state_data *state_data = &mvdev->vmig.vhca_state_data; + char __user *from_buff = buf; + u32 curr_offset; + u32 win_page_offset; + u32 copy_count; + struct page *page; + char *to_buff; + int ret; + + curr_offset = state_data->win_start_offset + offset; + + do { + page = mlx5vf_get_migration_page(&state_data->mig_data, + curr_offset); + if (!page) + return -EINVAL; + + win_page_offset = curr_offset % PAGE_SIZE; + copy_count = min_t(u32, PAGE_SIZE - win_page_offset, count); + + to_buff = kmap_local_page(page); + ret = copy_from_user(to_buff + win_page_offset, from_buff, + copy_count); + kunmap_local(to_buff); + if (ret) + return -EFAULT; + + from_buff += copy_count; + curr_offset += copy_count; + count -= copy_count; + } while (count > 0); + + return 0; +} + +static ssize_t +mlx5vf_pci_copy_device_state_to_user(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, u64 offset, size_t count) +{ + struct mlx5_vhca_state_data *state_data = &mvdev->vmig.vhca_state_data; + char __user *to_buff = buf; + u32 win_available_bytes; + u32 win_page_offset; + u32 copy_count; + u32 curr_offset; + char *from_buff; + struct page *page; + int ret; + + win_available_bytes = + min_t(u64, MLX5VF_MIG_REGION_DATA_SIZE, + mvdev->vmig.vhca_state_data.state_size - + mvdev->vmig.vhca_state_data.win_start_offset); + + if (count + offset > win_available_bytes) + return -EINVAL; + + curr_offset = state_data->win_start_offset + offset; + + do { + page = mlx5vf_get_migration_page(&state_data->mig_data, + curr_offset); + if (!page) + return -EINVAL; + + win_page_offset = curr_offset % PAGE_SIZE; + copy_count = min_t(u32, PAGE_SIZE - win_page_offset, count); + + from_buff = kmap_local_page(page); + ret = copy_to_user(buf, from_buff + win_page_offset, + copy_count); + kunmap_local(from_buff); + if (ret) + return -EFAULT; + + curr_offset += copy_count; + count -= copy_count; + to_buff += copy_count; + } while (count); + + return 0; +} + +static ssize_t +mlx5vf_pci_migration_data_rw(struct mlx5vf_pci_core_device *mvdev, + char __user *buf, size_t count, u64 offset, + bool iswrite) +{ + int ret; + + if (offset + count > MLX5VF_MIG_REGION_DATA_SIZE) + return -EINVAL; + + if (iswrite) + ret = mlx5vf_pci_copy_user_data_to_device_state(mvdev, buf, + count, offset); + else + ret = mlx5vf_pci_copy_device_state_to_user(mvdev, buf, offset, + count); + if (ret) + return ret; + return count; +} + +static ssize_t mlx5vf_pci_mig_rw(struct vfio_pci_core_device *vdev, + char __user *buf, size_t count, loff_t *ppos, + bool iswrite) +{ + struct mlx5vf_pci_core_device *mvdev = + container_of(vdev, struct mlx5vf_pci_core_device, core_device); + u64 pos = *ppos & VFIO_PCI_OFFSET_MASK; + int ret; + + mutex_lock(&mvdev->state_mutex); + /* Copy to/from the migration region data section */ + if (pos >= MLX5VF_MIG_REGION_DATA_OFFSET) { + ret = mlx5vf_pci_migration_data_rw( + mvdev, buf, count, pos - MLX5VF_MIG_REGION_DATA_OFFSET, + iswrite); + goto end; + } + + switch (pos) { + case VFIO_DEVICE_MIGRATION_OFFSET(device_state): + /* This is RW field. */ + if (count != sizeof(mvdev->vmig.vfio_dev_state)) { + ret = -EINVAL; + break; + } + ret = mlx5vf_pci_handle_migration_device_state(mvdev, buf, + iswrite); + break; + case VFIO_DEVICE_MIGRATION_OFFSET(pending_bytes): + /* + * The number of pending bytes still to be migrated from the + * vendor driver. This is RO field. + * Reading this field indicates on the start of a new iteration + * to get device data. + * + */ + ret = mlx5vf_pci_handle_migration_pending_bytes(mvdev, buf, + iswrite); + break; + case VFIO_DEVICE_MIGRATION_OFFSET(data_offset): + /* + * The user application should read data_offset field from the + * migration region. The user application should read the + * device data from this offset within the migration region + * during the _SAVING mode or write the device data during the + * _RESUMING mode. This is RO field. + */ + ret = mlx5vf_pci_handle_migration_data_offset(mvdev, buf, + iswrite); + break; + case VFIO_DEVICE_MIGRATION_OFFSET(data_size): + /* + * The user application should read data_size to get the size + * in bytes of the data copied to the migration region during + * the _SAVING state by the device. The user application should + * write the size in bytes of the data that was copied to + * the migration region during the _RESUMING state by the user. + * This is RW field. + */ + ret = mlx5vf_pci_handle_migration_data_size(mvdev, buf, + iswrite); + break; + default: + ret = -EFAULT; + break; + } + +end: + mutex_unlock(&mvdev->state_mutex); + return ret; +} + +static struct vfio_pci_regops migration_ops = { + .rw = mlx5vf_pci_mig_rw, +}; + +static int mlx5vf_pci_open_device(struct vfio_device *core_vdev) +{ + struct mlx5vf_pci_core_device *mvdev = container_of( + core_vdev, struct mlx5vf_pci_core_device, core_device.vdev); + struct vfio_pci_core_device *vdev = &mvdev->core_device; + int vf_id; + int ret; + + ret = vfio_pci_core_enable(vdev); + if (ret) + return ret; + + if (!mvdev->migrate_cap) { + vfio_pci_core_finish_enable(vdev); + return 0; + } + + vf_id = pci_iov_vf_id(vdev->pdev); + if (vf_id < 0) { + ret = vf_id; + goto out_disable; + } + + ret = mlx5vf_cmd_get_vhca_id(vdev->pdev, vf_id + 1, + &mvdev->vmig.vhca_id); + if (ret) + goto out_disable; + + ret = vfio_pci_register_dev_region(vdev, VFIO_REGION_TYPE_MIGRATION, + VFIO_REGION_SUBTYPE_MIGRATION, + &migration_ops, + MLX5VF_MIG_REGION_DATA_OFFSET + + MLX5VF_MIG_REGION_DATA_SIZE, + VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE, + NULL); + if (ret) + goto out_disable; + + mvdev->vmig.vfio_dev_state = VFIO_DEVICE_STATE_RUNNING; + vfio_pci_core_finish_enable(vdev); + return 0; +out_disable: + vfio_pci_core_disable(vdev); + return ret; +} + +static void mlx5vf_pci_close_device(struct vfio_device *core_vdev) +{ + struct mlx5vf_pci_core_device *mvdev = container_of( + core_vdev, struct mlx5vf_pci_core_device, core_device.vdev); + + vfio_pci_core_close_device(core_vdev); + mlx5vf_reset_mig_state(mvdev); +} + +static const struct vfio_device_ops mlx5vf_pci_ops = { + .name = "mlx5-vfio-pci", + .open_device = mlx5vf_pci_open_device, + .close_device = mlx5vf_pci_close_device, + .ioctl = vfio_pci_core_ioctl, + .read = vfio_pci_core_read, + .write = vfio_pci_core_write, + .mmap = vfio_pci_core_mmap, + .request = vfio_pci_core_request, + .match = vfio_pci_core_match, +}; + +static int mlx5vf_pci_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + struct mlx5vf_pci_core_device *mvdev; + int ret; + + mvdev = kzalloc(sizeof(*mvdev), GFP_KERNEL); + if (!mvdev) + return -ENOMEM; + vfio_pci_core_init_device(&mvdev->core_device, pdev, &mlx5vf_pci_ops); + + if (pdev->is_virtfn) { + struct mlx5_core_dev *mdev = + mlx5_vf_get_core_dev(pdev); + + if (mdev) { + if (MLX5_CAP_GEN(mdev, migration)) { + mvdev->migrate_cap = 1; + mutex_init(&mvdev->state_mutex); + } + mlx5_vf_put_core_dev(mdev); + } + } + + ret = vfio_pci_core_register_device(&mvdev->core_device); + if (ret) + goto out_free; + + dev_set_drvdata(&pdev->dev, mvdev); + return 0; + +out_free: + vfio_pci_core_uninit_device(&mvdev->core_device); + kfree(mvdev); + return ret; +} + +static void mlx5vf_pci_remove(struct pci_dev *pdev) +{ + struct mlx5vf_pci_core_device *mvdev = dev_get_drvdata(&pdev->dev); + + vfio_pci_core_unregister_device(&mvdev->core_device); + vfio_pci_core_uninit_device(&mvdev->core_device); + kfree(mvdev); +} + +static const struct pci_device_id mlx5vf_pci_table[] = { + { PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_MELLANOX, 0x101e) }, /* ConnectX Family mlx5Gen Virtual Function */ + {} +}; + +MODULE_DEVICE_TABLE(pci, mlx5vf_pci_table); + +static struct pci_driver mlx5vf_pci_driver = { + .name = KBUILD_MODNAME, + .id_table = mlx5vf_pci_table, + .probe = mlx5vf_pci_probe, + .remove = mlx5vf_pci_remove, + .err_handler = &vfio_pci_core_err_handlers, +}; + +static void __exit mlx5vf_pci_cleanup(void) +{ + pci_unregister_driver(&mlx5vf_pci_driver); +} + +static int __init mlx5vf_pci_init(void) +{ + return pci_register_driver(&mlx5vf_pci_driver); +} + +module_init(mlx5vf_pci_init); +module_exit(mlx5vf_pci_cleanup); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Max Gurtovoy "); +MODULE_AUTHOR("Yishai Hadas "); +MODULE_DESCRIPTION( + "MLX5 VFIO PCI - User Level meta-driver for MLX5 device family"); From patchwork Tue Oct 19 10:58:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543215 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=QYyKl1sB; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW4N0Qn1z9sP7 for ; Tue, 19 Oct 2021 22:00:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235399AbhJSLCq (ORCPT ); Tue, 19 Oct 2021 07:02:46 -0400 Received: from mail-dm6nam12on2086.outbound.protection.outlook.com ([40.107.243.86]:27649 "EHLO NAM12-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235408AbhJSLCM (ORCPT ); Tue, 19 Oct 2021 07:02:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fEgAldvft6tD2q1s7/YyNh8i/XogO5qP59WYVmtsO7sMaTQXdF93/dniRs5RNN8gI9e1nd3cgE/fEe306yYR6jpxjlswDD+pqXbO26Z0cRvcXyDDVln/Qij1T2kylV5GWyu0wAoFCt6lg7RfPk0qI054c0EOmTAV1yIHl0vQ03BRaoH0fM4KDeihUqnohbrhRCQ1lCpLIRLCEqvWrM5wfEV/tjrQ2n80BG55a7yTkoinKwSQZqAfiHEe9r+zHuo1psUxDnZ5DBwmB1wDnamrxosqCqkbkD4sDr34T42Ae0OKqjWraqt9M9qRiVzq6MnVV9L2igD8TsVWcOsMSwEWgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eLKFU+sGvBKXZ6cjTNa+FhQ1WxILKlXRvDl9qJoQPqQ=; b=Vn95dwrSR1/dfAIomg4zLotckRJkPCRmnPHdQL/acXFSd2jphUEKbStD2p6BxR5iE2SaA5+N0jWcs28+pNIp9ZgEmoZvxtgc6bdUw5E7Jj7hjAh/RhaQuJo0DKiOtLDgq0NDZQnjOj+c2jdKCfwrG3oU1O8YegvAdUjtD/1nwk2udmsl48cNuaY3rB1EqEUha4eFy9IGNlzzp6/mWn1LR5Io9S1gBFl71E94g2j5Udj5ZohVqPhTGCSr7JBli+maa5jqmhi5pbyxjYDte9QIQxYzr4XCGAdzsY+BZSLv0z0Wa7zuG+dLqkinTDpPkqZJLhi7rP7kX1PzVHbj8ANPRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eLKFU+sGvBKXZ6cjTNa+FhQ1WxILKlXRvDl9qJoQPqQ=; b=QYyKl1sBLpYPjMpxn+vPOdllvU1kg65/uRjNelOMtEkmN5nVzhaPttxEVENtWHV8aAkXaxBxpVJgnZa3CemVC4SS3FXYjW9VsDBIK8ASnDsJpXa0O8wqIQjCuw2FnpncKNGngdhWZgO9fXpxxMTq3ckONWLG0+GQr2SE1VLmDJfz8a3FRD+XfnNlLyUXmsl8xtQGruJwEQtL4a5dY4X354GH6xrtL0Xg0x6fDkMM2NuAk02aE7GaVwQDaXMaKKixX1zvBcy3WVUSrcr17RCucK0cJsAiUOMQ5eg4yUB7TdaJkz+uR+eNW66CdHZD44vRt4uNZRD8lhlOoPQO3EU5Kg== Received: from BN8PR04CA0002.namprd04.prod.outlook.com (2603:10b6:408:70::15) by DM4PR12MB5357.namprd12.prod.outlook.com (2603:10b6:5:39b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16; Tue, 19 Oct 2021 10:59:58 +0000 Received: from BN8NAM11FT051.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::14) by BN8PR04CA0002.outlook.office365.com (2603:10b6:408:70::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.16 via Frontend Transport; Tue, 19 Oct 2021 10:59:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by BN8NAM11FT051.mail.protection.outlook.com (10.13.177.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:57 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:53 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:50 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 13/14] vfio/pci: Expose vfio_pci_aer_err_detected() Date: Tue, 19 Oct 2021 13:58:37 +0300 Message-ID: <20211019105838.227569-14-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 32fd51bf-0b16-4154-1861-08d992ef972b X-MS-TrafficTypeDiagnostic: DM4PR12MB5357: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3968; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GEysapSabDjvQfgyesbLaYShRmozP/101WK3nAJ52FwnaV7CDPTZHxMQJ4mIsl37BDrtUjSXD7X/+cltel4YZqIxqMpeU0yI0puA9KHsjuI5XiLtI8bRxDN7GW2ZBqv/WBOCsQma9wre9X7xVqxZ5YrGYxNjzE5Hk/GErhho79Z1ZUA9hXiDoGAuMaMJxX0e58CPkYIBvU/rtpWS6JBF9bJ9UP5xHUrKi+HhKav3d2WiiINVIKSaLic5/SXlkzmxPaV0+B608U00dtj4F4Z57ZtWneDbZvumuTQN0anLV2/uvfE3WeNtWNRiD6xBqCToTuZHXoRpN9NGed3KH5tRk+2DAQez/phxB4IRLsR1rRaZdNSkZNG5uBEYaDsmo7ACBJNQHEAvIwpmEnCVEnCzLSh3wuDp0fxxI8mqDf6g7tvDTtz9OqQSpTzkBF6XPb0zJvXLmh6QinVkXDspnSwzQTfmTEA6rVkw28dVvU49LuewGT26POXJ3/tLZtrHYHtTJiPaGwDxAtndOG4ywLNvUcGpEBlOeAg172fRpI+ZhZOpnOqBJ2jaHEXxgHqIPFtemzNfsd9/vvO0qVPMFilcW2Df1kKjaCalkfZ6eAOxZFMH4bmiOpCzfN8o2D6IIstKmjzDANpVYmguzVXrv0P6a7ECiRrVm5KpWG1JgR5VsFn4QQt/Ls/XvmALL4GO+jrsLvI6h9+F+IrnPxLPHEKzKA== X-Forefront-Antispam-Report: CIP:216.228.112.36;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid05.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(336012)(316002)(2616005)(7696005)(47076005)(26005)(8676002)(36756003)(110136005)(426003)(508600001)(8936002)(7636003)(356005)(6636002)(83380400001)(82310400003)(70206006)(107886003)(186003)(86362001)(6666004)(70586007)(54906003)(2906002)(4326008)(1076003)(36860700001)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:57.5821 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32fd51bf-0b16-4154-1861-08d992ef972b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.36];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT051.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5357 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Expose vfio_pci_aer_err_detected() to be used by drivers as part of their pci_error_handlers structure. Next patch for mlx5 driver will use it. Signed-off-by: Yishai Hadas --- drivers/vfio/pci/vfio_pci_core.c | 5 +++-- include/linux/vfio_pci_core.h | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index e581a327f90d..0f4a50de913f 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1901,8 +1901,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) } EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_device); -static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, - pci_channel_state_t state) +pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state) { struct vfio_pci_core_device *vdev; struct vfio_device *device; @@ -1924,6 +1924,7 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, return PCI_ERS_RESULT_CAN_RECOVER; } +EXPORT_SYMBOL_GPL(vfio_pci_aer_err_detected); int vfio_pci_core_sriov_configure(struct pci_dev *pdev, int nr_virtfn) { diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index ef9a44b6cf5d..768336b02fd6 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -230,6 +230,8 @@ int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf); int vfio_pci_core_enable(struct vfio_pci_core_device *vdev); void vfio_pci_core_disable(struct vfio_pci_core_device *vdev); void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev); +pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, + pci_channel_state_t state); static inline bool vfio_pci_is_vga(struct pci_dev *pdev) { From patchwork Tue Oct 19 10:58:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 1543216 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=LVg/Bk9K; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by bilbo.ozlabs.org (Postfix) with ESMTP id 4HYW4N2n8pz9sPB for ; Tue, 19 Oct 2021 22:00:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235415AbhJSLCr (ORCPT ); Tue, 19 Oct 2021 07:02:47 -0400 Received: from mail-dm6nam10on2043.outbound.protection.outlook.com ([40.107.93.43]:14849 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235388AbhJSLCM (ORCPT ); Tue, 19 Oct 2021 07:02:12 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NaX5rY8ooHy+6Q4byJxWy9tqh40f6vqrwHh3MiazYFfnpymAZv13a0059zyG6FFzAj2pXsBhPfwiyUcyCv32HT753OTHGHLFL4htsC9L3k7sqbVuavEwq1VpNQfEY3jjydCcEW2bTkisjZKcW/nGRxn0YB8lbHk0YI6GX8o5b29AIvfL6BBKsIWbcnt028LRMgJ6E+TWWHPZ10pek85MBw2mqD4uDjikgyPqew8ictNQncYiyreo9JXfll0ASs+oIow/4WNVeO2gbhSDmKsjlfYr6YpG74VJMCnxLJ6/et/5lkWb6OuOSeMF0YTlty1z20pCrGRazrEssAEhLmLJqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gvbYG69wMjK0H/k9CGU5MIxs8tnAxFc7USLq1c6B0+4=; b=HKioM8qPONcBOpjtSep84EeHU7/VM1ecY5/nc9XwVVNYuvUtsc5kR4hx+Zfa+pP2nBte4LhnpPxJCVY9KvFhJF8DE8SbFNoDkOh6en9SQNN3KhszLdLr/AV2baCEVz/LBIml0nYOistTR6xQ2Ud8DLO+1qJjJ8pImMJOGmtrZeqZg6ksbLNy5DyNYK+Hm4/vUSrQ8b31ymottjAXanN0loQbdViuZQRO4mdTA39vI9JZk7e7mARSK74EcmuSYqb42wlrac/SXoF3AKE+RxdTOinf8GV4T3MQ+geTADrzwFw2B9s+ZmDkrMlMLiZqBJr0DSHTCDO/iJgdbH/jvSKpew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gvbYG69wMjK0H/k9CGU5MIxs8tnAxFc7USLq1c6B0+4=; b=LVg/Bk9Kg9chMGXjs1NJP1CLIL59fwBqQGNQWDHuQzjCvq0+kdzf3h2juKl56ihgiBc95gDP3LXiLbMIKBrCvalWBVk+yPQNiSN37zRoboajr8++0WJZGLxmJkWnzk6YmYPe3vWXOYANuKN8uZpFgP4LF4EZIUQQcB2/kEWy0yIW2iTtGoMFENQs/HnsdG6VMnhaBjH0I98jSK6spYSf7VIMBi6mk/GWX7otTcEJVMJiVkJsqJ6yFY505CO0dFQ7AJ9ZPJAZWD8II1MRW4Il+/tXsDbs+Emo3Gwfk3dD184nP81O7sXbK9tGIVryjCwEtAj5sZzWfcFsOPkg2DXe8Q== Received: from DM6PR02CA0062.namprd02.prod.outlook.com (2603:10b6:5:177::39) by DM6PR12MB2858.namprd12.prod.outlook.com (2603:10b6:5:182::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Tue, 19 Oct 2021 10:59:59 +0000 Received: from DM6NAM11FT027.eop-nam11.prod.protection.outlook.com (2603:10b6:5:177:cafe::a6) by DM6PR02CA0062.outlook.office365.com (2603:10b6:5:177::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT027.mail.protection.outlook.com (10.13.172.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Tue, 19 Oct 2021 10:59:58 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 03:59:57 -0700 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 19 Oct 2021 10:59:57 +0000 Received: from vdi.nvidia.com (172.20.187.5) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Tue, 19 Oct 2021 10:59:54 +0000 From: Yishai Hadas To: , , , CC: , , , , , , , , Subject: [PATCH V2 mlx5-next 14/14] vfio/mlx5: Use its own PCI reset_done error handler Date: Tue, 19 Oct 2021 13:58:38 +0300 Message-ID: <20211019105838.227569-15-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20211019105838.227569-1-yishaih@nvidia.com> References: <20211019105838.227569-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 34352b90-7e02-4d95-f4bc-08d992ef977c X-MS-TrafficTypeDiagnostic: DM6PR12MB2858: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1443; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WZoqKUn72M/DsTec5opTj7Z1dwUOWRIrIyiAUFsGIzYwko/ziC3AtbUYyKi23y4hsx65MhFfVskEXHG/F0q6I/BFkcNoC90NtkuquMSZZA3Sls5+U8LjPEVa2YH27CnWFwxRfW4Fuk86LduU61qfDJQJupNxDvIWJcwis9w+W9g4oobLz2fC0m2f+ZX2VfigsaJGA1UmohaerpxpFoJe10yFJwqtdbquIZ9oEBli6G6igXukbgc6D3sAVKhSUUcMkv4yD6d+D6MwTHI4jhvPcr0kjF2z9bMPngam38ooO4K7W+5tDEIpPwQd9uT+cJ0XyaymozUHvIVT1alHYpcADH8WVnptZSxi0mctiXAFOiqCPALv4WrGLs/wfhD43qq+Jh1q5142G3kG2/MTnP7tyPns2aC2VgN4hr3N3fPzYEdta2Sw1ZhzkXejCROPQE7q9GtlXP+cHSCtPwkXK+XS3UqigRv/MuYrjlm7XiwTPezHTWA6WlvlJqcffLO2sEf5hUt67//fAk+bMlesrIaDScFPZRYc9Kl8wYQjnnC9shAT4oH3YkwCScoSdcr3vgndP60rb0t91zidcMWMJISH1/ujJZSsaUkLBZuvWO3XjDw2fZAdPO7Ayyyaafve9UfECm0nfdi88mpGo6uphP5XINalJfFIkAw7ZUlu8L9XUXphdBEZsG3REwA5wDacuNjXv66+iuzsVCs3oSGaQV1y7g== X-Forefront-Antispam-Report: CIP:216.228.112.32;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid01.nvidia.com;CAT:NONE;SFS:(4636009)(46966006)(36840700001)(426003)(6666004)(82310400003)(70586007)(2906002)(47076005)(70206006)(8936002)(86362001)(6636002)(4326008)(5660300002)(336012)(316002)(36756003)(83380400001)(110136005)(8676002)(26005)(7696005)(107886003)(356005)(186003)(1076003)(7636003)(36860700001)(54906003)(508600001)(2616005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Oct 2021 10:59:58.4854 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 34352b90-7e02-4d95-f4bc-08d992ef977c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT027.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2858 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Register its own handler for pci_error_handlers.reset_done and update state accordingly. Signed-off-by: Yishai Hadas Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/main.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index 621b7fc60544..b759c6e153b6 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -58,6 +58,7 @@ struct mlx5vf_pci_core_device { /* protect migartion state */ struct mutex state_mutex; struct mlx5vf_pci_migration_info vmig; + struct work_struct work; }; static int mlx5vf_pci_unquiesce_device(struct mlx5vf_pci_core_device *mvdev) @@ -615,6 +616,27 @@ static const struct vfio_device_ops mlx5vf_pci_ops = { .match = vfio_pci_core_match, }; +static void mlx5vf_reset_work_handler(struct work_struct *work) +{ + struct mlx5vf_pci_core_device *mvdev = + container_of(work, struct mlx5vf_pci_core_device, work); + + mutex_lock(&mvdev->state_mutex); + mlx5vf_reset_mig_state(mvdev); + mvdev->vmig.vfio_dev_state = VFIO_DEVICE_STATE_RUNNING; + mutex_unlock(&mvdev->state_mutex); +} + +static void mlx5vf_pci_aer_reset_done(struct pci_dev *pdev) +{ + struct mlx5vf_pci_core_device *mvdev = dev_get_drvdata(&pdev->dev); + + if (!mvdev->migrate_cap) + return; + + schedule_work(&mvdev->work); +} + static int mlx5vf_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { @@ -634,6 +656,8 @@ static int mlx5vf_pci_probe(struct pci_dev *pdev, if (MLX5_CAP_GEN(mdev, migration)) { mvdev->migrate_cap = 1; mutex_init(&mvdev->state_mutex); + INIT_WORK(&mvdev->work, + mlx5vf_reset_work_handler); } mlx5_vf_put_core_dev(mdev); } @@ -656,6 +680,8 @@ static void mlx5vf_pci_remove(struct pci_dev *pdev) { struct mlx5vf_pci_core_device *mvdev = dev_get_drvdata(&pdev->dev); + if (mvdev->migrate_cap) + cancel_work_sync(&mvdev->work); vfio_pci_core_unregister_device(&mvdev->core_device); vfio_pci_core_uninit_device(&mvdev->core_device); kfree(mvdev); @@ -668,12 +694,17 @@ static const struct pci_device_id mlx5vf_pci_table[] = { MODULE_DEVICE_TABLE(pci, mlx5vf_pci_table); +const struct pci_error_handlers mlx5vf_err_handlers = { + .reset_done = mlx5vf_pci_aer_reset_done, + .error_detected = vfio_pci_aer_err_detected, +}; + static struct pci_driver mlx5vf_pci_driver = { .name = KBUILD_MODNAME, .id_table = mlx5vf_pci_table, .probe = mlx5vf_pci_probe, .remove = mlx5vf_pci_remove, - .err_handler = &vfio_pci_core_err_handlers, + .err_handler = &mlx5vf_err_handlers, }; static void __exit mlx5vf_pci_cleanup(void)