{"id":2224155,"url":"http://patchwork.ozlabs.org/api/patches/2224155/?format=json","web_url":"http://patchwork.ozlabs.org/project/linux-pci/patch/5e2c2ddd25f80a165fc96db88f765b418a6ed038.1776381841.git.nicolinc@nvidia.com/","project":{"id":28,"url":"http://patchwork.ozlabs.org/api/projects/28/?format=json","name":"Linux PCI development","link_name":"linux-pci","list_id":"linux-pci.vger.kernel.org","list_email":"linux-pci@vger.kernel.org","web_url":null,"scm_url":null,"webscm_url":null,"list_archive_url":"","list_archive_url_format":"","commit_url_format":""},"msgid":"<5e2c2ddd25f80a165fc96db88f765b418a6ed038.1776381841.git.nicolinc@nvidia.com>","list_archive_url":null,"date":"2026-04-16T23:28:40","name":"[v3,11/11] iommu/arm-smmu-v3: Block ATS upon an ATC invalidation timeout","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"43c6e97beafd3d9392bfffcba853d7d781239c5d","submitter":{"id":82183,"url":"http://patchwork.ozlabs.org/api/people/82183/?format=json","name":"Nicolin Chen","email":"nicolinc@nvidia.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/linux-pci/patch/5e2c2ddd25f80a165fc96db88f765b418a6ed038.1776381841.git.nicolinc@nvidia.com/mbox/","series":[{"id":500217,"url":"http://patchwork.ozlabs.org/api/series/500217/?format=json","web_url":"http://patchwork.ozlabs.org/project/linux-pci/list/?series=500217","date":"2026-04-16T23:28:31","name":"iommu/arm-smmu-v3: Quarantine device upon ATC invalidation timeout","version":3,"mbox":"http://patchwork.ozlabs.org/series/500217/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/2224155/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/2224155/checks/","tags":{},"related":[],"headers":{"Return-Path":"\n ","X-Original-To":["incoming@patchwork.ozlabs.org","linux-pci@vger.kernel.org"],"Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256\n header.s=selector2 header.b=HAVKbn8i;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org\n (client-ip=172.234.253.10; helo=sea.lore.kernel.org;\n envelope-from=linux-pci+bounces-52671-incoming=patchwork.ozlabs.org@vger.kernel.org;\n receiver=patchwork.ozlabs.org)","smtp.subspace.kernel.org;\n\tdkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com\n header.b=\"HAVKbn8i\"","smtp.subspace.kernel.org;\n arc=fail smtp.client-ip=40.93.198.4","smtp.subspace.kernel.org;\n dmarc=pass (p=reject dis=none) header.from=nvidia.com","smtp.subspace.kernel.org;\n spf=fail smtp.mailfrom=nvidia.com"],"Received":["from sea.lore.kernel.org (sea.lore.kernel.org [172.234.253.10])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4fxZFv6TPHz1yDF\n\tfor ; Fri, 17 Apr 2026 09:38:27 +1000 (AEST)","from smtp.subspace.kernel.org (conduit.subspace.kernel.org\n [100.90.174.1])\n\tby sea.lore.kernel.org (Postfix) with ESMTP id BF4E5306B080\n\tfor ; Thu, 16 Apr 2026 23:30:24 +0000 (UTC)","from localhost.localdomain (localhost.localdomain [127.0.0.1])\n\tby smtp.subspace.kernel.org (Postfix) with ESMTP id D5CCA3A3E6C;\n\tThu, 16 Apr 2026 23:30:22 +0000 (UTC)","from CY7PR03CU001.outbound.protection.outlook.com\n (mail-westcentralusazon11010004.outbound.protection.outlook.com\n [40.93.198.4])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby smtp.subspace.kernel.org (Postfix) with ESMTPS id 688CC3A8720;\n\tThu, 16 Apr 2026 23:30:21 +0000 (UTC)","from BY5PR17CA0046.namprd17.prod.outlook.com (2603:10b6:a03:167::23)\n by CH0PR12MB8464.namprd12.prod.outlook.com (2603:10b6:610:184::11) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9818.20; Thu, 16 Apr\n 2026 23:30:14 +0000","from CO1PEPF00012E83.namprd03.prod.outlook.com\n (2603:10b6:a03:167:cafe::3c) by BY5PR17CA0046.outlook.office365.com\n (2603:10b6:a03:167::23) with Microsoft SMTP Server (version=TLS1_3,\n cipher=TLS_AES_256_GCM_SHA384) id 15.20.9769.52 via Frontend Transport; Thu,\n 16 Apr 2026 23:30:13 +0000","from mail.nvidia.com (216.228.117.161) by\n CO1PEPF00012E83.mail.protection.outlook.com (10.167.249.58) with Microsoft\n SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n 15.20.9769.17 via Frontend Transport; Thu, 16 Apr 2026 23:30:13 +0000","from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com\n (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 16 Apr\n 2026 16:29:56 -0700","from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail204.nvidia.com\n (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Thu, 16 Apr\n 2026 16:29:55 -0700","from Asurada-Nvidia.nvidia.com (10.127.8.11) by mail.nvidia.com\n (10.129.68.9) with Microsoft SMTP Server id 15.2.2562.20 via Frontend\n Transport; Thu, 16 Apr 2026 16:29:49 -0700"],"ARC-Seal":["i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;\n\tt=1776382222; cv=fail;\n b=FJDXZrvNRIXMQcm4z4zoiV2Ufk7cB3F+ew1iZA1Ekcy13ksoJm7wjtg7wEpXMRWRtAhaj81AJYakH8KLZiaKUTj93RMn4b9h3A/mU04jsRbGehAgKwKNi6Ogx78UT8j2uiv6NyCRWdi0cLLonkI8Q5xv9581dqzrP5rnIt+nhYY=","i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=SMQ7q6jDf2i8ztsQtM76nWKDCjWRt4wz0nSfyhLb9gGTNtMuE84z+wRZwDqU3SliZ3xF80GULRsPC7IKO7rjMt+eywEIORdPKKBO7v4jf38F5mFDREKQOAIvv4Og1BNJIP+TAm8A2tbozz9rU0YLFZokX+dHJVUMCz426n0eR2x6QzA6kDawft0rWuoEONw18gN8MiAiaMieHtjZDVVLJIDaFmwsjNe8t7CmxvHIX96IMvOS9kMiih8CsIXQ2CfG3OOa4pb66F6PAYEcs6vjDFpqqVUcLQYxLGpAjLRKuz0eYe0pkPAzTuGaJBg4vVblD9zHWr1EZypebtF4n7v/lQ=="],"ARC-Message-Signature":["i=2; a=rsa-sha256; d=subspace.kernel.org;\n\ts=arc-20240116; t=1776382222; c=relaxed/simple;\n\tbh=BZMBBeUiRqQJaOWG7bMBv2C5/f73ImUb0yKOTFRm/z4=;\n\th=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References:\n\t MIME-Version:Content-Type;\n b=uYiu4qoqXxzVAlCjIpMg7UGz2YuPSlIDWVZEKmcPDjRAncrlTTEfL5ysG5KrqeTGj3EVO77godH8C6HJ6yYTO5149LrxSLSvkBrWIl05pEshjPiGszoBDEyLtDnKE7xg2PUcZ6+YDsmJ/8BlVCIuXSYzNn7gLySs5IbFBBXROBQ=","i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=8v2VF+E/uGiwaF8t4eiA68Qrp7GgNBrucQYc6qOzZaw=;\n b=uWiiWMhCGP+V6CnRonLeIgdfwb1/8mXZBkrzk+QDvjfv/5d4SwMLw8K8utRWodinX9uSOsNGhUoTJiebfewLx8vpYyG3Lw5r8cpUOg8CJwQRV8DaEvilbkv5DlT3ABdShW5kqy+O4OVXOwIxir8iC3jxMjPp+mUIfRVD/KS2suyL/dICS2ba2sS0VShEslt/AZPMcT2Rq5EU+rYGSKNysLh+lVRv884F1VeLQY2tDVHfo7NEdY3VG0OyxnKTe5h/S36u2yIozO5s6w3NMpD58wym7f8IhATYI31QT0s+iF2Ht6aJN9igwEm/W85Ufo3TBNUTxckZ+cDzFOIaHKzrtQ=="],"ARC-Authentication-Results":["i=2; smtp.subspace.kernel.org;\n dmarc=pass (p=reject dis=none) header.from=nvidia.com;\n spf=fail smtp.mailfrom=nvidia.com;\n dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com\n header.b=HAVKbn8i; arc=fail smtp.client-ip=40.93.198.4","i=1; mx.microsoft.com 1; spf=pass (sender ip is\n 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com;\n dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com;\n dkim=none (message not signed); arc=none (0)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;\n s=selector2;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=8v2VF+E/uGiwaF8t4eiA68Qrp7GgNBrucQYc6qOzZaw=;\n b=HAVKbn8ij81Ke2eql+QdAACZ2tztY6IlGeuzxW8msShT6AGq57NQMaffozrVXB3x7bfc6QwN1GQNl3ijlG/yX+/MgVtv0gPHtk3A3/zxmWNHUhl8uGaeaYupFjgmpsWy6kiwLElI4uapwENOp785Ra1hWpZUMR32CrLKzlkVwm+7UigKG3bmeUG7e5PUvjGyW5zMzqxNA0BVytRA0gqn2IRw67v1bDOg/YHpLjnIDv1kjcpz4M6gpo0jV+Skz/NBqzTTu9isUHAgjtR2J6n/woIDOPbKyyULrNg40iVULYikAi8MZq4Cn3LSli2bDsw6t4WryTglVfHdVXLi3JxJxA==","X-MS-Exchange-Authentication-Results":"spf=pass (sender IP is 216.228.117.161)\n smtp.mailfrom=nvidia.com; dkim=none (message not signed)\n header.d=none;dmarc=pass action=none header.from=nvidia.com;","Received-SPF":"Pass (protection.outlook.com: domain of nvidia.com designates\n 216.228.117.161 as permitted sender) receiver=protection.outlook.com;\n client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C","From":"Nicolin Chen ","To":"Will Deacon , Robin Murphy , \"Joerg\n Roedel\" , Bjorn Helgaas , \"Jason\n Gunthorpe\" ","CC":"\"Rafael J . Wysocki\" , Len Brown ,\n\tPranjal Shrivastava , Mostafa Saleh ,\n\tLu Baolu , Kevin Tian ,\n\t, ,\n\t, ,\n\t, , Shuai Xue\n\t","Subject":"[PATCH v3 11/11] iommu/arm-smmu-v3: Block ATS upon an ATC\n invalidation timeout","Date":"Thu, 16 Apr 2026 16:28:40 -0700","Message-ID":"\n <5e2c2ddd25f80a165fc96db88f765b418a6ed038.1776381841.git.nicolinc@nvidia.com>","X-Mailer":"git-send-email 2.43.0","In-Reply-To":"","References":"","Precedence":"bulk","X-Mailing-List":"linux-pci@vger.kernel.org","List-Id":"","List-Subscribe":"","List-Unsubscribe":"","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Content-Type":"text/plain","X-NV-OnPremToCloud":"ExternallySecured","X-EOPAttributedMessage":"0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"CO1PEPF00012E83:EE_|CH0PR12MB8464:EE_","X-MS-Office365-Filtering-Correlation-Id":"f2b84d67-b811-465b-99b9-08de9c101bf3","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"\n\tBCL:0;ARA:13230040|1800799024|36860700016|7416014|376014|82310400026|18002099003|56012099003|22082099003;","X-Microsoft-Antispam-Message-Info":"\n\tjVPtvX/pNrb03MfdUO8k/xDoC8xakvDbrcsusHGK59d1IlRgdS2gAtQJwEcYipFBR7btGxPlDzrNwipu1UhMn4G5TI+oCI6DGJ4IdAyFqzqjvzkg0oT3mJ9UP8sZCp2QLpHp9qlJDw4M8XgM2MWISN3/hDmVdHdfT3XggugOZl5LKT0begfMMMEnxZvYAAfMAgJcJvkyOX97OBAcaLAI7w27HACejFL4r7XLv9EEPFBrYSHAY7pVbs5mkE7kUTF86X0+mKLYmblDjy76R757XogZQMcx9HRfhIGViQX1awUUP1TepDxgXv/0cR/CQXULVW12Wdp8+Z4vKX99RTkja35PyiR9x1lBakio2VuLljwcaNkDlDIF3oo68u6SvJ+AvS+anIf16Og6f0tilMXr9TwjmpDbWZYa/uiKBylzGKAZKFNCvvQJQKzGP6ehaNCyx4yP0CrEeCIRF1cTQQ781ufO+3CUol+L8mSI8RV756xg2ceoHEP4UpVULYF9ZRSobej3mTbLiwNz94mGmHjK4pfD5QDYCdf6hzsITot+eH9gYkfMU9KGD2KAUWBLzZK5ND4Vf5lh/Yk6+RiM+tOqlt6YAbvYmrI4tIbTH+TRqRAUfXqkVaQsMyMNgrXP12uE4R2TH7hNu4gbzymgutyQPIUxTWIMCsVNfJLDsrvfyfG5VWXvv3xcHwH8hvsCD+JVf7PvltcmifI4SFyBvi1O6Rxj9m8pbhkv28pmebkyLY3Z2LM+yiRUqZdF8GxRn6decdaP+mN0atpSGk7bSRcgmA==","X-Forefront-Antispam-Report":"\n\tCIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700016)(7416014)(376014)(82310400026)(18002099003)(56012099003)(22082099003);DIR:OUT;SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"\n\tX7gPsO67FM26KGQPxCW2D2r4yU/OxGKZROyVtElUdeFJKYOefHWEO/jy9mzr11e4i1iX8Nh3xIVOVqgXJIy/RUAAOnSiRP3Rp724ylUHt/YIgff5MgVPnEhtZ5robK/rSl/++ISsz/CkbQuHnTbJppvkhx0n2sz7KqArRfPb9KC6OeIKdou1Nax7etlA4TLFQHDk9CSFoopBTxAlSHCErZoI8f/YgqMXVqZXcMyOpbGNFTLcyM6BUdm0wuc+q4K/G6KBbWLdOs7OM9cW44W0zzD5A0Z3LXt+Tdr7/in7D7OJMEKcCqcBjq1mfSWaOfqz5lHa3T+hUtPekp5cHWU4L3K6yJr/nf17/aLuTt7vRxy1uCmUG9NPpELbXmlCHtRON4g4l3Mhswk6zW8qV7p/KjoujNqhuxIHXtBmOQwGSs6Eie9uiPCwD6KV3soEUymv","X-OriginatorOrg":"Nvidia.com","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"16 Apr 2026 23:30:13.4564\n (UTC)","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n f2b84d67-b811-465b-99b9-08de9c101bf3","X-MS-Exchange-CrossTenant-Id":"43083d15-7273-40c1-b7db-39efd9ccc17a","X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp":"\n TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com]","X-MS-Exchange-CrossTenant-AuthSource":"\n\tCO1PEPF00012E83.namprd03.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Anonymous","X-MS-Exchange-CrossTenant-FromEntityHeader":"HybridOnPrem","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CH0PR12MB8464"},"content":"Currently, when GERROR_CMDQ_ERR occurs, the arm_smmu_cmdq_skip_err() won't\ndo anything for the CMDQ_ERR_CERROR_ATC_INV_IDX.\n\nWhen a device wasn't responsive to an ATC invalidation request, this often\nresults in constant CMDQ errors:\n unexpected global error reported (0x00000001), this could be serious\n CMDQ error (cons 0x0302bb84): ATC invalidate timeout\n unexpected global error reported (0x00000001), this could be serious\n CMDQ error (cons 0x0302bb88): ATC invalidate timeout\n unexpected global error reported (0x00000001), this could be serious\n CMDQ error (cons 0x0302bb8c): ATC invalidate timeout\n ...\n\nAn ATC invalidation timeout indicates that the device failed to respond to\na protocol-critical coherency request, which means that device's internal\nATS state is desynchronized from the SMMU.\n\nFurthermore, ignoring the timeout leaves the system in an unsafe state, as\nthe device cache may retain stale ATC entries for memory pages that the OS\nhas already reclaimed and reassigned. This might lead to data corruption.\n\nIsolate the device that is confirmed to be unresponsive by a surgical STE\nupdate to unset its EATS bit so as to reject any further ATS transaction,\nwhich could corrupt the memory.\n\nAlso, set the master->ats_broken flag that is revertible after the device\ncompletes a reset. This flag avoids further ATS requests and invalidations\nfrom happening.\n\nFinally, report this broken device to the IOMMU core to isolate the device\nin the core level too.\n\nFor batched ATC_INV commands, SMMU hardware only reports a timeout at the\nCMD_SYNC, which could follow the batch issued for multiple devices. So, it\nisn't straightforward to identify which command in a batch resulted in the\ntimeout. Fortunately, the invs array has a sorted list of ATC entries. So,\nthe issued batch must be sorted as well. This makes it possible to retry\nthe ATC_INV command for each unique Stream ID in the batch to identify the\nunresponsive master.\n\nSigned-off-by: Nicolin Chen \n---\n drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 100 +++++++++++++++++++-\n 1 file changed, 97 insertions(+), 3 deletions(-)","diff":"diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c\nindex 5dead82cf1186..7dbd9c5834314 100644\n--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c\n+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c\n@@ -107,6 +107,8 @@ static const char * const event_class_str[] = {\n \t[3] = \"Reserved\",\n };\n \n+static struct arm_smmu_ste *\n+arm_smmu_get_step_for_sid(struct arm_smmu_device *smmu, u32 sid);\n static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);\n \n static void parse_driver_options(struct arm_smmu_device *smmu)\n@@ -2516,10 +2518,49 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size,\n \tcmd->atc.size\t= log2_span;\n }\n \n+static void arm_smmu_disable_eats_for_sid(struct arm_smmu_device *smmu,\n+\t\t\t\t\t struct arm_smmu_cmdq *cmdq, u32 sid)\n+{\n+\tstruct arm_smmu_cmdq_ent ent = {\n+\t\t.opcode = CMDQ_OP_CFGI_STE,\n+\t\t.cfgi\t= {\n+\t\t\t.sid = sid,\n+\t\t\t.leaf = true,\n+\t\t},\n+\t};\n+\tstruct arm_smmu_ste *step;\n+\tu64 cmd[CMDQ_ENT_DWORDS];\n+\t__le64 old, new;\n+\n+\tstep = arm_smmu_get_step_for_sid(smmu, sid);\n+\n+\told = READ_ONCE(step->data[1]);\n+\tdo {\n+\t\tnew = old & cpu_to_le64(~STRTAB_STE_1_EATS);\n+\t} while (!try_cmpxchg64(&step->data[1], &old, new));\n+\n+\tarm_smmu_cmdq_build_cmd(cmd, &ent);\n+\tif (arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmd, 1, true))\n+\t\tdev_err_ratelimited(smmu->dev,\n+\t\t\t\t \"failed to disable ATS for sid %#x\\n\", sid);\n+}\n+\n+static void arm_smmu_master_disable_ats(struct arm_smmu_master *master,\n+\t\t\t\t\tstruct arm_smmu_cmdq *cmdq)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < master->num_streams; i++)\n+\t\tarm_smmu_disable_eats_for_sid(master->smmu, cmdq,\n+\t\t\t\t\t master->streams[i].id);\n+\tWRITE_ONCE(master->ats_broken, true);\n+\tiommu_report_device_broken(master->dev);\n+}\n+\n static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,\n \t\t\t\t ioasid_t ssid)\n {\n-\tint i;\n+\tint i, ret;\n \tstruct arm_smmu_cmdq_ent cmd;\n \tstruct arm_smmu_cmdq_batch cmds;\n \n@@ -2535,7 +2576,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,\n \t\tarm_smmu_cmdq_batch_add(master->smmu, &cmds, &cmd);\n \t}\n \n-\treturn arm_smmu_cmdq_batch_submit(master->smmu, &cmds);\n+\tret = arm_smmu_cmdq_batch_submit(master->smmu, &cmds);\n+\tif (ret == -EIO)\n+\t\tarm_smmu_master_disable_ats(master, cmds.cmdq);\n+\treturn ret;\n }\n \n /* IO_PGTABLE API */\n@@ -2682,6 +2726,55 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,\n \treturn false;\n }\n \n+static void arm_smmu_invs_disable_ats(struct arm_smmu_invs *invs,\n+\t\t\t\t struct arm_smmu_cmdq *cmdq,\n+\t\t\t\t struct arm_smmu_device *smmu, u32 sid)\n+{\n+\tstruct arm_smmu_inv *cur;\n+\tsize_t i;\n+\n+\tarm_smmu_invs_for_each_entry(invs, i, cur) {\n+\t\tif (cur->master->smmu == smmu && arm_smmu_inv_is_ats(cur) &&\n+\t\t cur->id == sid) {\n+\t\t\tarm_smmu_master_disable_ats(cur->master, cmdq);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n+static void arm_smmu_cmdq_batch_retry(struct arm_smmu_device *smmu,\n+\t\t\t\t struct arm_smmu_invs *invs,\n+\t\t\t\t struct arm_smmu_cmdq_batch *cmds)\n+{\n+\tu64 atc[CMDQ_ENT_DWORDS] = {0};\n+\tint i;\n+\n+\t/* Only a timed out ATC_INV command needs a retry */\n+\tif (!invs->has_ats)\n+\t\treturn;\n+\n+\tfor (i = 0; i < cmds->num * CMDQ_ENT_DWORDS; i += CMDQ_ENT_DWORDS) {\n+\t\tstruct arm_smmu_cmdq *cmdq = cmds->cmdq;\n+\t\tu32 sid;\n+\t\tint ret;\n+\n+\t\t/* Only need to retry ATC invalidations */\n+\t\tif (FIELD_GET(CMDQ_0_OP, cmds->cmds[i]) != CMDQ_OP_ATC_INV)\n+\t\t\tcontinue;\n+\n+\t\t/* Only need to retry with one ATC_INV per Stream ID (device) */\n+\t\tsid = FIELD_GET(CMDQ_ATC_0_SID, cmds->cmds[i]);\n+\t\tif (atc[0] && sid == FIELD_GET(CMDQ_ATC_0_SID, atc[0]))\n+\t\t\tcontinue;\n+\n+\t\tatc[0] = cmds->cmds[i];\n+\t\tatc[1] = cmds->cmds[i + 1];\n+\t\tret = arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, atc, 1, true);\n+\t\tif (ret == -EIO)\n+\t\t\tarm_smmu_invs_disable_ats(invs, cmdq, smmu, sid);\n+\t}\n+}\n+\n static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,\n \t\t\t\t\tunsigned long iova, size_t size,\n \t\t\t\t\tunsigned int granule, bool leaf)\n@@ -2760,7 +2853,8 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,\n \n \t\tif (cmds.num &&\n \t\t (next == end || arm_smmu_invs_end_batch(cur, next))) {\n-\t\t\tarm_smmu_cmdq_batch_submit(smmu, &cmds);\n+\t\t\tif (arm_smmu_cmdq_batch_submit(smmu, &cmds) == -EIO)\n+\t\t\t\tarm_smmu_cmdq_batch_retry(smmu, invs, &cmds);\n \t\t\tcmds.num = 0;\n \t\t}\n \t\tcur = next;\n","prefixes":["v3","11/11"]}