[{"id":1786531,"web_url":"http://patchwork.ozlabs.org/comment/1786531/","msgid":"<20171013183237.GA30572@arm.com>","list_archive_url":null,"date":"2017-10-13T18:32:38","subject":"Re: [PATCH v2 3/4] iommu/arm-smmu-v3: Use CMD_SYNC completion MSI","submitter":{"id":7916,"url":"http://patchwork.ozlabs.org/api/people/7916/","name":"Will Deacon","email":"will.deacon@arm.com"},"content":"Hi Robin,\n\nThis mostly looks good. Just a few comments below.\n\nOn Thu, Aug 31, 2017 at 02:44:27PM +0100, Robin Murphy wrote:\n> As an IRQ, the CMD_SYNC interrupt is not particularly useful, not least\n> because we often need to wait for sync completion within someone else's\n> IRQ handler anyway. However, when the SMMU is both coherent and supports\n> MSIs, we can have a lot more fun by not using it as an interrupt at all.\n> Following the example suggested in the architecture and using a write\n> targeting normal memory, we can let callers wait on a status variable\n> outside the lock instead of having to stall the entire queue or even\n> touch MMIO registers. Since multiple sync commands are guaranteed to\n> complete in order, a simple incrementing sequence count is all we need\n> to unambiguously support any realistic number of overlapping waiters.\n> \n> Signed-off-by: Robin Murphy \n> ---\n> \n> v2: Remove redundant 'bool msi' command member, other cosmetic tweaks\n> \n> drivers/iommu/arm-smmu-v3.c | 47 +++++++++++++++++++++++++++++++++++++++++++--\n> 1 file changed, 45 insertions(+), 2 deletions(-)\n> \n> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c\n> index f066725298cd..311f482b93d5 100644\n> --- a/drivers/iommu/arm-smmu-v3.c\n> +++ b/drivers/iommu/arm-smmu-v3.c\n> @@ -377,7 +377,16 @@\n> \n> #define CMDQ_SYNC_0_CS_SHIFT\t\t12\n> #define CMDQ_SYNC_0_CS_NONE\t\t(0UL << CMDQ_SYNC_0_CS_SHIFT)\n> +#define CMDQ_SYNC_0_CS_IRQ\t\t(1UL << CMDQ_SYNC_0_CS_SHIFT)\n> #define CMDQ_SYNC_0_CS_SEV\t\t(2UL << CMDQ_SYNC_0_CS_SHIFT)\n> +#define CMDQ_SYNC_0_MSH_SHIFT\t\t22\n> +#define CMDQ_SYNC_0_MSH_ISH\t\t(3UL << CMDQ_SYNC_0_MSH_SHIFT)\n> +#define CMDQ_SYNC_0_MSIATTR_SHIFT\t24\n> +#define CMDQ_SYNC_0_MSIATTR_OIWB\t(0xfUL << CMDQ_SYNC_0_MSIATTR_SHIFT)\n> +#define CMDQ_SYNC_0_MSIDATA_SHIFT\t32\n> +#define CMDQ_SYNC_0_MSIDATA_MASK\t0xffffffffUL\n> +#define CMDQ_SYNC_1_MSIADDR_SHIFT\t0\n> +#define CMDQ_SYNC_1_MSIADDR_MASK\t0xffffffffffffcUL\n> \n> /* Event queue */\n> #define EVTQ_ENT_DWORDS\t\t\t4\n> @@ -409,6 +418,7 @@\n> /* High-level queue structures */\n> #define ARM_SMMU_POLL_TIMEOUT_US\t100\n> #define ARM_SMMU_CMDQ_DRAIN_TIMEOUT_US\t1000000 /* 1s! */\n> +#define ARM_SMMU_SYNC_TIMEOUT_US\t1000000 /* 1s! */\n\nWe only ever do this when waiting for the queue to drain, so may as well\njust reuse the drain timeout.\n\n> #define MSI_IOVA_BASE\t\t\t0x8000000\n> #define MSI_IOVA_LENGTH\t\t\t0x100000\n> @@ -504,6 +514,10 @@ struct arm_smmu_cmdq_ent {\n> \t\t} pri;\n> \n> \t\t#define CMDQ_OP_CMD_SYNC\t0x46\n> +\t\tstruct {\n> +\t\t\tu32\t\t\tmsidata;\n> +\t\t\tu64\t\t\tmsiaddr;\n> +\t\t} sync;\n> \t};\n> };\n> \n> @@ -617,6 +631,9 @@ struct arm_smmu_device {\n> \tint\t\t\t\tgerr_irq;\n> \tint\t\t\t\tcombined_irq;\n> \n> +\tatomic_t\t\t\tsync_nr;\n> +\tu32\t\t\t\tsync_count;\n\nIt's probably worth sticking these in separate cachelines so we don't\nget spurious wakeups when sync_nr is incremented. (yes, I know it should\nbe the ERG, but that can be unreasonably huge!).\n\n> +\n> \tunsigned long\t\t\tias; /* IPA */\n> \tunsigned long\t\t\toas; /* PA */\n> \tunsigned long\t\t\tpgsize_bitmap;\n> @@ -878,7 +895,13 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)\n> \t\t}\n> \t\tbreak;\n> \tcase CMDQ_OP_CMD_SYNC:\n> -\t\tcmd[0] |= CMDQ_SYNC_0_CS_SEV;\n> +\t\tif (ent->sync.msiaddr)\n> +\t\t\tcmd[0] |= CMDQ_SYNC_0_CS_IRQ;\n> +\t\telse\n> +\t\t\tcmd[0] |= CMDQ_SYNC_0_CS_SEV;\n> +\t\tcmd[0] |= CMDQ_SYNC_0_MSH_ISH | CMDQ_SYNC_0_MSIATTR_OIWB;\n> +\t\tcmd[0] |= (u64)ent->sync.msidata << CMDQ_SYNC_0_MSIDATA_SHIFT;\n> +\t\tcmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;\n> \t\tbreak;\n> \tdefault:\n> \t\treturn -ENOENT;\n> @@ -964,21 +987,40 @@ static void arm_smmu_cmdq_issue_cmd(struct arm_smmu_device *smmu,\n> \tspin_unlock_irqrestore(&smmu->cmdq.lock, flags);\n> }\n> \n> +static int arm_smmu_sync_poll_msi(struct arm_smmu_device *smmu, u32 sync_idx)\n> +{\n> +\tktime_t timeout = ktime_add_us(ktime_get(), ARM_SMMU_SYNC_TIMEOUT_US);\n> +\tu32 val = smp_cond_load_acquire(&smmu->sync_count,\n> +\t\t\t\t\t(int)(VAL - sync_idx) >= 0 ||\n> +\t\t\t\t\t!ktime_before(ktime_get(), timeout));\n> +\n> +\treturn (int)(val - sync_idx) < 0 ? -ETIMEDOUT : 0;\n\nThere are some theoretical overflow issues here which I don't think will\never occur in practice, but deserve at least a comment to explain why.\n\n> +}\n> +\n> static void arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu)\n> {\n> \tu64 cmd[CMDQ_ENT_DWORDS];\n> \tunsigned long flags;\n> \tbool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);\n> +\tbool msi = (smmu->features & ARM_SMMU_FEAT_MSI) &&\n> +\t\t (smmu->features & ARM_SMMU_FEAT_COHERENCY);\n\nI don't think this is sufficient for the case where we fail to setup MSIs\nand fall back on legacy IRQs.\n\n> \tstruct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC };\n> \tint ret;\n> \n> +\tif (msi) {\n> +\t\tent.sync.msidata = atomic_inc_return(&smmu->sync_nr);\n\nI don't think you need barrier semantics here.\n\n> +\t\tent.sync.msiaddr = virt_to_phys(&smmu->sync_count);\n> +\t}\n> \tarm_smmu_cmdq_build_cmd(cmd, &ent);\n> \n> \tspin_lock_irqsave(&smmu->cmdq.lock, flags);\n> \tarm_smmu_cmdq_insert_cmd(smmu, cmd);\n> -\tret = queue_poll_cons(&smmu->cmdq.q, true, wfe);\n> +\tif (!msi)\n> +\t\tret = queue_poll_cons(&smmu->cmdq.q, true, wfe);\n> \tspin_unlock_irqrestore(&smmu->cmdq.lock, flags);\n> \n> +\tif (msi)\n> +\t\tret = arm_smmu_sync_poll_msi(smmu, ent.sync.msidata);\n\nThis looks like the queue polling should be wrapped up in a function that\nreturns with the spinlock released.\n\nWill","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"qyz1nKW/\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yDGYV2sD2z9s1h\n\tfor ;\n\tSat, 14 Oct 2017 05:33:02 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e34lX-0001qG-47; Fri, 13 Oct 2017 18:32:59 +0000","from foss.arm.com ([217.140.101.70])\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e34lT-0001nX-CJ for linux-arm-kernel@lists.infradead.org;\n\tFri, 13 Oct 2017 18:32:57 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B70B31596;\n\tFri, 13 Oct 2017 11:32:34 -0700 (PDT)","from edgewater-inn.cambridge.arm.com\n\t(usa-sjc-imap-foss1.foss.arm.com [10.72.51.249])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id\n\t87DC23F58C; Fri, 13 Oct 2017 11:32:34 -0700 (PDT)","by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000)\n\tid A49211AE2D71; Fri, 13 Oct 2017 19:32:38 +0100 (BST)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:\n\tMessage-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description:\n\tResent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=sA9CZK6HdAxfhZszLdsOUTEaKDHAa8wEWV+AU1VonIk=;\n\tb=qyz1nKW/5tqLWe\n\tZw/DgHOSn9v0cFbAWN6DU5C7eJv0l9n3+P5YqZXE8wplbiTPEzLMpv+IXmn84mrcYDXRo3F5/hENN\n\tSHotcoa2KoIcqeD6yvRBEaL93smsXmyQYXfLye//UCwt8W8jYx4Cvuq0wwqprKuQgGcwEt8KxIP0V\n\tr18fv8wjj42f1iJDzyO6hIWaB3D288z8MZXvbj2rxSQ8vXlIki/aQTImIkiLVIn/m/JToS1QfjIvK\n\tDYQo1738kqtbI9eb3ynN/GSAgwzA08h9W5pb+Wyzrg0k4IGM0X1B4LxMa1KiCtxwuhn9ZZrwyDrxb\n\tvM7lAq4toK9ptZ2jpHBw==;","Date":"Fri, 13 Oct 2017 19:32:38 +0100","From":"Will Deacon ","To":"Robin Murphy ","Subject":"Re: [PATCH v2 3/4] iommu/arm-smmu-v3: Use CMD_SYNC completion MSI","Message-ID":"<20171013183237.GA30572@arm.com>","References":"\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.23 (2014-03-12)","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171013_113255_444602_77261213 ","X-CRM114-Status":"GOOD ( 23.64 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}},{"id":1786559,"web_url":"http://patchwork.ozlabs.org/comment/1786559/","msgid":"<20171013185917.GB30572@arm.com>","list_archive_url":null,"date":"2017-10-13T18:59:17","subject":"Re: [PATCH v2 4/4] iommu/arm-smmu-v3: Poll for CMD_SYNC outside cmdq\n\tlock","submitter":{"id":7916,"url":"http://patchwork.ozlabs.org/api/people/7916/","name":"Will Deacon","email":"will.deacon@arm.com"},"content":"Hi Robin,\n\nSome of my comments on patch 3 are addressed here, but I'm really struggling\nto convince myself that this algorithm is correct. My preference would\nbe to leave the code as it is for SMMUs that don't implement MSIs, but\ncomments below anyway because it's an interesting idea.\n\nOn Thu, Aug 31, 2017 at 02:44:28PM +0100, Robin Murphy wrote:\n> Even without the MSI trick, we can still do a lot better than hogging\n> the entire queue while it drains. All we actually need to do for the\n> necessary guarantee of completion is wait for our particular command to\n> have been consumed, and as long as we keep track of where it is there is\n> no need to block other CPUs from adding further commands in the\n> meantime. There is one theoretical (but incredibly unlikely) edge case\n> to avoid, where cons has wrapped twice to still appear 'behind' the sync\n> position - this is easily disambiguated by adding a generation count to\n> the queue to indicate when prod wraps, since cons cannot wrap twice\n> without prod having wrapped at least once.\n> \n> Signed-off-by: Robin Murphy \n> ---\n> \n> v2: New\n> \n> drivers/iommu/arm-smmu-v3.c | 81 ++++++++++++++++++++++++++++++++-------------\n> 1 file changed, 58 insertions(+), 23 deletions(-)\n> \n> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c\n> index 311f482b93d5..f5c5da553803 100644\n> --- a/drivers/iommu/arm-smmu-v3.c\n> +++ b/drivers/iommu/arm-smmu-v3.c\n> @@ -417,7 +417,6 @@\n> \n> /* High-level queue structures */\n> #define ARM_SMMU_POLL_TIMEOUT_US\t100\n> -#define ARM_SMMU_CMDQ_DRAIN_TIMEOUT_US\t1000000 /* 1s! */\n> #define ARM_SMMU_SYNC_TIMEOUT_US\t1000000 /* 1s! */\n> \n> #define MSI_IOVA_BASE\t\t\t0x8000000\n> @@ -540,6 +539,7 @@ struct arm_smmu_queue {\n> struct arm_smmu_cmdq {\n> \tstruct arm_smmu_queue\t\tq;\n> \tspinlock_t\t\t\tlock;\n> +\tint\t\t\t\tgeneration;\n> };\n> \n> struct arm_smmu_evtq {\n> @@ -770,21 +770,12 @@ static void queue_inc_prod(struct arm_smmu_queue *q)\n> \twritel(q->prod, q->prod_reg);\n> }\n> \n> -/*\n> - * Wait for the SMMU to consume items. If drain is true, wait until the queue\n> - * is empty. Otherwise, wait until there is at least one free slot.\n> - */\n> -static int queue_poll_cons(struct arm_smmu_queue *q, bool drain, bool wfe)\n> +/* Wait for the SMMU to consume items, until there is at least one free slot */\n> +static int queue_poll_cons(struct arm_smmu_queue *q, bool wfe)\n> {\n> -\tktime_t timeout;\n> -\tunsigned int delay = 1;\n> +\tktime_t timeout = ktime_add_us(ktime_get(), ARM_SMMU_POLL_TIMEOUT_US);\n> \n> -\t/* Wait longer if it's queue drain */\n> -\ttimeout = ktime_add_us(ktime_get(), drain ?\n> -\t\t\t\t\t ARM_SMMU_CMDQ_DRAIN_TIMEOUT_US :\n> -\t\t\t\t\t ARM_SMMU_POLL_TIMEOUT_US);\n> -\n> -\twhile (queue_sync_cons(q), (drain ? !queue_empty(q) : queue_full(q))) {\n> +\twhile (queue_sync_cons(q), queue_full(q)) {\n> \t\tif (ktime_compare(ktime_get(), timeout) > 0)\n> \t\t\treturn -ETIMEDOUT;\n> \n> @@ -792,8 +783,7 @@ static int queue_poll_cons(struct arm_smmu_queue *q, bool drain, bool wfe)\n> \t\t\twfe();\n> \t\t} else {\n> \t\t\tcpu_relax();\n> -\t\t\tudelay(delay);\n> -\t\t\tdelay *= 2;\n> +\t\t\tudelay(1);\n> \t\t}\n> \t}\n> \n> @@ -959,15 +949,20 @@ static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu)\n> \tqueue_write(Q_ENT(q, cons), cmd, q->ent_dwords);\n> }\n> \n> -static void arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd)\n> +static u32 arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd)\n> {\n> \tstruct arm_smmu_queue *q = &smmu->cmdq.q;\n> \tbool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);\n> \n> \twhile (queue_insert_raw(q, cmd) == -ENOSPC) {\n> -\t\tif (queue_poll_cons(q, false, wfe))\n> +\t\tif (queue_poll_cons(q, wfe))\n> \t\t\tdev_err_ratelimited(smmu->dev, \"CMDQ timeout\\n\");\n> \t}\n> +\n> +\tif (Q_IDX(q, q->prod) == 0)\n> +\t\tsmmu->cmdq.generation++;\n\nThe readers of generation are using READ_ONCE, so you're missing something\nhere.\n\n> +\n> +\treturn q->prod;\n> }\n> \n> static void arm_smmu_cmdq_issue_cmd(struct arm_smmu_device *smmu,\n> @@ -997,15 +992,54 @@ static int arm_smmu_sync_poll_msi(struct arm_smmu_device *smmu, u32 sync_idx)\n> \treturn (int)(val - sync_idx) < 0 ? -ETIMEDOUT : 0;\n> }\n> \n> +static int arm_smmu_sync_poll_cons(struct arm_smmu_device *smmu, u32 sync_idx,\n> +\t\t\t\t int sync_gen)\n> +{\n> +\tktime_t timeout = ktime_add_us(ktime_get(), ARM_SMMU_SYNC_TIMEOUT_US);\n> +\tstruct arm_smmu_queue *q = &smmu->cmdq.q;\n> +\tbool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);\n> +\tunsigned int delay = 1;\n> +\n> +\tdo {\n> +\t\tqueue_sync_cons(q);\n> +\t\t/*\n> +\t\t * If we see updates quickly enough, cons has passed sync_idx,\n> +\t\t * but not yet wrapped. At worst, cons might have actually\n> +\t\t * wrapped an even number of times, but that still guarantees\n> +\t\t * the original sync must have been consumed.\n> +\t\t */\n> +\t\tif (Q_IDX(q, q->cons) >= Q_IDX(q, sync_idx) &&\n> +\t\t Q_WRP(q, sync_idx) == Q_WRP(q, q->cons))\n\nCan you move this into a separate function please, like we have for\nqueue_full and queue_empty?\n\n> +\t\t\treturn 0;\n> +\t\t/*\n> +\t\t * Otherwise, cons may have passed sync_idx and wrapped one or\n> +\t\t * more times to appear behind it again, but in that case prod\n> +\t\t * must also be one or more generations ahead.\n> +\t\t */\n> +\t\tif (Q_IDX(q, q->cons) < Q_IDX(q, sync_idx) &&\n> +\t\t READ_ONCE(smmu->cmdq.generation) != sync_gen)\n\nThere's another daft overflow case here which deserves a comment (and even\nif it *did* happen, we'll recover gracefully).\n\n> +\t\t\treturn 0;\n> +\n> +\t\tif (wfe) {\n> +\t\t\twfe();\n\nThis is a bit scary... if we miss a generation update, just due to store\npropagation latency (maybe it's buffered by the updater), I think we can\nend up going into wfe when there's not an event pending. Using xchg\neverywhere might work, but there's still a race on having somebody update\ngeneration. The ordering here just looks generally suspicious to me because\nyou have the generation writer in arm_smmu_cmdq_insert_cmd effectively\ndoing:\n\n Write prod\n Write generation\n\nand the reader in arm_smmu_sync_poll_cons doing:\n\n Read cons\n Read generation\n\nso I can't see anything that gives you order to guarantee that the\ngeneration is seen to be up-to-date.\n\nBut it's also Friday evening and my brain is pretty fried ;)\n\nWill","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"QvWdV8/t\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yDH8F5Q41z9s7c\n\tfor ;\n\tSat, 14 Oct 2017 05:59:41 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e35BK-0005Wd-Go; Fri, 13 Oct 2017 18:59:38 +0000","from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]\n\thelo=foss.arm.com)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e35BG-0005A8-MS for linux-arm-kernel@lists.infradead.org;\n\tFri, 13 Oct 2017 18:59:36 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 11B501596;\n\tFri, 13 Oct 2017 11:59:14 -0700 (PDT)","from edgewater-inn.cambridge.arm.com\n\t(usa-sjc-imap-foss1.foss.arm.com [10.72.51.249])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id\n\tD5C773F58C; Fri, 13 Oct 2017 11:59:13 -0700 (PDT)","by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000)\n\tid F2C011AE2D71; Fri, 13 Oct 2017 19:59:17 +0100 (BST)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:\n\tMessage-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description:\n\tResent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=+phN+KXXveL9MgD5zXyvPRStYaWSiA2lGbbZxZ3O4OU=;\n\tb=QvWdV8/tFdcF+e\n\tZyuQoEOZ6c5R4fNwS1g3eDYh/G5dgrB1bVdtkol60gdCu3E4JQR2Z5pHDZfpMhRI6f4ab18XeTPpS\n\tsyRxzdpjLpPnJgiDVbi1oH9nRqAYk3Z554qjUB/b/ZgDNkbZhMJFrLPLCaJba07jVJvxv7ffpY84S\n\tAFJS5QcvOON50zUYDUs1J6bVTUj73/RPLMk+Wt76aYqDQJMZZ7sQPSy2885myRUBq8cK0/GIgOwHR\n\teGTckzp+hM0xXjmibjZ/pdgfJZf78lKH8otRQOiHpKqViubIYCcDaRuh1KGvA1EKy65zxBmRI7aXr\n\tqZN5n02C70Q0AVC9X0Vg==;","Date":"Fri, 13 Oct 2017 19:59:17 +0100","From":"Will Deacon ","To":"Robin Murphy ","Subject":"Re: [PATCH v2 4/4] iommu/arm-smmu-v3: Poll for CMD_SYNC outside cmdq\n\tlock","Message-ID":"<20171013185917.GB30572@arm.com>","References":"\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.23 (2014-03-12)","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171013_115934_753903_11A2E8D9 ","X-CRM114-Status":"GOOD ( 28.70 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}},{"id":1786568,"web_url":"http://patchwork.ozlabs.org/comment/1786568/","msgid":"<20171013190521.GD30572@arm.com>","list_archive_url":null,"date":"2017-10-13T19:05:22","subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","submitter":{"id":7916,"url":"http://patchwork.ozlabs.org/api/people/7916/","name":"Will Deacon","email":"will.deacon@arm.com"},"content":"Hi Robin,\n\nOn Thu, Aug 31, 2017 at 02:44:24PM +0100, Robin Murphy wrote:\n> Since Nate reported a reasonable performance boost from the out-of-line\n> MSI polling in v1 [1], I've now implemented the equivalent for cons\n> polling as well - that has been boot-tested on D05 with some trivial I/O\n> and at least doesn't seem to lock up or explode. There's also a little\n> cosmetic tweaking to make the patches a bit cleaner as a series.\n> \n> Robin.\n> \n> [1] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg19657.html\n> \n> Robin Murphy (5):\n> iommu/arm-smmu-v3: Specialise CMD_SYNC handling\n> iommu/arm-smmu-v3: Forget about cmdq-sync interrupt\n> iommu/arm-smmu-v3: Use CMD_SYNC completion interrupt\n> iommu/arm-smmu-v3: Poll for CMD_SYNC outside cmdq lock\n> iommu/arm-smmu-v3: Use burst-polling for sync completion\n\nWhat's this final mythical patch about? I don't see it in the series.\n\nAnyway, the first two patches look fine to me, but this doesn't apply\non top of my iommu/devel branch so they will need rebasing.\n\nCheers,\n\nWill","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"XZV9YWAt\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yDHHH1KmCz9s7h\n\tfor ;\n\tSat, 14 Oct 2017 06:05:47 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e35HC-0001Dq-L7; Fri, 13 Oct 2017 19:05:42 +0000","from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]\n\thelo=foss.arm.com)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e35H8-0001CT-Tm for linux-arm-kernel@lists.infradead.org;\n\tFri, 13 Oct 2017 19:05:40 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 782001596;\n\tFri, 13 Oct 2017 12:05:18 -0700 (PDT)","from edgewater-inn.cambridge.arm.com\n\t(usa-sjc-imap-foss1.foss.arm.com [10.72.51.249])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id\n\t48FEC3F58C; Fri, 13 Oct 2017 12:05:18 -0700 (PDT)","by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000)\n\tid 676491AE2D76; Fri, 13 Oct 2017 20:05:22 +0100 (BST)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:\n\tMessage-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description:\n\tResent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=sPo7x9ZzMADNe8sJT+znikF4v6W0MGjTV5c05ru8IAE=;\n\tb=XZV9YWAtWMDx0V\n\t+iiSYs996CP2yhF1mxL05MJc534x9Xi/jsYPeHSUnDnAfm4XVVIbsUX/A4F+4GC8SDswyviOe3rlZ\n\tGclvCeH/7u27UHV5iKMk4l37G18Q4Tqsf1b224yG6+pCUNDJZli/VIcAPkegGsvYqlVuKD6V3KAXq\n\tvwNbrhQg+TxJLSREyms5un5uLr7vFFR9mFQASX2sWPJD5ZsqEodtg7V97FBvmc00dKbOkGuPJ14Rc\n\t8DiG1f9mYnx8Ag1prvXtgosBNVN2Nw18q/cojxFC2WD2eTPH+dBVI4fqnF6mOEzAcm77Ffl07JrFH\n\tU8MvYVyuk2orM6/+hNMg==;","Date":"Fri, 13 Oct 2017 20:05:22 +0100","From":"Will Deacon ","To":"Robin Murphy ","Subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","Message-ID":"<20171013190521.GD30572@arm.com>","References":"","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.23 (2014-03-12)","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171013_120538_984127_E4FB12C7 ","X-CRM114-Status":"GOOD ( 11.23 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}},{"id":1787384,"web_url":"http://patchwork.ozlabs.org/comment/1787384/","msgid":"<9dbbac49-669e-9294-7e8e-26a55ebb1d04@arm.com>","list_archive_url":null,"date":"2017-10-16T12:25:17","subject":"Re: [PATCH v2 3/4] iommu/arm-smmu-v3: Use CMD_SYNC completion MSI","submitter":{"id":65641,"url":"http://patchwork.ozlabs.org/api/people/65641/","name":"Robin Murphy","email":"robin.murphy@arm.com"},"content":"On 13/10/17 19:32, Will Deacon wrote:\n> Hi Robin,\n> \n> This mostly looks good. Just a few comments below.\n> \n> On Thu, Aug 31, 2017 at 02:44:27PM +0100, Robin Murphy wrote:\n>> As an IRQ, the CMD_SYNC interrupt is not particularly useful, not least\n>> because we often need to wait for sync completion within someone else's\n>> IRQ handler anyway. However, when the SMMU is both coherent and supports\n>> MSIs, we can have a lot more fun by not using it as an interrupt at all.\n>> Following the example suggested in the architecture and using a write\n>> targeting normal memory, we can let callers wait on a status variable\n>> outside the lock instead of having to stall the entire queue or even\n>> touch MMIO registers. Since multiple sync commands are guaranteed to\n>> complete in order, a simple incrementing sequence count is all we need\n>> to unambiguously support any realistic number of overlapping waiters.\n>>\n>> Signed-off-by: Robin Murphy \n>> ---\n>>\n>> v2: Remove redundant 'bool msi' command member, other cosmetic tweaks\n>>\n>> drivers/iommu/arm-smmu-v3.c | 47 +++++++++++++++++++++++++++++++++++++++++++--\n>> 1 file changed, 45 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c\n>> index f066725298cd..311f482b93d5 100644\n>> --- a/drivers/iommu/arm-smmu-v3.c\n>> +++ b/drivers/iommu/arm-smmu-v3.c\n>> @@ -377,7 +377,16 @@\n>> \n>> #define CMDQ_SYNC_0_CS_SHIFT\t\t12\n>> #define CMDQ_SYNC_0_CS_NONE\t\t(0UL << CMDQ_SYNC_0_CS_SHIFT)\n>> +#define CMDQ_SYNC_0_CS_IRQ\t\t(1UL << CMDQ_SYNC_0_CS_SHIFT)\n>> #define CMDQ_SYNC_0_CS_SEV\t\t(2UL << CMDQ_SYNC_0_CS_SHIFT)\n>> +#define CMDQ_SYNC_0_MSH_SHIFT\t\t22\n>> +#define CMDQ_SYNC_0_MSH_ISH\t\t(3UL << CMDQ_SYNC_0_MSH_SHIFT)\n>> +#define CMDQ_SYNC_0_MSIATTR_SHIFT\t24\n>> +#define CMDQ_SYNC_0_MSIATTR_OIWB\t(0xfUL << CMDQ_SYNC_0_MSIATTR_SHIFT)\n>> +#define CMDQ_SYNC_0_MSIDATA_SHIFT\t32\n>> +#define CMDQ_SYNC_0_MSIDATA_MASK\t0xffffffffUL\n>> +#define CMDQ_SYNC_1_MSIADDR_SHIFT\t0\n>> +#define CMDQ_SYNC_1_MSIADDR_MASK\t0xffffffffffffcUL\n>> \n>> /* Event queue */\n>> #define EVTQ_ENT_DWORDS\t\t\t4\n>> @@ -409,6 +418,7 @@\n>> /* High-level queue structures */\n>> #define ARM_SMMU_POLL_TIMEOUT_US\t100\n>> #define ARM_SMMU_CMDQ_DRAIN_TIMEOUT_US\t1000000 /* 1s! */\n>> +#define ARM_SMMU_SYNC_TIMEOUT_US\t1000000 /* 1s! */\n> \n> We only ever do this when waiting for the queue to drain, so may as well\n> just reuse the drain timeout.\n\nAs you've discovered, we remove the \"drain\" case entirely in the end.\n\n>> #define MSI_IOVA_BASE\t\t\t0x8000000\n>> #define MSI_IOVA_LENGTH\t\t\t0x100000\n>> @@ -504,6 +514,10 @@ struct arm_smmu_cmdq_ent {\n>> \t\t} pri;\n>> \n>> \t\t#define CMDQ_OP_CMD_SYNC\t0x46\n>> +\t\tstruct {\n>> +\t\t\tu32\t\t\tmsidata;\n>> +\t\t\tu64\t\t\tmsiaddr;\n>> +\t\t} sync;\n>> \t};\n>> };\n>> \n>> @@ -617,6 +631,9 @@ struct arm_smmu_device {\n>> \tint\t\t\t\tgerr_irq;\n>> \tint\t\t\t\tcombined_irq;\n>> \n>> +\tatomic_t\t\t\tsync_nr;\n>> +\tu32\t\t\t\tsync_count;\n> \n> It's probably worth sticking these in separate cachelines so we don't\n> get spurious wakeups when sync_nr is incremented. (yes, I know it should\n> be the ERG, but that can be unreasonably huge!).\n\nGood point - we've got 8K of bitmaps embedded in the structure anyway,\nso even maximum ERG separation is easily possible.\n\n>> +\n>> \tunsigned long\t\t\tias; /* IPA */\n>> \tunsigned long\t\t\toas; /* PA */\n>> \tunsigned long\t\t\tpgsize_bitmap;\n>> @@ -878,7 +895,13 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)\n>> \t\t}\n>> \t\tbreak;\n>> \tcase CMDQ_OP_CMD_SYNC:\n>> -\t\tcmd[0] |= CMDQ_SYNC_0_CS_SEV;\n>> +\t\tif (ent->sync.msiaddr)\n>> +\t\t\tcmd[0] |= CMDQ_SYNC_0_CS_IRQ;\n>> +\t\telse\n>> +\t\t\tcmd[0] |= CMDQ_SYNC_0_CS_SEV;\n>> +\t\tcmd[0] |= CMDQ_SYNC_0_MSH_ISH | CMDQ_SYNC_0_MSIATTR_OIWB;\n>> +\t\tcmd[0] |= (u64)ent->sync.msidata << CMDQ_SYNC_0_MSIDATA_SHIFT;\n>> +\t\tcmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;\n>> \t\tbreak;\n>> \tdefault:\n>> \t\treturn -ENOENT;\n>> @@ -964,21 +987,40 @@ static void arm_smmu_cmdq_issue_cmd(struct arm_smmu_device *smmu,\n>> \tspin_unlock_irqrestore(&smmu->cmdq.lock, flags);\n>> }\n>> \n>> +static int arm_smmu_sync_poll_msi(struct arm_smmu_device *smmu, u32 sync_idx)\n>> +{\n>> +\tktime_t timeout = ktime_add_us(ktime_get(), ARM_SMMU_SYNC_TIMEOUT_US);\n>> +\tu32 val = smp_cond_load_acquire(&smmu->sync_count,\n>> +\t\t\t\t\t(int)(VAL - sync_idx) >= 0 ||\n>> +\t\t\t\t\t!ktime_before(ktime_get(), timeout));\n>> +\n>> +\treturn (int)(val - sync_idx) < 0 ? -ETIMEDOUT : 0;\n> \n> There are some theoretical overflow issues here which I don't think will\n> ever occur in practice, but deserve at least a comment to explain why.\n\nEven if we did have 2^31 or more CPUs, the size of a queue is bounded at\n2^20, so we can never have enough in-flight syncs to get near to an\noverflow problem. I can certainly document that if you like.\n\n>> +}\n>> +\n>> static void arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu)\n>> {\n>> \tu64 cmd[CMDQ_ENT_DWORDS];\n>> \tunsigned long flags;\n>> \tbool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);\n>> +\tbool msi = (smmu->features & ARM_SMMU_FEAT_MSI) &&\n>> +\t\t (smmu->features & ARM_SMMU_FEAT_COHERENCY);\n> \n> I don't think this is sufficient for the case where we fail to setup MSIs\n> and fall back on legacy IRQs.\n\nRemember this 'MSI' is going nowhere near the GIC, so the IRQ\nconfiguration is irrelevant (especially after patch #2) - the feature\nbits tell us \"is the SMMU capable of generating sync-completion writes?\"\nand \"are those writes coherent?\", which is precisely what matters here.\n\n>> \tstruct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC };\n>> \tint ret;\n>> \n>> +\tif (msi) {\n>> +\t\tent.sync.msidata = atomic_inc_return(&smmu->sync_nr);\n> \n> I don't think you need barrier semantics here.\n\nYou mean atomic_inc_return_relaxed() would be sufficient? TBH I don't\nthink I'd given this any thought - I guess the coherency protocols make\nit impossible to do an atomic op on stale data, so that seems reasonable.\n>> +\t\tent.sync.msiaddr = virt_to_phys(&smmu->sync_count);\n>> +\t}\n>> \tarm_smmu_cmdq_build_cmd(cmd, &ent);\n>> \n>> \tspin_lock_irqsave(&smmu->cmdq.lock, flags);\n>> \tarm_smmu_cmdq_insert_cmd(smmu, cmd);\n>> -\tret = queue_poll_cons(&smmu->cmdq.q, true, wfe);\n>> +\tif (!msi)\n>> +\t\tret = queue_poll_cons(&smmu->cmdq.q, true, wfe);\n>> \tspin_unlock_irqrestore(&smmu->cmdq.lock, flags);\n>> \n>> +\tif (msi)\n>> +\t\tret = arm_smmu_sync_poll_msi(smmu, ent.sync.msidata);\n> \n> This looks like the queue polling should be wrapped up in a function that\n> returns with the spinlock released.\n\nI did ponder that, but I can't help finding such asymmetric interfaces\npretty grim, and things do get better again once both cases can wait\noutside the lock.\n\nRobin.","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"SNSJzpe5\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yFyGP4RRzz9sDB\n\tfor ;\n\tMon, 16 Oct 2017 23:25:49 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e44Sn-0005Pq-DB; Mon, 16 Oct 2017 12:25:45 +0000","from foss.arm.com ([217.140.101.70])\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e44Si-0005Ni-QB for linux-arm-kernel@lists.infradead.org;\n\tMon, 16 Oct 2017 12:25:42 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3DE991596;\n\tMon, 16 Oct 2017 05:25:20 -0700 (PDT)","from [10.1.210.88] (e110467-lin.cambridge.arm.com [10.1.210.88])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id\n\t085E53F53D; Mon, 16 Oct 2017 05:25:18 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:\n\tMessage-ID:From:References:To:Subject:Reply-To:Content-ID:Content-Description\n\t:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=DD37GK9AJvBKjXEzGNjcXZH58i+yX0OKS8wmZBoSNoE=;\n\tb=SNSJzpe547PhsQ\n\tSxeHIJutiGsj7RCfvJj0ytmRl05xsOEGsNVl86D0xgvclqM9I/KINZ7EUbCuRlroCpHM/4R3VALeB\n\tJFwh2gdboDaqj8aQBH0ATQfrpc7ze8Jb6Sstf9oQ8tGY6VYmkXkzpoEXm2BecJF42mF7q7NcVbKJT\n\t1WkBODzlOsKT+OrWkriSQw8c20qckI+mEtZEG9xBDp1uYvozZ+H8bqN1lTq3UrsnSMblc1bQxaqzb\n\tmTqL1D+Y5ahFfLgVDpuy/4anwv3cjaIg6jiKwMlVk8lp7hiXgfPXVUppUwUWo6lPMDV7rvtykr24A\n\t44otdnBa4MjUP2XoRzQQ==;","Subject":"Re: [PATCH v2 3/4] iommu/arm-smmu-v3: Use CMD_SYNC completion MSI","To":"Will Deacon ","References":"\n\t\n\t<20171013183237.GA30572@arm.com>","From":"Robin Murphy ","Message-ID":"<9dbbac49-669e-9294-7e8e-26a55ebb1d04@arm.com>","Date":"Mon, 16 Oct 2017 13:25:17 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20171013183237.GA30572@arm.com>","Content-Language":"en-US","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171016_052540_876662_B4EBD31D ","X-CRM114-Status":"GOOD ( 25.63 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}},{"id":1787427,"web_url":"http://patchwork.ozlabs.org/comment/1787427/","msgid":"<0414f685-1673-bb4e-bd7a-b99f2054b86e@arm.com>","list_archive_url":null,"date":"2017-10-16T13:18:00","subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","submitter":{"id":65641,"url":"http://patchwork.ozlabs.org/api/people/65641/","name":"Robin Murphy","email":"robin.murphy@arm.com"},"content":"On 13/10/17 20:05, Will Deacon wrote:\n> Hi Robin,\n> \n> On Thu, Aug 31, 2017 at 02:44:24PM +0100, Robin Murphy wrote:\n>> Since Nate reported a reasonable performance boost from the out-of-line\n>> MSI polling in v1 [1], I've now implemented the equivalent for cons\n>> polling as well - that has been boot-tested on D05 with some trivial I/O\n>> and at least doesn't seem to lock up or explode. There's also a little\n>> cosmetic tweaking to make the patches a bit cleaner as a series.\n>>\n>> Robin.\n>>\n>> [1] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg19657.html\n>>\n>> Robin Murphy (5):\n>> iommu/arm-smmu-v3: Specialise CMD_SYNC handling\n>> iommu/arm-smmu-v3: Forget about cmdq-sync interrupt\n>> iommu/arm-smmu-v3: Use CMD_SYNC completion interrupt\n>> iommu/arm-smmu-v3: Poll for CMD_SYNC outside cmdq lock\n>> iommu/arm-smmu-v3: Use burst-polling for sync completion\n> \n> What's this final mythical patch about? I don't see it in the series.\n\nIt's in the place 5/5 would be, I just tagged it as [RFT] since I have\nzero evidence whether it's worth the bother or not.\n\n> Anyway, the first two patches look fine to me, but this doesn't apply\n> on top of my iommu/devel branch so they will need rebasing.\n\nWill do.\n\nThanks for the review,\nRobin.","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"QPHsyQxY\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yFzRB07LKz9t3R\n\tfor ;\n\tTue, 17 Oct 2017 00:18:30 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e45Hm-0004Iy-NA; Mon, 16 Oct 2017 13:18:26 +0000","from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]\n\thelo=foss.arm.com)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e45Hj-0004I3-Tp for linux-arm-kernel@lists.infradead.org;\n\tMon, 16 Oct 2017 13:18:25 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 38D241529;\n\tMon, 16 Oct 2017 06:18:03 -0700 (PDT)","from [10.1.210.88] (e110467-lin.cambridge.arm.com [10.1.210.88])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id\n\t044603F53D; Mon, 16 Oct 2017 06:18:01 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:\n\tMessage-ID:From:References:To:Subject:Reply-To:Content-ID:Content-Description\n\t:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=llVvgp33ZUmuu2EVURvdRMWtlhib0SVRrKZ5xUmG3TA=;\n\tb=QPHsyQxYrYiqkn\n\txzhwtJ7g4nAoh0ePusEibEoLDFpDxSnsiuzkXTAr+TOtX7ttv4/0Gv/Nyi7+ikn+m3S29MLGWuWtT\n\tAlp2XfgqilpapMdOjwbYocZhwwoOJzR91yiv6zinsYeKK5o6MWwQrIveTAPAb+uvujAWLo3cZSWSP\n\t8Vivm2Ngqdug3nellUCuz+M06S87pjRoTOxO6PJVJVM4KgxSUKn4+Bn/ug9ys0lBBid0QhUmloF2J\n\ttKkjJsVqG2jJHvZYigiZQP9/0/KnRui+X3andW4jsCT13efTeVGEVBgVYU8x5vSrMNsDkeRRENd+R\n\tQ1M4zT9bgHHB2K7JxbEQ==;","Subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","To":"Will Deacon ","References":"\n\t<20171013190521.GD30572@arm.com>","From":"Robin Murphy ","Message-ID":"<0414f685-1673-bb4e-bd7a-b99f2054b86e@arm.com>","Date":"Mon, 16 Oct 2017 14:18:00 +0100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20171013190521.GD30572@arm.com>","Content-Language":"en-US","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171016_061823_986674_C02901A9 ","X-CRM114-Status":"GOOD ( 11.82 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}},{"id":1787491,"web_url":"http://patchwork.ozlabs.org/comment/1787491/","msgid":"<20171016150208.GA10500@arm.com>","list_archive_url":null,"date":"2017-10-16T15:02:09","subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","submitter":{"id":7916,"url":"http://patchwork.ozlabs.org/api/people/7916/","name":"Will Deacon","email":"will.deacon@arm.com"},"content":"On Fri, Oct 13, 2017 at 08:05:21PM +0100, Will Deacon wrote:\n> On Thu, Aug 31, 2017 at 02:44:24PM +0100, Robin Murphy wrote:\n> > Since Nate reported a reasonable performance boost from the out-of-line\n> > MSI polling in v1 [1], I've now implemented the equivalent for cons\n> > polling as well - that has been boot-tested on D05 with some trivial I/O\n> > and at least doesn't seem to lock up or explode. There's also a little\n> > cosmetic tweaking to make the patches a bit cleaner as a series.\n> > \n> > Robin.\n> > \n> > [1] https://www.mail-archive.com/iommu@lists.linux-foundation.org/msg19657.html\n> > \n> > Robin Murphy (5):\n> > iommu/arm-smmu-v3: Specialise CMD_SYNC handling\n> > iommu/arm-smmu-v3: Forget about cmdq-sync interrupt\n> > iommu/arm-smmu-v3: Use CMD_SYNC completion interrupt\n> > iommu/arm-smmu-v3: Poll for CMD_SYNC outside cmdq lock\n> > iommu/arm-smmu-v3: Use burst-polling for sync completion\n> \n> What's this final mythical patch about? I don't see it in the series.\n> \n> Anyway, the first two patches look fine to me, but this doesn't apply\n> on top of my iommu/devel branch so they will need rebasing.\n\nNo idea what I was doing wrong on Friday, but the first two patches apply\ncleanly now, so I'll push them out in a bit.\n\nSorry for the noise,\n\nWill","headers":{"Return-Path":"","X-Original-To":"incoming-imx@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming-imx@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=lists.infradead.org\n\t(client-ip=65.50.211.133; helo=bombadil.infradead.org;\n\tenvelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=lists.infradead.org\n\theader.i=@lists.infradead.org\n\theader.b=\"cN/+LUY8\"; dkim-atps=neutral"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n\t[65.50.211.133])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3yG1lJ1vHcz9t3w\n\tfor ;\n\tTue, 17 Oct 2017 02:02:36 +1100 (AEDT)","from localhost ([127.0.0.1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e46uW-0000aQ-Pt; Mon, 16 Oct 2017 15:02:32 +0000","from foss.arm.com ([217.140.101.70])\n\tby bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux))\n\tid 1e46uP-0000X3-00 for linux-arm-kernel@lists.infradead.org;\n\tMon, 16 Oct 2017 15:02:31 +0000","from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249])\n\tby usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 500AA1529;\n\tMon, 16 Oct 2017 08:02:04 -0700 (PDT)","from edgewater-inn.cambridge.arm.com\n\t(usa-sjc-imap-foss1.foss.arm.com [10.72.51.249])\n\tby usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id\n\t21AE13F483; Mon, 16 Oct 2017 08:02:04 -0700 (PDT)","by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000)\n\tid 28C391AE2F39; Mon, 16 Oct 2017 16:02:09 +0100 (BST)"],"DKIM-Signature":"v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20170209; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:\n\tMessage-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description:\n\tResent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=cIvAlnik8ZGX6/tbiJO7tSq7ASdGrs+HC/qeSdkfts8=;\n\tb=cN/+LUY8PjH9WD\n\tYcz7I0e9simXXfs0N9YiIGX9HvD/Vp8QgfLhxlYRi0buFnDYhfLEi8H4J2kgvUql3M6kCb2kOw/AH\n\tBUCsLA6nlTcdQrsErdOmu0QdjRfAhdRlByuGlW1Ofq0B6KyflhFIfyV6CypSLAQOiZKsoWz4stQUS\n\toulhQA8PGUySvSsTF5P7hq53rePTeOFyTjPGcAydCYXwZan2bdirOGWiKq/xZ6c85RrwuU/y4cLs3\n\tX3yMTmgLaIgw6deW48iZPwPR2ADc35TeQAKR1zV2EBZCoISoOyF4DqLPebw6myksqb2VrUl1vKqaU\n\tBoD3XD9mhkzSWTzjBy8A==;","Date":"Mon, 16 Oct 2017 16:02:09 +0100","From":"Will Deacon ","To":"Robin Murphy ","Subject":"Re: [PATCH v2 0/4] SMMUv3 CMD_SYNC optimisation","Message-ID":"<20171016150208.GA10500@arm.com>","References":"\n\t<20171013190521.GD30572@arm.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20171013190521.GD30572@arm.com>","User-Agent":"Mutt/1.5.23 (2014-03-12)","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20171016_080225_134985_EF5116AF ","X-CRM114-Status":"GOOD ( 14.59 )","X-Spam-Score":"-6.9 (------)","X-Spam-Report":"SpamAssassin version 3.4.1 on bombadil.infradead.org summary:\n\tContent analysis details: (-6.9 points)\n\tpts rule name description\n\t---- ----------------------\n\t--------------------------------------------------\n\t-5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/,\n\thigh trust [217.140.101.70 listed in list.dnswl.org]\n\t-0.0 SPF_PASS SPF: sender matches SPF record\n\t-0.0 RP_MATCHES_RCVD Envelope sender domain matches handover relay\n\tdomain\n\t-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n\t[score: 0.0000]","X-BeenThere":"linux-arm-kernel@lists.infradead.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"salil.mehta@huawei.com, jean-philippe.brucker@arm.com,\n\tiommu@lists.linux-foundation.org, nwatters@codeaurora.org,\n\tsgoutham@cavium.com, linux-arm-kernel@lists.infradead.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"linux-arm-kernel\" ","Errors-To":"linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org","List-Id":"linux-imx-kernel.lists.patchwork.ozlabs.org"}}]