{"id":1152983,"url":"http://patchwork.ozlabs.org/api/patches/1152983/?format=json","web_url":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20190826061705.92048-3-aik@ozlabs.ru/","project":{"id":2,"url":"http://patchwork.ozlabs.org/api/projects/2/?format=json","name":"Linux PPC development","link_name":"linuxppc-dev","list_id":"linuxppc-dev.lists.ozlabs.org","list_email":"linuxppc-dev@lists.ozlabs.org","web_url":"https://github.com/linuxppc/wiki/wiki","scm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git","webscm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/","list_archive_url":"https://lore.kernel.org/linuxppc-dev/","list_archive_url_format":"https://lore.kernel.org/linuxppc-dev/{}/","commit_url_format":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id={}"},"msgid":"<20190826061705.92048-3-aik@ozlabs.ru>","list_archive_url":"https://lore.kernel.org/linuxppc-dev/20190826061705.92048-3-aik@ozlabs.ru/","date":"2019-08-26T06:17:03","name":"[kernel,v2,2/4] KVM: PPC: Invalidate multiple TCEs at once","commit_ref":null,"pull_url":null,"state":"changes-requested","archived":false,"hash":"66a903c605a4593de21e578f009e3c3b2c8cfcec","submitter":{"id":7621,"url":"http://patchwork.ozlabs.org/api/people/7621/?format=json","name":"Alexey Kardashevskiy","email":"aik@ozlabs.ru"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20190826061705.92048-3-aik@ozlabs.ru/mbox/","series":[{"id":127248,"url":"http://patchwork.ozlabs.org/api/series/127248/?format=json","web_url":"http://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=127248","date":"2019-08-26T06:17:02","name":"powerpc/powernv/kvm: Invalidate multiple TCEs at once","version":2,"mbox":"http://patchwork.ozlabs.org/series/127248/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/1152983/comments/","check":"fail","checks":"http://patchwork.ozlabs.org/api/patches/1152983/checks/","tags":{},"related":[],"headers":{"Return-Path":"<linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org>","X-Original-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Delivered-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Received":["from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\tkey-exchange X25519 server-signature RSA-PSS (4096 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 46H25y40vnz9sMr\n\tfor <patchwork-incoming@ozlabs.org>;\n\tMon, 26 Aug 2019 16:24:22 +1000 (AEST)","from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 46H25y2rmXzDqJ8\n\tfor <patchwork-incoming@ozlabs.org>;\n\tMon, 26 Aug 2019 16:24:22 +1000 (AEST)","from ozlabs.ru (ozlabs.ru [107.173.13.209])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 46H1yS5zPjzDqb9\n\tfor <linuxppc-dev@lists.ozlabs.org>;\n\tMon, 26 Aug 2019 16:17:52 +1000 (AEST)","from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1])\n\tby ozlabs.ru (Postfix) with ESMTP id 0FE83AE80570;\n\tMon, 26 Aug 2019 02:16:58 -0400 (EDT)"],"Authentication-Results":["ozlabs.org;\n\tdmarc=none (p=none dis=none) header.from=ozlabs.ru","lists.ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=ozlabs.ru\n\t(client-ip=107.173.13.209; helo=ozlabs.ru; envelope-from=aik@ozlabs.ru;\n\treceiver=<UNKNOWN>)","lists.ozlabs.org;\n\tdmarc=none (p=none dis=none) header.from=ozlabs.ru"],"From":"Alexey Kardashevskiy <aik@ozlabs.ru>","To":"linuxppc-dev@lists.ozlabs.org","Subject":"[PATCH kernel v2 2/4] KVM: PPC: Invalidate multiple TCEs at once","Date":"Mon, 26 Aug 2019 16:17:03 +1000","Message-Id":"<20190826061705.92048-3-aik@ozlabs.ru>","X-Mailer":"git-send-email 2.17.1","In-Reply-To":"<20190826061705.92048-1-aik@ozlabs.ru>","References":"<20190826061705.92048-1-aik@ozlabs.ru>","X-BeenThere":"linuxppc-dev@lists.ozlabs.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"Linux on PowerPC Developers Mail List\n\t<linuxppc-dev.lists.ozlabs.org>","List-Unsubscribe":"<https://lists.ozlabs.org/options/linuxppc-dev>,\n\t<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=unsubscribe>","List-Archive":"<http://lists.ozlabs.org/pipermail/linuxppc-dev/>","List-Post":"<mailto:linuxppc-dev@lists.ozlabs.org>","List-Help":"<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=help>","List-Subscribe":"<https://lists.ozlabs.org/listinfo/linuxppc-dev>,\n\t<mailto:linuxppc-dev-request@lists.ozlabs.org?subject=subscribe>","Cc":"kvm@vger.kernel.org, Alexey Kardashevskiy <aik@ozlabs.ru>,\n\tkvm-ppc@vger.kernel.org, Alex Williamson <alex.williamson@redhat.com>,\n\tAlistair Popple <alistair@popple.id.au>,\n\tDavid Gibson <david@gibson.dropbear.id.au>","Errors-To":"linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org","Sender":"\"Linuxppc-dev\"\n\t<linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org>"},"content":"Invalidating a TCE cache entry for each updated TCE is quite expensive.\nThis makes use of the new iommu_table_ops::xchg_no_kill()/tce_kill()\ncallbacks to bring down the time spent in mapping a huge guest DMA window;\nroughly 20s to 10s for each guest's 100GB of DMA space.\n\nSigned-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>\n---\n arch/powerpc/kvm/book3s_64_vio.c    | 29 +++++++++++++++-------\n arch/powerpc/kvm/book3s_64_vio_hv.c | 38 +++++++++++++++++++++--------\n 2 files changed, 48 insertions(+), 19 deletions(-)","diff":"diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c\nindex c4b606fe73eb..5834db0a54c6 100644\n--- a/arch/powerpc/kvm/book3s_64_vio.c\n+++ b/arch/powerpc/kvm/book3s_64_vio.c\n@@ -416,7 +416,7 @@ static void kvmppc_clear_tce(struct mm_struct *mm, struct iommu_table *tbl,\n \tunsigned long hpa = 0;\n \tenum dma_data_direction dir = DMA_NONE;\n \n-\tiommu_tce_xchg(mm, tbl, entry, &hpa, &dir);\n+\tiommu_tce_xchg_no_kill(mm, tbl, entry, &hpa, &dir);\n }\n \n static long kvmppc_tce_iommu_mapped_dec(struct kvm *kvm,\n@@ -447,7 +447,8 @@ static long kvmppc_tce_iommu_do_unmap(struct kvm *kvm,\n \tunsigned long hpa = 0;\n \tlong ret;\n \n-\tif (WARN_ON_ONCE(iommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir)))\n+\tif (WARN_ON_ONCE(iommu_tce_xchg_no_kill(kvm->mm, tbl, entry, &hpa,\n+\t\t\t\t\t&dir)))\n \t\treturn H_TOO_HARD;\n \n \tif (dir == DMA_NONE)\n@@ -455,7 +456,7 @@ static long kvmppc_tce_iommu_do_unmap(struct kvm *kvm,\n \n \tret = kvmppc_tce_iommu_mapped_dec(kvm, tbl, entry);\n \tif (ret != H_SUCCESS)\n-\t\tiommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir);\n+\t\tiommu_tce_xchg_no_kill(kvm->mm, tbl, entry, &hpa, &dir);\n \n \treturn ret;\n }\n@@ -501,7 +502,7 @@ long kvmppc_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl,\n \tif (mm_iommu_mapped_inc(mem))\n \t\treturn H_TOO_HARD;\n \n-\tret = iommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir);\n+\tret = iommu_tce_xchg_no_kill(kvm->mm, tbl, entry, &hpa, &dir);\n \tif (WARN_ON_ONCE(ret)) {\n \t\tmm_iommu_mapped_dec(mem);\n \t\treturn H_TOO_HARD;\n@@ -579,6 +580,8 @@ long kvmppc_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,\n \t\t\tret = kvmppc_tce_iommu_map(vcpu->kvm, stt, stit->tbl,\n \t\t\t\t\tentry, ua, dir);\n \n+\t\tiommu_tce_kill(stit->tbl, entry, 1);\n+\n \t\tif (ret != H_SUCCESS) {\n \t\t\tkvmppc_clear_tce(vcpu->kvm->mm, stit->tbl, entry);\n \t\t\tgoto unlock_exit;\n@@ -656,13 +659,13 @@ long kvmppc_h_put_tce_indirect(struct kvm_vcpu *vcpu,\n \t\t */\n \t\tif (get_user(tce, tces + i)) {\n \t\t\tret = H_TOO_HARD;\n-\t\t\tgoto unlock_exit;\n+\t\t\tgoto invalidate_exit;\n \t\t}\n \t\ttce = be64_to_cpu(tce);\n \n \t\tif (kvmppc_tce_to_ua(vcpu->kvm, tce, &ua)) {\n \t\t\tret = H_PARAMETER;\n-\t\t\tgoto unlock_exit;\n+\t\t\tgoto invalidate_exit;\n \t\t}\n \n \t\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next) {\n@@ -673,13 +676,17 @@ long kvmppc_h_put_tce_indirect(struct kvm_vcpu *vcpu,\n \t\t\tif (ret != H_SUCCESS) {\n \t\t\t\tkvmppc_clear_tce(vcpu->kvm->mm, stit->tbl,\n \t\t\t\t\t\tentry);\n-\t\t\t\tgoto unlock_exit;\n+\t\t\t\tgoto invalidate_exit;\n \t\t\t}\n \t\t}\n \n \t\tkvmppc_tce_put(stt, entry + i, tce);\n \t}\n \n+invalidate_exit:\n+\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next)\n+\t\tiommu_tce_kill(stit->tbl, entry, npages);\n+\n unlock_exit:\n \tsrcu_read_unlock(&vcpu->kvm->srcu, idx);\n \n@@ -718,7 +725,7 @@ long kvmppc_h_stuff_tce(struct kvm_vcpu *vcpu,\n \t\t\t\tcontinue;\n \n \t\t\tif (ret == H_TOO_HARD)\n-\t\t\t\treturn ret;\n+\t\t\t\tgoto invalidate_exit;\n \n \t\t\tWARN_ON_ONCE(1);\n \t\t\tkvmppc_clear_tce(vcpu->kvm->mm, stit->tbl, entry);\n@@ -728,6 +735,10 @@ long kvmppc_h_stuff_tce(struct kvm_vcpu *vcpu,\n \tfor (i = 0; i < npages; ++i, ioba += (1ULL << stt->page_shift))\n \t\tkvmppc_tce_put(stt, ioba >> stt->page_shift, tce_value);\n \n-\treturn H_SUCCESS;\n+invalidate_exit:\n+\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next)\n+\t\tiommu_tce_kill(stit->tbl, ioba >> stt->page_shift, npages);\n+\n+\treturn ret;\n }\n EXPORT_SYMBOL_GPL(kvmppc_h_stuff_tce);\ndiff --git a/arch/powerpc/kvm/book3s_64_vio_hv.c b/arch/powerpc/kvm/book3s_64_vio_hv.c\nindex b4f20f13b860..ab6eeb8e753e 100644\n--- a/arch/powerpc/kvm/book3s_64_vio_hv.c\n+++ b/arch/powerpc/kvm/book3s_64_vio_hv.c\n@@ -218,13 +218,14 @@ static long kvmppc_rm_ioba_validate(struct kvmppc_spapr_tce_table *stt,\n \treturn H_SUCCESS;\n }\n \n-static long iommu_tce_xchg_rm(struct mm_struct *mm, struct iommu_table *tbl,\n+static long iommu_tce_xchg_no_kill_rm(struct mm_struct *mm,\n+\t\tstruct iommu_table *tbl,\n \t\tunsigned long entry, unsigned long *hpa,\n \t\tenum dma_data_direction *direction)\n {\n \tlong ret;\n \n-\tret = tbl->it_ops->exchange_rm(tbl, entry, hpa, direction);\n+\tret = tbl->it_ops->xchg_no_kill(tbl, entry, hpa, direction, true);\n \n \tif (!ret && ((*direction == DMA_FROM_DEVICE) ||\n \t\t\t\t(*direction == DMA_BIDIRECTIONAL))) {\n@@ -240,13 +241,20 @@ static long iommu_tce_xchg_rm(struct mm_struct *mm, struct iommu_table *tbl,\n \treturn ret;\n }\n \n+extern void iommu_tce_kill_rm(struct iommu_table *tbl,\n+\t\tunsigned long entry, unsigned long pages)\n+{\n+\tif (tbl->it_ops->tce_kill)\n+\t\ttbl->it_ops->tce_kill(tbl, entry, pages, true);\n+}\n+\n static void kvmppc_rm_clear_tce(struct kvm *kvm, struct iommu_table *tbl,\n \t\tunsigned long entry)\n {\n \tunsigned long hpa = 0;\n \tenum dma_data_direction dir = DMA_NONE;\n \n-\tiommu_tce_xchg_rm(kvm->mm, tbl, entry, &hpa, &dir);\n+\tiommu_tce_xchg_no_kill_rm(kvm->mm, tbl, entry, &hpa, &dir);\n }\n \n static long kvmppc_rm_tce_iommu_mapped_dec(struct kvm *kvm,\n@@ -278,7 +286,7 @@ static long kvmppc_rm_tce_iommu_do_unmap(struct kvm *kvm,\n \tunsigned long hpa = 0;\n \tlong ret;\n \n-\tif (iommu_tce_xchg_rm(kvm->mm, tbl, entry, &hpa, &dir))\n+\tif (iommu_tce_xchg_no_kill_rm(kvm->mm, tbl, entry, &hpa, &dir))\n \t\t/*\n \t\t * real mode xchg can fail if struct page crosses\n \t\t * a page boundary\n@@ -290,7 +298,7 @@ static long kvmppc_rm_tce_iommu_do_unmap(struct kvm *kvm,\n \n \tret = kvmppc_rm_tce_iommu_mapped_dec(kvm, tbl, entry);\n \tif (ret)\n-\t\tiommu_tce_xchg_rm(kvm->mm, tbl, entry, &hpa, &dir);\n+\t\tiommu_tce_xchg_no_kill_rm(kvm->mm, tbl, entry, &hpa, &dir);\n \n \treturn ret;\n }\n@@ -336,7 +344,7 @@ static long kvmppc_rm_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl,\n \tif (WARN_ON_ONCE_RM(mm_iommu_mapped_inc(mem)))\n \t\treturn H_TOO_HARD;\n \n-\tret = iommu_tce_xchg_rm(kvm->mm, tbl, entry, &hpa, &dir);\n+\tret = iommu_tce_xchg_no_kill_rm(kvm->mm, tbl, entry, &hpa, &dir);\n \tif (ret) {\n \t\tmm_iommu_mapped_dec(mem);\n \t\t/*\n@@ -417,6 +425,8 @@ long kvmppc_rm_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,\n \t\t\tret = kvmppc_rm_tce_iommu_map(vcpu->kvm, stt,\n \t\t\t\t\tstit->tbl, entry, ua, dir);\n \n+\t\tiommu_tce_kill_rm(stit->tbl, entry, 1);\n+\n \t\tif (ret != H_SUCCESS) {\n \t\t\tkvmppc_rm_clear_tce(vcpu->kvm, stit->tbl, entry);\n \t\t\treturn ret;\n@@ -558,7 +568,7 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,\n \t\tua = 0;\n \t\tif (kvmppc_rm_tce_to_ua(vcpu->kvm, tce, &ua, NULL)) {\n \t\t\tret = H_PARAMETER;\n-\t\t\tgoto unlock_exit;\n+\t\t\tgoto invalidate_exit;\n \t\t}\n \n \t\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next) {\n@@ -569,13 +579,17 @@ long kvmppc_rm_h_put_tce_indirect(struct kvm_vcpu *vcpu,\n \t\t\tif (ret != H_SUCCESS) {\n \t\t\t\tkvmppc_rm_clear_tce(vcpu->kvm, stit->tbl,\n \t\t\t\t\t\tentry);\n-\t\t\t\tgoto unlock_exit;\n+\t\t\t\tgoto invalidate_exit;\n \t\t\t}\n \t\t}\n \n \t\tkvmppc_rm_tce_put(stt, entry + i, tce);\n \t}\n \n+invalidate_exit:\n+\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next)\n+\t\tiommu_tce_kill_rm(stit->tbl, entry, npages);\n+\n unlock_exit:\n \tif (rmap)\n \t\tunlock_rmap(rmap);\n@@ -618,7 +632,7 @@ long kvmppc_rm_h_stuff_tce(struct kvm_vcpu *vcpu,\n \t\t\t\tcontinue;\n \n \t\t\tif (ret == H_TOO_HARD)\n-\t\t\t\treturn ret;\n+\t\t\t\tgoto invalidate_exit;\n \n \t\t\tWARN_ON_ONCE_RM(1);\n \t\t\tkvmppc_rm_clear_tce(vcpu->kvm, stit->tbl, entry);\n@@ -628,7 +642,11 @@ long kvmppc_rm_h_stuff_tce(struct kvm_vcpu *vcpu,\n \tfor (i = 0; i < npages; ++i, ioba += (1ULL << stt->page_shift))\n \t\tkvmppc_rm_tce_put(stt, ioba >> stt->page_shift, tce_value);\n \n-\treturn H_SUCCESS;\n+invalidate_exit:\n+\tlist_for_each_entry_lockless(stit, &stt->iommu_tables, next)\n+\t\tiommu_tce_kill_rm(stit->tbl, ioba >> stt->page_shift, npages);\n+\n+\treturn ret;\n }\n \n /* This can be called in either virtual mode or real mode */\n","prefixes":["kernel","v2","2/4"]}