[{"id":1915260,"web_url":"http://patchwork.ozlabs.org/comment/1915260/","msgid":"","list_archive_url":null,"date":"2018-05-17T17:23:33","subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","submitter":{"id":69187,"url":"http://patchwork.ozlabs.org/api/people/69187/","name":"Eric Auger","email":"eric.auger@redhat.com"},"content":"Hi Peter,\n\nOn 05/04/2018 05:08 AM, Peter Xu wrote:\n> IOMMU replay was carried out before in many use cases, e.g., context\n> cache invalidations, domain flushes. We used this mechanism to sync the\n> shadow page table by firstly (1) unmap the whole address space, then\n> (2) walk the page table to remap what's in the table.\n> \n> This is very dangerous.\n> \n> The problem is that we'll have a very small window (in my measurement,\n> it can be about 3ms) during above step (1) and (2) that the device will\n> see no (or incomplete) device page table. Howerver the device never\n> knows that. This can cause DMA error of devices, who assumes the page\n> table is always there.\nBut now you have the QemuMutex can we have a translate and an\ninvalidation that occur concurrently? Don't the iotlb flush and replay\nhappen while the lock is held?\n\nThanks\n\nEric\n> \n> So the point is that, for MAP typed notifiers (vfio-pci, for example)\n> they'll need the mapped page entries always be there. We can never\n> unmap any existing page entries like what we did in (1) above.\n> \n> The only solution is to remove step (1). We can't do that before since\n> we didn't know what device page was mapped and what was not, so we unmap\n> them all. Now with the new IOVA tree QEMU knows what has mapped and\n> what has not. We don't need this step (1) any more. Remove it.\n> \n> Note that after removing that global unmap flushing, we'll need to\n> notify unmap now during page walkings.\n> \n> This should fix the DMA error problem that Jintack Lim reported with\n> nested device assignment. This problem won't not happen always, e.g., I\n> cannot reproduce the error. However after collecting logs it shows that\n> this is the possible cause to Jintack's problem.\n> \n> Reported-by: Jintack Lim \n> Signed-off-by: Peter Xu \n> ---\n> hw/i386/intel_iommu.c | 8 ++++----\n> 1 file changed, 4 insertions(+), 4 deletions(-)\n> \n> diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c\n> index 16b3514afb..9439103cac 100644\n> --- a/hw/i386/intel_iommu.c\n> +++ b/hw/i386/intel_iommu.c\n> @@ -3003,10 +3003,8 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n)\n> \n> /*\n> * The replay can be triggered by either a invalidation or a newly\n> - * created entry. No matter what, we release existing mappings\n> - * (it means flushing caches for UNMAP-only registers).\n> + * created entry.\n> */\n> - vtd_address_space_unmap(vtd_as, n);\n> \n> if (vtd_dev_to_context_entry(s, bus_n, vtd_as->devfn, &ce) == 0) {\n> trace_vtd_replay_ce_valid(bus_n, PCI_SLOT(vtd_as->devfn),\n> @@ -3015,8 +3013,10 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n)\n> ce.hi, ce.lo);\n> if (vtd_as_notify_mappings(vtd_as)) {\n> /* This is required only for MAP typed notifiers */\n> - vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n> + vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, true,\n> vtd_as);\n> + } else {\n> + vtd_address_space_unmap(vtd_as, n);\n> }\n> } else {\n> trace_vtd_replay_ce_invalid(bus_n, PCI_SLOT(vtd_as->devfn),\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40mypQ11Kjz9s33\n\tfor ;\n\tFri, 18 May 2018 03:24:12 +1000 (AEST)","from localhost ([::1]:37969 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJMdN-000446-Aq\n\tfor incoming@patchwork.ozlabs.org; Thu, 17 May 2018 13:24:09 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:45663)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJMd2-00041R-98\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 13:23:51 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJMcx-0007Tm-GP\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 13:23:48 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:50504\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1fJMcx-0007T4-AD\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 13:23:43 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 14EEC79D36;\n\tThu, 17 May 2018 17:23:42 +0000 (UTC)","from localhost.localdomain (ovpn-117-111.ams2.redhat.com\n\t[10.36.117.111])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 67CBBD74CB;\n\tThu, 17 May 2018 17:23:34 +0000 (UTC)"],"To":"Peter Xu , qemu-devel@nongnu.org","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-10-peterx@redhat.com>","From":"Auger Eric ","Message-ID":"","Date":"Thu, 17 May 2018 19:23:33 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20180504030811.28111-10-peterx@redhat.com>","Content-Type":"text/plain; charset=windows-1252","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.11.54.5","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.2]);\n\tThu, 17 May 2018 17:23:42 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.2]); \n\tThu, 17 May 2018 17:23:42 +0000 (UTC) for IP:'10.11.54.5'\n\tDOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'eric.auger@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Jintack Lim , Tian Kevin ,\n\tAlex Williamson ,\n\tJason Wang , \"Michael S . Tsirkin\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1915641,"web_url":"http://patchwork.ozlabs.org/comment/1915641/","msgid":"<20180518060604.GG2569@xz-mi>","list_archive_url":null,"date":"2018-05-18T06:06:04","subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, May 17, 2018 at 07:23:33PM +0200, Auger Eric wrote:\n> Hi Peter,\n> \n> On 05/04/2018 05:08 AM, Peter Xu wrote:\n> > IOMMU replay was carried out before in many use cases, e.g., context\n> > cache invalidations, domain flushes. We used this mechanism to sync the\n> > shadow page table by firstly (1) unmap the whole address space, then\n> > (2) walk the page table to remap what's in the table.\n> > \n> > This is very dangerous.\n> > \n> > The problem is that we'll have a very small window (in my measurement,\n> > it can be about 3ms) during above step (1) and (2) that the device will\n> > see no (or incomplete) device page table. Howerver the device never\n> > knows that. This can cause DMA error of devices, who assumes the page\n> > table is always there.\n> But now you have the QemuMutex can we have a translate and an\n> invalidation that occur concurrently? Don't the iotlb flush and replay\n> happen while the lock is held?\n\nNote that when we are using vfio-pci devices we can't really know when\nthe device started a DMA since that's totally happening only between\nthe host IOMMU and the hardware. Say, vfio-pci device page\ntranslation is happening in the shadow page table, not really in QEMU.\nSo IMO we aren't protected by anything.\n\nAlso, this patch is dropped in version 3, and I did something else to\nachieve similar goal (by introducing the shadow page sync helper, and\nthen for DSIs we'll use that instead of calling IOMMU replay here).\nPlease have a look. Thanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40nHk823d2z9s37\n\tfor ;\n\tFri, 18 May 2018 16:06:38 +1000 (AEST)","from localhost ([::1]:36831 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJYXD-0005KK-6p\n\tfor incoming@patchwork.ozlabs.org; Fri, 18 May 2018 02:06:35 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:59823)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJYWt-0005KA-Pa\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 02:06:16 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJYWp-0000Ad-QD\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 02:06:15 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:35594\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1fJYWp-0000AP-LA\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 02:06:11 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 2BA7C81663C0;\n\tFri, 18 May 2018 06:06:11 +0000 (UTC)","from xz-mi (dhcp-14-151.nay.redhat.com [10.66.14.151])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 368242023582;\n\tFri, 18 May 2018 06:06:06 +0000 (UTC)"],"Date":"Fri, 18 May 2018 14:06:04 +0800","From":"Peter Xu ","To":"Auger Eric ","Message-ID":"<20180518060604.GG2569@xz-mi>","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-10-peterx@redhat.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.9.5 (2018-04-13)","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.4","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.8]);\n\tFri, 18 May 2018 06:06:11 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.8]); \n\tFri, 18 May 2018 06:06:11 +0000 (UTC) for IP:'10.11.54.4'\n\tDOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'peterx@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Tian Kevin , \"Michael S . Tsirkin\" ,\n\tJason Wang , qemu-devel@nongnu.org,\n\tAlex Williamson ,\n\tJintack Lim ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1915685,"web_url":"http://patchwork.ozlabs.org/comment/1915685/","msgid":"<84983cd6-39c2-3951-8017-6619fe1215fb@redhat.com>","list_archive_url":null,"date":"2018-05-18T07:31:03","subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","submitter":{"id":69187,"url":"http://patchwork.ozlabs.org/api/people/69187/","name":"Eric Auger","email":"eric.auger@redhat.com"},"content":"Hi Peter,\n\nOn 05/18/2018 08:06 AM, Peter Xu wrote:\n> On Thu, May 17, 2018 at 07:23:33PM +0200, Auger Eric wrote:\n>> Hi Peter,\n>>\n>> On 05/04/2018 05:08 AM, Peter Xu wrote:\n>>> IOMMU replay was carried out before in many use cases, e.g., context\n>>> cache invalidations, domain flushes. We used this mechanism to sync the\n>>> shadow page table by firstly (1) unmap the whole address space, then\n>>> (2) walk the page table to remap what's in the table.\n>>>\n>>> This is very dangerous.\n>>>\n>>> The problem is that we'll have a very small window (in my measurement,\n>>> it can be about 3ms) during above step (1) and (2) that the device will\n>>> see no (or incomplete) device page table. Howerver the device never\n>>> knows that. This can cause DMA error of devices, who assumes the page\n>>> table is always there.\n>> But now you have the QemuMutex can we have a translate and an\n>> invalidation that occur concurrently? Don't the iotlb flush and replay\n>> happen while the lock is held?\n> \n> Note that when we are using vfio-pci devices we can't really know when\n> the device started a DMA since that's totally happening only between\n> the host IOMMU and the hardware. \n\nOh yes that's fully relevant in vfio-pci use case. thank you for the\nclarification.\n\nSay, vfio-pci device page\n> translation is happening in the shadow page table, not really in QEMU.\n> So IMO we aren't protected by anything.\n> \n> Also, this patch is dropped in version 3, and I did something else to\n> achieve similar goal (by introducing the shadow page sync helper, and\n> then for DSIs we'll use that instead of calling IOMMU replay here).\n> Please have a look. Thanks,\n\nOK\n\nThanks\n\nEric\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40nKhX1zsNz9s0q\n\tfor ;\n\tFri, 18 May 2018 17:35:23 +1000 (AEST)","from localhost ([::1]:37096 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJZv6-0005kF-UJ\n\tfor incoming@patchwork.ozlabs.org; Fri, 18 May 2018 03:35:20 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:45221)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJZr4-0003UU-HE\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:31:11 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJZr1-0000GW-Dd\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:31:10 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:35864\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1fJZr1-0000GM-8Y\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:31:07 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id B87DE5BCB6;\n\tFri, 18 May 2018 07:31:06 +0000 (UTC)","from localhost.localdomain (ovpn-117-111.ams2.redhat.com\n\t[10.36.117.111])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 11F6C2166BAD;\n\tFri, 18 May 2018 07:31:04 +0000 (UTC)"],"To":"Peter Xu ","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-10-peterx@redhat.com>\n\t\n\t<20180518060604.GG2569@xz-mi>","From":"Auger Eric ","Message-ID":"<84983cd6-39c2-3951-8017-6619fe1215fb@redhat.com>","Date":"Fri, 18 May 2018 09:31:03 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20180518060604.GG2569@xz-mi>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.6","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.1]);\n\tFri, 18 May 2018 07:31:06 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.1]); \n\tFri, 18 May 2018 07:31:06 +0000 (UTC) for IP:'10.11.54.6'\n\tDOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'eric.auger@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 09/10] intel-iommu: don't unmap all for\n\tshadow page table","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Tian Kevin , \"Michael S . Tsirkin\" ,\n\tJason Wang , qemu-devel@nongnu.org,\n\tAlex Williamson ,\n\tJintack Lim ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]