[{"id":1915006,"web_url":"http://patchwork.ozlabs.org/comment/1915006/","msgid":"<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>","list_archive_url":null,"date":"2018-05-17T13:39:50","subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","submitter":{"id":69187,"url":"http://patchwork.ozlabs.org/api/people/69187/","name":"Eric Auger","email":"eric.auger@redhat.com"},"content":"Hi Peter,\n\nOn 05/04/2018 05:08 AM, Peter Xu wrote:\n> For UNMAP-only IOMMU notifiers, we don't really need to walk the page\ns/really// ;-)\n> tables. Fasten that procedure by skipping the page table walk. That\n> should boost performance for UNMAP-only notifiers like vhost.\n> \n> Signed-off-by: Peter Xu \n> ---\n> include/hw/i386/intel_iommu.h | 2 ++\n> hw/i386/intel_iommu.c | 43 +++++++++++++++++++++++++++++++----\n> 2 files changed, 40 insertions(+), 5 deletions(-)\n> \n> diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h\n> index ee517704e7..9e0a6c1c6a 100644\n> --- a/include/hw/i386/intel_iommu.h\n> +++ b/include/hw/i386/intel_iommu.h\n> @@ -93,6 +93,8 @@ struct VTDAddressSpace {\n> IntelIOMMUState *iommu_state;\n> VTDContextCacheEntry context_cache_entry;\n> QLIST_ENTRY(VTDAddressSpace) next;\n> + /* Superset of notifier flags that this address space has */\n> + IOMMUNotifierFlag notifier_flags;\n> };\n> \n> struct VTDBus {\n> diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c\n> index 112971638d..9a418abfb6 100644\n> --- a/hw/i386/intel_iommu.c\n> +++ b/hw/i386/intel_iommu.c\n> @@ -138,6 +138,12 @@ static inline void vtd_iommu_unlock(IntelIOMMUState *s)\n> qemu_mutex_unlock(&s->iommu_lock);\n> }\n> \n> +/* Whether the address space needs to notify new mappings */\n> +static inline gboolean vtd_as_notify_mappings(VTDAddressSpace *as)\nwould suggest vtd_as_has_map_notifier()? But tastes & colours ;-)\n> +{\n> + return as->notifier_flags & IOMMU_NOTIFIER_MAP;\n> +}\n> +\n> /* GHashTable functions */\n> static gboolean vtd_uint64_equal(gconstpointer v1, gconstpointer v2)\n> {\n> @@ -1433,14 +1439,35 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s,\n> VTDAddressSpace *vtd_as;\n> VTDContextEntry ce;\n> int ret;\n> + hwaddr size = (1 << am) * VTD_PAGE_SIZE;\n> \n> QLIST_FOREACH(vtd_as, &(s->notifiers_list), next) {\n> ret = vtd_dev_to_context_entry(s, pci_bus_num(vtd_as->bus),\n> vtd_as->devfn, &ce);\n> if (!ret && domain_id == VTD_CONTEXT_ENTRY_DID(ce.hi)) {\n> - vtd_page_walk(&ce, addr, addr + (1 << am) * VTD_PAGE_SIZE,\n> - vtd_page_invalidate_notify_hook,\n> - (void *)&vtd_as->iommu, true, s->aw_bits);\n> + if (vtd_as_notify_mappings(vtd_as)) {\n> + /*\n> + * For MAP-inclusive notifiers, we need to walk the\n> + * page table to sync the shadow page table.\n> + */\nPotentially we may have several notifiers attached to the IOMMU MR ~\nvtd_as, each of them having different flags. Those flags are OR'ed in\nmemory_region_update_iommu_notify_flags and this is the one you now\nstore in the vtd_as. So maybe your comment may rather state:\nas soon as we have at least one MAP notifier, we need to do the PTW?\n\nnit: not related to this patch: vtd_page_walk kerneldoc comments misses\n@notify_unmap param comment\nside note: the name of the hook is a bit misleading as it suggests we\ninvalidate the entry, whereas we update any valid entry and invalidate\nstale ones (if notify_unmap=true)?\n> + vtd_page_walk(&ce, addr, addr + size,\n> + vtd_page_invalidate_notify_hook,\n> + (void *)&vtd_as->iommu, true, s->aw_bits);\n> + } else {\n> + /*\n> + * For UNMAP-only notifiers, we don't need to walk the\n> + * page tables. We just deliver the PSI down to\n> + * invalidate caches.\n\nWe just unmap the range?\n> + */\n> + IOMMUTLBEntry entry = {\n> + .target_as = &address_space_memory,\n> + .iova = addr,\n> + .translated_addr = 0,\n> + .addr_mask = size - 1,\n> + .perm = IOMMU_NONE,\n> + };\n> + memory_region_notify_iommu(&vtd_as->iommu, entry);\n> + }\n> }\n> }\n> }\n> @@ -2380,6 +2407,9 @@ static void vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,\n> exit(1);\n> }\n> \n> + /* Update per-address-space notifier flags */\n> + vtd_as->notifier_flags = new;\n> +\n> if (old == IOMMU_NOTIFIER_NONE) {\n> /* Insert new ones */\n> QLIST_INSERT_HEAD(&s->notifiers_list, vtd_as, next);\n> @@ -2890,8 +2920,11 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n)\n> PCI_FUNC(vtd_as->devfn),\n> VTD_CONTEXT_ENTRY_DID(ce.hi),\n> ce.hi, ce.lo);\n> - vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n> - s->aw_bits);\n> + if (vtd_as_notify_mappings(vtd_as)) {\n> + /* This is required only for MAP typed notifiers */\n> + vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n> + s->aw_bits);\n> + }\n> } else {\n> trace_vtd_replay_ce_invalid(bus_n, PCI_SLOT(vtd_as->devfn),\n> PCI_FUNC(vtd_as->devfn));\n> \nA worthwhile improvement indeed!\n\nThanks\n\nEric","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40msrP62H6z9s0x\n\tfor ;\n\tThu, 17 May 2018 23:40:36 +1000 (AEST)","from localhost ([::1]:56400 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJJ8v-0000bi-R3\n\tfor incoming@patchwork.ozlabs.org; Thu, 17 May 2018 09:40:29 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:48237)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJJ8V-0000ad-8t\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 09:40:09 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJJ8S-0002xG-54\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 09:40:03 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:60472\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1fJJ8S-0002uj-0C\n\tfor qemu-devel@nongnu.org; Thu, 17 May 2018 09:40:00 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 97DEF40200A4;\n\tThu, 17 May 2018 13:39:58 +0000 (UTC)","from localhost.localdomain (ovpn-117-111.ams2.redhat.com\n\t[10.36.117.111])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 68C7110B2B4E;\n\tThu, 17 May 2018 13:39:51 +0000 (UTC)"],"To":"Peter Xu , qemu-devel@nongnu.org","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-5-peterx@redhat.com>","From":"Auger Eric ","Message-ID":"<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>","Date":"Thu, 17 May 2018 15:39:50 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20180504030811.28111-5-peterx@redhat.com>","Content-Type":"text/plain; charset=windows-1252","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.3","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.6]);\n\tThu, 17 May 2018 13:39:58 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.6]); \n\tThu, 17 May 2018 13:39:58 +0000 (UTC) for IP:'10.11.54.3'\n\tDOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'eric.auger@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Jintack Lim , Tian Kevin ,\n\tAlex Williamson ,\n\tJason Wang , \"Michael S . Tsirkin\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1915634,"web_url":"http://patchwork.ozlabs.org/comment/1915634/","msgid":"<20180518055307.GD2569@xz-mi>","list_archive_url":null,"date":"2018-05-18T05:53:07","subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, May 17, 2018 at 03:39:50PM +0200, Auger Eric wrote:\n> Hi Peter,\n> \n> On 05/04/2018 05:08 AM, Peter Xu wrote:\n> > For UNMAP-only IOMMU notifiers, we don't really need to walk the page\n> s/really// ;-)\n\nOk.\n\n> > tables. Fasten that procedure by skipping the page table walk. That\n> > should boost performance for UNMAP-only notifiers like vhost.\n> > \n> > Signed-off-by: Peter Xu \n> > ---\n> > include/hw/i386/intel_iommu.h | 2 ++\n> > hw/i386/intel_iommu.c | 43 +++++++++++++++++++++++++++++++----\n> > 2 files changed, 40 insertions(+), 5 deletions(-)\n> > \n> > diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h\n> > index ee517704e7..9e0a6c1c6a 100644\n> > --- a/include/hw/i386/intel_iommu.h\n> > +++ b/include/hw/i386/intel_iommu.h\n> > @@ -93,6 +93,8 @@ struct VTDAddressSpace {\n> > IntelIOMMUState *iommu_state;\n> > VTDContextCacheEntry context_cache_entry;\n> > QLIST_ENTRY(VTDAddressSpace) next;\n> > + /* Superset of notifier flags that this address space has */\n> > + IOMMUNotifierFlag notifier_flags;\n> > };\n> > \n> > struct VTDBus {\n> > diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c\n> > index 112971638d..9a418abfb6 100644\n> > --- a/hw/i386/intel_iommu.c\n> > +++ b/hw/i386/intel_iommu.c\n> > @@ -138,6 +138,12 @@ static inline void vtd_iommu_unlock(IntelIOMMUState *s)\n> > qemu_mutex_unlock(&s->iommu_lock);\n> > }\n> > \n> > +/* Whether the address space needs to notify new mappings */\n> > +static inline gboolean vtd_as_notify_mappings(VTDAddressSpace *as)\n> would suggest vtd_as_has_map_notifier()? But tastes & colours ;-)\n\nYeah it is. But okay, I can switch to that especially it's only used\nin this patch and it's new.\n\n> > +{\n> > + return as->notifier_flags & IOMMU_NOTIFIER_MAP;\n> > +}\n> > +\n> > /* GHashTable functions */\n> > static gboolean vtd_uint64_equal(gconstpointer v1, gconstpointer v2)\n> > {\n> > @@ -1433,14 +1439,35 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s,\n> > VTDAddressSpace *vtd_as;\n> > VTDContextEntry ce;\n> > int ret;\n> > + hwaddr size = (1 << am) * VTD_PAGE_SIZE;\n> > \n> > QLIST_FOREACH(vtd_as, &(s->notifiers_list), next) {\n> > ret = vtd_dev_to_context_entry(s, pci_bus_num(vtd_as->bus),\n> > vtd_as->devfn, &ce);\n> > if (!ret && domain_id == VTD_CONTEXT_ENTRY_DID(ce.hi)) {\n> > - vtd_page_walk(&ce, addr, addr + (1 << am) * VTD_PAGE_SIZE,\n> > - vtd_page_invalidate_notify_hook,\n> > - (void *)&vtd_as->iommu, true, s->aw_bits);\n> > + if (vtd_as_notify_mappings(vtd_as)) {\n> > + /*\n> > + * For MAP-inclusive notifiers, we need to walk the\n> > + * page table to sync the shadow page table.\n> > + */\n> Potentially we may have several notifiers attached to the IOMMU MR ~\n> vtd_as, each of them having different flags. Those flags are OR'ed in\n> memory_region_update_iommu_notify_flags and this is the one you now\n> store in the vtd_as. So maybe your comment may rather state:\n> as soon as we have at least one MAP notifier, we need to do the PTW?\n\nActually this is not 100% clear too, since all the \"MAP notifiers\" are\nactually both MAP+UNMAP notifiers... Maybe:\n\n As long as we have MAP notifications registered in any of our IOMMU\n notifiers, we need to sync the shadow page table.\n\n> \n> nit: not related to this patch: vtd_page_walk kerneldoc comments misses\n> @notify_unmap param comment\n> side note: the name of the hook is a bit misleading as it suggests we\n> invalidate the entry, whereas we update any valid entry and invalidate\n> stale ones (if notify_unmap=true)?\n> > + vtd_page_walk(&ce, addr, addr + size,\n> > + vtd_page_invalidate_notify_hook,\n> > + (void *)&vtd_as->iommu, true, s->aw_bits);\n> > + } else {\n> > + /*\n> > + * For UNMAP-only notifiers, we don't need to walk the\n> > + * page tables. We just deliver the PSI down to\n> > + * invalidate caches.\n> \n> We just unmap the range?\n\nIsn't it the same thing? :)\n\nIf to be explicit, here we know we only registered UNMAP\nnotifications, it's not really \"unmap\", it's really cache\ninvalidations only.\n\n> > + */\n> > + IOMMUTLBEntry entry = {\n> > + .target_as = &address_space_memory,\n> > + .iova = addr,\n> > + .translated_addr = 0,\n> > + .addr_mask = size - 1,\n> > + .perm = IOMMU_NONE,\n> > + };\n> > + memory_region_notify_iommu(&vtd_as->iommu, entry);\n> > + }\n> > }\n> > }\n> > }\n> > @@ -2380,6 +2407,9 @@ static void vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,\n> > exit(1);\n> > }\n> > \n> > + /* Update per-address-space notifier flags */\n> > + vtd_as->notifier_flags = new;\n> > +\n> > if (old == IOMMU_NOTIFIER_NONE) {\n> > /* Insert new ones */\n> > QLIST_INSERT_HEAD(&s->notifiers_list, vtd_as, next);\n> > @@ -2890,8 +2920,11 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n)\n> > PCI_FUNC(vtd_as->devfn),\n> > VTD_CONTEXT_ENTRY_DID(ce.hi),\n> > ce.hi, ce.lo);\n> > - vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n> > - s->aw_bits);\n> > + if (vtd_as_notify_mappings(vtd_as)) {\n> > + /* This is required only for MAP typed notifiers */\n> > + vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n> > + s->aw_bits);\n> > + }\n> > } else {\n> > trace_vtd_replay_ce_invalid(bus_n, PCI_SLOT(vtd_as->devfn),\n> > PCI_FUNC(vtd_as->devfn));\n> > \n> A worthwhile improvement indeed!\n\nI hope so. :) Thanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40nHRQ5y68z9s33\n\tfor ;\n\tFri, 18 May 2018 15:53:53 +1000 (AEST)","from localhost ([::1]:36785 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJYKp-0000J6-Qd\n\tfor incoming@patchwork.ozlabs.org; Fri, 18 May 2018 01:53:47 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:58027)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJYKT-0000J0-Js\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 01:53:26 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJYKO-0000MR-Nb\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 01:53:25 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:35266\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1fJYKO-0000M3-Hm\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 01:53:20 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id BA9B581FE15D;\n\tFri, 18 May 2018 05:53:19 +0000 (UTC)","from xz-mi (dhcp-14-151.nay.redhat.com [10.66.14.151])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 7C0F011166F4;\n\tFri, 18 May 2018 05:53:09 +0000 (UTC)"],"Date":"Fri, 18 May 2018 13:53:07 +0800","From":"Peter Xu ","To":"Auger Eric ","Message-ID":"<20180518055307.GD2569@xz-mi>","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-5-peterx@redhat.com>\n\t<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>","User-Agent":"Mutt/1.9.5 (2018-04-13)","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.3","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.8]);\n\tFri, 18 May 2018 05:53:19 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.8]); \n\tFri, 18 May 2018 05:53:19 +0000 (UTC) for IP:'10.11.54.3'\n\tDOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'peterx@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Tian Kevin , \"Michael S . Tsirkin\" ,\n\tJason Wang , qemu-devel@nongnu.org,\n\tAlex Williamson ,\n\tJintack Lim ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1915686,"web_url":"http://patchwork.ozlabs.org/comment/1915686/","msgid":"","list_archive_url":null,"date":"2018-05-18T07:38:07","subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","submitter":{"id":69187,"url":"http://patchwork.ozlabs.org/api/people/69187/","name":"Eric Auger","email":"eric.auger@redhat.com"},"content":"Hi Peter,\n\nOn 05/18/2018 07:53 AM, Peter Xu wrote:\n> On Thu, May 17, 2018 at 03:39:50PM +0200, Auger Eric wrote:\n>> Hi Peter,\n>>\n>> On 05/04/2018 05:08 AM, Peter Xu wrote:\n>>> For UNMAP-only IOMMU notifiers, we don't really need to walk the page\n>> s/really// ;-)\n> \n> Ok.\n> \n>>> tables. Fasten that procedure by skipping the page table walk. That\n>>> should boost performance for UNMAP-only notifiers like vhost.\n>>>\n>>> Signed-off-by: Peter Xu \n>>> ---\n>>> include/hw/i386/intel_iommu.h | 2 ++\n>>> hw/i386/intel_iommu.c | 43 +++++++++++++++++++++++++++++++----\n>>> 2 files changed, 40 insertions(+), 5 deletions(-)\n>>>\n>>> diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h\n>>> index ee517704e7..9e0a6c1c6a 100644\n>>> --- a/include/hw/i386/intel_iommu.h\n>>> +++ b/include/hw/i386/intel_iommu.h\n>>> @@ -93,6 +93,8 @@ struct VTDAddressSpace {\n>>> IntelIOMMUState *iommu_state;\n>>> VTDContextCacheEntry context_cache_entry;\n>>> QLIST_ENTRY(VTDAddressSpace) next;\n>>> + /* Superset of notifier flags that this address space has */\n>>> + IOMMUNotifierFlag notifier_flags;\n>>> };\n>>> \n>>> struct VTDBus {\n>>> diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c\n>>> index 112971638d..9a418abfb6 100644\n>>> --- a/hw/i386/intel_iommu.c\n>>> +++ b/hw/i386/intel_iommu.c\n>>> @@ -138,6 +138,12 @@ static inline void vtd_iommu_unlock(IntelIOMMUState *s)\n>>> qemu_mutex_unlock(&s->iommu_lock);\n>>> }\n>>> \n>>> +/* Whether the address space needs to notify new mappings */\n>>> +static inline gboolean vtd_as_notify_mappings(VTDAddressSpace *as)\n>> would suggest vtd_as_has_map_notifier()? But tastes & colours ;-)\n> \n> Yeah it is. But okay, I can switch to that especially it's only used\n> in this patch and it's new.\n> \n>>> +{\n>>> + return as->notifier_flags & IOMMU_NOTIFIER_MAP;\n>>> +}\n>>> +\n>>> /* GHashTable functions */\n>>> static gboolean vtd_uint64_equal(gconstpointer v1, gconstpointer v2)\n>>> {\n>>> @@ -1433,14 +1439,35 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s,\n>>> VTDAddressSpace *vtd_as;\n>>> VTDContextEntry ce;\n>>> int ret;\n>>> + hwaddr size = (1 << am) * VTD_PAGE_SIZE;\n>>> \n>>> QLIST_FOREACH(vtd_as, &(s->notifiers_list), next) {\n>>> ret = vtd_dev_to_context_entry(s, pci_bus_num(vtd_as->bus),\n>>> vtd_as->devfn, &ce);\n>>> if (!ret && domain_id == VTD_CONTEXT_ENTRY_DID(ce.hi)) {\n>>> - vtd_page_walk(&ce, addr, addr + (1 << am) * VTD_PAGE_SIZE,\n>>> - vtd_page_invalidate_notify_hook,\n>>> - (void *)&vtd_as->iommu, true, s->aw_bits);\n>>> + if (vtd_as_notify_mappings(vtd_as)) {\n>>> + /*\n>>> + * For MAP-inclusive notifiers, we need to walk the\n>>> + * page table to sync the shadow page table.\n>>> + */\n>> Potentially we may have several notifiers attached to the IOMMU MR ~\n>> vtd_as, each of them having different flags. Those flags are OR'ed in\n>> memory_region_update_iommu_notify_flags and this is the one you now\n>> store in the vtd_as. So maybe your comment may rather state:\n>> as soon as we have at least one MAP notifier, we need to do the PTW?\n> \n> Actually this is not 100% clear too, since all the \"MAP notifiers\" are\n> actually both MAP+UNMAP notifiers... Maybe:\n\nCan't IOMMU_NOTIFIER_MAP flag value be used without\nIOMMU_NOTIFIER_UNMAP? I don't see such restriction in the\nmemory_region_register_iommu_notifier API.\n> \n> As long as we have MAP notifications registered in any of our IOMMU\n> notifiers, we need to sync the shadow page table.\n> \n>>\n>> nit: not related to this patch: vtd_page_walk kerneldoc comments misses\n>> @notify_unmap param comment\n>> side note: the name of the hook is a bit misleading as it suggests we\n>> invalidate the entry, whereas we update any valid entry and invalidate\n>> stale ones (if notify_unmap=true)?\n>>> + vtd_page_walk(&ce, addr, addr + size,\n>>> + vtd_page_invalidate_notify_hook,\n>>> + (void *)&vtd_as->iommu, true, s->aw_bits);\n>>> + } else {\n>>> + /*\n>>> + * For UNMAP-only notifiers, we don't need to walk the\n>>> + * page tables. We just deliver the PSI down to\n>>> + * invalidate caches.\n>>\n>> We just unmap the range?\n> \n> Isn't it the same thing? :)\n> \n> If to be explicit, here we know we only registered UNMAP\n> notifications, it's not really \"unmap\", it's really cache\n> invalidations only.\nyes you're right I meant We just invalidate the range in cache. The\nsentence \"We just deliver the PSI down to invalidate caches.\" was not\ncrystal clear to me at first reading.\n\nThanks\n\nEric\n> \n>>> + */\n>>> + IOMMUTLBEntry entry = {\n>>> + .target_as = &address_space_memory,\n>>> + .iova = addr,\n>>> + .translated_addr = 0,\n>>> + .addr_mask = size - 1,\n>>> + .perm = IOMMU_NONE,\n>>> + };\n>>> + memory_region_notify_iommu(&vtd_as->iommu, entry);\n>>> + }\n>>> }\n>>> }\n>>> }\n>>> @@ -2380,6 +2407,9 @@ static void vtd_iommu_notify_flag_changed(IOMMUMemoryRegion *iommu,\n>>> exit(1);\n>>> }\n>>> \n>>> + /* Update per-address-space notifier flags */\n>>> + vtd_as->notifier_flags = new;\n>>> +\n>>> if (old == IOMMU_NOTIFIER_NONE) {\n>>> /* Insert new ones */\n>>> QLIST_INSERT_HEAD(&s->notifiers_list, vtd_as, next);\n>>> @@ -2890,8 +2920,11 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iommu_mr, IOMMUNotifier *n)\n>>> PCI_FUNC(vtd_as->devfn),\n>>> VTD_CONTEXT_ENTRY_DID(ce.hi),\n>>> ce.hi, ce.lo);\n>>> - vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n>>> - s->aw_bits);\n>>> + if (vtd_as_notify_mappings(vtd_as)) {\n>>> + /* This is required only for MAP typed notifiers */\n>>> + vtd_page_walk(&ce, 0, ~0ULL, vtd_replay_hook, (void *)n, false,\n>>> + s->aw_bits);\n>>> + }\n>>> } else {\n>>> trace_vtd_replay_ce_invalid(bus_n, PCI_SLOT(vtd_as->devfn),\n>>> PCI_FUNC(vtd_as->devfn));\n>>>\n>> A worthwhile improvement indeed!\n> \n> I hope so. :) Thanks,\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40nKmF0Bq2z9s2k\n\tfor ;\n\tFri, 18 May 2018 17:38:37 +1000 (AEST)","from localhost ([::1]:37115 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJZyE-0008LQ-OG\n\tfor incoming@patchwork.ozlabs.org; Fri, 18 May 2018 03:38:34 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:46659)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJZxu-0008KM-Gx\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:38:15 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJZxr-0003kN-EK\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:38:14 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:42758\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from )\n\tid 1fJZxr-0003k6-7X\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 03:38:11 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 37B03401EF18;\n\tFri, 18 May 2018 07:38:10 +0000 (UTC)","from localhost.localdomain (ovpn-117-111.ams2.redhat.com\n\t[10.36.117.111])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 8A7792166BAD;\n\tFri, 18 May 2018 07:38:08 +0000 (UTC)"],"To":"Peter Xu ","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-5-peterx@redhat.com>\n\t<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>\n\t<20180518055307.GD2569@xz-mi>","From":"Auger Eric ","Message-ID":"","Date":"Fri, 18 May 2018 09:38:07 +0200","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20180518055307.GD2569@xz-mi>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.6","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.5]);\n\tFri, 18 May 2018 07:38:10 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.5]); \n\tFri, 18 May 2018 07:38:10 +0000 (UTC) for IP:'10.11.54.6'\n\tDOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'eric.auger@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Tian Kevin , \"Michael S . Tsirkin\" ,\n\tJason Wang , qemu-devel@nongnu.org,\n\tAlex Williamson ,\n\tJintack Lim ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1915823,"web_url":"http://patchwork.ozlabs.org/comment/1915823/","msgid":"<20180518100249.GM2569@xz-mi>","list_archive_url":null,"date":"2018-05-18T10:02:49","subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, May 18, 2018 at 09:38:07AM +0200, Auger Eric wrote:\n> Hi Peter,\n> \n> On 05/18/2018 07:53 AM, Peter Xu wrote:\n> > On Thu, May 17, 2018 at 03:39:50PM +0200, Auger Eric wrote:\n> >> Hi Peter,\n> >>\n> >> On 05/04/2018 05:08 AM, Peter Xu wrote:\n> >>> For UNMAP-only IOMMU notifiers, we don't really need to walk the page\n> >> s/really// ;-)\n> > \n> > Ok.\n> > \n> >>> tables. Fasten that procedure by skipping the page table walk. That\n> >>> should boost performance for UNMAP-only notifiers like vhost.\n> >>>\n> >>> Signed-off-by: Peter Xu \n> >>> ---\n> >>> include/hw/i386/intel_iommu.h | 2 ++\n> >>> hw/i386/intel_iommu.c | 43 +++++++++++++++++++++++++++++++----\n> >>> 2 files changed, 40 insertions(+), 5 deletions(-)\n> >>>\n> >>> diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h\n> >>> index ee517704e7..9e0a6c1c6a 100644\n> >>> --- a/include/hw/i386/intel_iommu.h\n> >>> +++ b/include/hw/i386/intel_iommu.h\n> >>> @@ -93,6 +93,8 @@ struct VTDAddressSpace {\n> >>> IntelIOMMUState *iommu_state;\n> >>> VTDContextCacheEntry context_cache_entry;\n> >>> QLIST_ENTRY(VTDAddressSpace) next;\n> >>> + /* Superset of notifier flags that this address space has */\n> >>> + IOMMUNotifierFlag notifier_flags;\n> >>> };\n> >>> \n> >>> struct VTDBus {\n> >>> diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c\n> >>> index 112971638d..9a418abfb6 100644\n> >>> --- a/hw/i386/intel_iommu.c\n> >>> +++ b/hw/i386/intel_iommu.c\n> >>> @@ -138,6 +138,12 @@ static inline void vtd_iommu_unlock(IntelIOMMUState *s)\n> >>> qemu_mutex_unlock(&s->iommu_lock);\n> >>> }\n> >>> \n> >>> +/* Whether the address space needs to notify new mappings */\n> >>> +static inline gboolean vtd_as_notify_mappings(VTDAddressSpace *as)\n> >> would suggest vtd_as_has_map_notifier()? But tastes & colours ;-)\n> > \n> > Yeah it is. But okay, I can switch to that especially it's only used\n> > in this patch and it's new.\n> > \n> >>> +{\n> >>> + return as->notifier_flags & IOMMU_NOTIFIER_MAP;\n> >>> +}\n> >>> +\n> >>> /* GHashTable functions */\n> >>> static gboolean vtd_uint64_equal(gconstpointer v1, gconstpointer v2)\n> >>> {\n> >>> @@ -1433,14 +1439,35 @@ static void vtd_iotlb_page_invalidate_notify(IntelIOMMUState *s,\n> >>> VTDAddressSpace *vtd_as;\n> >>> VTDContextEntry ce;\n> >>> int ret;\n> >>> + hwaddr size = (1 << am) * VTD_PAGE_SIZE;\n> >>> \n> >>> QLIST_FOREACH(vtd_as, &(s->notifiers_list), next) {\n> >>> ret = vtd_dev_to_context_entry(s, pci_bus_num(vtd_as->bus),\n> >>> vtd_as->devfn, &ce);\n> >>> if (!ret && domain_id == VTD_CONTEXT_ENTRY_DID(ce.hi)) {\n> >>> - vtd_page_walk(&ce, addr, addr + (1 << am) * VTD_PAGE_SIZE,\n> >>> - vtd_page_invalidate_notify_hook,\n> >>> - (void *)&vtd_as->iommu, true, s->aw_bits);\n> >>> + if (vtd_as_notify_mappings(vtd_as)) {\n> >>> + /*\n> >>> + * For MAP-inclusive notifiers, we need to walk the\n> >>> + * page table to sync the shadow page table.\n> >>> + */\n> >> Potentially we may have several notifiers attached to the IOMMU MR ~\n> >> vtd_as, each of them having different flags. Those flags are OR'ed in\n> >> memory_region_update_iommu_notify_flags and this is the one you now\n> >> store in the vtd_as. So maybe your comment may rather state:\n> >> as soon as we have at least one MAP notifier, we need to do the PTW?\n> > \n> > Actually this is not 100% clear too, since all the \"MAP notifiers\" are\n> > actually both MAP+UNMAP notifiers... Maybe:\n> \n> Can't IOMMU_NOTIFIER_MAP flag value be used without\n> IOMMU_NOTIFIER_UNMAP? I don't see such restriction in the\n> memory_region_register_iommu_notifier API.\n\nYes from the API it can, but I can hardly think of a use case of\nthat.\n\n> > \n> > As long as we have MAP notifications registered in any of our IOMMU\n> > notifiers, we need to sync the shadow page table.\n> > \n> >>\n> >> nit: not related to this patch: vtd_page_walk kerneldoc comments misses\n> >> @notify_unmap param comment\n> >> side note: the name of the hook is a bit misleading as it suggests we\n> >> invalidate the entry, whereas we update any valid entry and invalidate\n> >> stale ones (if notify_unmap=true)?\n> >>> + vtd_page_walk(&ce, addr, addr + size,\n> >>> + vtd_page_invalidate_notify_hook,\n> >>> + (void *)&vtd_as->iommu, true, s->aw_bits);\n> >>> + } else {\n> >>> + /*\n> >>> + * For UNMAP-only notifiers, we don't need to walk the\n> >>> + * page tables. We just deliver the PSI down to\n> >>> + * invalidate caches.\n> >>\n> >> We just unmap the range?\n> > \n> > Isn't it the same thing? :)\n> > \n> > If to be explicit, here we know we only registered UNMAP\n> > notifications, it's not really \"unmap\", it's really cache\n> > invalidations only.\n> yes you're right I meant We just invalidate the range in cache. The\n> sentence \"We just deliver the PSI down to invalidate caches.\" was not\n> crystal clear to me at first reading.\n\nOkay. I just posted a new version, please feel free to comment again\nif you have better suggestions. Otherwise I'll just keep the comment\nfor now. Thanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdmarc=fail (p=none dis=none) header.from=redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 40nNzP72crz9s2k\n\tfor ;\n\tFri, 18 May 2018 20:03:28 +1000 (AEST)","from localhost ([::1]:37797 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1fJcEP-0006dn-Cr\n\tfor incoming@patchwork.ozlabs.org; Fri, 18 May 2018 06:03:25 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:46083)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1fJcDy-0006cT-Tt\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 06:03:01 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1fJcDv-0004Ih-Q3\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 06:02:58 -0400","from mx3-rdu2.redhat.com ([66.187.233.73]:46530\n\thelo=mx1.redhat.com)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1fJcDv-0004IN-Jq\n\tfor qemu-devel@nongnu.org; Fri, 18 May 2018 06:02:55 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 81FE7401EF14;\n\tFri, 18 May 2018 10:02:54 +0000 (UTC)","from xz-mi (dhcp-14-151.nay.redhat.com [10.66.14.151])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id AB6F82023582;\n\tFri, 18 May 2018 10:02:51 +0000 (UTC)"],"Date":"Fri, 18 May 2018 18:02:49 +0800","From":"Peter Xu ","To":"Auger Eric ","Message-ID":"<20180518100249.GM2569@xz-mi>","References":"<20180504030811.28111-1-peterx@redhat.com>\n\t<20180504030811.28111-5-peterx@redhat.com>\n\t<5f5b6f33-12fc-1bd1-a60f-035196270b23@redhat.com>\n\t<20180518055307.GD2569@xz-mi>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.9.5 (2018-04-13)","X-Scanned-By":"MIMEDefang 2.78 on 10.11.54.4","X-Greylist":["Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.5]);\n\tFri, 18 May 2018 10:02:54 +0000 (UTC)","inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.5]); \n\tFri, 18 May 2018 10:02:54 +0000 (UTC) for IP:'10.11.54.4'\n\tDOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'peterx@redhat.com' RCPT:''"],"X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"66.187.233.73","Subject":"Re: [Qemu-devel] [PATCH v2 04/10] intel-iommu: only do page walk\n\tfor MAP notifiers","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Tian Kevin , \"Michael S . Tsirkin\" ,\n\tJason Wang , qemu-devel@nongnu.org,\n\tAlex Williamson ,\n\tJintack Lim ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]