[{"id":1762390,"web_url":"http://patchwork.ozlabs.org/comment/1762390/","msgid":"<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>","list_archive_url":null,"date":"2017-09-04T01:38:47","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":69629,"url":"http://patchwork.ozlabs.org/api/people/69629/","name":"Dou Liyang","email":"douly.fnst@cn.fujitsu.com"},"content":"Hi Eduardo, Thadeu,\n\nAt 09/02/2017 12:11 AM, Eduardo Habkost wrote:\n> On Fri, Sep 01, 2017 at 12:45:42PM -0300, Thadeu Lima de Souza Cascardo wrote:\n>> Linux uses SRAT to determine the maximum memory in a system, which is\n>> used to determine whether to use the swiotlb for IOMMU or not for a\n>> device that supports only 32 bits of addresses.\n>\n> Do you have a pointer to the corresponding Linux code, for\n> reference? Which SRAT entries Linux uses to make this decision?\n>\n>>\n>> When there is no NUMA configuration, qemu will not build SRAT. And when\n>> memory hotplug is done, some Linux device drivers start failing.\n>>\n>> Tested by running with -m 512M,slots=8,maxmem=1G, adding the memory,\n>> putting that online and using the system. Without the patch, swiotlb is\n>> not used and ATA driver fails. With the patch, swiotlb is used, no\n>> driver failure is observed.\n>>\n>> Signed-off-by: Thadeu Lima de Souza Cascardo \n>\n> As far as I can see, this will only add APIC entries and a memory\n> affinity entry for the first 640KB (which would be obviously\n> wrong) if pcms->numa_nodes is 0.\n>\n\nIn my opinion, this may also add the hotpluggable memory, and see the \nfollowing commemts.\n\n /*\n * Entry is required for Windows to enable memory hotplug in OS\n * and for Linux to enable SWIOTLB when booted with less than\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n * 4G of RAM. Windows works better if the entry sets proximity\n * to the highest NUMA node in the machine.\n * Memory devices may override proximity set by this entry,\n * providing _PXM method if necessary.\n */\n if (hotplugabble_address_space_size) {\n numamem = acpi_data_push(table_data, sizeof *numamem);\n build_srat_memory(numamem, pcms->hotplug_memory.base,\n hotplugabble_address_space_size, \npcms->numa_nodes - 1,\n MEM_AFFINITY_HOTPLUGGABLE | \nMEM_AFFINITY_ENABLED);\n }\n\n\nThanks,\n\tdou.\n\n> Once we apply the \"Fix SRAT memory building in case of node 0\n> without RAM\" patch from Dou Liyang, no memory affinity entries\n> will be generated if pcms->numa_nodes is 0. Would this cause the\n> problem to happen again?\n>\n\n\n\n>\n>> ---\n>> hw/i386/acpi-build.c | 5 ++++-\n>> 1 file changed, 4 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c\n>> index 98dd424678..fb94249779 100644\n>> --- a/hw/i386/acpi-build.c\n>> +++ b/hw/i386/acpi-build.c\n>> @@ -2645,6 +2645,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n>> GArray *tables_blob = tables->table_data;\n>> AcpiSlicOem slic_oem = { .id = NULL, .table_id = NULL };\n>> Object *vmgenid_dev;\n>> + ram_addr_t hotplugabble_address_space_size =\n>> + object_property_get_int(OBJECT(pcms), PC_MACHINE_MEMHP_REGION_SIZE,\n>> + NULL);\n>>\n>> acpi_get_pm_info(&pm);\n>> acpi_get_misc_info(&misc);\n>> @@ -2708,7 +2711,7 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n>> build_tpm2(tables_blob, tables->linker);\n>> }\n>> }\n>> - if (pcms->numa_nodes) {\n>> + if (pcms->numa_nodes || hotplugabble_address_space_size) {\n>> acpi_add_table(table_offsets, tables_blob);\n>> build_srat(tables_blob, tables->linker, machine);\n>> if (have_numa_distance) {\n>> --\n>> 2.11.0\n>>\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xlswV6Xlkz9s7f\n\tfor ;\n\tMon, 4 Sep 2017 11:39:53 +1000 (AEST)","from localhost ([::1]:48242 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dogMf-0003VZ-Uq\n\tfor incoming@patchwork.ozlabs.org; Sun, 03 Sep 2017 21:39:49 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:46388)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dogM9-0003UV-TS\n\tfor qemu-devel@nongnu.org; Sun, 03 Sep 2017 21:39:22 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dogM4-00018l-Td\n\tfor qemu-devel@nongnu.org; Sun, 03 Sep 2017 21:39:17 -0400","from mail.cn.fujitsu.com ([183.91.158.132]:54396\n\thelo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dogM4-00016S-2Y\n\tfor qemu-devel@nongnu.org; Sun, 03 Sep 2017 21:39:12 -0400","from localhost (HELO cn.fujitsu.com) ([10.167.33.5])\n\tby heian.cn.fujitsu.com with ESMTP; 04 Sep 2017 09:38:52 +0800","from G08CNEXCHPEKD03.g08.fujitsu.local (unknown [10.167.33.85])\n\tby cn.fujitsu.com (Postfix) with ESMTP id 5AD124724E72;\n\tMon, 4 Sep 2017 09:38:51 +0800 (CST)","from localhost.localdomain (10.167.226.106) by\n\tG08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP\n\tServer (TLS) id 14.3.361.1; Mon, 4 Sep 2017 09:38:56 +0800"],"X-IronPort-AV":"E=Sophos;i=\"5.41,473,1498492800\"; d=\"scan'208\";a=\"25169199\"","To":"Eduardo Habkost , Thadeu Lima de Souza Cascardo\n\t","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>","From":"Dou Liyang ","Message-ID":"<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>","Date":"Mon, 4 Sep 2017 09:38:47 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20170901161118.GQ7570@localhost.localdomain>","Content-Type":"text/plain; charset=\"windows-1252\"; format=flowed","Content-Transfer-Encoding":"7bit","X-Originating-IP":"[10.167.226.106]","X-yoursite-MailScanner-ID":"5AD124724E72.AA14F","X-yoursite-MailScanner":"Found to be clean","X-yoursite-MailScanner-From":"douly.fnst@cn.fujitsu.com","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"183.91.158.132","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Igor Mammedov , Richard Henderson ,\n\tPaolo Bonzini , qemu-devel@nongnu.org,\n\t\"Michael S. Tsirkin\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1762474,"web_url":"http://patchwork.ozlabs.org/comment/1762474/","msgid":"<20170904092305.6adbf505@nial.brq.redhat.com>","list_archive_url":null,"date":"2017-09-04T07:23:05","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":11305,"url":"http://patchwork.ozlabs.org/api/people/11305/","name":"Igor Mammedov","email":"imammedo@redhat.com"},"content":"On Fri, 1 Sep 2017 13:11:18 -0300\nEduardo Habkost wrote:\n\n> On Fri, Sep 01, 2017 at 12:45:42PM -0300, Thadeu Lima de Souza Cascardo wrote:\n> > Linux uses SRAT to determine the maximum memory in a system, which is\n> > used to determine whether to use the swiotlb for IOMMU or not for a\n> > device that supports only 32 bits of addresses. \n> \n> Do you have a pointer to the corresponding Linux code, for\n> reference? Which SRAT entries Linux uses to make this decision?\nec941c5f x86/mm/64: Enable SWIOTLB if system has SRAT memory regions above MAX_DMA32_PFN\n\n\n> > When there is no NUMA configuration, qemu will not build SRAT. And when\n> > memory hotplug is done, some Linux device drivers start failing.\n> > \n> > Tested by running with -m 512M,slots=8,maxmem=1G, adding the memory,\n> > putting that online and using the system. Without the patch, swiotlb is\n> > not used and ATA driver fails. With the patch, swiotlb is used, no\n> > driver failure is observed.\n> > \n> > Signed-off-by: Thadeu Lima de Souza Cascardo \n> \n> As far as I can see, this will only add APIC entries and a memory\n> affinity entry for the first 640KB (which would be obviously\n> wrong) if pcms->numa_nodes is 0.\nI think safest way to achieve what patch needs is to\nenable add numa node implicitly if user hasn't done it explicitly on CLI\nwhen memory hotplug is enabled i.e. slots != 0.\n\n> \n> Once we apply the \"Fix SRAT memory building in case of node 0\n> without RAM\" patch from Dou Liyang, no memory affinity entries\n> will be generated if pcms->numa_nodes is 0. Would this cause the\n> problem to happen again?\n> \n> \n> > ---\n> > hw/i386/acpi-build.c | 5 ++++-\n> > 1 file changed, 4 insertions(+), 1 deletion(-)\n> > \n> > diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c\n> > index 98dd424678..fb94249779 100644\n> > --- a/hw/i386/acpi-build.c\n> > +++ b/hw/i386/acpi-build.c\n> > @@ -2645,6 +2645,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n> > GArray *tables_blob = tables->table_data;\n> > AcpiSlicOem slic_oem = { .id = NULL, .table_id = NULL };\n> > Object *vmgenid_dev;\n> > + ram_addr_t hotplugabble_address_space_size =\n> > + object_property_get_int(OBJECT(pcms), PC_MACHINE_MEMHP_REGION_SIZE,\n> > + NULL);\n> > \n> > acpi_get_pm_info(&pm);\n> > acpi_get_misc_info(&misc);\n> > @@ -2708,7 +2711,7 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n> > build_tpm2(tables_blob, tables->linker);\n> > }\n> > }\n> > - if (pcms->numa_nodes) {\n> > + if (pcms->numa_nodes || hotplugabble_address_space_size) {\n> > acpi_add_table(table_offsets, tables_blob);\n> > build_srat(tables_blob, tables->linker, machine);\n> > if (have_numa_distance) {\n> > -- \n> > 2.11.0\n> > \n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=imammedo@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xm1YH3Nhnz9s82\n\tfor ;\n\tMon, 4 Sep 2017 17:23:44 +1000 (AEST)","from localhost ([::1]:39100 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1doljR-0006PX-DE\n\tfor incoming@patchwork.ozlabs.org; Mon, 04 Sep 2017 03:23:41 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:49696)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dolj3-0006P7-VB\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 03:23:19 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dolj0-0008JJ-MI\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 03:23:17 -0400","from mx1.redhat.com ([209.132.183.28]:33142)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dolj0-0008Hc-Cg\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 03:23:14 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 75697C058EAE;\n\tMon, 4 Sep 2017 07:23:12 +0000 (UTC)","from nial.brq.redhat.com (unknown [10.43.2.241])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id AED418471C;\n\tMon, 4 Sep 2017 07:23:06 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 75697C058EAE","Date":"Mon, 4 Sep 2017 09:23:05 +0200","From":"Igor Mammedov ","To":"Eduardo Habkost ","Message-ID":"<20170904092305.6adbf505@nial.brq.redhat.com>","In-Reply-To":"<20170901161118.GQ7570@localhost.localdomain>","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.32]);\n\tMon, 04 Sep 2017 07:23:12 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Thadeu Lima de Souza Cascardo ,\n\tDou Liyang ,\n\t\"Michael S. Tsirkin\" , qemu-devel@nongnu.org,\n\tPaolo Bonzini , Richard Henderson ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1762626,"web_url":"http://patchwork.ozlabs.org/comment/1762626/","msgid":"<20170904130814.GR7570@localhost.localdomain>","list_archive_url":null,"date":"2017-09-04T13:08:14","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":195,"url":"http://patchwork.ozlabs.org/api/people/195/","name":"Eduardo Habkost","email":"ehabkost@redhat.com"},"content":"On Mon, Sep 04, 2017 at 09:38:47AM +0800, Dou Liyang wrote:\n> Hi Eduardo, Thadeu,\n> \n> At 09/02/2017 12:11 AM, Eduardo Habkost wrote:\n> > On Fri, Sep 01, 2017 at 12:45:42PM -0300, Thadeu Lima de Souza Cascardo wrote:\n> > > Linux uses SRAT to determine the maximum memory in a system, which is\n> > > used to determine whether to use the swiotlb for IOMMU or not for a\n> > > device that supports only 32 bits of addresses.\n> > \n> > Do you have a pointer to the corresponding Linux code, for\n> > reference? Which SRAT entries Linux uses to make this decision?\n> > \n> > > \n> > > When there is no NUMA configuration, qemu will not build SRAT. And when\n> > > memory hotplug is done, some Linux device drivers start failing.\n> > > \n> > > Tested by running with -m 512M,slots=8,maxmem=1G, adding the memory,\n> > > putting that online and using the system. Without the patch, swiotlb is\n> > > not used and ATA driver fails. With the patch, swiotlb is used, no\n> > > driver failure is observed.\n> > > \n> > > Signed-off-by: Thadeu Lima de Souza Cascardo \n> > \n> > As far as I can see, this will only add APIC entries and a memory\n> > affinity entry for the first 640KB (which would be obviously\n> > wrong) if pcms->numa_nodes is 0.\n> > \n> \n> In my opinion, this may also add the hotpluggable memory, and see the\n> following commemts.\n> \n> /*\n> * Entry is required for Windows to enable memory hotplug in OS\n> * and for Linux to enable SWIOTLB when booted with less than\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> * 4G of RAM. Windows works better if the entry sets proximity\n> * to the highest NUMA node in the machine.\n> * Memory devices may override proximity set by this entry,\n> * providing _PXM method if necessary.\n> */\n> if (hotplugabble_address_space_size) {\n> numamem = acpi_data_push(table_data, sizeof *numamem);\n> build_srat_memory(numamem, pcms->hotplug_memory.base,\n> hotplugabble_address_space_size, pcms->numa_nodes\n> - 1,\n> MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);\n> }\n\nYou are correct, I didn't see that part of the code. If that's\nthe entry that's missing, the patch makes sense. Thanks!\n\nHowever, the resulting tables still don't look correct: it will\ngenerate an entry assigned to NUMA node (uint32_t)-1 if no NUMA\nnodes are configured elsewhere, some APIC entries, but no entries\nfor the rest of the memory.\n\nIgor's suggestion to enable NUMA implicitly sounds safer to me.\n\n> \n> \n> Thanks,\n> \tdou.\n> \n> > Once we apply the \"Fix SRAT memory building in case of node 0\n> > without RAM\" patch from Dou Liyang, no memory affinity entries\n> > will be generated if pcms->numa_nodes is 0. Would this cause the\n> > problem to happen again?\n> > \n> \n> \n> \n> > \n> > > ---\n> > > hw/i386/acpi-build.c | 5 ++++-\n> > > 1 file changed, 4 insertions(+), 1 deletion(-)\n> > > \n> > > diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c\n> > > index 98dd424678..fb94249779 100644\n> > > --- a/hw/i386/acpi-build.c\n> > > +++ b/hw/i386/acpi-build.c\n> > > @@ -2645,6 +2645,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n> > > GArray *tables_blob = tables->table_data;\n> > > AcpiSlicOem slic_oem = { .id = NULL, .table_id = NULL };\n> > > Object *vmgenid_dev;\n> > > + ram_addr_t hotplugabble_address_space_size =\n> > > + object_property_get_int(OBJECT(pcms), PC_MACHINE_MEMHP_REGION_SIZE,\n> > > + NULL);\n> > > \n> > > acpi_get_pm_info(&pm);\n> > > acpi_get_misc_info(&misc);\n> > > @@ -2708,7 +2711,7 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)\n> > > build_tpm2(tables_blob, tables->linker);\n> > > }\n> > > }\n> > > - if (pcms->numa_nodes) {\n> > > + if (pcms->numa_nodes || hotplugabble_address_space_size) {\n> > > acpi_add_table(table_offsets, tables_blob);\n> > > build_srat(tables_blob, tables->linker, machine);\n> > > if (have_numa_distance) {\n> > > --\n> > > 2.11.0\n> > > \n> > \n> \n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=ehabkost@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xm9KX2ttYz9sR9\n\tfor ;\n\tMon, 4 Sep 2017 23:14:07 +1000 (AEST)","from localhost ([::1]:37553 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dorCW-0001i4-Ll\n\tfor incoming@patchwork.ozlabs.org; Mon, 04 Sep 2017 09:14:04 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:51936)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dor85-0006mm-5J\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 09:13:32 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dor71-0006l3-3h\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 09:09:29 -0400","from mx1.redhat.com ([209.132.183.28]:44910)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dor70-0006fK-Os\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 09:08:23 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 645A5C047B6B;\n\tMon, 4 Sep 2017 13:08:18 +0000 (UTC)","from localhost (ovpn-116-66.gru2.redhat.com [10.97.116.66])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 9E4148208E;\n\tMon, 4 Sep 2017 13:08:15 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 645A5C047B6B","Date":"Mon, 4 Sep 2017 10:08:14 -0300","From":"Eduardo Habkost ","To":"Dou Liyang ","Message-ID":"<20170904130814.GR7570@localhost.localdomain>","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>\n\t<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>","X-Fnord":"you can see the fnord","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tMon, 04 Sep 2017 13:08:18 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Thadeu Lima de Souza Cascardo ,\n\t\"Michael S. Tsirkin\" , qemu-devel@nongnu.org,\n\tPaolo Bonzini ,\n\tIgor Mammedov , Richard Henderson ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1762905,"web_url":"http://patchwork.ozlabs.org/comment/1762905/","msgid":"","list_archive_url":null,"date":"2017-09-05T01:17:06","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":69629,"url":"http://patchwork.ozlabs.org/api/people/69629/","name":"Dou Liyang","email":"douly.fnst@cn.fujitsu.com"},"content":"Hi Eduardo,\n\nAt 09/04/2017 09:08 PM, Eduardo Habkost wrote:\n[...]\n>> In my opinion, this may also add the hotpluggable memory, and see the\n>> following commemts.\n>>\n>> /*\n>> * Entry is required for Windows to enable memory hotplug in OS\n>> * and for Linux to enable SWIOTLB when booted with less than\n>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>> * 4G of RAM. Windows works better if the entry sets proximity\n>> * to the highest NUMA node in the machine.\n>> * Memory devices may override proximity set by this entry,\n>> * providing _PXM method if necessary.\n>> */\n>> if (hotplugabble_address_space_size) {\n>> numamem = acpi_data_push(table_data, sizeof *numamem);\n>> build_srat_memory(numamem, pcms->hotplug_memory.base,\n>> hotplugabble_address_space_size, pcms->numa_nodes\n>> - 1,\n>> MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);\n>> }\n>\n> You are correct, I didn't see that part of the code. If that's\n> the entry that's missing, the patch makes sense. Thanks!\n>\n> However, the resulting tables still don't look correct: it will\n> generate an entry assigned to NUMA node (uint32_t)-1 if no NUMA\n> nodes are configured elsewhere, some APIC entries, but no entries\n> for the rest of the memory.\n\nYes, indeed.\n\n>\n> Igor's suggestion to enable NUMA implicitly sounds safer to me.\n>\n\nI agree with Igor too.\n\nIs anybody doing this? If not, may I make a patch to enable adding NUMA\nnode implicitly first. let's see what it looks like.\n\nThanks,\n\tdou.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xmTNn2Mjlz9s7h\n\tfor ;\n\tTue, 5 Sep 2017 11:18:00 +1000 (AEST)","from localhost ([::1]:56282 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dp2V3-0008DK-Nm\n\tfor incoming@patchwork.ozlabs.org; Mon, 04 Sep 2017 21:17:57 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:45042)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dp2UQ-0008Cw-1R\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 21:17:22 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dp2UL-0002mx-3U\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 21:17:18 -0400","from mail.cn.fujitsu.com ([183.91.158.132]:53075\n\thelo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dp2UK-0002mT-OV\n\tfor qemu-devel@nongnu.org; Mon, 04 Sep 2017 21:17:13 -0400","from localhost (HELO cn.fujitsu.com) ([10.167.33.5])\n\tby heian.cn.fujitsu.com with ESMTP; 05 Sep 2017 09:17:10 +0800","from G08CNEXCHPEKD03.g08.fujitsu.local (unknown [10.167.33.85])\n\tby cn.fujitsu.com (Postfix) with ESMTP id 695514725542;\n\tTue, 5 Sep 2017 09:17:09 +0800 (CST)","from localhost.localdomain (10.167.226.106) by\n\tG08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP\n\tServer (TLS) id 14.3.361.1; Tue, 5 Sep 2017 09:17:08 +0800"],"X-IronPort-AV":"E=Sophos;i=\"5.41,477,1498492800\"; d=\"scan'208\";a=\"25214187\"","To":"Eduardo Habkost ","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>\n\t<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>\n\t<20170904130814.GR7570@localhost.localdomain>","From":"Dou Liyang ","Message-ID":"","Date":"Tue, 5 Sep 2017 09:17:06 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20170904130814.GR7570@localhost.localdomain>","Content-Type":"text/plain; charset=\"windows-1252\"; format=flowed","Content-Transfer-Encoding":"7bit","X-Originating-IP":"[10.167.226.106]","X-yoursite-MailScanner-ID":"695514725542.A5D66","X-yoursite-MailScanner":"Found to be clean","X-yoursite-MailScanner-From":"douly.fnst@cn.fujitsu.com","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"183.91.158.132","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Thadeu Lima de Souza Cascardo ,\n\t\"Michael S. Tsirkin\" , qemu-devel@nongnu.org,\n\tPaolo Bonzini ,\n\tIgor Mammedov , Richard Henderson ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1766221,"web_url":"http://patchwork.ozlabs.org/comment/1766221/","msgid":"<20170911125814.27a8d496@nial.brq.redhat.com>","list_archive_url":null,"date":"2017-09-11T10:58:14","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":11305,"url":"http://patchwork.ozlabs.org/api/people/11305/","name":"Igor Mammedov","email":"imammedo@redhat.com"},"content":"On Tue, 5 Sep 2017 09:17:06 +0800\nDou Liyang wrote:\n\n> Hi Eduardo,\n> \n> At 09/04/2017 09:08 PM, Eduardo Habkost wrote:\n> [...]\n> >> In my opinion, this may also add the hotpluggable memory, and see the\n> >> following commemts.\n> >>\n> >> /*\n> >> * Entry is required for Windows to enable memory hotplug in OS\n> >> * and for Linux to enable SWIOTLB when booted with less than\n> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> >> * 4G of RAM. Windows works better if the entry sets proximity\n> >> * to the highest NUMA node in the machine.\n> >> * Memory devices may override proximity set by this entry,\n> >> * providing _PXM method if necessary.\n> >> */\n> >> if (hotplugabble_address_space_size) {\n> >> numamem = acpi_data_push(table_data, sizeof *numamem);\n> >> build_srat_memory(numamem, pcms->hotplug_memory.base,\n> >> hotplugabble_address_space_size, pcms->numa_nodes\n> >> - 1,\n> >> MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED);\n> >> } \n> >\n> > You are correct, I didn't see that part of the code. If that's\n> > the entry that's missing, the patch makes sense. Thanks!\n> >\n> > However, the resulting tables still don't look correct: it will\n> > generate an entry assigned to NUMA node (uint32_t)-1 if no NUMA\n> > nodes are configured elsewhere, some APIC entries, but no entries\n> > for the rest of the memory. \n> \n> Yes, indeed.\n> \n> >\n> > Igor's suggestion to enable NUMA implicitly sounds safer to me.\n> > \n> \n> I agree with Igor too.\n> \n> Is anybody doing this? If not, may I make a patch to enable adding NUMA\n> node implicitly first. let's see what it looks like.\nAs far as I'm aware nobody is doing it, so fill free to look into it.\n\n> \n> Thanks,\n> \tdou.\n> \n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=imammedo@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xrQ2S5PtSz9s83\n\tfor ;\n\tMon, 11 Sep 2017 21:00:48 +1000 (AEST)","from localhost ([::1]:56703 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1drMSM-0001OF-Sx\n\tfor incoming@patchwork.ozlabs.org; Mon, 11 Sep 2017 07:00:46 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:39653)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1drMQA-0008QJ-BB\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 06:58:31 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1drMQ5-0002Un-TG\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 06:58:30 -0400","from mx1.redhat.com ([209.132.183.28]:43670)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1drMQ5-0002UQ-Mi\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 06:58:25 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id A0AA1C0A525D;\n\tMon, 11 Sep 2017 10:58:23 +0000 (UTC)","from nial.brq.redhat.com (unknown [10.43.2.209])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 1774D17CE7;\n\tMon, 11 Sep 2017 10:58:15 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com A0AA1C0A525D","Date":"Mon, 11 Sep 2017 12:58:14 +0200","From":"Igor Mammedov ","To":"Dou Liyang ","Message-ID":"<20170911125814.27a8d496@nial.brq.redhat.com>","In-Reply-To":"","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>\n\t<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>\n\t<20170904130814.GR7570@localhost.localdomain>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tMon, 11 Sep 2017 10:58:23 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Thadeu Lima de Souza Cascardo ,\n\tEduardo Habkost ,\n\t\"Michael S. Tsirkin\" , qemu-devel@nongnu.org,\n\tPaolo Bonzini , Richard Henderson ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1766641,"web_url":"http://patchwork.ozlabs.org/comment/1766641/","msgid":"<70fffb76-2273-7184-6dea-2b5788ec3b57@cn.fujitsu.com>","list_archive_url":null,"date":"2017-09-12T00:58:49","subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","submitter":{"id":69629,"url":"http://patchwork.ozlabs.org/api/people/69629/","name":"Dou Liyang","email":"douly.fnst@cn.fujitsu.com"},"content":"Hi Igor,\n\nAt 09/11/2017 06:58 PM, Igor Mammedov wrote:\n>>> > >\n>>> > > Igor's suggestion to enable NUMA implicitly sounds safer to me.\n>>> > >\n>> >\n>> > I agree with Igor too.\n>> >\n>> > Is anybody doing this? If not, may I make a patch to enable adding NUMA\n>> > node implicitly first. let's see what it looks like.\n> As far as I'm aware nobody is doing it, so fill free to look into it.\n>\n\nGot it! I will do it right now.\n\nThanks,\n\tdou.\n\n>> >\n>> > Thanks,\n>> > \tdou.\n>> >\n>> >","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xrmr23WK7z9s81\n\tfor ;\n\tTue, 12 Sep 2017 11:08:00 +1000 (AEST)","from localhost ([::1]:33029 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1drZgD-00024C-PS\n\tfor incoming@patchwork.ozlabs.org; Mon, 11 Sep 2017 21:07:57 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:44082)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1drZfn-000240-R7\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 21:07:32 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1drZfk-0002Ae-Nk\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 21:07:31 -0400","from mail.cn.fujitsu.com ([183.91.158.132]:48674\n\thelo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1drZfk-000267-CM\n\tfor qemu-devel@nongnu.org; Mon, 11 Sep 2017 21:07:28 -0400","from localhost (HELO cn.fujitsu.com) ([10.167.33.5])\n\tby heian.cn.fujitsu.com with ESMTP; 12 Sep 2017 09:07:23 +0800","from G08CNEXCHPEKD03.g08.fujitsu.local (unknown [10.167.33.85])\n\tby cn.fujitsu.com (Postfix) with ESMTP id 634DD47CA67C;\n\tTue, 12 Sep 2017 08:58:51 +0800 (CST)","from localhost.localdomain (10.167.226.106) by\n\tG08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP\n\tServer (TLS) id 14.3.361.1; Tue, 12 Sep 2017 08:58:50 +0800"],"X-IronPort-AV":"E=Sophos;i=\"5.42,380,1500912000\"; d=\"scan'208\";a=\"25799451\"","To":"Igor Mammedov ","References":"<20170901154542.5687-1-cascardo@canonical.com>\n\t<20170901161118.GQ7570@localhost.localdomain>\n\t<2ab3aaef-c6d7-ae3d-808b-f947dcb3ed1c@cn.fujitsu.com>\n\t<20170904130814.GR7570@localhost.localdomain>\n\t\n\t<20170911125814.27a8d496@nial.brq.redhat.com>","From":"Dou Liyang ","Message-ID":"<70fffb76-2273-7184-6dea-2b5788ec3b57@cn.fujitsu.com>","Date":"Tue, 12 Sep 2017 08:58:49 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20170911125814.27a8d496@nial.brq.redhat.com>","Content-Type":"text/plain; charset=\"windows-1252\"; format=flowed","Content-Transfer-Encoding":"7bit","X-Originating-IP":"[10.167.226.106]","X-yoursite-MailScanner-ID":"634DD47CA67C.ACF04","X-yoursite-MailScanner":"Found to be clean","X-yoursite-MailScanner-From":"douly.fnst@cn.fujitsu.com","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"183.91.158.132","Subject":"Re: [Qemu-devel] [PATCH] x86/acpi: build SRAT when memory hotplug\n\tis enabled","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Thadeu Lima de Souza Cascardo ,\n\tEduardo Habkost ,\n\t\"Michael S. Tsirkin\" , qemu-devel@nongnu.org,\n\tPaolo Bonzini , Richard Henderson ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]