[{"id":1784916,"web_url":"http://patchwork.ozlabs.org/comment/1784916/","msgid":"<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-11T20:05:24","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n> This patch adds the machine dependent call for\n> pcibios_bus_add_device, since the previous patch\n> separated the calls out between the PowerNV and PowerVM.\n> \n> The difference here is that for the PowerVM environment\n> we do not want match_driver set because in this environment\n> we do not want the VF device drivers to load immediately, due to\n> firmware loading the device node when VF device is assigned to the\n> logical partition.\n> \n> This patch will depend on the patch linked below, which is under\n> review.\n> \n> https://patchwork.kernel.org/patch/9882915/\n> \n> Signed-off-by: Bryant G. Ly \n> Signed-off-by: Juan J. Alvarez \n> ---\n> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n> 1 file changed, 24 insertions(+)\n> \n> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> index 6b812ad990e4..45946ee90985 100644\n> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> @@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n> static DEFINE_SPINLOCK(slot_errbuf_lock);\n> static int eeh_error_buf_size;\n> \n> +void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n> +{\n> +\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n> +\n> +\tif (!pdev->is_virtfn)\n> +\t\treturn;\n> +\n> +\tpdn->device_id = pdev->device;\n> +\tpdn->vendor_id = pdev->vendor;\n> +\tpdn->class_code = pdev->class;\n> +\n> +\t/*\n> +\t * The following operations will fail if VF's sysfs files\n> +\t * aren't created or its resources aren't finalized.\n> +\t */\n> +\teeh_add_device_early(pdn);\n> +\teeh_add_device_late(pdev);\n> +\teeh_sysfs_add_device(pdev);\n> +\tpdev->match_driver = -1;\n\nmatch_driver is a bool, which should be assigned \"true\" or \"false\".\n\n> +}\n> +\n> /**\n> * pseries_eeh_init - EEH platform dependent initialization\n> *\n> @@ -120,6 +141,9 @@ static int pseries_eeh_init(void)\n> \t/* Set EEH probe mode */\n> \teeh_add_flag(EEH_PROBE_MODE_DEVTREE | EEH_ENABLE_IO_FOR_LOG);\n> \n> +\t/* Set EEH machine dependent code */\n> +\tppc_md.pcibios_bus_add_device = pseries_pcibios_bus_add_device;\n> +\n> \treturn 0;\n> }\n> \n> -- \n> 2.11.0 (Apple Git-81)\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yC4j55fZLz9ryT\n\tfor ;\n\tThu, 12 Oct 2017 07:05:29 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1757389AbdJKUF2 (ORCPT );\n\tWed, 11 Oct 2017 16:05:28 -0400","from mail.kernel.org ([198.145.29.99]:49252 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1752275AbdJKUF1 (ORCPT );\n\tWed, 11 Oct 2017 16:05:27 -0400","from localhost (unknown [69.71.4.159])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 94150218CA;\n\tWed, 11 Oct 2017 20:05:26 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 94150218CA","Date":"Wed, 11 Oct 2017 15:05:24 -0500","From":"Bjorn Helgaas ","To":"\"Bryant G. Ly\" ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tmpe@ellerman.id.au, seroyer@linux.vnet.ibm.com,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1785128,"web_url":"http://patchwork.ozlabs.org/comment/1785128/","msgid":"<87efq92ei6.fsf@concordia.ellerman.id.au>","list_archive_url":null,"date":"2017-10-12T04:09:53","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":46580,"url":"http://patchwork.ozlabs.org/api/people/46580/","name":"Michael Ellerman","email":"mpe@ellerman.id.au"},"content":"Bjorn Helgaas writes:\n\n> On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n>> This patch adds the machine dependent call for\n>> pcibios_bus_add_device, since the previous patch\n>> separated the calls out between the PowerNV and PowerVM.\n>> \n>> The difference here is that for the PowerVM environment\n>> we do not want match_driver set because in this environment\n>> we do not want the VF device drivers to load immediately, due to\n>> firmware loading the device node when VF device is assigned to the\n>> logical partition.\n>> \n>> This patch will depend on the patch linked below, which is under\n>> review.\n>> \n>> https://patchwork.kernel.org/patch/9882915/\n>> \n>> Signed-off-by: Bryant G. Ly \n>> Signed-off-by: Juan J. Alvarez \n>> ---\n>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n>> 1 file changed, 24 insertions(+)\n>> \n>> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> index 6b812ad990e4..45946ee90985 100644\n>> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> @@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n>> static int eeh_error_buf_size;\n>> \n>> +void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n>> +{\n>> +\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n>> +\n>> +\tif (!pdev->is_virtfn)\n>> +\t\treturn;\n>> +\n>> +\tpdn->device_id = pdev->device;\n>> +\tpdn->vendor_id = pdev->vendor;\n>> +\tpdn->class_code = pdev->class;\n>> +\n>> +\t/*\n>> +\t * The following operations will fail if VF's sysfs files\n>> +\t * aren't created or its resources aren't finalized.\n>> +\t */\n>> +\teeh_add_device_early(pdn);\n>> +\teeh_add_device_late(pdev);\n>> +\teeh_sysfs_add_device(pdev);\n>> +\tpdev->match_driver = -1;\n>\n> match_driver is a bool, which should be assigned \"true\" or \"false\".\n\nAbove he mentioned a dependency on:\n\n [04/10] PCI: extend pci device match_driver state\n https://patchwork.kernel.org/patch/9882915/\n\n\nWhich makes it an int.\n\nOr has that patch been rejected or something?\n\ncheers","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yCHS500k5z9t2Z\n\tfor ;\n\tThu, 12 Oct 2017 15:09:56 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751544AbdJLEJz (ORCPT );\n\tThu, 12 Oct 2017 00:09:55 -0400","from ozlabs.org ([103.22.144.67]:44781 \"EHLO ozlabs.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751347AbdJLEJz (ORCPT );\n\tThu, 12 Oct 2017 00:09:55 -0400","from authenticated.ozlabs.org (localhost [127.0.0.1])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPSA id 3yCHS13HVBz9t2V;\n\tThu, 12 Oct 2017 15:09:53 +1100 (AEDT)"],"From":"Michael Ellerman ","To":"Bjorn Helgaas ,\n\t\"Bryant G. Ly\" ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","In-Reply-To":"<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>","Date":"Thu, 12 Oct 2017 15:09:53 +1100","Message-ID":"<87efq92ei6.fsf@concordia.ellerman.id.au>","MIME-Version":"1.0","Content-Type":"text/plain","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1785720,"web_url":"http://patchwork.ozlabs.org/comment/1785720/","msgid":"<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-12T18:29:32","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n> Bjorn Helgaas writes:\n> \n> > On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n> >> This patch adds the machine dependent call for\n> >> pcibios_bus_add_device, since the previous patch\n> >> separated the calls out between the PowerNV and PowerVM.\n> >> \n> >> The difference here is that for the PowerVM environment\n> >> we do not want match_driver set because in this environment\n> >> we do not want the VF device drivers to load immediately, due to\n> >> firmware loading the device node when VF device is assigned to the\n> >> logical partition.\n> >> \n> >> This patch will depend on the patch linked below, which is under\n> >> review.\n> >> \n> >> https://patchwork.kernel.org/patch/9882915/\n> >> \n> >> Signed-off-by: Bryant G. Ly \n> >> Signed-off-by: Juan J. Alvarez \n> >> ---\n> >> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n> >> 1 file changed, 24 insertions(+)\n> >> \n> >> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >> index 6b812ad990e4..45946ee90985 100644\n> >> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >> @@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n> >> static DEFINE_SPINLOCK(slot_errbuf_lock);\n> >> static int eeh_error_buf_size;\n> >> \n> >> +void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n> >> +{\n> >> +\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n> >> +\n> >> +\tif (!pdev->is_virtfn)\n> >> +\t\treturn;\n> >> +\n> >> +\tpdn->device_id = pdev->device;\n> >> +\tpdn->vendor_id = pdev->vendor;\n> >> +\tpdn->class_code = pdev->class;\n> >> +\n> >> +\t/*\n> >> +\t * The following operations will fail if VF's sysfs files\n> >> +\t * aren't created or its resources aren't finalized.\n> >> +\t */\n> >> +\teeh_add_device_early(pdn);\n> >> +\teeh_add_device_late(pdev);\n> >> +\teeh_sysfs_add_device(pdev);\n> >> +\tpdev->match_driver = -1;\n> >\n> > match_driver is a bool, which should be assigned \"true\" or \"false\".\n> \n> Above he mentioned a dependency on:\n> \n> [04/10] PCI: extend pci device match_driver state\n> https://patchwork.kernel.org/patch/9882915/\n> \n> \n> Which makes it an int.\n\nOh, right, I missed that, thanks.\n\n> Or has that patch been rejected or something?\n\nI haven't *rejected* it, but it's low on my priority list, so you\nshouldn't depend on it unless it adds functionality you really need.\nIf I did apply that particular patch, I would want some rework because\nit currently obfuscates the match_driver logic. There's no clue when\nreading the code what -1/0/1 mean.\n\nApparently here you *do* want the \"-1 means the PCI core will never\nset match_driver to 1\" functionality, so maybe you do depend on it.\n\nIf that's the case, how to you ever bind a driver to these VFs? The\nchangelog says you don't want VF drivers to load *immediately*, so I\nassume you do want them to load eventually.\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yCfX02vWgz9sNc\n\tfor ;\n\tFri, 13 Oct 2017 05:29:36 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751373AbdJLS3f (ORCPT );\n\tThu, 12 Oct 2017 14:29:35 -0400","from mail.kernel.org ([198.145.29.99]:54510 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1750992AbdJLS3e (ORCPT );\n\tThu, 12 Oct 2017 14:29:34 -0400","from localhost (unknown [69.55.156.165])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id D9E8521903;\n\tThu, 12 Oct 2017 18:29:33 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org D9E8521903","Date":"Thu, 12 Oct 2017 13:29:32 -0500","From":"Bjorn Helgaas ","To":"Michael Ellerman ","Cc":"\"Bryant G. Ly\" , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<87efq92ei6.fsf@concordia.ellerman.id.au>","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1785789,"web_url":"http://patchwork.ozlabs.org/comment/1785789/","msgid":"","list_archive_url":null,"date":"2017-10-12T19:59:23","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72247,"url":"http://patchwork.ozlabs.org/api/people/72247/","name":"Bryant G. Ly","email":"bryantly@linux.vnet.ibm.com"},"content":"On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n> On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n>> Bjorn Helgaas writes:\n>>\n>>> On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n>>>> This patch adds the machine dependent call for\n>>>> pcibios_bus_add_device, since the previous patch\n>>>> separated the calls out between the PowerNV and PowerVM.\n>>>>\n>>>> The difference here is that for the PowerVM environment\n>>>> we do not want match_driver set because in this environment\n>>>> we do not want the VF device drivers to load immediately, due to\n>>>> firmware loading the device node when VF device is assigned to the\n>>>> logical partition.\n>>>>\n>>>> This patch will depend on the patch linked below, which is under\n>>>> review.\n>>>>\n>>>> https://patchwork.kernel.org/patch/9882915/\n>>>>\n>>>> Signed-off-by: Bryant G. Ly \n>>>> Signed-off-by: Juan J. Alvarez \n>>>> ---\n>>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n>>>> 1 file changed, 24 insertions(+)\n>>>>\n>>>> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>> index 6b812ad990e4..45946ee90985 100644\n>>>> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>> @@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n>>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n>>>> static int eeh_error_buf_size;\n>>>> \n>>>> +void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n>>>> +{\n>>>> +\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n>>>> +\n>>>> +\tif (!pdev->is_virtfn)\n>>>> +\t\treturn;\n>>>> +\n>>>> +\tpdn->device_id = pdev->device;\n>>>> +\tpdn->vendor_id = pdev->vendor;\n>>>> +\tpdn->class_code = pdev->class;\n>>>> +\n>>>> +\t/*\n>>>> +\t * The following operations will fail if VF's sysfs files\n>>>> +\t * aren't created or its resources aren't finalized.\n>>>> +\t */\n>>>> +\teeh_add_device_early(pdn);\n>>>> +\teeh_add_device_late(pdev);\n>>>> +\teeh_sysfs_add_device(pdev);\n>>>> +\tpdev->match_driver = -1;\n>>> match_driver is a bool, which should be assigned \"true\" or \"false\".\n>> Above he mentioned a dependency on:\n>>\n>> [04/10] PCI: extend pci device match_driver state\n>> https://patchwork.kernel.org/patch/9882915/\n>>\n>>\n>> Which makes it an int.\n> Oh, right, I missed that, thanks.\n>\n>> Or has that patch been rejected or something?\n> I haven't *rejected* it, but it's low on my priority list, so you\n> shouldn't depend on it unless it adds functionality you really need.\n> If I did apply that particular patch, I would want some rework because\n> it currently obfuscates the match_driver logic. There's no clue when\n> reading the code what -1/0/1 mean.\nSo do you prefer enum's? - If so I can make a change for that.\n> Apparently here you *do* want the \"-1 means the PCI core will never\n> set match_driver to 1\" functionality, so maybe you do depend on it.\nWe depend on the patch because we want that ability to never set \nmatch_driver,\nfor SRIOV on PowerVM.\n>\n> If that's the case, how to you ever bind a driver to these VFs? The\n> changelog says you don't want VF drivers to load *immediately*, so I\n> assume you do want them to load eventually.\n>\nThe VF's that get dynamically created within the configure SR-IOV call, \non the\nPseries Platform, wont be matched with a driver. - We do not want it to \nmatch.\n\nThe Power Hypervisor will load the VFs. The VF's will get assigned(by \nthe user)\nvia the HMC or Novalink in this environment which will then trigger PHYP\nto load the VF device node to the device tree.\n\n-Bryant","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yChWm0SYgz9sP1\n\tfor ;\n\tFri, 13 Oct 2017 06:59:32 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752549AbdJLT7a (ORCPT );\n\tThu, 12 Oct 2017 15:59:30 -0400","from mx0a-001b2d01.pphosted.com ([148.163.156.1]:47414 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-OK)\n\tby vger.kernel.org with ESMTP id S1752246AbdJLT73 (ORCPT\n\t); Thu, 12 Oct 2017 15:59:29 -0400","from pps.filterd (m0098410.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9CJxFWt010864\n\tfor ; Thu, 12 Oct 2017 15:59:29 -0400","from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2djbekh18h-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Thu, 12 Oct 2017 15:59:29 -0400","from localhost\n\tby e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tThu, 12 Oct 2017 15:59:28 -0400","from b01cxnp23032.gho.pok.ibm.com (9.57.198.27)\n\tby e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tThu, 12 Oct 2017 15:59:25 -0400","from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com\n\t[9.57.199.108])\n\tby b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9CJxP4G38076544; Thu, 12 Oct 2017 19:59:25 GMT","from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 3B624B2054;\n\tThu, 12 Oct 2017 15:56:42 -0400 (EDT)","from Bryants-MacBook-Pro-2.local (unknown [9.85.188.239])\n\tby b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP id 67489B204E;\n\tThu, 12 Oct 2017 15:56:41 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Bjorn Helgaas , Michael Ellerman ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>","From":"\"Bryant G. Ly\" ","Date":"Thu, 12 Oct 2017 14:59:23 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"7bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101219-0044-0000-0000-0000039F1A80","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007886; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000236; SDB=6.00930209; UDB=6.00468249;\n\tIPR=6.00710476; \n\tBA=6.00005635; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017511;\n\tXFM=3.00000015; UTC=2017-10-12 19:59:27","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101219-0045-0000-0000-000007CE1D1D","Message-Id":"","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-12_10:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710120283","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1785980,"web_url":"http://patchwork.ozlabs.org/comment/1785980/","msgid":"<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-13T03:34:56","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"[+cc Alex, Bodong, Eli, Saeed]\n\nOn Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote:\n> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n> >On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n> >>Bjorn Helgaas writes:\n> >>\n> >>>On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n> >>>>This patch adds the machine dependent call for\n> >>>>pcibios_bus_add_device, since the previous patch\n> >>>>separated the calls out between the PowerNV and PowerVM.\n> >>>>\n> >>>>The difference here is that for the PowerVM environment\n> >>>>we do not want match_driver set because in this environment\n> >>>>we do not want the VF device drivers to load immediately, due to\n> >>>>firmware loading the device node when VF device is assigned to the\n> >>>>logical partition.\n> >>>>\n> >>>>This patch will depend on the patch linked below, which is under\n> >>>>review.\n> >>>>\n> >>>>https://patchwork.kernel.org/patch/9882915/\n> >>>>\n> >>>>Signed-off-by: Bryant G. Ly \n> >>>>Signed-off-by: Juan J. Alvarez \n> >>>>---\n> >>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n> >>>> 1 file changed, 24 insertions(+)\n> >>>>\n> >>>>diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>>>index 6b812ad990e4..45946ee90985 100644\n> >>>>--- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>>>+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>>>@@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n> >>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n> >>>> static int eeh_error_buf_size;\n> >>>>+void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n> >>>>+{\n> >>>>+\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n> >>>>+\n> >>>>+\tif (!pdev->is_virtfn)\n> >>>>+\t\treturn;\n> >>>>+\n> >>>>+\tpdn->device_id = pdev->device;\n> >>>>+\tpdn->vendor_id = pdev->vendor;\n> >>>>+\tpdn->class_code = pdev->class;\n> >>>>+\n> >>>>+\t/*\n> >>>>+\t * The following operations will fail if VF's sysfs files\n> >>>>+\t * aren't created or its resources aren't finalized.\n> >>>>+\t */\n> >>>>+\teeh_add_device_early(pdn);\n> >>>>+\teeh_add_device_late(pdev);\n> >>>>+\teeh_sysfs_add_device(pdev);\n> >>>>+\tpdev->match_driver = -1;\n> >>>match_driver is a bool, which should be assigned \"true\" or \"false\".\n> >>Above he mentioned a dependency on:\n> >>\n> >> [04/10] PCI: extend pci device match_driver state\n> >> https://patchwork.kernel.org/patch/9882915/\n> >>\n> >>\n> >>Which makes it an int.\n> >Oh, right, I missed that, thanks.\n> >\n> >>Or has that patch been rejected or something?\n> >I haven't *rejected* it, but it's low on my priority list, so you\n> >shouldn't depend on it unless it adds functionality you really need.\n> >If I did apply that particular patch, I would want some rework because\n> >it currently obfuscates the match_driver logic. There's no clue when\n> >reading the code what -1/0/1 mean.\n> So do you prefer enum's? - If so I can make a change for that.\n> >Apparently here you *do* want the \"-1 means the PCI core will never\n> >set match_driver to 1\" functionality, so maybe you do depend on it.\n> We depend on the patch because we want that ability to never set\n> match_driver,\n> for SRIOV on PowerVM.\n\nIs this really new PowerVM-specific functionality? ISTR recent discussions\nabout inhibiting driver binding in a generic way, e.g.,\nhttp://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n\n> >If that's the case, how to you ever bind a driver to these VFs? The\n> >changelog says you don't want VF drivers to load *immediately*, so I\n> >assume you do want them to load eventually.\n> >\n> The VF's that get dynamically created within the configure SR-IOV\n> call, on the Pseries Platform, wont be matched with a driver. - We\n> do not want it to match.\n> \n> The Power Hypervisor will load the VFs. The VF's will get\n> assigned(by the user) via the HMC or Novalink in this environment\n> which will then trigger PHYP to load the VF device node to the\n> device tree.\n\nI don't know what it means for the Hypervisor to \"load the VFs.\" Can\nyou explain that in PCI-speak?\n\nThe things I know about are:\n\n - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n - now the VFs respond to config accesses\n - the PCI core enumerates the VFs by reading their config space\n - the PCI core builds pci_dev structs for the VFs\n - the PCI core adds these pci_devs to the bus\n - we try to bind drivers to the VFs\n - the VF driver probe function may read VF config space and VF BARs\n - the VF may be assigned to a guest VM\n\nWhere does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\nor PHYP are. I don't *need* to know what they are, as long as you can\nexplain what's happening in terms of the PCI concepts and generic Linux VMs\nand device assignment.\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yCtdL4jZRz9s7m\n\tfor ;\n\tFri, 13 Oct 2017 14:35:02 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1753323AbdJMDfA (ORCPT );\n\tThu, 12 Oct 2017 23:35:00 -0400","from mail.kernel.org ([198.145.29.99]:40242 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1753305AbdJMDe7 (ORCPT );\n\tThu, 12 Oct 2017 23:34:59 -0400","from localhost (unknown [69.71.4.159])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 6689221877;\n\tFri, 13 Oct 2017 03:34:58 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 6689221877","Date":"Thu, 12 Oct 2017 22:34:56 -0500","From":"Bjorn Helgaas ","To":"\"Bryant G. Ly\" ","Cc":"Michael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ,\n\tAlex Williamson ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1786213,"web_url":"http://patchwork.ozlabs.org/comment/1786213/","msgid":"<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-13T11:53:06","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":68334,"url":"http://patchwork.ozlabs.org/api/people/68334/","name":"Steven Royer","email":"seroyer@linux.vnet.ibm.com"},"content":"On 2017-10-12 22:34, Bjorn Helgaas wrote:\n> [+cc Alex, Bodong, Eli, Saeed]\n> \n> On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote:\n>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n>> >On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n>> >>Bjorn Helgaas writes:\n>> >>\n>> >>>On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n>> >>>>This patch adds the machine dependent call for\n>> >>>>pcibios_bus_add_device, since the previous patch\n>> >>>>separated the calls out between the PowerNV and PowerVM.\n>> >>>>\n>> >>>>The difference here is that for the PowerVM environment\n>> >>>>we do not want match_driver set because in this environment\n>> >>>>we do not want the VF device drivers to load immediately, due to\n>> >>>>firmware loading the device node when VF device is assigned to the\n>> >>>>logical partition.\n>> >>>>\n>> >>>>This patch will depend on the patch linked below, which is under\n>> >>>>review.\n>> >>>>\n>> >>>>https://patchwork.kernel.org/patch/9882915/\n>> >>>>\n>> >>>>Signed-off-by: Bryant G. Ly \n>> >>>>Signed-off-by: Juan J. Alvarez \n>> >>>>---\n>> >>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n>> >>>> 1 file changed, 24 insertions(+)\n>> >>>>\n>> >>>>diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> >>>>index 6b812ad990e4..45946ee90985 100644\n>> >>>>--- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> >>>>+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>> >>>>@@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n>> >>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n>> >>>> static int eeh_error_buf_size;\n>> >>>>+void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n>> >>>>+{\n>> >>>>+\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n>> >>>>+\n>> >>>>+\tif (!pdev->is_virtfn)\n>> >>>>+\t\treturn;\n>> >>>>+\n>> >>>>+\tpdn->device_id = pdev->device;\n>> >>>>+\tpdn->vendor_id = pdev->vendor;\n>> >>>>+\tpdn->class_code = pdev->class;\n>> >>>>+\n>> >>>>+\t/*\n>> >>>>+\t * The following operations will fail if VF's sysfs files\n>> >>>>+\t * aren't created or its resources aren't finalized.\n>> >>>>+\t */\n>> >>>>+\teeh_add_device_early(pdn);\n>> >>>>+\teeh_add_device_late(pdev);\n>> >>>>+\teeh_sysfs_add_device(pdev);\n>> >>>>+\tpdev->match_driver = -1;\n>> >>>match_driver is a bool, which should be assigned \"true\" or \"false\".\n>> >>Above he mentioned a dependency on:\n>> >>\n>> >> [04/10] PCI: extend pci device match_driver state\n>> >> https://patchwork.kernel.org/patch/9882915/\n>> >>\n>> >>\n>> >>Which makes it an int.\n>> >Oh, right, I missed that, thanks.\n>> >\n>> >>Or has that patch been rejected or something?\n>> >I haven't *rejected* it, but it's low on my priority list, so you\n>> >shouldn't depend on it unless it adds functionality you really need.\n>> >If I did apply that particular patch, I would want some rework because\n>> >it currently obfuscates the match_driver logic. There's no clue when\n>> >reading the code what -1/0/1 mean.\n>> So do you prefer enum's? - If so I can make a change for that.\n>> >Apparently here you *do* want the \"-1 means the PCI core will never\n>> >set match_driver to 1\" functionality, so maybe you do depend on it.\n>> We depend on the patch because we want that ability to never set\n>> match_driver,\n>> for SRIOV on PowerVM.\n> \n> Is this really new PowerVM-specific functionality? ISTR recent \n> discussions\n> about inhibiting driver binding in a generic way, e.g.,\n> http://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n> \n>> >If that's the case, how to you ever bind a driver to these VFs? The\n>> >changelog says you don't want VF drivers to load *immediately*, so I\n>> >assume you do want them to load eventually.\n>> >\n>> The VF's that get dynamically created within the configure SR-IOV\n>> call, on the Pseries Platform, wont be matched with a driver. - We\n>> do not want it to match.\n>> \n>> The Power Hypervisor will load the VFs. The VF's will get\n>> assigned(by the user) via the HMC or Novalink in this environment\n>> which will then trigger PHYP to load the VF device node to the\n>> device tree.\n> \n> I don't know what it means for the Hypervisor to \"load the VFs.\" Can\n> you explain that in PCI-speak?\n> \n> The things I know about are:\n> \n> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n> - now the VFs respond to config accesses\n> - the PCI core enumerates the VFs by reading their config space\n> - the PCI core builds pci_dev structs for the VFs\n> - the PCI core adds these pci_devs to the bus\n> - we try to bind drivers to the VFs\n> - the VF driver probe function may read VF config space and VF BARs\n> - the VF may be assigned to a guest VM\n> \n> Where does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\n> or PHYP are. I don't *need* to know what they are, as long as you can\n> explain what's happening in terms of the PCI concepts and generic Linux \n> VMs\n> and device assignment.\n> \n> Bjorn\n\nThe VFs will be hotplugged into the VM separately from the enable \nSR-IOV, so the driver will load as part of the hotplug operation.\n\nSteve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yD5Zq6gp0z9sPm\n\tfor ;\n\tFri, 13 Oct 2017 22:48:35 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1757859AbdJMLsf (ORCPT );\n\tFri, 13 Oct 2017 07:48:35 -0400","from mx0b-001b2d01.pphosted.com ([148.163.158.5]:49688 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-FAIL)\n\tby vger.kernel.org with ESMTP id S1753413AbdJMLse (ORCPT\n\t); Fri, 13 Oct 2017 07:48:34 -0400","from pps.filterd (m0098419.ppops.net [127.0.0.1])\n\tby mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9DBhoAU001644\n\tfor ; Fri, 13 Oct 2017 07:48:33 -0400","from e33.co.us.ibm.com (e33.co.us.ibm.com [32.97.110.151])\n\tby mx0b-001b2d01.pphosted.com with ESMTP id 2djpr3by3v-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Fri, 13 Oct 2017 07:48:33 -0400","from localhost\n\tby e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tFri, 13 Oct 2017 05:48:32 -0600","from b03cxnp07029.gho.boulder.ibm.com (9.17.130.16)\n\tby e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tFri, 13 Oct 2017 05:48:26 -0600","from b03ledav001.gho.boulder.ibm.com\n\t(b03ledav001.gho.boulder.ibm.com [9.17.130.232])\n\tby b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with\n\tESMTP id v9DBmPws5112120; Fri, 13 Oct 2017 04:48:26 -0700","from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 646436E050;\n\tFri, 13 Oct 2017 05:48:26 -0600 (MDT)","from ltc.linux.ibm.com (unknown [9.16.170.189])\n\tby b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTP id F00A56E04A;\n\tFri, 13 Oct 2017 05:48:25 -0600 (MDT)"],"MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII;\n format=flowed","Content-Transfer-Encoding":"7bit","Date":"Fri, 13 Oct 2017 06:53:06 -0500","From":"Steven Royer ","To":"Bjorn Helgaas ","Cc":"\"Bryant G. Ly\" ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tAlex Williamson ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","In-Reply-To":"<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>","X-Sender":"seroyer@linux.vnet.ibm.com","User-Agent":"Roundcube Webmail/1.0.1","X-TM-AS-GCONF":"00","x-cbid":"17101311-0008-0000-0000-000008B52AC0","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007890; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000236; SDB=6.00930526; UDB=6.00468430;\n\tIPR=6.00710791; \n\tBA=6.00005636; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017523;\n\tXFM=3.00000015; UTC=2017-10-13 11:48:30","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101311-0009-0000-0000-00004458A033","Message-Id":"<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-13_03:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710130166","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1786217,"web_url":"http://patchwork.ozlabs.org/comment/1786217/","msgid":"<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-13T12:01:48","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":68334,"url":"http://patchwork.ozlabs.org/api/people/68334/","name":"Steven Royer","email":"seroyer@linux.vnet.ibm.com"},"content":"On 2017-10-13 06:53, Steven Royer wrote:\n> On 2017-10-12 22:34, Bjorn Helgaas wrote:\n>> [+cc Alex, Bodong, Eli, Saeed]\n>> \n>> On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote:\n>>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n>>> >On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n>>> >>Bjorn Helgaas writes:\n>>> >>\n>>> >>>On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n>>> >>>>This patch adds the machine dependent call for\n>>> >>>>pcibios_bus_add_device, since the previous patch\n>>> >>>>separated the calls out between the PowerNV and PowerVM.\n>>> >>>>\n>>> >>>>The difference here is that for the PowerVM environment\n>>> >>>>we do not want match_driver set because in this environment\n>>> >>>>we do not want the VF device drivers to load immediately, due to\n>>> >>>>firmware loading the device node when VF device is assigned to the\n>>> >>>>logical partition.\n>>> >>>>\n>>> >>>>This patch will depend on the patch linked below, which is under\n>>> >>>>review.\n>>> >>>>\n>>> >>>>https://patchwork.kernel.org/patch/9882915/\n>>> >>>>\n>>> >>>>Signed-off-by: Bryant G. Ly \n>>> >>>>Signed-off-by: Juan J. Alvarez \n>>> >>>>---\n>>> >>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n>>> >>>> 1 file changed, 24 insertions(+)\n>>> >>>>\n>>> >>>>diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>> >>>>index 6b812ad990e4..45946ee90985 100644\n>>> >>>>--- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>> >>>>+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>> >>>>@@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n>>> >>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n>>> >>>> static int eeh_error_buf_size;\n>>> >>>>+void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n>>> >>>>+{\n>>> >>>>+\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n>>> >>>>+\n>>> >>>>+\tif (!pdev->is_virtfn)\n>>> >>>>+\t\treturn;\n>>> >>>>+\n>>> >>>>+\tpdn->device_id = pdev->device;\n>>> >>>>+\tpdn->vendor_id = pdev->vendor;\n>>> >>>>+\tpdn->class_code = pdev->class;\n>>> >>>>+\n>>> >>>>+\t/*\n>>> >>>>+\t * The following operations will fail if VF's sysfs files\n>>> >>>>+\t * aren't created or its resources aren't finalized.\n>>> >>>>+\t */\n>>> >>>>+\teeh_add_device_early(pdn);\n>>> >>>>+\teeh_add_device_late(pdev);\n>>> >>>>+\teeh_sysfs_add_device(pdev);\n>>> >>>>+\tpdev->match_driver = -1;\n>>> >>>match_driver is a bool, which should be assigned \"true\" or \"false\".\n>>> >>Above he mentioned a dependency on:\n>>> >>\n>>> >> [04/10] PCI: extend pci device match_driver state\n>>> >> https://patchwork.kernel.org/patch/9882915/\n>>> >>\n>>> >>\n>>> >>Which makes it an int.\n>>> >Oh, right, I missed that, thanks.\n>>> >\n>>> >>Or has that patch been rejected or something?\n>>> >I haven't *rejected* it, but it's low on my priority list, so you\n>>> >shouldn't depend on it unless it adds functionality you really need.\n>>> >If I did apply that particular patch, I would want some rework because\n>>> >it currently obfuscates the match_driver logic. There's no clue when\n>>> >reading the code what -1/0/1 mean.\n>>> So do you prefer enum's? - If so I can make a change for that.\n>>> >Apparently here you *do* want the \"-1 means the PCI core will never\n>>> >set match_driver to 1\" functionality, so maybe you do depend on it.\n>>> We depend on the patch because we want that ability to never set\n>>> match_driver,\n>>> for SRIOV on PowerVM.\n>> \n>> Is this really new PowerVM-specific functionality? ISTR recent \n>> discussions\n>> about inhibiting driver binding in a generic way, e.g.,\n>> http://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n>> \n>>> >If that's the case, how to you ever bind a driver to these VFs? The\n>>> >changelog says you don't want VF drivers to load *immediately*, so I\n>>> >assume you do want them to load eventually.\n>>> >\n>>> The VF's that get dynamically created within the configure SR-IOV\n>>> call, on the Pseries Platform, wont be matched with a driver. - We\n>>> do not want it to match.\n>>> \n>>> The Power Hypervisor will load the VFs. The VF's will get\n>>> assigned(by the user) via the HMC or Novalink in this environment\n>>> which will then trigger PHYP to load the VF device node to the\n>>> device tree.\n>> \n>> I don't know what it means for the Hypervisor to \"load the VFs.\" Can\n>> you explain that in PCI-speak?\n>> \n>> The things I know about are:\n>> \n>> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n>> - now the VFs respond to config accesses\n>> - the PCI core enumerates the VFs by reading their config space\n>> - the PCI core builds pci_dev structs for the VFs\n>> - the PCI core adds these pci_devs to the bus\n>> - we try to bind drivers to the VFs\n>> - the VF driver probe function may read VF config space and VF BARs\n>> - the VF may be assigned to a guest VM\n>> \n>> Where does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\n>> or PHYP are. I don't *need* to know what they are, as long as you can\n>> explain what's happening in terms of the PCI concepts and generic \n>> Linux VMs\n>> and device assignment.\n>> \n>> Bjorn\n> \n> The VFs will be hotplugged into the VM separately from the enable\n> SR-IOV, so the driver will load as part of the hotplug operation.\n> \n> Steve\n\nOne more point of clarification: when the hotplug happens, the VF will \nshow up on a virtual PCI bus that is not directly correlated to the real \nPCI bus that the PF is on. On that virtual PCI bus, the driver will \nmatch because it won't be set to -1.\n\nSteve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yD5ms5Bk1z9s81\n\tfor ;\n\tFri, 13 Oct 2017 22:57:17 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1753406AbdJML5Q (ORCPT );\n\tFri, 13 Oct 2017 07:57:16 -0400","from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50208 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-OK)\n\tby vger.kernel.org with ESMTP id S1753374AbdJML5P (ORCPT\n\t); Fri, 13 Oct 2017 07:57:15 -0400","from pps.filterd (m0098410.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9DBub8e120185\n\tfor ; Fri, 13 Oct 2017 07:57:15 -0400","from e16.ny.us.ibm.com (e16.ny.us.ibm.com [129.33.205.206])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2dju6jy6d9-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Fri, 13 Oct 2017 07:57:14 -0400","from localhost\n\tby e16.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tFri, 13 Oct 2017 07:57:13 -0400","from b01cxnp23032.gho.pok.ibm.com (9.57.198.27)\n\tby e16.ny.us.ibm.com (146.89.104.203) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tFri, 13 Oct 2017 07:57:09 -0400","from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com\n\t[9.57.199.110])\n\tby b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9DBv9M637683316; Fri, 13 Oct 2017 11:57:09 GMT","from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 8841AAE034;\n\tFri, 13 Oct 2017 07:57:48 -0400 (EDT)","from ltc.linux.ibm.com (unknown [9.16.170.189])\n\tby b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP id 4458FAE03B;\n\tFri, 13 Oct 2017 07:57:47 -0400 (EDT)"],"MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII;\n format=flowed","Content-Transfer-Encoding":"7bit","Date":"Fri, 13 Oct 2017 07:01:48 -0500","From":"Steven Royer ","To":"Bjorn Helgaas ","Cc":"\"Bryant G. Ly\" ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tAlex Williamson ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","In-Reply-To":"<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>","X-Sender":"seroyer@linux.vnet.ibm.com","User-Agent":"Roundcube Webmail/1.0.1","X-TM-AS-GCONF":"00","x-cbid":"17101311-0024-0000-0000-000002E2722D","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007890; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000236; SDB=6.00930529; UDB=6.00468432;\n\tIPR=6.00710794; \n\tBA=6.00005636; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017524;\n\tXFM=3.00000015; UTC=2017-10-13 11:57:13","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101311-0025-0000-0000-000045B7C19B","Message-Id":"<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-13_03:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710130168","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1786514,"web_url":"http://patchwork.ozlabs.org/comment/1786514/","msgid":"<20171013120558.1a129183@t450s.home>","list_archive_url":null,"date":"2017-10-13T18:05:58","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":4123,"url":"http://patchwork.ozlabs.org/api/people/4123/","name":"Alex Williamson","email":"alex.williamson@redhat.com"},"content":"On Fri, 13 Oct 2017 07:01:48 -0500\nSteven Royer wrote:\n\n> On 2017-10-13 06:53, Steven Royer wrote:\n> > On 2017-10-12 22:34, Bjorn Helgaas wrote: \n> >> [+cc Alex, Bodong, Eli, Saeed]\n> >> \n> >> On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote: \n> >>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote: \n> >>> >On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote: \n> >>> >>Bjorn Helgaas writes:\n> >>> >> \n> >>> >>>On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote: \n> >>> >>>>This patch adds the machine dependent call for\n> >>> >>>>pcibios_bus_add_device, since the previous patch\n> >>> >>>>separated the calls out between the PowerNV and PowerVM.\n> >>> >>>>\n> >>> >>>>The difference here is that for the PowerVM environment\n> >>> >>>>we do not want match_driver set because in this environment\n> >>> >>>>we do not want the VF device drivers to load immediately, due to\n> >>> >>>>firmware loading the device node when VF device is assigned to the\n> >>> >>>>logical partition.\n> >>> >>>>\n> >>> >>>>This patch will depend on the patch linked below, which is under\n> >>> >>>>review.\n> >>> >>>>\n> >>> >>>>https://patchwork.kernel.org/patch/9882915/\n> >>> >>>>\n> >>> >>>>Signed-off-by: Bryant G. Ly \n> >>> >>>>Signed-off-by: Juan J. Alvarez \n> >>> >>>>---\n> >>> >>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n> >>> >>>> 1 file changed, 24 insertions(+)\n> >>> >>>>\n> >>> >>>>diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>> >>>>index 6b812ad990e4..45946ee90985 100644\n> >>> >>>>--- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>> >>>>+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n> >>> >>>>@@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n> >>> >>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n> >>> >>>> static int eeh_error_buf_size;\n> >>> >>>>+void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n> >>> >>>>+{\n> >>> >>>>+\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n> >>> >>>>+\n> >>> >>>>+\tif (!pdev->is_virtfn)\n> >>> >>>>+\t\treturn;\n> >>> >>>>+\n> >>> >>>>+\tpdn->device_id = pdev->device;\n> >>> >>>>+\tpdn->vendor_id = pdev->vendor;\n> >>> >>>>+\tpdn->class_code = pdev->class;\n> >>> >>>>+\n> >>> >>>>+\t/*\n> >>> >>>>+\t * The following operations will fail if VF's sysfs files\n> >>> >>>>+\t * aren't created or its resources aren't finalized.\n> >>> >>>>+\t */\n> >>> >>>>+\teeh_add_device_early(pdn);\n> >>> >>>>+\teeh_add_device_late(pdev);\n> >>> >>>>+\teeh_sysfs_add_device(pdev);\n> >>> >>>>+\tpdev->match_driver = -1; \n> >>> >>>match_driver is a bool, which should be assigned \"true\" or \"false\". \n> >>> >>Above he mentioned a dependency on:\n> >>> >>\n> >>> >> [04/10] PCI: extend pci device match_driver state\n> >>> >> https://patchwork.kernel.org/patch/9882915/\n> >>> >>\n> >>> >>\n> >>> >>Which makes it an int. \n> >>> >Oh, right, I missed that, thanks.\n> >>> > \n> >>> >>Or has that patch been rejected or something? \n> >>> >I haven't *rejected* it, but it's low on my priority list, so you\n> >>> >shouldn't depend on it unless it adds functionality you really need.\n> >>> >If I did apply that particular patch, I would want some rework because\n> >>> >it currently obfuscates the match_driver logic. There's no clue when\n> >>> >reading the code what -1/0/1 mean. \n> >>> So do you prefer enum's? - If so I can make a change for that. \n> >>> >Apparently here you *do* want the \"-1 means the PCI core will never\n> >>> >set match_driver to 1\" functionality, so maybe you do depend on it. \n> >>> We depend on the patch because we want that ability to never set\n> >>> match_driver,\n> >>> for SRIOV on PowerVM. \n> >> \n> >> Is this really new PowerVM-specific functionality? ISTR recent \n> >> discussions\n> >> about inhibiting driver binding in a generic way, e.g.,\n> >> http://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n> >> \n> >>> >If that's the case, how to you ever bind a driver to these VFs? The\n> >>> >changelog says you don't want VF drivers to load *immediately*, so I\n> >>> >assume you do want them to load eventually.\n> >>> > \n> >>> The VF's that get dynamically created within the configure SR-IOV\n> >>> call, on the Pseries Platform, wont be matched with a driver. - We\n> >>> do not want it to match.\n> >>> \n> >>> The Power Hypervisor will load the VFs. The VF's will get\n> >>> assigned(by the user) via the HMC or Novalink in this environment\n> >>> which will then trigger PHYP to load the VF device node to the\n> >>> device tree. \n> >> \n> >> I don't know what it means for the Hypervisor to \"load the VFs.\" Can\n> >> you explain that in PCI-speak?\n> >> \n> >> The things I know about are:\n> >> \n> >> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n> >> - now the VFs respond to config accesses\n> >> - the PCI core enumerates the VFs by reading their config space\n> >> - the PCI core builds pci_dev structs for the VFs\n> >> - the PCI core adds these pci_devs to the bus\n> >> - we try to bind drivers to the VFs\n> >> - the VF driver probe function may read VF config space and VF BARs\n> >> - the VF may be assigned to a guest VM\n> >> \n> >> Where does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\n> >> or PHYP are. I don't *need* to know what they are, as long as you can\n> >> explain what's happening in terms of the PCI concepts and generic \n> >> Linux VMs\n> >> and device assignment.\n> >> \n> >> Bjorn \n> > \n> > The VFs will be hotplugged into the VM separately from the enable\n> > SR-IOV, so the driver will load as part of the hotplug operation.\n> > \n> > Steve \n> \n> One more point of clarification: when the hotplug happens, the VF will \n> show up on a virtual PCI bus that is not directly correlated to the real \n> PCI bus that the PF is on. On that virtual PCI bus, the driver will \n> match because it won't be set to -1.\n\nI'm pretty lost too, but I think what's being said is that the\nparavirtualized SR-IOV enable creates VFs according to the SR-IOV\noffset and stride capabilities of the PF, but we're supposed to ignore\nthose (why are we even creating pci_devs for them?) and the hypervisor\nwill actually hotplug the VFs somewhere else. How's that still\nSR-IOV? Why wouldn't the hypervisor just add the real VFs that we're\nsupposed to use at the offset and stride indicated by the PF SR-IOV\ncapability and mask the VFs that we're not supposed to see? Thanks,\n\nAlex","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=alex.williamson@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yDFyP0bFxz9sRm\n\tfor ;\n\tSat, 14 Oct 2017 05:06:02 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752509AbdJMSGB (ORCPT );\n\tFri, 13 Oct 2017 14:06:01 -0400","from mx1.redhat.com ([209.132.183.28]:33234 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1752759AbdJMSGA (ORCPT );\n\tFri, 13 Oct 2017 14:06:00 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 2674A81E0C;\n\tFri, 13 Oct 2017 18:06:00 +0000 (UTC)","from t450s.home (ovpn-116-96.phx2.redhat.com [10.3.116.96])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id DEEA85D722;\n\tFri, 13 Oct 2017 18:05:58 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 2674A81E0C","Date":"Fri, 13 Oct 2017 12:05:58 -0600","From":"Alex Williamson ","To":"Steven Royer ","Cc":"Bjorn Helgaas ,\n\t\"Bryant G. Ly\" ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171013120558.1a129183@t450s.home>","In-Reply-To":"<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tFri, 13 Oct 2017 18:06:00 +0000 (UTC)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1786577,"web_url":"http://patchwork.ozlabs.org/comment/1786577/","msgid":"<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-13T19:12:32","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72247,"url":"http://patchwork.ozlabs.org/api/people/72247/","name":"Bryant G. Ly","email":"bryantly@linux.vnet.ibm.com"},"content":"On 10/13/17 1:05 PM, Alex Williamson wrote:\n> On Fri, 13 Oct 2017 07:01:48 -0500\n> Steven Royer wrote:\n>\n>> On 2017-10-13 06:53, Steven Royer wrote:\n>>> On 2017-10-12 22:34, Bjorn Helgaas wrote:\n>>>> [+cc Alex, Bodong, Eli, Saeed]\n>>>>\n>>>> On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote:\n>>>>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n>>>>>> On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n>>>>>>> Bjorn Helgaas writes:\n>>>>>>> \n>>>>>>>> On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n>>>>>>>>> This patch adds the machine dependent call for\n>>>>>>>>> pcibios_bus_add_device, since the previous patch\n>>>>>>>>> separated the calls out between the PowerNV and PowerVM.\n>>>>>>>>>\n>>>>>>>>> The difference here is that for the PowerVM environment\n>>>>>>>>> we do not want match_driver set because in this environment\n>>>>>>>>> we do not want the VF device drivers to load immediately, due to\n>>>>>>>>> firmware loading the device node when VF device is assigned to the\n>>>>>>>>> logical partition.\n>>>>>>>>>\n>>>>>>>>> This patch will depend on the patch linked below, which is under\n>>>>>>>>> review.\n>>>>>>>>>\n>>>>>>>>> https://patchwork.kernel.org/patch/9882915/\n>>>>>>>>>\n>>>>>>>>> Signed-off-by: Bryant G. Ly \n>>>>>>>>> Signed-off-by: Juan J. Alvarez \n>>>>>>>>> ---\n>>>>>>>>> arch/powerpc/platforms/pseries/eeh_pseries.c | 24 ++++++++++++++++++++++++\n>>>>>>>>> 1 file changed, 24 insertions(+)\n>>>>>>>>>\n>>>>>>>>> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>>>>>>> index 6b812ad990e4..45946ee90985 100644\n>>>>>>>>> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>>>>>>> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c\n>>>>>>>>> @@ -64,6 +64,27 @@ static unsigned char slot_errbuf[RTAS_ERROR_LOG_MAX];\n>>>>>>>>> static DEFINE_SPINLOCK(slot_errbuf_lock);\n>>>>>>>>> static int eeh_error_buf_size;\n>>>>>>>>> +void pseries_pcibios_bus_add_device(struct pci_dev *pdev)\n>>>>>>>>> +{\n>>>>>>>>> +\tstruct pci_dn *pdn = pci_get_pdn(pdev);\n>>>>>>>>> +\n>>>>>>>>> +\tif (!pdev->is_virtfn)\n>>>>>>>>> +\t\treturn;\n>>>>>>>>> +\n>>>>>>>>> +\tpdn->device_id = pdev->device;\n>>>>>>>>> +\tpdn->vendor_id = pdev->vendor;\n>>>>>>>>> +\tpdn->class_code = pdev->class;\n>>>>>>>>> +\n>>>>>>>>> +\t/*\n>>>>>>>>> +\t * The following operations will fail if VF's sysfs files\n>>>>>>>>> +\t * aren't created or its resources aren't finalized.\n>>>>>>>>> +\t */\n>>>>>>>>> +\teeh_add_device_early(pdn);\n>>>>>>>>> +\teeh_add_device_late(pdev);\n>>>>>>>>> +\teeh_sysfs_add_device(pdev);\n>>>>>>>>> +\tpdev->match_driver = -1;\n>>>>>>>> match_driver is a bool, which should be assigned \"true\" or \"false\".\n>>>>>>> Above he mentioned a dependency on:\n>>>>>>>\n>>>>>>> [04/10] PCI: extend pci device match_driver state\n>>>>>>> https://patchwork.kernel.org/patch/9882915/\n>>>>>>>\n>>>>>>>\n>>>>>>> Which makes it an int.\n>>>>>> Oh, right, I missed that, thanks.\n>>>>>> \n>>>>>>> Or has that patch been rejected or something?\n>>>>>> I haven't *rejected* it, but it's low on my priority list, so you\n>>>>>> shouldn't depend on it unless it adds functionality you really need.\n>>>>>> If I did apply that particular patch, I would want some rework because\n>>>>>> it currently obfuscates the match_driver logic. There's no clue when\n>>>>>> reading the code what -1/0/1 mean.\n>>>>> So do you prefer enum's? - If so I can make a change for that.\n>>>>>> Apparently here you *do* want the \"-1 means the PCI core will never\n>>>>>> set match_driver to 1\" functionality, so maybe you do depend on it.\n>>>>> We depend on the patch because we want that ability to never set\n>>>>> match_driver,\n>>>>> for SRIOV on PowerVM.\n>>>> Is this really new PowerVM-specific functionality? ISTR recent\n>>>> discussions\n>>>> about inhibiting driver binding in a generic way, e.g.,\n>>>> http://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n>>>> \n>>>>>> If that's the case, how to you ever bind a driver to these VFs? The\n>>>>>> changelog says you don't want VF drivers to load *immediately*, so I\n>>>>>> assume you do want them to load eventually.\n>>>>>> \n>>>>> The VF's that get dynamically created within the configure SR-IOV\n>>>>> call, on the Pseries Platform, wont be matched with a driver. - We\n>>>>> do not want it to match.\n>>>>>\n>>>>> The Power Hypervisor will load the VFs. The VF's will get\n>>>>> assigned(by the user) via the HMC or Novalink in this environment\n>>>>> which will then trigger PHYP to load the VF device node to the\n>>>>> device tree.\n>>>> I don't know what it means for the Hypervisor to \"load the VFs.\" Can\n>>>> you explain that in PCI-speak?\n>>>>\n>>>> The things I know about are:\n>>>>\n>>>> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n>>>> - now the VFs respond to config accesses\n>>>> - the PCI core enumerates the VFs by reading their config space\n>>>> - the PCI core builds pci_dev structs for the VFs\n>>>> - the PCI core adds these pci_devs to the bus\n>>>> - we try to bind drivers to the VFs\n>>>> - the VF driver probe function may read VF config space and VF BARs\n>>>> - the VF may be assigned to a guest VM\n>>>>\n>>>> Where does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\n>>>> or PHYP are. I don't *need* to know what they are, as long as you can\n>>>> explain what's happening in terms of the PCI concepts and generic\n>>>> Linux VMs\n>>>> and device assignment.\n>>>>\n>>>> Bjorn\n>>> The VFs will be hotplugged into the VM separately from the enable\n>>> SR-IOV, so the driver will load as part of the hotplug operation.\n>>>\n>>> Steve\n>> One more point of clarification: when the hotplug happens, the VF will\n>> show up on a virtual PCI bus that is not directly correlated to the real\n>> PCI bus that the PF is on. On that virtual PCI bus, the driver will\n>> match because it won't be set to -1.\nSo lets refer to Bjorn's list of things for SRIOV.\n\n - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n - now the VFs respond to config accesses\n - the PCI core enumerates the VFs by reading their config space\n - the PCI core builds pci_dev structs for the VFs\n - the PCI core adds these pci_devs to the bus\n\nSo everything is the same up to here.\n  - we try to bind drivers to the VFs\n - the VF driver probe function may read VF config space and VF BARs\n - the VF may be assigned to a guest VM\n\nPowerVM environment is very different than traditional KVM in terms of SRIOV.\nIn our environment the VFs are not usable or view-able by the Hosting Partition\nin this case Linux. This is a very important point in that the Host CAN NOT\ndo anything to any of the VFs available.\n\nSo like existing way of enabling SRIOV we still rely on the PF driver to\nenable VFs - but in this case the attachment phase is done via a user\naction via a management console in our case (novalink or hmc) triggered\nevent that will essentially act like a hotplug.\n\nSo in the fine details of that user triggered action the system firmware\nwill bind the VFs, allowing resources to be allocated to the VF.\n- Which essentially does all the attaching as we know it today but is\nmanaged by PHYP not by the kernel.\n> I'm pretty lost too, but I think what's being said is that the\n> paravirtualized SR-IOV enable creates VFs according to the SR-IOV\n> offset and stride capabilities of the PF, but we're supposed to ignore\n> those (why are we even creating pci_devs for them?) and the hypervisor\n> will actually hotplug the VFs somewhere else. How's that still\n> SR-IOV? Why wouldn't the hypervisor just add the real VFs that we're\n> supposed to use at the offset and stride indicated by the PF SR-IOV\n> capability and mask the VFs that we're not supposed to see? Thanks,\n>\n> Alex\n>\nIn our current environment (PSeries) we still need to manage the VFs say\nwhen doing error recovery. Therefore, we need pci_devs to be mapped\nto system resources in this case a PE (Partitionable Endpoint).\nThere are more patches that will map the system resources to a pci_dn\nstructure. This patch was merely the start of separating PowerVM\nand PowerNV configure sriov call.\n\n-Bryant","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yDHRL2MhHz9sDB\n\tfor ;\n\tSat, 14 Oct 2017 06:12:46 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752901AbdJMTMo (ORCPT );\n\tFri, 13 Oct 2017 15:12:44 -0400","from mx0b-001b2d01.pphosted.com ([148.163.158.5]:53540 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-FAIL)\n\tby vger.kernel.org with ESMTP id S1752870AbdJMTMl (ORCPT\n\t); Fri, 13 Oct 2017 15:12:41 -0400","from pps.filterd (m0098416.ppops.net [127.0.0.1])\n\tby mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9DJ9Ek6175989\n\tfor ; Fri, 13 Oct 2017 15:12:41 -0400","from e16.ny.us.ibm.com (e16.ny.us.ibm.com [129.33.205.206])\n\tby mx0b-001b2d01.pphosted.com with ESMTP id 2djyevhm4n-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Fri, 13 Oct 2017 15:12:40 -0400","from localhost\n\tby e16.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tFri, 13 Oct 2017 15:12:40 -0400","from b01cxnp23032.gho.pok.ibm.com (9.57.198.27)\n\tby e16.ny.us.ibm.com (146.89.104.203) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tFri, 13 Oct 2017 15:12:35 -0400","from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com\n\t[9.57.199.111])\n\tby b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9DJCYT434603090; Fri, 13 Oct 2017 19:12:34 GMT","from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 1E605AC040;\n\tFri, 13 Oct 2017 15:13:15 -0400 (EDT)","from bryants-mbp-3.rchland.ibm.com (unknown [9.10.79.61])\n\tby b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP id 6CC9BAC03F;\n\tFri, 13 Oct 2017 15:13:14 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Alex Williamson ,\n\tSteven Royer ","Cc":"Bjorn Helgaas ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed , jjalvare@linux.vnet.ibm.com","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>","From":"\"Bryant G. Ly\" ","Date":"Fri, 13 Oct 2017 14:12:32 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<20171013120558.1a129183@t450s.home>","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101319-0024-0000-0000-000002E298F7","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007892; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000236; SDB=6.00930673; UDB=6.00468519;\n\tIPR=6.00710940; \n\tBA=6.00005636; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017530;\n\tXFM=3.00000015; UTC=2017-10-13 19:12:38","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101319-0025-0000-0000-000045B89A19","Message-Id":"<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-13_08:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710130266","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1787978,"web_url":"http://patchwork.ozlabs.org/comment/1787978/","msgid":"<87r2u2jvfb.fsf@concordia.ellerman.id.au>","list_archive_url":null,"date":"2017-10-17T03:38:16","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":46580,"url":"http://patchwork.ozlabs.org/api/people/46580/","name":"Michael Ellerman","email":"mpe@ellerman.id.au"},"content":"\"Bryant G. Ly\" writes:\n> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n...\n>>\n>> If that's the case, how to you ever bind a driver to these VFs? The\n>> changelog says you don't want VF drivers to load *immediately*, so I\n>> assume you do want them to load eventually.\n>>\n> The VF's that get dynamically created within the configure SR-IOV call, \n> on the Pseries Platform, wont be matched with a driver. - We do not\n> want it to match.\n>\n> The Power Hypervisor will load the VFs. The VF's will get assigned(by \n> the user) > via the HMC or Novalink in this environment which will\n> then trigger PHYP to load the VF device node to the device tree.\n\nWhat about the other \"Power Hypervisor\"? ie. KVM running on Power.\n\nWe also use the pseries platform when running under KVM.\n\ncheers","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGLWM2mwvz9sNr\n\tfor ;\n\tTue, 17 Oct 2017 14:38:23 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1756785AbdJQDiV (ORCPT );\n\tMon, 16 Oct 2017 23:38:21 -0400","from ozlabs.org ([103.22.144.67]:49033 \"EHLO ozlabs.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1754964AbdJQDiU (ORCPT );\n\tMon, 16 Oct 2017 23:38:20 -0400","from authenticated.ozlabs.org (localhost [127.0.0.1])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128\n\tbits)) (No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPSA id 3yGLWG6sHwz9sNr;\n\tTue, 17 Oct 2017 14:38:18 +1100 (AEDT)"],"From":"Michael Ellerman ","To":"\"Bryant G. Ly\" ,\n\tBjorn Helgaas ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","In-Reply-To":"","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t","Date":"Tue, 17 Oct 2017 14:38:16 +1100","Message-ID":"<87r2u2jvfb.fsf@concordia.ellerman.id.au>","MIME-Version":"1.0","Content-Type":"text/plain","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788427,"web_url":"http://patchwork.ozlabs.org/comment/1788427/","msgid":"<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-17T13:51:23","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Fri, Oct 13, 2017 at 02:12:32PM -0500, Bryant G. Ly wrote:\n> \n> \n> On 10/13/17 1:05 PM, Alex Williamson wrote:\n> >On Fri, 13 Oct 2017 07:01:48 -0500\n> >Steven Royer wrote:\n> >\n> >>On 2017-10-13 06:53, Steven Royer wrote:\n> >>>On 2017-10-12 22:34, Bjorn Helgaas wrote:\n> >>>>[+cc Alex, Bodong, Eli, Saeed]\n> >>>>\n> >>>>On Thu, Oct 12, 2017 at 02:59:23PM -0500, Bryant G. Ly wrote:\n> >>>>>On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n> >>>>>>On Thu, Oct 12, 2017 at 03:09:53PM +1100, Michael Ellerman wrote:\n> >>>>>>>Bjorn Helgaas writes:\n> >>>>>>>>On Fri, Sep 22, 2017 at 09:19:28AM -0500, Bryant G. Ly wrote:\n> >>>>>>reading the code what -1/0/1 mean.\n\n> >>>>>>Apparently here you *do* want the \"-1 means the PCI core will never\n> >>>>>>set match_driver to 1\" functionality, so maybe you do depend on it.\n> >>>>>We depend on the patch because we want that ability to never set\n> >>>>>match_driver,\n> >>>>>for SRIOV on PowerVM.\n> >>>>Is this really new PowerVM-specific functionality? ISTR recent\n> >>>>discussions\n> >>>>about inhibiting driver binding in a generic way, e.g.,\n> >>>>http://lkml.kernel.org/r/1490022874-54718-1-git-send-email-bodong@mellanox.com\n> >>>>>>If that's the case, how to you ever bind a driver to these VFs? The\n> >>>>>>changelog says you don't want VF drivers to load *immediately*, so I\n> >>>>>>assume you do want them to load eventually.\n> >>>>>The VF's that get dynamically created within the configure SR-IOV\n> >>>>>call, on the Pseries Platform, wont be matched with a driver. - We\n> >>>>>do not want it to match.\n> >>>>>\n> >>>>>The Power Hypervisor will load the VFs. The VF's will get\n> >>>>>assigned(by the user) via the HMC or Novalink in this environment\n> >>>>>which will then trigger PHYP to load the VF device node to the\n> >>>>>device tree.\n> >>>>I don't know what it means for the Hypervisor to \"load the VFs.\" Can\n> >>>>you explain that in PCI-speak?\n> >>>>\n> >>>>The things I know about are:\n> >>>>\n> >>>> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n> >>>> - now the VFs respond to config accesses\n> >>>> - the PCI core enumerates the VFs by reading their config space\n> >>>> - the PCI core builds pci_dev structs for the VFs\n> >>>> - the PCI core adds these pci_devs to the bus\n> >>>> - we try to bind drivers to the VFs\n> >>>> - the VF driver probe function may read VF config space and VF BARs\n> >>>> - the VF may be assigned to a guest VM\n> >>>>\n> >>>>Where does \"loading the VFs\" fit in? I don't know what HMC, Novalink,\n> >>>>or PHYP are. I don't *need* to know what they are, as long as you can\n> >>>>explain what's happening in terms of the PCI concepts and generic\n> >>>>Linux VMs\n> >>>>and device assignment.\n> >>>>\n> >>>>Bjorn\n> >>>The VFs will be hotplugged into the VM separately from the enable\n> >>>SR-IOV, so the driver will load as part of the hotplug operation.\n> >>>\n> >>>Steve\n> >>One more point of clarification: when the hotplug happens, the VF will\n> >>show up on a virtual PCI bus that is not directly correlated to the real\n> >>PCI bus that the PF is on. On that virtual PCI bus, the driver will\n> >>match because it won't be set to -1.\n> So lets refer to Bjorn's list of things for SRIOV.\n> \n> - we set PCI_SRIOV_CTRL_VFE in the PF, which enables VFs\n> - now the VFs respond to config accesses\n> - the PCI core enumerates the VFs by reading their config space\n> - the PCI core builds pci_dev structs for the VFs\n> - the PCI core adds these pci_devs to the bus\n> \n> So everything is the same up to here.\n>  - we try to bind drivers to the VFs\n> - the VF driver probe function may read VF config space and VF BARs\n> - the VF may be assigned to a guest VM\n> \n> PowerVM environment is very different than traditional KVM in terms\n> of SRIOV. In our environment the VFs are not usable or view-able by\n> the Hosting Partition in this case Linux. This is a very important\n> point in that the Host CAN NOT do anything to any of the VFs\n> available.\n\nThis is where I get confused. I guess the Linux that sets\nPCI_SRIOV_CTRL_VFE to enable the VFs can also perform config accesses\nto the VFs, since it can enumerate them and build pci_dev structs for\nthem, right?\n\nAnd the Linux in the \"Hosting Partition\" is a guest that cannot see a\nVF until a management console attaches the VF to the Hosting\nPartition? I'm not a VFIO or KVM expert but that sounds vaguely like\nwhat they would do when assigning a VF to a guest.\n\n> So like existing way of enabling SRIOV we still rely on the PF driver to\n> enable VFs - but in this case the attachment phase is done via a user\n> action via a management console in our case (novalink or hmc) triggered\n> event that will essentially act like a hotplug.\n> \n> So in the fine details of that user triggered action the system\n> firmware will bind the VFs, allowing resources to be allocated to\n> the VF. - Which essentially does all the attaching as we know it\n> today but is managed by PHYP not by the kernel.\n\nWhat exactly does \"firmware binding the VFs\" mean? I guess this must\nmean assigning a VF to a partition, injecting a hotplug add event to\nthat partition, and making the VF visible in config space?\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGc6n45PBz9s72\n\tfor ;\n\tWed, 18 Oct 2017 00:51:29 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1762588AbdJQNv1 (ORCPT );\n\tTue, 17 Oct 2017 09:51:27 -0400","from mail.kernel.org ([198.145.29.99]:35390 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1762589AbdJQNv0 (ORCPT );\n\tTue, 17 Oct 2017 09:51:26 -0400","from localhost (173-17-104-160.client.mchsi.com [173.17.104.160])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 98C9921871;\n\tTue, 17 Oct 2017 13:51:25 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 98C9921871","Date":"Tue, 17 Oct 2017 08:51:23 -0500","From":"Bjorn Helgaas ","To":"\"Bryant G. Ly\" ","Cc":"Alex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed , jjalvare@linux.vnet.ibm.com","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>","References":"<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788449,"web_url":"http://patchwork.ozlabs.org/comment/1788449/","msgid":"<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-17T14:11:01","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72247,"url":"http://patchwork.ozlabs.org/api/people/72247/","name":"Bryant G. Ly","email":"bryantly@linux.vnet.ibm.com"},"content":"Adding Juan back into the cc: jjalvare@linux.vnet.ibm.com\n\n\nOn 10/16/17 10:38 PM, Michael Ellerman wrote:\n> \"Bryant G. Ly\" writes:\n>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n> ...\n>>> If that's the case, how to you ever bind a driver to these VFs? The\n>>> changelog says you don't want VF drivers to load *immediately*, so I\n>>> assume you do want them to load eventually.\n>>>\n>> The VF's that get dynamically created within the configure SR-IOV call, \n>> on the Pseries Platform, wont be matched with a driver. - We do not\n>> want it to match.\n>>\n>> The Power Hypervisor will load the VFs. The VF's will get assigned(by \n>> the user) > via the HMC or Novalink in this environment which will\n>> then trigger PHYP to load the VF device node to the device tree.\n> What about the other \"Power Hypervisor\"? ie. KVM running on Power.\nThis path is only exercised when configuring SR-IOV for Pseries LPAR,\ntherefore it will not affect PowerNV or KVM(opal). Which is the reason for\nthe separation of calls to the machine dependent stuff.\n> We also use the pseries platform when running under KVM.\n>\n> cheers\n>\nIf anyone plans to enable SR-IOV on Pseries platform, firmware must provide the\nresources to enable the VFs and map them with system resources. A new version\nof the PAPR Document will be added to document these system resources. Lastly,\nwe were not aware that there is an intention to enable SR-IOV in adapters assigned\nto a VM with Pseries running on KVM. Furthermore, this could be left as a todo\nfor the future if this type of configuration is needed.\n\n-Bryant","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGcYk48CQz9s7g\n\tfor ;\n\tWed, 18 Oct 2017 01:11:22 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1762688AbdJQOLV (ORCPT );\n\tTue, 17 Oct 2017 10:11:21 -0400","from mx0b-001b2d01.pphosted.com ([148.163.158.5]:44784 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-FAIL)\n\tby vger.kernel.org with ESMTP id S1751368AbdJQOLU (ORCPT\n\t); Tue, 17 Oct 2017 10:11:20 -0400","from pps.filterd (m0098420.ppops.net [127.0.0.1])\n\tby mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9HEA55e060015\n\tfor ; Tue, 17 Oct 2017 10:11:20 -0400","from e16.ny.us.ibm.com (e16.ny.us.ibm.com [129.33.205.206])\n\tby mx0b-001b2d01.pphosted.com with ESMTP id 2dngf31ykm-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Tue, 17 Oct 2017 10:11:19 -0400","from localhost\n\tby e16.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tTue, 17 Oct 2017 10:11:19 -0400","from b01cxnp22033.gho.pok.ibm.com (9.57.198.23)\n\tby e16.ny.us.ibm.com (146.89.104.203) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tTue, 17 Oct 2017 10:11:03 -0400","from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com\n\t[9.57.199.106])\n\tby b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9HEB2W440370288; Tue, 17 Oct 2017 14:11:02 GMT","from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id DAAF72803E;\n\tTue, 17 Oct 2017 10:10:55 -0400 (EDT)","from bryants-mbp-3.rchland.ibm.com (unknown [9.10.79.61])\n\tby b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP id 784AF2803D;\n\tTue, 17 Oct 2017 10:10:55 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Michael Ellerman , Bjorn Helgaas ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ,\n\tjjalvare@linux.vnet.ibm.com","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<87r2u2jvfb.fsf@concordia.ellerman.id.au>","From":"\"Bryant G. Ly\" ","Date":"Tue, 17 Oct 2017 09:11:01 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<87r2u2jvfb.fsf@concordia.ellerman.id.au>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101714-0024-0000-0000-000002E4116A","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007906; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000237; SDB=6.00932446; UDB=6.00469563;\n\tIPR=6.00712736; \n\tBA=6.00005642; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017576;\n\tXFM=3.00000015; UTC=2017-10-17 14:11:17","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101714-0025-0000-0000-000045C22694","Message-Id":"<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-17_10:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710170198","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788474,"web_url":"http://patchwork.ozlabs.org/comment/1788474/","msgid":"<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-17T14:33:34","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72586,"url":"http://patchwork.ozlabs.org/api/people/72586/","name":"Juan Alvarez","email":"jjalvare@linux.vnet.ibm.com"},"content":"On 10/17/17 8:51 AM, Bjorn Helgaas wrote:\n> This is where I get confused. I guess the Linux that sets\n> PCI_SRIOV_CTRL_VFE to enable the VFs can also perform config accesses\n> to the VFs, since it can enumerate them and build pci_dev structs for\n> them, right?\n\nCorrect, we are not changing anything related to how the VF gets initialized\nin the kernel.\n>\n> And the Linux in the \"Hosting Partition\" is a guest that cannot see a\n> VF until a management console attaches the VF to the Hosting\n> Partition? \n\nCorrect, this is how PHYP does normal adapter assignment.\n> I'm not a VFIO or KVM expert but that sounds vaguely like\n> what they would do when assigning a VF to a guest.\nI do not know the specifics on VFIO and KVM. However, in this\ncase there is no KVM running on the Linux LPAR. PHYP owns all\nthe system resources.\n>\n>> So like existing way of enabling SRIOV we still rely on the PF driver to\n>> enable VFs - but in this case the attachment phase is done via a user\n>> action via a management console in our case (novalink or hmc) triggered\n>> event that will essentially act like a hotplug.\n>>\n>> So in the fine details of that user triggered action the system\n>> firmware will bind the VFs, allowing resources to be allocated to\n>> the VF. - Which essentially does all the attaching as we know it\n>> today but is managed by PHYP not by the kernel.\n> What exactly does \"firmware binding the VFs\" mean? I guess this must\n> mean assigning a VF to a partition, injecting a hotplug add event to\n> that partition, and making the VF visible in config space?\nFirmware basically adds a device node to the device tree as defined\nin the (PAPR) Power Architecture Platform Requirements document.\n\nJuan","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGd3X3rSvz9sP1\n\tfor ;\n\tWed, 18 Oct 2017 01:33:44 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1762850AbdJQOdn (ORCPT );\n\tTue, 17 Oct 2017 10:33:43 -0400","from mx0b-001b2d01.pphosted.com ([148.163.158.5]:59946 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-FAIL)\n\tby vger.kernel.org with ESMTP id S1762870AbdJQOdm (ORCPT\n\t); Tue, 17 Oct 2017 10:33:42 -0400","from pps.filterd (m0098421.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9HEWt76082254\n\tfor ; Tue, 17 Oct 2017 10:33:42 -0400","from e14.ny.us.ibm.com (e14.ny.us.ibm.com [129.33.205.204])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2dnk8920q4-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Tue, 17 Oct 2017 10:33:41 -0400","from localhost\n\tby e14.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tTue, 17 Oct 2017 10:33:41 -0400","from b01cxnp23033.gho.pok.ibm.com (9.57.198.28)\n\tby e14.ny.us.ibm.com (146.89.104.201) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tTue, 17 Oct 2017 10:33:37 -0400","from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com\n\t[9.57.199.107])\n\tby b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9HEXaom62455858; Tue, 17 Oct 2017 14:33:36 GMT","from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id F3AD212403F;\n\tTue, 17 Oct 2017 10:30:45 -0400 (EDT)","from juans-mbp.austin.ibm.com (unknown [9.41.243.12])\n\tby b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id E772612403D;\n\tTue, 17 Oct 2017 10:30:44 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Bjorn Helgaas ,\n\t\"Bryant G. Ly\" ","Cc":"Alex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","References":"<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>","From":"Juan Alvarez ","Date":"Tue, 17 Oct 2017 09:33:34 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101714-0052-0000-0000-000002732E8A","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007906; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000237; SDB=6.00932454; UDB=6.00469567;\n\tIPR=6.00712743; \n\tBA=6.00005642; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017576;\n\tXFM=3.00000015; UTC=2017-10-17 14:33:40","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101714-0053-0000-0000-00005259454F","Message-Id":"<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-17_11:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710170204","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788619,"web_url":"http://patchwork.ozlabs.org/comment/1788619/","msgid":"<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-17T16:26:21","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Tue, Oct 17, 2017 at 09:33:34AM -0500, Juan Alvarez wrote:\n> On 10/17/17 8:51 AM, Bjorn Helgaas wrote:\n> > This is where I get confused. I guess the Linux that sets\n> > PCI_SRIOV_CTRL_VFE to enable the VFs can also perform config accesses\n> > to the VFs, since it can enumerate them and build pci_dev structs for\n> > them, right?\n> \n> Correct, we are not changing anything related to how the VF gets initialized\n> in the kernel.\n> >\n> > And the Linux in the \"Hosting Partition\" is a guest that cannot see a\n> > VF until a management console attaches the VF to the Hosting\n> > Partition? \n> \n> Correct, this is how PHYP does normal adapter assignment.\n> \n> > I'm not a VFIO or KVM expert but that sounds vaguely like\n> > what they would do when assigning a VF to a guest.\n>\n> I do not know the specifics on VFIO and KVM. However, in this\n> case there is no KVM running on the Linux LPAR. PHYP owns all\n> the system resources.\n\nTo pop back up to the top of the stack, I think the main point of this\npatch is to keep from binding a driver to the VFs in the kernel that\nset PCI_SRIOV_CTRL_VFE. This patch does that by setting\npdev->match_driver to -1 in powerpc-specific code.\n\nI think all systems, not just powerpc, want to prevent this VF driver\nbinding, so I hope there's a generic solution that everybody can use.\n\n> >> So like existing way of enabling SRIOV we still rely on the PF driver to\n> >> enable VFs - but in this case the attachment phase is done via a user\n> >> action via a management console in our case (novalink or hmc) triggered\n> >> event that will essentially act like a hotplug.\n> >>\n> >> So in the fine details of that user triggered action the system\n> >> firmware will bind the VFs, allowing resources to be allocated to\n> >> the VF. - Which essentially does all the attaching as we know it\n> >> today but is managed by PHYP not by the kernel.\n> >\n> > What exactly does \"firmware binding the VFs\" mean? I guess this must\n> > mean assigning a VF to a partition, injecting a hotplug add event to\n> > that partition, and making the VF visible in config space?\n>\n> Firmware basically adds a device node to the device tree as defined\n> in the (PAPR) Power Architecture Platform Requirements document.\n\nFrom the point of view of the kernel that consumes this device tree\nand owns the VF, I guess this looks like a hot-add. It would be nice\nif this could be exposed to that kernel by having the firmware inject\na normal PCIe hotplug interrupt, but I'm guessing it's not that\nsimple.\n\nBut if I understand correctly, this patch series isn't concerned with\nthat kernel; it's concerned with the kernel that owns the PF and sets\nPCI_SRIOV_CTRL_VFE.\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGgYd08kKz9s7f\n\tfor ;\n\tWed, 18 Oct 2017 03:26:29 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S934381AbdJQQ00 (ORCPT );\n\tTue, 17 Oct 2017 12:26:26 -0400","from mail.kernel.org ([198.145.29.99]:54786 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S932914AbdJQQ0Z (ORCPT );\n\tTue, 17 Oct 2017 12:26:25 -0400","from localhost (unknown [69.55.156.165])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 4EE53218B2;\n\tTue, 17 Oct 2017 16:26:24 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 4EE53218B2","Date":"Tue, 17 Oct 2017 11:26:21 -0500","From":"Bjorn Helgaas ","To":"Juan Alvarez ","Cc":"\"Bryant G. Ly\" ,\n\tAlex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>","References":"<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>\n\t<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788678,"web_url":"http://patchwork.ozlabs.org/comment/1788678/","msgid":"","list_archive_url":null,"date":"2017-10-17T17:23:01","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72586,"url":"http://patchwork.ozlabs.org/api/people/72586/","name":"Juan Alvarez","email":"jjalvare@linux.vnet.ibm.com"},"content":"On 10/17/17 11:26 AM, Bjorn Helgaas wrote:\n> To pop back up to the top of the stack, I think the main point of this\n> patch is to keep from binding a driver to the VFs in the kernel that\n> set PCI_SRIOV_CTRL_VFE. This patch does that by setting\n> pdev->match_driver to -1 in powerpc-specific code.\nCorrect, to add to your comment, we will only add to the PSeries platform\n (in PowerPC)  configure SR-IOV path.\n> I think all systems, not just powerpc, want to prevent this VF driver\n> binding, so I hope there's a generic solution that everybody can use.\nThe patch that we made this change dependent on:\n\nhttps://patchwork.kernel.org/patch/9882915/\n\nIs a generic form of not binding a pci device to a device driver\nand can be used in another environment if needed.\n>\n>>>> So like existing way of enabling SRIOV we still rely on the PF driver to\n>>>> enable VFs - but in this case the attachment phase is done via a user\n>>>> action via a management console in our case (novalink or hmc) triggered\n>>>> event that will essentially act like a hotplug.\n>>>>\n>>>> So in the fine details of that user triggered action the system\n>>>> firmware will bind the VFs, allowing resources to be allocated to\n>>>> the VF. - Which essentially does all the attaching as we know it\n>>>> today but is managed by PHYP not by the kernel.\n>>> What exactly does \"firmware binding the VFs\" mean? I guess this must\n>>> mean assigning a VF to a partition, injecting a hotplug add event to\n>>> that partition, and making the VF visible in config space?\n>> Firmware basically adds a device node to the device tree as defined\n>> in the (PAPR) Power Architecture Platform Requirements document.\n> From the point of view of the kernel that consumes this device tree\n> and owns the VF, I guess this looks like a hot-add. \nCorrect, this is intended. We want to minimize change and only focus\non configure SR-IOV path in the PF on PSeries platform.\n> It would be nice\n> if this could be exposed to that kernel by having the firmware inject\n> a normal PCIe hotplug interrupt, but I'm guessing it's not that\n> simple.\nI don't understand entirely your suggestion. However, in the case of\ninterrupts PHYP owns all the resources and will be associated accordingly\nwith a partitionable endpoint (PE).\n> But if I understand correctly, this patch series isn't concerned with\n> that kernel; it's concerned with the kernel that owns the PF and sets\n> PCI_SRIOV_CTRL_VFE.\nCorrect, we will enable SR-IOV in the PF through the Linux kernel and map system resources\nfor the new VFs in powerpc platform specific kernel code. Furthermore, not binding\nthe device drivers for the VFs that get mapped within the configure SR-IOV path\nis a requirement for this PSeries SR-IOV environment.\n\nJuan J. Alvarez","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGhq50KJ3z9t2m\n\tfor ;\n\tWed, 18 Oct 2017 04:23:12 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1759392AbdJQRXK (ORCPT );\n\tTue, 17 Oct 2017 13:23:10 -0400","from mx0a-001b2d01.pphosted.com ([148.163.156.1]:52120 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-OK)\n\tby vger.kernel.org with ESMTP id S1757506AbdJQRXJ (ORCPT\n\t); Tue, 17 Oct 2017 13:23:09 -0400","from pps.filterd (m0098399.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9HHL4xi087689\n\tfor ; Tue, 17 Oct 2017 13:23:09 -0400","from e19.ny.us.ibm.com (e19.ny.us.ibm.com [129.33.205.209])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2dnk4wa5ny-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Tue, 17 Oct 2017 13:23:09 -0400","from localhost\n\tby e19.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tTue, 17 Oct 2017 13:23:08 -0400","from b01cxnp23032.gho.pok.ibm.com (9.57.198.27)\n\tby e19.ny.us.ibm.com (146.89.104.206) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tTue, 17 Oct 2017 13:23:03 -0400","from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com\n\t[9.57.199.107])\n\tby b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9HHN2PQ34930822; Tue, 17 Oct 2017 17:23:02 GMT","from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 804F0124047;\n\tTue, 17 Oct 2017 13:20:12 -0400 (EDT)","from juans-mbp.austin.ibm.com (unknown [9.41.243.12])\n\tby b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id 7C158124044;\n\tTue, 17 Oct 2017 13:20:11 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Bjorn Helgaas ","Cc":"\"Bryant G. Ly\" ,\n\tAlex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","References":"<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>\n\t<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>\n\t<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>","From":"Juan Alvarez ","Date":"Tue, 17 Oct 2017 12:23:01 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101717-0056-0000-0000-000003DB245B","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007907; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000237; SDB=6.00932510; UDB=6.00469601;\n\tIPR=6.00712800; \n\tBA=6.00005642; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017576;\n\tXFM=3.00000015; UTC=2017-10-17 17:23:06","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101717-0057-0000-0000-000008122AA8","Message-Id":"","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-17_11:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=0\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710170245","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1788745,"web_url":"http://patchwork.ozlabs.org/comment/1788745/","msgid":"<20171017185242.GG5641@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-17T18:52:42","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Tue, Oct 17, 2017 at 12:23:01PM -0500, Juan Alvarez wrote:\n> On 10/17/17 11:26 AM, Bjorn Helgaas wrote:\n> > To pop back up to the top of the stack, I think the main point of this\n> > patch is to keep from binding a driver to the VFs in the kernel that\n> > set PCI_SRIOV_CTRL_VFE. This patch does that by setting\n> > pdev->match_driver to -1 in powerpc-specific code.\n> Correct, to add to your comment, we will only add to the PSeries platform\n>  (in PowerPC)  configure SR-IOV path.\n> > I think all systems, not just powerpc, want to prevent this VF driver\n> > binding, so I hope there's a generic solution that everybody can use.\n> The patch that we made this change dependent on:\n> \n> https://patchwork.kernel.org/patch/9882915/\n> \n> Is a generic form of not binding a pci device to a device driver\n> and can be used in another environment if needed.\n\nRight. But that patch isn't really on the path to inclusion (mainly\nbecause it's test framework and doesn't fix a bug or add support for\nnew hardware or features).\n\nI'm suggesting that maybe there should be a generic way to prevent\nbinding to VF devices that works for everybody and doesn't require any\narch-specific code at all.\n\n> >>>> So like existing way of enabling SRIOV we still rely on the PF driver to\n> >>>> enable VFs - but in this case the attachment phase is done via a user\n> >>>> action via a management console in our case (novalink or hmc) triggered\n> >>>> event that will essentially act like a hotplug.\n> >>>>\n> >>>> So in the fine details of that user triggered action the system\n> >>>> firmware will bind the VFs, allowing resources to be allocated to\n> >>>> the VF. - Which essentially does all the attaching as we know it\n> >>>> today but is managed by PHYP not by the kernel.\n> >>> What exactly does \"firmware binding the VFs\" mean? I guess this must\n> >>> mean assigning a VF to a partition, injecting a hotplug add event to\n> >>> that partition, and making the VF visible in config space?\n> >> Firmware basically adds a device node to the device tree as defined\n> >> in the (PAPR) Power Architecture Platform Requirements document.\n> > From the point of view of the kernel that consumes this device tree\n> > and owns the VF, I guess this looks like a hot-add. \n> Correct, this is intended. We want to minimize change and only focus\n> on configure SR-IOV path in the PF on PSeries platform.\n> > It would be nice\n> > if this could be exposed to that kernel by having the firmware inject\n> > a normal PCIe hotplug interrupt, but I'm guessing it's not that\n> > simple.\n>\n> I don't understand entirely your suggestion. However, in the case of\n> interrupts PHYP owns all the resources and will be associated accordingly\n> with a partitionable endpoint (PE).\n\nAssume you have a Linux kernel, and PHYP assigns a VF to it. What\nhappens in that Linux kernel? How does it discover this new device?\n\nI'm suggesting that it would be cool if that kernel received a hotplug\ninterrupt from the Downstream Port leading to the new VF, and the\npciehp driver could read the Slot Status register and see that\n\"Presence Detect Changed\" was set. If that happened, the pciehp\ndriver should automatically enumerate the new device and there\nwouldn't be much if any powerpc-specific code in the path.\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGkpP5Mfpz9t30\n\tfor ;\n\tWed, 18 Oct 2017 05:52:45 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1753950AbdJQSwo (ORCPT );\n\tTue, 17 Oct 2017 14:52:44 -0400","from mail.kernel.org ([198.145.29.99]:43468 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1753635AbdJQSwn (ORCPT );\n\tTue, 17 Oct 2017 14:52:43 -0400","from localhost (unknown [69.55.156.165])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 1597A21871;\n\tTue, 17 Oct 2017 18:52:43 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 1597A21871","Date":"Tue, 17 Oct 2017 13:52:42 -0500","From":"Bjorn Helgaas ","To":"Juan Alvarez ","Cc":"\"Bryant G. Ly\" ,\n\tAlex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171017185242.GG5641@bhelgaas-glaptop.roam.corp.google.com>","References":"\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>\n\t<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>\n\t<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1789000,"web_url":"http://patchwork.ozlabs.org/comment/1789000/","msgid":"","list_archive_url":null,"date":"2017-10-17T23:13:35","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72586,"url":"http://patchwork.ozlabs.org/api/people/72586/","name":"Juan Alvarez","email":"jjalvare@linux.vnet.ibm.com"},"content":"On 10/17/17 1:52 PM, Bjorn Helgaas wrote:\n> Right. But that patch isn't really on the path to inclusion (mainly\n> because it's test framework and doesn't fix a bug or add support for\n> new hardware or features).\nI was not aware of this decision and this will cause changes to this patch.\n>\n> I'm suggesting that maybe there should be a generic way to prevent\n> binding to VF devices that works for everybody and doesn't require any\n> arch-specific code at all.\n\nThe patch that you have suggested in kernel 4.12 is also a generic way.\n\nhttps://marc.info/?l=linux-kernel&m=149002335105804&w=2\n\nPerhaps we can use the same constructs that this patch uses at our level. Nonetheless,\nthat will take a rework to this patch and possibly an export of a function to set drivers_autoprobe\nglobally. I know that was frowned upon on first email thread. Let me know if this is an\nacceptable solution.\n>>>>>> So like existing way of enabling SRIOV we still rely on the PF driver to\n>>>>>> enable VFs - but in this case the attachment phase is done via a user\n>>>>>> action via a management console in our case (novalink or hmc) triggered\n>>>>>> event that will essentially act like a hotplug.\n>>>>>>\n>>>>>> So in the fine details of that user triggered action the system\n>>>>>> firmware will bind the VFs, allowing resources to be allocated to\n>>>>>> the VF. - Which essentially does all the attaching as we know it\n>>>>>> today but is managed by PHYP not by the kernel.\n>>>>> What exactly does \"firmware binding the VFs\" mean? I guess this must\n>>>>> mean assigning a VF to a partition, injecting a hotplug add event to\n>>>>> that partition, and making the VF visible in config space?\n>>>> Firmware basically adds a device node to the device tree as defined\n>>>> in the (PAPR) Power Architecture Platform Requirements document.\n>>> From the point of view of the kernel that consumes this device tree\n>>> and owns the VF, I guess this looks like a hot-add. \n>> Correct, this is intended. We want to minimize change and only focus\n>> on configure SR-IOV path in the PF on PSeries platform.\n>>> It would be nice\n>>> if this could be exposed to that kernel by having the firmware inject\n>>> a normal PCIe hotplug interrupt, but I'm guessing it's not that\n>>> simple.\n>> I don't understand entirely your suggestion. However, in the case of\n>> interrupts PHYP owns all the resources and will be associated accordingly\n>> with a partitionable endpoint (PE).\n> Assume you have a Linux kernel, and PHYP assigns a VF to it. What\n> happens in that Linux kernel? \n\nWhen  kernel boots it goes through the lists of buses in the device tree. For each (physical,virtual)\nbus the kernel reads the list of devices. After resources are read and assigned it will probe the devices.\n\nIf the kernel is up and running then this will start its flow through the dlpar add bus code.\nWhich then will read the device node and same operation as boot will entail of going\nthrough the list of devices under the bus.\n\nOne should note that dynamically created VFs in configure SR-IOV path will have is_virtfn\nset and others do not, therefore pcibios_bus_add_device will only be exercised in certain\ncases.\n> How does it discover this new device?\nThis is treated as a normal hot plug operation for the Operating System that\nthe device gets assigned which can be  IBMi, AIX or Linux in this environment.\n>\n> I'm suggesting that it would be cool if that kernel received a hotplug\n> interrupt from the Downstream Port leading to the new VF, and the\n> pciehp driver could read the Slot Status register and see that\n> \"Presence Detect Changed\" was set. If that happened, the pciehp\n> driver should automatically enumerate the new device and there\n> wouldn't be much if any powerpc-specific code in the path.\nTo re-state Steve's earlier comment, PHYP will enumerate the virtual pci bus and is not\ndirectly correlated to the real PCI bus that the PF is on. We have no control of the pci\nbus enumeration for the device node added to the device tree.\n\nJuan J. Alvarez","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGrbZ68Nkz9t3m\n\tfor ;\n\tWed, 18 Oct 2017 10:13:46 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S933366AbdJQXNo (ORCPT );\n\tTue, 17 Oct 2017 19:13:44 -0400","from mx0b-001b2d01.pphosted.com ([148.163.158.5]:41404 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-FAIL)\n\tby vger.kernel.org with ESMTP id S933132AbdJQXNn (ORCPT\n\t); Tue, 17 Oct 2017 19:13:43 -0400","from pps.filterd (m0098421.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9HNAGk4010578\n\tfor ; Tue, 17 Oct 2017 19:13:42 -0400","from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2dnssnucpc-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Tue, 17 Oct 2017 19:13:42 -0400","from localhost\n\tby e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tTue, 17 Oct 2017 19:13:41 -0400","from b01cxnp22036.gho.pok.ibm.com (9.57.198.26)\n\tby e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tTue, 17 Oct 2017 19:13:37 -0400","from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com\n\t[9.57.199.107])\n\tby b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP\n\tid v9HNDaGO49086520; Tue, 17 Oct 2017 23:13:36 GMT","from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id DA25A12403D;\n\tTue, 17 Oct 2017 19:10:46 -0400 (EDT)","from juans-mbp.austin.ibm.com (unknown [9.41.243.12])\n\tby b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id D873512403F;\n\tTue, 17 Oct 2017 19:10:45 -0400 (EDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Bjorn Helgaas ","Cc":"\"Bryant G. Ly\" ,\n\tAlex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","References":"\n\t<20171013033456.GC25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>\n\t<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>\n\t<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171017185242.GG5641@bhelgaas-glaptop.roam.corp.google.com>","From":"Juan Alvarez ","Date":"Tue, 17 Oct 2017 18:13:35 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<20171017185242.GG5641@bhelgaas-glaptop.roam.corp.google.com>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101723-0044-0000-0000-000003A13BC3","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007908; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000237; SDB=6.00932627; UDB=6.00469671;\n\tIPR=6.00712917; \n\tBA=6.00005643; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017582;\n\tXFM=3.00000015; UTC=2017-10-17 23:13:40","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101723-0045-0000-0000-000007D04255","Message-Id":"","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-17_14:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=3\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710170322","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1789038,"web_url":"http://patchwork.ozlabs.org/comment/1789038/","msgid":"","list_archive_url":null,"date":"2017-10-18T01:01:27","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":7621,"url":"http://patchwork.ozlabs.org/api/people/7621/","name":"Alexey Kardashevskiy","email":"aik@ozlabs.ru"},"content":"On 18/10/17 01:11, Bryant G. Ly wrote:\n> Adding Juan back into the cc: jjalvare@linux.vnet.ibm.com\n> \n> \n> On 10/16/17 10:38 PM, Michael Ellerman wrote:\n>> \"Bryant G. Ly\" writes:\n>>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n>> ...\n>>>> If that's the case, how to you ever bind a driver to these VFs? The\n>>>> changelog says you don't want VF drivers to load *immediately*, so I\n>>>> assume you do want them to load eventually.\n>>>>\n>>> The VF's that get dynamically created within the configure SR-IOV call, \n>>> on the Pseries Platform, wont be matched with a driver. - We do not\n>>> want it to match.\n>>>\n>>> The Power Hypervisor will load the VFs. The VF's will get assigned(by \n>>> the user) > via the HMC or Novalink in this environment which will\n>>> then trigger PHYP to load the VF device node to the device tree.\n>> What about the other \"Power Hypervisor\"? ie. KVM running on Power.\n> This path is only exercised when configuring SR-IOV for Pseries LPAR,\n> therefore it will not affect PowerNV or KVM(opal).\n\nPowerNV KVM guest is a pseries machine so this code will execute there.\n\n> Which is the reason for\n> the separation of calls to the machine dependent stuff.\n>> We also use the pseries platform when running under KVM.\n>>\n>> cheers\n>>\n> If anyone plans to enable SR-IOV on Pseries platform, firmware must provide the\n> resources to enable the VFs and map them with system resources.\n\nThis is what the PowerNV platform does.\n\n> A new version\n> of the PAPR Document will be added to document these system resources.\n\nThe guest simply gets yet another PCI device, how is IOV different here?\n\nIn regard of EEH, the API does not change afaik, it is up to the hypervisor\n(KVM+QEMU) to handle IOV case correctly.\n\n\n> Lastly,\n> we were not aware that there is an intention to enable SR-IOV in adapters assigned\n> to a VM with Pseries running on KVM.\n\nThere is no any special enablement of IOV for a VM on powernv, once\nconfigured in the powernv host, we can just pass VFs to QEMU and therefore\nto a pseries guest, it is just a normal PCI device.\n\nDo you assign a PF to a VM and create VFs from inside the VM? Or only pHyp\nis allowed to do that? Sorry, I know nothing about pHyp on this matter.\n\n\n> Furthermore, this could be left as a todo\n> for the future if this type of configuration is needed.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=ozlabs-ru.20150623.gappssmtp.com\n\theader.i=@ozlabs-ru.20150623.gappssmtp.com\n\theader.b=\"t+6pRJPF\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGv021brvz9rxj\n\tfor ;\n\tWed, 18 Oct 2017 12:01:38 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752108AbdJRBBg (ORCPT );\n\tTue, 17 Oct 2017 21:01:36 -0400","from mail-pf0-f177.google.com ([209.85.192.177]:47327 \"EHLO\n\tmail-pf0-f177.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750920AbdJRBBf (ORCPT\n\t); Tue, 17 Oct 2017 21:01:35 -0400","by mail-pf0-f177.google.com with SMTP id z11so2647629pfk.4\n\tfor ; Tue, 17 Oct 2017 18:01:35 -0700 (PDT)","from [10.61.2.175] ([122.99.82.10])\n\tby smtp.googlemail.com with ESMTPSA id\n\tv15sm20283340pfa.50.2017.10.17.18.01.30\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tTue, 17 Oct 2017 18:01:34 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ozlabs-ru.20150623.gappssmtp.com; s=20150623;\n\th=subject:to:cc:references:from:message-id:date:user-agent\n\t:mime-version:in-reply-to:content-language:content-transfer-encoding; \n\tbh=IIFsxYhb0L6mtgDPGIe5hxYTe4LLY5eCxxBR+M4cLUY=;\n\tb=t+6pRJPFzHVleqIJeyZ4QcvPznY1xL2FuIvmsCsy3xokD7W6Rl6QkTbwLVwtPQ7zBZ\n\tdXNNx6gxC3MVAbZbkOrwCawMpekSaqUTIIxz9G+Geg8UF21NTAeBCSaXRJX8VX/S091Z\n\tcbhHjI/OnvyiiB/m1m/GBnPTY+NDkmA48KVIjFesmiNsccZEaYQ+QAWy8isYSWV8O8ez\n\tWPKmg0XqriMnRmG8vsYHaTi2qO8Nq3GavVxQLjEcyCt72DBDSKLKpDy4DsDLmvu8fsU1\n\txa9aRs40vj97Pa+umzG4gH15kY3WRBq7LMgphvBVp22hjjNapwbdf8JOPmGr/WQCOfdx\n\te1YA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:to:cc:references:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-language\n\t:content-transfer-encoding;\n\tbh=IIFsxYhb0L6mtgDPGIe5hxYTe4LLY5eCxxBR+M4cLUY=;\n\tb=iqyL7/zCm62ohjjlzfCpVy4Cdqas07LeLyMgQBN99v6N78S/r32taC8LueEbT00sWZ\n\tkJXt4t96HnlMOMDuUJVaIthiwjY1jTcMdd5XoMTDX3EJEeogFgcUkDi6fV2mzXIW8Idr\n\te5TkS/8/qMss1zf+tYLBZN3pM0pgoDiMxdT1g9dxmX6l1qaN3mRJ12aaEk8Waf8S0Bfl\n\tfG6gG4v13ilQj9a/EweBDAOs0avp+0czzwHQgFx5HE+mNl/o+mbe6L/0rFVR88xV+fqr\n\teUdUISUDW4WFef6YPZXv+0HygU6NGvv38dpP9Ul68cgX5p0DgvJolBtUKgp84Hs5g/IW\n\t+QFw==","X-Gm-Message-State":"AMCzsaUAfBlEk0KraqsF//d6+XUTR7+xAYZlP0XlUjvwYV0uSF8ijtxK\n\tkBRdkaSU7I8R/ihfSATOAtl0Nw==","X-Google-Smtp-Source":"AOwi7QBIHVWIH1C5dOXDQNnBu6xY0ykGl3NUA7ue7hqngKuzSsiSvFREbcWbi9+8/Kh5Ug+vQqTXbg==","X-Received":"by 10.99.134.200 with SMTP id x191mr12003447pgd.86.1508288494847;\n\tTue, 17 Oct 2017 18:01:34 -0700 (PDT)","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"\"Bryant G. Ly\" ,\n\tMichael Ellerman , Bjorn Helgaas ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ,\n\tjjalvare@linux.vnet.ibm.com","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<87r2u2jvfb.fsf@concordia.ellerman.id.au>\n\t<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>","From":"Alexey Kardashevskiy ","Message-ID":"","Date":"Wed, 18 Oct 2017 12:01:27 +1100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.2.1","MIME-Version":"1.0","In-Reply-To":"<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-AU","Content-Transfer-Encoding":"7bit","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1789053,"web_url":"http://patchwork.ozlabs.org/comment/1789053/","msgid":"<5b7b8e8f-2d16-875f-ef53-ccf593064a53@ozlabs.ru>","list_archive_url":null,"date":"2017-10-18T01:36:30","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":7621,"url":"http://patchwork.ozlabs.org/api/people/7621/","name":"Alexey Kardashevskiy","email":"aik@ozlabs.ru"},"content":"On 18/10/17 12:01, Alexey Kardashevskiy wrote:\n> On 18/10/17 01:11, Bryant G. Ly wrote:\n>> Adding Juan back into the cc: jjalvare@linux.vnet.ibm.com\n>>\n>>\n>> On 10/16/17 10:38 PM, Michael Ellerman wrote:\n>>> \"Bryant G. Ly\" writes:\n>>>> On 10/12/17 1:29 PM, Bjorn Helgaas wrote:\n>>> ...\n>>>>> If that's the case, how to you ever bind a driver to these VFs? The\n>>>>> changelog says you don't want VF drivers to load *immediately*, so I\n>>>>> assume you do want them to load eventually.\n>>>>>\n>>>> The VF's that get dynamically created within the configure SR-IOV call, \n>>>> on the Pseries Platform, wont be matched with a driver. - We do not\n>>>> want it to match.\n>>>>\n>>>> The Power Hypervisor will load the VFs. The VF's will get assigned(by \n>>>> the user) > via the HMC or Novalink in this environment which will\n>>>> then trigger PHYP to load the VF device node to the device tree.\n>>> What about the other \"Power Hypervisor\"? ie. KVM running on Power.\n>> This path is only exercised when configuring SR-IOV for Pseries LPAR,\n>> therefore it will not affect PowerNV or KVM(opal).\n> \n> PowerNV KVM guest is a pseries machine so this code will execute there.\n> \n>> Which is the reason for\n>> the separation of calls to the machine dependent stuff.\n>>> We also use the pseries platform when running under KVM.\n>>>\n>>> cheers\n>>>\n>> If anyone plans to enable SR-IOV on Pseries platform, firmware must provide the\n>> resources to enable the VFs and map them with system resources.\n> \n> This is what the PowerNV platform does.\n> \n>> A new version\n>> of the PAPR Document will be added to document these system resources.\n> \n> The guest simply gets yet another PCI device, how is IOV different here?\n> \n> In regard of EEH, the API does not change afaik, it is up to the hypervisor\n> (KVM+QEMU) to handle IOV case correctly.\n> \n> \n>> Lastly,\n>> we were not aware that there is an intention to enable SR-IOV in adapters assigned\n>> to a VM with Pseries running on KVM.\n> \n> There is no any special enablement of IOV for a VM on powernv, once\n> configured in the powernv host, we can just pass VFs to QEMU and therefore\n> to a pseries guest, it is just a normal PCI device.\n> \n> Do you assign a PF to a VM and create VFs from inside the VM? Or only pHyp\n> is allowed to do that? Sorry, I know nothing about pHyp on this matter.\n> \n\nNever mind this last question, I've just read the entire thread about this\nhosting partition thing.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=ozlabs-ru.20150623.gappssmtp.com\n\theader.i=@ozlabs-ru.20150623.gappssmtp.com\n\theader.b=\"RSG1O4b/\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yGvmR3jxzz9t3H\n\tfor ;\n\tWed, 18 Oct 2017 12:36:39 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1756179AbdJRBgi (ORCPT );\n\tTue, 17 Oct 2017 21:36:38 -0400","from mail-pg0-f46.google.com ([74.125.83.46]:55472 \"EHLO\n\tmail-pg0-f46.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1755377AbdJRBgh (ORCPT\n\t); Tue, 17 Oct 2017 21:36:37 -0400","by mail-pg0-f46.google.com with SMTP id y184so1709966pgd.12\n\tfor ; Tue, 17 Oct 2017 18:36:37 -0700 (PDT)","from [10.61.2.175] ([122.99.82.10])\n\tby smtp.googlemail.com with ESMTPSA id\n\tj6sm22398622pfk.159.2017.10.17.18.36.32\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tTue, 17 Oct 2017 18:36:36 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ozlabs-ru.20150623.gappssmtp.com; s=20150623;\n\th=subject:from:to:cc:references:message-id:date:user-agent\n\t:mime-version:in-reply-to:content-language:content-transfer-encoding; \n\tbh=6vnlcbqxyLRAhFprojbzqlmBZQtxDX7sC+hSlEqm6cc=;\n\tb=RSG1O4b/h/ntDZND0x9uxTl8CfNFGgrlzhYXVDov+C9oPlaljw/2h7oU/H/cYZV7TO\n\tgpTOQmYWNf1KogUNZK1mJ9qOGFukchj6UzIKN9dijU6QQr89lXUdIhbeIBOdwCidJeAF\n\twEfpim5xUjO013qal7zVqQRXfaBxB8tWx10AKlXBKL+WAf0VLMEokY2U8/JfBa+iXzO7\n\tB+zAuch1jw8eyTfjyybEaFk3MN0AGtzX43enINtz4iOJ0t8fMtu17iAv+Py2v/+H5qz4\n\tyAn7atIX1zL+yE7wIMMHTtIoCqIzwLUo9En5fPCtXnH1Z5Ww5vOPxYKJ0jcEKnycPr1P\n\tHGYA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:from:to:cc:references:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-language\n\t:content-transfer-encoding;\n\tbh=6vnlcbqxyLRAhFprojbzqlmBZQtxDX7sC+hSlEqm6cc=;\n\tb=fmakCyQU+hvc3zbSv+qBP85YYa4Q+e3p7Zgdx6FQ3xeinnFp5906Nqglh3z/DddPrX\n\tP4SSWi0+hJyd4mxMi46OrOTtJg1VqxpZHXnmlqT0YL+UdqXzMPUfaproAqdxFNDg/5y2\n\t5NTH1pSJ4EYy6en5Dvmr9F2zkh4W0n0i2JonADTFwrJqr7QckrbxOW2K4QqmyLxTaWFb\n\tJA9Ieo6Cp+mmUfK+A7KC44QuIw3+tfb77iwp0hW0Qa2yJfAsvGEG5ckI4bftt454tgtd\n\tNRGfEnykTJsQJQ3DprjXNAroRJusr+ikiAA8XydPm7SWppoA+GcCCLwEyBD14odQCwLO\n\t2mtA==","X-Gm-Message-State":"AMCzsaXuyFpB8+uYFnY/x0tZJocn0rD/Qt+E8NpmJ0YhYEUNfChKg0QA\n\tTUufqdnq5jR+ypGwfkQLW52rNQC3","X-Google-Smtp-Source":"AOwi7QAwmdMWmRQRM6m0XB+7NxSEr4PUmiWJicE8O4Q7u7YkYUcLPJba9h51A+Nf/BNVqka7e8XLBw==","X-Received":"by 10.84.136.36 with SMTP id 33mr13960518plk.108.1508290596937; \n\tTue, 17 Oct 2017 18:36:36 -0700 (PDT)","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","From":"Alexey Kardashevskiy ","To":"\"Bryant G. Ly\" ,\n\tMichael Ellerman , Bjorn Helgaas ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ,\n\tjjalvare@linux.vnet.ibm.com","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<87r2u2jvfb.fsf@concordia.ellerman.id.au>\n\t<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>\n\t","Message-ID":"<5b7b8e8f-2d16-875f-ef53-ccf593064a53@ozlabs.ru>","Date":"Wed, 18 Oct 2017 12:36:30 +1100","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.2.1","MIME-Version":"1.0","In-Reply-To":"","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-AU","Content-Transfer-Encoding":"7bit","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1789554,"web_url":"http://patchwork.ozlabs.org/comment/1789554/","msgid":"<9893e011-d524-06b6-b041-3f1b24e5f03d@linux.vnet.ibm.com>","list_archive_url":null,"date":"2017-10-18T13:20:29","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":72586,"url":"http://patchwork.ozlabs.org/api/people/72586/","name":"Juan Alvarez","email":"jjalvare@linux.vnet.ibm.com"},"content":"On 10/17/17 8:36 PM, Alexey Kardashevskiy wrote:\n> PowerNV KVM guest is a pseries machine so this code will execute there.\n>\nThe configure sriov path will fail and not enable sriov if resources are\nnot met. I.e. the IOV Bar is not set in PF IOV Resources, which in this\ncase gets assigned by firmware.\n\nWe have separated the calls to put PowerNV and PSeries as machine dependent\ncalls. Furthermore, we are adding device node properties in the device tree to identify if\nthis is an SR-IOV slot on Phyp Pseries platform. Verification will be in place to\ndistinguish between platforms that support SR-IOV in PSeries.\n>> Which is the reason for\n>> the separation of calls to the machine dependent stuff.\n>>> We also use the pseries platform when running under KVM.\n>>>\n>>> cheers\n>>>\n>> If anyone plans to enable SR-IOV on Pseries platform, firmware must provide the\n>> resources to enable the VFs and map them with system resources.\n> This is what the PowerNV platform does.\nAgain, we have separated the machine dependent calls to different platforms. In our\ncase we don't use opal and our dependent on phyp to associate resources.\n>\n>> A new version\n>> of the PAPR Document will be added to document these system resources.\n> The guest simply gets yet another PCI device, how is IOV different here?\n>\n> In regard of EEH, the API does not change afaik, it is up to the hypervisor\n> (KVM+QEMU) to handle IOV case correctly.\n\nI don't understand your question entirely, can you rephrase?","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yHCNz2CYYz9t5Q\n\tfor ;\n\tThu, 19 Oct 2017 00:20:51 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752358AbdJRNUr (ORCPT );\n\tWed, 18 Oct 2017 09:20:47 -0400","from mx0a-001b2d01.pphosted.com ([148.163.156.1]:47516 \"EHLO\n\tmx0a-001b2d01.pphosted.com\" rhost-flags-OK-OK-OK-OK)\n\tby vger.kernel.org with ESMTP id S1751988AbdJRNUh (ORCPT\n\t); Wed, 18 Oct 2017 09:20:37 -0400","from pps.filterd (m0098410.ppops.net [127.0.0.1])\n\tby mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv9IDJYF0087265\n\tfor ; Wed, 18 Oct 2017 09:20:35 -0400","from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.153])\n\tby mx0a-001b2d01.pphosted.com with ESMTP id 2dp2nmcjj2-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Wed, 18 Oct 2017 09:20:35 -0400","from localhost\n\tby e35.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ;\n\tWed, 18 Oct 2017 07:20:34 -0600","from b03cxnp08025.gho.boulder.ibm.com (9.17.130.17)\n\tby e35.co.us.ibm.com (192.168.1.135) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tWed, 18 Oct 2017 07:20:30 -0600","from b03ledav005.gho.boulder.ibm.com\n\t(b03ledav005.gho.boulder.ibm.com [9.17.130.236])\n\tby b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with\n\tESMTP id v9IDKUN865143010; Wed, 18 Oct 2017 06:20:30 -0700","from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1])\n\tby IMSVA (Postfix) with ESMTP id 5E285BE05A;\n\tWed, 18 Oct 2017 07:20:30 -0600 (MDT)","from juans-mbp.austin.ibm.com (unknown [9.41.243.12])\n\tby b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP id AED72BE059;\n\tWed, 18 Oct 2017 07:20:29 -0600 (MDT)"],"Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","To":"Alexey Kardashevskiy ,\n\t\"Bryant G. Ly\" ,\n\tMichael Ellerman , Bjorn Helgaas ","Cc":"bhelgaas@google.com, benh@kernel.crashing.org, paulus@samba.org,\n\tseroyer@linux.vnet.ibm.com, linux-pci@vger.kernel.org,\n\tlinuxppc-dev@lists.ozlabs.org, \"Juan J . Alvarez\" ","References":"<20170922141928.49141-1-bryantly@linux.vnet.ibm.com>\n\t<20170922141928.49141-3-bryantly@linux.vnet.ibm.com>\n\t<20171011200524.GR25517@bhelgaas-glaptop.roam.corp.google.com>\n\t<87efq92ei6.fsf@concordia.ellerman.id.au>\n\t<20171012182932.GA653@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<87r2u2jvfb.fsf@concordia.ellerman.id.au>\n\t<3507ee2e-00ee-7bd0-36a3-292af5112734@linux.vnet.ibm.com>\n\t\n\t<5b7b8e8f-2d16-875f-ef53-ccf593064a53@ozlabs.ru>","From":"Juan Alvarez ","Date":"Wed, 18 Oct 2017 08:20:29 -0500","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.4.0","MIME-Version":"1.0","In-Reply-To":"<5b7b8e8f-2d16-875f-ef53-ccf593064a53@ozlabs.ru>","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"7bit","Content-Language":"en-US","X-TM-AS-GCONF":"00","x-cbid":"17101813-0012-0000-0000-00001529C99C","X-IBM-SpamModules-Scores":"","X-IBM-SpamModules-Versions":"BY=3.00007914; HX=3.00000241; KW=3.00000007;\n\tPH=3.00000004; SC=3.00000237; SDB=6.00932909; UDB=6.00469841;\n\tIPR=6.00713199; \n\tBA=6.00005646; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009;\n\tZB=6.00000000; \n\tZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017590;\n\tXFM=3.00000015; UTC=2017-10-18 13:20:32","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17101813-0013-0000-0000-00004FEC6981","Message-Id":"<9893e011-d524-06b6-b041-3f1b24e5f03d@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-10-18_05:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=18\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000\n\tdefinitions=main-1710180186","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1795130,"web_url":"http://patchwork.ozlabs.org/comment/1795130/","msgid":"<20171027213039.GB9235@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-10-27T21:30:39","subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Tue, Oct 17, 2017 at 06:13:35PM -0500, Juan Alvarez wrote:\n> On 10/17/17 1:52 PM, Bjorn Helgaas wrote:\n> > I'm suggesting that maybe there should be a generic way to prevent\n> > binding to VF devices that works for everybody and doesn't require any\n> > arch-specific code at all.\n> \n> The patch that you have suggested in kernel 4.12 is also a generic way.\n> \n> https://marc.info/?l=linux-kernel&m=149002335105804&w=2\n\nThe patch mentioned above is now upstream:\n\nhttp://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0e7df22401a3\n\n> Perhaps we can use the same constructs that this patch uses at our level.\n> Nonetheless, that will take a rework to this patch and possibly an export\n> of a function to set drivers_autoprobe globally. I know that was frowned\n> upon on first email thread. Let me know if this is an acceptable\n> solution.\n\nI would definitely like to see you use a solution that is somehow\nconnected with the sriov->drivers_autoprobe mechanism added by\n0e7df22401a3.\n\n0e7df22401a3 is a per-PF solution, and you're looking for a\nsystem-wide switch. I could imagine a system-wide flag exposed in\nsysfs and used by pci_device_can_probe(). Or maybe some arch hook in\nthe pci_device_add() path that clears sriov->drivers_autoprobe for\nevery PF.\n\nI think the important thing is that it is coordinated as much as\npossible with other driver binding controls and not unnecessarily\narch-dependent.\n\nBjorn","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3yNxrG29qtz9t4X\n\tfor ;\n\tSat, 28 Oct 2017 08:30:54 +1100 (AEDT)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S932411AbdJ0Vaw (ORCPT );\n\tFri, 27 Oct 2017 17:30:52 -0400","from mail.kernel.org ([198.145.29.99]:51662 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751189AbdJ0Vaw (ORCPT );\n\tFri, 27 Oct 2017 17:30:52 -0400","from localhost (unknown [69.71.4.159])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id 39006218A5;\n\tFri, 27 Oct 2017 21:30:49 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org 39006218A5","Date":"Fri, 27 Oct 2017 16:30:39 -0500","From":"Bjorn Helgaas ","To":"Juan Alvarez ","Cc":"\"Bryant G. Ly\" ,\n\tAlex Williamson ,\n\tSteven Royer ,\n\tMichael Ellerman , bhelgaas@google.com,\n\tbenh@kernel.crashing.org, paulus@samba.org,\n\tlinux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,\n\t\"Juan J . Alvarez\" ,\n\tBodong Wang , Eli Cohen ,\n\tSaeed Mahameed ","Subject":"Re: [PATCH v3 2/2] pseries/eeh: Add Pseries pcibios_bus_add_device","Message-ID":"<20171027213039.GB9235@bhelgaas-glaptop.roam.corp.google.com>","References":"<88eab212351a662dda330071d9afe9dc@linux.vnet.ibm.com>\n\t<11c15c773aefcc0490e197051c2eaf41@linux.vnet.ibm.com>\n\t<20171013120558.1a129183@t450s.home>\n\t<24fdec01-8df4-f027-0b3c-0472a2d73946@linux.vnet.ibm.com>\n\t<20171017135123.GA5641@bhelgaas-glaptop.roam.corp.google.com>\n\t<7e23c703-750b-b4f8-f252-0c872a40ae8a@linux.vnet.ibm.com>\n\t<20171017162621.GB5641@bhelgaas-glaptop.roam.corp.google.com>\n\t\n\t<20171017185242.GG5641@bhelgaas-glaptop.roam.corp.google.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}}]