[{"id":1759640,"web_url":"http://patchwork.ozlabs.org/comment/1759640/","msgid":"<20170829201631.GO8154@bhelgaas-glaptop.roam.corp.google.com>","list_archive_url":null,"date":"2017-08-29T20:16:31","subject":"Re: [PATCH] PCI/portdrv: Fix MSI/MSI-X bug for PCIe port service\n\tdrivers","submitter":{"id":67298,"url":"http://patchwork.ozlabs.org/api/people/67298/","name":"Bjorn Helgaas","email":"helgaas@kernel.org"},"content":"On Fri, Aug 25, 2017 at 05:50:42PM +0800, Dongdong Liu wrote:\n> Current code is broken as calling pci_free_irq_vectors()\n> invalidates the IRQ numbers returned before by pci_irq_vectors();\n> so we need to move all the assignment of the Linux IRQ numbers at\n> the bottom of the function.\n> \n> After removing and adding back the PCI root port device,\n> we see the PCIe port service drivers request irq failed.\n> \n> root@(none)$ lspci -tv\n> -[0000:00]-+-00.0-[01]----00.0 Device 19e5:0123\n> \\-08.0-[02-03]--+-00.0 Device 8086:10fb\n> \\-00.1 Device 8086:10fb\n\nIt doesn't look like the hierarchy *below* the root port is relevant.\n\n> root@(none)$ echo 1 > /sys/devices/pci0000\\:00/0000\\:00\\:00.0/remove\n> iommu: Removing device 0000:00:00.0 from group 2\n> root@(none)$ echo 1 > /sys/devices/pci0000\\:00/pci_bus/0000\\:00/rescan\n> pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it\n> with 'pcie_aspm=force'\n> pci 0000:00:00.0: BAR 14: assigned [mem 0xe0100000-0xe03fffff]\n> pci 0000:00:00.0: BAR 15: assigned [mem 0x80000e00000-0x80000ffffff 64bit\n> pref]\n> pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]\n> pci 0000:01:00.0: BAR 0: assigned [mem 0xe0100000-0xe013ffff 64bit]\n> pci 0000:01:00.0: BAR 6: assigned [mem 0xe0140000-0xe015ffff pref]\n> pci 0000:00:00.0: PCI bridge to [bus 01]\n> pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]\n> pci 0000:00:00.0: bridge window [mem 0xe0100000-0xe03fffff]\n> pci 0000:00:00.0: bridge window [mem 0x80000e00000-0x80000ffffff 64bit\n> pref]\n> iommu: Adding device 0000:00:00.0 to group 2\n> pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22\n> aer: probe of 0000:00:00.0:pcie002 failed with error -22\n> pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd-\n> PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+\n> pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller\n> pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)\n> dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22\n> dpc: probe of 0000:00:00.0:pcie010 failed with error -22\n\nI think the ASPM, iommu, BAR, and window message are irrelevant to this issue.\nIf so, please remove them so they aren't distractions.\n\nThis looks like it might be a fix for a regression. Can you add any\nrelevant Fixes: tags and possibly stable tags?\n\n> Signed-off-by: Dongdong Liu \n> Signed-off-by: Gabriele Paoloni \n> ---\n> drivers/pci/pcie/portdrv_core.c | 41 ++++++++++++++++++++++++-----------------\n> 1 file changed, 24 insertions(+), 17 deletions(-)\n> \n> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c\n> index 313a21d..4cac558 100644\n> --- a/drivers/pci/pcie/portdrv_core.c\n> +++ b/drivers/pci/pcie/portdrv_core.c\n> @@ -54,7 +54,10 @@ static void release_pcie_device(struct device *dev)\n> */\n> static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)\n> {\n> -\tint nr_entries, entry, nvec = 0;\n> +\tint nr_entries, nvec = 0;\n> +\tint entry_hp = 0;\n> +\tint entry_aer = 0;\n> +\tint entry_dpc = 0;\n> \n> \t/*\n> \t * Allocate as many entries as the port wants, so that we can check\n> @@ -86,14 +89,11 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)\n> \t\t * interrupt message.\"\n> \t\t */\n> \t\tpcie_capability_read_word(dev, PCI_EXP_FLAGS, ®16);\n> -\t\tentry = (reg16 & PCI_EXP_FLAGS_IRQ) >> 9;\n> -\t\tif (entry >= nr_entries)\n> +\t\tentry_hp = (reg16 & PCI_EXP_FLAGS_IRQ) >> 9;\n> +\t\tif (entry_hp >= nr_entries)\n> \t\t\tgoto out_free_irqs;\n> \n> -\t\tirqs[PCIE_PORT_SERVICE_PME_SHIFT] = pci_irq_vector(dev, entry);\n> -\t\tirqs[PCIE_PORT_SERVICE_HP_SHIFT] = pci_irq_vector(dev, entry);\n> -\n> -\t\tnvec = max(nvec, entry + 1);\n> +\t\tnvec = max(nvec, entry_hp + 1);\n> \t}\n> \n> \tif (mask & PCIE_PORT_SERVICE_AER) {\n> @@ -114,13 +114,11 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)\n> \t\t */\n> \t\tpos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ERR);\n> \t\tpci_read_config_dword(dev, pos + PCI_ERR_ROOT_STATUS, ®32);\n> -\t\tentry = reg32 >> 27;\n> -\t\tif (entry >= nr_entries)\n> +\t\tentry_aer = reg32 >> 27;\n> +\t\tif (entry_aer >= nr_entries)\n> \t\t\tgoto out_free_irqs;\n> \n> -\t\tirqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, entry);\n> -\n> -\t\tnvec = max(nvec, entry + 1);\n> +\t\tnvec = max(nvec, entry_aer + 1);\n> \t}\n> \n> \tif (mask & PCIE_PORT_SERVICE_DPC) {\n> @@ -141,13 +139,11 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)\n> \t\t */\n> \t\tpos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_DPC);\n> \t\tpci_read_config_word(dev, pos + PCI_EXP_DPC_CAP, ®16);\n> -\t\tentry = reg16 & 0x1f;\n> -\t\tif (entry >= nr_entries)\n> +\t\tentry_dpc = reg16 & 0x1f;\n> +\t\tif (entry_dpc >= nr_entries)\n> \t\t\tgoto out_free_irqs;\n> \n> -\t\tirqs[PCIE_PORT_SERVICE_DPC_SHIFT] = pci_irq_vector(dev, entry);\n> -\n> -\t\tnvec = max(nvec, entry + 1);\n> +\t\tnvec = max(nvec, entry_dpc + 1);\n> \t}\n> \n> \t/*\n> @@ -166,6 +162,17 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask)\n> \t\t\treturn nr_entries;\n> \t}\n> \n> +\tif (mask & (PCIE_PORT_SERVICE_PME | PCIE_PORT_SERVICE_HP)) {\n> +\t\tirqs[PCIE_PORT_SERVICE_PME_SHIFT] = pci_irq_vector(dev, entry_hp);\n> +\t\tirqs[PCIE_PORT_SERVICE_HP_SHIFT] = pci_irq_vector(dev, entry_hp);\n> +\t}\n> +\n> +\tif (mask & PCIE_PORT_SERVICE_AER)\n> +\t\tirqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, entry_aer);\n> +\n> +\tif (mask & PCIE_PORT_SERVICE_DPC)\n> +\t\tirqs[PCIE_PORT_SERVICE_DPC_SHIFT] = pci_irq_vector(dev, entry_dpc);\n> +\n> \treturn 0;\n> \n> out_free_irqs:\n> -- \n> 1.9.1\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","mail.kernel.org;\n\tdmarc=none (p=none dis=none) header.from=kernel.org","mail.kernel.org;\n\tspf=none smtp.mailfrom=helgaas@kernel.org"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xhfzl4d3dz9sPt\n\tfor ;\n\tWed, 30 Aug 2017 06:16:35 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751231AbdH2UQe (ORCPT );\n\tTue, 29 Aug 2017 16:16:34 -0400","from mail.kernel.org ([198.145.29.99]:54102 \"EHLO mail.kernel.org\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751186AbdH2UQd (ORCPT );\n\tTue, 29 Aug 2017 16:16:33 -0400","from localhost (unknown [69.71.4.159])\n\t(using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits))\n\t(No client certificate requested)\n\tby mail.kernel.org (Postfix) with ESMTPSA id BEFA02199E;\n\tTue, 29 Aug 2017 20:16:32 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mail.kernel.org BEFA02199E","Date":"Tue, 29 Aug 2017 15:16:31 -0500","From":"Bjorn Helgaas ","To":"Dongdong Liu ","Cc":"linux-pci@vger.kernel.org, gabriele.paoloni@huawei.com,\n\tcharles.chenxin@huawei.com, linuxarm@huawei.com","Subject":"Re: [PATCH] PCI/portdrv: Fix MSI/MSI-X bug for PCIe port service\n\tdrivers","Message-ID":"<20170829201631.GO8154@bhelgaas-glaptop.roam.corp.google.com>","References":"<1503654642-59977-1-git-send-email-liudongdong3@huawei.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<1503654642-59977-1-git-send-email-liudongdong3@huawei.com>","User-Agent":"Mutt/1.5.21 (2010-09-15)","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}},{"id":1760037,"web_url":"http://patchwork.ozlabs.org/comment/1760037/","msgid":"<014021d0-51e1-1169-e4b5-2a347ca12598@huawei.com>","list_archive_url":null,"date":"2017-08-30T10:35:37","subject":"Re: [PATCH] PCI/portdrv: Fix MSI/MSI-X bug for PCIe port service\n\tdrivers","submitter":{"id":67753,"url":"http://patchwork.ozlabs.org/api/people/67753/","name":"Dongdong Liu","email":"liudongdong3@huawei.com"},"content":"Hi Bjorn\n\nMany thanks for your review.\n\ne\u001c( 2017/8/30 4:16, Bjorn Helgaas e\u0006\u0019i\u0001\u0013:\n> On Fri, Aug 25, 2017 at 05:50:42PM +0800, Dongdong Liu wrote:\n>> Current code is broken as calling pci_free_irq_vectors()\n>> invalidates the IRQ numbers returned before by pci_irq_vectors();\n>> so we need to move all the assignment of the Linux IRQ numbers at\n>> the bottom of the function.\n>>\n>> After removing and adding back the PCI root port device,\n>> we see the PCIe port service drivers request irq failed.\n>>\n>> root@(none)$ lspci -tv\n>> -[0000:00]-+-00.0-[01]----00.0 Device 19e5:0123\n>> \\-08.0-[02-03]--+-00.0 Device 8086:10fb\n>> \\-00.1 Device 8086:10fb\n>\n> It doesn't look like the hierarchy *below* the root port is relevant.\nOK, will delete.\n>\n>> root@(none)$ echo 1 > /sys/devices/pci0000\\:00/0000\\:00\\:00.0/remove\n>> iommu: Removing device 0000:00:00.0 from group 2\n>> root@(none)$ echo 1 > /sys/devices/pci0000\\:00/pci_bus/0000\\:00/rescan\n>> pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it\n>> with 'pcie_aspm=force'\n>> pci 0000:00:00.0: BAR 14: assigned [mem 0xe0100000-0xe03fffff]\n>> pci 0000:00:00.0: BAR 15: assigned [mem 0x80000e00000-0x80000ffffff 64bit\n>> pref]\n>> pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]\n>> pci 0000:01:00.0: BAR 0: assigned [mem 0xe0100000-0xe013ffff 64bit]\n>> pci 0000:01:00.0: BAR 6: assigned [mem 0xe0140000-0xe015ffff pref]\n>> pci 0000:00:00.0: PCI bridge to [bus 01]\n>> pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]\n>> pci 0000:00:00.0: bridge window [mem 0xe0100000-0xe03fffff]\n>> pci 0000:00:00.0: bridge window [mem 0x80000e00000-0x80000ffffff 64bit\n>> pref]\n>> iommu: Adding device 0000:00:00.0 to group 2\n>> pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22\n>> aer: probe of 0000:00:00.0:pcie002 failed with error -22\n>> pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd-\n>> PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+\n>> pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller\n>> pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1)\n>> dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22\n>> dpc: probe of 0000:00:00.0:pcie010 failed with error -22\n>\n> I think the ASPM, iommu, BAR, and window message are irrelevant to this issue.\n> If so, please remove them so they aren't distractions.\nOK, I will remove them in PATCH V2.\n>\n> This looks like it might be a fix for a regression. Can you add any\n> relevant Fixes: tags and possibly stable tags?\n\nSure, I will add in PATCH V2.\nFixes: 3674cc4 (\"PCI/portdrv: Use pci_irq_alloc_vectors()\")\n\nThanks\nDongdong","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=linux-pci-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xj23D2Mxqz9s83\n\tfor ;\n\tWed, 30 Aug 2017 20:35:52 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751340AbdH3Kfv (ORCPT );\n\tWed, 30 Aug 2017 06:35:51 -0400","from szxga04-in.huawei.com ([45.249.212.190]:5493 \"EHLO\n\tszxga04-in.huawei.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751186AbdH3Kfu (ORCPT\n\t); Wed, 30 Aug 2017 06:35:50 -0400","from 172.30.72.60 (EHLO DGGEMS409-HUB.china.huawei.com)\n\t([172.30.72.60])\n\tby dggrg04-dlp.huawei.com (MOS 4.4.6-GA FastPath queued)\n\twith ESMTP id DGE63820; Wed, 30 Aug 2017 18:35:44 +0800 (CST)","from [127.0.0.1] (10.63.141.25) by DGGEMS409-HUB.china.huawei.com\n\t(10.3.19.209) with Microsoft SMTP Server id 14.3.301.0;\n\tWed, 30 Aug 2017 18:35:37 +0800"],"Subject":"Re: [PATCH] PCI/portdrv: Fix MSI/MSI-X bug for PCIe port service\n\tdrivers","To":"Bjorn Helgaas ","References":"<1503654642-59977-1-git-send-email-liudongdong3@huawei.com>\n\t<20170829201631.GO8154@bhelgaas-glaptop.roam.corp.google.com>","CC":", ,\n\t, ","From":"Dongdong Liu ","Message-ID":"<014021d0-51e1-1169-e4b5-2a347ca12598@huawei.com>","Date":"Wed, 30 Aug 2017 18:35:37 +0800","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101\n\tThunderbird/45.4.0","MIME-Version":"1.0","In-Reply-To":"<20170829201631.GO8154@bhelgaas-glaptop.roam.corp.google.com>","Content-Type":"text/plain; charset=\"UTF-8\"; format=flowed","Content-Transfer-Encoding":"7bit","X-Originating-IP":"[10.63.141.25]","X-CFilter-Loop":"Reflected","X-Mirapoint-Virus-RAPID-Raw":"score=unknown(0),\n\trefid=str=0001.0A020205.59A69501.0038, ss=1, re=0.000, recu=0.000,\n\treip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0,\n\tso=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32","X-Mirapoint-Loop-Id":"cf41004e84b5bb92572c4eea80c7415f","Sender":"linux-pci-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"linux-pci@vger.kernel.org"}}]