Message ID | 78C9135A3D2ECE4B8162EBDCE82CAD770579029A@nekter |
---|---|
State | Not Applicable, archived |
Delegated to: | David Miller |
Headers | show |
Sivakumar Subramani wrote: > Hi, > > Our card supports 4 virtual function. And I am getting following error > message from srio_enable function. Any idea on this error message? I > have attached the lspci output for reference. Your BIOS doesn't support SR-IOV. Please contact the vendor for a new one. Or plug the device into a PCIe slot that supports ARI Forwarding. If neither of the ways are easy for you, use pci=assign-busses to boot kernel. > > Thanks, > ~Siva > > > Jul 20 01:11:58 SuSE11 kernel: vxge: Copyright(c) 2002-2009 Neterion Inc > Jul 20 01:11:58 SuSE11 kernel: vxge: Driver version: 0.0.0.0-svn > Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: PCI INT A -> GSI 16 > (level, low) -> IRQ 16 > Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: setting latency timer > to 64 > Jul 20 01:11:58 SuSE11 kernel: virtfn_bus is 4 > Jul 20 01:11:58 SuSE11 kernel: nr_virtfn is 4 > Jul 20 01:11:58 SuSE11 kernel: dev->bus->subordinate is 3 > Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: SR-IOV: bus number out > of range <<===============================Error > Jul 20 01:11:58 SuSE11 kernel: eth2: SERIAL NUMBER: SXJ0841025 > Jul 20 01:11:58 SuSE11 kernel: eth2: PART NUMBER: X3110SR0001 > Jul 20 01:11:58 SuSE11 kernel: eth2: Neterion PCIe 10 GbE Adapter, > single port, SFP+ SR Server Adapter > Jul 20 01:11:58 SuSE11 kernel: eth2: MAC ADDR: 00:0C:FC:00:B6:B5 > Jul 20 01:11:58 SuSE11 kernel: eth2: Link Width x4 > Jul 20 01:11:58 SuSE11 kernel: eth2: Firmware version : 1.1.0 Date : > 06/17/2009 > Jul 20 01:11:58 SuSE11 kernel: eth2: Single Root IOV Mode Enabled > Jul 20 01:11:58 SuSE11 kernel: eth2: 1 Vpath(s) opened > Jul 20 01:11:58 SuSE11 kernel: eth2: Interrupt type MSI-X > Jul 20 01:11:58 SuSE11 kernel: eth2: RTH steering disabled > Jul 20 01:11:58 SuSE11 kernel: eth2: Tx steering disabled > Jul 20 01:11:58 SuSE11 kernel: eth2: Large receive offload enabled > Jul 20 01:11:58 SuSE11 kernel: eth2: NAPI enabled > Jul 20 01:11:58 SuSE11 kernel: eth2: Rx doorbell mode enabled > Jul 20 01:11:58 SuSE11 kernel: eth2: VLAN tag stripping enabled > Jul 20 01:11:58 SuSE11 kernel: eth2: Ring blocks : 2 > Jul 20 01:11:58 SuSE11 kernel: eth2: Fifo blocks : 12 > Thanks, > ~Siva > > -----Original Message----- > From: Yu Zhao [mailto:yu.zhao@intel.com] > Sent: Wednesday, June 17, 2009 10:28 AM > To: Sivakumar Subramani > Cc: netdev@vger.kernel.org > Subject: Re: SRIOV fails with "not enough MMIO resources" > > Sivakumar Subramani wrote: >> Sorry for the delay in reply. I got the setup back today. The PF BAR4 >> size is 32M. BAR0 of each VF is 8M. I loading the driver with 4 VFs. > I >> am facing this problem, even if I load it with 3 VFs. > > OK, I see the problem. The total VFs supported by your SR-IOV device is > 8 (0a:00.0/1) and 16 (09:00.0). And the VF MMIO area is calculated by > size of VF BAR multiplied total VFs. Passing 3 or 4 to > pci_enable_sriov() can not reduce the VF MMIO area size. > > So please hardcode the total VFs to 4 before you get a BIOS that > supports the SR-IOV: > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > index b497daa..5ba6da0 100644 > --- a/drivers/pci/iov.c > +++ b/drivers/pci/iov.c > @@ -430,6 +430,7 @@ static int sriov_init(struct pci_dev *dev, int pos) > if (!total) > return 0; > > + total = 4; > ctrl = 0; > list_for_each_entry(pdev, &dev->bus->devices, bus_list) > if (pdev->is_physfn) > > >> I have attached the code of sriov_init / sriov_enable / > quirk_vxge_sriov >> and the log of the debug statements that I have added in these > function. >> Please let you know your comments. >> >> </Siva> >> >> Thanks, >> ~Siva >> -----Original Message----- >> From: Yu Zhao [mailto:yu.zhao@intel.com] >> Sent: Tuesday, June 09, 2009 11:40 AM >> To: Sreenivasa Honnur >> Cc: netdev@vger.kernel.org; Sivakumar Subramani >> Subject: Re: SRIOV fails with "not enough MMIO resources" >> >> Sreenivasa Honnur wrote: >>> Hi, >>> >>> I added pci_enable_sriov / pci_disable_sriov in probe and remove >>> function for Neterion X3100 card to test SRIOV feature. While testing >> I >>> got following error. >>> >>> Jun 8 00:02:59 bethweidel kernel: vxge: Copyright(c) 2002-2009 >> Neterion >>> Inc Jun 8 00:02:59 bethweidel kernel: vxge: Driver version: >>> 2.0.1.17129-k Jun 8 00:02:59 bethweidel kernel: vxge 0000:03:00.0: > PCI >>> INT A -> GSI 16 (level, low) -> IRQ 16 Jun 8 00:02:59 bethweidel >> kernel: >>> res->parent is ffff88013f129d28 for 2 Jun 8 00:02:59 bethweidel >> kernel: >>> res->parent is ffff88013f129d28 for 4 Jun 8 00:02:59 bethweidel >> kernel: >>> nres = 2 - iov->nres = 3 Jun 8 00:02:59 bethweidel kernel: vxge >>> 0000:03:00.0: not enough MMIO resources for SR-IOV ----> ERROR Jun 8 >>> 00:02:59 bethweidel kernel: pci_enable_sriov returned -12 Jun 8 >> 00:02:59 >>> bethweidel kernel: eth2: SERIAL NUMBER: SXC0919196 Jun 8 00:02:59 >>> bethweidel kernel: eth2: PART NUMBER: X3110SR0003 >>> >>> >>> >>> When I dumped the config space of SRIOV Capability, I found BAR0 of >> VFs >>> was not configured by BIOS and it was zero. So I added following >>> "quirk_vxge_sriov" quirk function to configure the BAR2 of PF as BAR0 >> of >>> VFs. >>> >>> Now lspci shows proper value. But I am still get same error. The code >>> that dumps this error is as follow. Here I trying to find out the >> reason >>> why res->parent is not updated for BAR0 of VF. From the above log, I >> can >>> see that Bar2 and BAR4's res->parent are update. >>> >>> Any possible reason why res->parent of VF's BAR0 is not updated? >> Is the size of the PF BAR 4 big enough to cover the VF MMIO area >> indicated by VF BAR 0? (size of the PF BAR 4 >= size of VF BAR 0 * >> NumVFs). >> >> Can you please post the whole dmesg? I might be able to root cause the > >> problem after looking at it. >> >>> sriov_enable : >>> >>> -------------- >>> >>> nres = 0; >>> >>> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { >>> >>> res = dev->resource + PCI_IOV_RESOURCES + i; >>> >>> if (res->parent) { >>> >>> printk("res->parent is %llx for %d \n", res->parent,i ); >>> >>> nres++; >>> >>> } >>> >>> } >>> >>> printk("nres = %d - iov->nres = %d \n", nres, iov->nres); >>> >>> if (nres != iov->nres) { >>> >>> dev_err(&dev->dev, "not enough MMIO resources for SR-IOV\n"); >>> >>> return -ENOMEM; >>> >>> } >>> >>> >>> >>> quirk_vxge_sriov: >>> >>> ----------------- >>> >>> static void __devinit quirk_vxge_sriov(struct pci_dev *dev) { >>> >>> int pos, flags; >>> >>> u32 bar, start, size; >>> >>> printk("Entering quirk_vxge_sriov ...\n"); >>> >>> if (PAGE_SIZE > 0x10000) { >>> >>> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >>> >>> return; >>> >>> } >>> >>> pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV); >>> >>> printk("the pos is %d \n", pos); >>> >>> if (!pos) { >>> >>> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >>> >>> return; >>> >>> } >>> >>> pci_read_config_dword(dev, pos + PCI_SRIOV_BAR, &bar); >>> >>> if (bar & PCI_BASE_ADDRESS_MEM_MASK) >>> >>> { >>> >>> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >>> >>> return; >>> >>> } >>> >>> start = pci_resource_start(dev, 4); >>> >>> size = pci_resource_len(dev, 4); >>> >>> printk("Bar4 start %llx - size %llx \n", start, size); >>> >>> if (!start || size != 0x2000000 || start & (size - 1)) { >>> >>> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >>> >>> return; >>> >>> } >>> >>> pci_resource_flags(dev, 4) = 0; >>> >>> pci_write_config_dword(dev, PCI_BASE_ADDRESS_4, 0); >>> >>> pci_write_config_dword(dev, pos + PCI_SRIOV_BAR, start); >>> >>> pci_read_config_dword(dev, pos+PCI_SRIOV_BAR,&bar); >>> >>> printk("VF's Bar0 is %llx \n", bar); >>> >>> dev_info(&dev->dev, "use Flash Memory Space for SR-IOV BARs\n"); >>> >>> printk("Exiting quirk_vxge_sriov ...\n"); } >>> >>> >>> >>> Thanks, >>> >>> ~Siva >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe netdev" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index b497daa..5ba6da0 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -430,6 +430,7 @@ static int sriov_init(struct pci_dev *dev, int pos) if (!total) return 0; + total = 4; ctrl = 0; list_for_each_entry(pdev, &dev->bus->devices, bus_list) if (pdev->is_physfn)