From patchwork Mon Jul 20 08:16:29 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sivakumar Subramani X-Patchwork-Id: 30000 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@bilbo.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from ozlabs.org (ozlabs.org [203.10.76.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mx.ozlabs.org", Issuer "CA Cert Signing Authority" (verified OK)) by bilbo.ozlabs.org (Postfix) with ESMTPS id B43D9B70A6 for ; Mon, 20 Jul 2009 18:16:42 +1000 (EST) Received: by ozlabs.org (Postfix) id A6D1FDDD01; Mon, 20 Jul 2009 18:16:42 +1000 (EST) Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by ozlabs.org (Postfix) with ESMTP id 8A5BDDDD0B for ; Mon, 20 Jul 2009 18:16:41 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752977AbZGTIQf (ORCPT ); Mon, 20 Jul 2009 04:16:35 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752943AbZGTIQe (ORCPT ); Mon, 20 Jul 2009 04:16:34 -0400 Received: from mx.neterion.com ([72.1.205.142]:15452 "EHLO owa.neterion.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752904AbZGTIQd (ORCPT ); Mon, 20 Jul 2009 04:16:33 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: SRIOV fails with "SR-IOV: bus number out of range" Date: Mon, 20 Jul 2009 04:16:29 -0400 Message-ID: <78C9135A3D2ECE4B8162EBDCE82CAD770579029A@nekter> In-Reply-To: <4A3877EB.7020109@intel.com> X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Thread-Topic: SRIOV fails with "SR-IOV: bus number out of range" Thread-Index: AcnvCFmoN2ZNOpQETmSkGROKhAZZqAaCTg4A References: <78C9135A3D2ECE4B8162EBDCE82CAD77055985C4@nekter> <4A2DFCCC.10506@intel.com> <78C9135A3D2ECE4B8162EBDCE82CAD7705640305@nekter> <4A3877EB.7020109@intel.com> From: "Sivakumar Subramani" To: "Yu Zhao" Cc: Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi, Our card supports 4 virtual function. And I am getting following error message from srio_enable function. Any idea on this error message? I have attached the lspci output for reference. Thanks, ~Siva Jul 20 01:11:58 SuSE11 kernel: vxge: Copyright(c) 2002-2009 Neterion Inc Jul 20 01:11:58 SuSE11 kernel: vxge: Driver version: 0.0.0.0-svn Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: setting latency timer to 64 Jul 20 01:11:58 SuSE11 kernel: virtfn_bus is 4 Jul 20 01:11:58 SuSE11 kernel: nr_virtfn is 4 Jul 20 01:11:58 SuSE11 kernel: dev->bus->subordinate is 3 Jul 20 01:11:58 SuSE11 kernel: vxge 0000:03:00.0: SR-IOV: bus number out of range <<===============================Error Jul 20 01:11:58 SuSE11 kernel: eth2: SERIAL NUMBER: SXJ0841025 Jul 20 01:11:58 SuSE11 kernel: eth2: PART NUMBER: X3110SR0001 Jul 20 01:11:58 SuSE11 kernel: eth2: Neterion PCIe 10 GbE Adapter, single port, SFP+ SR Server Adapter Jul 20 01:11:58 SuSE11 kernel: eth2: MAC ADDR: 00:0C:FC:00:B6:B5 Jul 20 01:11:58 SuSE11 kernel: eth2: Link Width x4 Jul 20 01:11:58 SuSE11 kernel: eth2: Firmware version : 1.1.0 Date : 06/17/2009 Jul 20 01:11:58 SuSE11 kernel: eth2: Single Root IOV Mode Enabled Jul 20 01:11:58 SuSE11 kernel: eth2: 1 Vpath(s) opened Jul 20 01:11:58 SuSE11 kernel: eth2: Interrupt type MSI-X Jul 20 01:11:58 SuSE11 kernel: eth2: RTH steering disabled Jul 20 01:11:58 SuSE11 kernel: eth2: Tx steering disabled Jul 20 01:11:58 SuSE11 kernel: eth2: Large receive offload enabled Jul 20 01:11:58 SuSE11 kernel: eth2: NAPI enabled Jul 20 01:11:58 SuSE11 kernel: eth2: Rx doorbell mode enabled Jul 20 01:11:58 SuSE11 kernel: eth2: VLAN tag stripping enabled Jul 20 01:11:58 SuSE11 kernel: eth2: Ring blocks : 2 Jul 20 01:11:58 SuSE11 kernel: eth2: Fifo blocks : 12 Thanks, ~Siva -----Original Message----- From: Yu Zhao [mailto:yu.zhao@intel.com] Sent: Wednesday, June 17, 2009 10:28 AM To: Sivakumar Subramani Cc: netdev@vger.kernel.org Subject: Re: SRIOV fails with "not enough MMIO resources" Sivakumar Subramani wrote: > Sorry for the delay in reply. I got the setup back today. The PF BAR4 > size is 32M. BAR0 of each VF is 8M. I loading the driver with 4 VFs. I > am facing this problem, even if I load it with 3 VFs. OK, I see the problem. The total VFs supported by your SR-IOV device is 8 (0a:00.0/1) and 16 (09:00.0). And the VF MMIO area is calculated by size of VF BAR multiplied total VFs. Passing 3 or 4 to pci_enable_sriov() can not reduce the VF MMIO area size. So please hardcode the total VFs to 4 before you get a BIOS that supports the SR-IOV: > I have attached the code of sriov_init / sriov_enable / quirk_vxge_sriov > and the log of the debug statements that I have added in these function. > Please let you know your comments. > > > > Thanks, > ~Siva > -----Original Message----- > From: Yu Zhao [mailto:yu.zhao@intel.com] > Sent: Tuesday, June 09, 2009 11:40 AM > To: Sreenivasa Honnur > Cc: netdev@vger.kernel.org; Sivakumar Subramani > Subject: Re: SRIOV fails with "not enough MMIO resources" > > Sreenivasa Honnur wrote: >> Hi, >> >> I added pci_enable_sriov / pci_disable_sriov in probe and remove >> function for Neterion X3100 card to test SRIOV feature. While testing > I >> got following error. >> >> Jun 8 00:02:59 bethweidel kernel: vxge: Copyright(c) 2002-2009 > Neterion >> Inc Jun 8 00:02:59 bethweidel kernel: vxge: Driver version: >> 2.0.1.17129-k Jun 8 00:02:59 bethweidel kernel: vxge 0000:03:00.0: PCI >> INT A -> GSI 16 (level, low) -> IRQ 16 Jun 8 00:02:59 bethweidel > kernel: >> res->parent is ffff88013f129d28 for 2 Jun 8 00:02:59 bethweidel > kernel: >> res->parent is ffff88013f129d28 for 4 Jun 8 00:02:59 bethweidel > kernel: >> nres = 2 - iov->nres = 3 Jun 8 00:02:59 bethweidel kernel: vxge >> 0000:03:00.0: not enough MMIO resources for SR-IOV ----> ERROR Jun 8 >> 00:02:59 bethweidel kernel: pci_enable_sriov returned -12 Jun 8 > 00:02:59 >> bethweidel kernel: eth2: SERIAL NUMBER: SXC0919196 Jun 8 00:02:59 >> bethweidel kernel: eth2: PART NUMBER: X3110SR0003 >> >> >> >> When I dumped the config space of SRIOV Capability, I found BAR0 of > VFs >> was not configured by BIOS and it was zero. So I added following >> "quirk_vxge_sriov" quirk function to configure the BAR2 of PF as BAR0 > of >> VFs. >> >> Now lspci shows proper value. But I am still get same error. The code >> that dumps this error is as follow. Here I trying to find out the > reason >> why res->parent is not updated for BAR0 of VF. From the above log, I > can >> see that Bar2 and BAR4's res->parent are update. >> >> Any possible reason why res->parent of VF's BAR0 is not updated? > > Is the size of the PF BAR 4 big enough to cover the VF MMIO area > indicated by VF BAR 0? (size of the PF BAR 4 >= size of VF BAR 0 * > NumVFs). > > Can you please post the whole dmesg? I might be able to root cause the > problem after looking at it. > >> sriov_enable : >> >> -------------- >> >> nres = 0; >> >> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { >> >> res = dev->resource + PCI_IOV_RESOURCES + i; >> >> if (res->parent) { >> >> printk("res->parent is %llx for %d \n", res->parent,i ); >> >> nres++; >> >> } >> >> } >> >> printk("nres = %d - iov->nres = %d \n", nres, iov->nres); >> >> if (nres != iov->nres) { >> >> dev_err(&dev->dev, "not enough MMIO resources for SR-IOV\n"); >> >> return -ENOMEM; >> >> } >> >> >> >> quirk_vxge_sriov: >> >> ----------------- >> >> static void __devinit quirk_vxge_sriov(struct pci_dev *dev) { >> >> int pos, flags; >> >> u32 bar, start, size; >> >> printk("Entering quirk_vxge_sriov ...\n"); >> >> if (PAGE_SIZE > 0x10000) { >> >> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >> >> return; >> >> } >> >> pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV); >> >> printk("the pos is %d \n", pos); >> >> if (!pos) { >> >> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >> >> return; >> >> } >> >> pci_read_config_dword(dev, pos + PCI_SRIOV_BAR, &bar); >> >> if (bar & PCI_BASE_ADDRESS_MEM_MASK) >> >> { >> >> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >> >> return; >> >> } >> >> start = pci_resource_start(dev, 4); >> >> size = pci_resource_len(dev, 4); >> >> printk("Bar4 start %llx - size %llx \n", start, size); >> >> if (!start || size != 0x2000000 || start & (size - 1)) { >> >> printk("Returning quirk_vxge_sriov @%d \n", __LINE__); >> >> return; >> >> } >> >> pci_resource_flags(dev, 4) = 0; >> >> pci_write_config_dword(dev, PCI_BASE_ADDRESS_4, 0); >> >> pci_write_config_dword(dev, pos + PCI_SRIOV_BAR, start); >> >> pci_read_config_dword(dev, pos+PCI_SRIOV_BAR,&bar); >> >> printk("VF's Bar0 is %llx \n", bar); >> >> dev_info(&dev->dev, "use Flash Memory Space for SR-IOV BARs\n"); >> >> printk("Exiting quirk_vxge_sriov ...\n"); } >> >> >> >> Thanks, >> >> ~Siva >> >> >> >> >> >> >> >> >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index b497daa..5ba6da0 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -430,6 +430,7 @@ static int sriov_init(struct pci_dev *dev, int pos) if (!total) return 0; + total = 4; ctrl = 0; list_for_each_entry(pdev, &dev->bus->devices, bus_list) if (pdev->is_physfn)