From patchwork Wed Sep 8 00:01:48 2010 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cam Macdonell X-Patchwork-Id: 64081 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id F1E84B6EE9 for ; Wed, 8 Sep 2010 10:03:49 +1000 (EST) Received: from localhost ([127.0.0.1]:52988 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ot88g-0005zB-K4 for incoming@patchwork.ozlabs.org; Tue, 07 Sep 2010 20:03:46 -0400 Received: from [140.186.70.92] (port=43252 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ot86q-0005EY-KK for qemu-devel@nongnu.org; Tue, 07 Sep 2010 20:02:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ot86o-00081V-P5 for qemu-devel@nongnu.org; Tue, 07 Sep 2010 20:01:52 -0400 Received: from mail-vw0-f45.google.com ([209.85.212.45]:49407) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ot86o-00081A-L3 for qemu-devel@nongnu.org; Tue, 07 Sep 2010 20:01:50 -0400 Received: by vws19 with SMTP id 19so4935539vws.4 for ; Tue, 07 Sep 2010 17:01:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.84.133 with SMTP id j5mr327271vcl.103.1283904108157; Tue, 07 Sep 2010 17:01:48 -0700 (PDT) Received: by 10.220.84.17 with HTTP; Tue, 7 Sep 2010 17:01:48 -0700 (PDT) In-Reply-To: References: <4C83CA5A.7030404@redhat.com> Date: Tue, 7 Sep 2010 18:01:48 -0600 X-Google-Sender-Auth: buIFG25sYsoDrDzPP4bkmTPtsGA Message-ID: Subject: Re: [Qemu-devel] [solved] Guest cannot handle a PCI BAR > 1GB From: Cam Macdonell To: Avi Kivity X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) Cc: Chris Wright , "qemu-devel@nongnu.org Developers" , linux-kernel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org It seems it was the alignment value being passed back from pci_resource_alignment(). The return type is an int, which was causing value of 2GB to be sign extended to to 0xffffffff80000000. Changing the return type to resource_size_t allows BAR values >= 2GB to be successfully assigned. On Mon, Sep 6, 2010 at 11:24 AM, Cam Macdonell wrote: > On Mon, Sep 6, 2010 at 10:37 AM, Cam Macdonell wrote: >> >> >> On Sun, Sep 5, 2010 at 10:50 AM, Avi Kivity wrote: >>> >>>  On 09/04/2010 01:22 AM, Cam Macdonell wrote: >>>> >>>> Hi, >>>> >>>> I'm trying to test 2 GB (and eventually larger) BARs with ivshmem and >>>> I get an error in the guest that it is able to find a mem resource for >>>> a BAR larger than 1GB.  I'm using 64-bit BARs. >>>> >>>> when running with 6GB of RAM and a 1GB BAR for ivshmem, it finds a >>>> resource (and searches beyond 32-bit values to find it).  Here is a >>>> log from printfs added to the loop that searches the resources from >>>> find_resource() in kernel/resource.c:363. >>>> >>> >>> This is a kernel question, not a qemu issue.  Copying lkml. >>> >>>> trying 'tmp.start' 1000 to >>>>         'tmp.end' fff >>>> trying 'tmp.start' 9f400 to >>>>         'tmp.end' 9f3ff >>>> trying 'tmp.start' a0000 to >>>>         'tmp.end' effff >>>> trying 'tmp.start' 100000 to >>>>         'tmp.end' fffff >>>> trying 'tmp.start' dfffd000 to >>>>         'tmp.end' dfffcfff >>>> trying 'tmp.start' e0000000 to >>>>         'tmp.end' efffffff >>>> trying 'tmp.start' f2000000 to >>>>         'tmp.end' f1ffffff >>>> trying 'tmp.start' f2001000 to >>>>         'tmp.end' f200ffff >>>> trying 'tmp.start' f2020000 to >>>>         'tmp.end' f201ffff >>>> trying 'tmp.start' f2021000 to >>>>         'tmp.end' f202ffff >>>> trying 'tmp.start' f2040000 to >>>>         'tmp.end' f203ffff >>>> trying 'tmp.start' f2040100 to >>>>         'tmp.end' febfffff >>>> trying 'tmp.start' fec00400 to >>>>         'tmp.end' fffbffff >>>> trying 'tmp.start' 100000000 to >>>>         'tmp.end' ffffffff >>>> trying 'tmp.start' 1a0000000 to >>>>         'tmp.end' ffffffffffffffff >>>> pci 0000:00:04.0: BAR 2: assigned [mem 0x1c0000000-0x1ffffffff 64bit] >>>> pci 0000:00:04.0: BAR 2: set to [mem 0x1c0000000-0x1ffffffff 64bit] >>>> (PCI address [0x1c0000000-0x1ffffffff] >>>> >>>> and you can see the BAR is successfully assigned. >>>> >>>> However, with a 2GB BAR (below), the search fails, but it also never >>>> searches beyong 32-bits.  Again, all that's changed is the size of the >>>> ivshmem region. >>>> >>>> trying 'tmp.start' 1000 to >>>>         'tmp.end' fff >>>> trying 'tmp.start' 9f400 to >>>>         'tmp.end' 9f3ff >>>> trying 'tmp.start' a0000 to >>>>         'tmp.end' effff >>>> trying 'tmp.start' 100000 to >>>>         'tmp.end' fffff >>>> trying 'tmp.start' dfffd000 to >>>>         'tmp.end' dfffcfff >>>> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) >>>> >>>> Is there a limit to PCI BAR sizes or resources?  Any pointers or >>>> further debugging tips are greatly appreciated. >>>> >>> >>> What kernel version are you looking at? >> >> latest kvm git, 2.6.36-rc2+ >> >>> >>> Please add printks to the loop so we can see this->start and this->end.  It smells like a truncation issue. >> >> Success with a 1GB BAR >> this->start 1000, this->end 9f3ff >> this->start 9f400, this->end 9ffff >> this->start f0000, this->end fffff >> this->start 100000, this->end dfffcfff >> this->start dfffd000, this->end dfffffff >> this->start f0000000, this->end f1ffffff >> this->start f2000000, this->end f2000fff >> this->start f2010000, this->end f201ffff >> this->start f2020000, this->end f2020fff >> this->start f2030000, this->end f203ffff >> this->start f2040000, this->end f20400ff >> this->start fec00000, this->end fec003ff >> this->start fffc0000, this->end ffffffff >> this->start 100000000, this->end 11fffffff >> tmp.start 120000000, tmp.end ffffffffffffffff >> pci 0000:00:04.0: BAR 2: assigned [mem 0x140000000-0x17fffffff 64bit] >> and when it fails with a 2GB BAR, the following is printed >> this->start 1000, this->end 9f3ff >> this->start 9f400, this->end 9ffff >> this->start f0000, this->end fffff >> this->start 100000, this->end dfffcfff >> this->start dfffd000, this->end dfffffff >> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) >> I added a few more debug statements and found that in the failure case, the function returns that it found a region (the last one printed before the error).  I've added printfs for the two tests in the if that determine when a region is found: >>        if (tmp.start < tmp.end && tmp.end - tmp.start >= size - 1) { >>             new->start = tmp.start; >>             new->end = tmp.start + size - 1; >>             printk(KERN_INFO "returning 0\n"); >>             return 0; >>         } >> this->start 1000, this->end 9f3ff >> tmp.start 80000000, tmp.end fff >>     true: ffffffff80000fff >= 7fffffff >> this->start 9f400, this->end 9ffff >> tmp.start 80000000, tmp.end 9f3ff >>     true: ffffffff8009f3ff >= 7fffffff >> this->start f0000, this->end fffff >> tmp.start 80000000, tmp.end effff >>     true: ffffffff800effff >= 7fffffff >> this->start 100000, this->end dfffcfff >> tmp.start 80000000, tmp.end fffff >>     true: ffffffff800fffff >= 7fffffff >> this->start dfffd000, this->end dfffffff >> tmp.start 100000, tmp.end dfffcfff >>     true: 100000 < dfffcfff >>     true: dfefcfff >= 7fffffff >> returning 0 >> pci 0000:00:04.0: BAR 2: can't assign mem (size 0x80000000) > > Further to this, it seems tmp.start is getting set to zero by the ALIGN macro > > 2GB BAR: > this->start dfffd000, this->end dfffffff > tmp.start dfffd000 > tmp.start 0 > tmp.start 100000, tmp.end dfffcfff > > 1GB BAR: > this->start dfffd000, this->end dfffffff > tmp.start dfffd000 > tmp.start 100000000 > tmp.start 100000000, tmp.end dfffcfff > > I'll dig into the ALIGN macro. > >> >> >> >> >>> >>> -- >>> error compiling committee.c: too many arguments to function >>> >> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 679c39d..3d23522 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -318,7 +318,7 @@ static inline int pci_ats_enabled(struct pci_dev *dev) } #endif /* CONFIG_PCI_IOV */ -static inline int pci_resource_alignment(struct pci_dev *dev, +static inline resource_size_t pci_resource_alignment(struct pci_dev *dev, struct resource *res) { #ifdef CONFIG_PCI_IOV