From patchwork Thu Apr 27 08:51:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Huth X-Patchwork-Id: 755898 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wD9fM3y3kz9s7x for ; Thu, 27 Apr 2017 18:51:23 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3wD9fM38jgzDqFt for ; Thu, 27 Apr 2017 18:51:23 +1000 (AEST) X-Original-To: slof@lists.ozlabs.org Delivered-To: slof@lists.ozlabs.org Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3wD9fH581czDqFL for ; Thu, 27 Apr 2017 18:51:19 +1000 (AEST) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D7904C07F688; Thu, 27 Apr 2017 08:51:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D7904C07F688 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=thuth@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D7904C07F688 Received: from [10.36.116.27] (ovpn-116-27.ams2.redhat.com [10.36.116.27]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9751917183; Thu, 27 Apr 2017 08:51:15 +0000 (UTC) To: Alexey Kardashevskiy References: <20170424030131.32311-1-aik@ozlabs.ru> <7d1b54d2-fb16-8b69-a3a5-10d84324b83b@redhat.com> <20170427165448.67fa6d60@aik.ozlabs.ibm.com> <99870e49-739b-d7e0-1c7e-294989237486@redhat.com> <20170427183022.54569b63@aik.ozlabs.ibm.com> From: Thomas Huth Message-ID: Date: Thu, 27 Apr 2017 10:51:12 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170427183022.54569b63@aik.ozlabs.ibm.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 27 Apr 2017 08:51:17 +0000 (UTC) Subject: Re: [SLOF] [PATCH slof] pci: Put non-prefetchable 64bit BARs into 32bit MMIO window X-BeenThere: slof@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Patches for https://github.com/aik/SLOF" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: slof@lists.ozlabs.org Errors-To: slof-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "SLOF" On 27.04.2017 10:30, Alexey Kardashevskiy wrote: > On Thu, 27 Apr 2017 09:50:36 +0200 > Thomas Huth wrote: > >> On 27.04.2017 09:10, Thomas Huth wrote: >>> On 27.04.2017 08:54, Alexey Kardashevskiy wrote: >>>> On Wed, 26 Apr 2017 15:23:40 +0200 >>>> Thomas Huth wrote: >>>> >>>>> On 24.04.2017 05:01, Alexey Kardashevskiy wrote: >>>>>> At the moment 64bit non-prefetchable BARs of devices behind PCI >>>>>> p2p bridge go to a 64bit prefetchable windows which is not >>>>>> correct and causes linux guests to fail to ioremap() such >>>>>> resources. >>>>>> >>>>>> This moves 64bit non-prefetchable BARs 32bit non-prefetchable >>>>>> window. >>>>>> >>>>>> Note that this does not make distinction between P2P and PHB so >>>>>> from now on XHCI BARs will be allocated from 32bit MMIO space. >>>>>> However since most 64bit-MMIO-capable devices have prefetchable >>>>>> BARs, and XHCI BAR is just 4K (so it is unlikely to cause any >>>>>> space problems), this should not affect usual behavior. >>>>>> >>>>>> Signed-off-by: Alexey Kardashevskiy >>>>>> --- >>>>>> >>>>>> This fixes QEMU's XHCI when it is put on a P2P PCI bridge. >>>>>> >>>>>> There is a little naming confusion as it may look like the patch >>>>>> is changing assignment for all 64bit BAR but it does not as: >>>>>> - "mmio" is used for non-prefetchable memory, >>>>>> - "mem" is used for prefetchable memory. >>>>>> --- >>>>>> slof/fs/pci-properties.fs | 6 +----- >>>>>> 1 file changed, 1 insertion(+), 5 deletions(-) >>>>>> >>>>>> diff --git a/slof/fs/pci-properties.fs >>>>>> b/slof/fs/pci-properties.fs index e6fd843..8594e5d 100644 >>>>>> --- a/slof/fs/pci-properties.fs >>>>>> +++ b/slof/fs/pci-properties.fs >>>>>> @@ -166,11 +166,7 @@ >>>>>> \ Setup a non-prefetchable 64bit BAR and return its size >>>>>> : assign-mmio64-bar ( bar-addr -- 8 ) >>>>>> dup pci-bar-size-mem64 \ fetch size >>>>>> - pci-next-mem64 @ 0 = IF \ Check if we have >>>>>> 64-bit memory range >>>>>> - pci-next-mmio >>>>>> - ELSE >>>>>> - pci-next-mem64 \ for board-qemu we will >>>>>> use same range >>>>>> - THEN >>>>>> + pci-next-mmio >>>>>> assign-bar-value64 \ and set it all >>>>>> ; >>>>> >>>>> I'm sorry, but this is now causing trouble with the USB keyboard >>>>> in *SLOF* instead. If I start QEMU now like this: >>>>> >>>>> qemu-system-ppc64 -nodefaults -serial mon:stdio \ >>>>> -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1 \ >>>>> -device nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3 -vga std \ >>>>> -device usb-kbd,bus=xhci1.0 -bios boot_rom.bin >>>>> >>>>> then SLOF complains: "usb-xhci: failed to initialize XHCI >>>>> controller" and the keyboard is not working at the SLOF prompt >>>>> anymore. >>>>> >>>>> Any idea why this is happening now? >>>> >>>> >>>> XHCI's usb_hcd_dev::base == 0xc0000000 which is a bus address >>>> while it is supposed to be a mapped address, i.e. 800000020000000. >>>> >>>> It used to work before the patch as pci address is the same as >>>> mapped address - 210000000000 - and I am not sure if this is not >>>> an accident, could be QEMU's PCI MMIO windows/spacing rework David >>>> did some time ago. >>>> >>>> I am trying to figure out now where usb_hcd_dev::base is set in >>>> SLOF, it is tricky :-/ >>> >>> I think the value comes from usb-setup-hcidev in >>> pci-class_0c.fs ... so it's likely that translate-my-address is >>> failing now. >>> >>> ... but thinking about this issue again, I now wonder whether your >>> original patch was really the right solution here, since QEMU only >>> provides three memory regions to the guest: >>> 1) I/O >>> 2) 32-bit memory >>> 3) 64-bit memory >>> There is no distinction between prefetchable and non-prefetchable >>> here. So I think the original problem might have been caused by >>> something else instead... (not sure about that yet, though) >> >> Yup, looks like the real culprit is the "ranges" property of the PCI >> bridges. This can not deal with 64-bit memory regions yet. I think we >> should revert the patch to assign-mmio64-bar and fix >> pci-bridge-range-props and friends instead... > > Try this: > > diff --git a/slof/fs/pci-properties.fs b/slof/fs/pci-properties.fs > index 8594e5d..66e364d 100644 > --- a/slof/fs/pci-properties.fs > +++ b/slof/fs/pci-properties.fs > @@ -349,7 +349,7 @@ > 1 OF gen-io-bar-prop ENDOF \ - IO-BAR > 2 OF gen-mem32-bar-prop ENDOF \ - MEM32 > 3 OF gen-pmem32-bar-prop ENDOF \ - MEM32 prefetchable > - 4 OF gen-mem64-bar-prop ENDOF \ - MEM64 > + 4 OF gen-mem32-bar-prop ENDOF \ - MEM64 > 5 OF gen-pmem64-bar-prop ENDOF \ - MEM64 prefetchable > ENDCASE \ ESAC ( paddr plen bsize ) > ; > > Works for me, for both PHB and P2P bridges. That might work as long as the device has only one BAR. But as soon as you've got a device with multiple BARs, you'll run into problems since pci-add-assigned-address now returns the wrong BAR size. Please try this instead: Revert your original patch to assign-mmio64-bar and apply this patch instead: Thomas diff --git a/slof/fs/pci-properties.fs b/slof/fs/pci-properties.fs index e6fd843..625692a 100644 --- a/slof/fs/pci-properties.fs +++ b/slof/fs/pci-properties.fs @@ -411,12 +411,14 @@ : pci-bridge-gen-mem-range ( addr prop-addr prop-len -- addr prop-addr prop-len ) 2 pick 24 + rtas-config-l@ \ fetch Value ( addr paddr plen val ) dup 000FFFFF or \ calc limit Bits 31:0 ( addr paddr plen val limit.31:0 ) - swap 0000FFF0 and 10 lshift \ calc base Bits 31:0 ( addr paddr plen limit.31:0 base.31:0 ) + swap dup F and >r \ R: 64-bit? + 0000FFF0 and 10 lshift \ calc base Bits 31:0 ( addr paddr plen limit.31:0 base.31:0 ) 4 pick 28 + rtas-config-l@ \ fetch upper Basebits ( addr paddr plen limit.31:0 base.31:0 base.63:32 ) 20 lshift or swap \ and calc Base ( addr paddr plen base.63:0 limit.31:0 ) 4 pick 2C + rtas-config-l@ \ fetch upper Limitbits ( addr paddr plen base.63:0 limit.31:0 limit.63:32 ) 20 lshift or \ and calc Limit ( addr paddr plen base.63:0 limit.63:0 ) - 42000000 pci-bridge-gen-range \ and generate it ( addr paddr plen ) + r> IF 03000000 ELSE 42000000 THEN \ 64-bit or 32-bit ? + pci-bridge-gen-range \ and generate it ( addr paddr plen ) ; This fixes all the XHCI problems for me.