diff mbox series

[v1] PCI: Set no io resource for bridges that behind VMD controller

Message ID 1663075485-20591-1-git-send-email-lixiaochun.2888@163.com
State New
Headers show
Series [v1] PCI: Set no io resource for bridges that behind VMD controller | expand

Commit Message

Xiaochun Lee Sept. 13, 2022, 1:24 p.m. UTC 
From: Xiaochun Lee <lixc17@lenovo.com>

When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
recognized by VMD driver and there are many failed messages of
BAR 13 when scan the bridges and assign IO resource behind it
as listed below, the bridge wants to get 0x6000 as its IO
resource, but there is no IO resources on the host bridge.

VMD host bridge resources:
vmd 0000:64:00.5: PCI host bridge to bus 10000:80
pci_bus 10000:80: root bus resource [bus 80-9f]
pci_bus 10000:80: root bus resource [mem 0xe0000000-0xe1ffffff]
pci_bus 10000:80: root bus resource [mem 0x24ffff02010-0x24fffffffff 64bit]

Failed messages of BAR#13:
pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000]
pci 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000]
pci 10000:80:03.0: BAR 13: no space for [io  size 0x1000]
pci 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]

VMD-enabled root ports use
Enhanced Configuration Access Mechanism (ECAM) access
PCI Express configuration space, and offer VMD_CFGBAR as
base of PCI Express configuration space for the bridges
behind it. The configuration space includes IO resources,
but these IO resources are not actually used on X86,
especially the NVMes as device connected on this hot plug
bridges, and it can result in BAR#13 assign IO resource
failed. So we clear IO resources by setting an IO base value
greater than limit to these bridges. Hence, we can leverage
kernel parameter "pci=hpiosize=0KB" to avoid this failed
messages show out.

Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>
---
 drivers/pci/quirks.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

Comments

kernel test robot Sept. 23, 2022, 3:21 p.m. UTC | #1 
Hi Xiaochun,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on helgaas-pci/next]
[also build test ERROR on linus/master v6.0-rc6 next-20220923]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
base:   https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: um-allmodconfig (https://download.01.org/0day-ci/archive/20220924/202209240714.ronvmf1X-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-5) 11.3.0
reproduce (this is a W=1 build):
        # https://github.com/intel-lab-lkp/linux/commit/f97a8ba561d7cf5a755c8f42421138e8b1073cf9
        git remote add linux-review https://github.com/intel-lab-lkp/linux
        git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-for-bridges-that-behind-VMD-controller/20220913-213745
        git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
        # save the config file
        mkdir build_dir && cp config build_dir/.config
        make W=1 O=build_dir ARCH=um SHELL=/bin/bash

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
>> drivers/pci/quirks.c:5985:13: error: implicit declaration of function 'is_vmd' [-Werror=implicit-function-declaration]
    5985 |         if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
         |             ^~~~~~
   cc1: some warnings being treated as errors


vim +/is_vmd +5985 drivers/pci/quirks.c

  5959	
  5960	#if defined(CONFIG_X86_64) || defined(CONFIG_X86)
  5961	/*
  5962	 * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
  5963	 * access PCI Express configuration space, and offer VMD_CFGBAR as
  5964	 * base of PCI Express configuration space for the bridges behind it.
  5965	 * The configuration space includes IO resources, but these IO
  5966	 * resources are not actually used on X86, especially the NVMes as
  5967	 * device connnected on this hot plug bridges, and it can result
  5968	 * in BAR#13 assign IO resource failed. So we clear IO resources
  5969	 * by setting an IO base value greater than limit to these bridges.
  5970	 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
  5971	 * this BAR#13 failed messages show out.
  5972	 */
  5973	static void quirk_vmd_no_iosize(struct pci_dev *bridge)
  5974	{
  5975		u8 io_base_lo, io_limit_lo;
  5976		u16 io_low;
  5977		u32 io_upper16;
  5978		unsigned long io_mask,  base, limit;
  5979	
  5980		io_mask = PCI_IO_RANGE_MASK;
  5981		if (bridge->io_window_1k)
  5982			io_mask = PCI_IO_1K_RANGE_MASK;
  5983	
  5984		/* VMD Domain */
> 5985		if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
Xiaochun XC17 Li Sept. 25, 2022, 9:52 a.m. UTC | #2 
Hi,

> -----Original Message-----
> From: kernel test robot <lkp@intel.com>
> Sent: Friday, September 23, 2022 11:21 PM
> To: Xiaochun Lee <lixiaochun.2888@163.com>;
> nirmal.patel@linux.intel.com; jonathan.derrick@linux.dev
> Cc: kbuild-all@lists.01.org; lpieralisi@kernel.org; robh@kernel.org;
> kw@linux.com; bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> kernel@vger.kernel.org; Xiaochun XC17 Li <lixc17@lenovo.com>
> Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> behind VMD controller
> 
> Hi Xiaochun,
> 
> Thank you for the patch! Yet something to improve:
> 
> [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> git tree, kindly drop us a note.
Hi, thanks for your reply, this patch is only expected to be applied
on x86 or x64, actually function "is_vmd()" is just defined in
arch/x86/include/asm/pci.h, do we need to support it on arch um?

> And when submitting patch, we suggest to use '--base' as documented in
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-
> scm.com%2Fdocs%2Fgit-format-
> patch%23_base_tree_information&amp;data=05%7C01%7Clixc17%40lenov
> o.com%7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa9
> 34df372b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWF
> pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=TGZ2LnHsHCrJLW7WBBMZ
> lFxv8SIaQNmCGC1bEv2BEDE%3D&amp;reserved=0]
> 
> url:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-lkp%2Flinux%2Fcommits%2FXiaochun-Lee%2FPCI-Set-
> no-io-resource-for-bridges-that-behind-VMD-controller%2F20220913-
> 213745&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4d
> d9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=3NULvkgYg0fWTopgYY5EYHsLIBPdTEcMxbz5Tj12D0E%
> 3D&amp;reserved=0
> base:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.k
> ernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fhelgaas%2Fpci.git&a
> mp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc08
> da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C63799
> 5433448231363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&a
> mp;sdata=ZTZ7kKlwOk7IB3MN0rRwqHaNGGl4OiwrR7AjImBbTqc%3D&amp;
> reserved=0 next
> config: um-allmodconfig
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdow
> nload.01.org%2F0day-
> ci%2Farchive%2F20220924%2F202209240714.ronvmf1X-
> lkp%40intel.com%2Fconfig&amp;data=05%7C01%7Clixc17%40lenovo.com%
> 7C63bd7990b33f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372
> b16203%7C0%7C0%7C637995433448231363%7CUnknown%7CTWFpbGZsb3
> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000%7C%7C%7C&amp;sdata=6xMM7Fh9bKGnzBlmeu8kki3yvGGQ
> 2k%2FKCp%2BqvoLt4zU%3D&amp;reserved=0)
> compiler: gcc-11 (Debian 11.3.0-5) 11.3.0 reproduce (this is a W=1 build):
>         #
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux%2Fcommit%2Ff97a8ba561d7cf5a755c8f42421138e8b1073cf9&
> amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4dd9b4bc0
> 8da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379
> 95433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAi
> LCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&
> amp;sdata=4mjCA86AEbwN9eIiw1%2BLUPlf0%2BV2juqoDo7ojKSMweY%3D
> &amp;reserved=0
>         git remote add linux-review
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fintel-lab-
> lkp%2Flinux&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b3
> 3f4dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%
> 7C0%7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&amp;sdata=YusFni1tvPN89d2e%2B8yADL3jbwjuydLs0RD3NSjj
> 2MA%3D&amp;reserved=0
>         git fetch --no-tags linux-review Xiaochun-Lee/PCI-Set-no-io-resource-
> for-bridges-that-behind-VMD-controller/20220913-213745
>         git checkout f97a8ba561d7cf5a755c8f42421138e8b1073cf9
>         # save the config file
>         mkdir build_dir && cp config build_dir/.config
>         make W=1 O=build_dir ARCH=um SHELL=/bin/bash
> 
> If you fix the issue, kindly add following tag where applicable
> | Reported-by: kernel test robot <lkp@intel.com>
> 
> All errors (new ones prefixed by >>):
> 
>    drivers/pci/quirks.c: In function 'quirk_vmd_no_iosize':
> >> drivers/pci/quirks.c:5985:13: error: implicit declaration of function
> >> 'is_vmd' [-Werror=implicit-function-declaration]
>     5985 |         if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
>          |             ^~~~~~
>    cc1: some warnings being treated as errors
> 
> 
> vim +/is_vmd +5985 drivers/pci/quirks.c
> 
>   5959
>   5960	#if defined(CONFIG_X86_64) || defined(CONFIG_X86)
>   5961	/*
>   5962	 * VMD-enabled root ports use Enhanced Configuration Access
> Mechanism (ECAM)
>   5963	 * access PCI Express configuration space, and offer VMD_CFGBAR
> as
>   5964	 * base of PCI Express configuration space for the bridges behind it.
>   5965	 * The configuration space includes IO resources, but these IO
>   5966	 * resources are not actually used on X86, especially the NVMes as
>   5967	 * device connnected on this hot plug bridges, and it can result
>   5968	 * in BAR#13 assign IO resource failed. So we clear IO resources
>   5969	 * by setting an IO base value greater than limit to these bridges.
>   5970	 * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
>   5971	 * this BAR#13 failed messages show out.
>   5972	 */
>   5973	static void quirk_vmd_no_iosize(struct pci_dev *bridge)
>   5974	{
>   5975		u8 io_base_lo, io_limit_lo;
>   5976		u16 io_low;
>   5977		u32 io_upper16;
>   5978		unsigned long io_mask,  base, limit;
>   5979
>   5980		io_mask = PCI_IO_RANGE_MASK;
>   5981		if (bridge->io_window_1k)
>   5982			io_mask = PCI_IO_1K_RANGE_MASK;
>   5983
>   5984		/* VMD Domain */
> > 5985		if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
> 
> --
> 0-DAY CI Kernel Test Service
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2F01.or
> g%2Flkp&amp;data=05%7C01%7Clixc17%40lenovo.com%7C63bd7990b33f4
> dd9b4bc08da9d77696c%7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0
> %7C637995433448387557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&amp;sdata=4qYtyE86mw9VyFbB3QKJqtldq6relV2Cunv9STAmmqA
> %3D&amp;reserved=0
Bjorn Helgaas Sept. 25, 2022, 12:57 p.m. UTC | #3 
On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > -----Original Message-----
> > From: kernel test robot <lkp@intel.com>
> > Sent: Friday, September 23, 2022 11:21 PM
> > To: Xiaochun Lee <lixiaochun.2888@163.com>;
> > nirmal.patel@linux.intel.com; jonathan.derrick@linux.dev
> > Cc: kbuild-all@lists.01.org; lpieralisi@kernel.org; robh@kernel.org;
> > kw@linux.com; bhelgaas@google.com; linux-pci@vger.kernel.org; linux-
> > kernel@vger.kernel.org; Xiaochun XC17 Li <lixc17@lenovo.com>
> > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for bridges that
> > behind VMD controller
> > 
> > Hi Xiaochun,
> > 
> > Thank you for the patch! Yet something to improve:
> > 
> > [auto build test ERROR on helgaas-pci/next] [also build test ERROR on
> > linus/master v6.0-rc6 next-20220923] [If your patch is applied to the wrong
> > git tree, kindly drop us a note.
> Hi, thanks for your reply, this patch is only expected to be applied
> on x86 or x64, actually function "is_vmd()" is just defined in
> arch/x86/include/asm/pci.h, do we need to support it on arch um?

Yes, all possible Kconfig configurations must build cleanly.  You
may have to add additional ifdef tests or an is_vmd() stub.

For future reference, your email reply doesn't follow the usual Linux
mailing list style, so it is unnecessarily hard to read.  In
particular, it lacks the line that shows what you're responding to.
It would look something like this:

  On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:

Here's a sample:
https://lore.kernel.org/r/YyGghUdcrOdrR0ep@smile.fi.intel.com

More background:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
Xiaochun XC17 Li Sept. 27, 2022, 2:40 a.m. UTC | #4 
On Sun, Sep 25, 2022 at 8:57 PM +0800, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > > -----Original Message-----
> > > From: kernel test robot <lkp@intel.com>
> > > Sent: Friday, September 23, 2022 11:21 PM
> > > To: Xiaochun Lee <lixiaochun.2888@163.com>;
> > > nirmal.patel@linux.intel.com; jonathan.derrick@linux.dev
> > > Cc: kbuild-all@lists.01.org; lpieralisi@kernel.org; robh@kernel.org;
> > > kw@linux.com; bhelgaas@google.com; linux-pci@vger.kernel.org;
> linux-
> > > kernel@vger.kernel.org; Xiaochun XC17 Li <lixc17@lenovo.com>
> > > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for
> > > bridges that behind VMD controller
> > >
> > > Hi Xiaochun,
> > >
> > > Thank you for the patch! Yet something to improve:
> > >
> > > [auto build test ERROR on helgaas-pci/next] [also build test ERROR
> > > on linus/master v6.0-rc6 next-20220923] [If your patch is applied to
> > > the wrong git tree, kindly drop us a note.
> > Hi, thanks for your reply, this patch is only expected to be applied
> > on x86 or x64, actually function "is_vmd()" is just defined in
> > arch/x86/include/asm/pci.h, do we need to support it on arch um?
> 
> Yes, all possible Kconfig configurations must build cleanly.  You may have
> to add additional ifdef tests or an is_vmd() stub.

Thanks, I'll fix it and complier it base arch um, the patch v2 will be sent out.

> 
> For future reference, your email reply doesn't follow the usual Linux
> mailing list style, so it is unnecessarily hard to read.  In particular, it lacks
> the line that shows what you're responding to.
> It would look something like this:
> 
>   On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:

OK, thanks! 
I did a test just now, it seems to meet the format requirements.
I sent emails via outlook, and I've tried to set the reply format, 
unfortunately, the time zone and date time  are not resolved correctly.

> 
> Here's a sample:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fr%2FYyGghUdcrOdrR0ep%40smile.fi.intel.com&amp;data=05
> %7C01%7Clixc17%40lenovo.com%7C27c59eea4afb46c8e24608da9ef5752d%
> 7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379970750209852
> 91%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&amp;sdata=Y8
> O9DzJ%2FW8JlAz0LM8o6LgnQcWUbYlxseM1wWbLvUKI%3D&amp;reserved
> =0
> 
> More background:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.
> wikipedia.org%2Fwiki%2FPosting_style%23Interleaved_style&amp;data=05
> %7C01%7Clixc17%40lenovo.com%7C27c59eea4afb46c8e24608da9ef5752d%
> 7C5c7d0b28bdf8410caa934df372b16203%7C0%7C0%7C6379970750209852
> 91%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&amp;sdata=3Q
> oamhkXi78AOmFMuCRLnL52Ed8yEZ79Vx72pEpE4cs%3D&amp;reserved=0
Bjorn Helgaas Sept. 27, 2022, 3:52 p.m. UTC | #5 
On Tue, Sep 27, 2022 at 02:40:39AM +0000, Xiaochun XC17 Li wrote:
> On Sun, Sep 25, 2022 at 8:57 PM +0800, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Sun, Sep 25, 2022 at 09:52:21AM +0000, Xiaochun XC17 Li wrote:
> > > > -----Original Message-----
> > > > From: kernel test robot <lkp@intel.com>
> > > > Sent: Friday, September 23, 2022 11:21 PM
> > > > To: Xiaochun Lee <lixiaochun.2888@163.com>;
> > > > nirmal.patel@linux.intel.com; jonathan.derrick@linux.dev
> > > > Cc: kbuild-all@lists.01.org; lpieralisi@kernel.org; robh@kernel.org;
> > > > kw@linux.com; bhelgaas@google.com; linux-pci@vger.kernel.org;
> > linux-
> > > > kernel@vger.kernel.org; Xiaochun XC17 Li <lixc17@lenovo.com>
> > > > Subject: [External] Re: [PATCH v1] PCI: Set no io resource for
> > > > bridges that behind VMD controller

> > ...
> > For future reference, your email reply doesn't follow the usual Linux
> > mailing list style, so it is unnecessarily hard to read.  In particular, it lacks
> > the line that shows what you're responding to.
> > It would look something like this:
> > 
> >   On Fri, Sep 23, 2022 at 11:21PM +0800, kernel test robot wrote:
> 
> OK, thanks! 
> I did a test just now, it seems to meet the format requirements.
> I sent emails via outlook, and I've tried to set the reply format, 
> unfortunately, the time zone and date time  are not resolved correctly.

Much better, thanks!  It's much easier to read without all the
redundant headers.

Bjorn
Bjorn Helgaas Nov. 11, 2022, 11:54 p.m. UTC | #6 
On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> From: Xiaochun Lee <lixc17@lenovo.com>
> 
> When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> recognized by VMD driver and there are many failed messages of
> BAR 13 when scan the bridges and assign IO resource behind it
> as listed below, the bridge wants to get 0x6000 as its IO
> resource, but there is no IO resources on the host bridge.
> 
> VMD host bridge resources:
> vmd 0000:64:00.5: PCI host bridge to bus 10000:80
> pci_bus 10000:80: root bus resource [bus 80-9f]
> pci_bus 10000:80: root bus resource [mem 0xe0000000-0xe1ffffff]
> pci_bus 10000:80: root bus resource [mem 0x24ffff02010-0x24fffffffff 64bit]
> 
> Failed messages of BAR#13:
> pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000]
> pci 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000]
> pci 10000:80:03.0: BAR 13: no space for [io  size 0x1000]
> pci 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]
> 
> VMD-enabled root ports use
> Enhanced Configuration Access Mechanism (ECAM) access
> PCI Express configuration space, and offer VMD_CFGBAR as
> base of PCI Express configuration space for the bridges
> behind it. The configuration space includes IO resources,
> but these IO resources are not actually used on X86,
> especially the NVMes as device connected on this hot plug
> bridges, and it can result in BAR#13 assign IO resource
> failed. So we clear IO resources by setting an IO base value
> greater than limit to these bridges. Hence, we can leverage
> kernel parameter "pci=hpiosize=0KB" to avoid this failed
> messages show out.
> 
> Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>

Some of the discussion here got lost because of email issues.  Lore
has some:
https://lore.kernel.org/all/1663075485-20591-1-git-send-email-lixiaochun.2888@163.com/T/#u,
and patchwork has a v2 with a little more discussion:
https://patchwork.kernel.org/project/linux-pci/patch/1664288166-7432-1-git-send-email-lixiaochun.2888@163.com/

But the v2 patch doesn't seem to have made it to the mailing lists or
to lore (https://lore.kernel.org/all/?q=f%3Alixc17) and I don't apply
things until they appear on the mailing list.

I *would* like to get rid of those "no space" and "failed to assign"
messages.  This is an issue for platforms other than VMD, too.  Just
an FYI that you need to follow up on this if we want make progress.

Bjorn
Xiaochun XC17 Li Nov. 16, 2022, 3:34 a.m. UTC | #7 
On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <helgaas@kernel.org> wrote:
> On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > From: Xiaochun Lee <lixc17@lenovo.com>
> >
> > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > recognized by VMD driver and there are many failed messages of BAR 13
> > when scan the bridges and assign IO resource behind it as listed
> > below, the bridge wants to get 0x6000 as its IO resource, but there is
> > no IO resources on the host bridge.
> >
> > VMD host bridge resources:
> > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource [mem
> > 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource [mem
> > 0x24ffff02010-0x24fffffffff 64bit]
> >
> > Failed messages of BAR#13:
> > pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000] pci
> > 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000] pci
> > 10000:80:03.0: BAR 13: no space for [io  size 0x1000] pci
> > 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]
> >
> > VMD-enabled root ports use
> > Enhanced Configuration Access Mechanism (ECAM) access PCI Express
> > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > configuration space for the bridges behind it. The configuration space
> > includes IO resources, but these IO resources are not actually used on
> > X86, especially the NVMes as device connected on this hot plug
> > bridges, and it can result in BAR#13 assign IO resource failed. So we
> > clear IO resources by setting an IO base value greater than limit to
> > these bridges. Hence, we can leverage kernel parameter
> > "pci=hpiosize=0KB" to avoid this failed messages show out.
> >
> > Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>
> 
> Some of the discussion here got lost because of email issues.  Lore has
> some:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> lixiaochun.2888%40163.com%2FT%2F%23u&amp;data=05%7C01%7Clixc17
> %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf
> 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JzWYkIsaEfINofiqo
> XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&amp;reserved=0,
> and patchwork has a v2 with a little more discussion:
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-7432-1-
> git-send-email-
> lixiaochun.2888%40163.com%2F&amp;data=05%7C01%7Clixc17%40lenovo.
> com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> tiMjY0TONfK7zNCKXQ90%3D&amp;reserved=0
> 
> But the v2 patch doesn't seem to have made it to the mailing lists or to
> lore
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
> e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&amp;data=05%7C01%7Clixc1
> 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=zxGz1hZOD2tvQP
> EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&amp;reserved=0) and I don't
> apply things until they appear on the mailing list.
> 
> I *would* like to get rid of those "no space" and "failed to assign"
> messages.  This is an issue for platforms other than VMD, too.  Just an FYI
> that you need to follow up on this if we want make progress.
> 
> Bjorn

Thanks for your comments, so do you mean we'd better come up with a
solution to avoid messages like "no space" and "failed to assign" using
a common way, for both VMD and other platforms?

Xiaochun
Bjorn Helgaas Nov. 16, 2022, 4:05 a.m. UTC | #8 
On Wed, Nov 16, 2022 at 03:34:34AM +0000, Xiaochun XC17 Li wrote:
> On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <helgaas@kernel.org> wrote:
> > On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > > From: Xiaochun Lee <lixc17@lenovo.com>
> > >
> > > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > > recognized by VMD driver and there are many failed messages of BAR 13
> > > when scan the bridges and assign IO resource behind it as listed
> > > below, the bridge wants to get 0x6000 as its IO resource, but there is
> > > no IO resources on the host bridge.
> > >
> > > VMD host bridge resources:
> > > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource [mem
> > > 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource [mem
> > > 0x24ffff02010-0x24fffffffff 64bit]
> > >
> > > Failed messages of BAR#13:
> > > pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000] pci
> > > 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000] pci
> > > 10000:80:03.0: BAR 13: no space for [io  size 0x1000] pci
> > > 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]
> > >
> > > VMD-enabled root ports use
> > > Enhanced Configuration Access Mechanism (ECAM) access PCI Express
> > > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > > configuration space for the bridges behind it. The configuration space
> > > includes IO resources, but these IO resources are not actually used on
> > > X86, especially the NVMes as device connected on this hot plug
> > > bridges, and it can result in BAR#13 assign IO resource failed. So we
> > > clear IO resources by setting an IO base value greater than limit to
> > > these bridges. Hence, we can leverage kernel parameter
> > > "pci=hpiosize=0KB" to avoid this failed messages show out.
> > >
> > > Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>
> > 
> > Some of the discussion here got lost because of email issues.  Lore has
> > some:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> > lixiaochun.2888%40163.com%2FT%2F%23u&amp;data=05%7C01%7Clixc17
> > %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf
> > 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> > WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JzWYkIsaEfINofiqo
> > XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&amp;reserved=0,
> > and patchwork has a v2 with a little more discussion:
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> > hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-7432-1-
> > git-send-email-
> > lixiaochun.2888%40163.com%2F&amp;data=05%7C01%7Clixc17%40lenovo.
> > com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> > df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> > tiMjY0TONfK7zNCKXQ90%3D&amp;reserved=0
> > 
> > But the v2 patch doesn't seem to have made it to the mailing lists or to
> > lore
> > (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
> > e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&amp;data=05%7C01%7Clixc1
> > 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> > f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> > n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> > aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=zxGz1hZOD2tvQP
> > EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&amp;reserved=0) and I don't
> > apply things until they appear on the mailing list.
> > 
> > I *would* like to get rid of those "no space" and "failed to assign"
> > messages.  This is an issue for platforms other than VMD, too.  Just an FYI
> > that you need to follow up on this if we want make progress.
> 
> Thanks for your comments, so do you mean we'd better come up with a
> solution to avoid messages like "no space" and "failed to assign" using
> a common way, for both VMD and other platforms?

I tried to say two separate things:

1) It doesn't seem like a VMD-specific thing, so it would be ideal if
the solution were generic instead of being VMD-specific.

2) Some of your previous patches didn't make it to the mailing list,
so we couldn't really do anything with them.

Bjorn
Xiaochun XC17 Li Nov. 16, 2022, 4:26 a.m. UTC | #9 
On Wed, Nov 16, 2022 at 12:10 PM +0800, "Bjorn Helgaas" <helgaas@kernel.org> wrote:
> On Wed, Nov 16, 2022 at 03:34:34AM +0000, Xiaochun XC17 Li wrote:
> > On Sat, Nov 11, 2022 at 07:54:25, "Bjorn Helgaas" <helgaas@kernel.org>
> wrote:
> > > On Tue, Sep 13, 2022 at 09:24:45PM +0800, Xiaochun Lee wrote:
> > > > From: Xiaochun Lee <lixc17@lenovo.com>
> > > >
> > > > When enable VMDs on Intel CPUs, VMD controllers(8086:28c0) be
> > > > recognized by VMD driver and there are many failed messages of BAR
> > > > 13 when scan the bridges and assign IO resource behind it as
> > > > listed below, the bridge wants to get 0x6000 as its IO resource,
> > > > but there is no IO resources on the host bridge.
> > > >
> > > > VMD host bridge resources:
> > > > vmd 0000:64:00.5: PCI host bridge to bus 10000:80 pci_bus 10000:80:
> > > > root bus resource [bus 80-9f] pci_bus 10000:80: root bus resource
> > > > [mem 0xe0000000-0xe1ffffff] pci_bus 10000:80: root bus resource
> > > > [mem 0x24ffff02010-0x24fffffffff 64bit]
> > > >
> > > > Failed messages of BAR#13:
> > > > pci 10000:80:02.0: BAR 13: no space for [io  size 0x1000] pci
> > > > 10000:80:02.0: BAR 13: failed to assign [io  size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: no space for [io  size 0x1000] pci
> > > > 10000:80:03.0: BAR 13: failed to assign [io  size 0x1000]
> > > >
> > > > VMD-enabled root ports use
> > > > Enhanced Configuration Access Mechanism (ECAM) access PCI
> Express
> > > > configuration space, and offer VMD_CFGBAR as base of PCI Express
> > > > configuration space for the bridges behind it. The configuration
> > > > space includes IO resources, but these IO resources are not
> > > > actually used on X86, especially the NVMes as device connected on
> > > > this hot plug bridges, and it can result in BAR#13 assign IO
> > > > resource failed. So we clear IO resources by setting an IO base
> > > > value greater than limit to these bridges. Hence, we can leverage
> > > > kernel parameter "pci=hpiosize=0KB" to avoid this failed messages
> show out.
> > > >
> > > > Signed-off-by: Xiaochun Lee <lixc17@lenovo.com>
> > >
> > > Some of the discussion here got lost because of email issues.  Lore
> > > has
> > > some:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> > > kernel.org%2Fall%2F1663075485-20591-1-git-send-email-
> > >
> lixiaochun.2888%40163.com%2FT%2F%23u&amp;data=05%7C01%7Clixc17
> > > %40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28
> bdf
> > >
> 8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknown
> > > %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> ha
> > >
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=JzWYkIsaEfINofiqo
> > > XyjEh43VjXO3HZw2JLSsmhpUiQ%3D&amp;reserved=0,
> > > and patchwork has a v2 with a little more discussion:
> > >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpa
> > > tc
> > > hwork.kernel.org%2Fproject%2Flinux-pci%2Fpatch%2F1664288166-
> 7432-1-
> > > git-send-email-
> > >
> lixiaochun.2888%40163.com%2F&amp;data=05%7C01%7Clixc17%40lenovo.
> > >
> com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bdf8410caa934
> > >
> df372b16203%7C0%7C0%7C638038076734438158%7CUnknown%7CTWFpb
> > >
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > >
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y%2BtIsepTpyLeoHW6CrgkZz2
> > > tiMjY0TONfK7zNCKXQ90%3D&amp;reserved=0
> > >
> > > But the v2 patch doesn't seem to have made it to the mailing lists
> > > or to lore
> > >
> (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> > > or
> > >
> e.kernel.org%2Fall%2F%3Fq%3Df%253Alixc17&amp;data=05%7C01%7Clixc1
> > >
> 7%40lenovo.com%7C9cd095ffdb584e492dec08dac440139b%7C5c7d0b28bd
> > >
> f8410caa934df372b16203%7C0%7C0%7C638038076734438158%7CUnknow
> > >
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
> > >
> aWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=zxGz1hZOD2tvQP
> > > EsbxQTzjHwQvXvqeO%2FUd6I9S%2Fj314%3D&amp;reserved=0) and I
> don't
> > > apply things until they appear on the mailing list.
> > >
> > > I *would* like to get rid of those "no space" and "failed to assign"
> > > messages.  This is an issue for platforms other than VMD, too.  Just
> > > an FYI that you need to follow up on this if we want make progress.
> >
> > Thanks for your comments, so do you mean we'd better come up with a
> > solution to avoid messages like "no space" and "failed to assign"
> > using a common way, for both VMD and other platforms?
> 
> I tried to say two separate things:
> 
> 1) It doesn't seem like a VMD-specific thing, so it would be ideal if the
> solution were generic instead of being VMD-specific.

OK, I'll try to find out a generic approach.

> 
> 2) Some of your previous patches didn't make it to the mailing list, so we
> couldn't really do anything with them.
> 
> Bjorn

Actually, the V2 patch didn't following the mailing of V1 patch,
I sent it out separately like the first time for V1 patch, I don't know if there
is a mistake in the way I sent the V2 patch. I was wondering if I was going to
send out V2 patches based on the mailing list of V1 patch, if it is yes, I must
study how to summit the V2 patch, so would you please correct me?

Xiaochun
Bjorn Helgaas Nov. 16, 2022, 5:42 p.m. UTC | #10 
On Wed, Nov 16, 2022 at 04:26:02AM +0000, Xiaochun XC17 Li wrote:
> On Wed, Nov 16, 2022 at 12:10 PM +0800, "Bjorn Helgaas" <helgaas@kernel.org> wrote:

> > 2) Some of your previous patches didn't make it to the mailing list, so we
> > couldn't really do anything with them.
> > 
> > Bjorn
> 
> Actually, the V2 patch didn't following the mailing of V1 patch, I
> sent it out separately like the first time for V1 patch, I don't
> know if there is a mistake in the way I sent the V2 patch. I was
> wondering if I was going to send out V2 patches based on the mailing
> list of V1 patch, if it is yes, I must study how to summit the V2
> patch, so would you please correct me?

I was mistaken, the V2 patch *was* on the mailing list:
  https://lore.kernel.org/r/1664288166-7432-1-git-send-email-lixiaochun.2888@163.com

I guess I didn't find it because I was searching for the email address
"lixc17@lenovo.com", and v2 was posted by "lixiaochun.2888@163.com".

Sorry, my mistake!  It looks like the sending of v2 was just fine.

Bjorn
diff mbox series

Patch

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 4944798..f8a37f0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5956,3 +5956,60 @@  static void aspm_l1_acceptable_latency(struct pci_dev *dev)
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency);
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency);
 #endif
+
+#if defined(CONFIG_X86_64) || defined(CONFIG_X86)
+/*
+ * VMD-enabled root ports use Enhanced Configuration Access Mechanism (ECAM)
+ * access PCI Express configuration space, and offer VMD_CFGBAR as
+ * base of PCI Express configuration space for the bridges behind it.
+ * The configuration space includes IO resources, but these IO
+ * resources are not actually used on X86, especially the NVMes as
+ * device connnected on this hot plug bridges, and it can result
+ * in BAR#13 assign IO resource failed. So we clear IO resources
+ * by setting an IO base value greater than limit to these bridges.
+ * Hence, append kernel parameter "pci=hpiosize=0KB" can avoid
+ * this BAR#13 failed messages show out.
+ */
+static void quirk_vmd_no_iosize(struct pci_dev *bridge)
+{
+	u8 io_base_lo, io_limit_lo;
+	u16 io_low;
+	u32 io_upper16;
+	unsigned long io_mask,  base, limit;
+
+	io_mask = PCI_IO_RANGE_MASK;
+	if (bridge->io_window_1k)
+		io_mask = PCI_IO_1K_RANGE_MASK;
+
+	/* VMD Domain */
+	if (is_vmd(bridge->bus) && bridge->is_hotplug_bridge) {
+		pci_read_config_byte(bridge, PCI_IO_BASE, &io_base_lo);
+		pci_read_config_byte(bridge, PCI_IO_LIMIT, &io_limit_lo);
+		base = (io_base_lo & io_mask) << 8;
+		limit = (io_limit_lo & io_mask) << 8;
+		if (limit >= base) {
+			/* if there are defined io ports behind the bridge on x86,
+			 * we clear it, since there is only 64KB IO resource on it,
+			 * beyond that, hotplug io bridges don't needs IO port resource,
+			 * such as NVMes attach on it. So the corresponding range must be
+			 * turned off by writing base value greater than limit to the
+			 * bridge's base/limit registers.
+			 */
+
+			/* Clear upper 16 bits of I/O base/limit */
+			io_upper16 = 0;
+			/* set base value greater than limit */
+			io_low = 0x00f0;
+
+			/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+			pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, 0x0000ffff);
+			/* Update lower 16 bits of I/O base/limit */
+			pci_write_config_word(bridge, PCI_IO_BASE, io_low);
+			/* Update upper 16 bits of I/O base/limit */
+			pci_write_config_dword(bridge, PCI_IO_BASE_UPPER16, io_upper16);
+		}
+	}
+}
+DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_ANY_ID, PCI_ANY_ID,
+		PCI_CLASS_BRIDGE_PCI, 8, quirk_vmd_no_iosize);
+#endif