diff mbox

[Xen-devel] 3.18 xen-pcifront regression?

Message ID 20150324172702.GC2495@google.com
State Not Applicable
Headers show

Commit Message

Bjorn Helgaas March 24, 2015, 5:27 p.m. UTC
[+cc Rafael, linux-pci, linux-acpi]

On Tue, Mar 24, 2015 at 11:28:06AM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Mar 24, 2015 at 11:14:29AM -0400, Michael D Labriola wrote:
> > I'm having problems booting a 3.18 or newer domU w/ PCI devices passed 
> > through.  It only seems to be the domU kernel that's upset (i.e., Behavior 
> > is identical whether I'm running 3.19 or 3.13 dom0).  I'm running a 32bit 
> > dom0 (3.13.11) w/ 64bit 4.4.0 hypervisor and 32bit domU.  I get the 
> > following Oops when trying to boot my domU with a couple PCI cards passed 
> > through:
> > 
> > BUG: unable to handle kernel paging request at 0030303e
> > IP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a
> > *pdpt = 00000000019f1027 *pde = 0000000000000000 
> > Oops: 0000 [#1] PREEMPT SMP 
> > Modules linked in: xen_pcifront(+) pcspkr xen_blkfront loop
> > CPU: 0 PID: 18 Comm: xenwatch Not tainted 3.17.0-test+ #6
> > task: cb869950 ti: cb8ae000 task.ti: cb8ae000
> > EIP: 0061:[<c06ed0e6>] EFLAGS: 00010246 CPU: 0
> > EIP is at acpi_ns_validate_handle+0x12/0x1a
> > EAX: 00000000 EBX: cb895dc0 ECX: 00000000 EDX: 0030303a
> > ESI: c0a6bccd EDI: 00000000 EBP: 00000004 ESP: cb8afd00
> >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
> > CR0: 8005003b CR2: 0030303e CR3: 0a68e000 CR4: 00040660
> > Stack:
> >  c06eda4d 00000000 c096a21d 00000000 00000000 00006462 00000000 c0c102c0
> >  0030303a 00040004 0030303a cb8afd94 cb8afdec cb8afd60 c06b78e1 cb8afd60
> >  00000061 00000246 c0407bc7 c0c102c0 00000000 cb8afda0 cb8afda8 cb8afdb0
> > Call Trace:
> >  [<c06eda4d>] ? acpi_evaluate_object+0x31/0x1fc
> 
> We should not be calling in any acpi code in PV domU guests.
> 
> WE actually disable it (acpi=0) to make sure we don't call it - as
> there is no ACPI AML data at all in the guest.
> 
> CC-ing Bjorn.
> >  [<c096a21d>] ? resume_kernel+0x5/0x7
> >  [<c06b78e1>] ? pci_get_hp_params+0x111/0x4e0
> >  [<c0407bc7>] ? xen_force_evtchn_callback+0x17/0x30
> >  [<c04085fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
> >  [<c0699d34>] ? pci_device_add+0x24/0x450
> >  [<c06975ce>] ? pci_bus_read_config_word+0x6e/0x80
> >  [<c069a66d>] ? pci_scan_single_device+0x8d/0xb0
> >  [<c069a6cc>] ? pci_scan_slot+0x3c/0xf0
> >  [<c069b1bc>] ? pci_scan_child_bus+0x1c/0x90
> >  [<c069b71a>] ? pci_scan_bus_parented+0x6a/0x90
> >  [<d088d241>] ? pcifront_scan_root+0x91/0x130 [xen_pcifront]
> >  [<d088e69f>] ? pcifront_backend_changed+0x4af/0x654 [xen_pcifront]
> >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> >  [<c070c393>] ? xenbus_read_driver_state+0x33/0x50
> >  [<c070f375>] ? xenbus_otherend_changed+0x95/0xa0
> >  [<c0710a5f>] ? backend_changed+0xf/0x20
> >  [<c070d712>] ? xenwatch_thread+0x72/0x110
> >  [<c0486140>] ? bit_waitqueue+0x50/0x50
> >  [<c070d6a0>] ? join+0x70/0x70
> >  [<c046e59b>] ? kthread+0xab/0xd0
> >  [<c096a1c1>] ? ret_from_kernel_thread+0x21/0x30
> >  [<c046e4f0>] ? flush_kthread_worker+0xa0/0xa0
> > Code: 03 10 00 00 eb 0e 46 83 c2 04 4b 85 db 75 b9 c6 02 00 31 c0 5b 5e 5f 
> > 5d c3 89 c2 8d 40 ff 83 f8 fd 76 06 a1 2c 32 c1 c0 c3 31 c0 <80> 7a 04 0f 
> > 0f 44 c2 c3 83 ec 10 83 f8 1d 76 24 89 44 24 0c c7
> > EIP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a SS:ESP 0069:cb8afd00
> > CR2: 000000000030303e
> > ---[ end trace d4ddeb038cbcbdf7 ]---
> > 
> > 
> > I've bisected down to the following commit in 3.18, which breaks my 
> > system.
> > 
> > 6cd33649fa83d97ba7b66f1d871a360e867c5220 is the first bad commit
> > commit 6cd33649fa83d97ba7b66f1d871a360e867c5220
> > Author: Bjorn Helgaas <bhelgaas@google.com>
> > Date:   Wed Aug 27 14:29:47 2014 -0600
> > 
> >     PCI: Add pci_configure_device() during enumeration
> >  
> >     Some platforms can tell the OS how to configure PCI devices, e.g., how 
> > to
> >     set cache line size, error reporting enables, etc.  ACPI defines _HPP 
> > and
> >     _HPX methods for this purpose.
> >  
> >     This configuration was previously done by some of the hotplug drivers 
> > using
> >     pci_configure_slot().  But not all hotplug drivers did this, and per 
> > the
> >     spec (ACPI rev 5.0, sec 6.2.7), we can also do it for "devices not
> >     configured by the BIOS at system boot."
> >  
> >     Move this configuration into the PCI core by adding 
> > pci_configure_device()
> >     and calling it from pci_device_add(), so we do this for all devices as 
> > we
> >     enumerate them.
> >  
> >     This is based on pci_configure_slot(), which is used by hotplug 
> > drivers.
> >     I omitted:
> >  
> >       - pcie_bus_configure_settings() because it configures MPS and MRRS, 
> > which
> >         requires global knowledge of the fabric and must be done later, 
> > and
> >  
> >       - configuration of subordinate devices; that will happen when we 
> > call
> >         pci_device_add() for those devices.
> >  
> >     Because pci_configure_slot() was only done by hotplug drivers, this 
> > initial
> >     version of pci_configure_device() only configures hot-added devices,
> >     ignoring anything added during boot.
> >  
> >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> >     Acked-by: Yinghai Lu <yinghai@kernel.org>
> > 
> > :040000 040000 4fadbe1e5f8f18daa6be7bdb7c9c1d6def0a2615 
> > 9aef037aa35ca156ac46553f7fc4c5b1b3980c19 M      drivers
> > 
> > 
> > I've reverted that commit on top of 3.19, which feels incredibly wrong, 
> > but does fix the problem on my system.  This is a little over my head, 
> > though...  ;-)
> > 
> > Thoughts?

Thanks for the report, Michael, and sorry for the inconvenience.  I think
the patch below will fix it, but I don't think it's the right fix either
because it seems a little ad hoc to sprinkle "acpi_pci_disabled" tests
around like fairy dust.  I wonder if we can set things up so ACPI methods
would fail gracefully like they do when ACPI is disabled at compile-time.

I can boot with "acpi=off" on qemu just fine, and when we look up the ACPI
device handles, we just get NULL pointers, so everything works out even
without a fix like the one below.

There must be something different about the way things get set up in that
domU kernel.  I'll try to look into that some more, but I'm going on
vacation for the next week, so if you learn anything before then, let me
know.

Bjorn


commit 6678b0fb6504c890481863b4916089b41e6042bf
Author: Bjorn Helgaas <bhelgaas@google.com>
Date:   Tue Mar 24 11:12:45 2015 -0500

    PCI: Don't look for ACPI hotplug parameters if ACPI is disabled
    
    In a kernel with CONFIG_ACPI=y, pci_get_hp_params() evaluates ACPI methods
    (_HPX, _HPP, etc.) to learn how to configure devices.  If ACPI has been
    disabled at runtime, e.g., with "acpi=off", this causes an oops because
    there's no AML at all.
    
    Before 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration"),
    we only used pci_get_hp_params() for hot-added devices, but after it, we
    use it for all devices, so we're much more likely to see the oops.
    
    Don't bother looking for ACPI configuration information if ACPI has been
    disabled.
    
    Fixes: 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration")
    Reported-by: Michael D Labriola <mlabriol@gdeb.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
    CC: stable@vger.kernel.org	# v3.18+

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Michael D Labriola March 24, 2015, 10:27 p.m. UTC | #1
Bjorn Helgaas <bhelgaas@google.com> wrote on 03/24/2015 01:27:02 PM:

> From: Bjorn Helgaas <bhelgaas@google.com>
> To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, 
> Cc: Michael D Labriola <mlabriol@gdeb.com>, xen-
> devel@lists.xenproject.org, Stuart Wehrly <swehrly@gdeb.com>, 
> michael.d.labriola@gmail.com, Jayson A Dyke <jdyke@gdeb.com>, "Rafael 
> J. Wysocki" <rjw@rjwysocki.net>, linux-pci@vger.kernel.org, linux-
> acpi@vger.kernel.org
> Date: 03/24/2015 01:29 PM
> Subject: Re: [Xen-devel] 3.18 xen-pcifront regression?
> 
> [+cc Rafael, linux-pci, linux-acpi]
> 
> On Tue, Mar 24, 2015 at 11:28:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Tue, Mar 24, 2015 at 11:14:29AM -0400, Michael D Labriola wrote:
> > > I'm having problems booting a 3.18 or newer domU w/ PCI devices 
passed 
> > > through.  It only seems to be the domU kernel that's upset (i.e., 
Behavior 
> > > is identical whether I'm running 3.19 or 3.13 dom0).  I'm running a 
32bit 
> > > dom0 (3.13.11) w/ 64bit 4.4.0 hypervisor and 32bit domU.  I get the 
> > > following Oops when trying to boot my domU with a couple PCI cards 
passed 
> > > through:
> > > 
> > > BUG: unable to handle kernel paging request at 0030303e
> > > IP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a
> > > *pdpt = 00000000019f1027 *pde = 0000000000000000 
> > > Oops: 0000 [#1] PREEMPT SMP 
> > > Modules linked in: xen_pcifront(+) pcspkr xen_blkfront loop
> > > CPU: 0 PID: 18 Comm: xenwatch Not tainted 3.17.0-test+ #6
> > > task: cb869950 ti: cb8ae000 task.ti: cb8ae000
> > > EIP: 0061:[<c06ed0e6>] EFLAGS: 00010246 CPU: 0
> > > EIP is at acpi_ns_validate_handle+0x12/0x1a
> > > EAX: 00000000 EBX: cb895dc0 ECX: 00000000 EDX: 0030303a
> > > ESI: c0a6bccd EDI: 00000000 EBP: 00000004 ESP: cb8afd00
> > >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
> > > CR0: 8005003b CR2: 0030303e CR3: 0a68e000 CR4: 00040660
> > > Stack:
> > >  c06eda4d 00000000 c096a21d 00000000 00000000 00006462 00000000 
c0c102c0
> > >  0030303a 00040004 0030303a cb8afd94 cb8afdec cb8afd60 c06b78e1 
cb8afd60
> > >  00000061 00000246 c0407bc7 c0c102c0 00000000 cb8afda0 cb8afda8 
cb8afdb0
> > > Call Trace:
> > >  [<c06eda4d>] ? acpi_evaluate_object+0x31/0x1fc
> > 
> > We should not be calling in any acpi code in PV domU guests.
> > 
> > WE actually disable it (acpi=0) to make sure we don't call it - as
> > there is no ACPI AML data at all in the guest.
> > 
> > CC-ing Bjorn.
> > >  [<c096a21d>] ? resume_kernel+0x5/0x7
> > >  [<c06b78e1>] ? pci_get_hp_params+0x111/0x4e0
> > >  [<c0407bc7>] ? xen_force_evtchn_callback+0x17/0x30
> > >  [<c04085fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
> > >  [<c0699d34>] ? pci_device_add+0x24/0x450
> > >  [<c06975ce>] ? pci_bus_read_config_word+0x6e/0x80
> > >  [<c069a66d>] ? pci_scan_single_device+0x8d/0xb0
> > >  [<c069a6cc>] ? pci_scan_slot+0x3c/0xf0
> > >  [<c069b1bc>] ? pci_scan_child_bus+0x1c/0x90
> > >  [<c069b71a>] ? pci_scan_bus_parented+0x6a/0x90
> > >  [<d088d241>] ? pcifront_scan_root+0x91/0x130 [xen_pcifront]
> > >  [<d088e69f>] ? pcifront_backend_changed+0x4af/0x654 [xen_pcifront]
> > >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> > >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> > >  [<c070c393>] ? xenbus_read_driver_state+0x33/0x50
> > >  [<c070f375>] ? xenbus_otherend_changed+0x95/0xa0
> > >  [<c0710a5f>] ? backend_changed+0xf/0x20
> > >  [<c070d712>] ? xenwatch_thread+0x72/0x110
> > >  [<c0486140>] ? bit_waitqueue+0x50/0x50
> > >  [<c070d6a0>] ? join+0x70/0x70
> > >  [<c046e59b>] ? kthread+0xab/0xd0
> > >  [<c096a1c1>] ? ret_from_kernel_thread+0x21/0x30
> > >  [<c046e4f0>] ? flush_kthread_worker+0xa0/0xa0
> > > Code: 03 10 00 00 eb 0e 46 83 c2 04 4b 85 db 75 b9 c6 02 00 31 c0 5b 
5e 5f 
> > > 5d c3 89 c2 8d 40 ff 83 f8 fd 76 06 a1 2c 32 c1 c0 c3 31 c0 <80> 7a 
04 0f 
> > > 0f 44 c2 c3 83 ec 10 83 f8 1d 76 24 89 44 24 0c c7
> > > EIP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a SS:ESP 
0069:cb8afd00
> > > CR2: 000000000030303e
> > > ---[ end trace d4ddeb038cbcbdf7 ]---
> > > 
> > > 
> > > I've bisected down to the following commit in 3.18, which breaks my 
> > > system.
> > > 
> > > 6cd33649fa83d97ba7b66f1d871a360e867c5220 is the first bad commit
> > > commit 6cd33649fa83d97ba7b66f1d871a360e867c5220
> > > Author: Bjorn Helgaas <bhelgaas@google.com>
> > > Date:   Wed Aug 27 14:29:47 2014 -0600
> > > 
> > >     PCI: Add pci_configure_device() during enumeration
> > > 
> > >     Some platforms can tell the OS how to configure PCI devices, 
e.g., how 
> > > to
> > >     set cache line size, error reporting enables, etc.  ACPI defines 
_HPP 
> > > and
> > >     _HPX methods for this purpose.
> > > 
> > >     This configuration was previously done by some of the hotplug 
drivers 
> > > using
> > >     pci_configure_slot().  But not all hotplug drivers did this, and 
per 
> > > the
> > >     spec (ACPI rev 5.0, sec 6.2.7), we can also do it for "devices 
not
> > >     configured by the BIOS at system boot."
> > > 
> > >     Move this configuration into the PCI core by adding 
> > > pci_configure_device()
> > >     and calling it from pci_device_add(), so we do this for all 
devices as 
> > > we
> > >     enumerate them.
> > > 
> > >     This is based on pci_configure_slot(), which is used by hotplug 
> > > drivers.
> > >     I omitted:
> > > 
> > >       - pcie_bus_configure_settings() because it configures MPS and 
MRRS, 
> > > which
> > >         requires global knowledge of the fabric and must be done 
later, 
> > > and
> > > 
> > >       - configuration of subordinate devices; that will happen when 
we 
> > > call
> > >         pci_device_add() for those devices.
> > > 
> > >     Because pci_configure_slot() was only done by hotplug drivers, 
this 
> > > initial
> > >     version of pci_configure_device() only configures hot-added 
devices,
> > >     ignoring anything added during boot.
> > > 
> > >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> > >     Acked-by: Yinghai Lu <yinghai@kernel.org>
> > > 
> > > :040000 040000 4fadbe1e5f8f18daa6be7bdb7c9c1d6def0a2615 
> > > 9aef037aa35ca156ac46553f7fc4c5b1b3980c19 M      drivers
> > > 
> > > 
> > > I've reverted that commit on top of 3.19, which feels incredibly 
wrong, 
> > > but does fix the problem on my system.  This is a little over my 
head, 
> > > though...  ;-)
> > > 
> > > Thoughts?
> 
> Thanks for the report, Michael, and sorry for the inconvenience.  I 
think
> the patch below will fix it, but I don't think it's the right fix either
> because it seems a little ad hoc to sprinkle "acpi_pci_disabled" tests
> around like fairy dust.  I wonder if we can set things up so ACPI 
methods
> would fail gracefully like they do when ACPI is disabled at 
compile-time.
> 
> I can boot with "acpi=off" on qemu just fine, and when we look up the 
ACPI
> device handles, we just get NULL pointers, so everything works out even
> without a fix like the one below.

FYI, I'm not passing "acpi=off" on the guest's command line...  I believe 
it's getting turned off dynamically by the guest kernel.  I get the 
following in dmesg within the 1st dozen lines:

ACPI in unprivileged domain disabled

> 
> There must be something different about the way things get set up in 
that
> domU kernel.  I'll try to look into that some more, but I'm going on
> vacation for the next week, so if you learn anything before then, let me
> know.

I can confirm that your patch definitely fixes my problem.

If you need logs, kernel config, or someone to test patches, let me know.

> 
> Bjorn
> 
> 
> commit 6678b0fb6504c890481863b4916089b41e6042bf
> Author: Bjorn Helgaas <bhelgaas@google.com>
> Date:   Tue Mar 24 11:12:45 2015 -0500
> 
>     PCI: Don't look for ACPI hotplug parameters if ACPI is disabled
> 
>     In a kernel with CONFIG_ACPI=y, pci_get_hp_params() evaluates ACPI 
methods
>     (_HPX, _HPP, etc.) to learn how to configure devices.  If ACPI has 
been
>     disabled at runtime, e.g., with "acpi=off", this causes an oops 
because
>     there's no AML at all.
> 
>     Before 6cd33649fa83 ("PCI: Add pci_configure_device() during 
enumeration"),
>     we only used pci_get_hp_params() for hot-added devices, but after 
it, we
>     use it for all devices, so we're much more likely to see the oops.
> 
>     Don't bother looking for ACPI configuration information if ACPI has 
been
>     disabled.
> 
>     Fixes: 6cd33649fa83 ("PCI: Add pci_configure_device() during 
enumeration")
>     Reported-by: Michael D Labriola <mlabriol@gdeb.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     CC: stable@vger.kernel.org   # v3.18+
> 
> diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
> index 489063987325..c93fbe76d281 100644
> --- a/drivers/pci/pci-acpi.c
> +++ b/drivers/pci/pci-acpi.c
> @@ -248,6 +248,9 @@ int pci_get_hp_params(struct pci_dev *dev, struct 
> hotplug_params *hpp)
>     acpi_handle handle, phandle;
>     struct pci_bus *pbus;
> 
> +   if (acpi_pci_disabled)
> +      return -ENODEV;
> +
>     handle = NULL;
>     for (pbus = dev->bus; pbus; pbus = pbus->parent) {
>        handle = acpi_pci_get_bridge_handle(pbus);

---
Michael D Labriola
Electric Boat
mlabriol@gdeb.com
401-848-8871 (desk)
401-848-8513 (lab)
401-316-9844 (cell)



<<<DO_NOT_REMOVE_AUTOMATIC_FOOTER_GOES_HERE>>>



--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Bjorn Helgaas March 24, 2015, 11:31 p.m. UTC | #2
On Tue, Mar 24, 2015 at 5:27 PM, Michael D Labriola <mlabriol@gdeb.com> wrote:
> Bjorn Helgaas <bhelgaas@google.com> wrote on 03/24/2015 01:27:02 PM:

>> Thanks for the report, Michael, and sorry for the inconvenience.  I
> think
>> the patch below will fix it, but I don't think it's the right fix either
>> because it seems a little ad hoc to sprinkle "acpi_pci_disabled" tests
>> around like fairy dust.  I wonder if we can set things up so ACPI
> methods
>> would fail gracefully like they do when ACPI is disabled at
> compile-time.
>>
>> I can boot with "acpi=off" on qemu just fine, and when we look up the
> ACPI
>> device handles, we just get NULL pointers, so everything works out even
>> without a fix like the one below.
>
> FYI, I'm not passing "acpi=off" on the guest's command line...  I believe
> it's getting turned off dynamically by the guest kernel.  I get the
> following in dmesg within the 1st dozen lines:
>
> ACPI in unprivileged domain disabled

That's from xen_arch_setup(), which prints it when it calls
disable_acpi().  Booting with "acpi=off" also calls disable_acpi(), so
the effect should be similar.  But of course xen's PCI enumeration is
started a little differently and my guess is that difference is what
leads to this oops.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Konrad Rzeszutek Wilk March 25, 2015, 8:27 p.m. UTC | #3
On Tue, Mar 24, 2015 at 12:27:02PM -0500, Bjorn Helgaas wrote:
> [+cc Rafael, linux-pci, linux-acpi]
> 
> On Tue, Mar 24, 2015 at 11:28:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Tue, Mar 24, 2015 at 11:14:29AM -0400, Michael D Labriola wrote:
> > > I'm having problems booting a 3.18 or newer domU w/ PCI devices passed 
> > > through.  It only seems to be the domU kernel that's upset (i.e., Behavior 
> > > is identical whether I'm running 3.19 or 3.13 dom0).  I'm running a 32bit 
> > > dom0 (3.13.11) w/ 64bit 4.4.0 hypervisor and 32bit domU.  I get the 
> > > following Oops when trying to boot my domU with a couple PCI cards passed 
> > > through:
> > > 
> > > BUG: unable to handle kernel paging request at 0030303e
> > > IP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a
> > > *pdpt = 00000000019f1027 *pde = 0000000000000000 
> > > Oops: 0000 [#1] PREEMPT SMP 
> > > Modules linked in: xen_pcifront(+) pcspkr xen_blkfront loop
> > > CPU: 0 PID: 18 Comm: xenwatch Not tainted 3.17.0-test+ #6
> > > task: cb869950 ti: cb8ae000 task.ti: cb8ae000
> > > EIP: 0061:[<c06ed0e6>] EFLAGS: 00010246 CPU: 0
> > > EIP is at acpi_ns_validate_handle+0x12/0x1a
> > > EAX: 00000000 EBX: cb895dc0 ECX: 00000000 EDX: 0030303a
> > > ESI: c0a6bccd EDI: 00000000 EBP: 00000004 ESP: cb8afd00
> > >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
> > > CR0: 8005003b CR2: 0030303e CR3: 0a68e000 CR4: 00040660
> > > Stack:
> > >  c06eda4d 00000000 c096a21d 00000000 00000000 00006462 00000000 c0c102c0
> > >  0030303a 00040004 0030303a cb8afd94 cb8afdec cb8afd60 c06b78e1 cb8afd60
> > >  00000061 00000246 c0407bc7 c0c102c0 00000000 cb8afda0 cb8afda8 cb8afdb0
> > > Call Trace:
> > >  [<c06eda4d>] ? acpi_evaluate_object+0x31/0x1fc
> > 
> > We should not be calling in any acpi code in PV domU guests.
> > 
> > WE actually disable it (acpi=0) to make sure we don't call it - as
> > there is no ACPI AML data at all in the guest.
> > 
> > CC-ing Bjorn.
> > >  [<c096a21d>] ? resume_kernel+0x5/0x7
> > >  [<c06b78e1>] ? pci_get_hp_params+0x111/0x4e0
> > >  [<c0407bc7>] ? xen_force_evtchn_callback+0x17/0x30
> > >  [<c04085fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
> > >  [<c0699d34>] ? pci_device_add+0x24/0x450
> > >  [<c06975ce>] ? pci_bus_read_config_word+0x6e/0x80
> > >  [<c069a66d>] ? pci_scan_single_device+0x8d/0xb0
> > >  [<c069a6cc>] ? pci_scan_slot+0x3c/0xf0
> > >  [<c069b1bc>] ? pci_scan_child_bus+0x1c/0x90
> > >  [<c069b71a>] ? pci_scan_bus_parented+0x6a/0x90
> > >  [<d088d241>] ? pcifront_scan_root+0x91/0x130 [xen_pcifront]
> > >  [<d088e69f>] ? pcifront_backend_changed+0x4af/0x654 [xen_pcifront]
> > >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> > >  [<c070dd0f>] ? xenbus_gather+0x5f/0x90
> > >  [<c070c393>] ? xenbus_read_driver_state+0x33/0x50
> > >  [<c070f375>] ? xenbus_otherend_changed+0x95/0xa0
> > >  [<c0710a5f>] ? backend_changed+0xf/0x20
> > >  [<c070d712>] ? xenwatch_thread+0x72/0x110
> > >  [<c0486140>] ? bit_waitqueue+0x50/0x50
> > >  [<c070d6a0>] ? join+0x70/0x70
> > >  [<c046e59b>] ? kthread+0xab/0xd0
> > >  [<c096a1c1>] ? ret_from_kernel_thread+0x21/0x30
> > >  [<c046e4f0>] ? flush_kthread_worker+0xa0/0xa0
> > > Code: 03 10 00 00 eb 0e 46 83 c2 04 4b 85 db 75 b9 c6 02 00 31 c0 5b 5e 5f 
> > > 5d c3 89 c2 8d 40 ff 83 f8 fd 76 06 a1 2c 32 c1 c0 c3 31 c0 <80> 7a 04 0f 
> > > 0f 44 c2 c3 83 ec 10 83 f8 1d 76 24 89 44 24 0c c7
> > > EIP: [<c06ed0e6>] acpi_ns_validate_handle+0x12/0x1a SS:ESP 0069:cb8afd00
> > > CR2: 000000000030303e
> > > ---[ end trace d4ddeb038cbcbdf7 ]---
> > > 
> > > 
> > > I've bisected down to the following commit in 3.18, which breaks my 
> > > system.
> > > 
> > > 6cd33649fa83d97ba7b66f1d871a360e867c5220 is the first bad commit
> > > commit 6cd33649fa83d97ba7b66f1d871a360e867c5220
> > > Author: Bjorn Helgaas <bhelgaas@google.com>
> > > Date:   Wed Aug 27 14:29:47 2014 -0600
> > > 
> > >     PCI: Add pci_configure_device() during enumeration
> > >  
> > >     Some platforms can tell the OS how to configure PCI devices, e.g., how 
> > > to
> > >     set cache line size, error reporting enables, etc.  ACPI defines _HPP 
> > > and
> > >     _HPX methods for this purpose.
> > >  
> > >     This configuration was previously done by some of the hotplug drivers 
> > > using
> > >     pci_configure_slot().  But not all hotplug drivers did this, and per 
> > > the
> > >     spec (ACPI rev 5.0, sec 6.2.7), we can also do it for "devices not
> > >     configured by the BIOS at system boot."
> > >  
> > >     Move this configuration into the PCI core by adding 
> > > pci_configure_device()
> > >     and calling it from pci_device_add(), so we do this for all devices as 
> > > we
> > >     enumerate them.
> > >  
> > >     This is based on pci_configure_slot(), which is used by hotplug 
> > > drivers.
> > >     I omitted:
> > >  
> > >       - pcie_bus_configure_settings() because it configures MPS and MRRS, 
> > > which
> > >         requires global knowledge of the fabric and must be done later, 
> > > and
> > >  
> > >       - configuration of subordinate devices; that will happen when we 
> > > call
> > >         pci_device_add() for those devices.
> > >  
> > >     Because pci_configure_slot() was only done by hotplug drivers, this 
> > > initial
> > >     version of pci_configure_device() only configures hot-added devices,
> > >     ignoring anything added during boot.
> > >  
> > >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> > >     Acked-by: Yinghai Lu <yinghai@kernel.org>
> > > 
> > > :040000 040000 4fadbe1e5f8f18daa6be7bdb7c9c1d6def0a2615 
> > > 9aef037aa35ca156ac46553f7fc4c5b1b3980c19 M      drivers
> > > 
> > > 
> > > I've reverted that commit on top of 3.19, which feels incredibly wrong, 
> > > but does fix the problem on my system.  This is a little over my head, 
> > > though...  ;-)
> > > 
> > > Thoughts?
> 
> Thanks for the report, Michael, and sorry for the inconvenience.  I think
> the patch below will fix it, but I don't think it's the right fix either
> because it seems a little ad hoc to sprinkle "acpi_pci_disabled" tests
> around like fairy dust.  I wonder if we can set things up so ACPI methods
> would fail gracefully like they do when ACPI is disabled at compile-time.
> 
> I can boot with "acpi=off" on qemu just fine, and when we look up the ACPI
> device handles, we just get NULL pointers, so everything works out even
> without a fix like the one below.

<nods>
> 
> There must be something different about the way things get set up in that
> domU kernel.  I'll try to look into that some more, but I'm going on
> vacation for the next week, so if you learn anything before then, let me
> know.

And I don't see where 'resume_kernel' gets called.


And under my PV guest I see:
# lspci
00:00.0 USB Controller: NEC Corporation Device 0194 (rev 04)
00:01.0 USB Controller: NEC Corporation Device 0194 (rev 03)

when the guest boots up. Hm, what kind of cards are these?
Could you also provide the config file and the guest config please?

Thank you!

> 
> Bjorn
> 
> 
> commit 6678b0fb6504c890481863b4916089b41e6042bf
> Author: Bjorn Helgaas <bhelgaas@google.com>
> Date:   Tue Mar 24 11:12:45 2015 -0500
> 
>     PCI: Don't look for ACPI hotplug parameters if ACPI is disabled
>     
>     In a kernel with CONFIG_ACPI=y, pci_get_hp_params() evaluates ACPI methods
>     (_HPX, _HPP, etc.) to learn how to configure devices.  If ACPI has been
>     disabled at runtime, e.g., with "acpi=off", this causes an oops because
>     there's no AML at all.
>     
>     Before 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration"),
>     we only used pci_get_hp_params() for hot-added devices, but after it, we
>     use it for all devices, so we're much more likely to see the oops.
>     
>     Don't bother looking for ACPI configuration information if ACPI has been
>     disabled.
>     
>     Fixes: 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration")
>     Reported-by: Michael D Labriola <mlabriol@gdeb.com>
>     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>     CC: stable@vger.kernel.org	# v3.18+
> 
> diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
> index 489063987325..c93fbe76d281 100644
> --- a/drivers/pci/pci-acpi.c
> +++ b/drivers/pci/pci-acpi.c
> @@ -248,6 +248,9 @@ int pci_get_hp_params(struct pci_dev *dev, struct hotplug_params *hpp)
>  	acpi_handle handle, phandle;
>  	struct pci_bus *pbus;
>  
> +	if (acpi_pci_disabled)
> +		return -ENODEV;
> +
>  	handle = NULL;
>  	for (pbus = dev->bus; pbus; pbus = pbus->parent) {
>  		handle = acpi_pci_get_bridge_handle(pbus);
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Konrad Rzeszutek Wilk March 25, 2015, 8:40 p.m. UTC | #4
> > Thanks for the report, Michael, and sorry for the inconvenience.  I think
> > the patch below will fix it, but I don't think it's the right fix either
> > because it seems a little ad hoc to sprinkle "acpi_pci_disabled" tests
> > around like fairy dust.  I wonder if we can set things up so ACPI methods
> > would fail gracefully like they do when ACPI is disabled at compile-time.
> > 
> > I can boot with "acpi=off" on qemu just fine, and when we look up the ACPI
> > device handles, we just get NULL pointers, so everything works out even
> > without a fix like the one below.
> 
> <nods>
> > 
> > There must be something different about the way things get set up in that
> > domU kernel.  I'll try to look into that some more, but I'm going on
> > vacation for the next week, so if you learn anything before then, let me
> > know.
> 
> And I don't see where 'resume_kernel' gets called.
> 
> 
> And under my PV guest I see:
> # lspci
> 00:00.0 USB Controller: NEC Corporation Device 0194 (rev 04)
> 00:01.0 USB Controller: NEC Corporation Device 0194 (rev 03)
> 
> when the guest boots up. Hm, what kind of cards are these?
> Could you also provide the config file and the guest config please?


It helps when I boot an proper kernel (I had booted 3.16 by mistake).

With the 4.0 kernel I see this as well:

[    4.059387] pci 0000:00:00.0: reg 0x10: [mem 0xfbd00000-0xfbd01fff 64bit]
[    4.114184] BUG: unable to handle kernel paging request at 0030303e
[    4.114199] IP: [<c1379c1c>] acpi_ns_validate_handle+0x1c/0x26
[    4.114216] *pdpt = 0000000000000000 *pde = c2c2c2c2c2c2c2c2 
[    4.114230] Oops: 0000 [#1] SMP 
[    4.114241] Modules linked in:
[    4.114252] CPU: 0 PID: 22 Comm: xenwatch Not tainted 4.0.0-rc5upstream-00070-g3a88f16 #1
[    4.114268] task: dd557370 ti: dd55a000 task.ti: dd55a000 
[    4.114278] EIP: e019:[<c1379c1c>] EFLAGS: 00010246 CPU: 0
[    4.114289] EIP is at acpi_ns_validate_handle+0x1c/0x26
[    4.114299] EAX: 00000000 EBX: 0030303a ECX: 00000000 EDX: 00000000
[    4.114310] ESI: dd66e7c0 EDI: 0030303a EBP: dd55bcd8 ESP: dd55bcd4
[    4.114322]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
[    4.114334] CR0: 80050033 CR2: 0030303e CR3: 019a4000 CR4: 00042660
[    4.114347] Stack:
[    4.114354]  c17ae853 dd55bd14 c137ae18 00000010 c1a602c0 dd55bcf0 c109ae9a dd55bcfc
[    4.114376]  c13a9c57 deadbeef dd55bd50 00000000 deadbeef 0030303a dd55bd78 dd55bdd0
[    4.114397]  dd55bd58 c1336d90 dd55bd40 0000007b 0000007b 000000d8 dd55bd94 dd55bd8c
[    4.114419] Call Trace:
[    4.114429]  [<c137ae18>] acpi_evaluate_object+0x6f/0x366
[    4.114443]  [<c109ae9a>] ? irq_exit+0x4a/0xc0
[    4.114456]  [<c13a9c57>] ? xen_evtchn_do_upcall+0x27/0x40
[    4.114468]  [<c1336d90>] pci_get_hp_params+0x110/0x4b0
[    4.114480]  [<c1316436>] pci_device_add+0x26/0x450
[    4.114494]  [<c10409db>] ? xen_restore_fl_direct_reloc+0x4/0x4  
[    4.115132]  [<c16509c6>] ? _raw_spin_unlock_irqrestore+0x16/0x40
[    4.115132]  [<c13171eb>] pci_scan_single_device+0x8b/0xb0
[    4.115132]  [<c1317253>] pci_scan_slot+0x43/0x100
[    4.115132]  [<c1317cfc>] pci_scan_child_bus+0x1c/0xa0   
[    4.115132]  [<c1318254>] pci_scan_bus_parented+0x64/0x90
[    4.115132]  [<c1337730>] pcifront_scan_root+0x90/0x120
[    4.115132]  [<c164b8e5>] pcifront_backend_changed+0x475/0x63c
[    4.115132]  [<c164b200>] ? kmemleak_free+0x20/0x50
[    4.115132]  [<c119ec9d>] ? kfree+0x7d/0x100
[    4.115132]  [<c13a028d>] ? __cleanup+0xfd/0x180   
[    4.115132]  [<c13ae28d>] ? xenbus_gather+0x5d/0x90
[    4.115132]  [<c13ac475>] ? xenbus_read_driver_state+0x35/0x50
[    4.115132]  [<c13af3cd>] xenbus_otherend_changed+0x7d/0x80
[    4.115132]  [<c13b0bf2>] backend_changed+0x12/0x20 
[    4.115132]  [<c13adb12>] xenwatch_thread+0x72/0x120
[    4.115132]  [<c10ccc00>] ? woken_wake_function+0x20/0x20
[    4.115132]  [<c10b10bc>] kthread+0xac/0xd0
[    4.115132]  [<c13adaa0>] ? xenbus_transaction_start+0x60/0x60
[    4.115132]  [<c1650fc1>] ret_from_kernel_thread+0x21/0x30
[    4.115132]  [<c10b1010>] ? kthread_freezable_should_stop+0x60/0x60
[    4.115132] Code: 4f b2 00 00 31 c0 83 c4 2c 5b 5e 5f 5d c3 90 55 89 e5 53 89 c3 e8 45 b0 00 00 8d 43 ff 83 f8 fd 76 07 a1 20 80 a6 c1 eb 09 31 c0 <80> 7b 04 0f 0f 44 c3 5b 5d c3 55 89 e5 53 89 c3 e8 1f b0 00 00
[    4.115132] EIP: [<c1379c1c>] acpi_ns_validate_handle+0x1c/0x26 SS:ESP e021:dd55bcd4
[    4.115132] CR2: 000000000030303e
[    4.115132] ---[ end trace 21d8bfe52b77b825 ]---
[    4.115132] Kernel panic - not syncing: Fatal exception
[    4.115132] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xedbfdfff)

The interesting thing is that under 64-bit kernels I see this:

[    3.304732] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]^M
[    3.304748] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]^M
[    3.304764] pci_bus 0000:00: root bus resource [bus 00-ff]^M
[    3.305023] pci 0000:00:00.0: [1033:0194] type 00 class 0x0c0330^M
[    3.322181] pci 0000:00:00.0: reg 0x10: [mem 0xfbd00000-0xfbd01fff 64bit]^M
[    3.377359] ACPI Exception: AE_BAD_PARAMETER, Thread 520623344 could not acquire Mutex [0x1] (20150204/utmutex-285)^M
[    3.377379] ACPI Exception: AE_BAD_PARAMETER, Thread 520623344 could not acquire Mutex [0x1] (20150204/utmutex-285)^M
[    3.379202] pci 0000:00:01.0: [1033:0194] type 00 class 0x0c0330^M
[    3.396282] pci 0000:00:01.0: reg 0x10: [mem 0xfba00000-0xfba01fff 64bit]^M
[    3.451422] ACPI Exception: AE_BAD_PARAMETER, Thread 520623344 could not acquire Mutex [0x1] (20150204/utmutex-285)^M
[    3.451440] ACPI Exception: AE_BAD_PARAMETER, Thread 520623344 could not acquire Mutex [0x1] (20150204/utmutex-285)^M
[    3.456133] pcifront pci-0: claiming resource 0000:00:00.0/0^M
[    3.456147] pcifront pci-0: claiming resource 0000:00:01.0/0^M
[    3.456461] pci 0000:00:00.0: Xen PCI mapped GSI18 to IRQ25^M


> 
> Thank you!
> 
> > 
> > Bjorn
> > 
> > 
> > commit 6678b0fb6504c890481863b4916089b41e6042bf
> > Author: Bjorn Helgaas <bhelgaas@google.com>
> > Date:   Tue Mar 24 11:12:45 2015 -0500
> > 
> >     PCI: Don't look for ACPI hotplug parameters if ACPI is disabled
> >     
> >     In a kernel with CONFIG_ACPI=y, pci_get_hp_params() evaluates ACPI methods
> >     (_HPX, _HPP, etc.) to learn how to configure devices.  If ACPI has been
> >     disabled at runtime, e.g., with "acpi=off", this causes an oops because
> >     there's no AML at all.
> >     
> >     Before 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration"),
> >     we only used pci_get_hp_params() for hot-added devices, but after it, we
> >     use it for all devices, so we're much more likely to see the oops.
> >     
> >     Don't bother looking for ACPI configuration information if ACPI has been
> >     disabled.
> >     
> >     Fixes: 6cd33649fa83 ("PCI: Add pci_configure_device() during enumeration")
> >     Reported-by: Michael D Labriola <mlabriol@gdeb.com>
> >     Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> >     CC: stable@vger.kernel.org	# v3.18+
> > 
> > diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
> > index 489063987325..c93fbe76d281 100644
> > --- a/drivers/pci/pci-acpi.c
> > +++ b/drivers/pci/pci-acpi.c
> > @@ -248,6 +248,9 @@ int pci_get_hp_params(struct pci_dev *dev, struct hotplug_params *hpp)
> >  	acpi_handle handle, phandle;
> >  	struct pci_bus *pbus;
> >  
> > +	if (acpi_pci_disabled)
> > +		return -ENODEV;
> > +
> >  	handle = NULL;
> >  	for (pbus = dev->bus; pbus; pbus = pbus->parent) {
> >  		handle = acpi_pci_get_bridge_handle(pbus);
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
index 489063987325..c93fbe76d281 100644
--- a/drivers/pci/pci-acpi.c
+++ b/drivers/pci/pci-acpi.c
@@ -248,6 +248,9 @@  int pci_get_hp_params(struct pci_dev *dev, struct hotplug_params *hpp)
 	acpi_handle handle, phandle;
 	struct pci_bus *pbus;
 
+	if (acpi_pci_disabled)
+		return -ENODEV;
+
 	handle = NULL;
 	for (pbus = dev->bus; pbus; pbus = pbus->parent) {
 		handle = acpi_pci_get_bridge_handle(pbus);