| Message ID | 20220815065550.1303620-1-mpe@ellerman.id.au (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | powerpc/pci: Fix get_phb_number() locking | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/github-powerpc_ppctests | success | Successfully ran 10 jobs. |
| snowpatch_ozlabs/github-powerpc_selftests | success | Successfully ran 10 jobs. |
| snowpatch_ozlabs/github-powerpc_sparse | success | Successfully ran 4 jobs. |
| snowpatch_ozlabs/github-powerpc_clang | success | Successfully ran 6 jobs. |
| snowpatch_ozlabs/github-powerpc_kernel_qemu | success | Successfully ran 23 jobs. |
On Mon, Aug 15, 2022 at 04:55:50PM +1000, Michael Ellerman wrote: > The recent change to get_phb_number() causes a DEBUG_ATOMIC_SLEEP > warning on some systems: > > BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580 > in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper > preempt_count: 1, expected: 0 > RCU nest depth: 0, expected: 0 > 1 lock held by swapper/1: > #0: c157efb0 (hose_spinlock){+.+.}-{2:2}, at: pcibios_alloc_controller+0x64/0x220 > Preemption disabled at: > [<00000000>] 0x0 > CPU: 0 PID: 1 Comm: swapper Not tainted 5.19.0-yocto-standard+ #1 > Call Trace: > [d101dc90] [c073b264] dump_stack_lvl+0x50/0x8c (unreliable) > [d101dcb0] [c0093b70] __might_resched+0x258/0x2a8 > [d101dcd0] [c0d3e634] __mutex_lock+0x6c/0x6ec > [d101dd50] [c0a84174] of_alias_get_id+0x50/0xf4 > [d101dd80] [c002ec78] pcibios_alloc_controller+0x1b8/0x220 > [d101ddd0] [c140c9dc] pmac_pci_init+0x198/0x784 > [d101de50] [c140852c] discover_phbs+0x30/0x4c > [d101de60] [c0007fd4] do_one_initcall+0x94/0x344 > [d101ded0] [c1403b40] kernel_init_freeable+0x1a8/0x22c > [d101df10] [c00086e0] kernel_init+0x34/0x160 > [d101df30] [c001b334] ret_from_kernel_thread+0x5c/0x64 > > This is because pcibios_alloc_controller() holds hose_spinlock but > of_alias_get_id() takes of_mutex which can sleep. > > The hose_spinlock protects the phb_bitmap, and also the hose_list, but > it doesn't need to be held while get_phb_number() calls the OF routines, > because those are only looking up information in the device tree. > > So fix it by having get_phb_number() take the hose_spinlock itself, only > where required, and then dropping the lock before returning. > pcibios_alloc_controller() then needs to take the lock again before the > list_add() but that's safe, the order of the list is not important. > > Fixes: 0fe1e96fef0a ("powerpc/pci: Prefer PCI domain assignment via DT 'linux,pci-domain' and alias") > Reported-by: Guenter Roeck <linux@roeck-us.net> > Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> The problem is no longer seen with this patch applied. Tested-by: Guenter Roeck <linux@roeck-us.net> Guenter
On Monday 15 August 2022 16:55:50 Michael Ellerman wrote: > The recent change to get_phb_number() causes a DEBUG_ATOMIC_SLEEP > warning on some systems: > > BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580 > in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper > preempt_count: 1, expected: 0 > RCU nest depth: 0, expected: 0 > 1 lock held by swapper/1: > #0: c157efb0 (hose_spinlock){+.+.}-{2:2}, at: pcibios_alloc_controller+0x64/0x220 > Preemption disabled at: > [<00000000>] 0x0 > CPU: 0 PID: 1 Comm: swapper Not tainted 5.19.0-yocto-standard+ #1 > Call Trace: > [d101dc90] [c073b264] dump_stack_lvl+0x50/0x8c (unreliable) > [d101dcb0] [c0093b70] __might_resched+0x258/0x2a8 > [d101dcd0] [c0d3e634] __mutex_lock+0x6c/0x6ec > [d101dd50] [c0a84174] of_alias_get_id+0x50/0xf4 > [d101dd80] [c002ec78] pcibios_alloc_controller+0x1b8/0x220 > [d101ddd0] [c140c9dc] pmac_pci_init+0x198/0x784 > [d101de50] [c140852c] discover_phbs+0x30/0x4c > [d101de60] [c0007fd4] do_one_initcall+0x94/0x344 > [d101ded0] [c1403b40] kernel_init_freeable+0x1a8/0x22c > [d101df10] [c00086e0] kernel_init+0x34/0x160 > [d101df30] [c001b334] ret_from_kernel_thread+0x5c/0x64 > > This is because pcibios_alloc_controller() holds hose_spinlock but > of_alias_get_id() takes of_mutex which can sleep. > > The hose_spinlock protects the phb_bitmap, and also the hose_list, but > it doesn't need to be held while get_phb_number() calls the OF routines, > because those are only looking up information in the device tree. > > So fix it by having get_phb_number() take the hose_spinlock itself, only > where required, and then dropping the lock before returning. > pcibios_alloc_controller() then needs to take the lock again before the > list_add() but that's safe, the order of the list is not important. > > Fixes: 0fe1e96fef0a ("powerpc/pci: Prefer PCI domain assignment via DT 'linux,pci-domain' and alias") > Reported-by: Guenter Roeck <linux@roeck-us.net> > Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Thanks for fixing it! Acked-by: Pali Rohár <pali@kernel.org> > --- > arch/powerpc/kernel/pci-common.c | 16 ++++++++++------ > 1 file changed, 10 insertions(+), 6 deletions(-) > > diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c > index bdd3332200c5..31de91c8359c 100644 > --- a/arch/powerpc/kernel/pci-common.c > +++ b/arch/powerpc/kernel/pci-common.c > @@ -68,10 +68,6 @@ void __init set_pci_dma_ops(const struct dma_map_ops *dma_ops) > pci_dma_ops = dma_ops; > } > > -/* > - * This function should run under locking protection, specifically > - * hose_spinlock. > - */ > static int get_phb_number(struct device_node *dn) > { > int ret, phb_id = -1; > @@ -108,15 +104,20 @@ static int get_phb_number(struct device_node *dn) > if (!ret) > phb_id = (int)(prop & (MAX_PHBS - 1)); > > + spin_lock(&hose_spinlock); > + > /* We need to be sure to not use the same PHB number twice. */ > if ((phb_id >= 0) && !test_and_set_bit(phb_id, phb_bitmap)) > - return phb_id; > + goto out_unlock; > > /* If everything fails then fallback to dynamic PHB numbering. */ > phb_id = find_first_zero_bit(phb_bitmap, MAX_PHBS); > BUG_ON(phb_id >= MAX_PHBS); > set_bit(phb_id, phb_bitmap); > > +out_unlock: > + spin_unlock(&hose_spinlock); > + > return phb_id; > } > > @@ -127,10 +128,13 @@ struct pci_controller *pcibios_alloc_controller(struct device_node *dev) > phb = zalloc_maybe_bootmem(sizeof(struct pci_controller), GFP_KERNEL); > if (phb == NULL) > return NULL; > - spin_lock(&hose_spinlock); > + > phb->global_number = get_phb_number(dev); > + > + spin_lock(&hose_spinlock); > list_add_tail(&phb->list_node, &hose_list); > spin_unlock(&hose_spinlock); > + > phb->dn = dev; > phb->is_dynamic = slab_is_available(); > #ifdef CONFIG_PPC64 > -- > 2.37.1 >
On Mon, 15 Aug 2022 16:55:50 +1000, Michael Ellerman wrote: > The recent change to get_phb_number() causes a DEBUG_ATOMIC_SLEEP > warning on some systems: > > BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580 > in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper > preempt_count: 1, expected: 0 > RCU nest depth: 0, expected: 0 > 1 lock held by swapper/1: > #0: c157efb0 (hose_spinlock){+.+.}-{2:2}, at: pcibios_alloc_controller+0x64/0x220 > Preemption disabled at: > [<00000000>] 0x0 > CPU: 0 PID: 1 Comm: swapper Not tainted 5.19.0-yocto-standard+ #1 > Call Trace: > [d101dc90] [c073b264] dump_stack_lvl+0x50/0x8c (unreliable) > [d101dcb0] [c0093b70] __might_resched+0x258/0x2a8 > [d101dcd0] [c0d3e634] __mutex_lock+0x6c/0x6ec > [d101dd50] [c0a84174] of_alias_get_id+0x50/0xf4 > [d101dd80] [c002ec78] pcibios_alloc_controller+0x1b8/0x220 > [d101ddd0] [c140c9dc] pmac_pci_init+0x198/0x784 > [d101de50] [c140852c] discover_phbs+0x30/0x4c > [d101de60] [c0007fd4] do_one_initcall+0x94/0x344 > [d101ded0] [c1403b40] kernel_init_freeable+0x1a8/0x22c > [d101df10] [c00086e0] kernel_init+0x34/0x160 > [d101df30] [c001b334] ret_from_kernel_thread+0x5c/0x64 > > [...] Applied to powerpc/fixes. [1/1] powerpc/pci: Fix get_phb_number() locking https://git.kernel.org/powerpc/c/8d48562a2729742f767b0fdd994d6b2a56a49c63 cheers
diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c index bdd3332200c5..31de91c8359c 100644 --- a/arch/powerpc/kernel/pci-common.c +++ b/arch/powerpc/kernel/pci-common.c @@ -68,10 +68,6 @@ void __init set_pci_dma_ops(const struct dma_map_ops *dma_ops) pci_dma_ops = dma_ops; } -/* - * This function should run under locking protection, specifically - * hose_spinlock. - */ static int get_phb_number(struct device_node *dn) { int ret, phb_id = -1; @@ -108,15 +104,20 @@ static int get_phb_number(struct device_node *dn) if (!ret) phb_id = (int)(prop & (MAX_PHBS - 1)); + spin_lock(&hose_spinlock); + /* We need to be sure to not use the same PHB number twice. */ if ((phb_id >= 0) && !test_and_set_bit(phb_id, phb_bitmap)) - return phb_id; + goto out_unlock; /* If everything fails then fallback to dynamic PHB numbering. */ phb_id = find_first_zero_bit(phb_bitmap, MAX_PHBS); BUG_ON(phb_id >= MAX_PHBS); set_bit(phb_id, phb_bitmap); +out_unlock: + spin_unlock(&hose_spinlock); + return phb_id; } @@ -127,10 +128,13 @@ struct pci_controller *pcibios_alloc_controller(struct device_node *dev) phb = zalloc_maybe_bootmem(sizeof(struct pci_controller), GFP_KERNEL); if (phb == NULL) return NULL; - spin_lock(&hose_spinlock); + phb->global_number = get_phb_number(dev); + + spin_lock(&hose_spinlock); list_add_tail(&phb->list_node, &hose_list); spin_unlock(&hose_spinlock); + phb->dn = dev; phb->is_dynamic = slab_is_available(); #ifdef CONFIG_PPC64
The recent change to get_phb_number() causes a DEBUG_ATOMIC_SLEEP warning on some systems: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper preempt_count: 1, expected: 0 RCU nest depth: 0, expected: 0 1 lock held by swapper/1: #0: c157efb0 (hose_spinlock){+.+.}-{2:2}, at: pcibios_alloc_controller+0x64/0x220 Preemption disabled at: [<00000000>] 0x0 CPU: 0 PID: 1 Comm: swapper Not tainted 5.19.0-yocto-standard+ #1 Call Trace: [d101dc90] [c073b264] dump_stack_lvl+0x50/0x8c (unreliable) [d101dcb0] [c0093b70] __might_resched+0x258/0x2a8 [d101dcd0] [c0d3e634] __mutex_lock+0x6c/0x6ec [d101dd50] [c0a84174] of_alias_get_id+0x50/0xf4 [d101dd80] [c002ec78] pcibios_alloc_controller+0x1b8/0x220 [d101ddd0] [c140c9dc] pmac_pci_init+0x198/0x784 [d101de50] [c140852c] discover_phbs+0x30/0x4c [d101de60] [c0007fd4] do_one_initcall+0x94/0x344 [d101ded0] [c1403b40] kernel_init_freeable+0x1a8/0x22c [d101df10] [c00086e0] kernel_init+0x34/0x160 [d101df30] [c001b334] ret_from_kernel_thread+0x5c/0x64 This is because pcibios_alloc_controller() holds hose_spinlock but of_alias_get_id() takes of_mutex which can sleep. The hose_spinlock protects the phb_bitmap, and also the hose_list, but it doesn't need to be held while get_phb_number() calls the OF routines, because those are only looking up information in the device tree. So fix it by having get_phb_number() take the hose_spinlock itself, only where required, and then dropping the lock before returning. pcibios_alloc_controller() then needs to take the lock again before the list_add() but that's safe, the order of the list is not important. Fixes: 0fe1e96fef0a ("powerpc/pci: Prefer PCI domain assignment via DT 'linux,pci-domain' and alias") Reported-by: Guenter Roeck <linux@roeck-us.net> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> --- arch/powerpc/kernel/pci-common.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)