| Message ID | 20200617162938.743439-1-clg@kaod.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | powerpc/pci: unmap interrupts when a PHB is removed | expand |
On 6/17/20 6:29 PM, Cédric Le Goater wrote: > Hello, > > When a passthrough IO adapter is removed from a pseries machine using > hash MMU and the XIVE interrupt mode, the POWER hypervisor expects the > guest OS to clear all page table entries related to the adapter. If > some are still present, the RTAS call which isolates the PCI slot > returns error 9001 "valid outstanding translations" and the removal of > the IO adapter fails. This is because when the PHBs are scanned, Linux > maps automatically some interrupts in the Linux interrupt number space > but these are never removed. > > To solve this problem, we introduce a PPC platform specific > pcibios_remove_bus() routine which clears all interrupt mappings when > the bus is removed. This also clears the associated page table entries > of the ESB pages when using XIVE. > > For this purpose, we record the logical interrupt numbers of the > mapped interrupt under the PHB structure and let pcibios_remove_bus() > do the clean up. > > Tested on : > > - PowerNV with PCI, OpenCAPI, CAPI and GPU adapters. I don't know > how to inject a failure on a PHB but that would be a good test. I found out that powering down the slot is enough : echo 0 > /sys/bus/pci/slots/<slot name>/power The IRQ cleanup is done as expected on baremetal also. Cheers, C. > - KVM P8+P9 guests with passthrough PCI adapters, but PHBs can not > be removed under QEMU/KVM. > - PowerVM with passthrough PCI adapters (main target) > > Thanks, > > C. > > Changes since v1: > > - extended the removal to interrupts other than the legacy INTx. > > Cédric Le Goater (2): > powerpc/pci: unmap legacy INTx interrupts when a PHB is removed > powerpc/pci: unmap all interrupts when a PHB is removed > > arch/powerpc/include/asm/pci-bridge.h | 6 ++ > arch/powerpc/kernel/pci-common.c | 114 ++++++++++++++++++++++++++ > 2 files changed, 120 insertions(+) >
Hello, When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor expects the guest OS to clear all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails. This is because when the PHBs are scanned, Linux maps automatically some interrupts in the Linux interrupt number space but these are never removed. To solve this problem, we introduce a PPC platform specific pcibios_remove_bus() routine which clears all interrupt mappings when the bus is removed. This also clears the associated page table entries of the ESB pages when using XIVE. For this purpose, we record the logical interrupt numbers of the mapped interrupt under the PHB structure and let pcibios_remove_bus() do the clean up. Tested on : - PowerNV with PCI, OpenCAPI, CAPI and GPU adapters. I don't know how to inject a failure on a PHB but that would be a good test. - KVM P8+P9 guests with passthrough PCI adapters, but PHBs can not be removed under QEMU/KVM. - PowerVM with passthrough PCI adapters (main target) Thanks, C. Changes since v1: - extended the removal to interrupts other than the legacy INTx. Cédric Le Goater (2): powerpc/pci: unmap legacy INTx interrupts when a PHB is removed powerpc/pci: unmap all interrupts when a PHB is removed arch/powerpc/include/asm/pci-bridge.h | 6 ++ arch/powerpc/kernel/pci-common.c | 114 ++++++++++++++++++++++++++ 2 files changed, 120 insertions(+)