| Message ID | 20180611222918.1708-1-keith.busch@intel.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Bjorn Helgaas |
| Headers | show |
| Series | PCI/AER: Prevent runtime power management during recovery | expand |
On 6/11/2018 6:29 PM, Keith Busch wrote: > A bridge that supports D3 but not hotplug will be subject to runtime > power management placing it in a non-operation power state if it doesn't > have any devices attached. This patch will prevent this power management > during error recovery so that the rescan at the end may be successful. If there is no card connected, why would the bridge observe a fatal error?
On 2018-06-12 10:10, Sinan Kaya wrote: > On 6/11/2018 6:29 PM, Keith Busch wrote: >> A bridge that supports D3 but not hotplug will be subject to runtime >> power management placing it in a non-operation power state if it >> doesn't >> have any devices attached. This patch will prevent this power >> management >> during error recovery so that the rescan at the end may be successful. > > If there is no card connected, why would the bridge observe a fatal > error? fatal error could be coming from anywhere let us say below RootPort.. and RP observes it and decides to take the tree down.... Is that the case Keith is talking about ? Why will re-enumeration be a problem even if runtime PM is active ? I assume that enumeration will get bridge out of D3.
On Tue, Jun 12, 2018 at 01:21:53PM +0530, poza@codeaurora.org wrote: > On 2018-06-12 10:10, Sinan Kaya wrote: > > On 6/11/2018 6:29 PM, Keith Busch wrote: > > > A bridge that supports D3 but not hotplug will be subject to runtime > > > power management placing it in a non-operation power state if it > > > doesn't > > > have any devices attached. This patch will prevent this power > > > management > > > during error recovery so that the rescan at the end may be successful. > > > > If there is no card connected, why would the bridge observe a fatal > > error? > > fatal error could be coming from anywhere let us say below RootPort.. > and RP observes it and decides to take the tree down.... > Is that the case Keith is talking about ? Right, the err fatal handling removes all the devices below a bridge, making that bridge allowed for run time d3. > Why will re-enumeration be a problem even if runtime PM is active ? > I assume that enumeration will get bridge out of D3. That doesn't seem to be the case.
On 2018-06-12 20:14, Keith Busch wrote: > On Tue, Jun 12, 2018 at 01:21:53PM +0530, poza@codeaurora.org wrote: >> On 2018-06-12 10:10, Sinan Kaya wrote: >> > On 6/11/2018 6:29 PM, Keith Busch wrote: >> > > A bridge that supports D3 but not hotplug will be subject to runtime >> > > power management placing it in a non-operation power state if it >> > > doesn't >> > > have any devices attached. This patch will prevent this power >> > > management >> > > during error recovery so that the rescan at the end may be successful. >> > >> > If there is no card connected, why would the bridge observe a fatal >> > error? >> >> fatal error could be coming from anywhere let us say below RootPort.. >> and RP observes it and decides to take the tree down.... >> Is that the case Keith is talking about ? > > Right, the err fatal handling removes all the devices below a bridge, > making that bridge allowed for run time d3. > >> Why will re-enumeration be a problem even if runtime PM is active ? >> I assume that enumeration will get bridge out of D3. > > That doesn't seem to be the case. reset_link(udev, service); will initiate SBR, and which should do hot reset the bridge and bring it out of D3 ! but it seems even SBR is not doing it for you. Although I am not sure if the EP has to be designed to reset its config space upon SBR.. even if they are, some just might not do it all correctly ! The code looks okay to me anyway. Reviewed-by: Oza Pawandeep <poza@codeaurora.org>
[+cc Rafael, linux-pm for a PM perspective] On Mon, Jun 11, 2018 at 04:29:18PM -0600, Keith Busch wrote: > A bridge that supports D3 but not hotplug will be subject to runtime > power management placing it in a non-operation power state if it doesn't > have any devices attached. This patch will prevent this power management > during error recovery so that the rescan at the end may be successful. I assume you're seeing a problem that's fixed by this patch? I'm not very familiar with the pm_runtime_*() interfaces. The fact that drivers/pci/ only contains two calls to pm_runtime_forbid() and two to pm_runtime_allow() (one in a quirk) makes me wonder if this is the right approach. Do we need something similar in other enumeration paths, e.g., boot-time enumeration, hot-add, sysfs "rescan" entry, etc? > Cc: Oza Pawandeep <poza@codeaurora.org> > Signed-off-by: Keith Busch <keith.busch@intel.com> > --- > drivers/pci/pcie/err.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c > index f7ce0cb0b0b7..247b6ce14f0d 100644 > --- a/drivers/pci/pcie/err.c > +++ b/drivers/pci/pcie/err.c > @@ -16,6 +16,7 @@ > #include <linux/kernel.h> > #include <linux/errno.h> > #include <linux/aer.h> > +#include <linux/pm_runtime.h> > #include "portdrv.h" > #include "../pci.h" > > @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) > udev = dev->bus->self; > > parent = udev->subordinate; > + pm_runtime_forbid(&udev->dev); > pci_lock_rescan_remove(); > list_for_each_entry_safe_reverse(pdev, temp, &parent->devices, > bus_list) { > @@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) > } > > pci_unlock_rescan_remove(); > + pm_runtime_allow(&udev->dev); > } > > /** > -- > 2.14.3 >
On Sat, Jun 30, 2018 at 02:58:27PM -0500, Bjorn Helgaas wrote: > On Mon, Jun 11, 2018 at 04:29:18PM -0600, Keith Busch wrote: > > A bridge that supports D3 but not hotplug will be subject to runtime > > power management placing it in a non-operation power state if it doesn't > > have any devices attached. This patch will prevent this power management > > during error recovery so that the rescan at the end may be successful. > > I assume you're seeing a problem that's fixed by this patch? > > I'm not very familiar with the pm_runtime_*() interfaces. The fact > that drivers/pci/ only contains two calls to pm_runtime_forbid() and > two to pm_runtime_allow() (one in a quirk) makes me wonder if this is > the right approach. The use of pm_runtime_forbid() / pm_runtime_allow() is wrong here. The user may forbid and allow runtime PM on a device by writing "on" or "auto" to its power/control file in sysfs. The two functions have the same effect, but the user may override them at any time. If the intention is to ensure the bridge is runtime resumed while a rescan is in progess, the appropriate functions to call are pm_runtime_get_sync() and pm_runtime_put(). > Do we need something similar in other enumeration paths, e.g., > boot-time enumeration, hot-add, sysfs "rescan" entry, etc? Definitely not on boot-time enumeration. PCI devices are kept in D0 if not bound to a driver, and the PCI core first scans all devices, only then attaches drivers. As for hot-add, hotplug bridges are currently always kept in D0, but I've posted a patch series two weeks ago to add runtime PM. Did you get a chance to look at it? Patch [28/32] of that series resumes the bridge to D0 on hot-add ("PCI: pciehp: Resume to D0 on enable/disable"). As for sysfs "rescan", this does lack runtime PM calls currently, as do many other sysfs accessors. The above-mentioned series adds runtime PM to sysfs "reset" in patch [30/32] ("PCI: sysfs: Resume to D0 on function reset"). The patch does not depend on the preceding patches in the series, it could be applied out of order. Amending all the other accessors would be another big project. It's not as trivial as sprinkling a few runtime PM calls, it requires careful thought. E.g. if only config space is accessed, it's sufficient to resume the parent with pci_config_pm_runtime_get() instead of the device itself with pm_runtime_get_sync(). In other cases the information is read from config space even though it's cached, e.g. secondary_bus_number_show(). In those cases it might be better to always return the cached data instead of adding runtime PM calls. FWIW, from a quick look proc.c seems fine as far as runtime PM is concerned. Thanks, Lukas > > > Cc: Oza Pawandeep <poza@codeaurora.org> > > Signed-off-by: Keith Busch <keith.busch@intel.com> > > --- > > drivers/pci/pcie/err.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c > > index f7ce0cb0b0b7..247b6ce14f0d 100644 > > --- a/drivers/pci/pcie/err.c > > +++ b/drivers/pci/pcie/err.c > > @@ -16,6 +16,7 @@ > > #include <linux/kernel.h> > > #include <linux/errno.h> > > #include <linux/aer.h> > > +#include <linux/pm_runtime.h> > > #include "portdrv.h" > > #include "../pci.h" > > > > @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) > > udev = dev->bus->self; > > > > parent = udev->subordinate; > > + pm_runtime_forbid(&udev->dev); > > pci_lock_rescan_remove(); > > list_for_each_entry_safe_reverse(pdev, temp, &parent->devices, > > bus_list) { > > @@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) > > } > > > > pci_unlock_rescan_remove(); > > + pm_runtime_allow(&udev->dev); > > } > > > > /** > > -- > > 2.14.3
diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c index f7ce0cb0b0b7..247b6ce14f0d 100644 --- a/drivers/pci/pcie/err.c +++ b/drivers/pci/pcie/err.c @@ -16,6 +16,7 @@ #include <linux/kernel.h> #include <linux/errno.h> #include <linux/aer.h> +#include <linux/pm_runtime.h> #include "portdrv.h" #include "../pci.h" @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) udev = dev->bus->self; parent = udev->subordinate; + pm_runtime_forbid(&udev->dev); pci_lock_rescan_remove(); list_for_each_entry_safe_reverse(pdev, temp, &parent->devices, bus_list) { @@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service) } pci_unlock_rescan_remove(); + pm_runtime_allow(&udev->dev); } /**
A bridge that supports D3 but not hotplug will be subject to runtime power management placing it in a non-operation power state if it doesn't have any devices attached. This patch will prevent this power management during error recovery so that the rescan at the end may be successful. Cc: Oza Pawandeep <poza@codeaurora.org> Signed-off-by: Keith Busch <keith.busch@intel.com> --- drivers/pci/pcie/err.c | 3 +++ 1 file changed, 3 insertions(+)