| Message ID | 20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] PCI: dwc: Enable runtime pm of the host bridge | expand |
On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > The Controller driver is the parent device of the PCIe host bridge, > PCI-PCI bridge and PCIe endpoint as shown below. Nit: add blank line here. > PCIe controller(Top level parent & parent of host bridge) > | > v > PCIe Host bridge(Parent of PCI-PCI bridge) > | > v > PCI-PCI bridge(Parent of endpoint driver) > | > v > PCIe endpoint driver Nit: use spaces instead of tabs to ensure this still looks good when "git log" indents this. In this case it doesn't seem to matter, > Since runtime PM is disabled for host bridge, the state of the child > devices under the host bridge is not taken into account by PM framework > for the top level parent, PCIe controller. So PM framework, allows > the controller driver to enter runtime PM irrespective of the state > of the devices under the host bridge. IIUC this says that we runtime suspend the controller even though runtime PM is disabled for the host bridge? I have a hard time parsing this; can you cite a function that does this or some relevant documentation about how this part of runtime PM works? > And this causes the topology breakage and also possible PM issues. Not sure what this refers to, since you didn't mention topology breakage earlier. And "possible PM" issues is too vague to be useful. > So enable pm runtime for the host bridge, so that controller driver > goes to suspend only when all child devices goes to runtime suspend. s/pm runtime/runtime PM/ so all the references match (unless you mean something different here) (also in subject line) > Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com> > --- > Changes in v2: > - Updated commit message as suggested by mani. > - Link to v1: https://lore.kernel.org/r/20240219-runtime_pm_enable-v1-1-d39660310504@quicinc.com > --- > drivers/pci/controller/dwc/pcie-designware-host.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > index d5fc31f8345f..57756a73df30 100644 > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > @@ -16,6 +16,7 @@ > #include <linux/of_pci.h> > #include <linux/pci_regs.h> > #include <linux/platform_device.h> > +#include <linux/pm_runtime.h> > > #include "../../pci.h" > #include "pcie-designware.h" > @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) > if (pp->ops->post_init) > pp->ops->post_init(pp); > > + pm_runtime_set_active(&bridge->dev); There are currently no callers of pm_runtime_set_active() in drivers/pci/controller/. This adds it to dw_pcie_host_init(), but it doesn't seem to be a DWC-specific issue, so I assume other drivers and driver cores like cadence and mobiveil should have this, too? > + pm_runtime_enable(&bridge->dev); There are several existing calls of pci_runtime_enable(), including from several DWC drivers. Are they now redundant? In addition, [1] suggests that pm_runtime_enable() should be called *after* pm_runtime_set_active(), but these existing calls (dra7xx_pcie_probe(), ks_pcie_probe(), qcom_pcie_probe(), rcar_gen4_pcie_prepare(), tegra_pcie_config_rp()) happen *before* dw_pcie_host_init() calls pm_runtime_set_active(). [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/power/runtime_pm.rst?id=v6.7#n582 > return 0; > > err_stop_link: > > --- > base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d > change-id: 20240219-runtime_pm_enable-bdc17914bd50 > > Best regards, > -- > Krishna chaitanya chundru <quic_krichai@quicinc.com> >
On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: >> The Controller driver is the parent device of the PCIe host bridge, >> PCI-PCI bridge and PCIe endpoint as shown below. > > Nit: add blank line here. > Ack. >> PCIe controller(Top level parent & parent of host bridge) >> | >> v >> PCIe Host bridge(Parent of PCI-PCI bridge) >> | >> v >> PCI-PCI bridge(Parent of endpoint driver) >> | >> v >> PCIe endpoint driver > > Nit: use spaces instead of tabs to ensure this still looks good when > "git log" indents this. In this case it doesn't seem to matter, > Ack. >> Since runtime PM is disabled for host bridge, the state of the child >> devices under the host bridge is not taken into account by PM framework >> for the top level parent, PCIe controller. So PM framework, allows >> the controller driver to enter runtime PM irrespective of the state >> of the devices under the host bridge. > > IIUC this says that we runtime suspend the controller even though > runtime PM is disabled for the host bridge? I have a hard time > parsing this; can you cite a function that does this or some relevant > documentation about how this part of runtime PM works? > Generally controller should go to runtime suspend when endpoint client drivers and pci-pci host bridge drivers goes to runtime suspend as the controller driver is the parent, but we are observing controller driver goes to runtime suspend even when client drivers and PCI-PCI bridge are in active state. Unfortunately I don't have any reference for any documentation about how runtime PM works. >> And this causes the topology breakage and also possible PM issues. > > Not sure what this refers to, since you didn't mention topology > breakage earlier. And "possible PM" issues is too vague to be useful. > >> So enable pm runtime for the host bridge, so that controller driver >> goes to suspend only when all child devices goes to runtime suspend. > > s/pm runtime/runtime PM/ so all the references match (unless you mean > something different here) (also in subject line) > >> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com> >> --- >> Changes in v2: >> - Updated commit message as suggested by mani. >> - Link to v1: https://lore.kernel.org/r/20240219-runtime_pm_enable-v1-1-d39660310504@quicinc.com >> --- >> drivers/pci/controller/dwc/pcie-designware-host.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c >> index d5fc31f8345f..57756a73df30 100644 >> --- a/drivers/pci/controller/dwc/pcie-designware-host.c >> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c >> @@ -16,6 +16,7 @@ >> #include <linux/of_pci.h> >> #include <linux/pci_regs.h> >> #include <linux/platform_device.h> >> +#include <linux/pm_runtime.h> >> >> #include "../../pci.h" >> #include "pcie-designware.h" >> @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) >> if (pp->ops->post_init) >> pp->ops->post_init(pp); >> >> + pm_runtime_set_active(&bridge->dev); > > There are currently no callers of pm_runtime_set_active() in > drivers/pci/controller/. This adds it to dw_pcie_host_init(), but it > doesn't seem to be a DWC-specific issue, so I assume other drivers and > driver cores like cadence and mobiveil should have this, too? > I think cadence and mobiveil also should have this. >> + pm_runtime_enable(&bridge->dev); > > There are several existing calls of pci_runtime_enable(), including > from several DWC drivers. Are they now redundant? > No These are not redundant, Those drivers are calling this pm_runtime_enable() for their controller dev node. Here we are calling runtime enable for the bridge dev which is allocated and used by the PCIe framework. - Krishna Chaitanya. > In addition, [1] suggests that pm_runtime_enable() should be called > *after* pm_runtime_set_active(), but these existing calls > (dra7xx_pcie_probe(), ks_pcie_probe(), qcom_pcie_probe(), > rcar_gen4_pcie_prepare(), tegra_pcie_config_rp()) happen *before* > dw_pcie_host_init() calls pm_runtime_set_active(). > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/power/runtime_pm.rst?id=v6.7#n582 > >> return 0; >> >> err_stop_link: >> >> --- >> base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d >> change-id: 20240219-runtime_pm_enable-bdc17914bd50 >> >> Best regards, >> -- >> Krishna chaitanya chundru <quic_krichai@quicinc.com> >>
[+to Rafael, sorry, another runtime PM question, beginning of thread: https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > The Controller driver is the parent device of the PCIe host bridge, > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > | > > > v > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > | > > > v > > > PCI-PCI bridge(Parent of endpoint driver) > > > | > > > v > > > PCIe endpoint driver > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > devices under the host bridge is not taken into account by PM framework > > > for the top level parent, PCIe controller. So PM framework, allows > > > the controller driver to enter runtime PM irrespective of the state > > > of the devices under the host bridge. > > > > IIUC this says that we runtime suspend the controller even though > > runtime PM is disabled for the host bridge? I have a hard time > > parsing this; can you cite a function that does this or some relevant > > documentation about how this part of runtime PM works? > > > Generally controller should go to runtime suspend when endpoint client > drivers and pci-pci host bridge drivers goes to runtime suspend as the > controller driver is the parent, but we are observing controller driver > goes to runtime suspend even when client drivers and PCI-PCI bridge are > in active state. It surprises me that a device could be suspended while children are active. A PCI-PCI bridge must be in D0 for any devices below it to be active. The controller is a platform device, not a PCI device, but I am similarly surprised that we would suspend it when children are active, which makes me think we didn't set the hierarchy up correctly. It doesn't seem like we should need to enable runtime PM for a parent to keep it from being suspended when children are active. > > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > > > @@ -16,6 +16,7 @@ > > > #include <linux/of_pci.h> > > > #include <linux/pci_regs.h> > > > #include <linux/platform_device.h> > > > +#include <linux/pm_runtime.h> > > > > > > #include "../../pci.h" > > > #include "pcie-designware.h" > > > @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) > > > if (pp->ops->post_init) > > > pp->ops->post_init(pp); > > > > > > + pm_runtime_set_active(&bridge->dev); > > > + pm_runtime_enable(&bridge->dev); > > > + > > > return 0; > > > > > > err_stop_link: Bjorn
On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > [+to Rafael, sorry, another runtime PM question, beginning of thread: > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: >> On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: >>> On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: >>>> The Controller driver is the parent device of the PCIe host bridge, >>>> PCI-PCI bridge and PCIe endpoint as shown below. >>>> >>>> PCIe controller(Top level parent & parent of host bridge) >>>> | >>>> v >>>> PCIe Host bridge(Parent of PCI-PCI bridge) >>>> | >>>> v >>>> PCI-PCI bridge(Parent of endpoint driver) >>>> | >>>> v >>>> PCIe endpoint driver >>>> >>>> Since runtime PM is disabled for host bridge, the state of the child >>>> devices under the host bridge is not taken into account by PM framework >>>> for the top level parent, PCIe controller. So PM framework, allows >>>> the controller driver to enter runtime PM irrespective of the state >>>> of the devices under the host bridge. >>> >>> IIUC this says that we runtime suspend the controller even though >>> runtime PM is disabled for the host bridge? I have a hard time >>> parsing this; can you cite a function that does this or some relevant >>> documentation about how this part of runtime PM works? >>> >> Generally controller should go to runtime suspend when endpoint client >> drivers and pci-pci host bridge drivers goes to runtime suspend as the >> controller driver is the parent, but we are observing controller driver >> goes to runtime suspend even when client drivers and PCI-PCI bridge are >> in active state. > > It surprises me that a device could be suspended while children are > active. A PCI-PCI bridge must be in D0 for any devices below it to be > active. The controller is a platform device, not a PCI device, but I > am similarly surprised that we would suspend it when children are > active, which makes me think we didn't set the hierarchy up correctly. > > It doesn't seem like we should need to enable runtime PM for a parent > to keep it from being suspended when children are active. Here we are not enabling runtime PM of the controller device, we are enabling runtime PM for the bridge device which is maintained by the PCIe framework. The bridge device is the parent of the PCI-PCI bridge and child of the controller device. As the bridge device's runtime PM is not enabled the PM framework is ignoring the child's runtime status. - Krishna Chaitanya. >>>> --- a/drivers/pci/controller/dwc/pcie-designware-host.c >>>> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c >>>> @@ -16,6 +16,7 @@ >>>> #include <linux/of_pci.h> >>>> #include <linux/pci_regs.h> >>>> #include <linux/platform_device.h> >>>> +#include <linux/pm_runtime.h> >>>> >>>> #include "../../pci.h" >>>> #include "pcie-designware.h" >>>> @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) >>>> if (pp->ops->post_init) >>>> pp->ops->post_init(pp); >>>> >>>> + pm_runtime_set_active(&bridge->dev); >>>> + pm_runtime_enable(&bridge->dev); >>>> + >>>> return 0; >>>> >>>> err_stop_link: > > Bjorn
On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: > On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > > [+to Rafael, sorry, another runtime PM question, beginning of thread: > > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > > > The Controller driver is the parent device of the PCIe host bridge, > > > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > > > | > > > > > v > > > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > > > | > > > > > v > > > > > PCI-PCI bridge(Parent of endpoint driver) > > > > > | > > > > > v > > > > > PCIe endpoint driver > > > > > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > > > devices under the host bridge is not taken into account by PM framework > > > > > for the top level parent, PCIe controller. So PM framework, allows > > > > > the controller driver to enter runtime PM irrespective of the state > > > > > of the devices under the host bridge. > > > > > > > > IIUC this says that we runtime suspend the controller even though > > > > runtime PM is disabled for the host bridge? I have a hard time > > > > parsing this; can you cite a function that does this or some relevant > > > > documentation about how this part of runtime PM works? > > > > > > > Generally controller should go to runtime suspend when endpoint client > > > drivers and pci-pci host bridge drivers goes to runtime suspend as the > > > controller driver is the parent, but we are observing controller driver > > > goes to runtime suspend even when client drivers and PCI-PCI bridge are > > > in active state. > > > > It surprises me that a device could be suspended while children are > > active. A PCI-PCI bridge must be in D0 for any devices below it to be > > active. The controller is a platform device, not a PCI device, but I > > am similarly surprised that we would suspend it when children are > > active, which makes me think we didn't set the hierarchy up correctly. > > > > It doesn't seem like we should need to enable runtime PM for a parent > > to keep it from being suspended when children are active. > > Here we are not enabling runtime PM of the controller device, we are > enabling runtime PM for the bridge device which is maintained by the > PCIe framework. The bridge device is the parent of the PCI-PCI > bridge and child of the controller device. As the bridge device's > runtime PM is not enabled the PM framework is ignoring the child's > runtime status. OK, it's the host bridge, not the controller. I'm still surprised that the PM framework will runtime suspend a device when child devices are active. And further confused about managing the host bridge runtime PM in a controller driver. Which other callers of pci_alloc_host_bridge() or devm_pci_alloc_host_bridge() will need similar changes? > > > > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > > > > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > > > > > @@ -16,6 +16,7 @@ > > > > > #include <linux/of_pci.h> > > > > > #include <linux/pci_regs.h> > > > > > #include <linux/platform_device.h> > > > > > +#include <linux/pm_runtime.h> > > > > > > > > > > #include "../../pci.h" > > > > > #include "pcie-designware.h" > > > > > @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) > > > > > if (pp->ops->post_init) > > > > > pp->ops->post_init(pp); > > > > > > > > > > + pm_runtime_set_active(&bridge->dev); > > > > > + pm_runtime_enable(&bridge->dev); > > > > > + > > > > > return 0; > > > > > > > > > > err_stop_link:
On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: > On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: > > On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > > > [+to Rafael, sorry, another runtime PM question, beginning of thread: > > > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > > > > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > > > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > > > > The Controller driver is the parent device of the PCIe host bridge, > > > > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > > > > | > > > > > > v > > > > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > > > > | > > > > > > v > > > > > > PCI-PCI bridge(Parent of endpoint driver) > > > > > > | > > > > > > v > > > > > > PCIe endpoint driver > > > > > > > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > > > > devices under the host bridge is not taken into account by PM framework > > > > > > for the top level parent, PCIe controller. So PM framework, allows > > > > > > the controller driver to enter runtime PM irrespective of the state > > > > > > of the devices under the host bridge. > > > > > > > > > > IIUC this says that we runtime suspend the controller even though > > > > > runtime PM is disabled for the host bridge? I have a hard time > > > > > parsing this; can you cite a function that does this or some relevant > > > > > documentation about how this part of runtime PM works? > > > > > > > > > Generally controller should go to runtime suspend when endpoint client > > > > drivers and pci-pci host bridge drivers goes to runtime suspend as the > > > > controller driver is the parent, but we are observing controller driver > > > > goes to runtime suspend even when client drivers and PCI-PCI bridge are > > > > in active state. > > > > > > It surprises me that a device could be suspended while children are > > > active. A PCI-PCI bridge must be in D0 for any devices below it to be > > > active. The controller is a platform device, not a PCI device, but I > > > am similarly surprised that we would suspend it when children are > > > active, which makes me think we didn't set the hierarchy up correctly. > > > > > > It doesn't seem like we should need to enable runtime PM for a parent > > > to keep it from being suspended when children are active. > > > > Here we are not enabling runtime PM of the controller device, we are > > enabling runtime PM for the bridge device which is maintained by the > > PCIe framework. The bridge device is the parent of the PCI-PCI > > bridge and child of the controller device. As the bridge device's > > runtime PM is not enabled the PM framework is ignoring the child's > > runtime status. > > OK, it's the host bridge, not the controller. > > I'm still surprised that the PM framework will runtime suspend a > device when child devices are active. > There is a catch here. Even though the child devices are funtionally active, PM framework will only consider their runtime_pm state, which is initially set to 'disabled' for all devices. It is upto the device drivers to enable it when required. Here is the initial runtime PM status of each device post boot: Controller device -> disabled initially but enabled by pcie-qcom.c Host bridge -> disabled initially PCIe bridge -> disabled initially but conditionally enabled by portdrv.c PCIe devices -> disabled initially but enabled by respective drivers like WLAN Now, when the controller device goes to runtime suspend, PM framework will check the runtime PM state of the child device (host bridge) and will find it to be disabled. So it will allow the parent (controller device) to go to runtime suspend. Only if the child device's state was 'active' it will prevent the parent to get suspended. But you may wonder if this is ideal? IMO NO. But we cannot blame the PM framework here. The responsibility is within the device drivers to handle the PM state based on the usecase. Ideally, the host bridge driver should've handled runtime PM state during the probe time. Otherwise, PM framework wouldn't know when would be the best time to suspend the devices. > And further confused about managing the host bridge runtime PM in a > controller driver. Which other callers of pci_alloc_host_bridge() or > devm_pci_alloc_host_bridge() will need similar changes? > This scenario applies to all host bridges. So I think we should enable it inside pci_host_probe(). - Mani
On 3/19/2024 4:41 PM, Manivannan Sadhasivam wrote: > On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: >> On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: >>> On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: >>>> [+to Rafael, sorry, another runtime PM question, beginning of thread: >>>> https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] >>>> >>>> On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: >>>>> On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: >>>>>> On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: >>>>>>> The Controller driver is the parent device of the PCIe host bridge, >>>>>>> PCI-PCI bridge and PCIe endpoint as shown below. >>>>>>> >>>>>>> PCIe controller(Top level parent & parent of host bridge) >>>>>>> | >>>>>>> v >>>>>>> PCIe Host bridge(Parent of PCI-PCI bridge) >>>>>>> | >>>>>>> v >>>>>>> PCI-PCI bridge(Parent of endpoint driver) >>>>>>> | >>>>>>> v >>>>>>> PCIe endpoint driver >>>>>>> >>>>>>> Since runtime PM is disabled for host bridge, the state of the child >>>>>>> devices under the host bridge is not taken into account by PM framework >>>>>>> for the top level parent, PCIe controller. So PM framework, allows >>>>>>> the controller driver to enter runtime PM irrespective of the state >>>>>>> of the devices under the host bridge. >>>>>> >>>>>> IIUC this says that we runtime suspend the controller even though >>>>>> runtime PM is disabled for the host bridge? I have a hard time >>>>>> parsing this; can you cite a function that does this or some relevant >>>>>> documentation about how this part of runtime PM works? >>>>>> >>>>> Generally controller should go to runtime suspend when endpoint client >>>>> drivers and pci-pci host bridge drivers goes to runtime suspend as the >>>>> controller driver is the parent, but we are observing controller driver >>>>> goes to runtime suspend even when client drivers and PCI-PCI bridge are >>>>> in active state. >>>> >>>> It surprises me that a device could be suspended while children are >>>> active. A PCI-PCI bridge must be in D0 for any devices below it to be >>>> active. The controller is a platform device, not a PCI device, but I >>>> am similarly surprised that we would suspend it when children are >>>> active, which makes me think we didn't set the hierarchy up correctly. >>>> >>>> It doesn't seem like we should need to enable runtime PM for a parent >>>> to keep it from being suspended when children are active. >>> >>> Here we are not enabling runtime PM of the controller device, we are >>> enabling runtime PM for the bridge device which is maintained by the >>> PCIe framework. The bridge device is the parent of the PCI-PCI >>> bridge and child of the controller device. As the bridge device's >>> runtime PM is not enabled the PM framework is ignoring the child's >>> runtime status. >> >> OK, it's the host bridge, not the controller. >> >> I'm still surprised that the PM framework will runtime suspend a >> device when child devices are active. >> > > There is a catch here. Even though the child devices are funtionally active, PM > framework will only consider their runtime_pm state, which is initially set to > 'disabled' for all devices. It is upto the device drivers to enable it when > required. > > Here is the initial runtime PM status of each device post boot: > > Controller device -> disabled initially but enabled by pcie-qcom.c > Host bridge -> disabled initially > PCIe bridge -> disabled initially but conditionally enabled by portdrv.c > PCIe devices -> disabled initially but enabled by respective drivers like WLAN > > Now, when the controller device goes to runtime suspend, PM framework will check > the runtime PM state of the child device (host bridge) and will find it to be > disabled. So it will allow the parent (controller device) to go to runtime > suspend. Only if the child device's state was 'active' it will prevent the > parent to get suspended. > > But you may wonder if this is ideal? IMO NO. But we cannot blame the PM > framework here. The responsibility is within the device drivers to handle the PM > state based on the usecase. Ideally, the host bridge driver should've handled > runtime PM state during the probe time. Otherwise, PM framework wouldn't know > when would be the best time to suspend the devices. > >> And further confused about managing the host bridge runtime PM in a >> controller driver. Which other callers of pci_alloc_host_bridge() or >> devm_pci_alloc_host_bridge() will need similar changes? >> > > This scenario applies to all host bridges. So I think we should enable it inside > pci_host_probe(). > > - Mani > I will these runtime enable inside the pci_host_probe(). - Krishna Chaitanya.
On Tue, Mar 19, 2024 at 04:41:48PM +0530, Manivannan Sadhasivam wrote: > On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: > > On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: > > > On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > > > > [+to Rafael, sorry, another runtime PM question, beginning of thread: > > > > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > > > > > > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > > > > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > > > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > > > > > The Controller driver is the parent device of the PCIe host bridge, > > > > > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > > > > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > > > > > | > > > > > > > v > > > > > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > > > > > | > > > > > > > v > > > > > > > PCI-PCI bridge(Parent of endpoint driver) > > > > > > > | > > > > > > > v > > > > > > > PCIe endpoint driver > > > > > > > > > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > > > > > devices under the host bridge is not taken into account by PM framework > > > > > > > for the top level parent, PCIe controller. So PM framework, allows > > > > > > > the controller driver to enter runtime PM irrespective of the state > > > > > > > of the devices under the host bridge. > > > > > > > > > > > > IIUC this says that we runtime suspend the controller even though > > > > > > runtime PM is disabled for the host bridge? I have a hard time > > > > > > parsing this; can you cite a function that does this or some relevant > > > > > > documentation about how this part of runtime PM works? > > > > > > > > > > > Generally controller should go to runtime suspend when endpoint client > > > > > drivers and pci-pci host bridge drivers goes to runtime suspend as the > > > > > controller driver is the parent, but we are observing controller driver > > > > > goes to runtime suspend even when client drivers and PCI-PCI bridge are > > > > > in active state. > > > > > > > > It surprises me that a device could be suspended while children are > > > > active. A PCI-PCI bridge must be in D0 for any devices below it to be > > > > active. The controller is a platform device, not a PCI device, but I > > > > am similarly surprised that we would suspend it when children are > > > > active, which makes me think we didn't set the hierarchy up correctly. > > > > > > > > It doesn't seem like we should need to enable runtime PM for a parent > > > > to keep it from being suspended when children are active. > > > > > > Here we are not enabling runtime PM of the controller device, we are > > > enabling runtime PM for the bridge device which is maintained by the > > > PCIe framework. The bridge device is the parent of the PCI-PCI > > > bridge and child of the controller device. As the bridge device's > > > runtime PM is not enabled the PM framework is ignoring the child's > > > runtime status. > > > > OK, it's the host bridge, not the controller. > > > > I'm still surprised that the PM framework will runtime suspend a > > device when child devices are active. > > There is a catch here. Even though the child devices are funtionally > active, PM framework will only consider their runtime_pm state, > which is initially set to 'disabled' for all devices. It is upto the > device drivers to enable it when required. > > Here is the initial runtime PM status of each device post boot: > > Controller device -> disabled initially but enabled by pcie-qcom.c > Host bridge -> disabled initially > PCIe bridge -> disabled initially but conditionally enabled by portdrv.c > PCIe devices -> disabled initially but enabled by respective drivers like WLAN > > Now, when the controller device goes to runtime suspend, PM > framework will check the runtime PM state of the child device (host > bridge) and will find it to be disabled. So it will allow the parent > (controller device) to go to runtime suspend. Only if the child > device's state was 'active' it will prevent the parent to get > suspended. > > But you may wonder if this is ideal? IMO NO. But we cannot blame the > PM framework here. The responsibility is within the device drivers > to handle the PM state based on the usecase. Ideally, the host > bridge driver should've handled runtime PM state during the probe > time. Otherwise, PM framework wouldn't know when would be the best > time to suspend the devices. My expectation is that adding new functionality should only require changes in drivers that want to take advantage of it. For example, if we add runtime PM support in the controller driver, the result should be functionally correct even if we don't update drivers for downstream devices. If that's not the way it works, I suggest that would be a problem in the PM framework. The host bridge might be a special case because we don't have a separate "host bridge" driver; that code is kind of integrated with the controller drivers. So maybe it's OK to do controller + host bridge runtime PM support at the same time, as long as any time we add runtime PM to a controller, we sure it's also set up for the host bridge. Bjorn
On Fri, Mar 22, 2024 at 05:04:56PM -0500, Bjorn Helgaas wrote: > On Tue, Mar 19, 2024 at 04:41:48PM +0530, Manivannan Sadhasivam wrote: > > On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: > > > On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: > > > > On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: > > > > > [+to Rafael, sorry, another runtime PM question, beginning of thread: > > > > > https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] > > > > > > > > > > On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: > > > > > > On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: > > > > > > > On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: > > > > > > > > The Controller driver is the parent device of the PCIe host bridge, > > > > > > > > PCI-PCI bridge and PCIe endpoint as shown below. > > > > > > > > > > > > > > > > PCIe controller(Top level parent & parent of host bridge) > > > > > > > > | > > > > > > > > v > > > > > > > > PCIe Host bridge(Parent of PCI-PCI bridge) > > > > > > > > | > > > > > > > > v > > > > > > > > PCI-PCI bridge(Parent of endpoint driver) > > > > > > > > | > > > > > > > > v > > > > > > > > PCIe endpoint driver > > > > > > > > > > > > > > > > Since runtime PM is disabled for host bridge, the state of the child > > > > > > > > devices under the host bridge is not taken into account by PM framework > > > > > > > > for the top level parent, PCIe controller. So PM framework, allows > > > > > > > > the controller driver to enter runtime PM irrespective of the state > > > > > > > > of the devices under the host bridge. > > > > > > > > > > > > > > IIUC this says that we runtime suspend the controller even though > > > > > > > runtime PM is disabled for the host bridge? I have a hard time > > > > > > > parsing this; can you cite a function that does this or some relevant > > > > > > > documentation about how this part of runtime PM works? > > > > > > > > > > > > > Generally controller should go to runtime suspend when endpoint client > > > > > > drivers and pci-pci host bridge drivers goes to runtime suspend as the > > > > > > controller driver is the parent, but we are observing controller driver > > > > > > goes to runtime suspend even when client drivers and PCI-PCI bridge are > > > > > > in active state. > > > > > > > > > > It surprises me that a device could be suspended while children are > > > > > active. A PCI-PCI bridge must be in D0 for any devices below it to be > > > > > active. The controller is a platform device, not a PCI device, but I > > > > > am similarly surprised that we would suspend it when children are > > > > > active, which makes me think we didn't set the hierarchy up correctly. > > > > > > > > > > It doesn't seem like we should need to enable runtime PM for a parent > > > > > to keep it from being suspended when children are active. > > > > > > > > Here we are not enabling runtime PM of the controller device, we are > > > > enabling runtime PM for the bridge device which is maintained by the > > > > PCIe framework. The bridge device is the parent of the PCI-PCI > > > > bridge and child of the controller device. As the bridge device's > > > > runtime PM is not enabled the PM framework is ignoring the child's > > > > runtime status. > > > > > > OK, it's the host bridge, not the controller. > > > > > > I'm still surprised that the PM framework will runtime suspend a > > > device when child devices are active. > > > > There is a catch here. Even though the child devices are funtionally > > active, PM framework will only consider their runtime_pm state, > > which is initially set to 'disabled' for all devices. It is upto the > > device drivers to enable it when required. > > > > Here is the initial runtime PM status of each device post boot: > > > > Controller device -> disabled initially but enabled by pcie-qcom.c > > Host bridge -> disabled initially > > PCIe bridge -> disabled initially but conditionally enabled by portdrv.c > > PCIe devices -> disabled initially but enabled by respective drivers like WLAN > > > > Now, when the controller device goes to runtime suspend, PM > > framework will check the runtime PM state of the child device (host > > bridge) and will find it to be disabled. So it will allow the parent > > (controller device) to go to runtime suspend. Only if the child > > device's state was 'active' it will prevent the parent to get > > suspended. > > > > But you may wonder if this is ideal? IMO NO. But we cannot blame the > > PM framework here. The responsibility is within the device drivers > > to handle the PM state based on the usecase. Ideally, the host > > bridge driver should've handled runtime PM state during the probe > > time. Otherwise, PM framework wouldn't know when would be the best > > time to suspend the devices. > > My expectation is that adding new functionality should only require > changes in drivers that want to take advantage of it. For example, if > we add runtime PM support in the controller driver, the result should > be functionally correct even if we don't update drivers for downstream > devices. > Well, IMO PM framework should disable runtime PM for the parent if the child's runtime PM state is disabled. It'd be good to get the opinion of Rafael. - Mani > If that's not the way it works, I suggest that would be a problem in > the PM framework. > > The host bridge might be a special case because we don't have a > separate "host bridge" driver; that code is kind of integrated with > the controller drivers. So maybe it's OK to do controller + host > bridge runtime PM support at the same time, as long as any time we add > runtime PM to a controller, we sure it's also set up for the host > bridge. > > Bjorn
On 3/25/2024 4:39 PM, Manivannan Sadhasivam wrote: > On Fri, Mar 22, 2024 at 05:04:56PM -0500, Bjorn Helgaas wrote: >> On Tue, Mar 19, 2024 at 04:41:48PM +0530, Manivannan Sadhasivam wrote: >>> On Fri, Mar 08, 2024 at 11:12:48AM -0600, Bjorn Helgaas wrote: >>>> On Fri, Mar 08, 2024 at 08:38:52AM +0530, Krishna Chaitanya Chundru wrote: >>>>> On 3/8/2024 3:25 AM, Bjorn Helgaas wrote: >>>>>> [+to Rafael, sorry, another runtime PM question, beginning of thread: >>>>>> https://lore.kernel.org/r/20240305-runtime_pm_enable-v2-1-a849b74091d1@quicinc.com] >>>>>> >>>>>> On Thu, Mar 07, 2024 at 07:28:54AM +0530, Krishna Chaitanya Chundru wrote: >>>>>>> On 3/6/2024 1:27 AM, Bjorn Helgaas wrote: >>>>>>>> On Tue, Mar 05, 2024 at 03:19:01PM +0530, Krishna chaitanya chundru wrote: >>>>>>>>> The Controller driver is the parent device of the PCIe host bridge, >>>>>>>>> PCI-PCI bridge and PCIe endpoint as shown below. >>>>>>>>> >>>>>>>>> PCIe controller(Top level parent & parent of host bridge) >>>>>>>>> | >>>>>>>>> v >>>>>>>>> PCIe Host bridge(Parent of PCI-PCI bridge) >>>>>>>>> | >>>>>>>>> v >>>>>>>>> PCI-PCI bridge(Parent of endpoint driver) >>>>>>>>> | >>>>>>>>> v >>>>>>>>> PCIe endpoint driver >>>>>>>>> >>>>>>>>> Since runtime PM is disabled for host bridge, the state of the child >>>>>>>>> devices under the host bridge is not taken into account by PM framework >>>>>>>>> for the top level parent, PCIe controller. So PM framework, allows >>>>>>>>> the controller driver to enter runtime PM irrespective of the state >>>>>>>>> of the devices under the host bridge. >>>>>>>> >>>>>>>> IIUC this says that we runtime suspend the controller even though >>>>>>>> runtime PM is disabled for the host bridge? I have a hard time >>>>>>>> parsing this; can you cite a function that does this or some relevant >>>>>>>> documentation about how this part of runtime PM works? >>>>>>>> >>>>>>> Generally controller should go to runtime suspend when endpoint client >>>>>>> drivers and pci-pci host bridge drivers goes to runtime suspend as the >>>>>>> controller driver is the parent, but we are observing controller driver >>>>>>> goes to runtime suspend even when client drivers and PCI-PCI bridge are >>>>>>> in active state. >>>>>> >>>>>> It surprises me that a device could be suspended while children are >>>>>> active. A PCI-PCI bridge must be in D0 for any devices below it to be >>>>>> active. The controller is a platform device, not a PCI device, but I >>>>>> am similarly surprised that we would suspend it when children are >>>>>> active, which makes me think we didn't set the hierarchy up correctly. >>>>>> >>>>>> It doesn't seem like we should need to enable runtime PM for a parent >>>>>> to keep it from being suspended when children are active. >>>>> >>>>> Here we are not enabling runtime PM of the controller device, we are >>>>> enabling runtime PM for the bridge device which is maintained by the >>>>> PCIe framework. The bridge device is the parent of the PCI-PCI >>>>> bridge and child of the controller device. As the bridge device's >>>>> runtime PM is not enabled the PM framework is ignoring the child's >>>>> runtime status. >>>> >>>> OK, it's the host bridge, not the controller. >>>> >>>> I'm still surprised that the PM framework will runtime suspend a >>>> device when child devices are active. >>> >>> There is a catch here. Even though the child devices are funtionally >>> active, PM framework will only consider their runtime_pm state, >>> which is initially set to 'disabled' for all devices. It is upto the >>> device drivers to enable it when required. >>> >>> Here is the initial runtime PM status of each device post boot: >>> >>> Controller device -> disabled initially but enabled by pcie-qcom.c >>> Host bridge -> disabled initially >>> PCIe bridge -> disabled initially but conditionally enabled by portdrv.c >>> PCIe devices -> disabled initially but enabled by respective drivers like WLAN >>> >>> Now, when the controller device goes to runtime suspend, PM >>> framework will check the runtime PM state of the child device (host >>> bridge) and will find it to be disabled. So it will allow the parent >>> (controller device) to go to runtime suspend. Only if the child >>> device's state was 'active' it will prevent the parent to get >>> suspended. >>> >>> But you may wonder if this is ideal? IMO NO. But we cannot blame the >>> PM framework here. The responsibility is within the device drivers >>> to handle the PM state based on the usecase. Ideally, the host >>> bridge driver should've handled runtime PM state during the probe >>> time. Otherwise, PM framework wouldn't know when would be the best >>> time to suspend the devices. >> >> My expectation is that adding new functionality should only require >> changes in drivers that want to take advantage of it. For example, if >> we add runtime PM support in the controller driver, the result should >> be functionally correct even if we don't update drivers for downstream >> devices. >> > > Well, IMO PM framework should disable runtime PM for the parent if the child's > runtime PM state is disabled. > > It'd be good to get the opinion of Rafael. > Hi Rafeal, can you please comment on this. > - Mani > >> If that's not the way it works, I suggest that would be a problem in >> the PM framework. >> >> The host bridge might be a special case because we don't have a >> separate "host bridge" driver; that code is kind of integrated with >> the controller drivers. So maybe it's OK to do controller + host >> bridge runtime PM support at the same time, as long as any time we add >> runtime PM to a controller, we sure it's also set up for the host >> bridge. >> >> Bjorn >
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index d5fc31f8345f..57756a73df30 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -16,6 +16,7 @@ #include <linux/of_pci.h> #include <linux/pci_regs.h> #include <linux/platform_device.h> +#include <linux/pm_runtime.h> #include "../../pci.h" #include "pcie-designware.h" @@ -505,6 +506,9 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) if (pp->ops->post_init) pp->ops->post_init(pp); + pm_runtime_set_active(&bridge->dev); + pm_runtime_enable(&bridge->dev); + return 0; err_stop_link:
The Controller driver is the parent device of the PCIe host bridge, PCI-PCI bridge and PCIe endpoint as shown below. PCIe controller(Top level parent & parent of host bridge) | v PCIe Host bridge(Parent of PCI-PCI bridge) | v PCI-PCI bridge(Parent of endpoint driver) | v PCIe endpoint driver Since runtime PM is disabled for host bridge, the state of the child devices under the host bridge is not taken into account by PM framework for the top level parent, PCIe controller. So PM framework, allows the controller driver to enter runtime PM irrespective of the state of the devices under the host bridge. And this causes the topology breakage and also possible PM issues. So enable pm runtime for the host bridge, so that controller driver goes to suspend only when all child devices goes to runtime suspend. Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com> --- Changes in v2: - Updated commit message as suggested by mani. - Link to v1: https://lore.kernel.org/r/20240219-runtime_pm_enable-v1-1-d39660310504@quicinc.com --- drivers/pci/controller/dwc/pcie-designware-host.c | 4 ++++ 1 file changed, 4 insertions(+) --- base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d change-id: 20240219-runtime_pm_enable-bdc17914bd50 Best regards,