[v8,5/8] PCI/portdrv: add mechanism to turn on subdev regulators

Adds a mechanism inside the root port device to identify standard PCIe
regulators in the DT, allocate them, and turn them on before the rest of
the bus is scanned during pci_host_probe().  A root complex driver can
leverage this mechanism by setting the pci_ops methods add_bus and
remove_bus to pci_subdev_regulators_{add,remove}_bus.

The allocated structure that contains the regulators is stored in
dev.driver_data.

The unabridged reason for doing this is as follows.  We would like the
Broadcom STB PCIe root complex driver (and others) to be able to turn
off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
the drivers of these endpoint devices are stock Linux drivers that are not
aware that these regulator(s) exist and must be turned on for the driver to
be probed.  The simple solution of course is to turn these regulators on at
boot and keep them on.  However, this solution does not satisfy at least
three of our usage modes:

1. For example, one customer uses multiple PCIe controllers, but wants the
ability to, by script invoking and unbind, turn any or all of them by and
their subdevices off to save power, e.g. when in battery mode.

2. Another example is when a watchdog script discovers that an endpoint
device is in an unresponsive state and would like to unbind, power toggle,
and re-bind just the PCIe endpoint and controller.

3. Of course we also want power turned off during suspend mode.  However,
some endpoint devices may be able to "wake" during suspend and we need to
recognise this case and veto the nominal act of turning off its regulator.
Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
end-point device needs to be kept powered on in order to receive network
packets and wake-up the system.

In all of these cases it is advantageous for the PCIe controller to govern
the turning off/on the regulators needed by the endpoint device.  The first
two cases can be done by simply unbinding and binding the PCIe controller,
if the controller has control of these regulators.

[1] These regulators typically govern the actual power supply to the
    endpoint chip.  Sometimes they may be a the official PCIe socket
    power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
    the regulator(s) that supply power to the EP chip.

[2] The 99% configuration of our boards is a single endpoint device
    attached to the PCIe controller.  I use the term endpoint but it could
    possible mean a switch as well.

Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
---
 drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h              |  8 ++++
 drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
 3 files changed, 112 insertions(+)

Hi Jim,

I love your patch! Yet something to improve:

[auto build test ERROR on 8e37395c3a5dceff62a5010ebbbc107f4145935c]

url:    https://github.com/0day-ci/linux/commits/Jim-Quinlan/PCI-brcmstb-have-portdrv-turn-on-sub-device-power/20211111-061940
base:   8e37395c3a5dceff62a5010ebbbc107f4145935c
config: ia64-defconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/9751f2d806d6efe0a7e389c3d64ad2fbd84b6be0
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Jim-Quinlan/PCI-brcmstb-have-portdrv-turn-on-sub-device-power/20211111-061940
        git checkout 9751f2d806d6efe0a7e389c3d64ad2fbd84b6be0
        # save the attached .config to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=ia64 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   ia64-linux-ld: drivers/pci/bus.o: in function `pci_subdev_regulators_remove_bus':
>> (.text+0x3f2): undefined reference to `pcie_is_port_dev'
   ia64-linux-ld: drivers/pci/bus.o: in function `pci_subdev_regulators_add_bus':
   (.text+0x442): undefined reference to `pcie_is_port_dev'

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

Capitalize "Add" in subject.

On Wed, Nov 10, 2021 at 05:14:45PM -0500, Jim Quinlan wrote:
> Adds a mechanism inside the root port device to identify standard PCIe
> regulators in the DT, allocate them, and turn them on before the rest of
> the bus is scanned during pci_host_probe().  A root complex driver can
> leverage this mechanism by setting the pci_ops methods add_bus and
> remove_bus to pci_subdev_regulators_{add,remove}_bus.

s/Adds a/Add a/

> The allocated structure that contains the regulators is stored in
> dev.driver_data.
> 
> The unabridged reason for doing this is as follows.  We would like the
> Broadcom STB PCIe root complex driver (and others) to be able to turn
> off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> the drivers of these endpoint devices are stock Linux drivers that are not
> aware that these regulator(s) exist and must be turned on for the driver to
> be probed.  The simple solution of course is to turn these regulators on at
> boot and keep them on.  However, this solution does not satisfy at least
> three of our usage modes:
> 
> 1. For example, one customer uses multiple PCIe controllers, but wants the
> ability to, by script invoking and unbind, turn any or all of them by and
> their subdevices off to save power, e.g. when in battery mode.
> 
> 2. Another example is when a watchdog script discovers that an endpoint
> device is in an unresponsive state and would like to unbind, power toggle,
> and re-bind just the PCIe endpoint and controller.
> 
> 3. Of course we also want power turned off during suspend mode.  However,
> some endpoint devices may be able to "wake" during suspend and we need to
> recognise this case and veto the nominal act of turning off its regulator.
> Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> end-point device needs to be kept powered on in order to receive network
> packets and wake-up the system.
> 
> In all of these cases it is advantageous for the PCIe controller to govern
> the turning off/on the regulators needed by the endpoint device.  The first
> two cases can be done by simply unbinding and binding the PCIe controller,
> if the controller has control of these regulators.
> 
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket
>     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
>     the regulator(s) that supply power to the EP chip.

s/may be a the/may be the/

> [2] The 99% configuration of our boards is a single endpoint device
>     attached to the PCIe controller.  I use the term endpoint but it could
>     possible mean a switch as well.

s/possible/possibly/

This adds generic code, so it needs some connection to the generic DT
binding for these things, e.g., a commit in this series that adds it
(I see commits that touch brcm,stb-pcie.yaml, but not a generic
place).

> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +	static const char * const supplies[] = {
> +		"vpcie3v3",
> +		"vpcie3v3aux",
> +		"vpcie12v",
> +	};
> +	const size_t size = sizeof(struct subdev_regulators)
> +		+ sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> +	struct subdev_regulators *sr;
> +	int i;
> +
> +	sr = devm_kzalloc(dev, size, GFP_KERNEL);
> +
> +	if (sr) {
> +		sr->num_supplies = ARRAY_SIZE(supplies);
> +		for (i = 0; i < ARRAY_SIZE(supplies); i++)
> +			sr->supplies[i].supply = supplies[i];
> +	}
> +
> +	return sr;
> +}
> +
> +

Spurious blank line.

> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> ...

> @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
>  	if (status)
>  		return status;
>  
> +	if (dev->bus->ops &&
> +	    dev->bus->ops->add_bus &&
> +	    dev->bus->dev.driver_data) {
> +		pcie_portdriver.resume = subdev_regulator_resume;
> +		pcie_portdriver.suspend = subdev_regulator_suspend;

The pci_driver.resume() and pci_driver.suspend() methods are going
away, so we shouldn't add new uses like this.

Doesn't this trigger the warning in pci_has_legacy_pm_support() about
supporting both legacy PM and the new PM?

Bjorn

On Thu, Nov 11, 2021 at 4:12 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> Capitalize "Add" in subject.
>
> On Wed, Nov 10, 2021 at 05:14:45PM -0500, Jim Quinlan wrote:
> > Adds a mechanism inside the root port device to identify standard PCIe
> > regulators in the DT, allocate them, and turn them on before the rest of
> > the bus is scanned during pci_host_probe().  A root complex driver can
> > leverage this mechanism by setting the pci_ops methods add_bus and
> > remove_bus to pci_subdev_regulators_{add,remove}_bus.
>
> s/Adds a/Add a/
>
> > The allocated structure that contains the regulators is stored in
> > dev.driver_data.
> >
> > The unabridged reason for doing this is as follows.  We would like the
> > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > the drivers of these endpoint devices are stock Linux drivers that are not
> > aware that these regulator(s) exist and must be turned on for the driver to
> > be probed.  The simple solution of course is to turn these regulators on at
> > boot and keep them on.  However, this solution does not satisfy at least
> > three of our usage modes:
> >
> > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > ability to, by script invoking and unbind, turn any or all of them by and
> > their subdevices off to save power, e.g. when in battery mode.
> >
> > 2. Another example is when a watchdog script discovers that an endpoint
> > device is in an unresponsive state and would like to unbind, power toggle,
> > and re-bind just the PCIe endpoint and controller.
> >
> > 3. Of course we also want power turned off during suspend mode.  However,
> > some endpoint devices may be able to "wake" during suspend and we need to
> > recognise this case and veto the nominal act of turning off its regulator.
> > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > end-point device needs to be kept powered on in order to receive network
> > packets and wake-up the system.
> >
> > In all of these cases it is advantageous for the PCIe controller to govern
> > the turning off/on the regulators needed by the endpoint device.  The first
> > two cases can be done by simply unbinding and binding the PCIe controller,
> > if the controller has control of these regulators.
> >
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
> >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> >     the regulator(s) that supply power to the EP chip.
>
> s/may be a the/may be the/
>
> > [2] The 99% configuration of our boards is a single endpoint device
> >     attached to the PCIe controller.  I use the term endpoint but it could
> >     possible mean a switch as well.
>
> s/possible/possibly/
>
> This adds generic code, so it needs some connection to the generic DT
> binding for these things, e.g., a commit in this series that adds it
> (I see commits that touch brcm,stb-pcie.yaml, but not a generic
> place).

That's pending here: https://github.com/devicetree-org/dt-schema/pull/63

Rob

On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
>
> Adds a mechanism inside the root port device to identify standard PCIe
> regulators in the DT, allocate them, and turn them on before the rest of
> the bus is scanned during pci_host_probe().  A root complex driver can
> leverage this mechanism by setting the pci_ops methods add_bus and
> remove_bus to pci_subdev_regulators_{add,remove}_bus.
>
> The allocated structure that contains the regulators is stored in
> dev.driver_data.
>
> The unabridged reason for doing this is as follows.  We would like the
> Broadcom STB PCIe root complex driver (and others) to be able to turn
> off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> the drivers of these endpoint devices are stock Linux drivers that are not
> aware that these regulator(s) exist and must be turned on for the driver to
> be probed.  The simple solution of course is to turn these regulators on at
> boot and keep them on.  However, this solution does not satisfy at least
> three of our usage modes:
>
> 1. For example, one customer uses multiple PCIe controllers, but wants the
> ability to, by script invoking and unbind, turn any or all of them by and
> their subdevices off to save power, e.g. when in battery mode.
>
> 2. Another example is when a watchdog script discovers that an endpoint
> device is in an unresponsive state and would like to unbind, power toggle,
> and re-bind just the PCIe endpoint and controller.
>
> 3. Of course we also want power turned off during suspend mode.  However,
> some endpoint devices may be able to "wake" during suspend and we need to
> recognise this case and veto the nominal act of turning off its regulator.
> Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> end-point device needs to be kept powered on in order to receive network
> packets and wake-up the system.
>
> In all of these cases it is advantageous for the PCIe controller to govern
> the turning off/on the regulators needed by the endpoint device.  The first
> two cases can be done by simply unbinding and binding the PCIe controller,
> if the controller has control of these regulators.
>
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket
>     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
>     the regulator(s) that supply power to the EP chip.
>
> [2] The 99% configuration of our boards is a single endpoint device
>     attached to the PCIe controller.  I use the term endpoint but it could
>     possible mean a switch as well.
>
> Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> ---
>  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
>  drivers/pci/pci.h              |  8 ++++
>  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
>  3 files changed, 112 insertions(+)
>
> diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> index 3cef835b375f..c39fdf36b0ad 100644
> --- a/drivers/pci/bus.c
> +++ b/drivers/pci/bus.c
> @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
>         if (bus)
>                 put_device(&bus->dev);
>  }
> +
> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +       static const char * const supplies[] = {
> +               "vpcie3v3",
> +               "vpcie3v3aux",
> +               "vpcie12v",
> +       };
> +       const size_t size = sizeof(struct subdev_regulators)
> +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> +       struct subdev_regulators *sr;
> +       int i;
> +
> +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> +
> +       if (sr) {
> +               sr->num_supplies = ARRAY_SIZE(supplies);
> +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> +                       sr->supplies[i].supply = supplies[i];
> +       }
> +
> +       return sr;
> +}
> +
> +
> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> +{
> +       struct device *dev = &bus->dev;
> +       struct subdev_regulators *sr;
> +       int ret;
> +
> +       if (!pcie_is_port_dev(bus->self))
> +               return 0;
> +
> +       if (WARN_ON(bus->dev.driver_data))
> +               dev_err(dev, "multiple clients using dev.driver_data\n");
> +
> +       sr = alloc_subdev_regulators(&bus->dev);
> +       if (!sr)
> +               return -ENOMEM;
> +
> +       bus->dev.driver_data = sr;
> +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> +       if (ret)
> +               return ret;
> +
> +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> +       if (ret) {
> +               dev_err(dev, "failed to enable regulators for downstream device\n");
> +               return ret;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);

Can't these just go in the portdrv probe and remove functions now?

Rob

Hi Jim,

[...]
> [1] These regulators typically govern the actual power supply to the
>     endpoint chip.  Sometimes they may be a the official PCIe socket

In the above, did you mean to say "be at the"?

> +static void *alloc_subdev_regulators(struct device *dev)
> +{
> +	static const char * const supplies[] = {
> +		"vpcie3v3",
> +		"vpcie3v3aux",
> +		"vpcie12v",
> +	};
> +	const size_t size = sizeof(struct subdev_regulators)
> +		+ sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);

[...]
> +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> +{
> +	struct device *dev = &bus->dev;
> +	struct subdev_regulators *sr;
> +	int ret;
> +
> +	if (!pcie_is_port_dev(bus->self))
> +		return 0;
> +
> +	if (WARN_ON(bus->dev.driver_data))
> +		dev_err(dev, "multiple clients using dev.driver_data\n");

I have to ask - is the WARN_ON() above adding value given the nature of the
error?  Would dumping a stack be of interest to someone?

Having said that, why do we even need to assert this?  Can there be some
sort of a race condition with access happening here?

I am asking as pci_subdev_regulators_remove_bus() does not seem to be
concerned about this sort of thing yet it also accesses the same driver
data, and such.

[...]
> +/* forward declaration */
> +static struct pci_driver pcie_portdriver;

The comment above might not be needed as it's quite obvious what the code
at this line is for, I believe.

[...]
> @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
>  	if (status)
>  		return status;
>  
> +	if (dev->bus->ops &&
> +	    dev->bus->ops->add_bus &&
> +	    dev->bus->dev.driver_data) {
> +		pcie_portdriver.resume = subdev_regulator_resume;
> +		pcie_portdriver.suspend = subdev_regulator_suspend;
> +	}
> +
>  	pci_save_state(dev);

[...]
> @@ -237,6 +268,7 @@ static struct pci_driver pcie_portdriver = {
>  	.err_handler	= &pcie_portdrv_err_handler,
>  
>  	.driver.pm	= PCIE_PORTDRV_PM_OPS,
> +	/* Note: suspend and resume may be set during probe */

This comment here is for the "driver.pm" line above, correct?  If so, then
I would move it above the statement.  It's a little bit confusing
otherwise.

	Krzysztof

On Thu, Nov 11, 2021 at 6:38 PM Krzysztof Wilczyński <kw@linux.com> wrote:
>
> Hi Jim,
>
> [...]
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
>
> In the above, did you mean to say "be at the"?
Yep.
>
> > +static void *alloc_subdev_regulators(struct device *dev)
> > +{
> > +     static const char * const supplies[] = {
> > +             "vpcie3v3",
> > +             "vpcie3v3aux",
> > +             "vpcie12v",
> > +     };
> > +     const size_t size = sizeof(struct subdev_regulators)
> > +             + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
>
> [...]
> > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > +{
> > +     struct device *dev = &bus->dev;
> > +     struct subdev_regulators *sr;
> > +     int ret;
> > +
> > +     if (!pcie_is_port_dev(bus->self))
> > +             return 0;
> > +
> > +     if (WARN_ON(bus->dev.driver_data))
> > +             dev_err(dev, "multiple clients using dev.driver_data\n");
>
> I have to ask - is the WARN_ON() above adding value given the nature of the
> error?  Would dumping a stack be of interest to someone?
Hello Krzysztof,

It doesn't need to be a warning.  You are right, the backtrace will
not help anyone figure out how to fix the problem.

>
> Having said that, why do we even need to assert this?  Can there be some
> sort of a race condition with access happening here?
This commit-set is claiming the driver_data field of the PCIe port
device and I am concerned  that something else in the future would
unknowingly do the same.  It would not be a race, just two separate
pieces of code stomping on the same variable.  If I am over-worrying I
can use a dev_err or nothing at all.


>
> I am asking as pci_subdev_regulators_remove_bus() does not seem to be
> concerned about this sort of thing yet it also accesses the same driver
> data, and such.
Yes, but when pci_subdev_regulators_remove_bus() accesses the port
driver driver_data and it is non-NULL  it  does not  know whether it
is the expected pointer or something else.

>
> [...]
> > +/* forward declaration */
> > +static struct pci_driver pcie_portdriver;
>
> The comment above might not be needed as it's quite obvious what the code
> at this line is for, I believe.
Okay.
>
> [...]
> > @@ -131,6 +155,13 @@ static int pcie_portdrv_probe(struct pci_dev *dev,
> >       if (status)
> >               return status;
> >
> > +     if (dev->bus->ops &&
> > +         dev->bus->ops->add_bus &&
> > +         dev->bus->dev.driver_data) {
> > +             pcie_portdriver.resume = subdev_regulator_resume;
> > +             pcie_portdriver.suspend = subdev_regulator_suspend;
> > +     }
> > +
> >       pci_save_state(dev);
>
> [...]
> > @@ -237,6 +268,7 @@ static struct pci_driver pcie_portdriver = {
> >       .err_handler    = &pcie_portdrv_err_handler,
> >
> >       .driver.pm      = PCIE_PORTDRV_PM_OPS,
> > +     /* Note: suspend and resume may be set during probe */
>
> This comment here is for the "driver.pm" line above, correct?  If so, then
> I would move it above the statement.  It's a little bit confusing
> otherwise.
I'm planning to remove this comment and the code that sets
pcie_portdriver.{resume,suspend} and instead put this code into the
int pcie_port_device_{suspend,remove}() functions.

Regards,
Jim Quinlan
Broadcom STB

>
>         Krzysztof
>

On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
>
> On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> >
> > Adds a mechanism inside the root port device to identify standard PCIe
> > regulators in the DT, allocate them, and turn them on before the rest of
> > the bus is scanned during pci_host_probe().  A root complex driver can
> > leverage this mechanism by setting the pci_ops methods add_bus and
> > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> >
> > The allocated structure that contains the regulators is stored in
> > dev.driver_data.
> >
> > The unabridged reason for doing this is as follows.  We would like the
> > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > the drivers of these endpoint devices are stock Linux drivers that are not
> > aware that these regulator(s) exist and must be turned on for the driver to
> > be probed.  The simple solution of course is to turn these regulators on at
> > boot and keep them on.  However, this solution does not satisfy at least
> > three of our usage modes:
> >
> > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > ability to, by script invoking and unbind, turn any or all of them by and
> > their subdevices off to save power, e.g. when in battery mode.
> >
> > 2. Another example is when a watchdog script discovers that an endpoint
> > device is in an unresponsive state and would like to unbind, power toggle,
> > and re-bind just the PCIe endpoint and controller.
> >
> > 3. Of course we also want power turned off during suspend mode.  However,
> > some endpoint devices may be able to "wake" during suspend and we need to
> > recognise this case and veto the nominal act of turning off its regulator.
> > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > end-point device needs to be kept powered on in order to receive network
> > packets and wake-up the system.
> >
> > In all of these cases it is advantageous for the PCIe controller to govern
> > the turning off/on the regulators needed by the endpoint device.  The first
> > two cases can be done by simply unbinding and binding the PCIe controller,
> > if the controller has control of these regulators.
> >
> > [1] These regulators typically govern the actual power supply to the
> >     endpoint chip.  Sometimes they may be a the official PCIe socket
> >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> >     the regulator(s) that supply power to the EP chip.
> >
> > [2] The 99% configuration of our boards is a single endpoint device
> >     attached to the PCIe controller.  I use the term endpoint but it could
> >     possible mean a switch as well.
> >
> > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > ---
> >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> >  drivers/pci/pci.h              |  8 ++++
> >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> >  3 files changed, 112 insertions(+)
> >
> > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > index 3cef835b375f..c39fdf36b0ad 100644
> > --- a/drivers/pci/bus.c
> > +++ b/drivers/pci/bus.c
> > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> >         if (bus)
> >                 put_device(&bus->dev);
> >  }
> > +
> > +static void *alloc_subdev_regulators(struct device *dev)
> > +{
> > +       static const char * const supplies[] = {
> > +               "vpcie3v3",
> > +               "vpcie3v3aux",
> > +               "vpcie12v",
> > +       };
> > +       const size_t size = sizeof(struct subdev_regulators)
> > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > +       struct subdev_regulators *sr;
> > +       int i;
> > +
> > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > +
> > +       if (sr) {
> > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > +                       sr->supplies[i].supply = supplies[i];
> > +       }
> > +
> > +       return sr;
> > +}
> > +
> > +
> > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > +{
> > +       struct device *dev = &bus->dev;
> > +       struct subdev_regulators *sr;
> > +       int ret;
> > +
> > +       if (!pcie_is_port_dev(bus->self))
> > +               return 0;
> > +
> > +       if (WARN_ON(bus->dev.driver_data))
> > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > +
> > +       sr = alloc_subdev_regulators(&bus->dev);
> > +       if (!sr)
> > +               return -ENOMEM;
> > +
> > +       bus->dev.driver_data = sr;
> > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > +       if (ret) {
> > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > +               return ret;
> > +       }
> > +
> > +       return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
>
> Can't these just go in the portdrv probe and remove functions now?
>
> Rob

Not really.  The idea is that  only when a host controller driver does this

static struct pci_ops my_pcie_ops = {
    .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
    .remove_bus = pci_subdev_regulators_remove_bus,
    ...
}

does it explicitly want this feature.  Without doing this, every PCI
port in the world will execute a devm_kzalloc() and
devm_regulator_bulk_get() to (likely) grab nothing, and then there
will be three superfluous lines in the boot log:

pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator

Secondly, our  HW needs to know when the  alloc/get/enable of
regulators is done so that the PCIe link can then be attempted.   This
is pretty much the cornerstone of this patchset.   To do this the brcm
RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
brcm_pcie_add_bus() so that we can do this:

static struct pci_ops my_pcie_ops = {
    .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
    .remove_bus = pci_subdev_regulators_remove_bus,
    ...
}

Regards,
Jim Quinlan
Broadcom STB

+Pali

On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
>
> On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > >
> > > Adds a mechanism inside the root port device to identify standard PCIe
> > > regulators in the DT, allocate them, and turn them on before the rest of
> > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > >
> > > The allocated structure that contains the regulators is stored in
> > > dev.driver_data.
> > >
> > > The unabridged reason for doing this is as follows.  We would like the
> > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > aware that these regulator(s) exist and must be turned on for the driver to
> > > be probed.  The simple solution of course is to turn these regulators on at
> > > boot and keep them on.  However, this solution does not satisfy at least
> > > three of our usage modes:
> > >
> > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > ability to, by script invoking and unbind, turn any or all of them by and
> > > their subdevices off to save power, e.g. when in battery mode.
> > >
> > > 2. Another example is when a watchdog script discovers that an endpoint
> > > device is in an unresponsive state and would like to unbind, power toggle,
> > > and re-bind just the PCIe endpoint and controller.
> > >
> > > 3. Of course we also want power turned off during suspend mode.  However,
> > > some endpoint devices may be able to "wake" during suspend and we need to
> > > recognise this case and veto the nominal act of turning off its regulator.
> > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > end-point device needs to be kept powered on in order to receive network
> > > packets and wake-up the system.
> > >
> > > In all of these cases it is advantageous for the PCIe controller to govern
> > > the turning off/on the regulators needed by the endpoint device.  The first
> > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > if the controller has control of these regulators.
> > >
> > > [1] These regulators typically govern the actual power supply to the
> > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > >     the regulator(s) that supply power to the EP chip.
> > >
> > > [2] The 99% configuration of our boards is a single endpoint device
> > >     attached to the PCIe controller.  I use the term endpoint but it could
> > >     possible mean a switch as well.
> > >
> > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > ---
> > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > >  drivers/pci/pci.h              |  8 ++++
> > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > >  3 files changed, 112 insertions(+)
> > >
> > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > index 3cef835b375f..c39fdf36b0ad 100644
> > > --- a/drivers/pci/bus.c
> > > +++ b/drivers/pci/bus.c
> > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > >         if (bus)
> > >                 put_device(&bus->dev);
> > >  }
> > > +
> > > +static void *alloc_subdev_regulators(struct device *dev)
> > > +{
> > > +       static const char * const supplies[] = {
> > > +               "vpcie3v3",
> > > +               "vpcie3v3aux",
> > > +               "vpcie12v",
> > > +       };
> > > +       const size_t size = sizeof(struct subdev_regulators)
> > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > +       struct subdev_regulators *sr;
> > > +       int i;
> > > +
> > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > +
> > > +       if (sr) {
> > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > +                       sr->supplies[i].supply = supplies[i];
> > > +       }
> > > +
> > > +       return sr;
> > > +}
> > > +
> > > +
> > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > +{
> > > +       struct device *dev = &bus->dev;
> > > +       struct subdev_regulators *sr;
> > > +       int ret;
> > > +
> > > +       if (!pcie_is_port_dev(bus->self))
> > > +               return 0;
> > > +
> > > +       if (WARN_ON(bus->dev.driver_data))
> > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > +
> > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > +       if (!sr)
> > > +               return -ENOMEM;
> > > +
> > > +       bus->dev.driver_data = sr;
> > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > +       if (ret)
> > > +               return ret;
> > > +
> > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > +       if (ret) {
> > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > +               return ret;
> > > +       }
> > > +
> > > +       return 0;
> > > +}
> > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> >
> > Can't these just go in the portdrv probe and remove functions now?
> >
> > Rob
>
> Not really.  The idea is that  only when a host controller driver does this
>
> static struct pci_ops my_pcie_ops = {
>     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
>     .remove_bus = pci_subdev_regulators_remove_bus,
>     ...
> }
>
> does it explicitly want this feature.  Without doing this, every PCI
> port in the world will execute a devm_kzalloc() and
> devm_regulator_bulk_get() to (likely) grab nothing, and then there
> will be three superfluous lines in the boot log:

You can opt-in based on there being a DT node.

> pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator

This would be annoying, but not really a reason for how to design this.

> Secondly, our  HW needs to know when the  alloc/get/enable of
> regulators is done so that the PCIe link can then be attempted.   This
> is pretty much the cornerstone of this patchset.   To do this the brcm
> RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> brcm_pcie_add_bus() so that we can do this:
>
> static struct pci_ops my_pcie_ops = {
>     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
>     .remove_bus = pci_subdev_regulators_remove_bus,

Do add_bus/remove_bus get called during resume/suspend? If not, how do
you handle the link during resume?

Maybe there needs to be explicit hooks for link handling. Pali has
been looking into this some.

Rob

On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> +Pali
> 
> On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> >
> > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > >
> > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > >
> > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > >
> > > > The allocated structure that contains the regulators is stored in
> > > > dev.driver_data.
> > > >
> > > > The unabridged reason for doing this is as follows.  We would like the
> > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > three of our usage modes:
> > > >
> > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > their subdevices off to save power, e.g. when in battery mode.
> > > >
> > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > and re-bind just the PCIe endpoint and controller.
> > > >
> > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > end-point device needs to be kept powered on in order to receive network
> > > > packets and wake-up the system.
> > > >
> > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > if the controller has control of these regulators.
> > > >
> > > > [1] These regulators typically govern the actual power supply to the
> > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > >     the regulator(s) that supply power to the EP chip.
> > > >
> > > > [2] The 99% configuration of our boards is a single endpoint device
> > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > >     possible mean a switch as well.
> > > >
> > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > ---
> > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > >  drivers/pci/pci.h              |  8 ++++
> > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > >  3 files changed, 112 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > --- a/drivers/pci/bus.c
> > > > +++ b/drivers/pci/bus.c
> > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > >         if (bus)
> > > >                 put_device(&bus->dev);
> > > >  }
> > > > +
> > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > +{
> > > > +       static const char * const supplies[] = {
> > > > +               "vpcie3v3",
> > > > +               "vpcie3v3aux",
> > > > +               "vpcie12v",
> > > > +       };
> > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > +       struct subdev_regulators *sr;
> > > > +       int i;
> > > > +
> > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > +
> > > > +       if (sr) {
> > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > +                       sr->supplies[i].supply = supplies[i];
> > > > +       }
> > > > +
> > > > +       return sr;
> > > > +}
> > > > +
> > > > +
> > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > +{
> > > > +       struct device *dev = &bus->dev;
> > > > +       struct subdev_regulators *sr;
> > > > +       int ret;
> > > > +
> > > > +       if (!pcie_is_port_dev(bus->self))
> > > > +               return 0;
> > > > +
> > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > +
> > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > +       if (!sr)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       bus->dev.driver_data = sr;
> > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > +               return ret;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > >
> > > Can't these just go in the portdrv probe and remove functions now?
> > >
> > > Rob
> >
> > Not really.  The idea is that  only when a host controller driver does this
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> >     ...
> > }
> >
> > does it explicitly want this feature.  Without doing this, every PCI
> > port in the world will execute a devm_kzalloc() and
> > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > will be three superfluous lines in the boot log:
> 
> You can opt-in based on there being a DT node.
> 
> > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> 
> This would be annoying, but not really a reason for how to design this.
> 
> > Secondly, our  HW needs to know when the  alloc/get/enable of
> > regulators is done so that the PCIe link can then be attempted.   This
> > is pretty much the cornerstone of this patchset.   To do this the brcm
> > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > brcm_pcie_add_bus() so that we can do this:
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> 
> Do add_bus/remove_bus get called during resume/suspend? If not, how do
> you handle the link during resume?
> 
> Maybe there needs to be explicit hooks for link handling. Pali has
> been looking into this some.
> 
> Rob

Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
needs to be supplied at the "correct" time during establishing link
procedure. I wrote it in my RFC email:
https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/

I'm not sure if regulator API is the most suitable for this task in PCI
core code as there are planty ways how it can be controllers. My idea
presented in that email was that driver provides power callback and core
pci code would use it.

Because power needs to be enabled at the "correct" time during link up,
I think that add/remove bus callbacks are unsuitable for this task. This
would just cause adding another msleep() calls on different places to
make correct timing of link up...

I think it is needed to implement generic function for establishing link
in pci core code with all required steps.

On Tue, Nov 16, 2021 at 12:41 PM Rob Herring <robh@kernel.org> wrote:
>
> +Pali
>
> On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> >
> > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > >
> > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > >
> > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > >
> > > > The allocated structure that contains the regulators is stored in
> > > > dev.driver_data.
> > > >
> > > > The unabridged reason for doing this is as follows.  We would like the
> > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > three of our usage modes:drivers/pci/pcie/aspm.cdrivers/pci/pcie/aspm.c
> > > >
> > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > their subdevices off to save power, e.g. when in battery mode.
> > > >
> > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > and re-bind just the PCIe endpoint and controller.
> > > >
> > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > end-point device needs to be kept powered on in order to receive network
> > > > packets and wake-up the system.
> > > >
> > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > if the controller has control of these regulators.
> > > >
> > > > [1] These regulators typically govern the actual power supply to the
> > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > >     the regulator(s) that supply power to the EP chip.
> > > >
> > > > [2] The 99% configuration of our boards is a single endpoint device
> > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > >     possible mean a switch as well.
> > > >
> > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > ---
> > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > >  drivers/pci/pci.h              |  8 ++++
> > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > >  3 files changed, 112 insertions(+)
> > > >
> > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > --- a/drivers/pci/bus.c
> > > > +++ b/drivers/pci/bus.c
> > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > >         if (bus)
> > > >                 put_device(&bus->dev);
> > > >  }
> > > > +
> > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > +{
> > > > +       static const char * const supplies[] = {
> > > > +               "vpcie3v3",
> > > > +               "vpcie3v3aux",
> > > > +               "vpcie12v",
> > > > +       };
> > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > +       struct subdev_regulators *sr;
> > > > +       int i;
> > > > +
> > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > +
> > > > +       if (sr) {
> > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > +                       sr->supplies[i].supply = supplies[i];
> > > > +       }
> > > > +
> > > > +       return sr;
> > > > +}
> > > > +
> > > > +
> > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > +{
> > > > +       struct device *dev = &bus->dev;
> > > > +       struct subdev_regulators *sr;
> > > > +       int ret;
> > > > +
> > > > +       if (!pcie_is_port_dev(bus->self))
> > > > +               return 0;
> > > > +
> > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > +
> > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > +       if (!sr)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       bus->dev.driver_data = sr;
> > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > +       if (ret)
> > > > +               return ret;
> > > > +
> > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > +       if (ret) {
> > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > +               return ret;
> > > > +       }
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > >
> > > Can't these just go in the portdrv probe and remove functions now?
> > >
> > > Rob
> >
> > Not really.  The idea is that  only when a host controller driver does this
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
> >     ...
> > }
> >
> > does it explicitly want this feature.  Without doing this, every PCI
> > port in the world will execute a devm_kzalloc() and
> > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > will be three superfluous lines in the boot log:
>
> You can opt-in based on there being a DT node.
Will do, thanks.
>
> > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
>
> This would be annoying, but not really a reason for how to design this.
>
> > Secondly, our  HW needs to know when the  alloc/get/enable of
> > regulators is done so that the PCIe link can then be attempted.   This
> > is pretty much the cornerstone of this patchset.   To do this the brcm
> > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > brcm_pcie_add_bus() so that we can do this:
> >
> > static struct pci_ops my_pcie_ops = {
> >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> >     .remove_bus = pci_subdev_regulators_remove_bus,
>
> Do add_bus/remove_bus get called during resume/suspend? If not, how do
> you handle the link during resume?
No to the first question, and incorrectly to the second.  As it turns
out, my suspend/resume test for this patchset was flawed and wasn't
testing what I thought it was.  The suspend/resume is broken in V8.

Using pcie_port_device_resume_noirq() is not early enough to turn on
the regulators during resume.  However,  if I have my RC driver handle
 only the suspend/resume of the regulators, all is fine and as an
added benefit the patchset has no modifications to
portdrv_{pci,core}.c.  This is what I will submit for V9, assuming
there is one.

>
> Maybe there needs to be explicit hooks for link handling. Pali has
> been looking into this some.
I will reply to Pali's recent email now.
Regards,
Jim Quinlan
Broadcom STB
>
> Rob

On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > +Pali
> >
> > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > >
> > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > >
> > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > >
> > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > >
> > > > > The allocated structure that contains the regulators is stored in
> > > > > dev.driver_data.
> > > > >
> > > > > The unabridged reason for doing this is as follows.  We would like the
> > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > three of our usage modes:
> > > > >
> > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > >
> > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > and re-bind just the PCIe endpoint and controller.
> > > > >
> > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > end-point device needs to be kept powered on in order to receive network
> > > > > packets and wake-up the system.
> > > > >
> > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > if the controller has control of these regulators.
> > > > >
> > > > > [1] These regulators typically govern the actual power supply to the
> > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > >     the regulator(s) that supply power to the EP chip.
> > > > >
> > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > >     possible mean a switch as well.
> > > > >
> > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > ---
> > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > >  drivers/pci/pci.h              |  8 ++++
> > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > >  3 files changed, 112 insertions(+)
> > > > >
> > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > --- a/drivers/pci/bus.c
> > > > > +++ b/drivers/pci/bus.c
> > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > >         if (bus)
> > > > >                 put_device(&bus->dev);
> > > > >  }
> > > > > +
> > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > +{
> > > > > +       static const char * const supplies[] = {
> > > > > +               "vpcie3v3",
> > > > > +               "vpcie3v3aux",
> > > > > +               "vpcie12v",
> > > > > +       };
> > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > +       struct subdev_regulators *sr;
> > > > > +       int i;
> > > > > +
> > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > +
> > > > > +       if (sr) {
> > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > +       }
> > > > > +
> > > > > +       return sr;
> > > > > +}
> > > > > +
> > > > > +
> > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > +{
> > > > > +       struct device *dev = &bus->dev;
> > > > > +       struct subdev_regulators *sr;
> > > > > +       int ret;
> > > > > +
> > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > +               return 0;
> > > > > +
> > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > +
> > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > +       if (!sr)
> > > > > +               return -ENOMEM;
> > > > > +
> > > > > +       bus->dev.driver_data = sr;
> > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > +       if (ret)
> > > > > +               return ret;
> > > > > +
> > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > +       if (ret) {
> > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > +               return ret;
> > > > > +       }
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > >
> > > > Can't these just go in the portdrv probe and remove functions now?
> > > >
> > > > Rob
> > >
> > > Not really.  The idea is that  only when a host controller driver does this
> > >
> > > static struct pci_ops my_pcie_ops = {
> > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > >     ...
> > > }
> > >
> > > does it explicitly want this feature.  Without doing this, every PCI
> > > port in the world will execute a devm_kzalloc() and
> > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > will be three superfluous lines in the boot log:
> >
> > You can opt-in based on there being a DT node.
> >
> > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> >
> > This would be annoying, but not really a reason for how to design this.
> >
> > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > regulators is done so that the PCIe link can then be attempted.   This
> > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > brcm_pcie_add_bus() so that we can do this:
> > >
> > > static struct pci_ops my_pcie_ops = {
> > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > >     .remove_bus = pci_subdev_regulators_remove_bus,
> >
> > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > you handle the link during resume?
> >
> > Maybe there needs to be explicit hooks for link handling. Pali has
> > been looking into this some.
> >
> > Rob
>
> Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> needs to be supplied at the "correct" time during establishing link
> procedure. I wrote it in my RFC email:
> https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
Hello Pali,

I really like your proposal although I would like to get my patchset
first :-) :-)

Suppose you came up with a patchset for your ideas-- would that include
changes to existing RC drivers to use the proposed framework?  If so,
I am wary that it would
break at least a few of them.  Or would you just present the framework
and allow the
RC drivers' authors to opt-in, one by one?

At any rate, if you want someone to test some of your ideas I can work
with you.

Regards,
Jim Quinlan
Broadcom STB


>
> I'm not sure if regulator API is the most suitable for this task in PCI
> core code as there are planty ways how it can be controllers. My idea
> presented in that email was that driver provides power callback and core
> pci code would use it.
>
> Because power needs to be enabled at the "correct" time during link up,
> I think that add/remove bus callbacks are unsuitable for this task. This
> would just cause adding another msleep() calls on different places to
> make correct timing of link up...
>
> I think it is needed to implement generic function for establishing link
> in pci core code with all required steps.

On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > > +Pali
> > >
> > > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > > >
> > > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > > >
> > > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > > >
> > > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > > >
> > > > > > The allocated structure that contains the regulators is stored in
> > > > > > dev.driver_data.
> > > > > >
> > > > > > The unabridged reason for doing this is as follows.  We would like the
> > > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > > three of our usage modes:
> > > > > >
> > > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > > >
> > > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > > and re-bind just the PCIe endpoint and controller.
> > > > > >
> > > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > > end-point device needs to be kept powered on in order to receive network
> > > > > > packets and wake-up the system.
> > > > > >
> > > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > > if the controller has control of these regulators.
> > > > > >
> > > > > > [1] These regulators typically govern the actual power supply to the
> > > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > > >     the regulator(s) that supply power to the EP chip.
> > > > > >
> > > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > > >     possible mean a switch as well.
> > > > > >
> > > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > > ---
> > > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > > >  drivers/pci/pci.h              |  8 ++++
> > > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > > >  3 files changed, 112 insertions(+)
> > > > > >
> > > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > > --- a/drivers/pci/bus.c
> > > > > > +++ b/drivers/pci/bus.c
> > > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > > >         if (bus)
> > > > > >                 put_device(&bus->dev);
> > > > > >  }
> > > > > > +
> > > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > > +{
> > > > > > +       static const char * const supplies[] = {
> > > > > > +               "vpcie3v3",
> > > > > > +               "vpcie3v3aux",
> > > > > > +               "vpcie12v",
> > > > > > +       };
> > > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > > +       struct subdev_regulators *sr;
> > > > > > +       int i;
> > > > > > +
> > > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > > +
> > > > > > +       if (sr) {
> > > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > > +       }
> > > > > > +
> > > > > > +       return sr;
> > > > > > +}
> > > > > > +
> > > > > > +
> > > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > > +{
> > > > > > +       struct device *dev = &bus->dev;
> > > > > > +       struct subdev_regulators *sr;
> > > > > > +       int ret;
> > > > > > +
> > > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > > +               return 0;
> > > > > > +
> > > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > > +
> > > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > > +       if (!sr)
> > > > > > +               return -ENOMEM;
> > > > > > +
> > > > > > +       bus->dev.driver_data = sr;
> > > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > > +       if (ret)
> > > > > > +               return ret;
> > > > > > +
> > > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > > +       if (ret) {
> > > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > > +               return ret;
> > > > > > +       }
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > > >
> > > > > Can't these just go in the portdrv probe and remove functions now?
> > > > >
> > > > > Rob
> > > >
> > > > Not really.  The idea is that  only when a host controller driver does this
> > > >
> > > > static struct pci_ops my_pcie_ops = {
> > > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > >     ...
> > > > }
> > > >
> > > > does it explicitly want this feature.  Without doing this, every PCI
> > > > port in the world will execute a devm_kzalloc() and
> > > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > > will be three superfluous lines in the boot log:
> > >
> > > You can opt-in based on there being a DT node.
> > >
> > > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> > >
> > > This would be annoying, but not really a reason for how to design this.
> > >
> > > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > > regulators is done so that the PCIe link can then be attempted.   This
> > > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > > brcm_pcie_add_bus() so that we can do this:
> > > >
> > > > static struct pci_ops my_pcie_ops = {
> > > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > >
> > > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > > you handle the link during resume?
> > >
> > > Maybe there needs to be explicit hooks for link handling. Pali has
> > > been looking into this some.
> > >
> > > Rob
> >
> > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > needs to be supplied at the "correct" time during establishing link
> > procedure. I wrote it in my RFC email:
> > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> Hello Pali,
> 
> I really like your proposal although I would like to get my patchset
> first :-) :-)
> 
> Suppose you came up with a patchset for your ideas-- would that include
> changes to existing RC drivers to use the proposed framework?  If so,
> I am wary that it would
> break at least a few of them.  Or would you just present the framework
> and allow the
> RC drivers' authors to opt-in, one by one?

My idea is to add new "framework" to allow drivers implement new
callbacks for this "framework". There would be no change in drivers
which do not provide these callbacks to ensure that nothing is going to
be broken. I'm planning to implement these callbacks only for RC drivers
for which I have hardware and can properly test to not introduce any
regression. For other existing RC drivers it is up to other authors +
testers. But to decrease future maintenance cost of all RC drivers I
expect that new drivers would not implement any ad-hoc solution in their
"probe" function and instead implement these new callbacks. That is my
idea.

> At any rate, if you want someone to test some of your ideas I can work
> with you.

Perfect! If you have any concerns or you see any issues, please reply
that my RFC email. So I can collect feedback.

Also I sent draft for updating DTS schema for PCIe devices:
https://github.com/devicetree-org/dt-schema/pull/64

> Regards,
> Jim Quinlan
> Broadcom STB
> 
> 
> >
> > I'm not sure if regulator API is the most suitable for this task in PCI
> > core code as there are planty ways how it can be controllers. My idea
> > presented in that email was that driver provides power callback and core
> > pci code would use it.
> >
> > Because power needs to be enabled at the "correct" time during link up,
> > I think that add/remove bus callbacks are unsuitable for this task. This
> > would just cause adding another msleep() calls on different places to
> > make correct timing of link up...
> >
> > I think it is needed to implement generic function for establishing link
> > in pci core code with all required steps.

On Wed, Nov 17, 2021 at 10:45 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> > On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Tuesday 16 November 2021 11:41:22 Rob Herring wrote:
> > > > +Pali
> > > >
> > > > On Mon, Nov 15, 2021 at 2:44 PM Jim Quinlan <james.quinlan@broadcom.com> wrote:
> > > > >
> > > > > On Thu, Nov 11, 2021 at 5:57 PM Rob Herring <robh@kernel.org> wrote:
> > > > > >
> > > > > > On Wed, Nov 10, 2021 at 4:15 PM Jim Quinlan <jim2101024@gmail.com> wrote:
> > > > > > >
> > > > > > > Adds a mechanism inside the root port device to identify standard PCIe
> > > > > > > regulators in the DT, allocate them, and turn them on before the rest of
> > > > > > > the bus is scanned during pci_host_probe().  A root complex driver can
> > > > > > > leverage this mechanism by setting the pci_ops methods add_bus and
> > > > > > > remove_bus to pci_subdev_regulators_{add,remove}_bus.
> > > > > > >
> > > > > > > The allocated structure that contains the regulators is stored in
> > > > > > > dev.driver_data.
> > > > > > >
> > > > > > > The unabridge    /* PCIe endpoint */d reason for doing this is as follows.  We would like the
> > > > > > > Broadcom STB PCIe root complex driver (and others) to be able to turn
> > > > > > > off/on regulators[1] that provide power to endpoint[2] devices.  Typically,
> > > > > > > the drivers of these endpoint devices are stock Linux drivers that are not
> > > > > > > aware that these regulator(s) exist and must be turned on for the driver to
> > > > > > > be probed.  The simple solution of course is to turn these regulators on at
> > > > > > > boot and keep them on.  However, this solution does not satisfy at least
> > > > > > > three of our usage modes:
> > > > > > >
> > > > > > > 1. For example, one customer uses multiple PCIe controllers, but wants the
> > > > > > > ability to, by script invoking and unbind, turn any or all of them by and
> > > > > > > their subdevices off to save power, e.g. when in battery mode.
> > > > > > >
> > > > > > > 2. Another example is when a watchdog script discovers that an endpoint
> > > > > > > device is in an unresponsive state and would like to unbind, power toggle,
> > > > > > > and re-bind just the PCIe endpoint and controller.
> > > > > > >
> > > > > > > 3. Of course we also want power turned off during suspend mode.  However,
> > > > > > > some endpoint devices may be able to "wake" during suspend and we need to
> > > > > > > recognise this case and veto the nominal act of turning off its regulator.
> > > > > > > Such is the case with Wake-on-LAN and Wake-on-WLAN support where PCIe
> > > > > > > end-point device needs to be kept powered on in order to receive network
> > > > > > > packets and wake-up the system.
> > > > > > >
> > > > > > > In all of these cases it is advantageous for the PCIe controller to govern
> > > > > > > the turning off/on the regulators needed by the endpoint device.  The first
> > > > > > > two cases can be done by simply unbinding and binding the PCIe controller,
> > > > > > > if the controller has control of these regulators.
> > > > > > >
> > > > > > > [1] These regulators typically govern the actual power supply to the
> > > > > > >     endpoint chip.  Sometimes they may be a the official PCIe socket
> > > > > > >     power -- such as 3.3v or aux-3.3v.  Sometimes they are truly
> > > > > > >     the regulator(s) that supply power to the EP chip.
> > > > > > >
> > > > > > > [2] The 99% configuration of our boards is a single endpoint device
> > > > > > >     attached to the PCIe controller.  I use the term endpoint but it could
> > > > > > >     possible mean a switch as well.
> > > > > > >
> > > > > > > Signed-off-by: Jim Quinlan <jim2101024@gmail.com>
> > > > > > > ---
> > > > > > >  drivers/pci/bus.c              | 72 ++++++++++++++++++++++++++++++++++
> > > > > > >  drivers/pci/pci.h              |  8 ++++
> > > > > > >  drivers/pci/pcie/portdrv_pci.c | 32 +++++++++++++++
> > > > > > >  3 files changed, 112 insertions(+)
> > > > > > >
> > > > > > > diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
> > > > > > > index 3cef835b375f..c39fdf36b0ad 100644
> > > > > > > --- a/drivers/pci/bus.c
> > > > > > > +++ b/drivers/pci/bus.c
> > > > > > > @@ -419,3 +419,75 @@ void pci_bus_put(struct pci_bus *bus)
> > > > > > >         if (bus)
> > > > > > >                 put_device(&bus->dev);
> > > > > > >  }
> > > > > > > +
> > > > > > > +static void *alloc_subdev_regulators(struct device *dev)
> > > > > > > +{
> > > > > > > +       static const char * const supplies[] = {
> > > > > > > +               "vpcie3v3",
> > > > > > > +               "vpcie3v3aux",
> > > > > > > +               "vpcie12v",
> > > > > > > +       };
> > > > > > > +       const size_t size = sizeof(struct subdev_regulators)
> > > > > > > +               + sizeof(struct regulator_bulk_data) * ARRAY_SIZE(supplies);
> > > > > > > +       struct subdev_regulators *sr;
> > > > > > > +       int i;
> > > > > > > +
> > > > > > > +       sr = devm_kzalloc(dev, size, GFP_KERNEL);
> > > > > > > +
> > > > > > > +       if (sr) {
> > > > > > > +               sr->num_supplies = ARRAY_SIZE(supplies);
> > > > > > > +               for (i = 0; i < ARRAY_SIZE(supplies); i++)
> > > > > > > +                       sr->supplies[i].supply = supplies[i];
> > > > > > > +       }
> > > > > > > +
> > > > > > > +       return sr;
> > > > > > > +}
> > > > > > > +
> > > > > > > +
> > > > > > > +int pci_subdev_regulators_add_bus(struct pci_bus *bus)
> > > > > > > +{
> > > > > > > +       struct device *dev = &bus->dev;
> > > > > > > +       struct subdev_regulators *sr;
> > > > > > > +       int ret;
> > > > > > > +
> > > > > > > +       if (!pcie_is_port_dev(bus->self))
> > > > > > > +               return 0;
> > > > > > > +
> > > > > > > +       if (WARN_ON(bus->dev.driver_data))
> > > > > > > +               dev_err(dev, "multiple clients using dev.driver_data\n");
> > > > > > > +
> > > > > > > +       sr = alloc_subdev_regulators(&bus->dev);
> > > > > > > +       if (!sr)
> > > > > > > +               return -ENOMEM;
> > > > > > > +
> > > > > > > +       bus->dev.driver_data = sr;
> > > > > > > +       ret = regulator_bulk_get(dev, sr->num_supplies, sr->supplies);
> > > > > > > +       if (ret)
> > > > > > > +               return ret;
> > > > > > > +
> > > > > > > +       ret = regulator_bulk_enable(sr->num_supplies, sr->supplies);
> > > > > > > +       if (ret) {
> > > > > > > +               dev_err(dev, "failed to enable regulators for downstream device\n");
> > > > > > > +               return ret;
> > > > > > > +       }
> > > > > > > +
> > > > > > > +       return 0;
> > > > > > > +}
> > > > > > > +EXPORT_SYMBOL_GPL(pci_subdev_regulators_add_bus);
> > > > > >
> > > > > > Can't these just go in the portdrv probe and remove functions now?
> > > > > >
> > > > > > Rob
> > > > >
> > > > > Not really.  The idea is that  only when a host controller driver does this
> > > > >
> > > > > static struct pci_ops my_pcie_ops = {
> > > > >     .add_bus = pci_subdev_regulators_add_bus , /* see  note below */
> > > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > > >     ...
> > > > > }
> > > > >
> > > > > does it explicitly want this feature.  Without doing this, every PCI
> > > > > port in the world will execute a devm_kzalloc() and
> > > > > devm_regulator_bulk_get() to (likely) grab nothing, and then there
> > > > > will be three superfluous lines in the boot log:
> > > >
> > > > You can opt-in based on there being a DT node.
> > > >
> > > > > pci_bus 0001:01: 0001:01 supply vpcie12v not found, using dummy regulator
> > > > > pci_bus 0001:01: 0001:01 supply vpcie3v3 not found, using dummy regulator
> > > > > pci_bus 0001:01: 0001:01 supply vpcie3v3aux not found, using dummy regulator
> > > >
> > > > This would be annoying, but not really a reason for how to design this.
> > > >
> > > > > Secondly, our  HW needs to know when the  alloc/get/enable of
> > > > > regulators is done so that the PCIe link can then be attempted.   This
> > > > > is pretty much the cornerstone of this patchset.   To do this the brcm
> > > > > RC driver's call to pci_subdev_regulators_add_bus() is wrapped by
> > > > > brcm_pcie_add_bus() so that we can do this:
> > > > >
> > > > > static struct pci_ops my_pcie_ops = {
> > > > >     .add_bus = brcm_pcie_add_bus ,   /* calls pci_subdev_regulators_add_bus() */
> > > > >     .remove_bus = pci_subdev_regulators_remove_bus,
> > > >
> > > > Do add_bus/remove_bus get called during resume/suspend? If not, how do
> > > > you handle the link during resume?
> > > >
> > > > Maybe there needs to be explicit hooks for link handling. Pali has
> > > > been looking into this some.
> > > >
> > > > Rob
> > >
> > > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > > needs to be supplied at the "correct" time during establishing link
> > > procedure. I wrote it in my RFC email:
> > > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> > Hello Pali,
> >
> > I really like your proposal although I would like to get my patchset
> > first :-) :-)
> >
> > Suppose you came up with a patchset for your ideas-- would that include
> > changes to existing RC drivers to use the proposed framework?  If so,
> > I am wary that it would
> > break at least a few of them.  Or would you just present the framework
> > and allow the
> > RC drivers' authors to opt-in, one by one?
>
> My idea is to add new "framework" to allow drivers implement new
> callbacks for this "framework". There would be no change in drivers
> which do not provide these callbacks to ensure that nothing is going to
> be broken. I'm planning to implement these callbacks only for RC drivers
> for which I have hardware and can properly test to not introduce any
> regression. For other existing RC drivers it is up to other authors +
> testers. But to decrease future maintenance cost of all RC drivers I
> expect that new drivers would not implement any ad-hoc solution in their
> "probe" function and instead implement these new callbacks. That is my
> idea.
>
> > At any rate, if you want someone to test some of your ideas I can work
> > with you.
>
> Perfect! If you have any concerns or you see any issues, please reply
> that my RFC email. So I can collect feedback.
>
> Also I sent draft for updating DTS schema for PCIe devices:
> https://github.com/devicetree-org/dt-schema/pull/64

Hi Pali,
I don't see any mention or placement of the regulator nodes for power;
do you agree with where
I proposed we place them, ie in the first bridge under the root-complex,  e.g.

            pcie0: pcie@7d500000 {                                /*
root complex */
                    compatible = "brcm,bcm2711-pcie";
                    reg = <0x0 0x7d500000 0x9310>;

                    /* PCIe bridge */
                    pci@0,0 {
                            #address-cells = <3>;
                            #size-cells = <2>;
                            reg = <0x0 0x0 0x0 0x0 0x0>;
                            compatible = "pciclass,0604";
                            device_type = "pci";
                            vpcie3v3-supply = <&vreg7>;     /*
<------------- HERE  */
                            ranges;

                            pci-ep@0,0 {        /* PCIe endpoint */
                                    assigned-addresses =
                                        <0x82010000 0x0 0xf8000000 0x6
0x00000000 0x0 0x2000>;
                                    reg = <0x0 0x0 0x0 0x0 0x0>;
                                    compatible = "pci14e4,1688";
                                    #address-cells = <3>;
                                    #size-cells = <2>;

                                    ranges;
                            };
                    };
            };


Regards,
Jim

>
> > Regards,
> > Jim Quinlan
> > Broadcom STB
> >
> >
> > >
> > > I'm not sure if regulator API is the most suitable for this task in PCI
> > > core code as there are planty ways how it can be controllers. My idea
> > > presented in that email was that driver provides power callback and core
> > > pci code would use it.
> > >
> > > Because power needs to be enabled at the "correct" time during link up,
> > > I think that add/remove bus callbacks are unsuitable for this task. This
> > > would just cause adding another msleep() calls on different places to
> > > make correct timing of link up...
> > >
> > > I think it is needed to implement generic function for establishing link
> > > in pci core code with all required steps.

On Thursday 18 November 2021 10:36:00 Jim Quinlan wrote:
> On Wed, Nov 17, 2021 at 10:45 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Wednesday 17 November 2021 10:14:19 Jim Quinlan wrote:
> > > On Tue, Nov 16, 2021 at 3:53 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > Yes, I was looking at it... main power (12V/3.3V) and AUX power (3.3V)
> > > > needs to be supplied at the "correct" time during establishing link
> > > > procedure. I wrote it in my RFC email:
> > > > https://lore.kernel.org/linux-pci/20211022183808.jdeo7vntnagqkg7g@pali/
> > > Hello Pali,
> > >
> > > I really like your proposal although I would like to get my patchset
> > > first :-) :-)
> > >
> > > Suppose you came up with a patchset for your ideas-- would that include
> > > changes to existing RC drivers to use the proposed framework?  If so,
> > > I am wary that it would
> > > break at least a few of them.  Or would you just present the framework
> > > and allow the
> > > RC drivers' authors to opt-in, one by one?
> >
> > My idea is to add new "framework" to allow drivers implement new
> > callbacks for this "framework". There would be no change in drivers
> > which do not provide these callbacks to ensure that nothing is going to
> > be broken. I'm planning to implement these callbacks only for RC drivers
> > for which I have hardware and can properly test to not introduce any
> > regression. For other existing RC drivers it is up to other authors +
> > testers. But to decrease future maintenance cost of all RC drivers I
> > expect that new drivers would not implement any ad-hoc solution in their
> > "probe" function and instead implement these new callbacks. That is my
> > idea.
> >
> > > At any rate, if you want someone to test some of your ideas I can work
> > > with you.
> >
> > Perfect! If you have any concerns or you see any issues, please reply
> > that my RFC email. So I can collect feedback.
> >
> > Also I sent draft for updating DTS schema for PCIe devices:
> > https://github.com/devicetree-org/dt-schema/pull/64
> 
> Hi Pali,
> I don't see any mention or placement of the regulator nodes for power;

I put in above pull request draft only existing attributes (from
pci.txt), I have not introduce anything new yet.

> do you agree with where
> I proposed we place them, ie in the first bridge under the root-complex,  e.g.
> 
>             pcie0: pcie@7d500000 {                                /*
> root complex */
>                     compatible = "brcm,bcm2711-pcie";
>                     reg = <0x0 0x7d500000 0x9310>;
> 
>                     /* PCIe bridge */
>                     pci@0,0 {
>                             #address-cells = <3>;
>                             #size-cells = <2>;
>                             reg = <0x0 0x0 0x0 0x0 0x0>;
>                             compatible = "pciclass,0604";
>                             device_type = "pci";
>                             vpcie3v3-supply = <&vreg7>;     /*
> <------------- HERE  */

This node 'pci@0,0' describes PCIe Root Port. So yes, it is place where
power regulators belongs. I agree with you.

(Note: I would suggest to use /* PCIe Root Port */ comment instead of
/* PCIe bridge */. As PCIe bridge is ambiguous name which could mean
also other devices.)

>                             ranges;
> 
>                             pci-ep@0,0 {        /* PCIe endpoint */
>                                     assigned-addresses =
>                                         <0x82010000 0x0 0xf8000000 0x6
> 0x00000000 0x0 0x2000>;
>                                     reg = <0x0 0x0 0x0 0x0 0x0>;
>                                     compatible = "pci14e4,1688";
>                                     #address-cells = <3>;
>                                     #size-cells = <2>;
> 
>                                     ranges;
>                             };
>                     };
>             };
> 
> 
> Regards,
> Jim