[net-next,1/3] net: phy: Support enabling clocks prior to bus probe

Some Ethernet PHYs may require that their clock, which typically drives
their logic to respond to reads on the MDIO bus be enabled before
issusing a MDIO bus scan.

We have a chicken and egg problem though which is that we cannot enable
a given Ethernet PHY's device clock until we have a phy_device instance
create and called the driver's probe function. This will not happen
unless we are successful in probing the PHY device, which requires its
clock(s) to be turned on.

For DT based systems we can solve this by using of_clk_get() which
operates on a device_node reference, and make sure that all clocks
associaed with the node are enabled prior to doing any reads towards the
device. In order to avoid drivers having to know the a priori reference
count of the resources, we drop them back to 0 right before calling
->probe() which is then supposed to manage the resources normally.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 drivers/net/phy/phy_device.c | 15 +++++-
 drivers/of/of_mdio.c         | 95 ++++++++++++++++++++++++++++++++++++
 include/linux/of_mdio.h      | 16 ++++++
 include/linux/phy.h          | 13 +++++
 4 files changed, 138 insertions(+), 1 deletion(-)

On Wed, Sep 02, 2020 at 09:39:45PM -0700, Florian Fainelli wrote:
> Some Ethernet PHYs may require that their clock, which typically drives
> their logic to respond to reads on the MDIO bus be enabled before
> issusing a MDIO bus scan.

issuing

> 
> We have a chicken and egg problem though which is that we cannot enable
> a given Ethernet PHY's device clock until we have a phy_device instance
> create and called the driver's probe function. This will not happen
> unless we are successful in probing the PHY device, which requires its
> clock(s) to be turned on.
> 
> For DT based systems we can solve this by using of_clk_get() which
> operates on a device_node reference, and make sure that all clocks
> associaed with the node are enabled prior to doing any reads towards the

associated.

> device. In order to avoid drivers having to know the a priori reference
> count of the resources, we drop them back to 0 right before calling
> ->probe() which is then supposed to manage the resources normally.

  Andrew

On Wed, Sep 02, 2020 at 09:39:45PM -0700, Florian Fainelli wrote:
> Some Ethernet PHYs may require that their clock, which typically drives
> their logic to respond to reads on the MDIO bus be enabled before
> issusing a MDIO bus scan.
> 
> We have a chicken and egg problem though which is that we cannot enable
> a given Ethernet PHY's device clock until we have a phy_device instance
> create and called the driver's probe function. This will not happen
> unless we are successful in probing the PHY device, which requires its
> clock(s) to be turned on.
> 
> For DT based systems we can solve this by using of_clk_get() which
> operates on a device_node reference, and make sure that all clocks
> associaed with the node are enabled prior to doing any reads towards the
> device. In order to avoid drivers having to know the a priori reference
> count of the resources, we drop them back to 0 right before calling
> ->probe() which is then supposed to manage the resources normally.

Hi Florian

This seems asymmetric. The resources are enabled in
of_mdiobus_phy_device_register() but disabled in phy_probe().

What we are really interested in is having the clock ticking while
trying to read the ID registers. Could the enable/disable be placed
around the call to get_phy_id()?

       Andrew

On Wed, Sep 2, 2020 at 10:39 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> Some Ethernet PHYs may require that their clock, which typically drives
> their logic to respond to reads on the MDIO bus be enabled before
> issusing a MDIO bus scan.
>
> We have a chicken and egg problem though which is that we cannot enable
> a given Ethernet PHY's device clock until we have a phy_device instance
> create and called the driver's probe function. This will not happen
> unless we are successful in probing the PHY device, which requires its
> clock(s) to be turned on.
>
> For DT based systems we can solve this by using of_clk_get() which
> operates on a device_node reference, and make sure that all clocks
> associaed with the node are enabled prior to doing any reads towards the
> device. In order to avoid drivers having to know the a priori reference
> count of the resources, we drop them back to 0 right before calling
> ->probe() which is then supposed to manage the resources normally.

What if a device requires clocks enabled in a certain order or timing?
It's not just clocks, you could have some GPIOs or a regulator that
need enabling first. It's device specific, so really needs a per
device solution. This is not just an issue with MDIO. I think we
really need some sort of pre-probe hook in the driver model in order
to do any non-discoverable init for discoverable buses. Or perhaps
forcing probe when there are devices defined in DT if they're not
discovered by normal means.

However, you're not trying to do this in DT (see mmc-pwrseq :( ) and
this can evolve within the kernel to a better solution.

>
> Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
> ---
>  drivers/net/phy/phy_device.c | 15 +++++-
>  drivers/of/of_mdio.c         | 95 ++++++++++++++++++++++++++++++++++++

BTW, I'd really like to move this to drivers/net/ as we've done with
all the other bus/subsystem DT code.

>  include/linux/of_mdio.h      | 16 ++++++
>  include/linux/phy.h          | 13 +++++
>  4 files changed, 138 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
> index 57d44648c8dd..bf2824ba056e 100644
> --- a/drivers/net/phy/phy_device.c
> +++ b/drivers/net/phy/phy_device.c
> @@ -23,6 +23,7 @@
>  #include <linux/mm.h>
>  #include <linux/module.h>
>  #include <linux/netdevice.h>
> +#include <linux/of_mdio.h>
>  #include <linux/phy.h>
>  #include <linux/phy_led_triggers.h>
>  #include <linux/property.h>
> @@ -2845,6 +2846,15 @@ static int phy_probe(struct device *dev)
>
>         mutex_lock(&phydev->lock);
>
> +       /* To allow PHY drivers to manage device resources such as
> +        * clocks, regulators or others, disable the resources that
> +        * were enabled during the bus->reset or the PHY registration
> +        * routine such that they work with a natural resource reference
> +        * count.
> +        */
> +       of_mdiobus_device_disable_resources(phydev->mdio.bus,
> +                                           phydev->mdio.addr);
> +
>         /* Deassert the reset signal */
>         phy_device_reset(phydev, 0);
>
> @@ -2914,8 +2924,11 @@ static int phy_probe(struct device *dev)
>
>  out:
>         /* Assert the reset signal */
> -       if (err)
> +       if (err) {
>                 phy_device_reset(phydev, 1);
> +               of_mdiobus_device_disable_resources(phydev->mdio.bus,
> +                                                   phydev->mdio.addr);
> +       }
>
>         mutex_unlock(&phydev->lock);
>
> diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
> index cb32d7ef4938..bbce4a70312c 100644
> --- a/drivers/of/of_mdio.c
> +++ b/drivers/of/of_mdio.c
> @@ -19,6 +19,7 @@
>  #include <linux/of_mdio.h>
>  #include <linux/of_net.h>
>  #include <linux/module.h>
> +#include <linux/clk.h>
>
>  #define DEFAULT_GPIO_RESET_DELAY       10      /* in microseconds */
>
> @@ -60,6 +61,92 @@ static struct mii_timestamper *of_find_mii_timestamper(struct device_node *node)
>         return register_mii_timestamper(arg.np, arg.args[0]);
>  }
>
> +int of_mdiobus_device_enable_resources(struct mii_bus *bus,
> +                                      struct device_node *child,
> +                                      u32 addr)
> +{
> +       struct mdio_device_resource *res = &bus->mdio_resources[addr];
> +       unsigned int i;
> +       int rc;
> +
> +       if (res->enabled_resources) {
> +               dev_dbg(&bus->dev,
> +                       "MDIO resources for %d already enabled\n", addr);
> +               return 0;
> +       }
> +
> +       rc = of_count_phandle_with_args(child, "clocks", "#clock-cells");
> +       if (rc < 0) {
> +               if (rc == -ENOENT)
> +                       rc = 0;
> +               else
> +                       return rc;
> +       }
> +
> +       res->num_clks = rc;
> +       if (rc == 0)
> +               return rc;
> +
> +       dev_dbg(&bus->dev, "Found %d clocks for child at %d\n", rc, addr);
> +
> +       res->clks = devm_kcalloc(&bus->dev, res->num_clks,
> +                                sizeof(struct clk *), GFP_KERNEL);
> +       if (!res->clks)
> +               return -ENOMEM;
> +
> +       for (i = 0; i < res->num_clks; i++) {
> +               res->clks[i] = of_clk_get(child, i);
> +               if (IS_ERR(res->clks[i])) {
> +                       if (PTR_ERR(res->clks[i]) == -ENOENT)
> +                               continue;
> +
> +                       return PTR_ERR(res->clks[i]);

If you return here, you are potentially leaving clocks enabled and
with a reference.

> +               }
> +
> +               rc = clk_prepare_enable(res->clks[i]);

Can't you use clk_bulk_prepare_enable?

Looks like there's no bulk version of of_clk_get though...

> +               if (rc) {
> +                       dev_err(&bus->dev,
> +                               "Failed to enabled clock for %d (rc: %d)\n",
> +                               addr, rc);
> +                       goto out_clk_disable;
> +               }
> +
> +               dev_dbg(&bus->dev,
> +                       "Enable clock %d for child at %d\n",
> +                       i, addr);
> +       }
> +
> +       res->enabled_resources = true;
> +
> +       return 0;
> +
> +out_clk_disable:
> +       while (i-- > 0) {
> +               clk_disable_unprepare(res->clks[i]);
> +               clk_put(res->clks[i]);
> +       }
> +       res->enabled_resources = false;
> +       return rc;
> +}
> +EXPORT_SYMBOL(of_mdiobus_device_enable_resources);
> +
> +void of_mdiobus_device_disable_resources(struct mii_bus *bus, u32 addr)
> +{
> +       struct mdio_device_resource *res = &bus->mdio_resources[addr];
> +       unsigned int i;
> +
> +       if (!res->enabled_resources || res->num_clks == 0)
> +               return;
> +
> +       for (i = 0; i < res->num_clks; i++) {
> +               clk_disable_unprepare(res->clks[i]);
> +               clk_put(res->clks[i]);
> +               dev_dbg(&bus->dev, "Disabled clk %d for %d\n", i, addr);
> +       }
> +       res->enabled_resources = false;
> +}
> +EXPORT_SYMBOL(of_mdiobus_device_disable_resources);
> +
>  int of_mdiobus_phy_device_register(struct mii_bus *mdio, struct phy_device *phy,
>                               struct device_node *child, u32 addr)
>  {
> @@ -117,6 +204,12 @@ static int of_mdiobus_register_phy(struct mii_bus *mdio,
>         if (IS_ERR(mii_ts))
>                 return PTR_ERR(mii_ts);
>
> +       rc = of_mdiobus_device_enable_resources(mdio, child, addr);
> +       if (rc) {
> +               dev_err(&mdio->dev, "enable resources: %d\n", rc);
> +               return rc;
> +       }
> +
>         is_c45 = of_device_is_compatible(child,
>                                          "ethernet-phy-ieee802.3-c45");
>
> @@ -125,6 +218,7 @@ static int of_mdiobus_register_phy(struct mii_bus *mdio,
>         else
>                 phy = get_phy_device(mdio, addr, is_c45);
>         if (IS_ERR(phy)) {
> +               of_mdiobus_device_disable_resources(mdio, addr);
>                 if (mii_ts)
>                         unregister_mii_timestamper(mii_ts);
>                 return PTR_ERR(phy);
> @@ -132,6 +226,7 @@ static int of_mdiobus_register_phy(struct mii_bus *mdio,
>
>         rc = of_mdiobus_phy_device_register(mdio, phy, child, addr);
>         if (rc) {
> +               of_mdiobus_device_disable_resources(mdio, addr);
>                 if (mii_ts)
>                         unregister_mii_timestamper(mii_ts);
>                 phy_device_free(phy);
> diff --git a/include/linux/of_mdio.h b/include/linux/of_mdio.h
> index 1efb88d9f892..f43f4bcb3f22 100644
> --- a/include/linux/of_mdio.h
> +++ b/include/linux/of_mdio.h
> @@ -58,6 +58,10 @@ static inline int of_mdio_parse_addr(struct device *dev,
>         return addr;
>  }
>
> +int of_mdiobus_device_enable_resources(struct mii_bus *mdio,
> +                                      struct device_node *child, u32 addr);
> +void of_mdiobus_device_disable_resources(struct mii_bus *mdio, u32 addr);
> +
>  #else /* CONFIG_OF_MDIO */
>  static inline bool of_mdiobus_child_is_phy(struct device_node *child)
>  {
> @@ -129,6 +133,18 @@ static inline int of_mdiobus_phy_device_register(struct mii_bus *mdio,
>  {
>         return -ENOSYS;
>  }
> +
> +static inline int of_mdiobus_device_enable_resources(struct mii_bus *mdio,
> +                                                    struct device_node *child,
> +                                                    u32 addr)
> +{
> +       return 0;
> +}
> +
> +static inline void of_mdiobus_device_disable_resources(struct mii_bus *mdio,
> +                                                      u32 addr)
> +{
> +}
>  #endif
>
>
> diff --git a/include/linux/phy.h b/include/linux/phy.h
> index 3a09d2bf69ea..a01953daea45 100644
> --- a/include/linux/phy.h
> +++ b/include/linux/phy.h
> @@ -247,6 +247,14 @@ struct phy_package_shared {
>  #define PHY_SHARED_F_INIT_DONE  0
>  #define PHY_SHARED_F_PROBE_DONE 1
>
> +struct clk;
> +
> +struct mdio_device_resource {
> +       bool enabled_resources;
> +       unsigned int num_clks;
> +       struct clk **clks;
> +};
> +
>  /*
>   * The Bus class for PHYs.  Devices which provide access to
>   * PHYs should register using this structure
> @@ -291,6 +299,11 @@ struct mii_bus {
>          */
>         int irq[PHY_MAX_ADDR];
>
> +       /* An array of MDIO device resources that must be enabled
> +        * during probe for identification to succeed.
> +        */
> +       struct mdio_device_resource mdio_resources[PHY_MAX_ADDR];
> +
>         /* GPIO reset pulse width in microseconds */
>         int reset_delay_us;
>         /* GPIO reset deassert delay in microseconds */
> --
> 2.25.1
>

On Thu, Sep 03, 2020 at 03:28:22PM -0600, Rob Herring wrote:
> What if a device requires clocks enabled in a certain order or timing?
> It's not just clocks, you could have some GPIOs or a regulator that
> need enabling first. It's device specific, so really needs a per
> device solution. This is not just an issue with MDIO. I think we
> really need some sort of pre-probe hook in the driver model in order
> to do any non-discoverable init for discoverable buses.

Hi Rob

How do you solve the chicken/egg of knowing what device specific init
is needed before you can discover what device you have on the bus?

> Or perhaps forcing probe when there are devices defined in DT if
> they're not discovered by normal means.

The PHY subsystem has this. You came specify in DT the ID of the
device which we would normally read during bus discovery. The correct
driver is then loaded and probed. But it is good practice to avoid
this. OEMs are known to change the PHY in order to perform cost
optimisation. So we prefer to do discover and do the right thing if
the PHY has changed.

As for GPIOS and regulators, i expect this code will expand pretty
soon after being merged to handle those. There are users wanting
it. We already have some standard properties defined, in terms of
gpios, delay while off, delay after turning it on. As for ordering, i
guess it would make sense to enable the clocks and then hit it with a
reset? If there is a device which cannot be handled like this, it can
always hard code its ID in device tree, and fully control its
resources in the driver.

	  Andrew

On 9/3/2020 2:28 PM, Rob Herring wrote:
> On Wed, Sep 2, 2020 at 10:39 PM Florian Fainelli <f.fainelli@gmail.com> wrote:
>>
>> Some Ethernet PHYs may require that their clock, which typically drives
>> their logic to respond to reads on the MDIO bus be enabled before
>> issusing a MDIO bus scan.
>>
>> We have a chicken and egg problem though which is that we cannot enable
>> a given Ethernet PHY's device clock until we have a phy_device instance
>> create and called the driver's probe function. This will not happen
>> unless we are successful in probing the PHY device, which requires its
>> clock(s) to be turned on.
>>
>> For DT based systems we can solve this by using of_clk_get() which
>> operates on a device_node reference, and make sure that all clocks
>> associaed with the node are enabled prior to doing any reads towards the
>> device. In order to avoid drivers having to know the a priori reference
>> count of the resources, we drop them back to 0 right before calling
>> ->probe() which is then supposed to manage the resources normally.
> 
> What if a device requires clocks enabled in a certain order or timing?
> It's not just clocks, you could have some GPIOs or a regulator that
> need enabling first. It's device specific, so really needs a per
> device solution. This is not just an issue with MDIO. I think we
> really need some sort of pre-probe hook in the driver model in order
> to do any non-discoverable init for discoverable buses. Or perhaps
> forcing probe when there are devices defined in DT if they're not
> discovered by normal means.

I like the pre-probe hook idea, and there are other devices that I have 
access to that would benefit from that, like PCIe end-points that 
require regulators to be turned on in order for them to be discoverable.

For MDIO we might actually be able to create the backing device 
reference without having read the device ID yet, provided that we know 
its address on the bus, which DT can tell us.

Bartosz attempted to do that not so long ago and we sort of stalled 
there, too:

https://lkml.org/lkml/2020/6/22/253

Let me see if I can just add a pre-probe hook, make use of it in the 
MDIO layer, and we see how we can apply it to other subsystems?

On 9/3/2020 2:42 PM, Andrew Lunn wrote:
> On Thu, Sep 03, 2020 at 03:28:22PM -0600, Rob Herring wrote:
>> What if a device requires clocks enabled in a certain order or timing?
>> It's not just clocks, you could have some GPIOs or a regulator that
>> need enabling first. It's device specific, so really needs a per
>> device solution. This is not just an issue with MDIO. I think we
>> really need some sort of pre-probe hook in the driver model in order
>> to do any non-discoverable init for discoverable buses.
> 
> Hi Rob
> 
> How do you solve the chicken/egg of knowing what device specific init
> is needed before you can discover what device you have on the bus?

For MDIO since we have a fixed number of devices on the bus, we could 
pre-populate the MDIO map for all addresses, and free up the devices 
that we did not probe.

When using DT we can first parse the address, create a mdio_device 
there, and then turn on clocks/regulators/GPIOs whatever since we now 
have a device reference. Only then do we bind the device with its driver.

If we are using the DT scanning loop because the node did not provide a 
"reg" property, then the PHY must be in a functional state to be probed, 
we cannot guess what we do not know.

All of this uses MDIO implementation knowledge though.

> 
>> Or perhaps forcing probe when there are devices defined in DT if
>> they're not discovered by normal means.
> 
> The PHY subsystem has this. You came specify in DT the ID of the
> device which we would normally read during bus discovery. The correct
> driver is then loaded and probed. But it is good practice to avoid
> this. OEMs are known to change the PHY in order to perform cost
> optimisation. So we prefer to do discover and do the right thing if
> the PHY has changed.
> 
> As for GPIOS and regulators, i expect this code will expand pretty
> soon after being merged to handle those. There are users wanting
> it. We already have some standard properties defined, in terms of
> gpios, delay while off, delay after turning it on. As for ordering, i
> guess it would make sense to enable the clocks and then hit it with a
> reset? If there is a device which cannot be handled like this, it can
> always hard code its ID in device tree, and fully control its
> resources in the driver.
> 
> 	  Andrew
>