diff mbox

[net-next,2/4] nfp: register ports as devlink ports

Message ID 20170523151247.8535-3-jakub.kicinski@netronome.com
State Changes Requested, archived
Delegated to: David Miller
Headers show

Commit Message

Jakub Kicinski May 23, 2017, 3:12 p.m. UTC
Extend nfp_port to contain devlink_port structures.  Register the
ports to allow users inspecting device ports.

Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: Simon Horman <simon.horman@netronome.com>
---
 drivers/net/ethernet/netronome/nfp/nfp_devlink.c  | 49 +++++++++++++++++++++++
 drivers/net/ethernet/netronome/nfp/nfp_main.c     |  8 ++++
 drivers/net/ethernet/netronome/nfp/nfp_main.h     |  2 +
 drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 34 ++++++++++++++++
 drivers/net/ethernet/netronome/nfp/nfp_port.h     |  8 ++++
 5 files changed, 101 insertions(+)

Comments

Jiri Pirko May 24, 2017, 12:35 p.m. UTC | #1
Tue, May 23, 2017 at 05:12:45PM CEST, jakub.kicinski@netronome.com wrote:
>Extend nfp_port to contain devlink_port structures.  Register the
>ports to allow users inspecting device ports.
>
>Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
>Reviewed-by: Simon Horman <simon.horman@netronome.com>
>---

[...]

>+void nfp_devlink_port_unregister(struct nfp_port *port)
>+{
>+	/* Due to unpleasant lock ordering we may see the port go away before
>+	 * we have fully probed.

Could you elaborate on this a bit more please?


>+	 */
>+	if (port->dl_port.registered)
>+		devlink_port_unregister(&port->dl_port);
>+}
Jakub Kicinski May 24, 2017, 7:24 p.m. UTC | #2
On Wed, 24 May 2017 14:35:14 +0200, Jiri Pirko wrote:
> >+void nfp_devlink_port_unregister(struct nfp_port *port)
> >+{
> >+	/* Due to unpleasant lock ordering we may see the port go away before
> >+	 * we have fully probed.  
> 
> Could you elaborate on this a bit more please?

It's partially due to peculiarities of the management FW more than
kernel stuff.  Unfortunately some ethtool media config requires reboot
to be applied, so we print a friendly message to the logs and
unregister the associated netdevs.  Which means once netdevs get
registered ports may go away.

Enter devlink, I need the ability to grab the adapater lock in
split/unsplit callbacks to find the ports, which implies having to drop
that lock before I register devlink.  And only after I register devlink
can I register the ports.

I could do init without registering anything, drop the adapter lock,
register devlink, and then grab the adapter lock back and register
devlink ports and netdevs.  But there is another issue...

Since I look for ports on a list maintained in the adapter struct,
driver code doesn't care if devlink_port has been registered or not.
The moment devlink is registered, split/unsplit requests will be
accepted - potentially trying to unregister devlink_port before the
register could happen.

Further down the line, also, the eswitch mode setting is coming.  Which
means the moment I register devlink itself ports will get shuffled (due
to the plan of registering VFs as ports :)). 

I feel like registering devlink should be the last action of the
driver, really.  My plan was to keep that simple if() for now, and once
we get to extending devlink with SR-IOV stuff also add the ability to
pre-register ports.  Allow registering ports on not-yet-registered
devlink (probably put them on a private list within struct devlink).
This would make devlink_register() a single point when everything
devlink becomes visible, atomically, instead of devlink itself coming
first and then ports following.

Does that make sense?  Am I misreading the code (again :S)?
Jiri Pirko May 25, 2017, 8:20 a.m. UTC | #3
Wed, May 24, 2017 at 09:24:13PM CEST, kubakici@wp.pl wrote:
>On Wed, 24 May 2017 14:35:14 +0200, Jiri Pirko wrote:
>> >+void nfp_devlink_port_unregister(struct nfp_port *port)
>> >+{
>> >+	/* Due to unpleasant lock ordering we may see the port go away before
>> >+	 * we have fully probed.  
>> 
>> Could you elaborate on this a bit more please?
>
>It's partially due to peculiarities of the management FW more than
>kernel stuff.  Unfortunately some ethtool media config requires reboot
>to be applied, so we print a friendly message to the logs and
>unregister the associated netdevs.  Which means once netdevs get
>registered ports may go away.
>
>Enter devlink, I need the ability to grab the adapater lock in
>split/unsplit callbacks to find the ports, which implies having to drop
>that lock before I register devlink.  And only after I register devlink
>can I register the ports.
>
>I could do init without registering anything, drop the adapter lock,
>register devlink, and then grab the adapter lock back and register
>devlink ports and netdevs.  But there is another issue...
>
>Since I look for ports on a list maintained in the adapter struct,
>driver code doesn't care if devlink_port has been registered or not.
>The moment devlink is registered, split/unsplit requests will be
>accepted - potentially trying to unregister devlink_port before the
>register could happen.
>
>Further down the line, also, the eswitch mode setting is coming.  Which
>means the moment I register devlink itself ports will get shuffled (due
>to the plan of registering VFs as ports :)). 
>
>I feel like registering devlink should be the last action of the
>driver, really.  My plan was to keep that simple if() for now, and once
>we get to extending devlink with SR-IOV stuff also add the ability to
>pre-register ports.  Allow registering ports on not-yet-registered
>devlink (probably put them on a private list within struct devlink).
>This would make devlink_register() a single point when everything
>devlink becomes visible, atomically, instead of devlink itself coming
>first and then ports following.
>
>Does that make sense?  Am I misreading the code (again :S)?

Well in mlxsw, we internally maintain a list of port and we assign a
pointer to the port struct only after all is initialized:

...
mlxsw_sp->ports[local_port] = mlxsw_sp_port;
err = register_netdev(dev);
...

Then in split, we check:
mlxsw_sp_port = mlxsw_sp->ports[local_port];
if (!mlxsw_sp_port) {
        dev_err(mlxsw_sp->bus_info->dev, "Port number \"%d\" does not exist\n",
                local_port);
        return -EINVAL;
}

I guess that you can do something similar. You should ensure that split
won't happen for half-initialized ports.
diff mbox

Patch

diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
index a8a52b3ff42b..b65c50c0d211 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
@@ -31,9 +31,58 @@ 
  * SOFTWARE.
  */
 
+#include <linux/rtnetlink.h>
 #include <net/devlink.h>
 
+#include "nfpcore/nfp_nsp.h"
+#include "nfp_app.h"
 #include "nfp_main.h"
+#include "nfp_port.h"
+
+static int
+nfp_devlink_fill_eth_port(struct nfp_port *port,
+			  struct nfp_eth_table_port *copy)
+{
+	struct nfp_eth_table_port *eth_port;
+
+	eth_port = __nfp_port_get_eth_port(port);
+	if (!eth_port)
+		return -EINVAL;
+
+	memcpy(copy, eth_port, sizeof(*eth_port));
+
+	return 0;
+}
 
 const struct devlink_ops nfp_devlink_ops = {
 };
+
+int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
+{
+	struct nfp_eth_table_port eth_port;
+	struct devlink *devlink;
+	int ret;
+
+	rtnl_lock();
+	ret = nfp_devlink_fill_eth_port(port, &eth_port);
+	rtnl_unlock();
+	if (ret)
+		return ret;
+
+	devlink_port_type_eth_set(&port->dl_port, port->netdev);
+	if (eth_port.is_split)
+		devlink_port_split_set(&port->dl_port, eth_port.label_port);
+
+	devlink = priv_to_devlink(app->pf);
+
+	return devlink_port_register(devlink, &port->dl_port, port->eth_id);
+}
+
+void nfp_devlink_port_unregister(struct nfp_port *port)
+{
+	/* Due to unpleasant lock ordering we may see the port go away before
+	 * we have fully probed.
+	 */
+	if (port->dl_port.registered)
+		devlink_port_unregister(&port->dl_port);
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index f22c8f083ab0..ea7c122092e7 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -377,8 +377,14 @@  static int nfp_pci_probe(struct pci_dev *pdev,
 	if (err)
 		goto err_kill_ndevs;
 
+	err = nfp_net_devlink_register(pf);
+	if (err)
+		goto err_devlink_unregiser;
+
 	return 0;
 
+err_devlink_unregiser:
+	devlink_unregister(devlink);
 err_kill_ndevs:
 	nfp_net_pci_remove(pf);
 err_fw_unload:
@@ -405,6 +411,8 @@  static void nfp_pci_remove(struct pci_dev *pdev)
 
 	devlink = priv_to_devlink(pf);
 
+	nfp_net_devlink_unregister(pf);
+
 	devlink_unregister(devlink);
 
 	nfp_net_pci_remove(pf);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 526db8029dea..7fdb9036f666 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -112,5 +112,7 @@  extern const struct devlink_ops nfp_devlink_ops;
 
 int nfp_net_pci_probe(struct nfp_pf *pf);
 void nfp_net_pci_remove(struct nfp_pf *pf);
+int nfp_net_devlink_register(struct nfp_pf *pf);
+void nfp_net_devlink_unregister(struct nfp_pf *pf);
 
 #endif /* NFP_MAIN_H */
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index dd1118c7e1a4..bf10c0a316f3 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -584,6 +584,8 @@  static void nfp_net_refresh_vnics(struct work_struct *work)
 		if (!nn->port || nn->port->type != NFP_PORT_INVALID)
 			continue;
 
+		nfp_devlink_port_unregister(nn->port);
+
 		nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
 		nfp_net_clean(nn);
 
@@ -774,3 +776,35 @@  void nfp_net_pci_remove(struct nfp_pf *pf)
 
 	cancel_work_sync(&pf->port_refresh_work);
 }
+
+int nfp_net_devlink_register(struct nfp_pf *pf)
+{
+	struct nfp_port *port;
+	int err;
+
+	mutex_lock(&pf->lock);
+	list_for_each_entry(port, &pf->ports, port_list) {
+		err = nfp_devlink_port_register(pf->app, port);
+		if (err)
+			goto err_prev_ports_unreg;
+	}
+	mutex_unlock(&pf->lock);
+
+	return 0;
+
+err_prev_ports_unreg:
+	list_for_each_entry_continue_reverse(port, &pf->ports, port_list)
+		nfp_devlink_port_unregister(port);
+	mutex_unlock(&pf->lock);
+	return err;
+}
+
+void nfp_net_devlink_unregister(struct nfp_pf *pf)
+{
+	struct nfp_port *port;
+
+	mutex_lock(&pf->lock);
+	list_for_each_entry(port, &pf->ports, port_list)
+		nfp_devlink_port_unregister(port);
+	mutex_unlock(&pf->lock);
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.h b/drivers/net/ethernet/netronome/nfp/nfp_port.h
index 641617de41cc..36a540b514be 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.h
@@ -34,6 +34,8 @@ 
 #ifndef _NFP_PORT_H_
 #define _NFP_PORT_H_
 
+#include <net/devlink.h>
+
 struct net_device;
 struct nfp_app;
 struct nfp_port;
@@ -66,6 +68,7 @@  enum nfp_port_flags {
  * @type:	what port type does the entity represent
  * @flags:	port flags
  * @app:	backpointer to the app structure
+ * @dl_port:	devlink port structure
  * @eth_id:	for %NFP_PORT_PHYS_PORT port ID in NFP enumeration scheme
  * @eth_port:	for %NFP_PORT_PHYS_PORT translated ETH Table port entry
  * @port_list:	entry on pf's list of ports
@@ -78,6 +81,8 @@  struct nfp_port {
 
 	struct nfp_app *app;
 
+	struct devlink_port dl_port;
+
 	unsigned int eth_id;
 	struct nfp_eth_table_port *eth_port;
 
@@ -99,4 +104,7 @@  void nfp_port_free(struct nfp_port *port);
 int nfp_net_refresh_eth_port(struct nfp_port *port);
 void nfp_net_refresh_port_table(struct nfp_port *port);
 
+int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port);
+void nfp_devlink_port_unregister(struct nfp_port *port);
+
 #endif