Patchwork [net-next,V0,18/21] mlx4_core: adjust catas operation for SRIOV mode

login
register
mail settings
Submitter Yevgeny Petrilin
Date Dec. 2, 2011, 10:19 a.m.
Message ID <4ED8A62D.8080008@mellanox.co.il>
Download mbox | patch
Permalink /patch/128845/
State Changes Requested
Delegated to: David Miller
Headers show

Comments

Yevgeny Petrilin - Dec. 2, 2011, 10:19 a.m.
From: Jack Morgenstein <jackm@dev.mellanox.co.il>

When running in SRIOV mode, driver should not automatically start/stop
the mlx4_core upon sensing an HCA internal error -- doing this disables/enables
sriov, which will cause the hypervisor to hang if there are running VMs with
attached VFs.

In addition, on VMs the catas process should not run at all, since the HCA
error buffer is not available to VMs in the BARs.

Signed-off-by: Jack Morgenstein <jackm@dev.mellanox.co.il>
---
 drivers/net/ethernet/mellanox/mlx4/catas.c |    7 ++++++-
 drivers/net/ethernet/mellanox/mlx4/intf.c  |    6 ++++--
 2 files changed, 10 insertions(+), 3 deletions(-)
Roland Dreier - Dec. 2, 2011, 5:07 p.m.
On Fri, Dec 2, 2011 at 2:19 AM, Yevgeny Petrilin
<yevgenyp@mellanox.co.il> wrote:
> When running in SRIOV mode, driver should not automatically start/stop
> the mlx4_core upon sensing an HCA internal error -- doing this disables/enables
> sriov, which will cause the hypervisor to hang if there are running VMs with
> attached VFs.

Not sure I understand this -- what happens if the driver doesn't reset
the device
after a catastrophic error?  Surely all the VFs are pretty screwed at
that point?

Which hypervisor are we talking about here anyway?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Liran Liss - Dec. 4, 2011, 3:45 p.m.
> On Fri, Dec 2, 2011 at 2:19 AM, Yevgeny Petrilin
> <yevgenyp@mellanox.co.il> wrote:
> > When running in SRIOV mode, driver should not automatically
> start/stop
> > the mlx4_core upon sensing an HCA internal error -- doing this
> disables/enables
> > sriov, which will cause the hypervisor to hang if there are running
> VMs with
> > attached VFs.
> 
> Not sure I understand this -- what happens if the driver doesn't reset
> the device
> after a catastrophic error?  Surely all the VFs are pretty screwed at
> that point?
> 
> Which hypervisor are we talking about here anyway?
> 
>  - R.

Indeed, if you don't reset the device after an internal error, the PF/VF might not work properly unless you reload the PF driver.

However, invoking the SW reset by the PF (upon detecting an internal error) currently affects the SRIOV capability on the PCI so we cannot do it before disabling SRIOV first.
Since the driver is not in charge of passing-through VFs to VMs, it cannot disable SRIOV either so we cannot reset the device while using SRIOV.
Note that the single-function behavior is not modified.

We intend to post a fix for this in a different patch-set.
--Liran

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/drivers/net/ethernet/mellanox/mlx4/catas.c b/drivers/net/ethernet/mellanox/mlx4/catas.c
index 45aea9c..915e947 100644
--- a/drivers/net/ethernet/mellanox/mlx4/catas.c
+++ b/drivers/net/ethernet/mellanox/mlx4/catas.c
@@ -48,7 +48,8 @@  static struct work_struct catas_work;
 static int internal_err_reset = 1;
 module_param(internal_err_reset, int, 0644);
 MODULE_PARM_DESC(internal_err_reset,
-		 "Reset device on internal errors if non-zero (default 1)");
+		 "Reset device on internal errors if non-zero"
+		 " (default 1, in SRIOV mode default is 0)");
 
 static void dump_err_buf(struct mlx4_dev *dev)
 {
@@ -116,6 +117,10 @@  void mlx4_start_catas_poll(struct mlx4_dev *dev)
 	struct mlx4_priv *priv = mlx4_priv(dev);
 	phys_addr_t addr;
 
+	/*If we are in SRIOV the default of the module param must be 0*/
+	if (mlx4_is_mfunc(dev))
+		internal_err_reset = 0;
+
 	INIT_LIST_HEAD(&priv->catas_err.list);
 	init_timer(&priv->catas_err.timer);
 	priv->catas_err.map = NULL;
diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c
index ca6feb5..b4e9f6f 100644
--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ -142,7 +142,8 @@  int mlx4_register_device(struct mlx4_dev *dev)
 		mlx4_add_device(intf, priv);
 
 	mutex_unlock(&intf_mutex);
-	mlx4_start_catas_poll(dev);
+	if (!mlx4_is_slave(dev))
+		mlx4_start_catas_poll(dev);
 
 	return 0;
 }
@@ -152,7 +153,8 @@  void mlx4_unregister_device(struct mlx4_dev *dev)
 	struct mlx4_priv *priv = mlx4_priv(dev);
 	struct mlx4_interface *intf;
 
-	mlx4_stop_catas_poll(dev);
+	if (!mlx4_is_slave(dev))
+		mlx4_stop_catas_poll(dev);
 	mutex_lock(&intf_mutex);
 
 	list_for_each_entry(intf, &intf_list, list)