[net-next,3/6] ixgbe: release lock for the duration of ixgbe_suspend_close()

From: Pavel Tatashin <pasha.tatashin@oracle.com>

Currently, during device_shutdown() ixgbe holds rtnl_lock for the duration
of lengthy ixgbe_close_suspend(). On machines with multiple ixgbe cards
this lock prevents scaling if device_shutdown() function is multi-threaded.

It is not necessary to hold this lock during ixgbe_close_suspend()
as it is not held when ixgbe_close() is called also during shutdown but for
kexec case.

Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

Hello!

On 5/17/2018 7:37 PM, Jeff Kirsher wrote:

> From: Pavel Tatashin <pasha.tatashin@oracle.com>
> 
> Currently, during device_shutdown() ixgbe holds rtnl_lock for the duration
> of lengthy ixgbe_close_suspend(). On machines with multiple ixgbe cards
> this lock prevents scaling if device_shutdown() function is multi-threaded.
> 
> It is not necessary to hold this lock during ixgbe_close_suspend()
> as it is not held when ixgbe_close() is called also during shutdown but for
> kexec case.
> 
> Signed-off-by: Pavel Tatashin <pasha.tatashin@oracle.com>
> Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> ---
>   drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> index a52d92e182ee..5ddfb93ed491 100644
> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
> @@ -6698,8 +6698,15 @@ static int __ixgbe_shutdown(struct pci_dev *pdev, bool *enable_wake)
>   	rtnl_lock();
>   	netif_device_detach(netdev);
>   
> -	if (netif_running(netdev))
> +	if (netif_running(netdev)) {
> +		/* Suspend takes a long time, device_shutdown may be
> +		 * parallelized this function, so drop lock for the

     Parallelizing? Else the sentence doesn't parse for me. :-)

> +		 * duration of this call.
> +		 */
> +		rtnl_unlock();
>   		ixgbe_close_suspend(adapter);
> +		rtnl_lock();
> +	}
>   
>   	ixgbe_clear_interrupt_scheme(adapter);
>   	rtnl_unlock();

MBR, Sergei

* parallelized this function, so drop lock for the
> 
>     Parallelizing? Else the sentence doesn't parse for me. :-)

Hi Sergei,

In a separate series I parallelized device_shutdown(), see:
http://lkml.kernel.org/r/20180516024004.28977-1-pasha.tatashin@oracle.com

But, this particular patch should be dropped, as discussed in this thread:
http://lkml.kernel.org/r/20180503035931.22439-2-pasha.tatashin@oracle.com


Alexander Duyck, made a point that a generic RTNL scalability fix should be done. This particular patch might introduce a race, since it relies on assumption that RTNL is not needed in this place because  ixgbe_close() does not have it, but Alexander Duyck, says that the callers of ixgbe_close() are assumed to own this lock.

Thank you,
Pavel

On 05/18/2018 02:37 PM, Pavel Tatashin wrote:

>       * parallelized this function, so drop lock for the
>>
>>     Parallelizing? Else the sentence doesn't parse for me. :-)

   My comment hardly makes sense when you removed all the context...

> Hi Sergei,
> 
> In a separate series I parallelized device_shutdown(), see:
> http://lkml.kernel.org/r/20180516024004.28977-1-pasha.tatashin@oracle.com
> 
> But, this particular patch should be dropped, as discussed in this thread:
> http://lkml.kernel.org/r/20180503035931.22439-2-pasha.tatashin@oracle.com
> 
> 
> Alexander Duyck, made a point that a generic RTNL scalability fix should be done. This particular patch might introduce a race, since it relies on assumption that RTNL is not needed in this place because  ixgbe_close() does not have it, but Alexander Duyck, says that the callers of ixgbe_close() are assumed to own this lock.

   My comment was about the English grammar only. :-)

> Thank you,
> Pavel

MBR, Sergei

On 05/18/2018 01:28 PM, Sergei Shtylyov wrote:
> On 05/18/2018 02:37 PM, Pavel Tatashin wrote:
> 
>>       * parallelized this function, so drop lock for the
>>>
>>>     Parallelizing? Else the sentence doesn't parse for me. :-)
> 
>    My comment hardly makes sense when you removed all the context...

Hi Sergei,

Ah gotcha:

+        /* Suspend takes a long time, device_shutdown may be
+         * parallelized this function, so drop lock for the 

The comment should have been:

ixgbe_close_suspend() takes a long time, and device_shutdown may parallelize ixgbe_shutdown(), therefore drop lock to allow concurrent execution of ixgbe_close_suspend().

Anyway, as I said, this patch should be dropped.

Thank you,
Pavel