[1/2] xen-netfront: don't nest queue locks in xennet_connect()

The nesting of the per-queue rx_lock and tx_lock in xennet_connect()
is confusing to both humans and lockdep.  The locking is safe because
this is the only place where the locks are nested in this way but
lockdep still warns.

Instead of adding the missing lockdep annotations, refactor the
locking to avoid the confusing nesting.  This is still safe, because
the xenbus connection state changes are all serialized by the xenwatch
thread.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
---
 drivers/net/xen-netfront.c |   16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

On 07/02/2014 11:09 AM, David Vrabel wrote:
> The nesting of the per-queue rx_lock and tx_lock in xennet_connect()
> is confusing to both humans and lockdep.  The locking is safe because
> this is the only place where the locks are nested in this way but
> lockdep still warns.
>
> Instead of adding the missing lockdep annotations, refactor the
> locking to avoid the confusing nesting.  This is still safe, because
> the xenbus connection state changes are all serialized by the xenwatch
> thread.

Tulin (GSoC student) is working on adding some concurrency to the 
xenwatch thread (possibly eliminating it and replacing with a 
workqueue). Which means that we will need to add some sort of 
synchronization in watch callbacks, whenever they are required.

-boris

>
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
> ---
>   drivers/net/xen-netfront.c |   16 ++++++++++++----
>   1 file changed, 12 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
> index 2ccb4a0..6a37d62 100644
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -2046,13 +2046,15 @@ static int xennet_connect(struct net_device *dev)
>   	/* By now, the queue structures have been set up */
>   	for (j = 0; j < num_queues; ++j) {
>   		queue = &np->queues[j];
> -		spin_lock_bh(&queue->rx_lock);
> -		spin_lock_irq(&queue->tx_lock);
>   
>   		/* Step 1: Discard all pending TX packet fragments. */
> +		spin_lock_irq(&queue->tx_lock);
>   		xennet_release_tx_bufs(queue);
> +		spin_unlock_irq(&queue->tx_lock);
>   
>   		/* Step 2: Rebuild the RX buffer freelist and the RX ring itself. */
> +		spin_lock_bh(&queue->rx_lock);
> +
>   		for (requeue_idx = 0, i = 0; i < NET_RX_RING_SIZE; i++) {
>   			skb_frag_t *frag;
>   			const struct page *page;
> @@ -2076,6 +2078,8 @@ static int xennet_connect(struct net_device *dev)
>   		}
>   
>   		queue->rx.req_prod_pvt = requeue_idx;
> +
> +		spin_unlock_bh(&queue->rx_lock);
>   	}
>   
>   	/*
> @@ -2087,13 +2091,17 @@ static int xennet_connect(struct net_device *dev)
>   	netif_carrier_on(np->netdev);
>   	for (j = 0; j < num_queues; ++j) {
>   		queue = &np->queues[j];
> +
>   		notify_remote_via_irq(queue->tx_irq);
>   		if (queue->tx_irq != queue->rx_irq)
>   			notify_remote_via_irq(queue->rx_irq);
> -		xennet_tx_buf_gc(queue);
> -		xennet_alloc_rx_buffers(queue);
>   
> +		spin_lock_irq(&queue->tx_lock);
> +		xennet_tx_buf_gc(queue);
>   		spin_unlock_irq(&queue->tx_lock);
> +
> +		spin_lock_bh(&queue->rx_lock);
> +		xennet_alloc_rx_buffers(queue);
>   		spin_unlock_bh(&queue->rx_lock);
>   	}
>   

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 02/07/14 18:36, Boris Ostrovsky wrote:
> On 07/02/2014 11:09 AM, David Vrabel wrote:
>> The nesting of the per-queue rx_lock and tx_lock in xennet_connect()
>> is confusing to both humans and lockdep.  The locking is safe because
>> this is the only place where the locks are nested in this way but
>> lockdep still warns.
>>
>> Instead of adding the missing lockdep annotations, refactor the
>> locking to avoid the confusing nesting.  This is still safe, because
>> the xenbus connection state changes are all serialized by the xenwatch
>> thread.
> 
> Tulin (GSoC student) is working on adding some concurrency to the
> xenwatch thread (possibly eliminating it and replacing with a
> workqueue). Which means that we will need to add some sort of
> synchronization in watch callbacks, whenever they are required.

The back/frontend changed callbacks should be serialized with the device
mutex in the same way that the device core does for probe and remove etc.

David
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html