mv643xx_eth: Fix a possible deadlock upon ifdown

From: Lubomir Rintel <lubo.rintel@gooddata.com>

On Fri, Jan 04, 2013 at 03:07:02PM +0100, Lubomir Rintel wrote:

> From: Lubomir Rintel <lubo.rintel@gooddata.com>
> 
> =================================
> [ INFO: inconsistent lock state ]
> 3.7.0-6.luboskovo.fc19.armv5tel.kirkwood #1 Tainted: G        W
> ---------------------------------
> inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
> NetworkManager/337 [HC0[0]:SC0[0]:HE1:SE1] takes:
>  (_xmit_ETHER#2){+.?...}, at: [<bf07adfc>] txq_reclaim+0x54/0x264 [mv643xx_eth]
> {IN-SOFTIRQ-W} state was registered at:
>   [<c0068480>] __lock_acquire+0x5b4/0x17d0
>   [<c0069d7c>] lock_acquire+0x160/0x1e0
>   [<c04f41a0>] _raw_spin_lock+0x50/0x88
>   [<c0407178>] sch_direct_xmit+0x4c/0x2d4
>   [<c03ec978>] dev_queue_xmit+0x4b8/0x8d8
>   [<c0492dc8>] ip6_finish_output2+0x350/0x42c
>   [<c04b7fd8>] mld_sendpack+0x2d0/0x514
>   [<c04b8834>] mld_ifc_timer_expire+0x228/0x278
>   [<c002afe8>] call_timer_fn+0x140/0x33c
>   [<c002bbf0>] run_timer_softirq+0x278/0x32c
>   [<c0024288>] __do_softirq+0x16c/0x398
>   [<c002488c>] irq_exit+0x5c/0xc0
>   [<c0009c64>] handle_IRQ+0x6c/0x8c
>   [<c04f5218>] __irq_svc+0x38/0x80
>   [<c0065684>] lock_is_held+0x4/0x54
>   [<c004d5a0>] __might_sleep+0x44/0x228
>   [<c04f25a4>] down_read+0x28/0x88
>   [<c0263c94>] __copy_to_user_memcpy+0xa8/0x140
>   [<c01374d0>] seq_read+0x3ac/0x474
>   [<c011623c>] vfs_read+0xac/0x184
>   [<c0116354>] sys_read+0x40/0x6c
>   [<c0008cc0>] ret_fast_syscall+0x0/0x38
> irq event stamp: 115119
> hardirqs last  enabled at (115119): [<c04f4cf0>] _raw_spin_unlock_irqrestore+0x44/0x64
> hardirqs last disabled at (115118): [<c04f430c>] _raw_spin_lock_irqsave+0x28/0xa0
> softirqs last  enabled at (114880): [<c00243d4>] __do_softirq+0x2b8/0x398
> softirqs last disabled at (114873): [<c002488c>] irq_exit+0x5c/0xc0
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>        CPU0
>        ----
>   lock(_xmit_ETHER#2);
>   <Interrupt>
>     lock(_xmit_ETHER#2);
> 
>  *** DEADLOCK ***
> 
> 1 lock held by NetworkManager/337:
>  #0:  (rtnl_mutex){+.+.+.}, at: [<c03fad04>] rtnetlink_rcv+0x14/0x2c
> 
> stack backtrace:
> [<c000f5a8>] (unwind_backtrace+0x0/0x124) from [<c04ebea8>] (print_usage_bug.part.29+0x20c/0x26c)
> [<c04ebea8>] (print_usage_bug.part.29+0x20c/0x26c) from [<c0067cc4>] (mark_lock+0x404/0x60c)
> [<c0067cc4>] (mark_lock+0x404/0x60c) from [<c0068504>] (__lock_acquire+0x638/0x17d0)
> [<c0068504>] (__lock_acquire+0x638/0x17d0) from [<c0069d7c>] (lock_acquire+0x160/0x1e0)
> [<c0069d7c>] (lock_acquire+0x160/0x1e0) from [<c04f41a0>] (_raw_spin_lock+0x50/0x88)
> [<c04f41a0>] (_raw_spin_lock+0x50/0x88) from [<bf07adfc>] (txq_reclaim+0x54/0x264 [mv643xx_eth])
> [<bf07adfc>] (txq_reclaim+0x54/0x264 [mv643xx_eth]) from [<bf07b03c>] (txq_deinit+0x30/0xec [mv643xx_eth])
> [<bf07b03c>] (txq_deinit+0x30/0xec [mv643xx_eth]) from [<bf07b21c>] (mv643xx_eth_stop+0x124/0x140 [mv643xx_eth])
> [<bf07b21c>] (mv643xx_eth_stop+0x124/0x140 [mv643xx_eth]) from [<c03e8bbc>] (__dev_close_many+0xb0/0xec)
> [<c03e8bbc>] (__dev_close_many+0xb0/0xec) from [<c03e8c28>] (__dev_close+0x30/0x44)
> [<c03e8c28>] (__dev_close+0x30/0x44) from [<c03ed154>] (__dev_change_flags+0x94/0x120)
> [<c03ed154>] (__dev_change_flags+0x94/0x120) from [<c03ed260>] (dev_change_flags+0x18/0x4c)
> [<c03ed260>] (dev_change_flags+0x18/0x4c) from [<c03fb174>] (do_setlink+0x2cc/0x7ac)
> [<c03fb174>] (do_setlink+0x2cc/0x7ac) from [<c03fc5ec>] (rtnl_newlink+0x26c/0x4a8)
> [<c03fc5ec>] (rtnl_newlink+0x26c/0x4a8) from [<c03fc104>] (rtnetlink_rcv_msg+0x280/0x29c)
> [<c03fc104>] (rtnetlink_rcv_msg+0x280/0x29c) from [<c041245c>] (netlink_rcv_skb+0x58/0xb4)
> [<c041245c>] (netlink_rcv_skb+0x58/0xb4) from [<c03fad10>] (rtnetlink_rcv+0x20/0x2c)
> [<c03fad10>] (rtnetlink_rcv+0x20/0x2c) from [<c0411dec>] (netlink_unicast+0x158/0x208)
> [<c0411dec>] (netlink_unicast+0x158/0x208) from [<c0412254>] (netlink_sendmsg+0x310/0x3c0)
> [<c0412254>] (netlink_sendmsg+0x310/0x3c0) from [<c03d209c>] (sock_sendmsg+0xa8/0xd0)
> [<c03d209c>] (sock_sendmsg+0xa8/0xd0) from [<c03d2314>] (__sys_sendmsg+0x1d8/0x280)
> [<c03d2314>] (__sys_sendmsg+0x1d8/0x280) from [<c03d4054>] (sys_sendmsg+0x44/0x68)
> [<c03d4054>] (sys_sendmsg+0x44/0x68) from [<c0008cc0>] (ret_fast_syscall+0x0/0x38)
> ---
>  drivers/net/ethernet/marvell/mv643xx_eth.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c b/drivers/net/ethernet/marvell/mv643xx_eth.c
> index 84c1326..67a3e78 100644
> --- a/drivers/net/ethernet/marvell/mv643xx_eth.c
> +++ b/drivers/net/ethernet/marvell/mv643xx_eth.c
> @@ -943,7 +943,7 @@ static int txq_reclaim(struct tx_queue *txq, int budget, int force)
>  	struct netdev_queue *nq = netdev_get_tx_queue(mp->dev, txq->index);
>  	int reclaimed;
>  
> -	__netif_tx_lock(nq, smp_processor_id());
> +	__netif_tx_lock_bh(nq);
>  
>  	reclaimed = 0;
>  	while (reclaimed < budget && txq->tx_desc_count > 0) {
> @@ -989,7 +989,7 @@ static int txq_reclaim(struct tx_queue *txq, int budget, int force)
>  		dev_kfree_skb(skb);
>  	}
>  
> -	__netif_tx_unlock(nq);
> +	__netif_tx_unlock_bh(nq);
>  
>  	if (reclaimed < budget)
>  		mp->work_tx &= ~(1 << txq->index);
> -- 

Maybe I'm not reading it right, but I doubt that this is an actual
deadlock or that the patch is needed.

txq_reclaim() indeed doesn't disable BHs, but that's because it's
always called in BH context.  Almost always -- the only exception is
txq_deinit(), called from ->ndo_stop(), but by that time we've
already napi_disable()'d and netif_carrier_off()'d and free_irq()'d.

How to explain that to lockdep, though, I don't know.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am 04.01.2013 21:25, schrieb Lennert Buytenhek:
> On Fri, Jan 04, 2013 at 03:07:02PM +0100, Lubomir Rintel wrote:
>
>> From: Lubomir Rintel <lubo.rintel@gooddata.com>
>>
>> =================================
>> [ INFO: inconsistent lock state ]
>> 3.7.0-6.luboskovo.fc19.armv5tel.kirkwood #1 Tainted: G        W
>> ---------------------------------
>> inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
>> NetworkManager/337 [HC0[0]:SC0[0]:HE1:SE1] takes:
>>   (_xmit_ETHER#2){+.?...}, at: [<bf07adfc>] txq_reclaim+0x54/0x264 [mv643xx_eth]

I get the same annoying warning when the MTU gets changed (through dhcp).

>
> Maybe I'm not reading it right, but I doubt that this is an actual
> deadlock or that the patch is needed.
>
> txq_reclaim() indeed doesn't disable BHs, but that's because it's
> always called in BH context.  Almost always -- the only exception is
> txq_deinit(), called from ->ndo_stop(), but by that time we've
> already napi_disable()'d and netif_carrier_off()'d and free_irq()'d.

Agreed. I've just read me through that too and don't think a deadlock is 
possible.

>
> How to explain that to lockdep, though, I don't know.

The patch helps with that. ;)

Regards,

Alexander
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Mar 01, 2013 at 06:30:13PM +0100, Alexander Holler wrote:

> >>From: Lubomir Rintel <lubo.rintel@gooddata.com>
> >>
> >>=================================
> >>[ INFO: inconsistent lock state ]
> >>3.7.0-6.luboskovo.fc19.armv5tel.kirkwood #1 Tainted: G        W
> >>---------------------------------
> >>inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
> >>NetworkManager/337 [HC0[0]:SC0[0]:HE1:SE1] takes:
> >>  (_xmit_ETHER#2){+.?...}, at: [<bf07adfc>] txq_reclaim+0x54/0x264 [mv643xx_eth]
> 
> I get the same annoying warning when the MTU gets changed (through dhcp).

That is actually an issue.


> >Maybe I'm not reading it right, but I doubt that this is an actual
> >deadlock or that the patch is needed.
> >
> >txq_reclaim() indeed doesn't disable BHs, but that's because it's
> >always called in BH context.  Almost always -- the only exception is
> >txq_deinit(), called from ->ndo_stop(), but by that time we've
> >already napi_disable()'d and netif_carrier_off()'d and free_irq()'d.
> 
> Agreed. I've just read me through that too and don't think a
> deadlock is possible.
> 
> >How to explain that to lockdep, though, I don't know.
> 
> The patch helps with that. ;)

It fixes a bug (the MTU change thing) and a non-bug (the lockdep
warning) at the expense of slowing down the much more common path,
and I don't like it for that reason.

Can you make a __txq_reclaim() which is basically txq_reclaim()
without grabbing the tx queue lock, and then move the lock grabbing
to the caller?

E.g. make __txq_reclaim() have two callers, txq_reclaim() and
txq_reclaim_bh(), and then use the appropriate wrapper depending on
the context.  (tx queue lock but no BH disable when called from
mv643xx_eth_poll(), tx queue lock plus BH disable for MTU change,
and no locking at all when called from ->ndo_stop().  Something
like that.)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am 12.03.2013 03:49, schrieb Lennert Buytenhek:
> On Fri, Mar 01, 2013 at 06:30:13PM +0100, Alexander Holler wrote:
>
>>>> From: Lubomir Rintel <lubo.rintel@gooddata.com>
>>>>
>>>> =================================
>>>> [ INFO: inconsistent lock state ]
>>>> 3.7.0-6.luboskovo.fc19.armv5tel.kirkwood #1 Tainted: G        W
>>>> ---------------------------------
>>>> inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
>>>> NetworkManager/337 [HC0[0]:SC0[0]:HE1:SE1] takes:
>>>>   (_xmit_ETHER#2){+.?...}, at: [<bf07adfc>] txq_reclaim+0x54/0x264 [mv643xx_eth]
>>
>> I get the same annoying warning when the MTU gets changed (through dhcp).
>
> That is actually an issue.
>

I don't think so. Internally the driver calls mv643xx_eth_stop() 
therefore lockdep issues almost the same warning as when the interface 
is shut down (see below). And reading the code, I haven't seen how a 
deadlock could happen. I just wanted to mention that it happens too, 
when the MTU gets changed, in case someone else will see that warning 
after enabling lockdep and wonders if it actually is a problem.

> It fixes a bug (the MTU change thing) and a non-bug (the lockdep
> warning) at the expense of slowing down the much more common path,
> and I don't like it for that reason.

I don't see how something important is slowed down. The patch only 
affects code which is either used when the link goes down or when the 
interface will be polled. I'm not sure where poll is used, but I believe 
it is only used by netconsole, so nothing really important is slowed 
down by the patch. At least if I understood everything correct. ;)

> Can you make a __txq_reclaim() which is basically txq_reclaim()
> without grabbing the tx queue lock, and then move the lock grabbing
> to the caller?
>
> E.g. make __txq_reclaim() have two callers, txq_reclaim() and
> txq_reclaim_bh(), and then use the appropriate wrapper depending on
> the context.  (tx queue lock but no BH disable when called from
> mv643xx_eth_poll(), tx queue lock plus BH disable for MTU change,
> and no locking at all when called from ->ndo_stop().  Something
> like that.)

Hmm, I'm have to take a deep look at the driver to understand what you 
mean (it's some days ago I've last did), but I will try. ;)

For completeness, below is the full output (when the MTU will be changed 
and lockdep is enabled), I've just removed the dates to shorten it.

Regards,

Alexander

=================================
[ INFO: inconsistent lock state ]
3.8.1-dockstar-00023-g524f9cf #280 Not tainted
---------------------------------
inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
ip/1169 [HC0[0]:SC0[0]:HE1:SE1] takes:
  (_xmit_ETHER#2){+.?...}, at: [<c0264f2c>] txq_reclaim+0x44/0x1c8
{IN-SOFTIRQ-W} state was registered at:
   [<c00504c0>] __lock_acquire+0x5ac/0x17cc
   [<c0051b54>] lock_acquire+0x64/0x78
   [<c03656e0>] _raw_spin_lock+0x40/0x50
   [<c0265764>] mv643xx_eth_poll+0x1e8/0x634
   [<c02bfd54>] net_rx_action+0x9c/0x218
   [<c001dc1c>] __do_softirq+0xb4/0x18c
   [<c001e0b0>] irq_exit+0x54/0xb8
   [<c00098ec>] handle_IRQ+0x64/0x84
   [<c0008b98>] __irq_svc+0x38/0xa0
   [<c00186e8>] console_unlock+0x194/0x398
   [<c0019934>] register_console+0x294/0x36c
   [<c04cc4ec>] init_netconsole+0x138/0x1d4
   [<c0008480>] do_one_initcall+0x90/0x16c
   [<c04b9858>] kernel_init_freeable+0xe8/0x1b0
   [<c035ca40>] kernel_init+0x8/0xe4
   [<c00090a8>] ret_from_fork+0x14/0x2c
irq event stamp: 3539
hardirqs last  enabled at (3539): [<c0360d78>] 
__slab_free.isra.53+0x1ac/0x2f8
hardirqs last disabled at (3538): [<c0360c98>] 
__slab_free.isra.53+0xcc/0x2f8
softirqs last  enabled at (3246): [<c0264d10>] 
mib_counters_update+0x56c/0x58c
softirqs last disabled at (3244): [<c0365a48>] _raw_spin_lock_bh+0x14/0x5c

other info that might help us debug this:
  Possible unsafe locking scenario:

        CPU0
        ----
   lock(_xmit_ETHER#2);
   <Interrupt>
     lock(_xmit_ETHER#2);

  *** DEADLOCK ***

1 lock held by ip/1169:
  #0:  (rtnl_mutex){+.+.+.}, at: [<c02cfe50>] rtnetlink_rcv+0xc/0x24

stack backtrace:
[<c000d650>] (unwind_backtrace+0x0/0xe0) from [<c035ff90>] 
(print_usage_bug.part.25+0x20c/0x26c)
[<c035ff90>] (print_usage_bug.part.25+0x20c/0x26c) from [<c004fd0c>] 
(mark_lock+0x404/0x60c)
[<c004fd0c>] (mark_lock+0x404/0x60c) from [<c0050544>] 
(__lock_acquire+0x630/0x17cc)
[<c0050544>] (__lock_acquire+0x630/0x17cc) from [<c0051b54>] 
(lock_acquire+0x64/0x78)
[<c0051b54>] (lock_acquire+0x64/0x78) from [<c03656e0>] 
(_raw_spin_lock+0x40/0x50)
[<c03656e0>] (_raw_spin_lock+0x40/0x50) from [<c0264f2c>] 
(txq_reclaim+0x44/0x1c8)
[<c0264f2c>] (txq_reclaim+0x44/0x1c8) from [<c02650d8>] 
(txq_deinit+0x28/0xc0)
[<c02650d8>] (txq_deinit+0x28/0xc0) from [<c0265284>] 
(mv643xx_eth_stop+0x114/0x130)
[<c0265284>] (mv643xx_eth_stop+0x114/0x130) from [<c02672b0>] 
(mv643xx_eth_change_mtu+0x4c/0x80)
[<c02672b0>] (mv643xx_eth_change_mtu+0x4c/0x80) from [<c02bf1e0>] 
(dev_set_mtu+0x3c/0x70)
[<c02bf1e0>] (dev_set_mtu+0x3c/0x70) from [<c02d0148>] 
(do_setlink+0x1ac/0x794)
[<c02d0148>] (do_setlink+0x1ac/0x794) from [<c02d1624>] 
(rtnl_newlink+0x24c/0x438)
[<c02d1624>] (rtnl_newlink+0x24c/0x438) from [<c02d117c>] 
(rtnetlink_rcv_msg+0x264/0x284)
[<c02d117c>] (rtnetlink_rcv_msg+0x264/0x284) from [<c02e36e4>] 
(netlink_rcv_skb+0x50/0xac)
[<c02e36e4>] (netlink_rcv_skb+0x50/0xac) from [<c02cfe5c>] 
(rtnetlink_rcv+0x18/0x24)
[<c02cfe5c>] (rtnetlink_rcv+0x18/0x24) from [<c02e30e8>] 
(netlink_unicast+0x14c/0x1fc)
[<c02e30e8>] (netlink_unicast+0x14c/0x1fc) from [<c02e3518>] 
(netlink_sendmsg+0x2e0/0x368)
[<c02e3518>] (netlink_sendmsg+0x2e0/0x368) from [<c02ab128>] 
(sock_sendmsg+0x80/0x9c)
[<c02ab128>] (sock_sendmsg+0x80/0x9c) from [<c02ab418>] 
(__sys_sendmsg+0x1c4/0x250)
[<c02ab418>] (__sys_sendmsg+0x1c4/0x250) from [<c02ad164>] 
(sys_sendmsg+0x3c/0x60)
[<c02ad164>] (sys_sendmsg+0x3c/0x60) from [<c0009000>] 
(ret_fast_syscall+0x0/0x38)
mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full 
duplex, flow control disabled

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am 12.03.2013 10:04, schrieb Alexander Holler:

> I don't think so. Internally the driver calls mv643xx_eth_stop()
> therefore lockdep issues almost the same warning as when the interface
> is shut down (see below). And reading the code, I haven't seen how a

Btw. I consider the shutdown of the interface for just changing the MTU 
more a problem than that lockdep-warning.

E.g. I had serious problems with dhcpcd which sets the MTU to 1500 when 
an interface appears. What then happened was the following:

(1) interface comes up
(2) dhcpcd sets MTU to 1500
(3) a different MTU comes through DHCP
(4) driver shuts down if, sets MTU, turns on IF
back to (1) (endless loop)

I'm not sure if dhcpcd still does that (it's some time since I 
discovered and reported that), but I consider it a crude way to shut 
down the IF to set the MTU.

Regards,

Alexander

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html