diff mbox

mmotm 2010-04-28 - RCU whinges

Message ID 1272865137.2173.179.camel@edumazet-laptop
State Not Applicable, archived
Delegated to: David Miller
Headers show

Commit Message

Eric Dumazet May 3, 2010, 5:38 a.m. UTC
Le dimanche 02 mai 2010 à 13:46 -0400, Valdis.Kletnieks@vt.edu a écrit :
> On Wed, 28 Apr 2010 16:53:32 PDT, akpm@linux-foundation.org said:
> > The mm-of-the-moment snapshot 2010-04-28-16-53 has been uploaded to
> > 
> >    http://userweb.kernel.org/~akpm/mmotm/
> 
> I thought we swatted all these, hit another one...
> 
> [    9.131490] ctnetlink v0.93: registering with nfnetlink.
> [    9.131535]
> [    9.131535] ===================================================
> [    9.131704] [ INFO: suspicious rcu_dereference_check() usage. ]
> [    9.131794] ---------------------------------------------------
> [    9.131883] net/netfilter/nf_conntrack_ecache.c:88 invoked rcu_dereference_check() without protection!
> [    9.131977]
> [    9.131977] other info that might help us debug this:
> [    9.131978]
> [    9.132218]
> [    9.132219] rcu_scheduler_active = 1, debug_locks = 0
> [    9.132434] 1 lock held by swapper/1:
> [    9.132519]  #0:  (nf_ct_ecache_mutex){+.+...}, at: [<ffffffff8148922d>] nf_conntrack_register_notifier+0x1a/0x75
> [    9.132938]
> [    9.132939] stack backtrace:
> [    9.133129] Pid: 1, comm: swapper Tainted: G        W   2.6.34-rc5-mmotm0428 #1
> [    9.133220] Call Trace:
> [    9.133319]  [<ffffffff81064832>] lockdep_rcu_dereference+0xaa/0xb2
> [    9.133410]  [<ffffffff81489250>] nf_conntrack_register_notifier+0x3d/0x75
> [    9.133521]  [<ffffffff81b5a157>] ctnetlink_init+0x71/0xd5
> [    9.133627]  [<ffffffff81b5a0e6>] ? ctnetlink_init+0x0/0xd5
> [    9.133735]  [<ffffffff810001ef>] do_one_initcall+0x59/0x14e
> [    9.133843]  [<ffffffff81b2e68a>] kernel_init+0x144/0x1ce
> [    9.133949]  [<ffffffff81003414>] kernel_thread_helper+0x4/0x10
> [    9.134060]  [<ffffffff81598a40>] ? restore_args+0x0/0x30
> [    9.134196]  [<ffffffff81b2e546>] ? kernel_init+0x0/0x1ce
> [    9.134328]  [<ffffffff81003410>] ? kernel_thread_helper+0x0/0x10
> [    9.134530] ip_tables: (C) 2000-2006 Netfilter Core Team
> [    9.134655] TCP bic registered
> 

Thanks for the report !

We can use rcu_dereference_protected() in those cases.

[PATCH] net: Use rcu_dereference_protected in nf_conntrack_ecache

Writers own nf_ct_ecache_mutex.

Reported-by: Valdis Kletnieks <Valdis.Kletnieks@vt.edu>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 net/netfilter/nf_conntrack_ecache.c |   12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Eric Dumazet May 3, 2010, 5:41 a.m. UTC | #1
Le lundi 03 mai 2010 à 07:38 +0200, Eric Dumazet a écrit :
> Le dimanche 02 mai 2010 à 13:46 -0400, Valdis.Kletnieks@vt.edu a écrit :
> > On Wed, 28 Apr 2010 16:53:32 PDT, akpm@linux-foundation.org said:
> > > The mm-of-the-moment snapshot 2010-04-28-16-53 has been uploaded to
> > > 
> > >    http://userweb.kernel.org/~akpm/mmotm/
> > 
> > I thought we swatted all these, hit another one...
> > 
> > [    9.131490] ctnetlink v0.93: registering with nfnetlink.
> > [    9.131535]
> > [    9.131535] ===================================================
> > [    9.131704] [ INFO: suspicious rcu_dereference_check() usage. ]
> > [    9.131794] ---------------------------------------------------
> > [    9.131883] net/netfilter/nf_conntrack_ecache.c:88 invoked rcu_dereference_check() without protection!
> > [    9.131977]
> > [    9.131977] other info that might help us debug this:
> > [    9.131978]
> > [    9.132218]
> > [    9.132219] rcu_scheduler_active = 1, debug_locks = 0
> > [    9.132434] 1 lock held by swapper/1:
> > [    9.132519]  #0:  (nf_ct_ecache_mutex){+.+...}, at: [<ffffffff8148922d>] nf_conntrack_register_notifier+0x1a/0x75
> > [    9.132938]
> > [    9.132939] stack backtrace:
> > [    9.133129] Pid: 1, comm: swapper Tainted: G        W   2.6.34-rc5-mmotm0428 #1
> > [    9.133220] Call Trace:
> > [    9.133319]  [<ffffffff81064832>] lockdep_rcu_dereference+0xaa/0xb2
> > [    9.133410]  [<ffffffff81489250>] nf_conntrack_register_notifier+0x3d/0x75
> > [    9.133521]  [<ffffffff81b5a157>] ctnetlink_init+0x71/0xd5
> > [    9.133627]  [<ffffffff81b5a0e6>] ? ctnetlink_init+0x0/0xd5
> > [    9.133735]  [<ffffffff810001ef>] do_one_initcall+0x59/0x14e
> > [    9.133843]  [<ffffffff81b2e68a>] kernel_init+0x144/0x1ce
> > [    9.133949]  [<ffffffff81003414>] kernel_thread_helper+0x4/0x10
> > [    9.134060]  [<ffffffff81598a40>] ? restore_args+0x0/0x30
> > [    9.134196]  [<ffffffff81b2e546>] ? kernel_init+0x0/0x1ce
> > [    9.134328]  [<ffffffff81003410>] ? kernel_thread_helper+0x0/0x10
> > [    9.134530] ip_tables: (C) 2000-2006 Netfilter Core Team
> > [    9.134655] TCP bic registered
> > 
> 
> Thanks for the report !
> 
> We can use rcu_dereference_protected() in those cases.
> 
> [PATCH] net: Use rcu_dereference_protected in nf_conntrack_ecache
> 
> Writers own nf_ct_ecache_mutex.
> 
> Reported-by: Valdis Kletnieks <Valdis.Kletnieks@vt.edu>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---


Oops scratch that, I'll resend a correct version.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet May 3, 2010, 5:43 a.m. UTC | #2
Le lundi 03 mai 2010 à 07:41 +0200, Eric Dumazet a écrit :

> Oops scratch that, I'll resend a correct version.
> 
> 

Sorry, patch _is_ fine, I had one brain collapse when re-reading it, I
thought a different mutex was in use in one of the functions.


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller May 3, 2010, 5:55 a.m. UTC | #3
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Mon, 03 May 2010 07:43:56 +0200

> Le lundi 03 mai 2010 à 07:41 +0200, Eric Dumazet a écrit :
> 
>> Oops scratch that, I'll resend a correct version.
>> 
>> 
> 
> Sorry, patch _is_ fine, I had one brain collapse when re-reading it, I
> thought a different mutex was in use in one of the functions.

Ok, Patrick please review, thanks.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Valdis Kl ē tnieks May 3, 2010, 2:30 p.m. UTC | #4
On Mon, 03 May 2010 07:38:57 +0200, Eric Dumazet said:
> Le dimanche 02 mai 2010 à 13:46 -0400, Valdis.Kletnieks@vt.edu a écrit :
> > On Wed, 28 Apr 2010 16:53:32 PDT, akpm@linux-foundation.org said:
> > > The mm-of-the-moment snapshot 2010-04-28-16-53 has been uploaded to
> > > 
> > >    http://userweb.kernel.org/~akpm/mmotm/
> > 
> > I thought we swatted all these, hit another one...

> Thanks for the report !
> 
> We can use rcu_dereference_protected() in those cases.
> 
> [PATCH] net: Use rcu_dereference_protected in nf_conntrack_ecache
> 
> Writers own nf_ct_ecache_mutex.

I *really* thought we swatted a bunch of these - did the fixes not make it
into linux-next or -mm?  Your patch fixed that one, but then:

[    9.128899] Netfilter messages via NETLINK v0.30.
[    9.128919] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    9.129108] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
[    9.129110] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
[    9.129113] sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
[    9.129135] ctnetlink v0.93: registering with nfnetlink.
[    9.129452] ip_tables: (C) 2000-2006 Netfilter Core Team
[    9.129506] 
[    9.129507] ===================================================
[    9.129683] [ INFO: suspicious rcu_dereference_check() usage. ]
[    9.129777] ---------------------------------------------------
[    9.129872] net/netfilter/nf_log.c:55 invoked rcu_dereference_check() without protection!
[    9.129969] 
[    9.129969] other info that might help us debug this:
[    9.129970] 
[    9.130232] 
[    9.130232] rcu_scheduler_active = 1, debug_locks = 0
[    9.130407] 1 lock held by swapper/1:
[    9.130525]  #0:  (nf_log_mutex){+.+...}, at: [<ffffffff81481154>] nf_log_register+0x57/0x10f
[    9.130955] 
[    9.130956] stack backtrace:
[    9.131162] Pid: 1, comm: swapper Tainted: G        W   2.6.34-rc5-mmotm0428 #2
[    9.131259] Call Trace:
[    9.131370]  [<ffffffff81064832>] lockdep_rcu_dereference+0xaa/0xb2
[    9.131466]  [<ffffffff814811db>] nf_log_register+0xde/0x10f
[    9.131579]  [<ffffffff81b5ca28>] ? log_tg_init+0x0/0x29
[    9.131689]  [<ffffffff81b5ca4d>] log_tg_init+0x25/0x29
[    9.131800]  [<ffffffff810001ef>] do_one_initcall+0x59/0x14e
[    9.131912]  [<ffffffff81b2e68a>] kernel_init+0x144/0x1ce
[    9.132033]  [<ffffffff81003414>] kernel_thread_helper+0x4/0x10
[    9.132146]  [<ffffffff81598a40>] ? restore_args+0x0/0x30
[    9.132257]  [<ffffffff81b2e546>] ? kernel_init+0x0/0x1ce
[    9.132370]  [<ffffffff81003410>] ? kernel_thread_helper+0x0/0x10
[    9.132513] TCP bic registered
Eric Dumazet May 3, 2010, 2:48 p.m. UTC | #5
Le lundi 03 mai 2010 à 10:30 -0400, Valdis.Kletnieks@vt.edu a écrit :

> 
> I *really* thought we swatted a bunch of these - did the fixes not make it
> into linux-next or -mm?  Your patch fixed that one, but then:
> 
> [    9.128899] Netfilter messages via NETLINK v0.30.
> [    9.128919] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> [    9.129108] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
> [    9.129110] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
> [    9.129113] sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
> [    9.129135] ctnetlink v0.93: registering with nfnetlink.
> [    9.129452] ip_tables: (C) 2000-2006 Netfilter Core Team
> [    9.129506] 
> [    9.129507] ===================================================
> [    9.129683] [ INFO: suspicious rcu_dereference_check() usage. ]
> [    9.129777] ---------------------------------------------------
> [    9.129872] net/netfilter/nf_log.c:55 invoked rcu_dereference_check() without protection!
> [    9.129969] 
> [    9.129969] other info that might help us debug this:
> [    9.129970] 
> [    9.130232] 
> [    9.130232] rcu_scheduler_active = 1, debug_locks = 0
> [    9.130407] 1 lock held by swapper/1:
> [    9.130525]  #0:  (nf_log_mutex){+.+...}, at: [<ffffffff81481154>] nf_log_register+0x57/0x10f
> [    9.130955] 
> [    9.130956] stack backtrace:
> [    9.131162] Pid: 1, comm: swapper Tainted: G        W   2.6.34-rc5-mmotm0428 #2
> [    9.131259] Call Trace:
> [    9.131370]  [<ffffffff81064832>] lockdep_rcu_dereference+0xaa/0xb2
> [    9.131466]  [<ffffffff814811db>] nf_log_register+0xde/0x10f
> [    9.131579]  [<ffffffff81b5ca28>] ? log_tg_init+0x0/0x29
> [    9.131689]  [<ffffffff81b5ca4d>] log_tg_init+0x25/0x29
> [    9.131800]  [<ffffffff810001ef>] do_one_initcall+0x59/0x14e
> [    9.131912]  [<ffffffff81b2e68a>] kernel_init+0x144/0x1ce
> [    9.132033]  [<ffffffff81003414>] kernel_thread_helper+0x4/0x10
> [    9.132146]  [<ffffffff81598a40>] ? restore_args+0x0/0x30
> [    9.132257]  [<ffffffff81b2e546>] ? kernel_init+0x0/0x1ce
> [    9.132370]  [<ffffffff81003410>] ? kernel_thread_helper+0x0/0x10
> [    9.132513] TCP bic registered
> 

You probably know this PROVE_RCU thing is new and reserved to
developpers ?

We yet have to change all spots were a rcu_dereference() was used
without rcu_read_lock(). Not a bug by itself, just lockdep is to be
instructed not to shout.

Maybe 30 patches already in, and maybe 30 other are still needed.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/net/netfilter/nf_conntrack_ecache.c b/net/netfilter/nf_conntrack_ecache.c
index f516961..cdcc764 100644
--- a/net/netfilter/nf_conntrack_ecache.c
+++ b/net/netfilter/nf_conntrack_ecache.c
@@ -85,7 +85,8 @@  int nf_conntrack_register_notifier(struct nf_ct_event_notifier *new)
 	struct nf_ct_event_notifier *notify;
 
 	mutex_lock(&nf_ct_ecache_mutex);
-	notify = rcu_dereference(nf_conntrack_event_cb);
+	notify = rcu_dereference_protected(nf_conntrack_event_cb,
+					   lockdep_is_held(&nf_ct_ecache_mutex));
 	if (notify != NULL) {
 		ret = -EBUSY;
 		goto out_unlock;
@@ -105,7 +106,8 @@  void nf_conntrack_unregister_notifier(struct nf_ct_event_notifier *new)
 	struct nf_ct_event_notifier *notify;
 
 	mutex_lock(&nf_ct_ecache_mutex);
-	notify = rcu_dereference(nf_conntrack_event_cb);
+	notify = rcu_dereference_protected(nf_conntrack_event_cb,
+					   lockdep_is_held(&nf_ct_ecache_mutex));
 	BUG_ON(notify != new);
 	rcu_assign_pointer(nf_conntrack_event_cb, NULL);
 	mutex_unlock(&nf_ct_ecache_mutex);
@@ -118,7 +120,8 @@  int nf_ct_expect_register_notifier(struct nf_exp_event_notifier *new)
 	struct nf_exp_event_notifier *notify;
 
 	mutex_lock(&nf_ct_ecache_mutex);
-	notify = rcu_dereference(nf_expect_event_cb);
+	notify = rcu_dereference_protected(nf_expect_event_cb,
+					   lockdep_is_held(&nf_ct_ecache_mutex));
 	if (notify != NULL) {
 		ret = -EBUSY;
 		goto out_unlock;
@@ -138,7 +141,8 @@  void nf_ct_expect_unregister_notifier(struct nf_exp_event_notifier *new)
 	struct nf_exp_event_notifier *notify;
 
 	mutex_lock(&nf_ct_ecache_mutex);
-	notify = rcu_dereference(nf_expect_event_cb);
+	notify = rcu_dereference_protected(nf_expect_event_cb,
+					   lockdep_is_held(&nf_ct_ecache_mutex));
 	BUG_ON(notify != new);
 	rcu_assign_pointer(nf_expect_event_cb, NULL);
 	mutex_unlock(&nf_ct_ecache_mutex);