From patchwork Wed Jul 15 19:54:01 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 29830 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@bilbo.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from ozlabs.org (ozlabs.org [203.10.76.45]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mx.ozlabs.org", Issuer "CA Cert Signing Authority" (verified OK)) by bilbo.ozlabs.org (Postfix) with ESMTPS id 9EA8DB7083 for ; Thu, 16 Jul 2009 05:54:38 +1000 (EST) Received: by ozlabs.org (Postfix) id 90624DDDA0; Thu, 16 Jul 2009 05:54:38 +1000 (EST) Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by ozlabs.org (Postfix) with ESMTP id EFBC5DDD1B for ; Thu, 16 Jul 2009 05:54:37 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756034AbZGOTyN (ORCPT ); Wed, 15 Jul 2009 15:54:13 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756087AbZGOTyN (ORCPT ); Wed, 15 Jul 2009 15:54:13 -0400 Received: from gw1.cosmosbay.com ([212.99.114.194]:38262 "EHLO gw1.cosmosbay.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756034AbZGOTyM (ORCPT ); Wed, 15 Jul 2009 15:54:12 -0400 Received: from [127.0.0.1] (localhost [127.0.0.1]) by gw1.cosmosbay.com (8.13.7/8.13.7) with ESMTP id n6FJs1tn005687; Wed, 15 Jul 2009 21:54:01 +0200 Message-ID: <4A5E33D9.2030602@gmail.com> Date: Wed, 15 Jul 2009 21:54:01 +0200 From: Eric Dumazet User-Agent: Thunderbird 2.0.0.22 (Windows/20090605) MIME-Version: 1.0 To: Patrick McHardy CC: David Miller , netdev@vger.kernel.org, paulmck@linux.vnet.ibm.com Subject: [PATCH] net: nf_conntrack_alloc() fixes References: <20090707.191424.167842005.davem@davemloft.net> <4A5441A0.3050504@gmail.com> <4A5581C5.5070409@gmail.com> <20090711.202727.18146102.davem@davemloft.net> <4A598BAB.6030400@gmail.com> <4A5DCB7C.9000502@gmail.com> <4A5DF5B4.5090809@trash.net> In-Reply-To: <4A5DF5B4.5090809@trash.net> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (gw1.cosmosbay.com [0.0.0.0]); Wed, 15 Jul 2009 21:54:02 +0200 (CEST) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Patrick McHardy a écrit : > Eric Dumazet wrote: >> [PATCH] net: nf_conntrack_alloc() should not use kmem_cache_zalloc() >> >> When a slab cache uses SLAB_DESTROY_BY_RCU, we must be careful when allocating >> objects, since slab allocator could give a freed object still used by lockless >> readers. >> >> In particular, nf_conntrack RCU lookups rely on ct->tuplehash[xxx].hnnode.next >> being always valid (ie containing a valid 'nulls' value, or a valid pointer to next >> object in hash chain.) >> >> kmem_cache_zalloc() setups object with NULL values, but a NULL value is not valid >> for ct->tuplehash[xxx].hnnode.next. >> >> Fix is to call kmem_cache_alloc() and do the zeroing ourself. > > I think this is still racy, please see below: Nice catch indeed ! > >> diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c >> index 7508f11..23feafa 100644 >> --- a/net/netfilter/nf_conntrack_core.c >> +++ b/net/netfilter/nf_conntrack_core.c >> @@ -561,17 +561,28 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, >> } >> } >> >> - ct = kmem_cache_zalloc(nf_conntrack_cachep, gfp); >> + /* >> + * Do not use kmem_cache_zalloc(), as this cache uses >> + * SLAB_DESTROY_BY_RCU. >> + */ >> + ct = kmem_cache_alloc(nf_conntrack_cachep, gfp); >> if (ct == NULL) { >> pr_debug("nf_conntrack_alloc: Can't alloc conntrack.\n"); >> atomic_dec(&net->ct.count); >> return ERR_PTR(-ENOMEM); >> } >> - > > __nf_conntrack_find() on another CPU finds the entry at this point. > >> + /* >> + * Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next >> + * and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. >> + */ >> + memset(&ct->tuplehash[IP_CT_DIR_MAX], 0, >> + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX])); >> spin_lock_init(&ct->lock); >> atomic_set(&ct->ct_general.use, 1); > > nf_conntrack_find_get() successfully tries to atomic_inc_not_zero() > at this point, following by another tuple comparison which is also > successful. > > Am I missing something? I think we need to make sure the reference > count is not increased until the new tuples are visible. Yes you are right, and Documentation/RCU/rculist_nulls.txt should be updated to reflect this as well (in insert algo, must use smp_wmb() between obj->key assignment and refcnt assigment) We'll have to change socket allocation too, this will be addressed by a followup patch Thanks Patrick ! [PATCH] net: nf_conntrack_alloc() fixes When a slab cache uses SLAB_DESTROY_BY_RCU, we must be careful when allocating objects, since slab allocator could give a freed object still used by lockless readers. In particular, nf_conntrack RCU lookups rely on ct->tuplehash[xxx].hnnode.next being always valid (ie containing a valid 'nulls' value, or a valid pointer to next object in hash chain.) kmem_cache_zalloc() setups object with NULL values, but a NULL value is not valid for ct->tuplehash[xxx].hnnode.next. Fix is to call kmem_cache_alloc() and do the zeroing ourself. As spotted by Patrick, we also need to make sure lookup keys are committed to memory before setting refcount to 1, or a lockless reader could get a reference on the old version of the object. Its key re-check could then pass the barrier. Signed-off-by: Eric Dumazet --- Documentation/RCU/rculist_nulls.txt | 7 ++++++- net/netfilter/nf_conntrack_core.c | 21 ++++++++++++++++++--- 2 files changed, 24 insertions(+), 4 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/Documentation/RCU/rculist_nulls.txt b/Documentation/RCU/rculist_nulls.txt index 93cb28d..18f9651 100644 --- a/Documentation/RCU/rculist_nulls.txt +++ b/Documentation/RCU/rculist_nulls.txt @@ -83,11 +83,12 @@ not detect it missed following items in original chain. obj = kmem_cache_alloc(...); lock_chain(); // typically a spin_lock() obj->key = key; -atomic_inc(&obj->refcnt); /* * we need to make sure obj->key is updated before obj->next + * or obj->refcnt */ smp_wmb(); +atomic_set(&obj->refcnt, 1); hlist_add_head_rcu(&obj->obj_node, list); unlock_chain(); // typically a spin_unlock() @@ -159,6 +160,10 @@ out: obj = kmem_cache_alloc(cachep); lock_chain(); // typically a spin_lock() obj->key = key; +/* + * changes to obj->key must be visible before refcnt one + */ +smp_wmb(); atomic_set(&obj->refcnt, 1); /* * insert obj in RCU way (readers might be traversing chain) diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c index 7508f11..b5869b9 100644 --- a/net/netfilter/nf_conntrack_core.c +++ b/net/netfilter/nf_conntrack_core.c @@ -561,23 +561,38 @@ struct nf_conn *nf_conntrack_alloc(struct net *net, } } - ct = kmem_cache_zalloc(nf_conntrack_cachep, gfp); + /* + * Do not use kmem_cache_zalloc(), as this cache uses + * SLAB_DESTROY_BY_RCU. + */ + ct = kmem_cache_alloc(nf_conntrack_cachep, gfp); if (ct == NULL) { pr_debug("nf_conntrack_alloc: Can't alloc conntrack.\n"); atomic_dec(&net->ct.count); return ERR_PTR(-ENOMEM); } - + /* + * Let ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.next + * and ct->tuplehash[IP_CT_DIR_REPLY].hnnode.next unchanged. + */ + memset(&ct->tuplehash[IP_CT_DIR_MAX], 0, + sizeof(*ct) - offsetof(struct nf_conn, tuplehash[IP_CT_DIR_MAX])); spin_lock_init(&ct->lock); - atomic_set(&ct->ct_general.use, 1); ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple = *orig; + ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode.pprev = NULL; ct->tuplehash[IP_CT_DIR_REPLY].tuple = *repl; + ct->tuplehash[IP_CT_DIR_REPLY].hnnode.pprev = NULL; /* Don't set timer yet: wait for confirmation */ setup_timer(&ct->timeout, death_by_timeout, (unsigned long)ct); #ifdef CONFIG_NET_NS ct->ct_net = net; #endif + /* + * changes to lookup keys must be done before setting refcnt to 1 + */ + smp_wmb(); + atomic_set(&ct->ct_general.use, 1); return ct; } EXPORT_SYMBOL_GPL(nf_conntrack_alloc);