[2/2] rhashtable: Restore RCU marking on rhash_lock_head

Message ID	20200724011830.GB8580@gondor.apana.org.au
State	Changes Requested
Delegated to:	David Miller
Headers	show Return-Path: <netdev-owner@vger.kernel.org> Date: Fri, 24 Jul 2020 11:18:30 +1000 From: Herbert Xu <herbert@gondor.apana.org.au> To: Eric Dumazet <eric.dumazet@gmail.com>, "Gong, Sishuai" <sishuai@purdue.edu>, "tgraf@suug.ch" <tgraf@suug.ch>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, "Sousa da Fonseca, Pedro Jose" <pfonseca@purdue.edu> Subject: [PATCH 2/2] rhashtable: Restore RCU marking on rhash_lock_head Message-ID: <20200724011830.GB8580@gondor.apana.org.au> References: <20200724010823.GA8549@gondor.apana.org.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200724010823.GA8549@gondor.apana.org.au> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: netdev-owner@vger.kernel.org Precedence: bulk
Series	rhashtable: Fix unprotected RCU dereference in __rht_ptr \| expand [0/2] rhashtable: Fix unprotected RCU dereference in __rht_ptr [1/2] rhashtable: Fix unprotected RCU dereference in __rht_ptr [2/2] rhashtable: Restore RCU marking on rhash_lock_head

Hi Herbert, I love your patch! Perhaps something to improve: [auto build test WARNING on linus/master] [also build test WARNING on v5.8-rc6 next-20200723] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Herbert-Xu/rhashtable-Fix-unprotected-RCU-dereference-in-__rht_ptr/20200724-092031 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git d15be546031cf65a0fc34879beca02fd90fe7ac7 config: i386-randconfig-s002-20200724 (attached as .config) compiler: gcc-9 (Debian 9.3.0-14) 9.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.2-93-g4c6cbe55-dirty # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=i386 If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> sparse warnings: (new ones prefixed by >>) net/sched/cls_flower.c:211:19: sparse: got restricted __be16 [usertype] dst net/sched/cls_flower.c:211:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:211:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:214:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:214:21: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] dst @@ net/sched/cls_flower.c:214:21: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:214:21: sparse: got restricted __be16 [usertype] dst net/sched/cls_flower.c:214:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:214:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:214:21: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:214:51: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:215:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:215:21: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] dst @@ net/sched/cls_flower.c:215:21: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:215:21: sparse: got restricted __be16 [usertype] dst net/sched/cls_flower.c:215:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:215:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:215:21: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:215:51: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:231:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:231:20: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:231:20: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:231:20: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:231:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:231:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:232:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:232:20: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:232:20: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:232:20: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:232:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:232:20: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:233:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:233:19: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:233:19: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:233:19: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:233:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:233:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:234:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:234:19: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:234:19: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:234:19: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:234:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:234:19: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:237:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:237:21: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:237:21: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:237:21: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:237:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:237:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:237:21: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:237:51: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:238:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:238:21: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:238:21: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:238:21: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:238:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:238:21: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:238:21: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:238:51: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:769:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:769:13: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] dst @@ net/sched/cls_flower.c:769:13: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:769:13: sparse: got restricted __be16 [usertype] dst net/sched/cls_flower.c:769:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:769:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:770:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:770:13: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] dst @@ net/sched/cls_flower.c:770:13: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:770:13: sparse: got restricted __be16 [usertype] dst net/sched/cls_flower.c:770:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:770:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:769:13: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:770:13: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:777:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:777:13: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:777:13: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:777:13: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:777:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:777:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:778:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:778:13: sparse: sparse: incorrect type in argument 1 (different base types) @@ expected unsigned short [usertype] val @@ got restricted __be16 [usertype] src @@ net/sched/cls_flower.c:778:13: sparse: expected unsigned short [usertype] val net/sched/cls_flower.c:778:13: sparse: got restricted __be16 [usertype] src net/sched/cls_flower.c:778:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:778:13: sparse: sparse: cast from restricted __be16 net/sched/cls_flower.c:777:13: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:778:13: sparse: sparse: restricted __be16 degrades to integer net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1030:15: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c:1031:16: sparse: sparse: cast to restricted __be32 net/sched/cls_flower.c: note: in included file: >> include/linux/rhashtable.h:1156:13: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct rhash_lock_head **bkt @@ got struct rhash_lock_head [noderef] __rcu ** @@ >> include/linux/rhashtable.h:1156:13: sparse: expected struct rhash_lock_head **bkt >> include/linux/rhashtable.h:1156:13: sparse: got struct rhash_lock_head [noderef] __rcu ** >> include/linux/rhashtable.h:1161:23: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct rhash_lock_head [noderef] __rcu **bkt @@ got struct rhash_lock_head **bkt @@ >> include/linux/rhashtable.h:1161:23: sparse: expected struct rhash_lock_head [noderef] __rcu **bkt >> include/linux/rhashtable.h:1161:23: sparse: got struct rhash_lock_head **bkt >> include/linux/rhashtable.h:1163:9: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct rhash_lock_head [noderef] __rcu *const *bkt @@ got struct rhash_lock_head **bkt @@ >> include/linux/rhashtable.h:1163:9: sparse: expected struct rhash_lock_head [noderef] __rcu *const *bkt include/linux/rhashtable.h:1163:9: sparse: got struct rhash_lock_head **bkt include/linux/rhashtable.h:1172:41: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct rhash_lock_head [noderef] __rcu **bkt @@ got struct rhash_lock_head **bkt @@ include/linux/rhashtable.h:1172:41: sparse: expected struct rhash_lock_head [noderef] __rcu **bkt include/linux/rhashtable.h:1172:41: sparse: got struct rhash_lock_head **bkt include/linux/rhashtable.h:1174:48: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct rhash_lock_head [noderef] __rcu **bkt @@ got struct rhash_lock_head **bkt @@ include/linux/rhashtable.h:1174:48: sparse: expected struct rhash_lock_head [noderef] __rcu **bkt include/linux/rhashtable.h:1174:48: sparse: got struct rhash_lock_head **bkt include/linux/rhashtable.h:1180:25: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct rhash_lock_head [noderef] __rcu **bkt @@ got struct rhash_lock_head **bkt @@ include/linux/rhashtable.h:1180:25: sparse: expected struct rhash_lock_head [noderef] __rcu **bkt include/linux/rhashtable.h:1180:25: sparse: got struct rhash_lock_head **bkt vim +1156 include/linux/rhashtable.h 02fd97c3d4a8a14 Herbert Xu 2015-03-20 1136 3502cad73c4bbf8 Tom Herbert 2015-12-15 1137 /* Internal function, please use rhashtable_replace_fast() instead */ 3502cad73c4bbf8 Tom Herbert 2015-12-15 1138 static inline int __rhashtable_replace_fast( 3502cad73c4bbf8 Tom Herbert 2015-12-15 1139 struct rhashtable *ht, struct bucket_table *tbl, 3502cad73c4bbf8 Tom Herbert 2015-12-15 1140 struct rhash_head *obj_old, struct rhash_head *obj_new, 3502cad73c4bbf8 Tom Herbert 2015-12-15 1141 const struct rhashtable_params params) 3502cad73c4bbf8 Tom Herbert 2015-12-15 1142 { ba6306e3f648a85 Herbert Xu 2019-05-16 1143 struct rhash_lock_head **bkt; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1144 struct rhash_head __rcu **pprev; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1145 struct rhash_head *he; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1146 unsigned int hash; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1147 int err = -ENOENT; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1148 3502cad73c4bbf8 Tom Herbert 2015-12-15 1149 /* Minimally, the old and new objects must have same hash 3502cad73c4bbf8 Tom Herbert 2015-12-15 1150 * (which should mean identifiers are the same). 3502cad73c4bbf8 Tom Herbert 2015-12-15 1151 */ 3502cad73c4bbf8 Tom Herbert 2015-12-15 1152 hash = rht_head_hashfn(ht, tbl, obj_old, params); 3502cad73c4bbf8 Tom Herbert 2015-12-15 1153 if (hash != rht_head_hashfn(ht, tbl, obj_new, params)) 3502cad73c4bbf8 Tom Herbert 2015-12-15 1154 return -EINVAL; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1155 8f0db018006a421 NeilBrown 2019-04-02 @1156 bkt = rht_bucket_var(tbl, hash); 8f0db018006a421 NeilBrown 2019-04-02 1157 if (!bkt) 8f0db018006a421 NeilBrown 2019-04-02 1158 return -ENOENT; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1159 8f0db018006a421 NeilBrown 2019-04-02 1160 pprev = NULL; 149212f07856b25 NeilBrown 2019-04-02 @1161 rht_lock(tbl, bkt); 3502cad73c4bbf8 Tom Herbert 2015-12-15 1162 adc6a3ab192eb40 NeilBrown 2019-04-12 @1163 rht_for_each_from(he, rht_ptr(bkt, tbl, hash), tbl, hash) { 3502cad73c4bbf8 Tom Herbert 2015-12-15 1164 if (he != obj_old) { 3502cad73c4bbf8 Tom Herbert 2015-12-15 1165 pprev = &he->next; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1166 continue; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1167 } 3502cad73c4bbf8 Tom Herbert 2015-12-15 1168 3502cad73c4bbf8 Tom Herbert 2015-12-15 1169 rcu_assign_pointer(obj_new->next, obj_old->next); 8f0db018006a421 NeilBrown 2019-04-02 1170 if (pprev) { 3502cad73c4bbf8 Tom Herbert 2015-12-15 1171 rcu_assign_pointer(*pprev, obj_new); 149212f07856b25 NeilBrown 2019-04-02 1172 rht_unlock(tbl, bkt); 8f0db018006a421 NeilBrown 2019-04-02 1173 } else { 149212f07856b25 NeilBrown 2019-04-02 1174 rht_assign_unlock(tbl, bkt, obj_new); 8f0db018006a421 NeilBrown 2019-04-02 1175 } 3502cad73c4bbf8 Tom Herbert 2015-12-15 1176 err = 0; 8f0db018006a421 NeilBrown 2019-04-02 1177 goto unlocked; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1178 } 3502cad73c4bbf8 Tom Herbert 2015-12-15 1179 149212f07856b25 NeilBrown 2019-04-02 1180 rht_unlock(tbl, bkt); 8f0db018006a421 NeilBrown 2019-04-02 1181 8f0db018006a421 NeilBrown 2019-04-02 1182 unlocked: 3502cad73c4bbf8 Tom Herbert 2015-12-15 1183 return err; 3502cad73c4bbf8 Tom Herbert 2015-12-15 1184 } 3502cad73c4bbf8 Tom Herbert 2015-12-15 1185 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h index e3def7bbe932..9a8d4b9dde50 100644 --- a/include/linux/rhashtable.h +++ b/include/linux/rhashtable.h @@ -84,7 +84,7 @@ struct bucket_table { struct lockdep_map dep_map; - struct rhash_lock_head *buckets[] ____cacheline_aligned_in_smp; + struct rhash_lock_head __rcu *buckets[] ____cacheline_aligned_in_smp; }; /* @@ -261,13 +261,12 @@ void rhashtable_free_and_destroy(struct rhashtable *ht, void *arg); void rhashtable_destroy(struct rhashtable *ht); -struct rhash_lock_head **rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash); -struct rhash_lock_head **__rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash); -struct rhash_lock_head **rht_bucket_nested_insert(struct rhashtable *ht, - struct bucket_table *tbl, - unsigned int hash); +struct rhash_lock_head __rcu **rht_bucket_nested( + const struct bucket_table *tbl, unsigned int hash); +struct rhash_lock_head __rcu **__rht_bucket_nested( + const struct bucket_table *tbl, unsigned int hash); +struct rhash_lock_head __rcu **rht_bucket_nested_insert( + struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash); #define rht_dereference(p, ht) \ rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht)) @@ -284,21 +283,21 @@ struct rhash_lock_head **rht_bucket_nested_insert(struct rhashtable *ht, #define rht_entry(tpos, pos, member) \ ({ tpos = container_of(pos, typeof(*tpos), member); 1; }) -static inline struct rhash_lock_head *const *rht_bucket( +static inline struct rhash_lock_head __rcu *const *rht_bucket( const struct bucket_table *tbl, unsigned int hash) { return unlikely(tbl->nest) ? rht_bucket_nested(tbl, hash) : &tbl->buckets[hash]; } -static inline struct rhash_lock_head **rht_bucket_var( +static inline struct rhash_lock_head __rcu **rht_bucket_var( struct bucket_table *tbl, unsigned int hash) { return unlikely(tbl->nest) ? __rht_bucket_nested(tbl, hash) : &tbl->buckets[hash]; } -static inline struct rhash_lock_head **rht_bucket_insert( +static inline struct rhash_lock_head __rcu **rht_bucket_insert( struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash) { return unlikely(tbl->nest) ? rht_bucket_nested_insert(ht, tbl, hash) : @@ -325,7 +324,7 @@ static inline struct rhash_lock_head **rht_bucket_insert( */ static inline void rht_lock(struct bucket_table *tbl, - struct rhash_lock_head **bkt) + struct rhash_lock_head __rcu **bkt) { local_bh_disable(); bit_spin_lock(0, (unsigned long *)bkt); @@ -333,7 +332,7 @@ static inline void rht_lock(struct bucket_table *tbl, } static inline void rht_lock_nested(struct bucket_table *tbl, - struct rhash_lock_head **bucket, + struct rhash_lock_head __rcu **bucket, unsigned int subclass) { local_bh_disable(); @@ -342,7 +341,7 @@ static inline void rht_lock_nested(struct bucket_table *tbl, } static inline void rht_unlock(struct bucket_table *tbl, - struct rhash_lock_head **bkt) + struct rhash_lock_head __rcu **bkt) { lock_map_release(&tbl->dep_map); bit_spin_unlock(0, (unsigned long *)bkt); @@ -365,48 +364,41 @@ static inline struct rhash_head *__rht_ptr( * access is guaranteed, such as when destroying the table. */ static inline struct rhash_head *rht_ptr_rcu( - struct rhash_lock_head *const *p) + struct rhash_lock_head __rcu *const *bkt) { - struct rhash_lock_head __rcu *const *bkt = (void *)p; return __rht_ptr(rcu_dereference(*bkt), bkt); } static inline struct rhash_head *rht_ptr( - struct rhash_lock_head *const *p, + struct rhash_lock_head __rcu *const *bkt, struct bucket_table *tbl, unsigned int hash) { - struct rhash_lock_head __rcu *const *bkt = (void *)p; return __rht_ptr(rht_dereference_bucket(*bkt, tbl, hash), bkt); } static inline struct rhash_head *rht_ptr_exclusive( - struct rhash_lock_head *const *p) + struct rhash_lock_head __rcu *const *bkt) { - struct rhash_lock_head __rcu *const *bkt = (void *)p; return __rht_ptr(rcu_dereference_protected(*bkt, 1), bkt); } -static inline void rht_assign_locked(struct rhash_lock_head **bkt, +static inline void rht_assign_locked(struct rhash_lock_head __rcu **bkt, struct rhash_head *obj) { - struct rhash_head __rcu **p = (struct rhash_head __rcu **)bkt; - if (rht_is_a_nulls(obj)) obj = NULL; - rcu_assign_pointer(*p, (void *)((unsigned long)obj | BIT(0))); + rcu_assign_pointer(*bkt, (void *)((unsigned long)obj | BIT(0))); } static inline void rht_assign_unlock(struct bucket_table *tbl, - struct rhash_lock_head **bkt, + struct rhash_lock_head __rcu **bkt, struct rhash_head *obj) { - struct rhash_head __rcu **p = (struct rhash_head __rcu **)bkt; - if (rht_is_a_nulls(obj)) obj = NULL; lock_map_release(&tbl->dep_map); - rcu_assign_pointer(*p, obj); + rcu_assign_pointer(*bkt, (void *)obj); preempt_enable(); __release(bitlock); local_bh_enable(); @@ -594,7 +586,7 @@ static inline struct rhash_head *__rhashtable_lookup( .ht = ht, .key = key, }; - struct rhash_lock_head *const *bkt; + struct rhash_lock_head __rcu *const *bkt; struct bucket_table *tbl; struct rhash_head *he; unsigned int hash; @@ -710,7 +702,7 @@ static inline void *__rhashtable_insert_fast( .ht = ht, .key = key, }; - struct rhash_lock_head **bkt; + struct rhash_lock_head __rcu **bkt; struct rhash_head __rcu **pprev; struct bucket_table *tbl; struct rhash_head *head; @@ -996,7 +988,7 @@ static inline int __rhashtable_remove_fast_one( struct rhash_head *obj, const struct rhashtable_params params, bool rhlist) { - struct rhash_lock_head **bkt; + struct rhash_lock_head __rcu **bkt; struct rhash_head __rcu **pprev; struct rhash_head *he; unsigned int hash; diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 9f6890aedd1a..c949c1e3b87c 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -31,7 +31,7 @@ union nested_table { union nested_table __rcu *table; - struct rhash_lock_head *bucket; + struct rhash_lock_head __rcu *bucket; }; static u32 head_hashfn(struct rhashtable *ht, @@ -222,7 +222,7 @@ static struct bucket_table *rhashtable_last_table(struct rhashtable *ht, } static int rhashtable_rehash_one(struct rhashtable *ht, - struct rhash_lock_head **bkt, + struct rhash_lock_head __rcu **bkt, unsigned int old_hash) { struct bucket_table *old_tbl = rht_dereference(ht->tbl, ht); @@ -275,7 +275,7 @@ static int rhashtable_rehash_chain(struct rhashtable *ht, unsigned int old_hash) { struct bucket_table *old_tbl = rht_dereference(ht->tbl, ht); - struct rhash_lock_head **bkt = rht_bucket_var(old_tbl, old_hash); + struct rhash_lock_head __rcu **bkt = rht_bucket_var(old_tbl, old_hash); int err; if (!bkt) @@ -485,7 +485,7 @@ static int rhashtable_insert_rehash(struct rhashtable *ht, } static void *rhashtable_lookup_one(struct rhashtable *ht, - struct rhash_lock_head **bkt, + struct rhash_lock_head __rcu **bkt, struct bucket_table *tbl, unsigned int hash, const void *key, struct rhash_head *obj) { @@ -535,12 +535,10 @@ static void *rhashtable_lookup_one(struct rhashtable *ht, return ERR_PTR(-ENOENT); } -static struct bucket_table *rhashtable_insert_one(struct rhashtable *ht, - struct rhash_lock_head **bkt, - struct bucket_table *tbl, - unsigned int hash, - struct rhash_head *obj, - void *data) +static struct bucket_table *rhashtable_insert_one( + struct rhashtable *ht, struct rhash_lock_head __rcu **bkt, + struct bucket_table *tbl, unsigned int hash, struct rhash_head *obj, + void *data) { struct bucket_table *new_tbl; struct rhash_head *head; @@ -591,7 +589,7 @@ static void *rhashtable_try_insert(struct rhashtable *ht, const void *key, { struct bucket_table *new_tbl; struct bucket_table *tbl; - struct rhash_lock_head **bkt; + struct rhash_lock_head __rcu **bkt; unsigned int hash; void *data; @@ -1173,8 +1171,8 @@ void rhashtable_destroy(struct rhashtable *ht) } EXPORT_SYMBOL_GPL(rhashtable_destroy); -struct rhash_lock_head **__rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash) +struct rhash_lock_head __rcu **__rht_bucket_nested( + const struct bucket_table *tbl, unsigned int hash) { const unsigned int shift = PAGE_SHIFT - ilog2(sizeof(void *)); unsigned int index = hash & ((1 << tbl->nest) - 1); @@ -1202,10 +1200,10 @@ struct rhash_lock_head **__rht_bucket_nested(const struct bucket_table *tbl, } EXPORT_SYMBOL_GPL(__rht_bucket_nested); -struct rhash_lock_head **rht_bucket_nested(const struct bucket_table *tbl, - unsigned int hash) +struct rhash_lock_head __rcu **rht_bucket_nested( + const struct bucket_table *tbl, unsigned int hash) { - static struct rhash_lock_head *rhnull; + static struct rhash_lock_head __rcu *rhnull; if (!rhnull) INIT_RHT_NULLS_HEAD(rhnull); @@ -1213,9 +1211,8 @@ struct rhash_lock_head **rht_bucket_nested(const struct bucket_table *tbl, } EXPORT_SYMBOL_GPL(rht_bucket_nested); -struct rhash_lock_head **rht_bucket_nested_insert(struct rhashtable *ht, - struct bucket_table *tbl, - unsigned int hash) +struct rhash_lock_head __rcu **rht_bucket_nested_insert( + struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash) { const unsigned int shift = PAGE_SHIFT - ilog2(sizeof(void *)); unsigned int index = hash & ((1 << tbl->nest) - 1);

[2/2] rhashtable: Restore RCU marking on rhash_lock_head

Commit Message

Comments

Patch