Message ID | 1491056064-19687-1-git-send-email-zlpnobody@163.com |
---|---|
State | Accepted |
Delegated to: | Pablo Neira |
Headers | show |
On Sat, Apr 01, 2017 at 10:14:24PM +0800, Liping Zhang wrote: > @@ -1960,9 +1955,7 @@ static int ctnetlink_new_conntrack(struct net *net, struct sock *ctnl, > err = -EEXIST; > ct = nf_ct_tuplehash_to_ctrack(h); > if (!(nlh->nlmsg_flags & NLM_F_EXCL)) { > - spin_lock_bh(&nf_conntrack_expect_lock); > err = ctnetlink_change_conntrack(ct, cda); > - spin_unlock_bh(&nf_conntrack_expect_lock); We used to have a central spinlock here. spin_lock_bh(&nf_conntrack_lock); that was removed time ago, so this go converted to use nf_conntrack_expect_lock. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Pablo, 2017-04-09 5:16 GMT+08:00 Pablo Neira Ayuso <pablo@netfilter.org>: > On Sat, Apr 01, 2017 at 10:14:24PM +0800, Liping Zhang wrote: >> @@ -1960,9 +1955,7 @@ static int ctnetlink_new_conntrack(struct net *net, struct sock *ctnl, >> err = -EEXIST; >> ct = nf_ct_tuplehash_to_ctrack(h); >> if (!(nlh->nlmsg_flags & NLM_F_EXCL)) { >> - spin_lock_bh(&nf_conntrack_expect_lock); >> err = ctnetlink_change_conntrack(ct, cda); >> - spin_unlock_bh(&nf_conntrack_expect_lock); > > We used to have a central spinlock here. > > spin_lock_bh(&nf_conntrack_lock); > > that was removed time ago, so this go converted to use > nf_conntrack_expect_lock. This patch should add: Fixes: ca7433df3a67 ("netfilter: conntrack: seperate expect locking from nf_conntrack_lock") Commit ca7433df3a67 add spin_lock_bh(&nf_conntrack_expect_lock) in nf_ct_remove_expectations, but we also lock the _expect_lock before calling ctnetlink_change_conntrack, so dead lock will happen: spin_lock_bh(&nf_conntrack_expect_lock): ->err = ctnetlink_change_conntrack(ct, cda) -->ctnetlink_change_helper --->if (!strcmp(helpname, "")) nf_ct_remove_expectations() ---->spin_lock_bh(&nf_conntrack_expect_lock); //lock _expect_lock again, dead lock! Since ctnetlink_change_conntrack is unrelated to nf_conntrack_expect_lock, so remove it can fix this issue. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Apr 09, 2017 at 12:21:22PM +0800, Liping Zhang wrote: > Hi Pablo, > > 2017-04-09 5:16 GMT+08:00 Pablo Neira Ayuso <pablo@netfilter.org>: > > On Sat, Apr 01, 2017 at 10:14:24PM +0800, Liping Zhang wrote: > >> @@ -1960,9 +1955,7 @@ static int ctnetlink_new_conntrack(struct net *net, struct sock *ctnl, > >> err = -EEXIST; > >> ct = nf_ct_tuplehash_to_ctrack(h); > >> if (!(nlh->nlmsg_flags & NLM_F_EXCL)) { > >> - spin_lock_bh(&nf_conntrack_expect_lock); > >> err = ctnetlink_change_conntrack(ct, cda); > >> - spin_unlock_bh(&nf_conntrack_expect_lock); > > > > We used to have a central spinlock here. > > > > spin_lock_bh(&nf_conntrack_lock); > > > > that was removed time ago, so this go converted to use > > nf_conntrack_expect_lock. > > This patch should add: > > Fixes: ca7433df3a67 ("netfilter: conntrack: seperate expect locking > from nf_conntrack_lock") > > Commit ca7433df3a67 add spin_lock_bh(&nf_conntrack_expect_lock) in > nf_ct_remove_expectations, but we also lock the _expect_lock before calling > ctnetlink_change_conntrack, so dead lock will happen: > > spin_lock_bh(&nf_conntrack_expect_lock): > ->err = ctnetlink_change_conntrack(ct, cda) > -->ctnetlink_change_helper > --->if (!strcmp(helpname, "")) nf_ct_remove_expectations() > ---->spin_lock_bh(&nf_conntrack_expect_lock); //lock _expect_lock > again, dead lock! I agree this is fixing the deadlock but see below. > Since ctnetlink_change_conntrack is unrelated to nf_conntrack_expect_lock, > so remove it can fix this issue. But packets may be updating a conntrack at the same time that we're mangling via ctnetlink, right? -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Pablo, 2017-04-10 20:02 GMT+08:00 Pablo Neira Ayuso <pablo@netfilter.org>: [...] >> Since ctnetlink_change_conntrack is unrelated to nf_conntrack_expect_lock, >> so remove it can fix this issue. > > But packets may be updating a conntrack at the same time that we're > mangling via ctnetlink, right? Yes, but in packets processing path, we use rcu_read_lock(), so using spin_lock_bh(&nf_conntrack_expect_lock) here won't help anything. As a quick summary(just a reference): 1. For CTA_TIMEOUT, there's no problem 2. For CTA_MARK, no problem too 3. For CTA_PROTOINFO, spin_lock_bh(&ct->lock) will be held, so no problem too 4. For CTA_LABELS, it may race with packets path, but it seems not a big problem 5. For CTA_SEQ_ADJ_ORIG... we should hold &ct->lock when do updating seqadj (this one should require a new patch) 6. For CTA_HELP, updating helpinfo may be a problem(I am not sure about this part) 7. For CTA_STATUS, I think it may cause a big problem, the bit set operation via ctnetlink_change_status is not atomic, so it may clear the IPS_DYING_BIT, for example: CPU0(update CTA_STATUS) CPU1(packet path, set _DYING_) ctnetlink_change_status -- olds = ct->status -- -- set_bit(IPS_DYING_BIT... ct->status = olds | new status --> Here DYING_BIT will be cleared! But I think we can convert "ct->status |= status & ~(IPS_NAT_DONE_MASK | IPS_NAT_MASK);" to a series of atomic bit set operations to solve the 7th issue. And the issues listed above won't be solved by holding _expect_lock, so I think we should get rid of the _expect_lock at first. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Apr 01, 2017 at 10:14:24PM +0800, Liping Zhang wrote: > From: Liping Zhang <zlpnobody@gmail.com> > > Currently, ctnetlink_change_helper() is always protected by _expect_lock, > this is unnecessary, since the operations are unrelated to _expect_lock. > > Also this will cause a deadlock when deleting the helper from a conntrack, > as _expect_lock will be locked again by nf_ct_remove_expectations(): Applied. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 10, 2017 at 09:53:17PM +0800, Liping Zhang wrote: > Hi Pablo, > > 2017-04-10 20:02 GMT+08:00 Pablo Neira Ayuso <pablo@netfilter.org>: > [...] > >> Since ctnetlink_change_conntrack is unrelated to nf_conntrack_expect_lock, > >> so remove it can fix this issue. > > > > But packets may be updating a conntrack at the same time that we're > > mangling via ctnetlink, right? > > Yes, but in packets processing path, we use rcu_read_lock(), so using > spin_lock_bh(&nf_conntrack_expect_lock) here won't help anything. > > As a quick summary(just a reference): > 1. For CTA_TIMEOUT, there's no problem > 2. For CTA_MARK, no problem too > 3. For CTA_PROTOINFO, spin_lock_bh(&ct->lock) will be held, so no problem too > 4. For CTA_LABELS, it may race with packets path, but it seems not a big problem > 5. For CTA_SEQ_ADJ_ORIG... we should hold &ct->lock when do updating seqadj > (this one should require a new patch) > 6. For CTA_HELP, updating helpinfo may be a problem(I am not sure > about this part) > 7. For CTA_STATUS, I think it may cause a big problem, the bit set operation via > ctnetlink_change_status is not atomic, so it may clear the > IPS_DYING_BIT, for example: > CPU0(update CTA_STATUS) CPU1(packet path, set _DYING_) > ctnetlink_change_status -- > olds = ct->status -- > -- set_bit(IPS_DYING_BIT... > ct->status = olds | new status --> Here DYING_BIT will be cleared! > > But I think we can convert "ct->status |= status & ~(IPS_NAT_DONE_MASK > | IPS_NAT_MASK);" > to a series of atomic bit set operations to solve the 7th issue. > > And the issues listed above won't be solved by holding _expect_lock, > so I think we should get rid of the _expect_lock at first. I'm tossing this. I would like to see a patch series to address all issues with conntrack updates in one go. By when the central spinlock was removed, this was incorrectly converted to be safe. Since then on this has been broken. This should refer to patch ca7433df3a67 when fixing this. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c index 7b83bbf..f776314 100644 --- a/net/netfilter/nf_conntrack_netlink.c +++ b/net/netfilter/nf_conntrack_netlink.c @@ -1514,14 +1514,9 @@ static int ctnetlink_change_helper(struct nf_conn *ct, nf_ct_protonum(ct)); if (helper == NULL) { #ifdef CONFIG_MODULES - spin_unlock_bh(&nf_conntrack_expect_lock); - - if (request_module("nfct-helper-%s", helpname) < 0) { - spin_lock_bh(&nf_conntrack_expect_lock); + if (request_module("nfct-helper-%s", helpname) < 0) return -EOPNOTSUPP; - } - spin_lock_bh(&nf_conntrack_expect_lock); helper = __nf_conntrack_helper_find(helpname, nf_ct_l3num(ct), nf_ct_protonum(ct)); if (helper) @@ -1960,9 +1955,7 @@ static int ctnetlink_new_conntrack(struct net *net, struct sock *ctnl, err = -EEXIST; ct = nf_ct_tuplehash_to_ctrack(h); if (!(nlh->nlmsg_flags & NLM_F_EXCL)) { - spin_lock_bh(&nf_conntrack_expect_lock); err = ctnetlink_change_conntrack(ct, cda); - spin_unlock_bh(&nf_conntrack_expect_lock); if (err == 0) { nf_conntrack_eventmask_report((1 << IPCT_REPLY) | (1 << IPCT_ASSURED) | @@ -2357,11 +2350,7 @@ ctnetlink_glue_parse(const struct nlattr *attr, struct nf_conn *ct) if (ret < 0) return ret; - spin_lock_bh(&nf_conntrack_expect_lock); - ret = ctnetlink_glue_parse_ct((const struct nlattr **)cda, ct); - spin_unlock_bh(&nf_conntrack_expect_lock); - - return ret; + return ctnetlink_glue_parse_ct((const struct nlattr **)cda, ct); } static int ctnetlink_glue_exp_parse(const struct nlattr * const *cda,