| Message ID | 7353554.n89QJXU3eh@gentoovm |
|---|---|
| State | Not Applicable |
| Headers | show |
On Thursday 30 August 2012 04:50:09 you wrote: > Not sure what you mean, you're still crashing with the patch below, > right? > > My proposal is to give a try to the ecache patch, that requires > removing the previous patch. Apologies for the confusion; the patch quoted is essentially the first patch you provided me, with my changes to make it work in 3.4.10 *plus* the deletion of the change to nf_conntrack_ecache.h where your patch deleted the nf_ct_is_dying() check (i.e I have this check left in) - with this modification, I find that conntrackd is well-behaved and I have thus far not successfully caused a kernel panic. Having tested your latest patch, I can also confirm that it also does not crash, including at exhaustion of the conntrack table. In terms of overall stability, I would presume your latest patch is superior to the previous (i.e. what I attached most recently) ? Kind Regards, Oliver -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 30, 2012 at 05:09:01AM +0200, Oliver wrote: > On Thursday 30 August 2012 04:50:09 you wrote: > > Not sure what you mean, you're still crashing with the patch below, > > right? > > > > My proposal is to give a try to the ecache patch, that requires > > removing the previous patch. > > Apologies for the confusion; the patch quoted is essentially the first patch > you provided me, with my changes to make it work in 3.4.10 *plus* the deletion > of the change to nf_conntrack_ecache.h where your patch deleted the > nf_ct_is_dying() check (i.e I have this check left in) - with this > modification, I find that conntrackd is well-behaved and I have thus far not > successfully caused a kernel panic. > > Having tested your latest patch, I can also confirm that it also does not > crash, including at exhaustion of the conntrack table. > > In terms of overall stability, I would presume your latest patch is superior > to the previous (i.e. what I attached most recently) ? Yes, I prefer the second patch. There is still races in the first patch I sent you, harder to trigger, but still there. There are several cleanups I'd like to recover from the first patch though. Would you help testing them? Thanks a lot for testing. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thursday 30 August 2012 12:34:37 you wrote: > Yes, I prefer the second patch. There is still races in the first > patch I sent you, harder to trigger, but still there. > > There are several cleanups I'd like to recover from the first patch > though. Would you help testing them? > > Thanks a lot for testing. HI Pablo, Yep, I'd be happy to test. I've also uncovered a new issue: I have two Active- Active machines (conntrackd running NOTRACK mode with both External and Internal cache disabled) In kernel 3.2 this pair works asymmetric and issue-free. Upgrade it to 3.4 and it immediately has around 50% failure of TCP connection attempts on systems behind them - ICMP on the other hand is flawless, DNS lookups also are OK so I *believe* that UDP may also be performing well - I've no idea where to even look on this one so any insight would be most appreciated. Kind Regards, Oliver -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thursday 30 August 2012 14:28:20 Oliver wrote: > Yep, I'd be happy to test. I've also uncovered a new issue: I have two > Active- Active machines (conntrackd running NOTRACK mode with both External > and Internal cache disabled) > > In kernel 3.2 this pair works asymmetric and issue-free. Upgrade it to 3.4 > and it immediately has around 50% failure of TCP connection attempts on > systems behind them - ICMP on the other hand is flawless, DNS lookups also > are OK so I *believe* that UDP may also be performing well - I've no idea > where to even look on this one so any insight would be most appreciated. > > Kind Regards, > Oliver Another thing that just entered my mind: I configured raw/PREROUTING to -j CT --notrack TCP port 80 (source and dest) on the appropriate interfaces and this resulted in total loss despite the fact that I had --ctstate UNTRACKED set to ACCEPT - and again, this behaviour only occurs under 3.4.[9|10] (probably earlier too but I didn't test) -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Oliver, On Thu, Aug 30, 2012 at 02:28:20PM +0200, Oliver wrote: > On Thursday 30 August 2012 12:34:37 you wrote: > > Yes, I prefer the second patch. There is still races in the first > > patch I sent you, harder to trigger, but still there. > > > > There are several cleanups I'd like to recover from the first patch > > though. Would you help testing them? > > > > Thanks a lot for testing. > > HI Pablo, > > Yep, I'd be happy to test. I've also uncovered a new issue: I have two Active- > Active machines (conntrackd running NOTRACK mode with both External and > Internal cache disabled) Thanks. I'll send you patches then. > In kernel 3.2 this pair works asymmetric and issue-free. Upgrade it to 3.4 and > it immediately has around 50% failure of TCP connection attempts on systems > behind them - ICMP on the other hand is flawless, DNS lookups also are OK so I > *believe* that UDP may also be performing well - I've no idea where to even > look on this one so any insight would be most appreciated. Unfortunately, asymmetric active-active is a crazy setup for conntrack (documentation already discuss this). The state synchronization that we are doing is asynchronous, so state-updates race with TCP packet. We don't support this, sorry. We can support active-active with hash-based load-sharing with the cluster match / arptables, it seems more sane to me, theory is described here: http://1984.lsi.us.es/~pablo/docs/intcomp09.pdf However, there is not documentation yet on how to make it. Last time I looked at existing HA daemons, I didn't find that they support active-active setup very well, so they require some changes / we need some small new HA daemon for this. I need to work on active/active load-sharing, to fully documented and support it. That's not in top of my priority list at the moment though. Another (simpler) alternative is, in case your firewall have two IPs, to statically distribute the load between your firewalls by assigning different gateway IP to your client nodes. That should not be hard to deploy. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thursday 30 August 2012 18:22:48 you wrote: > Unfortunately, asymmetric active-active is a crazy setup for conntrack > (documentation already discuss this). The state synchronization that > we are doing is asynchronous, so state-updates race with TCP packet. > We don't support this, sorry. The environment does fulfil the assumptions necessary for replication to happen within the handshake so under 3.2 there is no issue with handshakes completing under an asymmetric path. Nonetheless, what doesn't make sense is that this operates under 3.2 and not 3.4 - also is the fact that having a "-j CT --notrack" on specific traffic (i.e. asymmetric should not matter because there is no stateful tracking) Kind Regards, Oliver -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 30, 2012 at 07:49:24PM +0200, Oliver wrote: > On Thursday 30 August 2012 18:22:48 you wrote: > > Unfortunately, asymmetric active-active is a crazy setup for conntrack > > (documentation already discuss this). The state synchronization that > > we are doing is asynchronous, so state-updates race with TCP packet. > > We don't support this, sorry. > > The environment does fulfil the assumptions necessary for replication to happen > within the handshake so under 3.2 there is no issue with handshakes completing > under an asymmetric path. Interesting, how are those assumptions fulfilled? > Nonetheless, what doesn't make sense is that this operates under 3.2 and not > 3.4 - also is the fact that having a "-j CT --notrack" on specific traffic (i.e. > asymmetric should not matter because there is no stateful tracking) Agreed. But I don't come with any netfilter change that may result in that problem you're reporting. You'll have to debug this and get back to me with more information. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thursday 30 August 2012 20:39:50 Pablo Neira Ayuso wrote: > Interesting, how are those assumptions fulfilled? Well, timing of course ;) - essentially, traffic paths are ensured longer than the actual time for replication of conntrack state. > Agreed. But I don't come with any netfilter change that may result in > that problem you're reporting. You'll have to debug this and get back > to me with more information. You can disregard this, turned out to be due to the unfortunate fact that net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is of course replaced by net.netfilter.nf_conntrack_tcp_be_liberal under 3.4 Please feel free to send me your latest rework of the patch and I will be happy to test it out. Kind Regards, Oliver -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Aug 31, 2012 at 02:19:36AM +0200, Oliver wrote: > On Thursday 30 August 2012 20:39:50 Pablo Neira Ayuso wrote: > > Interesting, how are those assumptions fulfilled? > > Well, timing of course ;) - essentially, traffic paths are ensured longer than > the actual time for replication of conntrack state. I see. > > Agreed. But I don't come with any netfilter change that may result in > > that problem you're reporting. You'll have to debug this and get back > > to me with more information. > > You can disregard this, turned out to be due to the unfortunate fact that > net.ipv4.netfilter.ip_conntrack_tcp_be_liberal is of course replaced by > net.netfilter.nf_conntrack_tcp_be_liberal under 3.4 $ ls /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal Probably you forgot to set CONFIG_NF_CONNTRACK_PROC_COMPAT=y We haven't remove it yet, although it should be bad to schedule this for removal. > Please feel free to send me your latest rework of the patch and I will be > happy to test it out. Will do. -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c index 729f157..5c274f3 100644 --- a/net/netfilter/nf_conntrack_core.c +++ b/net/netfilter/nf_conntrack_core.c @@ -250,7 +250,8 @@ static void death_by_event(unsigned long ul_conntrack) struct nf_conn *ct = (void *)ul_conntrack; struct net *net = nf_ct_net(ct); - if (nf_conntrack_event(IPCT_DESTROY, ct) < 0) { + if (!test_bit(IPS_DYING_BIT, &ct->status) && + nf_conntrack_event(IPCT_DESTROY, ct) < 0) { /* bad luck, let's retry again */ ct->timeout.expires = jiffies + (random32() % net->ct.sysctl_events_retry_timeout);