diff mbox

[Bug,14470] New: freez in TCP stack

Message ID alpine.DEB.2.00.0911262352050.24189@melkinpaasi.cs.helsinki.fi
State Accepted, archived
Delegated to: David Miller
Headers show

Commit Message

Ilpo Järvinen Nov. 26, 2009, 9:54 p.m. UTC
On Thu, 29 Oct 2009, David Miller wrote:

> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Thu, 29 Oct 2009 06:59:41 +0100
> 
> > [PATCH] tcp: clear retrans hints in tcp_send_synack()
> > 
> > There is a small possibility the skb we unlink from write queue 
> > is still referenced by retrans hints.
> > 
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> 
> So, this would only be true if we were dealing with a data
> packet here.  We're not, this is a SYN+ACK which happens to
> be cloned in the write queue.
> 
> The hint SKBs pointers can only point to real data packets.
> 
> And we're only dealing with data packets once we enter established
> state, and when we enter established by definition we have unlinked
> and freed up any SYN and SYN+ACK SKBs in the write queue.

How about this then... Does the original reporter have NFS in use?

[PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)

Eric Dumazet mentioned in a context of another problem:

"Well, it seems NFS reuses its socket, so maybe we miss some
cleaning as spotted in this old patch"

I've not check under which conditions that actually happens but
if true, we need to make sure we don't accidently leave stale
hints behind when the write queue had to be purged (whether reusing
with NFS can actually happen if purging took place is something I'm
not sure of).

...At least it compiles.

Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
 include/net/tcp.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

Comments

David Miller Nov. 26, 2009, 11:37 p.m. UTC | #1
From: "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Thu, 26 Nov 2009 23:54:53 +0200 (EET)

> How about this then... Does the original reporter have NFS in use?
> 
> [PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)

I must be getting old and senile, but I specifically remembered that
we prevented a socket from ever being bound again once it has been
bound one time specifically so we didn't have to deal with issues
like this.

I really don't think it's valid for NFS to reuse the socket structure
like this over and over again.  And that's why only NFS can reproduce
this, the interfaces provided userland can't actually go through this
sequence after a socket goes down one time all the way to close.

Do we really want to audit each and every odd member of the socket
structure from the generic portion all the way down to INET and
TCP specifics to figure out what needs to get zero'd out?

So much relies upon the one-time full zero out during sock allocation.

Let's fix NFS instead.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet Nov. 27, 2009, 6:17 a.m. UTC | #2
David Miller a écrit :

> I must be getting old and senile, but I specifically remembered that
> we prevented a socket from ever being bound again once it has been
> bound one time specifically so we didn't have to deal with issues
> like this.
> 
> I really don't think it's valid for NFS to reuse the socket structure
> like this over and over again.  And that's why only NFS can reproduce
> this, the interfaces provided userland can't actually go through this
> sequence after a socket goes down one time all the way to close.
> 
> Do we really want to audit each and every odd member of the socket
> structure from the generic portion all the way down to INET and
> TCP specifics to figure out what needs to get zero'd out?

An audit is always welcomed, we might find bugs :)

> 
> So much relies upon the one-time full zero out during sock allocation.
> 
> Let's fix NFS instead.

bugzilla reference : http://bugzilla.kernel.org/show_bug.cgi?id=14580

Trond said :
  NFS MUST reuse the same port because on most servers, the replay cache is keyed
  to the port number. In other words, when we replay an RPC call, the server will
  only recognise it as a replay if it originates from the same port.
  See http://www.connectathon.org/talks96/werme1.html


Please note the socket stays bound to a given local port.

We want to connect() it to a possible other target, that's all.

In NFS case 'other target' is in fact the same target, but this
is a special case of a more general one.

Hmm... if an application wants to keep a local port for itself (not
allowing another one to get this (ephemeral ?) port during the 
close()/socket()/bind() window), this is the only way.
TCP state machine allows this IMHO.

google for "tcp AF_UNSPEC connect" to find many references and man pages
for this stuff.

http://kerneltrap.org/Linux/Connect_Specification_versus_Man_Page

How other Unixes / OS handle this ?
How many applications use this trick ?


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller Dec. 2, 2009, 11:10 p.m. UTC | #3
From: "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Thu, 26 Nov 2009 23:54:53 +0200 (EET)

> [PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)
> 
> Eric Dumazet mentioned in a context of another problem:
> 
> "Well, it seems NFS reuses its socket, so maybe we miss some
> cleaning as spotted in this old patch"
> 
> I've not check under which conditions that actually happens but
> if true, we need to make sure we don't accidently leave stale
> hints behind when the write queue had to be purged (whether reusing
> with NFS can actually happen if purging took place is something I'm
> not sure of).
> 
> ...At least it compiles.
> 
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>

I think this is a good safety net even if it doesn't specifically
fix a specific problem.

But I'd like to see this patch tested by the person seeing the
problem so we can know whether that is fixed or not.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller Dec. 3, 2009, 6:24 a.m. UTC | #4
From: "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Thu, 26 Nov 2009 23:54:53 +0200 (EET)

> [PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)

Ok, since Linus just released 2.6.32 I'm tossing this into net-next-2.6
so it gets wider exposure.

I still want to see test results from the bug reporter, and if it fixes
things we can toss this into -stable too.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Andrew Morton March 18, 2010, 9:04 p.m. UTC | #5
On Wed, 02 Dec 2009 22:24:46 -0800 (PST)
David Miller <davem@davemloft.net> wrote:

> From: "Ilpo J__rvinen" <ilpo.jarvinen@helsinki.fi>
> Date: Thu, 26 Nov 2009 23:54:53 +0200 (EET)
> 
> > [PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)
> 
> Ok, since Linus just released 2.6.32 I'm tossing this into net-next-2.6
> so it gets wider exposure.
> 
> I still want to see test results from the bug reporter, and if it fixes
> things we can toss this into -stable too.

Despite my request to take this to email, quite a few people have been
jumping onto this report via bugzilla:
http://bugzilla.kernel.org/show_bug.cgi?id=14470

Bit of a pita, but it'd be worth someone taking a look to ensure that
we're all talking about the same bug.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Ilpo Järvinen March 19, 2010, 3:52 p.m. UTC | #6
On Thu, 18 Mar 2010, Andrew Morton wrote:

> On Wed, 02 Dec 2009 22:24:46 -0800 (PST)
> David Miller <davem@davemloft.net> wrote:
> 
> > From: "Ilpo J__rvinen" <ilpo.jarvinen@helsinki.fi>
> > Date: Thu, 26 Nov 2009 23:54:53 +0200 (EET)
> > 
> > > [PATCH] tcp: clear hints to avoid a stale one (nfs only affected?)
> > 
> > Ok, since Linus just released 2.6.32 I'm tossing this into net-next-2.6
> > so it gets wider exposure.
> > 
> > I still want to see test results from the bug reporter, and if it fixes
> > things we can toss this into -stable too.
> 
> Despite my request to take this to email, quite a few people have been
> jumping onto this report via bugzilla:
> http://bugzilla.kernel.org/show_bug.cgi?id=14470
> 
> Bit of a pita, but it'd be worth someone taking a look to ensure that
> we're all talking about the same bug.

Could one try with this debug patch:

http://marc.info/?l=linux-kernel&m=126624014117610&w=2

It should prevent crashing too.
diff mbox

Patch

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 03a49c7..6b13faa 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1228,6 +1228,7 @@  static inline void tcp_write_queue_purge(struct sock *sk)
 	while ((skb = __skb_dequeue(&sk->sk_write_queue)) != NULL)
 		sk_wmem_free_skb(sk, skb);
 	sk_mem_reclaim(sk);
+	tcp_clear_all_retrans_hints(tcp_sk(sk));
 }
 
 static inline struct sk_buff *tcp_write_queue_head(struct sock *sk)