Patchwork net: fix setsockopt() locking errors

login
register
mail settings
Submitter Vegard Nossum
Date Jan. 24, 2009, 10:49 p.m.
Message ID <20090124224930.GA4456@localhost.localdomain>
Download mbox | patch
Permalink /patch/20183/
State Changes Requested
Delegated to: David Miller
Headers show

Comments

Vegard Nossum - Jan. 24, 2009, 10:49 p.m.
Hi,

This survives basic testing here, but I don't know what that counts for
when I couldn't reproduce the lockdep report in the first place. Please
review.


Vegard


From cc8bcd1c4fd219a31d6d191aefa4b4b57dadb9b0 Mon Sep 17 00:00:00 2001
From: Vegard Nossum <vegard.nossum@gmail.com>
Date: Sat, 24 Jan 2009 22:44:16 +0100
Subject: [PATCH] net: fix setsockopt() locking errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Martin MOKREJŠ <mmokrejs@ribosome.natur.cuni.cz> reported:
> =======================================================
> [ INFO: possible circular locking dependency detected ]
> 2.6.29-rc2-git1 #1
> -------------------------------------------------------
> tcpdump/3734 is trying to acquire lock:
>  (&mm->mmap_sem){----}, at: [<c1053294>] might_fault+0x30/0x6b
>
> but task is already holding lock:
>  (sk_lock-AF_PACKET){--..}, at: [<c12798c8>] sock_setsockopt+0x12b/0x4a4
>
> which lock already depends on the new lock.

It turns out that sock_setsockopt() is calling copy_from_user() while
holding the lock on the socket. We fix it by splitting the ioctl code
so that one switch handles the ioctls that have their own code for
reading from userspace, and one switch handles the cases that require
no additional reading.

Reported-by: Martin MOKREJŠ <mmokrejs@ribosome.natur.cuni.cz>
Signed-off-by: Vegard Nossum <vegard.nossum@gmail.com>
---
 net/core/sock.c |  134 +++++++++++++++++++++++++++++++++++-------------------
 1 files changed, 87 insertions(+), 47 deletions(-)
Jarek Poplawski - Jan. 26, 2009, 11:50 a.m.
On 24-01-2009 23:49, Vegard Nossum wrote:
> Hi,
> 
> This survives basic testing here, but I don't know what that counts for
> when I couldn't reproduce the lockdep report in the first place. Please
> review.
> 
> 
> Vegard
> 
> 
> From cc8bcd1c4fd219a31d6d191aefa4b4b57dadb9b0 Mon Sep 17 00:00:00 2001
> From: Vegard Nossum <vegard.nossum@gmail.com>
> Date: Sat, 24 Jan 2009 22:44:16 +0100
> Subject: [PATCH] net: fix setsockopt() locking errors
> MIME-Version: 1.0
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: 8bit
> 
> Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz> reported:
>> =======================================================
>> [ INFO: possible circular locking dependency detected ]
>> 2.6.29-rc2-git1 #1
>> -------------------------------------------------------
>> tcpdump/3734 is trying to acquire lock:
>>  (&mm->mmap_sem){----}, at: [<c1053294>] might_fault+0x30/0x6b
>>
>> but task is already holding lock:
>>  (sk_lock-AF_PACKET){--..}, at: [<c12798c8>] sock_setsockopt+0x12b/0x4a4
>>
>> which lock already depends on the new lock.
> 
> It turns out that sock_setsockopt() is calling copy_from_user() while
> holding the lock on the socket.

I guess it has been like this for some time, so it would be nice to
mention what scenario happens here, or IOW what exactly needs to get
these locks in reverse order.

> We fix it by splitting the ioctl code
> so that one switch handles the ioctls that have their own code for
> reading from userspace, and one switch handles the cases that require
> no additional reading.
> 
> Reported-by: Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz>
> Signed-off-by: Vegard Nossum <vegard.nossum@gmail.com>
> ---
>  net/core/sock.c |  134 +++++++++++++++++++++++++++++++++++-------------------
>  1 files changed, 87 insertions(+), 47 deletions(-)
> 
> diff --git a/net/core/sock.c b/net/core/sock.c
> index f3a0d08..6bd618d 100644
> --- a/net/core/sock.c
> +++ b/net/core/sock.c
> @@ -424,6 +424,80 @@ out:
>  	return ret;
>  }
>  
> +static int sock_linger(struct sock *sk, char __user *optval, int optlen)
...
> +static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
> +{
> +	int ret;
> +	long rcvtimeo;
> +
> +	ret = sock_set_timeout(&rcvtimeo, optval, optlen);

A check for error is needed here and below.

> +
> +	lock_sock(sk);
> +	sk->sk_rcvtimeo = rcvtimeo;
> +	release_sock(sk);
> +
> +	return ret;
> +}
> +
> +static int sock_set_sndtimeo(struct sock *sk, char __user *optval, int optlen)
> +{
> +	int ret;
> +	long sndtimeo;
> +
> +	ret = sock_set_timeout(&sndtimeo, optval, optlen);
> +
> +	lock_sock(sk);
> +	sk->sk_sndtimeo = sndtimeo;
> +	release_sock(sk);
> +
> +	return ret;
> +}
...

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller - Jan. 26, 2009, 8:33 p.m.
From: Jarek Poplawski <jarkao2@gmail.com>
Date: Mon, 26 Jan 2009 11:50:12 +0000

> On 24-01-2009 23:49, Vegard Nossum wrote:
> > +static int sock_linger(struct sock *sk, char __user *optval, int optlen)
> ...
> > +static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
> > +{
> > +	int ret;
> > +	long rcvtimeo;
> > +
> > +	ret = sock_set_timeout(&rcvtimeo, optval, optlen);
> 
> A check for error is needed here and below.

Right, you cannot continue and update the socket state if this
sock_set_timeout() call returns an error.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Martin MOKREJŠ - Jan. 26, 2009, 9:30 p.m.
The patch really did not help:
http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
Martin

Jarek Poplawski wrote:
> On 24-01-2009 23:49, Vegard Nossum wrote:
>> Hi,
>>
>> This survives basic testing here, but I don't know what that counts for
>> when I couldn't reproduce the lockdep report in the first place. Please
>> review.
>>
>>
>> Vegard
>>
>>
>> From cc8bcd1c4fd219a31d6d191aefa4b4b57dadb9b0 Mon Sep 17 00:00:00 2001
>> From: Vegard Nossum <vegard.nossum@gmail.com>
>> Date: Sat, 24 Jan 2009 22:44:16 +0100
>> Subject: [PATCH] net: fix setsockopt() locking errors
>> MIME-Version: 1.0
>> Content-Type: text/plain; charset=utf-8
>> Content-Transfer-Encoding: 8bit
>>
>> Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz> reported:
>>> =======================================================
>>> [ INFO: possible circular locking dependency detected ]
>>> 2.6.29-rc2-git1 #1
>>> -------------------------------------------------------
>>> tcpdump/3734 is trying to acquire lock:
>>>  (&mm->mmap_sem){----}, at: [<c1053294>] might_fault+0x30/0x6b
>>>
>>> but task is already holding lock:
>>>  (sk_lock-AF_PACKET){--..}, at: [<c12798c8>] sock_setsockopt+0x12b/0x4a4
>>>
>>> which lock already depends on the new lock.
>> It turns out that sock_setsockopt() is calling copy_from_user() while
>> holding the lock on the socket.
> 
> I guess it has been like this for some time, so it would be nice to
> mention what scenario happens here, or IOW what exactly needs to get
> these locks in reverse order.
> 
>> We fix it by splitting the ioctl code
>> so that one switch handles the ioctls that have their own code for
>> reading from userspace, and one switch handles the cases that require
>> no additional reading.
>>
>> Reported-by: Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz>
>> Signed-off-by: Vegard Nossum <vegard.nossum@gmail.com>
>> ---
>>  net/core/sock.c |  134 +++++++++++++++++++++++++++++++++++-------------------
>>  1 files changed, 87 insertions(+), 47 deletions(-)
>>
>> diff --git a/net/core/sock.c b/net/core/sock.c
>> index f3a0d08..6bd618d 100644
>> --- a/net/core/sock.c
>> +++ b/net/core/sock.c
>> @@ -424,6 +424,80 @@ out:
>>  	return ret;
>>  }
>>  
>> +static int sock_linger(struct sock *sk, char __user *optval, int optlen)
> ...
>> +static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
>> +{
>> +	int ret;
>> +	long rcvtimeo;
>> +
>> +	ret = sock_set_timeout(&rcvtimeo, optval, optlen);
> 
> A check for error is needed here and below.
> 
[cut]
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jarek Poplawski - Jan. 27, 2009, 8:45 a.m.
On Mon, Jan 26, 2009 at 10:30:30PM +0100, Martin MOKREJŠ wrote:
> The patch really did not help:
> http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
> Martin

Actually, there is a little change: the warning triggerd in another
place (sock_setsockopt() -> sk_attach_filter()). So we could go deeper
with these changes, but I'm not sure this is the right way to fix.

It looks like the scenario is very old, but probably wasn't reported
(maybe there is some lockdep improvement):

A) sys_mmap2() -> mm->mmap_sem -> packet_mmap() -> sk_lock
B) sock_setsockopt() -> sk_lock -> copy_from_user() -> mm->mmap_sem

packet_mmap() (net/packet/af_packet.c) seems to be the only place in
net to implement mmap method, and using this lock order btw. On the
other hand copy_from_user() could be more popular under sk_lock, and
I'm not sure these changes are necessary.

Since I don't know enough neither sock/packet nor sys_mmap, I guess
some advice would be precious. It looks like Peter Zijlstra solved
similar problems in nfs, so I CC him.

Thanks,
Jarek P.

> 
> Jarek Poplawski wrote:
> > On 24-01-2009 23:49, Vegard Nossum wrote:
> >> Hi,
> >>
> >> This survives basic testing here, but I don't know what that counts for
> >> when I couldn't reproduce the lockdep report in the first place. Please
> >> review.
> >>
> >>
> >> Vegard
> >>
> >>
> >> From cc8bcd1c4fd219a31d6d191aefa4b4b57dadb9b0 Mon Sep 17 00:00:00 2001
> >> From: Vegard Nossum <vegard.nossum@gmail.com>
> >> Date: Sat, 24 Jan 2009 22:44:16 +0100
> >> Subject: [PATCH] net: fix setsockopt() locking errors
> >> MIME-Version: 1.0
> >> Content-Type: text/plain; charset=utf-8
> >> Content-Transfer-Encoding: 8bit
> >>
> >> Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz> reported:
> >>> =======================================================
> >>> [ INFO: possible circular locking dependency detected ]
> >>> 2.6.29-rc2-git1 #1
> >>> -------------------------------------------------------
> >>> tcpdump/3734 is trying to acquire lock:
> >>>  (&mm->mmap_sem){----}, at: [<c1053294>] might_fault+0x30/0x6b
> >>>
> >>> but task is already holding lock:
> >>>  (sk_lock-AF_PACKET){--..}, at: [<c12798c8>] sock_setsockopt+0x12b/0x4a4
> >>>
> >>> which lock already depends on the new lock.
> >> It turns out that sock_setsockopt() is calling copy_from_user() while
> >> holding the lock on the socket.
> > 
> > I guess it has been like this for some time, so it would be nice to
> > mention what scenario happens here, or IOW what exactly needs to get
> > these locks in reverse order.
> > 
> >> We fix it by splitting the ioctl code
> >> so that one switch handles the ioctls that have their own code for
> >> reading from userspace, and one switch handles the cases that require
> >> no additional reading.
> >>
> >> Reported-by: Martin MOKREJ.  <mmokrejs@ribosome.natur.cuni.cz>
> >> Signed-off-by: Vegard Nossum <vegard.nossum@gmail.com>
> >> ---
> >>  net/core/sock.c |  134 +++++++++++++++++++++++++++++++++++-------------------
> >>  1 files changed, 87 insertions(+), 47 deletions(-)
> >>
> >> diff --git a/net/core/sock.c b/net/core/sock.c
> >> index f3a0d08..6bd618d 100644
> >> --- a/net/core/sock.c
> >> +++ b/net/core/sock.c
> >> @@ -424,6 +424,80 @@ out:
> >>  	return ret;
> >>  }
> >>  
> >> +static int sock_linger(struct sock *sk, char __user *optval, int optlen)
> > ...
> >> +static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
> >> +{
> >> +	int ret;
> >> +	long rcvtimeo;
> >> +
> >> +	ret = sock_set_timeout(&rcvtimeo, optval, optlen);
> > 
> > A check for error is needed here and below.
> > 
> [cut]
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Peter Zijlstra - Jan. 27, 2009, 8:52 a.m.
On Tue, 2009-01-27 at 08:45 +0000, Jarek Poplawski wrote:
> On Mon, Jan 26, 2009 at 10:30:30PM +0100, Martin MOKREJŠ wrote:
> > The patch really did not help:
> > http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
> > Martin
> 
> Actually, there is a little change: the warning triggerd in another
> place (sock_setsockopt() -> sk_attach_filter()). So we could go deeper
> with these changes, but I'm not sure this is the right way to fix.
> 
> It looks like the scenario is very old, but probably wasn't reported
> (maybe there is some lockdep improvement):

Yes, they likely are very old, and yes we added a lockdep annotation to
copy_to/from_user() to catch these.

> A) sys_mmap2() -> mm->mmap_sem -> packet_mmap() -> sk_lock
> B) sock_setsockopt() -> sk_lock -> copy_from_user() -> mm->mmap_sem
> 
> packet_mmap() (net/packet/af_packet.c) seems to be the only place in
> net to implement mmap method, and using this lock order btw. On the
> other hand copy_from_user() could be more popular under sk_lock, and
> I'm not sure these changes are necessary.
> 
> Since I don't know enough neither sock/packet nor sys_mmap, I guess
> some advice would be precious. It looks like Peter Zijlstra solved
> similar problems in nfs, so I CC him.

The NFS/sunrpc case was special in that it did copy_to/from_kernel, that
is, it never actually touched user memory -- we taught the might_fault()
annotation about that.

Can't you simply do the copy_from_user() before you take the sk_lock?

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jarek Poplawski - Jan. 27, 2009, 9:08 a.m.
On Tue, Jan 27, 2009 at 09:52:49AM +0100, Peter Zijlstra wrote:
> On Tue, 2009-01-27 at 08:45 +0000, Jarek Poplawski wrote:
> > On Mon, Jan 26, 2009 at 10:30:30PM +0100, Martin MOKREJŠ wrote:
> > > The patch really did not help:
> > > http://bugzilla.kernel.org/show_bug.cgi?id=12515#c5
> > > Martin
> > 
> > Actually, there is a little change: the warning triggerd in another
> > place (sock_setsockopt() -> sk_attach_filter()). So we could go deeper
> > with these changes, but I'm not sure this is the right way to fix.
> > 
> > It looks like the scenario is very old, but probably wasn't reported
> > (maybe there is some lockdep improvement):
> 
> Yes, they likely are very old, and yes we added a lockdep annotation to
> copy_to/from_user() to catch these.
> 
> > A) sys_mmap2() -> mm->mmap_sem -> packet_mmap() -> sk_lock
> > B) sock_setsockopt() -> sk_lock -> copy_from_user() -> mm->mmap_sem
> > 
> > packet_mmap() (net/packet/af_packet.c) seems to be the only place in
> > net to implement mmap method, and using this lock order btw. On the
> > other hand copy_from_user() could be more popular under sk_lock, and
> > I'm not sure these changes are necessary.
> > 
> > Since I don't know enough neither sock/packet nor sys_mmap, I guess
> > some advice would be precious. It looks like Peter Zijlstra solved
> > similar problems in nfs, so I CC him.
> 
> The NFS/sunrpc case was special in that it did copy_to/from_kernel, that
> is, it never actually touched user memory -- we taught the might_fault()
> annotation about that.
> 
> Can't you simply do the copy_from_user() before you take the sk_lock?
> 

Since it's really needed, and Vegard started doing it like this, I
guess he will try to add the missing pieces.

Thanks again,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Vegard Nossum - Jan. 27, 2009, 4:25 p.m.
On Mon, Jan 26, 2009 at 9:33 PM, David Miller <davem@davemloft.net> wrote:
> From: Jarek Poplawski <jarkao2@gmail.com>
> Date: Mon, 26 Jan 2009 11:50:12 +0000
>
>> On 24-01-2009 23:49, Vegard Nossum wrote:
>> > +static int sock_linger(struct sock *sk, char __user *optval, int optlen)
>> ...
>> > +static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
>> > +{
>> > +   int ret;
>> > +   long rcvtimeo;
>> > +
>> > +   ret = sock_set_timeout(&rcvtimeo, optval, optlen);
>>
>> A check for error is needed here and below.
>
> Right, you cannot continue and update the socket state if this
> sock_set_timeout() call returns an error.
>

Argh, thanks both. What a stupid mistake. What is the point in fixing
something if the fix introduces a different kind of error? :-(

I will fix the patch and also check for other places (like that
attach_filter) that need to be fixed.


Vegard

Patch

diff --git a/net/core/sock.c b/net/core/sock.c
index f3a0d08..6bd618d 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -424,6 +424,80 @@  out:
 	return ret;
 }
 
+static int sock_linger(struct sock *sk, char __user *optval, int optlen)
+{
+	struct linger ling;
+
+	if (optlen < sizeof(ling))
+		return -EINVAL; /* 1003.1g */
+	if (copy_from_user(&ling, optval, sizeof(ling)))
+		return -EFAULT;
+
+	lock_sock(sk);
+
+	if (!ling.l_onoff)
+		sock_reset_flag(sk, SOCK_LINGER);
+	else {
+#if (BITS_PER_LONG == 32)
+		if ((unsigned int) ling.l_linger >= MAX_SCHEDULE_TIMEOUT/HZ)
+			sk->sk_lingertime = MAX_SCHEDULE_TIMEOUT;
+		else
+#endif
+			sk->sk_lingertime = (unsigned int) ling.l_linger * HZ;
+		sock_set_flag(sk, SOCK_LINGER);
+	}
+
+	release_sock(sk);
+
+	return 0;
+}
+
+static int sock_set_rcvtimeo(struct sock *sk, char __user *optval, int optlen)
+{
+	int ret;
+	long rcvtimeo;
+
+	ret = sock_set_timeout(&rcvtimeo, optval, optlen);
+
+	lock_sock(sk);
+	sk->sk_rcvtimeo = rcvtimeo;
+	release_sock(sk);
+
+	return ret;
+}
+
+static int sock_set_sndtimeo(struct sock *sk, char __user *optval, int optlen)
+{
+	int ret;
+	long sndtimeo;
+
+	ret = sock_set_timeout(&sndtimeo, optval, optlen);
+
+	lock_sock(sk);
+	sk->sk_sndtimeo = sndtimeo;
+	release_sock(sk);
+
+	return ret;
+}
+
+static int sock_attach_filter(struct sock *sk, char __user *optval, int optlen)
+{
+	int ret;
+	struct sock_fprog fprog;
+
+	if (optlen != sizeof(struct sock_fprog))
+		return -EINVAL;
+
+	if (copy_from_user(&fprog, optval, sizeof(fprog)))
+		return -EFAULT;
+
+	lock_sock(sk);
+	ret = sk_attach_filter(&fprog, sk);
+	release_sock(sk);
+
+	return ret;
+}
+
 static inline void sock_valbool_flag(struct sock *sk, int bit, int valbool)
 {
 	if (valbool)
@@ -440,18 +514,27 @@  static inline void sock_valbool_flag(struct sock *sk, int bit, int valbool)
 int sock_setsockopt(struct socket *sock, int level, int optname,
 		    char __user *optval, int optlen)
 {
-	struct sock *sk=sock->sk;
+	struct sock *sk = sock->sk;
 	int val;
 	int valbool;
-	struct linger ling;
 	int ret = 0;
 
 	/*
-	 *	Options without arguments
+	 * Options with special locking requirements
 	 */
 
-	if (optname == SO_BINDTODEVICE)
+	switch (optname) {
+	case SO_BINDTODEVICE:
 		return sock_bindtodevice(sk, optval, optlen);
+	case SO_LINGER:
+		return sock_linger(sk, optval, optlen);
+	case SO_RCVTIMEO:
+		return sock_set_rcvtimeo(sk, optval, optlen);
+	case SO_SNDTIMEO:
+		return sock_set_sndtimeo(sk, optval, optlen);
+	case SO_ATTACH_FILTER:
+		return sock_attach_filter(sk, optval, optlen);
+	}
 
 	if (optlen < sizeof(int))
 		return -EINVAL;
@@ -573,28 +656,6 @@  set_rcvbuf:
 			ret = -EPERM;
 		break;
 
-	case SO_LINGER:
-		if (optlen < sizeof(ling)) {
-			ret = -EINVAL;	/* 1003.1g */
-			break;
-		}
-		if (copy_from_user(&ling,optval,sizeof(ling))) {
-			ret = -EFAULT;
-			break;
-		}
-		if (!ling.l_onoff)
-			sock_reset_flag(sk, SOCK_LINGER);
-		else {
-#if (BITS_PER_LONG == 32)
-			if ((unsigned int)ling.l_linger >= MAX_SCHEDULE_TIMEOUT/HZ)
-				sk->sk_lingertime = MAX_SCHEDULE_TIMEOUT;
-			else
-#endif
-				sk->sk_lingertime = (unsigned int)ling.l_linger * HZ;
-			sock_set_flag(sk, SOCK_LINGER);
-		}
-		break;
-
 	case SO_BSDCOMPAT:
 		sock_warn_obsolete_bsdism("setsockopt");
 		break;
@@ -627,27 +688,6 @@  set_rcvbuf:
 		sk->sk_rcvlowat = val ? : 1;
 		break;
 
-	case SO_RCVTIMEO:
-		ret = sock_set_timeout(&sk->sk_rcvtimeo, optval, optlen);
-		break;
-
-	case SO_SNDTIMEO:
-		ret = sock_set_timeout(&sk->sk_sndtimeo, optval, optlen);
-		break;
-
-	case SO_ATTACH_FILTER:
-		ret = -EINVAL;
-		if (optlen == sizeof(struct sock_fprog)) {
-			struct sock_fprog fprog;
-
-			ret = -EFAULT;
-			if (copy_from_user(&fprog, optval, sizeof(fprog)))
-				break;
-
-			ret = sk_attach_filter(&fprog, sk);
-		}
-		break;
-
 	case SO_DETACH_FILTER:
 		ret = sk_detach_filter(sk);
 		break;