diff mbox

Freeing dst when the reference count <0 causes general protection fault, it could be a major security flaw as rogue app can modify dst to crash kernel.

Message ID 1410596833-2548-1-git-send-email-shakilk1729@gmail.com
State Rejected, archived
Delegated to: David Miller
Headers show

Commit Message

Shakil A Khan Sept. 13, 2014, 8:27 a.m. UTC 
Signed-off-by: Shakil A Khan <shakilk1729@gmail.com>
---
 net/core/dst.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Comments

Hannes Frederic Sowa Sept. 13, 2014, 10:13 a.m. UTC | #1 
On Sa, 2014-09-13 at 01:27 -0700, Shakil A Khan wrote:
> Signed-off-by: Shakil A Khan <shakilk1729@gmail.com>
> ---
>  net/core/dst.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/net/core/dst.c b/net/core/dst.c
> index a028409..6a848b0 100644
> --- a/net/core/dst.c
> +++ b/net/core/dst.c
> @@ -284,7 +284,10 @@ void dst_release(struct dst_entry *dst)
>  		int newrefcnt;
>  
>  		newrefcnt = atomic_dec_return(&dst->__refcnt);
> -		WARN_ON(newrefcnt < 0);
> +
> +		if (WARN(newrefcnt < 0, "dst reference count less than zero"))
> +			return;
> +
>  		if (unlikely(dst->flags & DST_NOCACHE) && !newrefcnt)
>  			call_rcu(&dst->rcu_head, dst_destroy_rcu);
>  	}

So change this to a memory leak which also has reliable concerns...

You could just change this to a BUG_ON, but this will also allow a rogue
app to kill the kernel.


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet Sept. 13, 2014, 11:50 a.m. UTC | #2 
On Sat, 2014-09-13 at 01:27 -0700, Shakil A Khan wrote:
> Signed-off-by: Shakil A Khan <shakilk1729@gmail.com>
> ---
>  net/core/dst.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/net/core/dst.c b/net/core/dst.c
> index a028409..6a848b0 100644
> --- a/net/core/dst.c
> +++ b/net/core/dst.c
> @@ -284,7 +284,10 @@ void dst_release(struct dst_entry *dst)
>  		int newrefcnt;
>  
>  		newrefcnt = atomic_dec_return(&dst->__refcnt);
> -		WARN_ON(newrefcnt < 0);
> +
> +		if (WARN(newrefcnt < 0, "dst reference count less than zero"))
> +			return;
> +
>  		if (unlikely(dst->flags & DST_NOCACHE) && !newrefcnt)
>  			call_rcu(&dst->rcu_head, dst_destroy_rcu);
>  	}


A rogue application can not do trigger this, unless a major bug in the
kernel exists.

Instead of trying to hide the kernel bug, we need to fix it.

Can you describe how this could trigger with a pristine kernel ?



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Shakil A Khan Sept. 13, 2014, 6:35 p.m. UTC | #3 
On Saturday, September 13, 2014 04:50:22 AM Eric Dumazet wrote:
> On Sat, 2014-09-13 at 01:27 -0700, Shakil A Khan wrote:
> > Signed-off-by: Shakil A Khan <shakilk1729@gmail.com>
> > ---
> > 
> >  net/core/dst.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/net/core/dst.c b/net/core/dst.c
> > index a028409..6a848b0 100644
> > --- a/net/core/dst.c
> > +++ b/net/core/dst.c
> > @@ -284,7 +284,10 @@ void dst_release(struct dst_entry *dst)
> > 
> >  		int newrefcnt;
> >  		
> >  		newrefcnt = atomic_dec_return(&dst->__refcnt);
> > 
> > -		WARN_ON(newrefcnt < 0);
> > +
> > +		if (WARN(newrefcnt < 0, "dst reference count less than zero"))
> > +			return;
> > +
> > 
> >  		if (unlikely(dst->flags & DST_NOCACHE) && !newrefcnt)
> >  		
> >  			call_rcu(&dst->rcu_head, dst_destroy_rcu);
> >  	
> >  	}
> 
> A rogue application can not do trigger this, unless a major bug in the
> kernel exists.

Please check this kernel trace. It is able to crash the kernel.

general protection fault: 0000 [#1] PREEMPT SMP
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss oid_registry 
nfsv4 nfs fscache lockd sunrpc tun nbd ipmi_si ipmi_watchdog ipmi_devintf 
ipmi_msghandler xt_mark xt_owner ipt_MASQUERADE xt_physdev xt_state xt_LOG 
iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat 
nf_conntrack iptable_filter ip_tables xen_acpi_processor xen_pciback 
xen_netback xen_blkback xen_gntalloc xen_gntdev xenfs xen_privcmd bridge stp 
llc ipv6 ext4 jbd2 freq_table mperf coretemp crc32c_intel ghash_clmulni_intel 
microcode pcspkr sb_edac edac_core lpc_ich mfd_core i2c_i801 sg ioatdma igb 
dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif 
aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci 
libahci isci libsas scsi_transport_sas megaraid_sas wmi dm_mirror 
dm_region_hash dm_log dm_mod [last unloaded: iTCO_vendor_support]
CPU: 6 PID: 15324 Comm: XXXX Not tainted 3.10.45-xen.322.17.41238 #1
Hardware name: McAfee, Inc. ATD-6000/S4600LH...., BIOS 
SE5C600.86B.02.01.0002.082220131453 08/22/2013
task: ffff882bc6255000 ti: ffff882bc61aa000 task.ti: ffff882bc61aa000
RIP: e030:[<ffffffff8148473f>]  [<ffffffff8148473f>] ipv4_dst_destroy+0x3b/0x77
RSP: e02b:ffff882bc61abb48  EFLAGS: 00010296
RAX: dead000000200200 RBX: ffff882bc625bc80 RCX: 0001338a9b7110db
RDX: dead000000100100 RSI: ffffffff82267e30 RDI: 00000000000003f4
RBP: ffff882bc61abb58 R08: 00000000d5d6df8b R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff820e5880 R14: ffff88070e584b80 R15: 0000000000000000
FS:  00007f8d3fff2700(0000) GS:ffff88081e6c0000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000031d0e36ac0 CR3: 0000002db0165000 CR4: 0000000000042660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffff882bc61abb58 ffff882bc625bc80 ffff882bc61abb88 ffffffff8145bfc5
 ffff882bc61abba8 ffff882bc625bc80 0000000000000000 ffff882bc625bc80
 ffff882bc61abba8 ffffffff8145c2c5 ffff88070e584b80 ffff882b991c2300
Call Trace:
 [<ffffffff8145bfc5>] dst_destroy+0x29/0xbd
 [<ffffffff8145c2c5>] dst_release+0x58/0x67
 [<ffffffff814ad539>] tcp_v4_do_rcv+0x11b/0x287
 [<ffffffff81443e24>] __release_sock+0x7c/0xe7
 [<ffffffff81443ebd>] release_sock+0x2e/0x7c
 [<ffffffff81499cdf>] tcp_sendmsg+0xe0/0xd41
 [<ffffffff814bf369>] inet_sendmsg+0x7d/0xa0
 [<ffffffff8143d794>] sock_aio_write+0x159/0x17d
 [<ffffffff81005859>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8116d12b>] do_sync_write+0x7f/0xa7
 [<ffffffff8116d392>] ? rw_verify_area+0x56/0xd5
 [<ffffffff8116d555>] vfs_write+0x144/0x170
 [<ffffffff8116d977>] SyS_write+0x54/0x8f
 [<ffffffff810d078f>] ? __audit_syscall_exit+0x20c/0x29c
 [<ffffffff8151e959>] system_call_fastpath+0x16/0x1b
Code: fb 48 8d 87 b0 00 00 00 48 39 87 b0 00 00 00 74 4f 48 c7 c7 04 8f 3c 82 
e8 32 23 09 00 48 8b 93 b0 00 00 00 48 8b 83 b8 00 00 00 <48> 89 42 08 48 89 
10 48 b9 00 01 10 00 00 00 ad de 48 89 8b b0
RIP  [<ffffffff8148473f>] ipv4_dst_destroy+0x3b/0x77
 RSP <ffff882bc61abb48>
---[ end trace d56f90482c47af91 ]---
Kernel panic - not syncing: Fatal exception in interrupt

 
> 
> Instead of trying to hide the kernel bug, we need to fix it.
    There are two problems with this. First the list has somehow got out of 
sync in terms of number of delete and allocate, so we need to fix that.
But at the same time refcount if <0 signifies its been already freed so we need 
not free up.
We need fix for both and my patch targets later as my system works fine with 
this patch.
> 
> Can you describe how this could trigger with a pristine kernel ?
This is triggered with custom software to imitate malware traffic(Kernel was not 
having any custom patch whatsoever, it was a pristine kernel 3.10.45).
Point is if an application can crash this, then it would be big security flaw 
not exactly similar but like SSL love bug, which can be exploited to bring 
down systems.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller Sept. 13, 2014, 7:32 p.m. UTC | #4 
From: Shakil k <shakilk1729@gmail.com>
Date: Sat, 13 Sep 2014 10:46:39 -0700

> On Sat, Sep 13, 2014 at 4:50 AM, Eric Dumazet <eric.dumazet@gmail.com>
> wrote:
> 
>> Can you describe how this could trigger with a pristine kernel ?
>> This can be reproduced with our custom network traffic to simulate malware.
>>
> Point is the user can modify certain packets and can cause a kernel crash
> causing blackout to all the linux boxes hosting services :(

Eric is kindly asking you exactly how to reproduce the crash, so he
can 1) fix it and 2) generate a test case that gets run all the time
in the future.

Please answer his question directly instead of steering the conversation
endless towards other aspects.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet Sept. 14, 2014, 3:54 a.m. UTC | #5 
On Sat, 2014-09-13 at 11:35 -0700, shakil A Khan wrote:
> On Saturday, September 13, 2014 04:50:22 AM Eric Dumazet wrote:
> > On Sat, 2014-09-13 at 01:27 -0700, Shakil A Khan wrote:
> > > Signed-off-by: Shakil A Khan <shakilk1729@gmail.com>
> > > ---
> > > 
> > >  net/core/dst.c | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/net/core/dst.c b/net/core/dst.c
> > > index a028409..6a848b0 100644
> > > --- a/net/core/dst.c
> > > +++ b/net/core/dst.c
> > > @@ -284,7 +284,10 @@ void dst_release(struct dst_entry *dst)
> > > 
> > >  		int newrefcnt;
> > >  		
> > >  		newrefcnt = atomic_dec_return(&dst->__refcnt);
> > > 
> > > -		WARN_ON(newrefcnt < 0);
> > > +
> > > +		if (WARN(newrefcnt < 0, "dst reference count less than zero"))
> > > +			return;
> > > +
> > > 
> > >  		if (unlikely(dst->flags & DST_NOCACHE) && !newrefcnt)
> > >  		
> > >  			call_rcu(&dst->rcu_head, dst_destroy_rcu);
> > >  	
> > >  	}
> > 
> > A rogue application can not do trigger this, unless a major bug in the
> > kernel exists.
> 
> Please check this kernel trace. It is able to crash the kernel.
> 
> general protection fault: 0000 [#1] PREEMPT SMP
> Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss oid_registry 
> nfsv4 nfs fscache lockd sunrpc tun nbd ipmi_si ipmi_watchdog ipmi_devintf 
> ipmi_msghandler xt_mark xt_owner ipt_MASQUERADE xt_physdev xt_state xt_LOG 
> iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat 
> nf_conntrack iptable_filter ip_tables xen_acpi_processor xen_pciback 
> xen_netback xen_blkback xen_gntalloc xen_gntdev xenfs xen_privcmd bridge stp 
> llc ipv6 ext4 jbd2 freq_table mperf coretemp crc32c_intel ghash_clmulni_intel 
> microcode pcspkr sb_edac edac_core lpc_ich mfd_core i2c_i801 sg ioatdma igb 
> dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif 
> aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci 
> libahci isci libsas scsi_transport_sas megaraid_sas wmi dm_mirror 
> dm_region_hash dm_log dm_mod [last unloaded: iTCO_vendor_support]
> CPU: 6 PID: 15324 Comm: XXXX Not tainted 3.10.45-xen.322.17.41238 #1
> Hardware name: McAfee, Inc. ATD-6000/S4600LH...., BIOS 
> SE5C600.86B.02.01.0002.082220131453 08/22/2013
> task: ffff882bc6255000 ti: ffff882bc61aa000 task.ti: ffff882bc61aa000
> RIP: e030:[<ffffffff8148473f>]  [<ffffffff8148473f>] ipv4_dst_destroy+0x3b/0x77
> RSP: e02b:ffff882bc61abb48  EFLAGS: 00010296
> RAX: dead000000200200 RBX: ffff882bc625bc80 RCX: 0001338a9b7110db
> RDX: dead000000100100 RSI: ffffffff82267e30 RDI: 00000000000003f4
> RBP: ffff882bc61abb58 R08: 00000000d5d6df8b R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> R13: ffffffff820e5880 R14: ffff88070e584b80 R15: 0000000000000000
> FS:  00007f8d3fff2700(0000) GS:ffff88081e6c0000(0000) knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000031d0e36ac0 CR3: 0000002db0165000 CR4: 0000000000042660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Stack:
>  ffff882bc61abb58 ffff882bc625bc80 ffff882bc61abb88 ffffffff8145bfc5
>  ffff882bc61abba8 ffff882bc625bc80 0000000000000000 ffff882bc625bc80
>  ffff882bc61abba8 ffffffff8145c2c5 ffff88070e584b80 ffff882b991c2300
> Call Trace:
>  [<ffffffff8145bfc5>] dst_destroy+0x29/0xbd
>  [<ffffffff8145c2c5>] dst_release+0x58/0x67
>  [<ffffffff814ad539>] tcp_v4_do_rcv+0x11b/0x287
>  [<ffffffff81443e24>] __release_sock+0x7c/0xe7
>  [<ffffffff81443ebd>] release_sock+0x2e/0x7c
>  [<ffffffff81499cdf>] tcp_sendmsg+0xe0/0xd41
>  [<ffffffff814bf369>] inet_sendmsg+0x7d/0xa0
>  [<ffffffff8143d794>] sock_aio_write+0x159/0x17d
>  [<ffffffff81005859>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
>  [<ffffffff8116d12b>] do_sync_write+0x7f/0xa7
>  [<ffffffff8116d392>] ? rw_verify_area+0x56/0xd5
>  [<ffffffff8116d555>] vfs_write+0x144/0x170
>  [<ffffffff8116d977>] SyS_write+0x54/0x8f
>  [<ffffffff810d078f>] ? __audit_syscall_exit+0x20c/0x29c
>  [<ffffffff8151e959>] system_call_fastpath+0x16/0x1b
> Code: fb 48 8d 87 b0 00 00 00 48 39 87 b0 00 00 00 74 4f 48 c7 c7 04 8f 3c 82 
> e8 32 23 09 00 48 8b 93 b0 00 00 00 48 8b 83 b8 00 00 00 <48> 89 42 08 48 89 
> 10 48 b9 00 01 10 00 00 00 ad de 48 89 8b b0
> RIP  [<ffffffff8148473f>] ipv4_dst_destroy+0x3b/0x77
>  RSP <ffff882bc61abb48>
> ---[ end trace d56f90482c47af91 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> 
>  
> > 
> > Instead of trying to hide the kernel bug, we need to fix it.
>     There are two problems with this. First the list has somehow got out of 
> sync in terms of number of delete and allocate, so we need to fix that.
> But at the same time refcount if <0 signifies its been already freed so we need 
> not free up.
> We need fix for both and my patch targets later as my system works fine with 
> this patch.
> > 
> > Can you describe how this could trigger with a pristine kernel ?
> This is triggered with custom software to imitate malware traffic(Kernel was not 
> having any custom patch whatsoever, it was a pristine kernel 3.10.45).
> Point is if an application can crash this, then it would be big security flaw 
> not exactly similar but like SSL love bug, which can be exploited to bring 
> down systems.

3.10.45 is old, you are a bit late, as we already spent a good amount of
time fixing all this about 3 months ago.

3.10.45 misses this fix :

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7f502361531e9eecb396cf99bdc9e9a59f7ebd7f

And this one :

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f88649721268999bdff09777847080a52004f691

And also :

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9709674e68646cee5a24e3000b3558d25412203a

All these should already be in 3.10.54

I suggest you try a 3.10.54 kernel. If it still crashes, then try a 3.16
kernel. 

Then, we'll investigate, _if_ needed.

Thanks.



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/net/core/dst.c b/net/core/dst.c
index a028409..6a848b0 100644
--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -284,7 +284,10 @@  void dst_release(struct dst_entry *dst)
 		int newrefcnt;
 
 		newrefcnt = atomic_dec_return(&dst->__refcnt);
-		WARN_ON(newrefcnt < 0);
+
+		if (WARN(newrefcnt < 0, "dst reference count less than zero"))
+			return;
+
 		if (unlikely(dst->flags & DST_NOCACHE) && !newrefcnt)
 			call_rcu(&dst->rcu_head, dst_destroy_rcu);
 	}