Patchwork what's causing "ip_rt_bug"?

login
register
mail settings
Submitter Eric Dumazet
Date June 17, 2011, 8:36 p.m.
Message ID <1308342993.3539.30.camel@edumazet-laptop>
Download mbox | patch
Permalink /patch/100858/
State RFC
Delegated to: David Miller
Headers show

Comments

Eric Dumazet - June 17, 2011, 8:36 p.m.
Le vendredi 17 juin 2011 à 22:00 +0200, Tomasz Chmielewski a écrit :
> I have a system pushing around 800 Mbit/s, ~130 kpps.
> 
> It uses 2.6.35.12 kernel.
> 
> 
> Several times a minute, I can see entries like (a.b.c.d - IP of this system):
> 
> Jun 18 02:39:19 KOR-SV22 kernel: [37187.665951] ip_rt_bug: 110.x.x.x -> a.b.c.d, ?
> Jun 18 02:39:19 KOR-SV22 kernel: [37187.685419] ip_rt_bug: 110.x.x.x -> a.b.c.d, ?
> Jun 18 02:40:31 KOR-SV22 kernel: [37259.199315] ip_rt_bug: 124.x.x.x -> a.b.c.d, ?
> Jun 18 02:40:36 KOR-SV22 kernel: [37263.828000] ip_rt_bug: 124.x.x.x -> a.b.c.d, ?
> Jun 18 02:44:16 KOR-SV22 kernel: [37484.120689] ip_rt_bug: 110.x.x.x -> a.b.c.d, ?
> Jun 18 02:44:19 KOR-SV22 kernel: [37487.114357] ip_rt_bug: 110.x.x.x -> a.b.c.d, ?
> 

Hi

What your routing table looks like ? (ip ro)

You also could backport this patch so that we can catch where/why this
happens

commit c378a9c019cf5e017d1ed24954b54fae7bebd2bc
Author: Dave Jones <davej@redhat.com>
Date:   Sat May 21 07:16:42 2011 +0000

    ipv4: Give backtrace in ip_rt_bug().
    
    Add a stack backtrace to the ip_rt_bug path for debugging
    
    Signed-off-by: Dave Jones <davej@redhat.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>




--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski - June 17, 2011, 9:37 p.m.
On 17.06.2011 22:36, Eric Dumazet wrote:
> Le vendredi 17 juin 2011 à 22:00 +0200, Tomasz Chmielewski a écrit :
>> I have a system pushing around 800 Mbit/s, ~130 kpps.
>>
>> It uses 2.6.35.12 kernel.
>>
>>
>> Several times a minute, I can see entries like (a.b.c.d - IP of this system):
>>
>> Jun 18 02:39:19 KOR-SV22 kernel: [37187.665951] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:39:19 KOR-SV22 kernel: [37187.685419] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:40:31 KOR-SV22 kernel: [37259.199315] ip_rt_bug: 124.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:40:36 KOR-SV22 kernel: [37263.828000] ip_rt_bug: 124.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:44:16 KOR-SV22 kernel: [37484.120689] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:44:19 KOR-SV22 kernel: [37487.114357] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>>
> 
> Hi
> 
> What your routing table looks like ? (ip ro)

It's just a proxy, no special routing set:

# ip ro
58.185.117.18 via 119.46.110.193 dev eth0 
119.46.240.13 via 119.46.110.193 dev eth0 
58.185.117.29 via 119.46.110.193 dev eth0 
119.46.241.13 via 119.46.110.193 dev eth0 
58.185.117.28 via 119.46.110.193 dev eth0 
119.46.110.192/26 dev eth0  proto kernel  scope link  src 119.46.110.197 
169.254.0.0/16 dev eth0  scope link 
default via 119.46.110.195 dev eth0 


The box is also crashing every few days; and I really had no clue why (just connected a serial console to catch any new oops/panic).


The last time it crashed, I have this entry in syslog:

Jun 17 16:16:17 TRUE-SC02 kernel: [172488.602629] ip_rt_bug: 124.121.155.197 -> 119.46.110.197, ?
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239041] BUG: unable to handle kernel NULL pointer dereference at (null)
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239409] IP: [<ffffffff81361cae>] dev_queue_xmit+0x1e3/0x441
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239760] PGD 43c30b067 PUD 439e63067 PMD 0
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.240103] Oops: 0000 [#1] SMP
Jun 17 16:19:58 TRUE-SC02 syslogd 1.4.1: restart.



Right now, it uses the newest igb driver, and I started seeing "Out of socket memory" quite a bit (didn't have it with the original igb driver from 2.6.35.12).

So I doubled this value to be:

 net.ipv4.tcp_max_orphans = 256000


and "Out of socket memory" stopped showing up.

Instead, "ip_rt_bug" shows up.
Julian Anastasov - June 17, 2011, 11:56 p.m.
Hello,

On Fri, 17 Jun 2011, Tomasz Chmielewski wrote:

> > What your routing table looks like ? (ip ro)
> 
> It's just a proxy, no special routing set:

	Is transparent proxy used?

> # ip ro
> 58.185.117.18 via 119.46.110.193 dev eth0 
> 119.46.240.13 via 119.46.110.193 dev eth0 
> 58.185.117.29 via 119.46.110.193 dev eth0 
> 119.46.241.13 via 119.46.110.193 dev eth0 

	Same route for 58.185.117.28 2nd time? Is that possible?:

> 58.185.117.28 via 119.46.110.193 dev eth0 
> 119.46.110.192/26 dev eth0  proto kernel  scope link  src 119.46.110.197 
> 169.254.0.0/16 dev eth0  scope link 
> default via 119.46.110.195 dev eth0 
> 
> 
> The box is also crashing every few days; and I really had no clue why (just connected a serial console to catch any new oops/panic).
> 
> 
> The last time it crashed, I have this entry in syslog:
> 
> Jun 17 16:16:17 TRUE-SC02 kernel: [172488.602629] ip_rt_bug: 124.121.155.197 -> 119.46.110.197, ?

	The ip_rt_bug messages show that skb->dev is
NULL (OUTPUT hook), daddr in IP header is local address,
may be some original received packet. If such packet is
provided to ip_route_me_harder(skb, RTN_UNSPEC) an
ip_route_input call can happen. Calling later dst_output
should lead to this warning. The question is what can
cause received packet to appear in OUTPUT hook where
a change in mark or TOS can can trigger such ip_route_input
call. What kind of netfilter modules are used? nf_queue,
-j REJECT, NAT? Is 124.121.155.197 a local address?

> Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239041] BUG: unable to handle kernel NULL pointer dereference at (null)
> Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239409] IP: [<ffffffff81361cae>] dev_queue_xmit+0x1e3/0x441
> Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239760] PGD 43c30b067 PUD 439e63067 PMD 0
> Jun 17 16:17:00 TRUE-SC02 kernel: [172531.240103] Oops: 0000 [#1] SMP
> Jun 17 16:19:58 TRUE-SC02 syslogd 1.4.1: restart.

Regards

--
Julian Anastasov <ja@ssi.bg>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tomasz Chmielewski - June 18, 2011, 8:31 a.m.
On 18.06.2011 01:56, Julian Anastasov wrote:
> 
> 	Hello,
> 
> On Fri, 17 Jun 2011, Tomasz Chmielewski wrote:
> 
>>> What your routing table looks like ? (ip ro)
>>
>> It's just a proxy, no special routing set:
> 
> 	Is transparent proxy used?

Yes, it is.


>> # ip ro
>> 58.185.117.18 via 119.46.110.193 dev eth0
>> 119.46.240.13 via 119.46.110.193 dev eth0
>> 58.185.117.29 via 119.46.110.193 dev eth0
>> 119.46.241.13 via 119.46.110.193 dev eth0
> 
> 	Same route for 58.185.117.28 2nd time? Is that possible?:

Not second time, the addresses are similar, but different: 58.185.117.18, 58.185.117.29, 58.185.117.28. Unless there's something I don't see! ;)

 
>> 58.185.117.28 via 119.46.110.193 dev eth0
>> 119.46.110.192/26 dev eth0  proto kernel  scope link  src 119.46.110.197
>> 169.254.0.0/16 dev eth0  scope link
>> default via 119.46.110.195 dev eth0
>>
>>
>> The box is also crashing every few days; and I really had no clue why (just connected a serial console to catch any new oops/panic).
>>
>>
>> The last time it crashed, I have this entry in syslog:
>>
>> Jun 17 16:16:17 TRUE-SC02 kernel: [172488.602629] ip_rt_bug: 124.121.155.197 ->  119.46.110.197, ?
> 
> 	The ip_rt_bug messages show that skb->dev is
> NULL (OUTPUT hook), daddr in IP header is local address,
> may be some original received packet. If such packet is
> provided to ip_route_me_harder(skb, RTN_UNSPEC) an
> ip_route_input call can happen. Calling later dst_output
> should lead to this warning. The question is what can
> cause received packet to appear in OUTPUT hook where
> a change in mark or TOS can can trigger such ip_route_input
> call. What kind of netfilter modules are used? nf_queue,
> -j REJECT, NAT? Is 124.121.155.197 a local address?

No, it's not local.
With "ip_rt_bug: 124.121.184.77 -> 119.46.110.197, ?" lines, only the address on the right side is local.

# iptables -L -t nat -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         


# iptables -L -t mangle -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DIVERT     tcp  --  0.0.0.0/0            0.0.0.0/0           socket 

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain DIVERT (1 references)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            0.0.0.0/0           MARK xset 0x1/0xffffffff 
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0     


# lsmod
Module                  Size  Used by
xt_mark                 1171  1 
xt_socket               1922  1 
nf_tproxy_core          1752  1 xt_socket,[permanent]
ipt_REDIRECT            1093  1 
xt_tcpudp               2331  1 
ebt_redirect            1234  1 
ebt_ip                  1562  1 
ebtable_broute          1395  1 
bridge                 64647  1 ebtable_broute
stp                     1931  1 bridge
llc                     5071  2 bridge,stp
ebtables               20458  1 ebtable_broute
iptable_mangle          1351  1 
iptable_nat             3644  1 
nf_nat                 16977  2 ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11077  3 iptable_nat,nf_nat
nf_defrag_ipv4          1337  2 xt_socket,nf_conntrack_ipv4
i2c_dev                 4561  0 
i2c_core               21774  1 i2c_dev
nf_conntrack_netbios_ns     1486  0 
nf_conntrack           65085  5 xt_socket,iptable_nat,nf_nat,nf_conntrack_ipv4,nf_conntrack_netbios_ns
iptable_filter          1402  0 
ip_tables              14931  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               20316  11 xt_mark,xt_socket,ipt_REDIRECT,xt_tcpudp,ebt_redirect,ebt_ip,ebtables,iptable_mangle,iptable_nat,iptable_filter,ip_tables
dm_mirror              11724  0 
dm_multipath           14772  0 
scsi_dh                 5994  1 dm_multipath
video                  21310  0 
output                  2103  1 video
sbs                    11378  0 
sbshc                   4115  1 sbs
battery                10902  0 
acpi_memhotplug         4135  0 
ac                      3274  0 
parport_pc             21355  0 
lp                      9491  0 
parport                33290  2 parport_pc,lp
option                 16045  0 
usb_wwan               10222  1 option
usbserial              34477  2 option,usb_wwan
serio_raw               4064  0 
tpm_tis                 9203  0 
tpm                    14317  1 tpm_tis
tpm_bios                5252  1 tpm
rtc_cmos                8731  0 
rtc_core               14080  1 rtc_cmos
rtc_lib                 2497  1 rtc_core
button                  5662  0 
igb                   131680  0 
shpchp                 29302  0 
pcspkr                  1822  0 
dm_region_hash          9574  1 dm_mirror
dm_log                  8359  2 dm_mirror,dm_region_hash
usb_storage            45133  0 
ata_piix               22147  0 
libata                169650  1 ata_piix
cciss                  88474  24 
sd_mod                 28117  0 
scsi_mod              156163  5 scsi_dh,usb_storage,libata,cciss,sd_mod
ext3                  114308  12 
jbd                    43368  1 ext3
uhci_hcd               18941  0 
ohci_hcd               20027  0 
ehci_hcd               33605  0 


# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 18:A9:05:41:CC:CE  
          inet addr:119.46.110.197  Bcast:119.46.110.255  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4872707550 errors:0 dropped:1177767 overruns:1177767 frame:0
          TX packets:5066061004 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3719046973104 (3.3 TiB)  TX bytes:4237588228875 (3.8 TiB)

eth0:1    Link encap:Ethernet  HWaddr 18:A9:05:41:CC:CE  
          inet addr:119.46.110.249  Bcast:119.46.110.255  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Patch

diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index b24d58e..52b0b95 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1665,6 +1665,7 @@  static int ip_rt_bug(struct sk_buff *skb)
 		&ip_hdr(skb)->saddr, &ip_hdr(skb)->daddr,
 		skb->dev ? skb->dev->name : "?");
 	kfree_skb(skb);
+	WARN_ON(1);
 	return 0;
 }