Patchwork Ensure that we unshare skbs prior to calling pskb_may_pull in bonding driver

login
register
mail settings
Submitter Neil Horman
Date Jan. 20, 2011, 7:02 p.m.
Message ID <1295550151-25913-1-git-send-email-nhorman@tuxdriver.com>
Download mbox | patch
Permalink /patch/79751/
State Accepted
Delegated to: David Miller
Headers show

Comments

Neil Horman - Jan. 20, 2011, 7:02 p.m.
Recently reported oops:

kernel BUG at net/core/skbuff.c:813!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/net/bond0/broadcast
CPU 8
Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
scsi_transport_sas dm_mod [last unloaded: microcode]

Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
scsi_transport_sas dm_mod [last unloaded: microcode]
Pid: 0, comm: swapper Not tainted 2.6.32-71.el6.x86_64 #1 BladeCenter HS22
-[7870AC1]-
RIP: 0010:[<ffffffff81405b16>]  [<ffffffff81405b16>]
pskb_expand_head+0x36/0x1e0
RSP: 0018:ffff880028303b70  EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff880c6458ec80 RCX: 0000000000000020
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880c6458ec80
RBP: ffff880028303bc0 R08: ffffffff818a6180 R09: ffff880c6458ed64
R10: ffff880c622b36c0 R11: 0000000000000400 R12: 0000000000000000
R13: 0000000000000180 R14: ffff880c622b3000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000038653452a4 CR3: 0000000001001000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff8806649c2000, task ffff880c64f16ab0)
Stack:
 ffff880028303bc0 ffffffff8104fff9 000000000000001c 0000000100000000
<0> ffff880000047d80 ffff880c6458ec80 000000000000001c ffff880c6223da00
<0> ffff880c622b3000 0000000000000000 ffff880028303c10 ffffffff81407f7a
Call Trace:
<IRQ>
 [<ffffffff8104fff9>] ? __wake_up_common+0x59/0x90
 [<ffffffff81407f7a>] __pskb_pull_tail+0x2aa/0x360
 [<ffffffffa0244530>] bond_arp_rcv+0x2c0/0x2e0 [bonding]
 [<ffffffff814a0857>] ? packet_rcv+0x377/0x440
 [<ffffffff8140f21b>] netif_receive_skb+0x2db/0x670
 [<ffffffff8140f788>] napi_skb_finish+0x58/0x70
 [<ffffffff8140fc89>] napi_gro_receive+0x39/0x50
 [<ffffffffa01286eb>] ixgbe_clean_rx_irq+0x35b/0x900 [ixgbe]
 [<ffffffffa01290f6>] ixgbe_clean_rxtx_many+0x136/0x240 [ixgbe]
 [<ffffffff8140fe53>] net_rx_action+0x103/0x210
 [<ffffffff81073bd7>] __do_softirq+0xb7/0x1e0
 [<ffffffff810d8740>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff810142cc>] call_softirq+0x1c/0x30
 [<ffffffff81015f35>] do_softirq+0x65/0xa0
 [<ffffffff810739d5>] irq_exit+0x85/0x90
 [<ffffffff814cf915>] do_IRQ+0x75/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 <EOI>
 [<ffffffff8101bc01>] ? mwait_idle+0x71/0xd0
 [<ffffffff814cd80a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff81011e96>] cpu_idle+0xb6/0x110
 [<ffffffff814c17c8>] start_secondary+0x1fc/0x23f

Resulted from bonding driver registering packet handlers via dev_add_pack and
then trying to call pskb_may_pull. If another packet handler (like for AF_PACKET
sockets) gets called first, the delivered skb will have a user count > 1, which
causes pskb_may_pull to BUG halt when it does its skb_shared check.  Fix this by
calling skb_share_check prior to the may_pull call sites in the bonding driver
to clone the skb when needed.  Tested by myself and the reported successfully.

Signed-off-by: Neil Horman
CC: Andy Gospodarek <andy@greyhouse.net>
CC: Jay Vosburgh <fubar@us.ibm.com>
CC: "David S. Miller" <davem@davemloft.net>
---
 drivers/net/bonding/bond_3ad.c  |    4 ++++
 drivers/net/bonding/bond_alb.c  |    4 ++++
 drivers/net/bonding/bond_main.c |    4 ++++
 3 files changed, 12 insertions(+), 0 deletions(-)
Jay Vosburgh - Jan. 20, 2011, 7:43 p.m.
Neil Horman <nhorman@tuxdriver.com> wrote:

>Recently reported oops:
>
>kernel BUG at net/core/skbuff.c:813!
>invalid opcode: 0000 [#1] SMP
>last sysfs file: /sys/devices/virtual/net/bond0/broadcast
>CPU 8
>Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
>ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
>i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
>ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
>scsi_transport_sas dm_mod [last unloaded: microcode]
>
>Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
>ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
>i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
>ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
>scsi_transport_sas dm_mod [last unloaded: microcode]
>Pid: 0, comm: swapper Not tainted 2.6.32-71.el6.x86_64 #1 BladeCenter HS22
>-[7870AC1]-
>RIP: 0010:[<ffffffff81405b16>]  [<ffffffff81405b16>]
>pskb_expand_head+0x36/0x1e0
>RSP: 0018:ffff880028303b70  EFLAGS: 00010202
>RAX: 0000000000000002 RBX: ffff880c6458ec80 RCX: 0000000000000020
>RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880c6458ec80
>RBP: ffff880028303bc0 R08: ffffffff818a6180 R09: ffff880c6458ed64
>R10: ffff880c622b36c0 R11: 0000000000000400 R12: 0000000000000000
>R13: 0000000000000180 R14: ffff880c622b3000 R15: 0000000000000000
>FS:  0000000000000000(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>CR2: 00000038653452a4 CR3: 0000000001001000 CR4: 00000000000006e0
>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>Process swapper (pid: 0, threadinfo ffff8806649c2000, task ffff880c64f16ab0)
>Stack:
> ffff880028303bc0 ffffffff8104fff9 000000000000001c 0000000100000000
><0> ffff880000047d80 ffff880c6458ec80 000000000000001c ffff880c6223da00
><0> ffff880c622b3000 0000000000000000 ffff880028303c10 ffffffff81407f7a
>Call Trace:
><IRQ>
> [<ffffffff8104fff9>] ? __wake_up_common+0x59/0x90
> [<ffffffff81407f7a>] __pskb_pull_tail+0x2aa/0x360
> [<ffffffffa0244530>] bond_arp_rcv+0x2c0/0x2e0 [bonding]
> [<ffffffff814a0857>] ? packet_rcv+0x377/0x440
> [<ffffffff8140f21b>] netif_receive_skb+0x2db/0x670
> [<ffffffff8140f788>] napi_skb_finish+0x58/0x70
> [<ffffffff8140fc89>] napi_gro_receive+0x39/0x50
> [<ffffffffa01286eb>] ixgbe_clean_rx_irq+0x35b/0x900 [ixgbe]
> [<ffffffffa01290f6>] ixgbe_clean_rxtx_many+0x136/0x240 [ixgbe]
> [<ffffffff8140fe53>] net_rx_action+0x103/0x210
> [<ffffffff81073bd7>] __do_softirq+0xb7/0x1e0
> [<ffffffff810d8740>] ? handle_IRQ_event+0x60/0x170
> [<ffffffff810142cc>] call_softirq+0x1c/0x30
> [<ffffffff81015f35>] do_softirq+0x65/0xa0
> [<ffffffff810739d5>] irq_exit+0x85/0x90
> [<ffffffff814cf915>] do_IRQ+0x75/0xf0
> [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
> <EOI>
> [<ffffffff8101bc01>] ? mwait_idle+0x71/0xd0
> [<ffffffff814cd80a>] ? atomic_notifier_call_chain+0x1a/0x20
> [<ffffffff81011e96>] cpu_idle+0xb6/0x110
> [<ffffffff814c17c8>] start_secondary+0x1fc/0x23f
>
>Resulted from bonding driver registering packet handlers via dev_add_pack and
>then trying to call pskb_may_pull. If another packet handler (like for AF_PACKET
>sockets) gets called first, the delivered skb will have a user count > 1, which
>causes pskb_may_pull to BUG halt when it does its skb_shared check.  Fix this by
>calling skb_share_check prior to the may_pull call sites in the bonding driver
>to clone the skb when needed.  Tested by myself and the reported successfully.
>
>Signed-off-by: Neil Horman
>CC: Andy Gospodarek <andy@greyhouse.net>
>CC: Jay Vosburgh <fubar@us.ibm.com>
>CC: "David S. Miller" <davem@davemloft.net>

	Looks like a candidate for -stable as well.

	-J

Signed-off-by: Jay Vosburgh <fubar@us.ibm.com>


>---
> drivers/net/bonding/bond_3ad.c  |    4 ++++
> drivers/net/bonding/bond_alb.c  |    4 ++++
> drivers/net/bonding/bond_main.c |    4 ++++
> 3 files changed, 12 insertions(+), 0 deletions(-)
>
>diff --git a/drivers/net/bonding/bond_3ad.c b/drivers/net/bonding/bond_3ad.c
>index 171782e..1024ae1 100644
>--- a/drivers/net/bonding/bond_3ad.c
>+++ b/drivers/net/bonding/bond_3ad.c
>@@ -2470,6 +2470,10 @@ int bond_3ad_lacpdu_recv(struct sk_buff *skb, struct net_device *dev, struct pac
> 	if (!(dev->flags & IFF_MASTER))
> 		goto out;
>
>+	skb = skb_share_check(skb, GFP_ATOMIC);
>+	if (!skb)
>+		goto out;
>+
> 	if (!pskb_may_pull(skb, sizeof(struct lacpdu)))
> 		goto out;
>
>diff --git a/drivers/net/bonding/bond_alb.c b/drivers/net/bonding/bond_alb.c
>index f4e638c..5c6fba8 100644
>--- a/drivers/net/bonding/bond_alb.c
>+++ b/drivers/net/bonding/bond_alb.c
>@@ -326,6 +326,10 @@ static int rlb_arp_recv(struct sk_buff *skb, struct net_device *bond_dev, struct
> 		goto out;
> 	}
>
>+	skb = skb_share_check(skb, GFP_ATOMIC);
>+	if (!skb)
>+		goto out;
>+
> 	if (!pskb_may_pull(skb, arp_hdr_len(bond_dev)))
> 		goto out;
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index b1025b8..163e0b0 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -2733,6 +2733,10 @@ static int bond_arp_rcv(struct sk_buff *skb, struct net_device *dev, struct pack
> 	if (!slave || !slave_do_arp_validate(bond, slave))
> 		goto out_unlock;
>
>+	skb = skb_share_check(skb, GFP_ATOMIC);
>+	if (!skb)
>+		goto out_unlock;
>+
> 	if (!pskb_may_pull(skb, arp_hdr_len(dev)))
> 		goto out_unlock;
>
>-- 
>1.7.3.4
>

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Andy Gospodarek - Jan. 20, 2011, 7:59 p.m.
On Thu, Jan 20, 2011 at 02:02:31PM -0500, Neil Horman wrote:
> Recently reported oops:
> 
> kernel BUG at net/core/skbuff.c:813!
> invalid opcode: 0000 [#1] SMP
> last sysfs file: /sys/devices/virtual/net/bond0/broadcast
> CPU 8
> Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
> ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
> i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
> ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
> scsi_transport_sas dm_mod [last unloaded: microcode]
> 
> Modules linked in: sit tunnel4 cpufreq_ondemand acpi_cpufreq freq_table bonding
> ipv6 dm_mirror dm_region_hash dm_log cdc_ether usbnet mii serio_raw i2c_i801
> i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core bnx2
> ixgbe dca mdio sg ext4 mbcache jbd2 sd_mod crc_t10dif mptsas mptscsih mptbase
> scsi_transport_sas dm_mod [last unloaded: microcode]
> Pid: 0, comm: swapper Not tainted 2.6.32-71.el6.x86_64 #1 BladeCenter HS22
> -[7870AC1]-
> RIP: 0010:[<ffffffff81405b16>]  [<ffffffff81405b16>]
> pskb_expand_head+0x36/0x1e0
> RSP: 0018:ffff880028303b70  EFLAGS: 00010202
> RAX: 0000000000000002 RBX: ffff880c6458ec80 RCX: 0000000000000020
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880c6458ec80
> RBP: ffff880028303bc0 R08: ffffffff818a6180 R09: ffff880c6458ed64
> R10: ffff880c622b36c0 R11: 0000000000000400 R12: 0000000000000000
> R13: 0000000000000180 R14: ffff880c622b3000 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 00000038653452a4 CR3: 0000000001001000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 0, threadinfo ffff8806649c2000, task ffff880c64f16ab0)
> Stack:
>  ffff880028303bc0 ffffffff8104fff9 000000000000001c 0000000100000000
> <0> ffff880000047d80 ffff880c6458ec80 000000000000001c ffff880c6223da00
> <0> ffff880c622b3000 0000000000000000 ffff880028303c10 ffffffff81407f7a
> Call Trace:
> <IRQ>
>  [<ffffffff8104fff9>] ? __wake_up_common+0x59/0x90
>  [<ffffffff81407f7a>] __pskb_pull_tail+0x2aa/0x360
>  [<ffffffffa0244530>] bond_arp_rcv+0x2c0/0x2e0 [bonding]
>  [<ffffffff814a0857>] ? packet_rcv+0x377/0x440
>  [<ffffffff8140f21b>] netif_receive_skb+0x2db/0x670
>  [<ffffffff8140f788>] napi_skb_finish+0x58/0x70
>  [<ffffffff8140fc89>] napi_gro_receive+0x39/0x50
>  [<ffffffffa01286eb>] ixgbe_clean_rx_irq+0x35b/0x900 [ixgbe]
>  [<ffffffffa01290f6>] ixgbe_clean_rxtx_many+0x136/0x240 [ixgbe]
>  [<ffffffff8140fe53>] net_rx_action+0x103/0x210
>  [<ffffffff81073bd7>] __do_softirq+0xb7/0x1e0
>  [<ffffffff810d8740>] ? handle_IRQ_event+0x60/0x170
>  [<ffffffff810142cc>] call_softirq+0x1c/0x30
>  [<ffffffff81015f35>] do_softirq+0x65/0xa0
>  [<ffffffff810739d5>] irq_exit+0x85/0x90
>  [<ffffffff814cf915>] do_IRQ+0x75/0xf0
>  [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
>  <EOI>
>  [<ffffffff8101bc01>] ? mwait_idle+0x71/0xd0
>  [<ffffffff814cd80a>] ? atomic_notifier_call_chain+0x1a/0x20
>  [<ffffffff81011e96>] cpu_idle+0xb6/0x110
>  [<ffffffff814c17c8>] start_secondary+0x1fc/0x23f
> 
> Resulted from bonding driver registering packet handlers via dev_add_pack and
> then trying to call pskb_may_pull. If another packet handler (like for AF_PACKET
> sockets) gets called first, the delivered skb will have a user count > 1, which
> causes pskb_may_pull to BUG halt when it does its skb_shared check.  Fix this by
> calling skb_share_check prior to the may_pull call sites in the bonding driver
> to clone the skb when needed.  Tested by myself and the reported successfully.
> 
> Signed-off-by: Neil Horman
> CC: Andy Gospodarek <andy@greyhouse.net>
> CC: Jay Vosburgh <fubar@us.ibm.com>
> CC: "David S. Miller" <davem@davemloft.net>

Looks great.  Thanks for posting this, Neil.  I agree with Jay that this
would be good any stable kernel that also took:

ab12811c89e88f2e66746790b1fe4469ccb7bdd9 bonding: correctly process non-linear skbs

Signed-off-by: Andy Gospodarek <andy@greyhouse.net>

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller - Jan. 21, 2011, 12:47 a.m.
From: Neil Horman <nhorman@tuxdriver.com>
Date: Thu, 20 Jan 2011 14:02:31 -0500

> Recently reported oops:

Applied, but please compose reasonable Subject lines with your patches,
always begin the line with a subsystem tag followed by a colon.

This way we get

	bonding: Foo bar baz

instead of

	Foo bar baz in the bonding driver

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Neil Horman - Jan. 21, 2011, 11:51 a.m.
On Thu, Jan 20, 2011 at 04:47:23PM -0800, David Miller wrote:
> From: Neil Horman <nhorman@tuxdriver.com>
> Date: Thu, 20 Jan 2011 14:02:31 -0500
> 
> > Recently reported oops:
> 
> Applied, but please compose reasonable Subject lines with your patches,
> always begin the line with a subsystem tag followed by a colon.
> 
> This way we get
> 
> 	bonding: Foo bar baz
> 
> instead of
> 
> 	Foo bar baz in the bonding driver
> 
> Thanks.
> 
Yeah, my bad, I realized I screwed up the Subject the second I sent the email,
sorry about that.

Regards
Neil

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/drivers/net/bonding/bond_3ad.c b/drivers/net/bonding/bond_3ad.c
index 171782e..1024ae1 100644
--- a/drivers/net/bonding/bond_3ad.c
+++ b/drivers/net/bonding/bond_3ad.c
@@ -2470,6 +2470,10 @@  int bond_3ad_lacpdu_recv(struct sk_buff *skb, struct net_device *dev, struct pac
 	if (!(dev->flags & IFF_MASTER))
 		goto out;
 
+	skb = skb_share_check(skb, GFP_ATOMIC);
+	if (!skb)
+		goto out;
+
 	if (!pskb_may_pull(skb, sizeof(struct lacpdu)))
 		goto out;
 
diff --git a/drivers/net/bonding/bond_alb.c b/drivers/net/bonding/bond_alb.c
index f4e638c..5c6fba8 100644
--- a/drivers/net/bonding/bond_alb.c
+++ b/drivers/net/bonding/bond_alb.c
@@ -326,6 +326,10 @@  static int rlb_arp_recv(struct sk_buff *skb, struct net_device *bond_dev, struct
 		goto out;
 	}
 
+	skb = skb_share_check(skb, GFP_ATOMIC);
+	if (!skb)
+		goto out;
+
 	if (!pskb_may_pull(skb, arp_hdr_len(bond_dev)))
 		goto out;
 
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index b1025b8..163e0b0 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -2733,6 +2733,10 @@  static int bond_arp_rcv(struct sk_buff *skb, struct net_device *dev, struct pack
 	if (!slave || !slave_do_arp_validate(bond, slave))
 		goto out_unlock;
 
+	skb = skb_share_check(skb, GFP_ATOMIC);
+	if (!skb)
+		goto out_unlock;
+
 	if (!pskb_may_pull(skb, arp_hdr_len(dev)))
 		goto out_unlock;