af_packet: Don't use skb after dev_queue_xmit()

Message ID	20100106131044.25b4e500@nehalam
State	Changes Requested, archived
Delegated to:	David Miller
Headers	show Return-Path: <netdev-owner@vger.kernel.org> Date: Wed, 6 Jan 2010 13:10:44 -0800 From: Stephen Hemminger <shemminger@linux-foundation.org> To: Michael Breuer <mbreuer@majjas.com> Cc: Jarek Poplawski <jarkao2@gmail.com>, David Miller <davem@davemloft.net>, akpm@linux-foundation.org, flyboy@gmail.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: Re: [PATCH] af_packet: Don't use skb after dev_queue_xmit() Message-ID: <20100106131044.25b4e500@nehalam> In-Reply-To: <4B44E952.5000804@majjas.com> References: <20100105230746.GA6612@del.dom.local> <4B43F72C.9090004@majjas.com> <20100106072208.GA6711@ff.dom.local> <4B44E952.5000804@majjas.com> Organization: Linux Foundation Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk

Stephen Hemminger Jan. 6, 2010, 9:10 p.m. UTC

On Wed, 06 Jan 2010 14:49:38 -0500
Michael Breuer <mbreuer@majjas.com> wrote:

> This patch at first behaved similarly to the previous one - seemed to be 
> running a bit better... until the adapter went down :(
> 
> This is the syslog output at the time the network failed:
> Jan  6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt 
> status=0x40000008
> Jan  6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008

Could you go back to baseline sky2 driver.  The display code might be buggy.
These bits indicate an error in the MAC. The interrupt source enabled
is Transmit FIFO underrun.

Looking at how vendor driver handles this.
It looks like the Yukon EC_U chip doesn't really do Jumbo frames correctly.
Maybe not enough internal buffering to ensure that the whole packet
is in the chip.  Of course, none of this is in the chip manual.

Does this help
--------------

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 6, 2010, 9:20 p.m. UTC | #1

On 1/6/2010 4:10 PM, Stephen Hemminger wrote:
> On Wed, 06 Jan 2010 14:49:38 -0500
> Michael Breuer<mbreuer@majjas.com>  wrote:
>
>    
>> This patch at first behaved similarly to the previous one - seemed to be
>> running a bit better... until the adapter went down :(
>>
>> This is the syslog output at the time the network failed:
>> Jan  6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt
>> status=0x40000008
>> Jan  6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008
>>      
> Could you go back to baseline sky2 driver.  The display code might be buggy.
> These bits indicate an error in the MAC. The interrupt source enabled
> is Transmit FIFO underrun.
>
> Looking at how vendor driver handles this.
> It looks like the Yukon EC_U chip doesn't really do Jumbo frames correctly.
> Maybe not enough internal buffering to ensure that the whole packet
> is in the chip.  Of course, none of this is in the chip manual.
>
> Does this help
> --------------
> --- a/drivers/net/sky2.c	2010-01-06 12:48:43.012318966 -0800
> +++ b/drivers/net/sky2.c	2010-01-06 13:05:31.273987255 -0800
> @@ -792,33 +792,21 @@ static void sky2_set_tx_stfwd(struct sky
>   {
>   	struct net_device *dev = hw->dev[port];
>
> -	if ( (hw->chip_id == CHIP_ID_YUKON_EX&&
> -	      hw->chip_rev != CHIP_REV_YU_EX_A0) ||
> -	     hw->chip_id>= CHIP_ID_YUKON_FE_P) {
> -		/* Yukon-Extreme B0 and further Extreme devices */
> -		/* enable Store&  Forward mode for TX */
> -
> -		if (dev->mtu<= ETH_DATA_LEN)
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> -				     TX_JUMBO_DIS | TX_STFW_ENA);
> -
> -		else
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> -				     TX_JUMBO_ENA| TX_STFW_ENA);
> -	} else {
> -		if (dev->mtu<= ETH_DATA_LEN)
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> -		else {
> -			/* set Tx GMAC FIFO Almost Empty Threshold */
> -			sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
> -				     (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
> -
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
> -
> -			/* Can't do offload because of lack of store/forward */
> -			dev->features&= ~(NETIF_F_TSO | NETIF_F_SG | NETIF_F_ALL_CSUM);
> -		}
> -	}
> +       if ( (hw->chip_id == CHIP_ID_YUKON_EX&&  hw->chip_rev != CHIP_REV_YU_EX_A0) ||
> +	    hw->chip_id>= CHIP_ID_YUKON_FE_P) {
> +	       /* Yukon-Extreme B0 and further Extreme devices */
> +	       /* enable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> +       } else if (dev->mtu>  ETH_DATA_LEN) {
> +	       /* set Tx GMAC FIFO Almost Empty Threshold */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
> +			    (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
> +	       /* disable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
> +       } else {
> +	       /* enable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> +       }
>   }
>
>   static void sky2_mac_init(struct sky2_hw *hw, unsigned port)
> @@ -2185,11 +2173,16 @@ static int sky2_change_mtu(struct net_de
>   	if (new_mtu<  ETH_ZLEN || new_mtu>  ETH_JUMBO_MTU)
>   		return -EINVAL;
>
> +	/* MTU>  1500 on yukon FE and FE+ not allowed */
>   	if (new_mtu>  ETH_DATA_LEN&&
>   	(hw->chip_id == CHIP_ID_YUKON_FE ||
>   	     hw->chip_id == CHIP_ID_YUKON_FE_P))
>   		return -EINVAL;
>
> +	/* TSO on Yukon Ultra and MTU>  1500 not supported */
> +	if (new_mtu>  ETH_DATA_LEN&&  hw->chip_id == CHIP_ID_YUKON_EC_U)
> +		dev->features&= ~NETIF_F_TSO;
> +
>   	if (!netif_running(dev)) {
>   		dev->mtu = new_mtu;
>   		return 0;
> @@ -2233,6 +2226,15 @@ static int sky2_change_mtu(struct net_de
>   	if (err)
>   		dev_close(dev);
>   	else {
> +		/* WA for dev. #4.209 */
> +		if (hw->chip_id == CHIP_ID_YUKON_EC_U&&
> +		    hw->chip_rev == CHIP_REV_YU_EC_U_A1) {
> +			/* enable/disable Store&  Forward mode for TX */
> +			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> +				     sky2->speed != SPEED_1000
> +				     ? TX_STFW_ENA : TX_STFW_DIS);
> +		}
> +
>   		gma_write16(hw, port, GM_GP_CTRL, ctl);
>
>   		netif_wake_queue(dev);
> --- a/drivers/net/sky2.h	2010-01-06 12:48:48.632247424 -0800
> +++ b/drivers/net/sky2.h	2010-01-06 12:59:57.322078964 -0800
> @@ -1901,8 +1901,8 @@ enum {
>   	TX_VLAN_TAG_ON	= 1<<25,/* enable  VLAN tagging */
>   	TX_VLAN_TAG_OFF	= 1<<24,/* disable VLAN tagging */
>
> -	TX_JUMBO_ENA	= 1<<23,/* PCI Jumbo Mode enable (Yukon-EC Ultra) */
> -	TX_JUMBO_DIS	= 1<<22,/* PCI Jumbo Mode enable (Yukon-EC Ultra) */
> +	TX_PCI_JUM_ENA	= 1<<23,/* Enable  PCI Jumbo Mode (Yukon-EC Ultra) */
> +	TX_PCI_JUM_DIS	= 1<<22,/* Disable PCI Jumbo Mode (Yukon-EC Ultra) */
>
>   	GMF_WSP_TST_ON	= 1<<18,/* Write Shadow Pointer Test On */
>   	GMF_WSP_TST_OFF	= 1<<17,/* Write Shadow Pointer Test Off */
>    
I'll try this a bit later today. However, early on, I saw the same 
issues with MTU=1500. Also, maybe I'm missing something, but I can only 
recreate the issue with a high receive rate. Given the interaction with 
DHCP, for example, I'm thinking that there is some precondition that is 
as yet unknown. May be buggy hardware, or perhaps a race condition 
resulting in a corrupt i/o buffer somewhere. I'm wondering whether 
there's some useful place to insert some diagnostics on the RX side - at 
least we can see if there are any consistent events on the RX side 
preceding the TX error.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 6, 2010, 11:26 p.m. UTC | #2

On 1/6/2010 4:10 PM, Stephen Hemminger wrote:
> On Wed, 06 Jan 2010 14:49:38 -0500
> Michael Breuer<mbreuer@majjas.com>  wrote:
>
>    
>> This patch at first behaved similarly to the previous one - seemed to be
>> running a bit better... until the adapter went down :(
>>
>> This is the syslog output at the time the network failed:
>> Jan  6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt
>> status=0x40000008
>> Jan  6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008
>>      
> Could you go back to baseline sky2 driver.  The display code might be buggy.
> These bits indicate an error in the MAC. The interrupt source enabled
> is Transmit FIFO underrun.
>
> Looking at how vendor driver handles this.
> It looks like the Yukon EC_U chip doesn't really do Jumbo frames correctly.
> Maybe not enough internal buffering to ensure that the whole packet
> is in the chip.  Of course, none of this is in the chip manual.
>
> Does this help
> --------------
> --- a/drivers/net/sky2.c	2010-01-06 12:48:43.012318966 -0800
> +++ b/drivers/net/sky2.c	2010-01-06 13:05:31.273987255 -0800
> @@ -792,33 +792,21 @@ static void sky2_set_tx_stfwd(struct sky
>   {
>   	struct net_device *dev = hw->dev[port];
>
> -	if ( (hw->chip_id == CHIP_ID_YUKON_EX&&
> -	      hw->chip_rev != CHIP_REV_YU_EX_A0) ||
> -	     hw->chip_id>= CHIP_ID_YUKON_FE_P) {
> -		/* Yukon-Extreme B0 and further Extreme devices */
> -		/* enable Store&  Forward mode for TX */
> -
> -		if (dev->mtu<= ETH_DATA_LEN)
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> -				     TX_JUMBO_DIS | TX_STFW_ENA);
> -
> -		else
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> -				     TX_JUMBO_ENA| TX_STFW_ENA);
> -	} else {
> -		if (dev->mtu<= ETH_DATA_LEN)
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> -		else {
> -			/* set Tx GMAC FIFO Almost Empty Threshold */
> -			sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
> -				     (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
> -
> -			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
> -
> -			/* Can't do offload because of lack of store/forward */
> -			dev->features&= ~(NETIF_F_TSO | NETIF_F_SG | NETIF_F_ALL_CSUM);
> -		}
> -	}
> +       if ( (hw->chip_id == CHIP_ID_YUKON_EX&&  hw->chip_rev != CHIP_REV_YU_EX_A0) ||
> +	    hw->chip_id>= CHIP_ID_YUKON_FE_P) {
> +	       /* Yukon-Extreme B0 and further Extreme devices */
> +	       /* enable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> +       } else if (dev->mtu>  ETH_DATA_LEN) {
> +	       /* set Tx GMAC FIFO Almost Empty Threshold */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
> +			    (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
> +	       /* disable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
> +       } else {
> +	       /* enable Store&  Forward mode for TX */
> +	       sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
> +       }
>   }
>
>   static void sky2_mac_init(struct sky2_hw *hw, unsigned port)
> @@ -2185,11 +2173,16 @@ static int sky2_change_mtu(struct net_de
>   	if (new_mtu<  ETH_ZLEN || new_mtu>  ETH_JUMBO_MTU)
>   		return -EINVAL;
>
> +	/* MTU>  1500 on yukon FE and FE+ not allowed */
>   	if (new_mtu>  ETH_DATA_LEN&&
>   	(hw->chip_id == CHIP_ID_YUKON_FE ||
>   	     hw->chip_id == CHIP_ID_YUKON_FE_P))
>   		return -EINVAL;
>
> +	/* TSO on Yukon Ultra and MTU>  1500 not supported */
> +	if (new_mtu>  ETH_DATA_LEN&&  hw->chip_id == CHIP_ID_YUKON_EC_U)
> +		dev->features&= ~NETIF_F_TSO;
> +
>   	if (!netif_running(dev)) {
>   		dev->mtu = new_mtu;
>   		return 0;
> @@ -2233,6 +2226,15 @@ static int sky2_change_mtu(struct net_de
>   	if (err)
>   		dev_close(dev);
>   	else {
> +		/* WA for dev. #4.209 */
> +		if (hw->chip_id == CHIP_ID_YUKON_EC_U&&
> +		    hw->chip_rev == CHIP_REV_YU_EC_U_A1) {
> +			/* enable/disable Store&  Forward mode for TX */
> +			sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
> +				     sky2->speed != SPEED_1000
> +				     ? TX_STFW_ENA : TX_STFW_DIS);
> +		}
> +
>   		gma_write16(hw, port, GM_GP_CTRL, ctl);
>
>   		netif_wake_queue(dev);
> --- a/drivers/net/sky2.h	2010-01-06 12:48:48.632247424 -0800
> +++ b/drivers/net/sky2.h	2010-01-06 12:59:57.322078964 -0800
> @@ -1901,8 +1901,8 @@ enum {
>   	TX_VLAN_TAG_ON	= 1<<25,/* enable  VLAN tagging */
>   	TX_VLAN_TAG_OFF	= 1<<24,/* disable VLAN tagging */
>
> -	TX_JUMBO_ENA	= 1<<23,/* PCI Jumbo Mode enable (Yukon-EC Ultra) */
> -	TX_JUMBO_DIS	= 1<<22,/* PCI Jumbo Mode enable (Yukon-EC Ultra) */
> +	TX_PCI_JUM_ENA	= 1<<23,/* Enable  PCI Jumbo Mode (Yukon-EC Ultra) */
> +	TX_PCI_JUM_DIS	= 1<<22,/* Disable PCI Jumbo Mode (Yukon-EC Ultra) */
>
>   	GMF_WSP_TST_ON	= 1<<18,/* Write Shadow Pointer Test On */
>   	GMF_WSP_TST_OFF	= 1<<17,/* Write Shadow Pointer Test Off */
>    
Ok ...  results - and maybe some more clues...

Running with this patch; Jarek's "alternative 1", and the patch from the 
other thread. Not so good.

No reported errors (sky2, etc.) - however with mtu=9000, lots of stuff 
broke: XDMCP; http via MASQ/netfilter, ssh connections intermittently 
(when large frames involved perhaps), etc. Tried to change mtu to 1500 
on the fly, got a bunch of errors - and network watchdog kicked in. Have 
now rebooted with the same patches and mtu=1500.
... with mtu=1500, Everything is again working (i.e., XDMCP, netfilter, 
etc.)
Load test with mtu=1500 went well for a while - high throughput 
sustained for a few minutes - then similar crash as before... but no 
interrup error messages this time until after the oops:
<nothing of note before this>
Jan  6 18:17:54 mail kernel: DRHD: handling fault status reg 2
Jan  6 18:17:54 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
fault addr 1bbfe000
Jan  6 18:17:54 mail kernel: DMAR:[fault reason 06] PTE Read access is 
not set
Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x80000000
Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
Jan  6 18:18:04 mail kernel: ------------[ cut here ]------------
Jan  6 18:18:04 mail kernel: WARNING: at net/sched/sch_generic.c:261 
dev_watchdog+0xf3/0x164()
Jan  6 18:18:04 mail kernel: Hardware name: System Product Name
Jan  6 18:18:04 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
queue 0 timed out
Jan  6 18:18:04 mail kernel: Modules linked in: ip6table_filter 
ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat 
iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd 
nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit 
tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp 
xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm 
snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec 
snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq snd_seq_device 
gspca_spca505 gspca_main videodev v4l1_compat snd_pcm 
v4l2_compat_ioctl32 pcspkr asus_atk0110 hwmon i2c_i801 iTCO_wdt 
firewire_ohci iTCO_vendor_support firewire_core crc_itu_t snd_timer snd 
sky2 soundcore wmi snd_page_alloc fbcon tileblit font bitblit softcursor 
raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy 
async_tx raid1 ata_generic pata_acpi pata_marvell nouveau ttm 
drm_kms_helper drm agpgart fb i2c_algo_bit cfbcopyarea i2c_core 
cfbimgblt cfbfil
Jan  6 18:18:04 mail kernel: lrect [last unloaded: microcode]
Jan  6 18:18:04 mail kernel: Pid: 0, comm: swapper Tainted: G        W  
2.6.32-00840-gec8257c-dirty #41
Jan  6 18:18:04 mail kernel: Call Trace:
Jan  6 18:18:04 mail kernel: <IRQ>  [<ffffffff8105365a>] 
warn_slowpath_common+0x7c/0x94
Jan  6 18:18:04 mail kernel: [<ffffffff810536c9>] 
warn_slowpath_fmt+0x41/0x43
Jan  6 18:18:04 mail kernel: [<ffffffff813e12bf>] ? netif_tx_lock+0x44/0x6c
Jan  6 18:18:04 mail kernel: [<ffffffff813e1427>] dev_watchdog+0xf3/0x164
Jan  6 18:18:04 mail kernel: [<ffffffff81077696>] ? 
sched_clock_cpu+0x47/0xd1
Jan  6 18:18:04 mail kernel: [<ffffffff8106316b>] 
run_timer_softirq+0x1c8/0x270
Jan  6 18:18:04 mail kernel: [<ffffffff8105ae3b>] __do_softirq+0xf8/0x1cd
Jan  6 18:18:04 mail kernel: [<ffffffff8107ef33>] ? 
tick_program_event+0x2a/0x2c
Jan  6 18:18:04 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan  6 18:18:04 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan  6 18:18:04 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
Jan  6 18:18:04 mail kernel: [<ffffffff8146dd32>] 
smp_apic_timer_interrupt+0x86/0x94
Jan  6 18:18:04 mail kernel: [<ffffffff810127e3>] 
apic_timer_interrupt+0x13/0x20
Jan  6 18:18:04 mail kernel: <EOI>  [<ffffffff812c4a06>] ? 
acpi_idle_enter_c1+0xb2/0xd0
Jan  6 18:18:04 mail kernel: [<ffffffff812c49ff>] ? 
acpi_idle_enter_c1+0xab/0xd0
Jan  6 18:18:04 mail kernel: [<ffffffff813a43b8>] ? 
cpuidle_idle_call+0x9e/0xfa
Jan  6 18:18:04 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan  6 18:18:04 mail kernel: [<ffffffff81463312>] ? 
start_secondary+0x201/0x242
Jan  6 18:18:04 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
Jan  6 18:18:04 mail kernel: sky2 eth0: tx timeout
Jan  6 18:18:04 mail kernel: sky2 eth0: transmit ring 21 .. 108 
report=21 done=21
Jan  6 18:18:04 mail kernel: sky2 eth0: disabling interface
Jan  6 18:18:04 mail kernel: sky2 eth0: enabling interface
<eth0 dead after this>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 2:42 a.m. UTC | #3

On 1/6/2010 6:26 PM, Michael Breuer wrote:
> On 1/6/2010 4:10 PM, Stephen Hemminger wrote:
>> On Wed, 06 Jan 2010 14:49:38 -0500
>> Michael Breuer<mbreuer@majjas.com>  wrote:
>>
>>> This patch at first behaved similarly to the previous one - seemed 
>>> to be
>>> running a bit better... until the adapter went down :(
>>>
>>> This is the syslog output at the time the network failed:
>>> Jan  6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt
>>> status=0x40000008
>>> Jan  6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008
>> Could you go back to baseline sky2 driver.  The display code might be 
>> buggy.
>> These bits indicate an error in the MAC. The interrupt source enabled
>> is Transmit FIFO underrun.
>>
>> Looking at how vendor driver handles this.
>> It looks like the Yukon EC_U chip doesn't really do Jumbo frames 
>> correctly.
>> Maybe not enough internal buffering to ensure that the whole packet
>> is in the chip.  Of course, none of this is in the chip manual.
>>
>> Does this help
>> --------------
>> --- a/drivers/net/sky2.c    2010-01-06 12:48:43.012318966 -0800
>> +++ b/drivers/net/sky2.c    2010-01-06 13:05:31.273987255 -0800
>> @@ -792,33 +792,21 @@ static void sky2_set_tx_stfwd(struct sky
>>   {
>>       struct net_device *dev = hw->dev[port];
>>
>> -    if ( (hw->chip_id == CHIP_ID_YUKON_EX&&
>> -          hw->chip_rev != CHIP_REV_YU_EX_A0) ||
>> -         hw->chip_id>= CHIP_ID_YUKON_FE_P) {
>> -        /* Yukon-Extreme B0 and further Extreme devices */
>> -        /* enable Store&  Forward mode for TX */
>> -
>> -        if (dev->mtu<= ETH_DATA_LEN)
>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>> -                     TX_JUMBO_DIS | TX_STFW_ENA);
>> -
>> -        else
>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>> -                     TX_JUMBO_ENA| TX_STFW_ENA);
>> -    } else {
>> -        if (dev->mtu<= ETH_DATA_LEN)
>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
>> -        else {
>> -            /* set Tx GMAC FIFO Almost Empty Threshold */
>> -            sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
>> -                     (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
>> -
>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
>> -
>> -            /* Can't do offload because of lack of store/forward */
>> -            dev->features&= ~(NETIF_F_TSO | NETIF_F_SG | 
>> NETIF_F_ALL_CSUM);
>> -        }
>> -    }
>> +       if ( (hw->chip_id == CHIP_ID_YUKON_EX&&  hw->chip_rev != 
>> CHIP_REV_YU_EX_A0) ||
>> +        hw->chip_id>= CHIP_ID_YUKON_FE_P) {
>> +           /* Yukon-Extreme B0 and further Extreme devices */
>> +           /* enable Store&  Forward mode for TX */
>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
>> +       } else if (dev->mtu>  ETH_DATA_LEN) {
>> +           /* set Tx GMAC FIFO Almost Empty Threshold */
>> +           sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
>> +                (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
>> +           /* disable Store&  Forward mode for TX */
>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
>> +       } else {
>> +           /* enable Store&  Forward mode for TX */
>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
>> +       }
>>   }
>>
>>   static void sky2_mac_init(struct sky2_hw *hw, unsigned port)
>> @@ -2185,11 +2173,16 @@ static int sky2_change_mtu(struct net_de
>>       if (new_mtu<  ETH_ZLEN || new_mtu>  ETH_JUMBO_MTU)
>>           return -EINVAL;
>>
>> +    /* MTU>  1500 on yukon FE and FE+ not allowed */
>>       if (new_mtu>  ETH_DATA_LEN&&
>>       (hw->chip_id == CHIP_ID_YUKON_FE ||
>>            hw->chip_id == CHIP_ID_YUKON_FE_P))
>>           return -EINVAL;
>>
>> +    /* TSO on Yukon Ultra and MTU>  1500 not supported */
>> +    if (new_mtu>  ETH_DATA_LEN&&  hw->chip_id == CHIP_ID_YUKON_EC_U)
>> +        dev->features&= ~NETIF_F_TSO;
>> +
>>       if (!netif_running(dev)) {
>>           dev->mtu = new_mtu;
>>           return 0;
>> @@ -2233,6 +2226,15 @@ static int sky2_change_mtu(struct net_de
>>       if (err)
>>           dev_close(dev);
>>       else {
>> +        /* WA for dev. #4.209 */
>> +        if (hw->chip_id == CHIP_ID_YUKON_EC_U&&
>> +            hw->chip_rev == CHIP_REV_YU_EC_U_A1) {
>> +            /* enable/disable Store&  Forward mode for TX */
>> +            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>> +                     sky2->speed != SPEED_1000
>> +                     ? TX_STFW_ENA : TX_STFW_DIS);
>> +        }
>> +
>>           gma_write16(hw, port, GM_GP_CTRL, ctl);
>>
>>           netif_wake_queue(dev);
>> --- a/drivers/net/sky2.h    2010-01-06 12:48:48.632247424 -0800
>> +++ b/drivers/net/sky2.h    2010-01-06 12:59:57.322078964 -0800
>> @@ -1901,8 +1901,8 @@ enum {
>>       TX_VLAN_TAG_ON    = 1<<25,/* enable  VLAN tagging */
>>       TX_VLAN_TAG_OFF    = 1<<24,/* disable VLAN tagging */
>>
>> -    TX_JUMBO_ENA    = 1<<23,/* PCI Jumbo Mode enable (Yukon-EC 
>> Ultra) */
>> -    TX_JUMBO_DIS    = 1<<22,/* PCI Jumbo Mode enable (Yukon-EC 
>> Ultra) */
>> +    TX_PCI_JUM_ENA    = 1<<23,/* Enable  PCI Jumbo Mode (Yukon-EC 
>> Ultra) */
>> +    TX_PCI_JUM_DIS    = 1<<22,/* Disable PCI Jumbo Mode (Yukon-EC 
>> Ultra) */
>>
>>       GMF_WSP_TST_ON    = 1<<18,/* Write Shadow Pointer Test On */
>>       GMF_WSP_TST_OFF    = 1<<17,/* Write Shadow Pointer Test Off */
> Ok ...  results - and maybe some more clues...
>
> Running with this patch; Jarek's "alternative 1", and the patch from 
> the other thread. Not so good.
>
> No reported errors (sky2, etc.) - however with mtu=9000, lots of stuff 
> broke: XDMCP; http via MASQ/netfilter, ssh connections intermittently 
> (when large frames involved perhaps), etc. Tried to change mtu to 1500 
> on the fly, got a bunch of errors - and network watchdog kicked in. 
> Have now rebooted with the same patches and mtu=1500.
> ... with mtu=1500, Everything is again working (i.e., XDMCP, 
> netfilter, etc.)
> Load test with mtu=1500 went well for a while - high throughput 
> sustained for a few minutes - then similar crash as before... but no 
> interrup error messages this time until after the oops:
> <nothing of note before this>
> Jan  6 18:17:54 mail kernel: DRHD: handling fault status reg 2
> Jan  6 18:17:54 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
> fault addr 1bbfe000
> Jan  6 18:17:54 mail kernel: DMAR:[fault reason 06] PTE Read access is 
> not set
> Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: error interrupt 
> status=0x80000000
> Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: PCI hardware error 
> (0x2010)
> Jan  6 18:18:04 mail kernel: ------------[ cut here ]------------
> Jan  6 18:18:04 mail kernel: WARNING: at net/sched/sch_generic.c:261 
> dev_watchdog+0xf3/0x164()
> Jan  6 18:18:04 mail kernel: Hardware name: System Product Name
> Jan  6 18:18:04 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
> queue 0 timed out
> Jan  6 18:18:04 mail kernel: Modules linked in: ip6table_filter 
> ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat 
> iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd 
> nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq 
> sit tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP 
> xt_dscp xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath 
> kvm_intel kvm snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi 
> snd_ac97_codec snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq 
> snd_seq_device gspca_spca505 gspca_main videodev v4l1_compat snd_pcm 
> v4l2_compat_ioctl32 pcspkr asus_atk0110 hwmon i2c_i801 iTCO_wdt 
> firewire_ohci iTCO_vendor_support firewire_core crc_itu_t snd_timer 
> snd sky2 soundcore wmi snd_page_alloc fbcon tileblit font bitblit 
> softcursor raid456 async_raid6_recov async_pq raid6_pq async_xor xor 
> async_memcpy async_tx raid1 ata_generic pata_acpi pata_marvell nouveau 
> ttm drm_kms_helper drm agpgart fb i2c_algo_bit cfbcopyarea i2c_core 
> cfbimgblt cfbfil
> Jan  6 18:18:04 mail kernel: lrect [last unloaded: microcode]
> Jan  6 18:18:04 mail kernel: Pid: 0, comm: swapper Tainted: G        
> W  2.6.32-00840-gec8257c-dirty #41
> Jan  6 18:18:04 mail kernel: Call Trace:
> Jan  6 18:18:04 mail kernel: <IRQ>  [<ffffffff8105365a>] 
> warn_slowpath_common+0x7c/0x94
> Jan  6 18:18:04 mail kernel: [<ffffffff810536c9>] 
> warn_slowpath_fmt+0x41/0x43
> Jan  6 18:18:04 mail kernel: [<ffffffff813e12bf>] ? 
> netif_tx_lock+0x44/0x6c
> Jan  6 18:18:04 mail kernel: [<ffffffff813e1427>] dev_watchdog+0xf3/0x164
> Jan  6 18:18:04 mail kernel: [<ffffffff81077696>] ? 
> sched_clock_cpu+0x47/0xd1
> Jan  6 18:18:04 mail kernel: [<ffffffff8106316b>] 
> run_timer_softirq+0x1c8/0x270
> Jan  6 18:18:04 mail kernel: [<ffffffff8105ae3b>] __do_softirq+0xf8/0x1cd
> Jan  6 18:18:04 mail kernel: [<ffffffff8107ef33>] ? 
> tick_program_event+0x2a/0x2c
> Jan  6 18:18:04 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
> Jan  6 18:18:04 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
> Jan  6 18:18:04 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
> Jan  6 18:18:04 mail kernel: [<ffffffff8146dd32>] 
> smp_apic_timer_interrupt+0x86/0x94
> Jan  6 18:18:04 mail kernel: [<ffffffff810127e3>] 
> apic_timer_interrupt+0x13/0x20
> Jan  6 18:18:04 mail kernel: <EOI>  [<ffffffff812c4a06>] ? 
> acpi_idle_enter_c1+0xb2/0xd0
> Jan  6 18:18:04 mail kernel: [<ffffffff812c49ff>] ? 
> acpi_idle_enter_c1+0xab/0xd0
> Jan  6 18:18:04 mail kernel: [<ffffffff813a43b8>] ? 
> cpuidle_idle_call+0x9e/0xfa
> Jan  6 18:18:04 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
> Jan  6 18:18:04 mail kernel: [<ffffffff81463312>] ? 
> start_secondary+0x201/0x242
> Jan  6 18:18:04 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
> Jan  6 18:18:04 mail kernel: sky2 eth0: tx timeout
> Jan  6 18:18:04 mail kernel: sky2 eth0: transmit ring 21 .. 108 
> report=21 done=21
> Jan  6 18:18:04 mail kernel: sky2 eth0: disabling interface
> Jan  6 18:18:04 mail kernel: sky2 eth0: enabling interface
> <eth0 dead after this>
Walked through the code based on Jarek's patches... came upon 
NET_CLS_ACT. At least in some cases (sch_cbq.c for example), the net 
transmit error could be returned from here... after releasing the skb. A 
quick scan of the various files in net/sched suggests that with 
NET_CLS_ACT the skb may or may not have been freed in the event of an 
error. If I have time later I'll see if I can bypass NET_CLS_ACT and see 
whether this is even relevant.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 4 a.m. UTC | #4

On 1/6/2010 9:42 PM, Michael Breuer wrote:
> On 1/6/2010 6:26 PM, Michael Breuer wrote:
>> On 1/6/2010 4:10 PM, Stephen Hemminger wrote:
>>> On Wed, 06 Jan 2010 14:49:38 -0500
>>> Michael Breuer<mbreuer@majjas.com>  wrote:
>>>
>>>> This patch at first behaved similarly to the previous one - seemed 
>>>> to be
>>>> running a bit better... until the adapter went down :(
>>>>
>>>> This is the syslog output at the time the network failed:
>>>> Jan  6 14:11:01 mail kernel: sky2 0000:06:00.0: error interrupt
>>>> status=0x40000008
>>>> Jan  6 14:11:01 mail kernel: sky2 software interrupt status 0x40000008
>>> Could you go back to baseline sky2 driver.  The display code might 
>>> be buggy.
>>> These bits indicate an error in the MAC. The interrupt source enabled
>>> is Transmit FIFO underrun.
>>>
>>> Looking at how vendor driver handles this.
>>> It looks like the Yukon EC_U chip doesn't really do Jumbo frames 
>>> correctly.
>>> Maybe not enough internal buffering to ensure that the whole packet
>>> is in the chip.  Of course, none of this is in the chip manual.
>>>
>>> Does this help
>>> --------------
>>> --- a/drivers/net/sky2.c    2010-01-06 12:48:43.012318966 -0800
>>> +++ b/drivers/net/sky2.c    2010-01-06 13:05:31.273987255 -0800
>>> @@ -792,33 +792,21 @@ static void sky2_set_tx_stfwd(struct sky
>>>   {
>>>       struct net_device *dev = hw->dev[port];
>>>
>>> -    if ( (hw->chip_id == CHIP_ID_YUKON_EX&&
>>> -          hw->chip_rev != CHIP_REV_YU_EX_A0) ||
>>> -         hw->chip_id>= CHIP_ID_YUKON_FE_P) {
>>> -        /* Yukon-Extreme B0 and further Extreme devices */
>>> -        /* enable Store&  Forward mode for TX */
>>> -
>>> -        if (dev->mtu<= ETH_DATA_LEN)
>>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>>> -                     TX_JUMBO_DIS | TX_STFW_ENA);
>>> -
>>> -        else
>>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>>> -                     TX_JUMBO_ENA| TX_STFW_ENA);
>>> -    } else {
>>> -        if (dev->mtu<= ETH_DATA_LEN)
>>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), 
>>> TX_STFW_ENA);
>>> -        else {
>>> -            /* set Tx GMAC FIFO Almost Empty Threshold */
>>> -            sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
>>> -                     (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
>>> -
>>> -            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), 
>>> TX_STFW_DIS);
>>> -
>>> -            /* Can't do offload because of lack of store/forward */
>>> -            dev->features&= ~(NETIF_F_TSO | NETIF_F_SG | 
>>> NETIF_F_ALL_CSUM);
>>> -        }
>>> -    }
>>> +       if ( (hw->chip_id == CHIP_ID_YUKON_EX&&  hw->chip_rev != 
>>> CHIP_REV_YU_EX_A0) ||
>>> +        hw->chip_id>= CHIP_ID_YUKON_FE_P) {
>>> +           /* Yukon-Extreme B0 and further Extreme devices */
>>> +           /* enable Store&  Forward mode for TX */
>>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
>>> +       } else if (dev->mtu>  ETH_DATA_LEN) {
>>> +           /* set Tx GMAC FIFO Almost Empty Threshold */
>>> +           sky2_write32(hw, SK_REG(port, TX_GMF_AE_THR),
>>> +                (ECU_JUMBO_WM<<  16) | ECU_AE_THR);
>>> +           /* disable Store&  Forward mode for TX */
>>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_DIS);
>>> +       } else {
>>> +           /* enable Store&  Forward mode for TX */
>>> +           sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T), TX_STFW_ENA);
>>> +       }
>>>   }
>>>
>>>   static void sky2_mac_init(struct sky2_hw *hw, unsigned port)
>>> @@ -2185,11 +2173,16 @@ static int sky2_change_mtu(struct net_de
>>>       if (new_mtu<  ETH_ZLEN || new_mtu>  ETH_JUMBO_MTU)
>>>           return -EINVAL;
>>>
>>> +    /* MTU>  1500 on yukon FE and FE+ not allowed */
>>>       if (new_mtu>  ETH_DATA_LEN&&
>>>       (hw->chip_id == CHIP_ID_YUKON_FE ||
>>>            hw->chip_id == CHIP_ID_YUKON_FE_P))
>>>           return -EINVAL;
>>>
>>> +    /* TSO on Yukon Ultra and MTU>  1500 not supported */
>>> +    if (new_mtu>  ETH_DATA_LEN&&  hw->chip_id == CHIP_ID_YUKON_EC_U)
>>> +        dev->features&= ~NETIF_F_TSO;
>>> +
>>>       if (!netif_running(dev)) {
>>>           dev->mtu = new_mtu;
>>>           return 0;
>>> @@ -2233,6 +2226,15 @@ static int sky2_change_mtu(struct net_de
>>>       if (err)
>>>           dev_close(dev);
>>>       else {
>>> +        /* WA for dev. #4.209 */
>>> +        if (hw->chip_id == CHIP_ID_YUKON_EC_U&&
>>> +            hw->chip_rev == CHIP_REV_YU_EC_U_A1) {
>>> +            /* enable/disable Store&  Forward mode for TX */
>>> +            sky2_write32(hw, SK_REG(port, TX_GMF_CTRL_T),
>>> +                     sky2->speed != SPEED_1000
>>> +                     ? TX_STFW_ENA : TX_STFW_DIS);
>>> +        }
>>> +
>>>           gma_write16(hw, port, GM_GP_CTRL, ctl);
>>>
>>>           netif_wake_queue(dev);
>>> --- a/drivers/net/sky2.h    2010-01-06 12:48:48.632247424 -0800
>>> +++ b/drivers/net/sky2.h    2010-01-06 12:59:57.322078964 -0800
>>> @@ -1901,8 +1901,8 @@ enum {
>>>       TX_VLAN_TAG_ON    = 1<<25,/* enable  VLAN tagging */
>>>       TX_VLAN_TAG_OFF    = 1<<24,/* disable VLAN tagging */
>>>
>>> -    TX_JUMBO_ENA    = 1<<23,/* PCI Jumbo Mode enable (Yukon-EC 
>>> Ultra) */
>>> -    TX_JUMBO_DIS    = 1<<22,/* PCI Jumbo Mode enable (Yukon-EC 
>>> Ultra) */
>>> +    TX_PCI_JUM_ENA    = 1<<23,/* Enable  PCI Jumbo Mode (Yukon-EC 
>>> Ultra) */
>>> +    TX_PCI_JUM_DIS    = 1<<22,/* Disable PCI Jumbo Mode (Yukon-EC 
>>> Ultra) */
>>>
>>>       GMF_WSP_TST_ON    = 1<<18,/* Write Shadow Pointer Test On */
>>>       GMF_WSP_TST_OFF    = 1<<17,/* Write Shadow Pointer Test Off */
>> Ok ...  results - and maybe some more clues...
>>
>> Running with this patch; Jarek's "alternative 1", and the patch from 
>> the other thread. Not so good.
>>
>> No reported errors (sky2, etc.) - however with mtu=9000, lots of 
>> stuff broke: XDMCP; http via MASQ/netfilter, ssh connections 
>> intermittently (when large frames involved perhaps), etc. Tried to 
>> change mtu to 1500 on the fly, got a bunch of errors - and network 
>> watchdog kicked in. Have now rebooted with the same patches and 
>> mtu=1500.
>> ... with mtu=1500, Everything is again working (i.e., XDMCP, 
>> netfilter, etc.)
>> Load test with mtu=1500 went well for a while - high throughput 
>> sustained for a few minutes - then similar crash as before... but no 
>> interrup error messages this time until after the oops:
>> <nothing of note before this>
>> Jan  6 18:17:54 mail kernel: DRHD: handling fault status reg 2
>> Jan  6 18:17:54 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
>> fault addr 1bbfe000
>> Jan  6 18:17:54 mail kernel: DMAR:[fault reason 06] PTE Read access 
>> is not set
>> Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: error interrupt 
>> status=0x80000000
>> Jan  6 18:17:54 mail kernel: sky2 0000:06:00.0: PCI hardware error 
>> (0x2010)
>> Jan  6 18:18:04 mail kernel: ------------[ cut here ]------------
>> Jan  6 18:18:04 mail kernel: WARNING: at net/sched/sch_generic.c:261 
>> dev_watchdog+0xf3/0x164()
>> Jan  6 18:18:04 mail kernel: Hardware name: System Product Name
>> Jan  6 18:18:04 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
>> queue 0 timed out
>> Jan  6 18:18:04 mail kernel: Modules linked in: ip6table_filter 
>> ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat 
>> iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd 
>> nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq 
>> sit tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP 
>> xt_dscp xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath 
>> kvm_intel kvm snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi 
>> snd_ac97_codec snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq 
>> snd_seq_device gspca_spca505 gspca_main videodev v4l1_compat snd_pcm 
>> v4l2_compat_ioctl32 pcspkr asus_atk0110 hwmon i2c_i801 iTCO_wdt 
>> firewire_ohci iTCO_vendor_support firewire_core crc_itu_t snd_timer 
>> snd sky2 soundcore wmi snd_page_alloc fbcon tileblit font bitblit 
>> softcursor raid456 async_raid6_recov async_pq raid6_pq async_xor xor 
>> async_memcpy async_tx raid1 ata_generic pata_acpi pata_marvell 
>> nouveau ttm drm_kms_helper drm agpgart fb i2c_algo_bit cfbcopyarea 
>> i2c_core cfbimgblt cfbfil
>> Jan  6 18:18:04 mail kernel: lrect [last unloaded: microcode]
>> Jan  6 18:18:04 mail kernel: Pid: 0, comm: swapper Tainted: G        
>> W  2.6.32-00840-gec8257c-dirty #41
>> Jan  6 18:18:04 mail kernel: Call Trace:
>> Jan  6 18:18:04 mail kernel: <IRQ>  [<ffffffff8105365a>] 
>> warn_slowpath_common+0x7c/0x94
>> Jan  6 18:18:04 mail kernel: [<ffffffff810536c9>] 
>> warn_slowpath_fmt+0x41/0x43
>> Jan  6 18:18:04 mail kernel: [<ffffffff813e12bf>] ? 
>> netif_tx_lock+0x44/0x6c
>> Jan  6 18:18:04 mail kernel: [<ffffffff813e1427>] 
>> dev_watchdog+0xf3/0x164
>> Jan  6 18:18:04 mail kernel: [<ffffffff81077696>] ? 
>> sched_clock_cpu+0x47/0xd1
>> Jan  6 18:18:04 mail kernel: [<ffffffff8106316b>] 
>> run_timer_softirq+0x1c8/0x270
>> Jan  6 18:18:04 mail kernel: [<ffffffff8105ae3b>] 
>> __do_softirq+0xf8/0x1cd
>> Jan  6 18:18:04 mail kernel: [<ffffffff8107ef33>] ? 
>> tick_program_event+0x2a/0x2c
>> Jan  6 18:18:04 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
>> Jan  6 18:18:04 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
>> Jan  6 18:18:04 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
>> Jan  6 18:18:04 mail kernel: [<ffffffff8146dd32>] 
>> smp_apic_timer_interrupt+0x86/0x94
>> Jan  6 18:18:04 mail kernel: [<ffffffff810127e3>] 
>> apic_timer_interrupt+0x13/0x20
>> Jan  6 18:18:04 mail kernel: <EOI>  [<ffffffff812c4a06>] ? 
>> acpi_idle_enter_c1+0xb2/0xd0
>> Jan  6 18:18:04 mail kernel: [<ffffffff812c49ff>] ? 
>> acpi_idle_enter_c1+0xab/0xd0
>> Jan  6 18:18:04 mail kernel: [<ffffffff813a43b8>] ? 
>> cpuidle_idle_call+0x9e/0xfa
>> Jan  6 18:18:04 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
>> Jan  6 18:18:04 mail kernel: [<ffffffff81463312>] ? 
>> start_secondary+0x201/0x242
>> Jan  6 18:18:04 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
>> Jan  6 18:18:04 mail kernel: sky2 eth0: tx timeout
>> Jan  6 18:18:04 mail kernel: sky2 eth0: transmit ring 21 .. 108 
>> report=21 done=21
>> Jan  6 18:18:04 mail kernel: sky2 eth0: disabling interface
>> Jan  6 18:18:04 mail kernel: sky2 eth0: enabling interface
>> <eth0 dead after this>
> Walked through the code based on Jarek's patches... came upon 
> NET_CLS_ACT. At least in some cases (sch_cbq.c for example), the net 
> transmit error could be returned from here... after releasing the skb. 
> A quick scan of the various files in net/sched suggests that with 
> NET_CLS_ACT the skb may or may not have been freed in the event of an 
> error. If I have time later I'll see if I can bypass NET_CLS_ACT and 
> see whether this is even relevant.
Ok - so rerunning with Jarek's alternative #2 patch (the one that 
doesn't re-free the skb after the net_dev_enqueue error) and all else as 
above (your mtu patch on a clean 2.6.32.2 sky2.c, plus the devtpts inode 
patch) with an MTU of 1500 I can run without errors (including the 
interrupt errors previously reported).

Changing MTU to 9000, everything basically breaks - Can't use X11 (local 
or remote - get X11 screen after gdm login locally, but then goes back 
to greeter; remote gets no greeter); ssh sessions hang; etc. This time I 
was able to reset the MTU back to 1500 without a reboot - but I did have 
to ifconfig eth0 down and then up. Looking at the sk98lin code, it looks 
to me like they do a bit more work with existing buffers before 
completing the MTU switch. Note that even doing this, X11 did not work 
(it did with the old mtu change code). Tried changing to mtu 4500 - same 
effect as 9000... but when I switched back to 1500, ksoftirqd started 
spinning using 100% of one core.

Running with these patches and 1500 MTU seems stable, but the network is 
running rather slow. Latency is OK, but peak transfer rates seem to be 
running about 20% of what I saw while receiving the interrupt errors 
with earlier patches.

I'll leave this running overnight to confirm that at least the errors 
and hangs are resolved with this patch set.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stephen Hemminger Jan. 7, 2010, 4:53 a.m. UTC | #5

On Wed, 06 Jan 2010 23:00:34 -0500
Michael Breuer <mbreuer@majjas.com> wrote:

> Changing MTU to 9000, everything basically breaks - Can't use X11 (local 
> or remote - get X11 screen after gdm login locally, but then goes back 
> to greeter; remote gets no greeter); ssh sessions hang; etc. This time I 
> was able to reset the MTU back to 1500 without a reboot - but I did have 
> to ifconfig eth0 down and then up. Looking at the sk98lin code, it looks 
> to me like they do a bit more work with existing buffers before 
> completing the MTU switch. Note that even doing this, X11 did not work 
> (it did with the old mtu change code). Tried changing to mtu 4500 - same 
> effect as 9000... but when I switched back to 1500, ksoftirqd started 
> spinning using 100% of one core.

The problem is that patch was enabling scatter-gather and checksum offload
that won't work on EC_U hardware with 9K MTU.  At least, it never worked
for me when I tested it. So because of that it really doesn't change anything
for the better on that chip version.

What version chip is on that motherboard?  Mine is:
 Yukon-2 EC Ultra chip revision 3
which corresponds to B0 step.

Another possibility is the PHY register which controls number of ticks
of buffering.  The default is zero, which gives the most buffering (good),
but the firmware could be reprogramming it (bad).  In general, the driver
doesn't fiddle with bits that are already set correctly, because sometimes
vendors need to tweak PCI timing in firmware/BIOS.  It seems the firmware on this
chip is just a bunch of register setups done on power on.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 5:10 a.m. UTC | #6

On 1/6/2010 11:53 PM, Stephen Hemminger wrote:
> On Wed, 06 Jan 2010 23:00:34 -0500
> Michael Breuer<mbreuer@majjas.com>  wrote:
>
>    
>> Changing MTU to 9000, everything basically breaks - Can't use X11 (local
>> or remote - get X11 screen after gdm login locally, but then goes back
>> to greeter; remote gets no greeter); ssh sessions hang; etc. This time I
>> was able to reset the MTU back to 1500 without a reboot - but I did have
>> to ifconfig eth0 down and then up. Looking at the sk98lin code, it looks
>> to me like they do a bit more work with existing buffers before
>> completing the MTU switch. Note that even doing this, X11 did not work
>> (it did with the old mtu change code). Tried changing to mtu 4500 - same
>> effect as 9000... but when I switched back to 1500, ksoftirqd started
>> spinning using 100% of one core.
>>      
> The problem is that patch was enabling scatter-gather and checksum offload
> that won't work on EC_U hardware with 9K MTU.  At least, it never worked
> for me when I tested it. So because of that it really doesn't change anything
> for the better on that chip version.
>
> What version chip is on that motherboard?  Mine is:
>   Yukon-2 EC Ultra chip revision 3
> which corresponds to B0 step.
>
> Another possibility is the PHY register which controls number of ticks
> of buffering.  The default is zero, which gives the most buffering (good),
> but the firmware could be reprogramming it (bad).  In general, the driver
> doesn't fiddle with bits that are already set correctly, because sometimes
> vendors need to tweak PCI timing in firmware/BIOS.  It seems the firmware on this
> chip is just a bunch of register setups done on power on.
>    
So at this point, things are working as mentioned - but really slow... 
at least an order of magnitude slower than with the other set of 
patches. The other set also generated errors and was not stable :( But, 
that set worked with mtu=9000, this set doesn't seem to work with 
anything over 1500. The slowdown may also be (based on earlier testing) 
attributable to Jarek's alternative #2 patch.  As to the chip, I *think* 
we have the same chip - I'm including lspci -vv - perhaps there is 
something different.

06:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E 
Gigabit Ethernet Controller (rev 14)
         Subsystem: Giga-byte Technology Device e000
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR+ FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 256 bytes
         Interrupt: pin A routed to IRQ 58
         Region 0: Memory at fbdfc000 (64-bit, non-prefetchable) [size=16K]
         Region 2: I/O ports at d800 [size=256]
         Expansion ROM at fbdc0000 [disabled] [size=128K]
         Capabilities: [48] Power Management version 3
                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0+,D1+,D2+,D3hot+,D3cold+)
                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
         Capabilities: [50] Vital Product Data
                 Product Name: Marvell Yukon 88E8056 Gigabit Ethernet 
Controller
                 Read-only fields:
                         [PN] Part number: Yukon 88E8056
                         [EC] Engineering changes: Rev. 1.4
                         [MN] Manufacture ID: 4d 61 72 76 65 6c 6c
                         [SN] Serial number: AbCdEfG001C3B
                         [CP] Extended capability: 01 10 cc 03
                         [RV] Reserved: checksum good, 9 byte(s) reserved
                 Read/write fields:
                         [RW] Read-write area: 121 byte(s) free
                 End
         Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
                 Address: 00000000fee00458  Data: 0000
         Capabilities: [e0] Express (v1) Legacy Endpoint, MSI 00
                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s 
unlimited, L1 unlimited
                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                         MaxPayload 128 bytes, MaxReadReq 512 bytes
                 DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ 
AuxPwr+ TransPend-
                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, 
Latency L0 <256ns, L1 unlimited
                         ClockPM+ Surprise- LLActRep- BwNot-
                 LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- 
CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
         Capabilities: [100] Advanced Error Reporting
                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
NonFatalErr+
                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
NonFatalErr+
                 AERCap: First Error Pointer: 1f, GenCap- CGenEn- 
ChkCap- ChkEn-
         Kernel driver in use: sky2
         Kernel modules: sky2


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 5:32 a.m. UTC | #7

On 1/6/2010 11:53 PM, Stephen Hemminger wrote:
> On Wed, 06 Jan 2010 23:00:34 -0500
> Michael Breuer<mbreuer@majjas.com>  wrote:
>
>    
>> Changing MTU to 9000, everything basically breaks - Can't use X11 (local
>> or remote - get X11 screen after gdm login locally, but then goes back
>> to greeter; remote gets no greeter); ssh sessions hang; etc. This time I
>> was able to reset the MTU back to 1500 without a reboot - but I did have
>> to ifconfig eth0 down and then up. Looking at the sk98lin code, it looks
>> to me like they do a bit more work with existing buffers before
>> completing the MTU switch. Note that even doing this, X11 did not work
>> (it did with the old mtu change code). Tried changing to mtu 4500 - same
>> effect as 9000... but when I switched back to 1500, ksoftirqd started
>> spinning using 100% of one core.
>>      
> The problem is that patch was enabling scatter-gather and checksum offload
> that won't work on EC_U hardware with 9K MTU.  At least, it never worked
> for me when I tested it. So because of that it really doesn't change anything
> for the better on that chip version.
>
> What version chip is on that motherboard?  Mine is:
>   Yukon-2 EC Ultra chip revision 3
> which corresponds to B0 step.
>
> Another possibility is the PHY register which controls number of ticks
> of buffering.  The default is zero, which gives the most buffering (good),
> but the firmware could be reprogramming it (bad).  In general, the driver
> doesn't fiddle with bits that are already set correctly, because sometimes
> vendors need to tweak PCI timing in firmware/BIOS.  It seems the firmware on this
> chip is just a bunch of register setups done on power on.
>    
Also - I'm seeing a huge number of dropped packets  (RX) 200-300/second. 
Probably why this is so slow.

Current ifconfig:
eth0      Link encap:Ethernet  HWaddr 00:26:18:00:1C:3B
           inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
           inet6 addr: fe80::226:18ff:fe00:1c3b/64 Scope:Link
           UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
           RX packets:26647536 errors:0 dropped:517884 overruns:0 frame:0
           TX packets:12112780 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:38960063319 (36.2 GiB)  TX bytes:1889879762 (1.7 GiB)
           Interrupt:18



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 5:54 a.m. UTC | #8

On 1/7/2010 12:32 AM, Michael Breuer wrote:
> On 1/6/2010 11:53 PM, Stephen Hemminger wrote:
>> On Wed, 06 Jan 2010 23:00:34 -0500
>> Michael Breuer<mbreuer@majjas.com>  wrote:
>>
>>> Changing MTU to 9000, everything basically breaks - Can't use X11 
>>> (local
>>> or remote - get X11 screen after gdm login locally, but then goes back
>>> to greeter; remote gets no greeter); ssh sessions hang; etc. This 
>>> time I
>>> was able to reset the MTU back to 1500 without a reboot - but I did 
>>> have
>>> to ifconfig eth0 down and then up. Looking at the sk98lin code, it 
>>> looks
>>> to me like they do a bit more work with existing buffers before
>>> completing the MTU switch. Note that even doing this, X11 did not work
>>> (it did with the old mtu change code). Tried changing to mtu 4500 - 
>>> same
>>> effect as 9000... but when I switched back to 1500, ksoftirqd started
>>> spinning using 100% of one core.
>> The problem is that patch was enabling scatter-gather and checksum 
>> offload
>> that won't work on EC_U hardware with 9K MTU.  At least, it never worked
>> for me when I tested it. So because of that it really doesn't change 
>> anything
>> for the better on that chip version.
>>
>> What version chip is on that motherboard?  Mine is:
>>   Yukon-2 EC Ultra chip revision 3
>> which corresponds to B0 step.
>>
>> Another possibility is the PHY register which controls number of ticks
>> of buffering.  The default is zero, which gives the most buffering 
>> (good),
>> but the firmware could be reprogramming it (bad).  In general, the 
>> driver
>> doesn't fiddle with bits that are already set correctly, because 
>> sometimes
>> vendors need to tweak PCI timing in firmware/BIOS.  It seems the 
>> firmware on this
>> chip is just a bunch of register setups done on power on.
> Also - I'm seeing a huge number of dropped packets  (RX) 
> 200-300/second. Probably why this is so slow.
>
> Current ifconfig:
> eth0      Link encap:Ethernet  HWaddr 00:26:18:00:1C:3B
>           inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
>           inet6 addr: fe80::226:18ff:fe00:1c3b/64 Scope:Link
>           UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
>           RX packets:26647536 errors:0 dropped:517884 overruns:0 frame:0
>           TX packets:12112780 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:38960063319 (36.2 GiB)  TX bytes:1889879762 (1.7 GiB)
>           Interrupt:18
>
>
>

Never mind... spoke too soon. Crashed again. Just took longer:
Jan  7 00:37:39 mail kernel: DRHD: handling fault status reg 2
Jan  7 00:37:39 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
fault addr fff1401fe000
Jan  7 00:37:39 mail kernel: DMAR:[fault reason 06] PTE Read access is 
not set
Jan  7 00:37:39 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x80000000
Jan  7 00:37:39 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
Jan  7 00:37:40 mail smbd[4729]: [2010/01/07 00:37:40,  0] 
lib/util_sock.c:539(read_fd_with_timeout)
Jan  7 00:37:40 mail smbd[4729]: [2010/01/07 00:37:40,  0] 
lib/util_sock.c:1491(get_peer_addr_internal)
Jan  7 00:37:40 mail smbd[4729]:   getpeername failed. Error was 
Transport endpoint is not connected
Jan  7 00:37:40 mail smbd[4729]:   read_fd_with_timeout: client 0.0.0.0 
read error = Connection timed out.
Jan  7 00:37:40 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 
00:26:bb:aa:15:10 (mbitouch) via eth0
Jan  7 00:37:40 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 
(mbitouch) via eth0
Jan  7 00:37:41 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 
00:26:bb:aa:15:10 (mbitouch) via eth0
Jan  7 00:37:41 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 
(mbitouch) via eth0
Jan  7 00:37:44 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 
00:26:bb:aa:15:10 (mbitouch) via eth0
Jan  7 00:37:44 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 
(mbitouch) via eth0
Jan  7 00:37:47 mail kernel: ------------[ cut here ]------------
Jan  7 00:37:47 mail kernel: WARNING: at net/sched/sch_generic.c:261 
dev_watchdog+0xf3/0x164()
Jan  7 00:37:47 mail kernel: Hardware name: System Product Name
Jan  7 00:37:47 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
queue 0 timed out
Jan  7 00:37:47 mail kernel: Modules linked in: ip6table_filter 
ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat 
iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd 
nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit 
tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp 
xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm 
snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_ens1371 gameport 
snd_rawmidi gspca_spca505 gspca_main pcspkr snd_ac97_codec snd_hwdep 
i2c_i801 snd_seq firewire_ohci videodev v4l1_compat snd_seq_device 
v4l2_compat_ioctl32 ac97_bus firewire_core crc_itu_t iTCO_wdt snd_pcm 
iTCO_vendor_support wmi snd_timer snd sky2 asus_atk0110 hwmon soundcore 
snd_page_alloc fbcon tileblit font bitblit softcursor raid456 
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx 
raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm 
agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbfil
Jan  7 00:37:47 mail kernel: lrect [last unloaded: microcode]
Jan  7 00:37:47 mail kernel: Pid: 0, comm: swapper Tainted: G        W  
2.6.32-00840-gec8257c-dirty #44
Jan  7 00:37:47 mail kernel: Call Trace:
Jan  7 00:37:47 mail kernel: <IRQ>  [<ffffffff8105365a>] 
warn_slowpath_common+0x7c/0x94
Jan  7 00:37:47 mail kernel: [<ffffffff810536c9>] 
warn_slowpath_fmt+0x41/0x43
Jan  7 00:37:47 mail kernel: [<ffffffff813e12bf>] ? netif_tx_lock+0x44/0x6c
Jan  7 00:37:47 mail kernel: [<ffffffff813e1427>] dev_watchdog+0xf3/0x164
Jan  7 00:37:47 mail kernel: [<ffffffff8105f320>] ? cascade+0x6a/0x84
Jan  7 00:37:47 mail kernel: [<ffffffff8106316b>] 
run_timer_softirq+0x1c8/0x270
Jan  7 00:37:47 mail kernel: [<ffffffff8105ae3b>] __do_softirq+0xf8/0x1cd
Jan  7 00:37:47 mail kernel: [<ffffffff8107ef33>] ? 
tick_program_event+0x2a/0x2c
Jan  7 00:37:47 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan  7 00:37:47 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan  7 00:37:47 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
Jan  7 00:37:47 mail kernel: [<ffffffff8146dd32>] 
smp_apic_timer_interrupt+0x86/0x94
Jan  7 00:37:47 mail kernel: [<ffffffff810127e3>] 
apic_timer_interrupt+0x13/0x20
Jan  7 00:37:47 mail kernel: <EOI>  [<ffffffff812c4c7a>] ? 
acpi_idle_enter_bm+0x256/0x28a
Jan  7 00:37:47 mail kernel: [<ffffffff812c4c73>] ? 
acpi_idle_enter_bm+0x24f/0x28a
Jan  7 00:37:47 mail kernel: [<ffffffff813a43b8>] ? 
cpuidle_idle_call+0x9e/0xfa
Jan  7 00:37:47 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan  7 00:37:47 mail kernel: [<ffffffff81463312>] ? 
start_secondary+0x201/0x242
Jan  7 00:37:47 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
Jan  7 00:37:47 mail kernel: sky2 eth0: tx timeout
Jan  7 00:37:47 mail kernel: sky2 eth0: transmit ring 79 .. 39 report=79 
done=79
Jan  7 00:37:47 mail kernel: sky2 eth0: disabling interface
Jan  7 00:37:47 mail kernel: sky2 eth0: enabling interface
Jan  7 00:37:51 mail kernel: sky2 eth0: Link is up at 1000 Mbps, full 
duplex, flow control both
JJan  7 00:38:07 mail kernel: sky2 eth0: tx timeout
Jan  7 00:38:07 mail kernel: sky2 eth0: transmit ring 3 .. 90 report=3 
done=3
Jan  7 00:38:07 mail kernel: sky2 eth0: disabling interface
Jan  7 00:38:07 mail kernel: sky2 eth0: enabling interface
and so on.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 7:20 a.m. UTC | #9

On 1/7/2010 12:54 AM, Michael Breuer wrote:
> On 1/7/2010 12:32 AM, Michael Breuer wrote:
>> On 1/6/2010 11:53 PM, Stephen Hemminger wrote:
>>> On Wed, 06 Jan 2010 23:00:34 -0500
>>> Michael Breuer<mbreuer@majjas.com>  wrote:
>>>
>>>> Changing MTU to 9000, everything basically breaks - Can't use X11 
>>>> (local
>>>> or remote - get X11 screen after gdm login locally, but then goes back
>>>> to greeter; remote gets no greeter); ssh sessions hang; etc. This 
>>>> time I
>>>> was able to reset the MTU back to 1500 without a reboot - but I did 
>>>> have
>>>> to ifconfig eth0 down and then up. Looking at the sk98lin code, it 
>>>> looks
>>>> to me like they do a bit more work with existing buffers before
>>>> completing the MTU switch. Note that even doing this, X11 did not work
>>>> (it did with the old mtu change code). Tried changing to mtu 4500 - 
>>>> same
>>>> effect as 9000... but when I switched back to 1500, ksoftirqd started
>>>> spinning using 100% of one core.
>>> The problem is that patch was enabling scatter-gather and checksum 
>>> offload
>>> that won't work on EC_U hardware with 9K MTU.  At least, it never 
>>> worked
>>> for me when I tested it. So because of that it really doesn't change 
>>> anything
>>> for the better on that chip version.
>>>
>>> What version chip is on that motherboard?  Mine is:
>>>   Yukon-2 EC Ultra chip revision 3
>>> which corresponds to B0 step.
>>>
>>> Another possibility is the PHY register which controls number of ticks
>>> of buffering.  The default is zero, which gives the most buffering 
>>> (good),
>>> but the firmware could be reprogramming it (bad).  In general, the 
>>> driver
>>> doesn't fiddle with bits that are already set correctly, because 
>>> sometimes
>>> vendors need to tweak PCI timing in firmware/BIOS.  It seems the 
>>> firmware on this
>>> chip is just a bunch of register setups done on power on.
>> Also - I'm seeing a huge number of dropped packets  (RX) 
>> 200-300/second. Probably why this is so slow.
>>
>> Current ifconfig:
>> eth0      Link encap:Ethernet  HWaddr 00:26:18:00:1C:3B
>>           inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
>>           inet6 addr: fe80::226:18ff:fe00:1c3b/64 Scope:Link
>>           UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
>>           RX packets:26647536 errors:0 dropped:517884 overruns:0 frame:0
>>           TX packets:12112780 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 txqueuelen:1000
>>           RX bytes:38960063319 (36.2 GiB)  TX bytes:1889879762 (1.7 GiB)
>>           Interrupt:18
>>
>>
>>
>
> Never mind... spoke too soon. Crashed again. Just took longer:
...
Reapplied a couple of earlier patches - still can't do jumbo frames, but 
the rx errors are gone and speed has improved. Too early to assure that 
it's stable.

Patches that seem to fix the rx drops (all from Stephen):
1) Patch change to tx_init
2) Patch to lock netif_device_detach
3) Patch to sky2_tx_complete to add netif_device_present test
Also in the mix: Jarek's alternative 2

With this set and mtu=1500 all seems good - decent if not stellar 
throughput; no logged errors; no reported packet loss. As before, will 
leave running and see if anything falls apart.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 7:47 a.m. UTC | #10

On Thu, Jan 07, 2010 at 02:20:22AM -0500, Michael Breuer wrote:
> ...
> Reapplied a couple of earlier patches - still can't do jumbo frames, but  
> the rx errors are gone and speed has improved. Too early to assure that  
> it's stable.
>
> Patches that seem to fix the rx drops (all from Stephen):
> 1) Patch change to tx_init
> 2) Patch to lock netif_device_detach
> 3) Patch to sky2_tx_complete to add netif_device_present test
> Also in the mix: Jarek's alternative 2

BTW, the main difference between alt. 1 and 2 is error notification:
alternative 2 doesn't hide some (most) of drops, so, dependending on
app, there might be more and faster retransmits. (I don't know what
apps used by you (other than dhcp) can depend so much on this.)

>
> With this set and mtu=1500 all seems good - decent if not stellar  
> throughput; no logged errors; no reported packet loss. As before, will  
> leave running and see if anything falls apart.

Good news!

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 7:55 a.m. UTC | #11

On 1/7/2010 2:47 AM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 02:20:22AM -0500, Michael Breuer wrote:
>    
>> ...
>> Reapplied a couple of earlier patches - still can't do jumbo frames, but
>> the rx errors are gone and speed has improved. Too early to assure that
>> it's stable.
>>
>> Patches that seem to fix the rx drops (all from Stephen):
>> 1) Patch change to tx_init
>> 2) Patch to lock netif_device_detach
>> 3) Patch to sky2_tx_complete to add netif_device_present test
>> Also in the mix: Jarek's alternative 2
>>      
> BTW, the main difference between alt. 1 and 2 is error notification:
> alternative 2 doesn't hide some (most) of drops, so, dependending on
> app, there might be more and faster retransmits. (I don't know what
> apps used by you (other than dhcp) can depend so much on this.)
>
>    
Unless I misread the code, I think that in some cases e skb is actually 
freed if the cfq (among others perhaps) scheduler returns an error on 
enqueue (flow control perhaps).  Thus with alternative 1, it is possible 
that the skb is acted upon after being freed - this would be consistent 
with the DMAR errors I saw.

>> With this set and mtu=1500 all seems good - decent if not stellar
>> throughput; no logged errors; no reported packet loss. As before, will
>> leave running and see if anything falls apart.
>>      
> Good news!
>
> Jarek P.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>    

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 8:21 a.m. UTC | #12

On Thu, Jan 07, 2010 at 02:55:20AM -0500, Michael Breuer wrote:
> On 1/7/2010 2:47 AM, Jarek Poplawski wrote:
>> On Thu, Jan 07, 2010 at 02:20:22AM -0500, Michael Breuer wrote:
>>    
>>> ...
>>> Reapplied a couple of earlier patches - still can't do jumbo frames, but
>>> the rx errors are gone and speed has improved. Too early to assure that
>>> it's stable.
>>>
>>> Patches that seem to fix the rx drops (all from Stephen):
>>> 1) Patch change to tx_init
>>> 2) Patch to lock netif_device_detach
>>> 3) Patch to sky2_tx_complete to add netif_device_present test
>>> Also in the mix: Jarek's alternative 2
>>>      
>> BTW, the main difference between alt. 1 and 2 is error notification:
>> alternative 2 doesn't hide some (most) of drops, so, dependending on
>> app, there might be more and faster retransmits. (I don't know what
>> apps used by you (other than dhcp) can depend so much on this.)
>>
>>    
> Unless I misread the code, I think that in some cases e skb is actually  
> freed if the cfq (among others perhaps) scheduler returns an error on  
> enqueue (flow control perhaps).  Thus with alternative 1, it is possible  
> that the skb is acted upon after being freed - this would be consistent  
> with the DMAR errors I saw.

I can't see your point: could you give some scenario?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 3:03 p.m. UTC | #13

On 1/7/2010 3:21 AM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 02:55:20AM -0500, Michael Breuer wrote:
>    
>> On 1/7/2010 2:47 AM, Jarek Poplawski wrote:
>>      
>>> On Thu, Jan 07, 2010 at 02:20:22AM -0500, Michael Breuer wrote:
>>>
>>>        
>>>> ...
>>>> Reapplied a couple of earlier patches - still can't do jumbo frames, but
>>>> the rx errors are gone and speed has improved. Too early to assure that
>>>> it's stable.
>>>>
>>>> Patches that seem to fix the rx drops (all from Stephen):
>>>> 1) Patch change to tx_init
>>>> 2) Patch to lock netif_device_detach
>>>> 3) Patch to sky2_tx_complete to add netif_device_present test
>>>> Also in the mix: Jarek's alternative 2
>>>>
>>>>          
>>> BTW, the main difference between alt. 1 and 2 is error notification:
>>> alternative 2 doesn't hide some (most) of drops, so, dependending on
>>> app, there might be more and faster retransmits. (I don't know what
>>> apps used by you (other than dhcp) can depend so much on this.)
>>>
>>>
>>>        
>> Unless I misread the code, I think that in some cases e skb is actually
>> freed if the cfq (among others perhaps) scheduler returns an error on
>> enqueue (flow control perhaps).  Thus with alternative 1, it is possible
>> that the skb is acted upon after being freed - this would be consistent
>> with the DMAR errors I saw.
>>      
> I can't see your point: could you give some scenario?
>
> Jarek P.
>    
With NET_CLS_ACT set, net_dev_enqueue can return an error after freeing 
the skb. Alternative 1 disregards the error and assumes the skb is still 
valid. The original code and alternative 2 exit the loop assuming the 
skb has been freed.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 3:05 p.m. UTC | #14

On 1/7/2010 2:55 AM, Michael Breuer wrote:
> On 1/7/2010 2:47 AM, Jarek Poplawski wrote:
>> On Thu, Jan 07, 2010 at 02:20:22AM -0500, Michael Breuer wrote:
>>> ...
>>> Reapplied a couple of earlier patches - still can't do jumbo frames, 
>>> but
>>> the rx errors are gone and speed has improved. Too early to assure that
>>> it's stable.
>>>
>>> Patches that seem to fix the rx drops (all from Stephen):
>>> 1) Patch change to tx_init
>>> 2) Patch to lock netif_device_detach
>>> 3) Patch to sky2_tx_complete to add netif_device_present test
>>> Also in the mix: Jarek's alternative 2
>> BTW, the main difference between alt. 1 and 2 is error notification:
>> alternative 2 doesn't hide some (most) of drops, so, dependending on
>> app, there might be more and faster retransmits. (I don't know what
>> apps used by you (other than dhcp) can depend so much on this.)
>>
> Unless I misread the code, I think that in some cases e skb is 
> actually freed if the cfq (among others perhaps) scheduler returns an 
> error on enqueue (flow control perhaps).  Thus with alternative 1, it 
> is possible that the skb is acted upon after being freed - this would 
> be consistent with the DMAR errors I saw.
>
>>> With this set and mtu=1500 all seems good - decent if not stellar
>>> throughput; no logged errors; no reported packet loss. As before, will
>>> leave running and see if anything falls apart.
>> Good news!
>>
>> Jarek P.
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
Bad news - crashed about an hour after I wrote this email - under load - 
same crash as before. Network watchdog... lots of attempts to reset the 
adapter... then hw watchdog rebooted the system.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 5:56 p.m. UTC | #15

On Thu, Jan 07, 2010 at 10:03:51AM -0500, Michael Breuer wrote:
> On 1/7/2010 3:21 AM, Jarek Poplawski wrote:
> >On Thu, Jan 07, 2010 at 02:55:20AM -0500, Michael Breuer wrote:
> >>Unless I misread the code, I think that in some cases e skb is actually
> >>freed if the cfq (among others perhaps) scheduler returns an error on
> >>enqueue (flow control perhaps).  Thus with alternative 1, it is possible
> >>that the skb is acted upon after being freed - this would be consistent
> >>with the DMAR errors I saw.
> >I can't see your point: could you give some scenario?
> >
> >Jarek P.
> With NET_CLS_ACT set, net_dev_enqueue can return an error after
> freeing the skb. Alternative 1 disregards the error and assumes the
> skb is still valid. The original code and alternative 2 exit the
> loop assuming the skb has been freed.

Not exactly: alternative 1 disregards the error, and tries to send
next skbs if the message was longer. After consuming all the message
it returns without err code (at least wrt. dev_queue_xmit). This is
quite often practice to skip dev_queue_xmit() return (try to grep in
net\). It should never touch any part of an earlier sent skb.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 6:01 p.m. UTC | #16

On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
> Bad news - crashed about an hour after I wrote this email - under
> load - same crash as before. Network watchdog... lots of attempts to
> reset the adapter... then hw watchdog rebooted the system.

It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
if you find time.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 6:17 p.m. UTC | #17

On Thu, Jan 07, 2010 at 06:56:10PM +0100, Jarek Poplawski wrote:
> [...] (try to grep in net\). [...]

Did I say "net\"?!

Extremely sorry! ;-)
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 6:19 p.m. UTC | #18

On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
>    
>> Bad news - crashed about an hour after I wrote this email - under
>> load - same crash as before. Network watchdog... lots of attempts to
>> reset the adapter... then hw watchdog rebooted the system.
>>      
> It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
> if you find time.
>
> Jarek P.
>    
Ok - any particular patch set to try with? I'm going to start with a 
clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but can't 
get a usable console... will look at that later.)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 6:35 p.m. UTC | #19

On Thu, Jan 07, 2010 at 01:19:41PM -0500, Michael Breuer wrote:
> On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
> >On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
> >>Bad news - crashed about an hour after I wrote this email - under
> >>load - same crash as before. Network watchdog... lots of attempts to
> >>reset the adapter... then hw watchdog rebooted the system.
> >It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
> >if you find time.
> >
> >Jarek P.
> Ok - any particular patch set to try with? I'm going to start with a
> clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but
> can't get a usable console... will look at that later.)

My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW,
could you remind if it worked any better with 2.6.31 or earlier?)

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 6:40 p.m. UTC | #20

On 1/7/2010 1:35 PM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 01:19:41PM -0500, Michael Breuer wrote:
>    
>> On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
>>      
>>> On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
>>>        
>>>> Bad news - crashed about an hour after I wrote this email - under
>>>> load - same crash as before. Network watchdog... lots of attempts to
>>>> reset the adapter... then hw watchdog rebooted the system.
>>>>          
>>> It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
>>> if you find time.
>>>
>>> Jarek P.
>>>        
>> Ok - any particular patch set to try with? I'm going to start with a
>> clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but
>> can't get a usable console... will look at that later.)
>>      
> My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW,
> could you remind if it worked any better with 2.6.31 or earlier?)
>
> Jarek P.
>    
   Never tried earlier than 2.6.31 - new machine. I had a bunch of 
issues on 2.6.31, including sky2 DMAR errors... so probably not a 2.6.32 
regression.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 6:43 p.m. UTC | #21

On 1/7/2010 1:35 PM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 01:19:41PM -0500, Michael Breuer wrote:
>    
>> On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
>>      
>>> On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
>>>        
>>>> Bad news - crashed about an hour after I wrote this email - under
>>>> load - same crash as before. Network watchdog... lots of attempts to
>>>> reset the adapter... then hw watchdog rebooted the system.
>>>>          
>>> It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
>>> if you find time.
>>>
>>> Jarek P.
>>>        
>> Ok - any particular patch set to try with? I'm going to start with a
>> clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but
>> can't get a usable console... will look at that later.)
>>      
> My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW,
> could you remind if it worked any better with 2.6.31 or earlier?)
>
> Jarek P.
>    
I'm not sure my crash-and-burn runs yesterday included the pskb_may_pull 
patch :(

Going to rerun with these patches and with and without MMAP. Will also 
retry both with jumbo frames if possible.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 6:50 p.m. UTC | #22

On Thu, Jan 07, 2010 at 01:43:08PM -0500, Michael Breuer wrote:
> On 1/7/2010 1:35 PM, Jarek Poplawski wrote:
> >On Thu, Jan 07, 2010 at 01:19:41PM -0500, Michael Breuer wrote:
> >>On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
> >>>On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
> >>>>Bad news - crashed about an hour after I wrote this email - under
> >>>>load - same crash as before. Network watchdog... lots of attempts to
> >>>>reset the adapter... then hw watchdog rebooted the system.
> >>>It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
> >>>if you find time.
> >>>
> >>>Jarek P.
> >>Ok - any particular patch set to try with? I'm going to start with a
> >>clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but
> >>can't get a usable console... will look at that later.)
> >My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW,
> >could you remind if it worked any better with 2.6.31 or earlier?)
> >
> >Jarek P.
> I'm not sure my crash-and-burn runs yesterday included the
> pskb_may_pull patch :(
> 
> Going to rerun with these patches and with and without MMAP. Will
> also retry both with jumbo frames if possible.

If MMAP then some "alternative" too. But first no MMAP.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 7:36 p.m. UTC | #23

On Thu, Jan 07, 2010 at 07:50:40PM +0100, Jarek Poplawski wrote:
> > Going to rerun with these patches and with and without MMAP. Will
> > also retry both with jumbo frames if possible.
> 
> If MMAP then some "alternative" too. But first no MMAP.

Another things IMHO worth to try: a sky2 module parameter
"disable_msi=1", and CONFIG_DMAR off.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 7:55 p.m. UTC | #24

On 1/7/2010 2:36 PM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 07:50:40PM +0100, Jarek Poplawski wrote:
>    
>>> Going to rerun with these patches and with and without MMAP. Will
>>> also retry both with jumbo frames if possible.
>>>        
>> If MMAP then some "alternative" too. But first no MMAP.
>>      
> Another things IMHO worth to try: a sky2 module parameter
> "disable_msi=1", and CONFIG_DMAR off.
>
> Jarek P.
>    
Ok - that'd be with or without MMAP enabled? (note that so-far, without 
MMAP I'm not seeing any errors - throughput is running about half what I 
was seeing with MMAP enabled (before crashing that is). CPU is also way 
busier (also to be expected). One other observation - I had been seeing 
lots of DNS errors - IPV6 related format errors I really didn't think 
much of it as they were mostly .ru and seemed spam-related, but now I 
don't see any. Haven't updated bind; doubt the world has changed - so 
perhaps this is related to the network issue.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 7, 2010, 8:22 p.m. UTC | #25

On Thu, Jan 07, 2010 at 02:55:22PM -0500, Michael Breuer wrote:
> On 1/7/2010 2:36 PM, Jarek Poplawski wrote:
> >On Thu, Jan 07, 2010 at 07:50:40PM +0100, Jarek Poplawski wrote:
> >>>Going to rerun with these patches and with and without MMAP. Will
> >>>also retry both with jumbo frames if possible.
> >>If MMAP then some "alternative" too. But first no MMAP.
> >Another things IMHO worth to try: a sky2 module parameter
> >"disable_msi=1", and CONFIG_DMAR off.
> >
> >Jarek P.
> Ok - that'd be with or without MMAP enabled? (note that so-far,
> without MMAP I'm not seeing any errors - throughput is running about
> half what I was seeing with MMAP enabled (before crashing that is).
> CPU is also way busier (also to be expected). One other observation
> - I had been seeing lots of DNS errors - IPV6 related format errors
> I really didn't think much of it as they were mostly .ru and seemed
> spam-related, but now I don't see any. Haven't updated bind; doubt
> the world has changed - so perhaps this is related to the network
> issue.

MMAP enabled (with some "alternative" patch - to avoid known bugs)
should give as earlier the answer if these changes matter. But first
let's try longer (if possible) if this "no MMAP" could really heal
your hardware.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 7, 2010, 11:11 p.m. UTC | #26

On 1/7/2010 1:50 PM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 01:43:08PM -0500, Michael Breuer wrote:
>    
>> On 1/7/2010 1:35 PM, Jarek Poplawski wrote:
>>      
>>> On Thu, Jan 07, 2010 at 01:19:41PM -0500, Michael Breuer wrote:
>>>        
>>>> On 1/7/2010 1:01 PM, Jarek Poplawski wrote:
>>>>          
>>>>> On Thu, Jan 07, 2010 at 10:05:37AM -0500, Michael Breuer wrote:
>>>>>            
>>>>>> Bad news - crashed about an hour after I wrote this email - under
>>>>>> load - same crash as before. Network watchdog... lots of attempts to
>>>>>> reset the adapter... then hw watchdog rebooted the system.
>>>>>>              
>>>>> It's a pity. Anyway, I'd be still interested in CONFIG_PACKET_MMAP off
>>>>> if you find time.
>>>>>
>>>>> Jarek P.
>>>>>            
>>>> Ok - any particular patch set to try with? I'm going to start with a
>>>> clean tree using the latest 2.6.32 from git (tried 2.6.33-rc3, but
>>>> can't get a usable console... will look at that later.)
>>>>          
>>> My "Berck E. Nash" and Stephen's "pskb_may_pull" sky2 patches. (BTW,
>>> could you remind if it worked any better with 2.6.31 or earlier?)
>>>
>>> Jarek P.
>>>        
>> I'm not sure my crash-and-burn runs yesterday included the
>> pskb_may_pull patch :(
>>
>> Going to rerun with these patches and with and without MMAP. Will
>> also retry both with jumbo frames if possible.
>>      
> If MMAP then some "alternative" too. But first no MMAP.
>
> Jarek P.
>    
Results:
* no MMAP, mtu=1500, neither alternative patch loaded: adapter crashed:
Jan  7 15:44:23 mail kernel: DRHD: handling fault status reg 2
Jan  7 15:44:23 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
fault addr fffb7bffe000
Jan  7 15:44:23 mail kernel: DMAR:[fault reason 06] PTE Read access is 
not set
Jan  7 15:44:23 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x80000000
Jan  7 15:44:23 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
Jan  7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24,  0] 
lib/util_sock.c:539(read_fd_with_timeout)
Jan  7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24,  0] 
lib/util_sock.c:1491(get_peer_addr_internal)
Jan  7 15:44:24 mail smbd[6572]:   getpeername failed. Error was 
Transport endpoint is not connected
Jan  7 15:44:24 mail smbd[6572]:   read_fd_with_timeout: client 0.0.0.0 
read error = Connection timed out.
Jan  7 15:44:44 mail kernel: ------------[ cut here ]------------
Jan  7 15:44:44 mail kernel: WARNING: at net/sched/sch_generic.c:261 
dev_watchdog+0xf3/0x164()
Jan  7 15:44:44 mail kernel: Hardware name: System Product Name
Jan  7 15:44:44 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
queue 0 timed out
Jan  7 15:44:44 mail kernel: Modules linked in: ip6table_filter 
ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat 
iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd 
nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit 
tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp 
xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm 
snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec 
snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq snd_seq_device 
snd_pcm gspca_spca505 gspca_main firewire_ohci videodev v4l1_compat 
firewire_core pcspkr v4l2_compat_ioctl32 snd_timer iTCO_wdt i2c_i801 
crc_itu_t iTCO_vendor_support snd soundcore snd_page_alloc sky2 wmi 
asus_atk0110 hwmon fbcon tileblit font bitblit softcursor raid456 
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx 
raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm 
agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbfil
Jan  7 15:44:44 mail kernel: lrect [last unloaded: microcode]
Jan  7 15:44:44 mail kernel: Pid: 0, comm: swapper Tainted: G        W  
2.6.32NOMMAP-00847-g50ebb93-dirty #4
Jan  7 15:44:44 mail kernel: Call Trace:
Jan  7 15:44:44 mail kernel: <IRQ>  [<ffffffff8105365a>] 
warn_slowpath_common+0x7c/0x94
Jan  7 15:44:44 mail kernel: [<ffffffff810536c9>] 
warn_slowpath_fmt+0x41/0x43
Jan  7 15:44:44 mail kernel: [<ffffffff813e2dcf>] ? netif_tx_lock+0x44/0x6c
Jan  7 15:44:44 mail kernel: [<ffffffff813e2f37>] dev_watchdog+0xf3/0x164
Jan  7 15:44:44 mail kernel: [<ffffffff8106e8a4>] ? __queue_work+0x3a/0x42
Jan  7 15:44:44 mail kernel: [<ffffffff8106316b>] 
run_timer_softirq+0x1c8/0x270
Jan  7 15:44:44 mail kernel: [<ffffffff8105ae3b>] __do_softirq+0xf8/0x1cd
Jan  7 15:44:44 mail kernel: [<ffffffff8107ef33>] ? 
tick_program_event+0x2a/0x2c
Jan  7 15:44:44 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan  7 15:44:44 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan  7 15:44:44 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
Jan  7 15:44:44 mail kernel: [<ffffffff8146e3e2>] 
smp_apic_timer_interrupt+0x86/0x94
Jan  7 15:44:44 mail kernel: [<ffffffff810127e3>] 
apic_timer_interrupt+0x13/0x20
Jan  7 15:44:44 mail kernel: <EOI>  [<ffffffff812c678a>] ? 
acpi_idle_enter_bm+0x256/0x28a
Jan  7 15:44:44 mail kernel: [<ffffffff812c6783>] ? 
acpi_idle_enter_bm+0x24f/0x28a
Jan  7 15:44:44 mail kernel: [<ffffffff813a5ec8>] ? 
cpuidle_idle_call+0x9e/0xfa
Jan  7 15:44:44 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan  7 15:44:44 mail kernel: [<ffffffff814639c2>] ? 
start_secondary+0x201/0x242
Jan  7 15:44:44 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
Jan  7 15:44:44 mail kernel: sky2 eth0: tx timeout
Jan  7 15:44:44 mail kernel: sky2 eth0: transmit ring 77 .. 36 report=77 
done=77
Jan  7 15:44:44 mail kernel: sky2 eth0: disabling interface
Jan  7 15:44:44 mail kernel: sky2 eth0: enabling interface
--- adapter dead after this --- rebooted.
* no MMAP; alternative 1 patch, mtu=1500; no errors; sustained transfer 
rates about 25% lower than what I saw with mmap enabled...(before MMAP 
enabled crashed).
* no MMAP mtu=9000; ran ok at low transfer rates - when high rates 
kicked in, got the sky2 interrupt error & things went south:
Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x40000008
Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0x40000008
After this, remote connections broke and I rebooted... decided to rerun 
w/o MMAP again before going back to MMAP and trying those other sky2 
options...
* Retest of no MMAP + Alternative 1 - just to confirm consistency. 
Worked - no errors. Only version so far that allows the win7 backup to 
complete.
* MMAP + NO DMAR + disable_msi=1... also works w/o errors... leaving 
this one running for a while - also completed a backup successfully. 
Fastest of the lot... about 3x faster than any other version, working or 
not.

I'm leaving this one running for now. Not retesting jumbo for now. Be 
happy to help dig further.

Tentative recommendations:

1) The af alternative patch seems rather necessary. First alternative 
seems to be working, I'd suggest that be submitted and backported to 2.6.32.
2) Steven's pskb_may_pull patch also ought to be included and backported.
3) Jumbo frame support for yukon2 should probably be disabled until/if 
fixed.
4) When possible I'll test dmar and disable_msi, and no dmar and no 
disable_msi. When I first hit issues, I was running without DMAR, but 
also without the above patches. I suppose the non-working permutations 
need to be either fixed or invalidated (or well documented).
5) It would be nice if someone with comparable hardware could reproduce 
these issues. FWIW, I can only recreate the crash running windows backup 
to a cifs share. Copying large files doesn't seem to do it.  Could also 
be some other interaction going on here that perhaps others aren't 
running - would be happy to compare notes.

Notes:
This *could* be coincidental, but maybe not...
With MMAP+NO DMAR + disable_msi there are far fewer ... actually almost 
no... bind error reports... and no bind format error messages. With 
NOMMAP and alternative one there are a few more bind error messages and 
one format error message during the several hours that version was up. 
All other configurations going back perhaps for two weeks have 
significantly more bind error reports - and all versions show increasing 
frequency of bind format errors (IPV6 only) in the roughly 10-15 minutes 
preceding the lockup/crash/interrupt error messages. There are none 
immediately preceding any crash, but perhaps there is some correlation 
between the network errors and bind ipv6 packets.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 8, 2010, 7:45 a.m. UTC | #27

On Thu, Jan 07, 2010 at 06:11:34PM -0500, Michael Breuer wrote:
> Results:
> * no MMAP, mtu=1500, neither alternative patch loaded: adapter crashed:
> Jan  7 15:44:23 mail kernel: DRHD: handling fault status reg 2
> Jan  7 15:44:23 mail kernel: DMAR:[DMA Read] Request device [06:00.0]  
> fault addr fffb7bffe000
> Jan  7 15:44:23 mail kernel: DMAR:[fault reason 06] PTE Read access is  
> not set
> Jan  7 15:44:23 mail kernel: sky2 0000:06:00.0: error interrupt  
> status=0x80000000
> Jan  7 15:44:23 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
> Jan  7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24,  0]  
> lib/util_sock.c:539(read_fd_with_timeout)
> Jan  7 15:44:24 mail smbd[6572]: [2010/01/07 15:44:24,  0]  
> lib/util_sock.c:1491(get_peer_addr_internal)
> Jan  7 15:44:24 mail smbd[6572]:   getpeername failed. Error was  
> Transport endpoint is not connected
> Jan  7 15:44:24 mail smbd[6572]:   read_fd_with_timeout: client 0.0.0.0  
> read error = Connection timed out.
> Jan  7 15:44:44 mail kernel: ------------[ cut here ]------------
> Jan  7 15:44:44 mail kernel: WARNING: at net/sched/sch_generic.c:261  
> dev_watchdog+0xf3/0x164()
> Jan  7 15:44:44 mail kernel: Hardware name: System Product Name
> Jan  7 15:44:44 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit  
> queue 0 timed out
> Jan  7 15:44:44 mail kernel: Modules linked in: ip6table_filter  
> ip6table_mangle ip6_tables ipt_MASQUERADE iptable_nat nf_nat  
> iptable_mangle iptable_raw bridge stp appletalk psnap llc nfsd lockd  
> nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit  
> tunnel4 ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp xt_DSCP xt_dscp  
> xt_MARK nf_conntrack_ipv6 xt_multiport ipv6 dm_multipath kvm_intel kvm  
> snd_hda_codec_analog snd_ens1371 gameport snd_rawmidi snd_ac97_codec  
> snd_hda_intel snd_hda_codec ac97_bus snd_hwdep snd_seq snd_seq_device  
> snd_pcm gspca_spca505 gspca_main firewire_ohci videodev v4l1_compat  
> firewire_core pcspkr v4l2_compat_ioctl32 snd_timer iTCO_wdt i2c_i801  
> crc_itu_t iTCO_vendor_support snd soundcore snd_page_alloc sky2 wmi  
> asus_atk0110 hwmon fbcon tileblit font bitblit softcursor raid456  
> async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx  
> raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm  
> agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbfil
> Jan  7 15:44:44 mail kernel: lrect [last unloaded: microcode]
> Jan  7 15:44:44 mail kernel: Pid: 0, comm: swapper Tainted: G        W   

BTW, was there any other oops saved before this one?

...
> --- adapter dead after this --- rebooted.
> * no MMAP; alternative 1 patch, mtu=1500; no errors; sustained transfer  
> rates about 25% lower than what I saw with mmap enabled...(before MMAP  
> enabled crashed).

?? Read below...

> * no MMAP mtu=9000; ran ok at low transfer rates - when high rates  
> kicked in, got the sky2 interrupt error & things went south:
> Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt  
> status=0x40000008
> Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt  
> status=0x40000008
> After this, remote connections broke and I rebooted... decided to rerun  
> w/o MMAP again before going back to MMAP and trying those other sky2  
> options...
> * Retest of no MMAP + Alternative 1 - just to confirm consistency.  
> Worked - no errors. Only version so far that allows the win7 backup to  
> complete.

??? Hmm... Alternative 1 or 2 doesn't even compile into when no MMAP,
so it definitely needs re-retesting ;-)

> * MMAP + NO DMAR + disable_msi=1... also works w/o errors... leaving  
> this one running for a while - also completed a backup successfully.  
> Fastest of the lot... about 3x faster than any other version, working or  
> not.

Very interesting. It would be nice to give it a really long try, and
if still true, try MMAP + NO DMAR only.

>
> I'm leaving this one running for now. Not retesting jumbo for now. Be  
> happy to help dig further.
>
> Tentative recommendations:
>
> 1) The af alternative patch seems rather necessary. First alternative  
> seems to be working, I'd suggest that be submitted and backported to 
> 2.6.32.
> 2) Steven's pskb_may_pull patch also ought to be included and backported.
> 3) Jumbo frame support for yukon2 should probably be disabled until/if  
> fixed.
> 4) When possible I'll test dmar and disable_msi, and no dmar and no  
> disable_msi. When I first hit issues, I was running without DMAR, but  
> also without the above patches. I suppose the non-working permutations  
> need to be either fixed or invalidated (or well documented).
> 5) It would be nice if someone with comparable hardware could reproduce  
> these issues. FWIW, I can only recreate the crash running windows backup  
> to a cifs share. Copying large files doesn't seem to do it.  Could also  
> be some other interaction going on here that perhaps others aren't  
> running - would be happy to compare notes.
>
> Notes:
> This *could* be coincidental, but maybe not...
> With MMAP+NO DMAR + disable_msi there are far fewer ... actually almost  
> no... bind error reports... and no bind format error messages. With  
> NOMMAP and alternative one there are a few more bind error messages and  
> one format error message during the several hours that version was up.  
> All other configurations going back perhaps for two weeks have  
> significantly more bind error reports - and all versions show increasing  
> frequency of bind format errors (IPV6 only) in the roughly 10-15 minutes  
> preceding the lockup/crash/interrupt error messages. There are none  
> immediately preceding any crash, but perhaps there is some correlation  
> between the network errors and bind ipv6 packets.

OK, for now let's make sure this MMAP + NO DMAR + disable_msi is
really really working.

Thanks,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 8, 2010, 4:40 p.m. UTC | #28

On 1/8/2010 2:45 AM, Jarek Poplawski wrote:
> On Thu, Jan 07, 2010 at 06:11:34PM -0500, Michael Breuer wrote:
>    
>> Results:
>> ...
>> Jan  7 15:44:44 mail kernel: Pid: 0, comm: swapper Tainted: G        W
>>      
> BTW, was there any other oops saved before this one?
>
> ...
>    
Nope - just the one.
>> --- adapter dead after this --- rebooted.
>> * no MMAP; alternative 1 patch, mtu=1500; no errors; sustained transfer
>> rates about 25% lower than what I saw with mmap enabled...(before MMAP
>> enabled crashed).
>>      
> ?? Read below...
>
>    
>> * no MMAP mtu=9000; ran ok at low transfer rates - when high rates
>> kicked in, got the sky2 interrupt error&  things went south:
>> Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt
>> status=0x40000008
>> Jan  7 15:09:28 mail kernel: sky2 0000:06:00.0: error interrupt
>> status=0x40000008
>> After this, remote connections broke and I rebooted... decided to rerun
>> w/o MMAP again before going back to MMAP and trying those other sky2
>> options...
>> * Retest of no MMAP + Alternative 1 - just to confirm consistency.
>> Worked - no errors. Only version so far that allows the win7 backup to
>> complete.
>>      
> ??? Hmm... Alternative 1 or 2 doesn't even compile into when no MMAP,
> so it definitely needs re-retesting ;-)
>    
I see your point. I'm pretty sure that run failed miserably. Perhaps 
something else is going on - some sort if intermittent thing that just 
got caught there... have one thought - see below.
>    
>> * MMAP + NO DMAR + disable_msi=1... also works w/o errors... leaving
>> this one running for a while - also completed a backup successfully.
>> Fastest of the lot... about 3x faster than any other version, working or
>> not.
>>      
> Very interesting. It would be nice to give it a really long try, and
> if still true, try MMAP + NO DMAR only.
>    
Still up - no kernel errors reported. There was a large dropped packet 
rate  (RX) which seems to actually correlate with DNS format error 
messages (ipv6 only). I spent some time tracking those down. 
Interestingly, most pointed back to one netblock & one ISP (0qf.ru). I 
blocked that domain and the errors expectedly dropped - as did the RX 
dropped packet rate.  Since booting this version yesterday eth0 shows 
1752944 dropped packets. 1752939 of those happened before I blocked the 
domain about 8 hours ago. I have run load tests since as well.

I think this dns activity is sendmail attempting to validate spam - but 
not 100% sure yet as I can't correlate the .ru domain with anything 
sendmail has reported. I'm running a sniffer now hoping to catch the 
next one. I *think* but can't prove, that these are coming in via 
sendmail - i.e., bad email, not even spam really - just enough to get a 
system configured to do dns lookups as part of spam filtering to connect 
to the server in question. What comes back would seem to be corrupt ipv6 
packets.

This gets us back somewhat to Berck Nash's reported problem. His report 
of sky2 failure was due to external attack. Could this be related? Is it 
possible that absent some set of the patches & config settings in this 
version that ipv6 bind activity involving corrupt (perhaps 
intentionally) packets is breaking something?

Will try rerunning without disable_msi later (after I catch the dns 
thing in the sniffer).
>    
>> I'm leaving this one running for now. Not retesting jumbo for now. Be
>> happy to help dig further.
>>
>> Tentative recommendations:
>>
>> 1) The af alternative patch seems rather necessary. First alternative
>> seems to be working, I'd suggest that be submitted and backported to
>> 2.6.32.
>> 2) Steven's pskb_may_pull patch also ought to be included and backported.
>> 3) Jumbo frame support for yukon2 should probably be disabled until/if
>> fixed.
>> 4) When possible I'll test dmar and disable_msi, and no dmar and no
>> disable_msi. When I first hit issues, I was running without DMAR, but
>> also without the above patches. I suppose the non-working permutations
>> need to be either fixed or invalidated (or well documented).
>> 5) It would be nice if someone with comparable hardware could reproduce
>> these issues. FWIW, I can only recreate the crash running windows backup
>> to a cifs share. Copying large files doesn't seem to do it.  Could also
>> be some other interaction going on here that perhaps others aren't
>> running - would be happy to compare notes.
>>
>> Notes:
>> This *could* be coincidental, but maybe not...
>> With MMAP+NO DMAR + disable_msi there are far fewer ... actually almost
>> no... bind error reports... and no bind format error messages. With
>> NOMMAP and alternative one there are a few more bind error messages and
>> one format error message during the several hours that version was up.
>> All other configurations going back perhaps for two weeks have
>> significantly more bind error reports - and all versions show increasing
>> frequency of bind format errors (IPV6 only) in the roughly 10-15 minutes
>> preceding the lockup/crash/interrupt error messages. There are none
>> immediately preceding any crash, but perhaps there is some correlation
>> between the network errors and bind ipv6 packets.
>>      
> OK, for now let's make sure this MMAP + NO DMAR + disable_msi is
> really really working.
>    
Still running :)
> Thanks,
> Jarek P.
>    

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 8, 2010, 9:29 p.m. UTC | #29

On Fri, Jan 08, 2010 at 11:40:25AM -0500, Michael Breuer wrote:
> On 1/8/2010 2:45 AM, Jarek Poplawski wrote:
> >On Thu, Jan 07, 2010 at 06:11:34PM -0500, Michael Breuer wrote:
> >??? Hmm... Alternative 1 or 2 doesn't even compile into when no MMAP,
> >so it definitely needs re-retesting ;-)
> I see your point. I'm pretty sure that run failed miserably. Perhaps
> something else is going on - some sort if intermittent thing that
> just got caught there... have one thought - see below.

It looks like very timing dependent and without MMAP it can work just
below limit, which could trigger those dmar or msi bugs. Anyway, it
seems to show there is no another serious bug in the MMAP part (i.e.
except the one fixed be alternative 1 or 2 patches).

> >>* MMAP + NO DMAR + disable_msi=1... also works w/o errors... leaving
> >>this one running for a while - also completed a backup successfully.
> >>Fastest of the lot... about 3x faster than any other version, working or
> >>not.
> >Very interesting. It would be nice to give it a really long try, and
> >if still true, try MMAP + NO DMAR only.
> Still up - no kernel errors reported. There was a large dropped
> packet rate  (RX) which seems to actually correlate with DNS format
> error messages (ipv6 only). I spent some time tracking those down.
> Interestingly, most pointed back to one netblock & one ISP (0qf.ru).
> I blocked that domain and the errors expectedly dropped - as did the
> RX dropped packet rate.  Since booting this version yesterday eth0
> shows 1752944 dropped packets. 1752939 of those happened before I
> blocked the domain about 8 hours ago. I have run load tests since as
> well.
> 
> I think this dns activity is sendmail attempting to validate spam -
> but not 100% sure yet as I can't correlate the .ru domain with
> anything sendmail has reported. I'm running a sniffer now hoping to
> catch the next one. I *think* but can't prove, that these are coming
> in via sendmail - i.e., bad email, not even spam really - just
> enough to get a system configured to do dns lookups as part of spam
> filtering to connect to the server in question. What comes back
> would seem to be corrupt ipv6 packets.
> 
> This gets us back somewhat to Berck Nash's reported problem. His
> report of sky2 failure was due to external attack. Could this be
> related? Is it possible that absent some set of the patches & config
> settings in this version that ipv6 bind activity involving corrupt
> (perhaps intentionally) packets is breaking something?

Berck Nash reported oopses during sky2 TX timeout recovery, which are
generally hardware/driver problems, and shouldn't be triggered by ip
level bugs, so it should be queried as a separate bug report.

> 
> Will try rerunning without disable_msi later (after I catch the dns
> thing in the sniffer).
> >>I'm leaving this one running for now. Not retesting jumbo for now. Be
> >>happy to help dig further.
> >>
> >>Tentative recommendations:
> >>
> >>1) The af alternative patch seems rather necessary. First alternative
> >>seems to be working, I'd suggest that be submitted and backported to
> >>2.6.32.

BTW, don't hurry with that yet, but in the next test, please try
alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).

> >>2) Steven's pskb_may_pull patch also ought to be included and backported.

This patch is very helpful for debugging, but I doubt appropriate
for the mainline if it isn't triggered any more after 1) fix. But,
please keep it yourself for some time in all these tests (and of
course Berck Nash's patch).

> >OK, for now let's make sure this MMAP + NO DMAR + disable_msi is
> >really really working.
> Still running :)

Very nice :)
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 8, 2010, 9:48 p.m. UTC | #30

On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
> On Fri, Jan 08, 2010 at 11:40:25AM -0500, Michael Breuer wrote:
>    
>> On 1/8/2010 2:45 AM, Jarek Poplawski wrote:
>>      
>> ...
> Berck Nash reported oopses during sky2 TX timeout recovery, which are
> generally hardware/driver problems, and shouldn't be triggered by ip
> level bugs, so it should be queried as a separate bug report.
>
>    
My thought was that his crash was secondary to the netdev watchdog & 
subsequent reset that I saw.
>> Will try rerunning without disable_msi later (after I catch the dns
>> thing in the sniffer).
>>      
>>>> I'm leaving this one running for now. Not retesting jumbo for now. Be
>>>> happy to help dig further.
>>>>
>>>> Tentative recommendations:
>>>>
>>>> 1) The af alternative patch seems rather necessary. First alternative
>>>> seems to be working, I'd suggest that be submitted and backported to
>>>> 2.6.32.
>>>>          
> BTW, don't hurry with that yet, but in the next test, please try
> alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).
>
>    
Will do - still up from yesterday... no more dropped packets... none of 
the dns errors either. To be expected I suppose as long as I'm trying to 
sniff it. Assuming no immediate erorrs with alt2, no DMAR + disable_msi 
I'll report back after it's been up for a while.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 8, 2010, 10:02 p.m. UTC | #31

On Fri, Jan 08, 2010 at 04:48:11PM -0500, Michael Breuer wrote:
> On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
> >On Fri, Jan 08, 2010 at 11:40:25AM -0500, Michael Breuer wrote:
> >>On 1/8/2010 2:45 AM, Jarek Poplawski wrote:
> >>...
> >Berck Nash reported oopses during sky2 TX timeout recovery, which are
> >generally hardware/driver problems, and shouldn't be triggered by ip
> >level bugs, so it should be queried as a separate bug report.
> >
> My thought was that his crash was secondary to the netdev watchdog &
> subsequent reset that I saw.

Yes, the netdev watchdog which triggers on TX timeout.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 9, 2010, 4:45 a.m. UTC | #32

On 1/8/2010 4:48 PM, Michael Breuer wrote:
> On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
> ...
>> BTW, don't hurry with that yet, but in the next test, please try
>> alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).
>>
> Will do - still up from yesterday... no more dropped packets... none 
> of the dns errors either. To be expected I suppose as long as I'm 
> trying to sniff it. Assuming no immediate erorrs with alt2, no DMAR + 
> disable_msi I'll report back after it's been up for a while.
> -- 
Ok - ran alt1 and alt2, both with MMAP, no DMAR and disable_msi. Both 
seem to behave similarly. No logged errors; large numbers of dropped RX 
packets. One odd thing: when driving every sort of traffic through, I 
was able to hose the client adapter (win7) repeatedly by runnnig the 
win7 backup and connecting Windows Media Player to a Mediatomb stream 
while also running a remote X11 session. Looks like the SSDP traffic 
that occurs at the same time as the SMB traffic and X11 traffic takes 
out the adapter on Win7 - Nforce.
Reran with msi enabled, MMAP and no DMAR. Also no errors; much faster, 
and the Win7 side survives the same conditions that don't work when msi 
is disabled. Doesn't make sense to me, but it is what it is.

Going to leave this up for a while and see if things remain functional.
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 9, 2010, 5:44 a.m. UTC | #33

On 1/8/2010 11:45 PM, Michael Breuer wrote:
> On 1/8/2010 4:48 PM, Michael Breuer wrote:
>> On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
>> ...
>>> BTW, don't hurry with that yet, but in the next test, please try
>>> alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).
>>>
>> Will do - still up from yesterday... no more dropped packets... none 
>> of the dns errors either. To be expected I suppose as long as I'm 
>> trying to sniff it. Assuming no immediate erorrs with alt2, no DMAR + 
>> disable_msi I'll report back after it's been up for a while.
>> -- 
> Ok - ran alt1 and alt2, both with MMAP, no DMAR and disable_msi. Both 
> seem to behave similarly. No logged errors; large numbers of dropped 
> RX packets. One odd thing: when driving every sort of traffic through, 
> I was able to hose the client adapter (win7) repeatedly by runnnig the 
> win7 backup and connecting Windows Media Player to a Mediatomb stream 
> while also running a remote X11 session. Looks like the SSDP traffic 
> that occurs at the same time as the SMB traffic and X11 traffic takes 
> out the adapter on Win7 - Nforce.
> Reran with msi enabled, MMAP and no DMAR. Also no errors; much faster, 
> and the Win7 side survives the same conditions that don't work when 
> msi is disabled. Doesn't make sense to me, but it is what it is.
>
Tracked it down - this appears to be result of Win7 enabling 
"EnableDeadGWDetect" by default. Long story short, if this is set and a 
packet sent to the gw is retransmitted more than, 1/2 the value of 
maxdataretransmissions (default of 5), then the gateway is declared dead 
and the next one selected. If there is only one gateway, then the 
default gateway is effectively removed. So under load where the server 
is the default gateway, this is not a surprising result. With msi 
enabled, the system responds better, fewer retransmissions, no dropping 
the link.
> Going to leave this up for a while and see if things remain functional.
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 9, 2010, 12:28 p.m. UTC | #34

On Fri, Jan 08, 2010 at 11:45:35PM -0500, Michael Breuer wrote:
> On 1/8/2010 4:48 PM, Michael Breuer wrote:
> >On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
> >...
> >>BTW, don't hurry with that yet, but in the next test, please try
> >>alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).
> >>
> >Will do - still up from yesterday... no more dropped packets...
> >none of the dns errors either. To be expected I suppose as long as
> >I'm trying to sniff it. Assuming no immediate erorrs with alt2, no
> >DMAR + disable_msi I'll report back after it's been up for a
> >while.
> >-- 
> Ok - ran alt1 and alt2, both with MMAP, no DMAR and disable_msi.
> Both seem to behave similarly. No logged errors;

Ok - since alt2 introduces less changes and is acked by Stephen
already, I'll resend it with your "Tested-by" in a new thread to clear
things a bit.

> large numbers of dropped RX packets.

You might try some tweaking with another sky2 parameter "copybreak"
or even editing its "NAPI_WEIGHT" variable.

> One odd thing: when driving every sort of
> traffic through, I was able to hose the client adapter (win7)
> repeatedly by runnnig the win7 backup and connecting Windows Media
> Player to a Mediatomb stream while also running a remote X11
> session. Looks like the SSDP traffic that occurs at the same time as
> the SMB traffic and X11 traffic takes out the adapter on Win7 -
> Nforce.
> Reran with msi enabled, MMAP and no DMAR. Also no errors; much
> faster, and the Win7 side survives the same conditions that don't
> work when msi is disabled. Doesn't make sense to me, but it is what
> it is.
> 
> Going to leave this up for a while and see if things remain functional.

It looks like DMAR is the main candidate for a new linux-kernel@ or
bugzilla bug report. You should also consider reporting these ipv6
problems, especially if you think they can trigger TX timeouts. (In
both cases it would be good to try current mainline first, when you
get it workable.)

Thanks,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 9, 2010, 6:34 p.m. UTC | #35

On 1/9/2010 7:28 AM, Jarek Poplawski wrote:
> On Fri, Jan 08, 2010 at 11:45:35PM -0500, Michael Breuer wrote:
>    
>> On 1/8/2010 4:48 PM, Michael Breuer wrote:
>>      
>>> On 1/8/2010 4:29 PM, Jarek Poplawski wrote:
>>> ...
>>>        
>>>> BTW, don't hurry with that yet, but in the next test, please try
>>>> alternative 2 again (i.e. with MMAP + no DMAR + disable_msi).
>>>>
>>>>          
>>> Will do - still up from yesterday... no more dropped packets...
>>> none of the dns errors either. To be expected I suppose as long as
>>> I'm trying to sniff it. Assuming no immediate erorrs with alt2, no
>>> DMAR + disable_msi I'll report back after it's been up for a
>>> while.
>>> -- 
>>>        
>> Ok - ran alt1 and alt2, both with MMAP, no DMAR and disable_msi.
>> Both seem to behave similarly. No logged errors;
>>      
> Ok - since alt2 introduces less changes and is acked by Stephen
> already, I'll resend it with your "Tested-by" in a new thread to clear
> things a bit.
>
>    
Ok - good, and thank you for all your help.
>> large numbers of dropped RX packets.
>>      
> You might try some tweaking with another sky2 parameter "copybreak"
> or even editing its "NAPI_WEIGHT" variable.
>
>    
I'll play around with this after I figure out what's actually being 
dropped and why. Looking at ethtool, over 99% of RX packets are in the 
 >=1024 bucket. I'll play with weight and perhaps rmem as time permits.
>> One odd thing: when driving every sort of
>> traffic through, I was able to hose the client adapter (win7)
>> repeatedly by runnnig the win7 backup and connecting Windows Media
>> Player to a Mediatomb stream while also running a remote X11
>> session. Looks like the SSDP traffic that occurs at the same time as
>> the SMB traffic and X11 traffic takes out the adapter on Win7 -
>> Nforce.
>> Reran with msi enabled, MMAP and no DMAR. Also no errors; much
>> faster, and the Win7 side survives the same conditions that don't
>> work when msi is disabled. Doesn't make sense to me, but it is what
>> it is.
>>
>> Going to leave this up for a while and see if things remain functional.
>>      
> It looks like DMAR is the main candidate for a new linux-kernel@ or
> bugzilla bug report. You should also consider reporting these ipv6
> problems, especially if you think they can trigger TX timeouts. (In
> both cases it would be good to try current mainline first, when you
> get it workable.)
>    
Ok - I'll get onto mainline and then deal with DMAR.
> Thanks,
> Jarek P.
>    

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 13, 2010, 8:39 p.m. UTC | #36

On 1/9/2010 1:34 PM, Michael Breuer wrote:
> On 1/9/2010 7:28 AM, Jarek Poplawski wrote:   ...
>> It looks like DMAR is the main candidate for a new linux-kernel@ or
>> bugzilla bug report. You should also consider reporting these ipv6
>> problems, especially if you think they can trigger TX timeouts. (In
>> both cases it would be good to try current mainline first, when you
>> get it workable.)
> Ok - I'll get onto mainline and then deal with DMAR.
Just an FYI - 2.6.32.3 with alt 3 af_packet patch & sky2 pskb_may_pull 
runs OK with DMAR (re)enabled and msi enabled.
>> Thanks,
>> Jarek P.
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 13, 2010, 9:09 p.m. UTC | #37

On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
> Just an FYI - 2.6.32.3 with alt 3 af_packet patch & sky2
> pskb_may_pull runs OK with DMAR (re)enabled and msi enabled.

Hmm... What a pity! It was such a useful debugging tool for
networking ;-) BTW, I'm not sure if "runs OK" means with or without
those DHCP drops & large packets you described.

Thanks,
Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 13, 2010, 9:16 p.m. UTC | #38

On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>    
>> Just an FYI - 2.6.32.3 with alt 3 af_packet patch&  sky2
>> pskb_may_pull runs OK with DMAR (re)enabled and msi enabled.
>>      
> Hmm... What a pity! It was such a useful debugging tool for
> networking ;-) BTW, I'm not sure if "runs OK" means with or without
> those DHCP drops&  large packets you described.
>
> Thanks,
> Jarek P.
>    
As of now, no errors even when blasting traffic & forcing dhcp packets 
as before. I haven't tried putting mtu back to 9k yet. OK means that 
there are no obvious differences in behavior with or without DMAR all 
else being equal.

There were some updates made to stable that could have fixed this - I'd 
guess intel_iommu fixes helped.

If it helps, I'm still getting one error without DMAR enabled - at 
startup, there's a DMA sync oops - mismatch of 72 bytes coming from 
sky2. That oops was posted previously - with DMAR (re) enabled, there's 
no related oops.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 13, 2010, 9:34 p.m. UTC | #39

On Wed, Jan 13, 2010 at 04:16:36PM -0500, Michael Breuer wrote:
> If it helps, I'm still getting one error without DMAR enabled - at
> startup, there's a DMA sync oops - mismatch of 72 bytes coming from
> sky2. That oops was posted previously - with DMAR (re) enabled,
> there's no related oops.

I hope re-posting this oops with this information should be helpful.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 17, 2010, 4:26 p.m. UTC | #40

On 01/13/2010 04:16 PM, Michael Breuer wrote:
> On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
>> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>>> Just an FYI - 2.6.32.3 with alt 3 af_packet patch&  sky2
>>> pskb_may_pull runs OK with DMAR (re)enabled and msi enabled.
>> Hmm... What a pity! It was such a useful debugging tool for
>> networking ;-) BTW, I'm not sure if "runs OK" means with or without
>> those DHCP drops&  large packets you described.
>>
>> Thanks,
>> Jarek P.
> As of now, no errors even when blasting traffic & forcing dhcp packets 
> as before. I haven't tried putting mtu back to 9k yet. OK means that 
> there are no obvious differences in behavior with or without DMAR all 
> else being equal.
>
> There were some updates made to stable that could have fixed this - 
> I'd guess intel_iommu fixes helped.
>
> If it helps, I'm still getting one error without DMAR enabled - at 
> startup, there's a DMA sync oops - mismatch of 72 bytes coming from 
> sky2. That oops was posted previously - with DMAR (re) enabled, 
> there's no related oops.
Update: after leaving the system up for a few days, I hit the DMAR error 
again. This happened during a scheduled backup from my win7 box. A 
reboot was required to re-enable eth0. After the error, eth0 was 
receiving, but was unable to transmit. For example, the log reported arp 
bogons; DHCPINFORM/ACK sequences (where the ACK that was logged was not 
transmitted), etc. The log was filled with sky2 eth0: tx timeout 
messages; as well as disable/enable of eth0.

I attempted to get things up again without a reboot, but failed. Even 
rmmod & insmod did not fix whatever was broken on the TX side.

Note that this is similar to the earlier sky2 errors I had under load 
with the variety of patches, and with or without DMAR enabled. Just took 
way longer this time. Note that eth1 remained functional.

Unfortunately, with the latest set of patches installed, this is no 
longer reproducible at will. I'd guess therefore that the patches 
narrowed some hole, but didn't close it.

Relevant log portions:

Jan 17 05:29:49 mail dhcpd: DHCPREQUEST for 10.0.0.32 from 
00:26:bb:aa:15:10 (mbitouch) via eth0
Jan 17 05:29:49 mail dhcpd: DHCPACK on 10.0.0.32 to 00:26:bb:aa:15:10 
(mbitouch) via eth0
Jan 17 05:36:49 mail kernel: DRHD: handling fault status reg 2
Jan 17 05:36:49 mail kernel: DMAR:[DMA Read] Request device [06:00.0] 
fault addr ffe7957fe000
Jan 17 05:36:49 mail kernel: DMAR:[fault reason 06] PTE Read access is 
not set
Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: error interrupt 
status=0xc0000000
Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
Jan 17 05:36:49 mail smbd[14840]: [2010/01/17 05:36:49,  0] 
lib/util_sock.c:539(read_fd_with_timeout)
Jan 17 05:36:49 mail smbd[14840]: [2010/01/17 05:36:49,  0] 
lib/util_sock.c:1491(get_peer_addr_internal)
Jan 17 05:36:49 mail smbd[14840]:   getpeername failed. Error was 
Transport endpoint is not connected
Jan 17 05:36:49 mail smbd[14840]:   read_fd_with_timeout: client 0.0.0.0 
read error = Connection timed out.
Jan 17 05:37:51 mail kernel: ------------[ cut here ]------------
Jan 17 05:37:51 mail kernel: WARNING: at net/sched/sch_generic.c:261 
dev_watchdog+0xf3/0x164()
Jan 17 05:37:51 mail kernel: Hardware name: System Product Name
Jan 17 05:37:51 mail kernel: NETDEV WATCHDOG: eth0 (sky2): transmit 
queue 0 timed out
Jan 17 05:37:51 mail kernel: Modules linked in: nls_utf8 cifs 
cpufreq_stats ip6table_mangle ip6table_filter ip6_tables iptable_raw 
iptable_mangle ipt_MASQUERADE iptable_nat nf_nat appletalk psnap llc 
nfsd lockd nfs_acl auth_rpcgss exportfs hwmon_vid coretemp sunrpc 
acpi_cpufreq sit tunnel4 ipt_LOG nf_conntrack_netbios_ns 
nf_conntrack_ftp nf_conntrack_ipv6 xt_multiport xt_DSCP xt_dscp xt_MARK 
ipv6 dm_multipath kvm_intel kvm snd_hda_codec_analog snd_hda_intel 
snd_hda_codec snd_ens1371 gameport snd_rawmidi snd_ac97_codec snd_hwdep 
ac97_bus firewire_ohci snd_seq firewire_core snd_seq_device 
gspca_spca505 gspca_main videodev i2c_i801 snd_pcm crc_itu_t v4l1_compat 
pcspkr v4l2_compat_ioctl32 asus_atk0110 hwmon iTCO_wdt 
iTCO_vendor_support snd_timer snd soundcore sky2 snd_page_alloc wmi 
fbcon tileblit font bitblit softcursor raid456 async_raid6_recov 
async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 ata_generic 
pata_acpi pata_marvell nouveau ttm drm_kms_helper drm agpgart fb 
i2c_algo_bit cfbcopyarea i2c_core
Jan 17 05:37:51 mail kernel: cfbimgblt cfbfillrect [last unloaded: 
microcode]
Jan 17 05:37:51 mail kernel: Pid: 0, comm: swapper Tainted: G        W  
2.6.32WITHMMAPNODMARAF3SKY2TXRGCLNV4TX-00893-gb5d5baa-dirty #2
Jan 17 05:37:51 mail kernel: Call Trace:
Jan 17 05:37:51 mail kernel: <IRQ>  [<ffffffff8105365a>] 
warn_slowpath_common+0x7c/0x94
Jan 17 05:37:51 mail kernel: [<ffffffff810536c9>] 
warn_slowpath_fmt+0x41/0x43
Jan 17 05:37:51 mail kernel: [<ffffffff813e2e57>] ? netif_tx_lock+0x44/0x6c
Jan 17 05:37:51 mail kernel: [<ffffffff813e2fbf>] dev_watchdog+0xf3/0x164
Jan 17 05:37:51 mail kernel: [<ffffffff8106e8a4>] ? __queue_work+0x3a/0x42
Jan 17 05:37:51 mail kernel: [<ffffffff8106316b>] 
run_timer_softirq+0x1c8/0x270
Jan 17 05:37:51 mail kernel: [<ffffffff8105ae3b>] __do_softirq+0xf8/0x1cd
Jan 17 05:37:51 mail kernel: [<ffffffff8107ef33>] ? 
tick_program_event+0x2a/0x2c
Jan 17 05:37:51 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan 17 05:37:51 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan 17 05:37:51 mail kernel: [<ffffffff8105aa1b>] irq_exit+0x4a/0x8c
Jan 17 05:37:51 mail kernel: [<ffffffff8146f8f2>] 
smp_apic_timer_interrupt+0x86/0x94
Jan 17 05:37:51 mail kernel: [<ffffffff810127e3>] 
apic_timer_interrupt+0x13/0x20
Jan 17 05:37:51 mail kernel: <EOI>  [<ffffffff812c678a>] ? 
acpi_idle_enter_bm+0x256/0x28a
Jan 17 05:37:51 mail kernel: [<ffffffff812c6783>] ? 
acpi_idle_enter_bm+0x24f/0x28a
Jan 17 05:37:51 mail kernel: [<ffffffff813a5f50>] ? 
cpuidle_idle_call+0x9e/0xfa
Jan 17 05:37:51 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan 17 05:37:51 mail kernel: [<ffffffff81464ed2>] ? 
start_secondary+0x201/0x242
Jan 17 05:37:51 mail kernel: ---[ end trace 57f7151f6a5def07 ]---
Jan 17 05:37:51 mail kernel: sky2 eth0: tx timeout
Jan 17 05:37:51 mail kernel: sky2 eth0: transmit ring 85 .. 45 report=85 
done=85
Jan 17 05:37:51 mail kernel: sky2 eth0: disabling interface
Jan 17 05:37:51 mail kernel: sky2 eth0: enabling interface
<unrelated stuff>
Jan 17 05:39:14 mail kernel: sky2 eth0: tx timeout
Jan 17 05:39:14 mail kernel: sky2 eth0: transmit ring 2 .. 89 report=2 
done=2
Jan 17 05:39:14 mail kernel: sky2 eth0: disabling interface
Jan 17 05:39:14 mail kernel: sky2 eth0: enabling interface
<time passes>
Jan 17 05:40:22 mail kernel: sky2 eth0: tx timeout
Jan 17 05:40:22 mail kernel: sky2 eth0: transmit ring 2 .. 89 report=2 
done=2
Jan 17 05:40:22 mail kernel: sky2 eth0: disabling interface
Jan 17 05:40:22 mail kernel: sky2 eth0: enabling interface
Jan 17 05:40:22 mail NetworkManager: <info>  (eth0): carrier now OFF 
(device state 1)
Jan 17 05:40:25 mail kernel: sky2 eth0: Link is up at 1000 Mbps, full 
duplex, flow control both
<time passes>
Jan 17 05:42:05 mail kernel: sky2 eth0: tx timeout
Jan 17 05:42:05 mail kernel: sky2 eth0: transmit ring 2 .. 89 report=2 
done=2
Jan 17 05:42:05 mail kernel: sky2 eth0: disabling interface
Jan 17 05:42:05 mail kernel: sky2 eth0: enabling interface
Jan 17 05:42:08 mail kernel: sky2 eth0: Link is up at 1000 Mbps, full 
duplex, flow control both
<time passes>
Jan 17 05:44:13 mail kernel: sky2 eth0: tx timeout
Jan 17 05:44:13 mail kernel: sky2 eth0: transmit ring 3 .. 90 report=3 
done=3
Jan 17 05:44:13 mail kernel: sky2 eth0: disabling interface
Jan 17 05:44:13 mail kernel: sky2 eth0: enabling interface
Jan 17 05:44:16 mail kernel: sky2 eth0: Link is up at 1000 Mbps, full 
duplex, flow control both
<much of the same until I rebooted>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 17, 2010, 10:17 p.m. UTC | #41

On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
> On 01/13/2010 04:16 PM, Michael Breuer wrote:
> >On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
> >>On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
> >>>Just an FYI - 2.6.32.3 with alt 3 af_packet patch&  sky2
> >>>pskb_may_pull runs OK with DMAR (re)enabled and msi enabled.
> >>Hmm... What a pity! It was such a useful debugging tool for
> >>networking ;-) BTW, I'm not sure if "runs OK" means with or without
> >>those DHCP drops&  large packets you described.
> >>
> >>Thanks,
> >>Jarek P.
> >As of now, no errors even when blasting traffic & forcing dhcp
> >packets as before. I haven't tried putting mtu back to 9k yet. OK
> >means that there are no obvious differences in behavior with or
> >without DMAR all else being equal.
> >
> >There were some updates made to stable that could have fixed this
> >- I'd guess intel_iommu fixes helped.
> >
> >If it helps, I'm still getting one error without DMAR enabled - at
> >startup, there's a DMA sync oops - mismatch of 72 bytes coming
> >from sky2. That oops was posted previously - with DMAR (re)
> >enabled, there's no related oops.
> Update: after leaving the system up for a few days, I hit the DMAR
> error again.

My proposal is to send some summary as a new thread, with dmar in the
subject, and cc-ed dmar maintainers.

> This happened during a scheduled backup from my win7
> box. A reboot was required to re-enable eth0. After the error, eth0
> was receiving, but was unable to transmit. For example, the log
> reported arp bogons; DHCPINFORM/ACK sequences (where the ACK that
> was logged was not transmitted), etc. The log was filled with sky2
> eth0: tx timeout messages; as well as disable/enable of eth0.
> 
> I attempted to get things up again without a reboot, but failed.
> Even rmmod & insmod did not fix whatever was broken on the TX side.
> 
> Note that this is similar to the earlier sky2 errors I had under
> load with the variety of patches, and with or without DMAR enabled.
> Just took way longer this time. Note that eth1 remained functional.
> 
> Unfortunately, with the latest set of patches installed, this is no
> longer reproducible at will. I'd guess therefore that the patches
> narrowed some hole, but didn't close it.

It would be nice to name those patches each time. Anyway, try this
again without DMAR.

Thanks,
Jarek P.

> 
> Relevant log portions:
> 
> Jan 17 05:29:49 mail dhcpd: DHCPREQUEST for 10.0.0.32 from
> 00:26:bb:aa:15:10 (mbitouch) via eth0
> Jan 17 05:29:49 mail dhcpd: DHCPACK on 10.0.0.32 to
> 00:26:bb:aa:15:10 (mbitouch) via eth0
> Jan 17 05:36:49 mail kernel: DRHD: handling fault status reg 2
> Jan 17 05:36:49 mail kernel: DMAR:[DMA Read] Request device
> [06:00.0] fault addr ffe7957fe000
> Jan 17 05:36:49 mail kernel: DMAR:[fault reason 06] PTE Read access
> is not set
> Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: error interrupt
> status=0xc0000000
> Jan 17 05:36:49 mail kernel: sky2 0000:06:00.0: PCI hardware error (0x2010)
...
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 17, 2010, 10:34 p.m. UTC | #42

On 1/17/2010 5:17 PM, Jarek Poplawski wrote:
> On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
>    
>> On 01/13/2010 04:16 PM, Michael Breuer wrote:
>>      
>>> On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
>>>        
>>>> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>>>>          
>>>>
>> Update: after leaving the system up for a few days, I hit the DMAR
>> error again.
>>      
> My proposal is to send some summary as a new thread, with dmar in the
> subject, and cc-ed dmar maintainers.
>
>    
Not sure I agree. The symptoms are identical to those I hit without DMAR 
earlier on. Also, as this issue only happens when there is high receive 
load, I'm thinking there's some sort of race between TX and RX within 
the sky2 driver, or hardware. I think that DMAR is correctly catching 
the error.
>> This happened during a scheduled backup from my win7
>> box. A reboot was required to re-enable eth0. After the error, eth0
>> was receiving, but was unable to transmit. For example, the log
>> reported arp bogons; DHCPINFORM/ACK sequences (where the ACK that
>> was logged was not transmitted), etc. The log was filled with sky2
>> eth0: tx timeout messages; as well as disable/enable of eth0.
>>
>> I attempted to get things up again without a reboot, but failed.
>> Even rmmod&  insmod did not fix whatever was broken on the TX side.
>>
>> Note that this is similar to the earlier sky2 errors I had under
>> load with the variety of patches, and with or without DMAR enabled.
>> Just took way longer this time. Note that eth1 remained functional.
>>
>> Unfortunately, with the latest set of patches installed, this is no
>> longer reproducible at will. I'd guess therefore that the patches
>> narrowed some hole, but didn't close it.
>>      
> It would be nice to name those patches each time. Anyway, try this
> again without DMAR.
>
> Thanks,
> Jarek P.
>    
>
My bad: was running with the af_packet.c version 3 patch; and Stephen's 
v4 patch from last week. Both on 2.6.32 from git (so 2.6.32.4). Can't 
move back to head as I've hit two unrelated issues.

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 17, 2010, 11:05 p.m. UTC | #43

On Sun, Jan 17, 2010 at 05:34:19PM -0500, Michael Breuer wrote:
> On 1/17/2010 5:17 PM, Jarek Poplawski wrote:
> >On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
> >>On 01/13/2010 04:16 PM, Michael Breuer wrote:
> >>>On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
> >>>>On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
> >>>>
> >>Update: after leaving the system up for a few days, I hit the DMAR
> >>error again.
> >My proposal is to send some summary as a new thread, with dmar in the
> >subject, and cc-ed dmar maintainers.
> >
> Not sure I agree. The symptoms are identical to those I hit without
> DMAR earlier on. Also, as this issue only happens when there is high
> receive load, I'm thinking there's some sort of race between TX and
> RX within the sky2 driver, or hardware. I think that DMAR is
> correctly catching the error.

Hmm... OK, then let's wait with this report and go back to testing
it "really really long" ;-) without DMAR, and maybe without the
last Stephen's patch either? (So only the two things in the current
linux-2.6.)

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 17, 2010, 11:15 p.m. UTC | #44

On 1/17/2010 6:05 PM, Jarek Poplawski wrote:
> On Sun, Jan 17, 2010 at 05:34:19PM -0500, Michael Breuer wrote:
>    
>> On 1/17/2010 5:17 PM, Jarek Poplawski wrote:
>>      
>>> On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
>>>        
>>>> On 01/13/2010 04:16 PM, Michael Breuer wrote:
>>>>          
>>>>> On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
>>>>>            
>>>>>> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>>>>>>
>>>>>>              
>>>> Update: after leaving the system up for a few days, I hit the DMAR
>>>> error again.
>>>>          
>>> My proposal is to send some summary as a new thread, with dmar in the
>>> subject, and cc-ed dmar maintainers.
>>>
>>>        
>> Not sure I agree. The symptoms are identical to those I hit without
>> DMAR earlier on. Also, as this issue only happens when there is high
>> receive load, I'm thinking there's some sort of race between TX and
>> RX within the sky2 driver, or hardware. I think that DMAR is
>> correctly catching the error.
>>      
> Hmm... OK, then let's wait with this report and go back to testing
> it "really really long" ;-) without DMAR, and maybe without the
> last Stephen's patch either? (So only the two things in the current
> linux-2.6.)
>
> Jarek P.
>    
Ok - but absent the last patch, I think I still need the pskb_may_pull 
patch... so it'd be pskb_may_pull and afpacket v3 and no DMAR.

Also - not sure if related, but there's still the odd tx side behavior 
when RX is under load. That I CAN reproduce at will (yesterday's report 
- no crash, but I confirmed that DHCPOFFER packets are being dropped 
somewhere after wireshark sees them and before hitting the wire.

I am also wondering whether or not that testing I did yesterday set up 
today's hang - perhaps those lost TX packets are corrupting something 
that manifests worse later.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 7:30 a.m. UTC | #45

On Sun, Jan 17, 2010 at 06:15:22PM -0500, Michael Breuer wrote:
> On 1/17/2010 6:05 PM, Jarek Poplawski wrote:
>> On Sun, Jan 17, 2010 at 05:34:19PM -0500, Michael Breuer wrote:
>>    
>>> On 1/17/2010 5:17 PM, Jarek Poplawski wrote:
>>>      
>>>> On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
>>>>        
>>>>> On 01/13/2010 04:16 PM, Michael Breuer wrote:
>>>>>          
>>>>>> On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
>>>>>>            
>>>>>>> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>>>>>>>
>>>>>>>              
>>>>> Update: after leaving the system up for a few days, I hit the DMAR
>>>>> error again.
>>>>>          
>>>> My proposal is to send some summary as a new thread, with dmar in the
>>>> subject, and cc-ed dmar maintainers.
>>>>
>>>>        
>>> Not sure I agree. The symptoms are identical to those I hit without
>>> DMAR earlier on. Also, as this issue only happens when there is high
>>> receive load, I'm thinking there's some sort of race between TX and
>>> RX within the sky2 driver, or hardware. I think that DMAR is
>>> correctly catching the error.
>>>      
>> Hmm... OK, then let's wait with this report and go back to testing
>> it "really really long" ;-) without DMAR, and maybe without the
>> last Stephen's patch either? (So only the two things in the current
>> linux-2.6.)
>>
>> Jarek P.
>>    
> Ok - but absent the last patch, I think I still need the pskb_may_pull  
> patch... so it'd be pskb_may_pull and afpacket v3 and no DMAR.

Exactly. Or if it's working for you already, the mainline (2.6.33-rc4)
with the pskb_may_pull patch. And check for warnings from the latter.

>
> Also - not sure if related, but there's still the odd tx side behavior  
> when RX is under load. That I CAN reproduce at will (yesterday's report  
> - no crash, but I confirmed that DHCPOFFER packets are being dropped  
> somewhere after wireshark sees them and before hitting the wire.

I'm not sure either, but until there is no crash it might be some
minor bug or/and missing stat. Btw, you could probably try alternative
test with ping from this overloaded box to the router and win7.

>
> I am also wondering whether or not that testing I did yesterday set up  
> today's hang - perhaps those lost TX packets are corrupting something  
> that manifests worse later.

Maybe, but you wrote earlier they had to fix something around this
DMAR in the meantime, because it triggered much faster during your
previous tests. So, I don't know why you assume this DMAR has to be
correct this time.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 4:29 p.m. UTC | #46

On 01/18/2010 02:30 AM, Jarek Poplawski wrote:
> On Sun, Jan 17, 2010 at 06:15:22PM -0500, Michael Breuer wrote:
>    
>> On 1/17/2010 6:05 PM, Jarek Poplawski wrote:
>>      
>>> On Sun, Jan 17, 2010 at 05:34:19PM -0500, Michael Breuer wrote:
>>>
>>>        
>>>> On 1/17/2010 5:17 PM, Jarek Poplawski wrote:
>>>>
>>>>          
>>>>> On Sun, Jan 17, 2010 at 11:26:46AM -0500, Michael Breuer wrote:
>>>>>
>>>>>            
>>>>>> On 01/13/2010 04:16 PM, Michael Breuer wrote:
>>>>>>
>>>>>>              
>>>>>>> On 1/13/2010 4:09 PM, Jarek Poplawski wrote:
>>>>>>>
>>>>>>>                
>>>>>>>> On Wed, Jan 13, 2010 at 03:39:37PM -0500, Michael Breuer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>> Update: after leaving the system up for a few days, I hit the DMAR
>>>>>> error again.
>>>>>>
>>>>>>              
>>>>> My proposal is to send some summary as a new thread, with dmar in the
>>>>> subject, and cc-ed dmar maintainers.
>>>>>
>>>>>
>>>>>            
>>>> Not sure I agree. The symptoms are identical to those I hit without
>>>> DMAR earlier on. Also, as this issue only happens when there is high
>>>> receive load, I'm thinking there's some sort of race between TX and
>>>> RX within the sky2 driver, or hardware. I think that DMAR is
>>>> correctly catching the error.
>>>>
>>>>          
>>> Hmm... OK, then let's wait with this report and go back to testing
>>> it "really really long" ;-) without DMAR, and maybe without the
>>> last Stephen's patch either? (So only the two things in the current
>>> linux-2.6.)
>>>
>>> Jarek P.
>>>
>>>        
>> Ok - but absent the last patch, I think I still need the pskb_may_pull
>> patch... so it'd be pskb_may_pull and afpacket v3 and no DMAR.
>>      
> Exactly. Or if it's working for you already, the mainline (2.6.33-rc4)
> with the pskb_may_pull patch. And check for warnings from the latter.
>
>    
>> Also - not sure if related, but there's still the odd tx side behavior
>> when RX is under load. That I CAN reproduce at will (yesterday's report
>> - no crash, but I confirmed that DHCPOFFER packets are being dropped
>> somewhere after wireshark sees them and before hitting the wire.
>>      
> I'm not sure either, but until there is no crash it might be some
> minor bug or/and missing stat. Btw, you could probably try alternative
> test with ping from this overloaded box to the router and win7.
>
>    
>> I am also wondering whether or not that testing I did yesterday set up
>> today's hang - perhaps those lost TX packets are corrupting something
>> that manifests worse later.
>>      
> Maybe, but you wrote earlier they had to fix something around this
> DMAR in the meantime, because it triggered much faster during your
> previous tests. So, I don't know why you assume this DMAR has to be
> correct this time.
>
> Jarek P.
>    
Ok - up on the two patches, no DMAR. Some early observations:

1. There's an early on MMAP oops (see below). This happens once, at the 
completion of the transition to runlevel 5 (I've seen it entering 
runlevel 3 as well). This does not recur when runlevels are subsequently 
changed. I do not see this when running with DMAR enabled.

2. The dropped tx packet (DHCP) is a bit harder to recreate, but it 
still happens. Interestingly, I initially saw no dropped packets with 
ping - but after I went the DCHP route and eventually reconnected, I 
could then cause dropped tx packets with ping. To clarify:

a) start throughput
b) ping device - no packet loss - this was true for the entire test run.
c) start throughput again
d) ping - no loss.
e) drop wifi on the device & restart - first attempt worked. Repeat 
attempt yielded the dropped DHCPOFFER packets. After about 6 tries, the 
device reconnected to wifi.
f) ping again (after the reconnection) - packet loss rate about 80%.
g) simultaneously ping the wifi router - no loss.
h) After a while, packets are no longer dropped during ping. If I manage 
to cause the dhcp drop again, and then ping after the device finally 
reconnects, packet loss is significant for a while (maybe 30 sec to a 
minute). Then things return to normal. Note that the packet loss 
continues even if the reported throughput drops to nil.
i) I can't cause the initial packet loss at RX rates below about 
30,000KBPS (as reported by nethogs).  At rates over 40 I can reproduce 
this on this set of patches & config about 60% of the time.

The initial sky2 oops:

Jan 18 10:42:43 mail kernel: ------------[ cut here ]------------
Jan 18 10:42:43 mail kernel: WARNING: at lib/dma-debug.c:898 
check_sync+0xbd/0x426()
Jan 18 10:42:43 mail kernel: Hardware name: System Product Name
Jan 18 10:42:43 mail kernel: sky2 0000:06:00.0: DMA-API: device driver 
tries to sync DMA memory it has not allocated [device 
address=0x00000003249b4022] [size=98 bytes]
Jan 18 10:42:43 mail kernel: Modules linked in: microcode(+) 
ip6table_mangle ip6table_filter ip6_tables iptable_raw iptable_mangle 
ipt_MASQUERADE iptable_nat nf_nat appletalk psnap llc nfsd lockd nfs_acl 
auth_rpcgss exportfs hwmon_vid coretemp sunrpc acpi_cpufreq sit tunnel4 
ipt_LOG nf_conntrack_netbios_ns nf_conntrack_ftp nf_conntrack_ipv6 
xt_multiport xt_DSCP xt_dscp xt_MARK ipv6 dm_multipath kvm_intel kvm 
snd_hda_codec_analog snd_ens1371 gameport snd_hda_intel snd_rawmidi 
snd_hda_codec snd_ac97_codec gspca_spca505 ac97_bus snd_hwdep snd_seq 
gspca_main snd_seq_device firewire_ohci videodev firewire_core 
v4l1_compat snd_pcm i2c_i801 pcspkr v4l2_compat_ioctl32 crc_itu_t 
asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support wmi snd_timer snd sky2 
soundcore snd_page_alloc fbcon tileblit font bitblit softcursor raid456 
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx 
raid1 ata_generic pata_acpi pata_marvell nouveau ttm drm_kms_helper drm 
agpgart fb i2c_algo_bit cfbcopyarea i2c_core cfbimgblt cfbf
Jan 18 10:42:43 mail kernel: illrect [last unloaded: ip6_tables]
Jan 18 10:42:43 mail kernel: Pid: 0, comm: swapper Not tainted 
2.6.32NOMMAPNODMARAF3SKY2PSKBMAYPULL-00893-gb5d5baa-dirty #3
Jan 18 10:42:43 mail kernel: Call Trace:
Jan 18 10:42:43 mail kernel: <IRQ>  [<ffffffff81053676>] 
warn_slowpath_common+0x7c/0x94
Jan 18 10:42:43 mail kernel: [<ffffffff810536e5>] 
warn_slowpath_fmt+0x41/0x43
Jan 18 10:42:43 mail kernel: [<ffffffff8127ae7d>] check_sync+0xbd/0x426
Jan 18 10:42:43 mail kernel: [<ffffffff813c5b4c>] ? 
__netdev_alloc_skb+0x34/0x50
Jan 18 10:42:43 mail kernel: [<ffffffff8127b539>] 
debug_dma_sync_single_for_cpu+0x42/0x44
Jan 18 10:42:43 mail kernel: [<ffffffff812788d7>] ? 
swiotlb_sync_single+0x2a/0xb6
Jan 18 10:42:43 mail kernel: [<ffffffff81278a33>] ? 
swiotlb_sync_single_for_cpu+0xc/0xe
Jan 18 10:42:43 mail kernel: [<ffffffffa015eed6>] sky2_poll+0x4c6/0xae1 
[sky2]
Jan 18 10:42:43 mail kernel: [<ffffffff814673f2>] ? 
_spin_unlock_irqrestore+0x29/0x41
Jan 18 10:42:43 mail kernel: [<ffffffff813cc7ea>] net_rx_action+0xb5/0x1f3
Jan 18 10:42:43 mail kernel: [<ffffffff8105ae57>] __do_softirq+0xf8/0x1cd
Jan 18 10:42:43 mail kernel: [<ffffffff810a2e0e>] ? 
handle_IRQ_event+0x119/0x12b
Jan 18 10:42:43 mail kernel: [<ffffffff81012e1c>] call_softirq+0x1c/0x30
Jan 18 10:42:43 mail kernel: [<ffffffff810143a3>] do_softirq+0x4b/0xa6
Jan 18 10:42:43 mail kernel: [<ffffffff8105aa37>] irq_exit+0x4a/0x8c
Jan 18 10:42:43 mail kernel: [<ffffffff8146b445>] do_IRQ+0xa5/0xbc
Jan 18 10:42:43 mail kernel: [<ffffffff81012613>] ret_from_intr+0x0/0x16
Jan 18 10:42:43 mail kernel: <EOI>  [<ffffffff812c251e>] ? 
acpi_idle_enter_bm+0x256/0x28a
Jan 18 10:42:43 mail kernel: [<ffffffff812c2517>] ? 
acpi_idle_enter_bm+0x24f/0x28a
Jan 18 10:42:43 mail kernel: [<ffffffff813a1b78>] ? 
cpuidle_idle_call+0x9e/0xfa
Jan 18 10:42:43 mail kernel: [<ffffffff81010c90>] ? cpu_idle+0xb4/0xf6
Jan 18 10:42:43 mail kernel: [<ffffffff81460acf>] ? 
start_secondary+0x201/0x242
Jan 18 10:42:43 mail kernel: ---[ end trace 188c0cdbace3665e ]---

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 8:46 p.m. UTC | #47

On Mon, Jan 18, 2010 at 11:29:31AM -0500, Michael Breuer wrote:
> Ok - up on the two patches, no DMAR. Some early observations:
> 
> 1. There's an early on MMAP oops (see below). This happens once, at
> the completion of the transition to runlevel 5 (I've seen it
> entering runlevel 3 as well). This does not recur when runlevels are
> subsequently changed. I do not see this when running with DMAR
> enabled.

OK, you mentioned this oops (actually a warning only) happened during
previous tests too.

> 
> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
> still happens.

Btw, I guess you improved the test because you didn't mention it here,
even after my explicit question?:
http://permalink.gmane.org/gmane.linux.network/149171

> Interestingly, I initially saw no dropped packets
> with ping - but after I went the DCHP route and eventually
> reconnected, I could then cause dropped tx packets with ping. To
> clarify:
> 
> a) start throughput
> b) ping device - no packet loss - this was true for the entire test run.
> c) start throughput again
> d) ping - no loss.
> e) drop wifi on the device & restart - first attempt worked. Repeat
> attempt yielded the dropped DHCPOFFER packets. After about 6 tries,
> the device reconnected to wifi.
> f) ping again (after the reconnection) - packet loss rate about 80%.
> g) simultaneously ping the wifi router - no loss.
> h) After a while, packets are no longer dropped during ping. If I
> manage to cause the dhcp drop again, and then ping after the device
> finally reconnects, packet loss is significant for a while (maybe 30
> sec to a minute). Then things return to normal. Note that the packet
> loss continues even if the reported throughput drops to nil.
> i) I can't cause the initial packet loss at RX rates below about
> 30,000KBPS (as reported by nethogs).  At rates over 40 I can
> reproduce this on this set of patches & config about 60% of the
> time.

I forgot to mention, but did you try to check if these lost ping
packets are "being dropped somewhere after wireshark sees them and
before hitting the wire" like DHCPOFFER? Aren't there any sky2
warnings/resets while this happens?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 8:56 p.m. UTC | #48

On 1/18/2010 3:46 PM, Jarek Poplawski wrote:
> On Mon, Jan 18, 2010 at 11:29:31AM -0500, Michael Breuer wrote:
>    
>> Ok - up on the two patches, no DMAR. Some early observations:
>>
>> 1. There's an early on MMAP oops (see below). This happens once, at
>> the completion of the transition to runlevel 5 (I've seen it
>> entering runlevel 3 as well). This does not recur when runlevels are
>> subsequently changed. I do not see this when running with DMAR
>> enabled.
>>      
> OK, you mentioned this oops (actually a warning only) happened during
> previous tests too.
>    
Yes - dk if it's significant or not. Only obvious difference between 
DMAR and not.
>    
>> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
>> still happens.
>>      
> Btw, I guess you improved the test because you didn't mention it here,
> even after my explicit question?:
> http://permalink.gmane.org/gmane.linux.network/149171
>    
I had been focusing on the hangs - dhcp causing the initial crash from 
December. After things stabilized with the af patch & skb may pull I 
started noticing the dropped tx packets. I reported the TX loss on the 
16th of January after confirming the issue.
>    
>> Interestingly, I initially saw no dropped packets
>> with ping - but after I went the DCHP route and eventually
>> reconnected, I could then cause dropped tx packets with ping. To
>> clarify:
>>
>> a) start throughput
>> b) ping device - no packet loss - this was true for the entire test run.
>> c) start throughput again
>> d) ping - no loss.
>> e) drop wifi on the device&  restart - first attempt worked. Repeat
>> attempt yielded the dropped DHCPOFFER packets. After about 6 tries,
>> the device reconnected to wifi.
>> f) ping again (after the reconnection) - packet loss rate about 80%.
>> g) simultaneously ping the wifi router - no loss.
>> h) After a while, packets are no longer dropped during ping. If I
>> manage to cause the dhcp drop again, and then ping after the device
>> finally reconnects, packet loss is significant for a while (maybe 30
>> sec to a minute). Then things return to normal. Note that the packet
>> loss continues even if the reported throughput drops to nil.
>> i) I can't cause the initial packet loss at RX rates below about
>> 30,000KBPS (as reported by nethogs).  At rates over 40 I can
>> reproduce this on this set of patches&  config about 60% of the
>> time.
>>      
> I forgot to mention, but did you try to check if these lost ping
> packets are "being dropped somewhere after wireshark sees them and
> before hitting the wire" like DHCPOFFER? Aren't there any sky2
> warnings/resets while this happens?
>
> Jarek P.
>    
Yes. There are no errors, and no statistics anywhere that I know to look 
reflect the loss. Nothing in netstat; ethtool -S; etc. The only loss 
reported is RX. The recent TX warnings/resets happened while the machine 
was up for several days and while unattended and under high RX load.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stephen Hemminger Jan. 18, 2010, 9 p.m. UTC | #49

On Mon, 18 Jan 2010 15:56:45 -0500
Michael Breuer <mbreuer@majjas.com> wrote:

> >> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
> >> still happens.
> >>        

You might want to use tc filter rule to set priority of DHCP packets
higher. This would cause them to be in a separate queue and eliminate
the problem.

Jarek Poplawski Jan. 18, 2010, 9:06 p.m. UTC | #50

On Mon, Jan 18, 2010 at 01:00:38PM -0800, Stephen Hemminger wrote:
> On Mon, 18 Jan 2010 15:56:45 -0500
> Michael Breuer <mbreuer@majjas.com> wrote:
> 
> > >> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
> > >> still happens.
> > >>        
> 
> You might want to use tc filter rule to set priority of DHCP packets
> higher. This would cause them to be in a separate queue and eliminate
> the problem.
> 

But it looks like they are dropped in TX after passing a qdisc (and
wireshark).

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 9:24 p.m. UTC | #51

On 1/18/2010 4:00 PM, Stephen Hemminger wrote:
> On Mon, 18 Jan 2010 15:56:45 -0500
> Michael Breuer<mbreuer@majjas.com>  wrote:
>
>    
>>>> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
>>>> still happens.
>>>>
>>>>          
> You might want to use tc filter rule to set priority of DHCP packets
> higher. This would cause them to be in a separate queue and eliminate
> the problem.
>
>    
Ok - for fun, tried that - no change. Not sure I see why this might be a 
factor. The packet loss happens when TX load is low and RX high.
Also,  packets only being dropped if traversing a router vs.to  the 
router itself.  Keep in mind that pings to the router did not lose 
packets, pings through the router lost packets. The router was not under 
load (traffic is being generated from a device connected via the 1Gb 
switch, not the wifi router), and tcpdump on the router input port shows 
the pings to the router, but not the ones through the router.

One added note, when I just tried this, the test data ended while the 
packet loss was occurring. The DHCPOFFER packet loss did not clear until 
about a minute after the throughput abated. I really think something is 
getting hosed, and I'd but some weird interaction with the arp logic 
high on the list of suspects. Not sure what else would be a factor when 
looking at the extra hop on the same subnet.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 9:25 p.m. UTC | #52

On Mon, Jan 18, 2010 at 03:56:45PM -0500, Michael Breuer wrote:
> On 1/18/2010 3:46 PM, Jarek Poplawski wrote:
> >On Mon, Jan 18, 2010 at 11:29:31AM -0500, Michael Breuer wrote:
> >>Ok - up on the two patches, no DMAR. Some early observations:
> >>
> >>1. There's an early on MMAP oops (see below). This happens once, at
> >>the completion of the transition to runlevel 5 (I've seen it
> >>entering runlevel 3 as well). This does not recur when runlevels are
> >>subsequently changed. I do not see this when running with DMAR
> >>enabled.
> >OK, you mentioned this oops (actually a warning only) happened during
> >previous tests too.
> Yes - dk if it's significant or not. Only obvious difference between
> DMAR and not.

OK, let's try (as long as possible) if it can break so hard as with
DMAR.

> >>2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
> >>still happens.
> >Btw, I guess you improved the test because you didn't mention it here,
> >even after my explicit question?:
> >http://permalink.gmane.org/gmane.linux.network/149171
> I had been focusing on the hangs - dhcp causing the initial crash
> from December. After things stabilized with the af patch & skb may
> pull I started noticing the dropped tx packets. I reported the TX
> loss on the 16th of January after confirming the issue.

OK, but we need to establish some status quo after these patches
before any new things (including DMAR), so I'd suggest trying this
config really longer and harder.

> >>Interestingly, I initially saw no dropped packets
> >>with ping - but after I went the DCHP route and eventually
> >>reconnected, I could then cause dropped tx packets with ping. To
> >>clarify:
> >>
> >>a) start throughput
> >>b) ping device - no packet loss - this was true for the entire test run.
> >>c) start throughput again
> >>d) ping - no loss.
> >>e) drop wifi on the device&  restart - first attempt worked. Repeat
> >>attempt yielded the dropped DHCPOFFER packets. After about 6 tries,
> >>the device reconnected to wifi.
> >>f) ping again (after the reconnection) - packet loss rate about 80%.
> >>g) simultaneously ping the wifi router - no loss.
> >>h) After a while, packets are no longer dropped during ping. If I
> >>manage to cause the dhcp drop again, and then ping after the device
> >>finally reconnects, packet loss is significant for a while (maybe 30
> >>sec to a minute). Then things return to normal. Note that the packet
> >>loss continues even if the reported throughput drops to nil.
> >>i) I can't cause the initial packet loss at RX rates below about
> >>30,000KBPS (as reported by nethogs).  At rates over 40 I can
> >>reproduce this on this set of patches&  config about 60% of the
> >>time.
> >I forgot to mention, but did you try to check if these lost ping
> >packets are "being dropped somewhere after wireshark sees them and
> >before hitting the wire" like DHCPOFFER? Aren't there any sky2
> >warnings/resets while this happens?
> >
> >Jarek P.
> Yes. There are no errors, and no statistics anywhere that I know to
> look reflect the loss. Nothing in netstat; ethtool -S; etc. The only
> loss reported is RX. The recent TX warnings/resets happened while
> the machine was up for several days and while unattended and under
> high RX load.

Please check "tc -s qdisc" each time as well.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 9:39 p.m. UTC | #53

On 1/18/2010 4:25 PM, Jarek Poplawski wrote:
> On Mon, Jan 18, 2010 at 03:56:45PM -0500, Michael Breuer wrote:
>    
>> On 1/18/2010 3:46 PM, Jarek Poplawski wrote:
>>      
>>> On Mon, Jan 18, 2010 at 11:29:31AM -0500, Michael Breuer wrote:
>>>        
>>>> Ok - up on the two patches, no DMAR. Some early observations:
>>>>
>>>> 1. There's an early on MMAP oops (see below). This happens once, at
>>>> the completion of the transition to runlevel 5 (I've seen it
>>>> entering runlevel 3 as well). This does not recur when runlevels are
>>>> subsequently changed. I do not see this when running with DMAR
>>>> enabled.
>>>>          
>>> OK, you mentioned this oops (actually a warning only) happened during
>>> previous tests too.
>>>        
>> Yes - dk if it's significant or not. Only obvious difference between
>> DMAR and not.
>>      
> OK, let's try (as long as possible) if it can break so hard as with
> DMAR.
>
>    
>>>> 2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
>>>> still happens.
>>>>          
>>> Btw, I guess you improved the test because you didn't mention it here,
>>> even after my explicit question?:
>>> http://permalink.gmane.org/gmane.linux.network/149171
>>>        
>> I had been focusing on the hangs - dhcp causing the initial crash
>> from December. After things stabilized with the af patch&  skb may
>> pull I started noticing the dropped tx packets. I reported the TX
>> loss on the 16th of January after confirming the issue.
>>      
> OK, but we need to establish some status quo after these patches
> before any new things (including DMAR), so I'd suggest trying this
> config really longer and harder.
>
>    
>>>> Interestingly, I initially saw no dropped packets
>>>> with ping - but after I went the DCHP route and eventually
>>>> reconnected, I could then cause dropped tx packets with ping. To
>>>> clarify:
>>>>
>>>> a) start throughput
>>>> b) ping device - no packet loss - this was true for the entire test run.
>>>> c) start throughput again
>>>> d) ping - no loss.
>>>> e) drop wifi on the device&   restart - first attempt worked. Repeat
>>>> attempt yielded the dropped DHCPOFFER packets. After about 6 tries,
>>>> the device reconnected to wifi.
>>>> f) ping again (after the reconnection) - packet loss rate about 80%.
>>>> g) simultaneously ping the wifi router - no loss.
>>>> h) After a while, packets are no longer dropped during ping. If I
>>>> manage to cause the dhcp drop again, and then ping after the device
>>>> finally reconnects, packet loss is significant for a while (maybe 30
>>>> sec to a minute). Then things return to normal. Note that the packet
>>>> loss continues even if the reported throughput drops to nil.
>>>> i) I can't cause the initial packet loss at RX rates below about
>>>> 30,000KBPS (as reported by nethogs).  At rates over 40 I can
>>>> reproduce this on this set of patches&   config about 60% of the
>>>> time.
>>>>          
>>> I forgot to mention, but did you try to check if these lost ping
>>> packets are "being dropped somewhere after wireshark sees them and
>>> before hitting the wire" like DHCPOFFER? Aren't there any sky2
>>> warnings/resets while this happens?
>>>
>>> Jarek P.
>>>        
>> Yes. There are no errors, and no statistics anywhere that I know to
>> look reflect the loss. Nothing in netstat; ethtool -S; etc. The only
>> loss reported is RX. The recent TX warnings/resets happened while
>> the machine was up for several days and while unattended and under
>> high RX load.
>>      
> Please check "tc -s qdisc" each time as well.
>
> Jarek P
>    

Some output from tc -s qdisc:

Before test:
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 35279532 bytes 291080 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 377308 bytes 3107 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0

During test (after initial observed packet loss):
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 123389424 bytes 1781403 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 400862 bytes 3250 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0

During test - while packet loss occuring:
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 150518974 bytes 2138312 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 422003 bytes 3432 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0

After the conclusion of the test:
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 244900497 bytes 3416350 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 
0 0 1 1 1 1 1 1 1 1
  Sent 564380 bytes 4708 pkt (dropped 0, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0


During the test, 8.9GB received; 232.9MB sent).

I also connected a second device through the wifi router. I was able to 
ping that device w/o loss while DHCP packets were being dropped to the 
other connected device.

Last note: just moved to 2.6.32.4 from .3 for this test (from git).

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 9:50 p.m. UTC | #54

On Mon, Jan 18, 2010 at 04:24:52PM -0500, Michael Breuer wrote:
> On 1/18/2010 4:00 PM, Stephen Hemminger wrote:
> >On Mon, 18 Jan 2010 15:56:45 -0500
> >Michael Breuer<mbreuer@majjas.com>  wrote:
> >
> >>>>2. The dropped tx packet (DHCP) is a bit harder to recreate, but it
> >>>>still happens.
> >>>>
> >You might want to use tc filter rule to set priority of DHCP packets
> >higher. This would cause them to be in a separate queue and eliminate
> >the problem.
> >
> Ok - for fun, tried that - no change. Not sure I see why this might
> be a factor. The packet loss happens when TX load is low and RX
> high.
> Also,  packets only being dropped if traversing a router vs.to  the
> router itself.  Keep in mind that pings to the router did not lose
> packets, pings through the router lost packets. The router was not
> under load (traffic is being generated from a device connected via
> the 1Gb switch, not the wifi router), and tcpdump on the router
> input port shows the pings to the router, but not the ones through
> the router.
> 
> One added note, when I just tried this, the test data ended while
> the packet loss was occurring. The DHCPOFFER packet loss did not
> clear until about a minute after the throughput abated. I really
> think something is getting hosed, and I'd but some weird interaction
> with the arp logic high on the list of suspects. Not sure what else
> would be a factor when looking at the extra hop on the same subnet.

Good point! Actually, IIRC, your setup might be a problem: you seem
to have two switches on the path (I guess the router is a bridge for
these wireless), so I wonder if it's not something between them.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 10:08 p.m. UTC | #55

On Mon, Jan 18, 2010 at 04:39:24PM -0500, Michael Breuer wrote:
> Some output from tc -s qdisc:
...
> After the conclusion of the test:
> qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2
> 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 244900497 bytes 3416350 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2
> 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 564380 bytes 4708 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0

Great!

> 
> 
> During the test, 8.9GB received; 232.9MB sent).
> 
> I also connected a second device through the wifi router. I was able
> to ping that device w/o loss while DHCP packets were being dropped
> to the other connected device.

Could you remind us if the problem is always with this first device?
Btw, I wonder if you could test it skipping the (HP?) switch?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 10:17 p.m. UTC | #56

On Mon, Jan 18, 2010 at 11:08:14PM +0100, Jarek Poplawski wrote:
> Btw, I wonder if you could test it skipping the (HP?) switch?

If so, then of course don't forget to try tcpdump on the router.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 10:25 p.m. UTC | #57

On 1/18/2010 5:08 PM, Jarek Poplawski wrote:
> On Mon, Jan 18, 2010 at 04:39:24PM -0500, Michael Breuer wrote:
>    
>> Some output from tc -s qdisc:
>>      
> ...
>    
>> After the conclusion of the test:
>> qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2
>> 1 2 0 0 1 1 1 1 1 1 1 1
>>   Sent 244900497 bytes 3416350 pkt (dropped 0, overlimits 0 requeues 0)
>>   rate 0bit 0pps backlog 0b 0p requeues 0
>> qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2
>> 1 2 0 0 1 1 1 1 1 1 1 1
>>   Sent 564380 bytes 4708 pkt (dropped 0, overlimits 0 requeues 0)
>>   rate 0bit 0pps backlog 0b 0p requeues 0
>>      
> Great!
>
>    
>>
>> During the test, 8.9GB received; 232.9MB sent).
>>
>> I also connected a second device through the wifi router. I was able
>> to ping that device w/o loss while DHCP packets were being dropped
>> to the other connected device.
>>      
> Could you remind us if the problem is always with this first device?
> Btw, I wonder if you could test it skipping the (HP?) switch?
>
> Jarek P.
>    
No - can be any device connected via the wifi router - wired or wireless.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 18, 2010, 10:40 p.m. UTC | #58

On Mon, Jan 18, 2010 at 05:25:43PM -0500, Michael Breuer wrote:
> On 1/18/2010 5:08 PM, Jarek Poplawski wrote:
> >Could you remind us if the problem is always with this first device?
> >Btw, I wonder if you could test it skipping the (HP?) switch?
> >
> >Jarek P.
> No - can be any device connected via the wifi router - wired or wireless.

Anyway, if it can't be repeated with this Win7 box or even the router
getting dhcp itself, then this router's interaction seems most
suspected.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 18, 2010, 10:47 p.m. UTC | #59

On 1/18/2010 5:17 PM, Jarek Poplawski wrote:
> On Mon, Jan 18, 2010 at 11:08:14PM +0100, Jarek Poplawski wrote:
>    
>> Btw, I wonder if you could test it skipping the (HP?) switch?
>>      
> If so, then of course don't forget to try tcpdump on the router.
>
> Jarek P.
>    
Well - no.... but I'm not sure that would show anything.

Setup diagram:

Server->gb switch-> (100mb) wifi router -> devices
                     |
               Win7 PC (gb)

The problem does not occur (at least I haven't been able to recreate it) 
at 100mb, and the wifi router doesn't do 1Gb. I drive the traffic from 
the win7 PC to the server. I've seen the loss when the only traffic 
going through the wifi router was ping & dhcp. I've also never seen any 
loss on a device directly attached to the 1GB switch. I can drive load 
through the wifi router while driving load from the Win7 box, but don't 
see TX packet loss at all when not doing DHCP RELEASE/RENEW.

As there is no packet loss to devices not involved in the DHCP sequence 
through the same path, I'm not really sure that the GB switch is implicated.

As I don't have a standalone sniffer, I'm thinking that it might be 
easier to instrument places where the TX packet could be dropped and see 
at least whether it's getting to the card.

Given the circumstances of the TX drop, and that it was DHCP traffic 
while under load that caused the oops rectified with the two patches, 
I'm thinking that the packet loss is the current manifestation of 
whatever the underlying problem is. Given the extra hop required to 
break things, and given that a dhcp release/renew seems to trigger 
things, I keep coming back to arp logic as being somehow implicated.

If arp is somehow involved, then I'd expect to see manifestations under 
similar circumstances with other drivers. As the pskb_may_pull patch 
stopped the crash, perhaps other drivers do suffer packet loss and it's 
just not been widely noticed or attributed to the kernel - especially if 
the network topology is a factor. I do know people at large enterprises 
who have been complaining of what *could* be this same issue, however 
they're currently blaming their switch vendors. As most traffic is TCP, 
this is really only noticed by those few places deeply concerned with 
latency. It's likely something altogether different, but then again, 
maybe not.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 19, 2010, 5:46 a.m. UTC | #60

On 1/18/2010 5:47 PM, Michael Breuer wrote:
> On 1/18/2010 5:17 PM, Jarek Poplawski wrote:
>> On Mon, Jan 18, 2010 at 11:08:14PM +0100, Jarek Poplawski wrote:
>>> Btw, I wonder if you could test it skipping the (HP?) switch?
>> If so, then of course don't forget to try tcpdump on the router.
>>
>> Jarek P.
> Well - no.... but I'm not sure that would show anything.
>
> Setup diagram:
>
> Server->gb switch-> (100mb) wifi router -> devices
>                     |
>               Win7 PC (gb)
>
> The problem does not occur (at least I haven't been able to recreate 
> it) at 100mb, and the wifi router doesn't do 1Gb. I drive the traffic 
> from the win7 PC to the server. I've seen the loss when the only 
> traffic going through the wifi router was ping & dhcp. I've also never 
> seen any loss on a device directly attached to the 1GB switch. I can 
> drive load through the wifi router while driving load from the Win7 
> box, but don't see TX packet loss at all when not doing DHCP 
> RELEASE/RENEW.
>
> As there is no packet loss to devices not involved in the DHCP 
> sequence through the same path, I'm not really sure that the GB switch 
> is implicated.
>
> As I don't have a standalone sniffer, I'm thinking that it might be 
> easier to instrument places where the TX packet could be dropped and 
> see at least whether it's getting to the card.
>
> Given the circumstances of the TX drop, and that it was DHCP traffic 
> while under load that caused the oops rectified with the two patches, 
> I'm thinking that the packet loss is the current manifestation of 
> whatever the underlying problem is. Given the extra hop required to 
> break things, and given that a dhcp release/renew seems to trigger 
> things, I keep coming back to arp logic as being somehow implicated.
>
> If arp is somehow involved, then I'd expect to see manifestations 
> under similar circumstances with other drivers. As the pskb_may_pull 
> patch stopped the crash, perhaps other drivers do suffer packet loss 
> and it's just not been widely noticed or attributed to the kernel - 
> especially if the network topology is a factor. I do know people at 
> large enterprises who have been complaining of what *could* be this 
> same issue, however they're currently blaming their switch vendors. As 
> most traffic is TCP, this is really only noticed by those few places 
> deeply concerned with latency. It's likely something altogether 
> different, but then again, maybe not.
Ok - one last update for a while ...not sure what's next... I put some 
printk's into sky2.c xmit logic - the packets are being sent to the 
card, and the i/o's are completing successfully. So it would seem either 
the switch is dropping the packets, or else the wifi router is. As 
tcpdump doesn't show the packets arriving on the wifi router, I'm 
leaning towards the switch. I ran wireshark on the win7 box to see what 
is coming off the switch. I did notice one thing that's visible to the 
win7 box but is not showing up on the linux wireshark - before every 
successful dhcpoffer, there's an XID message broadcast from the device. 
I'm wondering why I don't see this on the linux side:

The packet is from the mac of the device, dst ff:ff:ff:ff:ff:ff; 
protocol eth:llc... hex packet: ffffffffffff001cccf39ff600060001af810100.

Now I guess I've got some reading to do... I've got no idea what the 
correct application of llc messages would be given my topology :(. I do 
suspect that the llc stuff (or lack thereof under some conditions) is 
causing the switch to fail to forward the dhcpoffer message. As the 
dhcpoffer message is not broadcast, but directed to the remote mac 
address and as that address is not connected directly to the switch, I'm 
guessing that under some conditions whatever tells the switch how to 
find the mac is missing. I'd guess that the wifi router should be 
letting the switch know around the time it forwards the first arp and/or 
DHCP broadcast message from the client... or maybe the linux box should 
be doing something before the offer.

So net-net, as far as my TX packet loss issue, sky2 is in the clear. If 
something on the linux side should be informing the switch about 
something then there may still be an issue. If the wifi router should be 
doing something differently, then it's unfortunately likely a 2.4.37 
kernel issue (That's what dd-wrt is using).



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jarek Poplawski Jan. 19, 2010, 8:41 a.m. UTC | #61

On Tue, Jan 19, 2010 at 12:46:24AM -0500, Michael Breuer wrote:
> So net-net, as far as my TX packet loss issue, sky2 is in the clear. If  
> something on the linux side should be informing the switch about  
> something then there may still be an issue. If the wifi router should be  
> doing something differently, then it's unfortunately likely a 2.4.37  
> kernel issue (That's what dd-wrt is using).

IMHO until there is no proof from a sniffer or some regs dumps the
switch and the router are more suspicious than your NIC or linux box.
Then debugging these other things isn't so much interesting from my
POV ;-)

Anyway, if you only want to get it working (instead of debugging),
it seems you might try moving the dhcp server to the router or maybe
even using two separate servers with their pools - unless I missed
something in your config.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 19, 2010, 3:28 p.m. UTC | #62

On 1/19/2010 3:41 AM, Jarek Poplawski wrote:
> On Tue, Jan 19, 2010 at 12:46:24AM -0500, Michael Breuer wrote:
>    
>> So net-net, as far as my TX packet loss issue, sky2 is in the clear. If
>> something on the linux side should be informing the switch about
>> something then there may still be an issue. If the wifi router should be
>> doing something differently, then it's unfortunately likely a 2.4.37
>> kernel issue (That's what dd-wrt is using).
>>      
> IMHO until there is no proof from a sniffer or some regs dumps the
> switch and the router are more suspicious than your NIC or linux box.
> Then debugging these other things isn't so much interesting from my
> POV ;-)
>
> Anyway, if you only want to get it working (instead of debugging),
> it seems you might try moving the dhcp server to the router or maybe
> even using two separate servers with their pools - unless I missed
> something in your config.
>
> Jarek P.
>    
I'm going to take a stab at debugging before creating a new subnet. Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Breuer Jan. 21, 2010, 7:48 p.m. UTC | #63

On 1/19/2010 3:41 AM, Jarek Poplawski wrote:
> On Tue, Jan 19, 2010 at 12:46:24AM -0500, Michael Breuer wrote:
>    
>> So net-net, as far as my TX packet loss issue, sky2 is in the clear. If
>> something on the linux side should be informing the switch about
>> something then there may still be an issue. If the wifi router should be
>> doing something differently, then it's unfortunately likely a 2.4.37
>> kernel issue (That's what dd-wrt is using).
>>      
> IMHO until there is no proof from a sniffer or some regs dumps the
> switch and the router are more suspicious than your NIC or linux box.
> Then debugging these other things isn't so much interesting from my
> POV ;-)
>
> Anyway, if you only want to get it working (instead of debugging),
> it seems you might try moving the dhcp server to the router or maybe
> even using two separate servers with their pools - unless I missed
> something in your config.
>
> Jarek P.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>    
Postscript: HP has confirmed that the 1400-8g switch doesn't work, 
"well," in a mixed 100Mb/1000Mb environment, and is dropping DHCP packets.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

af_packet: Don't use skb after dev_queue_xmit()

Commit Message

Comments

Patch