diff mbox

Difficulties to get 1Gbps on be2net ethernet card

Message ID 1338980484.2760.4219.camel@edumazet-glaptop
State RFC, archived
Delegated to: David Miller
Headers show

Commit Message

Eric Dumazet June 6, 2012, 11:01 a.m. UTC 
On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:

> Well, well, well, after having tested several configurations, several
> drivers, I have a big difference between an old 2.6.26 kernel and a
> newer one (I tried 3.2 and 3.4).
> 
> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
> set to 4096. I am sending packets only, nothing on RX.
> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
> kernel, but a lot of drops with a newer kernel.
> So, I don't know if I missed something in my kernel configuration, but
> I have used the 2.6.26 one as a reference, in order to set the same
> options (DMA related, etc).
> 
> I easily reproduce this problem and setting a bigger txqueuelen solves
> it partially.
> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
> 
> If you have any idea, I am interested, as this is a big issue for my use case.
> 

Yep.

This driver wants to limit number of tx completions, thats just wrong.

Fix and dirty patch:




--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Jean-Michel Hautbois June 6, 2012, 12:34 p.m. UTC | #1 
2012/6/6 Eric Dumazet <eric.dumazet@gmail.com>:
> On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
>
>> Well, well, well, after having tested several configurations, several
>> drivers, I have a big difference between an old 2.6.26 kernel and a
>> newer one (I tried 3.2 and 3.4).
>>
>> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
>> set to 4096. I am sending packets only, nothing on RX.
>> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
>> kernel, but a lot of drops with a newer kernel.
>> So, I don't know if I missed something in my kernel configuration, but
>> I have used the 2.6.26 one as a reference, in order to set the same
>> options (DMA related, etc).
>>
>> I easily reproduce this problem and setting a bigger txqueuelen solves
>> it partially.
>> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
>>
>> If you have any idea, I am interested, as this is a big issue for my use case.
>>
>
> Yep.
>
> This driver wants to limit number of tx completions, thats just wrong.
>
> Fix and dirty patch:
>
>
> diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
> index c5c4c0e..1e8f8a6 100644
> --- a/drivers/net/ethernet/emulex/benet/be.h
> +++ b/drivers/net/ethernet/emulex/benet/be.h
> @@ -105,7 +105,7 @@ static inline char *nic_name(struct pci_dev *pdev)
>  #define MAX_TX_QS              8
>  #define MAX_ROCE_EQS           5
>  #define MAX_MSIX_VECTORS       (MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
> -#define BE_TX_BUDGET           256
> +#define BE_TX_BUDGET           65535
>  #define BE_NAPI_WEIGHT         64
>  #define MAX_RX_POST            BE_NAPI_WEIGHT /* Frags posted at a time */
>  #define RX_FRAGS_REFILL_WM     (RX_Q_LEN - MAX_RX_POST)
>

I will try that in a few minutes.
I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
a broadcom (bnx2x).

JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 6, 2012, 1:07 p.m. UTC | #2 
2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
> 2012/6/6 Eric Dumazet <eric.dumazet@gmail.com>:
>> On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
>>
>>> Well, well, well, after having tested several configurations, several
>>> drivers, I have a big difference between an old 2.6.26 kernel and a
>>> newer one (I tried 3.2 and 3.4).
>>>
>>> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
>>> set to 4096. I am sending packets only, nothing on RX.
>>> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
>>> kernel, but a lot of drops with a newer kernel.
>>> So, I don't know if I missed something in my kernel configuration, but
>>> I have used the 2.6.26 one as a reference, in order to set the same
>>> options (DMA related, etc).
>>>
>>> I easily reproduce this problem and setting a bigger txqueuelen solves
>>> it partially.
>>> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
>>>
>>> If you have any idea, I am interested, as this is a big issue for my use case.
>>>
>>
>> Yep.
>>
>> This driver wants to limit number of tx completions, thats just wrong.
>>
>> Fix and dirty patch:
>>
>>
>> diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
>> index c5c4c0e..1e8f8a6 100644
>> --- a/drivers/net/ethernet/emulex/benet/be.h
>> +++ b/drivers/net/ethernet/emulex/benet/be.h
>> @@ -105,7 +105,7 @@ static inline char *nic_name(struct pci_dev *pdev)
>>  #define MAX_TX_QS              8
>>  #define MAX_ROCE_EQS           5
>>  #define MAX_MSIX_VECTORS       (MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
>> -#define BE_TX_BUDGET           256
>> +#define BE_TX_BUDGET           65535
>>  #define BE_NAPI_WEIGHT         64
>>  #define MAX_RX_POST            BE_NAPI_WEIGHT /* Frags posted at a time */
>>  #define RX_FRAGS_REFILL_WM     (RX_Q_LEN - MAX_RX_POST)
>>
>
> I will try that in a few minutes.
> I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
> a broadcom (bnx2x).
>

And it is not really better, still need about 18000 at 2.4Gbps in
order to avoid drops...
I really think there is something in the networking stack or in my
configuration (DMA ? Something else ?)...
As it doesn't seem to be driver related as I said...

JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 6, 2012, 2:36 p.m. UTC | #3 
2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
> 2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
>> 2012/6/6 Eric Dumazet <eric.dumazet@gmail.com>:
>>> On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
>>>
>>>> Well, well, well, after having tested several configurations, several
>>>> drivers, I have a big difference between an old 2.6.26 kernel and a
>>>> newer one (I tried 3.2 and 3.4).
>>>>
>>>> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
>>>> set to 4096. I am sending packets only, nothing on RX.
>>>> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
>>>> kernel, but a lot of drops with a newer kernel.
>>>> So, I don't know if I missed something in my kernel configuration, but
>>>> I have used the 2.6.26 one as a reference, in order to set the same
>>>> options (DMA related, etc).
>>>>
>>>> I easily reproduce this problem and setting a bigger txqueuelen solves
>>>> it partially.
>>>> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
>>>>
>>>> If you have any idea, I am interested, as this is a big issue for my use case.
>>>>
>>>
>>> Yep.
>>>
>>> This driver wants to limit number of tx completions, thats just wrong.
>>>
>>> Fix and dirty patch:
>>>
>>>
>>> diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
>>> index c5c4c0e..1e8f8a6 100644
>>> --- a/drivers/net/ethernet/emulex/benet/be.h
>>> +++ b/drivers/net/ethernet/emulex/benet/be.h
>>> @@ -105,7 +105,7 @@ static inline char *nic_name(struct pci_dev *pdev)
>>>  #define MAX_TX_QS              8
>>>  #define MAX_ROCE_EQS           5
>>>  #define MAX_MSIX_VECTORS       (MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
>>> -#define BE_TX_BUDGET           256
>>> +#define BE_TX_BUDGET           65535
>>>  #define BE_NAPI_WEIGHT         64
>>>  #define MAX_RX_POST            BE_NAPI_WEIGHT /* Frags posted at a time */
>>>  #define RX_FRAGS_REFILL_WM     (RX_Q_LEN - MAX_RX_POST)
>>>
>>
>> I will try that in a few minutes.
>> I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
>> a broadcom (bnx2x).
>>
>
> And it is not really better, still need about 18000 at 2.4Gbps in
> order to avoid drops...
> I really think there is something in the networking stack or in my
> configuration (DMA ? Something else ?)...
> As it doesn't seem to be driver related as I said...
>

If it can help, on a 3.0 kernel a txqueuelen of 9000 is sufficient in
order to get this bandwith on TX.

JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 7, 2012, 12:27 p.m. UTC | #4 
2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
> 2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
>> 2012/6/6 Jean-Michel Hautbois <jhautbois@gmail.com>:
>>> 2012/6/6 Eric Dumazet <eric.dumazet@gmail.com>:
>>>> On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
>>>>
>>>>> Well, well, well, after having tested several configurations, several
>>>>> drivers, I have a big difference between an old 2.6.26 kernel and a
>>>>> newer one (I tried 3.2 and 3.4).
>>>>>
>>>>> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
>>>>> set to 4096. I am sending packets only, nothing on RX.
>>>>> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
>>>>> kernel, but a lot of drops with a newer kernel.
>>>>> So, I don't know if I missed something in my kernel configuration, but
>>>>> I have used the 2.6.26 one as a reference, in order to set the same
>>>>> options (DMA related, etc).
>>>>>
>>>>> I easily reproduce this problem and setting a bigger txqueuelen solves
>>>>> it partially.
>>>>> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
>>>>>
>>>>> If you have any idea, I am interested, as this is a big issue for my use case.
>>>>>
>>>>
>>>> Yep.
>>>>
>>>> This driver wants to limit number of tx completions, thats just wrong.
>>>>
>>>> Fix and dirty patch:
>>>>
>>>>
>>>> diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
>>>> index c5c4c0e..1e8f8a6 100644
>>>> --- a/drivers/net/ethernet/emulex/benet/be.h
>>>> +++ b/drivers/net/ethernet/emulex/benet/be.h
>>>> @@ -105,7 +105,7 @@ static inline char *nic_name(struct pci_dev *pdev)
>>>>  #define MAX_TX_QS              8
>>>>  #define MAX_ROCE_EQS           5
>>>>  #define MAX_MSIX_VECTORS       (MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
>>>> -#define BE_TX_BUDGET           256
>>>> +#define BE_TX_BUDGET           65535
>>>>  #define BE_NAPI_WEIGHT         64
>>>>  #define MAX_RX_POST            BE_NAPI_WEIGHT /* Frags posted at a time */
>>>>  #define RX_FRAGS_REFILL_WM     (RX_Q_LEN - MAX_RX_POST)
>>>>
>>>
>>> I will try that in a few minutes.
>>> I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
>>> a broadcom (bnx2x).
>>>
>>
>> And it is not really better, still need about 18000 at 2.4Gbps in
>> order to avoid drops...
>> I really think there is something in the networking stack or in my
>> configuration (DMA ? Something else ?)...
>> As it doesn't seem to be driver related as I said...
>>
>
> If it can help, on a 3.0 kernel a txqueuelen of 9000 is sufficient in
> order to get this bandwith on TX.
>
> JM

All,

I made some tests, and I didn't mention it : I am using the bonding
driver over my ethernet drivers (be2net/mlx4 etc.).
When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
When I disable bonding, and use directly the NIC then I don't see any
drops in qdisc and it works well.
So, I think there is something between 2.6.26 and 3.0 in the bonding
driver which causes this issue.

Any help would be appreciated :).
Regards,
JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet June 7, 2012, 12:31 p.m. UTC | #5 
On Thu, 2012-06-07 at 14:27 +0200, Jean-Michel Hautbois wrote:

> I made some tests, and I didn't mention it : I am using the bonding
> driver over my ethernet drivers (be2net/mlx4 etc.).
> When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
> When I disable bonding, and use directly the NIC then I don't see any
> drops in qdisc and it works well.
> So, I think there is something between 2.6.26 and 3.0 in the bonding
> driver which causes this issue.
> 

What your bond configuration looks like ?

cat /proc/net/bonding/bond0
ifconfig -a
tc -s -d qdisc



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 7, 2012, 12:54 p.m. UTC | #6 
2012/6/7 Eric Dumazet <eric.dumazet@gmail.com>:
> On Thu, 2012-06-07 at 14:27 +0200, Jean-Michel Hautbois wrote:
>
>> I made some tests, and I didn't mention it : I am using the bonding
>> driver over my ethernet drivers (be2net/mlx4 etc.).
>> When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
>> When I disable bonding, and use directly the NIC then I don't see any
>> drops in qdisc and it works well.
>> So, I think there is something between 2.6.26 and 3.0 in the bonding
>> driver which causes this issue.
>>
>
> What your bond configuration looks like ?
>
> cat /proc/net/bonding/bond0
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 50
Up Delay (ms): 100
Down Delay (ms): 0

Slave Interface: eth1
MII Status: up
Speed: 4000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:b5:99:b9:8d:d4
Slave queue ID: 0

Slave Interface: eth9
MII Status: up
Speed: 4000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 78:e7:d1:68:bb:38
Slave queue ID: 0

> ifconfig -a
bond0     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d0
          inet addr:192.168.250.11  Bcast:192.168.250.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:6570 errors:0 dropped:74 overruns:0 frame:0
          TX packets:5208 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:900993 (879.8 KiB)  TX bytes:863735 (843.4 KiB)

bond1     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd4/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:4096  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)

bond2     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d1
          inet addr:10.11.17.190  Bcast:10.11.17.255  Mask:255.255.255.128
          inet6 addr: fe80::6ab5:99ff:feb9:8dd1/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1301996 errors:0 dropped:27 overruns:0 frame:0
          TX packets:959 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1760302182 (1.6 GiB)  TX bytes:502828 (491.0 KiB)

bond3     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d5
          inet6 addr: fe80::6ab5:99ff:feb9:8dd5/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:942641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1278313720 (1.1 GiB)  TX bytes:2616 (2.5 KiB)

bond4     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d2
          inet addr:192.168.202.1  Bcast:192.168.202.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd2/64 Scope:Link
          UP BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:90 (90.0 B)

bond5     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d6
          inet addr:192.168.203.1  Bcast:192.168.203.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd6/64 Scope:Link
          UP BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:90 (90.0 B)

bond6     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d3
          inet6 addr: fe80::6ab5:99ff:feb9:8dd3/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:25531 (24.9 KiB)  TX bytes:1046 (1.0 KiB)

bond7     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d7
          UP BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

bond3.4   Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d5
          inet addr:192.168.201.1  Bcast:192.168.201.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:942641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1265116746 (1.1 GiB)  TX bytes:1980 (1.9 KiB)

bond6.7   Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d3
          inet addr:192.168.204.1  Bcast:192.168.204.255  Mask:255.255.255.0
          inet6 addr: fe80::6ab5:99ff:feb9:8dd3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:21765 (21.2 KiB)  TX bytes:468 (468.0 B)

eth0      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:6496 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5208 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:896849 (875.8 KiB)  TX bytes:863735 (843.4 KiB)

eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)

eth2      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d1
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1301996 errors:0 dropped:27 overruns:0 frame:0
          TX packets:959 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1760302182 (1.6 GiB)  TX bytes:502828 (491.0 KiB)

eth3      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d5
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth4      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d2
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth5      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d6
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth6      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d3
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:25531 (24.9 KiB)  TX bytes:1046 (1.0 KiB)

eth7      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d7
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth8      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:74 errors:0 dropped:74 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4144 (4.0 KiB)  TX bytes:0 (0.0 B)

eth9      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth10     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d1
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth11     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d5
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:942641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1278313720 (1.1 GiB)  TX bytes:2616 (2.5 KiB)

eth12     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d2
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:90 (90.0 B)

eth13     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d6
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:90 (90.0 B)

eth14     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d3
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth15     Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d7
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:667883 errors:0 dropped:0 overruns:0 frame:0
          TX packets:667883 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:109537849 (104.4 MiB)  TX bytes:109537849 (104.4 MiB)


> tc -s -d qdisc
 tc -s -d qdisc
qdisc mq 0: dev eth0 root
 Sent 873668 bytes 5267 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth1 root
 Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
requeues 17480)
 backlog 0b 0p requeues 17480
qdisc mq 0: dev eth2 root
 Sent 516248 bytes 983 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth3 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth4 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth5 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth6 root
 Sent 1022 bytes 13 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth7 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth8 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth9 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth10 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth11 root
 Sent 2448 bytes 40 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth12 root
 Sent 90 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth13 root
 Sent 90 bytes 1 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth14 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth15 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet June 8, 2012, 6:08 a.m. UTC | #7 
On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:

> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)

> qdisc mq 0: dev eth1 root
>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
> requeues 17480)

OK, and "tc -s -d cl show dev eth1"

(How many queues are really used)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 8, 2012, 8:14 a.m. UTC | #8 
2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
> On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
>
>> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
>>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 txqueuelen:1000
>>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
>
>> qdisc mq 0: dev eth1 root
>>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
>> requeues 17480)
>
> OK, and "tc -s -d cl show dev eth1"
>
> (How many queues are really used)
>
>
>

tc -s -d cl show dev eth1
class mq :1 root
 Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
 backlog 0b 0p requeues 2747
class mq :2 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :3 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :4 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :5 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :6 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :7 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class mq :8 root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet June 8, 2012, 8:22 a.m. UTC | #9 
On Fri, 2012-06-08 at 10:14 +0200, Jean-Michel Hautbois wrote:
> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
> > On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
> >
> >> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
> >>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
> >>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
> >>           collisions:0 txqueuelen:1000
> >>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
> >
> >> qdisc mq 0: dev eth1 root
> >>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
> >> requeues 17480)
> >
> > OK, and "tc -s -d cl show dev eth1"
> >
> > (How many queues are really used)
> >
> >
> >
> 
> tc -s -d cl show dev eth1
> class mq :1 root
>  Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
>  backlog 0b 0p requeues 2747
> class mq :2 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :3 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :4 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :5 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :6 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :7 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> class mq :8 root
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0


Do you have the same distribution on old kernels as well ?
(ie only queue 0 is used)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 8, 2012, 2:53 p.m. UTC | #10 
2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
> On Fri, 2012-06-08 at 10:14 +0200, Jean-Michel Hautbois wrote:
>> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
>> > On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
>> >
>> >> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
>> >>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
>> >>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> >>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
>> >>           collisions:0 txqueuelen:1000
>> >>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
>> >
>> >> qdisc mq 0: dev eth1 root
>> >>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
>> >> requeues 17480)
>> >
>> > OK, and "tc -s -d cl show dev eth1"
>> >
>> > (How many queues are really used)
>> >
>> >
>> >
>>
>> tc -s -d cl show dev eth1
>> class mq :1 root
>>  Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
>>  backlog 0b 0p requeues 2747
>> class mq :2 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :3 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :4 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :5 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :6 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :7 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>> class mq :8 root
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>
>
> Do you have the same distribution on old kernels as well ?
> (ie only queue 0 is used)
>
>
>

On the old kernel, there is nothing returned by this command.

JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 12, 2012, 8:24 a.m. UTC | #11 
2012/6/8 Jean-Michel Hautbois <jhautbois@gmail.com>:
> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
>> On Fri, 2012-06-08 at 10:14 +0200, Jean-Michel Hautbois wrote:
>>> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
>>> > On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
>>> >
>>> >> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
>>> >>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
>>> >>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>> >>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
>>> >>           collisions:0 txqueuelen:1000
>>> >>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
>>> >
>>> >> qdisc mq 0: dev eth1 root
>>> >>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
>>> >> requeues 17480)
>>> >
>>> > OK, and "tc -s -d cl show dev eth1"
>>> >
>>> > (How many queues are really used)
>>> >
>>> >
>>> >
>>>
>>> tc -s -d cl show dev eth1
>>> class mq :1 root
>>>  Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
>>>  backlog 0b 0p requeues 2747
>>> class mq :2 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :3 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :4 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :5 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :6 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :7 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>> class mq :8 root
>>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>>  backlog 0b 0p requeues 0
>>
>>
>> Do you have the same distribution on old kernels as well ?
>> (ie only queue 0 is used)
>>
>>
>>
>
> On the old kernel, there is nothing returned by this command.
>
> JM

I used perf in order to get more information.
Here is the perf record -a sleep 10 result (I took only kernel) :
     6.93%    ModuleTester  [kernel.kallsyms]                 [k]
copy_user_generic_string
     2.99%         swapper  [kernel.kallsyms]                 [k] mwait_idle
     2.60%          kipmi0  [ipmi_si]                         [k] port_inb
     1.75%         swapper  [kernel.kallsyms]                 [k] rb_prev
     1.63%    ModuleTester  [kernel.kallsyms]                 [k] _raw_spin_lock
     1.43%     NodeManager  [kernel.kallsyms]                 [k] delay_tsc
     0.90%    ModuleTester  [kernel.kallsyms]                 [k] clear_page_c
     0.88%    ModuleTester  [kernel.kallsyms]                 [k] dev_queue_xmit
     0.80%              ip  [kernel.kallsyms]                 [k]
snmp_fold_field
     0.73%    ModuleTester  [kernel.kallsyms]                 [k]
clflush_cache_range
     0.69%            grep  [kernel.kallsyms]                 [k] page_fault
     0.61%              sh  [kernel.kallsyms]                 [k] page_fault
     0.59%    ModuleTester  [kernel.kallsyms]                 [k] udp_sendmsg
     0.55%    ModuleTester  [kernel.kallsyms]                 [k] _raw_read_lock
     0.53%              sh  [kernel.kallsyms]                 [k] unmap_vmas
     0.52%    ModuleTester  [kernel.kallsyms]                 [k] rb_prev
     0.51%    ModuleTester  [kernel.kallsyms]                 [k]
find_busiest_group
     0.49%    ModuleTester  [kernel.kallsyms]                 [k] __ip_make_skb
     0.48%    ModuleTester  [kernel.kallsyms]                 [k]
sock_alloc_send_pskb
     0.48%    ModuleTester  libpthread-2.7.so                 [.]
pthread_mutex_lock
     0.47%    ModuleTester  [kernel.kallsyms]                 [k]
__netif_receive_skb
     0.44%              ip  [kernel.kallsyms]                 [k] find_next_bit
     0.43%         swapper  [kernel.kallsyms]                 [k]
clflush_cache_range
     0.41%              ps  [kernel.kallsyms]                 [k] format_decode
     0.41%    ModuleTester  [bonding]                         [k]
bond_start_xmit
     0.39%    ModuleTester  [be2net]                          [k] be_xmit
     0.39%    ModuleTester  [kernel.kallsyms]                 [k]
__ip_append_data
     0.38%    ModuleTester  [kernel.kallsyms]                 [k] netif_rx
     0.37%         swapper  [be2net]                          [k] be_poll
     0.37%         swapper  [kernel.kallsyms]                 [k] ktime_get
     0.37%              sh  [kernel.kallsyms]                 [k] copy_page_c
     0.36%         swapper  [kernel.kallsyms]                 [k]
irq_entries_start
     0.36%    ModuleTester  [kernel.kallsyms]                 [k]
__alloc_pages_nodemask
     0.35%    ModuleTester  [kernel.kallsyms]                 [k] __slab_free
     0.35%    ModuleTester  [kernel.kallsyms]                 [k] ip_mc_output
     0.34%    ModuleTester  [kernel.kallsyms]                 [k]
skb_release_data
     0.33%              ip  [kernel.kallsyms]                 [k] page_fault
     0.33%    ModuleTester  [kernel.kallsyms]                 [k] udp_send_skb

And here is the perf record -a result without bonding :
     2.49%     ModuleTester  [kernel.kallsyms]               [k]
csum_partial_copy_generic
     1.35%     ModuleTester  [kernel.kallsyms]               [k] _raw_spin_lock
     1.29%     ModuleTester  [kernel.kallsyms]               [k]
clflush_cache_range
     1.16%       jobprocess  [kernel.kallsyms]               [k] rb_prev
     1.01%       jobprocess  [kernel.kallsyms]               [k]
clflush_cache_range
     0.81%     ModuleTester  [be2net]                        [k] be_xmit
     0.78%       jobprocess  [kernel.kallsyms]               [k] __slab_free
     0.77%          swapper  [kernel.kallsyms]               [k] mwait_idle
     0.72%     ModuleTester  [kernel.kallsyms]               [k]
__domain_mapping
     0.66%       jobprocess  [kernel.kallsyms]               [k] _raw_spin_lock
     0.59%       jobprocess  [kernel.kallsyms]               [k]
_raw_spin_lock_irqsave
     0.56%     ModuleTester  [kernel.kallsyms]               [k] rb_prev
     0.53%          swapper  [kernel.kallsyms]               [k] rb_prev
     0.49%     ModuleTester  [kernel.kallsyms]               [k] sock_wmalloc
     0.47%       jobprocess  [be2net]                        [k] be_poll
     0.47%     ModuleTester  [kernel.kallsyms]               [k]
kmem_cache_alloc
     0.47%          swapper  [kernel.kallsyms]               [k]
clflush_cache_range
     0.45%           kipmi0  [ipmi_si]                       [k] port_inb
     0.42%          swapper  [kernel.kallsyms]               [k] __slab_free
     0.41%       jobprocess  [kernel.kallsyms]               [k] try_to_wake_up
     0.40%     ModuleTester  [kernel.kallsyms]               [k]
kmem_cache_alloc_node
     0.40%       jobprocess  [kernel.kallsyms]               [k] tg_load_down
     0.39%       jobprocess  libodyssey.so.1.8.2             [.]
y8_deblocking_luma_vert_edge_h264_sse2
     0.38%       jobprocess  libodyssey.so.1.8.2             [.]
y8_deblocking_luma_horz_edge_h264_ssse3
     0.38%     ModuleTester  [kernel.kallsyms]               [k] rb_insert_color
     0.37%       jobprocess  [kernel.kallsyms]               [k] find_iova
     0.37%       jobprocess  [kernel.kallsyms]               [k]
find_busiest_group
     0.36%       jobprocess  libpthread-2.7.so               [.]
pthread_mutex_lock
     0.35%          swapper  [kernel.kallsyms]               [k]
_raw_spin_unlock_irqrestore
     0.34%     ModuleTester  [kernel.kallsyms]               [k]
_raw_spin_lock_irqsave
     0.33%     ModuleTester  [kernel.kallsyms]               [k]
pfifo_fast_dequeue
     0.32%     ModuleTester  [kernel.kallsyms]               [k]
__kmalloc_node_track_caller
     0.32%       jobprocess  [be2net]                        [k]
be_tx_compl_process
     0.31%     ModuleTester  [kernel.kallsyms]               [k] ip_fragment
     0.29%          swapper  [kernel.kallsyms]               [k]
__hrtimer_start_range_ns
     0.29%       jobprocess  [kernel.kallsyms]               [k] __schedule
     0.29%     ModuleTester  [kernel.kallsyms]               [k] dev_queue_xmit
     0.28%          swapper  [kernel.kallsyms]               [k] __schedule

First thing I notice is the difference in copy_user_generic_string (it
is only 0.11% on the second measure, I didn't reported it here).
I think perf can help in finding the issue I observe with bonding,
maybe do you have  suggestions on the parameters to use ?
FYI, with bonding, TX goes up to 640Mbps, without bonding, I can send
2.4Gbps without suffering...

JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet June 12, 2012, 8:55 a.m. UTC | #12 
On Tue, 2012-06-12 at 10:24 +0200, Jean-Michel Hautbois wrote:
> 2012/6/8 Jean-Michel Hautbois <jhautbois@gmail.com>:
> > 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
> >> On Fri, 2012-06-08 at 10:14 +0200, Jean-Michel Hautbois wrote:
> >>> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
> >>> > On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
> >>> >
> >>> >> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
> >>> >>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
> >>> >>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >>> >>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
> >>> >>           collisions:0 txqueuelen:1000
> >>> >>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
> >>> >
> >>> >> qdisc mq 0: dev eth1 root
> >>> >>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
> >>> >> requeues 17480)
> >>> >
> >>> > OK, and "tc -s -d cl show dev eth1"
> >>> >
> >>> > (How many queues are really used)
> >>> >
> >>> >
> >>> >
> >>>
> >>> tc -s -d cl show dev eth1
> >>> class mq :1 root
> >>>  Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
> >>>  backlog 0b 0p requeues 2747
> >>> class mq :2 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :3 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :4 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :5 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :6 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :7 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>> class mq :8 root
> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> >>>  backlog 0b 0p requeues 0
> >>
> >>
> >> Do you have the same distribution on old kernels as well ?
> >> (ie only queue 0 is used)
> >>
> >>
> >>
> >
> > On the old kernel, there is nothing returned by this command.
> >
> > JM
> 
> I used perf in order to get more information.

What happens if you force some traffic in the other way (say 50.000
(small) packets per second in RX) while doing your tests ?




--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 12, 2012, 9:01 a.m. UTC | #13 
2012/6/12 Eric Dumazet <eric.dumazet@gmail.com>:
> On Tue, 2012-06-12 at 10:24 +0200, Jean-Michel Hautbois wrote:
>> 2012/6/8 Jean-Michel Hautbois <jhautbois@gmail.com>:
>> > 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
>> >> On Fri, 2012-06-08 at 10:14 +0200, Jean-Michel Hautbois wrote:
>> >>> 2012/6/8 Eric Dumazet <eric.dumazet@gmail.com>:
>> >>> > On Thu, 2012-06-07 at 14:54 +0200, Jean-Michel Hautbois wrote:
>> >>> >
>> >>> >> eth1      Link encap:Ethernet  HWaddr 68:b5:99:b9:8d:d4
>> >>> >>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:4096  Metric:1
>> >>> >>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>> >>> >>           TX packets:15215387 errors:0 dropped:0 overruns:0 carrier:0
>> >>> >>           collisions:0 txqueuelen:1000
>> >>> >>           RX bytes:0 (0.0 B)  TX bytes:61476524359 (57.2 GiB)
>> >>> >
>> >>> >> qdisc mq 0: dev eth1 root
>> >>> >>  Sent 61476524359 bytes 15215387 pkt (dropped 45683472, overlimits 0
>> >>> >> requeues 17480)
>> >>> >
>> >>> > OK, and "tc -s -d cl show dev eth1"
>> >>> >
>> >>> > (How many queues are really used)
>> >>> >
>> >>> >
>> >>> >
>> >>>
>> >>> tc -s -d cl show dev eth1
>> >>> class mq :1 root
>> >>>  Sent 9798071746 bytes 2425410 pkt (dropped 3442405, overlimits 0 requeues 2747)
>> >>>  backlog 0b 0p requeues 2747
>> >>> class mq :2 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :3 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :4 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :5 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :6 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :7 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>> class mq :8 root
>> >>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> >>>  backlog 0b 0p requeues 0
>> >>
>> >>
>> >> Do you have the same distribution on old kernels as well ?
>> >> (ie only queue 0 is used)
>> >>
>> >>
>> >>
>> >
>> > On the old kernel, there is nothing returned by this command.
>> >
>> > JM
>>
>> I used perf in order to get more information.
>
> What happens if you force some traffic in the other way (say 50.000
> (small) packets per second in RX) while doing your tests ?
>

Can I do that using netperf ?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet June 12, 2012, 9:06 a.m. UTC | #14 
On Tue, 2012-06-12 at 11:01 +0200, Jean-Michel Hautbois wrote:

> Can I do that using netperf ?


Sure, you could use netperf -t UDP_RR


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jean-Michel Hautbois June 12, 2012, 9:10 a.m. UTC | #15 
2012/6/12 Eric Dumazet <eric.dumazet@gmail.com>:
> On Tue, 2012-06-12 at 11:01 +0200, Jean-Michel Hautbois wrote:
>
>> Can I do that using netperf ?
>
>
> Sure, you could use netperf -t UDP_RR
>
>

It sends, but no change on TX...
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
index c5c4c0e..1e8f8a6 100644
--- a/drivers/net/ethernet/emulex/benet/be.h
+++ b/drivers/net/ethernet/emulex/benet/be.h
@@ -105,7 +105,7 @@  static inline char *nic_name(struct pci_dev *pdev)
 #define MAX_TX_QS		8
 #define MAX_ROCE_EQS		5
 #define MAX_MSIX_VECTORS	(MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
-#define BE_TX_BUDGET		256
+#define BE_TX_BUDGET		65535
 #define BE_NAPI_WEIGHT		64
 #define MAX_RX_POST		BE_NAPI_WEIGHT /* Frags posted at a time */
 #define RX_FRAGS_REFILL_WM	(RX_Q_LEN - MAX_RX_POST)