myr10ge: again fix lro_gen_skb() alignment

On Wed, Apr 22, 2009 at 11:37:24AM -0400, Andrew Gallatin wrote:
>
> I booted the sender into a kernel.org 2.6.18.2 so as to try to have  
> results as close to yours as possible (I was running 2.6.22 on the
> sender before).

OK I've got my hands on a myricom card.  I've tested it using the
same 2.6.18 sender that I used against the eariler cxgb3 test.
I wasn't able to discern any significant deviations between LRO
and GRO.

Unfortunately it seems that this machine is a little too fast
so even with the IRQ bound to a single CPU it's way overspeced
for 10GbE:

    Idle at 10Gb IRQ rate	soaker IRQ rate soaker throuput
GRO 43-45	 14700		13300		7933
LRO 43-45	 14700		13300		7943

But even with the soaker running they seem to be neck and neck.

Here's the patch I used BTW.  I got the checksums to work by
just setting skb->csum.


Cheers,

Herbert Xu wrote:
> On Wed, Apr 22, 2009 at 11:37:24AM -0400, Andrew Gallatin wrote:
>> I booted the sender into a kernel.org 2.6.18.2 so as to try to have  
>> results as close to yours as possible (I was running 2.6.22 on the
>> sender before).
> 
> OK I've got my hands on a myricom card.  I've tested it using the
> same 2.6.18 sender that I used against the eariler cxgb3 test.
> I wasn't able to discern any significant deviations between LRO
> and GRO.
> 
> Unfortunately it seems that this machine is a little too fast
> so even with the IRQ bound to a single CPU it's way overspeced
> for 10GbE:
> 
>     Idle at 10Gb IRQ rate	soaker IRQ rate soaker throuput
> GRO 43-45	 14700		13300		7933
> LRO 43-45	 14700		13300		7943
> 
> But even with the soaker running they seem to be neck and neck.

This is strange.  I wonder if it might be a cache footprint issue?
My intentionally weak receiver is an athlon64 x2 "Toledo", and
has only 512KB L2 cache.  I can re-test with a core-2 based Xeon.

But can you describe your setup in more detail?  What CPU does the
receiver have? You say the sender is running 2.6.18.  Is this
a RHEL5 kernel, or a kernel.org kernel?

> Here's the patch I used BTW.  I got the checksums to work by
> just setting skb->csum.

Yes, sorry about that stupidity.

Drew
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Apr 24, 2009 at 08:45:28AM -0400, Andrew Gallatin wrote:
>
> This is strange.  I wonder if it might be a cache footprint issue?
> My intentionally weak receiver is an athlon64 x2 "Toledo", and
> has only 512KB L2 cache.  I can re-test with a core-2 based Xeon.
>
> But can you describe your setup in more detail?  What CPU does the
> receiver have? You say the sender is running 2.6.18.  Is this
> a RHEL5 kernel, or a kernel.org kernel?

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5440  @ 2.83GHz
stepping        : 6
cpu MHz         : 2833.155
cache size      : 6144 KB
physical id     : 1
siblings        : 1
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips        : 5665.85
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

It's a RHEL-5 kernel, -128 to be exact.  But I rebooted into
2.6.30-rc1 and it made no difference.  The previous observed
problem with 30-rc1 seems to only affect cxgb3.

Cheers,

Herbert Xu wrote:
> On Wed, Apr 22, 2009 at 11:37:24AM -0400, Andrew Gallatin wrote:
>> I booted the sender into a kernel.org 2.6.18.2 so as to try to have  
>> results as close to yours as possible (I was running 2.6.22 on the
>> sender before).
> 
> OK I've got my hands on a myricom card.  I've tested it using the
> same 2.6.18 sender that I used against the eariler cxgb3 test.
> I wasn't able to discern any significant deviations between LRO
> and GRO.
> 
> Unfortunately it seems that this machine is a little too fast
> so even with the IRQ bound to a single CPU it's way overspeced
> for 10GbE:
> 
>     Idle at 10Gb IRQ rate	soaker IRQ rate soaker throuput
> GRO 43-45	 14700		13300		7933
> LRO 43-45	 14700		13300		7943
> 
> But even with the soaker running they seem to be neck and neck.

 From what I can tell,  CPU utilization is only broken when a CPU is
otherwise idle, so it should be accurate when you bind the IRQ and the
netserver to the same CPU.   Here are results from an older, slower
core-2 Xeon with a 4MB L2 cache:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5150  @ 2.66GHz
stepping        : 6
cpu MHz         : 2659.916
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow
bogomips        : 5319.83
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

The Xeon was running net-next, had DCA enabled, ioatdma disabled for
TCP (CONFIG_NET_DMA is not set).  The sender was the weak athlon64,
running 2.6.22.

LRO, no soaker: (13,200 intrs/sec)
Recv   Send    Send                          Utilization       Service 
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local 
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB
  87380  65536  65536    60.02      9469.74   17.44    13.31    0.302 
0.461

LRO, soaker: (6,500 intrs/sec)

  87380  65536  65536    60.06      3955.74   7.11     25.02    0.294 
2.072

GRO, no soaker, (13,200 intrs/sec)
  87380  65536  65536    60.02      9467.90   16.76    14.16    0.290 
0.490

GRO, soaker: (6,500 intrs/sec)
  87380  65536  65536    60.02      3774.88   6.20     25.01    0.269 
2.171


These results are indeed quite close, so the performance problem seems
isolated to AMD CPUS, and perhaps due to the smaller caches.
Do you have any AMD you can use as a receiver?

Note that the GRO results were still obtained by (bogusly) setting
CHECKSUM_UNNECESSARY.  I tried to use your patch, and I see
terrible performance. Netperf shows between 1Gb/s to 2Gb/s (compared
to 5Gb/s with GRO disabled).  I don't see bad checksums in netstat
on the receiver, but it *feels* like something like that.

Here's a diff of netstat -st taken on the sender before and after
a 5 second netperf:
2c2
<     157 active connections openings
---
 >     159 active connections openings
7,9c7,9
<     31465934 segments received
<     72887021 segments send out
<     679 segments retransmited
---
 >     32184827 segments received
 >     73473546 segments send out
 >     698 segments retransmited
16c16
<     4596 packets directly queued to recvmsg prequeue.
---
 >     4603 packets directly queued to recvmsg prequeue.
18,21c18,21
<     15928 packets header predicted
<     18100148 acknowledgments not containing data received
<     13351873 predicted acknowledgments
<     343 times recovered from packet loss due to SACK data
---
 >     15930 packets header predicted
 >     18464095 acknowledgments not containing data received
 >     13706813 predicted acknowledgments
 >     365 times recovered from packet loss due to SACK data
23,25c23,25
<     53 congestion windows fully recovered
<     221 congestion windows partially recovered using Hoe heuristic
<     TCPDSACKUndo: 268
---
 >     60 congestion windows fully recovered
 >     228 congestion windows partially recovered using Hoe heuristic
 >     TCPDSACKUndo: 281
27,28c27,28
<     584 fast retransmits
<     93 forward retransmits
---
 >     597 fast retransmits
 >     99 forward retransmits
30c30
<     674 DSACKs received
---
 >     693 DSACKs received

And on the receiver (whose netstat is confused, and cannot read ext 
stats in a net-next kernel):
diff /tmp/a /tmp/b
3c3
<     12 passive connection openings
---
 >     14 passive connection openings
7,8c7,8
<     3776478 segments received
<     3775846 segments send out
---
 >     4495385 segments received
 >     4494747 segments send out



This was using a net-next pulled 1/2 hour ago.  The only patch was your 
GRO patch applied to myri10ge.  Do you have some other local patch
which might be helping you?

Drew
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Apr 24, 2009 at 12:16:08PM -0400, Andrew Gallatin wrote:
>
> Note that the GRO results were still obtained by (bogusly) setting
> CHECKSUM_UNNECESSARY.  I tried to use your patch, and I see
> terrible performance. Netperf shows between 1Gb/s to 2Gb/s (compared
> to 5Gb/s with GRO disabled).  I don't see bad checksums in netstat
> on the receiver, but it *feels* like something like that.

Well if the hardware checksum ends up being wrong then we'll always
fall back to using software checksums.  So somehow I doubt it's
causing what you're seeing.

> Here's a diff of netstat -st taken on the sender before and after
> a 5 second netperf:
> 2c2
> <     157 active connections openings
> ---
> >     159 active connections openings
> 7,9c7,9
> <     31465934 segments received
> <     72887021 segments send out
> <     679 segments retransmited
> ---
> >     32184827 segments received
> >     73473546 segments send out
> >     698 segments retransmited

So you're losing packets.  This is indeed something that I didn't
see here at all.  I'll see if I can get the card moved to an AMD
machine.

> This was using a net-next pulled 1/2 hour ago.  The only patch was your  
> GRO patch applied to myri10ge.  Do you have some other local patch
> which might be helping you?

I was using Linus's tree + the GRO patches from net-next.  I do
have two further optimisation patches applied but they don't
actually make much difference (I made them while trying to figure
out why cxgb3's GRO became slow again, which turned out to be sender
related).

Cheers,

On Sat, Apr 25, 2009 at 12:30:09AM +0800, Herbert Xu wrote:
>
> > This was using a net-next pulled 1/2 hour ago.  The only patch was your  
> > GRO patch applied to myri10ge.  Do you have some other local patch
> > which might be helping you?
> 
> I was using Linus's tree + the GRO patches from net-next.  I do
> have two further optimisation patches applied but they don't
> actually make much difference (I made them while trying to figure
> out why cxgb3's GRO became slow again, which turned out to be sender
> related).

I'll also retest using net-next.

Thanks,

> This is strange.  I wonder if it might be a cache footprint issue?
> My intentionally weak receiver is an athlon64 x2 "Toledo", and
> has only 512KB L2 cache.  I can re-test with a core-2 based Xeon.

A point about netperf :)  By default, it will use one more buffer than the 
initial size of the socket buffer divided by the send/recv buffer size - this 
goes back to days of copy-avoidance, a flavor of which can be found in reading:

ftp://ftp.cup.hp.com/dist/networking/briefs/copyavoid.pdf

particularly section 3.2.  It can be overridden with the global -W option:

     -W send,recv      Set the number of send,recv buffers

Of course, this will interact with other things such as:

The default send/recv size will be the send/recv socket buffer size. That can be 
overridden with the test-specific -m/-M options.

The default socket buffer size will be whatever the system gives it.  That can be 
overridden with the test-specific -s/-S options.

So, the various options can have a non-trivial effect on the cache footprint of 
the data netperf is shoving around.

rick jones
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html