[net,2/2] ixgbe: Limit use of 2K buffers on architectures with 256B or larger cache lines

From: Alexander Duyck <alexander.h.duyck@intel.com>

On architectures that have a cache line size larger than 64 Bytes we start
running into issues where the amount of headroom for the frame starts
shrinking.

The size of skb_shared_info on a system with a 64B L1 cache line size is
320.  This increases to 384 with a 128B cache line, and 512 with a 256B
cache line.

In addition the NET_SKB_PAD value increases as well consistent with the
cache line size.  As a result when we get to a 256B cache line as seen on
the s390 we end up 768 bytes used by padding and shared info leaving us
with only 1280 bytes to use for data storage.  On architectures such as
this we should default to using 3K Rx buffers out of a 8K page instead of
trying to do 1.5K buffers out of a 4K page.

To take all of this into account I have added one small check so that we
compare the max_frame to the amount of actual data we can store.  This was
already occurring for igb, but I had overlooked it for ixgbe as it doesn't
have strict limits for 82599 once we enable jumbo frames.  By adding this
check we will automatically enable 3K Rx buffers as soon as the maximum
frame size we can handle drops below the standard Ethernet MTU.

I also went through and fixed one small typo that I found where I had left
an IGB in a variable name due to a copy/paste error.

Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h      | 2 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

From: Jeff Kirsher
> Sent: 03 March 2017 02:25
> From: Alexander Duyck <alexander.h.duyck@intel.com>
> 
> On architectures that have a cache line size larger than 64 Bytes we start
> running into issues where the amount of headroom for the frame starts
> shrinking.
> 
> The size of skb_shared_info on a system with a 64B L1 cache line size is
> 320.  This increases to 384 with a 128B cache line, and 512 with a 256B
> cache line.

Perhaps some of the CACHE_LINE_ALIGNED markers don't actually need
to force alignment with large line sizes?

I realise some things have hard requirements for cache alignment
(eg non-coherent dma), but others are just there to limit the number
of cache lines read and/or dirtied.

	David

> -----Original Message-----
> From: David Laight [mailto:David.Laight@ACULAB.COM]
> Sent: Friday, March 3, 2017 4:25 AM
> To: Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>; davem@davemloft.net
> Cc: Duyck, Alexander H <alexander.h.duyck@intel.com>;
> netdev@vger.kernel.org; nhorman@redhat.com; sassmann@redhat.com;
> jogreene@redhat.com
> Subject: RE: [net 2/2] ixgbe: Limit use of 2K buffers on architectures with 256B
> or larger cache lines
> 
> From: Jeff Kirsher
> > Sent: 03 March 2017 02:25
> > From: Alexander Duyck <alexander.h.duyck@intel.com>
> >
> > On architectures that have a cache line size larger than 64 Bytes we
> > start running into issues where the amount of headroom for the frame
> > starts shrinking.
> >
> > The size of skb_shared_info on a system with a 64B L1 cache line size
> > is 320.  This increases to 384 with a 128B cache line, and 512 with a
> > 256B cache line.
> 
> Perhaps some of the CACHE_LINE_ALIGNED markers don't actually need to
> force alignment with large line sizes?
> 
> I realise some things have hard requirements for cache alignment (eg non-
> coherent dma), but others are just there to limit the number of cache lines read
> and/or dirtied.
> 
> 	David

For our purposes I think this works well enough.  Basically we wanted to guarantee we have enough headroom for XDP.  In the case of the Mellanox drivers they are guaranteeing 256 if I recall correctly.

I have some follow-up patches for net-next that will make it so that we can just do a build-time test that will determine the padding size and allow us to always guaranteed at least NET_SKB_PAD + NET_IP_ALIGN.

- Alex