[net-next] i40e: limit lan queue count in large cpu count machine

Message ID 1500945462-33176-1-git-send-email-shannon.nelson@oracle.com
State Accepted
Delegated to: Jeff Kirsher
Headers show

Commit Message

Shannon Nelson July 25, 2017, 1:17 a.m.
When a machine has more cpus than queuepairs, e.g. 512 cores, the
counting gets a little funky and turns off Flow Director with the
message:
  not enough queues for Flow Director. Flow Director feature is disabled

This patch limits the number of lan queues initially allocated to
be sure we have some left for FD and other features.

Signed-off-by: Shannon Nelson <shannon.nelson@oracle.com>
---
 drivers/net/ethernet/intel/i40e/i40e_main.c |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)

Comments

Bowers, AndrewX Aug. 9, 2017, 5:46 p.m. | #1
> -----Original Message-----
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Shannon Nelson
> Sent: Monday, July 24, 2017 6:18 PM
> To: davem@davemloft.net; netdev@vger.kernel.org
> Cc: intel-wired-lan@lists.osuosl.org
> Subject: [Intel-wired-lan] [PATCH net-next] i40e: limit lan queue count in
> large cpu count machine
> 
> When a machine has more cpus than queuepairs, e.g. 512 cores, the counting
> gets a little funky and turns off Flow Director with the
> message:
>   not enough queues for Flow Director. Flow Director feature is disabled
> 
> This patch limits the number of lan queues initially allocated to be sure we
> have some left for FD and other features.
> 
> Signed-off-by: Shannon Nelson <shannon.nelson@oracle.com>
> ---
>  drivers/net/ethernet/intel/i40e/i40e_main.c |   11 +++++++----
>  1 files changed, 7 insertions(+), 4 deletions(-)

Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
Shannon Nelson Sept. 5, 2017, 8:31 p.m. | #2
Hey Jeff,

Any clues on when this patch, still sitting at "Under Review", will get 
moved along?

sln

On 7/24/2017 6:17 PM, Shannon Nelson wrote:
> When a machine has more cpus than queuepairs, e.g. 512 cores, the
> counting gets a little funky and turns off Flow Director with the
> message:
>    not enough queues for Flow Director. Flow Director feature is disabled
> 
> This patch limits the number of lan queues initially allocated to
> be sure we have some left for FD and other features.
> 
> Signed-off-by: Shannon Nelson <shannon.nelson@oracle.com>
> ---
>   drivers/net/ethernet/intel/i40e/i40e_main.c |   11 +++++++----
>   1 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
> index 2db93d3..a694e99 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> @@ -11059,6 +11059,7 @@ static int i40e_setup_pf_switch(struct i40e_pf *pf, bool reinit)
>   static void i40e_determine_queue_usage(struct i40e_pf *pf)
>   {
>   	int queues_left;
> +	int q_max;
>   
>   	pf->num_lan_qps = 0;
>   
> @@ -11105,10 +11106,12 @@ static void i40e_determine_queue_usage(struct i40e_pf *pf)
>   					I40E_FLAG_DCB_ENABLED);
>   			dev_info(&pf->pdev->dev, "not enough queues for DCB. DCB is disabled.\n");
>   		}
> -		pf->num_lan_qps = max_t(int, pf->rss_size_max,
> -					num_online_cpus());
> -		pf->num_lan_qps = min_t(int, pf->num_lan_qps,
> -					pf->hw.func_caps.num_tx_qp);
> +
> +		/* limit lan qps to the smaller of qps, cpus or msix */
> +		q_max = max_t(int, pf->rss_size_max, num_online_cpus());
> +		q_max = min_t(int, q_max, pf->hw.func_caps.num_tx_qp);
> +		q_max = min_t(int, q_max, pf->hw.func_caps.num_msix_vectors);
> +		pf->num_lan_qps = q_max;
>   
>   		queues_left -= pf->num_lan_qps;
>   	}
>
Jeff Kirsher Sept. 5, 2017, 9:09 p.m. | #3
On Tue, 2017-09-05 at 13:31 -0700, Shannon Nelson wrote:
> Hey Jeff,
> 
> Any clues on when this patch, still sitting at "Under Review", will
> get 
> moved along?

I have 15 patches ready to push to Dave, and your patch is literally
the next in line.  But the window is closing, if not closed already.  I
could try and push a series of 16 patches, but I do not want to push my
luck with Dave especially that this would be a pull request right at
the deadline of net-next closing.

Your patch does "fix" the driver, so I have no problem pushing it to
net (4.14) once Dave and Linus get done syncing their trees this week.

> 
> On 7/24/2017 6:17 PM, Shannon Nelson wrote:
> > When a machine has more cpus than queuepairs, e.g. 512 cores, the
> > counting gets a little funky and turns off Flow Director with the
> > message:
> >    not enough queues for Flow Director. Flow Director feature is
> > disabled
> > 
> > This patch limits the number of lan queues initially allocated to
> > be sure we have some left for FD and other features.
> > 
> > Signed-off-by: Shannon Nelson <shannon.nelson@oracle.com>
> > ---
> >   drivers/net/ethernet/intel/i40e/i40e_main.c |   11 +++++++----
> >   1 files changed, 7 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c
> > b/drivers/net/ethernet/intel/i40e/i40e_main.c
> > index 2db93d3..a694e99 100644
> > --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> > +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> > @@ -11059,6 +11059,7 @@ static int i40e_setup_pf_switch(struct
> > i40e_pf *pf, bool reinit)
> >   static void i40e_determine_queue_usage(struct i40e_pf *pf)
> >   {
> >   	int queues_left;
> > +	int q_max;
> >   
> >   	pf->num_lan_qps = 0;
> >   
> > @@ -11105,10 +11106,12 @@ static void
> > i40e_determine_queue_usage(struct i40e_pf *pf)
> >   					I40E_FLAG_DCB_ENABLED);
> >   			dev_info(&pf->pdev->dev, "not enough
> > queues for DCB. DCB is disabled.\n");
> >   		}
> > -		pf->num_lan_qps = max_t(int, pf->rss_size_max,
> > -					num_online_cpus());
> > -		pf->num_lan_qps = min_t(int, pf->num_lan_qps,
> > -					pf-
> > >hw.func_caps.num_tx_qp);
> > +
> > +		/* limit lan qps to the smaller of qps, cpus or
> > msix */
> > +		q_max = max_t(int, pf->rss_size_max,
> > num_online_cpus());
> > +		q_max = min_t(int, q_max, pf-
> > >hw.func_caps.num_tx_qp);
> > +		q_max = min_t(int, q_max, pf-
> > >hw.func_caps.num_msix_vectors);
> > +		pf->num_lan_qps = q_max;
> >   
> >   		queues_left -= pf->num_lan_qps;
> >   	}
> > 
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan@osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Patch

diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
index 2db93d3..a694e99 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_main.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
@@ -11059,6 +11059,7 @@  static int i40e_setup_pf_switch(struct i40e_pf *pf, bool reinit)
 static void i40e_determine_queue_usage(struct i40e_pf *pf)
 {
 	int queues_left;
+	int q_max;
 
 	pf->num_lan_qps = 0;
 
@@ -11105,10 +11106,12 @@  static void i40e_determine_queue_usage(struct i40e_pf *pf)
 					I40E_FLAG_DCB_ENABLED);
 			dev_info(&pf->pdev->dev, "not enough queues for DCB. DCB is disabled.\n");
 		}
-		pf->num_lan_qps = max_t(int, pf->rss_size_max,
-					num_online_cpus());
-		pf->num_lan_qps = min_t(int, pf->num_lan_qps,
-					pf->hw.func_caps.num_tx_qp);
+
+		/* limit lan qps to the smaller of qps, cpus or msix */
+		q_max = max_t(int, pf->rss_size_max, num_online_cpus());
+		q_max = min_t(int, q_max, pf->hw.func_caps.num_tx_qp);
+		q_max = min_t(int, q_max, pf->hw.func_caps.num_msix_vectors);
+		pf->num_lan_qps = q_max;
 
 		queues_left -= pf->num_lan_qps;
 	}