diff mbox series

net/mlx5e: Set EQE based as default TX interrupt moderation mode

Message ID 1523529499-3326-1-git-send-email-talatb@mellanox.com
State New
Headers show
Series net/mlx5e: Set EQE based as default TX interrupt moderation mode | expand

Commit Message

Talat Batheesh April 12, 2018, 10:38 a.m. UTC 
From: Tal Gilboa <talgi@mellanox.com>

BugLink: http://bugs.launchpad.net/bugs/1763325

The default TX moderation mode was mistakenly set to CQE based. The
intention was to add a control ability in order to improve some specific
use-cases. In general, we prefer to use EQE based moderation as it gives
much better numbers for the common cases.

CQE based causes a degradation in the common case since it resets the
moderation timer on CQE generation. This causes an issue when TSO is
well utilized (large TSO sessions). The timer is set to 16us so traffic
of ~64KB TSO sessions per second would mean timer reset (CQE per TSO
session -> long time between CQEs). In this case we quickly reach the
tcp_limit_output_bytes (256KB by default) and cause a halt in TX traffic.

By setting EQE based moderation we make sure timer would expire after
16us regardless of the packet rate.
This fixes an up to 40% packet rate and up to 23% bandwidth degradtions.

Fixes: 0088cbbc4b66 ("net/mlx5e: Enable CQE based moderation on TX CQ")
Signed-off-by: Tal Gilboa <talgi@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
(cherry picked from commit 48bfc39791b8b4a25f165e711f18b9c1617cefbc)
Signed-off-by: Talat Batheesh <talatb@mellanox.com>

 Conflicts:
	drivers/net/ethernet/mellanox/mlx5/core/en_main.c

Was conflict sine the mlx5e_build_nic_params changes over 4.16 kernel.
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Comments

Kleber Sacilotto de Souza April 12, 2018, 1:39 p.m. UTC | #1 
On 04/12/18 12:38, Talat Batheesh wrote:
> From: Tal Gilboa <talgi@mellanox.com>
> 
> BugLink: http://bugs.launchpad.net/bugs/1763325
> 
> The default TX moderation mode was mistakenly set to CQE based. The
> intention was to add a control ability in order to improve some specific
> use-cases. In general, we prefer to use EQE based moderation as it gives
> much better numbers for the common cases.
> 
> CQE based causes a degradation in the common case since it resets the
> moderation timer on CQE generation. This causes an issue when TSO is
> well utilized (large TSO sessions). The timer is set to 16us so traffic
> of ~64KB TSO sessions per second would mean timer reset (CQE per TSO
> session -> long time between CQEs). In this case we quickly reach the
> tcp_limit_output_bytes (256KB by default) and cause a halt in TX traffic.
> 
> By setting EQE based moderation we make sure timer would expire after
> 16us regardless of the packet rate.
> This fixes an up to 40% packet rate and up to 23% bandwidth degradtions.
> 
> Fixes: 0088cbbc4b66 ("net/mlx5e: Enable CQE based moderation on TX CQ")
> Signed-off-by: Tal Gilboa <talgi@mellanox.com>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> (cherry picked from commit 48bfc39791b8b4a25f165e711f18b9c1617cefbc)
> Signed-off-by: Talat Batheesh <talatb@mellanox.com>
> 
>  Conflicts:
> 	drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> 
> Was conflict sine the mlx5e_build_nic_params changes over 4.16 kernel.
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> index 0d352d4..33faac2 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> @@ -4075,7 +4075,7 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
>  			    struct mlx5e_params *params,
>  			    u16 max_channels)
>  {
> -	u8 cq_period_mode = 0;
> +	u8 rx_cq_period_mode;
>  	u32 link_speed = 0;
>  	u32 pci_bw = 0;
>  
> @@ -4111,12 +4111,12 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
>  	params->lro_timeout = mlx5e_choose_lro_timeout(mdev, MLX5E_DEFAULT_LRO_TIMEOUT);
>  
>  	/* CQ moderation params */
> -	cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
> +	rx_cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
>  			MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
>  			MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
>  	params->rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
> -	mlx5e_set_rx_cq_mode_params(params, cq_period_mode);
> -	mlx5e_set_tx_cq_mode_params(params, cq_period_mode);
> +	mlx5e_set_rx_cq_mode_params(params, rx_cq_period_mode);
> +	mlx5e_set_tx_cq_mode_params(params, MLX5_CQ_PERIOD_MODE_START_FROM_EQE);
>  
>  	/* TX inline */
>  	params->tx_max_inline = mlx5e_get_max_inline_cap(mdev);
> 

Note: this patch is targeted for BIONIC.

Kleber
Seth Forshee April 12, 2018, 3:51 p.m. UTC | #2 
On Thu, Apr 12, 2018 at 01:38:19PM +0300, Talat Batheesh wrote:
> From: Tal Gilboa <talgi@mellanox.com>
> 
> BugLink: http://bugs.launchpad.net/bugs/1763325
> 
> The default TX moderation mode was mistakenly set to CQE based. The
> intention was to add a control ability in order to improve some specific
> use-cases. In general, we prefer to use EQE based moderation as it gives
> much better numbers for the common cases.
> 
> CQE based causes a degradation in the common case since it resets the
> moderation timer on CQE generation. This causes an issue when TSO is
> well utilized (large TSO sessions). The timer is set to 16us so traffic
> of ~64KB TSO sessions per second would mean timer reset (CQE per TSO
> session -> long time between CQEs). In this case we quickly reach the
> tcp_limit_output_bytes (256KB by default) and cause a halt in TX traffic.
> 
> By setting EQE based moderation we make sure timer would expire after
> 16us regardless of the packet rate.
> This fixes an up to 40% packet rate and up to 23% bandwidth degradtions.
> 
> Fixes: 0088cbbc4b66 ("net/mlx5e: Enable CQE based moderation on TX CQ")
> Signed-off-by: Tal Gilboa <talgi@mellanox.com>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> (cherry picked from commit 48bfc39791b8b4a25f165e711f18b9c1617cefbc)
> Signed-off-by: Talat Batheesh <talatb@mellanox.com>
> 
>  Conflicts:
> 	drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> 
> Was conflict sine the mlx5e_build_nic_params changes over 4.16 kernel.

This patch has already been applied to bionic throught v4.15.17 stable
update. Thanks!
diff mbox series

Patch

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 0d352d4..33faac2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -4075,7 +4075,7 @@  void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
 			    struct mlx5e_params *params,
 			    u16 max_channels)
 {
-	u8 cq_period_mode = 0;
+	u8 rx_cq_period_mode;
 	u32 link_speed = 0;
 	u32 pci_bw = 0;
 
@@ -4111,12 +4111,12 @@  void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
 	params->lro_timeout = mlx5e_choose_lro_timeout(mdev, MLX5E_DEFAULT_LRO_TIMEOUT);
 
 	/* CQ moderation params */
-	cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
+	rx_cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
 			MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
 			MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
 	params->rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
-	mlx5e_set_rx_cq_mode_params(params, cq_period_mode);
-	mlx5e_set_tx_cq_mode_params(params, cq_period_mode);
+	mlx5e_set_rx_cq_mode_params(params, rx_cq_period_mode);
+	mlx5e_set_tx_cq_mode_params(params, MLX5_CQ_PERIOD_MODE_START_FROM_EQE);
 
 	/* TX inline */
 	params->tx_max_inline = mlx5e_get_max_inline_cap(mdev);