diff mbox

[net-next] mlx4: change TX coalescing defaults

Message ID 1352168442.3140.32.camel@edumazet-glaptop
State Accepted, archived
Delegated to: David Miller
Headers show

Commit Message

Eric Dumazet Nov. 6, 2012, 2:20 a.m. UTC 
From: Eric Dumazet <edumazet@google.com>

mlx4 currently uses a too high tx coalescing setting, deferring
TX completion interrupts by up to 128 us.

With the recent skb_orphan() removal in commit 8112ec3b872,
performance of a single TCP flow is capped to ~4 Gbps, unless
we increase tcp_limit_output_bytes.

I suggest using 16 us instead of 128 us, allowing a finer control.

Performance of a single TCP flow is restored to previous levels,
while keeping TCP small queues fully enabled with default sysctl.

This patch is also a BQL prereq.

Reported-by: Vimalkumar <j.vimal@gmail.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yevgeny Petrilin <yevgenyp@mellanox.com>
Cc: Or Gerlitz <ogerlitz@mellanox.com>
---
By the way, is BQL addition planned for this driver ?

 drivers/net/ethernet/mellanox/mlx4/en_netdev.c |    2 +-
 drivers/net/ethernet/mellanox/mlx4/mlx4_en.h   |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Amir Mellanox Nov. 6, 2012, 8:04 a.m. UTC | #1 
Eric Hi,

I will check and review your patch and return to you ASAP.

Regarding BQL, the driver has it over half a year now, In commit:
5b263f5374ea70daa61ed9f5cc5d8f4ac236efbd

Amir.

On Tue, Nov 6, 2012 at 4:20 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> From: Eric Dumazet <edumazet@google.com>
>
> mlx4 currently uses a too high tx coalescing setting, deferring
> TX completion interrupts by up to 128 us.
>
> With the recent skb_orphan() removal in commit 8112ec3b872,
> performance of a single TCP flow is capped to ~4 Gbps, unless
> we increase tcp_limit_output_bytes.
>
> I suggest using 16 us instead of 128 us, allowing a finer control.
>
> Performance of a single TCP flow is restored to previous levels,
> while keeping TCP small queues fully enabled with default sysctl.
>
> This patch is also a BQL prereq.
>
> Reported-by: Vimalkumar <j.vimal@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Yevgeny Petrilin <yevgenyp@mellanox.com>
> Cc: Or Gerlitz <ogerlitz@mellanox.com>
> ---
> By the way, is BQL addition planned for this driver ?
>
>  drivers/net/ethernet/mellanox/mlx4/en_netdev.c |    2 +-
>  drivers/net/ethernet/mellanox/mlx4/mlx4_en.h   |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> index edd9cb8..2b23ca2 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> @@ -870,7 +870,7 @@ static void mlx4_en_set_default_moderation(struct mlx4_en_priv *priv)
>         /* If we haven't received a specific coalescing setting
>          * (module param), we set the moderation parameters as follows:
>          * - moder_cnt is set to the number of mtu sized packets to
> -        *   satisfy our coelsing target.
> +        *   satisfy our coalescing target.
>          * - moder_time is set to a fixed value.
>          */
>         priv->rx_frames = MLX4_EN_RX_COAL_TARGET;
> diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> index 9d27e42..8a5e70d 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> @@ -126,7 +126,7 @@ enum {
>  #define MLX4_EN_RX_COAL_TIME   0x10
>
>  #define MLX4_EN_TX_COAL_PKTS   16
> -#define MLX4_EN_TX_COAL_TIME   0x80
> +#define MLX4_EN_TX_COAL_TIME   0x10
>
>  #define MLX4_EN_RX_RATE_LOW            400000
>  #define MLX4_EN_RX_COAL_TIME_LOW       0
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet Nov. 6, 2012, 11:50 a.m. UTC | #2 
On Tue, 2012-11-06 at 10:04 +0200, Amir Mellanox wrote:
> Eric Hi,
> 
> I will check and review your patch and return to you ASAP.
> 
> Regarding BQL, the driver has it over half a year now, In commit:
> 5b263f5374ea70daa61ed9f5cc5d8f4ac236efbd

Oh right, it seems I was not looking for right verbs or changelog
keywords ;)

Then I suspect my patch will help to keep inflight BQL counter lower and
reduce latencies.

Thanks


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Amir Vadai Nov. 7, 2012, 3:30 p.m. UTC | #3 
Acked.

BTW, a mail will be sent - I'm replacing Yevgeny Petrilin as mlx4_en maintainer.

(sorry for the previous non plain text mail)

Amir

On Tue, Nov 6, 2012 at 4:20 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> From: Eric Dumazet <edumazet@google.com>
>
> mlx4 currently uses a too high tx coalescing setting, deferring
> TX completion interrupts by up to 128 us.
>
> With the recent skb_orphan() removal in commit 8112ec3b872,
> performance of a single TCP flow is capped to ~4 Gbps, unless
> we increase tcp_limit_output_bytes.
>
> I suggest using 16 us instead of 128 us, allowing a finer control.
>
> Performance of a single TCP flow is restored to previous levels,
> while keeping TCP small queues fully enabled with default sysctl.
>
> This patch is also a BQL prereq.
>
> Reported-by: Vimalkumar <j.vimal@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Yevgeny Petrilin <yevgenyp@mellanox.com>
> Cc: Or Gerlitz <ogerlitz@mellanox.com>
> ---
> By the way, is BQL addition planned for this driver ?
>
>  drivers/net/ethernet/mellanox/mlx4/en_netdev.c |    2 +-
>  drivers/net/ethernet/mellanox/mlx4/mlx4_en.h   |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> index edd9cb8..2b23ca2 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> @@ -870,7 +870,7 @@ static void mlx4_en_set_default_moderation(struct
> mlx4_en_priv *priv)
>         /* If we haven't received a specific coalescing setting
>          * (module param), we set the moderation parameters as follows:
>          * - moder_cnt is set to the number of mtu sized packets to
> -        *   satisfy our coelsing target.
> +        *   satisfy our coalescing target.
>          * - moder_time is set to a fixed value.
>          */
>         priv->rx_frames = MLX4_EN_RX_COAL_TARGET;
> diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> index 9d27e42..8a5e70d 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
> @@ -126,7 +126,7 @@ enum {
>  #define MLX4_EN_RX_COAL_TIME   0x10
>
>  #define MLX4_EN_TX_COAL_PKTS   16
> -#define MLX4_EN_TX_COAL_TIME   0x80
> +#define MLX4_EN_TX_COAL_TIME   0x10
>
>  #define MLX4_EN_RX_RATE_LOW            400000
>  #define MLX4_EN_RX_COAL_TIME_LOW       0
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Eric Dumazet Nov. 7, 2012, 4:05 p.m. UTC | #4 
On Wed, 2012-11-07 at 17:30 +0200, Amir Vadai wrote:
> Acked.
> 
> BTW, a mail will be sent - I'm replacing Yevgeny Petrilin as mlx4_en maintainer.
> 

OK, thanks.

BTW proper way to Ack a patch so that its automatically taken into
account by tools is to use :

Acked-by: Amir Vadai <amirv@mellanox.com>



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Miller Nov. 7, 2012, 8:31 p.m. UTC | #5 
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Mon, 05 Nov 2012 18:20:42 -0800

> From: Eric Dumazet <edumazet@google.com>
> 
> mlx4 currently uses a too high tx coalescing setting, deferring
> TX completion interrupts by up to 128 us.
> 
> With the recent skb_orphan() removal in commit 8112ec3b872,
> performance of a single TCP flow is capped to ~4 Gbps, unless
> we increase tcp_limit_output_bytes.
> 
> I suggest using 16 us instead of 128 us, allowing a finer control.
> 
> Performance of a single TCP flow is restored to previous levels,
> while keeping TCP small queues fully enabled with default sysctl.
> 
> This patch is also a BQL prereq.
> 
> Reported-by: Vimalkumar <j.vimal@gmail.com>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Applied, thanks Eric.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index edd9cb8..2b23ca2 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -870,7 +870,7 @@  static void mlx4_en_set_default_moderation(struct mlx4_en_priv *priv)
 	/* If we haven't received a specific coalescing setting
 	 * (module param), we set the moderation parameters as follows:
 	 * - moder_cnt is set to the number of mtu sized packets to
-	 *   satisfy our coelsing target.
+	 *   satisfy our coalescing target.
 	 * - moder_time is set to a fixed value.
 	 */
 	priv->rx_frames = MLX4_EN_RX_COAL_TARGET;
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 9d27e42..8a5e70d 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -126,7 +126,7 @@  enum {
 #define MLX4_EN_RX_COAL_TIME	0x10
 
 #define MLX4_EN_TX_COAL_PKTS	16
-#define MLX4_EN_TX_COAL_TIME	0x80
+#define MLX4_EN_TX_COAL_TIME	0x10
 
 #define MLX4_EN_RX_RATE_LOW		400000
 #define MLX4_EN_RX_COAL_TIME_LOW	0