From patchwork Tue Dec 20 12:02:05 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tariq Toukan X-Patchwork-Id: 707415 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3tjbxg1tqFz9t1F for ; Tue, 20 Dec 2016 23:02:15 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rnVGgwYa"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757698AbcLTMCL (ORCPT ); Tue, 20 Dec 2016 07:02:11 -0500 Received: from mail-wj0-f195.google.com ([209.85.210.195]:36280 "EHLO mail-wj0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757498AbcLTMCJ (ORCPT ); Tue, 20 Dec 2016 07:02:09 -0500 Received: by mail-wj0-f195.google.com with SMTP id j10so27359715wjb.3 for ; Tue, 20 Dec 2016 04:02:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=O65uobM9y4tzx3piP2dkJKAYbdVjCOcJKllP6gHCJAw=; b=rnVGgwYaMaGfmlNafWW/vYPsoUEpKKb4MWrH7/CgisIXfOfyKHJ6OC+1i9lc9sSnQL 4bv/LlL9zyVhPus2pcfERVLqkT7PFYmUp6qlkmYQ3JtdVfkCnxmkjFuQWSFrjCuvO/wr /LGYYdf1MXzFgkRdPB8omb556Qmxt5QzIQGm0OeYlhiBucQMntyFPJAROqfl/xURN04c mtqRMVRpcUYkWCU+p/21t+2vnGq2V4ErTtT2qD3P9zIRRn1q5d4oTfGM79wA4uKaldeZ fxlm6G4VzW5UmfiRZWiMaksSXrTe1xHUWmwDSh+dxdGZXyJsg/yRKV+kJxr66ijjvHdU HELA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=O65uobM9y4tzx3piP2dkJKAYbdVjCOcJKllP6gHCJAw=; b=QehkDBtkrDFs+xHoyVUtJDku6omEViLPYK17V26aiAQjjKvq3N8/Eva4qvK0gb1bUx CGokXbtQgmxCVwg9xHu2uBrnFpKDxEjXWWRAHaj3pxV3KUhpXmxpPGE2zEmgMJCimj9n GLE4Tja0hewIJLHmHIpVO8E8m/4O0GqBj6lEWKie+2xusYw3f5ouFPZq7ec+LzPsy6WL OInodcslvIV5VMX6vjZ8Fz1ddUusozqV4eWcSAv4r0mbJNLrgErj14aiTARf+x3AB557 ADV1gXFqb9yfm70a9vccwNFkFdSFWKe42HpOHt0srFuUE7MII6g5v7rSfm6RHxcH0SvX Pv1g== X-Gm-Message-State: AKaTC02tPxM8aCHi3o5o++IwQhZkqQpFJvDcGpuszuwkQmnG2WuwHNYZpWwmTQ0jZ9X52w== X-Received: by 10.194.90.135 with SMTP id bw7mr17145916wjb.34.1482235327864; Tue, 20 Dec 2016 04:02:07 -0800 (PST) Received: from [10.8.1.96] ([193.47.165.251]) by smtp.gmail.com with ESMTPSA id d17sm25112658wjr.14.2016.12.20.04.02.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Dec 2016 04:02:07 -0800 (PST) Subject: Re: mlx4: Bug in XDP_TX + 16 rx-queues To: Martin KaFai Lau , Tariq Toukan References: <20161219233709.GA29858@kafai-mba.local> Cc: Saeed Mahameed , "netdev@vger.kernel.org" , Alexei Starovoitov From: Tariq Toukan Message-ID: Date: Tue, 20 Dec 2016 14:02:05 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20161219233709.GA29858@kafai-mba.local> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Thanks Martin, nice catch! On 20/12/2016 1:37 AM, Martin KaFai Lau wrote: > Hi Tariq, > > On Sat, Dec 17, 2016 at 02:18:03AM -0800, Martin KaFai Lau wrote: >> Hi All, >> >> I have been debugging with XDP_TX and 16 rx-queues. >> >> 1) When 16 rx-queues is used and an XDP prog is doing XDP_TX, >> it seems that the packet cannot be XDP_TX out if the pkt >> is received from some particular CPUs (/rx-queues). >> >> 2) If 8 rx-queues is used, it does not have problem. >> >> 3) The 16 rx-queues problem also went away after reverting these >> two patches: >> 15fca2c8eb41 net/mlx4_en: Add ethtool statistics for XDP cases >> 67f8b1dcb9ee net/mlx4_en: Refactor the XDP forwarding rings scheme >> > After taking a closer look at 67f8b1dcb9ee ("net/mlx4_en: Refactor the XDP forwarding rings scheme") > and armed with the fact that '>8 rx-queues does not work', I have > made the attached change that fixed the issue. > > Making change in mlx4_en_fill_qp_context() could be an easier fix > but I think this change will be easier for discussion purpose. > > I don't want to lie that I know anything about how this variable > works in CX3. If this change makes sense, I can cook up a diff. > Otherwise, can you shed some light on what could be happening > and hopefully can lead to a diff? > > Thanks > --Martin > > > diff --git i/drivers/net/ethernet/mellanox/mlx4/en_netdev.c w/drivers/net/ethernet/mellanox/mlx4/en_netdev.c > index bcd955339058..b3bfb987e493 100644 > --- i/drivers/net/ethernet/mellanox/mlx4/en_netdev.c > +++ w/drivers/net/ethernet/mellanox/mlx4/en_netdev.c > @@ -1638,10 +1638,10 @@ int mlx4_en_start_port(struct net_device *dev) > > /* Configure tx cq's and rings */ > for (t = 0 ; t < MLX4_EN_NUM_TX_TYPES; t++) { > - u8 num_tx_rings_p_up = t == TX ? priv->num_tx_rings_p_up : 1; The bug lies in this line. Number of rings per UP in case of TX_XDP should be priv->tx_ring_num[TX_XDP ], not 1. Please try the following fix. I can prepare and send it once the window opens again. /* Configure cq */ > - > for (i = 0; i < priv->tx_ring_num[t]; i++) { > /* Configure cq */ > + int user_prio; > + > cq = priv->tx_cq[t][i]; > err = mlx4_en_activate_cq(priv, cq, i); > if (err) { > @@ -1660,9 +1660,14 @@ int mlx4_en_start_port(struct net_device *dev) > > /* Configure ring */ > tx_ring = priv->tx_ring[t][i]; > + if (t != TX_XDP) > + user_prio = i / priv->num_tx_rings_p_up; > + else > + user_prio = i & 0x07; > + > err = mlx4_en_activate_tx_ring(priv, tx_ring, > cq->mcq.cqn, > - i / num_tx_rings_p_up); > + user_prio); > if (err) { > en_err(priv, "Failed allocating Tx ring\n"); > mlx4_en_deactivate_cq(priv, cq); Regards, Tariq Toukan. diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index bcd955339058..edbe200ac2fa 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -1638,7 +1638,8 @@ int mlx4_en_start_port(struct net_device *dev) /* Configure tx cq's and rings */ for (t = 0 ; t < MLX4_EN_NUM_TX_TYPES; t++) { - u8 num_tx_rings_p_up = t == TX ? priv->num_tx_rings_p_up : 1; + u8 num_tx_rings_p_up = t == TX ? + priv->num_tx_rings_p_up : priv->tx_ring_num[t]; for (i = 0; i < priv->tx_ring_num[t]; i++) {