From patchwork Wed Jul 12 18:50:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Gardner X-Patchwork-Id: 1806948 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=ub1EU4nh; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4R1RfL6Lvcz242X for ; Thu, 13 Jul 2023 04:51:21 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1qJevf-0002g5-2O; Wed, 12 Jul 2023 18:51:11 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1qJevc-0002fg-Kd for kernel-team@lists.ubuntu.com; Wed, 12 Jul 2023 18:51:08 +0000 Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 986653F189 for ; Wed, 12 Jul 2023 18:51:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1689187867; bh=XXzmykRnaH/KiZCbAN6AN+O+mleAlA0NO1L/uBzzBt0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ub1EU4nhoMYuIduPXhBpWU/yYiZHdxKzoWP0y67A/mCv9Uf8eyqiIBLGXrvuoXhMu FnF6JlEwhrICHydPuKNJEdlb+6nJFobtCJ6ZEGbWZFVXN/wEJhsI+r6IHh46Lwt9P8 MrGixZK5YKXHofYNMvN7hDhpCJ0wGyzjxucOwYCzjbLTjSgQazPDasPXxH2T/DQG9L 3+Gzpy5h7ObQ1uKVwU8/rRXY2sWfjvIXmD3PAAJEI54IXsFvDrgY9h/Fv70AKQPJFs uk1UHG3k/IA16QMXCD//7bztFBqXVGablGnJxMEbiZMu2BLydUEKmA5VCHQoat2Yjw aTAstJ4z+nDEQ== Received: by mail-pg1-f197.google.com with SMTP id 41be03b00d2f7-53feeb13906so9385186a12.1 for ; Wed, 12 Jul 2023 11:51:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689187865; x=1691779865; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XXzmykRnaH/KiZCbAN6AN+O+mleAlA0NO1L/uBzzBt0=; b=keSF7r0JtbWotRDeWCI1DGInLR9vamVZ4cLIsHCIovAHbWCixKq1INkzd7EUdQxFPY U/l9id4k5FgWj6BfMP9D0npvGhSQ8WB+FwbQp3LPv/UX79a75A9LLuBNS3VG05L2MKJS +t6fqUGQmmBsKcmbmtn/8vMR6KKl4LJ32yJvqjVqfKFvjDc9bHgtlku0hh9iUsUWP4El cyebWbM5ZRqpBAyr2UT80ctc5CmMcpuAQmEshUSDPProNZHxYvNPVOpynMiY662JXAdA StiE7LoQV/UZx9AE+QHlNBjuou58LuymwHYOvuHwFToGVway/H9OdkiqHTDIZnBIoZl9 tV3Q== X-Gm-Message-State: ABy/qLbFcCfPncH7IDe+sfNWqOGr13KlsCyi7mU/Y7vtqEJt4sk6od/6 Qomf/A5+V67TYSOS/RvwEgtMrKroLqUuaefCagnI5nKVOZ+PL2yhaXB3hsdQfaRUynFx+layc9d fJG+EPIssa9flr90FOiEl/CqA0PZw3J1T05EgIqaAUOIbCZMvgw== X-Received: by 2002:a05:6a00:3920:b0:682:4edf:b9c7 with SMTP id fh32-20020a056a00392000b006824edfb9c7mr25344789pfb.2.1689187865654; Wed, 12 Jul 2023 11:51:05 -0700 (PDT) X-Google-Smtp-Source: APBJJlHqQUrHK4zIW1KKZsQQpl2YZGS9MHYcSTY+RCcw/usnzZBNLCZBFaSM0wr0eImGyKy3Q3miJQ== X-Received: by 2002:a05:6a00:3920:b0:682:4edf:b9c7 with SMTP id fh32-20020a056a00392000b006824edfb9c7mr25344777pfb.2.1689187865361; Wed, 12 Jul 2023 11:51:05 -0700 (PDT) Received: from smtp.gmail.com (174-045-099-030.res.spectrum.com. [174.45.99.30]) by smtp.gmail.com with ESMTPSA id x41-20020a056a000be900b00682a9325ffcsm3984968pfu.5.2023.07.12.11.51.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 11:51:04 -0700 (PDT) From: Tim Gardner To: kernel-team@lists.ubuntu.com Subject: [PATCH 1/2] UBUNTU: SAUCE: net: mana: Batch ringing RX queue doorbell on receiving packets Date: Wed, 12 Jul 2023 12:50:58 -0600 Message-Id: <20230712185059.723271-2-tim.gardner@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712185059.723271-1-tim.gardner@canonical.com> References: <20230712185059.723271-1-tim.gardner@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Long Li BugLink: https://bugs.launchpad.net/bugs/2027615 It's inefficient to ring the doorbell page every time a WQE is posted to the received queue. Excessive MMIO writes result in CPU spending more time waiting on LOCK instructions (atomic operations), resulting in poor scaling performance. Move the code for ringing doorbell page to where after we have posted all WQEs to the receive queue during a callback from napi_poll(). With this change, tests showed an improvement from 120G/s to 160G/s on a 200G physical link, with 16 or 32 hardware queues. Tests showed no regression in network latency benchmarks on single connection. Cc: stable@vger.kernel.org Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)") Reviewed-by: Haiyang Zhang Reviewed-by: Dexuan Cui Signed-off-by: Long Li Signed-off-by: Tim Gardner --- drivers/net/ethernet/microsoft/mana/mana_en.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c index 2bf3f999ce97..146b4f99a5d8 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -1383,8 +1383,8 @@ static void mana_post_pkt_rxq(struct mana_rxq *rxq) recv_buf_oob = &rxq->rx_oobs[curr_index]; - err = mana_gd_post_and_ring(rxq->gdma_rq, &recv_buf_oob->wqe_req, - &recv_buf_oob->wqe_inf); + err = mana_gd_post_work_request(rxq->gdma_rq, &recv_buf_oob->wqe_req, + &recv_buf_oob->wqe_inf); if (WARN_ON_ONCE(err)) return; @@ -1654,6 +1654,12 @@ static void mana_poll_rx_cq(struct mana_cq *cq) mana_process_rx_cqe(rxq, cq, &comp[i]); } + if (comp_read > 0) { + struct gdma_context *gc = rxq->gdma_rq->gdma_dev->gdma_context; + + mana_gd_wq_ring_doorbell(gc, rxq->gdma_rq); + } + if (rxq->xdp_flush) xdp_do_flush(); } From patchwork Wed Jul 12 18:50:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Gardner X-Patchwork-Id: 1806949 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=qs01uR0q; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4R1RfL653Hz20Ph for ; Thu, 13 Jul 2023 04:51:21 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1qJevh-0002gQ-7j; Wed, 12 Jul 2023 18:51:13 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1qJeve-0002fx-Px for kernel-team@lists.ubuntu.com; Wed, 12 Jul 2023 18:51:10 +0000 Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 9D6D73F15D for ; Wed, 12 Jul 2023 18:51:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1689187870; bh=euppatP3y7WBuVtkyK3FpsoCMJ55WZQ54dvs+ZN1h+o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qs01uR0qQmwvCqL1bYwYWpCpC/uWe5HB4GsOQw0msKxxQFJojxD2mZ/29RghKRUS2 dHUvO3QnP0a7J95o+rfBqjYKySZsxNPoLOlfJ9g1kiAhc7xoBcGuDQR4Dx2OB8zWkf 5WGBn0zkQjVjtEgKQAfcNXnXiMavPmxqvuAexzEtVGJSLW6ND8+O7hxSRN+F3QqwN9 mIwfiVLF3gwtmqHy01DBzedI2mXjSo+EMTsQo70dw3RXv4qC4MAMm41dBHh9XUJwvr dvBGnfUQHMy07QygdfRlGKDPWlllS+fH7s0fQizW6UqYmZqcYaGpUhACCtx7jQN0rD zPwUev0GIKeQw== Received: by mail-pg1-f200.google.com with SMTP id 41be03b00d2f7-55c1c7f872bso4963897a12.1 for ; Wed, 12 Jul 2023 11:51:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689187867; x=1691779867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=euppatP3y7WBuVtkyK3FpsoCMJ55WZQ54dvs+ZN1h+o=; b=Kwp5YNZ6zO58Bo18hswnH7rrW/fdiZrh9/3xiwFSuvoby+d8kvVXJ3Y8bFjzgdWWoU T9rPkyvqtDauutVwrub4eE6XqIEhS7uhUIsB582bMtbr97TihO92Is8GViRcFQjkBcd1 W9sk5oORdUtUSlA2E/yucG3gBh67DJonh0pG9/AbpnC7+r9JBRyE08+LaNbpPmNcHMV7 ReJCB1hC8/pwjMGEyNj5soC875OPi3Q5o/daC71x2he64lLHWdDmQqbMDWnLIi21cq0H B3MhqMqvOdZ+2m0AGWXq83eToZh2A7VG1QSrBW8mVyUoFwCykzzdHNevY3LD3KdSekm+ UAnQ== X-Gm-Message-State: ABy/qLYIWA3luwFYA+Q3y0Xkdrh0aR4wMsBSFY5RzBaQUlF2yeSaX03G 1MVMQymu2m0rMgbD+OnZvQ9DZw3NS4zty5dNPmVSRPWfhZTsUUFKAxsASudqh6K7zmzyz3QXgxb nosxClS8G44fc3VW13FmFFQCVUsn0NvnIKit0gkQcZnBBo+I+Xg== X-Received: by 2002:a05:6a20:4421:b0:131:6464:2176 with SMTP id ce33-20020a056a20442100b0013164642176mr11692051pzb.29.1689187866751; Wed, 12 Jul 2023 11:51:06 -0700 (PDT) X-Google-Smtp-Source: APBJJlEF/gP3irJI6P8MXNwYyvuiMwk3WGPdyTv5r9d5eEbGkWgRgBfjmtRSS0CFGsTkETyY0ALWww== X-Received: by 2002:a05:6a20:4421:b0:131:6464:2176 with SMTP id ce33-20020a056a20442100b0013164642176mr11692040pzb.29.1689187866482; Wed, 12 Jul 2023 11:51:06 -0700 (PDT) Received: from smtp.gmail.com (174-045-099-030.res.spectrum.com. [174.45.99.30]) by smtp.gmail.com with ESMTPSA id x41-20020a056a000be900b00682a9325ffcsm3984968pfu.5.2023.07.12.11.51.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 11:51:06 -0700 (PDT) From: Tim Gardner To: kernel-team@lists.ubuntu.com Subject: [PATCH 2/2] UBUNTU: SAUCE: net: mana: Use the correct WQE count for ringing RQ doorbell Date: Wed, 12 Jul 2023 12:50:59 -0600 Message-Id: <20230712185059.723271-3-tim.gardner@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712185059.723271-1-tim.gardner@canonical.com> References: <20230712185059.723271-1-tim.gardner@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Long Li BugLink: https://bugs.launchpad.net/bugs/2027615 The hardware specification specifies that WQE_COUNT should set to 0 for the Receive Queue. Although currently the hardware doesn't enforce the check, in the future releases it may check on this value. Cc: stable@vger.kernel.org Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network Adapter (MANA)") Reviewed-by: Haiyang Zhang Reviewed-by: Dexuan Cui Signed-off-by: Long Li Signed-off-by: Tim Gardner --- drivers/net/ethernet/microsoft/mana/gdma_main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c index f9b8f372ec8a..9a08ed3eff01 100644 --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c @@ -300,8 +300,11 @@ static void mana_gd_ring_doorbell(struct gdma_context *gc, u32 db_index, void mana_gd_wq_ring_doorbell(struct gdma_context *gc, struct gdma_queue *queue) { + /* Hardware Spec specifies that software client should set 0 for + * wqe_cnt for Receive Queues. This value is not used in Send Queues. + */ mana_gd_ring_doorbell(gc, queue->gdma_dev->doorbell, queue->type, - queue->id, queue->head * GDMA_WQE_BU_SIZE, 1); + queue->id, queue->head * GDMA_WQE_BU_SIZE, 0); } void mana_gd_ring_cq(struct gdma_queue *cq, u8 arm_bit)