From patchwork Fri Aug 28 20:31:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Whitfield X-Patchwork-Id: 2294267 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4hWqnH5zv3z1xyd for ; Sat, 29 Aug 2026 06:32:18 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1x03F8-0002JA-SZ; Fri, 28 Aug 2026 20:32:06 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1x03F4-0002I6-T4 for kernel-team@lists.ubuntu.com; Fri, 28 Aug 2026 20:32:02 +0000 Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id A814C3F2DD for ; Fri, 28 Aug 2026 20:32:02 +0000 (UTC) Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-92e4f946461so163269485a.2 for ; Fri, 28 Aug 2026 13:32:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787949121; x=1788553921; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to:content-type; bh=biQuFj57ZQU25hotCT4VtdikINQGD+AlWNHFSkfvpG4=; b=JL3uGhGf8UU1YSQqJ5KoyavO43sXUhIXi8w1ciJxsuEiKuppcZRodJD8h2SgBN30dS S7KsUPRCSOESl8DADi3X7s6sg7u0JAWNDMAtPhgOBonnPzXfr5vVPrCV8wyNYFzrObCB ZXAfVr8cBTV99fF8/BXJK2JTlCd/UTTTGoPXGuv4NLN8Dk4J4PDWut9Ofux/fRkMykAW iyAtMXe2KcPIcYATUccHNd544TDDGhpkzZBGpQp7paO4dInwL/ScFxfEVjE/3NB41tV6 QWe1dsc+rkx8HJxSa2fwV3ZL7zrMBZeRmZ27qbd5VAggYUAZZaKlWn4gBhIAPAhATWVF EJSw== X-Gm-Message-State: AFuF++nuvWRA4E1b7qAGCAhxaJw9EbPvMuXX5ZnoEDPQt8zppCfvK5iF vPyvkZgyC/AypX4QuhfB1nJIXbxSupzybPrv+OL9qHrIba5rgJ/jJp0bWwPhOk5JdPS2zXHoS7U xTK4GIfx9TkzVsAZ8HCJO+5GF3WXB0XXT5zuLWx9+gBGJmtHMXmDquqo2AbKjKWt25GD7L2yStY LSOJkaJAJ+1LvOMA== X-Gm-Gg: AR+sD13Se28/djixGYb0EG1JNqY+XQJKPdaDH/JGf9lW2DEVuSYiFXFQ34podk9t/6B CMGcBAvkpj4LMTNt2tjLf5rkbHEU2LMsYI8Yw7vmvWlXf6/qVGxoTPEimISCtYdWGGahDbxJeDr UM3GQvPIfiV23xfTJIGPccF0eWW3mAGxFQ2h7wHU4ZH2NPC2/Bch0SM+bEy4RANZ7w/WXHtOGSI E+mQF0Klu4XtgDlc4XLWbusw2+OYUirH475LB0I/5scDzkoqUboumGmZ1Brzv2aGCgHHnM/SM7Z 6d4brSR/3E/OV4kdWFPTrbdc11ZRYi8o0pU8cJNAe7pt6Q/+EvfbRnG3cIqmoE0tOkZaGduZV/k mLqjeww== X-Received: by 2002:a05:620a:6488:b0:939:1fc6:3fd6 with SMTP id af79cd13be357-9391fc64239mr443314385a.18.1787949121133; Fri, 28 Aug 2026 13:32:01 -0700 (PDT) X-Received: by 2002:a05:620a:6488:b0:939:1fc6:3fd6 with SMTP id af79cd13be357-9391fc64239mr443303085a.18.1787949120512; Fri, 28 Aug 2026 13:32:00 -0700 (PDT) Received: from localhost ([208.89.33.89]) by smtp.gmail.com with ESMTPSA id af79cd13be357-9391723e87dsm225336685a.15.2026.08.28.13.31.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 13:32:00 -0700 (PDT) To: kernel-team@lists.ubuntu.com Subject: [SRU][R:linux-gcp][PATCH 1/9] RDMA/irdma: Provide scratch buffers to firmware for internal use Date: Fri, 28 Aug 2026 16:31:47 -0400 Message-ID: <20260828203155.75778-2-ian.whitfield@canonical.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260828203155.75778-1-ian.whitfield@canonical.com> References: <20260828203155.75778-1-ian.whitfield@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Ian Whitfield via kernel-team From: Ian Whitfield Reply-To: Ian Whitfield Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Jay Bhat BugLink: https://bugs.launchpad.net/bugs/2165062 For GEN3 and higher, FW requires scratch buffers for bookkeeping during cleanup, specifically during QP and MR destroy ops. Signed-off-by: Jay Bhat Signed-off-by: Tatyana Nikolova Signed-off-by: Leon Romanovsky (cherry picked from commit 9d6ba4ced73433df3d3c5909d73a49945bedd5c6) Signed-off-by: Ian Whitfield --- drivers/infiniband/hw/irdma/ctrl.c | 43 +++++++++++++++++++++++++++++- drivers/infiniband/hw/irdma/defs.h | 4 +++ drivers/infiniband/hw/irdma/hw.c | 11 ++++++++ drivers/infiniband/hw/irdma/type.h | 2 ++ drivers/infiniband/hw/irdma/user.h | 4 +-- 5 files changed, 61 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/irdma/ctrl.c b/drivers/infiniband/hw/irdma/ctrl.c index 45c7433c96f36..13820f1a48a40 100644 --- a/drivers/infiniband/hw/irdma/ctrl.c +++ b/drivers/infiniband/hw/irdma/ctrl.c @@ -3570,6 +3570,41 @@ static int irdma_sc_parse_fpm_query_buf(struct irdma_sc_dev *dev, __le64 *buf, hmc_fpm_misc->loc_mem_pages = (u32)FIELD_GET(IRDMA_QUERY_FPM_LOC_MEM_PAGES, temp); if (!hmc_fpm_misc->loc_mem_pages) return -EINVAL; + + get_64bit_val(buf, 184, &temp); + if (temp) { + hmc_fpm_misc->fw_scratch_buf0.size = temp; + hmc_fpm_misc->fw_scratch_buf0.va = + dma_alloc_coherent(dev->hw->device, + hmc_fpm_misc->fw_scratch_buf0.size, + &hmc_fpm_misc->fw_scratch_buf0.pa, + GFP_KERNEL); + + if (!hmc_fpm_misc->fw_scratch_buf0.va) { + hmc_fpm_misc->fw_scratch_buf0.size = 0; + return -ENOMEM; + } + } + get_64bit_val(buf, 192, &temp); + if (temp) { + hmc_fpm_misc->fw_scratch_buf1.size = temp; + hmc_fpm_misc->fw_scratch_buf1.va = + dma_alloc_coherent(dev->hw->device, + hmc_fpm_misc->fw_scratch_buf1.size, + &hmc_fpm_misc->fw_scratch_buf1.pa, + GFP_KERNEL); + + if (!hmc_fpm_misc->fw_scratch_buf1.va) { + hmc_fpm_misc->fw_scratch_buf1.size = 0; + dma_free_coherent(dev->hw->device, + hmc_fpm_misc->fw_scratch_buf0.size, + hmc_fpm_misc->fw_scratch_buf0.va, + hmc_fpm_misc->fw_scratch_buf0.pa); + hmc_fpm_misc->fw_scratch_buf0.va = NULL; + hmc_fpm_misc->fw_scratch_buf0.size = 0; + return -ENOMEM; + } + } } return 0; @@ -4187,6 +4222,8 @@ static int irdma_sc_commit_fpm_val(struct irdma_sc_cqp *cqp, u64 scratch, hdr = FIELD_PREP(IRDMA_CQPSQ_BUFSIZE, IRDMA_COMMIT_FPM_BUF_SIZE) | FIELD_PREP(IRDMA_CQPSQ_OPCODE, IRDMA_CQP_OP_COMMIT_FPM_VAL) | + FIELD_PREP(IRDMA_CQPSQ_CFPM_FW_SCRATCH_BUF_PRESENT, + cqp->dev->hmc_fpm_misc.fw_scratch_buf0.va != NULL) | FIELD_PREP(IRDMA_CQPSQ_WQEVALID, cqp->polarity); dma_wmb(); /* make sure WQE is written before valid bit is set */ @@ -5034,7 +5071,9 @@ static void irdma_set_loc_mem(__le64 *buf) for (offset = 0; offset < IRDMA_COMMIT_FPM_BUF_SIZE; offset += sizeof(__le64)) { - if (offset == IRDMA_PBLE_COMMIT_OFFSET) + if (offset == IRDMA_PBLE_COMMIT_OFFSET || + offset == IRDMA_SCRATCH_BUF0_COMMIT_OFFSET || + offset == IRDMA_SCRATCH_BUF1_COMMIT_OFFSET) continue; get_64bit_val(buf, offset, &temp); if (temp) @@ -5090,6 +5129,8 @@ static int irdma_sc_cfg_iw_fpm(struct irdma_sc_dev *dev, u8 hmc_fn_id) (u64)obj_info[IRDMA_HMC_IW_OOISC].cnt); set_64bit_val(buf, 168, (u64)obj_info[IRDMA_HMC_IW_OOISCFFL].cnt); + set_64bit_val(buf, 192, dev->hmc_fpm_misc.fw_scratch_buf0.pa); + set_64bit_val(buf, 200, dev->hmc_fpm_misc.fw_scratch_buf1.pa); if (dev->hw_attrs.uk_attrs.hw_rev >= IRDMA_GEN_3 && dev->hmc_fpm_misc.loc_mem_pages) irdma_set_loc_mem(buf); diff --git a/drivers/infiniband/hw/irdma/defs.h b/drivers/infiniband/hw/irdma/defs.h index 983b22d7ae233..d6a3152959dde 100644 --- a/drivers/infiniband/hw/irdma/defs.h +++ b/drivers/infiniband/hw/irdma/defs.h @@ -133,6 +133,8 @@ enum irdma_protocol_used { #define MAX_MR_PER_SD 0x8000 #define MAX_MR_SD_PER_FCN 0x80 #define IRDMA_PBLE_COMMIT_OFFSET 112 +#define IRDMA_SCRATCH_BUF0_COMMIT_OFFSET 192 +#define IRDMA_SCRATCH_BUF1_COMMIT_OFFSET 200 #define IRDMA_MAX_QUANTA_PER_WR 8 #define IRDMA_QP_SW_MAX_WQ_QUANTA 32768 @@ -658,6 +660,8 @@ enum irdma_cqp_op_type { #define IRDMA_COMMIT_FPM_QPCNT GENMASK_ULL(20, 0) #define IRDMA_COMMIT_FPM_BASE_S 32 #define IRDMA_CQPSQ_CFPM_HMCFNID GENMASK_ULL(15, 0) +#define IRDMA_CQPSQ_CFPM_FW_SCRATCH_BUF_PRESENT_S 38 +#define IRDMA_CQPSQ_CFPM_FW_SCRATCH_BUF_PRESENT BIT_ULL(38) #define IRDMA_CQPSQ_FWQE_AECODE GENMASK_ULL(15, 0) #define IRDMA_CQPSQ_FWQE_AESOURCE GENMASK_ULL(19, 16) diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c index f4ae530f56db0..879fb632ee7a9 100644 --- a/drivers/infiniband/hw/irdma/hw.c +++ b/drivers/infiniband/hw/irdma/hw.c @@ -1693,6 +1693,8 @@ static int irdma_hmc_setup(struct irdma_pci_f *rf) static void irdma_del_init_mem(struct irdma_pci_f *rf) { struct irdma_sc_dev *dev = &rf->sc_dev; + struct irdma_dma_mem *fw_scratch_buf0; + struct irdma_dma_mem *fw_scratch_buf1; if (!rf->sc_dev.privileged) irdma_vchnl_req_put_hmc_fcn(&rf->sc_dev); @@ -1713,6 +1715,15 @@ static void irdma_del_init_mem(struct irdma_pci_f *rf) rf->iw_msixtbl = NULL; kfree(rf->hmc_info_mem); rf->hmc_info_mem = NULL; + + fw_scratch_buf0 = &dev->hmc_fpm_misc.fw_scratch_buf0; + fw_scratch_buf1 = &dev->hmc_fpm_misc.fw_scratch_buf1; + if (fw_scratch_buf0->va) + dma_free_coherent(dev->hw->device, fw_scratch_buf0->size, + fw_scratch_buf0->va, fw_scratch_buf0->pa); + if (fw_scratch_buf1->va) + dma_free_coherent(dev->hw->device, fw_scratch_buf1->size, + fw_scratch_buf1->va, fw_scratch_buf1->pa); } /** diff --git a/drivers/infiniband/hw/irdma/type.h b/drivers/infiniband/hw/irdma/type.h index da8c54d1f035a..5557d9338796f 100644 --- a/drivers/infiniband/hw/irdma/type.h +++ b/drivers/infiniband/hw/irdma/type.h @@ -622,6 +622,8 @@ struct irdma_hmc_fpm_misc { u32 timer_bucket; u32 rrf_block_size; u32 ooiscf_block_size; + struct irdma_dma_mem fw_scratch_buf0; + struct irdma_dma_mem fw_scratch_buf1; }; #define IRDMA_VCHNL_MAX_MSG_SIZE 512 diff --git a/drivers/infiniband/hw/irdma/user.h b/drivers/infiniband/hw/irdma/user.h index 9eb7fd0b1cbf9..008af1acc9288 100644 --- a/drivers/infiniband/hw/irdma/user.h +++ b/drivers/infiniband/hw/irdma/user.h @@ -159,8 +159,8 @@ enum irdma_device_caps_const { IRDMA_CEQE_SIZE = 1, IRDMA_CQP_CTX_SIZE = 8, IRDMA_SHADOW_AREA_SIZE = 8, - IRDMA_QUERY_FPM_BUF_SIZE = 192, - IRDMA_COMMIT_FPM_BUF_SIZE = 192, + IRDMA_QUERY_FPM_BUF_SIZE = 200, + IRDMA_COMMIT_FPM_BUF_SIZE = 208, IRDMA_GATHER_STATS_BUF_SIZE = 1024, IRDMA_MIN_IW_QP_ID = 0, IRDMA_MAX_IW_QP_ID = 262143,