From patchwork Wed Dec 5 04:52:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 1007989 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 438mh24ssJz9s7W for ; Wed, 5 Dec 2018 15:57:54 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 438mh22VclzDqq5 for ; Wed, 5 Dec 2018 15:57:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=ozlabs.ru (client-ip=107.173.13.209; helo=ozlabs.ru; envelope-from=aik@ozlabs.ru; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Received: from ozlabs.ru (unknown [107.173.13.209]) by lists.ozlabs.org (Postfix) with ESMTP id 438mYl21KtzDqly for ; Wed, 5 Dec 2018 15:52:27 +1100 (AEDT) Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 463E6AE80001; Tue, 4 Dec 2018 23:52:24 -0500 (EST) From: Alexey Kardashevskiy To: skiboot@lists.ozlabs.org Date: Wed, 5 Dec 2018 15:52:22 +1100 Message-Id: <20181205045222.40053-1-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 Subject: [Skiboot] [PATCH skiboot] npu2: Allow ATSD for LPAR other than 0 X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alistair Popple , Reza Arbab MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Each XTS MMIO ATSD# register is accompanied by another register - XTS MMIO ATSD0 LPARID# - which controls LPID filtering for ATSD transactions. When a host system passes a GPU through to a guest, we need to enable some ATSD for an LPAR. At the moment the host assigns one ATSD to a NVLink bridge and this maps it to an LPAR when GPU is assigned to the LPAR. The link number is used for an ATSD index. ATSD6&7 stay mapped to the host (LPAR=0) all the time which seems to be acceptable price for the simplicity. Signed-off-by: Alexey Kardashevskiy --- include/npu2-regs.h | 2 ++ hw/npu2.c | 22 ++++++++++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 8273b2b..ae5e225 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -547,6 +547,8 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_XTS_MMIO_ATSD5_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x128) #define NPU2_XTS_MMIO_ATSD6_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x130) #define NPU2_XTS_MMIO_ATSD7_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x138) +#define NPU2_XTS_MMIO_ATSD_MSR_HV PPC_BIT(51) +#define NPU2_XTS_MMIO_ATSD_LPARID PPC_BITMASK(52,63) #define NPU2_XTS_BDF_MAP NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x4000) #define NPU2_XTS_BDF_MAP_VALID PPC_BIT(0) #define NPU2_XTS_BDF_MAP_UNFILT PPC_BIT(1) diff --git a/hw/npu2.c b/hw/npu2.c index 41563b4..767306f 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -2255,9 +2255,14 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, struct phb *phb = pci_get_phb(phb_id); struct npu2 *p; struct npu2_dev *ndev = NULL; - uint64_t xts_bdf_lpar, rc = OPAL_SUCCESS; + uint64_t xts_bdf_lpar, atsd_lpar, rc = OPAL_SUCCESS; int i; int id; + static uint64_t atsd_lpar_regs[] = { + NPU2_XTS_MMIO_ATSD0_LPARID, NPU2_XTS_MMIO_ATSD1_LPARID, + NPU2_XTS_MMIO_ATSD2_LPARID, NPU2_XTS_MMIO_ATSD3_LPARID, + NPU2_XTS_MMIO_ATSD4_LPARID, NPU2_XTS_MMIO_ATSD5_LPARID, + NPU2_XTS_MMIO_ATSD6_LPARID, NPU2_XTS_MMIO_ATSD7_LPARID }; if (!phb || phb->phb_type != phb_type_npu_v2) return OPAL_PARAMETER; @@ -2297,11 +2302,20 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_LPARID, xts_bdf_lpar, lparid); xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_LPARSHORT, xts_bdf_lpar, id); - /* Need to find an NVLink to send the ATSDs for this device over */ + /* + * Need to find an NVLink to send the ATSDs for this device over. + * Also, the host allocates an ATSD per NVLink, enable filtering now. + */ + atsd_lpar = SETFIELD(NPU2_XTS_MMIO_ATSD_LPARID, 0, lparid); + if (!lparid) + atsd_lpar = SETFIELD(NPU2_XTS_MMIO_ATSD_MSR_HV, atsd_lpar, 1); + for (i = 0; i < p->total_devices; i++) { if (p->devices[i].nvlink.gpu_bdfn == bdf) { - ndev = &p->devices[i]; - break; + if (!ndev) + ndev = &p->devices[i]; + if (i < ARRAY_SIZE(atsd_lpar_regs)) + npu2_write(p, atsd_lpar_regs[i], atsd_lpar); } }