| Message ID | 20181213055521.119903-1-aik@ozlabs.ru |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [v2] npu2: Allow ATSD for LPAR other than 0 | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | master/apply_patch Successfully applied |
| snowpatch_ozlabs/snowpatch_job_snowpatch-skiboot | success | Test snowpatch/job/snowpatch-skiboot on branch master |
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > Each XTS MMIO ATSD# register is accompanied by another register - > XTS MMIO ATSD0 LPARID# - which controls LPID filtering for ATSD > transactions. > > When a host system passes a GPU through to a guest, we need to enable > some ATSD for an LPAR. At the moment the host assigns one ATSD to > a NVLink bridge and this maps it to an LPAR when GPU is assigned to > the LPAR. The link number is used for an ATSD index. > > ATSD6&7 stay mapped to the host (LPAR=0) all the time which seems to be > acceptable price for the simplicity. Merged to master as of c0f17caddcd894f68340ba1c66151cc6b26aa5de. Is there any backwards-compat things we need to deal with here?
On 26/02/2019 17:12, Stewart Smith wrote: > Alexey Kardashevskiy <aik@ozlabs.ru> writes: >> Each XTS MMIO ATSD# register is accompanied by another register - >> XTS MMIO ATSD0 LPARID# - which controls LPID filtering for ATSD >> transactions. >> >> When a host system passes a GPU through to a guest, we need to enable >> some ATSD for an LPAR. At the moment the host assigns one ATSD to >> a NVLink bridge and this maps it to an LPAR when GPU is assigned to >> the LPAR. The link number is used for an ATSD index. >> >> ATSD6&7 stay mapped to the host (LPAR=0) all the time which seems to be >> acceptable price for the simplicity. > > Merged to master as of c0f17caddcd894f68340ba1c66151cc6b26aa5de. > > Is there any backwards-compat things we need to deal with here? I cannot think of any.
diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 86ca819..211cf2b 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -547,6 +547,8 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_XTS_MMIO_ATSD5_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x128) #define NPU2_XTS_MMIO_ATSD6_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x130) #define NPU2_XTS_MMIO_ATSD7_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x138) +#define NPU2_XTS_MMIO_ATSD_MSR_HV PPC_BIT(51) +#define NPU2_XTS_MMIO_ATSD_LPARID PPC_BITMASK(52, 63) #define NPU2_XTS_BDF_MAP NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x4000) #define NPU2_XTS_BDF_MAP_VALID PPC_BIT(0) #define NPU2_XTS_BDF_MAP_UNFILT PPC_BIT(1) diff --git a/hw/npu2.c b/hw/npu2.c index 1a7e3c4..d86ed2b 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -2261,9 +2261,15 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, struct phb *phb = pci_get_phb(phb_id); struct npu2 *p; struct npu2_dev *ndev = NULL; - uint64_t xts_bdf_lpar, rc = OPAL_SUCCESS; + uint64_t xts_bdf_lpar, atsd_lpar, rc = OPAL_SUCCESS; int i; int id; + static uint64_t atsd_lpar_regs[] = { + NPU2_XTS_MMIO_ATSD0_LPARID, NPU2_XTS_MMIO_ATSD1_LPARID, + NPU2_XTS_MMIO_ATSD2_LPARID, NPU2_XTS_MMIO_ATSD3_LPARID, + NPU2_XTS_MMIO_ATSD4_LPARID, NPU2_XTS_MMIO_ATSD5_LPARID, + NPU2_XTS_MMIO_ATSD6_LPARID, NPU2_XTS_MMIO_ATSD7_LPARID + }; if (!phb || phb->phb_type != phb_type_npu_v2) return OPAL_PARAMETER; @@ -2317,6 +2323,20 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, goto out; } + /* + * We need to allocate an ATSD per NVLink bridge if possible, + * use the ibm,npu-link-index property for that. + */ + atsd_lpar = SETFIELD(NPU2_XTS_MMIO_ATSD_LPARID, 0, lparid); + if (!lparid) + atsd_lpar = SETFIELD(NPU2_XTS_MMIO_ATSD_MSR_HV, atsd_lpar, 1); + + if (ndev->link_index < ARRAY_SIZE(atsd_lpar_regs)) + npu2_write(p, atsd_lpar_regs[ndev->link_index], atsd_lpar); + else + NPU2ERR(p, "Unable to assign ATSD for link index %u\n", + ndev->link_index); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_STACK, xts_bdf_lpar, 0x4 >> (ndev->brick_index / 2)); xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BRICK, xts_bdf_lpar,
Each XTS MMIO ATSD# register is accompanied by another register - XTS MMIO ATSD0 LPARID# - which controls LPID filtering for ATSD transactions. When a host system passes a GPU through to a guest, we need to enable some ATSD for an LPAR. At the moment the host assigns one ATSD to a NVLink bridge and this maps it to an LPAR when GPU is assigned to the LPAR. The link number is used for an ATSD index. ATSD6&7 stay mapped to the host (LPAR=0) all the time which seems to be acceptable price for the simplicity. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> --- Changes: v2: * use the actual link_index instead of the index into npu2::devices --- include/npu2-regs.h | 2 ++ hw/npu2.c | 22 +++++++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-)