From patchwork Fri Nov 9 05:43:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 995318 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42rpxb3gQWz9sBk for ; Fri, 9 Nov 2018 16:44:19 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42rpxb2W0jzF3X3 for ; Fri, 9 Nov 2018 16:44:19 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=ozlabs.ru (client-ip=107.173.13.209; helo=ozlabs.ru; envelope-from=aik@ozlabs.ru; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Received: from ozlabs.ru (unknown [107.173.13.209]) by lists.ozlabs.org (Postfix) with ESMTP id 42rpxW1d0GzF3Wt for ; Fri, 9 Nov 2018 16:44:14 +1100 (AEDT) Received: from fstn1-p1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id F18D5AE8003F; Fri, 9 Nov 2018 00:43:41 -0500 (EST) From: Alexey Kardashevskiy To: skiboot@lists.ozlabs.org Date: Fri, 9 Nov 2018 16:43:38 +1100 Message-Id: <20181109054338.33932-1-aik@ozlabs.ru> X-Mailer: git-send-email 2.17.1 Subject: [Skiboot] [PATCH skiboot v2] npu2: Add nvlink2 interconnect information X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alistair Popple , Reza Arbab MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" GPUs on Redbud and Sequoia platforms are interconnected between each other in groups of 2 or 3 GPUs. The problem with that is if we decide to pass one of GPUs in a group to the userspace (and potentially a guest), we need to make sure that interconnectd link does not get enabled. The GPU firmware provides a way to disable links on a GPU. However we want to disable only links to other GPUs which are not in the same guest so we need a map of what nvlink is connected to what. This adds an "ibm,nvlink-peers" property to every GPU in a "GPUn" slot with phandles to peer GPUs and NPU PHB, the index in the property is GPU's link number. Signed-off-by: Alexey Kardashevskiy --- Changes: v2: * s/ibm,nvlinks/ibm,nvlink-peers/ --- hw/npu2.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/hw/npu2.c b/hw/npu2.c index d7d94357..ba1264be 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -732,6 +732,91 @@ static void npu2_phb_fixup_scominit(struct dt_node *dn, int links_per_gpu) xscom_write_mask(gcid, 0x50114c0, val, mask); } +static int gpu_slot_to_num(const char *slot) +{ + char *p = NULL; + int ret; + + if (!slot) + return -1; + + if (memcmp(slot, "GPU", 3)) + return -1; + + ret = strtol(slot + 3, &p, 10); + if (*p || p == slot) + return -1; + + return ret; +} + +static void npu2_phb_nvlink_dt(struct phb *npuphb, int links_per_gpu) +{ + struct dt_node *g[3] = { 0 }; /* Current maximum is 3 GPUs per 1 NPU */ + const int max_gpus = 6 / links_per_gpu; + struct npu2 *npu2_phb = phb_to_npu2_nvlink(npuphb); + const u32 npuph = npuphb->dt_node->phandle; + int i, gpuid, first = max_gpus, last = 0; + + /* Find the indexes of GPUs connected to this NPU */ + for (i = 0; i < npu2_phb->total_devices; ++i) { + gpuid = gpu_slot_to_num(npu2_phb->devices[i].nvlink.slot_label); + if (gpuid < 0) + continue; + if (gpuid > last) + last = gpuid; + if (gpuid < first) + first = gpuid; + } + + /* Either no "GPUx" slots found or they are not consecutive, abort */ + if (!last || last + 1 - first > max_gpus) + return; + + /* Collect GPU device nodes, sorted by an index from "GPUn" */ + for (i = 0; i < npu2_phb->total_devices; ++i) { + gpuid = gpu_slot_to_num(npu2_phb->devices[i].nvlink.slot_label); + g[gpuid - first] = npu2_phb->devices[i].nvlink.pd->dn; + } + + /* + * Store interconnect phandles in the device tree. + * The mapping is from Witherspoon_Design_Workbook_v1.7_19June2018.pdf, + * pages 39 (Sequoia), 40 (Redbud): + * Figure 16: NVLink wiring diagram for planar with 6 GPUs + * Figure 17: NVLink wiring diagram for planar with 4 GPUs + */ + switch (last + 1 - first) { + case 2: /* Redbud */ + dt_add_property_cells(g[0], "ibm,nvlink-peers", + g[1]->phandle, npuph, + g[1]->phandle, npuph, + g[1]->phandle, npuph); + dt_add_property_cells(g[1], "ibm,nvlink-peers", + g[0]->phandle, npuph, + g[0]->phandle, npuph, + g[0]->phandle, npuph); + break; + case 3: /* Sequoia */ + dt_add_property_cells(g[0], "ibm,nvlink-peers", + g[1]->phandle, npuph, + g[2]->phandle, g[2]->phandle, + g[1]->phandle, npuph); + dt_add_property_cells(g[1], "ibm,nvlink-peers", + g[0]->phandle, npuph, + g[2]->phandle, g[2]->phandle, + g[0]->phandle, npuph); + dt_add_property_cells(g[2], "ibm,nvlink-peers", + g[1]->phandle, g[0]->phandle, + g[1]->phandle, npuph, + g[0]->phandle, npuph); + break; + default: + prlog(PR_NOTICE, "Failed to detect the exact platform\n"); + break; + } +} + static void npu2_phb_final_fixup(struct phb *phb) { int links_per_gpu = 0; @@ -746,6 +831,8 @@ static void npu2_phb_final_fixup(struct phb *phb) pci_walk_dev(phb, NULL, npu2_links_per_gpu, &links_per_gpu); dt_for_each_compatible(dt_root, np, "ibm,power9-npu") npu2_phb_fixup_scominit(np, links_per_gpu); + + npu2_phb_nvlink_dt(phb, links_per_gpu); } static void npu2_init_ioda_cache(struct npu2 *p)