From patchwork Tue May 24 11:22:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Munsie X-Patchwork-Id: 625582 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3rDY2w1BTLz9t4F for ; Tue, 24 May 2016 21:24:24 +1000 (AEST) Received: from ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3rDY2w0bXpzDqJL for ; Tue, 24 May 2016 21:24:24 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from e23smtp08.au.ibm.com (e23smtp08.au.ibm.com [202.81.31.141]) (using TLSv1.2 with cipher CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3rDY2r1f1YzDqHh for ; Tue, 24 May 2016 21:24:20 +1000 (AEST) Received: from localhost by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 24 May 2016 21:24:19 +1000 Received: from d23dlp03.au.ibm.com (202.81.31.214) by e23smtp08.au.ibm.com (202.81.31.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 24 May 2016 21:24:18 +1000 X-IBM-Helo: d23dlp03.au.ibm.com X-IBM-MailFrom: imunsie@au1.ibm.com X-IBM-RcptTo: skiboot@lists.ozlabs.org Received: from d23relay09.au.ibm.com (d23relay09.au.ibm.com [9.185.63.181]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id 6CD1F3578056 for ; Tue, 24 May 2016 21:24:17 +1000 (EST) Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97]) by d23relay09.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u4OBO9qa35455150 for ; Tue, 24 May 2016 21:24:17 +1000 Received: from d23av03.au.ibm.com (localhost [127.0.0.1]) by d23av03.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u4OBNivf001849 for ; Tue, 24 May 2016 21:23:45 +1000 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.192.253.14]) by d23av03.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u4OBNhKc001170; Tue, 24 May 2016 21:23:44 +1000 Received: from x230.au.ibm.com (unknown [9.192.207.124]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B558DA01D3; Tue, 24 May 2016 21:23:17 +1000 (AEST) From: "Ian Munsie" To: skiboot Date: Tue, 24 May 2016 21:22:53 +1000 Message-Id: <1464088973-16388-2-git-send-email-imunsie@au.ibm.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1464088973-16388-1-git-send-email-imunsie@au.ibm.com> References: <1464088973-16388-1-git-send-email-imunsie@au.ibm.com> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16052411-0029-0000-0000-000045CA7EC7 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Subject: [Skiboot] [PATCH 2/2] phb3: Add support for CAPP DMA mode X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Ian Munsie The XSL used in the Mellanox CX4 card uses a DMA mode of CAPI, which requires a few registers configured specially. In addition to enabling the mode, - The CAPP only owns some of the PHB read buffers, and must be configured to use the correct ones, and the self-snoop configured for the same ones. - The tve needs to be configured to allow the card to access all kernel memory as it uses DMA accesses to read the scheduled process area from the kernel, among other things. These cannot be configured unconditionally, as doing so will break existing CAPI devices that do not use DMA mode. This adds a new mode to the OPAL_PCI_SET_PHB_CAPI_MODE API to enable CAPI in DMA mode. Since the snoop on/off modes write to the capi snoop configuration register, which is configured differently in DMA mode, it uses the redundant bits from the apc master powerbus control register to determine if it should configure the register for DMA mode rather than requiring any more permutations of the mode parameter. Signed-off-by: Ian Munsie Reviewed-by: Frederic Barrat --- hw/phb3.c | 56 ++++++++++++++++++++++++++++++++++++++++++++---------- include/opal-api.h | 1 + 2 files changed, 47 insertions(+), 10 deletions(-) diff --git a/hw/phb3.c b/hw/phb3.c index b2be3e4..3bfa000 100644 --- a/hw/phb3.c +++ b/hw/phb3.c @@ -3324,14 +3324,22 @@ static int64_t phb3_get_diag_data(struct phb *phb, return OPAL_SUCCESS; } -static void phb3_init_capp_regs(struct phb3 *p) +static void phb3_init_capp_regs(struct phb3 *p, bool dma_mode) { uint64_t reg; uint32_t offset; + uint64_t read_buffers = 0; + + if (dma_mode) { + /* In DMA mode, the CAPP only owns some of the PHB read buffers */ + read_buffers = 0x1; + } offset = PHB3_CAPP_REG_OFFSET(p); xscom_read(p->chip_id, APC_MASTER_PB_CTRL + offset, ®); + reg &= ~PPC_BITMASK(10, 11); reg |= PPC_BIT(3); + reg |= read_buffers << PPC_BITLSHIFT(11); xscom_write(p->chip_id, APC_MASTER_PB_CTRL + offset, reg); /* Dynamically workout which PHB to connect to port 0 of the CAPP. @@ -3378,7 +3386,10 @@ static void phb3_init_capp_regs(struct phb3 *p) xscom_write(p->chip_id, FLUSH_UOP_CONFIG1 + offset, 0xB188280728000000); xscom_write(p->chip_id, FLUSH_UOP_CONFIG2 + offset, 0xB188400F00000000); - xscom_write(p->chip_id, SNOOP_CAPI_CONFIG + offset, 0xA1F0000000000000); + + reg = 0xA1F0000000000000; + reg |= read_buffers << PPC_BITLSHIFT(39); + xscom_write(p->chip_id, SNOOP_CAPI_CONFIG + offset, reg); } /* override some inits with CAPI defaults */ @@ -3395,7 +3406,7 @@ static void phb3_init_capp_errors(struct phb3 *p) #define PE_REG_OFFSET(p) \ ((PHB3_IS_NAPLES(p) && (p)->index) ? 0x40 : 0x0) -static int64_t enable_capi_mode(struct phb3 *p, uint64_t pe_number) +static int64_t enable_capi_mode(struct phb3 *p, uint64_t pe_number, bool dma_mode) { uint64_t reg; int i; @@ -3417,7 +3428,12 @@ static int64_t enable_capi_mode(struct phb3 *p, uint64_t pe_number) return OPAL_HARDWARE; } - xscom_write(p->chip_id, p->spci_xscom + 0x3, 0x8000000000000000ull); + /* pb aib capp enable */ + reg = PPC_BIT(0); /* capp enable */ + if (dma_mode) + reg |= PPC_BIT(1); /* capp dma mode */ + xscom_write(p->chip_id, p->spci_xscom + 0x3, reg); + /* FIXME security timer bar xscom_write(p->chip_id, p->spci_xscom + 0x4, 0x8000000000000000ull); */ @@ -3456,8 +3472,16 @@ static int64_t enable_capi_mode(struct phb3 *p, uint64_t pe_number) /* set tve no translate mode allow mmio window */ memset(p->tve_cache, 0x0, sizeof(p->tve_cache)); - /* Allow address range 0x0002000000000000: 0x0002FFFFFFFFFFF */ - p->tve_cache[pe_number * 2] = 0x000000FFFFFF0a00ULL; + if (dma_mode) { + /* + * CAPP DMA mode needs access to all of memory, set address + * range to 0x0000000000000000: 0x0002FFFFFFFFFFF + */ + p->tve_cache[pe_number * 2] = 0x000000FFFFFF0200ULL; + } else { + /* Allow address range 0x0002000000000000: 0x0002FFFFFFFFFFF */ + p->tve_cache[pe_number * 2] = 0x000000FFFFFF0a00ULL; + } phb3_ioda_sel(p, IODA2_TBL_TVT, 0, true); for (i = 0; i < ARRAY_SIZE(p->tve_cache); i++) @@ -3484,7 +3508,7 @@ static int64_t enable_capi_mode(struct phb3 *p, uint64_t pe_number) phb3_init_capp_errors(p); - phb3_init_capp_regs(p); + phb3_init_capp_regs(p, dma_mode); if (!chiptod_capp_timebase_sync(p)) { PHBERR(p, "CAPP: Failed to sync timebase\n"); @@ -3500,6 +3524,7 @@ static int64_t phb3_set_capi_mode(struct phb *phb, uint64_t mode, struct phb3 *p = phb_to_phb3(phb); struct proc_chip *chip = get_chip(p->chip_id); uint64_t reg; + uint64_t read_buffers; uint32_t offset; u8 mask; @@ -3545,7 +3570,10 @@ static int64_t phb3_set_capi_mode(struct phb *phb, uint64_t mode, return OPAL_UNSUPPORTED; case OPAL_PHB_CAPI_MODE_CAPI: - return enable_capi_mode(p, pe_number); + return enable_capi_mode(p, pe_number, false); + + case OPAL_PHB_CAPI_MODE_DMA: + return enable_capi_mode(p, pe_number, true); case OPAL_PHB_CAPI_MODE_SNOOP_OFF: xscom_write(p->chip_id, SNOOP_CAPI_CONFIG + offset, @@ -3555,8 +3583,16 @@ static int64_t phb3_set_capi_mode(struct phb *phb, uint64_t mode, case OPAL_PHB_CAPI_MODE_SNOOP_ON: xscom_write(p->chip_id, CAPP_ERR_STATUS_CTRL + offset, 0x0000000000000000); - xscom_write(p->chip_id, SNOOP_CAPI_CONFIG + offset, - 0xA1F0000000000000); + /* + * Make sure the PHB read buffers being snooped match those + * being used so we don't need another mode to set SNOOP+DMA + */ + xscom_read(p->chip_id, APC_MASTER_PB_CTRL + offset, ®); + read_buffers = (reg >> PPC_BITLSHIFT(11)) & 0x3; + reg = 0xA1F0000000000000; + reg |= read_buffers << PPC_BITLSHIFT(39); + xscom_write(p->chip_id, SNOOP_CAPI_CONFIG + offset, reg); + return OPAL_SUCCESS; } diff --git a/include/opal-api.h b/include/opal-api.h index 0b7b0bb..e116207 100644 --- a/include/opal-api.h +++ b/include/opal-api.h @@ -951,6 +951,7 @@ enum { OPAL_PHB_CAPI_MODE_CAPI = 1, OPAL_PHB_CAPI_MODE_SNOOP_OFF = 2, OPAL_PHB_CAPI_MODE_SNOOP_ON = 3, + OPAL_PHB_CAPI_MODE_DMA = 4, }; /* CAPI feature flags (in device-tree) */