From patchwork Tue Jun 25 23:48:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver O'Halloran X-Patchwork-Id: 1122433 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45YNCq0WW6z9s3Z for ; Wed, 26 Jun 2019 09:48:55 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nwXEKTUL"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 45YNCp6VXlzDqPC for ; Wed, 26 Jun 2019 09:48:54 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::441; helo=mail-pf1-x441.google.com; envelope-from=oohall@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nwXEKTUL"; dkim-atps=neutral Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 45YNCj6vntzDqHw for ; Wed, 26 Jun 2019 09:48:49 +1000 (AEST) Received: by mail-pf1-x441.google.com with SMTP id 19so241930pfa.4 for ; Tue, 25 Jun 2019 16:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=8t0V9rpNrFYhwXqyprofmz2GOAVjEGkoQpjsxuvSj2c=; b=nwXEKTULZD80wvR4YF/cqSbWj/2EUWjSAHE5416ztTKK8KixDXHjQcbGAGNz+mNIOL DdnqO5p/7tzMPxTtAxATeb45Ghubk+PjofXYJb3Kdw63vu3bZ2xuYJbpxD2MaMzDJMLB Gn3aiFZw4R6np8vwYrDfYyToZY0dWVxc+esb0cAUt5IPmMXGRP0l4/yGi2p+13WX9oV6 YIlQVAZHRtPKZknrAgvF3qhhcfgsrc+5o7c3e8jhwkBRr78oQ/9UfUvHMIu5Kq35Dkd9 rI1BxGIs0KIGAXCoEumeJWHI7k0Wzt3fqqA9bEYMKmVqL7638gx/MTnSVWEYkATGzmPA yS7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=8t0V9rpNrFYhwXqyprofmz2GOAVjEGkoQpjsxuvSj2c=; b=XlUqe9SzGBeyxlSRS2538Di+wLED1FwvirWuFvfjuiziWQDESKEyH93Xah7XZIagoO Cjd04yTcCe9DbWC3/Mdpa3Nx+Qw1rnwlAqp28+s2XrLU5oFAsuAdjtf/mCknsVw/4jzm FXbrAx4YJzOPm310GDj2cRuN0bAJmszknevCDf9FXIXcliVoMMtXOWn6l+2v8XQCN1MX zcfIDPjH3Q7oQ1upCONljCQCSeP8UNzj4/555EQlcARFStR5nvAOWwx6h7GOT1QWi11B 1Y5WWx/YiLIPWWhxIYqi2vZKi4CEitx+M8Koeuy7cD3Ik0WoCbu473RqxV9/+N6IIfKp NgBg== X-Gm-Message-State: APjAAAU2f+yeyFD+UrnJlOC/f9OU3ukwVK1RkSV+jkeWO9+mKtsDs5EB Tv6UdWYLR7JpM6c9m+PIDF6g/w6M X-Google-Smtp-Source: APXvYqw9paWzM9A4j8nAizs/6JxgsGS+jTctlolPNvZmNsXDapW/kjpSuPosG/qLh/tH730BGDslyw== X-Received: by 2002:a63:1316:: with SMTP id i22mr43053380pgl.274.1561506524262; Tue, 25 Jun 2019 16:48:44 -0700 (PDT) Received: from wafer.ozlabs.ibm.com.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id t24sm16007638pfh.113.2019.06.25.16.48.42 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 25 Jun 2019 16:48:43 -0700 (PDT) From: Oliver O'Halloran To: skiboot@lists.ozlabs.org Date: Wed, 26 Jun 2019 09:48:33 +1000 Message-Id: <20190625234834.8199-1-oohall@gmail.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Subject: [Skiboot] [PATCH 1/2] pci: Make the pci-eeh-verbose nvram option generic X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" We currently have the "pci-eeh-verbose" NVRAM flag that causes phb4 to print a register dump when it detects the PHB has been fenced. This is useful for debugging most EEH issues since the kernel may not be ready to handle EEH events when the problem is first detected. There's no real reason this needs to be specific to PHB4 so this patch moves the nvram flag handling into the generic init path (along with the pcie_max_link_speed flag) so we can add a similar function for PHB3. Signed-off-by: Oliver O'Halloran --- core/init.c | 5 +++++ hw/phb4.c | 10 +++------- include/pci.h | 3 +++ 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/core/init.c b/core/init.c index 3db9df314292..cde1dd3d367d 100644 --- a/core/init.c +++ b/core/init.c @@ -58,6 +58,7 @@ enum proc_gen proc_gen; unsigned int pcie_max_link_speed; +bool verbose_eeh; static uint64_t kernel_entry; static size_t kernel_size; @@ -866,6 +867,10 @@ static void pci_nvram_init(void) { const char *nvram_speed; + verbose_eeh = nvram_query_eq_safe("pci-eeh-verbose", "true"); + if (verbose_eeh) + prlog(PR_INFO, "PHB: Verbose EEH enabled\n"); + pcie_max_link_speed = 0; nvram_speed = nvram_query_dangerous("pcie-max-link-speed"); diff --git a/hw/phb4.c b/hw/phb4.c index 9a38dc7525ef..20809c878418 100644 --- a/hw/phb4.c +++ b/hw/phb4.c @@ -151,7 +151,6 @@ static void phb4_init_hw(struct phb4 *p); #define PHB4_CAN_STORE_EOI(p) XIVE_STORE_EOI_ENABLED -static bool verbose_eeh; static bool pci_tracing; static bool pci_eeh_mmio; static bool pci_retry_all; @@ -5934,6 +5933,9 @@ static void phb4_probe_pbcq(struct dt_node *pbcq) uint32_t nest_base, pci_base, pec_index; struct dt_node *stk; + /* REMOVEME: force this for now until we stabalise PCIe */ + verbose_eeh = 1; + nest_base = dt_get_address(pbcq, 0, NULL); pci_base = dt_get_address(pbcq, 1, NULL); pec_index = dt_prop_get_u32(pbcq, "ibm,pec-index"); @@ -5949,12 +5951,6 @@ void probe_phb4(void) struct dt_node *np; const char *s; - verbose_eeh = nvram_query_eq_safe("pci-eeh-verbose", "true"); - /* REMOVEME: force this for now until we stabalise PCIe */ - verbose_eeh = 1; - if (verbose_eeh) - prlog(PR_INFO, "PHB4: Verbose EEH enabled\n"); - pci_tracing = nvram_query_eq_safe("pci-tracing", "true"); pci_eeh_mmio = !nvram_query_eq_dangerous("pci-eeh-mmio", "disabled"); pci_retry_all = nvram_query_eq_dangerous("pci-retry-all", "true"); diff --git a/include/pci.h b/include/pci.h index 2b7a3c2893d5..c10d79418e70 100644 --- a/include/pci.h +++ b/include/pci.h @@ -368,6 +368,9 @@ enum phb_type { phb_type_npu_v2_opencapi, }; + +extern bool verbose_eeh; + struct phb { struct dt_node *dt_node; int opal_id; From patchwork Tue Jun 25 23:48:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver O'Halloran X-Patchwork-Id: 1122434 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45YND82yT4z9s3Z for ; Wed, 26 Jun 2019 09:49:12 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="RY8RTXZE"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 45YND76RkVzDqVv for ; Wed, 26 Jun 2019 09:49:11 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=oohall@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="RY8RTXZE"; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 45YNCl1PX4zDqHw for ; Wed, 26 Jun 2019 09:48:50 +1000 (AEST) Received: by mail-pf1-x442.google.com with SMTP id c85so254706pfc.1 for ; Tue, 25 Jun 2019 16:48:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NjfUQLSzLBRheTjxnTs4uSWbh6sUQqxm4sgO3JInQmc=; b=RY8RTXZEfI5NSvKvm+EwGXvUZATyC1i5v2zSBN+yjqBlIN2mVWOi8uvgRR3g6RbZgr nQ7MreuNuZ2RCV9AU1VCDdh9FHFtZ6yPZhMsSM26KwXELZVLLA0dK730idBZSPiBa19T YFi3wzfRKjK2e901deqLv/jvJg8uR0N/y+lMyX3KRA8FCCwr+QJvPEc3IZqq07Y4RJFg HzjqptzVX/r8PbkHp8Uj9rU2AqbXWrjRNSK7NcF98ESoA8vXEM5tCU0ay+cI++Tev8ik pJzL53fJKaYJhu7Ro4PLksWkbd02OX3e0NPm7ZGmEFEskSY1ISjWl2UBVk45HZpaBi5f A8ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NjfUQLSzLBRheTjxnTs4uSWbh6sUQqxm4sgO3JInQmc=; b=Wt7bWQPUZdFHl6zYjYzC9Kb7EQIs4QoKfdIrFhI2MvHt6Q1EfNiGaUj499rd8jQIBG ZRN21/vUlJ8ueCpYZRdKWhYIMOHoV6m6qMHPCxiPnLxQK+pDVbkZQETwV4f1ujW82ctT j3iwmQ1hLjrkGx5+sdAoUF5pLEt0jCx9ptBeYj05QjivHLRl8FXMtb8Vlv8ZcIde/Lud gpQaAx2BeEQrvYCexhl/xMwtWnIawNEOkGX3vwODJwUZYjAM32X2hJt0bQZvGmECLqtp qdOTv8z+FimoASs1UYVwSfjNbyBVmBbu56wc2k66vgTa6Lg21pdMzrofTkroK0xtDu48 cBDg== X-Gm-Message-State: APjAAAWsNPOPoiv7s21HQwiA2cVYKAb8uo/K1k2q1N3ngkhjM04y4wwn 89KQ5h8Lyh1rHv0eS4P0HvlAVkqK X-Google-Smtp-Source: APXvYqyEf9QXejIO60bFepASXvPxluug3nsFdb0lqWEgutKeyq2mukUYgIWWW5iqUDwQxJ+wutq+xg== X-Received: by 2002:a65:56c5:: with SMTP id w5mr40729273pgs.434.1561506528656; Tue, 25 Jun 2019 16:48:48 -0700 (PDT) Received: from wafer.ozlabs.ibm.com.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id t24sm16007638pfh.113.2019.06.25.16.48.46 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 25 Jun 2019 16:48:48 -0700 (PDT) From: Oliver O'Halloran To: skiboot@lists.ozlabs.org Date: Wed, 26 Jun 2019 09:48:34 +1000 Message-Id: <20190625234834.8199-2-oohall@gmail.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190625234834.8199-1-oohall@gmail.com> References: <20190625234834.8199-1-oohall@gmail.com> MIME-Version: 1.0 Subject: [Skiboot] [PATCH 2/2] hw/phb3: Add verbose EEH output X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Add support for the pci-eeh-verbose NVRAM flag on PHB3. We've had this on PHB4 since forever and it has proven very useful when debugging EEH issues. When testing changes to the Linux kernel's EEH implementation it's fairly common for the kernel to crash before printing the EEH log so it's helpful to have it in the OPAL log where it can be dumped from XMON. Note that unlike PHB4 we do not enable verbose mode by default. The nvram option must be used to explicitly enable it. Signed-off-by: Oliver O'Halloran --- hw/phb3.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-) diff --git a/hw/phb3.c b/hw/phb3.c index 3042c3e69cbf..ecc298055803 100644 --- a/hw/phb3.c +++ b/hw/phb3.c @@ -63,6 +63,9 @@ static inline void phb3_ioda_sel(struct phb3 *p, uint32_t table, SETFIELD(PHB_IODA_AD_TADR, 0ul, addr)); } +static void phb3_eeh_dump_regs(struct phb3 *p, + struct OpalIoPhb3ErrorData *regs); + /* Check if AIB is fenced via PBCQ NFIR */ static bool phb3_fenced(struct phb3 *p) { @@ -72,6 +75,8 @@ static bool phb3_fenced(struct phb3 *p) xscom_read(p->chip_id, p->pe_xscom + 0x0, &nfir); if (nfir & PPC_BIT(16)) { p->flags |= PHB3_AIB_FENCED; + + phb3_eeh_dump_regs(p, NULL); return true; } return false; @@ -1787,6 +1792,92 @@ static void phb3_read_phb_status(struct phb3 *p, } } +static void phb3_eeh_dump_regs(struct phb3 *p, struct OpalIoPhb3ErrorData *regs) +{ + struct OpalIoPhb3ErrorData *s; + unsigned int i; + + if (!verbose_eeh) + return; + + if (!regs) { + s = zalloc(sizeof(struct OpalIoPhb3ErrorData)); + if (!s) { + PHBERR(p, "Failed to allocate error info !\n"); + return; + } + + phb3_read_phb_status(p, s); + } else { + s = regs; + } + + PHBERR(p, "Error detected!\n"); + + PHBERR(p, " portStatusReg = %08x\n", s->portStatusReg); + PHBERR(p, " rootCmplxStatus = %08x\n", s->rootCmplxStatus); + PHBERR(p, " busAgentStatus = %08x\n", s->busAgentStatus); + + PHBERR(p, " errorClass = %016llx\n", s->errorClass); + PHBERR(p, " correlator = %016llx\n", s->correlator); + + PHBERR(p, " brdgCtl = %08x\n", s->brdgCtl); + PHBERR(p, " deviceStatus = %08x\n", s->deviceStatus); + PHBERR(p, " slotStatus = %08x\n", s->slotStatus); + PHBERR(p, " linkStatus = %08x\n", s->linkStatus); + PHBERR(p, " devCmdStatus = %08x\n", s->devCmdStatus); + PHBERR(p, " devSecStatus = %08x\n", s->devSecStatus); + PHBERR(p, " rootErrorStatus = %08x\n", s->rootErrorStatus); + PHBERR(p, " corrErrorStatus = %08x\n", s->corrErrorStatus); + PHBERR(p, " uncorrErrorStatus = %08x\n", s->uncorrErrorStatus); + + /* Byte swap TLP headers so they are the same as the PCIe spec */ + PHBERR(p, " tlpHdr1 = %08x\n", bswap_32(s->tlpHdr1)); + PHBERR(p, " tlpHdr2 = %08x\n", bswap_32(s->tlpHdr2)); + PHBERR(p, " tlpHdr3 = %08x\n", bswap_32(s->tlpHdr3)); + PHBERR(p, " tlpHdr4 = %08x\n", bswap_32(s->tlpHdr4)); + PHBERR(p, " sourceId = %08x\n", s->sourceId); + + PHBERR(p, " nFir = %016llx\n", s->nFir); + PHBERR(p, " nFirMask = %016llx\n", s->nFirMask); + PHBERR(p, " nFirWOF = %016llx\n", s->nFirWOF); + PHBERR(p, " phbPlssr = %016llx\n", s->phbPlssr); + PHBERR(p, " phbCsr = %016llx\n", s->phbCsr); + PHBERR(p, " lemFir = %016llx\n", s->lemFir); + PHBERR(p, " lemErrorMask = %016llx\n", s->lemErrorMask); + PHBERR(p, " lemWOF = %016llx\n", s->lemWOF); + + PHBERR(p, " phbErrorStatus = %016llx\n", s->phbErrorStatus); + PHBERR(p, " phbFirstErrorStatus = %016llx\n", s->phbFirstErrorStatus); + PHBERR(p, " phbErrorLog0 = %016llx\n", s->phbErrorLog0); + PHBERR(p, " phbErrorLog1 = %016llx\n", s->phbErrorLog1); + + PHBERR(p, " mmioErrorStatus = %016llx\n", s->mmioErrorStatus); + PHBERR(p, "mmioFirstErrorStatus = %016llx\n", s->mmioFirstErrorStatus); + PHBERR(p, " mmioErrorLog0 = %016llx\n", s->mmioErrorLog0); + PHBERR(p, " mmioErrorLog1 = %016llx\n", s->mmioErrorLog1); + + PHBERR(p, " dma0ErrorStatus = %016llx\n", s->dma0ErrorStatus); + PHBERR(p, "dma0FirstErrorStatus = %016llx\n", s->dma0FirstErrorStatus); + PHBERR(p, " dma0ErrorLog0 = %016llx\n", s->dma0ErrorLog0); + PHBERR(p, " dma0ErrorLog1 = %016llx\n", s->dma0ErrorLog1); + + PHBERR(p, " dma1ErrorStatus = %016llx\n", s->dma1ErrorStatus); + PHBERR(p, "dma1FirstErrorStatus = %016llx\n", s->dma1FirstErrorStatus); + PHBERR(p, " dma1ErrorLog0 = %016llx\n", s->dma1ErrorLog0); + PHBERR(p, " dma1ErrorLog1 = %016llx\n", s->dma1ErrorLog1); + + for (i = 0; i < OPAL_PHB3_NUM_PEST_REGS; i++) { + if (!s->pestA[i] && !s->pestB[i]) + continue; + PHBERR(p, " PEST[%03x] = %016llx %016llx\n", + i, s->pestA[i], s->pestB[i]); + } + + if (s != regs) + free(s); +} + static int64_t phb3_msi_get_xive(struct irq_source *is, uint32_t isn, uint16_t *server, uint8_t *prio) { @@ -3387,6 +3478,7 @@ static int64_t phb3_get_diag_data(struct phb *phb, { struct phb3 *p = phb_to_phb3(phb); struct OpalIoPhb3ErrorData *data = diag_buffer; + bool fenced; if (diag_buffer_len < sizeof(struct OpalIoPhb3ErrorData)) return OPAL_PARAMETER; @@ -3397,9 +3489,12 @@ static int64_t phb3_get_diag_data(struct phb *phb, * Dummy check for fence so that phb3_read_phb_status knows * whether to use ASB or AIB */ - phb3_fenced(p); + fenced = phb3_fenced(p); phb3_read_phb_status(p, data); + if (!fenced) + phb3_eeh_dump_regs(p, data); + /* * We're running to here probably because of errors * (INF class). For that case, we need clear the error