From patchwork Tue Oct 29 02:11:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Niethe X-Patchwork-Id: 1185800 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 472FTH02BRz9sPV for ; Tue, 29 Oct 2019 13:12:02 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="I+hZHg7c"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 472FTG2MJczDwND for ; Tue, 29 Oct 2019 13:12:02 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=jniethe5@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="I+hZHg7c"; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 472FT31GsZzDsNc for ; Tue, 29 Oct 2019 13:11:50 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id b25so8345238pfi.13 for ; Mon, 28 Oct 2019 19:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i9wkK+6kxgPBxGGqzq6MQigHvA+oXKB9h/YN19OvJO4=; b=I+hZHg7crYZ6gzqSrFodC3jH2US86KGPozVVUeUdsaYDvkcY0LiSMmP87UcT89hPTL kMoEEKSVC8yrdaAo95Ol2d380wIWgtlXA4jKsqZO+jD4Z+uxjaBehEpWdNVXL/dHn+/e cJY1ANNk03cxdh2DGNr2WHbPyRKL7wSb0f5tjPWgcrS5n5TZf6h5tmvWFtL/n0p7O/Z8 v5DvmY33nhFuVCPKW2Pr0KhV8bEcGBYW2ZrK4dcu0PBrqRZDvsdCbhacz/lNsBx25R1v ZigYTSnx86QugGyMvBEoC/iNec0r3iqaAC+OYwUFTGY/HRiYS2WK0/X0lS64Q8bG0Nmf 3Tpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i9wkK+6kxgPBxGGqzq6MQigHvA+oXKB9h/YN19OvJO4=; b=nk50TmEAawJG8oSoEqzjdMMHnkLID35TDNYosVMFzpef2chLXYoH1NDznxczlggnS+ RuZC7T1k6J46IT2h3vM7YS4I9Iwbs4XGgE1RutldGGmbZEh8rH+rJwMSgjUglw+UXIkk Sn+pvEkXjiogNgDo4PP4YaePuroQJro+Jnqydx+HEjKfJpXmNcHLVtxzfN9JTVYC+807 12Du2iLkqQZ6F8NMUOLZNpL8LQdxNhwJzMmYjyNEMlIkBhAOqG3ZkoDgcv5tqC0GAoop BwSG6pMbLt+nT6s4qwrnXOinFo28pJMQoLRFwHjmlIJ+Uejk/omMtTYFRWkR692lcxhG tpQQ== X-Gm-Message-State: APjAAAWoyaBLN0CStU5SN9Qt+2e5RPcABAO7GhJviy3UN8pUhO8+gZ0D k51ie4CaX5VdRIWI/R7rxnemba3Q X-Google-Smtp-Source: APXvYqygL9exQTgSFBD+lyNgN0X6mMfTMQbXAGivfqoGp112E3Fp5+mcrQLqFbvCFZFyYDAydKu90w== X-Received: by 2002:a63:fa4a:: with SMTP id g10mr23569797pgk.432.1572315108373; Mon, 28 Oct 2019 19:11:48 -0700 (PDT) Received: from sol.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id w14sm13710999pge.56.2019.10.28.19.11.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2019 19:11:47 -0700 (PDT) From: Jordan Niethe To: skiboot@lists.ozlabs.org Date: Tue, 29 Oct 2019 13:11:04 +1100 Message-Id: <20191029021106.29396-2-jniethe5@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191029021106.29396-1-jniethe5@gmail.com> References: <20191029021106.29396-1-jniethe5@gmail.com> MIME-Version: 1.0 Subject: [Skiboot] [RFC PATCH v2 1/3] core/pci: Add functions for iterating PCI devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Currently to visit all the devices under a PHB/PCI device the recursive pci_walk_dev() function is used. This does not make the state of where the walk is up to accessible which limits how it can be used. For example we might like to visit each device and perform tasks that require waiting. Rather than waiting in skiboot we would like to be able to return a timeout value to the caller. This requires keeping the state of the walk so that we can return to the device after waiting. To do this give each pci_device a member to record if it has been visited or not and add a member to each PHB to keep track of the current PCI device in the walk. Signed-off-by: Jordan Niethe --- v2: No changes core/pci.c | 37 +++++++++++++++++++++++++++++++++++++ include/pci.h | 6 ++++++ 2 files changed, 43 insertions(+) diff --git a/core/pci.c b/core/pci.c index 9ee70f4fddc5..547c8e32272b 100644 --- a/core/pci.c +++ b/core/pci.c @@ -1889,6 +1889,43 @@ struct pci_device *pci_find_dev(struct phb *phb, uint16_t bdfn) return pci_walk_dev(phb, NULL, __pci_find_dev, &bdfn); } +static int __pci_unsee_device(struct phb *phb, + struct pci_device *pd, + void *data __unused) +{ + if (!phb || !pd) { + return 0; + } + + pd->seen = false; + return 0; +} + +void pci_device_iter_reset(struct phb *phb, struct pci_device *pd) +{ + pci_walk_dev(phb, pd, __pci_unsee_device, NULL); +} + +static int __pci_device_iter_next(struct phb *phb, + struct pci_device *pd, + void *data __unused) +{ + if (!phb || !pd) { + return 0; + } + + if (pd->seen) + return 0; + + pd->seen = true; + return 1; +} + +struct pci_device *pci_device_iter_next(struct phb *phb, struct pci_device *pd) +{ + return pci_walk_dev(phb, pd, __pci_device_iter_next, NULL); +} + static int __pci_restore_bridge_buses(struct phb *phb, struct pci_device *pd, void *data __unused) diff --git a/include/pci.h b/include/pci.h index dcd354a7cfef..9672a2d342f6 100644 --- a/include/pci.h +++ b/include/pci.h @@ -75,6 +75,7 @@ struct pci_device { bool is_bridge; bool is_multifunction; bool is_vf; + bool seen; uint8_t dev_type; /* PCIE */ uint8_t primary_bus; uint8_t secondary_bus; @@ -378,6 +379,9 @@ struct phb { /* PCI-X only slot info, for PCI-E this is in the RC bridge */ struct pci_slot *slot; + /* Used for iteration */ + struct pci_device *current_pd; + /* Base location code used to generate the children one */ const char *base_loc_code; @@ -451,6 +455,8 @@ extern void pci_init_capabilities(struct phb *phb, struct pci_device *pd); extern bool pci_wait_crs(struct phb *phb, uint16_t bdfn, uint32_t *out_vdid); extern void pci_restore_slot_bus_configs(struct pci_slot *slot); extern void pci_device_init(struct phb *phb, struct pci_device *pd); +extern void pci_device_iter_reset(struct phb *phb, struct pci_device *pd); +extern struct pci_device *pci_device_iter_next(struct phb *phb, struct pci_device *pd); extern struct pci_device *pci_walk_dev(struct phb *phb, struct pci_device *pd, int (*cb)(struct phb *, From patchwork Tue Oct 29 02:11:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Niethe X-Patchwork-Id: 1185802 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 472FV06G7qz9sP3 for ; Tue, 29 Oct 2019 13:12:40 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NsoirtF8"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 472FV05LDQzDvJ2 for ; Tue, 29 Oct 2019 13:12:40 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::441; helo=mail-pf1-x441.google.com; envelope-from=jniethe5@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NsoirtF8"; dkim-atps=neutral Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 472FT4663DzDsFk for ; Tue, 29 Oct 2019 13:11:52 +1100 (AEDT) Received: by mail-pf1-x441.google.com with SMTP id u9so3639559pfn.4 for ; Mon, 28 Oct 2019 19:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=akKUTKx2uamws3EHVVsYbujnk4eVaTnYeI0dsDcjQ3c=; b=NsoirtF8mOSy3Ql+I/9j7T5pcDrEDWf7+8rphPUZ5Zldle97U2eaL2bWqVERJ/14TO FXf5lIHp29SyyXbWXzkgZ9eMjcykKxWgHqo7Hak/oPAeDv0dekxZR+6tBd7m+Spyqy4I t/nbCzvtAVEM7cpuQFdNTutAERsbQ3yT80XcRp6MdHMi/t1INrFcDlZGmTLitc5Sie1r XILggDO83KXx4gKxb7Y5doYIg0NOnRPERN/biUhekidUh9ZNnU8f3u1gONMBUI+7UmZk pFwRcZvsTSgO0VOi6mYPXcXepdqkqollr3/BrICTTjPHQxYMIJ0Rcd7ipCod/5H5hmq4 72rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=akKUTKx2uamws3EHVVsYbujnk4eVaTnYeI0dsDcjQ3c=; b=WqFBy6+lDuxozuPQkDssN2hQQKrT5dKF1JHF+zfo8LH2z2dgWduzxesbcOw6FkHXgN m8xYR2M+xdX6ZPAwSGUBBjzTcbPdMCNtU6uLVIKKpUUa00cK75bHPQ3vcY7aO3zuc46l QR4Hn4u2qZGQ+va1mEiAnKSCWJVPdL6LfJBLgoBc84Kgmd5cQzqBNqNwnEQI6SvzuYTS wohTQxRbGJTmFf6IyREwSIeONseFv+Yz7nwWBvLr16lrVOBayya+SE/J4YhXaJC/oefA WtsO1IDd5o7JMHGLRj8Uw+G0eq4r1a9jjJLURzCR0PFUwD4Xa3tp+DGIuQqx9b+sFzkw G/Xg== X-Gm-Message-State: APjAAAUlBJqwiLjabZdaDYvWyOX8nLacJWU7jcOEc3WxvB5+wH+pGtmX S+6UIPVBtvBMbBN+lvGtxQU6YLVG X-Google-Smtp-Source: APXvYqwdVkvq3uW19SjGV8oSqNyssp9njmGbwr3voB889hbS55sBUoxzozV70qm2nD6pMaJnnqO5WQ== X-Received: by 2002:a62:5258:: with SMTP id g85mr8358544pfb.180.1572315110619; Mon, 28 Oct 2019 19:11:50 -0700 (PDT) Received: from sol.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id w14sm13710999pge.56.2019.10.28.19.11.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2019 19:11:50 -0700 (PDT) From: Jordan Niethe To: skiboot@lists.ozlabs.org Date: Tue, 29 Oct 2019 13:11:05 +1100 Message-Id: <20191029021106.29396-3-jniethe5@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191029021106.29396-1-jniethe5@gmail.com> References: <20191029021106.29396-1-jniethe5@gmail.com> MIME-Version: 1.0 Subject: [Skiboot] [RFC PATCH v2 2/3] core/pcie-slot: Extend slot state machine to restore bus numbers X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" After performing a fundamental reset bus numbers need to be restored. This is done by pci_restore_slot_bus_configs() which is called after the PHB's link is active. Restoring the bus numbers can require waiting for certain delays, which are currently performed using time_wait_ms(). This leads to excessive times spent waiting in skiboot during the OPAL_PCI_RESET opal call. If a device is a behind a switch, it needs to wait for the switch's downstream port link to become active before the buses can be restored. Additionally all devices potentially need to wait for CRS. Slots already have state machine based functions that are used for reset. Add a new state machine function, restore_buses(), with associated states for restoring buses. Call restore_buses() instead of pci_restore_slot_bus_configs(). The actually process for this is not specific to the PHB so we only implement a generic PCIe slot function. pci_restore_slot_bus_configs() is only called by PHB4, so for now only it calls restore_buses(). Signed-off-by: Jordan Niethe --- v2: - Move out from PHB4 and into PCIe generic slot - Changes to debug print messages core/pci-slot.c | 3 ++ core/pci.c | 7 +++ core/pcie-slot.c | 126 +++++++++++++++++++++++++++++++++++++++++++++ hw/phb4.c | 6 +-- include/pci-slot.h | 13 +++++ include/pci.h | 1 + 6 files changed, 153 insertions(+), 3 deletions(-) diff --git a/core/pci-slot.c b/core/pci-slot.c index ffb54cbb7a49..4321f30f98aa 100644 --- a/core/pci-slot.c +++ b/core/pci-slot.c @@ -79,6 +79,9 @@ static int64_t pci_slot_run_sm(struct pci_slot *slot) slot->delay_tgt_tb = 0; switch (slot->state & PCI_SLOT_STATE_MASK) { + case PCI_SLOT_STATE_BUSES: + ret = slot->ops.restore_buses(slot); + break; case PCI_SLOT_STATE_LINK: ret = slot->ops.poll_link(slot); break; diff --git a/core/pci.c b/core/pci.c index 547c8e32272b..732ba68d5d23 100644 --- a/core/pci.c +++ b/core/pci.c @@ -1981,6 +1981,13 @@ void pci_restore_slot_bus_configs(struct pci_slot *slot) slot->phb->ops->device_init, NULL); } +void pci_init_all_devices(struct pci_slot *slot) +{ + if (slot->phb->ops->device_init) + pci_walk_dev(slot->phb, slot->pd, + slot->phb->ops->device_init, NULL); +} + struct pci_cfg_reg_filter *pci_find_cfg_reg_filter(struct pci_device *pd, uint32_t start, uint32_t len) { diff --git a/core/pcie-slot.c b/core/pcie-slot.c index 894fd40513c3..048a8b279c71 100644 --- a/core/pcie-slot.c +++ b/core/pcie-slot.c @@ -262,6 +262,131 @@ static int64_t pcie_slot_set_power_state(struct pci_slot *slot, uint8_t val) return pcie_slot_set_power_state_ext(slot, val, true); } +int64_t pcie_slot_sm_restore_buses(struct pci_slot *slot) +{ + struct pci_device *pd, *parent; + uint32_t link_cap = 0; + uint16_t link_sts = 0; + int32_t ecap = 0; + struct phb *phb; + uint32_t vdid; + int64_t rc; + + phb = slot->phb; + pd = slot->phb->current_pd; + parent = pd ? pd->parent : NULL; + + switch (slot->state) { + case PCI_SLOT_STATE_NORMAL: + case PCI_SLOT_BUSES_START: + PCIE_SLOT_DBG(slot, "BUSES: Starts\n"); + pci_device_iter_reset(slot->phb, NULL); + slot->phb->current_pd = pci_device_iter_next(slot->phb, NULL); + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_START); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + case PCI_SLOT_BUSES_PD_START: + if (slot->phb->current_pd == NULL) { + PCIE_SLOT_DBG(slot, "BUSES: Finished all PCI devices\n"); + pci_slot_set_state(slot, PCI_SLOT_STATE_NORMAL); + pci_init_all_devices(slot); + return OPAL_SUCCESS; + } + + PCIDBG(phb, pd->bdfn, "BUSES: PCI Device Starts\n"); + if (!pd->is_vf && !(pd->bdfn & 7) && pd->parent != NULL && + pd->parent->dev_type == PCIE_TYPE_SWITCH_DNPORT) { + PCIDBG(phb, pd->bdfn, "BUSES: Behind a switch\n"); + slot->retries = PCI_BUS_SWITCH_LINK_RETRIES; + pci_slot_set_state(slot, + PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + } else { + PCIDBG(phb, pd->bdfn, "BUSES: Not behind a switch\n"); + slot->retries = PCI_BUS_CRS_RETRIES; + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_CRS); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + } + case PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK: + PCIDBG(phb, pd->bdfn, "BUSES: Wait for Switch Link\n"); + + if (pci_has_cap(parent, PCI_CFG_CAP_ID_EXP, false)) { + ecap = pci_cap(parent, PCI_CFG_CAP_ID_EXP, false); + pci_cfg_read32(phb, parent->bdfn, + ecap + PCICAP_EXP_LCAP, &link_cap); + } + + if (!(link_cap & PCICAP_EXP_LCAP_DL_ACT_REP)) { + PCIDBG(phb, parent->bdfn, + "BUSES: Parent: No link state reporting\n"); + slot->retries = PCI_BUS_CRS_RETRIES; + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_CRS); + return pci_slot_set_sm_timeout(slot, secs_to_tb(1)); + } + + pci_cfg_read16(phb, parent->bdfn, + ecap + PCICAP_EXP_LSTAT, &link_sts); + + if (link_sts & PCICAP_EXP_LSTAT_DLLL_ACT) { + /* Have to wait 100ms before touch config space */ + PCIDBG(phb, parent->bdfn, "BUSES: Parent: Link active\n"); + slot->retries = PCI_BUS_CRS_RETRIES; + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_CRS); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(100)); + } + + if (slot->retries-- == 0) { + PCIDBG(phb, pd->bdfn, + "BUSES: Timeout waiting downstream link\n"); + pci_slot_set_state(slot, PCI_SLOT_STATE_NORMAL); + return OPAL_HARDWARE; + } + return pci_slot_set_sm_timeout(slot, msecs_to_tb(100)); + case PCI_SLOT_BUSES_PD_WAIT_CRS: + rc = pci_cfg_read32(phb, pd->bdfn, PCI_CFG_VENDOR_ID, &vdid); + + if (rc || vdid == 0xffffffff || vdid == 0x00000000) { + PCIERR(phb, pd->bdfn, "Error reading VENDOR ID\n"); + pci_slot_set_state(slot, PCI_SLOT_STATE_NORMAL); + return OPAL_HARDWARE; + } + + if (vdid == 0xffff0001) { + PCIDBG(phb, pd->bdfn, "BUSES: Got a CRS\n"); + if (slot->retries-- == 0) { + PCIERR(phb, pd->bdfn, "Timeout waiting for CRS\n"); + pci_slot_set_state(slot, PCI_SLOT_STATE_NORMAL); + return OPAL_HARDWARE; + } + return pci_slot_set_sm_timeout(slot, msecs_to_tb(100)); + } + + /* Actual Vendor ID */ + if (slot->retries != PCI_BUS_CRS_RETRIES) + PCIDBG(phb, pd->bdfn, "BUSES: Probe success after CRS\n"); + + /* Make devices below a bridge "re-capture" the bdfn */ + pci_cfg_write32(phb, pd->bdfn, PCI_CFG_VENDOR_ID, vdid); + + if (pd->is_bridge) { + PCIDBG(phb, pd->bdfn, + "BUSES: Device is a bridge. Restoring buses\n"); + pci_cfg_write8(phb, pd->bdfn, PCI_CFG_PRIMARY_BUS, + pd->primary_bus); + pci_cfg_write8(phb, pd->bdfn, PCI_CFG_SECONDARY_BUS, + pd->secondary_bus); + pci_cfg_write8(phb, pd->bdfn, PCI_CFG_SUBORDINATE_BUS, + pd->subordinate_bus); + } + + PCIDBG(phb, pd->bdfn, "BUSES: PCI Device Finished\n"); + slot->phb->current_pd = pci_device_iter_next(slot->phb, NULL); + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_START); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + } + return 0; +} + + static int64_t pcie_slot_sm_poll_link(struct pci_slot *slot) { struct phb *phb = slot->phb; @@ -511,6 +636,7 @@ struct pci_slot *pcie_slot_create(struct phb *phb, struct pci_device *pd) * State machine (SM) based reset stuff. The poll function is always * unified for all cases. */ + slot->ops.restore_buses = pcie_slot_sm_restore_buses; slot->ops.poll_link = pcie_slot_sm_poll_link; slot->ops.hreset = pcie_slot_sm_hreset; slot->ops.freset = pcie_slot_sm_freset; diff --git a/hw/phb4.c b/hw/phb4.c index 3c71427aef2f..fd13abee4099 100644 --- a/hw/phb4.c +++ b/hw/phb4.c @@ -2851,9 +2851,8 @@ static int64_t phb4_poll_link(struct pci_slot *slot) */ PHBERR(p, "LINK: Degraded but no more retries\n"); } - pci_restore_slot_bus_configs(slot); - pci_slot_set_state(slot, PHB4_SLOT_NORMAL); - return OPAL_SUCCESS; + pci_slot_set_state(slot, PCI_SLOT_BUSES_START); + return slot->ops.restore_buses(slot); } PHBERR(p, "LINK: Went down waiting for stabilty\n"); PHBDBG(p, "LINK: DLP train control: 0x%016llx\n", reg); @@ -3431,6 +3430,7 @@ static struct pci_slot *phb4_slot_create(struct phb *phb) * individual platforms. */ slot->ops.prepare_link_change = phb4_prepare_link_change; + slot->ops.restore_buses = pcie_slot_sm_restore_buses; slot->ops.poll_link = phb4_poll_link; slot->ops.hreset = phb4_hreset; slot->ops.freset = phb4_freset; diff --git a/include/pci-slot.h b/include/pci-slot.h index 6e7dc1c06970..353efc7e96e1 100644 --- a/include/pci-slot.h +++ b/include/pci-slot.h @@ -92,6 +92,7 @@ struct pci_slot_ops { /* SM based functions for reset */ void (*prepare_link_change)(struct pci_slot *slot, bool is_up); + int64_t (*restore_buses)(struct pci_slot *slot); int64_t (*poll_link)(struct pci_slot *slot); int64_t (*creset)(struct pci_slot *slot); int64_t (*freset)(struct pci_slot *slot); @@ -129,6 +130,17 @@ struct pci_slot_ops { #define PCI_SLOT_STATE_SPOWER_DONE (PCI_SLOT_STATE_SPOWER + 2) #define PCI_SLOT_STATE_GPRESENCE 0x00000700 #define PCI_SLOT_STATE_GPRESENCE_START (PCI_SLOT_STATE_GPRESENCE + 1) +#define PCI_SLOT_STATE_BUSES 0x00000800 +#define PCI_SLOT_BUSES PCI_SLOT_STATE_BUSES +#define PCI_SLOT_BUSES_START (PCI_SLOT_BUSES + 1) +#define PCI_SLOT_BUSES_PD_START (PCI_SLOT_BUSES + 2) +#define PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK (PCI_SLOT_BUSES + 3) +#define PCI_SLOT_BUSES_PD_WAIT_CRS (PCI_SLOT_BUSES + 4) +#define PCI_SLOT_BUSES_PD_END (PCI_SLOT_BUSES + 5) + +#define PCI_BUS_CRS_RETRIES 40 +#define PCI_BUS_SWITCH_LINK_RETRIES 100 + struct pci_slot { @@ -244,6 +256,7 @@ extern struct pci_slot *pcie_slot_create(struct phb *phb, struct pci_device *pd); extern struct pci_slot *pcie_slot_create_dynamic(struct phb *phb, struct pci_device *pd); +extern int64_t pcie_slot_sm_restore_buses(struct pci_slot *slot); extern void pci_slot_add_dt_properties(struct pci_slot *slot, struct dt_node *np); diff --git a/include/pci.h b/include/pci.h index 9672a2d342f6..9bb82cc30cb4 100644 --- a/include/pci.h +++ b/include/pci.h @@ -454,6 +454,7 @@ extern int64_t pci_find_ecap(struct phb *phb, uint16_t bdfn, uint16_t cap, extern void pci_init_capabilities(struct phb *phb, struct pci_device *pd); extern bool pci_wait_crs(struct phb *phb, uint16_t bdfn, uint32_t *out_vdid); extern void pci_restore_slot_bus_configs(struct pci_slot *slot); +extern void pci_init_all_devices(struct pci_slot *slot); extern void pci_device_init(struct phb *phb, struct pci_device *pd); extern void pci_device_iter_reset(struct phb *phb, struct pci_device *pd); extern struct pci_device *pci_device_iter_next(struct phb *phb, struct pci_device *pd); From patchwork Tue Oct 29 02:11:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Niethe X-Patchwork-Id: 1185803 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 472FVJ1Q0Hz9sP3 for ; Tue, 29 Oct 2019 13:12:56 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CCNnEEEP"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 472FVH6t2YzF07t for ; Tue, 29 Oct 2019 13:12:55 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=jniethe5@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CCNnEEEP"; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 472FT62sFBzDsFk for ; Tue, 29 Oct 2019 13:11:54 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id v4so8365234pff.6 for ; Mon, 28 Oct 2019 19:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=d7pE1f3AO4sjRfLcsm3FyvwfhQ4s1AGUa07bV+q/U/0=; b=CCNnEEEPHRC55QU2pHfLtOg9YgQWH7BlcElHtHKc0KdnH4IEZO3A83w4gXpmL8Q0t9 hCia9wHTGk2BFpuT0k7776UA05mRCgI4vANHxvZIKODZ8e7FpmXfCrMh8wjPI3D1idsS yf9U6cdbLFTPjgGiJfuK/lmNNqafL4zySo27VoIbRZFFc7DSVHigwXYIlVVXeBK12eIi rjCq8KKpOzYFUVZaxWRMYK//BdSRruPRS9fJrMKRjiljjgpVtt9hch4zZYwkoLMv6q8P c8hKAa68uqN6gJVlwAatXYW6ASZKkvs+hogEX13wyZqOEeYh04t/CqddniCxVqXMfvkM ZCAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d7pE1f3AO4sjRfLcsm3FyvwfhQ4s1AGUa07bV+q/U/0=; b=JMH9qjfHm/GBh503SKTYYCgBvo8tXJs8Gi4xKDfOdBlvwM05RIzfJdGAM82YkI4Ecz Zs9l+1ZEb2dh5e/0T0pqExh3LcxX2VhxHAx/+dwBe8X59cxyTMqlN4beou/la4La7RfZ JCeQub8BGXnZS728vaOUAcaN/1HLmd/j7mSOqTGbDHpjPdaIo14BeneGhhlL1S5M5prg 56ka8fLHvOlDr18tU7k9RxHXXvs4zly4K7mJLis2CRETJtIbwOjm+1yWHVzjfdi0atFQ j8rdvsOLEKT4DA6gy91adkRTsGs1RwfWIAOdHTiHOv2FBiqu9WxXLZHWzzpwrQaR6N56 9zug== X-Gm-Message-State: APjAAAUZOZrs3awSxu8ak2eTXssoAEc1hoDRfdKtxu0pFiBWzwSDxQU3 64hWtywtWIQIANSfaz2RM//1fhkK X-Google-Smtp-Source: APXvYqwcnoLqV4WJo0zNxc/EJeUCRVlSA7JjTT5zK5H1uzpxbY/OYWldhmgzBNo4KJ2SEIivlbraMg== X-Received: by 2002:a62:ce4a:: with SMTP id y71mr157985pfg.54.1572315112421; Mon, 28 Oct 2019 19:11:52 -0700 (PDT) Received: from sol.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id w14sm13710999pge.56.2019.10.28.19.11.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2019 19:11:52 -0700 (PDT) From: Jordan Niethe To: skiboot@lists.ozlabs.org Date: Tue, 29 Oct 2019 13:11:06 +1100 Message-Id: <20191029021106.29396-4-jniethe5@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191029021106.29396-1-jniethe5@gmail.com> References: <20191029021106.29396-1-jniethe5@gmail.com> MIME-Version: 1.0 Subject: [Skiboot] [RFC PATCH v2 3/3] core/pcie-slot: Restore slot power controller state X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" After a fundamental reset we currently assume that each slot's power controllers maintain their correct state. This is an invalid assumption. We need to make sure that this state gets restored. If the power controller does not get turned on but we think it is on things will go badly. We can handle this with extra states in our restore_buses() state machine function. If a PCI devices is behind a switch, check for a power controller and if present restore the power state before waiting for the link to activate. Signed-off-by: Jordan Niethe --- v2: - Make sure we actually change to PHB4_SLOT_BUSES_PD_WAIT_SWITCH_OFF state - Use get_power_state() for reading the slot power state core/pcie-slot.c | 55 ++++++++++++++++++++++++++++++++++++++++++++-- include/pci-slot.h | 8 ++++--- 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/core/pcie-slot.c b/core/pcie-slot.c index 048a8b279c71..06e80ee3d689 100644 --- a/core/pcie-slot.c +++ b/core/pcie-slot.c @@ -267,6 +267,9 @@ int64_t pcie_slot_sm_restore_buses(struct pci_slot *slot) struct pci_device *pd, *parent; uint32_t link_cap = 0; uint16_t link_sts = 0; + uint32_t slot_cap = 0; + uint16_t slot_ctl = 0; + uint8_t power_state; int32_t ecap = 0; struct phb *phb; uint32_t vdid; @@ -296,9 +299,8 @@ int64_t pcie_slot_sm_restore_buses(struct pci_slot *slot) if (!pd->is_vf && !(pd->bdfn & 7) && pd->parent != NULL && pd->parent->dev_type == PCIE_TYPE_SWITCH_DNPORT) { PCIDBG(phb, pd->bdfn, "BUSES: Behind a switch\n"); - slot->retries = PCI_BUS_SWITCH_LINK_RETRIES; pci_slot_set_state(slot, - PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK); + PCI_SLOT_BUSES_PD_SWITCH_POWER); return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); } else { PCIDBG(phb, pd->bdfn, "BUSES: Not behind a switch\n"); @@ -306,6 +308,55 @@ int64_t pcie_slot_sm_restore_buses(struct pci_slot *slot) pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_CRS); return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); } + case PCI_SLOT_BUSES_PD_SWITCH_POWER: + if (pci_has_cap(parent, PCI_CFG_CAP_ID_EXP, false)) { + ecap = pci_cap(parent, PCI_CFG_CAP_ID_EXP, false); + pci_cfg_read32(phb, parent->bdfn, + ecap + PCICAP_EXP_SLOTCAP, &slot_cap); + } + + if (!(slot_cap & PCICAP_EXP_SLOTCAP_PWCTRL)) { + PCIDBG(phb, parent->bdfn, + "BUSES: Parent: No power control\n"); + slot->retries = PCI_BUS_SWITCH_LINK_RETRIES; + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + } + + PCIDBG(phb, parent->bdfn, "BUSES: Parent: Have power control\n"); + /* Always turn it off */ + PCIDBG(phb, parent->bdfn, "BUSES: Parent: Turning slot off\n"); + pci_cfg_read16(phb, parent->bdfn, + ecap + PCICAP_EXP_SLOTCTL, &slot_ctl); + slot_ctl |= PCICAP_EXP_SLOTCTL_PWRCTLR; + slot_ctl |= SETFIELD(PCICAP_EXP_SLOTCTL_PWRI, 0, PCIE_INDIC_OFF); + pci_cfg_write16(phb, parent->bdfn, + ecap + PCICAP_EXP_SLOTCTL, slot_ctl); + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_SWITCH_OFF); + return pci_slot_set_sm_timeout(slot, secs_to_tb(1)); + case PCI_SLOT_BUSES_PD_WAIT_SWITCH_OFF: + parent->slot->ops.get_power_state(parent->slot, &power_state); + if (power_state == PCI_SLOT_POWER_OFF) { + PCIDBG(phb, parent->bdfn, + "BUSES: Parent: Slot power state is off. Nothing to do\n"); + slot->phb->current_pd = pci_device_iter_next(slot->phb, + NULL); + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_START); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); + } + + PCIDBG(phb, parent->bdfn, "BUSES: Parent: Turning slot on\n"); + ecap = pci_cap(parent, PCI_CFG_CAP_ID_EXP, false); + pci_cfg_read16(phb, parent->bdfn, ecap + PCICAP_EXP_SLOTCTL, + &slot_ctl); + slot_ctl &= ~PCICAP_EXP_SLOTCTL_PWRCTLR; + slot_ctl |= SETFIELD(PCICAP_EXP_SLOTCTL_PWRI, 0, PCIE_INDIC_ON); + pci_cfg_write16(phb, parent->bdfn, + ecap + PCICAP_EXP_SLOTCTL, slot_ctl); + + slot->retries = PCI_BUS_SWITCH_LINK_RETRIES; + pci_slot_set_state(slot, PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK); + return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); case PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK: PCIDBG(phb, pd->bdfn, "BUSES: Wait for Switch Link\n"); diff --git a/include/pci-slot.h b/include/pci-slot.h index 353efc7e96e1..a95a387bf6aa 100644 --- a/include/pci-slot.h +++ b/include/pci-slot.h @@ -134,9 +134,11 @@ struct pci_slot_ops { #define PCI_SLOT_BUSES PCI_SLOT_STATE_BUSES #define PCI_SLOT_BUSES_START (PCI_SLOT_BUSES + 1) #define PCI_SLOT_BUSES_PD_START (PCI_SLOT_BUSES + 2) -#define PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK (PCI_SLOT_BUSES + 3) -#define PCI_SLOT_BUSES_PD_WAIT_CRS (PCI_SLOT_BUSES + 4) -#define PCI_SLOT_BUSES_PD_END (PCI_SLOT_BUSES + 5) +#define PCI_SLOT_BUSES_PD_SWITCH_POWER (PCI_SLOT_BUSES + 3) +#define PCI_SLOT_BUSES_PD_WAIT_SWITCH_OFF (PCI_SLOT_BUSES + 4) +#define PCI_SLOT_BUSES_PD_WAIT_SWITCH_LINK (PCI_SLOT_BUSES + 5) +#define PCI_SLOT_BUSES_PD_WAIT_CRS (PCI_SLOT_BUSES + 6) +#define PCI_SLOT_BUSES_PD_END (PCI_SLOT_BUSES + 7) #define PCI_BUS_CRS_RETRIES 40 #define PCI_BUS_SWITCH_LINK_RETRIES 100