From patchwork Fri Feb 15 00:48:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver O'Halloran X-Patchwork-Id: 1042560 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 440wBW5T18z9s3l for ; Fri, 15 Feb 2019 12:08:51 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QmpkMcw5"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 440wBW4PZ8zDqGG for ; Fri, 15 Feb 2019 12:08:51 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::643; helo=mail-pl1-x643.google.com; envelope-from=oohall@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QmpkMcw5"; dkim-atps=neutral Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 440vlH5YhwzDqYP for ; Fri, 15 Feb 2019 11:48:43 +1100 (AEDT) Received: by mail-pl1-x643.google.com with SMTP id g9so4065577plo.3 for ; Thu, 14 Feb 2019 16:48:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PQYTLwTUTUNQitC9VV0okYczJ8dcLrtFO9YklGXLjzc=; b=QmpkMcw52HKDDASp583Zifgp97igSBaN0A+TygBBkD2315ZyOOeznh0hM7/U0wUlbE xsXxzEeeH1iBed3iYpthUcwrUfZ8QHbJVRBjzh+Ir9XzFJ9cf16XsUya+LWaCKPDpQ85 aI0ZjlVHqy0J4jicJcmytOtokVCT0VOp28y7hZnYPXWU8vNu0v63LqexZGBm8zTYFpzo skys+hLBzrgJcS/d4IDEM8pbJdoJ0wNAdGUuEPgs5UVNfc3wAnnvbZ6u8dcz0bEkoPkl FGh/E7sJlO5yYooH0IiFKruvvd8X4uijHcWYD9El69fsF5pSuQLhS0Cxw4nooSooL4YD nDNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PQYTLwTUTUNQitC9VV0okYczJ8dcLrtFO9YklGXLjzc=; b=Hey06hvJD1qShUZrpruydXuFiWHe6kWRl8raWf8nSIXjXd/dJH8dcYsXCSQXbAp4yd gkhaaJx7+wqIdsaMXxqkbH9iKGcZg1lQjIhfTARszEEz5y7BdeIbVg3C0u70tKBbwiGr IWxZrCKw1yVhkYsoabWrU6kKVbOko4ng9tNhNft9h7lG0xJe7VTGqzJjfB4rbNeUwBBj bPMrN8WP0TY9qejGXUjYDS7tl8CvAN0LIDZHmhESpe43s9nkEoQy725h70Vyx0aSqQtI WqLSakaVpwf0FLg8nxjUcAhgddsAvncCa4T+NFt+3sRR3mgTWKauvv7CE/QwiMDdaCh3 lRxA== X-Gm-Message-State: AHQUAuZOm7HwiILXLuwCrLljDbEFWDt3Nw9dgeaGpdExSm7NnyG4TEwO n8vANr8HemWRRCd3jc2koLtvZGV+ X-Google-Smtp-Source: AHgI3IZPXaC+HKAygspqig6rjRtjL3dvCWEk0JDUKUP3kb0t4CcdkYmPOqa/ZYuBknc8vuGjDUQdig== X-Received: by 2002:a17:902:7590:: with SMTP id j16mr4881936pll.304.1550191721951; Thu, 14 Feb 2019 16:48:41 -0800 (PST) Received: from wafer.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id v26sm4993751pfg.62.2019.02.14.16.48.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Feb 2019 16:48:41 -0800 (PST) From: Oliver O'Halloran To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 7/7] powerpc/eeh: Add eeh_force_recover to debugfs Date: Fri, 15 Feb 2019 11:48:17 +1100 Message-Id: <20190215004817.19961-7-oohall@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190215004817.19961-1-oohall@gmail.com> References: <20190215004817.19961-1-oohall@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: sbobroff@linux.ibm.com, Oliver O'Halloran Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This patch adds a debugfs interface to force scheduling a recovery event. This can be used to recover a specific PE or schedule a "special" recovery even that checks for errors at the PHB level. To force a recovery of a normal PE, use: echo '<#pe>:<#phb>' > /sys/kernel/debug/powerpc/eeh_force_recover To force a scan for broken PHBs: echo 'hwcheck' > /sys/kernel/debug/powerpc/eeh_force_recover Signed-off-by: Oliver O'Halloran --- v2: Rename from pci_find_hose_for_domain() to pci_find_controller_for_domain() Use the more descriptive "hwcheck" to send a special event rather than "null" --- arch/powerpc/include/asm/eeh_event.h | 1 + arch/powerpc/kernel/eeh.c | 59 ++++++++++++++++++++++++++++ arch/powerpc/kernel/eeh_event.c | 25 +++++++----- 3 files changed, 75 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/include/asm/eeh_event.h b/arch/powerpc/include/asm/eeh_event.h index 9884e872686f..6d0412b846ac 100644 --- a/arch/powerpc/include/asm/eeh_event.h +++ b/arch/powerpc/include/asm/eeh_event.h @@ -33,6 +33,7 @@ struct eeh_event { int eeh_event_init(void); int eeh_send_failure_event(struct eeh_pe *pe); +int __eeh_send_failure_event(struct eeh_pe *pe); void eeh_remove_event(struct eeh_pe *pe, bool force); void eeh_handle_normal_event(struct eeh_pe *pe); void eeh_handle_special_event(void); diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c index 9f20099ce2d9..37677468be6d 100644 --- a/arch/powerpc/kernel/eeh.c +++ b/arch/powerpc/kernel/eeh.c @@ -1805,6 +1805,62 @@ static int eeh_enable_dbgfs_get(void *data, u64 *val) DEFINE_DEBUGFS_ATTRIBUTE(eeh_enable_dbgfs_ops, eeh_enable_dbgfs_get, eeh_enable_dbgfs_set, "0x%llx\n"); + +static ssize_t eeh_force_recover_write(struct file *filp, + const char __user *user_buf, + size_t count, loff_t *ppos) +{ + struct pci_controller *hose; + uint32_t phbid, pe_no; + struct eeh_pe *pe; + char buf[20]; + int ret; + + ret = simple_write_to_buffer(buf, sizeof(buf), ppos, user_buf, count); + if (!ret) + return -EFAULT; + + /* + * When PE is NULL the event is a "special" event. Rather than + * recovering a specific PE it forces the EEH core to scan for failed + * PHBs and recovers each. This needs to be done before any device + * recoveries can occur. + */ + if (!strncmp(buf, "hwcheck", 7)) { + __eeh_send_failure_event(NULL); + return count; + } + + ret = sscanf(buf, "%x:%x", &phbid, &pe_no); + if (ret != 2) + return -EINVAL; + + hose = pci_find_controller_for_domain(phbid); + if (!hose) + return -ENODEV; + + /* Retrieve PE */ + pe = eeh_pe_get(hose, pe_no, 0); + if (!pe) + return -ENODEV; + + /* + * We don't do any state checking here since the detection + * process is async to the recovery process. The recovery + * thread *should* not break even if we schedule a recovery + * from an odd state (e.g. PE removed, or recovery of a + * non-isolated PE) + */ + __eeh_send_failure_event(pe); + + return ret < 0 ? ret : count; +} + +static const struct file_operations eeh_force_recover_fops = { + .open = simple_open, + .llseek = no_llseek, + .write = eeh_force_recover_write, +}; #endif static int __init eeh_init_proc(void) @@ -1820,6 +1876,9 @@ static int __init eeh_init_proc(void) debugfs_create_bool("eeh_disable_recovery", 0600, powerpc_debugfs_root, &eeh_debugfs_no_recover); + debugfs_create_file_unsafe("eeh_force_recover", 0600, + powerpc_debugfs_root, NULL, + &eeh_force_recover_fops); eeh_cache_debugfs_init(); #endif } diff --git a/arch/powerpc/kernel/eeh_event.c b/arch/powerpc/kernel/eeh_event.c index 19837798bb1d..539aca055d70 100644 --- a/arch/powerpc/kernel/eeh_event.c +++ b/arch/powerpc/kernel/eeh_event.c @@ -121,20 +121,11 @@ int eeh_event_init(void) * the actual event will be delivered in a normal context * (from a workqueue). */ -int eeh_send_failure_event(struct eeh_pe *pe) +int __eeh_send_failure_event(struct eeh_pe *pe) { unsigned long flags; struct eeh_event *event; - /* - * If we've manually supressed recovery events via debugfs - * then just drop it on the floor. - */ - if (eeh_debugfs_no_recover) { - pr_err("EEH: Event dropped due to no_recover setting\n"); - return 0; - } - event = kzalloc(sizeof(*event), GFP_ATOMIC); if (!event) { pr_err("EEH: out of memory, event not handled\n"); @@ -153,6 +144,20 @@ int eeh_send_failure_event(struct eeh_pe *pe) return 0; } +int eeh_send_failure_event(struct eeh_pe *pe) +{ + /* + * If we've manually supressed recovery events via debugfs + * then just drop it on the floor. + */ + if (eeh_debugfs_no_recover) { + pr_err("EEH: Event dropped due to no_recover setting\n"); + return 0; + } + + return __eeh_send_failure_event(pe); +} + /** * eeh_remove_event - Remove EEH event from the queue * @pe: Event binding to the PE