From patchwork Wed Sep 23 06:12:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver O'Halloran X-Patchwork-Id: 1369631 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Bx7TH6S7gz9sTC for ; Wed, 23 Sep 2020 16:25:23 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=UALgh1p/; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Bx7TH46VjzDqRX for ; Wed, 23 Sep 2020 16:25:23 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1042; helo=mail-pj1-x1042.google.com; envelope-from=oohall@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=UALgh1p/; dkim-atps=neutral Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Bx7BX3sGzzDqYY for ; Wed, 23 Sep 2020 16:12:34 +1000 (AEST) Received: by mail-pj1-x1042.google.com with SMTP id kk9so2614731pjb.2 for ; Tue, 22 Sep 2020 23:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hV8ObAlKn59+0I/Dgg2I63RUCLQ/dSf3IrzqfMLThk0=; b=UALgh1p/DkZQIpJFZRHUzyrgcvbScm4hL7FEJA8r+q44AkO6EsvMOkU5/paymm526i 5UVKTIBa9yNZVpj8VNws52itqSRjqsbr0CRHyq6Rkd0ex2JL+6zX4JADyUxrkKrt228K 9sBAum+ue1kNaYf6rOLQB3BCyk4oCsApKKYKaVAQseZF7yB5jMow9RcPenpOGm0AL+Gw gsiObZHTaXMugtoM6TTWScYaBQE0Lnv6kGtMpPmZzukIIUAB5KaiLGSqTXeRB2wMuWS3 ebjjzwMyM0ol9ijdOKMItPh74QWTTYzUdBfEjIPcRX+Wk1505fJ7ilOxnr6PZ1kxPnuW eMjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hV8ObAlKn59+0I/Dgg2I63RUCLQ/dSf3IrzqfMLThk0=; b=gWgQj5TpdJ9qtz6VYk3d9IPp6vSCGLm85eeTp0v5NEvbjIck0seKXc6lkPlxMR4LyK rsLr2m1xcrZyP9kRWD0z8IQoh4V7VQMmURe5C0rjAVpmv4lM7ncTv2TJFDvLAvqfQxbK 2Kf3ipMi2pSo36oP19+i+qvOe3Grx8wsY5O7YxuiDBG+6YutUPTIaADiaKFAe/pN60rh 7Ox+fGDyazISI2IELplkAkvQU4+d/hjDZIpz6qBMlAnl6lOozBjLzd7WJZfngo+a11qD 2nFTvi6tX1FI9ucX8AiuQHDMZfgc5GNZrU/lhMCeb/3d9k4bwYCHmkVKr9RpTMDGIRhV kJdw== X-Gm-Message-State: AOAM530rPk8unay/jMVQ4bWIn1bXUHc2GllSi4ZBWs9hMHB7iTpNlzTe hZireuma680X7yCYkYCWxgwy9Z5fK0Q0Ow== X-Google-Smtp-Source: ABdhPJyJ3GBjpKbLwwum94Hnh1h5pBMlEytCOozIAv0gEx9uG9/ty7rgd/tCT2TvJ4lIIb2qhMOzWQ== X-Received: by 2002:a17:90b:88d:: with SMTP id bj13mr6897421pjb.80.1600841551997; Tue, 22 Sep 2020 23:12:31 -0700 (PDT) Received: from localhost.ibm.com (194-193-34-182.tpgi.com.au. [194.193.34.182]) by smtp.gmail.com with ESMTPSA id f12sm4637450pjm.5.2020.09.22.23.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2020 23:12:31 -0700 (PDT) From: Oliver O'Halloran To: skiboot@lists.ozlabs.org Date: Wed, 23 Sep 2020 16:12:20 +1000 Message-Id: <20200923061220.231458-1-oohall@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Subject: [Skiboot] [PATCH v2] opal-prd: Have a worker process handle page offlining X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" The memory_error() hservice interface expects the memory_error() call to just accept the offline request and return without actually offlining the memory. Currently we will attempt to offline the marked pages before returning to HBRT which can result in an excessively long time spent in the memory_error() hservice call which blocks HBRT from processing other errors. Fix this by adding a worker process which performs the page offlining via the sysfs memory error interfaces. Signed-off-by: Oliver O'Halloran Reviewed-by: Vasant Hegde --- v2: switched to using sigaction() to auto-reap the workers and handle system call restarting, mainly for poll(). --- external/opal-prd/opal-prd.c | 83 +++++++++++++++++++++++++++--------- 1 file changed, 64 insertions(+), 19 deletions(-) diff --git a/external/opal-prd/opal-prd.c b/external/opal-prd/opal-prd.c index 40e5a9841427..d74d80398da0 100644 --- a/external/opal-prd/opal-prd.c +++ b/external/opal-prd/opal-prd.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include @@ -696,13 +697,42 @@ out: return rc; } +static int memory_error_worker(const char *sysfsfile, const char *type, + uint64_t i_start_addr, uint64_t i_endAddr) +{ + int memfd, rc, n, ret = 0; + char buf[ADDR_STRING_SZ]; + uint64_t addr; + + memfd = open(sysfsfile, O_WRONLY); + if (memfd < 0) { + pr_log(LOG_CRIT, "MEM: Failed to offline memory! " + "Unable to open sysfs node %s: %m", sysfsfile); + return -1; + } + + for (addr = i_start_addr; addr <= i_endAddr; addr += ctx->page_size) { + n = snprintf(buf, ADDR_STRING_SZ, "0x%lx", addr); + rc = write(memfd, buf, n); + if (rc != n) { + pr_log(LOG_CRIT, "MEM: Failed to offline memory! " + "page addr: %016lx type: %s: %m", + addr, type); + ret = 1; + } + } + pr_log(LOG_CRIT, "MEM: Offlined %016lx,%016lx, type %s: %m\n", + i_start_addr, addr, type); + + close(memfd); + return ret; +} + int hservice_memory_error(uint64_t i_start_addr, uint64_t i_endAddr, enum MemoryError_t i_errorType) { const char *sysfsfile, *typestr; - char buf[ADDR_STRING_SZ]; - int memfd, rc, n, ret = 0; - uint64_t addr; + pid_t pid; switch(i_errorType) { case MEMORY_ERROR_CE: @@ -722,26 +752,21 @@ int hservice_memory_error(uint64_t i_start_addr, uint64_t i_endAddr, pr_log(LOG_ERR, "MEM: Memory error: range %016lx-%016lx, type: %s", i_start_addr, i_endAddr, typestr); + /* + * HBRT expects the memory offlining process to happen in the background + * after the notification is delivered. + */ + pid = fork(); + if (pid > 0) + exit(memory_error_worker(sysfsfile, typestr, i_start_addr, i_endAddr)); - memfd = open(sysfsfile, O_WRONLY); - if (memfd < 0) { - pr_log(LOG_CRIT, "MEM: Failed to offline memory! " - "Unable to open sysfs node %s: %m", sysfsfile); + if (pid < 0) { + perror("MEM: unable to fork worker to offline memory!\n"); return -1; } - for (addr = i_start_addr; addr <= i_endAddr; addr += ctx->page_size) { - n = snprintf(buf, ADDR_STRING_SZ, "0x%lx", addr); - rc = write(memfd, buf, n); - if (rc != n) { - pr_log(LOG_CRIT, "MEM: Failed to offline memory! " - "page addr: %016lx type: %d: %m", - addr, i_errorType); - ret = rc; - } - } - - return ret; + pr_log(LOG_INFO, "MEM: forked off %d to handle mem error\n", pid); + return 0; } uint64_t hservice_get_interface_capabilities(uint64_t set) @@ -2112,6 +2137,10 @@ static int init_control_socket(struct opal_prd_ctx *ctx) return 0; } +static struct sigaction sigchild_action = { + .sa_flags = SA_NOCLDWAIT | SA_RESTART, + .sa_handler = SIG_DFL, +}; static int run_prd_daemon(struct opal_prd_ctx *ctx) { @@ -2243,6 +2272,22 @@ static int run_prd_daemon(struct opal_prd_ctx *ctx) pr_debug("SCOM: f00f: %lx", be64toh(val)); } + /* + * Setup the SIGCHLD handler to automatically reap the worker threads + * we use for memory offlining. We can't do this earlier since the + * modprobe helper spawns workers and wants to check their exit status + * with waitpid(). Auto-reaping breaks that so enable it just before + * entering the attn loop. + * + * We also setup system call restarting on SIGCHLD since opal-prd + * doesn't make any real attempt to handle blocking functions exiting + * due to EINTR. + */ + if (sigaction(SIGCHLD, &sigchild_action, NULL)) { + pr_log(LOG_ERR, "CTRL: Failed to register signal handler %m\n"); + return -1; + } + run_attn_loop(ctx); rc = 0;