From patchwork Thu Nov 8 14:46:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mauricio Faria de Oliveira X-Patchwork-Id: 994928 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42rR2K5c7xz9sDB; Fri, 9 Nov 2018 01:47:05 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1gKlaG-0002Ov-1q; Thu, 08 Nov 2018 14:47:00 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.86_2) (envelope-from ) id 1gKlaD-0002Nt-KB for kernel-team@lists.ubuntu.com; Thu, 08 Nov 2018 14:46:57 +0000 Received: from mail-qk1-f197.google.com ([209.85.222.197]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1gKlaD-0006D4-9q for kernel-team@lists.ubuntu.com; Thu, 08 Nov 2018 14:46:57 +0000 Received: by mail-qk1-f197.google.com with SMTP id n68so37636183qkn.8 for ; Thu, 08 Nov 2018 06:46:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tLAf0rYQSJ1+lkrSkwTfY1dNvnzg4l6bbd8Zb6/Ll8A=; b=LqJfVBJUlJrO6C9DEtjy6G8T1WsQot4+M/uh2AnewlqPRwYeHXwbeMAX2AkaEo6iBi fXcTcdHpduFjeJyncoKyhtq7jV1M3fnhGB8FUBnWtdIGCAzBL+zjzFzB2JroInU3rCdn Bheb6MpxWGu1iRqeb7l8U2FntrgyBjuyfek7tpRGUe39rOevcO8nffsC4NseNQ4Tts+u S7VtAszfnI8DVnzHqGXTUGsFl4m45fEg3v5gfSiLff79yDc0oARO16ryi8kkf8MkpmOo a4e1o3BBmldghAR1OJwUaWMrYN59vW3NDWYuk0hjUhfCCChldmrH9Nxx0sutwW4VEbHa YMzw== X-Gm-Message-State: AGRZ1gK+MXB8bW4ztzEHKA2pDx9pQv7ayx8xdXuijG+TEJUcyZ4l5hjX p7GTYdwQUTmgFTjwHQUvnGRUfTVxL5Bh/5QXaW9vJSK7rCkm1RnnwMkO86s9TM/3aflJ3e08PXY LC8CA8OjiJt++5etq+WGW3TmwEUJPvAQIXBKKhY9vtA== X-Received: by 2002:ac8:4055:: with SMTP id j21mr4485314qtl.389.1541688416204; Thu, 08 Nov 2018 06:46:56 -0800 (PST) X-Google-Smtp-Source: AJdET5dDwPY2oYVGHyZ9MY5AhNEdgtNnRxxmU8ekosTF/OQ+h4FAM9YrDoqvphVEpn6QOMQc6UgGxA== X-Received: by 2002:ac8:4055:: with SMTP id j21mr4485296qtl.389.1541688415938; Thu, 08 Nov 2018 06:46:55 -0800 (PST) Received: from localhost.localdomain ([179.159.57.206]) by smtp.gmail.com with ESMTPSA id x141-v6sm2145396qka.9.2018.11.08.06.46.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Nov 2018 06:46:55 -0800 (PST) From: Mauricio Faria de Oliveira To: kernel-team@lists.ubuntu.com Subject: [SRU B][PATCH v2 2/3] UBUNTU: SAUCE: x86/quirks: Add parameter to clear MSIs early on boot Date: Thu, 8 Nov 2018 12:46:35 -0200 Message-Id: <20181108144636.15887-3-mfo@canonical.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181108144636.15887-1-mfo@canonical.com> References: <20181108144636.15887-1-mfo@canonical.com> X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gpiccoli@canonical.com MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "Guilherme G. Piccoli" BugLink: https://bugs.launchpad.net/bugs/1797990 We observed a kdump failure in x86 that was narrowed down to MSI irq storm coming from a PCI network device. The bug manifests as a lack of progress in the boot process of kdump kernel, and a flood of kernel messages like: [...] [ 342.265294] do_IRQ: 0.155 No irq handler for vector [ 342.266916] do_IRQ: 0.155 No irq handler for vector [ 347.258422] do_IRQ: 14053260 callbacks suppressed [...] The root cause of the issue is that kexec process of the kdump kernel doesn't ensure PCI devices are reset or MSI capabilities are disabled, so a PCI adapter could produce a huge amount of irqs which would steal all the processing time for the CPU (specially since we usually restrict kdump kernel to use a single CPU only). This patch implements the kernel parameter "pci=clearmsi" to clear the MSI/MSI-X enable bits in the Message Control register for all PCI devices during early boot time, thus preventing potential issues in the kexec'ed kernel. PCI spec also supports/enforces this need (see PCI Local Bus spec sections 6.8.1.3 and 6.8.2.3). Suggested-by: Dan Streetman Suggested-by: Gavin Shan Signed-off-by: Guilherme G. Piccoli [mfo: backport to ubuntu-bionic: - update context lines in pci-direct.h and early-quirks.c] Signed-off-by: Mauricio Faria de Oliveira --- .../admin-guide/kernel-parameters.txt | 6 ++++ arch/x86/include/asm/pci-direct.h | 1 + arch/x86/kernel/early-quirks.c | 32 +++++++++++++++++++ arch/x86/pci/common.c | 4 +++ 4 files changed, 43 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 51210d10d905..c163c3ce2902 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3049,6 +3049,12 @@ nomsi [MSI] If the PCI_MSI kernel config parameter is enabled, this kernel boot option can be used to disable the use of MSI interrupts system-wide. + clearmsi [X86] Clears MSI/MSI-X enable bits early in boot + time in order to avoid issues like adapters + screaming irqs and preventing boot progress. + Also, it enforces the PCI Local Bus spec + rule that those bits should be 0 in system reset + events (useful for kexec/kdump cases). noioapicquirk [APIC] Disable all boot interrupt quirks. Safety option to keep boot IRQs enabled. This should never be necessary. diff --git a/arch/x86/include/asm/pci-direct.h b/arch/x86/include/asm/pci-direct.h index 3f5b5158bbd7..14d4aa4e1aed 100644 --- a/arch/x86/include/asm/pci-direct.h +++ b/arch/x86/include/asm/pci-direct.h @@ -15,6 +15,7 @@ extern void write_pci_config(u8 bus, u8 slot, u8 func, u8 offset, u32 val); extern void write_pci_config_byte(u8 bus, u8 slot, u8 func, u8 offset, u8 val); extern void write_pci_config_16(u8 bus, u8 slot, u8 func, u8 offset, u16 val); +extern unsigned int pci_early_clear_msi; extern int early_pci_allowed(void); extern unsigned int pci_early_dump_regs; diff --git a/arch/x86/kernel/early-quirks.c b/arch/x86/kernel/early-quirks.c index c87560e1e3ef..f7faa3341492 100644 --- a/arch/x86/kernel/early-quirks.c +++ b/arch/x86/kernel/early-quirks.c @@ -28,6 +28,37 @@ #include #include +static void __init early_pci_clear_msi(int bus, int slot, int func) +{ + int pos; + u16 ctrl; + + if (likely(!pci_early_clear_msi)) + return; + + pr_info_once("Clearing MSI/MSI-X enable bits early in boot (quirk)\n"); + + pos = pci_early_find_cap(bus, slot, func, PCI_CAP_ID_MSI); + if (pos) { + ctrl = read_pci_config_16(bus, slot, func, pos + PCI_MSI_FLAGS); + ctrl &= ~PCI_MSI_FLAGS_ENABLE; + write_pci_config_16(bus, slot, func, pos + PCI_MSI_FLAGS, ctrl); + + /* Read again to flush previous write */ + ctrl = read_pci_config_16(bus, slot, func, pos + PCI_MSI_FLAGS); + } + + pos = pci_early_find_cap(bus, slot, func, PCI_CAP_ID_MSIX); + if (pos) { + ctrl = read_pci_config_16(bus, slot, func, pos + PCI_MSIX_FLAGS); + ctrl &= ~PCI_MSIX_FLAGS_ENABLE; + write_pci_config_16(bus, slot, func, pos + PCI_MSIX_FLAGS, ctrl); + + /* Read again to flush previous write */ + ctrl = read_pci_config_16(bus, slot, func, pos + PCI_MSIX_FLAGS); + } +} + #define dev_err(msg) pr_err("pci 0000:%02x:%02x.%d: %s", bus, slot, func, msg) static void __init fix_hypertransport_config(int num, int slot, int func) @@ -683,6 +714,7 @@ static struct chipset early_qrk[] __initdata = { PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet}, { PCI_VENDOR_ID_BROADCOM, 0x4331, PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset}, + { PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0, early_pci_clear_msi}, {} }; diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c index 563049c483a1..94b156f85573 100644 --- a/arch/x86/pci/common.c +++ b/arch/x86/pci/common.c @@ -33,6 +33,7 @@ int noioapicreroute = 1; #endif int pcibios_last_bus = -1; unsigned long pirq_table_addr; +unsigned int pci_early_clear_msi; const struct pci_raw_ops *__read_mostly raw_pci_ops; const struct pci_raw_ops *__read_mostly raw_pci_ext_ops; @@ -608,6 +609,9 @@ char *__init pcibios_setup(char *str) } else if (!strcmp(str, "skip_isa_align")) { pci_probe |= PCI_CAN_SKIP_ISA_ALIGN; return NULL; + } else if (!strcmp(str, "clearmsi")) { + pci_early_clear_msi = 1; + return NULL; } else if (!strcmp(str, "noioapicquirk")) { noioapicquirk = 1; return NULL;