From patchwork Tue Nov 27 00:43:13 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takao Indoh X-Patchwork-Id: 202054 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id D245E2C0086 for ; Tue, 27 Nov 2012 11:43:50 +1100 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757598Ab2K0AnT (ORCPT ); Mon, 26 Nov 2012 19:43:19 -0500 Received: from fgwmail6.fujitsu.co.jp ([192.51.44.36]:52172 "EHLO fgwmail6.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757605Ab2K0AnR (ORCPT ); Mon, 26 Nov 2012 19:43:17 -0500 Received: from m4.gw.fujitsu.co.jp (unknown [10.0.50.74]) by fgwmail6.fujitsu.co.jp (Postfix) with ESMTP id 81A353EE0C0; Tue, 27 Nov 2012 09:43:16 +0900 (JST) Received: from smail (m4 [127.0.0.1]) by outgoing.m4.gw.fujitsu.co.jp (Postfix) with ESMTP id 2842045DE56; Tue, 27 Nov 2012 09:43:14 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (s4.gw.fujitsu.co.jp [10.0.50.94]) by m4.gw.fujitsu.co.jp (Postfix) with ESMTP id B75D645DE52; Tue, 27 Nov 2012 09:43:13 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id A50741DB8042; Tue, 27 Nov 2012 09:43:13 +0900 (JST) Received: from m1000.s.css.fujitsu.com (m1000.s.css.fujitsu.com [10.240.81.136]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id 518201DB8040; Tue, 27 Nov 2012 09:43:13 +0900 (JST) Received: from m1000.css.fujitsu.com (m1000 [127.0.0.1]) by m1000.s.css.fujitsu.com (Postfix) with ESMTP id 204396124E; Tue, 27 Nov 2012 09:43:13 +0900 (JST) Received: from tindoh.g01.fujitsu.local (tindoh.g01.fujitsu.local [10.124.101.134]) by m1000.s.css.fujitsu.com (Postfix) with ESMTP id 88E0D6122E; Tue, 27 Nov 2012 09:43:12 +0900 (JST) X-SecurityPolicyCheck: OK by SHieldMailChecker v1.7.4 From: Takao Indoh To: linux-pci@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org Cc: tokunaga.keiich@jp.fujitsu.com, kexec@lists.infradead.org, hbabu@us.ibm.com, andi@firstfloor.org, ddutile@redhat.com, vgoyal@redhat.com, ishii.hironobu@jp.fujitsu.com, hpa@zytor.com, bhelgaas@google.com, tglx@linutronix.de, yinghai@kernel.org, mingo@redhat.com, Takao Indoh , khalid@gonehiking.org Message-Id: <20121127004223.3604.78831.sendpatchset@tindoh.g01.fujitsu.local> In-Reply-To: <20121127004144.3604.61708.sendpatchset@tindoh.g01.fujitsu.local> References: <20121127004144.3604.61708.sendpatchset@tindoh.g01.fujitsu.local> Subject: [PATCH v7 4/5] x86, pci: Reset PCIe devices at boot time Date: Tue, 27 Nov 2012 09:43:13 +0900 (JST) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This patch resets PCIe devices at boot time when "reset_devices" is specified. Kdump with intel_iommu=on fails becasue ongoing DMA from first kernel causes DMAR fault when page table of DMAR is initialized while kdump kernel is booting up. To solve this problem, this patch resets PCIe devices during boot to stop its DMA. Signed-off-by: Takao Indoh --- arch/x86/include/asm/pci-direct.h | 1 + arch/x86/kernel/setup.c | 3 + arch/x86/pci/early.c | 241 +++++++++++++++++++++++++++++++++++++ 3 files changed, 245 insertions(+), 0 deletions(-) diff --git a/arch/x86/include/asm/pci-direct.h b/arch/x86/include/asm/pci-direct.h index b6360d3..5620070 100644 --- a/arch/x86/include/asm/pci-direct.h +++ b/arch/x86/include/asm/pci-direct.h @@ -18,6 +18,7 @@ extern int early_pci_allowed(void); extern unsigned int pci_early_dump_regs; extern void early_dump_pci_device(u8 bus, u8 slot, u8 func); extern void early_dump_pci_devices(void); +extern void early_reset_pcie_devices(void); struct pci_dev *get_early_pci_dev(int num, int slot, int func); #endif /* _ASM_X86_PCI_DIRECT_H */ diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index ca45696..2e7928e 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1001,6 +1001,9 @@ void __init setup_arch(char **cmdline_p) generic_apic_probe(); early_quirks(); +#ifdef CONFIG_PCI + early_reset_pcie_devices(); +#endif /* * Read APIC and some other early information from ACPI tables. diff --git a/arch/x86/pci/early.c b/arch/x86/pci/early.c index 024def7..ff737f3 100644 --- a/arch/x86/pci/early.c +++ b/arch/x86/pci/early.c @@ -1,5 +1,6 @@ #include #include +#include #include #include #include @@ -183,3 +184,243 @@ __init struct pci_dev *get_early_pci_dev(int num, int slot, int func) return pdev; } + +struct pcie_dev { + int cap; /* position of PCI Express capability */ + int flags; /* PCI_EXP_FLAGS */ + + /* saved configration register */ + u32 pci_cfg[16]; + u16 pcie_cfg[7]; +}; + +struct pcie_port { + struct list_head dev; + u8 bus; + u8 slot; + u8 func; + u8 secondary; + struct pcie_dev child[PCI_MAX_FUNCTIONS]; +}; + +static __initdata LIST_HEAD(device_list); + +static void __init early_udelay(int loops) +{ + while (loops--) { + /* Approximately 1 us */ + native_io_delay(); + } +} + +static void __init do_reset(u8 bus, u8 slot, u8 func) +{ + struct pci_dev *dev; + u16 ctrl; + + dev = get_early_pci_dev(bus, slot, func); + + printk(KERN_INFO "pci 0000:%02x:%02x.%d reset\n", bus, slot, func); + + /* Assert Secondary Bus Reset */ + pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); + ctrl |= PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + + /* + * PCIe spec requires software to ensure a minimum reset duration + * (Trst == 1ms). We have here 5ms safety margin because early_udelay + * is not precise. + */ + early_udelay(5000); + + /* De-assert Secondary Bus Reset */ + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); +} + +static void __init save_state(u8 bus, u8 slot, u8 func, struct pcie_dev *pdev) +{ + struct pci_dev *dev; + int i; + + dev = get_early_pci_dev(bus, slot, func); + dev->is_pcie = 1; + dev->pcie_cap = pdev->cap; + dev->pcie_flags_reg = pdev->flags; + + printk(KERN_INFO "pci 0000:%02x:%02x.%d save state\n", bus, slot, func); + + for (i = 0; i < 16; i++) + pci_read_config_dword(dev, i * 4, pdev->pci_cfg + i); + i = 0; + pcie_capability_read_word(dev, PCI_EXP_DEVCTL, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_SLTCTL, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_RTCTL, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &pdev->pcie_cfg[i++]); + pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &pdev->pcie_cfg[i++]); +} + +static void __init restore_state(u8 bus, u8 slot, u8 func, + struct pcie_dev *pdev) +{ + struct pci_dev *dev; + int i = 0; + + dev = get_early_pci_dev(bus, slot, func); + dev->is_pcie = 1; + dev->pcie_cap = pdev->cap; + dev->pcie_flags_reg = pdev->flags; + + printk(KERN_INFO "pci 0000:%02x:%02x.%d restore state\n", + bus, slot, func); + + pcie_capability_write_word(dev, PCI_EXP_DEVCTL, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_LNKCTL, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_SLTCTL, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_RTCTL, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_DEVCTL2, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, pdev->pcie_cfg[i++]); + pcie_capability_write_word(dev, PCI_EXP_SLTCTL2, pdev->pcie_cfg[i++]); + + for (i = 15; i >= 0; i--) + pci_write_config_dword(dev, i * 4, pdev->pci_cfg[i]); +} + +static void __init find_pcie_device(unsigned bus, unsigned slot, unsigned func) +{ + struct pci_dev *dev; + int f, pcie_type, count; + u8 secondary, type; + u16 vendor; + u32 class; + struct pcie_port *port; + int pcie_cap[PCI_MAX_FUNCTIONS]; + int pcie_flags[PCI_MAX_FUNCTIONS]; + + dev = get_early_pci_dev(bus, slot, func); + set_pcie_port_type(dev); + if (!pci_is_pcie(dev)) + return; + + pcie_type = pci_pcie_type(dev); + if ((pcie_type != PCI_EXP_TYPE_ROOT_PORT) && + (pcie_type != PCI_EXP_TYPE_DOWNSTREAM)) + return; + + pci_read_config_byte(dev, PCI_HEADER_TYPE, &type); + if ((type & 0x7f) != PCI_HEADER_TYPE_BRIDGE) + return; + pci_read_config_byte(dev, PCI_SECONDARY_BUS, &secondary); + + memset(pcie_cap, 0, sizeof(pcie_cap)); + memset(pcie_flags, 0, sizeof(pcie_flags)); + for (count = 0, f = 0; f < PCI_MAX_FUNCTIONS; f++) { + dev = get_early_pci_dev(secondary, 0, f); + pci_read_config_word(dev, PCI_VENDOR_ID, &vendor); + if (vendor == 0xffff) + continue; + + set_pcie_port_type(dev); + if (!pci_is_pcie(dev)) + continue; + + pcie_type = pci_pcie_type(dev); + if ((pcie_type == PCI_EXP_TYPE_UPSTREAM) || + (pcie_type == PCI_EXP_TYPE_PCI_BRIDGE)) + /* Don't reset switch, bridge */ + return; + + pci_read_config_dword(dev, PCI_CLASS_REVISION, &class); + if ((class >> 24) == PCI_BASE_CLASS_DISPLAY) + /* Don't reset VGA device */ + return; + + count++; + pcie_cap[f] = dev->pcie_cap; + pcie_flags[f] = dev->pcie_flags_reg; + } + + if (!count) + return; + + port = (struct pcie_port *)alloc_bootmem(sizeof(struct pcie_port)); + if (port == NULL) { + printk(KERN_ERR "pci 0000:%02x:%02x.%d alloc_bootmem failed\n", + bus, slot, func); + return; + } + memset(port, 0, sizeof(*port)); + port->bus = bus; + port->slot = slot; + port->func = func; + port->secondary = secondary; + for (f = 0; f < PCI_MAX_FUNCTIONS; f++) + if (pcie_cap[f]) { + port->child[f].cap = pcie_cap[f]; + port->child[f].flags = pcie_flags[f]; + save_state(secondary, 0, f, &port->child[f]); + } + list_add_tail(&port->dev, &device_list); +} + +void __init early_reset_pcie_devices(void) +{ + unsigned bus, slot, func; + struct pcie_port *port, *tmp; + struct pci_dev *dev; + + if (!early_pci_allowed() || !reset_devices) + return; + + /* + * Find PCIe port(root port and downstream port) and save config + * registers of its downstream devices + */ + for (bus = 0; bus < 256; bus++) { + for (slot = 0; slot < 32; slot++) { + for (func = 0; func < PCI_MAX_FUNCTIONS; func++) { + u16 vendor; + u8 type; + + dev = get_early_pci_dev(bus, slot, func); + pci_read_config_word(dev, PCI_VENDOR_ID, + &vendor); + if (vendor == 0xffff) + continue; + + pci_read_config_byte(dev, PCI_HEADER_TYPE, + &type); + find_pcie_device(bus, slot, func); + + if ((func == 0) && !(type & 0x80)) + break; + } + } + } + + if (list_empty(&device_list)) + return; + + /* Do bus reset */ + list_for_each_entry(port, &device_list, dev) + do_reset(port->bus, port->slot, port->func); + + /* + * According to PCIe spec, software must wait a minimum of 100 ms + * before sending a configuration request. We have 500ms safety margin + * here. + */ + early_udelay(500000); + + /* Restore config registers and free memory */ + list_for_each_entry_safe(port, tmp, &device_list, dev) { + for (func = 0; func < PCI_MAX_FUNCTIONS; func++) + if (port->child[func].cap) + restore_state(port->secondary, 0, func, + &port->child[func]); + free_bootmem(__pa(port), sizeof(*port)); + } +}