From patchwork Tue Dec 4 09:56:24 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel J Blueman X-Patchwork-Id: 203587 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id ED2C52C0098 for ; Tue, 4 Dec 2012 20:56:55 +1100 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751688Ab2LDJ4y (ORCPT ); Tue, 4 Dec 2012 04:56:54 -0500 Received: from mail-pa0-f46.google.com ([209.85.220.46]:39229 "EHLO mail-pa0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751608Ab2LDJ4w (ORCPT ); Tue, 4 Dec 2012 04:56:52 -0500 Received: by mail-pa0-f46.google.com with SMTP id bh2so2637236pad.19 for ; Tue, 04 Dec 2012 01:56:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:to:cc:subject:date:message-id:x-mailer:x-gm-message-state; bh=n/Ixg9Hze56JCmtkkm6U/P713oYR/Boma+QIdf6Az2w=; b=le0jdTc38bBIcTw6HnQCqNzkfx75kIKW9yART9B+T2HcoG1I+0uD3KRs7BOSOFDGM7 H+nvhxa16lnNipAdLn7XF/kaXS/AW/GQnnDqLnlwZbZpAVFLJQ09qpkSKMSL/5iBO0Z7 0fHAcSmI5Y/qf6uDqMw7L4rnCFkTXUI8pQdbbbImoXS0nEkNehnxK3kNr7XHpD71NsYd hO4eiCof8ISA57q5UWFALBrzdZSUZfh5wdQmjMUAlmrUZ99rHKBtp8yOtljXJiBOo1CD Xw+3XzHWm7R2MAgoYY9GRNY1y3bIPlcoTtM/4Z5wYoGRY/31uB6InSzKgcY6KVczr1TO d76Q== Received: by 10.68.223.230 with SMTP id qx6mr37496463pbc.159.1354615012092; Tue, 04 Dec 2012 01:56:52 -0800 (PST) Received: from meiko.1dn ([115.42.131.38]) by mx.google.com with ESMTPS id gl9sm752089pbc.51.2012.12.04.01.56.48 (version=SSLv3 cipher=OTHER); Tue, 04 Dec 2012 01:56:51 -0800 (PST) From: Daniel J Blueman To: Bjorn Helgaas Cc: Daniel J Blueman , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Steffen Persvold Subject: [PATCH v3] Add NumaChip remote PCI support Date: Tue, 4 Dec 2012 17:56:24 +0800 Message-Id: <1354614984-6094-1-git-send-email-daniel@numascale-asia.com> X-Mailer: git-send-email 1.7.10.4 X-Gm-Message-State: ALoCoQmGttCdTx7rttn514lzXnXGzD10TUMlwRu9ZIqspNelqh3fL1BkjhimQETNRrn22l0yDZ8U Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add NumaChip-specific PCI access mechanism via MMCONFIG cycles, but preventing access to AMD Northbridges which shouldn't respond. v2: Use PCI_DEVFN in precomputed constant limit; drop unneeded includes v3: Express dependency on MMCONFIG Signed-off-by: Daniel J Blueman --- arch/x86/Kconfig | 2 + arch/x86/include/asm/numachip/numachip.h | 20 +++++ arch/x86/kernel/apic/apic_numachip.c | 2 + arch/x86/pci/Makefile | 1 + arch/x86/pci/numachip.c | 129 ++++++++++++++++++++++++++++++ 5 files changed, 154 insertions(+) create mode 100644 arch/x86/include/asm/numachip/numachip.h create mode 100644 arch/x86/pci/numachip.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 46c3bff..50e8700 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -374,6 +374,7 @@ config X86_NUMACHIP depends on NUMA depends on SMP depends on X86_X2APIC + depends on PCI_MMCONFIG ---help--- Adds support for Numascale NumaChip large-SMP systems. Needed to enable more than ~168 cores. diff --git a/arch/x86/include/asm/numachip/numachip.h b/arch/x86/include/asm/numachip/numachip.h new file mode 100644 index 0000000..fe7f60c --- /dev/null +++ b/arch/x86/include/asm/numachip/numachip.h @@ -0,0 +1,20 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Numascale NumaConnect-specific header file + * + * Copyright (C) 2012 Numascale AS. All rights reserved. + * + * Send feedback to + * + */ + +#ifndef _ASM_X86_NUMACHIP_NUMACHIP_H +#define _ASM_X86_NUMACHIP_NUMACHIP_H + +extern int __init pci_numachip_init(void); + +#endif /* _ASM_X86_NUMACHIP_NUMACHIP_H */ + diff --git a/arch/x86/kernel/apic/apic_numachip.c b/arch/x86/kernel/apic/apic_numachip.c index a65829a..9c2aa89 100644 --- a/arch/x86/kernel/apic/apic_numachip.c +++ b/arch/x86/kernel/apic/apic_numachip.c @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -179,6 +180,7 @@ static int __init numachip_system_init(void) return 0; x86_cpuinit.fixup_cpu_id = fixup_cpu_id; + x86_init.pci.arch_init = pci_numachip_init; map_csrs(); diff --git a/arch/x86/pci/Makefile b/arch/x86/pci/Makefile index 3af5a1e..ee0af58 100644 --- a/arch/x86/pci/Makefile +++ b/arch/x86/pci/Makefile @@ -16,6 +16,7 @@ obj-$(CONFIG_STA2X11) += sta2x11-fixup.o obj-$(CONFIG_X86_VISWS) += visws.o obj-$(CONFIG_X86_NUMAQ) += numaq_32.o +obj-$(CONFIG_X86_NUMACHIP) += numachip.o obj-$(CONFIG_X86_INTEL_MID) += mrst.o diff --git a/arch/x86/pci/numachip.c b/arch/x86/pci/numachip.c new file mode 100644 index 0000000..7307d9d --- /dev/null +++ b/arch/x86/pci/numachip.c @@ -0,0 +1,129 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Numascale NumaConnect-specific PCI code + * + * Copyright (C) 2012 Numascale AS. All rights reserved. + * + * Send feedback to + * + * PCI accessor functions derived from mmconfig_64.c + * + */ + +#include +#include + +static u8 limit __read_mostly; + +static inline char __iomem *pci_dev_base(unsigned int seg, unsigned int bus, unsigned int devfn) +{ + struct pci_mmcfg_region *cfg = pci_mmconfig_lookup(seg, bus); + + if (cfg && cfg->virt) + return cfg->virt + (PCI_MMCFG_BUS_OFFSET(bus) | (devfn << 12)); + return NULL; +} + +static int pci_mmcfg_read_numachip(unsigned int seg, unsigned int bus, + unsigned int devfn, int reg, int len, u32 *value) +{ + char __iomem *addr; + + /* Why do we have this when nobody checks it. How about a BUG()!? -AK */ + if (unlikely((bus > 255) || (devfn > 255) || (reg > 4095))) { +err: *value = -1; + return -EINVAL; + } + + /* Ensure AMD Northbridges don't decode reads to other devices */ + if (unlikely(bus == 0 && devfn >= limit)) { + *value = -1; + return 0; + } + + rcu_read_lock(); + addr = pci_dev_base(seg, bus, devfn); + if (!addr) { + rcu_read_unlock(); + goto err; + } + + switch (len) { + case 1: + *value = mmio_config_readb(addr + reg); + break; + case 2: + *value = mmio_config_readw(addr + reg); + break; + case 4: + *value = mmio_config_readl(addr + reg); + break; + } + rcu_read_unlock(); + + return 0; +} + +static int pci_mmcfg_write_numachip(unsigned int seg, unsigned int bus, + unsigned int devfn, int reg, int len, u32 value) +{ + char __iomem *addr; + + /* Why do we have this when nobody checks it. How about a BUG()!? -AK */ + if (unlikely((bus > 255) || (devfn > 255) || (reg > 4095))) + return -EINVAL; + + /* Ensure AMD Northbridges don't decode writes to other devices */ + if (unlikely(bus == 0 && devfn >= limit)) + return 0; + + rcu_read_lock(); + addr = pci_dev_base(seg, bus, devfn); + if (!addr) { + rcu_read_unlock(); + return -EINVAL; + } + + switch (len) { + case 1: + mmio_config_writeb(addr + reg, value); + break; + case 2: + mmio_config_writew(addr + reg, value); + break; + case 4: + mmio_config_writel(addr + reg, value); + break; + } + rcu_read_unlock(); + + return 0; +} + +const struct pci_raw_ops pci_mmcfg_numachip = { + .read = pci_mmcfg_read_numachip, + .write = pci_mmcfg_write_numachip, +}; + +int __init pci_numachip_init(void) +{ + int ret = 0; + u32 val; + + /* For remote I/O, restrict bus 0 access to the actual number of AMD + Northbridges, which starts at device number 0x18 */ + ret = raw_pci_read(0, 0, PCI_DEVFN(0x18, 0), 0x60, sizeof(val), &val); + if (ret) + goto out; + + /* HyperTransport fabric size in bits 6:4 */ + limit = PCI_DEVFN(0x18 + ((val >> 4) & 7) + 1, 0); + + /* Use NumaChip PCI accessors for non-extended and extended access */ + raw_pci_ops = raw_pci_ext_ops = &pci_mmcfg_numachip; +out: + return ret; +}