From patchwork Tue Apr 10 12:48:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 896644 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jEwpXU/f"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40L6Rs3g2Dz9s4t for ; Tue, 10 Apr 2018 22:48:57 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752923AbeDJMs5 (ORCPT ); Tue, 10 Apr 2018 08:48:57 -0400 Received: from mail-pf0-f195.google.com ([209.85.192.195]:39550 "EHLO mail-pf0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752759AbeDJMs4 (ORCPT ); Tue, 10 Apr 2018 08:48:56 -0400 Received: by mail-pf0-f195.google.com with SMTP id c78so8216186pfj.6 for ; Tue, 10 Apr 2018 05:48:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=z27GieaoKGerBPW7txqIgdu7iBhwLiD7kgZkaFrsjWY=; b=jEwpXU/fBX8yepxHJFqklQf+NrBFT0TjmGoy5UBk5SQnh0IxxLXlOmWJbQQauOgLeD 5AXUSo4kL0GhhIrX/9+EiLYdiguCSth+nUjw6tHI2SWQp8qRuJWcBwPz/dZvkfJa2Wy1 pnnqU4pTmcy9tHuqqiZkwdo0azrek6cqfcH9a7Kq/ENBbk/4Q6mV8J7hQLVCpKnbaX2C Sbx/aJqFux8N7oyEh+tIeDtCT6hYxZdv0BI/3+TODCQhJaldzplAkMZq8qQcGy0MXnVG qANhNlOzUC3GHUCt7hdHVU8W3ibDNqYDCD9vULTgk1bTO2q0n1VpZzM2QKEfp4w3gICD +moQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=z27GieaoKGerBPW7txqIgdu7iBhwLiD7kgZkaFrsjWY=; b=TV9aKDTy/9CzMqdeIGjS0vTdDnK/RW8ormdnlJJ7IJZWsHYklVAgf1cezbQA9Jit78 YOjNaw/NZezFBewGrzShnedLRdqDDkvjLaiZxuk8KES0Asoo4U0arBjMG0EYFSLWrcST wQgZJA0WG8yF/42kJO/lHEpuVpA0JLTHrT4nDmPARChHCPs2muCNCydQRibw1kxczMtL xi+8JI5BfkfIt8oEg12+YE8770/b7v8g4U7Ng/G8Jozcc6Ew9GMulleIYzNPJLPFLScd 6cFPsnO56kRJNuyWUMxpxNfoNji2WUEKf5uSQcSkAQdF9szDKHOL9fR5KsNqXD3nsF/A wTGw== X-Gm-Message-State: ALQs6tC9KM9RX+FfqzsyOnQMFTw4EYmgCe3LezKmFLx4oPNiX8i04+pp FrDidAoj9f9+7yEFSxEfrpc4oA== X-Google-Smtp-Source: AIpwx48JsJcREUmrovxiMJmNtxyWiLz36qdr98+r4tah2H9NKMtJia7xTMwZuW9ZjOrghxtG5KMf8w== X-Received: by 10.101.97.200 with SMTP id j8mr174773pgv.443.1523364535554; Tue, 10 Apr 2018 05:48:55 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id j20sm6037529pfa.149.2018.04.10.05.48.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Apr 2018 05:48:54 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/5] powerpc/64s/mm: Implement LPID based TLB flushes to be used by KVM Date: Tue, 10 Apr 2018 22:48:38 +1000 Message-Id: <20180410124842.30184-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180410124842.30184-1-npiggin@gmail.com> References: <20180410124842.30184-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org Implent local TLB flush for entire LPID, for hash and radix, and a global TLB flush for a partition scoped page in an LPID, for radix. These will be used by KVM in subsequent patches. Signed-off-by: Nicholas Piggin --- .../include/asm/book3s/64/tlbflush-hash.h | 2 + .../include/asm/book3s/64/tlbflush-radix.h | 5 ++ arch/powerpc/mm/hash_native_64.c | 8 ++ arch/powerpc/mm/tlb-radix.c | 87 +++++++++++++++++++ 4 files changed, 102 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h index 64d02a704bcb..8b328fd87722 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h @@ -53,6 +53,8 @@ static inline void arch_leave_lazy_mmu_mode(void) extern void hash__tlbiel_all(unsigned int action); +extern void hash__local_flush_tlb_lpid(unsigned int lpid); + extern void flush_hash_page(unsigned long vpn, real_pte_t pte, int psize, int ssize, unsigned long flags); extern void flush_hash_range(unsigned long number, int local); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 19b45ba6caf9..2ddaadf3e9ea 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -51,4 +51,9 @@ extern void radix__flush_tlb_all(void); extern void radix__flush_tlb_pte_p9_dd1(unsigned long old_pte, struct mm_struct *mm, unsigned long address); +extern void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size); +extern void radix__local_flush_tlb_lpid(unsigned int lpid); + #endif diff --git a/arch/powerpc/mm/hash_native_64.c b/arch/powerpc/mm/hash_native_64.c index 1d049c78c82a..2f02cd780c19 100644 --- a/arch/powerpc/mm/hash_native_64.c +++ b/arch/powerpc/mm/hash_native_64.c @@ -294,6 +294,14 @@ static inline void tlbie(unsigned long vpn, int psize, int apsize, raw_spin_unlock(&native_tlbie_lock); } +void hash__local_flush_tlb_lpid(unsigned int lpid) +{ + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + hash__tlbiel_all(TLB_INVAL_SCOPE_LPID); +} +EXPORT_SYMBOL_GPL(hash__local_flush_tlb_lpid); + static inline void native_lock_hpte(struct hash_pte *hptep) { unsigned long *word = (unsigned long *)&hptep->v; diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 2fba6170ab3f..f246fb0ac049 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -119,6 +119,22 @@ static inline void __tlbie_pid(unsigned long pid, unsigned long ric) trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbiel_lpid(unsigned long lpid, int set, + unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rb |= set << PPC_BITLSHIFT(51); + rs = 0; /* LPID comes from LPIDR */ + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 1, rb, rs, ric, prs, r); +} + static inline void __tlbiel_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { @@ -151,6 +167,22 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid, trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long ap, unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = va & ~(PPC_BITMASK(52, 63)); + rb |= ap << PPC_BITLSHIFT(58); + rs = lpid; + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 0, rb, rs, ric, prs, r); +} + static inline void fixup_tlbie(void) { unsigned long pid = 0; @@ -215,6 +247,34 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric) +{ + int set; + + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + asm volatile("ptesync": : :"memory"); + + /* + * Flush the first set of the TLB, and if we're doing a RIC_FLUSH_ALL, + * also flush the entire Page Walk Cache. + */ + __tlbiel_lpid(lpid, 0, ric); + + /* For PWC, only one flush is needed */ + if (ric == RIC_FLUSH_PWC) { + asm volatile("ptesync": : :"memory"); + return; + } + + /* For the remaining sets, just flush the TLB */ + for (set = 1; set < POWER9_TLB_SETS_RADIX ; set++) + __tlbiel_lpid(lpid, set, RIC_FLUSH_TLB); + + asm volatile("ptesync": : :"memory"); + asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory"); +} + static inline void __tlbiel_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize) @@ -269,6 +329,17 @@ static inline void _tlbie_va(unsigned long va, unsigned long pid, asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long psize, unsigned long ric) +{ + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + __tlbie_lpid_va(va, lpid, ap, ric); + fixup_tlbie(); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + static inline void _tlbie_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize, bool also_pwc) @@ -535,6 +606,22 @@ static int radix_get_mmu_psize(int page_size) return psize; } +void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size) +{ + int psize = radix_get_mmu_psize(page_size); + + _tlbie_lpid_va(addr, lpid, psize, RIC_FLUSH_TLB); +} +EXPORT_SYMBOL_GPL(radix__flush_tlb_lpid_page); + +void radix__local_flush_tlb_lpid(unsigned int lpid) +{ + _tlbiel_lpid(lpid, RIC_FLUSH_ALL); +} +EXPORT_SYMBOL_GPL(radix__local_flush_tlb_lpid); + static void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); From patchwork Tue Apr 10 12:48:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 896645 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="L7njjOew"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40L6Rw56r1z9s52 for ; Tue, 10 Apr 2018 22:49:00 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752999AbeDJMtA (ORCPT ); Tue, 10 Apr 2018 08:49:00 -0400 Received: from mail-pl0-f67.google.com ([209.85.160.67]:40823 "EHLO mail-pl0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752997AbeDJMs7 (ORCPT ); Tue, 10 Apr 2018 08:48:59 -0400 Received: by mail-pl0-f67.google.com with SMTP id x4-v6so7450366pln.7 for ; Tue, 10 Apr 2018 05:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yTMv2pR3P1xIK+dgMkT8NxITMpA6Wa0KHRqxbJBRLbs=; b=L7njjOewtPijvan7Tt7sxRhstxj4UosyFlZfsMoEnsdmxSR5Vy0gu3Pak1yFqIkqu/ ucrItnVBm3lmdRzrqtAaM2uy8F5rHL5P5nqDr4E9GmC0Zg3KEDTb3IBum/0NFxeaX5n8 V8QETtmsMeHtqOPetp0um/BhyyLAvnUBZTnOho0/1Togh3XH8GKuYVp34sBZ581cwd0A ah2hDwpoF75POumG2T/fDGcHlO94zsPB0wzcAZZrRcnOBn92s3jdgKopfrLFU1yoZqMR B3CPn0J7KJRH6eILUTdaxcih28DGGwOsvSOXJ0BpMzh2xlX4AEmw0gdFGJ+QpiIInZCw NsdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yTMv2pR3P1xIK+dgMkT8NxITMpA6Wa0KHRqxbJBRLbs=; b=lJYPWR+BZOz1z24FJFJ+0Qv5Vf48jL1MbwpOEUQUtSZLJx8s5MwtfW0/7cCNdLf9JR E05SjDEePH9XBHzT09bXk76yJ+s4DNapx9X4n4PS3YiZ/9SDDUIDBfM0VWbQzZ42GPk2 s5hfD/uK0rU3NbnYMI3Ji8p3+A2ar0PPyOp4Glw264Cl7dr5m1EkrheJtrd/xd75JB8S 00t04/sJAMjoPcH7nUi2tf+4DTF4SapuN6/Xp9NUcYdKrHWs3Aj3E6pWhaOky3bGGZNf KJvWpT3UR2/s2+mB8WDmQv2vQ5xIj8P4QzpEMu8qtQRqur3DVJYlUCF1XO5SXT1xX8iO DiGg== X-Gm-Message-State: ALQs6tAlXWm8jf+ElqDuob2IJDzlJvuYXXsviM/j9tTMNtS1LUywbW5i FOpBtIdcFcwZBgoi/5MEYwDQow== X-Google-Smtp-Source: AIpwx4+n0aFypBEPOulsuFc9CjJrzwii7VwauFWmuyVvNOm766GFANbVfTfaokUSmYW1IHrZypBxuA== X-Received: by 2002:a17:902:5a0b:: with SMTP id q11-v6mr282241pli.199.1523364538918; Tue, 10 Apr 2018 05:48:58 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id j20sm6037529pfa.149.2018.04.10.05.48.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Apr 2018 05:48:58 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/5] KVM: PPC: Book3S HV: kvmppc_radix_tlbie_page use Linux flush function Date: Tue, 10 Apr 2018 22:48:39 +1000 Message-Id: <20180410124842.30184-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180410124842.30184-1-npiggin@gmail.com> References: <20180410124842.30184-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org This has the advantage of consolidating TLB flush code in fewer places, and it also implements powerpc:tlbie trace events. 1GB pages should be handled without further modification. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 26 +++++++------------------- 1 file changed, 7 insertions(+), 19 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 81d5ad26f9a1..dab6b622011c 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -139,28 +139,16 @@ int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, return 0; } -#ifdef CONFIG_PPC_64K_PAGES -#define MMU_BASE_PSIZE MMU_PAGE_64K -#else -#define MMU_BASE_PSIZE MMU_PAGE_4K -#endif - static void kvmppc_radix_tlbie_page(struct kvm *kvm, unsigned long addr, unsigned int pshift) { - int psize = MMU_BASE_PSIZE; - - if (pshift >= PMD_SHIFT) - psize = MMU_PAGE_2M; - addr &= ~0xfffUL; - addr |= mmu_psize_defs[psize].ap << 5; - asm volatile("ptesync": : :"memory"); - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - asm volatile("eieio ; tlbsync ; ptesync": : :"memory"); + unsigned long psize = PAGE_SIZE; + + if (pshift) + psize = 1UL << pshift; + + addr &= ~(psize - 1); + radix__flush_tlb_lpid_page(kvm->arch.lpid, addr, psize); } unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep, From patchwork Tue Apr 10 12:48:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 896646 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Gl3XU6KO"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40L6Rz4SNkz9s4Y for ; Tue, 10 Apr 2018 22:49:03 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753036AbeDJMtD (ORCPT ); Tue, 10 Apr 2018 08:49:03 -0400 Received: from mail-pl0-f65.google.com ([209.85.160.65]:34666 "EHLO mail-pl0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752997AbeDJMtC (ORCPT ); Tue, 10 Apr 2018 08:49:02 -0400 Received: by mail-pl0-f65.google.com with SMTP id y12-v6so7343967plt.1 for ; Tue, 10 Apr 2018 05:49:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0bkJfrGvGPz3S8u54aSlAoTLdFnjRWHUVMWzkejBiJs=; b=Gl3XU6KO+Nzzk8PdSfxvUUR6h1iepzJH5YVi9W2FSoPNDDgYTtusJ94V/f5SX0d4Yb afrCCyrsD09tyEGhFivTHDwMnFxGjUqCG5P6Mw0ixEUdmePcwcxrWJiG07mhT5EQ+bBJ IVzs9DSOiSQ+u+Be5Yb7SaWfGyNYn4hZzwoiB9zlDrkZ5Dq1pc+8wVy54W41EHsY53v/ da0SsdoxqQOuiWCtKbMziXSRqbZIt02nad5TUWBkNM/jEVE+yzzfC4e1Fcstsy3uRPGI GaZMJA+hcJUXXZlbrfRWdGowL+qYSEOgaoflnNOHFatsjwD1eXxTgLcR15u2V7vRpBxN hb4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0bkJfrGvGPz3S8u54aSlAoTLdFnjRWHUVMWzkejBiJs=; b=YHKLRscFWgBhuL3yRPjBdmQJfE83YbkkHtZXrELF6egHFeKVzW2A5ZS3ZZEFlvoATV xlgWyWLz4s8z3CTLFLOL0ArIC7goDmVSublOBIH/nfcD2uAzKrt0MjlqC/00ckrbVkhW hBm4tDcVC6U62HKwnyocbPeEc4Hc0yyFwMKxILDbIFqRRmPchX+YhFOou9t5E7CnVU1e LR9rQioxbuso0W5YiDLPRYDJvUnqHwOD7/7uIfDcpQtKJIMkaySraiRGLTPPry30Zq8+ UDI+UOQturgGrLzKk293GRW4Ke8oFuwE+JRntpC2Ja/ZCABbcKeGhyYEuGkAYKRbB/2i MwOA== X-Gm-Message-State: ALQs6tAd5DSvZARgkBPeUXeBNELBDEGXdvtSdAMu2llaZq6uGGOJ1mOe exdBs5ObiakVKuOZb2Da793+4A== X-Google-Smtp-Source: AIpwx491xtZWuSBHa01qe/UrCFnX/udruK8IknMAAyEIEI6Ey5lI5ubliCyBTNADVuYt5wSElrcGIA== X-Received: by 2002:a17:902:4545:: with SMTP id m63-v6mr278386pld.149.1523364541916; Tue, 10 Apr 2018 05:49:01 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id j20sm6037529pfa.149.2018.04.10.05.48.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Apr 2018 05:49:01 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/5] KVM: PPC: Book3S HV: kvmhv_p9_set_lpcr use Linux flush function Date: Tue, 10 Apr 2018 22:48:40 +1000 Message-Id: <20180410124842.30184-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180410124842.30184-1-npiggin@gmail.com> References: <20180410124842.30184-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org The existing flush uses the radix value for sets, and uses R=0 tlbiel instructions. This can't be quite right, but I'm not entirely sure if this is the right way to fix it. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_hv_builtin.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c index 0b9b8e188bfa..577769fbfae9 100644 --- a/arch/powerpc/kvm/book3s_hv_builtin.c +++ b/arch/powerpc/kvm/book3s_hv_builtin.c @@ -676,7 +676,7 @@ static void wait_for_sync(struct kvm_split_mode *sip, int phase) void kvmhv_p9_set_lpcr(struct kvm_split_mode *sip) { - unsigned long rb, set; + struct kvm *kvm = local_paca->kvm_hstate.kvm_vcpu->kvm; /* wait for every other thread to get to real mode */ wait_for_sync(sip, PHASE_REALMODE); @@ -689,14 +689,10 @@ void kvmhv_p9_set_lpcr(struct kvm_split_mode *sip) /* Invalidate the TLB on thread 0 */ if (local_paca->kvm_hstate.tid == 0) { sip->do_set = 0; - asm volatile("ptesync" : : : "memory"); - for (set = 0; set < POWER9_TLB_SETS_RADIX; ++set) { - rb = TLBIEL_INVAL_SET_LPID + - (set << TLBIEL_INVAL_SET_SHIFT); - asm volatile(PPC_TLBIEL(%0, %1, 0, 0, 0) : : - "r" (rb), "r" (0)); - } - asm volatile("ptesync" : : : "memory"); + if (kvm_is_radix(kvm)) + radix__local_flush_tlb_lpid(kvm->arch.lpid); + else + hash__local_flush_tlb_lpid(kvm->arch.lpid); } /* indicate that we have done so and wait for others */ From patchwork Tue Apr 10 12:48:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 896647 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Lv15yWoo"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40L6S40ygdz9s4r for ; Tue, 10 Apr 2018 22:49:08 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753096AbeDJMtH (ORCPT ); Tue, 10 Apr 2018 08:49:07 -0400 Received: from mail-pl0-f68.google.com ([209.85.160.68]:43753 "EHLO mail-pl0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753016AbeDJMtH (ORCPT ); Tue, 10 Apr 2018 08:49:07 -0400 Received: by mail-pl0-f68.google.com with SMTP id a39-v6so7432567pla.10 for ; Tue, 10 Apr 2018 05:49:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RNDO28U9y25nuBDXmhiq0uvDQCJSbiEnA/vvjyB7dxc=; b=Lv15yWoocgH6i6kGLCz5fC8FwHabe0PjWlL/c5ORrR6uN51YoUBdA3fGqYAuHCx+aC z+mgivk1uazSNjUr9IzB1W2cHaGFlvD5ODbqzjOFjUZbXAPwctpxf6C29eRwHUwUIzFn Q9EGXkSQljMbdekdeZLy9EvJkBxiQgEHGfurL4x0HxxjjgWtbFxHGDXefyKsFlSVqSXc ZRjU9pFBmbyU4AvXDjBHB/sH+FlhEHeVE8HvQZSW8k4FLnabrRkdZb5xY8VRti3XmRlo QnpxEvROt9QFzMacFQv3VZDhshgwYGIn5at6fPWxAC+/trRVEHaY+sXYaahqJ/HAsDDe JawQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RNDO28U9y25nuBDXmhiq0uvDQCJSbiEnA/vvjyB7dxc=; b=XHVK0PLXFZXnStvSj9sPBV9sLnMDRlTmI3/7oSF3qFX+t7CX6lUUriUF0gx0UHnQmi Nty9vj1IRSCrDNZbmHD5Zp+m3zm5VoYn65UoV30DijxQ9yRxJ57ToeajauVS1etpiujX o0Y4sXWZfTa0mqJzLjgaRKBhs4RsDjiyg9qLH/0QZSeOYh0JjbQIPxxmS6+l3p0i2i+5 k+ajI6moraL5r1vxlSLiJlM2Lv5/XRnHsP3EudgBGYAZONwmUKZSw8UOD3ZO72hPcefY 1mXF3m0lhEeSM1KKrz8q2GbVVLZ2UDZR9W9w036th7TMgXFmXT3x1R7Th5js5pslbpAh P+Tw== X-Gm-Message-State: ALQs6tB63oARHV/e7eziiHohstWwt0WGZ8j88xPOij13y0lYKjwBAYcy O2rRIFhPrw4EJaTavyhLZiyfdA== X-Google-Smtp-Source: AIpwx4/M9PwVmp4UL9tnXOTROoq/fuol4W1ihIWXM3czbrGCjbBwQcnTkAn+4AMWABpty2nypnlTpw== X-Received: by 2002:a17:902:6547:: with SMTP id d7-v6mr267968pln.253.1523364546277; Tue, 10 Apr 2018 05:49:06 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id j20sm6037529pfa.149.2018.04.10.05.49.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Apr 2018 05:49:05 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/5] KVM: PPC: Book3S HV: handle need_tlb_flush in C before low-level guest entry Date: Tue, 10 Apr 2018 22:48:41 +1000 Message-Id: <20180410124842.30184-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180410124842.30184-1-npiggin@gmail.com> References: <20180410124842.30184-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org Move this flushing out of assembly and have it use Linux TLB flush implementations introduced earlier. This allows powerpc:tlbie trace events to be used. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_hv.c | 21 +++++++++++- arch/powerpc/kvm/book3s_hv_rmhandlers.S | 43 +------------------------ 2 files changed, 21 insertions(+), 43 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 81e2ea882d97..5d4783b5b47a 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -2680,7 +2680,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) int sub; bool thr0_done; unsigned long cmd_bit, stat_bit; - int pcpu, thr; + int pcpu, thr, tmp; int target_threads; int controlled_threads; int trap; @@ -2780,6 +2780,25 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) return; } + /* + * Do we need to flush the TLB for the LPAR? (see TLB comment above) + * On POWER9, individual threads can come in here, but the + * TLB is shared between the 4 threads in a core, hence + * invalidating on one thread invalidates for all. + * Thus we make all 4 threads use the same bit here. + */ + tmp = pcpu; + if (cpu_has_feature(CPU_FTR_ARCH_300)) + tmp &= ~0x3UL; + if (cpumask_test_cpu(tmp, &vc->kvm->arch.need_tlb_flush)) { + if (kvm_is_radix(vc->kvm)) + radix__local_flush_tlb_lpid(vc->kvm->arch.lpid); + else + hash__local_flush_tlb_lpid(vc->kvm->arch.lpid); + /* Clear the bit after the TLB flush */ + cpumask_clear_cpu(tmp, &vc->kvm->arch.need_tlb_flush); + } + kvmppc_clear_host_core(pcpu); /* Decide on micro-threading (split-core) mode */ diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S index bd63fa8a08b5..6a23a0f3ceea 100644 --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S @@ -647,49 +647,8 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300) mtspr SPRN_LPID,r7 isync - /* See if we need to flush the TLB */ - lhz r6,PACAPACAINDEX(r13) /* test_bit(cpu, need_tlb_flush) */ -BEGIN_FTR_SECTION - /* - * On POWER9, individual threads can come in here, but the - * TLB is shared between the 4 threads in a core, hence - * invalidating on one thread invalidates for all. - * Thus we make all 4 threads use the same bit here. - */ - clrrdi r6,r6,2 -END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) - clrldi r7,r6,64-6 /* extract bit number (6 bits) */ - srdi r6,r6,6 /* doubleword number */ - sldi r6,r6,3 /* address offset */ - add r6,r6,r9 - addi r6,r6,KVM_NEED_FLUSH /* dword in kvm->arch.need_tlb_flush */ - li r8,1 - sld r8,r8,r7 - ld r7,0(r6) - and. r7,r7,r8 - beq 22f - /* Flush the TLB of any entries for this LPID */ - lwz r0,KVM_TLB_SETS(r9) - mtctr r0 - li r7,0x800 /* IS field = 0b10 */ - ptesync - li r0,0 /* RS for P9 version of tlbiel */ - bne cr7, 29f -28: tlbiel r7 /* On P9, rs=0, RIC=0, PRS=0, R=0 */ - addi r7,r7,0x1000 - bdnz 28b - b 30f -29: PPC_TLBIEL(7,0,2,1,1) /* for radix, RIC=2, PRS=1, R=1 */ - addi r7,r7,0x1000 - bdnz 29b -30: ptesync -23: ldarx r7,0,r6 /* clear the bit after TLB flushed */ - andc r7,r7,r8 - stdcx. r7,0,r6 - bne 23b - /* Add timebase offset onto timebase */ -22: ld r8,VCORE_TB_OFFSET(r5) + ld r8,VCORE_TB_OFFSET(r5) cmpdi r8,0 beq 37f mftb r6 /* current host timebase */ From patchwork Tue Apr 10 12:48:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 896648 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EmHbJTQ0"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40L6S721ktz9s52 for ; Tue, 10 Apr 2018 22:49:11 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753130AbeDJMtK (ORCPT ); Tue, 10 Apr 2018 08:49:10 -0400 Received: from mail-pl0-f66.google.com ([209.85.160.66]:32854 "EHLO mail-pl0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753063AbeDJMtK (ORCPT ); Tue, 10 Apr 2018 08:49:10 -0400 Received: by mail-pl0-f66.google.com with SMTP id s10-v6so7451143plp.0 for ; Tue, 10 Apr 2018 05:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kt6s5OIHB5AxXxWa6R0dBucBB3m+NQERaPJJzWIT/h0=; b=EmHbJTQ0K70bPMpDEGhrzgHGUvviU9H2z4J7Hu/pwIriIup809vIS+R+K3F7MrPbws +wYdcWh893qsNzNtUBZOiRfTFcj2fy7oJGF9c9iyR14t598G1RCTDiNu1q+GEdRkmO/B oYQlQKgsuIzCoWNB1bnRrwki9scwNwypCcjXiyndFrYLregYjIRf/wAVE+dmy8ccP2Mj zRay6VnROp9uX/JYL2gAkTSTcXvxtciYMA3O8YzN2Upk2hMLHMmx665aQS+nU4zuIE3m G3dtZsn8QlmbS5C+Wkdi4XrwlZ/PN/IpFUqnDIoDHURaExrkvVxBRp1/SmksgsxOsL2Q zewA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kt6s5OIHB5AxXxWa6R0dBucBB3m+NQERaPJJzWIT/h0=; b=AGs0kHqv4+inhj/qesD6Ovaw5yULMihiQLd1kegnkkBQqGO/tmjan7pJUtdikua8qR tofEyO7+yt1QkWv/oz9kncKOpPVJjsVOXcNCjSxv3wc9BfJd05llGxUtG73vJcETKbaZ HHelHSRuujKM9MOZ5dtxPEr8U5cElrBX7+bGFWE+ZMlB2CUz7xXxpakGj3oLwo0CkW2m LtM0l6eROxVbF/B3FFbsUONDRGOoc+rOFIbSoGKgh13SVhLTbsTYgHPk/RZGFaJBxwMv c/huKgDX/yFj/mWBK30OjsspZYTp7AkkaSlZDQJCQeOEL5mRGoKQdX5QIug574kKwGcU XYSg== X-Gm-Message-State: ALQs6tApE8vPGbcT7dewdkwKXhRT+WekBqLboZet7+xSTWGF+dth++R/ IjTExg8gOnnaXnkOskWVNrG8iw== X-Google-Smtp-Source: AIpwx49wrQ9eN9dKMuAgkVIXquPFzubrbwUg4ll63AMY5/QTmmrtBLzp+ZtaeRi465iNFhE3ZRTKjw== X-Received: by 2002:a17:902:464:: with SMTP id 91-v6mr318126ple.126.1523364549426; Tue, 10 Apr 2018 05:49:09 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id j20sm6037529pfa.149.2018.04.10.05.49.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Apr 2018 05:49:08 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 5/5] KVM: PPC: Book3S HV: Radix do not clear partition scoped page table when page fault races with other vCPUs. Date: Tue, 10 Apr 2018 22:48:42 +1000 Message-Id: <20180410124842.30184-6-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180410124842.30184-1-npiggin@gmail.com> References: <20180410124842.30184-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org KVM with an SMP radix guest can get into storms of page faults and tlbies due to the partition scopd page tables being invalidated and TLB flushed if they were found to race with another page fault that set them up. This tends to cause vCPUs to pile up if several hit common addresses, then page faults will get serialized on common locks, and then they each invalidate the previous entry and it's long enough before installing the new entry that will cause more CPUs to hit page faults and they will invalidate that new entry. There doesn't seem to be a need to invalidate in the case of an existing entry. This solves the tlbie storms. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 39 +++++++++++++++----------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index dab6b622011c..4af177d24f6c 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -243,6 +243,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, pmd = pmd_offset(pud, gpa); if (pmd_is_leaf(*pmd)) { unsigned long lgpa = gpa & PMD_MASK; + pte_t old_pte = *pmdp_ptep(pmd); /* * If we raced with another CPU which has just put @@ -252,18 +253,17 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, ret = -EAGAIN; goto out_unlock; } - /* Valid 2MB page here already, remove it */ - old = kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), - ~0UL, 0, lgpa, PMD_SHIFT); - kvmppc_radix_tlbie_page(kvm, lgpa, PMD_SHIFT); - if (old & _PAGE_DIRTY) { - unsigned long gfn = lgpa >> PAGE_SHIFT; - struct kvm_memory_slot *memslot; - memslot = gfn_to_memslot(kvm, gfn); - if (memslot && memslot->dirty_bitmap) - kvmppc_update_dirty_map(memslot, - gfn, PMD_SIZE); + WARN_ON_ONCE(pte_pfn(old_pte) != pte_pfn(pte)); + if (pte_val(old_pte) == pte_val(pte)) { + ret = -EAGAIN; + goto out_unlock; } + + /* Valid 2MB page here already, remove it */ + kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), + 0, pte_val(pte), lgpa, PMD_SHIFT); + ret = 0; + goto out_unlock; } else if (level == 1 && !pmd_none(*pmd)) { /* * There's a page table page here, but we wanted @@ -274,6 +274,8 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, goto out_unlock; } if (level == 0) { + pte_t old_pte; + if (pmd_none(*pmd)) { if (!new_ptep) goto out_unlock; @@ -281,13 +283,16 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, new_ptep = NULL; } ptep = pte_offset_kernel(pmd, gpa); - if (pte_present(*ptep)) { + old_pte = *ptep; + if (pte_present(old_pte)) { /* PTE was previously valid, so invalidate it */ - old = kvmppc_radix_update_pte(kvm, ptep, _PAGE_PRESENT, - 0, gpa, 0); - kvmppc_radix_tlbie_page(kvm, gpa, 0); - if (old & _PAGE_DIRTY) - mark_page_dirty(kvm, gpa >> PAGE_SHIFT); + WARN_ON_ONCE(pte_pfn(old_pte) != pte_pfn(pte)); + if (pte_val(old_pte) == pte_val(pte)) { + ret = -EAGAIN; + goto out_unlock; + } + kvmppc_radix_update_pte(kvm, ptep, 0, + pte_val(pte), gpa, 0); } kvmppc_radix_set_pte_at(kvm, gpa, ptep, pte); } else {