From patchwork Mon Apr 16 04:32:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898450 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="O0yRKo5z"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Pb8l2TGPz9s27 for ; Mon, 16 Apr 2018 14:32:55 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751908AbeDPEcz (ORCPT ); Mon, 16 Apr 2018 00:32:55 -0400 Received: from mail-pg0-f65.google.com ([74.125.83.65]:39912 "EHLO mail-pg0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751113AbeDPEcy (ORCPT ); Mon, 16 Apr 2018 00:32:54 -0400 Received: by mail-pg0-f65.google.com with SMTP id b9so3236065pgf.6 for ; Sun, 15 Apr 2018 21:32:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OFNKo8tEpRIGrsRwGHyMDp6eZcpQLuio1wWrh50mUAk=; b=O0yRKo5z5rfdVmXvUaGmoQrhZMBBTcDfXWoc3SxRND67VLBTpxlTonwrJ4I+R4KFvl 8C6fiL+H1/91infaXgZxjRMAaBLqoTUdor1Li4RqI5acgk22NJCzHAl8PJdGjhoIm0sf DTifI6ofYyfS9BdCi2r9DGo2X8+tQR1lwpG+/a4M7Md5kFGAQVJ+e7i68vKzTIlDDuEP uT49WWqloZ76mWVUyHwkUsL+mHAW+S+AIyt0yM41i+04Ss6rI9+5R7+A5Dh7VATqIpg2 vSVpRfymFgYFSejTBAHyw6JgIfMf0rra+GGRG0gmMI5hklLnJ2SNEWyimgrDJNo+qM9K I7Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OFNKo8tEpRIGrsRwGHyMDp6eZcpQLuio1wWrh50mUAk=; b=kdafkCMLZTf42av+Hac0GSOqSGheLJs3d2xK1y6ghsWbxuh4Of25Tq9pZXT4FfAr1f 2yt1oD9mid5fex8s6G+Qn+kSPz8Z9PM4WOGcRFwLg+aIRhTZxaIrFBFxrWyS5N7GgP2O aaV3VAAAaTEApxZkvHyjNJVsTBwdq8IL+yKRjP1m67wGnZeF4+MwA+2ISmfl23hTAwME dnLtCGfLXZqgafO5Hylp/ghGOLhVads++TdgUqB9t9AlSdGaGYHqjmBBgaIPnoFwfhDl cqRtPi1sH0se+wE0wAogM3/9D99xQmCS/CFo9LDiiTehtk5PnQnCR+TKqtKRgdxSh1pW kvEg== X-Gm-Message-State: ALQs6tD5MP0fv3tbIvuN9AO/CMMAEuQHmDpItknSafs0rl7z+U6VNVNi qe1OMUuOQMYtWAYiP5THBSK7hg== X-Google-Smtp-Source: AIpwx4/LgmTA25Y4lAn8qYjpOu3Odymkyi8pA/AxW5hhUeAn6zbMYfKKa1/Vvswe/IUcW7pJuX1WUQ== X-Received: by 10.99.177.9 with SMTP id r9mr11805796pgf.60.1523853173650; Sun, 15 Apr 2018 21:32:53 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id e87sm23029614pfd.136.2018.04.15.21.32.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:32:52 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 1/5] KVM: PPC: Book3S HV: radix use correct tlbie sequence in kvmppc_radix_tlbie_page Date: Mon, 16 Apr 2018 14:32:36 +1000 Message-Id: <20180416043240.8796-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180416043240.8796-1-npiggin@gmail.com> References: <20180416043240.8796-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org The standard eieio ; tlbsync ; ptesync must follow tlbie to ensure it is ordered with respect to subsequent operations. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 5d9bafe9a371..81d5ad26f9a1 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -160,7 +160,7 @@ static void kvmppc_radix_tlbie_page(struct kvm *kvm, unsigned long addr, if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - asm volatile("ptesync": : :"memory"); + asm volatile("eieio ; tlbsync ; ptesync": : :"memory"); } unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep, From patchwork Mon Apr 16 04:32:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898451 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Zn+GfMfe"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Pb8q0Ydrz9s21 for ; Mon, 16 Apr 2018 14:32:59 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752593AbeDPEc6 (ORCPT ); Mon, 16 Apr 2018 00:32:58 -0400 Received: from mail-pg0-f65.google.com ([74.125.83.65]:32860 "EHLO mail-pg0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751113AbeDPEc6 (ORCPT ); Mon, 16 Apr 2018 00:32:58 -0400 Received: by mail-pg0-f65.google.com with SMTP id i194so3235457pgd.0 for ; Sun, 15 Apr 2018 21:32:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=te8nXYx8M3VblWujPzDNfLsxk3hhBprNrxgzj8V5Xd8=; b=Zn+GfMfeJJOZfmriJyltJQgoyru+Rzciwm+U8ByQ1gOEOvdfO1gP6mcwuiPPa6Y5gT RX5hQd87GFOVcCBS8SbJIFWf5DUvl0qwrsMsfp+89eBJglgg+HKlBvXCSlElhd1GBdo1 qHbFfnDJ7UVXjeiAucOPG8YVIiJt426NVZEfmpwCgPS3puwdVX0nJo57Xch8Gfe4OhAj 1HDbjsPmwYiPnrIApbnOmQa8IpPdbHqTZV6hh0v4gktpwKssU2PGx+i14c8Be9WlLLzJ FW8RsviegYlf16nU9/shEEDreUpeItln/bnsJ2/GOcFrxx6dd2gnX0dr61rHU4pcQX/a SzOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=te8nXYx8M3VblWujPzDNfLsxk3hhBprNrxgzj8V5Xd8=; b=CKmp3HFGXHQdlWNzBhvVDGAYmZgQksbLeQ0SzXojmIuJBbj8zZe+1kdyVLR6WkhlhM Iozds0xgeDAqIeIzz110rf2/wsOz5pBhmw8W8Pym+2FiRhzChItO7C6xfROjEm1JXelG QwMJZMaWzxCpig0xa1BC8OLQO1hXEq5WZ6GqtKIZPx9HNDjpQAvTmWBs3GSBQYSOfc2/ //rS/IVYbzBYsPhgcBXR2EHA7/1gCFc5RtVFxtCZqrHwnP4vaUXCrk+x+TXDLPRwl5hU wAhdpKiTeAL00b9E2vxnx4L5vjE+kFWN1Z6BiSVsUhCIkgW8kX+cpNpD/O6UcbaqfzUf k3XQ== X-Gm-Message-State: ALQs6tAguiqKJcX/uEl4/WoNjzf05uAUqSFjOM1J8kAvJIQANYvKImEY qf1d6XqRqT8TrohPQn0JSLd1Dg== X-Google-Smtp-Source: AIpwx4+OSiI5parZ/wIcO0YFa55vrmILZNjfv7GYt8US2McueGvwUzv9Wjzv/2HfG2RCXBVneanBeg== X-Received: by 10.101.66.6 with SMTP id c6mr11623243pgq.133.1523853176992; Sun, 15 Apr 2018 21:32:56 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id e87sm23029614pfd.136.2018.04.15.21.32.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:32:55 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 2/5] powerpc/mm/radix: implement LPID based TLB flushes to be used by KVM Date: Mon, 16 Apr 2018 14:32:37 +1000 Message-Id: <20180416043240.8796-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180416043240.8796-1-npiggin@gmail.com> References: <20180416043240.8796-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org Implement a local TLB flush for invalidating an LPID with variants for process or partition scope. And a global TLB flush for invalidating a partition scoped page of an LPID. These will be used by KVM in subsequent patches. Signed-off-by: Nicholas Piggin --- .../include/asm/book3s/64/tlbflush-radix.h | 6 + arch/powerpc/mm/tlb-radix.c | 160 ++++++++++++++++++ 2 files changed, 166 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 19b45ba6caf9..12c02c0e5a4b 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -51,4 +51,10 @@ extern void radix__flush_tlb_all(void); extern void radix__flush_tlb_pte_p9_dd1(unsigned long old_pte, struct mm_struct *mm, unsigned long address); +extern void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size); +extern void radix__local_flush_tlb_lpid(unsigned int lpid); +extern void radix__local_flush_tlb_lpid_guest(unsigned int lpid); + #endif diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 2fba6170ab3f..c2b001dc4dd9 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -119,6 +119,39 @@ static inline void __tlbie_pid(unsigned long pid, unsigned long ric) trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbiel_lpid(unsigned long lpid, int set, + unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rb |= set << PPC_BITLSHIFT(51); + rs = 0; /* LPID comes from LPIDR */ + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 1, rb, rs, ric, prs, r); +} + +static inline void __tlbiel_lpid_guest(unsigned long lpid, int set, + unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = PPC_BIT(52); /* IS = 2 */ + rb |= set << PPC_BITLSHIFT(51); + rs = 0; /* LPID comes from LPIDR */ + prs = 1; /* process scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 1, rb, rs, ric, prs, r); +} + + static inline void __tlbiel_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { @@ -151,6 +184,22 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid, trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void __tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long ap, unsigned long ric) +{ + unsigned long rb,rs,prs,r; + + rb = va & ~(PPC_BITMASK(52, 63)); + rb |= ap << PPC_BITLSHIFT(58); + rs = lpid; + prs = 0; /* partition scoped */ + r = 1; /* radix format */ + + asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) + : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); + trace_tlbie(lpid, 0, rb, rs, ric, prs, r); +} + static inline void fixup_tlbie(void) { unsigned long pid = 0; @@ -162,6 +211,16 @@ static inline void fixup_tlbie(void) } } +static inline void fixup_tlbie_lpid(unsigned long lpid) +{ + unsigned long va = ((1UL << 52) - 1); + + if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) { + asm volatile("ptesync": : :"memory"); + __tlbie_lpid_va(va, lpid, mmu_get_ap(MMU_PAGE_64K), RIC_FLUSH_TLB); + } +} + /* * We use 128 set in radix mode and 256 set in hpt mode. */ @@ -215,6 +274,62 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric) +{ + int set; + + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + asm volatile("ptesync": : :"memory"); + + /* + * Flush the first set of the TLB, and if we're doing a RIC_FLUSH_ALL, + * also flush the entire Page Walk Cache. + */ + __tlbiel_lpid(lpid, 0, ric); + + /* For PWC, only one flush is needed */ + if (ric == RIC_FLUSH_PWC) { + asm volatile("ptesync": : :"memory"); + return; + } + + /* For the remaining sets, just flush the TLB */ + for (set = 1; set < POWER9_TLB_SETS_RADIX ; set++) + __tlbiel_lpid(lpid, set, RIC_FLUSH_TLB); + + asm volatile("ptesync": : :"memory"); + asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory"); +} + +static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric) +{ + int set; + + VM_BUG_ON(mfspr(SPRN_LPID) != lpid); + + asm volatile("ptesync": : :"memory"); + + /* + * Flush the first set of the TLB, and if we're doing a RIC_FLUSH_ALL, + * also flush the entire Page Walk Cache. + */ + __tlbiel_lpid_guest(lpid, 0, ric); + + /* For PWC, only one flush is needed */ + if (ric == RIC_FLUSH_PWC) { + asm volatile("ptesync": : :"memory"); + return; + } + + /* For the remaining sets, just flush the TLB */ + for (set = 1; set < POWER9_TLB_SETS_RADIX ; set++) + __tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB); + + asm volatile("ptesync": : :"memory"); +} + + static inline void __tlbiel_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize) @@ -269,6 +384,17 @@ static inline void _tlbie_va(unsigned long va, unsigned long pid, asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbie_lpid_va(unsigned long va, unsigned long lpid, + unsigned long psize, unsigned long ric) +{ + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + __tlbie_lpid_va(va, lpid, ap, ric); + fixup_tlbie_lpid(lpid); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + static inline void _tlbie_va_range(unsigned long start, unsigned long end, unsigned long pid, unsigned long page_size, unsigned long psize, bool also_pwc) @@ -535,6 +661,40 @@ static int radix_get_mmu_psize(int page_size) return psize; } +/* + * Flush partition scoped LPID address translation for all CPUs. + */ +void radix__flush_tlb_lpid_page(unsigned int lpid, + unsigned long addr, + unsigned long page_size) +{ + int psize = radix_get_mmu_psize(page_size); + + _tlbie_lpid_va(addr, lpid, psize, RIC_FLUSH_TLB); +} +EXPORT_SYMBOL_GPL(radix__flush_tlb_lpid_page); + +/* + * Flush partition scoped translations from LPID (=LPIDR) + */ +void radix__local_flush_tlb_lpid(unsigned int lpid) +{ + _tlbiel_lpid(lpid, RIC_FLUSH_ALL); +} +EXPORT_SYMBOL_GPL(radix__local_flush_tlb_lpid); + +/* + * Flush process scoped translations from LPID (=LPIDR). + * Important difference, the guest normally manages its own translations, + * but some cases e.g., vCPU CPU migration require KVM to flush. + */ +void radix__local_flush_tlb_lpid_guest(unsigned int lpid) +{ + _tlbiel_lpid_guest(lpid, RIC_FLUSH_ALL); +} +EXPORT_SYMBOL_GPL(radix__local_flush_tlb_lpid_guest); + + static void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); From patchwork Mon Apr 16 04:32:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898452 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="dV8nFTTH"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Pb8s5sWQz9s1p for ; Mon, 16 Apr 2018 14:33:01 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752641AbeDPEdB (ORCPT ); Mon, 16 Apr 2018 00:33:01 -0400 Received: from mail-pf0-f194.google.com ([209.85.192.194]:39676 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751113AbeDPEdA (ORCPT ); Mon, 16 Apr 2018 00:33:00 -0400 Received: by mail-pf0-f194.google.com with SMTP id z9so3110585pfe.6 for ; Sun, 15 Apr 2018 21:33:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yTMv2pR3P1xIK+dgMkT8NxITMpA6Wa0KHRqxbJBRLbs=; b=dV8nFTTH5y6I8XO9leN/2uIrHNXvaTNKG0Cg+oUo2lYIeakRdm8GC7dApBoHe/aO+2 3pSPQTZRrBxud1docaseBW5U2+Q8BJZ6mNocgwFPBpC9BqzlkqmsAfKdCZIpoSVAlgzW LJQHt8bx6P2qUj9KrrCxRNTJ9nQq2yNn7WrMVOveFmeLwYyoX8Sk2R0BTQzYz2nsghSI 37GP/ncTTO8lCZD0HMVjd18umlJwO9mkYyi0GEMBt8ky1Url0gtaj9AdkXcaGi7WFo4I WSdPazLM+j8+qi4g+VAD2vcJU6XLpDOwt2FHXRYUscmtc0jWW8rAwWlp7cNt1gRHZcUz KsRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yTMv2pR3P1xIK+dgMkT8NxITMpA6Wa0KHRqxbJBRLbs=; b=NU8e0DZp8QmMGwdpEZHkTeh4e2/GMjTgutr19uXFJRXamakRhQ/TbGPJMBwPhoW+F9 9j9Fpe4UsmEYgGGnkT4qhoeofDGuQsQxL+rd0gmyJQOGKnZqrj04fE0Jz1FI/43r/k3l hsV3VPjcV+RBlXUkYgpGAX0Dwv84FGLW+NUU0Nj3rTXCm5tlCDGkTOa88jk9W+3dV9t6 FOyx1jEzq3+WF/7BRW5SqJ8BqIt603IBw1l66m9B3yp18R6pnCK9PgG9RJhjp3ZABOIN Unzw178FJUc5n7y5WZ1vVA4YUSfqBFfnwWERO/fX2K54h+VbB2TSVGO3NQswIh9k05yH pesQ== X-Gm-Message-State: ALQs6tB1j/d9kkxnMij6XTkiuBJRZ0ecW4OOAe5DRRDwSVVJgY1A7sEe fSzkh0auhM23zmxuUju0q+RFFA== X-Google-Smtp-Source: AIpwx4/2GXnhXsrow2Yu0+CTWng0TkHtSYijvHwBn/ZOLFc2OS6MMxR8+o/G2IPlnwsQjeS//vXZUw== X-Received: by 10.99.183.77 with SMTP id w13mr5068182pgt.231.1523853179951; Sun, 15 Apr 2018 21:32:59 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id e87sm23029614pfd.136.2018.04.15.21.32.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:32:59 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 3/5] KVM: PPC: Book3S HV: radix use the Linux TLB flush function in kvmppc_radix_tlbie_page Date: Mon, 16 Apr 2018 14:32:38 +1000 Message-Id: <20180416043240.8796-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180416043240.8796-1-npiggin@gmail.com> References: <20180416043240.8796-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org This has the advantage of consolidating TLB flush code in fewer places, and it also implements powerpc:tlbie trace events. 1GB pages should be handled without further modification. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 26 +++++++------------------- 1 file changed, 7 insertions(+), 19 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 81d5ad26f9a1..dab6b622011c 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -139,28 +139,16 @@ int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr, return 0; } -#ifdef CONFIG_PPC_64K_PAGES -#define MMU_BASE_PSIZE MMU_PAGE_64K -#else -#define MMU_BASE_PSIZE MMU_PAGE_4K -#endif - static void kvmppc_radix_tlbie_page(struct kvm *kvm, unsigned long addr, unsigned int pshift) { - int psize = MMU_BASE_PSIZE; - - if (pshift >= PMD_SHIFT) - psize = MMU_PAGE_2M; - addr &= ~0xfffUL; - addr |= mmu_psize_defs[psize].ap << 5; - asm volatile("ptesync": : :"memory"); - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - if (cpu_has_feature(CPU_FTR_P9_TLBIE_BUG)) - asm volatile(PPC_TLBIE_5(%0, %1, 0, 0, 1) - : : "r" (addr), "r" (kvm->arch.lpid) : "memory"); - asm volatile("eieio ; tlbsync ; ptesync": : :"memory"); + unsigned long psize = PAGE_SIZE; + + if (pshift) + psize = 1UL << pshift; + + addr &= ~(psize - 1); + radix__flush_tlb_lpid_page(kvm->arch.lpid, addr, psize); } unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep, From patchwork Mon Apr 16 04:32:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898453 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="KBdyZulY"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Pb8x14wfz9s21 for ; Mon, 16 Apr 2018 14:33:05 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752841AbeDPEdE (ORCPT ); Mon, 16 Apr 2018 00:33:04 -0400 Received: from mail-pl0-f66.google.com ([209.85.160.66]:38663 "EHLO mail-pl0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751113AbeDPEdD (ORCPT ); Mon, 16 Apr 2018 00:33:03 -0400 Received: by mail-pl0-f66.google.com with SMTP id c7-v6so9313738plr.5 for ; Sun, 15 Apr 2018 21:33:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vdE1j1c4PH4zcKSRRFNdK9lUI78hS+IfpeavSoAgqxA=; b=KBdyZulYjoHaM8oISxPZ3K/KhrbSxeofEToTWZqwgsZJHdtGpctWXIhCStbo/NmoM7 lu9OAXVRd/yPlNm8FmYRpLF8iC7RJSRq6JmA7WyYn5CDIud8oeMemFlidBugLUArNGOi Ly8OWgBUjkI8F4R3d+oWgqvGk0J+sgnI5IM49y0rsGD9Fs5T2siSE4M69p/YjhQbO5RZ EIAIsBKMcOnjTZ+iqj5wNArXcyD6wDDAXdY7sfcTCor3+AU3UHzVD/rOy0lYgVU46zRh ul8dzq5iN+8zo+pokOEMSiufe2d7oxLvb6h3Bf5oI4C98vimLIlUvZWbeHztE7TBjfZu 3Aww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vdE1j1c4PH4zcKSRRFNdK9lUI78hS+IfpeavSoAgqxA=; b=YcTblG5Omdrux2JQUcaa1FXP44LKKNrs8VAJXcHu3Vh5v5TxwgbUBA4iFkguqB6Zsy YaVFIQWlui0sgNS4ufMejMh2rjqHDfq0ZvAcS1sy1Cvdv2JinNKGxb9GQkWYxs/gTsgY To3C/Q1+UzwJl2RZIdOx8UHqPP/NxC4bC88UPr9FuE490SGlcdDfXHQGVP8PjMWSCn7y VN8Ol9hvkrjm2oIZiIgj/Up/fdPGYEv519QTnEquUbTgwpbq1kykNty0lPI2Py6mZB5/ GeCsX6AgyvNPtGR38u09zFtwCxo5LK4bDvKsjfvGh1KiXghReRKhqCk2u8/ubFYlu5YC YvXA== X-Gm-Message-State: ALQs6tA48LzSihwoj2jKCiE+CGvV029NYUaw1f8VDsp0aVdmHoBPY5U9 OeoWn1St2S3/577739KzJBPbtA== X-Google-Smtp-Source: AIpwx4+Pzt7V1f84gr+TyekNcreZ96EgIclH+v/XiPuwZc1xOHFEB4MbSKQKw2kncKdIVnWtSZ0rbA== X-Received: by 2002:a17:902:8d98:: with SMTP id v24-v6mr3574642plo.146.1523853182887; Sun, 15 Apr 2018 21:33:02 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id e87sm23029614pfd.136.2018.04.15.21.33.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:33:02 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 4/5] KVM: PPC: Book3S HV: radix handle process scoped LPID flush in C, with relocation on Date: Mon, 16 Apr 2018 14:32:39 +1000 Message-Id: <20180416043240.8796-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180416043240.8796-1-npiggin@gmail.com> References: <20180416043240.8796-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org The radix guest code can has fewer restrictions about what context it can run in, so move this flushing out of assembly and have it use the Linux TLB flush implementations introduced previously. This allows powerpc:tlbie trace events to be used. This changes the tlbiel sequence to only execute RIC=2 flush once on the first set flushed, which matches the rest of the Linux flushing. This does not change semantics of the flush. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_hv.c | 26 +++++++++++++++++++++++++ arch/powerpc/kvm/book3s_hv_rmhandlers.S | 13 ++++++------- 2 files changed, 32 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 81e2ea882d97..c1660df41190 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -2901,6 +2901,32 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc) for (sub = 0; sub < core_info.n_subcores; ++sub) spin_unlock(&core_info.vc[sub]->lock); + if (kvm_is_radix(vc->kvm)) { + int tmp = pcpu; + + /* + * Do we need to flush the process scoped TLB for the LPAR? + * + * On POWER9, individual threads can come in here, but the + * TLB is shared between the 4 threads in a core, hence + * invalidating on one thread invalidates for all. + * Thus we make all 4 threads use the same bit here. + * + * Hash must be flushed in realmode in order to use tlbiel. + */ + mtspr(SPRN_LPID, vc->kvm->arch.lpid); + isync(); + + if (cpu_has_feature(CPU_FTR_ARCH_300)) + tmp &= ~0x3UL; + + if (cpumask_test_cpu(tmp, &vc->kvm->arch.need_tlb_flush)) { + radix__local_flush_tlb_lpid_guest(vc->kvm->arch.lpid); + /* Clear the bit after the TLB flush */ + cpumask_clear_cpu(tmp, &vc->kvm->arch.need_tlb_flush); + } + } + /* * Interrupts will be enabled once we get into the guest, * so tell lockdep that we're about to enable interrupts. diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S index bd63fa8a08b5..d4c7bb3e777e 100644 --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S @@ -636,6 +636,10 @@ kvmppc_hv_entry: /* Primary thread switches to guest partition. */ cmpwi r6,0 bne 10f + + /* Radix has already switched LPID and flushed core TLB */ + bne cr7,22f + lwz r7,KVM_LPID(r9) BEGIN_FTR_SECTION ld r6,KVM_SDR1(r9) @@ -647,7 +651,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300) mtspr SPRN_LPID,r7 isync - /* See if we need to flush the TLB */ + /* See if we need to flush the TLB. Hash has to be done in RM */ lhz r6,PACAPACAINDEX(r13) /* test_bit(cpu, need_tlb_flush) */ BEGIN_FTR_SECTION /* @@ -674,15 +678,10 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300) li r7,0x800 /* IS field = 0b10 */ ptesync li r0,0 /* RS for P9 version of tlbiel */ - bne cr7, 29f 28: tlbiel r7 /* On P9, rs=0, RIC=0, PRS=0, R=0 */ addi r7,r7,0x1000 bdnz 28b - b 30f -29: PPC_TLBIEL(7,0,2,1,1) /* for radix, RIC=2, PRS=1, R=1 */ - addi r7,r7,0x1000 - bdnz 29b -30: ptesync + ptesync 23: ldarx r7,0,r6 /* clear the bit after TLB flushed */ andc r7,r7,r8 stdcx. r7,0,r6 From patchwork Mon Apr 16 04:32:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898454 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ErznZEWz"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Pb8z6Xqkz9s2M for ; Mon, 16 Apr 2018 14:33:07 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753151AbeDPEdH (ORCPT ); Mon, 16 Apr 2018 00:33:07 -0400 Received: from mail-pf0-f193.google.com ([209.85.192.193]:42223 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752906AbeDPEdG (ORCPT ); Mon, 16 Apr 2018 00:33:06 -0400 Received: by mail-pf0-f193.google.com with SMTP id o16so9916733pfk.9 for ; Sun, 15 Apr 2018 21:33:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SHblShcL6cWD2qCo1ccHlzhlgXwebqDRmpQvroUmA1k=; b=ErznZEWzOfnq8Xsu9+GbbEGX8p37bdreO83WIfxJhnkpEwx3x3cfeb6QdkDS/U6UX5 Nt58pOgnzTDSIadDxY2lJ5lZdEhMQcZgv6WuqNQLE0IjxdX6NUsp57QqT77upRE0hYvC Q1BM4exjbvYp2qKnRpKGpIhHD+v0ZLu8+YrobwQXw3qUrVXJdPTNlt8oC8lx56WKtKMs anQCEJitkC+yuNtNUdHvXGXPWi3Hi/Oua3a0vPto77JNgcF0dddZFA2IJhm9XvbH9k1N +KPESeJGzwfJ1ay7eY+tYyR8oieoSPpuxo/7ISLPmC6Gs1f7yGub/Gk7bkv7tXtaRL3u gstA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SHblShcL6cWD2qCo1ccHlzhlgXwebqDRmpQvroUmA1k=; b=qNwlkV9lqk659L7EjTLi+2qXOtryJEw+Q9uPqDWcyNdkNA8vDLtepUvoh/mUcVVqT3 MFuLmVN/+Apg3vYHh/vLNsgVAG8Vq8fqvbhiKZqImB2ZaQTU1tG4kErJiUmWTPv9nTS6 cyHNLgodomWjvMLZz92nL/dFkPr0Hmm/4s9zX2qzFFMpU4qGEFa5+cui8g9gjQNOA6+V Oij1mMRPkOtCbKRX47brd1HS+S/ImUnp4apbsU5V7XpdX3OI1tpgWk16tDw52ebreaMC ITJJNlSQ3Awe5kcSfEOQRUP7oBq2oZ2xiU09WgSyhjSPMpU+ZFDmTW9twKW3QHfNylFn HCHg== X-Gm-Message-State: ALQs6tBwJHN3yAdCqj6GR4NrtjRFqVDhrF2bBzD+ZHsg7gPIVG8dOTHg YOdZcrPwQ4e3JASRLa7DbRJhLg== X-Google-Smtp-Source: AIpwx4/wh58LbaHla8sRRo6fz6DMw3jFNnn4LQg6ERkCz3piKfdOeaoc1o8wGIMJHI0SCw4ZrX4/pw== X-Received: by 10.101.97.205 with SMTP id j13mr11560802pgv.266.1523853185789; Sun, 15 Apr 2018 21:33:05 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id e87sm23029614pfd.136.2018.04.15.21.33.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:33:05 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 5/5] KVM: PPC: Book3S HV: radix do not clear partition scoped page table when page fault races with other vCPUs. Date: Mon, 16 Apr 2018 14:32:40 +1000 Message-Id: <20180416043240.8796-6-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180416043240.8796-1-npiggin@gmail.com> References: <20180416043240.8796-1-npiggin@gmail.com> Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org When running a SMP radix guest, KVM can get into page fault / tlbie storms -- hundreds of thousands to the same address from different threads -- due to partition scoped page faults invalidating the page table entry if it was found to be already set up by a racing CPU. What can happen is that guest threads can hit page faults for the same addresses, this can happen when KSM or THP takes out a commonly used page. gRA zero (the interrupt vectors and important kernel text) was a common one. Multiple CPUs will page fault and contend on the same lock, when one CPU sets up the page table and releases the lock, the next will find the new entry and invalidate it before installing its own, which causes other page faults which invalidate that entry, etc. The solution to this is to avoid invalidating the entry or flushing TLBs in case of a race. The pte may still need bits updated, but those are to add R/C or relax access restrictions so no flush is required. This solves the page fault / tlbie storms. Signed-off-by: Nicholas Piggin --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 52 ++++++++++++++++---------- 1 file changed, 33 insertions(+), 19 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index dab6b622011c..2d3af22f90dd 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -199,7 +199,6 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, pud_t *pud, *new_pud = NULL; pmd_t *pmd, *new_pmd = NULL; pte_t *ptep, *new_ptep = NULL; - unsigned long old; int ret; /* Traverse the guest's 2nd-level tree, allocate new levels needed */ @@ -243,6 +242,7 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, pmd = pmd_offset(pud, gpa); if (pmd_is_leaf(*pmd)) { unsigned long lgpa = gpa & PMD_MASK; + pte_t old_pte = *pmdp_ptep(pmd); /* * If we raced with another CPU which has just put @@ -252,18 +252,22 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, ret = -EAGAIN; goto out_unlock; } - /* Valid 2MB page here already, remove it */ - old = kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), - ~0UL, 0, lgpa, PMD_SHIFT); - kvmppc_radix_tlbie_page(kvm, lgpa, PMD_SHIFT); - if (old & _PAGE_DIRTY) { - unsigned long gfn = lgpa >> PAGE_SHIFT; - struct kvm_memory_slot *memslot; - memslot = gfn_to_memslot(kvm, gfn); - if (memslot && memslot->dirty_bitmap) - kvmppc_update_dirty_map(memslot, - gfn, PMD_SIZE); + + /* PTE was previously valid, so update it */ + if (pte_val(old_pte) == pte_val(pte)) { + ret = -EAGAIN; + goto out_unlock; } + + /* Make sure we're weren't trying to take bits away */ + WARN_ON_ONCE(pte_pfn(old_pte) != pte_pfn(pte)); + WARN_ON_ONCE((pte_val(old_pte) & ~pte_val(pte)) & + (_PAGE_PRESENT | _PAGE_READ | _PAGE_WRITE)); + + kvmppc_radix_update_pte(kvm, pmdp_ptep(pmd), + 0, pte_val(pte), lgpa, PMD_SHIFT); + ret = 0; + goto out_unlock; } else if (level == 1 && !pmd_none(*pmd)) { /* * There's a page table page here, but we wanted @@ -274,6 +278,8 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, goto out_unlock; } if (level == 0) { + pte_t old_pte; + if (pmd_none(*pmd)) { if (!new_ptep) goto out_unlock; @@ -281,13 +287,21 @@ static int kvmppc_create_pte(struct kvm *kvm, pte_t pte, unsigned long gpa, new_ptep = NULL; } ptep = pte_offset_kernel(pmd, gpa); - if (pte_present(*ptep)) { - /* PTE was previously valid, so invalidate it */ - old = kvmppc_radix_update_pte(kvm, ptep, _PAGE_PRESENT, - 0, gpa, 0); - kvmppc_radix_tlbie_page(kvm, gpa, 0); - if (old & _PAGE_DIRTY) - mark_page_dirty(kvm, gpa >> PAGE_SHIFT); + old_pte = *ptep; + if (pte_present(old_pte)) { + /* PTE was previously valid, so update it */ + if (pte_val(old_pte) == pte_val(pte)) { + ret = -EAGAIN; + goto out_unlock; + } + + /* Make sure we're weren't trying to take bits away */ + WARN_ON_ONCE(pte_pfn(old_pte) != pte_pfn(pte)); + WARN_ON_ONCE((pte_val(old_pte) & ~pte_val(pte)) & + (_PAGE_PRESENT | _PAGE_READ | _PAGE_WRITE)); + + kvmppc_radix_update_pte(kvm, ptep, 0, + pte_val(pte), gpa, 0); } kvmppc_radix_set_pte_at(kvm, gpa, ptep, pte); } else {