From patchwork Mon Apr 16 04:36:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 898456 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=kvm-ppc-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HUu+lgAy"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40PbFK6Qs1z9s1p for ; Mon, 16 Apr 2018 14:36:53 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751189AbeDPEgx (ORCPT ); Mon, 16 Apr 2018 00:36:53 -0400 Received: from mail-pl0-f66.google.com ([209.85.160.66]:43268 "EHLO mail-pl0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751151AbeDPEgw (ORCPT ); Mon, 16 Apr 2018 00:36:52 -0400 Received: by mail-pl0-f66.google.com with SMTP id a39-v6so9306634pla.10 for ; Sun, 15 Apr 2018 21:36:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=8yGcflUL2+FCQCnQscjeXj3uYQ+lsbb0MLaPeH1kzQ8=; b=HUu+lgAyvVCRy4xsCB0BcmalySx7U0mWBAhABXwcDux/3cPHwaaKKvvMkBiLhNDT9V e9Oir+J5W90By6g0qOsiea/43ieJ41ugHPHBrjrqF5jiUhy6h41A+yvbDIF6uA4aHJzL mnr6/QtWOqA0QQRjrpGU9BfIZQ/RLncfYMEIVNsn5Vq5jl8QYYnDgmnsF8GfwOxU4S9C tMnLHL03dvAMW6heQy1EsdJjDz4iBwPuPIevWkbTke4VZZKP0hmcaSSgw5/xTymPs63B vxvFCK3P/TGcfvwHaCriKB/Wx2Tyf/202bsE5Au9LlSRrrs8tsAPdB0ycW/lbWpK/tk1 TGpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=8yGcflUL2+FCQCnQscjeXj3uYQ+lsbb0MLaPeH1kzQ8=; b=HUiLWDmpl8HBflTcYNxVnbimlfg0O7j6rQrHI+qxRDb+F4eEQUWgG8VV6jRa/Kws3J oqwW6QJPPB+GLFz4J44gzEljK3GWe8oYHqYRlzg8LHleba+J8XtubtN9xANXMP/JiaZY IOk+ouWCh0eKvkL7hBQhOjWM2rz2LT62ybNTtiXguXmGYfXWPTmhvAuhGFrthvQOi3MF goh3wDZ57dMRvPLql51uZVA5N9XZSOvu4X+QkLZNhN/+9nt5jDsNJn75OqmhuOzbVMmC fpnr/Aj0xctRRPXDOrLArm792ziO1a/VTJ8gXtSn2ht/aaj3j3H3l3QDo53XlMMUVt8H zIuA== X-Gm-Message-State: ALQs6tByfPH/t0Pq7UXfbqmhfHBvyDCyzqhw837dvTVxI65HexeMTYEA xE7GvHmhktiOxVEK8qxI/Gsn4g== X-Google-Smtp-Source: AIpwx4+qCDM4j37b18MwKbOFo0l/5EnjYeZICOoFwLRq/o7jWgsASCOo2hde9IcT9lW5W56DyWCpPw== X-Received: by 2002:a17:902:4d46:: with SMTP id o6-v6mr8994383plh.243.1523853411602; Sun, 15 Apr 2018 21:36:51 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id y7sm19639605pfn.63.2018.04.15.21.36.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Apr 2018 21:36:50 -0700 (PDT) From: Nicholas Piggin To: kvm-ppc@vger.kernel.org Cc: Nicholas Piggin Subject: [PATCH v2] KVM: PPC: Book3S HV: lockless tlbie for HPT hcalls Date: Mon, 16 Apr 2018 14:36:42 +1000 Message-Id: <20180416043642.10119-1-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 Sender: kvm-ppc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm-ppc@vger.kernel.org tlbies to an LPAR do not have to be serialised since POWER4, MMU_FTR_LOCKLESS_TLBIE can be used to avoid the spin lock in do_tlbies. Testing was done on a POWER9 system in HPT mode, with a -smp 32 guest in HPT mode. 32 instances of the powerpc fork benchmark from selftests were run with --fork, and the results measured. Without this patch, total throughput was about 13.5K/sec, and this is the top of the host profile: 74.52% [k] do_tlbies 2.95% [k] kvmppc_book3s_hv_page_fault 1.80% [k] calc_checksum 1.80% [k] kvmppc_vcpu_run_hv 1.49% [k] kvmppc_run_core After this patch, throughput was about 51K/sec, with this profile: 21.28% [k] do_tlbies 5.26% [k] kvmppc_run_core 4.88% [k] kvmppc_book3s_hv_page_fault 3.30% [k] _raw_spin_lock_irqsave 3.25% [k] gup_pgd_range Signed-off-by: Nicholas Piggin --- Since v1: - Bug fix for unlock !MMU_FTR_LOCKLESS_TLBIE path arch/powerpc/kvm/book3s_hv_rm_mmu.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kvm/book3s_hv_rm_mmu.c b/arch/powerpc/kvm/book3s_hv_rm_mmu.c index e1c083fbe434..4af26cfc4800 100644 --- a/arch/powerpc/kvm/book3s_hv_rm_mmu.c +++ b/arch/powerpc/kvm/book3s_hv_rm_mmu.c @@ -439,6 +439,9 @@ static inline int try_lock_tlbie(unsigned int *lock) unsigned int tmp, old; unsigned int token = LOCK_TOKEN; + if (mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE)) + return 1; + asm volatile("1:lwarx %1,0,%2\n" " cmpwi cr0,%1,0\n" " bne 2f\n" @@ -452,6 +455,13 @@ static inline int try_lock_tlbie(unsigned int *lock) return old == 0; } +static inline void unlock_tlbie_after_sync(unsigned int *lock) +{ + if (mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE)) + return; + WRITE_ONCE(*lock, 0); +} + static void do_tlbies(struct kvm *kvm, unsigned long *rbvalues, long npages, int global, bool need_sync) { @@ -485,7 +495,7 @@ static void do_tlbies(struct kvm *kvm, unsigned long *rbvalues, } asm volatile("eieio; tlbsync; ptesync" : : : "memory"); - kvm->arch.tlbie_lock = 0; + unlock_tlbie_after_sync(&kvm->arch.tlbie_lock); } else { if (need_sync) asm volatile("ptesync" : : : "memory");