From patchwork Sun May 20 00:43:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916964 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNgT2J5Hz9s3x for ; Sun, 20 May 2018 10:53:09 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNgT0dzGzDqlg for ; Sun, 20 May 2018 10:53:09 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT20mlVzDrTn for ; Sun, 20 May 2018 10:44:06 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id c41-v6so6671531plj.10 for ; Sat, 19 May 2018 17:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vixH0jmn3eXEDpLEJlRrdy4hsil6LS4CrZDFf+QRtD4=; b=jKTfakWRNPqUkfWKsaozy/aDlKI83/yz05tYuJzVihwzmSPRnoEdK4s1QfVGPH0L1e PmaHVbv0aaTVtCzV7HctzqFhgfxKYGwo9oBGVqlJnQx74Ris6EFi/btcjx5jqv9fqbOn qtBtRwtjQrsu5epacLO1DOdzgw3wf5ajwFCpKbzwHTsqzmVLv+dikFwJkwx/xr3d1jdg wlT/XFy+iCwKhsdAC5RdZC6s8qzGuuKKVkoRiaubBMnnvvLjcDphVflnKBcXNnFNlX8g GK0axXBU+ff/Mq+pOi99tpu8jNv06gZIbrkIKgH4iCE2aJ+FvpYPtw8cdeuC/5bdVd67 uHKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vixH0jmn3eXEDpLEJlRrdy4hsil6LS4CrZDFf+QRtD4=; b=L2izcuRcAUrSTjkta/uVRM70agxm15Jma2GY1mK+YFEn5Xj7KfcYjstQ/fHwu6iFGp 5z1Is0Z9w5AsSH26GFggDDX18J4r80i0vmnS9zvEQjWZkIL2NiRmCir0EuDCMcwAxL2O LrxXwTpd8AEBbDRt1IroYXVv2li2pnzoxlmUuirg6JzvN/r3Fy5prphHf7eC6I8tAbcQ WweDSF7LOtQ2eB6wa/B4v+0aWv9ZPQH5N6yz2Y2yM8kyc2S/ZDF0YXtero/r32EmRjYO W2Kn7IQNNAMxUi/FLxvYTKEAq4YUemfFpZOt3vadq/lJXGor8yHDIQeAACPTQSXz2L33 CgaA== X-Gm-Message-State: ALKqPwc8nm34BXgyE5Ayiubnh7mB6NaGjncohDM5H5N5Qqveqi6711ex UMSMpgLfyoXXjMKLE5wALkOO0A== X-Google-Smtp-Source: AB8JxZrc8Pw7xytXevMRmrC4mwqha0d04ml+vv2/BLs2V4kFBfXOC4wEaFQjc2daO5GzuBU3qm0sJQ== X-Received: by 2002:a17:902:206:: with SMTP id 6-v6mr7134626plc.294.1526777044067; Sat, 19 May 2018 17:44:04 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:03 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 3/7] powerpc/64s/radix: make single threaded mms always flush all translations from non-local CPUs Date: Sun, 20 May 2018 10:43:43 +1000 Message-Id: <20180520004347.19508-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Go one step further, if we're going to put a tlbie on the bus at all, make it count. Make any global invalidation from a single threaded mm do a full PID flush so the mm_cpumask can be reset. The tradeoff is that it will over-flush one time the local CPU's TLB if there was a small number of pages to flush that could be done with specific address tlbies. If the workload is invalidate-heavy enough for this to be a concern, this should be outweighed by the benefit that it can subsequently avoid the global flush. This reduces tlbies for a kernel compile workload from 0.40M to 0.18M, tlbiels are increased from 22.5M to 23.8M because local pid flushes take 128 tlbiels vs 1 for global pid flush. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 45 ++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index d5593a78702a..55f93d66c8d2 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -587,10 +587,16 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, return; preempt_disable(); - if (!mm_is_thread_local(mm)) - _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); - else + if (mm_is_thread_local(mm)) { _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } else { + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else { + _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } + } preempt_enable(); } @@ -659,14 +665,14 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, nr_pages > tlb_single_page_flush_ceiling); } - if (full) { + if (!local && mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else if (full) { if (local) { _tlbiel_pid(pid, RIC_FLUSH_TLB); } else { - if (mm_is_singlethreaded(mm)) { - _tlbie_pid(pid, RIC_FLUSH_ALL); - mm_reset_thread_local(mm); - } else if (mm_needs_flush_escalation(mm)) { + if (mm_needs_flush_escalation(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); } else { _tlbie_pid(pid, RIC_FLUSH_TLB); @@ -824,19 +830,17 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, nr_pages > tlb_single_page_flush_ceiling); } - if (full) { + if (!local && mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else if (full) { if (local) { _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } else { - if (mm_is_singlethreaded(mm)) { - _tlbie_pid(pid, RIC_FLUSH_ALL); - mm_reset_thread_local(mm); - } else { - if (mm_needs_flush_escalation(mm)) - also_pwc = true; + if (mm_needs_flush_escalation(mm)) + also_pwc = true; - _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); - } + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } } else { if (local) @@ -882,7 +886,12 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) if (mm_is_thread_local(mm)) { _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } else { - _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else { + _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + } } preempt_enable();