From patchwork Thu May 24 17:58:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920037 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHLQ5KFsz9s0w for ; Fri, 25 May 2018 04:03:26 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AUE1kfFr"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHLQ3smBzF1Q8 for ; Fri, 25 May 2018 04:03:26 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AUE1kfFr"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::244; helo=mail-pl0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AUE1kfFr"; dkim-atps=neutral Received: from mail-pl0-x244.google.com (mail-pl0-x244.google.com [IPv6:2607:f8b0:400e:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFW6dHSzDqG0 for ; Fri, 25 May 2018 03:59:11 +1000 (AEST) Received: by mail-pl0-x244.google.com with SMTP id f1-v6so1068104plt.6 for ; Thu, 24 May 2018 10:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+p1PoTaIis7RzZPUh75WZci+HWngrhM6bsHwDuA8wFo=; b=AUE1kfFrOTXCK1kzzGEIhyYsNpbng2UR2j+K1Jj0oAJjz+AxAMwhmD+7rliFqAnhSj BCQag/XKKiK8NLW7Ev1MruWqd9ZEG4WNQMEzzWsF4TqjGd0oVk+E78pDxVwX0tfWQAEP 20qspoeyDeuR0tSQj7B5rdv4gVdJaLv3Xn3TEMdbqqydPZu/BMpw5ITUXtMVkDtJJ2/k fArXpIMHfMAwGskXbiO5GZLSSG8K6c3lv9qWYzx/8K4pCep9UxYBEDxcDO5R/xoTrL6D CbfZ+yP9qDoEFUBT/gsne0CxHy14ghqS5a4d0zWMpjEY1hROV9h8w64L3bSpkYmtD6yU ABxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+p1PoTaIis7RzZPUh75WZci+HWngrhM6bsHwDuA8wFo=; b=kMtuaxPwWU7sCw2dLXmjXBrhYzdZ5HWDzGiVfIeC0AFzQXRgpdoij7x3vgE61uWKmi ywmIgmoeIWIm9NtXn7ik91jolB8rQ1KxTfViVX4Bf35MCfQy5x9E7RqglvSL0IuSBY0b PmpPH5OibORS4akX81EQjlb3Tlaq0sQPuzyFP7HBVNnz1vp6T25zQUhvpBHfrM4ACWqt A8vGnCDbjR3xPYbkI3EJjQdGddAKTYUZC0iIRTfDGN2lToSjsGlGiXH7JHU3vW9NoIV7 ZcJe4ro/rEwr65LQ9VOQsxfIZGSKQO1ANQmVAbOfyi1XVy9se8MPGgcqKuSQ7Et2JtHm HEBQ== X-Gm-Message-State: ALKqPwen9HW1ug/jK2nMciYY8Oyc2qV92qpp8S3fSviMvcSGn69otpQP VN/UB0NLivX8zThV4eR3wlpTbtlo X-Google-Smtp-Source: AB8JxZo3Zs4q0aUxBIpI4I2aWRns26a3i1KJ4AZj5xmcUYIwdEClZ6U2ZjKkPEUWIFVSWUFxNePZlg== X-Received: by 2002:a17:902:8bc4:: with SMTP id r4-v6mr8325703plo.381.1527184749593; Thu, 24 May 2018 10:59:09 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:08 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 1/7] powerpc/64s/radix: do not flush TLB when relaxing access Date: Fri, 25 May 2018 03:58:47 +1000 Message-Id: <20180524175853.19695-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Radix flushes the TLB when updating ptes to increase permissiveness of protection (increase access authority). Book3S does not require TLB flushing in this case, and it is not done on hash. This patch avoids the flush for radix. From Power ISA v3.0B, p.1090: Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates) If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Reference or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority. The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases. This reduces tlbies for a kernel compile workload from 1.28M to 0.95M, tlbiels from 20.17M 19.68M. fork --fork --exec benchmark improved 2.77% (12000->12300). Reviewed-by: Aneesh Kumar K.V Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/pgtable-book3s64.c | 10 +++++++--- arch/powerpc/mm/pgtable.c | 25 +++++++++++++++++++++++-- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 518518fb7c45..994492453f0e 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -31,16 +31,20 @@ int (*register_process_table)(unsigned long base, unsigned long page_size, int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { + struct mm_struct *mm = vma->vm_mm; int changed; #ifdef CONFIG_DEBUG_VM WARN_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); - assert_spin_locked(&vma->vm_mm->page_table_lock); + assert_spin_locked(&mm->page_table_lock); #endif changed = !pmd_same(*(pmdp), entry); if (changed) { - __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp), + __ptep_set_access_flags(mm, pmdp_ptep(pmdp), pmd_pte(entry), address); - flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); + /* See ptep_set_access_flags comments */ + if (atomic_read(&mm->context.copros) > 0) + flush_pmd_tlb_range(vma, address, + address + HPAGE_PMD_SIZE); } return changed; } diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 9f361ae571e9..02a24bce7e51 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -217,14 +217,35 @@ void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { + struct mm_struct *mm = vma->vm_mm; int changed; + entry = set_access_flags_filter(entry, vma, dirty); changed = !pte_same(*(ptep), entry); if (changed) { if (!is_vm_hugetlb_page(vma)) - assert_pte_locked(vma->vm_mm, address); - __ptep_set_access_flags(vma->vm_mm, ptep, entry, address); + assert_pte_locked(mm, address); + __ptep_set_access_flags(mm, ptep, entry, address); +#ifdef CONFIG_PPC_BOOK3S_64 + /* + * Book3S does not require a TLB flush when relaxing access + * restrictions because the core MMU will reload the pte after + * taking an access fault. However the NMMU on POWER9 does not + * re-load the pte, so flush if we have a coprocessor attached + * to this address space. + * + * This could be further refined and pushed out to NMMU drivers + * so TLBIEs are only done for NMMU faults, but this is a more + * minimal fix. The NMMU fault handler does + * get_user_pages_remote or similar to bring the page tables + * in, and this flush_tlb_page will do a global TLBIE because + * the coprocessor is attached to the address space. + */ + if (atomic_read(&mm->context.copros) > 0) + flush_tlb_page(vma, address); +#else flush_tlb_page(vma, address); +#endif } return changed; } From patchwork Thu May 24 17:58:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920039 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHPS0PtFz9s0w for ; Fri, 25 May 2018 04:06:04 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ue8bm5Tp"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHPR5TVCzF1bQ for ; Fri, 25 May 2018 04:06:03 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ue8bm5Tp"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::242; helo=mail-pl0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ue8bm5Tp"; dkim-atps=neutral Received: from mail-pl0-x242.google.com (mail-pl0-x242.google.com [IPv6:2607:f8b0:400e:c01::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFb0NLFzF1Qt for ; Fri, 25 May 2018 03:59:14 +1000 (AEST) Received: by mail-pl0-x242.google.com with SMTP id w19-v6so1515383plq.4 for ; Thu, 24 May 2018 10:59:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZYyUG2AcAEcCY7v57oSPXnLfOMTkjS6Xfwhxl5zjKok=; b=Ue8bm5TpL2Upx9qmCR9/2HzwicGKyQNOXVlp1NPoJoVOMFPQXI7T73+P3kIgqd0b1I riCc5RqF5R351iQgPK7X96Mz4INAtp8UeJ7ePtjN4MPSLbr2pSHXXYMDrM4cGM/87bwe QaDWhRdbvr8ul97XMdYqq/RLBq+o7vct3MV4j/3bUyyKIahfEQS+H2GxosiDnMaRys2/ TowQgEpoD2oo5yd3XcYrCQRsoRLFtGwpwjvPx7KgXKBIfGvODpX2XxpzWcTx4WXrh0QI w/gHA3ZI0Ltw5tUQ03rbDg5kIqnF6SDfNVatptLk6XoxQ8qrFLz5Fg9porvSRnLmORJF /76w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZYyUG2AcAEcCY7v57oSPXnLfOMTkjS6Xfwhxl5zjKok=; b=UqvxBR0dAL1IOTqPSgd63laJMrCSp0CCbBW3+3U4Qhda30PpX4LNRKRj94RTjfuCp7 Qvo/4eBOWB+4rLWl3VqOZ3bM0oYrMIX0//eDznFiJXT13iA5p+l54wxLVvvZ8AtZqt0X JTQHUSS+ECjVBuIX27LTFC3k7B1ZGQgXKGuL/tVITNCX9hchdq5ijiIjkR6pjE5lscVI 7AgG6IdFcocSoxppkNTtQNWKxFjN+MVqurgeve077zhFOJy8LwzPoQXEAmvIEm2Ul/wo vZ4vc76TpJVjvdhGdE8Wuo3XHwpnZrHmYGhwXTlObq4YtkAgsAIo/JhvJ50BgYzcUUe0 cENw== X-Gm-Message-State: ALKqPwee944M14uMNVA1RGx5n8PtEGAas9fe1u/BTors5JHsJ1pztucq jdy4KVlSoaUrvWaPZlWHg/fxuoXI X-Google-Smtp-Source: AB8JxZqkW2hgtOFNhQGVWiEKzCituSjmrRDGI0IwOOvvB1d0DQPAdn/rsOTSif8QdBuZarqR4+Qf8w== X-Received: by 2002:a17:902:6b47:: with SMTP id g7-v6mr8421835plt.251.1527184752697; Thu, 24 May 2018 10:59:12 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:11 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 2/7] powerpc/64s/radix: do not flush TLB on spurious fault Date: Fri, 25 May 2018 03:58:48 +1000 Message-Id: <20180524175853.19695-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" In the case of a spurious fault (which can happen due to a race with another thread that changes the page table), the default Linux mm code calls flush_tlb_page for that address. This is not required because the pte will be re-fetched. Hash does not wire this up to a hardware TLB flush for this reason. This patch avoids the flush for radix. From Power ISA v3.0B, p.1090: Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates) If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Refer- ence or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority. The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases. This reduces tlbies for a kernel compile workload from 0.95M to 0.90M. fork --fork --exec benchmark improved 0.5% (12300->12400). Reviewed-by: Aneesh Kumar K.V Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/tlbflush.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0cac17253513..ebf572ea621e 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -4,7 +4,7 @@ #define MMU_NO_CONTEXT ~0UL - +#include #include #include @@ -137,6 +137,16 @@ static inline void flush_all_mm(struct mm_struct *mm) #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #define flush_all_mm(mm) local_flush_all_mm(mm) #endif /* CONFIG_SMP */ + +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault +static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address) +{ + /* See ptep_set_access_flags comment */ + if (atomic_read(&vma->vm_mm->context.copros) > 0) + flush_tlb_page(vma, address); +} + /* * flush the page walk cache for the address */ From patchwork Thu May 24 17:58:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920044 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHSc2TQtz9s0y for ; Fri, 25 May 2018 04:08:48 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="q7qceJ7X"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHSc0V23zF1RW for ; Fri, 25 May 2018 04:08:48 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="q7qceJ7X"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="q7qceJ7X"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFd2bd8zF1RN for ; Fri, 25 May 2018 03:59:17 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id c11-v6so1512712plr.5 for ; Thu, 24 May 2018 10:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hvag+NSnjD+bOjB7jAHzwIebQMpDE3JgxbO5+IjbYFQ=; b=q7qceJ7XbaMyekFEf+8S8M93K058VUJaRaWg9bW7hUO1HwzQzvsRYbcSnmb1DnFsGA qYmAkTCod3w28UH154vEjjcv76cTq4qqYooiCQTKfHS/3S4OH0GrLij94n4uG0mhRJ1H 4a3BQPPqEPfHtTlWlNCGolJu/T5Ds1DIOdzhYNNEGjkxnDxGXXr9GPYYRAQ9DVSj77HE P6nFQzPLOGcck+kutJPjcBCTL0xu1Pl4c7hlf6JOYC7IQX+KFhyINDZlwtwqKQ/aUhK+ u9v7DJkJq3ZJ5q4VbdNN2Jfnjj7F1q7U6y7j+cCqVXwFokBzpzXgg9UIwBG9dTSIDs2R ke6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hvag+NSnjD+bOjB7jAHzwIebQMpDE3JgxbO5+IjbYFQ=; b=b1xqBvL7FYtJPCDGAS+tsAkr+66EPnLsyOkxVV5kzycocAvLeyRgK4c9ZYShdurpVG iame80B6zNHpYo04LDpqdqis+cQR5S5dTE+fcf2ye1OvR+iVdnzspfHVNGsbaiuh+Bui bAjuAYexBcw1mYMsNBugcgofoHol3rhJhNT1l0rUJ5WYJWdwDvMWb5J7k40gf2ahwd1R 5YJ8Qrn0vpUk9+rFmZdk83H9nC68/eq1WFli1kzlRwMRliWg/N8iYBQekyvIkhDJQi5y agUU8lc+1IYIjOtiDkFgIgdUi2tXT0eQkQP26xY2/KW8Rhg9UHcsqUzzhePhwOfidim0 D2PA== X-Gm-Message-State: ALKqPwdbKfLrVc9rxrQ2Jhqa4XsMM4JdAHDYeFvr+LFs1R6EfyJk66zb dwmQEDfYSLIG9KBwgrxYE8KkeWxs X-Google-Smtp-Source: AB8JxZpH+VB5rAffeGRFZlSET6y4ha3LM7nJIRMBUATiXm5fxT0NycGmFQeR0YQ0RWOZ19XYmgHB/g== X-Received: by 2002:a17:902:2927:: with SMTP id g36-v6mr8304366plb.303.1527184755214; Thu, 24 May 2018 10:59:15 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:14 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 3/7] powerpc/64s/radix: make ptep_get_and_clear_full non-atomic for the full case Date: Fri, 25 May 2018 03:58:49 +1000 Message-Id: <20180524175853.19695-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This matches other architectures, when we know there will be no further accesses to the address (e.g., for teardown), page table entries can be cleared non-atomically. The comments about NMMU are bogus: all MMU notifiers (including NMMU) are released at this point, with their TLBs flushed. An NMMU access at this point would be a bug. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 705193e7192f..fcd92f9b6ec0 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -176,14 +176,8 @@ static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long old_pte; if (full) { - /* - * If we are trying to clear the pte, we can skip - * the DD1 pte update sequence and batch the tlb flush. The - * tlb flush batching is done by mmu gather code. We - * still keep the cmp_xchg update to make sure we get - * correct R/C bit which might be updated via Nest MMU. - */ - old_pte = __radix_pte_update(ptep, ~0ul, 0); + old_pte = pte_val(*ptep); + *ptep = __pte(0); } else old_pte = radix__pte_update(mm, addr, ptep, ~0ul, 0, 0); From patchwork Thu May 24 17:58:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920047 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHWN0VRKz9s19 for ; Fri, 25 May 2018 04:11:12 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nGM8nper"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHWM5v0XzF1Qy for ; Fri, 25 May 2018 04:11:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nGM8nper"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::242; helo=mail-pg0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nGM8nper"; dkim-atps=neutral Received: from mail-pg0-x242.google.com (mail-pg0-x242.google.com [IPv6:2607:f8b0:400e:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFg5VrLzF1Rb for ; Fri, 25 May 2018 03:59:19 +1000 (AEST) Received: by mail-pg0-x242.google.com with SMTP id e21-v6so1127990pgv.0 for ; Thu, 24 May 2018 10:59:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=R+lFYoroCbtw3G0UsBZurh+Z8+Vpx4wQpt9kugh/0Lo=; b=nGM8nperFRHPHChxYhq/1RuYodjma4RoretVIx8ru0vWZR7DTUmrbKen8lheoB6NUt W0iWqH7O+ifp/el6dtrbGfl+KmWRCg1fVyBrvYeUbbhxVYaDLD8CS3t3xjPgwEKMy9G3 /KvgfVF6NBBMpLc3av8l2uuCljeTq4fBYN5GuZoSrrOVzCtyZ2Tp1RxHoJJA/MNwL/w0 Y4gONJQpEi+rB6zX84eFFuZsjugW3qqI8yJCEZ9gB9c6j1SI1ncxMqlK46NVoefXboMP Ntk+B6aYVLAT4fDiHbHkiswHA13z2iU3JBAddybnirYxSWKo/lvdMMMvN7H/kEpCSIek P8Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=R+lFYoroCbtw3G0UsBZurh+Z8+Vpx4wQpt9kugh/0Lo=; b=H5WpzDp1g9Qd8s5YBGxrqYGvnlEn3LEqR44YLS1lqKIqs/oQ6VgHXaB5hQbjDvbIKB 1RbmzGDzbnTo0fb8gQFVjqtpUTZ7Zuyms4NlFe8LUibQRPocLj1s+X1UO5LyjQYW7ZiK ujafyV/pZgIOON9cAYagoAmUnsQgx/+IWOn/6iMnuVYMk88vT9zfUSBQNmZHsA3MLePY 5g4abmMfybHXOJy5c6Ovjo7vPDIY+BiUqE29IvhzJVc1hxTOEEvM4Q2ULbsvi2PFtcvI MwMd55tMzRGdVL+mX+sYjncsFIcS0lAdDJya8Ucb6nfFZqx8fuGdD70ETdU/uZKItjEg DAxg== X-Gm-Message-State: ALKqPwcEwyu5DE8uJ/E5V5u3aCnWsz8OTTnpQcEjpDsW+lSxPs5V5t5X qdqxaGHY4QB/SJ4XXRuk8PDsfULt X-Google-Smtp-Source: AB8JxZrEEZQLYx0DUaOUa8Ea3ZZlCtTvDh+vNd3kKa82maklzHJx9aUhfyJ+T16oJ5/zWeSKBjBVWA== X-Received: by 2002:a62:66dd:: with SMTP id s90-v6mr8275365pfj.123.1527184757841; Thu, 24 May 2018 10:59:17 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:17 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 4/7] powerpc/64s/radix: prefetch user address in update_mmu_cache Date: Fri, 25 May 2018 03:58:50 +1000 Message-Id: <20180524175853.19695-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Prefetch the faulting address in update_mmu_cache to give the page table walker perhaps 100 cycles head start as locks are dropped and the interrupt completed. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/mem.c | 4 +++- arch/powerpc/mm/pgtable-book3s64.c | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index c3c39b02b2ba..8cecda4bd66a 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -509,8 +509,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, */ unsigned long access, trap; - if (radix_enabled()) + if (radix_enabled()) { + prefetch((void *)address); return; + } /* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */ if (!pte_young(*ptep) || address >= TASK_SIZE) diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 994492453f0e..7ce889a7e5ce 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -145,7 +145,8 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { - return; + if (radix_enabled()) + prefetch((void *)addr); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Thu May 24 17:58:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920048 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHZ45DgMz9ryk for ; Fri, 25 May 2018 04:13:32 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BGjxx1Ai"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHZ43gr3zF1Q8 for ; Fri, 25 May 2018 04:13:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BGjxx1Ai"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::244; helo=mail-pf0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BGjxx1Ai"; dkim-atps=neutral Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFk4ff6zF1QS for ; Fri, 25 May 2018 03:59:22 +1000 (AEST) Received: by mail-pf0-x244.google.com with SMTP id q22-v6so1257879pff.11 for ; Thu, 24 May 2018 10:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+B2cPAVQCD9aE8L1VMjvQ8RjTXkhQbFtDIcYirz6jhQ=; b=BGjxx1AiWgl55AnlILSJjsm/ZQlroSWP4WJIK01BhrnqSSUucpj98jsOtMgmEOdVX4 aFlm9iswZYtIBa/gSN1FEJJcOF6Xy72AFHm9Ft6BtSXIaYqTik9cEs7iRQU2M/YcW2bk Maflw64vYZtXqbDr27IVSzIbJad7RFfC2KQZeuOGj93hB+R9AIVQa7+ODRjyuLosEu24 YfdmlyVWDQc77WBbM/ONYuZN438aV+c6dmaQZnZzHAMnDYb87Yac3Zk1qMxNGIa2KJfP X2FzuHVW0tl0aU94oTMCqABZa0OW0SHq1VvQKxS6V+6joNS4DxdmuAXM/FK5crTY7rop 3CVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+B2cPAVQCD9aE8L1VMjvQ8RjTXkhQbFtDIcYirz6jhQ=; b=bYFHLU11RWGQRMl+CE4491l4D0z2w5bbNok8aJ8aj+AN+bFu2pqxGnrxFTq0i3/hKv iL0+bEb3400XPIGlBzZ0Zf2HVw8+A2oAeYz5NRBPJFV4yG3qVC41H1g3wr4FTab+IkZz 3JxODcBPUBdUHdAGDcFO0Rptz5pgQ34iFBW8VIO3yhY0ARnnwzXvk28keOik3U55KJ0Z LMse7p9g2p8FIp2PYq6RcWzRHQLiuWppOJ4X1ijZ+OqGv8BoOV+BWF57Ufw2UdlkO9uz uAXeqT816qsl++HMnCQANoCenw6F6RiiHDw2C3TTWh7dOkCPNV1XInUqGOBt9wB1EeMm nvkw== X-Gm-Message-State: ALKqPwcTJ0bOraZUc7mqhW4yONjA9gUJy0Ocdv1M2Jwfjae0vCKXhYD1 +OnszYoWS6PWFS7jv3HGsM5BeCMp X-Google-Smtp-Source: AB8JxZrpg9r4OnlP23+UU0f209jCFMvVLI5ogsFEemvIYT9aITqkdPXPxCWgdfGxG/V3izigfN6QVg== X-Received: by 2002:aa7:864d:: with SMTP id a13-v6mr8353893pfo.199.1527184760482; Thu, 24 May 2018 10:59:20 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:19 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 5/7] powerpc/64s/radix: avoid ptesync after set_pte and ptep_set_access_flags Date: Fri, 25 May 2018 03:58:51 +1000 Message-Id: <20180524175853.19695-6-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The ISA suggests ptesync after setting a pte, to prevent a table walk initiated by a subsequent access from missing that store and causing a spurious fault. This is an architectual allowance that allows an implementation's page table walker to be incoherent with the store queue. However there is no correctness problem in taking a spurious fault in userspace -- the kernel copes with these at any time, so the updated pte will be found eventually. Spurious kernel faults on vmap memory must be avoided, so a ptesync is put into flush_cache_vmap. On POWER9 so far I have not found a measurable window where this can result in more minor faults, so as an optimisation, remove the costly ptesync from pte updates. If an implementation benefits from ptesync, it would be better to add it back in update_mmu_cache, so it's not done for things like fork(2). fork --fork --exec benchmark improved 5.2% (12400->13100). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 2 -- arch/powerpc/include/asm/cacheflush.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index fcd92f9b6ec0..45bf1e1b1d33 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -209,7 +209,6 @@ static inline void radix__ptep_set_access_flags(struct mm_struct *mm, __radix_pte_update(ptep, 0, new_pte); } else __radix_pte_update(ptep, 0, set); - asm volatile("ptesync" : : : "memory"); } static inline int radix__pte_same(pte_t pte_a, pte_t pte_b) @@ -226,7 +225,6 @@ static inline void radix__set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, int percpu) { *ptep = pte; - asm volatile("ptesync" : : : "memory"); } static inline int radix__pmd_bad(pmd_t pmd) diff --git a/arch/powerpc/include/asm/cacheflush.h b/arch/powerpc/include/asm/cacheflush.h index 11843e37d9cf..e9662648e72d 100644 --- a/arch/powerpc/include/asm/cacheflush.h +++ b/arch/powerpc/include/asm/cacheflush.h @@ -26,6 +26,19 @@ #define flush_cache_vmap(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) +#ifdef CONFIG_BOOK3S_64 +/* + * Book3s has no ptesync after setting a pte, so without this ptesync it's + * possible for a kernel virtual mapping access to return a spurious fault + * if it's accessed right after the pte is set. The page fault handler does + * not expect this type of fault. flush_cache_vmap is not exactly the right + * place to put this, but it seems to work well enough. + */ +#define flush_cache_vmap(start, end) do { asm volatile("ptesync"); } while (0) +#else +#define flush_cache_vmap(start, end) do { } while (0) +#endif + #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 extern void flush_dcache_page(struct page *page); #define flush_dcache_mmap_lock(mapping) do { } while (0) From patchwork Thu May 24 17:58:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920049 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHcm1Mbrz9s19 for ; Fri, 25 May 2018 04:15:52 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hjjfa/1b"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHcl6K7wzF0cy for ; Fri, 25 May 2018 04:15:51 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hjjfa/1b"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hjjfa/1b"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFn3R5SzF1R4 for ; Fri, 25 May 2018 03:59:25 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id u6-v6so1511428pls.9 for ; Thu, 24 May 2018 10:59:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=s6JKj8xkMfEioME2R8WrDjVUwmWcEvmp3+FfT8x+tPE=; b=hjjfa/1b3w1DDNgdJwmwF0hg3iOm9Sdq3zf04hI1fLd9go5Gjde+3QZzHjj/n4Z3Da 85DcnoKeHbt5gr0IUHbkquZ6tNhDnid9ztV1od3SNHYBx8bp3T5WEpQ2ve0rVW2tcLfT 3oPlMgb+SvkDY3TwK4wFXAZZhizK9AvRfjFnoHtVcAo9ytcgH5fFVbF//uzm5HfhSrQ7 D4oaXQbE9f6InxOwPoRpLXWrMw3ey74TWS3/F7IxQBeajX6+i79mb58FcG3KI7/TunkR Tz/lFWPeTG3OTtiJ6TMJ8raC2leHfMFLtDl25inwAE7Jd4CqeN0frIUAmX9d7jPvTPiu JdFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=s6JKj8xkMfEioME2R8WrDjVUwmWcEvmp3+FfT8x+tPE=; b=Ku4wWAgM9b9Hvn9lcpth2vHpEB3DRGnyw9873ZV5mnMr8n54FAlmAphLBH4dTEXqr6 J+diTeq3u9mZfMMAV210p21Y8y+Im8MBIZ/XPXzR8r0k1BLGl4LclYVrSz56WW37+hVq 1UKt8vKlL1Z4Abea45aLHA+XaIiA2y0xermxE2zru0YMjMwOqc2oYdDr1NMoF1IXxEXH Bxmt/S2e1Yz7t0yWOOPx4LHivN89b1sXuviys71sFUvM7VwVWBXLREpL+VO9oHv+Fy6S lYqicIq1VZWIIXxx8vSw5wxJsyhfNSUsMw4+GBAx6cfMDvJ/0m/m/Q6h0by7IRXgNheD NKvw== X-Gm-Message-State: ALKqPwe9gCrdFgWNbviAmZ++fdU4zzmK/ItNJzCmp+hfWswndBjEhybA F8Jmye3GXlX8QLW66COeZRKn2kU0 X-Google-Smtp-Source: AB8JxZo54ZleIsWBTndDDsrYH6zLf9jntjxkMyfH8dD2gRIUC+uj5nRkq3jCP1XP6ibgKzaFpfduig== X-Received: by 2002:a17:902:b489:: with SMTP id y9-v6mr8321284plr.385.1527184763217; Thu, 24 May 2018 10:59:23 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 6/7] powerpc/64s/radix: optimise pte_update Date: Fri, 25 May 2018 03:58:52 +1000 Message-Id: <20180524175853.19695-7-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implementing pte_update with pte_xchg (which uses cmpxchg) is inefficient. A single larx/stcx. works fine, no need for the less efficient cmpxchg sequence. Then remove the memory barriers from the operation. There is a requirement for TLB flushing to load mm_cpumask after the store that reduces pte permissions, which is moved into the TLB flush code. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 25 +++++++++++----------- arch/powerpc/mm/mmu_context.c | 6 ++++-- arch/powerpc/mm/tlb-radix.c | 11 +++++++++- 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 45bf1e1b1d33..cc9437a542cc 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -127,20 +127,21 @@ extern void radix__mark_initmem_nx(void); static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, unsigned long set) { - pte_t pte; - unsigned long old_pte, new_pte; - - do { - pte = READ_ONCE(*ptep); - old_pte = pte_val(pte); - new_pte = (old_pte | set) & ~clr; - - } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); - - return old_pte; + __be64 old_be, tmp_be; + + __asm__ __volatile__( + "1: ldarx %0,0,%3 # pte_update\n" + " andc %1,%0,%5 \n" + " or %1,%1,%4 \n" + " stdcx. %1,0,%3 \n" + " bne- 1b" + : "=&r" (old_be), "=&r" (tmp_be), "=m" (*ptep) + : "r" (ptep), "r" (cpu_to_be64(set)), "r" (cpu_to_be64(clr)) + : "cc" ); + + return be64_to_cpu(old_be); } - static inline unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long clr, diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 0ab297c4cfad..f84e14f23e50 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -57,8 +57,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * in switch_slb(), and/or the store of paca->mm_ctx_id in * copy_mm_to_paca(). * - * On the read side the barrier is in pte_xchg(), which orders - * the store to the PTE vs the load of mm_cpumask. + * On the other side, the barrier is in mm/tlb-radix.c for + * radix which orders earlier stores to clear the PTEs vs + * the load of mm_cpumask. And pte_xchg which does the same + * thing for hash. * * This full barrier is needed by membarrier when switching * between processes after store to rq->curr, before user-space diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 5ac3206c51cc..cdc50398fd60 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -524,6 +524,11 @@ void radix__flush_tlb_mm(struct mm_struct *mm) return; preempt_disable(); + /* + * Order loads of mm_cpumask vs previous stores to clear ptes before + * the invalidate. See barrier in switch_mm_irqs_off + */ + smp_mb(); if (!mm_is_thread_local(mm)) { if (mm_needs_flush_escalation(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); @@ -544,6 +549,7 @@ void radix__flush_all_mm(struct mm_struct *mm) return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (!mm_is_thread_local(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); else @@ -568,6 +574,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (!mm_is_thread_local(mm)) _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); else @@ -630,6 +637,7 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { local = true; full = (end == TLB_FLUSH_ALL || @@ -791,6 +799,7 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { local = true; full = (end == TLB_FLUSH_ALL || @@ -849,7 +858,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) /* Otherwise first do the PWC, then iterate the pages. */ preempt_disable(); - + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } else { From patchwork Thu May 24 17:58:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 920050 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40sHgx2NJ2z9s15 for ; Fri, 25 May 2018 04:18:37 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C8DcXz3Y"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40sHgx0sTjzF1S9 for ; Fri, 25 May 2018 04:18:37 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C8DcXz3Y"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::244; helo=mail-pl0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C8DcXz3Y"; dkim-atps=neutral Received: from mail-pl0-x244.google.com (mail-pl0-x244.google.com [IPv6:2607:f8b0:400e:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40sHFr2wb0zF1QY for ; Fri, 25 May 2018 03:59:28 +1000 (AEST) Received: by mail-pl0-x244.google.com with SMTP id c41-v6so1507396plj.10 for ; Thu, 24 May 2018 10:59:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qb8U7Dx17FMVRbbJa2CWgCznpvQl2LQl8Rcs4WLvIyM=; b=C8DcXz3YPkSvtxWfmIE54HtKCKqsEBAmaMA9WHv1tc0NcEMbaRDsNYAVTgGQMiC21b Zb9bCriQC5mpWyI50gNOXXq3KJqEFrd3YyIHwAXH9a4ZKOFlgdre4bxIQup2bdJmRoT7 F2oV4d7ndeh0x8f0Us9ftqEFKM78ewJLSizfml/DGQj50W+3F4TkkrSeVIrOyjywMKdo E6mY9nMc5B3ZL2t4cpjunHjY+MgdSMCsAVj7bBhSSh1RikbxsXvzwX1kYVxwq5dTzeQj YkrSWuML6IeF2ZrIcVYsF2zpaUZMqjlSayTXULIMugcmpnjVmESnCX9snG/Zmx5Gmlwh Pg8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qb8U7Dx17FMVRbbJa2CWgCznpvQl2LQl8Rcs4WLvIyM=; b=B+regsCWEFp8md+awynrx/rl2kC5URjn06NrcWSgqwXr6xV8nVbDwNnjOvy1UXr9JQ cPBFycXzAkrKwphBkjUyC/y1MHZYKrwwVECjw5c+KTD401BztL87AghrS6LpAxiytqam R7q8vM0ZqkdPKS0sV/Qw4weYaSH9WhZO+4aqxxnVZtIBT5xk+Kk15+XXdo0AaGl2L5No S3Bn6dsXXA4Vfx8zVa7LtVaZGsFSXFDrflTa+9/IsJyTfmb0QwU6gCi4ycQtXJXU1kNQ OIPzoCARAJUQMb1IkJydnhvjqPc7nWc/YB5psCdMWHr8/cdxK3CPrlyTwSgpS1fjhEBq FhmQ== X-Gm-Message-State: ALKqPweGUhIQM+qNJgeAqJRpf0tQ4hWBpUdsZxzMPZZnzoxvCgmV8Bhg 3fX5oaKgF/1Ry+RVz4XLa0Gk+jSh X-Google-Smtp-Source: AB8JxZoLFm44uCKp+Zmk1GfM1K09yCbAU2Hxe2wOPAeZ2ltudcxxczQc4Drqp0QpsSR1rluGaFUYrA== X-Received: by 2002:a17:902:64d0:: with SMTP id y16-v6mr8276305pli.349.1527184765930; Thu, 24 May 2018 10:59:25 -0700 (PDT) Received: from roar.au.ibm.com ([1.128.253.12]) by smtp.gmail.com with ESMTPSA id v14-v6sm34833483pfn.105.2018.05.24.10.59.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 10:59:25 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 7/7] powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask Date: Fri, 25 May 2018 03:58:53 +1000 Message-Id: <20180524175853.19695-8-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524175853.19695-1-npiggin@gmail.com> References: <20180524175853.19695-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When a single-threaded process has a non-local mm_cpumask, try to use that point to flush the TLBs out of other CPUs in the cpumask. An IPI is used for clearing remote CPUs for a few reasons: - An IPI can end lazy TLB use of the mm, which is required to prevent TLB entries being created on the remote CPU. The alternative is to drop lazy TLB switching completely, which costs 7.5% in a context switch ping-pong test betwee a process and kernel idle thread. - An IPI can have remote CPUs flush the entire PID, but the local CPU can flush a specific VA. tlbie would require over-flushing of the local CPU (where the process is running). - A single threaded process that is migrated to a different CPU is likely to have a relatively small mm_cpumask, so IPI is reasonable. No other thread can concurrently switch to this mm, because it must have been given a reference to mm_users by the current thread before it can use_mm. mm_users can be asynchronously incremented (by mm_activate or mmget_not_zero), but those users must use remote mm access and can't use_mm or access user address space. Existing code makes the this assumption already, for example sparc64 has reset mm_cpumask using this condition since the start of history, see arch/sparc/kernel/smp_64.c. This reduces tlbies for a kernel compile workload from 0.90M to 0.12M, tlbiels are increased significantly due to the PID flushing for the cleaning up remote CPUs, and increased local flushes (PID flushes take 128 tlbiels vs 1 tlbie). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 13 +++ arch/powerpc/mm/tlb-radix.c | 148 +++++++++++++++++++++++++++------ 2 files changed, 134 insertions(+), 27 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index a7eabff27a0f..9138baccebb0 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -76,6 +76,19 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } +static inline void mm_reset_thread_local(struct mm_struct *mm) +{ + WARN_ON(atomic_read(&mm->context.copros) > 0); + /* + * It's possible for mm_access to take a reference on mm_users to + * access the remote mm from another thread, but it's not allowed + * to set mm_cpumask, so mm_users may be > 1 here. + */ + WARN_ON(current->mm != mm); + atomic_set(&mm->context.active_cpus, 1); + cpumask_clear(mm_cpumask(mm)); + cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); +} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index cdc50398fd60..67a6e86d3e7e 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -12,6 +12,8 @@ #include #include #include +#include +#include #include #include @@ -504,6 +506,15 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd } EXPORT_SYMBOL(radix__local_flush_tlb_page); +static bool mm_is_singlethreaded(struct mm_struct *mm) +{ + if (atomic_read(&mm->context.copros) > 0) + return false; + if (atomic_read(&mm->mm_users) <= 1 && current->mm == mm) + return true; + return false; +} + static bool mm_needs_flush_escalation(struct mm_struct *mm) { /* @@ -511,10 +522,47 @@ static bool mm_needs_flush_escalation(struct mm_struct *mm) * caching PTEs and not flushing them properly when * RIC = 0 for a PID/LPID invalidate */ - return atomic_read(&mm->context.copros) != 0; + if (atomic_read(&mm->context.copros) > 0) + return true; + return false; } #ifdef CONFIG_SMP +static void do_exit_flush_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + unsigned long pid = mm->context.id; + + if (current->mm == mm) + return; /* Local CPU */ + + if (current->active_mm == mm) { + /* + * Must be a kernel thread because sender is single-threaded. + */ + BUG_ON(current->mm); + mmgrab(&init_mm); + switch_mm(mm, &init_mm, current); + current->active_mm = &init_mm; + mmdrop(mm); + } + _tlbiel_pid(pid, RIC_FLUSH_ALL); +} + +static void exit_flush_lazy_tlbs(struct mm_struct *mm) +{ + /* + * Would be nice if this was async so it could be run in + * parallel with our local flush, but generic code does not + * give a good API for it. Could extend the generic code or + * make a special powerpc IPI for flushing TLBs. + * For now it's not too performance critical. + */ + smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, + (void *)mm, 1); + mm_reset_thread_local(mm); +} + void radix__flush_tlb_mm(struct mm_struct *mm) { unsigned long pid; @@ -530,17 +578,24 @@ void radix__flush_tlb_mm(struct mm_struct *mm) */ smp_mb(); if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } + if (mm_needs_flush_escalation(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); else _tlbie_pid(pid, RIC_FLUSH_TLB); - } else + } else { +local: _tlbiel_pid(pid, RIC_FLUSH_TLB); + } preempt_enable(); } EXPORT_SYMBOL(radix__flush_tlb_mm); -void radix__flush_all_mm(struct mm_struct *mm) +static void __flush_all_mm(struct mm_struct *mm, bool fullmm) { unsigned long pid; @@ -550,12 +605,24 @@ void radix__flush_all_mm(struct mm_struct *mm) preempt_disable(); smp_mb(); /* see radix__flush_tlb_mm */ - if (!mm_is_thread_local(mm)) + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (!fullmm) { + exit_flush_lazy_tlbs(mm); + goto local; + } + } _tlbie_pid(pid, RIC_FLUSH_ALL); - else + } else { +local: _tlbiel_pid(pid, RIC_FLUSH_ALL); + } preempt_enable(); } +void radix__flush_all_mm(struct mm_struct *mm) +{ + __flush_all_mm(mm, false); +} EXPORT_SYMBOL(radix__flush_all_mm); void radix__flush_tlb_pwc(struct mmu_gather *tlb, unsigned long addr) @@ -575,10 +642,16 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, preempt_disable(); smp_mb(); /* see radix__flush_tlb_mm */ - if (!mm_is_thread_local(mm)) + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); - else + } else { +local: _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } preempt_enable(); } @@ -638,14 +711,21 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, preempt_disable(); smp_mb(); /* see radix__flush_tlb_mm */ - if (mm_is_thread_local(mm)) { - local = true; - full = (end == TLB_FLUSH_ALL || - nr_pages > tlb_local_single_page_flush_ceiling); - } else { + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (end != TLB_FLUSH_ALL) { + exit_flush_lazy_tlbs(mm); + goto is_local; + } + } local = false; full = (end == TLB_FLUSH_ALL || nr_pages > tlb_single_page_flush_ceiling); + } else { +is_local: + local = true; + full = (end == TLB_FLUSH_ALL || + nr_pages > tlb_local_single_page_flush_ceiling); } if (full) { @@ -766,7 +846,7 @@ void radix__tlb_flush(struct mmu_gather *tlb) * See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm) { - radix__flush_all_mm(mm); + __flush_all_mm(mm, true); } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { if (!tlb->need_flush_all) radix__flush_tlb_mm(mm); @@ -800,24 +880,32 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, preempt_disable(); smp_mb(); /* see radix__flush_tlb_mm */ - if (mm_is_thread_local(mm)) { - local = true; - full = (end == TLB_FLUSH_ALL || - nr_pages > tlb_local_single_page_flush_ceiling); - } else { + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + if (end != TLB_FLUSH_ALL) { + exit_flush_lazy_tlbs(mm); + goto is_local; + } + } local = false; full = (end == TLB_FLUSH_ALL || nr_pages > tlb_single_page_flush_ceiling); + } else { +is_local: + local = true; + full = (end == TLB_FLUSH_ALL || + nr_pages > tlb_local_single_page_flush_ceiling); } if (full) { - if (!local && mm_needs_flush_escalation(mm)) - also_pwc = true; - - if (local) + if (local) { _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); - else - _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL: RIC_FLUSH_TLB); + } else { + if (mm_needs_flush_escalation(mm)) + also_pwc = true; + + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); + } } else { if (local) _tlbiel_va_range(start, end, pid, page_size, psize, also_pwc); @@ -859,10 +947,16 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) /* Otherwise first do the PWC, then iterate the pages. */ preempt_disable(); smp_mb(); /* see radix__flush_tlb_mm */ - if (mm_is_thread_local(mm)) { - _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); - } else { + if (!mm_is_thread_local(mm)) { + if (unlikely(mm_is_singlethreaded(mm))) { + exit_flush_lazy_tlbs(mm); + goto local; + } _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + goto local; + } else { +local: + _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } preempt_enable();