From patchwork Sat Dec 19 23:48:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1418787 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cz2X00hsyz9sVH for ; Sun, 20 Dec 2020 10:49:36 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=LyyIwyTl; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cz2Wz6H9jzDqSD for ; Sun, 20 Dec 2020 10:49:35 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=LyyIwyTl; dkim-atps=neutral Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cz2Vf4wnrzDqS5 for ; Sun, 20 Dec 2020 10:48:25 +1100 (AEDT) Received: by mail-pj1-x1035.google.com with SMTP id f14so3673475pju.4 for ; Sat, 19 Dec 2020 15:48:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H4Iq/9eTVp0IUe98ScS8UQL+J5vj9xeqoWM1trXMCGo=; b=LyyIwyTlunUOiOd5aZ+waTdXF+K6wRyf+sL2faUjAKZo/K1hezSmjMDzJ+P4/yK1XL Jo8eBnwDgbq7LUxho8NysNhK4iASE/wCR9fREHfpwdiwQzaxjuLR5OXXevcvKxaqZXCO Pc+8giGJezd7DKwwRcqTBr2FRlJ4c61onbBIBUEzyhJUAX/1leu9qfOB0OZZ8btY6XCI e3C/P/RhpWm9Qruc8CPxc+x8Wcj3Fqs1Dv40ZZ9InCAHdIljuEzABQ6D2PlALF4cdll2 U3SMpfD/1byFhJ0iJBVHDEWCLZLsfuMlViuhKY0xwa6fr5G96VzCMH+362JtbPYXRIIi tojw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H4Iq/9eTVp0IUe98ScS8UQL+J5vj9xeqoWM1trXMCGo=; b=HO4cOPnmpm+dXEF7RPvkV+u/Q7MdmHEVBxrBxiIqzkfVws9shRC8+d09ZCHte1tzSg eJmxi0yL7XZBNG0ay6xcMdcHknTdmK7cuTPoeA8G+c322TF4bgA0d1u5Y5MDRR2ZxSLi azW9Nx7naTRQ7/boYdWPdzL8HoyvmcWkrXykQ29hTrWCgmJvdxUZ7PeNt/Uthj+CjrH2 xYQ3tF2G8I57+cf9R1GqiD/EJPI/mjiXd/t41hBpygmr+bUFT/qcHI1GNuKahhJYFurr kxtZAdvBgini3UIfgjDYh5vm6f3+2t7m45H/11AKYi4R/4jhR5I9SlFb8bOIDfVbtfat MhYA== X-Gm-Message-State: AOAM532SWji9WeShn4y1NxMkEqSQacEct75hEuJRNFRFO2iS8qtlx4La tjFqJyfUBgIPARgJrpthW1haQvaoK4s= X-Google-Smtp-Source: ABdhPJy675pJt34ULW+n2p+eqQ2zW/Zd8Q4hLcmm8zlqWEciRf98/sTjVoWIACtYoRNk4ZMWy6QqCQ== X-Received: by 2002:a17:90b:68d:: with SMTP id m13mr10751085pjz.164.1608421703176; Sat, 19 Dec 2020 15:48:23 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id k15sm12729275pfh.40.2020.12.19.15.48.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 15:48:22 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/5] powerpc/64s: update_mmu_cache inline the radix test Date: Sun, 20 Dec 2020 09:48:09 +1000 Message-Id: <20201219234813.830747-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201219234813.830747-1-npiggin@gmail.com> References: <20201219234813.830747-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This allows the function to be entirely noped if hash support is compiled out (not possible yet). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/pgtable.h | 11 ++++++++++- arch/powerpc/mm/book3s32/mmu.c | 4 ++-- arch/powerpc/mm/book3s64/hash_utils.c | 7 ++----- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/pgtable.h b/arch/powerpc/include/asm/book3s/pgtable.h index 0e1263455d73..914e9fc7b069 100644 --- a/arch/powerpc/include/asm/book3s/pgtable.h +++ b/arch/powerpc/include/asm/book3s/pgtable.h @@ -35,7 +35,16 @@ extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, * corresponding HPTE into the hash table ahead of time, instead of * waiting for the inevitable extra hash-table miss exception. */ -void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); +void hash__update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); + +static inline void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) +{ +#ifdef CONFIG_PPC64 + if (radix_enabled()) + return; +#endif + hash__update_mmu_cache(vma, address, ptep); +} #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/powerpc/mm/book3s32/mmu.c b/arch/powerpc/mm/book3s32/mmu.c index 859e5bd603ac..c5a570ca37ff 100644 --- a/arch/powerpc/mm/book3s32/mmu.c +++ b/arch/powerpc/mm/book3s32/mmu.c @@ -325,8 +325,8 @@ static void hash_preload(struct mm_struct *mm, unsigned long ea) * * This must always be called with the pte lock held. */ -void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, - pte_t *ptep) +void hash__update_mmu_cache(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep) { if (!mmu_has_feature(MMU_FTR_HPTE_TABLE)) return; diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 73b06adb6eeb..d52a3dee7cf2 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1667,8 +1667,8 @@ static void hash_preload(struct mm_struct *mm, pte_t *ptep, unsigned long ea, * * This must always be called with the pte lock held. */ -void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, - pte_t *ptep) +void hash__update_mmu_cache(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep) { /* * We don't need to worry about _PAGE_PRESENT here because we are @@ -1677,9 +1677,6 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, unsigned long trap; bool is_exec; - if (radix_enabled()) - return; - /* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */ if (!pte_young(*ptep) || address >= TASK_SIZE) return; From patchwork Sat Dec 19 23:48:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1418789 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cz2c30pGTz9sVY for ; Sun, 20 Dec 2020 10:53:07 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=m5+ZRj1b; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cz2c24ntZzDqWH for ; Sun, 20 Dec 2020 10:53:06 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::533; helo=mail-pg1-x533.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=m5+ZRj1b; dkim-atps=neutral Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cz2Vj42zLzDqS5 for ; Sun, 20 Dec 2020 10:48:29 +1100 (AEDT) Received: by mail-pg1-x533.google.com with SMTP id c22so3829388pgg.13 for ; Sat, 19 Dec 2020 15:48:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=icGd0d9KwCkYYneSTMyPusq7Jqg9a5nwUdntjXfQzRA=; b=m5+ZRj1b1liSsf/FJ+Cc3JdtDNsLX72BwBmcJnYpsjJPKM/djoly6DLUSXGJrtVKUW grGwxZQ3uwXGZ4/qQiLFxSpuoVHiKLp67bwfKp7t/DlWR8ve1pZ2l4j5sHJ/3+vdr+mu p4Q0CDcZkxPLMp8v1OLPuksx0bsWP5QO9Q4yY50M6QoP4/iIDftpJjMCk0buVVXO59e+ QajK8vRCWiBUm392FSuUALZ6CYI+zsGrtUAQyfRyL4OkZYbv5DjueOVaeU9/Aiw/6HFF 33mBTzghx+Fapsk1B0FDtfFqYRqdIBaQMlxGPgSVYenLLj8zi+PSxNVHyz7CT2D9/DB+ BrJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=icGd0d9KwCkYYneSTMyPusq7Jqg9a5nwUdntjXfQzRA=; b=K+YuCPi8WkD16/LLBRiQjzDwNRU0Pk78LU36P0xvcXACGRcCbn79r1lgkz6FzhyFOY Kp0gTgA5UATvJYmNMef/uvYvtkNchkdBpWt5DHtUanO1HRURJesXP7I00Bxzhc9gZ0iW JPP6Des9fG3/KsUqAHEGKcXHtgw1OfBp3SsTBIerGE63mIMdWVrb4DolftCXy5APfjeo 7eweTA8j096WbshPrCvSpnIosXwA0n7/fJXCDDYCrNiOvm5KG698QwpEtFe2jVtfWbEl T2CJSSAbfWvpyJoJiqKmbvc9CnvTU4oeYWn6doHXFpC+jpOlEzIYnT5GPJR9JN/Atd/T UemA== X-Gm-Message-State: AOAM530PvC0KntCWFqRyxz4O/7x9T6rI8kEEPSwJQuW8MqN6t1rdCVgw LvHV9EYVqbpd5fRBANrnEKBbV1Ame5U= X-Google-Smtp-Source: ABdhPJzMVVMhU8KdbygNAFx9I2CkNenbx4ubAj6aj6g7Nob62LnZTaUaZLXMic6ZFXxCkokeh4pzZg== X-Received: by 2002:aa7:9afb:0:b029:19e:45d8:b102 with SMTP id y27-20020aa79afb0000b029019e45d8b102mr9609608pfp.48.1608421705449; Sat, 19 Dec 2020 15:48:25 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id k15sm12729275pfh.40.2020.12.19.15.48.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 15:48:25 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/5] powerpc/64s: implement mm_nmmu_has_tlbs Date: Sun, 20 Dec 2020 09:48:10 +1000 Message-Id: <20201219234813.830747-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201219234813.830747-1-npiggin@gmail.com> References: <20201219234813.830747-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Rather than read the copros field directly. This allows coprocessors to be compiled out. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/tlbflush.h | 9 ++++++- arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 3 +-- arch/powerpc/mm/book3s64/radix_pgtable.c | 5 ++-- arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++---------- 4 files changed, 23 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index dcb5c3839d2f..0a7431e954c6 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -14,6 +14,13 @@ enum { TLB_INVAL_SCOPE_LPID = 1, /* invalidate TLBs for current LPID */ }; +static inline bool mm_has_nmmu(struct mm_struct *mm) +{ + if (unlikely(atomic_read(&mm->context.copros) > 0)) + return true; + return false; +} + #ifdef CONFIG_PPC_NATIVE static inline void tlbiel_all(void) { @@ -143,7 +150,7 @@ static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, unsigned long address) { /* See ptep_set_access_flags comment */ - if (atomic_read(&vma->vm_mm->context.copros) > 0) + if (mm_has_nmmu(vma->vm_mm)) flush_tlb_page(vma, address); } diff --git a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c index cb91071eef52..2dfe1416d7db 100644 --- a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c +++ b/arch/powerpc/mm/book3s64/radix_hugetlbpage.c @@ -100,8 +100,7 @@ void radix__huge_ptep_modify_prot_commit(struct vm_area_struct *vma, * To avoid NMMU hang while relaxing access we need to flush the tlb before * we set the new value. */ - if (is_pte_rw_upgrade(pte_val(old_pte), pte_val(pte)) && - (atomic_read(&mm->context.copros) > 0)) + if (is_pte_rw_upgrade(pte_val(old_pte), pte_val(pte)) && mm_has_nmmu(mm)) radix__flush_hugetlb_page(vma, addr); set_huge_pte_at(vma->vm_mm, addr, ptep, pte); diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 98f0b243c1ab..9495206b9b91 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1042,7 +1042,7 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, * To avoid NMMU hang while relaxing access, we need mark * the pte invalid in between. */ - if ((change & _PAGE_RW) && atomic_read(&mm->context.copros) > 0) { + if ((change & _PAGE_RW) && mm_has_nmmu(mm)) { unsigned long old_pte, new_pte; old_pte = __radix_pte_update(ptep, _PAGE_PRESENT, _PAGE_INVALID); @@ -1075,8 +1075,7 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, * we set the new value. We need to do this only for radix, because hash * translation does flush when updating the linux pte. */ - if (is_pte_rw_upgrade(pte_val(old_pte), pte_val(pte)) && - (atomic_read(&mm->context.copros) > 0)) + if (is_pte_rw_upgrade(pte_val(old_pte), pte_val(pte)) && mm_has_nmmu(mm)) radix__flush_tlb_page(vma, addr); set_pte_at(mm, addr, ptep, pte); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 3455947a1b08..6633b47abee8 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -371,7 +371,7 @@ static inline void _tlbiel_pid_multicast(struct mm_struct *mm, * these paths, so while coprocessors must use tlbie, we can not * optimise away the tlbiel component. */ - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); } @@ -504,7 +504,7 @@ static inline void _tlbiel_va_multicast(struct mm_struct *mm, struct cpumask *cpus = mm_cpumask(mm); struct tlbiel_va t = { .va = va, .pid = pid, .psize = psize, .ric = ric }; on_each_cpu_mask(cpus, do_tlbiel_va, &t, 1); - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) _tlbie_va(va, pid, psize, RIC_FLUSH_TLB); } @@ -558,7 +558,7 @@ static inline void _tlbiel_va_range_multicast(struct mm_struct *mm, .psize = psize, .also_pwc = also_pwc }; on_each_cpu_mask(cpus, do_tlbiel_va_range, &t, 1); - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) _tlbie_va_range(start, end, pid, page_size, psize, also_pwc); } @@ -634,9 +634,7 @@ static bool mm_needs_flush_escalation(struct mm_struct *mm) * caching PTEs and not flushing them properly when * RIC = 0 for a PID/LPID invalidate */ - if (atomic_read(&mm->context.copros) > 0) - return true; - return false; + return mm_has_nmmu(mm); } /* @@ -759,8 +757,8 @@ static enum tlb_flush_type flush_type_needed(struct mm_struct *mm, bool fullmm) return FLUSH_TYPE_LOCAL; } - /* Coprocessors require TLBIE to invalidate nMMU. */ - if (atomic_read(&mm->context.copros) > 0) + /* The nest MMU requires TLBIE to invalidate its TLBs. */ + if (mm_has_nmmu(mm)) return FLUSH_TYPE_GLOBAL; /* @@ -833,7 +831,7 @@ void radix__flush_tlb_mm(struct mm_struct *mm) if (!mmu_has_feature(MMU_FTR_GTSE)) { unsigned long tgt = H_RPTI_TARGET_CMMU; - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, H_RPTI_TYPE_TLB, H_RPTI_PAGE_ALL, 0, -1UL); @@ -870,7 +868,7 @@ static void __flush_all_mm(struct mm_struct *mm, bool fullmm) unsigned long type = H_RPTI_TYPE_TLB | H_RPTI_TYPE_PWC | H_RPTI_TYPE_PRT; - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, type, H_RPTI_PAGE_ALL, 0, -1UL); @@ -911,7 +909,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, pg_sizes = psize_to_rpti_pgsize(psize); size = 1UL << mmu_psize_to_shift(psize); - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, H_RPTI_TYPE_TLB, pg_sizes, vmaddr, @@ -1024,7 +1022,7 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm, if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) pg_sizes |= psize_to_rpti_pgsize(MMU_PAGE_2M); - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, H_RPTI_TYPE_TLB, pg_sizes, start, end); @@ -1216,7 +1214,7 @@ static __always_inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, if (also_pwc) type |= H_RPTI_TYPE_PWC; - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, type, pg_sizes, start, end); } else if (flush_pid) { @@ -1293,7 +1291,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) H_RPTI_TYPE_PRT; pg_sizes = psize_to_rpti_pgsize(mmu_virtual_psize); - if (atomic_read(&mm->context.copros) > 0) + if (mm_has_nmmu(mm)) tgt |= H_RPTI_TARGET_NMMU; pseries_rpt_invalidate(pid, tgt, type, pg_sizes, addr, end); From patchwork Sat Dec 19 23:48:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1418790 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cz2dt74JTz9sVJ for ; Sun, 20 Dec 2020 10:54:42 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=rrypSkDa; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cz2dt4FTWzDqVK for ; Sun, 20 Dec 2020 10:54:42 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102d; helo=mail-pj1-x102d.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=rrypSkDa; dkim-atps=neutral Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cz2Vk6HkQzDqS5 for ; Sun, 20 Dec 2020 10:48:30 +1100 (AEDT) Received: by mail-pj1-x102d.google.com with SMTP id f14so3673986pju.4 for ; Sat, 19 Dec 2020 15:48:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QybGjdSIsDRYE3ibaVlzKWhdb3UETSBreqtTrYRLhIM=; b=rrypSkDa1PmIimWyTQ7VP+kpEnbpeFzKqKYsSklf9wXqBKhuTvJngnd/SIbVJSqwjc 4aG+xkOBZINERmFB1R1KCuTGRLK/CdqUlxfduvKKr8hPIsZ89hmj7yD3s1xnSKNggwgp i74S9f1oz1jwQ3Lbylr+m4uUa8sjl6cQrQoi/UplNunWsJ8iEOyGev8h4AB1gjRWRj1X +a9V0SFAEuvOD25EO/O8yU6ZdCMN17bmE2OSvQDKYCSlczn7wlnUtNpSENaFbvmB+Uhb KxXryLBPYsO1c6Pq23q5FpK7QnDSueESyNT9gC9/RGGZxhRwtemXyv71eSK+9xGoOot0 t3HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QybGjdSIsDRYE3ibaVlzKWhdb3UETSBreqtTrYRLhIM=; b=b/QE0FHtWiOLhXMYfTwz0XPLAoBTaGt6xmhMoquvdtu+URQkH8iAnCo/9rlIEozE8d HYtx2+zHijnCUTn+Q/FnNF2YLHp6U9iodge/ImK01xtxcbTWyPMi6LCEE7aTX1O20yjp sugLxQ9/XDC58rXAhiAB76gV1GWwhZpuGsIjNZoY1eSmkjxQyTevPf6WezPebZGUJHd/ cfcWKr23kpfhUMmAXkkGNe/+5VxoGabl5OAfQX9PMgbPY5g2FDeCKnVkbDGbQhzVwVAY BgmaGnvJkJ9U67Nz3gsx5F/ORpbtfTm0V+ObS6987csnsT5T0s7lDLkdekAN94EsrEUl plog== X-Gm-Message-State: AOAM533J074n5+nzlhH6p0jpQD78O8VSBohX+/54RMAGaUKStdx8ht5W /W6JILCDrpuC/rIBjQ6kItlpxHuu47U= X-Google-Smtp-Source: ABdhPJz9jpZknV38JHEgZteoqTtzW8+vca+nUoyNaD4brssfm6K/m9l3Jv0R68FO8rP5hzMNtMG5bw== X-Received: by 2002:a17:902:fe87:b029:da:5d3b:4a84 with SMTP id x7-20020a170902fe87b02900da5d3b4a84mr10746327plm.51.1608421707942; Sat, 19 Dec 2020 15:48:27 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id k15sm12729275pfh.40.2020.12.19.15.48.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 15:48:27 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/5] powerpc/64s: add CONFIG_PPC_NMMU for nest MMU support Date: Sun, 20 Dec 2020 09:48:11 +1000 Message-Id: <20201219234813.830747-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201219234813.830747-1-npiggin@gmail.com> References: <20201219234813.830747-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This allows some nest MMU features to be compiled away if coprocessor support is not selected. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/mmu.h | 2 ++ arch/powerpc/include/asm/book3s/64/tlbflush.h | 2 ++ arch/powerpc/include/asm/mmu_context.h | 5 +++-- arch/powerpc/platforms/Kconfig | 3 +++ arch/powerpc/platforms/powernv/Kconfig | 1 + 6 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index ae7391627054..4376bf4c53b4 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -888,6 +888,7 @@ config PPC_PROT_SAO_LPAR config PPC_COPRO_BASE bool + select PPC_NMMU if PPC_BOOK3S_64 config SCHED_SMT bool "SMT (Hyperthreading) scheduler support" diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h index 995bbcdd0ef8..07850d68a624 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu.h +++ b/arch/powerpc/include/asm/book3s/64/mmu.h @@ -103,8 +103,10 @@ typedef struct { /* Number of bits in the mm_cpumask */ atomic_t active_cpus; +#ifdef CONFIG_PPC_NMMU /* Number of users of the external (Nest) MMU */ atomic_t copros; +#endif /* Number of user space windows opened in process mm_context */ atomic_t vas_windows; diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0a7431e954c6..c70a82851f78 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -16,8 +16,10 @@ enum { static inline bool mm_has_nmmu(struct mm_struct *mm) { +#ifdef CONFIG_PPC_NMMU if (unlikely(atomic_read(&mm->context.copros) > 0)) return true; +#endif return false; } diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index d5821834dba9..53eac0cc4929 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -143,6 +143,7 @@ static inline void dec_mm_active_cpus(struct mm_struct *mm) atomic_dec(&mm->context.active_cpus); } +#ifdef CONFIG_PPC_NMMU static inline void mm_context_add_copro(struct mm_struct *mm) { /* @@ -187,6 +188,7 @@ static inline void mm_context_remove_copro(struct mm_struct *mm) dec_mm_active_cpus(mm); } } +#endif /* * vas_windows counter shows number of open windows in the mm @@ -218,8 +220,7 @@ static inline void mm_context_remove_vas_window(struct mm_struct *mm) #else static inline void inc_mm_active_cpus(struct mm_struct *mm) { } static inline void dec_mm_active_cpus(struct mm_struct *mm) { } -static inline void mm_context_add_copro(struct mm_struct *mm) { } -static inline void mm_context_remove_copro(struct mm_struct *mm) { } +static inline bool mm_has_nmmu(struct mm_struct *mm) { return false; } #endif diff --git a/arch/powerpc/platforms/Kconfig b/arch/powerpc/platforms/Kconfig index 7a5e8f4541e3..b4b04b3f98d1 100644 --- a/arch/powerpc/platforms/Kconfig +++ b/arch/powerpc/platforms/Kconfig @@ -191,6 +191,9 @@ config PPC_INDIRECT_MMIO config PPC_IO_WORKAROUNDS bool +config PPC_NMMU + bool + source "drivers/cpufreq/Kconfig" menu "CPUIdle driver" diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index 619b093a0657..145009d74457 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -36,6 +36,7 @@ config PPC_MEMTRACE config PPC_VAS bool "IBM Virtual Accelerator Switchboard (VAS)" depends on PPC_POWERNV && PPC_64K_PAGES + select PPC_NMMU default y help This enables support for IBM Virtual Accelerator Switchboard (VAS). From patchwork Sat Dec 19 23:48:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1418791 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cz2hL6F7gz9sVY for ; Sun, 20 Dec 2020 10:56:50 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Ve9Rmq4e; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cz2hL5l2mzDqTL for ; Sun, 20 Dec 2020 10:56:50 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Ve9Rmq4e; dkim-atps=neutral Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cz2Vn3fb7zDqT6 for ; Sun, 20 Dec 2020 10:48:32 +1100 (AEDT) Received: by mail-pj1-x102e.google.com with SMTP id z12so3682726pjn.1 for ; Sat, 19 Dec 2020 15:48:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YkWZE9sjUg9xxYL4ZcA+07tjebgpZAPhGBWV+5f2kKY=; b=Ve9Rmq4el4TAX9ersDo7lFAQUnAEDCVJyoCGeoIR/si+qQUPgTg+lBuKJd7tls8qvT q08UTOqu2szrN5E7MeyTdnwKsObPb97Fl38R5hDgvDxDNPKGOqdzK/3RaDTmvsk4uxW9 DBE8/C+wCmeFMMEYnDtt5rz3Iqo5qzxMNgI3mhQmysTJf8Uxd16UaosTIQ+dTGmlbmlf IWkPMeemtbO4RtgZDPawdxB0Ys18IsPgKAHCEd7cQ4TsWnH0+PwZ7eQVT5t//0crluYd XO9TpDJB6AgH5K0SfOOGzTqvtzIkXDOzLBcSUK7t5ufKLfT4ws6FMqVM5sH12vcbQnc+ Ketw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YkWZE9sjUg9xxYL4ZcA+07tjebgpZAPhGBWV+5f2kKY=; b=QHIbzr4jqFc3I/MT3B14687I6YA4wKpCGihOv6Kcy+QBy2l+4JGoznQPGl40XcOJJq ZFyK/cQNQJG+iYuDR4BudwjVEtbvrm5APsUezkwUSgSzfIVxP70CvFeQ1EWOsk+4Ehur FInD0pssEVPVM9QJFLwktFWXa/26LwjnxTYDJCSsBGBuY8yXW2d8+MdIoPMqKFEY0+Lk xTgZS6mPDdiOeBjm/0WMbNi05sETLLYlIonwBcqeWQFpwW85/umDQ++5b9qS6YLNu5tY cm8vHYPzgHgSRQsNFaGUf4lboU0y88sjGmOmSISutrtz74OskvWc+6xNK6LZBnEWr6Ny aRMA== X-Gm-Message-State: AOAM533GFN6Dw2SNroxqFcojWFzquBAPBExkDmCGU4LQBSK4nL2X37NR LMm/IQRnr1RSNcD1fJWkzY5qqzOwEPk= X-Google-Smtp-Source: ABdhPJz4BgNMLAj2Ud5jARxXQcadx0cO0xKzaz6UjrjbxfwUJwlp9ZgzMR8jr2jb1xVjZNR6Koc2hQ== X-Received: by 2002:a17:902:7c92:b029:da:d86b:78be with SMTP id y18-20020a1709027c92b02900dad86b78bemr10576044pll.0.1608421710375; Sat, 19 Dec 2020 15:48:30 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id k15sm12729275pfh.40.2020.12.19.15.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 15:48:29 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/5] powerpc/64s/radix: implement complete radix__ptep_get_and_clear_full Date: Sun, 20 Dec 2020 09:48:12 +1000 Message-Id: <20201219234813.830747-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201219234813.830747-1-npiggin@gmail.com> References: <20201219234813.830747-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently the radix variant is only called when full=true. Allow radix to take over the entire function. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/pgtable.h | 10 +++------- arch/powerpc/include/asm/book3s/64/radix.h | 4 ++++ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 058601efbc8a..410a917c0c60 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -483,14 +483,10 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, unsigned long addr, pte_t *ptep, int full) { - if (full && radix_enabled()) { - /* - * We know that this is a full mm pte clear and - * hence can be sure there is no parallel set_pte. - */ + if (radix_enabled()) return radix__ptep_get_and_clear_full(mm, addr, ptep, full); - } - return ptep_get_and_clear(mm, addr, ptep); + else + return ptep_get_and_clear(mm, addr, ptep); } diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index c7813dc628fc..2491f3befda0 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -185,6 +185,10 @@ static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long old_pte; if (full) { + /* + * We know that this is a full mm pte clear and + * hence can be sure there is no parallel set_pte. + */ old_pte = pte_val(*ptep); *ptep = __pte(0); } else From patchwork Sat Dec 19 23:48:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1418792 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cz2kB4tQKz9sWH for ; Sun, 20 Dec 2020 10:58:26 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KstXafR1; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cz2kB46dTzDqbt for ; Sun, 20 Dec 2020 10:58:26 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KstXafR1; dkim-atps=neutral Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cz2Vp55TPzDqT7 for ; Sun, 20 Dec 2020 10:48:34 +1100 (AEDT) Received: by mail-pj1-x102e.google.com with SMTP id m5so3750859pjv.5 for ; Sat, 19 Dec 2020 15:48:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GyyTQXFMxJsrMrABK3N9juVNdcOA1lOIsRRvLqBrr3g=; b=KstXafR1CkK9Gg/55qJHCilXLC4QXyCj7ZuILru8kQWm36zE87zg+pES2W0nXuwUOi UABt5t6aLezwO2o5Kl2cZErpmGK/Wdb9BQ2OZiHQbj25c7XrEAOGp5bD8OAsnwJn6e7G s1e+1HQJk/Gsedmu/xfNZyBVi6q57xohOmVa0Whb+hf/ytrDsUa83AtkjqLJvNh8gh0Y /fbJ4YCwpadDV0HUFzEH6DkoLCROZCL6tWDa3vtwcNAd/nZZFB1nryeYlHxrDK5QeZ/v PSUk3nFDIDbcOQjD7Eg0bP9T3rb/GiyGQDaVOiIxMGfs6oshCEPujflPzmQ23p7goAl/ Q/OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GyyTQXFMxJsrMrABK3N9juVNdcOA1lOIsRRvLqBrr3g=; b=bmQhlyLQGsOERLj9VTJHd2KDs9QEidPnIOM/3khhUnncKhtXIFjgc4TTQSbXGKtio2 51Kz3vGJwJ4Rk04wgNdiaH0XMyEy3hRSuG0JnAYOTW5o7zj2QBe3TgmUF9wSN6VDw0gP hAqzvHYGrVLD+Tkc2CXElyLp26vlBUKKS9UsRgqauoKvv7rsB5OqkGZo1lyvLbKI2VL8 iud2Jbwluybi9jJELBF2BG9npeKBhbLi6kY8Th10vN3X5X/jkLjUH1bekwCqBRq6MkYZ sMTgW2qU9utXD4TmdzLQLf+tJm4q7t4EwZu/yHQjD6wd5dEiDrKuQwf+dOzts0W67EA2 8A0g== X-Gm-Message-State: AOAM531Y2KB6OZ+fJ3g2od2xVcBQ38BjBKrn4gQg11uyouxdy3Yp6F36 0hNdZUhO/MbfdGXJZmXflc7nzcPM+uM= X-Google-Smtp-Source: ABdhPJwJQJksidauYmUYMkPKfziuiUFmZojxRk+2y3N+1ZETorDs41RlufTDH/5eJg0cysfrJoSQlA== X-Received: by 2002:a17:90b:e0d:: with SMTP id ge13mr10699863pjb.111.1608421712702; Sat, 19 Dec 2020 15:48:32 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-97-30.tpgi.com.au. [193.116.97.30]) by smtp.gmail.com with ESMTPSA id k15sm12729275pfh.40.2020.12.19.15.48.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Dec 2020 15:48:32 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 5/5] powerpc/64s/radix: Use non-atomic PTE updates if the MMU does not modify the PTE Date: Sun, 20 Dec 2020 09:48:13 +1000 Message-Id: <20201219234813.830747-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201219234813.830747-1-npiggin@gmail.com> References: <20201219234813.830747-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" PTE updates performed by Linux are serialized by the page table locks. There is no need to use atomic operations for these modifications if the hardware does not perform updates asynchronously (e.g., via the MMU RC update mechanism). Current POWER CPUs that support radix do not perform such updates in the core MMU, but the nest MMU does so make this conditional on whether the nest mmu is active for the mm. This improves a page fault / mprotect microbenchmark by about 10% on POWER9. There is a question of how compatibility would be work if a future processor implements core MMU PTE udpates. This is being discussed with ISA developers at the moment... Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 54 ++++++--------- arch/powerpc/mm/book3s64/radix_pgtable.c | 79 ++++++++++++++++++---- 2 files changed, 83 insertions(+), 50 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 2491f3befda0..837ed6fb1c7d 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -137,16 +137,17 @@ extern void radix__mark_rodata_ro(void); extern void radix__mark_initmem_nx(void); #endif -extern void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, - pte_t entry, unsigned long address, - int psize); - -extern void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t old_pte, pte_t pte); - -static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, - unsigned long set) +void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, + pte_t entry, unsigned long address, int psize); +void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, + pte_t old_pte, pte_t pte); +unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + unsigned long clr, unsigned long set, int huge); +pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + int full); + +static inline unsigned long __radix_pte_update_atomic(pte_t *ptep, unsigned long clr, + unsigned long set) { __be64 old_be, tmp_be; @@ -163,38 +164,21 @@ static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, return be64_to_cpu(old_be); } -static inline unsigned long radix__pte_update(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, unsigned long clr, - unsigned long set, - int huge) +static inline unsigned long __radix_pte_update_nonatomic(pte_t *ptep, unsigned long clr, + unsigned long set) { - unsigned long old_pte; + unsigned long old_pte = pte_val(*ptep); - old_pte = __radix_pte_update(ptep, clr, set); - if (!huge) - assert_pte_locked(mm, addr); + *ptep = __pte((old_pte & ~clr) | set); return old_pte; } -static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, int full) +static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, + unsigned long set) { - unsigned long old_pte; - - if (full) { - /* - * We know that this is a full mm pte clear and - * hence can be sure there is no parallel set_pte. - */ - old_pte = pte_val(*ptep); - *ptep = __pte(0); - } else - old_pte = radix__pte_update(mm, addr, ptep, ~0ul, 0, 0); - - return __pte(old_pte); + /* Must use atomic updates because accelerators may be attached which can modify the pte */ + return __radix_pte_update_atomic(ptep, clr, set); } static inline int radix__pte_same(pte_t pte_a, pte_t pte_b) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 9495206b9b91..f7ecb90daf87 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1030,6 +1030,47 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool radix__mm_has_atomic_pte_update(struct mm_struct *mm) +{ + if (mm_has_nmmu(mm)) + return true; + return false; +} + +static inline bool radix__mm_has_core_atomic_pte_update(struct mm_struct *mm) +{ + return false; +} + +unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + unsigned long clr, unsigned long set, int huge) +{ + unsigned long old_pte; + + if (!huge) + assert_pte_locked(mm, addr); + + if (radix__mm_has_atomic_pte_update(mm)) + old_pte = __radix_pte_update_atomic(ptep, clr, set); + else + old_pte = __radix_pte_update_nonatomic(ptep, clr, set); + + return old_pte; +} + +pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + int full) +{ + unsigned long old_pte; + + if (full) + old_pte = __radix_pte_update_nonatomic(ptep, ~0ul, 0); + else + old_pte = radix__pte_update(mm, addr, ptep, ~0ul, 0, 0); + + return __pte(old_pte); +} + void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, pte_t entry, unsigned long address, int psize) { @@ -1037,23 +1078,31 @@ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, unsigned long set = pte_val(entry) & (_PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_RW | _PAGE_EXEC); - unsigned long change = pte_val(entry) ^ pte_val(*ptep); - /* - * To avoid NMMU hang while relaxing access, we need mark - * the pte invalid in between. - */ - if ((change & _PAGE_RW) && mm_has_nmmu(mm)) { - unsigned long old_pte, new_pte; + if (mm_has_nmmu(mm)) { + unsigned long change = pte_val(entry) ^ pte_val(*ptep); - old_pte = __radix_pte_update(ptep, _PAGE_PRESENT, _PAGE_INVALID); - /* - * new value of pte - */ - new_pte = old_pte | set; - radix__flush_tlb_page_psize(mm, address, psize); - __radix_pte_update(ptep, _PAGE_INVALID, new_pte); + if ((change & _PAGE_RW)) { + unsigned long old_pte, new_pte; + + /* + * To avoid NMMU hang while relaxing access, we need mark + * the pte invalid in between. + */ + old_pte = __radix_pte_update(ptep, _PAGE_PRESENT, _PAGE_INVALID); + /* + * new value of pte + */ + new_pte = old_pte | set; + radix__flush_tlb_page_psize(mm, address, psize); + __radix_pte_update(ptep, _PAGE_INVALID, new_pte); + } else { + __radix_pte_update(ptep, 0, set); + } } else { - __radix_pte_update(ptep, 0, set); + if (radix__mm_has_core_atomic_pte_update(mm)) + __radix_pte_update_atomic(ptep, 0, set); + else + __radix_pte_update_nonatomic(ptep, 0, set); /* * Book3S does not require a TLB flush when relaxing access * restrictions when the address space is not attached to a