From patchwork Wed May 20 08:30:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294107 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmLj2B26z9sT8 for ; Wed, 20 May 2020 18:36:29 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=Cy0hTnTA; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmLj17jmzDqRy for ; Wed, 20 May 2020 18:36:29 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::541; helo=mail-pg1-x541.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=Cy0hTnTA; dkim-atps=neutral Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmGx4Ht9zDqWv for ; Wed, 20 May 2020 18:33:13 +1000 (AEST) Received: by mail-pg1-x541.google.com with SMTP id p21so1081224pgm.13 for ; Wed, 20 May 2020 01:33:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DvGJDJ10NCwwfIoYiATTvuoTDqLIXlzfFKq1jKX4bOM=; b=Cy0hTnTAPs1JQEg5UTxTAbeYYI7lx7paxKnv1ZXiggmOqQ6yTQYOQBZWyxnUhDuhRS REHWbOlrE0wkyGkx2fk3lPLVGZzCLrrefyOoPWduJ9ZnsasvuF5teNjOxyj6RrghWAaq 7Rq2/mr7Fjh5fwJ2JbC2hqTp/3tpJLLgaRZcY+BXBqOY0HCUBi5I/nZeNbW56OQPkYNQ kKXE31RBVJqmJNzMBmSebse7yCJMoHzBtNthhlMTgTVSqi3wfjeHRJA4WNogi4jRo7YH t5xctEMCLfx41V3pdAx1TljO64UuDJWy5f0AO/gUwpYBLfO6+EhQiHe7niR/xKJbdV4n Vrfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DvGJDJ10NCwwfIoYiATTvuoTDqLIXlzfFKq1jKX4bOM=; b=e1xC41m1kVoyupCYuKSQNTFUgGxBZq9MsNugwmjFnYUaFVS1Sun/Ag7CBDXO8OOZC6 2xZ/o2rIt1rv6oTsnJvUNqIaFrzpn+41mGwTPq+qjS59WAhQFz5K7EK6960Rq+WzdnXk 06Pt/qcF2EeYgmxGCs/ocTBUyz4nt9SAKpXESkjkChoFJOa2/Pppa/fEE164wqx+w5k7 OwtvqnjGXpKCO153CBViJrs/YzuCLJ7+lUTrWZ5Xif8Of1NRmJOykLCpAO+nA8Aeh7w4 jMcn11/XDRO93SyAWVdD5tUJeZpM8Aaz6cDWKv0v9/kAUiNfkdjkUUOnVCEF9sqTNBR2 iFpg== X-Gm-Message-State: AOAM533+1OD8f01/KVqo+2u02NtVVjyyzudRzuoaIP7PjooZ5e8f51Rm j2vVwq+dYTaOX6Iy1RygSq2LDA== X-Google-Smtp-Source: ABdhPJzuMxVxkEIHchBwUmpORtcFOYOxowBV2WxC+d+eFA7YvFuscZXksRaJ690s125sVI8Tl+S5WA== X-Received: by 2002:aa7:8084:: with SMTP id v4mr3201873pff.39.1589963590255; Wed, 20 May 2020 01:33:10 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:08 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 1/6] asm-generic/tlb: Track freeing of page-table directories in struct mmu_gather Date: Wed, 20 May 2020 14:00:20 +0530 Message-Id: <20200520083025.229011-2-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Santosh Sivaraj , Peter Zijlstra , Will Deacon , Greg KH Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Zijlstra commit 22a61c3c4f1379ef8b0ce0d5cb78baf3178950e2 upstream Some architectures require different TLB invalidation instructions depending on whether it is only the last-level of page table being changed, or whether there are also changes to the intermediate (directory) entries higher up the tree. Add a new bit to the flags bitfield in struct mmu_gather so that the architecture code can operate accordingly if it's the intermediate levels being invalidated. Signed-off-by: Peter Zijlstra Signed-off-by: Will Deacon Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: prerequisite for tlbflush backports] --- include/asm-generic/tlb.h | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b3353e21f3b3..97306b32d8d2 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -97,12 +97,22 @@ struct mmu_gather { #endif unsigned long start; unsigned long end; - /* we are in the middle of an operation to clear - * a full mm and can make some optimizations */ - unsigned int fullmm : 1, - /* we have performed an operation which - * requires a complete flush of the tlb */ - need_flush_all : 1; + /* + * we are in the middle of an operation to clear + * a full mm and can make some optimizations + */ + unsigned int fullmm : 1; + + /* + * we have performed an operation which + * requires a complete flush of the tlb + */ + unsigned int need_flush_all : 1; + + /* + * we have removed page directories + */ + unsigned int freed_tables : 1; struct mmu_gather_batch *active; struct mmu_gather_batch local; @@ -137,6 +147,7 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->start = TASK_SIZE; tlb->end = 0; } + tlb->freed_tables = 0; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -278,6 +289,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pte_free_tlb(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) #endif @@ -285,7 +297,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #ifndef pmd_free_tlb #define pmd_free_tlb(tlb, pmdp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) #endif @@ -295,6 +308,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pud_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) #endif @@ -304,7 +318,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #ifndef p4d_free_tlb #define p4d_free_tlb(tlb, pudp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif From patchwork Wed May 20 08:30:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294117 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmNp6Wtxz9sTC for ; Wed, 20 May 2020 18:38:18 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=KoBiDgTY; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmNp4QkDzDqcJ for ; Wed, 20 May 2020 18:38:18 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::642; helo=mail-pl1-x642.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=KoBiDgTY; dkim-atps=neutral Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmH265THzDqLl for ; Wed, 20 May 2020 18:33:18 +1000 (AEST) Received: by mail-pl1-x642.google.com with SMTP id s20so1028609plp.6 for ; Wed, 20 May 2020 01:33:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jK3eak8AGYwL0wJRZqsVcAfH4fevuortaENnjYjREuc=; b=KoBiDgTYuD8IKAqJBdXII+sxD6sA6Xi0IPUUqUub+G+3EfhEhjhsSpHZL/JvhHO0dM 2HmkJd/HUr826uvD1Vp3f/hieDftN1Bj0Gt9Pj5/OPBqO0kaUEBNKBRikRilWTNxINnH CUi6XhdWliKmD07poHONfOzVzfFSNwkijYCUMh/fsdJTPoUc7qVFqqVokN4sD7a+JugC UlXaCfR61zsJ/DNdrlRdJXk4sc3pyv6Yz9/x/u67nggENWPChw+Y0vU2tK46gsVu0K6z rpuemkIFk3dA37qPpjIHwefNbzQGNSQ5cDK8KHkHMgg0lTXWgUsjb8nrh/27VNnm5X0U fuPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jK3eak8AGYwL0wJRZqsVcAfH4fevuortaENnjYjREuc=; b=tb2bupLw+jTwA6iWqqEYoU/Jd06saWaFExJRASvLinIbaWZete2PDNaIQCWwCVbyij 2A1RfJDEAHhHemASstjBf7m43dbsLRsCih4yplahFRBAeLUvAarxGBjzNWgl4Rrq0hT9 aGbnWqYv6kf7tLPjTOjs8pyS5aCIvmORx2tBUNE7xqtz0O7FiY5dRk9z4dTS9qYlsHdO 3MYTpoBjM1EbpOPBxRpzGMhNrGsfZh1GktktYMaRlyxHqcknL+8yrekIqGoSCKqbfC9b el5UoKzsthyoH6Co4oc8FxkpV3aLISDXgM2Re/n/bdTgwLrBaoaoKU8J4R6oln5XmbU2 Wx1g== X-Gm-Message-State: AOAM533kAxjZxwDEh5qZHvA/mR7MXkmzWBL4GbGPi6BYY6Cx/jQgPsc5 sP1qxLpwJEVfYT1J5NrOOb22fA== X-Google-Smtp-Source: ABdhPJzDTT7m+BSVW0Xo6BaI3Xo8msG0e5KvtMajBqg/QMPqaK6NDnNIXye7Ajo6vR+g/ZnmZp9tZw== X-Received: by 2002:a17:90a:2586:: with SMTP id k6mr4041874pje.121.1589963594861; Wed, 20 May 2020 01:33:14 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:14 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 2/6] asm-generic/tlb: Track which levels of the page tables have been cleared Date: Wed, 20 May 2020 14:00:21 +0530 Message-Id: <20200520083025.229011-3-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Santosh Sivaraj , Will Deacon , Greg KH Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Will Deacon commit a6d60245d6d9b1caf66b0d94419988c4836980af upstream It is common for architectures with hugepage support to require only a single TLB invalidation operation per hugepage during unmap(), rather than iterating through the mapping at a PAGE_SIZE increment. Currently, however, the level in the page table where the unmap() operation occurs is not stored in the mmu_gather structure, therefore forcing architectures to issue additional TLB invalidation operations or to give up and over-invalidate by e.g. invalidating the entire TLB. Ideally, we could add an interval rbtree to the mmu_gather structure, which would allow us to associate the correct mapping granule with the various sub-mappings within the range being invalidated. However, this is costly in terms of book-keeping and memory management, so instead we approximate by keeping track of the page table levels that are cleared and provide a means to query the smallest granule required for invalidation. Signed-off-by: Will Deacon Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: prerequisite for upcoming tlbflush backports] --- include/asm-generic/tlb.h | 58 +++++++++++++++++++++++++++++++++------ mm/memory.c | 4 ++- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 97306b32d8d2..f2b9dc9cbaf8 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -114,6 +114,14 @@ struct mmu_gather { */ unsigned int freed_tables : 1; + /* + * at which levels have we cleared entries? + */ + unsigned int cleared_ptes : 1; + unsigned int cleared_pmds : 1; + unsigned int cleared_puds : 1; + unsigned int cleared_p4ds : 1; + struct mmu_gather_batch *active; struct mmu_gather_batch local; struct page *__pages[MMU_GATHER_BUNDLE]; @@ -148,6 +156,10 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->end = 0; } tlb->freed_tables = 0; + tlb->cleared_ptes = 0; + tlb->cleared_pmds = 0; + tlb->cleared_puds = 0; + tlb->cleared_p4ds = 0; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -197,6 +209,25 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, } #endif +static inline unsigned long tlb_get_unmap_shift(struct mmu_gather *tlb) +{ + if (tlb->cleared_ptes) + return PAGE_SHIFT; + if (tlb->cleared_pmds) + return PMD_SHIFT; + if (tlb->cleared_puds) + return PUD_SHIFT; + if (tlb->cleared_p4ds) + return P4D_SHIFT; + + return PAGE_SHIFT; +} + +static inline unsigned long tlb_get_unmap_size(struct mmu_gather *tlb) +{ + return 1UL << tlb_get_unmap_shift(tlb); +} + /* * In the case of tlb vma handling, we can optimise these away in the * case where we're doing a full MM flush. When we're doing a munmap, @@ -230,13 +261,19 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->cleared_ptes = 1; \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ - do { \ - __tlb_adjust_range(tlb, address, huge_page_size(h)); \ - __tlb_remove_tlb_entry(tlb, ptep, address); \ +#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ + do { \ + unsigned long _sz = huge_page_size(h); \ + __tlb_adjust_range(tlb, address, _sz); \ + if (_sz == PMD_SIZE) \ + tlb->cleared_pmds = 1; \ + else if (_sz == PUD_SIZE) \ + tlb->cleared_puds = 1; \ + __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) /** @@ -250,6 +287,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE); \ + tlb->cleared_pmds = 1; \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0) @@ -264,6 +302,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pud_tlb_entry(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE); \ + tlb->cleared_puds = 1; \ __tlb_remove_pud_tlb_entry(tlb, pudp, address); \ } while (0) @@ -289,7 +328,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pte_free_tlb(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_pmds = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) #endif @@ -298,7 +338,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pmd_free_tlb(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_puds = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) #endif @@ -308,7 +349,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pud_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_p4ds = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) #endif @@ -319,7 +361,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define p4d_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif diff --git a/mm/memory.c b/mm/memory.c index bbf0cc4066c8..1832c5ed6ac0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -267,8 +267,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb, { struct mmu_gather_batch *batch, *next; - if (force) + if (force) { + __tlb_reset_range(tlb); __tlb_adjust_range(tlb, start, end - start); + } tlb_flush_mmu(tlb); From patchwork Wed May 20 08:30:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294119 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmQk6Dv4z9sT8 for ; Wed, 20 May 2020 18:39:58 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=lgiiQjKM; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmQk5Hm6zDqXq for ; Wed, 20 May 2020 18:39:58 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::441; helo=mail-pf1-x441.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=lgiiQjKM; dkim-atps=neutral Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmH524ZszDqXC for ; Wed, 20 May 2020 18:33:20 +1000 (AEST) Received: by mail-pf1-x441.google.com with SMTP id 145so1191566pfw.13 for ; Wed, 20 May 2020 01:33:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WEw2wGx1vvAyoxxZ9FPUEcrVmh8BFlemdBg8LI0s3UA=; b=lgiiQjKMJE/dVWoAw0AvHVU2UgJ+yZMqakQ8rhPmPx4MFyq1qv18DIG9HSWsDIv6+8 GaABporOSoZ5/i1lkGBK1+w6JJCt0aDDTtnTS6VA8UYlkePLFXiWYvRjHvw1Ph7g8iV9 bX3xb8MAzha7hzqAl4OSF4n44OeEnZ9J1381EYS6zI3LX0hyzbu/1lCDguTdnWg4Diku RBYxNBnYM2me47Fv0F6DTMRk6Kuv1+hbVrgeuS1fpchuwPEO1EizgD6xmbHkLaWgMuu9 9VEvxTjfvV+kasnPmr6XZVPLFDcMC05MYhjwZnP0Az7auCw2qxHjWahWxNbvVM2OJ/QA 20eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WEw2wGx1vvAyoxxZ9FPUEcrVmh8BFlemdBg8LI0s3UA=; b=iDzVkslHnnTzhKiLf97hJJKHV7nPKQEBQZaVCIg6cOasuMFHKzm1gdPFU8eMT0PswP T/A9KPieJTMCBDOJu0Cy4WUlkDaQlL29rjAsrAja9ua5LsihH50JP/akj+9QtkBQ0Frz Nz3SjJhsRl2hrGnDyNzakigehcUMMJvwAsyFOLMFvQO0LJEedQQzFYuxu0oo2FarN1e8 HaYmqTn7iwTgGb+/XdzZTb4EhLGTXUeW8kW1NXg5VlgMj2KZaUAEs6JErUHFFzeUbARy e5qI87kiHEgr7sOBQqh4FUZbgL+k6RQEcj1N7nlxpuz5s84rSxXzg+sC9Z5s4k/zJOc6 FzNQ== X-Gm-Message-State: AOAM533DuuqSpCaTDZ7MARwhzeIt1Vb/WpORILArJoFyUXx3ONTUpdRz 2BE0DGPixg/slifdRfjAATtoPg== X-Google-Smtp-Source: ABdhPJw5Qq3QN6wymXN/voLJ8FEV+C4e/lXnltEu3Z86k17QxxrTUCdDvqJQKCn91RQY+EEKI7RZoQ== X-Received: by 2002:a63:712:: with SMTP id 18mr3023617pgh.96.1589963598843; Wed, 20 May 2020 01:33:18 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:18 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 3/6] asm-generic/tlb, arch: Invert CONFIG_HAVE_RCU_TABLE_INVALIDATE Date: Wed, 20 May 2020 14:00:22 +0530 Message-Id: <20200520083025.229011-4-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Peter Zijlstra , Santosh Sivaraj , Greg KH Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Zijlstra commit 96bc9567cbe112e9320250f01b9c060c882e8619 upstream Make issuing a TLB invalidate for page-table pages the normal case. The reason is twofold: - too many invalidates is safer than too few, - most architectures use the linux page-tables natively and would thus require this. Make it an opt-out, instead of an opt-in. No change in behavior intended. Signed-off-by: Peter Zijlstra (Intel) Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: prerequisite for upcoming tlbflush backports] --- arch/Kconfig | 2 +- arch/powerpc/Kconfig | 1 + arch/sparc/Kconfig | 1 + arch/x86/Kconfig | 1 - mm/memory.c | 2 +- 5 files changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index a336548487e6..061a12b8140e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -363,7 +363,7 @@ config HAVE_ARCH_JUMP_LABEL config HAVE_RCU_TABLE_FREE bool -config HAVE_RCU_TABLE_INVALIDATE +config HAVE_RCU_TABLE_NO_INVALIDATE bool config ARCH_HAVE_NMI_SAFE_CMPXCHG diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 6f475dc5829b..e09cfb109b8c 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -216,6 +216,7 @@ config PPC select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE if SMP + select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC64 && CPU_LITTLE_ENDIAN select HAVE_SYSCALL_TRACEPOINTS diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index e6f2a38d2e61..d90d632868aa 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -64,6 +64,7 @@ config SPARC64 select HAVE_KRETPROBES select HAVE_KPROBES select HAVE_RCU_TABLE_FREE if SMP + select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_MEMBLOCK_NODE_MAP select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_DYNAMIC_FTRACE diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index af35f5caadbe..181d0d522977 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -181,7 +181,6 @@ config X86 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE if PARAVIRT - select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR diff --git a/mm/memory.c b/mm/memory.c index 1832c5ed6ac0..ba5689610c04 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -327,7 +327,7 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_ */ static inline void tlb_table_invalidate(struct mmu_gather *tlb) { -#ifdef CONFIG_HAVE_RCU_TABLE_INVALIDATE +#ifndef CONFIG_HAVE_RCU_TABLE_NO_INVALIDATE /* * Invalidate page-table caches used by hardware walkers. Then we still * need to RCU-sched wait while freeing the pages because software From patchwork Wed May 20 08:30:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294121 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmTB49jDz9sT8 for ; Wed, 20 May 2020 18:42:06 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=RJoZ4Xg9; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmTB1q2dzDqW9 for ; Wed, 20 May 2020 18:42:06 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=RJoZ4Xg9; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmH90rBVzDqSk for ; Wed, 20 May 2020 18:33:25 +1000 (AEST) Received: by mail-pf1-x443.google.com with SMTP id y18so1201353pfl.9 for ; Wed, 20 May 2020 01:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0WrLFXSRKQDZy0BVrG6fpwzhIiirPXa1ZZ2+F23h/xo=; b=RJoZ4Xg9ChrLjboaatcLi11P3/BfFu9/06Ez954FemKONvyNrhGgP9Qs0LwpVeg4qN aQc42ZbQuHmOm6vfNKmslrKcoNmzYWzJgnf+MSOsyRFXUKBm2ltW+iBqFX+IzgD514ru WkQM+Uka+sWIrwtQFdTJWmE/HOaxEIYUlmkZS5mpah7gmVboRk8vsfQ2q4eXk2/XJRn2 CD29DRxhipU3EnBWYRAXu6BGi6hGiibdb2YMIg2Nnhqru+Ys70dG7VjAnO+HPGmfwmyn ctTyAitOLgjZbdz7rDX7qPfPXe3r7n7U6eRQH/XkQdB9gWS0EXlYHc1NvyAjTqtBkBD/ tT2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0WrLFXSRKQDZy0BVrG6fpwzhIiirPXa1ZZ2+F23h/xo=; b=DcXzh726BAi/PGCqjk9RGL0YAONn7ZuPkLXGgU/2vRvlbij2ZciD4cS7GA8YqhgY/x U9A43hvn3G6XFL6dHxzRzk1EgggRxU8iI1eeMJye1UaLNHjpWN0vOIxPNWTnEJYRzpRE uxoLvn3zilpAQ5Qp9fU6OFuWTE3GdrooEwQalYqb2f1f7AkVrKOGL9ZBFbA01ExhPiHO rCKF+x9PGKh2bq4fkHwABiVk97Wpltqt3losmLCgjSNyiZrZEbzcHtqqamsMN2vIZXUC GLPF8HeBJdajU4jrOs7jsjxZt9CZJtM8h3qcFzWF6tSy/ZgYG+c017qmx/VBMrEE3wA5 qX3g== X-Gm-Message-State: AOAM533u+WJRIwHBmeLjEx9UXrnuwBwVVfmyEMOwC+K/NMb2NR0XJeZO /c0bTIxvUhXVqiUvcaNjRk+bHw== X-Google-Smtp-Source: ABdhPJxZ9CDZanXwI00iKof052hSihO6c0Z7hLztBSZhU7r7znhPh7idsoGP8V6L4cp8OvOwqhZk1w== X-Received: by 2002:a63:4c1d:: with SMTP id z29mr2968658pga.243.1589963602568; Wed, 20 May 2020 01:33:22 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:22 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 4/6] powerpc/mmu_gather: enable RCU_TABLE_FREE even for !SMP case Date: Wed, 20 May 2020 14:00:23 +0530 Message-Id: <20200520083025.229011-5-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Santosh Sivaraj , "Aneesh Kumar K.V" , Greg KH Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: "Aneesh Kumar K.V" commit 12e4d53f3f04e81f9e83d6fc10edc7314ab9f6b9 upstream Patch series "Fixup page directory freeing", v4. This is a repost of patch series from Peter with the arch specific changes except ppc64 dropped. ppc64 changes are added here because we are redoing the patch series on top of ppc64 changes. This makes it easy to backport these changes. Only the first 2 patches need to be backported to stable. The thing is, on anything SMP, freeing page directories should observe the exact same order as normal page freeing: 1) unhook page/directory 2) TLB invalidate 3) free page/directory Without this, any concurrent page-table walk could end up with a Use-after-Free. This is esp. trivial for anything that has software page-table walkers (HAVE_FAST_GUP / software TLB fill) or the hardware caches partial page-walks (ie. caches page directories). Even on UP this might give issues since mmu_gather is preemptible these days. An interrupt or preempted task accessing user pages might stumble into the free page if the hardware caches page directories. This patch series fixes ppc64 and add generic MMU_GATHER changes to support the conversion of other architectures. I haven't added patches w.r.t other architecture because they are yet to be acked. This patch (of 9): A followup patch is going to make sure we correctly invalidate page walk cache before we free page table pages. In order to keep things simple enable RCU_TABLE_FREE even for !SMP so that we don't have to fixup the !SMP case differently in the followup patch !SMP case is right now broken for radix translation w.r.t page walk cache flush. We can get interrupted in between page table free and that would imply we have page walk cache entries pointing to tables which got freed already. Michael said "both our platforms that run on Power9 force SMP on in Kconfig, so the !SMP case is unlikely to be a problem for anyone in practice, unless they've hacked their kernel to build it !SMP." Link: http://lkml.kernel.org/r/20200116064531.483522-2-aneesh.kumar@linux.ibm.com Signed-off-by: Aneesh Kumar K.V Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: backported for 4.19 stable] --- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/book3s/32/pgalloc.h | 8 -------- arch/powerpc/include/asm/book3s/64/pgalloc.h | 2 -- arch/powerpc/include/asm/nohash/32/pgalloc.h | 8 -------- arch/powerpc/mm/pgtable-book3s64.c | 7 ------- 5 files changed, 1 insertion(+), 26 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e09cfb109b8c..1a00ce4b0040 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -215,7 +215,7 @@ config PPC select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI && !HAVE_HARDLOCKUP_DETECTOR_ARCH select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select HAVE_RCU_TABLE_FREE if SMP + select HAVE_RCU_TABLE_FREE select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC64 && CPU_LITTLE_ENDIAN diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index 82e44b1a00ae..79ba3fbb512e 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -110,7 +110,6 @@ static inline void pgtable_free(void *table, unsigned index_size) #define check_pgt_cache() do { } while (0) #define get_hugepd_cache_index(x) (x) -#ifdef CONFIG_SMP static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { @@ -127,13 +126,6 @@ static inline void __tlb_remove_table(void *_table) pgtable_free(table, shift); } -#else -static inline void pgtable_free_tlb(struct mmu_gather *tlb, - void *table, int shift) -{ - pgtable_free(table, shift); -} -#endif static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, unsigned long address) diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h index f9019b579903..1013c0214213 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -47,9 +47,7 @@ extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long); extern void pte_fragment_free(unsigned long *, int); extern void pmd_fragment_free(unsigned long *); extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift); -#ifdef CONFIG_SMP extern void __tlb_remove_table(void *_table); -#endif static inline pgd_t *radix__pgd_alloc(struct mm_struct *mm) { diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h index 8825953c225b..96eed46d5684 100644 --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h @@ -111,7 +111,6 @@ static inline void pgtable_free(void *table, unsigned index_size) #define check_pgt_cache() do { } while (0) #define get_hugepd_cache_index(x) (x) -#ifdef CONFIG_SMP static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { @@ -128,13 +127,6 @@ static inline void __tlb_remove_table(void *_table) pgtable_free(table, shift); } -#else -static inline void pgtable_free_tlb(struct mmu_gather *tlb, - void *table, int shift) -{ - pgtable_free(table, shift); -} -#endif static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, unsigned long address) diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 297db665d953..5b4e9fd8990c 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -432,7 +432,6 @@ static inline void pgtable_free(void *table, int index) } } -#ifdef CONFIG_SMP void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index) { unsigned long pgf = (unsigned long)table; @@ -449,12 +448,6 @@ void __tlb_remove_table(void *_table) return pgtable_free(table, index); } -#else -void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index) -{ - return pgtable_free(table, index); -} -#endif #ifdef CONFIG_PROC_FS atomic_long_t direct_pages_count[MMU_PAGE_COUNT]; From patchwork Wed May 20 08:30:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294128 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmWQ47l5z9sT8 for ; Wed, 20 May 2020 18:44:02 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=sTNISCaj; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmWQ3jMGzDqVc for ; Wed, 20 May 2020 18:44:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::1041; helo=mail-pj1-x1041.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=sTNISCaj; dkim-atps=neutral Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmHG0Pl9zDqRj for ; Wed, 20 May 2020 18:33:29 +1000 (AEST) Received: by mail-pj1-x1041.google.com with SMTP id t40so919132pjb.3 for ; Wed, 20 May 2020 01:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6vUYnVtzIwHWJb6qjVbD2f9l+GduTHLNkDtlUmKXKK8=; b=sTNISCaj7RyO3LfhbyD/z0HifoLVQ+e+tV452kn2DsYKSa6Y3ZhGqxzb5RlQhMfZt2 QoGc2YnDV+OSvUJ47eoHIIBp3QpmVzPRJ4qeuup/wPi57Lo73jnPe59dYZ4M7azzALx6 KbMXPI/mOx0xhY/QdESEVWILztrxDUwCw1tt2m+ArhPH+n0Aw3gt4AApC3zu4C/WmlLH mPaKsI77Zugid0tX0pLtcr3v6M1nCzQTmmjPBn7AMNbF0sFm+jkiw3B24zWEJb8Tw3BI yZd4sBGM+ODBoFjvoSzKPS8AnRFJU5sqxwMZKwyOyTiTZAIr2cej1eFVmTUM4M9vM96q mrKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6vUYnVtzIwHWJb6qjVbD2f9l+GduTHLNkDtlUmKXKK8=; b=ZlLxDwXVJ0SYeL0sJYKvpg8WG1hH10geiW4zxGLzgPcrdYATvHUnrhSwGRQccHc6PA NHL1YTQxu2yZBPIu+70wD38zpSKhRZAx9Ljdq6+TH7ECYozHBi8TMdXlM2AXB57sR3kk w1Xb7kbXki6tEQp62ImX4rQjPeENNtqFGiAJlHmiG/6FHTHk5+B+j1kT+LFYLhhMVirJ YPBEaC8gr6uFgofBPrIFqkhbXeEFeOZU8ASpjI+byrdril1rnIThbPsFyGCigmQuOpQu S3DBciWEr/YmEWOZ7qgRI8Y/8KJxtlAoEUtXh8LklauacU1uHyLHgDSEyaTt3Iff/wBp cJMw== X-Gm-Message-State: AOAM531xznh96XvUieIzNV+ZKSZy8RXYvWJBJl6tCx8TLweFFWPNDzdM E2SyKOYVedldkdo8iuRvLyZlZA== X-Google-Smtp-Source: ABdhPJywcPpG6uSuWHx87lXNUW8fc0ZLaat13S2lX9XW6p9+3xmG3JjXkP5jmlQxVoYQcCmT24EZgA== X-Received: by 2002:a17:902:326:: with SMTP id 35mr3240466pld.188.1589963607187; Wed, 20 May 2020 01:33:27 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:26 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 5/6] mm/mmu_gather: invalidate TLB correctly on batch allocation failure and flush Date: Wed, 20 May 2020 14:00:24 +0530 Message-Id: <20200520083025.229011-6-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Santosh Sivaraj , Peter Zijlstra , Greg KH , "Aneesh Kumar K . V" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Zijlstra commit 0ed1325967ab5f7a4549a2641c6ebe115f76e228 upstream Architectures for which we have hardware walkers of Linux page table should flush TLB on mmu gather batch allocation failures and batch flush. Some architectures like POWER supports multiple translation modes (hash and radix) and in the case of POWER only radix translation mode needs the above TLBI. This is because for hash translation mode kernel wants to avoid this extra flush since there are no hardware walkers of linux page table. With radix translation, the hardware also walks linux page table and with that, kernel needs to make sure to TLB invalidate page walk cache before page table pages are freed. More details in commit d86564a2f085 ("mm/tlb, x86/mm: Support invalidating TLB caches for RCU_TABLE_FREE") The changes to sparc are to make sure we keep the old behavior since we are now removing HAVE_RCU_TABLE_NO_INVALIDATE. The default value for tlb_needs_table_invalidate is to always force an invalidate and sparc can avoid the table invalidate. Hence we define tlb_needs_table_invalidate to false for sparc architecture. Link: http://lkml.kernel.org/r/20200116064531.483522-3-aneesh.kumar@linux.ibm.com Fixes: a46cc7a90fd8 ("powerpc/mm/radix: Improve TLB/PWC flushes") Signed-off-by: Peter Zijlstra (Intel) Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: backported to 4.19 stable] --- arch/Kconfig | 3 --- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/tlb.h | 11 +++++++++++ arch/sparc/Kconfig | 1 - arch/sparc/include/asm/tlb_64.h | 9 +++++++++ include/asm-generic/tlb.h | 15 +++++++++++++++ mm/memory.c | 16 ++++++++-------- 7 files changed, 43 insertions(+), 13 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 061a12b8140e..3abbdb0cea44 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -363,9 +363,6 @@ config HAVE_ARCH_JUMP_LABEL config HAVE_RCU_TABLE_FREE bool -config HAVE_RCU_TABLE_NO_INVALIDATE - bool - config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1a00ce4b0040..e5bc0cfea2b1 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -216,7 +216,6 @@ config PPC select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE - select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC64 && CPU_LITTLE_ENDIAN select HAVE_SYSCALL_TRACEPOINTS diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index f0e571b2dc7c..63418275f402 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -30,6 +30,17 @@ #define tlb_remove_check_page_size_change tlb_remove_check_page_size_change extern void tlb_flush(struct mmu_gather *tlb); +/* + * book3s: + * Hash does not use the linux page-tables, so we can avoid + * the TLB invalidate for page-table freeing, Radix otoh does use the + * page-tables and needs the TLBI. + * + * nohash: + * We still do TLB invalidate in the __pte_free_tlb routine before we + * add the page table pages to mmu gather table batch. + */ +#define tlb_needs_table_invalidate() radix_enabled() /* Get the generic bits... */ #include diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index d90d632868aa..e6f2a38d2e61 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -64,7 +64,6 @@ config SPARC64 select HAVE_KRETPROBES select HAVE_KPROBES select HAVE_RCU_TABLE_FREE if SMP - select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_MEMBLOCK_NODE_MAP select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_DYNAMIC_FTRACE diff --git a/arch/sparc/include/asm/tlb_64.h b/arch/sparc/include/asm/tlb_64.h index a2f3fa61ee36..8cb8f3833239 100644 --- a/arch/sparc/include/asm/tlb_64.h +++ b/arch/sparc/include/asm/tlb_64.h @@ -28,6 +28,15 @@ void flush_tlb_pending(void); #define __tlb_remove_tlb_entry(tlb, ptep, address) do { } while (0) #define tlb_flush(tlb) flush_tlb_pending() +/* + * SPARC64's hardware TLB fill does not use the Linux page-tables + * and therefore we don't need a TLBI when freeing page-table pages. + */ + +#ifdef CONFIG_HAVE_RCU_TABLE_FREE +#define tlb_needs_table_invalidate() (false) +#endif + #include #endif /* _SPARC64_TLB_H */ diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index f2b9dc9cbaf8..19934cdd143e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -61,8 +61,23 @@ struct mmu_table_batch { extern void tlb_table_flush(struct mmu_gather *tlb); extern void tlb_remove_table(struct mmu_gather *tlb, void *table); +/* + * This allows an architecture that does not use the linux page-tables for + * hardware to skip the TLBI when freeing page tables. + */ +#ifndef tlb_needs_table_invalidate +#define tlb_needs_table_invalidate() (true) #endif +#else + +#ifdef tlb_needs_table_invalidate +#error tlb_needs_table_invalidate() requires HAVE_RCU_TABLE_FREE +#endif + +#endif /* CONFIG_HAVE_RCU_TABLE_FREE */ + + /* * If we can't allocate a page to make a big batch of page pointers * to work on, then just handle a few from the on-stack structure. diff --git a/mm/memory.c b/mm/memory.c index ba5689610c04..7daa7ae1b046 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -327,14 +327,14 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_ */ static inline void tlb_table_invalidate(struct mmu_gather *tlb) { -#ifndef CONFIG_HAVE_RCU_TABLE_NO_INVALIDATE - /* - * Invalidate page-table caches used by hardware walkers. Then we still - * need to RCU-sched wait while freeing the pages because software - * walkers can still be in-flight. - */ - tlb_flush_mmu_tlbonly(tlb); -#endif + if (tlb_needs_table_invalidate()) { + /* + * Invalidate page-table caches used by hardware walkers. Then + * we still need to RCU-sched wait while freeing the pages + * because software walkers can still be in-flight. + */ + tlb_flush_mmu_tlbonly(tlb); + } } static void tlb_remove_table_smp_sync(void *arg) From patchwork Wed May 20 08:30:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Sivaraj X-Patchwork-Id: 1294133 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49RmYl294zz9sTT for ; Wed, 20 May 2020 18:46:03 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=X9lGxgxE; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49RmYl1MgxzDqGN for ; Wed, 20 May 2020 18:46:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=fossix.org (client-ip=2607:f8b0:4864:20::1044; helo=mail-pj1-x1044.google.com; envelope-from=santosh@fossix.org; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=fossix.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fossix-org.20150623.gappssmtp.com header.i=@fossix-org.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=X9lGxgxE; dkim-atps=neutral Received: from mail-pj1-x1044.google.com (mail-pj1-x1044.google.com [IPv6:2607:f8b0:4864:20::1044]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49RmHK70ttzDqVq for ; Wed, 20 May 2020 18:33:33 +1000 (AEST) Received: by mail-pj1-x1044.google.com with SMTP id k7so901664pjs.5 for ; Wed, 20 May 2020 01:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fossix-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NlxdBI8wiLZenSs6j2nq89RHtqfPDggo6DaziIKFMVc=; b=X9lGxgxEPnFHSzzaOdMDLP7ATZXnbqTUa9hrsb9Z/3TMQ80e/6CC8Cs8MftCgeSYbW tWcwUGqj6bE/2RDfGVmERWU7RPf0O6SdL458fXe39rOozu8D6CrQV2zNWO6a5Az7EUuA xQkKuJkw0+s/Ti2gwcMztT4AZXIvphgs81Qiqd0Gl+HVXPpWunOQAHYV4TBJTQFguzcb F88Z2xxLrTpo2V1WlcK3bJKCow6rb8gUzz3+WbB5WG+tAC3GhxCGj+Z+T1FGeENlVvu+ xPjnBtW0wsp50C6tiPwC2CPQ8bKKmknPnKSgWiaOSFS0qWv9l19AinL1O8Ne7B17uMzi WnUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NlxdBI8wiLZenSs6j2nq89RHtqfPDggo6DaziIKFMVc=; b=qMqLw8bHCTtYPD25GQqjabhdrCMaNEEdHR+JBfbm3uUy0FBsazQEVMtfsFneP+bh7q Yo1vXUdpduQieZHoWzXDaC615PPofNRVnrUdj3Gu+hmHA+yXD4G2MiPv+iCoYqG1gdWZ wCQ1Y2++O/o0v5QXGWUp4xQ5JuavNH6gUAgoVfKDelhZP10qqRDVqiaHp97WRD+5jM4C KfBzu8VoCse+ZMPeUCi/4hzUGuJIvVP+hxNB8eYvhpFzAy91ICzYcCOa9W4joJBG80oH H5W9kaJG7H0KRe28Xl50Vuz2o1WfN5ckzn+BM69h18HMFG+NDxPlCOInCPz7BKh14qbl 5n/w== X-Gm-Message-State: AOAM530wmnhTKUBIZeHp0YFSmD5kdEEtRGwWHNVDX3ok4xCk0L8AIpEd 3diHbPBbXN9MNTfmNzQqfHo16A== X-Google-Smtp-Source: ABdhPJwEQgWQuk6x5EevptElzwnttM1fzDUykzVF8QoxGjOlm29ZKZuBRwmc+SfC8yz5Pv/hFbgtbg== X-Received: by 2002:a17:90a:1303:: with SMTP id h3mr3819470pja.44.1589963611955; Wed, 20 May 2020 01:33:31 -0700 (PDT) Received: from santosiv.in.ibm.com.com ([223.181.246.139]) by smtp.gmail.com with ESMTPSA id 2sm1553980pfz.39.2020.05.20.01.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 01:33:31 -0700 (PDT) From: Santosh Sivaraj To: , linuxppc-dev Subject: [PATCH v4 6/6] asm-generic/tlb: avoid potential double flush Date: Wed, 20 May 2020 14:00:25 +0530 Message-Id: <20200520083025.229011-7-santosh@fossix.org> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200520083025.229011-1-santosh@fossix.org> References: <20200520083025.229011-1-santosh@fossix.org> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Santosh Sivaraj , Peter Zijlstra , Greg KH , "Aneesh Kumar K . V" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Zijlstra commit 0758cd8304942292e95a0f750c374533db378b32 upstream Aneesh reported that: tlb_flush_mmu() tlb_flush_mmu_tlbonly() tlb_flush() <-- #1 tlb_flush_mmu_free() tlb_table_flush() tlb_table_invalidate() tlb_flush_mmu_tlbonly() tlb_flush() <-- #2 does two TLBIs when tlb->fullmm, because __tlb_reset_range() will not clear tlb->end in that case. Observe that any caller to __tlb_adjust_range() also sets at least one of the tlb->freed_tables || tlb->cleared_p* bits, and those are unconditionally cleared by __tlb_reset_range(). Change the condition for actually issuing TLBI to having one of those bits set, as opposed to having tlb->end != 0. Link: http://lkml.kernel.org/r/20200116064531.483522-4-aneesh.kumar@linux.ibm.com Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Aneesh Kumar K.V Reported-by: "Aneesh Kumar K.V" Cc: # 4.19 Signed-off-by: Santosh Sivaraj [santosh: backported to 4.19 stable] --- include/asm-generic/tlb.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 19934cdd143e..427a70c56ddd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -179,7 +179,12 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { - if (!tlb->end) + /* + * Anything calling __tlb_adjust_range() also sets at least one of + * these bits. + */ + if (!(tlb->freed_tables || tlb->cleared_ptes || tlb->cleared_pmds || + tlb->cleared_puds || tlb->cleared_p4ds)) return; tlb_flush(tlb);