From patchwork Tue Oct 16 13:13:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 984755 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42ZG8k3D83z9s9G for ; Wed, 17 Oct 2018 00:18:30 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Nm0YzlJB"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42ZG8k1X1GzF3WF for ; Wed, 17 Oct 2018 00:18:30 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Nm0YzlJB"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::543; helo=mail-pg1-x543.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Nm0YzlJB"; dkim-atps=neutral Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 42ZG3Z3hstzDqfW for ; Wed, 17 Oct 2018 00:14:02 +1100 (AEDT) Received: by mail-pg1-x543.google.com with SMTP id 23-v6so10839215pgc.8 for ; Tue, 16 Oct 2018 06:14:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=og3nps762l1wehgPrVvLZ4tOt6tgHDhsP/lk8/fPWl8=; b=Nm0YzlJBr/tFpHAsamNBQRtcflUZ1a3Fhs7jRJoXnpFPig4gbEgKXSD8GJg5pTuCzZ flZP8osFYXA9rcYPLY1PyxKzJemBVuQjRz30z6FwFB4TaoUDz2o5xuG/CMsPKZ2UFyqs RZ+GE2BV0OhCu0HksSYWPZDL+bdF8YLGRJ14FGakPYH/sq2hk0gVDXssBQ1yLJUjEr7Q Xs9PwkS8+UAIHo28AreMJSU7LdlhogWZ2UT0D0x7OwKMjaCFxzh/mt7sv551VBgwuR39 L/QfwPOstPLfmSkiEpLRxyhfsrZnq5zTvdQQsfeW8zv5Cuo1DGQSTzvpgO2MSayvrNPU O72Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=og3nps762l1wehgPrVvLZ4tOt6tgHDhsP/lk8/fPWl8=; b=slgCsPaxKcfLQaPQF+EkcMXwUwi1bartMkb4B09xqi8tbNt39kCJudEblEFJbUZbEh xqoBzupJ7IlAfGp87/2sUd4dHlh1FbuubkA7E4IQ21JU/NqHlM/9nzJhCkFlVv+Q83wq gEFFAqYSXc3F+c1qgjq3RL7w4RCt+Lz2gss177G8NUkETk+Kf7/CoENXavyaDYTJM49j gBICLoW/vBDyRXC2vMfcz2ujCWmgcl7erjSAqm4Ue5Tc5EDg2yz42RSNCdt8tybyGpvv zmycvH2OuSk2+bKgThZR0CJVyq7mnHCc3cCLdJPOsLTUoNj3qnEIjNMibQpCjHxvX9Es o67w== X-Gm-Message-State: ABuFfohQeKz0grxN1yZIUJEg2UCqoWSqKaGL4O4+qyTn+Rm60ahgOZjK 9hY/+rjpP9mQTkDv+mu3TKW83r9e X-Google-Smtp-Source: ACcGV633qW2OBNPwm3Of34MQ0PiCfw8SZWKbv1d8VPnZh7InlVEXQra/DC+Bn4ZaPxLg1AladyddzA== X-Received: by 2002:a63:4f20:: with SMTP id d32-v6mr18957422pgb.231.1539695640808; Tue, 16 Oct 2018 06:14:00 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.13.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:00 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v2 1/5] nios2: update_mmu_cache clear the old entry from the TLB Date: Tue, 16 Oct 2018 23:13:39 +1000 Message-Id: <20181016131343.20556-2-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch , Linus Torvalds , Linux Kernel Mailing List , Nicholas Piggin , linux-mm , Ley Foon Tan , ppc-dev Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Fault paths like do_read_fault will install a Linux pte with the young bit clear. The CPU will fault again because the TLB has not been updated, this time a valid pte exists so handle_pte_fault will just set the young bit with ptep_set_access_flags, which flushes the TLB. The TLB is flushed so the next attempt will go to the fast TLB handler which loads the TLB with the new Linux pte. The access then proceeds. This design is fragile to depend on the young bit being clear after the initial Linux fault. A proposed core mm change to immediately set the young bit upon such a fault, results in ptep_set_access_flags not flushing the TLB because it finds no change to the pte. The spurious fault fix path only flushes the TLB if the access was a store. If it was a load, then this results in an infinite loop of page faults. This change adds a TLB flush in update_mmu_cache, which removes that TLB entry upon the first fault. This will cause the fast TLB handler to load the new pte and avoid the Linux page fault entirely. Reviewed-by: Ley Foon Tan Signed-off-by: Nicholas Piggin --- arch/nios2/mm/cacheflush.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/nios2/mm/cacheflush.c b/arch/nios2/mm/cacheflush.c index 506f6e1c86d5..d58e7e80dc0d 100644 --- a/arch/nios2/mm/cacheflush.c +++ b/arch/nios2/mm/cacheflush.c @@ -204,6 +204,8 @@ void update_mmu_cache(struct vm_area_struct *vma, struct page *page; struct address_space *mapping; + flush_tlb_page(vma, address); + if (!pfn_valid(pfn)) return; From patchwork Tue Oct 16 13:13:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 984756 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42ZGCS335Lz9s9J for ; Wed, 17 Oct 2018 00:20:52 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="A/SSJVZx"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42ZGCS1PqLzF3WD for ; Wed, 17 Oct 2018 00:20:52 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="A/SSJVZx"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::643; helo=mail-pl1-x643.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="A/SSJVZx"; dkim-atps=neutral Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 42ZG3g5VqVzF3W6 for ; Wed, 17 Oct 2018 00:14:07 +1100 (AEDT) Received: by mail-pl1-x643.google.com with SMTP id 1-v6so11010934plv.7 for ; Tue, 16 Oct 2018 06:14:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vbOXilnyHM7Y3HZf42CmhT0al3ZotZPzM5/9KSS+Wl0=; b=A/SSJVZxVxROnzdtTJGd3Rl0mtQnn0aeP262cO0PG37AB5aJFbY1kcl0Q9RVU+AsSN hnGdUUZtKTqK8l4cdbSjbZ9BUzn/hPxOFdEPN4vlGKIB25R+NFVzhjzkDFWSAaFBLaSt xrSM6/l6UKnNfWzsSJg7ulrXaZrJXIUqdvEaGukSxIoLvU3Es6tmC1I+PECGOfrldZKg NLn/cJoSesrvsErHZ4uFkC2dmA03+a8M23DZATqifHAalHpK02AeaGxYq3X86uOI4su5 c7wNfOzT0sZv7rpzm5n9vvCGcZMKbl4VkQEOnVbCI02AICWJXxrRFSVG1IEGRym457Rs WgRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vbOXilnyHM7Y3HZf42CmhT0al3ZotZPzM5/9KSS+Wl0=; b=SJY2JxLR10qfo334Rl05KiOzS4DPrYWaeES1De7azdaptlQ+jqAAt8fqVBowcRwCxJ j9/a5IVND4jPJU5OJaa/Httgsy+nCtOuO6P/2XAYuiYYV/suQ3c/H9x7WZZJ4K8AXEDG W8b6h76Wt3Eg9m8A0G7e4aBd2NfeNBAR3szJIJfdRyzT+8eH97RviBSytBJyzuSWuJAQ OOkXVQN7lNsgMv3NETUYyM6fo/bQpQ8CbrM/3PpWqtTdB3tMdYJlkW3zcBU77lwKNjJJ Y3jPZukVIaqUWVGd9d6x6aukE44GMJkXmU2pZlPSLc8oieOT5mK+YG9TenE4SJDfTYUy PEHw== X-Gm-Message-State: ABuFfojfKhb5q8vtjnjWFRpELzrpKl4pLDwbHNGI1vUTYzcT349WxKsN Bvwl4p11HLr6bUa3aGbbu3w= X-Google-Smtp-Source: ACcGV625ozmXlwQNgWm1xWhrstSjyDp7scuMStIV7BLFGcBHbvseOYBjb3efegUxYSoldpVrEtpcww== X-Received: by 2002:a17:902:9b84:: with SMTP id y4-v6mr21817926plp.332.1539695646081; Tue, 16 Oct 2018 06:14:06 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:05 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v2 2/5] mm/cow: don't bother write protecting already write-protected huge pages Date: Tue, 16 Oct 2018 23:13:40 +1000 Message-Id: <20181016131343.20556-3-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch , Linus Torvalds , Linux Kernel Mailing List , Nicholas Piggin , linux-mm , Ley Foon Tan , ppc-dev Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This is the HugePage / THP equivalent for 1b2de5d039c8 ("mm/cow: don't bother write protecting already write-protected pages"). Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 14 ++++++++++---- mm/hugetlb.c | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 58269f8ba7c4..0fb0e3025f98 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -973,8 +973,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + if (pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1064,8 +1067,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, /* No huge zero pud yet */ } - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5c390f5a5207..54a4dcb6ee21 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3287,7 +3287,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_swap_pte_at(dst, addr, dst_pte, entry, sz); } else { - if (cow) { + if (cow && huge_pte_write(entry)) { /* * No need to notify as we are downgrading page * table protection not changing it to point From patchwork Tue Oct 16 13:13:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 984758 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42ZGGX2Bdqz9s55 for ; Wed, 17 Oct 2018 00:23:32 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WYUM7y/k"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42ZGGX0WDfzF3VD for ; Wed, 17 Oct 2018 00:23:32 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WYUM7y/k"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::642; helo=mail-pl1-x642.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WYUM7y/k"; dkim-atps=neutral Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 42ZG3n3GN1zF3WQ for ; Wed, 17 Oct 2018 00:14:13 +1100 (AEDT) Received: by mail-pl1-x642.google.com with SMTP id 1-v6so11011040plv.7 for ; Tue, 16 Oct 2018 06:14:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=b8n7ekTz4I5WPtItNykHetIxRP9150gP8kPb0BQtO90=; b=WYUM7y/kL8/+heRnnym3+/pjQXjKvyJ+4vExqnWSUKAt5OCcPWLDKAyjYXXX+3tTNR T6b0g4t9dmXMbmx3FIvcAqX7O329x5zhDruzeal6OmBiza0FOgjbU1RMeljPcpkbKp0/ TFo8sX1VGUzkmoHiR1PFKoUwS+KpHNocpFVRrcfhk+rx6fduiPzvcVAXqnIL0cBl6THr hbVSz2bbtVDrUnEsJdSAy25sFuItTMKF0xRIJvWJnmBDgjGpCixgOlx4aG9rDmg6RsAx /uuAKEyJO8ldmQhR9iNJHXbr4spuqd6ROpfB/V469ofmS6CbDMXdpP6hxlUKVxCe5x+R 3iTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=b8n7ekTz4I5WPtItNykHetIxRP9150gP8kPb0BQtO90=; b=EYTRWc1s0SKCNGSznREEYD6LHrtn7g/7iIBt2Z8qLkNUQLe9JcFYo6PSO7LbHegvgT 4qJQW6CvxNYlwZazY2jfKosu5zE6PO/yAb/NuGENnufOecj4eaY4u64RlTuDLqed9Q1C YS6lm0BwDVjPZhJe35hXgz68+jojZvGFwd2Dqva0tuDIBhn4PwcF663W4Z7XxU6xW3MP f8QhEKiZgunwmkQePBqDICUFMPI4YlIYwJbq6DFgcN+RdVbBOXIjATjhkaf0/o0bhGnd hzYGAjW6SPzJOMNWLkF3H0rs8mxXQ5kiHT1HGMfn3bpCclseU4lXgISo0m1xwBASaSjY ia6Q== X-Gm-Message-State: ABuFfogA8qCeVOuFSCihFR0pxJeLUB3fLLis4fSeF0/NQKaTcPiJetD8 190HyvG6yyB2scXJJLa5Yw4= X-Google-Smtp-Source: ACcGV63ooqapdfib9rvEhAj6eX85EavZn0yGkIHSWlSmi1u/6UNdDgdZJZkJZsL5NL3zvWoE/2y3vA== X-Received: by 2002:a17:902:4681:: with SMTP id p1-v6mr13592768pld.97.1539695651671; Tue, 16 Oct 2018 06:14:11 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:11 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v2 3/5] mm/cow: optimise pte accessed bit handling in fork Date: Tue, 16 Oct 2018 23:13:41 +1000 Message-Id: <20181016131343.20556-4-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch , Linus Torvalds , Linux Kernel Mailing List , Nicholas Piggin , linux-mm , Ley Foon Tan , ppc-dev Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits, so this patch avoids clearing the accessed bit in the child. Any accessed bit is treated similarly to many, with the difference today with > 1 referenced bit causing the page to be activated, while 1 bit causes it to be kept. This patch causes pages shared by fork(2) to be more readily activated, but this heuristic is very fuzzy anyway -- a page can be accessed by multiple threads via a single pte and be just as important as one that is accessed via multiple ptes, for example. In the end I don't believe fork(2) is a significant driver of page reclaim behaviour that this should matter too much. This and the following change eliminate a major source of faults that powerpc/radix requires to set dirty/accessed bits in ptes, speeding up a fork/exit microbenchmark by about 5% on POWER9 (16600 -> 17500 fork/execs per second). Skylake appears to have a micro-fault overhead too -- a test which allocates 4GB anonymous memory, reads each page, then forks, and times the child reading a byte from each page. The first pass over the pages takes about 1000 cycles per page, the second pass takes about 27 cycles (TLB miss). With no additional minor faults measured due to either child pass, and the page array well exceeding TLB capacity, the large cost must be caused by micro faults caused by setting accessed bit. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 2 -- mm/memory.c | 1 - mm/vmscan.c | 8 ++++++++ 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0fb0e3025f98..1f43265204d4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -977,7 +977,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); pmd = pmd_wrprotect(pmd); } - pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1071,7 +1070,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } - pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..0387ee1e3582 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1033,7 +1033,6 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, */ if (vm_flags & VM_SHARED) pte = pte_mkclean(pte); - pte = pte_mkold(pte); page = vm_normal_page(vma, addr, pte); if (page) { diff --git a/mm/vmscan.c b/mm/vmscan.c index c5ef7240cbcb..e72d5b3336a0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1031,6 +1031,14 @@ static enum page_references page_check_references(struct page *page, * to look twice if a mapped file page is used more * than once. * + * fork() will set referenced bits in child ptes despite + * not having been accessed, to avoid micro-faults of + * setting accessed bits. This heuristic is not perfectly + * accurate in other ways -- multiple map/unmap in the + * same time window would be treated as multiple references + * despite same number of actual memory accesses made by + * the program. + * * Mark it and spare it for another trip around the * inactive list. Another page table reference will * lead to its activation. From patchwork Tue Oct 16 13:13:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 984761 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42ZGPB2L4Sz9s55 for ; Wed, 17 Oct 2018 00:29:18 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="e11cs6jE"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42ZGPB0ffCzF3XJ for ; Wed, 17 Oct 2018 00:29:18 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="e11cs6jE"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::644; helo=mail-pl1-x644.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="e11cs6jE"; dkim-atps=neutral Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 42ZG3t147qzF3WG for ; Wed, 17 Oct 2018 00:14:18 +1100 (AEDT) Received: by mail-pl1-x644.google.com with SMTP id s4-v6so11034824plp.0 for ; Tue, 16 Oct 2018 06:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Uwz2MhBiZj2/RLBte8wwxNWfHSbBq1uBQUZRjpspgrw=; b=e11cs6jEoeMmwoBTYdNdNJksH7qI2wwNhzL8dHOfMYBv61ZXmE7ZyV/m0HuwprSE8P LFyAsDmOM0lbAcgAwqRMqM/kSCDNmRysuTjQBcOTsE+FYIuV6YatjeYM9khiyj/R3/AY QCOLgFdpTNNuFTtQ/BUGeOh8mtqwWkMaj1cDEQfLz/cYm4GL6lZaJ74Hjstyi58GeYBt wOKkZeW4oBUFptEPdrDwkxiVepZCC+JvI0CR0+o9yZckDgcyrP91npD4G3d5GNYQsbQN Zhnb8I+DxMlfZAf2J/KOOMUmZ9HMuxpolqtPk2C9td0SKgrYsShSDKNx8nkIWCHWN94Z 0Q0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Uwz2MhBiZj2/RLBte8wwxNWfHSbBq1uBQUZRjpspgrw=; b=l4b8mRTOHcZKvN2Di7kiOSG9uL7JbciHbpkuLGBOLMAhssP7Tz6mMq/wn0RakqI3dB eLfatmYKXn2K5C2J4dAHt3xlokJ2bu87nXFcJ2+WlcqFhFOHMupOEvANCdNcuRoGbgea R8Cu5DER7TrXIOs7UNcGbcXrrwfOqW3P0KEKkV292NO4oJaC0fjMgOTEvR4aGdmLISnM 6AJUrVZxyonmJ1MeRX3IxxMlF+b1drPGQHwySAdVlv81Dcx02VndWsayy/r686TGnQEr ldVQJANE2yuJQrv6Wz7XaET3YzUsqtVz+i/oxJUEHIj6zt99Ux3sbgdVRmWpsCf6tAHX C0/g== X-Gm-Message-State: ABuFfogUr+V2tJOGMyWSzaIQK3MqYilYPsLXZXlrcMBHX8vsiT3zqA/M 4FHT73vYbdse1zmUBgczFJCEVXRE X-Google-Smtp-Source: ACcGV62O4GEomn4Rji7i3zI4ZPz6+o/83UivhZoWG06/NphsLTdj3gWE9y8+Ptx4lChGWFwrT5esfA== X-Received: by 2002:a17:902:34a:: with SMTP id 68-v6mr21440986pld.39.1539695656502; Tue, 16 Oct 2018 06:14:16 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:16 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v2 4/5] mm/cow: optimise pte dirty bit handling in fork Date: Tue, 16 Oct 2018 23:13:42 +1000 Message-Id: <20181016131343.20556-5-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch , Linus Torvalds , Linux Kernel Mailing List , Nicholas Piggin , linux-mm , Ley Foon Tan , ppc-dev Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits, so this patch avoids clearing the dirty bit in the child. Dirty bits are all tested and cleared together, and any dirty bit is the same as many dirty bits, so from a correctness and writeback bandwidth point-of-view it does not matter if the child gets a dirty bit. Dirty ptes are more costly to unmap because they require flushing under the page table lock, but it is pretty rare to have a shared dirty mapping that is copied on fork, so just simplify the code and avoid this dirty clearing logic. Signed-off-by: Nicholas Piggin --- mm/memory.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0387ee1e3582..9e314339a0bd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1028,11 +1028,12 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * If it's a shared mapping, mark it clean in - * the child + * Child inherits dirty and young bits from parent. There is no + * point clearing them because any cleaning or aging has to walk + * all ptes anyway, and it will notice the bits set in the parent. + * Leaving them set avoids stalls and even page faults on CPUs that + * handle these bits in software. */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); page = vm_normal_page(vma, addr, pte); if (page) { From patchwork Tue Oct 16 13:13:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 984762 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42ZGRz2yG9z9s55 for ; Wed, 17 Oct 2018 00:31:43 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QRnnDkdr"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42ZGRz138rzF3X5 for ; Wed, 17 Oct 2018 00:31:43 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QRnnDkdr"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QRnnDkdr"; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 42ZG3y5ZhTzF3W9 for ; Wed, 17 Oct 2018 00:14:22 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id j23-v6so11445238pfi.4 for ; Tue, 16 Oct 2018 06:14:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Wozx3I6t1Z6fZ+0xttJ6rllGkcnzPt2KAsF0ALUBnnI=; b=QRnnDkdrX4fTuAStV0TssQmvg1tQGZDM8vwrwyePkqEPHeenfDLv/cItj0JX95n69G z/jVDqEaBiBrP+MCux6xsHe8ceWLckKJyYmwS2L38fAkinSn2GD/WWuGpyEw95WtG0ub V/Uhabhj24big4D6uo/4ZkeWohj7BQHqqv7c09ZBuAuEm34YxZgn5plnw1psItwCJGoL zzuBpbBoeYk0qesuFnzoPrdViEUg4i9rMh1GV/bP3wUAan7fBuk9DdbLzpFF2Vy1cosG 0uMbbR8trBZmW0mfb/w+Wc24TkIyRYwl39mgvpMJYP4uPJnprMkcGzs4cPlwH4A4lbMW Hs7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Wozx3I6t1Z6fZ+0xttJ6rllGkcnzPt2KAsF0ALUBnnI=; b=rKf/+mXu59+10vBtz6QYmtl0XQS1IIqDYatXlp4qlItBBj6Z7XgDVFNh/pDBncYXV5 ICT9ncZNKrUJY8IuHYpqo7gt7Kpsc7LqTuVT/NmTd4P95Sjzqgx35g0G1LMNgbX7EwTT lEckI5h51LKBhkkIaNt/0KFs5PbCrp2fahMsj/Kh2b/IIeO7D/mltlapNklbWMvpPSjp W1paLKu97iYt69vNrkfMJ76KIyrAMJ864pbrMTAEASimzRUw37E9xmoX5zp2dsvj8dn9 NTwkMu0sJfIsiDdE+bvgX66jaZRAPTW9SKTQMTFQm8XPAU06lQPcWSNQdC1Uh9NvONgD MdEw== X-Gm-Message-State: ABuFfojjFvIk8kcKFux2RVzW/CXpTtR0NKzDVFKg9MfoR0th1mt7lrNq c71rDUbhprK9BaSEQC8qMIs= X-Google-Smtp-Source: ACcGV62PowkKGuB1+DTkoNK49/BnGHkMHGcEPghSygGdFtY1eXOZ5nEe+W/s+bLONLOvO07moh8WRA== X-Received: by 2002:a63:db55:: with SMTP id x21-v6mr19632152pgi.365.1539695661103; Tue, 16 Oct 2018 06:14:21 -0700 (PDT) Received: from roar.local0.net ([60.240.252.156]) by smtp.gmail.com with ESMTPSA id j62-v6sm16043423pgd.40.2018.10.16.06.14.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 06:14:20 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v2 5/5] mm: optimise pte dirty/accessed bit setting by demand based pte insertion Date: Tue, 16 Oct 2018 23:13:43 +1000 Message-Id: <20181016131343.20556-6-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20181016131343.20556-1-npiggin@gmail.com> References: <20181016131343.20556-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch , Linus Torvalds , Linux Kernel Mailing List , Nicholas Piggin , linux-mm , Ley Foon Tan , ppc-dev Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Similarly to the previous patch, this tries to optimise dirty/accessed bits in ptes to avoid access costs of hardware setting them. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 12 ++++++++---- mm/memory.c | 9 ++++++--- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f43265204d4..38c2cd3b4879 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1197,6 +1197,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); @@ -2067,7 +2068,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; - bool young, write, soft_dirty, pmd_migration = false; + bool young, write, dirty, soft_dirty, pmd_migration = false; unsigned long addr; int i; @@ -2145,7 +2146,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pmd_page(old_pmd); VM_BUG_ON_PAGE(!page_count(page), page); page_ref_add(page, HPAGE_PMD_NR - 1); - if (pmd_dirty(old_pmd)) + dirty = pmd_dirty(old_pmd); + if (dirty) SetPageDirty(page); write = pmd_write(old_pmd); young = pmd_young(old_pmd); @@ -2176,8 +2178,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = maybe_mkwrite(entry, vma); if (!write) entry = pte_wrprotect(entry); - if (!young) - entry = pte_mkold(entry); + if (young) + entry = pte_mkyoung(entry); + if (dirty) + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); } diff --git a/mm/memory.c b/mm/memory.c index 9e314339a0bd..f907ea7a6303 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1804,10 +1804,9 @@ static int insert_pfn(struct vm_area_struct *vma, unsigned long addr, entry = pte_mkspecial(pfn_t_pte(pfn, prot)); out_mkwrite: - if (mkwrite) { - entry = pte_mkyoung(entry); + entry = pte_mkyoung(entry); + if (mkwrite) entry = maybe_mkwrite(pte_mkdirty(entry), vma); - } set_pte_at(mm, addr, pte, entry); update_mmu_cache(vma, addr, pte); /* XXX: why not for insert_page? */ @@ -2534,6 +2533,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* * Clear the pte entry and flush it first, before updating the @@ -3043,6 +3043,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); + pte = pte_mkyoung(pte); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3185,6 +3186,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) __SetPageUptodate(page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); @@ -3453,6 +3455,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */