From patchwork Tue Aug 28 11:20:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 962862 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4205yb52Tmz9rvt for ; Tue, 28 Aug 2018 21:25:11 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BZHp6QbN"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4205yb36QqzDrM8 for ; Tue, 28 Aug 2018 21:25:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BZHp6QbN"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BZHp6QbN"; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4205sb2xphzDrRN for ; Tue, 28 Aug 2018 21:20:51 +1000 (AEST) Received: by mail-pf1-x443.google.com with SMTP id h79-v6so561351pfk.8 for ; Tue, 28 Aug 2018 04:20:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PqIy7bFKrUvh3N2mZhaZbdyHPW+hgwvBa4i30Nw8Ybc=; b=BZHp6QbN59dTJzQoscwjj/i43FAP0Gjci3ePWXPKsGC8C1C6frC6pwM7ZO9Og1acP6 kalOnMjZXknjI8Nx28NzpbS+Bf3PiNWazSfTSbEb5Tfl+239vryJxYCtq1UZm1KWzn0H LErHEmA5WLAWDKAGAxnCjAXNSt84PWxsr1fyECPVZJQ/WqwNnCqAHpj1BOcu3dssf1Xt ri+KTIc8FfxXuQLLx0wAF3iLM12t2IOTD2KNAaa3tt6ARchgJ8OtW2orWdvTnAmDuRT0 34rrzhoTIK0LlmWUaEsu/2lFprRk1Z7eQ/jprd4BLoyNtsG+l4id/94KNevuGHZiRKdN ojKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PqIy7bFKrUvh3N2mZhaZbdyHPW+hgwvBa4i30Nw8Ybc=; b=ElbH0mTiKjsp3xkbysOBUHTatqY8Q0arjGKk9O24vYoludGaAAXRELdccD9GVc42Ev XKR4bBD70gIJzvLKI4NvlBluIOkSKYbz+7VmAmj2izfCNN9PcKdRLHSEH9u1j6e6nVMi WUoZZ0xZZzSkoryp+ytLqTn4q8RuppI8eJpEdwENMBi0HjHMt/J6vfukgj67r2S3RpbJ nvh1tFhhlSCsStnNfis07mNlxIitsHk6f69GszUU9pQ5D8y3FMjKvJQtDsOvBYgbVoPe hUPMFu7q0rH79F3BV3XRIS82de2jbYmgqnKIFcnonLDP1OuJ72kEIH3UhTPB5qsi1njU 4vwQ== X-Gm-Message-State: APzg51A2SA+2e9rumRnJ96XotsTT2XF4TpriXS/fr23GO6DXuCdJ5B6L Awl74xbRvvwWr7qFAwSs6g8= X-Google-Smtp-Source: ANB0Vda1rxfWQ20CAFltZX59amKlHwbHH1qDC49g8kZHwadojwF7gpVndoanEo69eLkZduJDG3UbpQ== X-Received: by 2002:a63:5751:: with SMTP id h17-v6mr1085425pgm.423.1535455249468; Tue, 28 Aug 2018 04:20:49 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH 1/3] mm/cow: don't bother write protectig already write-protected huge pages Date: Tue, 28 Aug 2018 21:20:32 +1000 Message-Id: <20180828112034.30875-2-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Nicholas Piggin , Andrew Morton , Linus Torvalds Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This is the THP equivalent for 1b2de5d039c8 ("mm/cow: don't bother write protecting already write-protected pages"). Explicit hugetlb pages don't get the same treatment because they don't appear to have the right accessor functions. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9592cbd8530a..d9bae12978ef 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -973,8 +973,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); - pmdp_set_wrprotect(src_mm, addr, src_pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + if (pmd_write(pmd)) { + pmdp_set_wrprotect(src_mm, addr, src_pmd); + pmd = pmd_wrprotect(pmd); + } + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1064,8 +1067,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, /* No huge zero pud yet */ } - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; From patchwork Tue Aug 28 11:20:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 962866 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42061G5VYCz9s2P for ; Tue, 28 Aug 2018 21:27:30 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rOguRM3J"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42061G3wzNzDrM8 for ; Tue, 28 Aug 2018 21:27:30 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rOguRM3J"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::641; helo=mail-pl1-x641.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rOguRM3J"; dkim-atps=neutral Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4205sh24cLzDrqP for ; Tue, 28 Aug 2018 21:20:56 +1000 (AEST) Received: by mail-pl1-x641.google.com with SMTP id ba4-v6so582832plb.11 for ; Tue, 28 Aug 2018 04:20:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rCMUeVqKxlpR8mAw1QaqwgN5G/ui6F18sxW16mRgzbg=; b=rOguRM3J3lcM/kPZaSPcAEQbaVyrefutmBcfZrlssC+KfOFnsXJ6Vjui5heOyza9Xt iK7+x8ng89Bjapc7F0EsHJVJL5mB4ONxLruzuwpj/fUVU1WeBA6YVerZ2bDzK8nAemv0 uiBq3kX8C2IYkp9o+e1KW8Ac4R6HS4UvGcSAbaickQf+2TuqDmHDAX2cpKOajfYDYmOA VAm6IQKvxoIOHHydgXN+4A1KtQN/jt1SfMGDmY9E5mZE9DQgP2DXejhn/yB6yWhb8JwG 0TUf+aWp/CUuaqFWPjc6qjO1WRuCZukxDNIRvlCHNhD7UCpEn6x9ZTVKVRx+zG6Q4Ttp u1YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rCMUeVqKxlpR8mAw1QaqwgN5G/ui6F18sxW16mRgzbg=; b=IiWLtm0T4TRGME4Ru6gf4mo8wWTFlU/YeK8vyjQP3Q2TgxC0lmzALxoLQ03oRC8qAD WswpCLR8IVOKaR0Uwfpl8aRTyhpMbiUCdPmWueJ99O+6CriG8CZznw0F6aUCuf5JCu9L w1eSpHADPztKjTZBE5YZjsislWiZmCH4KaxCAQeB+hsMufojEF+Cii9o7tLxR2Gfbbye BPpNac8XSOWnAyhTKHLfntwt/xKoUcVeFvZXdyTRvyCKn0o8bjFiH6iXVjKlxKH5GwA1 rbcinLrDRbg8dXq/rclgt4K4oo7+73kpPPiUN327Ee3ZYMPjYK3bIDXY2LyBShVjn5Nb 3ESw== X-Gm-Message-State: APzg51CQc6NhM2pR0xTsVKv7t2j66fsGlAgqwMCGI/VMCQM7rGEzVEQE 4C9VKmhhqGad1+bIQnUR/6I= X-Google-Smtp-Source: ANB0Vdbt9K3kz71sMSBn8c+JLrglp1376ALqJAIKdR2WDl+whCCjXliR4Y+1Pyz04+N2UEXNEBrAKQ== X-Received: by 2002:a17:902:6b4c:: with SMTP id g12-v6mr1111608plt.159.1535455253350; Tue, 28 Aug 2018 04:20:53 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:52 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH 2/3] mm/cow: optimise pte dirty/accessed bits handling in fork Date: Tue, 28 Aug 2018 21:20:33 +1000 Message-Id: <20180828112034.30875-3-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Nicholas Piggin , Andrew Morton , Linus Torvalds Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" fork clears dirty/accessed bits from new ptes in the child. This logic has existed since mapped page reclaim was done by scanning ptes when it may have been quite important. Today with physical based pte scanning, there is less reason to clear these bits. Dirty bits are all tested and cleared together and any dirty bit is the same as many dirty bits. Any young bit is treated similarly to many young bits, but not quite the same. A comment has been added where there is some difference. This eliminates a major source of faults powerpc/radix requires to set dirty/accessed bits in ptes, speeding up a fork/exit microbenchmark by about 5% on POWER9 (16600 -> 17500 fork/execs per second). Skylake appears to have a micro-fault overhead too -- a test which allocates 4GB anonymous memory, reads each page, then forks, and times the child reading a byte from each page. The first pass over the pages takes about 1000 cycles per page, the second pass takes about 27 cycles (TLB miss). With no additional minor faults measured due to either child pass, and the page array well exceeding TLB capacity, the large cost must be caused by micro faults caused by setting accessed bit. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 2 -- mm/memory.c | 10 +++++----- mm/vmscan.c | 8 ++++++++ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d9bae12978ef..5fb1a43e12e0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -977,7 +977,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); pmd = pmd_wrprotect(pmd); } - pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1071,7 +1070,6 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } - pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0; diff --git a/mm/memory.c b/mm/memory.c index b616a69ad770..3d8bf8220bd0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1038,12 +1038,12 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * If it's a shared mapping, mark it clean in - * the child + * Child inherits dirty and young bits from parent. There is no + * point clearing them because any cleaning or aging has to walk + * all ptes anyway, and it will notice the bits set in the parent. + * Leaving them set avoids stalls and even page faults on CPUs that + * handle these bits in software. */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); - pte = pte_mkold(pte); page = vm_normal_page(vma, addr, pte); if (page) { diff --git a/mm/vmscan.c b/mm/vmscan.c index 7e7d25504651..52fe64af3d80 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1021,6 +1021,14 @@ static enum page_references page_check_references(struct page *page, * to look twice if a mapped file page is used more * than once. * + * fork() will set referenced bits in child ptes despite + * not having been accessed, to avoid micro-faults of + * setting accessed bits. This heuristic is not perfectly + * accurate in other ways -- multiple map/unmap in the + * same time window would be treated as multiple references + * despite same number of actual memory accesses made by + * the program. + * * Mark it and spare it for another trip around the * inactive list. Another page table reference will * lead to its activation. From patchwork Tue Aug 28 11:20:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 962876 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42064d21qzz9rvt for ; Tue, 28 Aug 2018 21:30:25 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JwRKkOpx"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 42064c74ypzDrYf for ; Tue, 28 Aug 2018 21:30:24 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JwRKkOpx"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::644; helo=mail-pl1-x644.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JwRKkOpx"; dkim-atps=neutral Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4205sl0NPbzDrqP for ; Tue, 28 Aug 2018 21:20:58 +1000 (AEST) Received: by mail-pl1-x644.google.com with SMTP id b12-v6so591081plr.8 for ; Tue, 28 Aug 2018 04:20:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u5LLVwzqsd0W3khip77eczbmhBS8m0M7MliDfum5fbo=; b=JwRKkOpxwjV2ovQ7J2Ho5+UlDBygiVUre+hfgsY9ifuXf7lMftdoyq0EjG5Y4wV9jI edNRMwartMVSqXKLDu9KmlMzCUzsjm1SEnGMe2GKjqwu5rEkfsiRHxxqwBpQMsqp84uM 61qFr8Q4SUNZ52mt4/gmik+I7401vIvRBgcknLsZKC4trEXjn7k+zwZSc1V1G9+jFkyv BoW03rpPR4WSm6aaQuOxSQqpT2H0MJDPANu/Bs7ctr23p5f5jUTAAiVEgXGHIM0Ktpv1 1j6HNaUmy1Rnx0kpiaEMO7Ut3drfbl2yP3OpVKcWN5uUo30RDYFEChjPVH3nS0A+Qxzg 7TOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u5LLVwzqsd0W3khip77eczbmhBS8m0M7MliDfum5fbo=; b=kOJiNg4puvLJ6mVxeTEimDYJBn1hiyHx4ZWnwkq1qscH305G30p2j+H9TFbLuLh0fK 7gQYxMwIm42QjUOjZgXIntTjU7Ecj554wAE6SisVlJdT6NQOyjNYdrLzjTWkt088PhJb 3FjnvgfNcjuUH2iWvR0Iatc9Of4+kKOdWSWMpza1QX+Ofr3hF5XHCwSJCQrMsLnDpCnv cPD9nusM5St14a6ejEW75p945QhfzK+n3GYzW6L3TvSUjobb0J3l/NxV/iAOn2CgwtLc LbJqt0Ut7R9yj5af0geoiPK5tanP5IGxQ+A6YYpy2wzk6l1MdI5/JFKXS2iO+Zg/wQG4 vpUw== X-Gm-Message-State: APzg51BNxqZM0LGFoNfRBV/87auwUPFZkli/cGKAOaRVmQGgT+kI26RL IMfysESS0ZgI20K+Nczfnj4= X-Google-Smtp-Source: ANB0VdYR8Iq6a5/JlwAWF9iIuyrZ3FY50H+M9B7BfYKPAMTl0ScSRzxrBp5pb47wFpOFK+WGxxiTKg== X-Received: by 2002:a17:902:6845:: with SMTP id f5-v6mr1121540pln.17.1535455257066; Tue, 28 Aug 2018 04:20:57 -0700 (PDT) Received: from roar.au.ibm.com (59-102-81-67.tpgi.com.au. [59.102.81.67]) by smtp.gmail.com with ESMTPSA id s3-v6sm3287917pgj.84.2018.08.28.04.20.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Aug 2018 04:20:56 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH 3/3] mm: optimise pte dirty/accessed bit setting by demand based pte insertion Date: Tue, 28 Aug 2018 21:20:34 +1000 Message-Id: <20180828112034.30875-4-npiggin@gmail.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180828112034.30875-1-npiggin@gmail.com> References: <20180828112034.30875-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Nicholas Piggin , Andrew Morton , Linus Torvalds Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Similarly to the previous patch, this tries to optimise dirty/accessed bits in ptes to avoid access costs of hardware setting them. Signed-off-by: Nicholas Piggin --- mm/huge_memory.c | 12 +++++++----- mm/memory.c | 8 +++++--- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5fb1a43e12e0..2c169041317f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1197,6 +1197,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); @@ -2067,7 +2068,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; - bool young, write, soft_dirty, pmd_migration = false; + bool young, write, dirty, soft_dirty, pmd_migration = false; unsigned long addr; int i; @@ -2145,8 +2146,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pmd_page(old_pmd); VM_BUG_ON_PAGE(!page_count(page), page); page_ref_add(page, HPAGE_PMD_NR - 1); - if (pmd_dirty(old_pmd)) - SetPageDirty(page); + dirty = pmd_dirty(old_pmd); write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); @@ -2176,8 +2176,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = maybe_mkwrite(entry, vma); if (!write) entry = pte_wrprotect(entry); - if (!young) - entry = pte_mkold(entry); + if (young) + entry = pte_mkyoung(entry); + if (dirty) + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); } diff --git a/mm/memory.c b/mm/memory.c index 3d8bf8220bd0..d205ba69918c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1830,10 +1830,9 @@ static int insert_pfn(struct vm_area_struct *vma, unsigned long addr, entry = pte_mkspecial(pfn_t_pte(pfn, prot)); out_mkwrite: - if (mkwrite) { - entry = pte_mkyoung(entry); + entry = pte_mkyoung(entry); + if (mkwrite) entry = maybe_mkwrite(pte_mkdirty(entry), vma); - } set_pte_at(mm, addr, pte, entry); update_mmu_cache(vma, addr, pte); /* XXX: why not for insert_page? */ @@ -2560,6 +2559,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); + entry = pte_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* * Clear the pte entry and flush it first, before updating the @@ -3069,6 +3069,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); pte = mk_pte(page, vma->vm_page_prot); + pte = pte_mkyoung(pte); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3479,6 +3480,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); + entry = pte_mkyoung(entry); if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */