From patchwork Thu Aug 16 15:15:53 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 178039 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from ozlabs.org (localhost [IPv6:::1]) by ozlabs.org (Postfix) with ESMTP id 23FA82C0BF1 for ; Fri, 17 Aug 2012 01:20:48 +1000 (EST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by ozlabs.org (Postfix) with ESMTP id 576292C0222 for ; Fri, 17 Aug 2012 01:16:29 +1000 (EST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP; 16 Aug 2012 08:16:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.77,778,1336374000"; d="scan'208";a="209391446" Received: from blue.fi.intel.com ([10.237.72.50]) by fmsmga002.fm.intel.com with ESMTP; 16 Aug 2012 08:15:56 -0700 Received: by blue.fi.intel.com (Postfix, from userid 1000) id 470B5E008A; Thu, 16 Aug 2012 18:15:59 +0300 (EEST) From: "Kirill A. Shutemov" To: linux-mm@kvack.org Subject: [PATCH v3 6/7] mm: make clear_huge_page cache clear only around the fault address Date: Thu, 16 Aug 2012 18:15:53 +0300 Message-Id: <1345130154-9602-7-git-send-email-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1345130154-9602-1-git-send-email-kirill.shutemov@linux.intel.com> References: <1345130154-9602-1-git-send-email-kirill.shutemov@linux.intel.com> Cc: linux-mips@linux-mips.org, linux-sh@vger.kernel.org, Jan Beulich , "H. Peter Anvin" , sparclinux@vger.kernel.org, Andrea Arcangeli , Andi Kleen , Robert Richter , x86@kernel.org, Hugh Dickins , Ingo Molnar , Mel Gorman , Alex Shi , Thomas Gleixner , KAMEZAWA Hiroyuki , Tim Chen , linux-kernel@vger.kernel.org, Andy Lutomirski , Johannes Weiner , Andrew Morton , linuxppc-dev@lists.ozlabs.org, "Kirill A. Shutemov" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Andi Kleen Clearing a 2MB huge page will typically blow away several levels of CPU caches. To avoid this only cache clear the 4K area around the fault address and use a cache avoiding clears for the rest of the 2MB area. Signed-off-by: Andi Kleen Signed-off-by: Kirill A. Shutemov --- mm/memory.c | 34 +++++++++++++++++++++++++++++----- 1 files changed, 29 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dfc179b..d4626b9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3969,18 +3969,34 @@ EXPORT_SYMBOL(might_fault); #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) + +#ifndef ARCH_HAS_USER_NOCACHE +#define ARCH_HAS_USER_NOCACHE 0 +#endif + +#if ARCH_HAS_USER_NOCACHE == 0 +#define clear_user_highpage_nocache clear_user_highpage +#endif + static void clear_gigantic_page(struct page *page, - unsigned long addr, - unsigned int pages_per_huge_page) + unsigned long haddr, unsigned long fault_address, + unsigned int pages_per_huge_page) { int i; struct page *p = page; + unsigned long vaddr; + int target = (fault_address - haddr) >> PAGE_SHIFT; might_sleep(); + vaddr = haddr; for (i = 0; i < pages_per_huge_page; i++, p = mem_map_next(p, page, i)) { cond_resched(); - clear_user_highpage(p, addr + i * PAGE_SIZE); + vaddr = haddr + i*PAGE_SIZE; + if (!ARCH_HAS_USER_NOCACHE || i == target) + clear_user_highpage(p, vaddr); + else + clear_user_highpage_nocache(p, vaddr); } } void clear_huge_page(struct page *page, @@ -3988,16 +4004,24 @@ void clear_huge_page(struct page *page, unsigned int pages_per_huge_page) { int i; + unsigned long vaddr; + int target = (fault_address - haddr) >> PAGE_SHIFT; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { - clear_gigantic_page(page, haddr, pages_per_huge_page); + clear_gigantic_page(page, haddr, fault_address, + pages_per_huge_page); return; } might_sleep(); + vaddr = haddr; for (i = 0; i < pages_per_huge_page; i++) { cond_resched(); - clear_user_highpage(page + i, haddr + i * PAGE_SIZE); + vaddr = haddr + i*PAGE_SIZE; + if (!ARCH_HAS_USER_NOCACHE || i == target) + clear_user_highpage(page + i, vaddr); + else + clear_user_highpage_nocache(page + i, vaddr); } }