From patchwork Mon May 1 06:34:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Balbir Singh X-Patchwork-Id: 756991 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wGZVk2NsRz9s8N for ; Mon, 1 May 2017 16:38:06 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FCUIQ51h"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3wGZVk1LPtzDqK9 for ; Mon, 1 May 2017 16:38:06 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FCUIQ51h"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3wGZR50pvvzDq9m for ; Mon, 1 May 2017 16:34:57 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FCUIQ51h"; dkim-atps=neutral Received: by mail-pf0-x241.google.com with SMTP id v14so26689931pfd.3 for ; Sun, 30 Apr 2017 23:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=772s+0eiELaUtJLCZsbC9A6pwYTrXnIjdq59AoK7hso=; b=FCUIQ51hrVeYFWlgW5JgFjFVud5WTpfpgevMmyqzUSss0+lFlyRZXs7359fgTiENZ5 aL0p/Hd0fenKusGh6TQ+6+eBsm3L8P9Tv1Vx1SQwFwscY4vHWSw5hnvQ5suXiGDUvb28 XoEb1FSvPPBBFogCu1UXcpym+YDIM1LjGUDZZoKChKpRryWaE1NXSaP19hyPb6iC7gIY G4UInfWB/rSrjIVzh+L3YryK73Y8nhF33Y8Xq/pPNCdYZX4G7U11aNfEugBQHtk3CPeA 4V/z3H8HsuscxtA8DveKvA5GOCIrKxtlCYOX5Ow9gVlK4cxiEFPBfSCfzbG+7qJ+ZvVz 5/8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=772s+0eiELaUtJLCZsbC9A6pwYTrXnIjdq59AoK7hso=; b=YeiztmDK/5sL80AZfA1u9xkOq1PJt2sYuEB9/00caiMbBGaxbAJGNOnMN7nAupnfpK jhM1l74bNpum2KwBnIZbqoV5jO0AzOrWDIv58VPL1TOPcLQIDEC3uObxOG0/rfpaJRz0 RErKIzAzzXDrDjMA+WhVZvr/7hRAI/hsY3DYuyA1NbzWMl8pVd+9R8vzPV//H+liUeVy jLopc4K7Ao25WWb97GcjfF3yPbsERnA2GrcUKUsy6a/qFMc3MvmM0/FXJlbwTHn3fEJX Ah1DaQp3UXTFSDmoSRP4O9Gk47QVFcuAXTstYXqx3cIR7H1XBzlxvE6bDP3Sa/nK9YNy FyvA== X-Gm-Message-State: AN3rC/49d+vjUbOkR2VQ99a/kiqnyySRbnB09e1Z82LlxJbDv9lgTiM8 zdsXLaJ/cYbqYw== X-Received: by 10.99.102.195 with SMTP id a186mr25208618pgc.11.1493620495362; Sun, 30 Apr 2017 23:34:55 -0700 (PDT) Received: from firefly.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id v85sm24026402pfk.5.2017.04.30.23.34.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Apr 2017 23:34:54 -0700 (PDT) From: Balbir Singh To: vdavydov@virtuozzo.com, mpe@ellerman.id.au, oss@buserror.net Subject: [PATCH v2 1/3] powerpc/mm/book(e)(3s)/64: Add page table accounting Date: Mon, 1 May 2017 16:34:36 +1000 Message-Id: <20170501063438.25237-2-bsingharora@gmail.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20170501063438.25237-1-bsingharora@gmail.com> References: <20170501063438.25237-1-bsingharora@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Introduce a helper pgtable_gfp_flags() which just returns the current gfp flags and adds __GFP_ACCOUNT to account for page table allocation. The generic helper is added to include/asm/pgalloc.h and has two variants - WARNING ugly bits ahead 1. If the header is included from a module, no check for mm == &init_mm is done, since init_mm is not exported 2. For kernel includes, the check is done and required see (3e79ec7 arch: x86: charge page tables to kmemcg) The fundamental assumption is that no module should be doing pgd/pud/pmd and pte alloc's on behalf of init_mm directly. NOTE: This adds an overhead to pmd/pud/pgd allocations similar to x86. The other alternative was to implement pmd_alloc_kernel/pud_alloc_kernel and pgd_alloc_kernel with their offset variants. For 4k page size, pte_alloc_one no longer calls pte_alloc_one_kernel. Signed-off-by: Balbir Singh --- arch/powerpc/include/asm/book3s/32/pgalloc.h | 3 ++- arch/powerpc/include/asm/book3s/64/pgalloc.h | 17 +++++++++++------ arch/powerpc/include/asm/nohash/64/pgalloc.h | 12 ++++++++---- arch/powerpc/include/asm/pgalloc.h | 14 ++++++++++++++ arch/powerpc/mm/pgtable_64.c | 3 ++- 5 files changed, 37 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index d310546..a120e7f 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -31,7 +31,8 @@ extern struct kmem_cache *pgtable_cache[]; static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h index cd5e7aa..9aa904f 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -53,10 +53,11 @@ extern void __tlb_remove_table(void *_table); static inline pgd_t *radix__pgd_alloc(struct mm_struct *mm) { #ifdef CONFIG_PPC_64K_PAGES - return (pgd_t *)__get_free_page(PGALLOC_GFP); + return (pgd_t *)__get_free_page(pgtable_gfp_flags(mm, PGALLOC_GFP)); #else struct page *page; - page = alloc_pages(PGALLOC_GFP | __GFP_REPEAT, 4); + page = alloc_pages(pgtable_gfp_flags(mm, PGALLOC_GFP | __GFP_REPEAT), + 4); if (!page) return NULL; return (pgd_t *) page_address(page); @@ -76,7 +77,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) { if (radix_enabled()) return radix__pgd_alloc(mm); - return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) @@ -93,7 +95,8 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pud_free(struct mm_struct *mm, pud_t *pud) @@ -119,7 +122,8 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) @@ -168,7 +172,8 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, struct page *page; pte_t *pte; - pte = pte_alloc_one_kernel(mm, address); + pte = (pte_t *)__get_free_page(pgtable_gfp_flags(mm, + GFP_KERNEL | __GFP_ZERO)); if (!pte) return NULL; page = virt_to_page(pte); diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h index 897d2e1..0d5e37b 100644 --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h @@ -43,7 +43,8 @@ extern struct kmem_cache *pgtable_cache[]; static inline pgd_t *pgd_alloc(struct mm_struct *mm) { - return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) @@ -57,7 +58,8 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PUD_INDEX_SIZE), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pud_free(struct mm_struct *mm, pud_t *pud) @@ -96,7 +98,8 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, struct page *page; pte_t *pte; - pte = pte_alloc_one_kernel(mm, address); + pte = (pte_t *)__get_free_page( + pgtable_gfp_flags(mm, GFP_KERNEL | __GFP_ZERO)); if (!pte) return NULL; page = virt_to_page(pte); @@ -189,7 +192,8 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), GFP_KERNEL); + return kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX), + pgtable_gfp_flags(mm, GFP_KERNEL)); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h index 0413457..d795c5d 100644 --- a/arch/powerpc/include/asm/pgalloc.h +++ b/arch/powerpc/include/asm/pgalloc.h @@ -3,6 +3,20 @@ #include +#ifndef MODULE +static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm, gfp_t gfp) +{ + if (unlikely(mm == &init_mm)) + return gfp; + return gfp | __GFP_ACCOUNT; +} +#else /* !MODULE */ +static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm, gfp_t gfp) +{ + return gfp | __GFP_ACCOUNT; +} +#endif /* MODULE */ + #ifdef CONFIG_PPC_BOOK3S #include #else diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c index db93cf7..b322262 100644 --- a/arch/powerpc/mm/pgtable_64.c +++ b/arch/powerpc/mm/pgtable_64.c @@ -351,9 +351,10 @@ static pte_t *get_from_cache(struct mm_struct *mm) static pte_t *__alloc_for_cache(struct mm_struct *mm, int kernel) { void *ret = NULL; - struct page *page = alloc_page(GFP_KERNEL | __GFP_NOTRACK | __GFP_ZERO); + struct page *page = alloc_page(pgtable_gfp_flags(mm, PGALLOC_GFP)); if (!page) return NULL; + if (!kernel && !pgtable_page_ctor(page)) { __free_page(page); return NULL;