{"id":666824,"url":"http://patchwork.ozlabs.org/api/patches/666824/?format=json","web_url":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20160907061709.GD21443@fergus.ozlabs.ibm.com/","project":{"id":2,"url":"http://patchwork.ozlabs.org/api/projects/2/?format=json","name":"Linux PPC development","link_name":"linuxppc-dev","list_id":"linuxppc-dev.lists.ozlabs.org","list_email":"linuxppc-dev@lists.ozlabs.org","web_url":"https://github.com/linuxppc/wiki/wiki","scm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git","webscm_url":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/","list_archive_url":"https://lore.kernel.org/linuxppc-dev/","list_archive_url_format":"https://lore.kernel.org/linuxppc-dev/{}/","commit_url_format":"https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id={}"},"msgid":"<20160907061709.GD21443@fergus.ozlabs.ibm.com>","list_archive_url":"https://lore.kernel.org/linuxppc-dev/20160907061709.GD21443@fergus.ozlabs.ibm.com/","date":"2016-09-07T06:17:09","name":"[v2,3/3] powerpc/mm: Speed up computation of base and actual page size for a HPTE","commit_ref":null,"pull_url":null,"state":"accepted","archived":false,"hash":"480bf66030efcca35b7e1be72753ae2c62dd8630","submitter":{"id":67079,"url":"http://patchwork.ozlabs.org/api/people/67079/?format=json","name":"Paul Mackerras","email":"paulus@ozlabs.org"},"delegate":{"id":13,"url":"http://patchwork.ozlabs.org/api/users/13/?format=json","username":"paulus","first_name":"Paul","last_name":"Mackerras","email":"paulus@samba.org"},"mbox":"http://patchwork.ozlabs.org/project/linuxppc-dev/patch/20160907061709.GD21443@fergus.ozlabs.ibm.com/mbox/","series":[],"comments":"http://patchwork.ozlabs.org/api/patches/666824/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/666824/checks/","tags":{},"related":[],"headers":{"Return-Path":"","X-Original-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Delivered-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Received":["from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3sTYDx6ScGz9sDG\n\tfor ;\n\tWed, 7 Sep 2016 16:18:25 +1000 (AEST)","from ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 3sTYDx4grszDsM2\n\tfor ;\n\tWed, 7 Sep 2016 16:18:25 +1000 (AEST)","from ozlabs.org (ozlabs.org [103.22.144.67])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby lists.ozlabs.org (Postfix) with ESMTPS id 3sTYCk4BKGzDrpx\n\tfor ;\n\tWed, 7 Sep 2016 16:17:22 +1000 (AEST)","by ozlabs.org (Postfix, from userid 1003)\n\tid 3sTYCk3cwhz9sR9; Wed, 7 Sep 2016 16:17:22 +1000 (AEST)"],"Date":"Wed, 7 Sep 2016 16:17:09 +1000","From":"Paul Mackerras ","To":"linuxppc-dev@lists.ozlabs.org","Subject":"[PATCH v2 3/3] powerpc/mm: Speed up computation of base and actual\n\tpage size for a HPTE","Message-ID":"<20160907061709.GD21443@fergus.ozlabs.ibm.com>","References":"<20160902114759.GA12433@fergus.ozlabs.ibm.com>\n\t<20160902115055.GC12433@fergus.ozlabs.ibm.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20160902115055.GC12433@fergus.ozlabs.ibm.com>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-BeenThere":"linuxppc-dev@lists.ozlabs.org","X-Mailman-Version":"2.1.22","Precedence":"list","List-Id":"Linux on PowerPC Developers Mail List\n\t","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org","Sender":"\"Linuxppc-dev\"\n\t"},"content":"This replaces a 2-D search through an array with a simple 8-bit table\nlookup for determining the actual and/or base page size for a HPT entry.\n\nThe encoding in the second doubleword of the HPTE is designed to encode\nthe actual and base page sizes without using any more bits than would be\nneeded for a 4k page number, by using between 1 and 8 low-order bits of\nthe RPN (real page number) field to encode the page sizes. A single\n\"large page\" bit in the first doubleword indicates that these low-order\nbits are to be interpreted like this.\n\nWe can determine the page sizes by using the low-order 8 bits of the RPN\nto look up a 256-entry table. For actual page sizes less than 1MB, some\nof the upper bits of these 8 bits are going to be real address bits, but\nwe can cope with that by replicating the entries for those smaller page\nsizes.\n\nWhile we're at it, let's move the hpte_page_size() and hpte_base_page_size()\nfunctions from a KVM-specific header to a header for 64-bit HPT systems,\nsince this computation doesn't have anything specifically to do with KVM.\n\nSigned-off-by: Paul Mackerras \n---\nv2: added more comments as suggested by Aneesh\n\n arch/powerpc/include/asm/book3s/64/mmu-hash.h | 37 ++++++++++++\n arch/powerpc/include/asm/kvm_book3s_64.h | 87 +++------------------------\n arch/powerpc/include/asm/mmu.h | 1 +\n arch/powerpc/mm/hash_native_64.c | 42 +------------\n arch/powerpc/mm/hash_utils_64.c | 55 +++++++++++++++++\n 5 files changed, 102 insertions(+), 120 deletions(-)","diff":"diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h\nindex 287a656..e407af2 100644\n--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h\n+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h\n@@ -245,6 +245,43 @@ static inline int segment_shift(int ssize)\n }\n \n /*\n+ * This array is indexed by the LP field of the HPTE second dword.\n+ * Since this field may contain some RPN bits, some entries are\n+ * replicated so that we get the same value irrespective of RPN.\n+ * The top 4 bits are the page size index (MMU_PAGE_*) for the\n+ * actual page size, the bottom 4 bits are the base page size.\n+ */\n+extern u8 hpte_page_sizes[1 << LP_BITS];\n+\n+static inline unsigned long __hpte_page_size(unsigned long h, unsigned long l,\n+\t\t\t\t\t bool is_base_size)\n+{\n+\tunsigned int i, lp;\n+\n+\tif (!(h & HPTE_V_LARGE))\n+\t\treturn 1ul << 12;\n+\n+\t/* Look at the 8 bit LP value */\n+\tlp = (l >> LP_SHIFT) & ((1 << LP_BITS) - 1);\n+\ti = hpte_page_sizes[lp];\n+\tif (!i)\n+\t\treturn 0;\n+\tif (!is_base_size)\n+\t\ti >>= 4;\n+\treturn 1ul << mmu_psize_defs[i & 0xf].shift;\n+}\n+\n+static inline unsigned long hpte_page_size(unsigned long h, unsigned long l)\n+{\n+\treturn __hpte_page_size(h, l, 0);\n+}\n+\n+static inline unsigned long hpte_base_page_size(unsigned long h, unsigned long l)\n+{\n+\treturn __hpte_page_size(h, l, 1);\n+}\n+\n+/*\n * The current system page and segment sizes\n */\n extern int mmu_kernel_ssize;\ndiff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h\nindex 88d17b4..4ffd5a1 100644\n--- a/arch/powerpc/include/asm/kvm_book3s_64.h\n+++ b/arch/powerpc/include/asm/kvm_book3s_64.h\n@@ -20,6 +20,8 @@\n #ifndef __ASM_KVM_BOOK3S_64_H__\n #define __ASM_KVM_BOOK3S_64_H__\n \n+#include \n+\n #ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE\n static inline struct kvmppc_book3s_shadow_vcpu *svcpu_get(struct kvm_vcpu *vcpu)\n {\n@@ -97,56 +99,20 @@ static inline void __unlock_hpte(__be64 *hpte, unsigned long hpte_v)\n \thpte[0] = cpu_to_be64(hpte_v);\n }\n \n-static inline int __hpte_actual_psize(unsigned int lp, int psize)\n-{\n-\tint i, shift;\n-\tunsigned int mask;\n-\n-\t/* start from 1 ignoring MMU_PAGE_4K */\n-\tfor (i = 1; i < MMU_PAGE_COUNT; i++) {\n-\n-\t\t/* invalid penc */\n-\t\tif (mmu_psize_defs[psize].penc[i] == -1)\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * encoding bits per actual page size\n-\t\t * PTE LP actual page size\n-\t\t * rrrr rrrz\t\t>=8KB\n-\t\t * rrrr rrzz\t\t>=16KB\n-\t\t * rrrr rzzz\t\t>=32KB\n-\t\t * rrrr zzzz\t\t>=64KB\n-\t\t * .......\n-\t\t */\n-\t\tshift = mmu_psize_defs[i].shift - LP_SHIFT;\n-\t\tif (shift > LP_BITS)\n-\t\t\tshift = LP_BITS;\n-\t\tmask = (1 << shift) - 1;\n-\t\tif ((lp & mask) == mmu_psize_defs[psize].penc[i])\n-\t\t\treturn i;\n-\t}\n-\treturn -1;\n-}\n-\n static inline unsigned long compute_tlbie_rb(unsigned long v, unsigned long r,\n \t\t\t\t\t unsigned long pte_index)\n {\n-\tint b_psize = MMU_PAGE_4K, a_psize = MMU_PAGE_4K;\n+\tint i, b_psize = MMU_PAGE_4K, a_psize = MMU_PAGE_4K;\n \tunsigned int penc;\n \tunsigned long rb = 0, va_low, sllp;\n \tunsigned int lp = (r >> LP_SHIFT) & ((1 << LP_BITS) - 1);\n \n \tif (v & HPTE_V_LARGE) {\n-\t\tfor (b_psize = 0; b_psize < MMU_PAGE_COUNT; b_psize++) {\n-\n-\t\t\t/* valid entries have a shift value */\n-\t\t\tif (!mmu_psize_defs[b_psize].shift)\n-\t\t\t\tcontinue;\n-\n-\t\t\ta_psize = __hpte_actual_psize(lp, b_psize);\n-\t\t\tif (a_psize != -1)\n-\t\t\t\tbreak;\n-\t\t}\n+\t\ti = hpte_page_sizes[lp];\n+\t\tb_psize = i & 0xf;\n+\t\ta_psize = i >> 4;\n \t}\n+\n \t/*\n \t * Ignore the top 14 bits of va\n \t * v have top two bits covering segment size, hence move\n@@ -215,45 +181,6 @@ static inline unsigned long compute_tlbie_rb(unsigned long v, unsigned long r,\n \treturn rb;\n }\n \n-static inline unsigned long __hpte_page_size(unsigned long h, unsigned long l,\n-\t\t\t\t\t bool is_base_size)\n-{\n-\n-\tint size, a_psize;\n-\t/* Look at the 8 bit LP value */\n-\tunsigned int lp = (l >> LP_SHIFT) & ((1 << LP_BITS) - 1);\n-\n-\t/* only handle 4k, 64k and 16M pages for now */\n-\tif (!(h & HPTE_V_LARGE))\n-\t\treturn 1ul << 12;\n-\telse {\n-\t\tfor (size = 0; size < MMU_PAGE_COUNT; size++) {\n-\t\t\t/* valid entries have a shift value */\n-\t\t\tif (!mmu_psize_defs[size].shift)\n-\t\t\t\tcontinue;\n-\n-\t\t\ta_psize = __hpte_actual_psize(lp, size);\n-\t\t\tif (a_psize != -1) {\n-\t\t\t\tif (is_base_size)\n-\t\t\t\t\treturn 1ul << mmu_psize_defs[size].shift;\n-\t\t\t\treturn 1ul << mmu_psize_defs[a_psize].shift;\n-\t\t\t}\n-\t\t}\n-\n-\t}\n-\treturn 0;\n-}\n-\n-static inline unsigned long hpte_page_size(unsigned long h, unsigned long l)\n-{\n-\treturn __hpte_page_size(h, l, 0);\n-}\n-\n-static inline unsigned long hpte_base_page_size(unsigned long h, unsigned long l)\n-{\n-\treturn __hpte_page_size(h, l, 1);\n-}\n-\n static inline unsigned long hpte_rpn(unsigned long ptel, unsigned long psize)\n {\n \treturn ((ptel & HPTE_R_RPN) & ~(psize - 1)) >> PAGE_SHIFT;\ndiff --git a/arch/powerpc/include/asm/mmu.h b/arch/powerpc/include/asm/mmu.h\nindex e2fb408..b78e8d3 100644\n--- a/arch/powerpc/include/asm/mmu.h\n+++ b/arch/powerpc/include/asm/mmu.h\n@@ -271,6 +271,7 @@ static inline bool early_radix_enabled(void)\n #define MMU_PAGE_16G\t13\n #define MMU_PAGE_64G\t14\n \n+/* N.B. we need to change the type of hpte_page_sizes if this gets to be > 16 */\n #define MMU_PAGE_COUNT\t15\n \n #ifdef CONFIG_PPC_BOOK3S_64\ndiff --git a/arch/powerpc/mm/hash_native_64.c b/arch/powerpc/mm/hash_native_64.c\nindex 0e4e965..83ddc0e 100644\n--- a/arch/powerpc/mm/hash_native_64.c\n+++ b/arch/powerpc/mm/hash_native_64.c\n@@ -493,36 +493,6 @@ static void native_hugepage_invalidate(unsigned long vsid,\n }\n #endif\n \n-static inline int __hpte_actual_psize(unsigned int lp, int psize)\n-{\n-\tint i, shift;\n-\tunsigned int mask;\n-\n-\t/* start from 1 ignoring MMU_PAGE_4K */\n-\tfor (i = 1; i < MMU_PAGE_COUNT; i++) {\n-\n-\t\t/* invalid penc */\n-\t\tif (mmu_psize_defs[psize].penc[i] == -1)\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * encoding bits per actual page size\n-\t\t * PTE LP actual page size\n-\t\t * rrrr rrrz\t\t>=8KB\n-\t\t * rrrr rrzz\t\t>=16KB\n-\t\t * rrrr rzzz\t\t>=32KB\n-\t\t * rrrr zzzz\t\t>=64KB\n-\t\t * .......\n-\t\t */\n-\t\tshift = mmu_psize_defs[i].shift - LP_SHIFT;\n-\t\tif (shift > LP_BITS)\n-\t\t\tshift = LP_BITS;\n-\t\tmask = (1 << shift) - 1;\n-\t\tif ((lp & mask) == mmu_psize_defs[psize].penc[i])\n-\t\t\treturn i;\n-\t}\n-\treturn -1;\n-}\n-\n static void hpte_decode(struct hash_pte *hpte, unsigned long slot,\n \t\t\tint *psize, int *apsize, int *ssize, unsigned long *vpn)\n {\n@@ -538,16 +508,8 @@ static void hpte_decode(struct hash_pte *hpte, unsigned long slot,\n \t\tsize = MMU_PAGE_4K;\n \t\ta_size = MMU_PAGE_4K;\n \t} else {\n-\t\tfor (size = 0; size < MMU_PAGE_COUNT; size++) {\n-\n-\t\t\t/* valid entries have a shift value */\n-\t\t\tif (!mmu_psize_defs[size].shift)\n-\t\t\t\tcontinue;\n-\n-\t\t\ta_size = __hpte_actual_psize(lp, size);\n-\t\t\tif (a_size != -1)\n-\t\t\t\tbreak;\n-\t\t}\n+\t\tsize = hpte_page_sizes[lp] & 0xf;\n+\t\ta_size = hpte_page_sizes[lp] >> 4;\n \t}\n \t/* This works for all page sizes, and for 256M and 1T segments */\n \tif (cpu_has_feature(CPU_FTR_ARCH_300))\ndiff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c\nindex 0821556..ef3ae89 100644\n--- a/arch/powerpc/mm/hash_utils_64.c\n+++ b/arch/powerpc/mm/hash_utils_64.c\n@@ -93,6 +93,9 @@ static unsigned long _SDR1;\n struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT];\n EXPORT_SYMBOL_GPL(mmu_psize_defs);\n \n+u8 hpte_page_sizes[1 << LP_BITS];\n+EXPORT_SYMBOL_GPL(hpte_page_sizes);\n+\n struct hash_pte *htab_address;\n unsigned long htab_size_bytes;\n unsigned long htab_hash_mask;\n@@ -564,8 +567,60 @@ static void __init htab_scan_page_sizes(void)\n #endif /* CONFIG_HUGETLB_PAGE */\n }\n \n+/*\n+ * Fill in the hpte_page_sizes[] array.\n+ * We go through the mmu_psize_defs[] array looking for all the\n+ * supported base/actual page size combinations. Each combination\n+ * has a unique pagesize encoding (penc) value in the low bits of\n+ * the LP field of the HPTE. For actual page sizes less than 1MB,\n+ * some of the upper LP bits are used for RPN bits, meaning that\n+ * we need to fill in several entries in hpte_page_sizes[].\n+ *\n+ * In diagrammatic form, with r = RPN bits and z = page size bits:\n+ * PTE LP actual page size\n+ * rrrr rrrz\t\t>=8KB\n+ * rrrr rrzz\t\t>=16KB\n+ * rrrr rzzz\t\t>=32KB\n+ * rrrr zzzz\t\t>=64KB\n+ * ...\n+ *\n+ * The zzzz bits are implementation-specific but are chosen so that\n+ * no encoding for a larger page size uses the same value in its\n+ * low-order N bits as the encoding for the 2^(12+N) byte page size\n+ * (if it exists).\n+ */\n+static void init_hpte_page_sizes(void)\n+{\n+\tlong int ap, bp;\n+\tlong int shift, penc;\n+\n+\tfor (bp = 0; bp < MMU_PAGE_COUNT; ++bp) {\n+\t\tif (!mmu_psize_defs[bp].shift)\n+\t\t\tcontinue;\t/* not a supported page size */\n+\t\tfor (ap = bp; ap < MMU_PAGE_COUNT; ++ap) {\n+\t\t\tpenc = mmu_psize_defs[bp].penc[ap];\n+\t\t\tif (penc == -1)\n+\t\t\t\tcontinue;\n+\t\t\tshift = mmu_psize_defs[ap].shift - LP_SHIFT;\n+\t\t\tif (shift <= 0)\n+\t\t\t\tcontinue;\t/* should never happen */\n+\t\t\t/*\n+\t\t\t * For page sizes less than 1MB, this loop\n+\t\t\t * replicates the entry for all possible values\n+\t\t\t * of the rrrr bits.\n+\t\t\t */\n+\t\t\twhile (penc < (1 << LP_BITS)) {\n+\t\t\t\thpte_page_sizes[penc] = (ap << 4) | bp;\n+\t\t\t\tpenc += 1 << shift;\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n static void __init htab_init_page_sizes(void)\n {\n+\tinit_hpte_page_sizes();\n+\n \tif (!debug_pagealloc_enabled()) {\n \t\t/*\n \t\t * Pick a size for the linear mapping. Currently, we only\n","prefixes":["v2","3/3"]}