From patchwork Tue Apr 17 07:38:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 899122 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40QHPm58Vrz9s19 for ; Tue, 17 Apr 2018 17:46:36 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=c-s.fr Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40QHPm3Z9vzF1xm for ; Tue, 17 Apr 2018 17:46:36 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=c-s.fr X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=c-s.fr (client-ip=93.17.236.30; helo=pegase1.c-s.fr; envelope-from=christophe.leroy@c-s.fr; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=c-s.fr Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40QHDj2cGnzDrJw for ; Tue, 17 Apr 2018 17:38:45 +1000 (AEST) Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 40QHDT4t45z9ty7y; Tue, 17 Apr 2018 09:38:33 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id Xu7q8_yljDEC; Tue, 17 Apr 2018 09:38:33 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 40QHDT4NjMz9ty7C; Tue, 17 Apr 2018 09:38:33 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B48C78B888; Tue, 17 Apr 2018 09:38:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id oL9VzNHM30Py; Tue, 17 Apr 2018 09:38:41 +0200 (CEST) Received: from po15720vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.231.2]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 7F09F8B887; Tue, 17 Apr 2018 09:38:41 +0200 (CEST) Received: by po15720vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 702046C030; Tue, 17 Apr 2018 09:38:41 +0200 (CEST) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [PATCH 4/7] powerpc/lib: inline memcmp() for small constant sizes To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman Date: Tue, 17 Apr 2018 09:38:41 +0200 (CEST) X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" In my 8xx configuration, I get 208 calls to memcmp() Within those 208 calls, about half of them have constant sizes, 46 have a size of 8, 17 have a size of 16, only a few have a size over 16. Other fixed sizes are mostly 4, 6 and 10. This patch inlines calls to memcmp() when size is constant and lower than or equal to 16 In my 8xx configuration, this reduces the number of calls to memcmp() from 208 to 123 The following table shows the number of TB timeticks to perform a constant size memcmp() before and after the patch depending on the size Before After Improvement 01: 7577 5682 25% 02: 41668 5682 86% 03: 51137 13258 74% 04: 45455 5682 87% 05: 58713 13258 77% 06: 58712 13258 77% 07: 68183 20834 70% 08: 56819 15153 73% 09: 70077 28411 60% 10: 70077 28411 60% 11: 79546 35986 55% 12: 68182 28411 58% 13: 81440 35986 55% 14: 81440 39774 51% 15: 94697 43562 54% 16: 79546 37881 52% Signed-off-by: Christophe Leroy --- arch/powerpc/include/asm/string.h | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/arch/powerpc/include/asm/string.h b/arch/powerpc/include/asm/string.h index cf6f495134c3..196ac5d587fb 100644 --- a/arch/powerpc/include/asm/string.h +++ b/arch/powerpc/include/asm/string.h @@ -4,6 +4,8 @@ #ifdef __KERNEL__ +#include + #define __HAVE_ARCH_STRNCPY #define __HAVE_ARCH_STRNCMP #define __HAVE_ARCH_MEMSET @@ -28,10 +30,45 @@ extern void * memchr(const void *,int,__kernel_size_t); extern void * memcpy_flushcache(void *,const void *,__kernel_size_t); #ifndef CONFIG_FORTIFY_SOURCE +static inline int ___memcmp(const void *p,const void *q,__kernel_size_t size, int offset) +{ + int dif; + + BUILD_BUG_ON(!size || size > 8); + + p += offset, q += offset; + if (size == 1) + return *(u8*)p - *(u8*)q; + if (size == 2) + return be16_to_cpu(*(u16*)p) - be16_to_cpu(*(u16*)q); + if (size == 3) { + dif = be16_to_cpu(*(u16*)p) - be16_to_cpu(*(u16*)q); + if (dif) + return dif; + return *(u8*)(p + 2) - *(u8*)(q + 2); + } + if (size == 8) { + s64 tmp = be64_to_cpu(*(u64*)p) - be64_to_cpu(*(u64*)q); + return tmp >> 32 ? : (int)tmp; + } + + dif = be32_to_cpu(*(u32*)p) - be32_to_cpu(*(u32*)q); + if (size == 4 || dif) + return dif; + + return ___memcmp(p, q, size - 4, 4); +} + static inline int __memcmp(const void *p,const void *q,__kernel_size_t size) { if (unlikely(!size)) return 0; + if (__builtin_constant_p(size) && size <= 16) { + int dif = ___memcmp(p, q, size < 8 ? size : 8, 0); + if (size <= 8 || dif) + return dif; + return ___memcmp(p, q, size - 8, 8); + } return memcmp(p, q, size); } #define memcmp __memcmp