From patchwork Wed Aug 18 14:19:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 1518048 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=RhfoGzWZ; dkim-atps=neutral Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GqVSZ0mnvz9s1l for ; Thu, 19 Aug 2021 00:21:18 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BF3773986407 for ; Wed, 18 Aug 2021 14:21:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BF3773986407 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1629296475; bh=jcM+C6JZQ1A3uo9l1MlqaP46WTLcRHVMqwYvbBGqhLU=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=RhfoGzWZqfIUID2A5wB37lXTokex/6LzoehpV9/7VaBBMSSlVWmLYGy76Hd0sW8+G guGcDOYOlXfz1lLAYzCryg0x/sTbXfk/qR88aZuTLFgqd5cGgnpcIx9N/O+woS1JKm XkwNSa639as5GstElGSIPaWua3dot7EjhJqTGsGc= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by sourceware.org (Postfix) with ESMTPS id 3DE083986430 for ; Wed, 18 Aug 2021 14:20:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3DE083986430 Received: by mail-pf1-x434.google.com with SMTP id y11so2283514pfl.13 for ; Wed, 18 Aug 2021 07:20:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jcM+C6JZQ1A3uo9l1MlqaP46WTLcRHVMqwYvbBGqhLU=; b=IGIcNskkstBe6WfVfZZYFxauG3nnehxk5/n6crtu4kF3GDm7hTe69FJZyvRGqWLast sqeTCngxMcFmZAlJgKwI2Zx4RKdT2RsMQ2p9Upmijefnz7V6SwfSrHSyfr79FgSFqSWS a/VboHkXiWCZ5Z7xe7/pzMpFqtWiosSbZ6J5iezoYYaL1y0WOul7k3MhBf/9QTRb7tFZ 23pcDjnyvhub6oD+yTfHi6X6wniHIgSqYyskP/9BwkTUT6ZdMd1MgsCeslqmiSx4M9Y1 BKNTzirWOUXBTI8Axu8QhsR9PZEee7YRVmaZ5IUJqkMGyryXkzepRT22c+WmLlbMjyv0 vXrQ== X-Gm-Message-State: AOAM533nP9jf7lr0hQC+wiYDEEYflsBFH0Z/7vAFwNc/O8H5wrqp8RwN plX1eIk4ifXcrD+awykwTZLH2wKLynt7AQ== X-Google-Smtp-Source: ABdhPJzitzoOgboIaNHJhlG3s7KdM4HTe1QQ8E6Xb6VFFV9u6+Z+kKUJDPJSwydFzWFtZXOorAR3Tg== X-Received: by 2002:aa7:9056:0:b0:3e3:332:c8b7 with SMTP id n22-20020aa79056000000b003e30332c8b7mr1593898pfo.69.1629296407955; Wed, 18 Aug 2021 07:20:07 -0700 (PDT) Received: from birita.. ([2804:431:c7ca:cd83:8c0a:d250:6dae:d807]) by smtp.gmail.com with ESMTPSA id c133sm6805015pfb.39.2021.08.18.07.20.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Aug 2021 07:20:07 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v2 2/4] malloc: Add THP/madvise support for sbrk Date: Wed, 18 Aug 2021 11:19:58 -0300 Message-Id: <20210818142000.128752-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210818142000.128752-1-adhemerval.zanella@linaro.org> References: <20210818142000.128752-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin , Siddhesh Poyarekar Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() might the preferable syscall instead of mmap(). And the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled with the new tunable 'glibc.malloc.thp_pagesize'. Checked on x86_64-linux-gnu. --- include/libc-pointer-arith.h | 10 ++++++++++ malloc/malloc.c | 35 ++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/include/libc-pointer-arith.h b/include/libc-pointer-arith.h index 04ba537617..f592cbafec 100644 --- a/include/libc-pointer-arith.h +++ b/include/libc-pointer-arith.h @@ -37,6 +37,16 @@ /* Cast an integer or a pointer VAL to integer with proper type. */ # define cast_to_integer(val) ((__integer_if_pointer_type (val)) (val)) +/* Check if SIZE is aligned on SIZE */ +#define IS_ALIGNED(base, size) \ + (((base) & (size - 1)) == 0) + +#define PTR_IS_ALIGNED(base, size) \ + ((((uintptr_t) (base)) & (size - 1)) == 0) + +#define PTR_DIFF(p1, p2) \ + ((ptrdiff_t)((uintptr_t)(p1) - (uintptr_t)(p2))) + /* Cast an integer VAL to void * pointer. */ # define cast_to_pointer(val) ((void *) (uintptr_t) (val)) diff --git a/malloc/malloc.c b/malloc/malloc.c index ad3eec41ac..1a2c798a35 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2024,6 +2024,17 @@ sysmadvise_thp (void *p, INTERNAL_SIZE_T size) not active. */ if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) return; + + /* madvise() requires at least the input to be aligned to system page and + MADV_HUGEPAGE should handle unaligned address. Also unaligned inputs + should happen only for the initial data segment. */ + if (__glibc_unlikely (!PTR_IS_ALIGNED (p, GLRO (dl_pagesize)))) + { + void *q = PTR_ALIGN_DOWN (p, GLRO (dl_pagesize)); + size += PTR_DIFF (p, q); + p = q; + } + __madvise (p, size, MADV_HUGEPAGE); #endif } @@ -2610,14 +2621,25 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t) __curbrk + size, + mp_.thp_pagesize); + size = top - (uintptr_t) __curbrk; + } + else +#endif + size = ALIGN_UP (size, GLRO(dl_pagesize)); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2922,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2931,12 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + extra = ALIGN_DOWN (top_area - pad, mp_.thp_pagesize); + else +#endif + extra = ALIGN_DOWN (top_area - pad, GLRO(dl_pagesize)); if (extra == 0) return 0;