From patchwork Tue Aug 7 11:31:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 954478 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-483300-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="H3AJKfP1"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41lC641T1Zz9ryt for ; Tue, 7 Aug 2018 21:31:55 +1000 (AEST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; q=dns; s= default; b=FM0SLbuhgNVBVr73SEtLznxgRPW9igSpwx45MSY6hFE2tEE05tEak LH9OKwGTIha4PalfYKDZy6wDK4kPS0S2NXbzO12B6h+RtkUNq59dqKdCe3sgtBOx rxC7WyrKaGbhOxyYJc9JA2OqYhOQPBptiYn9iB2CEwhNEX5PO7JE8k= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; s= default; bh=197H/2mm6FzY5JoTw4aA0xrgnao=; b=H3AJKfP1JOMvVMEL0R8p zu6cCkDAxG3NE7NFJaTR+qUckrOPx/5xiRMaf+xoIOBPorUGjO3dTrvfxABNnExk XV3/wb9Rf/ZGshdWty3pKpNyCFgs7YePFPvYo154P2Mkd2wdk6/2pfZ82DICvr25 WgGxVLZJo3hSLgO2rNwHzeY= Received: (qmail 61822 invoked by alias); 7 Aug 2018 11:31:40 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 61788 invoked by uid 89); 7 Aug 2018 11:31:39 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.0 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=worrying, Leave X-HELO: mx1.redhat.com Received: from mx3-rdu2.redhat.com (HELO mx1.redhat.com) (66.187.233.73) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 07 Aug 2018 11:31:35 +0000 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2E2EA4023336; Tue, 7 Aug 2018 11:31:34 +0000 (UTC) Received: from localhost (unknown [10.33.36.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id E1441215670D; Tue, 7 Aug 2018 11:31:33 +0000 (UTC) Date: Tue, 7 Aug 2018 12:31:33 +0100 From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH] Define monotonic_buffer_resource members out-of-line Message-ID: <20180807113133.GA2878@redhat.com> MIME-Version: 1.0 Content-Disposition: inline X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.9.2 (2017-12-15) Move the allocation logic into libstdc++.so so that it can be changed without worrying about inlined code in existing binaries. Leave do_allocate inline so that calls to it can be devirtualized, and only the slow path needs to call into the library. * config/abi/pre/gnu.ver: Export monotonic_buffer_resource members. * include/std/memory_resource (monotonic_buffer_resource::release): Call _M_release_buffers to free buffers. (monotonic_buffer_resource::do_allocate): Call _M_new_buffer to allocate a new buffer from upstream. (monotonic_buffer_resource::_M_new_buffer): Declare. (monotonic_buffer_resource::_M_release_buffers): Declare. (monotonic_buffer_resource::_Chunk): Replace definition with declaration as opaque type. * src/c++17/memory_resource.cc (monotonic_buffer_resource::_Chunk): Define. (monotonic_buffer_resource::_M_new_buffer): Define. (monotonic_buffer_resource::_M_release_buffers): Define. Tested powerpc64le-linux, committed to trunk. commit 736856a34ec11bdf4a4019eccead58b5c6a6b0cc Author: Jonathan Wakely Date: Mon Aug 6 23:50:54 2018 +0100 Define monotonic_buffer_resource members out-of-line Move the allocation logic into libstdc++.so so that it can be changed without worrying about inlined code in existing binaries. Leave do_allocate inline so that calls to it can be devirtualized, and only the slow path needs to call into the library. * config/abi/pre/gnu.ver: Export monotonic_buffer_resource members. * include/std/memory_resource (monotonic_buffer_resource::release): Call _M_release_buffers to free buffers. (monotonic_buffer_resource::do_allocate): Call _M_new_buffer to allocate a new buffer from upstream. (monotonic_buffer_resource::_M_new_buffer): Declare. (monotonic_buffer_resource::_M_release_buffers): Declare. (monotonic_buffer_resource::_Chunk): Replace definition with declaration as opaque type. * src/c++17/memory_resource.cc (monotonic_buffer_resource::_Chunk): Define. (monotonic_buffer_resource::_M_new_buffer): Define. (monotonic_buffer_resource::_M_release_buffers): Define. diff --git a/libstdc++-v3/config/abi/pre/gnu.ver b/libstdc++-v3/config/abi/pre/gnu.ver index 36459e88b6a..593783da1aa 100644 --- a/libstdc++-v3/config/abi/pre/gnu.ver +++ b/libstdc++-v3/config/abi/pre/gnu.ver @@ -2043,6 +2043,8 @@ GLIBCXX_3.4.26 { _ZNSt3pmr20null_memory_resourceEv; _ZNSt3pmr20get_default_resourceEv; _ZNSt3pmr20set_default_resourceEPNS_15memory_resourceE; + _ZNSt3pmr25monotonic_buffer_resource13_M_new_bufferE[jmy][jmy]; + _ZNSt3pmr25monotonic_buffer_resource18_M_release_buffersEv; } GLIBCXX_3.4.25; diff --git a/libstdc++-v3/include/std/memory_resource b/libstdc++-v3/include/std/memory_resource index b3f8f7d9477..bb4e31551e6 100644 --- a/libstdc++-v3/include/std/memory_resource +++ b/libstdc++-v3/include/std/memory_resource @@ -365,7 +365,8 @@ namespace pmr void release() noexcept { - _Chunk::release(_M_head, _M_upstream); + if (_M_head) + _M_release_buffers(); // reset to initial state at contruction: if ((_M_current_buf = _M_orig_buf)) @@ -392,19 +393,14 @@ namespace pmr if (__bytes == 0) __bytes = 1; // Ensures we don't return the same pointer twice. - if (auto __p = std::align(__alignment, __bytes, _M_current_buf, _M_avail)) + void* __p = std::align(__alignment, __bytes, _M_current_buf, _M_avail); + if (!__p) { - _M_current_buf = (char*)_M_current_buf + __bytes; - _M_avail -= __bytes; - return __p; + _M_new_buffer(__bytes, __alignment); + __p = _M_current_buf; } - - const size_t __n = std::max(__bytes, _M_next_bufsiz); - const size_t __m = std::max(__alignment, alignof(std::max_align_t)); - auto [__p, __size] = _Chunk::allocate(_M_upstream, __n, __m, _M_head); - _M_current_buf = (char*)__p + __bytes; - _M_avail = __size - __bytes; - _M_next_bufsiz *= _S_growth_factor; + _M_current_buf = (char*)_M_current_buf + __bytes; + _M_avail -= __bytes; return __p; } @@ -417,6 +413,15 @@ namespace pmr { return this == &__other; } private: + // Update _M_current_buf and _M_avail to refer to a new buffer with + // at least the specified size and alignment, allocated from upstream. + void + _M_new_buffer(size_t __bytes, size_t __alignment); + + // Deallocate all buffers obtained from upstream. + void + _M_release_buffers() noexcept; + static size_t _S_next_bufsize(size_t __buffer_size) noexcept { @@ -437,68 +442,7 @@ namespace pmr void* const _M_orig_buf = nullptr; size_t const _M_orig_size = _M_next_bufsiz; - // Memory allocated by the upstream resource is managed in a linked list - // of _Chunk objects. A _Chunk object recording the size and alignment of - // the allocated block and a pointer to the previous chunk is placed - // at end of the block. - class _Chunk - { - public: - // Return the address and size of a block of memory allocated from __r, - // of at least __size bytes and aligned to __align. - // Add a new _Chunk to the front of the linked list at __head. - static pair - allocate(memory_resource* __r, size_t __size, size_t __align, - _Chunk*& __head) - { - __size = std::__ceil2(__size + sizeof(_Chunk)); - void* __p = __r->allocate(__size, __align); - // Add a chunk defined by (__p, __size, __align) to linked list __head. - void* const __back = (char*)__p + __size - sizeof(_Chunk); - __head = ::new(__back) _Chunk(__size, __align, __head); - return { __p, __size - sizeof(_Chunk) }; - } - - // Return every chunk in linked list __head to resource __r. - static void - release(_Chunk*& __head, memory_resource* __r) noexcept - { - _Chunk* __next = __head; - __head = nullptr; - while (__next) - { - _Chunk* __ch = __next; - __builtin_memcpy(&__next, __ch->_M_next, sizeof(_Chunk*)); - - __glibcxx_assert(__ch->_M_canary != 0); - __glibcxx_assert(__ch->_M_canary == (__ch->_M_size|__ch->_M_align)); - - if (__ch->_M_canary != (__ch->_M_size | __ch->_M_align)) - return; // buffer overflow detected! - - size_t __size = (1u << __ch->_M_size); - size_t __align = (1u << __ch->_M_align); - void* __start = (char*)(__ch + 1) - __size; - __r->deallocate(__start, __size, __align); - } - } - - private: - _Chunk(size_t __size, size_t __align, _Chunk* __next) noexcept - : _M_size(std::__log2p1(__size) - 1), - _M_align(std::__log2p1(__align) - 1) - { - __builtin_memcpy(_M_next, &__next, sizeof(__next)); - _M_canary = _M_size | _M_align; - } - - unsigned char _M_canary; - unsigned char _M_size; - unsigned char _M_align; - unsigned char _M_next[sizeof(_Chunk*)]; - }; - static_assert(alignof(_Chunk) == 1); - + class _Chunk; _Chunk* _M_head = nullptr; }; diff --git a/libstdc++-v3/src/c++17/memory_resource.cc b/libstdc++-v3/src/c++17/memory_resource.cc index dd418c1b1aa..c3ae2b69f71 100644 --- a/libstdc++-v3/src/c++17/memory_resource.cc +++ b/libstdc++-v3/src/c++17/memory_resource.cc @@ -104,6 +104,89 @@ namespace pmr get_default_resource() noexcept { return default_res.obj.load(); } + // Member functions for std::pmr::monotonic_buffer_resource + + // Memory allocated by the upstream resource is managed in a linked list + // of _Chunk objects. A _Chunk object recording the size and alignment of + // the allocated block and a pointer to the previous chunk is placed + // at end of the block. + class monotonic_buffer_resource::_Chunk + { + public: + // Return the address and size of a block of memory allocated from __r, + // of at least __size bytes and aligned to __align. + // Add a new _Chunk to the front of the linked list at __head. + static pair + allocate(memory_resource* __r, size_t __size, size_t __align, + _Chunk*& __head) + { + __size = std::__ceil2(__size + sizeof(_Chunk)); + void* __p = __r->allocate(__size, __align); + // Add a chunk defined by (__p, __size, __align) to linked list __head. + void* const __back = (char*)__p + __size - sizeof(_Chunk); + __head = ::new(__back) _Chunk(__size, __align, __head); + return { __p, __size - sizeof(_Chunk) }; + } + + // Return every chunk in linked list __head to resource __r. + static void + release(_Chunk*& __head, memory_resource* __r) noexcept + { + _Chunk* __next = __head; + __head = nullptr; + while (__next) + { + _Chunk* __ch = __next; + __builtin_memcpy(&__next, __ch->_M_next, sizeof(_Chunk*)); + + __glibcxx_assert(__ch->_M_canary != 0); + __glibcxx_assert(__ch->_M_canary == (__ch->_M_size|__ch->_M_align)); + + if (__ch->_M_canary != (__ch->_M_size | __ch->_M_align)) + return; // buffer overflow detected! + + size_t __size = (1u << __ch->_M_size); + size_t __align = (1u << __ch->_M_align); + void* __start = (char*)(__ch + 1) - __size; + __r->deallocate(__start, __size, __align); + } + } + + private: + _Chunk(size_t __size, size_t __align, _Chunk* __next) noexcept + : _M_size(std::__log2p1(__size) - 1), + _M_align(std::__log2p1(__align) - 1) + { + __builtin_memcpy(_M_next, &__next, sizeof(__next)); + _M_canary = _M_size | _M_align; + } + + unsigned char _M_canary; + unsigned char _M_size; + unsigned char _M_align; + unsigned char _M_next[sizeof(_Chunk*)]; + }; + + void + monotonic_buffer_resource::_M_new_buffer(size_t bytes, size_t alignment) + { + // Need to check this somewhere, so put it here: + static_assert(alignof(monotonic_buffer_resource::_Chunk) == 1); + + const size_t n = std::max(bytes, _M_next_bufsiz); + const size_t m = std::max(alignment, alignof(std::max_align_t)); + auto [p, size] = _Chunk::allocate(_M_upstream, n, m, _M_head); + _M_current_buf = p; + _M_avail = size; + _M_next_bufsiz *= _S_growth_factor; + } + + void + monotonic_buffer_resource::_M_release_buffers() noexcept + { + _Chunk::release(_M_head, _M_upstream); + } + } // namespace pmr _GLIBCXX_END_NAMESPACE_VERSION } // namespace std