From patchwork Fri May 9 05:01:17 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Lance Taylor X-Patchwork-Id: 347294 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 14FBF1400F5 for ; Fri, 9 May 2014 15:01:31 +1000 (EST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; q=dns; s= default; b=j8s1qz+UyZhHj7nq6NYKVEAHUrbQ9Rpn/ic4M6crGMpwoq7dU7kdJ mxeC5rJvhGLuDdwq0qyNqLa7hFGxp9DaPcrpTwNMPpLugFyRo9EGjYns7Ibyt91V L2Fh0c+8zUPI5/ZhhSzbJtAjH+Gg+0Hk3OzRlybuITExCKGrfkhWAQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; s= default; bh=9PuL2Gf6HxxYjFpIUTO70YK01Fw=; b=Psoq6PfsYAP4xIMvYSaK lT1MtU53M6hKrZumrS4G6giaP8H+40a9jEu70OpKcyBAwkUHo00OFobmE8zMelaU LKF/gR7sOQ8qZbLweD2ySUYx9zEMWawBBQ+NWQ2NnOgARhbXQK6qWrcZsk+7oGIR DLun/OHMTh77MjuZ6ij2uTI= Received: (qmail 23577 invoked by alias); 9 May 2014 05:01:25 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 23566 invoked by uid 89); 9 May 2014 05:01:24 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.0 required=5.0 tests=AWL, BAYES_00, RCVD_IN_DNSWL_NONE, RP_MATCHES_RCVD, SPF_PASS, T_TVD_MIME_NO_HEADERS autolearn=ham version=3.3.2 X-HELO: mail-pd0-f174.google.com Received: from mail-pd0-f174.google.com (HELO mail-pd0-f174.google.com) (209.85.192.174) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Fri, 09 May 2014 05:01:22 +0000 Received: by mail-pd0-f174.google.com with SMTP id w10so3176186pde.19 for ; Thu, 08 May 2014 22:01:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :mime-version:content-type; bh=VOHC0FMhuV+rYW6nTTRjXfw7xi0cD/tfeEuO7Xl9Gs8=; b=G2dSeRTW1DUcsqxgJ5eAH8KCkrdmDyp00oO/xdHymz/TES/e6v+APbCfmPr4s+l41q c9XrIwVGHB+7VAuKDTcruLUCIJ9vl9fz25gpPRU6zZ9jHzF6Mz4JboM4iFPcv2zB9chp OEG1y6KgisEzI65JsmhUk14pJY2eNwAm1ZX5dOSohdxyTMRNxssnp4fC9rHxAhX7gsPe cicSCrEe2IAKZR7CpabECztqOFBM/aKs+mkcKmG0GAUdYcEMPm9p7d8ktQFl+wA3bPHp /pEDFAUgRad6cps576skTOtCfU0E2lRMOrLNSWvbcjr/a9RgBWxgTQ2ticPdlpOr8K+r pjbA== X-Gm-Message-State: ALoCoQmflGv/VBWmT4TgeCN4dpfYvV71oHl8Ac9c//cXcUCGxDV841tXnmJltK8g7pDaaGaobM3/WnnGxF13a4lbTgcaBtXiDugonQmCgbO2M02jraQm95Lkj1XM0Qqz79XJgLa5e2Uj X-Received: by 10.66.66.225 with SMTP id i1mr2922515pat.0.1399611679410; Thu, 08 May 2014 22:01:19 -0700 (PDT) Received: from iant-glaptop.roam.corp.google.com.google.com (adsl-71-133-8-30.dsl.pltn13.pacbell.net. [71.133.8.30]) by mx.google.com with ESMTPSA id xc1sm2506370pab.39.2014.05.08.22.01.18 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Thu, 08 May 2014 22:01:18 -0700 (PDT) From: Ian Lance Taylor To: gcc-patches@gcc.gnu.org Subject: libbacktrace patch committed: Fixes for large binaries Date: Thu, 08 May 2014 22:01:17 -0700 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux) MIME-Version: 1.0 X-IsSubscribed: yes While testing on a large Google binary, I noticed that libbacktrace is allocating an inordinate amount of memory. The binary winds up with 377,944 entries in the unit_addrs vector. Each entry is 24 bytes, so this is 9,070,656 bytes, which is not too terrible. Unfortunately, for some reason I thought that when a libbacktrace vector is larger than a page the code should only allocate one additional page at a time. This vector requires 2215 4096-byte pages. Growing the vector one page at a time allocates a total of something like (2215 * 2214) / 2 pages, which turns out to be nearly 1.5G. Allocating 1.5G to represent a vector of size 9M is not desirable. It's true that when the vector grows, the old memory can be reused. But there is nothing in libbacktrace that is going to reuse that much memory. And even worse, there was a bug in the vector_grow routine that caused it to fail to correctly report the size of the old vector, so the memory had no chance of being reused anyhow. This patch fixes vector growth to double the number of pages requested each time. It fixes vector growth to record the correct size of the old vector being freed. The patch also adds some code to simply munmap large blocks of allocated memory. It's unlikely in practice that libbacktrace will ever be able to reuse a large block, so it's probably better to hand the memory back rather than hold onto it for no purpose. Bootstrapped and tested on x86_64-unknown-linux-gnu. Committed to 4.9 branch and mainline. Ian 2014-05-08 Ian Lance Taylor * mmap.c (backtrace_free): If freeing a large aligned block of memory, call munmap rather than holding onto it. (backtrace_vector_grow): When growing a vector, double the number of pages requested. When releasing the old version of a grown vector, pass the correct size to backtrace_free. Index: ChangeLog =================================================================== --- ChangeLog (revision 210248) +++ ChangeLog (working copy) @@ -1,3 +1,11 @@ +2014-05-08 Ian Lance Taylor + + * mmap.c (backtrace_free): If freeing a large aligned block of + memory, call munmap rather than holding onto it. + (backtrace_vector_grow): When growing a vector, double the number + of pages requested. When releasing the old version of a grown + vector, pass the correct size to backtrace_free. + 2014-03-07 Ian Lance Taylor * sort.c (backtrace_qsort): Use middle element as pivot. Index: mmap.c =================================================================== --- mmap.c (revision 210248) +++ mmap.c (working copy) @@ -164,6 +164,26 @@ backtrace_free (struct backtrace_state * { int locked; + /* If we are freeing a large aligned block, just release it back to + the system. This case arises when growing a vector for a large + binary with lots of debug info. Calling munmap here may cause us + to call mmap again if there is also a large shared library; we + just live with that. */ + if (size >= 16 * 4096) + { + size_t pagesize; + + pagesize = getpagesize (); + if (((uintptr_t) addr & (pagesize - 1)) == 0 + && (size & (pagesize - 1)) == 0) + { + /* If munmap fails for some reason, just add the block to + the freelist. */ + if (munmap (addr, size) == 0) + return; + } + } + /* If we can acquire the lock, add the new space to the free list. If we can't acquire the lock, just leak the memory. __sync_lock_test_and_set returns the old state of the lock, so we @@ -209,14 +229,18 @@ backtrace_vector_grow (struct backtrace_ alc = pagesize; } else - alc = (alc + pagesize - 1) & ~ (pagesize - 1); + { + alc *= 2; + alc = (alc + pagesize - 1) & ~ (pagesize - 1); + } base = backtrace_alloc (state, alc, error_callback, data); if (base == NULL) return NULL; if (vec->base != NULL) { memcpy (base, vec->base, vec->size); - backtrace_free (state, vec->base, vec->alc, error_callback, data); + backtrace_free (state, vec->base, vec->size + vec->alc, + error_callback, data); } vec->base = base; vec->alc = alc - vec->size;