From patchwork Tue Sep 4 16:43:17 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andi Kleen X-Patchwork-Id: 181626 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) by ozlabs.org (Postfix) with SMTP id AA9C82C0098 for ; Wed, 5 Sep 2012 02:43:55 +1000 (EST) Comment: DKIM? See http://www.dkim.org DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=gcc.gnu.org; s=default; x=1347381836; h=Comment: DomainKey-Signature:Received:Received:Received:Received:Received: From:To:Cc:Subject:Date:Message-Id:Mailing-List:Precedence: List-Id:List-Unsubscribe:List-Archive:List-Post:List-Help:Sender: Delivered-To; bh=YRQdQlWGQsDrAyni4BP/GUjnmmI=; b=m0Y9Z+NCQ90x9HA GQcBCOqNTr1TZgU/0l8IybEyhmlBt/iC5Jk3RbXp/KhlfYatyslk3vf1fyjIeuT6 3lKWspYL6EQjPeRd2oop03rWA/8mqBVulgPI5FIXtUmSvNVOMRfnX52HBaHS9yPm QzDJEY2Vq0FT36oxHn0Utjt2uxeI= Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=gcc.gnu.org; h=Received:Received:X-SWARE-Spam-Status:X-Spam-Check-By:Received:Received:Received:From:To:Cc:Subject:Date:Message-Id:Mailing-List:Precedence:List-Id:List-Unsubscribe:List-Archive:List-Post:List-Help:Sender:Delivered-To; b=dPyRzRLvFdQzWk1az15LeSXwhXIeY9fE+Mz64VLCVA5LJpGVKTQocoZ7XSv7OM XdLns1hOaDHnWvO4Km625raUCKkR705AtZJh8tQyHR7W+40nmbVs/Iij1hJepwnv RqhijlEA45VHJ6wSTLIcfP8kAdy1AYC7rY/39yJdA9om8=; Received: (qmail 5521 invoked by alias); 4 Sep 2012 16:43:42 -0000 Received: (qmail 5511 invoked by uid 22791); 4 Sep 2012 16:43:40 -0000 X-SWARE-Spam-Status: No, hits=-2.1 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from one.firstfloor.org (HELO one.firstfloor.org) (213.235.205.2) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Tue, 04 Sep 2012 16:43:22 +0000 Received: from basil.firstfloor.org (71-222-31-204.ptld.qwest.net [71.222.31.204]) by one.firstfloor.org (Postfix) with ESMTP id 003121ED8024; Tue, 4 Sep 2012 18:43:21 +0200 (CEST) Received: by basil.firstfloor.org (Postfix, from userid 1000) id 06894B1506; Tue, 4 Sep 2012 09:43:18 -0700 (PDT) From: Andi Kleen To: gcc-patches@gcc.gnu.org Cc: Andi Kleen Subject: [PATCH] Reduce memory usage for storing LTO decl resolutions Date: Tue, 4 Sep 2012 09:43:17 -0700 Message-Id: <1346776997-8726-1-git-send-email-andi@firstfloor.org> Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org From: Andi Kleen With a LTO build of a large project (>11k subfiles incrementially linked) storing the LTO resolutions took over 0.5GB memory: lto/lto.c:1087 (lto_resolution_read) 0: 0.0% 540398500 15903: 0.0% The reason is that the declaration indexes are quite sparse, but every subfile got a full continuous vector for them. Since there are so many of them the many vectors add up. This patch instead stores the resolutions initially in a compact (index, resolution) format. This is only expanded into a sparse vector for fast lookup when the subfile is actually read, but then immediately freed. This means only one vector is allocated at a time. This brings the overhead for this down to less than 3MB for the test case: lto/lto.c:1087 (lto_resolution_read) 0: 0.0% 2821456 42186: 0.0% Passed bootstrap and test suite on x86_64-linux. Ok for 4.8 and possibly for 4.7? -Andi 2012-09-04 Andi Kleen * gcc/lto-streamer.h (res_pair): Add. (lto_file_decl_data): Replace resolutions with respairs. Add max_index. * gcc/lto/lto.c (lto_resolution_read): Remove max_index. Add rp. Initialize respairs. (lto_file_finalize): Set up resolutions vector lazily from respairs. --- gcc/lto-streamer.h | 15 ++++++++++++++- gcc/lto/lto.c | 31 ++++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/gcc/lto-streamer.h b/gcc/lto-streamer.h index bed408a..19a35cb 100644 --- a/gcc/lto-streamer.h +++ b/gcc/lto-streamer.h @@ -513,6 +513,18 @@ typedef struct lto_out_decl_state *lto_out_decl_state_ptr; DEF_VEC_P(lto_out_decl_state_ptr); DEF_VEC_ALLOC_P(lto_out_decl_state_ptr, heap); +/* Compact representation of a index <-> resolution pair. Unpacked to an + vector later. */ +struct res_pair +{ + ld_plugin_symbol_resolution_t res; + unsigned index; +}; +typedef struct res_pair res_pair; + +DEF_VEC_P(res_pair); +DEF_VEC_ALLOC_P(res_pair, heap); + /* One of these is allocated for each object file that being compiled by lto. This structure contains the tables that are needed by the serialized functions and ipa passes to connect themselves to the @@ -548,7 +560,8 @@ struct GTY(()) lto_file_decl_data unsigned HOST_WIDE_INT id; /* Symbol resolutions for this file */ - VEC(ld_plugin_symbol_resolution_t,heap) * GTY((skip)) resolutions; + VEC(res_pair, heap) * GTY((skip)) respairs; + unsigned max_index; struct gcov_ctr_summary GTY((skip)) profile_info; }; diff --git a/gcc/lto/lto.c b/gcc/lto/lto.c index bd91c39..5da5412 100644 --- a/gcc/lto/lto.c +++ b/gcc/lto/lto.c @@ -1012,7 +1012,6 @@ lto_resolution_read (splay_tree file_ids, FILE *resolution, lto_file *file) unsigned int num_symbols; unsigned int i; struct lto_file_decl_data *file_data; - unsigned max_index = 0; splay_tree_node nd = NULL; if (!resolution) @@ -1054,13 +1053,12 @@ lto_resolution_read (splay_tree file_ids, FILE *resolution, lto_file *file) unsigned int j; unsigned int lto_resolution_str_len = sizeof (lto_resolution_str) / sizeof (char *); + res_pair rp; t = fscanf (resolution, "%u " HOST_WIDE_INT_PRINT_HEX_PURE " %26s %*[^\n]\n", &index, &id, r_str); if (t != 3) internal_error ("invalid line in the resolution file"); - if (index > max_index) - max_index = index; for (j = 0; j < lto_resolution_str_len; j++) { @@ -1082,11 +1080,13 @@ lto_resolution_read (splay_tree file_ids, FILE *resolution, lto_file *file) } file_data = (struct lto_file_decl_data *)nd->value; - VEC_safe_grow_cleared (ld_plugin_symbol_resolution_t, heap, - file_data->resolutions, - max_index + 1); - VEC_replace (ld_plugin_symbol_resolution_t, - file_data->resolutions, index, r); + /* The indexes are very sparse. To save memory save them in a compact + format that is only unpacked later when the subfile is processed. */ + rp.res = r; + rp.index = index; + VEC_safe_push (res_pair, heap, file_data->respairs, rp); + if (file_data->max_index < index) + file_data->max_index = index; } } @@ -1166,6 +1166,18 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file) { const char *data; size_t len; + VEC(ld_plugin_symbol_resolution_t,heap) *resolutions = NULL; + int i; + res_pair *rp; + + /* Create vector for fast access of resolution. We do this lazily + to save memory. */ + VEC_safe_grow_cleared (ld_plugin_symbol_resolution_t, heap, + resolutions, + file_data->max_index + 1); + for (i = 0; VEC_iterate (res_pair, file_data->respairs, i, rp); i++) + VEC_replace (ld_plugin_symbol_resolution_t, resolutions, rp->index, rp->res); + VEC_free (res_pair, heap, file_data->respairs); file_data->renaming_hash_table = lto_create_renaming_table (); file_data->file_name = file->filename; @@ -1175,7 +1187,8 @@ lto_file_finalize (struct lto_file_decl_data *file_data, lto_file *file) internal_error ("cannot read LTO decls from %s", file_data->file_name); return; } - lto_read_decls (file_data, data, file_data->resolutions); + /* Frees resolutions */ + lto_read_decls (file_data, data, resolutions); lto_free_section_data (file_data, LTO_section_decls, NULL, data, len); }