From patchwork Wed Jul 30 11:37:04 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 374799 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id C64381400D7 for ; Wed, 30 Jul 2014 21:37:16 +1000 (EST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :mime-version:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; q=dns; s= default; b=tXU7TzqrFAcHnYARA6FdDnp7vKeW4T3mvaVUkn5QQ9DesY9w7pDOC t1/c0bGZcG0iRrRJgITNo1weftmrRtS6FbVCBI85dPUAcsMQWreLusWatwPTV1Ew WLmEXDlq5HzTuCrbC02tTJwnDtr/RmMsMSZWoMKPyIyk9S2raKG55E= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :mime-version:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; s=default; bh=AB3H5A9S4EU44SJKwvqNSnRt+S4=; b=v6ayMbE+MKXVcyLrr1uKhQkJoPBl 4FIsCxUk2M3BRIPVXD2ftJ8HTljZcXLkBQrqRpv3j01ZosG03MC00LSnC5CnSa65 t1H2625VB0B+7lR1r6EJrIvz195TXK2oONpfvCXLT9CXcICvfhrjGRLoTHBtD5PE gY77YGkZloxYnI0= Received: (qmail 31604 invoked by alias); 30 Jul 2014 11:37:09 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 31591 invoked by uid 89); 30 Jul 2014 11:37:08 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.8 required=5.0 tests=AWL, BAYES_00, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wg0-f50.google.com Received: from mail-wg0-f50.google.com (HELO mail-wg0-f50.google.com) (74.125.82.50) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Wed, 30 Jul 2014 11:37:07 +0000 Received: by mail-wg0-f50.google.com with SMTP id n12so1036443wgh.33 for ; Wed, 30 Jul 2014 04:37:04 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.194.80.7 with SMTP id n7mr5453871wjx.8.1406720224253; Wed, 30 Jul 2014 04:37:04 -0700 (PDT) Received: by 10.195.13.76 with HTTP; Wed, 30 Jul 2014 04:37:04 -0700 (PDT) In-Reply-To: <53D8D3A7.6040204@suse.cz> References: <20140730055113.GE22904@x4> <53D8D3A7.6040204@suse.cz> Date: Wed, 30 Jul 2014 13:37:04 +0200 Message-ID: Subject: Re: [PATCH] LTO streamer reorg - try to reduce WPA memory use From: Richard Biener To: =?UTF-8?Q?Martin_Li=C5=A1ka?= Cc: GCC Patches X-IsSubscribed: yes On Wed, Jul 30, 2014 at 1:14 PM, Martin Liška wrote: > > On 07/30/2014 11:41 AM, Richard Biener wrote: >> >> On Wed, 30 Jul 2014, Richard Biener wrote: >> >>> On Wed, Jul 30, 2014 at 7:51 AM, Markus Trippelsdorf >>> wrote: >>>> >>>> On 2014.07.29 at 15:10 +0200, Richard Biener wrote: >>>>> >>>>> On Tue, 29 Jul 2014, Richard Biener wrote: >>>>> >>>>>> This re-organizes the LTO streamer to do compression transparently >>>>>> in the data-streamer routines (and disables section compression >>>>>> by defaulting to -flto-compression-level=0). This avoids >>>>>> keeping the whole uncompressed sections in memory, only retaining >>>>>> the compressed ones. >>>>>> >>>>>> The downside is that we lose compression of at least the string >>>>>> parts (they are abusing the streaming interface quite awkwardly >>>>>> and doing random-accesses with offsets into the uncompressed >>>>>> section). With a little bit of surgery we can get that back I >>>>>> think (but we'd have to keep the uncompressed piece in memory >>>>>> somewhere which means losing the memory use advantage). >>>>>> >>>>>> Very lightly tested sofar (running lto.exp). I'll try a LTO >>>>>> bootstrap now. >>>>>> >>>>>> I wonder what the change is on WPA memory use for larger >>>>>> projects and what the effect on object file size is. >>>>> >>>>> Updated patch passing LTO bootstrap (one warning fix) and >>>>> with a memory leak fixed. >>>> >>>> Testing with Firefox is impossible at the moment because of PR61885. >>>> One thing I've noticed (before the ICE) is that virtual memory usage is >>>> very high: >>>> >>>> Address Kbytes RSS Dirty Mode Mapping >>>> 0000000000400000 16344 9084 0 r-x-- lto1 >>>> 00000000013f6000 36 36 28 rw--- lto1 >>>> 00000000013ff000 1072 276 276 rw--- [ anon ] >>>> 00000000034aa000 10154940 1540384 1540384 rw--- [ anon ] >>>> 00002acf04af2000 136 136 0 r-x-- ld-2.19.90.so >>>> 00002acf04b14000 88 88 88 rw--- [ anon ] >>>> ... >>>> ---------------- ------- ------- ------- >>>> total kB 12022060 3388396 3377708 >>> >>> Maybe there is still a memleak (just checked that LTOing int main() {} >>> doesn't leak). >> >> Found it: >> >> Index: gcc/lto-section-in.c >> =================================================================== >> --- gcc/lto-section-in.c.orig 2014-07-30 12:40:27.950225826 +0200 >> +++ gcc/lto-section-in.c 2014-07-30 12:37:44.179237102 +0200 >> @@ -249,7 +249,7 @@ lto_destroy_simple_input_block (struct l >> struct lto_input_block *ib, >> const char *data, size_t len) >> { >> - free (ib); >> + delete ib; >> lto_free_section_data (file_data, section_type, NULL, data, len); >> } >> Richard. > > Hello, > there's memory/CPU usage for the patch. for both, I used sync and > drop_caches. > > Url: > https://drive.google.com/file/d/0B0pisUJ80pO1andOX19JMHV3LVE/edit?usp=sharing Ok, it turns out setting -flto-compression-level to 0 doesn't really short-circuit zlib for sections. So the following does that the hard but effective way. does that change anything? Thanks, Richard. > Martin > Index: gcc/lto-section-out.c =================================================================== --- gcc/lto-section-out.c.orig 2014-07-30 13:33:06.634008355 +0200 +++ gcc/lto-section-out.c 2014-07-30 13:29:19.468023995 +0200 @@ -80,7 +80,7 @@ lto_begin_section (const char *name, boo data is anything other than assembler output. The effect here is that we get compression of IL only in non-ltrans object files. */ gcc_assert (compression_stream == NULL); - if (compress) + if (compress && 0) compression_stream = lto_start_compression (lto_append_data, NULL); } Index: gcc/lto-section-in.c =================================================================== --- gcc/lto-section-in.c.orig 2014-07-30 13:33:06.637008355 +0200 +++ gcc/lto-section-in.c 2014-07-30 13:31:57.329013126 +0200 @@ -153,7 +153,7 @@ lto_get_section_data (struct lto_file_de /* FIXME lto: WPA mode does not write compressed sections, so for now suppress uncompression if flag_ltrans. */ - if (!flag_ltrans) + if (!flag_ltrans && 0) { /* Create a mapping header containing the underlying data and length, and prepend this to the uncompression buffer. The uncompressed data @@ -200,7 +200,7 @@ lto_free_section_data (struct lto_file_d /* FIXME lto: WPA mode does not write compressed sections, so for now suppress uncompression mapping if flag_ltrans. */ - if (flag_ltrans) + if (flag_ltrans || 1) { (free_section_f) (file_data, section_type, name, data, len); return;