From patchwork Sat Mar 3 12:48:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Malcolm X-Patchwork-Id: 881024 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-474243-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="v86PspYq"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3ztm6W5sc0z9s8h for ; Sat, 3 Mar 2018 23:42:56 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id; q=dns; s=default; b=XUlxGRdDZoij uQGZ2TdX9+Dhnx6gB9b/ju3WjbnrNsO40an0Ksj0hTyrIXcSrMwHibkZ1VV/7H8I Z2DMkwqFhKAINoeiLUB15PAJMisuLIe4JBqGjJlm67xZU/4RDwkwPltW4e8r2zZv zcCejTU1MFCUuQygHHzgqJorcA7Ci60= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id; s=default; bh=89egA3BDAqQOGYMvce uLSJLYAlk=; b=v86PspYqWTxcNC6t8SpHKWRBls2QQSB5di7Pr1y5V6oO5VmYlM WvMTW0HOiNdsz7tlVLDG2yWFEqQQTS6F43lUzQaYmrz4F3G7UG7wn46Kmijt+p4B DaUAtNwBPT+m1/kSLNy1XcpchXn+BYpKc7qbhSe4ss09U6J/ddtHZ+Fl4= Received: (qmail 117483 invoked by alias); 3 Mar 2018 12:42:49 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 117461 invoked by uid 89); 3 Mar 2018 12:42:49 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-24.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, LOTS_OF_MONEY, T_RP_MATCHES_RCVD autolearn=ham version=3.3.2 spammy=39996, ____ X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sat, 03 Mar 2018 12:42:47 +0000 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 781F68B100 for ; Sat, 3 Mar 2018 12:42:46 +0000 (UTC) Received: from c64.redhat.com (ovpn-112-12.phx2.redhat.com [10.3.112.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id B93C360933; Sat, 3 Mar 2018 12:42:45 +0000 (UTC) From: David Malcolm To: gcc-patches@gcc.gnu.org Cc: David Malcolm Subject: [PATCH] libiberty: fix memory usage explosion for invalid templates (PR demangler/84668) Date: Sat, 3 Mar 2018 07:48:54 -0500 Message-Id: <1520081334-5959-1-git-send-email-dmalcolm@redhat.com> X-IsSubscribed: yes PR demangler/84668 reports this failure of c++filt (found by fuzzing): $ c++filt '______H5z5555555555_____H5z55555555555555555555555' c++filt: out of memory allocating 18446744071696285694 bytes after a total of 135168 bytes internal_cplus_demangle handles the "H5" as a template with 5 arguments; the "z5555555555" is handled as a template parameter length of 5555555555, though this is truncated to 32-bits to 1260588259: (gdb) p /x 5555555555 $19 = 0x14b230ce3 (gdb) p /x r $18 = 0x4b230ce3 (gdb) p r $17 = 1260588259 demangle_template_template_parm repeatedly calls do_type for each of these 1.2 billion template params, and each call attempts to handle the "_", but hits this within demangle_fund_type: 3996 /* Now pick off the fundamental type. There can be only one. */ 3997 3998 switch (**mangled) 3999 { 4000 case '\0': 4001 case '_': 4002 break; and thus returns true for success. It does this without consuming any of the input string. At each iteration, it appends ", ", leading to the construction of a string of the form: "____