From patchwork Thu Aug 24 09:48:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 805398 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ozlabs-ru.20150623.gappssmtp.com header.i=@ozlabs-ru.20150623.gappssmtp.com header.b="RRy+ctc4"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xdKK84l48z9sR9 for ; Thu, 24 Aug 2017 19:50:02 +1000 (AEST) Received: from localhost ([::1]:47478 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkoly-0006eO-Oj for incoming@patchwork.ozlabs.org; Thu, 24 Aug 2017 05:49:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50656) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkolF-0006e9-7n for qemu-devel@nongnu.org; Thu, 24 Aug 2017 05:49:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkol9-0002gJ-AO for qemu-devel@nongnu.org; Thu, 24 Aug 2017 05:49:13 -0400 Received: from mail-it0-x235.google.com ([2607:f8b0:4001:c0b::235]:35305) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dkol8-0002g1-OX for qemu-devel@nongnu.org; Thu, 24 Aug 2017 05:49:07 -0400 Received: by mail-it0-x235.google.com with SMTP id f135so631585ita.0 for ; Thu, 24 Aug 2017 02:49:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to; bh=1EEiN+LF3QmJpmFBvMDVysUmANiRJQCIVpqULZZc8kw=; b=RRy+ctc465xPPrmqTJc9SBlVuREVVFJjZCQZDKL4Ys25FcJVovvLP49G3xeywvo2N4 haAyn6dBLmqUwmlIkKrOm1ftFs735imO39LWXADQ/0YEvu1EPUfu8yJY/wdGDtuFKvAz kbHtCslMgt26pGEod15cNc8B9SPIrbsUdxcafL/3dzMNOA9KG0cWXmNaj4H2Ncav+aZu mYWfgw9laFj+TY9z8QkVJ2ydS0e53sFTXvYYdXxQLCOLkLtdqk7wtP3yN0YvsUCFwcpg 8Mvhszs7h3dtfouJBLfbkzHP7S1wNJ8KYDFOdLchFWtYwFVLl5GQfmxB6c9Lx39j7kdZ 9q/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to; bh=1EEiN+LF3QmJpmFBvMDVysUmANiRJQCIVpqULZZc8kw=; b=iC4EUqMjq9J2qLwIEbATp/GRlxEmF60+wSUR+pW5xSnkDgFEllkZf1OXxQ2mSbeWyV nv4jNziyAk7PETkvrB0pyZVdYEa8tvIE4HXhUCB0xx163SIRhEZnQaL4Bimqhd4InVNo wT7JtO7uKIYt629s7mNKWBi+tTwDb308/AiRxASbqJQyiF8a2Jr8JX6JCRgtVS8G06bL qYsbA5tfA6QSUO6JwbqKDZK7A9y04Iy2gI12EH1Xggr/q8hICscPDxhkgnlmUgu1JPbX 3vbXkPFlr23AlXLElAyqo8N5gMGnDhPt+7zf25yCmplOl9njJl6e4dsspmfUvdrLH9Kr g7tA== X-Gm-Message-State: AHYfb5jx9GJWPTTL3jeDMNYzFbhsRM1BtLW6RBON7RFRaCVIEcEl62Mv 4Xc/jT2fQs3122t/ X-Received: by 10.36.107.3 with SMTP id v3mr6393186itc.137.1503568144119; Thu, 24 Aug 2017 02:49:04 -0700 (PDT) Received: from [10.61.2.175] ([122.99.82.10]) by smtp.googlemail.com with ESMTPSA id 86sm1737762ios.68.2017.08.24.02.49.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Aug 2017 02:49:03 -0700 (PDT) From: Alexey Kardashevskiy To: David Gibson , Stefan Hajnoczi References: <283a2a27-7fb5-5fa7-4921-60ba91f13666@ozlabs.ru> <20170818131853.GB17654@stefanha-x1.localdomain> <20170821043125.GH12356@umbus.fritz.box> <6e2ae379-9cff-9615-c19f-747867328cb5@ozlabs.ru> Message-ID: Date: Thu, 24 Aug 2017 19:48:57 +1000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <6e2ae379-9cff-9615-c19f-747867328cb5@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4001:c0b::235 X-Content-Filtered-By: Mailman/MimeDel 2.1.21 Subject: Re: [Qemu-devel] Memory use with >100 virtio devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gerd Hoffmann , "qemu-devel@nongnu.org" , "Michael S. Tsirkin" Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" On 21/08/17 15:50, Alexey Kardashevskiy wrote: > On 21/08/17 14:31, David Gibson wrote: >> On Fri, Aug 18, 2017 at 02:18:53PM +0100, Stefan Hajnoczi wrote: >>> On Fri, Aug 18, 2017 at 03:39:20PM +1000, Alexey Kardashevskiy wrote: >>>> ==94451== -------------------- 4 of 10 -------------------- >>>> ==94451== max-live: 314,649,600 in 150 blocks >>>> ==94451== tot-alloc: 314,649,600 in 150 blocks (avg size 2097664.00) >>>> ==94451== deaths: none (none of these blocks were freed) >>>> ==94451== acc-ratios: 0.00 rd, 0.00 wr (0 b-read, 0 b-written) >>>> ==94451== at 0x4895600: memalign (in >>>> /usr/lib/valgrind/vgpreload_exp-dhat-ppc64le-linux.so) >>>> ==94451== by 0x48957E7: posix_memalign (in >>>> /usr/lib/valgrind/vgpreload_exp-dhat-ppc64le-linux.so) >>>> ==94451== by 0xB744AB: qemu_try_memalign (oslib-posix.c:106) >>>> ==94451== by 0xA92053: qemu_try_blockalign (io.c:2493) >>>> ==94451== by 0xA34DDF: qcow2_do_open (qcow2.c:1365) >>>> ==94451== by 0xA35627: qcow2_open (qcow2.c:1526) >>>> ==94451== by 0x9FB94F: bdrv_open_driver (block.c:1109) >>>> ==94451== by 0x9FC413: bdrv_open_common (block.c:1365) >>>> ==94451== by 0x9FF823: bdrv_open_inherit (block.c:2542) >>>> ==94451== by 0x9FFC17: bdrv_open (block.c:2626) >>>> ==94451== by 0xA71027: blk_new_open (block-backend.c:267) >>>> ==94451== by 0x6D3E6B: blockdev_init (blockdev.c:588) >>> >>> This allocation is unnecessary. Most qcow2 files are not encrypted so >>> s->cluster_data does not need to be allocated upfront. >>> >>> I'll send a patch. >> >> Is that sufficient to explain the problem, I can't quickly see how big >> that unnecessary allocation is - but would it account for the 10s of >> gigabytes usage we're seeing here? >> >> I'm suspecting we accidentally have a O(n^2) or worse space complexity >> going on here. >> > > No, it is a small fraction only. See "[PATCH] qcow2: allocate > cluster_cache/cluster_data on demand" thread for more details. The information was lost there so I'll continue in this thread. I run QEMU again, with 2GB of RAM, -initrd+-kernel, pseries, 64 PCI bridges, -S, no KVM, some virtio-block devices; I run it under "valgrind --tool=exp-dhat" and exited via "c-a x" as soon as possible. The summary of each run is: 50 virtio-block devices: guest_insns: 2,728,740,444 max_live: 1,214,121,770 in 226,958 blocks tot_alloc: 1,384,726,690 in 310,930 blocks 150 virtio-block devices: guest_insns: 17,576,279,582 max_live: 7,454,182,031 in 1,286,128 blocks tot_alloc: 7,958,747,994 in 1,469,719 blocks 250 virtio-block devices: guest_insns: 46,100,928,249 max_live: 19,423,868,479 in 3,264,833 blocks tot_alloc: 20,262,409,839 in 3,548,220 blocks 350 virtio-block devices: guest_insns: 88,046,403,555 max_live: 36,994,652,991 in 6,140,203 blocks tot_alloc: 38,167,153,779 in 6,523,206 blocks Memory usage 1) grows a lot 2) grows out of proportion 3) QEMU becomes incredibly slow. With the hack (below) and 350 virtio-block devices, the summary is: guest_insns: 7,873,805,573 max_live: 2,577,738,019 in 2,567,682 blocks tot_alloc: 3,750,238,807 in 2,950,685 blocks insns per allocated byte: 2 I am also attaching 2 snapshots from the valgrind's "massif" tool, with and without the hack. Ideas what to tweak or what valgrind tool to try? The hack is basically excluding virtio-pci-cfg-as from the address_spaces list (yeah, it breaks QEMU, this is just a hint): address_space_init_dispatch(as); diff --git a/memory.c b/memory.c index 02c95d1..118ac7f 100644 --- a/memory.c +++ b/memory.c @@ -2589,6 +2589,7 @@ void address_space_init(AddressSpace *as, MemoryRegion *root, const char *name) as->ioeventfd_nb = 0; as->ioeventfds = NULL; QTAILQ_INIT(&as->listeners); + if (strcmp(name, "virtio-pci-cfg-as")) QTAILQ_INSERT_TAIL(&address_spaces, as, address_spaces_link); as->name = g_strdup(name ? name : "anonymous");