From patchwork Fri Jun 19 10:56:58 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre DERUMIER X-Patchwork-Id: 486671 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id A755B140271 for ; Fri, 19 Jun 2015 21:37:13 +1000 (AEST) Received: from localhost ([::1]:57496 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5ubd-0005vV-F0 for incoming@patchwork.ozlabs.org; Fri, 19 Jun 2015 07:37:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50170) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5ubG-0005I4-2r for qemu-devel@nongnu.org; Fri, 19 Jun 2015 07:36:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z5ub8-0003Qk-QF for qemu-devel@nongnu.org; Fri, 19 Jun 2015 07:36:44 -0400 Received: from mailpro.odiso.net ([89.248.209.98]:52865) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5ub8-0003QP-HK for qemu-devel@nongnu.org; Fri, 19 Jun 2015 07:36:38 -0400 Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id C81D644DC099A; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id sC5ayE6PyEZE; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 8ADC944D80D1B; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ojQuylMU_3Tv; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) Received: from kvmtest1.odiso.net (unknown [10.3.94.31]) by mailpro.odiso.net (Postfix) with ESMTP id 5C5A8442D92D8; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) Received: by kvmtest1.odiso.net (Postfix, from userid 0) id 41F0B30610E; Fri, 19 Jun 2015 12:58:24 +0200 (CEST) From: Alexandre Derumier To: qemu-devel@nongnu.org Date: Fri, 19 Jun 2015 12:56:58 +0200 Message-Id: <1434711418-20429-1-git-send-email-aderumier@odiso.com> X-Mailer: git-send-email 2.1.4 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 89.248.209.98 Cc: Alexandre Derumier Subject: [Qemu-devel] [PATCH] configure: Add support for jemalloc X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking to jemalloc memory allocator. We have already tcmalloc support, but it seem to not working well with a lot of iothreads/disks. The main problem is that tcmalloc use a shared thread cache of 16MB by default. With more threads, this cache is shared, and some bad garbage collections can occur if the cache is too low. It's possible to tcmalloc cache increase it with a env var: TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB With default 16MB, performances are really bad with more than 2 disks. Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads. Jemalloc don't have performance problem with default configuration. Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32, with rbd block backend (librbd is doing a lot of memory allocation), 1 iothread by disk glibc malloc ------------ 1 disk 29052 2 disks 55878 4 disks 127899 8 disks 240566 15 disks 269976 jemalloc -------- 1 disk 41278 2 disks 75781 4 disks 195351 8 disks 294241 15 disks 298199 tcmalloc 2.2.1 default 16M cache -------------------------------- 1 disk 37911 2 disks 67698 4 disks 41076 8 disks 43312 15 disks 37569 tcmalloc : 256M cache --------------------------- 1 disk 33914 2 disks 58839 4 disks 148205 8 disks 213298 15 disks 218383 Signed-off-by: Alexandre Derumier Reviewed-by: Fam Zheng --- configure | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/configure b/configure index 222694f..2fe1e05 100755 --- a/configure +++ b/configure @@ -336,6 +336,7 @@ vhdx="" quorum="" numa="" tcmalloc="no" +jemalloc="no" # parse CC options first for opt do @@ -1147,6 +1148,10 @@ for opt do ;; --enable-tcmalloc) tcmalloc="yes" ;; + --disable-jemalloc) jemalloc="no" + ;; + --enable-jemalloc) jemalloc="yes" + ;; *) echo "ERROR: unknown option $opt" echo "Try '$0 --help' for more information" @@ -1420,6 +1425,8 @@ Advanced options (experts only): --enable-numa enable libnuma support --disable-tcmalloc disable tcmalloc support --enable-tcmalloc enable tcmalloc support + --disable-jemalloc disable jemalloc support + --enable-jemalloc enable jemalloc support NOTE: The object files are built at the place where configure is launched EOF @@ -3344,6 +3351,11 @@ EOF fi fi +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then + echo "ERROR: tcmalloc && jemalloc can't be used at the same time" + exit 1 +fi + ########################################## # tcmalloc probe @@ -3361,6 +3373,22 @@ EOF fi ########################################## +# jemalloc probe + +if test "$jemalloc" = "yes" ; then + cat > $TMPC << EOF +#include +int main(void) { malloc(1); return 0; } +EOF + + if compile_prog "" "-ljemalloc" ; then + LIBS="-ljemalloc $LIBS" + else + feature_not_found "jemalloc" "install jemalloc devel" + fi +fi + +########################################## # signalfd probe signalfd="no" cat > $TMPC << EOF @@ -4499,6 +4527,7 @@ echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "NUMA host support $numa" echo "tcmalloc support $tcmalloc" +echo "jemalloc support $jemalloc" if test "$sdl_too_old" = "yes"; then echo "-> Your SDL version is too old - please upgrade to have SDL support"