From patchwork Fri Aug 28 10:40:54 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353162
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=ULG82S0L;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGPf0tt5z9sTC
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:42:13 +1000 (AEST)
Received: from localhost ([::1]:52858 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbpj-0003Nf-QR
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:42:11 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34384)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpB-0003MZ-Ex
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400
Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:42827)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbp8-0005fA-Ql
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400
Received: by mail-wr1-x442.google.com with SMTP id c18so833999wrm.9
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=;
 b=ULG82S0L3kCib5VT4GaouqmZivM+bay/mzr4qks3za+D7dtBqR++RqUjn5NKpkuVLE
 /3WjJcnTTdwSFiaGo8qwVLUJahcQ7YdGddB3xqtaldRO0TWnk8RyT7Cau2kciF5uc7V7
 Nk/GzfRznwhn6IjcEL5I8C3h1O59R/HBQrVnu7SUJ3+UrmTDzW1TF7XsuaY1bR65VcNx
 sZ3AFsVXCOcMZP1jtQhLb8qc/1cQPTORJ/AxlxH/uybNPNQO95i10kyWpJ8XPe87hLJA
 AQLF6MfAcC6rwAgym1SY74AMz2zNBol60n2pAN+xDpkD9mAz2NmUuitE7eVfTahJz+cX
 4BPw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=;
 b=edhvkGgaW9FNDljG3BIyWtLg3vd0nXEB+/VdohZOpFWeWNs3yQU+HeWPI6aFOXHqip
 w9iA0fpMObgbfEg/q9ms4kU1Td1b/+f5x4Ujjxb4tbeSLEfHVuBXTCrmeu+dzRBVRdOd
 KTaTcdGN+h854DLMezFJJJE+lqYJyY7gep7kYzCj5BzIqTS9cqSIIVPBBcFB1hC+3U4d
 7RX1jMrvlxe7E5Qh1aZIcQ2ntgdnxUDhsbZhfIezgHAVJt9rbhqI/58mbqcy9rwhmJLr
 FXKvpSlFTuI4+ereHs7Wv3rBgAizzUm+s/azfzQAc6n7LkmlymlVZCZkfCZ4KHaN585s
 GQig==
X-Gm-Message-State: AOAM532cer+BEXV1Y4xxukMb6m+oXF80a4vYTa7iq27DCeVULpPw8SUI
 B7Snuz/ED3TY/vhWtxrk80YWy3tKpaP0UA==
X-Google-Smtp-Source: 
 ABdhPJy+UxZw0jQnURExU+I0JXJh37Lpqu5CD/VUPQVnG0uk7ICLiSSbKv/D/w+rB/FxGK0ERvMoDA==
X-Received: by 2002:adf:a1d6:: with SMTP id v22mr954931wrv.185.1598611292937;
 Fri, 28 Aug 2020 03:41:32 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.30
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:32 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
Date: Fri, 28 Aug 2020 12:40:54 +0200
Message-Id: <20200828104102.4490-2-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::442;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
---
 scripts/performance/topN_perf.py | 174 +++++++++++++++----------------
 1 file changed, 87 insertions(+), 87 deletions(-)

diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_perf.py
index 07be195fc8..56b100da87 100755
--- a/scripts/performance/topN_perf.py
+++ b/scripts/performance/topN_perf.py
@@ -1,72 +1,77 @@
 #!/usr/bin/env python3
 
-#  Print the top N most executed functions in QEMU using perf.
-#  Syntax:
-#  topN_perf.py [-h] [-n] <number of displayed top functions>  -- \
-#           <qemu executable> [<qemu executable options>] \
-#           <target executable> [<target execurable options>]
-#
-#  [-h] - Print the script arguments help message.
-#  [-n] - Specify the number of top functions to print.
-#       - If this flag is not specified, the tool defaults to 25.
-#
-#  Example of usage:
-#  topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
-#
-#  This file is a part of the project "TCG Continuous Benchmarking".
-#
-#  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-#  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 2 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the top N most executed functions in QEMU using perf.
+
+Syntax:
+topN_perf.py [-h] [-n <number of displayed top functions>] -- \
+         <qemu executable> [<qemu executable options>] \
+         <target executable> [<target execurable options>]
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+     - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
 
 import argparse
 import os
 import subprocess
 import sys
+import tempfile
 
 
 # Parse the command line arguments
-parser = argparse.ArgumentParser(
-    usage='topN_perf.py [-h] [-n] <number of displayed top functions >  -- '
+PARSER = argparse.ArgumentParser(
+    usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- '
           '<qemu executable> [<qemu executable options>] '
           '<target executable> [<target executable options>]')
 
-parser.add_argument('-n', dest='top', type=int, default=25,
+PARSER.add_argument('-n', dest='top', type=int, default=25,
                     help='Specify the number of top functions to print.')
 
-parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
+PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
 
-args = parser.parse_args()
+ARGS = PARSER.parse_args()
 
 # Extract the needed variables from the args
-command = args.command
-top = args.top
+COMMAND = ARGS.command
+TOP = ARGS.top
 
 # Insure that perf is installed
-check_perf_presence = subprocess.run(["which", "perf"],
-                                     stdout=subprocess.DEVNULL)
-if check_perf_presence.returncode:
+CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
+                                     stdout=subprocess.DEVNULL,
+                                     check=False)
+if CHECK_PERF_PRESENCE.returncode:
     sys.exit("Please install perf before running the script!")
 
 # Insure user has previllage to run perf
-check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
+CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
                                           stdout=subprocess.DEVNULL,
-                                          stderr=subprocess.DEVNULL)
-if check_perf_executability.returncode:
-    sys.exit(
-"""
+                                          stderr=subprocess.DEVNULL,
+                                          check=False)
+if CHECK_PERF_EXECUTABILITY.returncode:
+    sys.exit("""
 Error:
 You may not have permission to collect stats.
 
@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
    kernel.perf_event_paranoid = -1
 
 * Alternatively, you can run this script under sudo privileges.
-"""
-)
-
-# Run perf record
-perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] +
-                              command),
-                             stdout=subprocess.DEVNULL,
-                             stderr=subprocess.PIPE)
-if perf_record.returncode:
-    os.unlink('/tmp/perf.data')
-    sys.exit(perf_record.stderr.decode("utf-8"))
-
-# Save perf report output to /tmp/perf_report.out
-with open("/tmp/perf_report.out", "w") as output:
-    perf_report = subprocess.run(
-        ["perf", "report", "--input=/tmp/perf.data", "--stdio"],
-        stdout=output,
-        stderr=subprocess.PIPE)
-    if perf_report.returncode:
-        os.unlink('/tmp/perf.data')
-        output.close()
-        os.unlink('/tmp/perf_report.out')
-        sys.exit(perf_report.stderr.decode("utf-8"))
-
-# Read the reported data to functions[]
-functions = []
-with open("/tmp/perf_report.out", "r") as data:
-    # Only read lines that are not comments (comments start with #)
-    # Only read lines that are not empty
-    functions = [line for line in data.readlines() if line and line[0]
-                 != '#' and line[0] != "\n"]
-
-# Limit the number of top functions to "top"
-number_of_top_functions = top if len(functions) > top else len(functions)
-
-# Store the data of the top functions in top_functions[]
-top_functions = functions[:number_of_top_functions]
+""")
+
+# Run perf and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+    RECORD_PATH = os.path.join(tmpdir, "record.data")
+    REPORT_PATH = os.path.join(tmpdir, "report.txt")
+
+    PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] +
+                                  COMMAND),
+                                 stdout=subprocess.DEVNULL,
+                                 stderr=subprocess.PIPE,
+                                 check=False)
+    if PERF_RECORD.returncode:
+        sys.exit(PERF_RECORD.stderr.decode("utf-8"))
+
+    with open(REPORT_PATH, "w") as output:
+        PERF_REPORT = subprocess.run(
+            ["perf", "report", "--input="+RECORD_PATH, "--stdio"],
+            stdout=output,
+            stderr=subprocess.PIPE,
+            check=False)
+        if PERF_REPORT.returncode:
+            sys.exit(PERF_REPORT.stderr.decode("utf-8"))
+
+    # Save the reported data to FUNCTIONS[]
+    with open(REPORT_PATH, "r") as data:
+        # Only read lines that are not comments (comments start with #)
+        # Only read lines that are not empty
+        FUNCTIONS = [line for line in data.readlines() if line and
+                     line[0] != '#' and line[0] != "\n"]
+
+# Limit the number of top functions to "TOP"
+NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
+
+# Store the data of the top functions in TOP_FUNCTIONS[]
+TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
 
 # Print table header
 print('{:>4}  {:>10}  {:<30}  {}\n{}  {}  {}  {}'.format('No.',
@@ -134,7 +138,7 @@ print('{:>4}  {:>10}  {:<30}  {}\n{}  {}  {}  {}'.format('No.',
                                                          '-' * 25))
 
 # Print top N functions
-for (index, function) in enumerate(top_functions, start=1):
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
     function_data = function.split()
     function_percentage = function_data[0]
     function_name = function_data[-1]
@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1):
                                              function_percentage,
                                              function_name,
                                              function_invoker))
-
-# Remove intermediate files
-os.unlink('/tmp/perf.data')
-os.unlink('/tmp/perf_report.out')

From patchwork Fri Aug 28 10:40:55 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353167
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=HK/+cklW;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGRn5ZM2z9sSn
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:44:05 +1000 (AEST)
Received: from localhost ([::1]:32792 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbrX-0006iv-Oq
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:44:03 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34414)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpD-0003Pe-Ia
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400
Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:45770)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpA-0005gS-RI
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400
Received: by mail-wr1-x443.google.com with SMTP id h15so830654wrt.12
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=;
 b=HK/+cklW0orMXNl4Is5lsX4XkmBhx776gGavdwJf8IBs89t4vZJzXPAaCtxc6yKIvc
 RzQB1Qus6S9Bn6Mzp/bHtBKVuU99Vcl4/E54OUhOshOmMFSRImyfLeEu3yKgjaZNuKZr
 Q/R91V2p8iDPXFPeLzOrwTJzcHoK+Rwdtb5Izj+iATpzclxbu7oMxWbdaKr5AbmjWolH
 WQEPE/DkQ7BvxFwG2cz0hL0+6qnx3otbnGYt7t0mxUeUiEXYIxZeDOGK0qs9WxCdaBSo
 ZNaz31UqFtszJ4YvYS+YHjq3NuI1ipWTa24bHx8MYl6pALyPRYiokQMv3shLjDT4F9bH
 NkiQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=;
 b=NriiN+nz4yZkfD74HZulD9tIomLmngDWDA8gazf8+8/WJHHvFU5zcT5gGhzY5GBOQQ
 E+WlbiNH+y2SDEIjCdO7wq/KE26t8X/PZaUepCvMWKRVxyxifYmXvJrfQMn89RuDPkIJ
 sYzFCFV+adVzh8kg4R26iZWSpisLfzwYbeoiBsUvP3DdYwcYKzywaia/4Tu+yLTK7a+/
 11BbMJiuu9+c8iIjVEX4LUBEhEg8bL/fSEfxWtOJtIVqfld2ausPS8moJRsmx93PLtNC
 n7liJJuATdjssXl9WRFwM+zup86OT3AXSAmeKCOe59p1sI5+ACWWMErExjyzkKAKXZRY
 iTdA==
X-Gm-Message-State: AOAM533QXLo3R0Uz/AxpcfakWt4+/QsIVEYutCI00pdz4sFd8rW4P2Rl
 rZ3xit9u9SaskK8XOYJn6gdLKIUiLPu4/A==
X-Google-Smtp-Source: 
 ABdhPJzJsAjybpcBhmfxWDs+uG3yuDlTJWs5Nnd98OPlyvZdKNUSixe42FeYCDweCVGzHmXJt3aKPA==
X-Received: by 2002:adf:9d44:: with SMTP id o4mr864841wre.99.1598611294899;
 Fri, 28 Aug 2020 03:41:34 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.33
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:34 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py
Date: Fri, 28 Aug 2020 12:40:55 +0200
Message-Id: <20200828104102.4490-3-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::443;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x443.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

- Apply pylint and flake8 formatting rules to the script.
    - Use 'tempfile' instead of '/tmp' for creating temporary files.

    Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
---
 scripts/performance/topN_callgrind.py | 169 +++++++++++++-------------
 1 file changed, 87 insertions(+), 82 deletions(-)

diff --git a/scripts/performance/topN_callgrind.py b/scripts/performance/topN_callgrind.py
index 67c59197af..f8a554f393 100755
--- a/scripts/performance/topN_callgrind.py
+++ b/scripts/performance/topN_callgrind.py
@@ -1,113 +1,122 @@
 #!/usr/bin/env python3
 
-#  Print the top N most executed functions in QEMU using callgrind.
-#  Syntax:
-#  topN_callgrind.py [-h] [-n] <number of displayed top functions>  -- \
-#           <qemu executable> [<qemu executable options>] \
-#           <target executable> [<target execurable options>]
-#
-#  [-h] - Print the script arguments help message.
-#  [-n] - Specify the number of top functions to print.
-#       - If this flag is not specified, the tool defaults to 25.
-#
-#  Example of usage:
-#  topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
-#
-#  This file is a part of the project "TCG Continuous Benchmarking".
-#
-#  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-#  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 2 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the top N most executed functions in QEMU using callgrind.
+
+Syntax:
+topN_callgrind.py [-h] [-n <number of displayed top functions>] -- \
+         <qemu executable> [<qemu executable options>] \
+         <target executable> [<target execurable options>]
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+     - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
 
 import argparse
 import os
 import subprocess
 import sys
+import tempfile
 
 
 # Parse the command line arguments
-parser = argparse.ArgumentParser(
-    usage='topN_callgrind.py [-h] [-n] <number of displayed top functions>  -- '
+PARSER = argparse.ArgumentParser(
+    usage='topN_callgrind.py [-h] [-n] <number of displayed top functions> -- '
           '<qemu executable> [<qemu executable options>] '
           '<target executable> [<target executable options>]')
 
-parser.add_argument('-n', dest='top', type=int, default=25,
+PARSER.add_argument('-n', dest='top', type=int, default=25,
                     help='Specify the number of top functions to print.')
 
-parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
+PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
 
-args = parser.parse_args()
+ARGS = PARSER.parse_args()
 
 # Extract the needed variables from the args
-command = args.command
-top = args.top
+COMMAND = ARGS.command
+TOP = ARGS.top
 
 # Insure that valgrind is installed
-check_valgrind_presence = subprocess.run(["which", "valgrind"],
-                                         stdout=subprocess.DEVNULL)
-if check_valgrind_presence.returncode:
+CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"],
+                                         stdout=subprocess.DEVNULL,
+                                         check=False)
+if CHECK_VALGRIND_PRESENCE.returncode:
     sys.exit("Please install valgrind before running the script!")
 
-# Run callgrind
-callgrind = subprocess.run((
-    ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/callgrind.data"]
-    + command),
-    stdout=subprocess.DEVNULL,
-    stderr=subprocess.PIPE)
-if callgrind.returncode:
-    sys.exit(callgrind.stderr.decode("utf-8"))
-
-# Save callgrind_annotate output to /tmp/callgrind_annotate.out
-with open("/tmp/callgrind_annotate.out", "w") as output:
-    callgrind_annotate = subprocess.run(["callgrind_annotate",
-                                         "/tmp/callgrind.data"],
-                                        stdout=output,
-                                        stderr=subprocess.PIPE)
-    if callgrind_annotate.returncode:
-        os.unlink('/tmp/callgrind.data')
-        output.close()
-        os.unlink('/tmp/callgrind_annotate.out')
-        sys.exit(callgrind_annotate.stderr.decode("utf-8"))
-
-# Read the callgrind_annotate output to callgrind_data[]
-callgrind_data = []
-with open('/tmp/callgrind_annotate.out', 'r') as data:
-    callgrind_data = data.readlines()
+# Run callgrind and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+    CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data")
+    ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out")
+
+    # Run callgrind
+    CALLGRIND = subprocess.run((["valgrind",
+                                 "--tool=callgrind",
+                                 "--callgrind-out-file="+CALLGRIND_DATA_PATH]
+                                + COMMAND),
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.PIPE,
+                               check=False)
+    if CALLGRIND.returncode:
+        sys.exit(CALLGRIND.stderr.decode("utf-8"))
+
+    with open(ANNOTATE_OUT_PATH, "w") as output:
+        CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate",
+                                             CALLGRIND_DATA_PATH],
+                                            stdout=output,
+                                            stderr=subprocess.PIPE,
+                                            check=False)
+        if CALLGRIND_ANNOTATE.returncode:
+            sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8"))
+
+    # Read the callgrind_annotate output to CALLGRIND_DATA[]
+    CALLGRIND_DATA = []
+    with open(ANNOTATE_OUT_PATH, 'r') as data:
+        CALLGRIND_DATA = data.readlines()
 
 # Line number with the total number of instructions
-total_instructions_line_number = 20
+TOTAL_INSTRUCTIONS_LINE_NO = 20
 
 # Get the total number of instructions
-total_instructions_line_data = callgrind_data[total_instructions_line_number]
-total_number_of_instructions = total_instructions_line_data.split(' ')[0]
-total_number_of_instructions = int(
-    total_number_of_instructions.replace(',', ''))
+TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO]
+TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0]
+TOTAL_NUMBER_OF_INSTRUCTIONS = int(
+    TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', ''))
 
 # Line number with the top function
-first_func_line = 25
+FIRST_FUNC_LINE = 25
 
 # Number of functions recorded by callgrind, last two lines are always empty
-number_of_functions = len(callgrind_data) - first_func_line - 2
+NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2
 
 # Limit the number of top functions to "top"
-number_of_top_functions = (top if number_of_functions >
-                           top else number_of_functions)
+NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS >
+                           TOP else NUMBER_OF_FUNCTIONS)
 
 # Store the data of the top functions in top_functions[]
-top_functions = callgrind_data[first_func_line:
-                               first_func_line + number_of_top_functions]
+TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE:
+                               FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS]
 
 # Print table header
 print('{:>4}  {:>10}  {:<30}  {}\n{}  {}  {}  {}'.format('No.',
@@ -121,12 +130,12 @@ print('{:>4}  {:>10}  {:<30}  {}\n{}  {}  {}  {}'.format('No.',
                                                          ))
 
 # Print top N functions
-for (index, function) in enumerate(top_functions, start=1):
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
     function_data = function.split()
     # Calculate function percentage
     function_instructions = float(function_data[0].replace(',', ''))
     function_percentage = (function_instructions /
-                           total_number_of_instructions)*100
+                           TOTAL_NUMBER_OF_INSTRUCTIONS)*100
     # Get function name and source files path
     function_source_file, function_name = function_data[1].split(':')
     # Print extracted data
@@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions, start=1):
                                                 round(function_percentage, 3),
                                                 function_name,
                                                 function_source_file))
-
-# Remove intermediate files
-os.unlink('/tmp/callgrind.data')
-os.unlink('/tmp/callgrind_annotate.out')

From patchwork Fri Aug 28 10:40:56 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353164
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=O+XmnZCn;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGPj6FD1z9sPB
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:42:17 +1000 (AEST)
Received: from localhost ([::1]:53124 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbpn-0003Uq-QV
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:42:15 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34422)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpE-0003Re-M4
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:40 -0400
Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]:46399)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpC-0005gg-Fx
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:40 -0400
Received: by mail-wr1-x42a.google.com with SMTP id r15so828425wrp.13
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=EJjPWrg0i8aqjuW0NGnomyE8WzQ7HVuoMsi3T2ewGf0=;
 b=O+XmnZCn19gSQ9LhKKOi5ptnw7DICMn0z17F4uo1VVTkzXwb2ScJWwleT9vSnBgGuj
 wTh3Zc+jwzHhwmOcdb3U7pjhZdoamRmOZ/jSFUzcvbsk8eweunRayngHIL7rjoa+/eVK
 JPVI2q09eKw7LJI4qwfB3JT32NfIjKFoVj9IlbeoO2PpiA3p0v/DxmEgN49zvbDOjZ4/
 U82KeCfn4VYTbhe7HBaxx0E41dnyqI1Z8piJ0fVdLyuDy7IR0BtHs0XyS/LrAaAxyeYL
 ZIFudBPRpRM5aH7hyqB4Fg+pLfQUbmwhKdiQLNvIUOiPYI2hU/rRrIUvZLYejCck3JeX
 3EAQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=EJjPWrg0i8aqjuW0NGnomyE8WzQ7HVuoMsi3T2ewGf0=;
 b=hEyCFR39ji16AXv7G3xOMFY/PE6FDpY4nCkffrNxVPkdR3W1JtI5lgBougtjiWjkU5
 9zrFm02UubIOpHxxjZVSDZxSDneZ/rlyfHuiI1Tdv/ESKBEu5kcfTBk/QBd4ez9+D4eR
 gcjRAO5DTxsNgskFPXcgFRIsO+OPuqAAgm9I3zdSn4QqRxfBQra625WqRQQUbb7i7mB6
 5FN1Y9sp78xF19UVdfTzXZj+T0BaE/cP+m8pCM1QET3Vtb7eQ1FwM4ih4tpt29MwCbmT
 VaKjXajHzsQVTs8Kz6LKwJiB83qj1Akmf7T7E+MLjGBvHFAZeSi0jdQdA25Z0n1Z0Z4V
 J8hg==
X-Gm-Message-State: AOAM533VW2tu8hEUDUoLsrG35SbLapUUjYIKfvp2tXhnkPLyfsUtIh0g
 h/WfOSOk5W/mmec4o5yD5lGPjvsBqXvPBw==
X-Google-Smtp-Source: 
 ABdhPJwYHcBmyWUqISf1+MDWEgjeN49KGbx78XbWEICTBGoADTQXsb4aYsE666vllJTQwD7y/kKXvw==
X-Received: by 2002:adf:ea0f:: with SMTP id q15mr904644wrm.113.1598611296846;
 Fri, 28 Aug 2020 03:41:36 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.35
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:36 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 3/9] scripts/performance: Refactor dissect.py
Date: Fri, 28 Aug 2020 12:40:56 +0200
Message-Id: <20200828104102.4490-4-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::42a;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x42a.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

- Apply pylint and flake8 formatting rules to the script.
- Move syntax and usage exmaple to main() docstring.
- Update get_jit_line() to only detect the main jit call.
- Use mypy for specifying parameters and return types in functions.

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
---
 scripts/performance/dissect.py | 123 ++++++++++++++++++---------------
 1 file changed, 68 insertions(+), 55 deletions(-)

diff --git a/scripts/performance/dissect.py b/scripts/performance/dissect.py
index bf24f50922..d4df884b75 100755
--- a/scripts/performance/dissect.py
+++ b/scripts/performance/dissect.py
@@ -1,34 +1,27 @@
 #!/usr/bin/env python3
 
-#  Print the percentage of instructions spent in each phase of QEMU
-#  execution.
-#
-#  Syntax:
-#  dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
-#                   <target executable> [<target executable options>]
-#
-#  [-h] - Print the script arguments help message.
-#
-#  Example of usage:
-#  dissect.py -- qemu-arm coulomb_double-arm
-#
-#  This file is a part of the project "TCG Continuous Benchmarking".
-#
-#  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-#  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 2 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the percentage of instructions spent in each phase of QEMU
+execution.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
 
 import argparse
 import os
@@ -36,23 +29,26 @@ import subprocess
 import sys
 import tempfile
 
+from typing import List
+
 
-def get_JIT_line(callgrind_data):
+def get_jit_line(callgrind_data: List[str]) -> int:
     """
     Search for the first instance of the JIT call in
     the callgrind_annotate output when ran using --tree=caller
     This is equivalent to the self number of instructions of JIT.
 
     Parameters:
-    callgrind_data (list): callgrind_annotate output
+    callgrind_data (List[str]): callgrind_annotate output
 
     Returns:
     (int): Line number
     """
     line = -1
-    for i in range(len(callgrind_data)):
-        if callgrind_data[i].strip('\n') and \
-                callgrind_data[i].split()[-1] == "[???]":
+    for (i, callgrind_datum) in enumerate(callgrind_data):
+        if callgrind_datum.strip('\n') and \
+                callgrind_datum.split()[-1] == "[???]" and \
+                callgrind_datum.split()[1] == "*":
             line = i
             break
     if line == -1:
@@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data):
 
 
 def main():
+    """
+    Parse the command line arguments then start the execution.
+    Syntax:
+    dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
+                 <target executable> [<target executable options>]
+
+    [-h] - Print the script arguments help message.
+
+    Example of usage:
+    dissect.py -- qemu-arm coulomb_double-arm
+    """
+
     # Parse the command line arguments
     parser = argparse.ArgumentParser(
         usage='dissect.py [-h] -- '
@@ -76,7 +84,7 @@ def main():
 
     # Insure that valgrind is installed
     check_valgrind = subprocess.run(
-        ["which", "valgrind"], stdout=subprocess.DEVNULL)
+        ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
     if check_valgrind.returncode:
         sys.exit("Please install valgrind before running the script.")
 
@@ -93,7 +101,8 @@ def main():
                                      "--callgrind-out-file=" + data_path]
                                     + command),
                                    stdout=subprocess.DEVNULL,
-                                   stderr=subprocess.PIPE)
+                                   stderr=subprocess.PIPE,
+                                   check=False)
         if callgrind.returncode:
             sys.exit(callgrind.stderr.decode("utf-8"))
 
@@ -102,7 +111,8 @@ def main():
             callgrind_annotate = subprocess.run(
                 ["callgrind_annotate", data_path, "--tree=caller"],
                 stdout=output,
-                stderr=subprocess.PIPE)
+                stderr=subprocess.PIPE,
+                check=False)
             if callgrind_annotate.returncode:
                 sys.exit(callgrind_annotate.stderr.decode("utf-8"))
 
@@ -120,25 +130,28 @@ def main():
         total_instructions = int(total_instructions.replace(',', ''))
 
         # Line number with the JIT self number of instructions
-        JIT_self_instructions_line_number = get_JIT_line(callgrind_data)
+        jit_self_instructions_line_number = get_jit_line(callgrind_data)
         # Get the JIT self number of instructions
-        JIT_self_instructions_line_data = \
-            callgrind_data[JIT_self_instructions_line_number]
-        JIT_self_instructions = JIT_self_instructions_line_data.split()[0]
-        JIT_self_instructions = int(JIT_self_instructions.replace(',', ''))
+        jit_self_instructions_line_data = \
+            callgrind_data[jit_self_instructions_line_number]
+        jit_self_instructions = jit_self_instructions_line_data.split()[0]
+        jit_self_instructions = int(jit_self_instructions.replace(',', ''))
 
         # Line number with the JIT self + inclusive number of instructions
-        # It's the line above the first JIT call when running with --tree=caller
-        JIT_total_instructions_line_number = JIT_self_instructions_line_number-1
+        # It's the line above the first JIT call when running with
+        # --tree=caller
+        jit_total_instructions_line_number = \
+            jit_self_instructions_line_number-1
         # Get the JIT self + inclusive number of instructions
-        JIT_total_instructions_line_data = \
-            callgrind_data[JIT_total_instructions_line_number]
-        JIT_total_instructions = JIT_total_instructions_line_data.split()[0]
-        JIT_total_instructions = int(JIT_total_instructions.replace(',', ''))
+        jit_total_instructions_line_data = \
+            callgrind_data[jit_total_instructions_line_number]
+        jit_total_instructions = jit_total_instructions_line_data.split()[0]
+        jit_total_instructions = int(jit_total_instructions.replace(',', ''))
 
         # Calculate number of instructions in helpers and code generation
-        helpers_instructions = JIT_total_instructions-JIT_self_instructions
-        code_generation_instructions = total_instructions-JIT_total_instructions
+        helpers_instructions = jit_total_instructions-jit_self_instructions
+        code_generation_instructions = \
+            total_instructions-jit_total_instructions
 
         # Print results (Insert commas in large numbers)
         # Print total number of instructions
@@ -149,12 +162,12 @@ def main():
         print('{:<20}{:>20}\t{:>6.3f}%'.
               format("Code Generation:",
                      format(code_generation_instructions, ","),
-                     (code_generation_instructions / total_instructions) * 100))
-        # Print JIT instructions and percentage
+                     (code_generation_instructions/total_instructions)*100))
+        # Print jit instructions and percentage
         print('{:<20}{:>20}\t{:>6.3f}%'.
-              format("JIT Execution:",
-                     format(JIT_self_instructions, ","),
-                     (JIT_self_instructions / total_instructions) * 100))
+              format("jit Execution:",
+                     format(jit_self_instructions, ","),
+                     (jit_self_instructions / total_instructions) * 100))
         # Print helpers instructions and percentage
         print('{:<20}{:>20}\t{:>6.3f}%'.
               format("Helpers:",

From patchwork Fri Aug 28 10:40:57 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353165
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=Ee1yLB37;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGPj71vyz9sTC
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:42:17 +1000 (AEST)
Received: from localhost ([::1]:53158 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbpn-0003Va-Tu
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:42:15 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34440)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpH-0003To-FF
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400
Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]:39586)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpE-0005gz-JQ
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400
Received: by mail-wm1-x329.google.com with SMTP id b79so461882wmb.4
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=;
 b=Ee1yLB37j//pyLubHmHwGE8x2jNKvKKRnrZaLj1/xd5pSgwuowx6nfn0DOwOpSopLR
 lGrY3IzEbRA9n29sUMWiAFi5Nj2kl0YaJ/mhdAW/uodYb0B7NNvBctZw+DZXFQ3yidzC
 EP9i1sM/0hQ2O7xlHkMT6RC679fmLATZv4VoQMCxTWt+zhdBTs6I14b7TXYDDG/KkRCZ
 86si0/dF9yibzmS08RLtZXmlOAvo88BSy/Ly0dVn0/ALObIPfXIAO08mTXce39zOOXZf
 kv9JgEYhlYyp+y5XTcjCCx0XcGHyghgbGVBw4WAQTtaQYY+5HAiIs0lfR5QrPs6MPB7N
 CXvw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=;
 b=A7mJaKGoZVAedVWUOv2sMnSRbe+WRYr5r+gHJs3hCyDC0+PItsbn+MH7zYaac3qPgU
 oXzUp0EByUSSp383wDWES+vsiNquyZMwj0qQBErrwFtHis8nKRZYZOg6VplliMw+rey8
 1v8zDcljO6rTiM10nM2HomWd2ry3rSz6hVcMbj5B9LoUQDaYIT9yHbzPJknThkaoSjuN
 CezMwJ3yhn2icmsk54U0xDTLCx2zXMnIff2WNrfSTxSEklbr8D8it5TrXi2O9GdckmS1
 muKWR3W4ToEFsnOhLcpQyVO+E8Yk1MooiftFUKQVwo5U+YgZBt4UOSE7eIKtR55iZ7cV
 yu1Q==
X-Gm-Message-State: AOAM5333ZetjEBn2uL10Mll/K7q/nQzpX04uhpuT9XH4KXcjM0STbCF0
 gsvUkaXDkT+NMIoPkYA2qeJoP0JO5oQqHg==
X-Google-Smtp-Source: 
 ABdhPJxzVzFgK/pQN+NIm9cfQdPnRVmfEJwuo9hH0KmgVb+A2Odn35Uv2iFuBuHTUbF0FlSOjwCBsQ==
X-Received: by 2002:a1c:e919:: with SMTP id q25mr953381wmc.123.1598611298903;
 Fri, 28 Aug 2020 03:41:38 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.37
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:38 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 4/9] scripts/performance: Add list_fn_callees.py script
Date: Fri, 28 Aug 2020 12:40:57 +0200
Message-Id: <20200828104102.4490-5-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::329;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wm1-x329.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

Python script that prints the callees of a given list of QEMU
functions.

Syntax:
list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \
               <qemu executable> [<qemu executable options>] \
               <target executable> [<target executable options>]

[-h] - Print the script arguments help message.
-f FUNCTION [FUNCTION ...] - List of function names

Example of usage:
list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \
                      qemu-mips coulomb_double-mips -n10

Example output:
 Total number of instructions: 108,952,851

 Callees of helper_float_sub_d:

 No. Instructions Percentage  Calls Ins/Call Function Name Source File
 --- ------------ ---------- ------ -------- ------------- ---------------
   1      153,160     0.141%  1,305     117  float64_sub   <qemu>/fpu/softfloat.c

 Callees of helper_float_mul_d:

 No. Instructions Percentage  Calls Ins/Call Function Name Source File
 --- ------------ ---------- ------ -------- ------------- ---------------
   1      131,137     0.120%  1,014      129 float64_mul   <qemu>/fpu/softfloat.c

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
 scripts/performance/list_fn_callees.py | 245 +++++++++++++++++++++++++
 1 file changed, 245 insertions(+)
 create mode 100755 scripts/performance/list_fn_callees.py

diff --git a/scripts/performance/list_fn_callees.py b/scripts/performance/list_fn_callees.py
new file mode 100755
index 0000000000..6aa8f6b6ca
--- /dev/null
+++ b/scripts/performance/list_fn_callees.py
@@ -0,0 +1,245 @@
+#!/usr/bin/env python3
+
+"""
+Print the callees of a given list of QEMU functions.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+from typing import List, Union
+
+
+def find_function_lines(function_name: str,
+                        callgrind_data: List[str]) -> List[int]:
+    """
+    Search for the line with the function name in the
+    callgrind_annotate output when ran using --tre=calling.
+    All the function callees should be listed after that line.
+
+    Parameters:
+    function_name (string): The desired function name to print its callees
+    callgrind_data (List[str]): callgrind_annotate output
+
+    Returns:
+    (List[int]): List of function line numbers
+    """
+    lines = []
+    for (i, callgrind_datum) in enumerate(callgrind_data):
+        split_line = callgrind_datum.split()
+        if len(split_line) > 2 and \
+                split_line[1] == "*" and \
+                split_line[2].split(":")[-1] == function_name:
+            # Function might be in the callgrind_annotate output more than
+            # once, so don't break after finding an instance
+            if callgrind_data[i + 1] != "\n":
+                # Only append the line number if the found instance has
+                # callees
+                lines.append(i)
+    return lines
+
+
+def get_function_calles(
+        function_lines: List[int],
+        callgrind_data: List[str]) -> List[List[Union[str, int]]]:
+    """
+    Get all callees data for a function given its list of line numbers in
+    callgrind_annotate output.
+
+    Parameters:
+    function_lines (List[int]): Line numbers of the function to get its callees
+    callgrind_data (List[str]): callgrind_annotate output
+
+    Returns:
+    (List[List[Union[str, int]]]):[[number_of_instructions(int),
+                                    callee_name(str),
+                                    number_of_calls(int),
+                                    source_file(str)],
+                                    ...]
+    """
+    callees: List[List[Union[str, int]]] = []
+    for function_line in function_lines:
+        next_callee = function_line + 1
+        while callgrind_data[next_callee] != "\n":
+            split_line = callgrind_data[next_callee].split()
+            number_of_instructions = int(split_line[0].replace(",", ""))
+            source_file = split_line[2].split(":")[0]
+            callee_name = split_line[2].split(":")[1]
+            number_of_calls = int(split_line[3][1:-2])
+            callees.append([number_of_instructions, callee_name,
+                            number_of_calls, source_file])
+            next_callee += 1
+    return sorted(callees, reverse=True)
+
+
+def main():
+    """
+    Parse the command line arguments then start execution.
+
+    Syntax:
+    list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \
+               <qemu executable> [<qemu executable options>] \
+               <target executable> [<target executable options>]
+
+    [-h] - Print the script arguments help message.
+    -f FUNCTION [FUNCTION ...] - List of function names
+
+    Example of usage:
+    list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \
+                      qemu-mips coulomb_double-mips
+    """
+
+    # Parse the command line arguments
+    parser = argparse.ArgumentParser(
+        usage="list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- "
+        "<qemu executable> [<qemu executable options>] "
+        "<target executable> [<target executable options>]")
+
+    parser.add_argument("-f", dest="function", type=str,
+                        nargs="+", required=True,
+                        help="list of function names to print their callees")
+
+    parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+    args = parser.parse_args()
+
+    # Extract the needed variables from the args
+    command = args.command
+    function_names = args.function
+
+    # Insure that valgrind is installed
+    check_valgrind = subprocess.run(
+        ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
+    if check_valgrind.returncode:
+        sys.exit("Please install valgrind before running the script.")
+
+    # Save all intermediate files in a temporary directory
+    with tempfile.TemporaryDirectory() as tmpdirname:
+        # callgrind output file path
+        data_path = os.path.join(tmpdirname, "callgrind.data")
+        # callgrind_annotate output file path
+        annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out")
+
+        # Run callgrind
+        callgrind = subprocess.run((["valgrind",
+                                     "--tool=callgrind",
+                                     "--callgrind-out-file=" + data_path]
+                                    + command),
+                                   stdout=subprocess.DEVNULL,
+                                   stderr=subprocess.PIPE,
+                                   check=False)
+        if callgrind.returncode:
+            sys.exit(callgrind.stderr.decode("utf-8"))
+
+        # Save callgrind_annotate output
+        with open(annotate_out_path, "w") as output:
+            callgrind_annotate = subprocess.run(["callgrind_annotate",
+                                                 data_path,
+                                                 "--threshold=100",
+                                                 "--tree=calling"],
+                                                stdout=output,
+                                                stderr=subprocess.PIPE,
+                                                check=False)
+            if callgrind_annotate.returncode:
+                sys.exit(callgrind_annotate.stderr.decode("utf-8"))
+
+        # Read the callgrind_annotate output to callgrind_data[]
+        callgrind_data = []
+        with open(annotate_out_path, "r") as data:
+            callgrind_data = data.readlines()
+
+        # Line number with the total number of instructions
+        total_instructions_line_number = 20
+        # Get the total number of instructions
+        total_instructions_line_data = \
+            callgrind_data[total_instructions_line_number]
+        total_instructions = total_instructions_line_data.split()[0]
+
+        print("Total number of instructions: {}\n".format(total_instructions))
+
+        # Remove commas and convert to int
+        total_instructions = int(total_instructions.replace(",", ""))
+
+        for function_name in function_names:
+            # Line numbers with the desired function
+            function_lines = find_function_lines(function_name, callgrind_data)
+
+            if len(function_lines) == 0:
+                print("Couldn't locate function: {}.\n".format(
+                    function_name))
+                continue
+
+            # Get function callees
+            function_callees = get_function_calles(
+                function_lines, callgrind_data)
+
+            print("Callees of {}:\n".format(function_name))
+
+            # Print table header
+            print("{:>4}  {:>15}  {:>10}  {:>15}  {:>10}  {:<25}  {}".
+                  format(
+                      "No.",
+                      "Instructions",
+                      "Percentage",
+                      "Calls",
+                      "Ins/Call",
+                      "Function Name",
+                      "Source File")
+                  )
+
+            print("{:>4}  {:>15}  {:>10}  {:>15}  {:>10}  {:<25}  {}".
+                  format(
+                      "-" * 4,
+                      "-" * 15,
+                      "-" * 10,
+                      "-" * 15,
+                      "-" * 10,
+                      "-" * 25,
+                      "-" * 30)
+                  )
+
+            for (index, callee) in enumerate(function_callees, start=1):
+                instructions = callee[0]
+                percentage = (callee[0] / total_instructions) * 100
+                calls = callee[2]
+                instruction_per_call = int(callee[0] / callee[2])
+                function_name = callee[1]
+                source_file = callee[3]
+                # Print extracted data
+                print("{:>4}  {:>15}  {:>9.3f}%  {:>15}  {:>10}  {:<25}  {}".
+                      format(
+                          index,
+                          format(instructions, ","),
+                          round(percentage, 3),
+                          format(calls, ","),
+                          format(instruction_per_call, ","),
+                          function_name,
+                          source_file)
+                      )
+
+            print("\n")
+
+
+if __name__ == "__main__":
+    main()

From patchwork Fri Aug 28 10:40:58 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353168
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=Zm/LCScj;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGRs3W37z9sPB
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:44:09 +1000 (AEST)
Received: from localhost ([::1]:33256 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbrb-0006vC-Cf
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:44:07 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34456)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpJ-0003W3-Do
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:45 -0400
Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:42832)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpH-0005hF-45
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:45 -0400
Received: by mail-wr1-x444.google.com with SMTP id c18so834373wrm.9
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=v7lvcz/T9X8UTSmvha1Vg7mQ7t6EJ96o/jbqvgFXbUM=;
 b=Zm/LCScjFpPa/OlSC90gjfzM/dO1Cbbd5MhzgHc9Av9kwuVbpJIujUhey643OVEB1d
 TT/D7HlKFQmnS9Q/4boVcHsch3eljfQZQHjoVG01OgX2Vdp2N5J1I5FawkA49Jm962DL
 MY2UmlBxSy2HvOFv7r8nQrmQZd/lYbDSYj0HpMra0uIMN6rxb5Srtyen7yRJUWmV3CXT
 SFpiLG2qgNMuX6st734EPDycMo0Rm3pYVC/MRiZjHBz7QeE84BnSfz9H+vUoxlO+OZ3y
 Gddeph3LBqzyWTF4LCmBxde3BbQjLQRmp/E0tmh44npvEsgrCxE8UL7qqEc+59O4mga8
 lgKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=v7lvcz/T9X8UTSmvha1Vg7mQ7t6EJ96o/jbqvgFXbUM=;
 b=Tv8P2f6ILztUAIY7DoyREs1wKjd4qaFtO+ybMAvqN/B/2obnhNhnjMwvF6uzXh/9Ok
 MvyR7/K4LLEkyJT9laJJNGByRPGqNyVhaEKNEizKIjIFCGyk6xJ93ollHM1GvxFD38uN
 hZSZozq9FwHdeABqYhST1vv1FOdTEqUBtlmevdIcxSxsR09ggmyuYTi3le4fbazzW8Om
 gRu7UUGFFEvdYTjHvxm/PwDziGi1zRk9cphZi6/WsGVjxqm00uJ7m4g1YPs6FkvVihxO
 K9nnjueyO6KixGE+XOawdCqFDvTYbYp+Er9/1kpMmcWXLmPic4X4TnYUKF7fnC+2dSEq
 aXLQ==
X-Gm-Message-State: AOAM531UJu/+YYBcpjhRLoVahiLfyT/9gTEV1omwSqS+J7MiGclnEIJG
 wyGiHz+6QWyYwHxPomP+uFk34gN9hKaU+g==
X-Google-Smtp-Source: 
 ABdhPJyyk0V9ys0F1gVgrOARWvF6p8zAVaB4JNP7nTqmgazkjjzo1v0ik5fyIsGCLdQhEQDDkO3rdg==
X-Received: by 2002:a5d:4c90:: with SMTP id z16mr1016638wrs.170.1598611300883;
 Fri, 28 Aug 2020 03:41:40 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.39
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:40 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 5/9] scripts/performance: Add list_helpers.py script
Date: Fri, 28 Aug 2020 12:40:58 +0200
Message-Id: <20200828104102.4490-6-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::444;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x444.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

Python script that prints executed helpers of a QEMU invocation.

Syntax:
list_helpers.py [-h] -- \
               <qemu executable> [<qemu executable options>] \
               <target executable> [<target executable options>]

[-h] - Print the script arguments help message.

Example of usage:
list_helpers.py -- qemu-mips coulomb_double-mips -n10

Example output:
 Total number of instructions: 108,933,695

 Executed QEMU Helpers:

 No. Ins     Percent  Calls Ins/Call Helper Name             Source File
 --- ------- ------- ------ -------- --------------------    ---------------
   1 183,021  0.168%  1,305      140 helper_float_sub_d      <qemu>/target/mips/fpu_helper.c
   2 177,111  0.163%    770      230 helper_float_madd_d     <qemu>/target/mips/fpu_helper.c
   3 171,537  0.157%  1,014      169 helper_float_mul_d      <qemu>/target/mips/fpu_helper.c
   4 157,298  0.144%  2,443       64 helper_lookup_tb_ptr    <qemu>/accel/tcg/tcg-runtime.c
   5 138,123  0.127%    897      153 helper_float_add_d      <qemu>/target/mips/fpu_helper.c
   6  47,083  0.043%    207      227 helper_float_msub_d     <qemu>/target/mips/fpu_helper.c
   7  24,062  0.022%    487       49 helper_cmp_d_lt         <qemu>/target/mips/fpu_helper.c
   8  22,910  0.021%    150      152 helper_float_div_d      <qemu>/target/mips/fpu_helper.c
   9  15,497  0.014%    321       48 helper_cmp_d_eq         <qemu>/target/mips/fpu_helper.c
  10   9,100  0.008%     52      175 helper_float_trunc_w_d  <qemu>/target/mips/fpu_helper.c
  11   7,059  0.006%     10      705 helper_float_sqrt_d     <qemu>/target/mips/fpu_helper.c
  12   3,000  0.003%     40       75 helper_cmp_d_ule        <qemu>/target/mips/fpu_helper.c
  13   2,720  0.002%     20      136 helper_float_cvtd_w     <qemu>/target/mips/fpu_helper.c
  14   2,477  0.002%     27       91 helper_swl              <qemu>/target/mips/op_helper.c
  15   2,000  0.002%     40       50 helper_cmp_d_le         <qemu>/target/mips/fpu_helper.c
  16   1,800  0.002%     40       45 helper_cmp_d_un         <qemu>/target/mips/fpu_helper.c
  17   1,164  0.001%     12       97 helper_raise_exception_ <qemu>/target/mips/op_helper.c
  18     720  0.001%     10       72 helper_cmp_d_ult        <qemu>/target/mips/fpu_helper.c
  19     560  0.001%    140        4 helper_cfc1             <qemu>/target/mips/fpu_helper.c

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
 scripts/performance/list_helpers.py | 221 ++++++++++++++++++++++++++++
 1 file changed, 221 insertions(+)
 create mode 100755 scripts/performance/list_helpers.py

diff --git a/scripts/performance/list_helpers.py b/scripts/performance/list_helpers.py
new file mode 100755
index 0000000000..823b1cef66
--- /dev/null
+++ b/scripts/performance/list_helpers.py
@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+
+"""
+Print the executed helpers of a QEMU invocation.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+from typing import List, Union
+
+
+def find_jit_line(callgrind_data: List[str]) -> int:
+    """
+    Search for the line with the JIT call in the callgrind_annotate
+    output when ran using --tre=calling.
+    All the helpers should be listed after that line.
+
+    Parameters:
+    callgrind_data (List[str]): callgrind_annotate output
+
+    Returns:
+    (int): Line number of JIT call
+    """
+    line = -1
+    for (i, callgrind_datum) in enumerate(callgrind_data):
+        split_line = callgrind_datum.split()
+        if len(split_line) > 2 and \
+                split_line[1] == "*" and \
+                split_line[-1] == "[???]":
+            line = i
+            break
+    return line
+
+
+def get_helpers(jit_line: int,
+                callgrind_data: List[str]) -> List[List[Union[str, int]]]:
+    """
+    Get all helpers data given the line number of the JIT call.
+
+    Parameters:
+    jit_line (int): Line number of the JIT call
+    callgrind_data (List[str]): callgrind_annotate output
+
+    Returns:
+    (List[List[Union[str, int]]]):[[number_of_instructions(int),
+                                    helper_name(str),
+                                    number_of_calls(int),
+                                    source_file(str)],
+                                    ...]
+    """
+    helpers: List[List[Union[str, int]]] = []
+    next_helper = jit_line + 1
+    while callgrind_data[next_helper] != "\n":
+        split_line = callgrind_data[next_helper].split()
+        number_of_instructions = int(split_line[0].replace(",", ""))
+        source_file = split_line[2].split(":")[0]
+        callee_name = split_line[2].split(":")[1]
+        number_of_calls = int(split_line[3][1:-2])
+        helpers.append([number_of_instructions, callee_name,
+                        number_of_calls, source_file])
+        next_helper += 1
+    return sorted(helpers, reverse=True)
+
+
+def main():
+    """
+    Parse the command line arguments then start execution
+
+    Syntax:
+    list_helpers.py [-h] -- \
+               <qemu executable> [<qemu executable options>] \
+               <target executable> [<target executable options>]
+
+    [-h] - Print the script arguments help message.
+
+    Example of usage:
+    list_helpers.py -- qemu-mips coulomb_double-mips
+    """
+    # Parse the command line arguments
+    parser = argparse.ArgumentParser(
+        usage="list_helpers.py [-h] -- "
+        "<qemu executable> [<qemu executable options>] "
+        "<target executable> [<target executable options>]")
+
+    parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+    args = parser.parse_args()
+
+    # Extract the needed variables from the args
+    command = args.command
+
+    # Insure that valgrind is installed
+    check_valgrind = subprocess.run(
+        ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
+    if check_valgrind.returncode:
+        sys.exit("Please install valgrind before running the script.")
+
+    # Save all intermediate files in a temporary directory
+    with tempfile.TemporaryDirectory() as tmpdirname:
+        # callgrind output file path
+        data_path = os.path.join(tmpdirname, "callgrind.data")
+        # callgrind_annotate output file path
+        annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out")
+
+        # Run callgrind
+        callgrind = subprocess.run((["valgrind",
+                                     "--tool=callgrind",
+                                     "--callgrind-out-file=" + data_path]
+                                    + command),
+                                   stdout=subprocess.DEVNULL,
+                                   stderr=subprocess.PIPE,
+                                   check=False)
+        if callgrind.returncode:
+            sys.exit(callgrind.stderr.decode("utf-8"))
+
+        # Save callgrind_annotate output
+        with open(annotate_out_path, "w") as output:
+            callgrind_annotate = subprocess.run(["callgrind_annotate",
+                                                 data_path,
+                                                 "--threshold=100",
+                                                 "--tree=calling"],
+                                                stdout=output,
+                                                stderr=subprocess.PIPE,
+                                                check=False)
+            if callgrind_annotate.returncode:
+                sys.exit(callgrind_annotate.stderr.decode("utf-8"))
+
+        # Read the callgrind_annotate output to callgrind_data[]
+        callgrind_data = []
+        with open(annotate_out_path, "r") as data:
+            callgrind_data = data.readlines()
+
+        # Line number with the total number of instructions
+        total_instructions_line_number = 20
+        # Get the total number of instructions
+        total_instructions_line_data = \
+            callgrind_data[total_instructions_line_number]
+        total_instructions = total_instructions_line_data.split()[0]
+
+        print("Total number of instructions: {}\n".format(total_instructions))
+
+        # Remove commas and convert to int
+        total_instructions = int(total_instructions.replace(",", ""))
+
+        # Line number with the JIT call
+        jit_line = find_jit_line(callgrind_data)
+        if jit_line == -1:
+            sys.exit("Couldn't locate the JIT call ... Exiting")
+
+        # Get helpers
+        helpers = get_helpers(jit_line, callgrind_data)
+
+        print("Executed QEMU Helpers:\n")
+
+        # Print table header
+        print("{:>4}  {:>15}  {:>10}  {:>15}  {:>10}  {:<25}  {}".
+              format(
+                  "No.",
+                  "Instructions",
+                  "Percentage",
+                  "Calls",
+                  "Ins/Call",
+                  "Helper Name",
+                  "Source File")
+              )
+
+        print("{:>4}  {:>15}  {:>10}  {:>15}  {:>10}  {:<25}  {}".
+              format(
+                  "-" * 4,
+                  "-" * 15,
+                  "-" * 10,
+                  "-" * 15,
+                  "-" * 10,
+                  "-" * 25,
+                  "-" * 30)
+              )
+
+        for (index, callee) in enumerate(helpers, start=1):
+            instructions = callee[0]
+            percentage = (callee[0] / total_instructions) * 100
+            calls = callee[2]
+            instruction_per_call = int(callee[0] / callee[2])
+            helper_name = callee[1]
+            source_file = callee[3]
+            # Print extracted data
+            print("{:>4}  {:>15}  {:>9.3f}%  {:>15}  {:>10}  {:<25}  {}".
+                  format(
+                      index,
+                      format(instructions, ","),
+                      round(percentage, 3),
+                      format(calls, ","),
+                      format(instruction_per_call, ","),
+                      helper_name,
+                      source_file)
+                  )
+
+
+if __name__ == "__main__":
+    main()

From patchwork Fri Aug 28 10:40:59 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353169
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=d22AIMjK;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGV12c95z9sPB
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:46:01 +1000 (AEST)
Received: from localhost ([::1]:40138 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbtP-0001Jv-BP
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:45:59 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34486)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpM-0003Yn-4j
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:48 -0400
Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:40951)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpJ-0005hg-3a
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:47 -0400
Received: by mail-wr1-x441.google.com with SMTP id b18so843147wrs.7
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=;
 b=d22AIMjKeRUgBAq6Me+AslRt0Z7/E/+lfAYk3Vhyj8k1DRf8LQV5OXxTbpE/8tCkRC
 frlF68m70idHOn5t4oyqLtwzxdjs7EiR0UzvYpV4K+XqBu0PKE6PHCzb2jhS7CDPuLFr
 e6l82FARRMrbNCzVWFcNX8xoYPvXz7IV8nrAr3gN+zTMuytZyPFx4RtHP0uIevCpmbCn
 d0C2HPig5+nf8mvn+Gq8UER1gcI3Vi7YAbNXUfQkj4v5mP40brC6JnPd/dO51+ZfXlqP
 KKel+EL8jPYe0ojuwXc/Qyj3AGzSWETumaaUC7xLZ+91uV1sYj4smmidCJfTClMEunx5
 /eIw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=;
 b=feEeLct9rkHjZ88mvTPUA4l7AWPH55VmgjDlav8Grcx26XOoq2RlMTk0buX2lCXkeg
 x03GsHgfixSS59dkFF+/2nle12X8jwBSvsFvpB87A4LoIlMs8EIKEwbnbZ1yNodWAPo2
 d2vn4PJnBZOucu+GUl65aOMYUmE4fXi9mAUTMiCsYU+fyEwezbp+PNohHVXvftwqVtLZ
 DktxSXdJnE90NuV6PrZQW3w2VA0kB0r1uIC5Xlr+/0074aqDGEHOwnlL8VTdiUP28s5F
 3cPk8OzFs+1H1dvz6R9FyhxsF8YgFH/tQlbnT1msTIhrxK/WrJA9w39fpNlnjkYnT9kt
 XlqA==
X-Gm-Message-State: AOAM532DQqWgeRH7B5EkZJz1RXs6MuN8PCxUHB3U7wvnmQLDRfFpfYpE
 VjAsOWrAltggJMxmA+zqq90nsyKBD7J7Rw==
X-Google-Smtp-Source: 
 ABdhPJzLgdt+VUg1kK5s5dAGQlueSyKj9ZHTrjhfUnKU3dUnC08oX7yEINk3+0r1h3AlB5U2mS4WfA==
X-Received: by 2002:a5d:6087:: with SMTP id w7mr916406wrt.290.1598611303029;
 Fri, 28 Aug 2020 03:41:43 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.40
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:42 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 6/9] scripts/performance: Add bisect.py script
Date: Fri, 28 Aug 2020 12:40:59 +0200
Message-Id: <20200828104102.4490-7-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::441;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x441.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

Python script that locates the commit that caused a performance
degradation or improvement in QEMU using the git bisect command
(binary search).

Syntax:
bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \
--target TARGET --tool {perf,callgrind} -- \
<target executable> [<target executable options>]

[-h] - Print the script arguments help message
-s,--start START - First commit hash in the search range
[-e,--end END] - Last commit hash in the search range
                (default: Latest commit)
[-q,--qemu QEMU] - QEMU path.
                (default: Path to a GitHub QEMU clone)
--target TARGET - QEMU target name
--tool {perf,callgrind} - Underlying tool used for measurements

Example of usage:
bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \
--tool=perf -- coulomb_double-ppc -n 1000

Example output:
Start Commit Instructions:     12,710,790,060
End Commit Instructions:       13,031,083,512
Performance Change:            -2.458%

Estimated Number of Steps:     10

*****************BISECT STEP 1*****************
Instructions:        13,031,097,790
Status:              slow commit
*****************BISECT STEP 2*****************
Instructions:        12,710,805,265
Status:              fast commit
*****************BISECT STEP 3*****************
Instructions:        13,031,028,053
Status:              slow commit
*****************BISECT STEP 4*****************
Instructions:        12,711,763,211
Status:              fast commit
*****************BISECT STEP 5*****************
Instructions:        13,031,027,292
Status:              slow commit
*****************BISECT STEP 6*****************
Instructions:        12,711,748,738
Status:              fast commit
*****************BISECT STEP 7*****************
Instructions:        12,711,748,788
Status:              fast commit
*****************BISECT STEP 8*****************
Instructions:        13,031,100,493
Status:              slow commit
*****************BISECT STEP 9*****************
Instructions:        12,714,472,954
Status:              fast commit
****************BISECT STEP 10*****************
Instructions:        12,715,409,153
Status:              fast commit
****************BISECT STEP 11*****************
Instructions:        12,715,394,739
Status:              fast commit

*****************BISECT RESULT*****************
commit 0673ecdf6cb2b1445a85283db8cbacb251c46516
Author: Richard Henderson <richard.henderson@linaro.org>
Date:   Tue May 5 10:40:23 2020 -0700

    softfloat: Inline float64 compare specializations

    Replace the float64 compare specializations with inline functions
    that call the standard float64_compare{,_quiet} functions.
    Use bool as the return type.
***********************************************

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
 scripts/performance/bisect.py | 425 ++++++++++++++++++++++++++++++++++
 1 file changed, 425 insertions(+)
 create mode 100755 scripts/performance/bisect.py

diff --git a/scripts/performance/bisect.py b/scripts/performance/bisect.py
new file mode 100755
index 0000000000..0c60be22ce
--- /dev/null
+++ b/scripts/performance/bisect.py
@@ -0,0 +1,425 @@
+#!/usr/bin/env python3
+
+"""
+Locate the commit that caused a performance degradation or improvement in
+QEMU using the git bisect command (binary search).
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import multiprocessing
+import tempfile
+import os
+import shutil
+import subprocess
+import sys
+
+from typing import List
+
+
+# --------------------------- GIT WRAPPERS --------------------------
+def git_bisect(qemu_path: str, qemu_build_path: str, command: str,
+               args: List[str] = None) -> str:
+    """
+    Wrapper function for running git bisect.
+
+    Parameters:
+    qemu_path (str): QEMU path
+    qemu_build_path (str): Path to the build directory with configuration files
+    command (str): bisect command (start|fast|slow|reset)
+    args (list): Optional arguments
+
+    Returns:
+    (str): git bisect stdout.
+    """
+    process = ["git", "bisect", command]
+    if args:
+        process += args
+    bisect = subprocess.run(process,
+                            cwd=qemu_path,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            check=False)
+    if bisect.returncode:
+        clean_exit(qemu_build_path, bisect.stderr.decode("utf-8"))
+
+    return bisect.stdout.decode("utf-8")
+
+
+def git_checkout(commit: str, qemu_path: str, qemu_build_path: str) -> None:
+    """
+    Wrapper function for checking out a given git commit.
+
+    Parameters:
+    commit (str): Commit hash of a git commit
+    qemu_path (str): QEMU path
+    qemu_build_path (str): Path to the build directory with configuration files
+    """
+    checkout_commit = subprocess.run(["git",
+                                      "checkout",
+                                      commit],
+                                     cwd=qemu_path,
+                                     stdout=subprocess.DEVNULL,
+                                     stderr=subprocess.PIPE,
+                                     check=False)
+    if checkout_commit.returncode:
+        clean_exit(qemu_build_path, checkout_commit.stderr.decode("utf-8"))
+
+
+def git_clone(qemu_path: str) -> None:
+    """
+    Wrapper function for cloning QEMU git repo from GitHub.
+
+    Parameters:
+    qemu_path (str): Path to clone the QEMU repo to
+    """
+    clone_qemu = subprocess.run(["git",
+                                 "clone",
+                                 "https://github.com/qemu/qemu.git",
+                                 qemu_path],
+                                stderr=subprocess.STDOUT,
+                                check=False)
+    if clone_qemu.returncode:
+        sys.exit("Failed to clone QEMU!")
+# -------------------------------------------------------------------
+
+
+def check_requirements(tool: str) -> None:
+    """
+    Verify that all script requirements are installed (perf|callgrind & git).
+
+    Parameters:
+    tool (str): Tool used for the measurement (perf or callgrind)
+    """
+    if tool == "perf":
+        check_perf_installation = subprocess.run(["which", "perf"],
+                                                 stdout=subprocess.DEVNULL,
+                                                 check=False)
+        if check_perf_installation.returncode:
+            sys.exit("Please install perf before running the script.")
+
+        # Insure user has previllage to run perf
+        check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
+                                                  stdout=subprocess.DEVNULL,
+                                                  stderr=subprocess.DEVNULL,
+                                                  check=False)
+        if check_perf_executability.returncode:
+            sys.exit("""
+        Error:
+        You may not have permission to collect stats.
+        Consider tweaking /proc/sys/kernel/perf_event_paranoid,
+        which controls use of the performance events system by
+        unprivileged users (without CAP_SYS_ADMIN).
+        -1: Allow use of (almost) all events by all users
+            Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
+        0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
+            Disallow raw tracepoint access by users without CAP_SYS_ADMIN
+        1: Disallow CPU event access by users without CAP_SYS_ADMIN
+        2: Disallow kernel profiling by users without CAP_SYS_ADMIN
+        To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
+        kernel.perf_event_paranoid = -1
+
+        *Alternatively, you can run this script under sudo privileges.
+        """)
+    elif tool == "callgrind":
+        check_valgrind_installation = subprocess.run(["which", "valgrind"],
+                                                     stdout=subprocess.DEVNULL,
+                                                     check=False)
+        if check_valgrind_installation.returncode:
+            sys.exit("Please install valgrind before running the script.")
+
+    # Insure that git is installed
+    check_git_installation = subprocess.run(["which", "git"],
+                                            stdout=subprocess.DEVNULL,
+                                            check=False)
+    if check_git_installation.returncode:
+        sys.exit("Please install git before running the script.")
+
+
+def clean_exit(qemu_build_path: str, error_message: str) -> None:
+    """
+    Clean up intermediate files and exit.
+
+    Parameters:
+    qemu_build_path (str): Path to the build directory with configuration files
+    error_message (str): Error message to display after exiting
+    """
+    shutil.rmtree(qemu_build_path)
+    sys.exit(error_message)
+
+
+def make(qemu_build_path: str) -> None:
+    """
+    Build QEMU by running the Makefile.
+
+    Parameters:
+    qemu_build_path (str): Path to the build directory with configuration files
+    """
+    run_make = subprocess.run(["make",
+                               "-j",
+                               str(multiprocessing.cpu_count())],
+                              cwd=qemu_build_path,
+                              stdout=subprocess.DEVNULL,
+                              stderr=subprocess.PIPE,
+                              check=False)
+    if run_make.returncode:
+        clean_exit(qemu_build_path, run_make.stderr.decode("utf-8"))
+
+
+def measure_instructions(tool: str, qemu_build_path: str, target: str,
+                         command: List[str]) -> int:
+    """
+    Measure the number of instructions when running an program with QEMU.
+
+    Parameters:
+    tool (str): Tool used for the measurement (perf|callgrind)
+    qemu_build_path (str): Path to the build directory with configuration files
+    target (str): QEMU target
+    command (list): Program path and arguments
+
+    Returns:
+    (int): Number of instructions.
+    """
+    qemu_exe_path = os.path.join(qemu_build_path,
+                                 "{}-linux-user".format(target),
+                                 "qemu-{}".format(target))
+    instructions = 0
+    if tool == "perf":
+        run_perf = subprocess.run((["perf",
+                                    "stat",
+                                    "-x",
+                                    " ",
+                                    "-e",
+                                    "instructions",
+                                    qemu_exe_path]
+                                   + command),
+                                  stdout=subprocess.DEVNULL,
+                                  stderr=subprocess.PIPE,
+                                  check=False)
+        if run_perf.returncode:
+            clean_exit(qemu_build_path, run_perf.stderr.decode("utf-8"))
+
+        else:
+            perf_output = run_perf.stderr.decode("utf-8").split(" ")
+            instructions = int(perf_output[0])
+
+    elif tool == "callgrind":
+        with tempfile.NamedTemporaryFile() as tmpfile:
+            run_callgrind = subprocess.run((["valgrind",
+                                             "--tool=callgrind",
+                                             "--callgrind-out-file={}".format(
+                                                 tmpfile.name),
+                                             qemu_exe_path]
+                                            + command),
+                                           stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.PIPE,
+                                           check=False)
+        if run_callgrind.returncode:
+            clean_exit(qemu_build_path, run_callgrind.stderr.decode("utf-8"))
+
+        else:
+            callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n")
+            instructions = int(callgrind_output[8].split(" ")[-1])
+
+    return instructions
+
+
+def main():
+    """
+    Parse the command line arguments then start the execution.
+
+    Syntax:
+        bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \
+        --target TARGET --tool {perf,callgrind} -- \
+        <target executable> [<target executable options>]
+
+    Arguments:
+        [-h] - Print the script arguments help message
+        -s,--start START - First commit hash in the search range
+        [-e,--end END] - Last commit hash in the search range
+                (default: Latest commit)
+        [-q,--qemu QEMU] - QEMU path.
+                    (default: Path to a GitHub QEMU clone)
+        --target TARGET - QEMU target name
+        --tool {perf,callgrind} - Underlying tool used for measurements
+
+    Example of usage:
+        bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \
+        --tool=perf coulomb_double-ppc -n 1000
+    """
+
+    # Parse the command line arguments
+    parser = argparse.ArgumentParser(
+        usage="bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU]"
+        " --target TARGET --tool {perf,callgrind} -- "
+        "<target executable> [<target executable options>]")
+
+    parser.add_argument("-s", "--start", dest="start", type=str, required=True,
+                        help="First commit hash in the search range")
+    parser.add_argument("-e", "--end", dest="end", type=str, default="master",
+                        help="Last commit hash in the search range")
+    parser.add_argument("-q", "--qemu", dest="qemu", type=str, default="",
+                        help="QEMU path")
+    parser.add_argument("--target", dest="target", type=str, required=True,
+                        help="QEMU target")
+    parser.add_argument("--tool", dest="tool", choices=["perf", "callgrind"],
+                        required=True, help="Tool used for measurements")
+
+    parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+    args = parser.parse_args()
+
+    # Extract the needed variables from the args
+    start_commit = args.start
+    end_commit = args.end
+    qemu = args.qemu
+    target = args.target
+    tool = args.tool
+    command = args.command
+
+    # Set QEMU path
+    if qemu == "":
+        # Create a temp directory for cloning QEMU
+        tmpdir = tempfile.TemporaryDirectory()
+        qemu_path = os.path.join(tmpdir.name, "qemu")
+
+        # Clone QEMU into the temporary directory
+        print("Fetching QEMU: ", end="", flush=True)
+        git_clone(qemu_path)
+        print()
+    else:
+        qemu_path = qemu
+
+    # Create the build directory
+    qemu_build_path = os.path.join(qemu_path, "tmp-build-gcc")
+
+    if not os.path.exists(qemu_build_path):
+        os.mkdir(qemu_build_path)
+    else:
+        sys.exit("A build directory with the same name (tmp-build-gcc) used in"
+                 " the script is already in the provided QEMU path.")
+
+    # Configure QEMU
+    configure = subprocess.run(["../configure",
+                                "--target-list={}-linux-user".format(target)],
+                               cwd=qemu_build_path,
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.PIPE,
+                               check=False)
+    if configure.returncode:
+        clean_exit(qemu_build_path, configure.stderr.decode("utf-8"))
+
+    # Do performance measurements for the start commit
+    git_checkout(start_commit, qemu_path, qemu_build_path)
+    make(qemu_build_path)
+    start_commit_instructions = measure_instructions(tool,
+                                                     qemu_build_path,
+                                                     target,
+                                                     command)
+    print("{:<30} {}".format("Start Commit Instructions:",
+                             format(start_commit_instructions, ",")))
+
+    # Do performance measurements for the end commit
+    git_checkout(end_commit, qemu_path, qemu_build_path)
+    make(qemu_build_path)
+    end_commit_instructions = measure_instructions(tool,
+                                                   qemu_build_path,
+                                                   target,
+                                                   command)
+    print("{:<30} {}".format("End Commit Instructions:",
+                             format(end_commit_instructions, ",")))
+
+    # Calculate performance difference between start and end commits
+    performance_difference = \
+        (start_commit_instructions - end_commit_instructions) / \
+        max(end_commit_instructions, start_commit_instructions) * 100
+    performance_change = "+" if performance_difference > 0 else "-"
+    print("{:<30} {}".format("Performance Change:",
+                             performance_change +
+                             str(round(abs(performance_difference), 3))+"%"))
+
+    # Set the custom terms used for progressing in "git bisect"
+    term_old = "fast" if performance_difference < 0 else "slow"
+    term_new = "slow" if term_old == "fast" else "fast"
+
+    # Start git bisect
+    git_bisect(qemu_path, qemu_build_path, "start",
+               ["--term-old", term_old, "--term-new", term_new])
+    # Set start commit state
+    git_bisect(qemu_path, qemu_build_path, term_old, [start_commit])
+    # Set end commit state
+    bisect_output = git_bisect(
+        qemu_path, qemu_build_path, term_new, [end_commit])
+    # Print estimated bisect steps
+    print("\n{:<30} {}\n".format(
+        "Estimated Number of Steps:", bisect_output.split()[9]))
+
+    # Initialize bisect_count to track the number of performed
+    bisect_count = 1
+
+    while True:
+        print("**************BISECT STEP {}**************".
+              format(bisect_count))
+
+        make(qemu_build_path)
+
+        instructions = measure_instructions(tool,
+                                            qemu_build_path,
+                                            target,
+                                            command)
+        # Find the difference between the current instructions and start/end
+        # instructions.
+        diff_end = abs(instructions - end_commit_instructions)
+        diff_start = abs(instructions - start_commit_instructions)
+
+        # If current number of insructions is closer to that of start,
+        # set current commit as term_old.
+        # Else, set current commit as term_new.
+        if diff_end > diff_start:
+            bisect_command = term_old
+        else:
+            bisect_command = term_new
+
+        print("{:<20} {}".format("Instructions:", format(instructions, ",")))
+        print("{:<20} {}".format("Status:", "{} commit".
+                                 format(bisect_command)))
+
+        bisect_output = git_bisect(qemu_path, qemu_build_path, bisect_command)
+
+        # Continue if still bisecting,
+        # else, print result and break.
+        if not bisect_output.split(" ")[0] == "Bisecting:":
+            print("\n*****************BISECT RESULT*****************")
+            commit_message_start = bisect_output.find("commit\n") + 7
+            commit_message_end = bisect_output.find(":040000") - 1
+            print(bisect_output[commit_message_start:commit_message_end])
+            break
+
+        bisect_count += 1
+
+    # Reset git bisect
+    git_bisect(qemu_path, qemu_build_path, "reset")
+
+    # Delete temp build directory
+    shutil.rmtree(qemu_build_path)
+
+
+if __name__ == "__main__":
+    main()

From patchwork Fri Aug 28 10:41:00 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353171
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=dX41ymz9;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGX33szMz9sPB
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:47:47 +1000 (AEST)
Received: from localhost ([::1]:44508 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbv7-0003BI-JR
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:47:45 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34590)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpU-0003rt-Tz
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400
Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:34674)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpP-0005jN-Rs
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400
Received: by mail-wr1-x433.google.com with SMTP id f7so857616wrw.1
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding;
 bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=;
 b=dX41ymz9tc93PMf0ZRirpDHXn9NI60cKLfOFjuDhBM1gUH6SuQItoBy6JDX/CJOvk6
 bDz6/aq3ZOE1IxebLaizUmTQkWn7Mz5zHGHn+HrNOhHXjjGoeVdPGn+5PoPp6HRUwEfp
 FTV+fQ+S9yrf7NDm0zHDjoGsJACo53WQDRRCEIDKIhvrv/aWtZl+w1PJinQyqz1z+gh9
 M79aCKCCrSWxwJA6/g2LHsSvIXFrDCu3Tnu0TlZ1ieQV3oyVWhj0HmDcOh7sqeqlGpCk
 fKYMHuC2qqtaF4h4gvEzINeRW9s6JlztrNEID7Gjo8L/Uw84Mc3tfOZCi0zDSdCUd0rp
 ABAg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references:mime-version:content-transfer-encoding;
 bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=;
 b=mb1l57QQfTdbklIqsDkSWNR77EcX+iAQSb3xsZd+/RZUBx4PY0mxygyGjRHAUYIZKG
 wY0fJbM065LqBCBI8VGSAn7g6Pk83rxyhdi75dAbgfvPthAzh4KUovl7B2OC9934c1N2
 pCvnP5GlX+jfeljiqgKlmownBCdrJ0CmIJorDl1OOif0JRfSWnmHOUAwkOObP5dSsw2Z
 BfXn2MYH+gUUVIZmJ1xB5LaCgejcJuMzgUssWN1GQK4pax4EGLhsO3sMF2NSW+x7k532
 HUjy3XViWHJfNTsSoAQ4d5EuZN+3o6jDAb1KL5ztdSy60T5r2LEAOEyT7yK4GVlRMfZr
 6Ofg==
X-Gm-Message-State: AOAM531X5tgLPxIq1FUHahh1HI+Ptk03nI1WDbyHLPkJLviYBMkh1W8j
 Xc4C27V5KgC0IjvJwaFFHYnnNjQeIFtW8g==
X-Google-Smtp-Source: 
 ABdhPJxZOldAOxbmDbmBbcG+/cHy/Z3PNSWRU55JhPE+ZYjn/P9C2pkVeNhEcVnBDOtvT5yTaUrhXA==
X-Received: by 2002:a5d:4145:: with SMTP id c5mr986131wrq.18.1598611306982;
 Fri, 28 Aug 2020 03:41:46 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.43
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:46 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 7/9] tests/performance: Add nightly tests
Date: Fri, 28 Aug 2020 12:41:00 +0200
Message-Id: <20200828104102.4490-8-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
MIME-Version: 1.0
Received-SPF: pass client-ip=2a00:1450:4864:20::433;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x433.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

A nightly performance testing system to monitor any change in QEMU
performance across seventeen different targets.

The system includes eight different benchmarks to provide a variety
of testing workloads.

dijkstra_double:
Find the shortest path between the source node and all other nodes
using Dijkstra’s algorithm. The graph contains n nodes where all nxn
distances are double values. The value of n can be specified using
the -n flag. The default value is 2000.

dijkstra_int32:
Find the shortest path between the source node and all other nodes
using Dijkstra’s algorithm. The graph contains n nodes where all nxn
distances are int32 values. The value of n can be specified using
the -n flag. The default value is 2000.

matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated
double numbers from 0 to 100. The value of n is passed as an argument
with the -n flag. The default value is 200.

matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated
integer numbers from 0 to 100. The value of n is passed as an
argument with the -n flag. The default value is 200.

qsort_double:
Quick sort of an array of n randomly generated double numbers from 0
to 1000. The value of n is passed as an argument with the -n flag.
The default value is 300000.

qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0
to 50000000. The value of n is passed as an argument with the -n
flag.The default value is 300000.

qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8
(including null terminating character). The sort process is repeated
n number of times. The value of n is passed as an argument with the
-n flag. The default value is 20.

search_string:
Search for the occurrence of a small string in a much larger random
string (“needle in a hay”). The search process is repeated n number
of times and each time, a different large random string (“hay”) is
generated. The value of n can be specified using the -n flag. The
default value is 20.

Syntax:
    nightly_tests_core.py [-h] [-r REF]
    Optional arguments:
        -h, --help            Show this help message and exit
        -r REF, --reference REF
                        Reference QEMU version - Default is v5.1.0
    Example of usage:
        nightly_tests_core.py -r v5.1.0 2>log.txt

The following report includes detailed setup and execution details
of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
---
 tests/performance/nightly-tests/README.md     | 243 +++++
 .../source/dijkstra_double/dijkstra_double.c  | 194 ++++
 .../source/dijkstra_int32/dijkstra_int32.c    | 192 ++++
 .../source/matmult_double/matmult_double.c    | 123 +++
 .../source/matmult_int32/matmult_int32.c      | 121 +++
 .../source/qsort_double/qsort_double.c        | 104 ++
 .../source/qsort_int32/qsort_int32.c          | 103 ++
 .../source/qsort_string/qsort_string.c        | 122 +++
 .../source/search_string/search_string.c      | 110 +++
 .../scripts/nightly_tests_core.py             | 920 ++++++++++++++++++
 .../scripts/run_nightly_tests.py              | 135 +++
 .../nightly-tests/scripts/send_email.py       |  56 ++
 12 files changed, 2423 insertions(+)
 create mode 100644 tests/performance/nightly-tests/README.md
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
 create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
 create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py
 create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py
 create mode 100644 tests/performance/nightly-tests/scripts/send_email.py

diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/nightly-tests/README.md
new file mode 100644
index 0000000000..6db3b351b3
--- /dev/null
+++ b/tests/performance/nightly-tests/README.md
@@ -0,0 +1,243 @@
+### QEMU Nightly Tests
+
+**Required settings:**
+
+Update the `GMAIL_USER` object in `send_email.py` with your credentials.
+
+For more details on how the system works, please check the [eighth report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series.
+
+**Running the System:**
+
+The default reference version is v5.1.0. To specify a custom version, please use the `-r, --reference` flag.
+
+```bash
+./run_nightly_tests.py
+```
+
+**Output:**
+
+```
+Host CPU         : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
+Host Memory      : 15.49 GB
+
+Start Time (UTC) : 2020-08-25 21:30:01
+End Time (UTC)   : 2020-08-25 22:02:37
+Execution Time   : 0:32:35.896990
+
+Status           : SUCCESS
+
+Note:
+Changes denoted by '-----' are less than 0.01%.
+
+--------------------------------------------------------
+            SUMMARY REPORT - COMMIT d1a2b51f
+--------------------------------------------------------
+                    AVERAGE RESULTS
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 158 355 274       -----     +1.693%
+alpha              1 914 967 171       -----     +3.524%
+arm                8 076 402 940       -----     +2.304%
+hppa               4 261 685 987     -0.182%     +3.164%
+m68k               2 690 273 044       -----     +7.131%
+mips               1 862 033 667       -----     +2.494%
+mipsel             2 008 211 069       -----     +2.674%
+mips64             1 918 635 565       -----     +2.818%
+mips64el           2 051 565 677       -----     +3.026%
+ppc                2 480 141 217       -----     +3.107%
+ppc64              2 576 713 959       -----     +3.143%
+ppc64le            2 558 853 539       -----     +3.173%
+riscv64            1 406 704 050       -----      +2.65%
+s390x              3 158 140 046       -----     +3.118%
+sh4                2 364 449 748       -----      +3.33%
+sparc64            3 318 544 783       -----     +3.851%
+x86_64             1 775 844 158       -----     +2.156%
+--------------------------------------------------------
+
+                   DETAILED RESULTS
+--------------------------------------------------------
+Test Program: dijkstra_double
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            3 062 583 464       -----     +1.424%
+alpha              3 191 864 698       -----     +3.696%
+arm               16 357 157 526       -----     +2.347%
+hppa               7 228 376 315     -0.139%     +3.086%
+m68k               4 294 016 587       -----     +9.692%
+mips               3 051 419 166       -----     +2.427%
+mipsel             3 231 509 618       -----     +2.869%
+mips64             3 245 837 754       -----     +2.596%
+mips64el           3 414 195 796       -----     +3.021%
+ppc                4 914 520 972     -0.041%      +4.74%
+ppc64              5 098 154 311       -----     +4.565%
+ppc64le            5 082 419 054       -----      +4.58%
+riscv64            2 192 294 915       -----     +1.955%
+s390x              4 584 503 977       -----     +2.896%
+sh4                3 949 036 447       -----     +3.464%
+sparc64            4 586 203 546       -----     +4.237%
+x86_64             2 484 092 105       -----      +1.75%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: dijkstra_int32
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 210 194 577       -----     +1.493%
+alpha              1 494 133 274       -----      +2.15%
+arm                8 262 935 967       -----     +2.665%
+hppa               5 207 318 306       -----     +3.047%
+m68k               1 725 856 962       -----     +2.527%
+mips               1 495 227 032       -----     +1.492%
+mipsel             1 497 147 869       -----     +1.479%
+mips64             1 715 388 570       -----     +1.892%
+mips64el           1 695 276 864       -----     +1.913%
+ppc                2 014 557 389       -----     +1.819%
+ppc64              2 206 267 901       -----     +2.139%
+ppc64le            2 197 998 781       -----     +2.146%
+riscv64            1 354 912 745       -----     +2.396%
+s390x              2 916 247 062       -----     +1.241%
+sh4                1 990 532 533       -----     +2.669%
+sparc64            2 872 231 051       -----     +3.758%
+x86_64             1 553 981 241       -----      +2.12%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: matmult_double
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            1 412 273 223       -----     +0.302%
+alpha              3 233 991 649       -----     +7.473%
+arm                8 545 173 979       -----     +1.088%
+hppa               3 483 597 802     -1.267%     +4.468%
+m68k               3 919 065 529       -----    +18.431%
+mips               2 344 774 894       -----     +4.091%
+mipsel             3 329 886 464       -----     +5.177%
+mips64             2 359 046 988       -----     +4.076%
+mips64el           3 343 664 785       -----     +5.167%
+ppc                3 209 457 051       -----     +3.246%
+ppc64              3 287 503 981       -----     +3.173%
+ppc64le            3 287 189 065       -----     +3.173%
+riscv64            1 221 603 682       -----     +0.277%
+s390x              2 874 199 923       -----     +5.827%
+sh4                3 543 943 634       -----     +6.416%
+sparc64            3 426 150 004       -----     +7.139%
+x86_64             1 248 917 276       -----     +0.322%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: matmult_int32
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64              598 681 621       -----     +0.585%
+alpha                372 437 418       -----     +0.677%
+arm                  746 583 193       -----     +1.462%
+hppa                 674 278 359       -----     +1.183%
+m68k                 410 495 553       -----       +0.9%
+mips                 499 698 837       -----     +0.531%
+mipsel               499 500 429       -----     +0.497%
+mips64               481 554 664       -----     +0.599%
+mips64el             465 057 054       -----     +0.619%
+ppc                  341 334 603       -----     +0.944%
+ppc64                393 796 203       -----     +0.966%
+ppc64le              393 977 298       -----     +0.965%
+riscv64              351 709 769       -----     +0.785%
+s390x                494 427 384       -----     +0.599%
+sh4                  402 668 444       -----     +0.899%
+sparc64              495 952 959       -----     +1.192%
+x86_64               402 928 461       -----     +0.833%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_double
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 709 683 624       -----     +2.417%
+alpha              1 969 460 172       -----      +3.68%
+arm                8 322 998 390       -----     +2.587%
+hppa               3 188 301 995     -0.047%       +2.9%
+m68k               4 953 930 065       -----    +15.153%
+mips               2 123 919 587       -----     +3.055%
+mipsel             2 124 212 187       -----     +3.048%
+mips64             1 999 047 826       -----     +3.405%
+mips64el           1 996 426 772       -----     +3.409%
+ppc                2 819 267 902     -0.021%     +5.435%
+ppc64              2 768 186 548       -----     +5.513%
+ppc64le            2 724 803 772     -0.011%     +5.603%
+riscv64            1 638 328 937       -----     +4.021%
+s390x              2 519 081 708       -----     +3.362%
+sh4                2 595 545 154       -----     +2.994%
+sparc64            3 988 986 587       -----     +2.747%
+x86_64             2 033 468 588       -----     +3.234%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_int32
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 193 392 565       -----     +2.916%
+alpha              1 521 291 933       -----     +4.193%
+arm                3 465 445 043       -----     +2.756%
+hppa               2 280 034 340       -----     +3.821%
+m68k               1 843 189 041       -----     +3.583%
+mips               1 558 024 873       -----     +3.863%
+mipsel             1 560 583 980       -----     +3.846%
+mips64             1 563 415 749       -----     +4.412%
+mips64el           1 542 677 320       -----     +4.474%
+ppc                1 728 698 880       -----     +3.665%
+ppc64              1 842 444 545       -----     +3.555%
+ppc64le            1 791 822 067       -----     +3.661%
+riscv64            1 348 866 430       -----     +4.654%
+s390x              2 184 073 151       -----     +3.319%
+sh4                1 946 492 539       -----     +3.624%
+sparc64            3 452 215 585       -----     +2.937%
+x86_64             1 813 544 414       -----     +3.537%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_string
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 592 218 418       -----     +2.468%
+alpha              1 855 834 626       -----     +3.487%
+arm                7 347 721 165       -----     +2.682%
+hppa               4 758 753 926       -----     +3.543%
+m68k               2 376 811 462       -----     +3.567%
+mips               2 166 608 045       -----     +2.532%
+mipsel             2 163 392 541       -----     +2.528%
+mips64             2 029 251 969       -----     +3.117%
+mips64el           2 011 628 621       -----     +3.145%
+ppc                2 492 942 463       -----     +2.673%
+ppc64              2 464 702 554       -----     +2.488%
+ppc64le            2 445 253 307       -----     +2.505%
+riscv64            1 625 053 328       -----     +3.953%
+s390x              4 194 608 798       -----     +6.623%
+sh4                2 164 142 539       -----     +3.166%
+sparc64            4 299 516 539       -----     +4.065%
+x86_64             2 940 456 780       -----     +2.649%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: search_string
+--------------------------------------------------------
+Target              Instructions      Latest      v5.1.0
+----------  --------------------  ----------  ----------
+aarch64            2 487 814 704       -----      +1.94%
+alpha              1 680 723 605       -----     +2.835%
+arm               11 563 208 260       -----     +2.848%
+hppa               7 272 826 858       -----     +3.263%
+m68k               1 998 819 159       -----     +3.198%
+mips               1 656 596 909       -----     +1.959%
+mipsel             1 659 455 464       -----     +1.947%
+mips64             1 955 541 001       -----     +2.447%
+mips64el           1 943 598 207       -----     +2.462%
+ppc                2 320 350 477       -----     +2.332%
+ppc64              2 552 655 634       -----     +2.742%
+ppc64le            2 547 364 971       -----     +2.748%
+riscv64            1 520 862 601       -----     +3.159%
+s390x              5 497 978 370       -----     +1.078%
+sh4                2 323 236 696       -----      +3.41%
+sparc64            3 427 101 999       -----      +4.73%
+x86_64             1 729 364 402       -----     +2.806%
+--------------------------------------------------------
+```
diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
new file mode 100644
index 0000000000..9c0bb804ac
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
@@ -0,0 +1,194 @@
+/*
+ *  Source file of a benchmark program involving calculations of the
+ *  shortest distances between a source node and all other nodes in a
+ *  graph of n nodes in which all nxn distances are defined as "double".
+ *  The number n can be given via command line, and the default is 2000.
+ *  The algorithm used is Dijsktra's.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <float.h>
+#include <limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_NODE_COUNT      2000
+#define MIN_NODE_COUNT          3
+#define MAX_NODE_COUNT          10000
+
+
+int32_t closest_index(int32_t count, double *distances, bool *flags)
+{
+    int32_t closest;
+    double minimum = DBL_MAX;
+
+    for (size_t i = 0; i < count; i++) {
+        if (flags[i] == false && distances[i] <= minimum) {
+            closest = i;
+            minimum = distances[i];
+        }
+    }
+
+    return closest;
+}
+
+/**
+ * Calculate the shortest distances from the source node using Dijkstra method.
+ * @param (out) distances  An array of shortest distances from the source node.
+ * @param (out) via  An array of nodes needed to be taken as the the last
+ *                   before destination, for each destination.
+ * @param (out) eccent  Eccentricity of the source node.
+ * @param (in) count  The number of nodes.
+ * @param (in) source  Source node.
+ * @param (in) matrix  Distance matrix.
+ */
+void find_shortest_distances(double *distances, int32_t *via, double *eccent,
+                             int32_t count, int32_t source, double **matrix)
+{
+    bool *flags;
+
+    flags = (bool *)malloc(count * sizeof(bool));
+
+    for (size_t i = 0; i < count; i++) {
+        distances[i] = DBL_MAX;
+        flags[i] = false;
+    }
+
+    distances[source] = 0.0;
+    via[source] = source;
+
+    for (size_t i = 0; i < count - 1; i++) {
+        int32_t closest = closest_index(count, distances, flags);
+        flags[closest] = true;
+        for (size_t j = 0; j < count; j++) {
+            if ((!flags[j]) &&
+                    (matrix[closest][j]) &&
+                    (distances[closest] != DBL_MAX) &&
+                    (distances[j] > distances[closest] + matrix[closest][j])) {
+                distances[j] = distances[closest] + matrix[closest][j];
+                via[j] = closest;
+            }
+        }
+    }
+
+    *eccent = 0;
+    for (size_t i = 0; i < count; i++) {
+        if (*eccent < distances[i]) {
+            *eccent = distances[i];
+        }
+    }
+
+    free(flags);
+}
+
+
+void main(int argc, char *argv[])
+{
+    double **distance_matrix;
+    double *shortest_distances;
+    int32_t *via_node;
+    int32_t node_count = DEFAULT_NODE_COUNT;
+    int32_t source_node = 0;
+    double node_eccentricity = 0.0;
+    double range_factor = 999.0 / (double)(RAND_MAX);
+    int32_t option;
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_node_count = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_node_count == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_node_count < MIN_NODE_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_NODE_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_node_count > MAX_NODE_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_NODE_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            node_count = user_node_count;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for all matrixes */
+    distance_matrix = (double **)malloc(node_count * sizeof(double *));
+    for (size_t i = 0; i < node_count; i++) {
+        distance_matrix[i] = (double *)malloc(node_count * sizeof(double));
+    }
+    shortest_distances = (double *)malloc(node_count * sizeof(double));
+    via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
+
+    /* Initialize helper arrays and populate distance_matrix */
+    srand(1);
+    for (size_t i = 0; i < node_count; i++) {
+        shortest_distances[i] = 0.0;
+        via_node[i] = -1;
+        distance_matrix[i][i] = 0.0;
+    }
+    for (size_t i = 0; i < node_count; i++) {
+        for (size_t j = i + 1; j < node_count; j++) {
+            distance_matrix[i][j] = 1.0 + range_factor * (double)rand();
+            distance_matrix[j][i] = distance_matrix[i][j];
+        }
+    }
+
+    find_shortest_distances(shortest_distances, via_node, &node_eccentricity,
+                            node_count, source_node, distance_matrix);
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf(" Distance matrix (top left part):\n");
+    for (size_t i = 0; i < 3; i++) {
+        for (size_t j = 0; j < 3; j++) {
+            printf("    %7.2f", distance_matrix[i][j]);
+        }
+        printf("\n");
+    }
+    printf(" Source: %d (eccentricity: %f)\n",
+           source_node, node_eccentricity);
+    printf(" Destination   Distance   Via Node\n");
+    for (size_t i = 0; i < 3; i++) {
+        printf("  %5d       %7.2f       %4d\n",
+               i, shortest_distances[i], via_node[i]);
+    }
+
+    /* Free all previously allocated space */
+    for (size_t i = 0; i < node_count; i++) {
+        free(distance_matrix[i]);
+    }
+    free(distance_matrix);
+    free(shortest_distances);
+    free(via_node);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
new file mode 100644
index 0000000000..2663cde943
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
@@ -0,0 +1,192 @@
+/*
+ *  Source file of a benchmark program involving calculations of the
+ *  shortest distances between a source node and all other nodes in a
+ *  graph of n nodes in which all nxn distances are defined as "int32".
+ *  The number n can be given via command line, and the default is 2000.
+ *  The algorithm used is Dijsktra's.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_NODE_COUNT      2000
+#define MIN_NODE_COUNT          3
+#define MAX_NODE_COUNT          10000
+
+
+int32_t closest_index(int32_t count, int32_t *distances, bool *flags)
+{
+    int32_t closest;
+    int32_t minimum = INT_MAX;
+
+    for (size_t i = 0; i < count; i++) {
+        if (flags[i] == false && distances[i] <= minimum) {
+            closest = i;
+            minimum = distances[i];
+        }
+    }
+
+    return closest;
+}
+
+/**
+ * Calculate the shortest distances from the source node using Dijkstra method.
+ * @param (out) distances  An array of shortest distances from the source node.
+ * @param (out) via  An array of nodes needed to be taken as the the last
+ *                   before destination, for each destination.
+ * @param (out) eccent  Eccentricity of the source node.
+ * @param (in) count  The number of nodes.
+ * @param (in) source  Source node.
+ * @param (in) matrix  Distance matrix.
+ */
+void find_shortest_distances(int32_t *distances, int32_t *via, int32_t *eccent,
+                             int32_t count, int32_t source, int32_t **matrix)
+{
+    bool *flags;
+
+    flags = (bool *)malloc(count * sizeof(bool));
+
+    for (size_t i = 0; i < count; i++) {
+        distances[i] = INT_MAX;
+        flags[i] = false;
+    }
+
+    distances[source] = 0;
+    via[source] = source;
+
+    for (size_t i = 0; i < count - 1; i++) {
+        int32_t closest = closest_index(count, distances, flags);
+        flags[closest] = true;
+        for (size_t j = 0; j < count; j++) {
+            if ((!flags[j]) &&
+                    (matrix[closest][j]) &&
+                    (distances[closest] != INT_MAX) &&
+                    (distances[j] > distances[closest] + matrix[closest][j])) {
+                distances[j] = distances[closest] + matrix[closest][j];
+                via[j] = closest;
+            }
+        }
+    }
+
+    *eccent = 0;
+    for (size_t i = 0; i < count; i++) {
+        if (*eccent < distances[i]) {
+            *eccent = distances[i];
+        }
+    }
+
+    free(flags);
+}
+
+
+void main(int argc, char *argv[])
+{
+    int32_t **distance_matrix;
+    int32_t *shortest_distances;
+    int32_t *via_node;
+    int32_t node_count = DEFAULT_NODE_COUNT;
+    int32_t source_node = 0;
+    int32_t node_eccentricity = 0;
+    int32_t option;
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_node_count = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_node_count == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_node_count < MIN_NODE_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_NODE_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_node_count > MAX_NODE_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_NODE_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            node_count = user_node_count;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for all matrixes */
+    distance_matrix = (int32_t **)malloc(node_count * sizeof(int32_t *));
+    for (size_t i = 0; i < node_count; i++) {
+        distance_matrix[i] = (int32_t *)malloc(node_count * sizeof(int32_t));
+    }
+    shortest_distances = (int32_t *)malloc(node_count * sizeof(int32_t));
+    via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
+
+    /* Initialize helper arrays and populate distance_matrix */
+    srand(1);
+    for (size_t i = 0; i < node_count; i++) {
+        shortest_distances[i] = 0;
+        via_node[i] = -1;
+        distance_matrix[i][i] = 0;
+    }
+    for (size_t i = 0; i < node_count; i++) {
+        for (size_t j = i + 1; j < node_count; j++) {
+            distance_matrix[i][j] = 1 + (rand()) / (RAND_MAX / 999);
+            distance_matrix[j][i] = distance_matrix[i][j];
+        }
+    }
+
+    find_shortest_distances(shortest_distances, via_node, &node_eccentricity,
+                            node_count, source_node, distance_matrix);
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf(" Distance matrix (top left part):\n");
+    for (size_t i = 0; i < 3; i++) {
+        for (size_t j = 0; j < 3; j++) {
+            printf("    %6d", distance_matrix[i][j]);
+        }
+        printf("\n");
+    }
+    printf(" Source: %d (eccentricity: %d)\n",
+           source_node, node_eccentricity);
+    printf(" Destination   Distance   Via Node\n");
+    for (size_t i = 0; i < 3; i++) {
+        printf("  %5d          %3d        %4d\n",
+               i, shortest_distances[i], via_node[i]);
+    }
+
+    /* Free all previously allocated space */
+    for (size_t i = 0; i < node_count; i++) {
+        free(distance_matrix[i]);
+    }
+    free(distance_matrix);
+    free(shortest_distances);
+    free(via_node);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
new file mode 100644
index 0000000000..42bbb4717a
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
@@ -0,0 +1,123 @@
+/*
+ *  Source file of a benchmark program involving calculations of
+ *  a product of two matrixes nxn whose elements are "double". The
+ *  number n can be given via command line, and the default is 200.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_MATRIX_SIZE     200
+#define MIN_MATRIX_SIZE         2
+#define MAX_MATRIX_SIZE         200000
+
+void main(int argc, char *argv[])
+{
+    double **matrix_a;
+    double **matrix_b;
+    double **matrix_res;
+    size_t i;
+    size_t j;
+    size_t k;
+    int32_t matrix_size = DEFAULT_MATRIX_SIZE;
+    int32_t option;
+    double range_factor = 100.0 / (double)(RAND_MAX);
+
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_matrix_size = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_matrix_size == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_matrix_size < MIN_MATRIX_SIZE) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_MATRIX_SIZE);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_matrix_size > MAX_MATRIX_SIZE) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_MATRIX_SIZE);
+                exit(EXIT_FAILURE);
+            }
+            matrix_size = user_matrix_size;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for all matrixes */
+    matrix_a = (double **)malloc(matrix_size * sizeof(double *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_a[i] = (double *)malloc(matrix_size * sizeof(double));
+    }
+    matrix_b = (double **)malloc(matrix_size * sizeof(double *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_b[i] = (double *)malloc(matrix_size * sizeof(double));
+    }
+    matrix_res = (double **)malloc(matrix_size * sizeof(double *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_res[i] = (double *)malloc(matrix_size * sizeof(double));
+    }
+
+    /* Populate matrix_a and matrix_b with random numbers */
+    srand(1);
+    for (i = 0; i < matrix_size; i++) {
+        for (j = 0; j < matrix_size; j++) {
+            matrix_a[i][j] = range_factor * (double)rand();
+            matrix_b[i][j] = range_factor * (double)rand();
+        }
+    }
+
+    /* Calculate the product of two matrixes */
+    for (i = 0; i < matrix_size; i++) {
+        for (j = 0; j < matrix_size; j++) {
+            matrix_res[i][j] = 0.0;
+            for (k = 0; k < matrix_size; k++) {
+                matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
+            }
+        }
+    }
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]);
+    printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]);
+
+    /* Free all previously allocated space */
+    for (i = 0; i < matrix_size; i++) {
+        free(matrix_a[i]);
+        free(matrix_b[i]);
+        free(matrix_res[i]);
+    }
+    free(matrix_a);
+    free(matrix_b);
+    free(matrix_res);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
new file mode 100644
index 0000000000..29a6eb000d
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
@@ -0,0 +1,121 @@
+/*
+ *  Source file of a benchmark program involving calculations of
+ *  a product of two matrixes nxn whose elements are "int32_t". The
+ *  number n can be given via command line, and the default is 200.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_MATRIX_SIZE     200
+#define MIN_MATRIX_SIZE         2
+#define MAX_MATRIX_SIZE         200000
+
+void main(int argc, char *argv[])
+{
+    int32_t **matrix_a;
+    int32_t **matrix_b;
+    int32_t **matrix_res;
+    size_t i;
+    size_t j;
+    size_t k;
+    int32_t matrix_size = DEFAULT_MATRIX_SIZE;
+    int32_t option;
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_matrix_size = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_matrix_size == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_matrix_size < MIN_MATRIX_SIZE) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_MATRIX_SIZE);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_matrix_size > MAX_MATRIX_SIZE) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_MATRIX_SIZE);
+                exit(EXIT_FAILURE);
+            }
+            matrix_size = user_matrix_size;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for all matrixes */
+    matrix_a = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_a[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+    }
+    matrix_b = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_b[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+    }
+    matrix_res = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+    for (i = 0; i < matrix_size; i++) {
+        matrix_res[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+    }
+
+    /* Populate matrix_a and matrix_b with random numbers */
+    srand(1);
+    for (i = 0; i < matrix_size; i++) {
+        for (j = 0; j < matrix_size; j++) {
+            matrix_a[i][j] = (rand()) / (RAND_MAX / 100);
+            matrix_b[i][j] = (rand()) / (RAND_MAX / 100);
+        }
+    }
+
+    /* Calculate the product of two matrixes */
+    for (i = 0; i < matrix_size; i++) {
+        for (j = 0; j < matrix_size; j++) {
+            matrix_res[i][j] = 0;
+            for (k = 0; k < matrix_size; k++) {
+                matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
+            }
+        }
+    }
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]);
+    printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]);
+
+    /* Free all previously allocated space */
+    for (i = 0; i < matrix_size; i++) {
+        free(matrix_a[i]);
+        free(matrix_b[i]);
+        free(matrix_res[i]);
+    }
+    free(matrix_a);
+    free(matrix_b);
+    free(matrix_res);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
new file mode 100644
index 0000000000..efc1b2eee1
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
@@ -0,0 +1,104 @@
+/*
+ *  Source file of a benchmark program involving sorting of an array
+ *  of length n whose elements are "double". The default value for n
+ *  is 300000, and it can be set via command line as well.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of elements in the array to be sorted */
+#define DEFAULT_ARRAY_LEN       300000
+#define MIN_ARRAY_LEN           3
+#define MAX_ARRAY_LEN           30000000
+
+/* Upper limit used for generation of random numbers */
+#define UPPER_LIMIT             1000.0
+
+/* Comparison function passed to qsort() */
+static int compare(const void *a, const void *b)
+{
+    if (*(const double *)a > *(const double *)b) {
+        return 1;
+    } else if (*(const double *)a < *(const double *)b) {
+        return -1;
+    }
+    return 0;
+}
+
+void main(int argc, char *argv[])
+{
+    double *array_to_be_sorted;
+    int32_t array_len = DEFAULT_ARRAY_LEN;
+    int32_t option;
+    double range_factor = UPPER_LIMIT / (double)(RAND_MAX);
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_array_len = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_array_len == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_array_len < MIN_ARRAY_LEN) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_ARRAY_LEN);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_array_len > MAX_ARRAY_LEN) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_ARRAY_LEN);
+                exit(EXIT_FAILURE);
+            }
+            array_len = user_array_len;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for the array */
+    array_to_be_sorted = (double *) malloc(array_len * sizeof(double));
+
+    /* Populate the_array with random numbers */
+    srand(1);
+    for (size_t i = 0; i < array_len; i++) {
+        array_to_be_sorted[i] = range_factor * (double)rand();
+    }
+
+    /* Sort the_array using qsort() */
+    qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]),
+          compare);
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf("%14.10f %14.10f %14.10f\n",
+           array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]);
+
+    /* Free all previously allocated space */
+    free(array_to_be_sorted);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
new file mode 100644
index 0000000000..76ca9c3490
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
@@ -0,0 +1,103 @@
+/*
+ *  Source file of a benchmark program involving sorting of an array
+ *  of length n whose elements are "int32_t". The default value for n
+ *  is 300000, and it can be set via command line as well.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of elements in the array to be sorted */
+#define DEFAULT_ARRAY_LEN       300000
+#define MIN_ARRAY_LEN           3
+#define MAX_ARRAY_LEN           30000000
+
+/* Upper limit used for generation of random numbers */
+#define UPPER_LIMIT             50000000
+
+/* Comparison function passed to qsort() */
+static int compare(const void *a, const void *b)
+{
+    if (*(const int32_t *)a > *(const int32_t *)b) {
+        return 1;
+    } else if (*(const int32_t *)a < *(const int32_t *)b) {
+        return -1;
+    }
+    return 0;
+}
+
+void main(int argc, char *argv[])
+{
+    int32_t *array_to_be_sorted;
+    int32_t array_len = DEFAULT_ARRAY_LEN;
+    int32_t option;
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_array_len = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_array_len == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_array_len < MIN_ARRAY_LEN) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_ARRAY_LEN);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_array_len > MAX_ARRAY_LEN) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_ARRAY_LEN);
+                exit(EXIT_FAILURE);
+            }
+            array_len = user_array_len;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /* Allocate the memory space for the array */
+    array_to_be_sorted = (int32_t *) malloc(array_len * sizeof(int32_t));
+
+    /* Populate the_array with random numbers */
+    srand(1);
+    for (size_t i = 0; i < array_len; i++) {
+        array_to_be_sorted[i] = (rand()) / (RAND_MAX / UPPER_LIMIT);
+    }
+
+    /* Sort the_array using qsort() */
+    qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]),
+          compare);
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf("%d %d %d\n",
+           array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]);
+
+    /* Free all previously allocated space */
+    free(array_to_be_sorted);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
new file mode 100644
index 0000000000..7d582b2dd0
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
@@ -0,0 +1,122 @@
+/*
+ *  Source file of a benchmark program involving sorting of an array
+ *  of 10000 random strings of length 8 (including terminating zero).
+ *  That sorting is repeated a number of times (default is 20 times),
+ *  and each time a different array of random strings is generated.
+ *  The number of repetitions can be set via command line.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <unistd.h>
+
+/* Length of an individual random string (including terminating zero) */
+#define RANDOM_STRING_LEN             8
+/* Number of elements of the array of random strings */
+#define NUMBER_OF_RANDOM_STRINGS      10000
+
+/* Number of repetitions to be performed each with different input */
+#define DEFAULT_REPETITION_COUNT      20
+#define MIN_REPETITION_COUNT          1
+#define MAX_REPETITION_COUNT          1000
+
+/* Structure that keeps an array of random strings to be sorted */
+struct StringStruct {
+    char chars[RANDOM_STRING_LEN];
+};
+
+/* Comparison function passed to qsort() */
+int compare_strings(const void *element1, const void *element2)
+{
+    int result;
+
+    result = strcmp((*((struct StringStruct *)element1)).chars,
+                    (*((struct StringStruct *)element2)).chars);
+
+    return (result < 0) ? -1 : ((result == 0) ? 0 : 1);
+}
+
+/* Generate a random string of given length and containing only small letters */
+static void gen_random_string(char *s, const int len)
+{
+    static const char letters[] = "abcdefghijklmnopqrstuvwxyz";
+
+    for (size_t i = 0; i < (len - 1); i++) {
+        s[i] = letters[rand() % (sizeof(letters) - 1)];
+    }
+
+    s[len - 1] = 0;
+}
+
+void main(int argc, char *argv[])
+{
+    struct StringStruct strings_to_be_sorted[NUMBER_OF_RANDOM_STRINGS];
+    int32_t repetition_count = DEFAULT_REPETITION_COUNT;
+    int32_t option;
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_repetition_count = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_repetition_count == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_repetition_count < MIN_REPETITION_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_REPETITION_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_repetition_count > MAX_REPETITION_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_REPETITION_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            repetition_count = user_repetition_count;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    srand(1);
+
+    for (size_t i = 0; i < repetition_count; ++i) {
+        /* Generate random strings, and, in turn, sort them */
+        for (size_t i = 0; i < NUMBER_OF_RANDOM_STRINGS; ++i) {
+            gen_random_string(strings_to_be_sorted[i].chars, RANDOM_STRING_LEN);
+        }
+        qsort(strings_to_be_sorted, NUMBER_OF_RANDOM_STRINGS,
+              sizeof(struct StringStruct), compare_strings);
+    }
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    for (size_t i = 0; i < 2; ++i) {
+        printf(" %s", strings_to_be_sorted[i].chars);
+    }
+    printf("\n");
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
new file mode 100644
index 0000000000..2827ea032e
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
@@ -0,0 +1,110 @@
+/*
+ *  Source file of a benchmark program that searches for the occurrence
+ *  of a small string in a much larger random string ("needle in a hay").
+ *  That searching is repeated a number of times (default is 20 times),
+ *  and each time a different large random string ("hay") is generated.
+ *  The number of repetitions can be set via command line.
+ *
+ *  This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ *  Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ *  Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <unistd.h>
+
+/* Length of a long string to be searched (including terminating zero) */
+#define HAYSTACK_LEN                  30000
+
+/* Number of repetitions to be performed each with different input */
+#define DEFAULT_REPETITION_COUNT      100
+#define MIN_REPETITION_COUNT          1
+#define MAX_REPETITION_COUNT          10000
+
+
+/* Generate a random string of given length and containing only small letters */
+static void gen_random_string(char *s, const int len)
+{
+    static const char letters[] = "abcdefghijklmnopqrstuvwxyz";
+
+    for (size_t i = 0; i < (len - 1); i++) {
+        s[i] = letters[rand() % (sizeof(letters) - 1)];
+    }
+
+    s[len - 1] = 0;
+}
+
+void main(int argc, char *argv[])
+{
+    char haystack[HAYSTACK_LEN];
+    const char needle[] = "aaa ";
+    char *found_needle;
+    int32_t found_cnt = 0;
+    int32_t not_found_cnt = 0;
+    int32_t repetition_count = DEFAULT_REPETITION_COUNT;
+    int32_t option;
+
+    printf("needle is %s, size %d\n", needle, sizeof(needle));
+
+    /* Parse command line options */
+    while ((option = getopt(argc, argv, "n:")) != -1) {
+        if (option == 'n') {
+            int32_t user_repetition_count = atoi(optarg);
+
+            /* Check if the value is a string or zero */
+            if (user_repetition_count == 0) {
+                fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is a negative number */
+            if (user_repetition_count < MIN_REPETITION_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+                                "number less than %d.\n", MIN_REPETITION_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            /* Check if the value is too large */
+            if (user_repetition_count > MAX_REPETITION_COUNT) {
+                fprintf(stderr, "Error ... Value for option '-n' cannot be "
+                                "more than %d.\n", MAX_REPETITION_COUNT);
+                exit(EXIT_FAILURE);
+            }
+            repetition_count = user_repetition_count;
+        } else {
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    srand(1);
+
+    for (size_t i = 0; i < repetition_count; ++i) {
+        /* Generate random hay, and, in turn, find a needle */
+        gen_random_string(haystack, HAYSTACK_LEN);
+        found_needle = strstr(haystack, needle);
+        if (found_needle != NULL) {
+            found_cnt++;
+        } else {
+            not_found_cnt++;
+        }
+    }
+
+    /* Control printing */
+    printf("CONTROL RESULT:\n");
+    printf(" Found %d times. Not found %d times.\n", found_cnt, not_found_cnt);
+}
diff --git a/tests/performance/nightly-tests/scripts/nightly_tests_core.py b/tests/performance/nightly-tests/scripts/nightly_tests_core.py
new file mode 100755
index 0000000000..da192c704a
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/nightly_tests_core.py
@@ -0,0 +1,920 @@
+#!/usr/bin/env python3
+
+"""
+Core script for performing nightly performance tests on QEMU.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import csv
+import datetime
+import glob
+import multiprocessing
+import os
+import pathlib
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from typing import Dict, List, Optional, Union
+
+
+def get_benchmark_name(benchmark_path: str) -> str:
+    """
+    Return the benchmark name given its path.
+
+    Parameters:
+    benchmarks_path (str): Absolute path to benchmark
+
+    Return:
+    (str): Benchmark name
+    """
+    benchmark_source_file = os.path.split(benchmark_path)[1]
+    return os.path.splitext(benchmark_source_file)[0]
+
+
+def get_benchmark_parent_dir(benchmark_path: str) -> str:
+    """
+    Return the benchmark parent directory name given the benchmark path.
+
+    Parameters:
+    benchmarks_path (str): Absolute path to benchmark
+
+    Return:
+    (str): Benchmark parent directory name
+    """
+    benchmark_parent_dir_path = os.path.split(benchmark_path)[0]
+    benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1]
+
+    return benchmark_parent_dir
+
+
+def get_executable_parent_dir_path(
+        benchmark_path: str, benchmarks_executables_dir_path: str) -> str:
+    """
+    Return the executables parent directory of a benchmark given its path.
+    This is the directory that includes all compiled executables for the
+    benchmark.
+
+    Parameters:
+    benchmarks_path (str): Absolute path to benchmark
+    benchmarks_executables_dir_path (str): Absolute path to the executables
+
+    Return:
+    (str): Executables parent directory path
+    """
+    benchmark_parent_dir_path = os.path.split(benchmark_path)[0]
+    benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1]
+    executable_parent_dir_path = os.path.join(benchmarks_executables_dir_path,
+                                              benchmark_parent_dir)
+
+    return executable_parent_dir_path
+
+
+def get_commit_hash(commit_tag: str, qemu_path: str) -> str:
+    """
+    Find commit hash given the Git commit tag.
+
+    Parameters:
+    commit_tag (str): Commit tag
+    qemu_path (str): Absolute path to QEMU
+
+    Returns:
+    (str): 8 digit commit hash
+    """
+
+    commit_hash = subprocess.run(["git",
+                                  "rev-parse",
+                                  commit_tag],
+                                 cwd=qemu_path,
+                                 stdout=subprocess.PIPE,
+                                 check=False)
+    if commit_hash.returncode:
+        clean_exit(qemu_path,
+                   "Failed to find the commit hash of {}.".format(commit_tag))
+
+    return commit_hash.stdout.decode("utf-8")[:8]
+
+
+def git_checkout(commit: str, qemu_path: str) -> None:
+    """
+    Checkout a given Git commit.
+    Also pull the latest changes from origin/master if the commit is "master".
+
+    Parameters:
+    commit (str): Commit hash or tag
+    qemu_path (str): Absolute path to QEMU
+    """
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Checking out {}".format(commit), file=sys.stderr, flush=True)
+
+    checkout_commit = subprocess.run(["git",
+                                      "checkout",
+                                      commit],
+                                     cwd=qemu_path,
+                                     stdout=subprocess.DEVNULL,
+                                     stderr=subprocess.PIPE,
+                                     check=False)
+    if checkout_commit.returncode:
+        clean_exit(qemu_path, checkout_commit.stderr.decode("utf-8"))
+
+    if commit == "master":
+        print(datetime.datetime.utcnow().isoformat(),
+              "- Pulling the latest changes from QEMU master",
+              file=sys.stderr, flush=True)
+        # Try pulling the latest changes.
+        # Limit the number of failed trials to 10.
+        failure_count, failure_limit = 0, 10
+        while True:
+            pull_latest = subprocess.run(["git",
+                                          "pull",
+                                          "origin",
+                                          "master"],
+                                         cwd=qemu_path,
+                                         stdout=subprocess.DEVNULL,
+                                         check=False)
+            if pull_latest.returncode:
+                failure_count += 1
+                if failure_count == failure_limit:
+                    print(datetime.datetime.utcnow().isoformat(),
+                          "- Trial {}/{}: Failed to pull QEMU".format(
+                              failure_count, failure_limit),
+                          file=sys.stderr, flush=True)
+                    clean_exit(qemu_path, "")
+                else:
+                    print(datetime.datetime.utcnow().isoformat(),
+                          "- Trial {}/{}: Failed to pull QEMU"
+                          " ... retrying again in a minute!".format(
+                              failure_count, failure_limit),
+                          file=sys.stderr, flush=True)
+                    time.sleep(60)
+            else:
+                break
+
+
+def git_clone(qemu_path: str) -> None:
+    """
+    Clone QEMU from Git.
+
+    Parameters:
+    qemu_path (str): Absolute path to clone the QEMU repo to
+    """
+    # Try cloning QEMU.
+    # Limit the number of failed trials to 10.
+    failure_count, failure_limit = 0, 10
+    while True:
+        clone_qemu = subprocess.run(["git",
+                                     "clone",
+                                     "https://git.qemu.org/git/qemu.git",
+                                     qemu_path],
+                                    check=False)
+        if clone_qemu.returncode:
+            failure_count += 1
+            if failure_count == failure_limit:
+                print(datetime.datetime.utcnow().isoformat(),
+                      "- Trial {}/{}: Failed to clone QEMU".format(
+                          failure_count, failure_limit),
+                      file=sys.stderr, flush=True)
+                clean_exit(qemu_path, "")
+            else:
+                print(datetime.datetime.utcnow().isoformat(),
+                      "- Trial {}/{}: Failed to clone QEMU"
+                      " ... retrying again in a minute!".format(
+                          failure_count, failure_limit),
+                      file=sys.stderr, flush=True)
+                time.sleep(60)
+        else:
+            break
+
+
+def build_qemu(qemu_path: str, git_tag: str, targets: List[str]) -> None:
+    """
+    Checkout the Git tag then configure and build QEMU.
+
+    Parameters:
+    qemu_path (str): Absolute path to QEMU
+    git_tag (str): Git tag to checkout before building
+    targets (List[str]): List of targets to configure QEMU for
+    """
+
+    # Clean the QEMU build path
+    qemu_build_path = os.path.join(qemu_path, "build-gcc")
+    if os.path.isdir(qemu_build_path):
+        shutil.rmtree(qemu_build_path)
+    os.mkdir(qemu_build_path)
+
+    git_checkout(git_tag, qemu_path)
+
+    # Specify target list for configuring QEMU
+    target_list = ["{}-linux-user".format(target) for target in targets]
+
+    # Configure QEMU
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Running 'configure' for {}".format(git_tag),
+          file=sys.stderr, flush=True)
+    configure = subprocess.run(["../configure",
+                                "--disable-system",
+                                "--disable-tools",
+                                "--target-list={}".
+                                format(",".join(target_list))],
+                               cwd=qemu_build_path,
+                               stdout=subprocess.DEVNULL,
+                               stderr=subprocess.PIPE,
+                               check=False)
+    if configure.returncode:
+        clean_exit(qemu_path, configure.stderr.decode("utf-8"))
+
+    # Run "make -j$(nproc)"
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Running 'make' for {}".format(git_tag), file=sys.stderr,
+          flush=True)
+    make = subprocess.run(["make",
+                           "-j",
+                           str(multiprocessing.cpu_count())],
+                          cwd=qemu_build_path,
+                          stdout=subprocess.DEVNULL,
+                          stderr=subprocess.PIPE,
+                          check=False)
+    if make.returncode:
+        clean_exit(qemu_path, make.stderr.decode("utf-8"))
+
+
+def compile_target(benchmark_path: str, compiled_benchmark_path: str,
+                   target_compiler: str) -> None:
+    """
+    Compile a benchmark using the provided cross compiler.
+
+    Parameters:
+    benchmarks_path (str): Absolute path to benchmark
+    compiled_benchmark_path (str): Path to the output executable
+    target_compiler (str): Cross compiler
+    """
+    compile_benchmark = subprocess.run([target_compiler,
+                                        "-O2",
+                                        "-static",
+                                        "-w",
+                                        benchmark_path,
+                                        "-o",
+                                        compiled_benchmark_path],
+                                       check=False)
+    if compile_benchmark.returncode:
+        sys.exit("Compilation of {} failed".format(
+            os.path.split(compiled_benchmark_path)[1]))
+
+
+def measure_instructions(
+        benchmark_path: str, benchmarks_executables_dir_path: str,
+        qemu_path: str, targets: List[str]) -> List[List[Union[str, int]]]:
+    """
+    Measure the number of instructions when running an program with QEMU.
+
+    Parameters:
+    benchmarks_path (str): Absolute path to benchmark
+    benchmarks_executables_dir_path (str): Absolute path to the executables
+    qemu_path (str): Absolute path to QEMU
+    targets (List[str]): List of QEMU targets
+
+    Returns:
+    (List[List[Union[str, int]]]): [[target_name, instructions],[...],...]
+    """
+
+    benchmark_name = get_benchmark_name(benchmark_path)
+    executable_parent_dir_path = get_executable_parent_dir_path(
+        benchmark_path, benchmarks_executables_dir_path)
+    qemu_build_path = os.path.join(qemu_path, "build-gcc")
+
+    instructions: List[List[Union[str, int]]] = []
+
+    for target in targets:
+        executable_path = os.path.join(
+            executable_parent_dir_path, "{}-{}".format(benchmark_name, target))
+
+        qemu_exe_path = os.path.join(qemu_build_path,
+                                     "{}-linux-user".format(target),
+                                     "qemu-{}".format(target))
+
+        with tempfile.NamedTemporaryFile() as tmpfile:
+            run_callgrind = subprocess.run(["valgrind",
+                                            "--tool=callgrind",
+                                            "--callgrind-out-file={}".format(
+                                                tmpfile.name),
+                                            qemu_exe_path,
+                                            executable_path],
+                                           stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.PIPE,
+                                           check=False)
+        if run_callgrind.returncode == 1:
+            clean_exit(qemu_path, run_callgrind.stderr.decode("utf-8"))
+
+        callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n")
+        instructions.append([target, int(callgrind_output[8].split(" ")[-1])])
+
+    return instructions
+
+
+def measure_master_instructions(
+        reference_version_path: str, reference_commit_hash: str,
+        latest_version_path: str, benchmark_path: str,
+        benchmarks_executables_dir_path: str, qemu_path: str,
+        targets: List[str]) -> List[List[Union[str, int]]]:
+    """
+    Measure the latest QEMU "master" instructions and also append the latest
+    instructions and reference version instructions to the instructions list.
+
+    Parameters:
+    reference_version_path (str): Absolute path to reference version results
+    reference_commit_hash (str): Git hash of the reference version
+    latest_version_path (str): Absolute path to the latest version results
+    benchmark_path (str): Absolute path to benchmark
+    benchmarks_executables_dir_path (str):
+                            Absolute path to the executables of the benchmark
+    qemu_path (str): Absolute path to QEMU
+    targets (List[str]): List of QEMU targets
+
+
+    Return:
+    (List[List[Union[str, int]]]):
+            [[target_name, instructions, comparison_instructions],[...],...]
+            comparsion_instructions: *[latest, reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    """
+    benchmark_name = get_benchmark_name(benchmark_path)
+
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Measuring instructions for master - {}".format(benchmark_name),
+          file=sys.stderr, flush=True)
+
+    instructions = measure_instructions(
+        benchmark_path, benchmarks_executables_dir_path, qemu_path, targets)
+
+    reference_result = "{}-{}-results.csv".format(
+        reference_commit_hash, benchmark_name)
+    reference_result_path = os.path.join(
+        reference_version_path, reference_result)
+
+    # Find if this benchmark has a record in the latest results
+    latest_result = ""
+    latest_results = os.listdir(latest_version_path)
+    for result in latest_results:
+        if result.split("-")[1] == benchmark_name:
+            latest_result = result
+            break
+
+    # Append instructions from latest version if available
+    if latest_result != "":
+        latest_result_path = os.path.join(latest_version_path, latest_result)
+        with open(latest_result_path, "r") as file:
+            file.readline()
+            for target_instructions in instructions:
+                target_instructions.append(
+                    int(file.readline().split('"')[1].replace(",", "")))
+        # Delete the latest results. The directory will contain the new latest
+        # when the new "master" results are stored later.
+        os.unlink(latest_result_path)
+
+    # Append instructions from reference version
+    with open(reference_result_path, "r") as file:
+        file.readline()
+        for target_instructions in instructions:
+            target_instructions.append(
+                int(file.readline().split('"')[1].replace(",", "")))
+
+    return instructions
+
+
+def calculate_percentage(old_instructions: int, new_instructions: int) -> str:
+    """
+    Calculate the change in percentage between two instruction counts
+
+    Parameters:
+    old_instructions (int): Old number
+    new_instructions (int): New number
+
+    Return:
+    (str): [+|-][change][%] or "-----" in case of 0.01% change
+    """
+    percentage = round(((new_instructions - old_instructions) /
+                        old_instructions) * 100, 3)
+    return format_percentage(percentage)
+
+
+def format_percentage(percentage: float) -> str:
+    """
+    Format the percentage value to add +|- and %.
+
+    Parameters:
+    percentage (float): Percentage
+
+    Returns:
+    (str): Formatted percentage string
+    """
+    if abs(percentage) <= 0.01:
+        return "-----"
+    return "+" + str(percentage) + "%" if percentage > 0 \
+        else str(percentage) + "%"
+
+
+def calculate_change(instructions: List[List[Union[str, int]]]) -> None:
+    """
+    Calculate the change in the recorded instructions for master compared to
+    latest results and reference version results.
+
+    Parameters:
+    instructions (List[List[Union[str, int]]]):
+            [[target_name, instructions, comparison_instructions],[...],...]
+            comparsion_instructions: *[latest, reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    """
+    for target_instructions in instructions:
+        target_instructions[-1] = calculate_percentage(
+            int(target_instructions[-1]), int(target_instructions[1]))
+        # If latest instructions exists
+        if len(target_instructions) == 4:
+            target_instructions[-2] = calculate_percentage(
+                int(target_instructions[-2]), int(target_instructions[1]))
+
+
+def calculate_average(results: List[List[List[Union[str, int]]]],
+                      targets: List[str],
+                      num_benchmarks: int) -> List[List[Union[str, int]]]:
+    """
+    Calculate the average results for each target for all benchmarks.
+
+    Parameters:
+    results (List[List[List[Union[str, int]]]]):
+            [[target_name, instructions, comparison_instructions],[...],...]
+            comparsion_instructions: *[latest, reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    targets (List[str]): List of target names
+    num_benchmarks (int): Number of benchmarks
+
+    Return:
+    (List[List[Union[str, int]]]):
+            [[target_name, average_instructions, \
+                comparison_instructions],[...],...]
+            comparsion_instructions: *[average_latest, average_reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    """
+    average_instructions: List[List[Union[str, int]]] = []
+
+    for i, target in enumerate(targets):
+        average_instructions.append([target])
+
+        total_instructions = 0
+        total_latest_percentages: Optional[float] = 0.0
+        total_reference_percentages = 0.0
+
+        for instructions in results:
+            total_instructions += int(instructions[i][1])
+            if instructions[i][3] != "-----":
+                total_reference_percentages += float(
+                    str(instructions[i][3])[:-1])
+            if total_latest_percentages is not None:
+                if instructions[i][2] != "N/A":
+                    if instructions[i][2] != "-----":
+                        total_latest_percentages += float(
+                            str(instructions[i][2])[:-1])
+                else:
+                    total_latest_percentages = None
+
+        avg_instructions = total_instructions // num_benchmarks
+        avg_reference_percentages = format_percentage(
+            round(total_reference_percentages / num_benchmarks, 3))
+        avg_latest_percentages = format_percentage(
+            round(total_latest_percentages / num_benchmarks, 3)) \
+            if total_latest_percentages is not None else "N/A"
+
+        average_instructions[-1].extend([avg_instructions,
+                                         avg_latest_percentages,
+                                         avg_reference_percentages])
+
+    return average_instructions
+
+
+def write_to_csv(instructions: List[List[Union[str, int]]],
+                 output_csv_path: str, percentages: bool = False,
+                 reference_version: str = "") -> None:
+    """
+    Write the [Target, Instructions] for each target in a CSV file.
+    comparison_instructions are ignored.
+
+    Parameters:
+    instructions (List[List[Union[str, int]]]):
+            [[target_name, instructions, comparison_instructions],[...],...]
+            comparsion_instructions: *[latest, reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    output_csv_path (str): Absolute path to output CSV file
+    percentages (bool): Add percentages to the output CSV file
+    """
+    with open(output_csv_path, "w") as file:
+        writer = csv.writer(file)
+        header = ["Target", "Instructions"]
+        if percentages:
+            header.extend(["Latest", reference_version])
+        writer.writerow(header)
+        for target_instructions in instructions:
+            row = []
+            row.extend([target_instructions[0], format(
+                target_instructions[1], ",")])
+            if percentages:
+                row.extend(target_instructions[2:])
+            writer.writerow(row)
+
+
+def print_table(instructions: str, text: str, reference_version: str) -> None:
+    """
+    Print the results in a tabular form
+
+    Parameters:
+    instructions (List[List[Union[str, int]]]):
+            [[target_name, instructions, comparison_instructions],[...],...]
+            comparsion_instructions: *[latest, reference]
+            If latest is not available,
+            then comparsion_instructions = reference
+    text (str): Text be added to the table header
+    reference_version (str): Reference version used in these results
+    """
+    print("{}\n{}\n{}".
+          format("-" * 56, text, "-" * 56))
+
+    print('{:<10}  {:>20}  {:>10}  {:>10}\n{}  {}  {}  {}'.
+          format('Target',
+                 'Instructions',
+                 'Latest',
+                 reference_version,
+                 '-' * 10,
+                 '-' * 20,
+                 '-' * 10,
+                 '-' * 10))
+
+    for target_change in instructions:
+        # Replace commas with spaces in instruction count
+        # for easier readability.
+        formatted_instructions = format(
+            target_change[1], ",").replace(",", " ")
+        print('{:<10}  {:>20}  {:>10}  {:>10}'.format(
+            target_change[0], formatted_instructions, *target_change[2:]))
+
+    print("-" * 56)
+
+
+def clean_exit(qemu_path: str, error_message: str) -> None:
+    """
+    Clean up intermediate files and exit.
+
+    Parameters:
+    qemu_path (str): Absolute path to QEMU
+    error_message (str): Error message to display after exiting
+    """
+    # Clean the QEMU build path
+    qemu_build_path = os.path.join(qemu_path, "build-gcc")
+    if os.path.isdir(qemu_build_path):
+        shutil.rmtree(qemu_build_path)
+    sys.exit(error_message)
+
+
+def verify_executables(benchmark_paths: List[str], targets: Dict[str, str],
+                       benchmarks: List[Dict[str, str]],
+                       benchmarks_executables_dir_path: str) -> None:
+    """
+    Verify that all executables exist for each benchmark.
+
+    Parameters:
+    benchmark_paths (List[str]): List of all paths to benchmarks
+    targets (Dict[str, str]): Dictionary the contains for each target,
+                              target_name: target_compiler
+    benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path)
+    benchmarks_executables_dir_path (str): Absolute path to the executables dir
+    """
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Verifying executables of {} benchmarks for {} targets".
+          format(len(benchmark_paths), len(targets)),
+          file=sys.stderr, flush=True)
+
+    for benchmark in benchmarks:
+        executable_parent_dir_path = get_executable_parent_dir_path(
+            benchmark["path"], benchmarks_executables_dir_path)
+
+        # Verify that the exists for this benchmark executables, if not,
+        # create it
+        if not os.path.isdir(executable_parent_dir_path):
+            os.mkdir(executable_parent_dir_path)
+
+        for target_name, target_compiler in targets.items():
+            compiled_benchmark = "{}-{}".format(
+                benchmark["name"], target_name)
+            compiled_benchmark_path = os.path.join(
+                executable_parent_dir_path, compiled_benchmark)
+            # Verify that the the executable for this target is available,
+            # if not, compile it
+            if not os.path.isfile(compiled_benchmark_path):
+                compile_target(benchmark["path"],
+                               compiled_benchmark_path,
+                               target_compiler)
+
+
+def verify_reference_results(reference_version: str, qemu_path: str,
+                             benchmarks: List[Dict[str, str]],
+                             reference_version_results_dir_path: str,
+                             targets: List[str],
+                             benchmarks_executables_dir_path: str) -> None:
+    """
+    Verify that results are available for reference version.
+    If results are missing, build QEMU for the reference version then perform
+    the measurements.
+
+    Paramters:
+    reference_version (str): Reference QEMU version
+    qemu_path (str): Absolute path to QEMU
+    benchmark_paths (List[str]): List of all paths to benchmarks
+    reference_version_results_dir_path (str): Absolute path to the reference
+                                              version results dir
+    targets (List[str]): Target names
+    benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path)
+    benchmarks_executables_dir_path (str): Path to the root executables dir
+    """
+    print(datetime.datetime.utcnow().isoformat(),
+          "- Verifying results of reference version {}".
+          format(reference_version), file=sys.stderr, flush=True)
+
+    # Set flag to know if QEMU was built for reference version before
+    did_build_reference = False
+
+    latest_commit_hash = get_commit_hash(reference_version, qemu_path)
+
+    for benchmark in benchmarks:
+        benchmark_results_dir_path = os.path.join(
+            reference_version_results_dir_path, benchmark["parent_dir"])
+
+        # Verify that the results directory for the benchmark exists, if not,
+        # create it
+        if not os.path.isdir(benchmark_results_dir_path):
+            os.mkdir(benchmark_results_dir_path)
+
+        # Verify that the the results.csv file for the benchmark exits, if not,
+        # create it
+        results_path = os.path.join(benchmark_results_dir_path,
+                                    "{}-{}-results.csv".
+                                    format(latest_commit_hash,
+                                           benchmark["name"]))
+        if not os.path.isfile(results_path):
+            # Only build qemu if reference version wasn't built before
+            if not did_build_reference:
+                build_qemu(qemu_path, reference_version, targets)
+                did_build_reference = True
+            print(datetime.datetime.utcnow().isoformat(),
+                  "- Measuring instructions for reference version {} - {}".
+                  format(reference_version, benchmark["name"]),
+                  file=sys.stderr, flush=True)
+            instructions = measure_instructions(
+                benchmark["path"],
+                benchmarks_executables_dir_path,
+                qemu_path,
+                targets)
+            write_to_csv(instructions, results_path)
+
+
+def verify_requirements() -> None:
+    """
+    Verify that all script requirements are installed (valgrind & git).
+    """
+    # Insure that valgrind is installed
+    check_valgrind_installation = subprocess.run(["which", "valgrind"],
+                                                 stdout=subprocess.DEVNULL,
+                                                 check=False)
+    if check_valgrind_installation.returncode:
+        sys.exit("Please install valgrind before running the script.")
+
+    # Insure that git is installed
+    check_git_installation = subprocess.run(["which", "git"],
+                                            stdout=subprocess.DEVNULL,
+                                            check=False)
+    if check_git_installation.returncode:
+        sys.exit("Please install git before running the script.")
+
+
+def main():
+    """
+    Parse the command line arguments then start the execution.
+    Output on STDOUT represents the nightly test results.
+    Output on STDERR represents the execution log and errors if any.
+    Output on STDERR must be redirected to either /dev/null or to a log file.
+
+    Syntax:
+        nightly_tests_core.py [-h] [-r REF]
+    Optional arguments:
+        -h, --help            Show this help message and exit
+        -r REF, --reference REF
+                            Reference QEMU version - Default is v5.1.0
+    Example of usage:
+        nightly_tests_core.py -r v5.1.0 2>log.txt
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-r", "--reference", dest="ref",
+                        default="v5.1.0",
+                        help="Reference QEMU version - Default is v5.1.0")
+    reference_version = parser.parse_args().ref
+
+    targets = {
+        "aarch64":  "aarch64-linux-gnu-gcc",
+        "alpha":    "alpha-linux-gnu-gcc",
+        "arm":      "arm-linux-gnueabi-gcc",
+        "hppa":     "hppa-linux-gnu-gcc",
+        "m68k":     "m68k-linux-gnu-gcc",
+        "mips":     "mips-linux-gnu-gcc",
+        "mipsel":   "mipsel-linux-gnu-gcc",
+        "mips64":   "mips64-linux-gnuabi64-gcc",
+        "mips64el": "mips64el-linux-gnuabi64-gcc",
+        "ppc":      "powerpc-linux-gnu-gcc",
+        "ppc64":    "powerpc64-linux-gnu-gcc",
+        "ppc64le":  "powerpc64le-linux-gnu-gcc",
+        "riscv64":  "riscv64-linux-gnu-gcc",
+        "s390x":    "s390x-linux-gnu-gcc",
+        "sh4":      "sh4-linux-gnu-gcc",
+        "sparc64":  "sparc64-linux-gnu-gcc",
+        "x86_64":   "gcc"
+    }
+
+    # Verify that the script requirements are installed
+    verify_requirements()
+
+    # Get required paths
+    nightly_tests_dir_path = pathlib.Path(__file__).parent.parent.absolute()
+    benchmarks_dir_path = os.path.join(nightly_tests_dir_path, "benchmarks")
+    benchmarks_source_dir_path = os.path.join(benchmarks_dir_path, "source")
+    benchmarks_executables_dir_path = os.path.join(
+        benchmarks_dir_path, "executables")
+
+    # Verify that If the executables directory exists, if not, create it
+    if not os.path.isdir(benchmarks_executables_dir_path):
+        os.mkdir(benchmarks_executables_dir_path)
+
+    # Get absolute path to all available benchmarks
+    benchmark_paths = sorted([y for x in os.walk(benchmarks_source_dir_path)
+                              for y in glob.glob(os.path.join(x[0], '*.c'))])
+
+    benchmarks = [{
+        "name": get_benchmark_name(benchmark_path),
+        "parent_dir": get_benchmark_parent_dir(benchmark_path),
+        "path": benchmark_path} for benchmark_path in benchmark_paths]
+
+    # Verify that all executables exist for each benchmark
+    verify_executables(benchmark_paths, targets, benchmarks,
+                       benchmarks_executables_dir_path)
+
+    # Set QEMU path and clone from Git if the path doesn't exist
+    qemu_path = os.path.join(nightly_tests_dir_path, "qemu-nightly")
+    if not os.path.isdir(qemu_path):
+        # Clone QEMU into the temporary directory
+        print(datetime.datetime.utcnow().isoformat(),
+              "- Fetching QEMU: ", end="", flush=True, file=sys.stderr)
+        git_clone(qemu_path)
+        print("\n", file=sys.stderr, flush=True)
+
+    # Verify that the results directory exists, if not, create it
+    results_dir_path = os.path.join(nightly_tests_dir_path, "results")
+    if not os.path.isdir(results_dir_path):
+        os.mkdir(results_dir_path)
+
+    # Verify that the reference version results directory exists, if not,
+    # create it
+    reference_version_results_dir_path = os.path.join(
+        results_dir_path, reference_version)
+    if not os.path.isdir(reference_version_results_dir_path):
+        os.mkdir(reference_version_results_dir_path)
+
+    # Verify that previous results are available for reference version
+    verify_reference_results(reference_version, qemu_path, benchmarks,
+                             reference_version_results_dir_path,
+                             targets.keys(), benchmarks_executables_dir_path)
+
+    # Compare results with the latest QEMU master
+    # -------------------------------------------------------------------------
+    # Verify that the "latest" results directory exists, if not,
+    # create it
+    latest_version_results_dir_path = os.path.join(results_dir_path, "latest")
+    if not os.path.isdir(latest_version_results_dir_path):
+        os.mkdir(latest_version_results_dir_path)
+
+    # Verify that the "history" results directory exists, if not, create it
+    history_results_dir_path = os.path.join(results_dir_path, "history")
+    if not os.path.isdir(history_results_dir_path):
+        os.mkdir(history_results_dir_path)
+
+    # Build QEMU for master
+    build_qemu(qemu_path, "master", targets.keys())
+
+    # Get the commit hash for the top commit at master
+    master_commit_hash = get_commit_hash("master", qemu_path)
+
+    # Print report summary header
+    print("{}\n{}".format("-" * 56, "{} SUMMARY REPORT - COMMIT {}".
+                          format(" "*11, master_commit_hash)))
+
+    # For each benchmark, compare the current master results with
+    # latest and reference version
+    results = []
+    for benchmark in benchmarks:
+        reference_version_benchmark_results_dir_path = os.path.join(
+            reference_version_results_dir_path, benchmark["parent_dir"])
+        latest_version_benchmark_results_dir_path = os.path.join(
+            latest_version_results_dir_path, benchmark["parent_dir"])
+        history_benchmark_results_dir_path = os.path.join(
+            history_results_dir_path, benchmark["parent_dir"])
+
+        # Verify that the the benchmark directory exists in the "latest"
+        # directory, if not, create it
+        if not os.path.isdir(latest_version_benchmark_results_dir_path):
+            os.mkdir(latest_version_benchmark_results_dir_path)
+
+        # Verify that the the benchmark directory exists in the "history"
+        # directory, if not, create it
+        if not os.path.isdir(history_benchmark_results_dir_path):
+            os.mkdir(history_benchmark_results_dir_path)
+
+        # Obtain the instructions array which will contain for each target,
+        # the target name, the number of "master" instructions,
+        # "latest" instructions if available,
+        # and "reference version" instructions
+        instructions = measure_master_instructions(
+            reference_version_benchmark_results_dir_path,
+            get_commit_hash(reference_version, qemu_path),
+            latest_version_benchmark_results_dir_path,
+            benchmark["path"],
+            benchmarks_executables_dir_path,
+            qemu_path,
+            targets.keys())
+
+        # Update the "latest" directory with the new results form master
+        updated_latest_version_benchmark_results = os.path.join(
+            latest_version_benchmark_results_dir_path,
+            "{}-{}-results.csv".format(master_commit_hash, benchmark["name"]))
+        write_to_csv(instructions, updated_latest_version_benchmark_results)
+
+        calculate_change(instructions)
+
+        # Insert "N/A" for targets that don't have "latest" instructions
+        for target_instructions in instructions:
+            if len(target_instructions) == 3:
+                target_instructions.insert(2, "N/A")
+
+        history_benchmark_results = os.path.join(
+            history_benchmark_results_dir_path,
+            "{}-{}-results.csv".format(master_commit_hash, benchmark["name"]))
+        write_to_csv(instructions, history_benchmark_results,
+                     True, reference_version)
+
+        # Store the results
+        results.append([benchmark["name"], instructions])
+
+    # Calculate the average instructions for each target
+    # Only send the instructions as results without the benchmark names
+    average_instructions = calculate_average(
+        [result[1] for result in results], targets.keys(), len(benchmarks))
+
+    # Save average results to results/history directory
+    average_results = os.path.join(
+        history_results_dir_path,
+        "{}-average-results.csv".format(master_commit_hash))
+    write_to_csv(average_instructions, average_results,
+                 True, reference_version)
+
+    # Print results
+    print_table(average_instructions, " "*20 +
+                "AVERAGE RESULTS", reference_version)
+    print("\n", " "*17, "DETAILED RESULTS")
+    for [benchmark_name, instructions] in results:
+        print_table(instructions,
+                    "Test Program: " + benchmark_name,
+                    reference_version)
+
+    # Cleanup (delete the build directory)
+    shutil.rmtree(os.path.join(qemu_path, "build-gcc"))
+    # -------------------------------------------------------------------------
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/performance/nightly-tests/scripts/run_nightly_tests.py b/tests/performance/nightly-tests/scripts/run_nightly_tests.py
new file mode 100755
index 0000000000..ea65be60dc
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/run_nightly_tests.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+
+"""
+Entry point script for running nightly performance tests on QEMU.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+import datetime
+import io
+import os
+import subprocess
+import time
+import sys
+from send_email import send_email
+
+
+# Record system hardware information
+with open('/proc/cpuinfo', 'r') as cpuinfo:
+    for line in cpuinfo:
+        if line.startswith('model name'):
+            HOST_CPU = line.rstrip('\n').split(':')[1].strip()
+            break
+with open('/proc/meminfo', 'r') as meminfo:
+    for line in meminfo:
+        if line.startswith('MemTotal'):
+            HOST_MEMORY_KB = int(line.rstrip('\n').split(':')[1].
+                                 strip().split(' ')[0])
+            HOST_MEMORY = str(round(HOST_MEMORY_KB / (1024 * 1024), 2)) + " GB"
+            break
+
+# Find path for the "nightly_tests_core.py" script
+NIGHTLY_TESTS_CORE_PATH = os.path.join(
+    os.path.dirname(os.path.abspath(__file__)),
+    "nightly_tests_core.py")
+
+NIGHTLY_TESTS_ARGUMENTS = sys.argv[1:]
+
+# Start the nightly test
+START_EPOCH = time.time()
+RUN_NIGHTLY_TESTS = subprocess.run([NIGHTLY_TESTS_CORE_PATH,
+                                    *NIGHTLY_TESTS_ARGUMENTS],
+                                   stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE,
+                                   check=False)
+END_EPOCH = time.time()
+
+# Perform time calculations
+EXECUTION_TIME = datetime.timedelta(seconds=END_EPOCH - START_EPOCH)
+TEST_DATE = datetime.datetime.utcfromtimestamp(
+    START_EPOCH).strftime('%A, %B %-d, %Y')
+START_TIME = datetime.datetime.utcfromtimestamp(
+    START_EPOCH).strftime('%Y-%m-%d %H:%M:%S')
+END_TIME = datetime.datetime.utcfromtimestamp(
+    END_EPOCH).strftime('%Y-%m-%d %H:%M:%S')
+
+# Get nightly test status
+if RUN_NIGHTLY_TESTS.returncode:
+    STATUS = "FAILURE"
+else:
+    STATUS = "SUCCESS"
+
+# Initialize a StringIO to print all the output into
+OUTPUT = io.StringIO()
+
+# Print the nightly test statistical information
+print("{:<17}: {}\n{:<17}: {}\n".
+      format("Host CPU",
+             HOST_CPU,
+             "Host Memory",
+             HOST_MEMORY), file=OUTPUT)
+print("{:<17}: {}\n{:<17}: {}\n{:<17}: {}\n".
+      format("Start Time (UTC)",
+             START_TIME,
+             "End Time (UTC)",
+             END_TIME,
+             "Execution Time",
+             EXECUTION_TIME), file=OUTPUT)
+print("{:<17}: {}\n".format("Status", STATUS), file=OUTPUT)
+
+if STATUS == "SUCCESS":
+    print("Note:\nChanges denoted by '-----' are less than 0.01%.\n",
+          file=OUTPUT)
+
+# Print the nightly test stdout (main output)
+print(RUN_NIGHTLY_TESTS.stdout.decode("utf-8"), file=OUTPUT)
+
+# If the nightly test failed, print the stderr (error logs)
+if STATUS == "FAILURE":
+    print("{}\n{}\n{}".format("-" * 56,
+                              " " * 18 + "ERROR LOGS",
+                              "-" * 56), file=OUTPUT)
+    print(RUN_NIGHTLY_TESTS.stderr.decode("utf-8"), file=OUTPUT)
+
+
+# Temp file to store the output in case sending the email failed
+# with open("temp.txt", "w") as file:
+#     file.write(OUTPUT.getvalue())
+
+# Use an HTML message to preserve monospace formatting
+HTML_MESSAGE = """\
+<html><body><pre>
+{body}
+</pre></body></html>
+""".format(body=OUTPUT.getvalue())
+OUTPUT.close()
+
+# Send the nightly test results email to the QEMU mailing list
+while True:
+    try:
+        send_email("[REPORT] Nightly Performance Tests - {}".format(TEST_DATE),
+                   ["qemu-devel@nongnu.org"], HTML_MESSAGE)
+    except Exception:  # pylint: disable=W0703
+        # Wait for a minute then retry sending
+        time.sleep(60)
+        continue
+    else:
+        break
diff --git a/tests/performance/nightly-tests/scripts/send_email.py b/tests/performance/nightly-tests/scripts/send_email.py
new file mode 100644
index 0000000000..e24e244f51
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/send_email.py
@@ -0,0 +1,56 @@
+"""
+Helper script for sending emails with the nightly performance test results.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+import smtplib
+from typing import List
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.utils import COMMASPACE, formatdate
+
+
+GMAIL_USER = {"name": "",
+              "email": "",
+              "pass": ""}
+
+
+def send_email(subject: str, send_to: List[str], html: str) -> None:
+    """
+    Send an HTML email.
+
+    Parameters:
+    subject (str): Email subject
+    send_to (List(str)): List of recipients
+    html (str): HTML message
+    """
+    msg = MIMEMultipart('alternative')
+    msg['From'] = "{} <{}>".format(GMAIL_USER["name"], GMAIL_USER["email"])
+    msg['To'] = COMMASPACE.join(send_to)
+    msg['Date'] = formatdate(localtime=True)
+    msg['Subject'] = subject
+
+    msg.attach(MIMEText(html, 'html'))
+
+    server = smtplib.SMTP_SSL('smtp.gmail.com', 465)
+    server.login(GMAIL_USER["email"], GMAIL_USER["pass"])
+    server.sendmail(msg['From'], send_to, msg.as_string())
+    server.quit()

From patchwork Fri Aug 28 10:41:01 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353170
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=sFkgZ2bo;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGW74Ktqz9sPB
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:46:59 +1000 (AEST)
Received: from localhost ([::1]:41562 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbuL-0001w4-IS
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:46:57 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34534)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpQ-0003fM-Ah
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:52 -0400
Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:45773)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpO-0005jD-C2
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:51 -0400
Received: by mail-wr1-x442.google.com with SMTP id h15so831313wrt.12
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=ocs60WDznaDScA/gvXj53ZTFJX4+fFayK+ZVchaf5to=;
 b=sFkgZ2boQgWhq4eXz6d09Ypkx8IqsyVoGLyWlG08+NeYR0k2QrTXXwqqVN0smG09Sb
 o89f2Vs0Upy7/Gm/+odj5QYCMICXmj2m8tTkvLXpIORj8cmv/QRO/oor7daXMgMiZw56
 JJZZS4tH9NQNzGc4AA6GtJhkVWwwKexihmyv512GrORDoe6+r/rf8EwwDjhEyziNpvh7
 L0rX+mp0WG+zTMyhuC+X8WWoGEwb8Ua6SqwwVz/GrndNXBQqmuweQgeeLcF2p45HQcL1
 Jq5Ra5zk7uAabEvpMUSrCOQzWqN2ryd+2SoIRJRS3h30PI5qdJ4FIdLMgNVSlmEiw23W
 V0fA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=ocs60WDznaDScA/gvXj53ZTFJX4+fFayK+ZVchaf5to=;
 b=UB2qNRDP9qKTfrFddQHDOrYSim/mODDD1PeGAxY5aHZhCZYV0HePEYX/8LHbKtXyUl
 +Fq0ENfkHVicElI7PJyVHtwJxI+mEXWlG2p0flchXNxBrZXJazd+1HW3sZHQ5LHCM70F
 1LATI/Dl/kEgAzHzgX5Ez+Ki3xL/43A0NsPxmZdySNHMdDX4lHmagGDUC1puBTlTDu7A
 yh54kCT3xnb0IxXOqqjdHAH/H+9R4rCF00QrbnWdpxTNRm+J4KuVVs4PBkZIIU8KGLwp
 KCaT9//7YImOngue6Lcm16QiMTxuzuHLJus148gsa/AtKPdCiqn0tfgfMu0QyfZPnzwi
 Vyag==
X-Gm-Message-State: AOAM530uDkDY6fCqjdBtQoJuiL4dPemHSEZ9qIwHjmye6JQUfiXn9xXI
 iF9Z9QeR+cMQx0JC75vO/XY0VY+WW493Xg==
X-Google-Smtp-Source: 
 ABdhPJzC+ez05Uou+atjjrhL2u5gcFrd8SM0IZ3ReNieo/RoGnrU8Gt8HLHuFj2S1Aw6BCQL6c88KA==
X-Received: by 2002:a5d:674c:: with SMTP id l12mr890586wrw.325.1598611308896;
 Fri, 28 Aug 2020 03:41:48 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.47
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:48 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance
 Tools and Tests' subsection
Date: Fri, 28 Aug 2020 12:41:01 +0200
Message-Id: <20200828104102.4490-9-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::442;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

Add a new 'tests/performance' directory under the 'Performance Tools
and Tests' subsection.

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 5a22c8be42..8923307642 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3124,3 +3124,4 @@ Performance Tools and Tests
 M: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
 S: Maintained
 F: scripts/performance/
+F: tests/performance/

From patchwork Fri Aug 28 10:41:02 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
X-Patchwork-Id: 1353166
Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org
 (client-ip=209.51.188.17; helo=lists.gnu.org;
 envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;
 receiver=<UNKNOWN>)
Authentication-Results: ozlabs.org;
 dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: ozlabs.org;
	dkim=pass (2048-bit key;
 unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256
 header.s=20161025 header.b=b7Lp06PI;
	dkim-atps=neutral
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ozlabs.org (Postfix) with ESMTPS id 4BdGRm1MM2z9sSn
	for <incoming@patchwork.ozlabs.org>; Fri, 28 Aug 2020 20:44:04 +1000 (AEST)
Received: from localhost ([::1]:60790 helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>)
	id 1kBbrW-0006cT-4I
	for incoming@patchwork.ozlabs.org; Fri, 28 Aug 2020 06:44:02 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34582)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpT-0003pL-V1
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:55 -0400
Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:43367)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <ahmedkhaledkaraman@gmail.com>)
 id 1kBbpQ-0005kD-Jh
 for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:55 -0400
Received: by mail-wr1-x442.google.com with SMTP id k15so834510wrn.10
 for <qemu-devel@nongnu.org>; Fri, 28 Aug 2020 03:41:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=from:to:cc:subject:date:message-id:in-reply-to:references;
 bh=NKjrR/1jJBl3PMLAgxGFOGP+xi3KNmymEVWR9lFRmRk=;
 b=b7Lp06PIVaYJOGD/iWKmSUt1QjjX41g5c9VeXHZhk3l7qeTyJropoduGwPmAMNJ0RC
 smQxmVX7ivvXTDaMNzWBiWCiIGajG5y1jonWPU/YKLQrwB20wBvHWUFYlM1nDdf6CFWQ
 zph5CvKoUrduoVOz2a4hk4BnRW2Is0uABwxVPA/rNsRGAeVQ3QlKQSO1N0z5Q15EDFVY
 M4YWJRHHYTQs+yVy1Ipyf+ObGU5d0UL0D4BVqw2I9t4jITTLakphE5OVbHZja7km9GsB
 /kHtzlJlRWExC9iPT7/DmO9A2oNZApQi2TB1Q/Olw8kK53bwhuf8LFVDdvnCoP39Q7P9
 t51A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
 :references;
 bh=NKjrR/1jJBl3PMLAgxGFOGP+xi3KNmymEVWR9lFRmRk=;
 b=JFw0lDw6e2fU7+ukZcqxus70HcXSlGlRRSdkTjJQHQXESD1FkbQ6A73ZAx5vd9Ag3v
 oNRwYFDRje0Rrin+c3fRPoOf5N94x2lHfs2PV89kuX7hKlvtvNTLT1b64CVyJsB9wTBm
 BcH2po1J4gsFYsVO5pRqBOoSVnujgNHN84nWwzZ4hsbjDbbYjuEPvPwJfu8KxYRfLICE
 7walVArnLs5iSWo74WBC1H0vvTWO0J6mt//f+gs1RiUdO6yMx3dvoRqrWxIjqVr6giZW
 EzqPPzxK2QXdAumDunKQ6f2x/qWshrJX5U6dZqjoLtzAB0/bSaTnxc4dS7mMf/3iEbXN
 tbXg==
X-Gm-Message-State: AOAM530pseFF1o4KXG7up1h2MihWyrvstAz3IF76meNuFzRmtjluOPW7
 968P62rRIM98xbrircy8sCNwydAr/BXfKg==
X-Google-Smtp-Source: 
 ABdhPJwsM0EBllpTnbeUyiaGJcaEEvoElZtbRzP6mYUFKdp3uqyWztCdJ+EosrbuJFcOdtYbGs7lKg==
X-Received: by 2002:adf:828e:: with SMTP id 14mr984753wrc.217.1598611310859;
 Fri, 28 Aug 2020 03:41:50 -0700 (PDT)
Received: from localhost.localdomain ([197.58.77.158])
 by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.49
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Fri, 28 Aug 2020 03:41:50 -0700 (PDT)
From: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com,
 alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com,
 jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com
Subject: [PATCH 9/9] scripts/performance: Add topN_system.py script
Date: Fri, 28 Aug 2020 12:41:02 +0200
Message-Id: <20200828104102.4490-10-ahmedkhaledkaraman@gmail.com>
X-Mailer: git-send-email 2.17.1
In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com>
Received-SPF: pass client-ip=2a00:1450:4864:20::442;
 envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com
X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache.
 That's all we know.
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Cc: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org
Sender: "Qemu-devel"
 <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>

Python script for listing the topN executed QEMU functions in
system mode.

Syntax:
topN_system.py [-h] [-n <number of displayed top functions>] -- \
    <qemu system executable> <qemu system options>

[-h] - Print the script arguments help message.
[-n] - Specify the number of top functions to print.
     - If this flag is not specified, the tool defaults to 25.

Example of usage:
topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \
    -initrd <initrd>

Example output:
Number of instructions: 150,991,381,071

 No.  Percentage  Name
 ---  ----------  ------------------------------
   1      11.30%  helper_lookup_tb_ptr
   2       7.01%  liveness_pass_1
   3       4.48%  tcg_gen_code
   4       3.41%  tcg_optimize
   5       1.84%  tcg_out_opc.isra.13
   6       1.78%  helper_pcmpeqb_xmm
   7       1.20%  object_dynamic_cast_assert
   8       1.00%  cpu_exec
   9       0.99%  tcg_temp_new_internal
  10       0.88%  tb_htable_lookup
  11       0.84%  object_class_dynamic_cast_assert
  12       0.81%  init_ts_info
  13       0.80%  tlb_set_page_with_attrs
  14       0.77%  victim_tlb_hit
  15       0.75%  tcg_out_sib_offset
  16       0.62%  tcg_op_alloc
  17       0.61%  helper_pmovmskb_xmm
  18       0.58%  disas_insn.isra.50
  19       0.56%  helper_pcmpgtb_xmm
  20       0.56%  address_space_ldq

Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
 scripts/performance/topN_system.py | 158 +++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)
 create mode 100755 scripts/performance/topN_system.py

diff --git a/scripts/performance/topN_system.py b/scripts/performance/topN_system.py
new file mode 100755
index 0000000000..9b1f1a66c7
--- /dev/null
+++ b/scripts/performance/topN_system.py
@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+
+"""
+Print the top N most executed functions in QEMU system mode emulation.
+
+Syntax:
+topN_system.py [-h] [-n <number of displayed top functions>] -- \
+    <qemu system executable> <qemu system options>
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+     - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \
+    -initrd <initrd>
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020  Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020  Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+
+# Parse the command line arguments
+PARSER = argparse.ArgumentParser(
+    usage="usage: topN_system.py [-h] [-n <number of displayed top functions>]"
+          " -- <qemu system executable> <qemu system options>")
+
+PARSER.add_argument("-n", dest="top", type=int, default=25,
+                    help="Specify the number of top functions to print.")
+
+PARSER.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+ARGS = PARSER.parse_args()
+
+# Extract the needed variables from the args
+COMMAND = ARGS.command
+TOP = ARGS.top
+
+# Ensure that perf is installed
+CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
+                                     stdout=subprocess.DEVNULL,
+                                     check=False)
+if CHECK_PERF_PRESENCE.returncode:
+    sys.exit("Please install perf before running the script!")
+
+# Ensure user has previllage to run perf
+CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
+                                          stdout=subprocess.DEVNULL,
+                                          stderr=subprocess.DEVNULL,
+                                          check=False)
+if CHECK_PERF_EXECUTABILITY.returncode:
+    sys.exit("""
+Error:
+You may not have permission to collect stats.
+Consider tweaking /proc/sys/kernel/perf_event_paranoid,
+which controls use of the performance events system by
+unprivileged users (without CAP_SYS_ADMIN).
+  -1: Allow use of (almost) all events by all users
+      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
+   0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
+      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
+   1: Disallow CPU event access by users without CAP_SYS_ADMIN
+   2: Disallow kernel profiling by users without CAP_SYS_ADMIN
+To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
+   kernel.perf_event_paranoid = -1
+
+* Alternatively, you can run this script under sudo privileges.
+""")
+
+# Run perf and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+    RECORD_PATH = os.path.join(tmpdir, "record.data")
+    REPORT_PATH = os.path.join(tmpdir, "report.txt")
+
+    PERF_RECORD = subprocess.run((["perf",
+                                   "record",
+                                   "-e",
+                                   "instructions",
+                                   "--output="+RECORD_PATH] +
+                                  COMMAND),
+                                 stdout=subprocess.DEVNULL,
+                                 stderr=subprocess.PIPE,
+                                 check=False)
+    if PERF_RECORD.returncode:
+        sys.exit(PERF_RECORD.stderr.decode("utf-8"))
+
+    with open(REPORT_PATH, "w") as output:
+        PERF_REPORT = subprocess.run(["perf",
+                                      "report",
+                                      "--input=" + RECORD_PATH,
+                                      "--stdio"],
+                                     stdout=output,
+                                     stderr=subprocess.PIPE,
+                                     check=False)
+        if PERF_REPORT.returncode:
+            sys.exit(PERF_REPORT.stderr.decode("utf-8"))
+
+    # Save the reported data to FUNCTIONS[] and INSTRUCTIONS
+    with open(REPORT_PATH, "r") as data:
+        LINES = data.readlines()
+        # Read the number of instructions
+        INSTRUCTIONS = int(LINES[6].split()[-1])
+        # Continue reading:
+        # Only read lines that are not empty
+        # Only read lines that are not comments (comments start with #)
+        # Only read functions executed by qemu-system
+        FUNCTIONS = [line for line in LINES if line
+                     and line[0] != "\n"
+                     and line[0] != "#"
+                     and "qemu-system" in line.split()[2]]
+
+
+# Limit the number of top functions to "TOP"
+NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
+
+# Store the data of the top functions in TOP_FUNCTIONS[]
+TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
+
+# Print total instructions
+print("\nNumber of instructions:", format(INSTRUCTIONS, ","))
+# Print table header
+print("\n{:>4}  {:>10}  {}\n{}  {}  {}".format("No.",
+                                               "Percentage",
+                                               "Name",
+                                               "-" * 4,
+                                               "-" * 10,
+                                               "-" * 30))
+
+# Print top N functions
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
+    function_data = function.split()
+    function_percentage = function_data[0]
+    function_name = function_data[-1]
+    function_invoker = " ".join(function_data[2:-2])
+    print("{:>4}  {:>10}  {}".format(index,
+                                     function_percentage,
+                                     function_name))