From patchwork Fri Dec 27 18:16:29 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 305460 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ozlabs.org (Postfix) with ESMTP id 5B3152C00A5 for ; Sat, 28 Dec 2013 05:16:43 +1100 (EST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 518B8816C9; Fri, 27 Dec 2013 18:16:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rq1ZIw9EFkM0; Fri, 27 Dec 2013 18:16:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 6C01581722; Fri, 27 Dec 2013 18:16:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id CB6871C22F3 for ; Fri, 27 Dec 2013 18:16:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C11E081722 for ; Fri, 27 Dec 2013 18:16:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hYFaXhWOuW5y for ; Fri, 27 Dec 2013 18:16:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) by whitealder.osuosl.org (Postfix) with ESMTPS id CF35A816C9 for ; Fri, 27 Dec 2013 18:16:37 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id hq4so14572333wib.9 for ; Fri, 27 Dec 2013 10:16:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id; bh=eblVgrDrR0Aulf/CX+Tb/XBj4epZkMXVB9Th3bSwjLo=; b=KVxwLluIFS0Yu40gIS9s/LbmGJGJR72LsZfoOo6As9uKKvDxKMNauqN9b0RYJtb70r 6M90wgaVPqEoXvi599S6e8sRLCeJaq/bAb5sGqnXkGtCWkyAalUALvZW5sTMUgZw3DP0 H0mFdl48+IhZ4J06P5LgNC35RAvwRE8dUIQTAi9hw5t5rgZXZGMgi+D2AMl57Khp21sG t6SGhuCTsthZ+g2g765AecM2vs2piW3MP0m3Gkdtf2nuStU7iagS0rgyTBQNURcP3mTA 6oPH0IqWiTVB4KO9+RtJYtP3QerJFgZTE3KTpymapZayFmzzmpsVlroH9CerdpoeDnb2 yO6A== X-Received: by 10.194.79.165 with SMTP id k5mr10788034wjx.44.1388168196282; Fri, 27 Dec 2013 10:16:36 -0800 (PST) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id d2sm55705985wik.11.2013.12.27.10.16.33 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 27 Dec 2013 10:16:34 -0800 (PST) From: "Yann E. MORIN" To: buildroot@busybox.net Date: Fri, 27 Dec 2013 19:16:29 +0100 Message-Id: <1388168189-5713-1-git-send-email-yann.morin.1998@free.fr> X-Mailer: git-send-email 1.8.1.2 Cc: Thomas Petazzoni , "Yann E. MORIN" Subject: [Buildroot] [PATCH] graph-build-time: generate graphs based on timing data X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: buildroot-bounces@busybox.net From: "Yann E. MORIN" This script generates graphs of packages build time, from the timing data generated by Buildroot in the $(O)/build-time.log file. Example usage: cat $(O)/build-time.data | \ ./support/scripts/graph-build-time \ --type=histogram --output=foobar.pdf Three graph types are available : * histogram, which creates an histogram of the build time for each package, decomposed by each step (extract, patch, configure, etc.). The order in which the packages are shown is configurable: by package name, by build order, or by duration order. See the --order option. * pie-packages, which creates a pie chart of the build time of each package (without decomposition in steps). Packages that contributed to less than 1% of the overall build time are all grouped together in an "Other" entry. * pie-steps, which creates a pie chart of the time spent globally on each step (extract, patch, configure, etc...) The default is to generate an histogram ordered by package name. Signed-off-by: Thomas Petazzoni [yann.morin.1998@free.fr: adapt to the format of the step-hooks build-time.log, add sort order by name] Signed-off-by: "Yann E. MORIN" --- So, this is a respin of Thomas' scripts from more than two years ago, that I adapted to the new build-time.log format we now have with step-hooks. I have put a few graphs on-line: http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/matplotlib/ For comparisson, some I made with gnuplot: http://ymorin.is-a-geek.org/download/tmp/buildroot/graphs/gnuplot/ To be honnest, I prefer the graphs made with gnuplot, but it is mostly because of the colors (hey, Thomas, are you colorblind? ;-p) But since this script is cleaner (much, much cleaner) than what I was able to get with gnuplot, I won't even try to submit my gnuplot scripts... :-/ --- support/scripts/graph-build-time | 273 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 273 insertions(+) create mode 100755 support/scripts/graph-build-time diff --git a/support/scripts/graph-build-time b/support/scripts/graph-build-time new file mode 100755 index 0000000..13509ef --- /dev/null +++ b/support/scripts/graph-build-time @@ -0,0 +1,273 @@ +#!/usr/bin/env python + +# Copyright (C) 2011 by Thomas Petazzoni +# Copyright (C) 2013 by Yann E. MORIN +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# This script generates graphs of packages build time, from the timing +# data generated by Buildroot in the $(O)/build-time.log file. +# +# Example usage: +# +# cat $(O)/build-time.data | ./support/scripts/graph-build-time --type=histogram --output=foobar.pdf +# +# Three graph types are available : +# +# * histogram, which creates an histogram of the build time for each +# package, decomposed by each step (extract, patch, configure, +# etc.). The order in which the packages are shown is +# configurable: by package name, by build order, or by duration +# order. See the --order option. +# +# * pie-packages, which creates a pie chart of the build time of +# each package (without decomposition in steps). Packages that +# contributed to less than 1% of the overall build time are all +# grouped together in an "Other" entry. +# +# * pie-steps, which creates a pie chart of the time spent globally +# on each step (extract, patch, configure, etc...) +# +# The default is to generate an histogram ordered by package name. +# +# Requirements: +# +# * matplotlib (python-matplotlib on Debian/Ubuntu systems) +# * numpy (python-numpy on Debian/Ubuntu systems) +# * argparse (by default in Python 2.7, requires python-argparse if +# Python 2.6 is used) + +import matplotlib +import numpy +matplotlib.use('PDF') + +import matplotlib.pyplot as plt +import matplotlib.font_manager as fm +import csv +import argparse +import sys + +steps = [ 'extract', 'patch', 'configure', 'build', + 'install-target', 'install-host', 'install-images', + 'install-staging' ] + +histogram_colors = ['#e60004', '#009836', '#2e1d86', '#ffed00', + '#0068b5', '#f28e00', '#940084', '#97c000', + '#00469b', '#f9c000' ] + +class Package: + def __init__(self, name): + self.name = name + self.steps_duration = {} + self.steps_start = {} + self.steps_end = {} + + def add_step(self, step, state, time): + if state == "start": + self.steps_start[step] = time + else: + self.steps_end[step] = time + if self.steps_start.has_key(step) and self.steps_end.has_key(step): + self.steps_duration[step] = self.steps_end[step] - self.steps_start[step] + + def get_duration(self, step=None): + if step == None: + duration = 0 + for step in self.steps_duration.keys(): + duration += self.steps_duration[step] + return duration + if self.steps_duration.has_key(step): + return self.steps_duration[step] + return 0 + +# Generate an histogram of the time spent in each step of each +# package. +def pkg_histogram(data, output, order="build"): + n_pkgs = len(data) + ind = numpy.arange(n_pkgs) + + if order == "duration": + data = sorted(data, key=lambda p: p.get_duration(), reverse=True) + elif order == "name": + data = sorted(data, key=lambda p: p.name, reverse=False) + + # Prepare the vals array, containing one entry for each step + vals = [] + for step in steps: + val = [] + for p in data: + val.append(p.get_duration(step)) + vals.append(val) + + bottom = [0] * n_pkgs + legenditems = [] + + plt.figure() + + # Draw the bars, step by step + for i in range(0, len(vals)): + b = plt.bar(ind+0.1, vals[i], width=0.8, color=histogram_colors[i], bottom=bottom, linewidth=0.25) + legenditems.append(b[0]) + bottom = [ bottom[j] + vals[i][j] for j in range(0, len(vals[i])) ] + + # Draw the package names + plt.xticks(ind + .6, [ p.name for p in data ], rotation=-60, rotation_mode="anchor", fontsize=8, ha='left') + + # Adjust size of graph (double the width) + sz = plt.gcf().get_size_inches() + plt.gcf().set_size_inches(sz[0] * 2, sz[1]) + + # Add more space for the package names at the bottom + plt.gcf().subplots_adjust(bottom=0.2) + + # Remove ticks in the graph for each package + axes = plt.gcf().gca() + for line in axes.get_xticklines(): + line.set_markersize(0) + + axes.set_ylabel('Time (seconds)') + + # Reduce size of legend text + leg_prop = fm.FontProperties(size=6) + + # Draw legend + plt.legend(legenditems, steps, prop=leg_prop) + + if order == "name": + plt.title('Build time of packages') + elif order == "build": + plt.title('Build time of packages, by build order') + elif order == "duration": + plt.title('Build time of packages, by duration order') + + # Save graph + plt.savefig(output) + +# Generate a pie chart with the time spent building each package. +def pkg_pie_time_per_package(data, output): + # Compute total build duration + total = 0 + for p in data: + total += p.get_duration() + + # Build the list of labels and values, and filter the packages + # that account for less than 1% of the build time. + labels = [] + values = [] + other_value = 0 + for p in data: + if p.get_duration() < (total * 0.01): + other_value += p.get_duration() + else: + labels.append(p.name) + values.append(p.get_duration()) + + labels.append('Other') + values.append(other_value) + + plt.figure() + + # Draw pie graph + patches, texts, autotexts = plt.pie(values, labels=labels, autopct='%1.1f%%', shadow=True) + + # Reduce text size + proptease = fm.FontProperties() + proptease.set_size('xx-small') + plt.setp(autotexts, fontproperties=proptease) + plt.setp(texts, fontproperties=proptease) + + plt.title('Build time per package') + plt.savefig(output) + +# Generate a pie chart with a portion for the overall time spent in +# each step for all packages. +def pkg_pie_time_per_step(data, output): + steps_values = [] + for step in steps: + val = 0 + for p in data: + val += p.get_duration(step) + steps_values.append(val) + + plt.figure() + + # Draw pie graph + patches, texts, autotexts = plt.pie(steps_values, labels=steps, + autopct='%1.1f%%', shadow=True) + + # Reduce text size + proptease = fm.FontProperties() + proptease.set_size('xx-small') + plt.setp(autotexts, fontproperties=proptease) + plt.setp(texts, fontproperties=proptease) + + plt.title('Build time per step') + plt.savefig(output) + +# Parses the csv file passed on standard input and returns a list of +# Package objects, filed with the duration of each step and the total +# duration of the package. +def read_data(): + reader = csv.reader(sys.stdin, delimiter=':') + pkgs = [] + + # Auxilliary function to find a package by name in the list. + def getpkg(name): + for p in pkgs: + if p.name == name: + return p + return None + + for row in reader: + time = int(row[0].strip()) + state = row[1].strip() + step = row[2].strip() + pkg = row[3].strip() + + p = getpkg(pkg) + if p is None: + p = Package(pkg) + pkgs.append(p) + + p.add_step(step, state, time) + + return pkgs + +parser = argparse.ArgumentParser(description='Draw build time graphs') +parser.add_argument("--type", metavar="GRAPH_TYPE", required=True, + help="Type of graph (histogram, pie-packages, pie-steps)") +parser.add_argument("--order", metavar="GRAPH_ORDER", + help="Ordering of packages: build or duration (for histogram only)") +parser.add_argument("--output", metavar="OUTPUT", required=True, + help="Output file (PDF extension)") +args = parser.parse_args() + +d = read_data() + +if args.type == "histogram" or args.type == None: + if args.order == "build" or args.order == "duration" or args.order == "name": + pkg_histogram(d, args.output, args.order) + elif args.order == None: + pkg_histogram(d, args.output, "name") + else: + sys.stderr.write("Unknown ordering: %s\n" % args.order) + exit(1) +elif args.type == "pie-packages": + pkg_pie_time_per_package(d, args.output) +elif args.type == "pie-steps": + pkg_pie_time_per_step(d, args.output) +else: + sys.stderr.write("Unknown type: %s\n" % args.type) + exit(1)