From patchwork Thu Jul 11 09:28:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Huesca X-Patchwork-Id: 1130723 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45krNP4YJyz9sBt for ; Thu, 11 Jul 2019 19:29:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3541184D89; Thu, 11 Jul 2019 09:29:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RIRcVeNGcP4P; Thu, 11 Jul 2019 09:29:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id DEBC6842F2; Thu, 11 Jul 2019 09:29:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id BD9F31BF3F4 for ; Thu, 11 Jul 2019 09:29:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B5870861B5 for ; Thu, 11 Jul 2019 09:29:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mklbrAgQ+F2S for ; Thu, 11 Jul 2019 09:29:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id A7FD4861B3 for ; Thu, 11 Jul 2019 09:28:59 +0000 (UTC) X-Originating-IP: 86.250.200.211 Received: from localhost.localdomain (lfbn-1-17395-211.w86-250.abo.wanadoo.fr [86.250.200.211]) (Authenticated sender: victor.huesca@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id EFC19E000D; Thu, 11 Jul 2019 09:28:24 +0000 (UTC) From: Victor Huesca To: buildroot@buildroot.org Date: Thu, 11 Jul 2019 11:28:02 +0200 Message-Id: <20190711092801.6314-1-victor.huesca@bootlin.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2] support/script/pkg-stats: add support for json output X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Huesca , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pkg-stats is a great script that get a lot of intersting infos from buildroot packages. Unfortunatly it is currently designed to output a static HTML page only. While html is great for human to read, it is a pain to parse for scripts. This patch provide a new option to output a JSON file in addition to the HTML one. This allow other scripts to use all the usefull informations computed by pkg-stats. Please note that the old 'output' option has been renamed to 'html' to distinguish from the new 'json' option. Also the 'npackages' and 'packages' now uses the argparse `mutualy_exclusive` group. Signed-off-by: Victor Huesca --- Changes v1 --> v2: - Make commit message more explicit - fix coding style using flake8 --- support/scripts/pkg-stats | 53 +++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index b0be7d919b..1ff3fb2bd1 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -23,7 +23,6 @@ import os from collections import defaultdict import re import subprocess -import sys import requests # URL checking import json import certifi @@ -696,22 +695,52 @@ def dump_html(packages, stats, output): f.write(html_footer) +def dump_json(packages, stats, output): + excluded_fields = ['url_worker', 'name'] + pkgs = { + pkg.name: { + k: v + for k, v in pkg.__dict__.items() + if k not in excluded_fields + } for pkg in packages + } + statistics = { + k: v + for k, v in stats.items() + if not k.startswith('infra-') + } + statistics['infra'] = {k[6:]: v for k, v in stats.items() if k.startswith('infra-')} + o = subprocess.check_output(['git', 'log', 'master', '-n', '1', '--pretty=format:%H']) + final = {'packages': pkgs, + 'stats': statistics, + 'commit': o.splitlines()[0], + 'date': str(datetime.datetime.utcnow())} + + with open(output, 'w') as f: + json.dump(final, f, indent=2, separators=(',', ': ')) + f.write('\n') + + def parse_args(): parser = argparse.ArgumentParser() - parser.add_argument('-o', dest='output', action='store', required=True, + group1 = parser.add_argument_group('output', 'Output file(s)') + group1.add_argument('--html', dest='html', action='store', help='HTML output file') - parser.add_argument('-n', dest='npackages', type=int, action='store', + group1.add_argument('--json', dest='json', action='store', + help='JSON output file') + group2 = parser.add_mutually_exclusive_group() + group2.add_argument('-n', dest='npackages', type=int, action='store', help='Number of packages') - parser.add_argument('-p', dest='packages', action='store', + group2.add_argument('-p', dest='packages', action='store', help='List of packages (comma separated)') - return parser.parse_args() + args = parser.parse_args() + if not args.html and not args.json: + parser.error('at least one of --html or --json (or both) is required') + return args def __main__(): args = parse_args() - if args.npackages and args.packages: - print("ERROR: -n and -p are mutually exclusive") - sys.exit(1) if args.packages: package_list = args.packages.split(",") else: @@ -735,8 +764,12 @@ def __main__(): check_package_latest_version(packages) print("Calculate stats") stats = calculate_stats(packages) - print("Write HTML") - dump_html(packages, stats, args.output) + if args.html: + print("Write HTML") + dump_html(packages, stats, args.html) + if args.json: + print("Write JSON") + dump_json(packages, stats, args.json) __main__()