From patchwork Thu Feb 15 22:03:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 874161 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zj9Kl5vg6z9t2f for ; Fri, 16 Feb 2018 09:04:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id EBE2288BCB; Thu, 15 Feb 2018 22:04:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U7gzSDkQWUyE; Thu, 15 Feb 2018 22:04:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id ABECE88BA5; Thu, 15 Feb 2018 22:04:18 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id D3CDD1C0015 for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CF86B8937D for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DmeUMXrJMMtF for ; Thu, 15 Feb 2018 22:04:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by whitealder.osuosl.org (Postfix) with ESMTP id 9FD898937F for ; Thu, 15 Feb 2018 22:04:11 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 6BA13207FE; Thu, 15 Feb 2018 23:04:10 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id 14694207FD; Thu, 15 Feb 2018 23:03:50 +0100 (CET) From: Thomas Petazzoni To: Buildroot List Date: Thu, 15 Feb 2018 23:03:41 +0100 Message-Id: <20180215220345.8532-2-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> References: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next 1/5] support/scripts/pkg-stats-new: rewrite in Python X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds a new version of the pkg-stats script, rewritten in Python. It is for now implemented in a separate file called, pkg-stats-new, in order to make the diff easily readable. A future commit will rename it to pkg-stats. Compared to the existing shell-based pkg-stats script, the functionality and output is basically the same. The main difference is that the output no longer goes to stdout, but to the file passed as argument using the -o option. This allows stdout to be used for more debugging related information. The way the script works is that a first function get_pkglist() creates a dict associating package names with an instance of a Package() object, containing basic information about the package. Then a number of other functions (add_infra_info, add_pkg_make_info, add_hash_info, add_patch_count, add_check_package_warnings) will calculate additional information about packages, and fill in fields in the Package objects. calculate_stats() then calculates global statistics (how packages have license information, how packages have a hash file, etc.). Finally, dump_html() produces the HTML output, using a number of sub-functions. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats-new | 424 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 424 insertions(+) create mode 100755 support/scripts/pkg-stats-new diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new new file mode 100755 index 0000000000..a5e056a948 --- /dev/null +++ b/support/scripts/pkg-stats-new @@ -0,0 +1,424 @@ +#!/usr/bin/env python + +import argparse +import fnmatch +import os +from collections import defaultdict +import re +import subprocess +import sys + +class Package: + def __init__(self, name): + self.name = name + self.path = None + self.infras = None + self.has_license = False + self.has_license_files = False + self.has_hash = False + self.patch_count = 0 + self.warnings = 0 + def __str__(self): + return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ + (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + +# +# Builds the list of Buildroot packages, returning a dict, where the +# key is the package name, and the value is an instance of the Package +# object. Only the .name and .path fields of the Package object are +# initialized. +# +# npackages: limit to N packages +# package_list: limit to those packages in this list +# +def get_pkglist(): + WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] + WALK_EXCLUDES = [ "boot/common.mk", + "linux/linux-ext-.*.mk", + "package/freescale-imx/freescale-imx.mk", + "package/gcc/gcc.mk", + "package/gstreamer/gstreamer.mk", + "package/gstreamer1/gstreamer1.mk", + "package/gtk2-themes/gtk2-themes.mk", + "package/matchbox/matchbox.mk", + "package/opengl/opengl.mk", + "package/qt5/qt5.mk", + "package/x11r7/x11r7.mk", + "package/doc-asciidoc.mk", + "package/pkg-.*.mk", + "package/nvidia-tegra23/nvidia-tegra23.mk", + "toolchain/toolchain-external/pkg-toolchain-external.mk", + "toolchain/toolchain-external/toolchain-external.mk", + "toolchain/toolchain.mk", + "toolchain/helpers.mk", + "toolchain/toolchain-wrapper.mk" ] + packages = dict() + for root, dirs, files in os.walk("."): + rootdir = root.split("/") + if len(rootdir) < 2: + continue + if rootdir[1] not in WALK_USEFUL_SUBDIRS: + continue + for f in files: + if not f.endswith(".mk"): + continue + # Strip ending ".mk" + pkgname = f[:-3] + pkgpath = os.path.join(root, f) + skip = False + for exclude in WALK_EXCLUDES: + # pkgpath[2:] strips the initial './' + if re.match(exclude, pkgpath[2:]): + skip = True + continue + if skip: + continue + p = Package(pkgname) + p.path = pkgpath + packages[pkgname] = p + return packages + +INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") + +def get_pkg_infra_info(pkgpath): + infras = list() + with open(pkgpath, 'r') as f: + lines = f.readlines() + for l in lines: + match = INFRA_RE.match(l) + if not match: + continue + infra = match.group(1) + if infra.startswith("host-"): + infras.append(("host", infra[5:])) + else: + infras.append(("target", infra)) + return infras + +# Fills in the .infras field of all Package objects +def add_infra_info(packages): + for name, pkg in packages.iteritems(): + pkg.infras = get_pkg_infra_info(pkg.path) + +def pkgname_to_pkgvar(pkgname): + return pkgname.upper().replace("-", "_") + +# Fills in the .has_license and .has_license_files fields of all +# Package objects +def add_pkg_make_info(packages): + licenses = list() + license_files = list() + + # Licenses + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + # Strip _LICENSE + pkgvar = pkgvar[:-8] + + # If value is "unknown", no license details available + if value == "unknown": + continue + licenses.append(pkgvar) + + # License files + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE_FILES"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): + continue + + # Strip _LICENSE_FILES + pkgvar = pkgvar[:-14] + + license_files.append(pkgvar) + + for name, pkg in packages.iteritems(): + var = pkgname_to_pkgvar(name) + if var in licenses: + pkg.has_license = True + if var in license_files: + pkg.has_license_files = True + +# Fills in the .has_hash field of all Package objects +def add_hash_info(packages): + for name, pkg in packages.iteritems(): + hashpath = pkg.path.replace(".mk", ".hash") + pkg.has_hash = os.path.exists(hashpath) + +# Fills in the .patch_count field of all Package objects +def add_patch_count(packages): + for name, pkg in packages.iteritems(): + pkgdir = os.path.dirname(pkg.path) + pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch')) + +def get_check_package_warnings(pkgdir): + cmd = ["./utils/check-package"] + for root, dirs, files in os.walk(pkgdir): + for f in files: + if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": + cmd.append(f) + o = subprocess.check_output(cmd, stderr=subprocess.STDOUT) + lines = o.splitlines() + for line in lines: + m = re.match("^([0-9]*) warnings generated", line) + if m: + return int(m.group(1)) + return None + +# Fills in the .warnings field of all Package objects +def add_check_package_warnings(packages): + for name, pkg in packages.iteritems(): + pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path)) + + +def calculate_stats(packages): + stats = defaultdict(int) + for name, pkg in packages.iteritems(): + # If packages have multiple infra, take the first one. For the + # vast majority of packages, the target and host infra are the + # same. There are very few packages that use a different infra + # for the host and target variants. + if len(pkg.infras) > 0: + infra = pkg.infras[0][1] + stats["infra-%s" % infra] += 1 + else: + stats["infra-unknown"] += 1 + if pkg.has_license: + stats["license"] += 1 + else: + stats["no-license"] += 1 + if pkg.has_license_files: + stats["license-files"] += 1 + else: + stats["no-license-files"] += 1 + if pkg.has_hash: + stats["hash"] += 1 + else: + stats["no-hash"] += 1 + stats["patches"] += pkg.patch_count + return stats + +html_header = """ + + + +Statistics of Buildroot packages + + +Results
+ +

+""" + +html_footer = """ + + + + +""" + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + +def dump_html_pkg(f, pkg): + f.write(" ") + f.write(" %s" % pkg.path[2:]) + + # Patch count + td_class = ["centered"] + if pkg.patch_count == 0: + td_class.append("nopatches") + elif pkg.patch_count < 5: + td_class.append("somepatches") + else: + td_class.append("lotsofpatches") + f.write(" %s" % + (" ".join(td_class), str(pkg.patch_count))) + + # Infrastructure + infra = infra_str(pkg.infras) + td_class = ["centered"] + if infra == "Unknown": + td_class.append("wrong") + else: + td_class.append("correct") + f.write(" %s" % \ + (" ".join(td_class), infra_str(pkg.infras))) + + # License + td_class = ["centered"] + if pkg.has_license: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_license))) + + # License files + td_class = ["centered"] + if pkg.has_license_files: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_license_files))) + + # Hash + td_class = ["centered"] + if pkg.has_hash: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_hash))) + + # Warnings + td_class = ["centered"] + if pkg.warnings == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %d" % \ + (" ".join(td_class), pkg.warnings)) + + f.write(" ") + +def dump_html_all_pkgs(f, packages): + f.write(""" + + + + + + + + + +""") + for name, pkg in sorted(packages.iteritems()): + dump_html_pkg(f, pkg) + f.write("
PackagePatch countInfrastructureLicenseLicense filesHash fileWarnings
") + +def dump_html_stats(f, stats): + f.write("") + infras = [ infra[6:] for infra in stats.keys() if infra.startswith("infra-") ] + for infra in infras: + f.write("" % \ + (infra, stats["infra-%s" % infra])) + f.write("" % + stats["license"]) + f.write("" % + stats["no-license"]) + f.write("" % + stats["license-files"]) + f.write("" % + stats["no-license-files"]) + f.write("" % + stats["hash"]) + f.write("" % + stats["no-hash"]) + f.write("" % + stats["patches"]) + f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
") + +def dump_html(packages, stats, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_stats(f, stats) + f.write(html_footer) + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-o', dest='output', action='store', required=True, + help='HTML output file') + return parser.parse_args() + +def __main__(): + args = parse_args() + print "Build package list ..." + packages = get_pkglist() + print "Get package infra ..." + add_infra_info(packages) + print "Get make info ..." + add_pkg_make_info(packages) + print "Get hash info ..." + add_hash_info(packages) + print "Get patch count ..." + add_patch_count(packages) + print "Get package warnings ..." + add_check_package_warnings(packages) + print "Calculate stats" + stats = calculate_stats(packages) + print "Write HTML" + dump_html(packages, stats, args.output) + +__main__() From patchwork Thu Feb 15 22:03:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 874159 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zj9Kd6ml8z9t2f for ; Fri, 16 Feb 2018 09:04:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 51F518937F; Thu, 15 Feb 2018 22:04:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j7ewy3R+5HMz; Thu, 15 Feb 2018 22:04:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 0AFA589382; Thu, 15 Feb 2018 22:04:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6A2BD1C0015 for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 657A28937D for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TqVuAAbgKoYi for ; Thu, 15 Feb 2018 22:04:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by whitealder.osuosl.org (Postfix) with ESMTP id A432789382 for ; Thu, 15 Feb 2018 22:04:11 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 4B2EC207FC; Thu, 15 Feb 2018 23:04:10 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id 58B6D207FE; Thu, 15 Feb 2018 23:03:50 +0100 (CET) From: Thomas Petazzoni To: Buildroot List Date: Thu, 15 Feb 2018 23:03:42 +0100 Message-Id: <20180215220345.8532-3-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> References: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next 2/5] support/scripts/pkg-stats-new: add -n and -p options X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds the following options to the pkg-stats-new script: -n, to specify a number of packages to parse instead of all packages -p, to specify a list of packages (comma-separated) to parse instead of all packages These options are basically only useful when debugging/developping this script, but they are very useful, because the script is rather slow to run completely with all 2000+ packages, especially once upstream versions will be fetched from release-monitoring.org. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats-new | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index a5e056a948..d018e1fed4 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -31,7 +31,7 @@ class Package: # npackages: limit to N packages # package_list: limit to those packages in this list # -def get_pkglist(): +def get_pkglist(npackages, package_list): WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] WALK_EXCLUDES = [ "boot/common.mk", "linux/linux-ext-.*.mk", @@ -53,6 +53,7 @@ def get_pkglist(): "toolchain/helpers.mk", "toolchain/toolchain-wrapper.mk" ] packages = dict() + count = 0 for root, dirs, files in os.walk("."): rootdir = root.split("/") if len(rootdir) < 2: @@ -64,6 +65,8 @@ def get_pkglist(): continue # Strip ending ".mk" pkgname = f[:-3] + if package_list and pkgname not in package_list: + continue pkgpath = os.path.join(root, f) skip = False for exclude in WALK_EXCLUDES: @@ -76,6 +79,9 @@ def get_pkglist(): p = Package(pkgname) p.path = pkgpath packages[pkgname] = p + count += 1 + if npackages and count == npackages: + return packages return packages INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") @@ -400,12 +406,23 @@ def parse_args(): parser = argparse.ArgumentParser() parser.add_argument('-o', dest='output', action='store', required=True, help='HTML output file') + parser.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + parser.add_argument('-p', dest='packages', action='store', + help='List of packages') return parser.parse_args() def __main__(): args = parse_args() + if args.npackages and args.packages: + print "ERROR: -n and -p are mutually exclusive" + sys.exit(1) + if args.packages: + package_list = args.packages.split(",") + else: + package_list = None print "Build package list ..." - packages = get_pkglist() + packages = get_pkglist(args.npackages, package_list) print "Get package infra ..." add_infra_info(packages) print "Get make info ..." From patchwork Thu Feb 15 22:03:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 874160 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zj9Kg2LD1z9t2f for ; Fri, 16 Feb 2018 09:04:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 568F988BC1; Thu, 15 Feb 2018 22:04:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DHyoL8_ElLWA; Thu, 15 Feb 2018 22:04:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E9ABD88B97; Thu, 15 Feb 2018 22:04:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 8173C1CF059 for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 7D8FA89C8C for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U+Hv40hYFz+L for ; Thu, 15 Feb 2018 22:04:12 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by hemlock.osuosl.org (Postfix) with ESMTP id 712F789ACC for ; Thu, 15 Feb 2018 22:04:12 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 14C22207FD; Thu, 15 Feb 2018 23:04:11 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id A120820804; Thu, 15 Feb 2018 23:03:50 +0100 (CET) From: Thomas Petazzoni To: Buildroot List Date: Thu, 15 Feb 2018 23:03:43 +0100 Message-Id: <20180215220345.8532-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> References: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next 3/5] support/scripts/pkg-stats-new: add current version information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds a new column in the HTML output containing the current version of a package in Buildroot. As such, it isn't terribly useful, but combined with the latest upstream version added in a follow-up commit, it will become very useful. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats-new | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index d018e1fed4..32227a8906 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -18,6 +18,7 @@ class Package: self.has_hash = False self.patch_count = 0 self.warnings = 0 + self.current_version = None def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) @@ -114,6 +115,7 @@ def pkgname_to_pkgvar(pkgname): def add_pkg_make_info(packages): licenses = list() license_files = list() + versions = dict() # Licenses o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE"]) @@ -151,12 +153,32 @@ def add_pkg_make_info(packages): license_files.append(pkgvar) + # Version + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_VERSION"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_DL_VERSION"): + continue + + # Strip _VERSION + pkgvar = pkgvar[:-8] + + versions[pkgvar] = value + for name, pkg in packages.iteritems(): var = pkgname_to_pkgvar(name) if var in licenses: pkg.has_license = True if var in license_files: pkg.has_license_files = True + if versions.has_key(var): + pkg.current_version = versions[var] # Fills in the .has_hash field of all Package objects def add_hash_info(packages): @@ -346,6 +368,9 @@ def dump_html_pkg(f, pkg): f.write(" %s" % \ (" ".join(td_class), boolean_str(pkg.has_hash))) + # Current version + f.write(" %s" % pkg.current_version) + # Warnings td_class = ["centered"] if pkg.warnings == 0: @@ -367,6 +392,7 @@ def dump_html_all_pkgs(f, packages): License License files Hash file +Current version Warnings """) for name, pkg in sorted(packages.iteritems()): From patchwork Thu Feb 15 22:03:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 874162 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zj9Kp16Ysz9t3G for ; Fri, 16 Feb 2018 09:04:30 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 49B6288BA0; Thu, 15 Feb 2018 22:04:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AhlU-aft57ya; Thu, 15 Feb 2018 22:04:22 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0F47288BA1; Thu, 15 Feb 2018 22:04:22 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 03CA01C0015 for ; Thu, 15 Feb 2018 22:04:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id F3E5F30770 for ; Thu, 15 Feb 2018 22:04:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AWBWd5b2cDrI for ; Thu, 15 Feb 2018 22:04:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by silver.osuosl.org (Postfix) with ESMTP id D2FF026F66 for ; Thu, 15 Feb 2018 22:04:12 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 2ED8D20804; Thu, 15 Feb 2018 23:04:11 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id E58B620807; Thu, 15 Feb 2018 23:03:50 +0100 (CET) From: Thomas Petazzoni To: Buildroot List Date: Thu, 15 Feb 2018 23:03:44 +0100 Message-Id: <20180215220345.8532-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> References: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next 4/5] support/scripts/pkg-stats-new: add latest upstream version information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. If there is no result, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Since release-monitoring.org is a bit slow, we have 8 threads that fetch information in parallel. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats-new | 120 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index 32227a8906..0d682656df 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -7,6 +7,10 @@ from collections import defaultdict import re import subprocess import sys +import json +import urllib2 +from Queue import Queue +from threading import Thread class Package: def __init__(self, name): @@ -19,6 +23,7 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.latest_version = None def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) @@ -211,6 +216,70 @@ def add_check_package_warnings(packages): for name, pkg in packages.iteritems(): pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path)) +RELEASE_MONITORING_API = "http://release-monitoring.org/api" + +def get_latest_version_by_distro(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + if len(data['versions']) > 0: + return (True, data['versions'][0], data['id']) + else: + return (True, None, data['id']) + +def get_latest_version_by_guess(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + for p in data['projects']: + if p['name'] == package and len(p['versions']) > 0: + return (False, p['versions'][0], p['id']) + return (False, None, None) + +def get_latest_version(package): + try: + # We first try by using the "Buildroot" distribution on + # release-monitoring.org, if it has a mapping for the current + # package name. + return get_latest_version_by_distro(package) + except urllib2.HTTPError, e: + # If that fails because there is no mapping, we try to search + # in all packages for a package of this name. + if e.code == 404: + return get_latest_version_by_guess(package) + else: + return (False, None, None) + +def get_version_worker(q): + while True: + name, pkg = q.get() + pkg.latest_version = get_latest_version(name) + print " [%04d] %s => %s" % (q.qsize(), name, str(pkg.latest_version)) + q.task_done() + +# Fills in the .latest_version field of all Package objects +# +# This field has a special format: +# (mapping, version, id) +# with: +# - mapping: boolean that indicates whether release-monitoring.org +# has a mapping for this package name in the Buildroot distribution +# or not +# - version: string containing the latest version known by +# release-monitoring.org for this package +# - id: string containing the id of the project corresponding to this +# package, as known by release-monitoring.org +def add_latest_version_info(packages): + q = Queue() + for name, pkg in packages.iteritems(): + q.put((name, pkg)) + # Since release-monitoring.org is rather slow, we create 8 threads + # that do HTTP requests to the site. + for i in range(8): + t = Thread(target=get_version_worker, args=[q]) + t.daemon = True + t.start() + q.join() def calculate_stats(packages): stats = defaultdict(int) @@ -236,6 +305,16 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.latest_version[0]: + stats["rmo-mapping"] += 1 + else: + stats["rmo-no-mapping"] += 1 + if not pkg.latest_version[1]: + stats["version-unknown"] += 1 + elif pkg.latest_version[1] == pkg.current_version: + stats["version-uptodate"] += 1 + else: + stats["version-not-uptodate"] += 1 stats["patches"] += pkg.patch_count return stats @@ -371,6 +450,34 @@ def dump_html_pkg(f, pkg): # Current version f.write(" %s" % pkg.current_version) + # Latest version + if pkg.latest_version[1] == None: + td_class.append("version-unknown") + elif pkg.latest_version[1] != pkg.current_version: + td_class.append("version-needs-update") + else: + td_class.append("version-good") + + if pkg.latest_version[1] is None: + latest_version_text = "Unknown" + else: + latest_version_text = "%s" % str(pkg.latest_version[1]) + + latest_version_text += "
" + + if pkg.latest_version[2]: + latest_version_text += "link, " % pkg.latest_version[2] + else: + latest_version_text += "no link, " + + if pkg.latest_version[0]: + latest_version_text += "has mapping" + else: + latest_version_text += "has no mapping" + + f.write(" %s" % \ + (" ".join(td_class), latest_version_text)) + # Warnings td_class = ["centered"] if pkg.warnings == 0: @@ -393,6 +500,7 @@ def dump_html_all_pkgs(f, packages): License files Hash file Current version +Latest version Warnings """) for name, pkg in sorted(packages.iteritems()): @@ -419,6 +527,16 @@ def dump_html_stats(f, stats): stats["no-hash"]) f.write("Total number of patches%s" % stats["patches"]) + f.write("Packages having a mapping on release-monitoring.org%s" % + stats["rmo-mapping"]) + f.write("Packages lacking a mapping on release-monitoring.org%s" % + stats["rmo-no-mapping"]) + f.write("Packages that are up-to-date%s" % + stats["version-uptodate"]) + f.write("Packages that are not up-to-date%s" % + stats["version-not-uptodate"]) + f.write("Packages with no known upstream version%s" % + stats["version-unknown"]) f.write("") def dump_html(packages, stats, output): @@ -459,6 +577,8 @@ def __main__(): add_patch_count(packages) print "Get package warnings ..." add_check_package_warnings(packages) + print "Get latest version ..." + add_latest_version_info(packages) print "Calculate stats" stats = calculate_stats(packages) print "Write HTML" From patchwork Thu Feb 15 22:03:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 874163 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zj9LV4mxlz9t3G for ; Fri, 16 Feb 2018 09:05:06 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D434B8955A; Thu, 15 Feb 2018 22:05:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4i-BUEmZai-7; Thu, 15 Feb 2018 22:04:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id C5BD0893BA; Thu, 15 Feb 2018 22:04:27 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id D42A51C0015 for ; Thu, 15 Feb 2018 22:04:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CF88D8955B for ; Thu, 15 Feb 2018 22:04:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qJuU-kb20TG5 for ; Thu, 15 Feb 2018 22:04:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by whitealder.osuosl.org (Postfix) with ESMTP id A2D468937D for ; Thu, 15 Feb 2018 22:04:18 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 7562C207F4; Thu, 15 Feb 2018 23:04:17 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id 32E8D2080B; Thu, 15 Feb 2018 23:03:51 +0100 (CET) From: Thomas Petazzoni To: Buildroot List Date: Thu, 15 Feb 2018 23:03:45 +0100 Message-Id: <20180215220345.8532-6-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> References: <20180215220345.8532-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next 5/5] support/scripts/pkg-stats: replace with new Python version X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Rename pkg-stats-new to pkg-stats. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats | 946 ++++++++++++++++++++++++------------------ support/scripts/pkg-stats-new | 587 -------------------------- 2 files changed, 539 insertions(+), 994 deletions(-) delete mode 100755 support/scripts/pkg-stats-new diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 48a2cc29a1..0d682656df 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -1,31 +1,325 @@ -#!/usr/bin/env bash +#!/usr/bin/env python + +import argparse +import fnmatch +import os +from collections import defaultdict +import re +import subprocess +import sys +import json +import urllib2 +from Queue import Queue +from threading import Thread + +class Package: + def __init__(self, name): + self.name = name + self.path = None + self.infras = None + self.has_license = False + self.has_license_files = False + self.has_hash = False + self.patch_count = 0 + self.warnings = 0 + self.current_version = None + self.latest_version = None + def __str__(self): + return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ + (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) -# Copyright (C) 2009 by Thomas Petazzoni # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. +# Builds the list of Buildroot packages, returning a dict, where the +# key is the package name, and the value is an instance of the Package +# object. Only the .name and .path fields of the Package object are +# initialized. # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. +# npackages: limit to N packages +# package_list: limit to those packages in this list # -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -# This script generates an HTML file that contains a report about all -# Buildroot packages, their usage of the different package -# infrastructure and possible cleanup actions -# -# Run the script from the Buildroot toplevel directory: -# -# ./support/scripts/pkg-stats > /tmp/pkg.html +def get_pkglist(npackages, package_list): + WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] + WALK_EXCLUDES = [ "boot/common.mk", + "linux/linux-ext-.*.mk", + "package/freescale-imx/freescale-imx.mk", + "package/gcc/gcc.mk", + "package/gstreamer/gstreamer.mk", + "package/gstreamer1/gstreamer1.mk", + "package/gtk2-themes/gtk2-themes.mk", + "package/matchbox/matchbox.mk", + "package/opengl/opengl.mk", + "package/qt5/qt5.mk", + "package/x11r7/x11r7.mk", + "package/doc-asciidoc.mk", + "package/pkg-.*.mk", + "package/nvidia-tegra23/nvidia-tegra23.mk", + "toolchain/toolchain-external/pkg-toolchain-external.mk", + "toolchain/toolchain-external/toolchain-external.mk", + "toolchain/toolchain.mk", + "toolchain/helpers.mk", + "toolchain/toolchain-wrapper.mk" ] + packages = dict() + count = 0 + for root, dirs, files in os.walk("."): + rootdir = root.split("/") + if len(rootdir) < 2: + continue + if rootdir[1] not in WALK_USEFUL_SUBDIRS: + continue + for f in files: + if not f.endswith(".mk"): + continue + # Strip ending ".mk" + pkgname = f[:-3] + if package_list and pkgname not in package_list: + continue + pkgpath = os.path.join(root, f) + skip = False + for exclude in WALK_EXCLUDES: + # pkgpath[2:] strips the initial './' + if re.match(exclude, pkgpath[2:]): + skip = True + continue + if skip: + continue + p = Package(pkgname) + p.path = pkgpath + packages[pkgname] = p + count += 1 + if npackages and count == npackages: + return packages + return packages + +INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") + +def get_pkg_infra_info(pkgpath): + infras = list() + with open(pkgpath, 'r') as f: + lines = f.readlines() + for l in lines: + match = INFRA_RE.match(l) + if not match: + continue + infra = match.group(1) + if infra.startswith("host-"): + infras.append(("host", infra[5:])) + else: + infras.append(("target", infra)) + return infras + +# Fills in the .infras field of all Package objects +def add_infra_info(packages): + for name, pkg in packages.iteritems(): + pkg.infras = get_pkg_infra_info(pkg.path) + +def pkgname_to_pkgvar(pkgname): + return pkgname.upper().replace("-", "_") + +# Fills in the .has_license and .has_license_files fields of all +# Package objects +def add_pkg_make_info(packages): + licenses = list() + license_files = list() + versions = dict() + + # Licenses + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + # Strip _LICENSE + pkgvar = pkgvar[:-8] + + # If value is "unknown", no license details available + if value == "unknown": + continue + licenses.append(pkgvar) + + # License files + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE_FILES"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): + continue + + # Strip _LICENSE_FILES + pkgvar = pkgvar[:-14] + + license_files.append(pkgvar) + + # Version + o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_VERSION"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_DL_VERSION"): + continue + + # Strip _VERSION + pkgvar = pkgvar[:-8] + + versions[pkgvar] = value + + for name, pkg in packages.iteritems(): + var = pkgname_to_pkgvar(name) + if var in licenses: + pkg.has_license = True + if var in license_files: + pkg.has_license_files = True + if versions.has_key(var): + pkg.current_version = versions[var] + +# Fills in the .has_hash field of all Package objects +def add_hash_info(packages): + for name, pkg in packages.iteritems(): + hashpath = pkg.path.replace(".mk", ".hash") + pkg.has_hash = os.path.exists(hashpath) + +# Fills in the .patch_count field of all Package objects +def add_patch_count(packages): + for name, pkg in packages.iteritems(): + pkgdir = os.path.dirname(pkg.path) + pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch')) + +def get_check_package_warnings(pkgdir): + cmd = ["./utils/check-package"] + for root, dirs, files in os.walk(pkgdir): + for f in files: + if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": + cmd.append(f) + o = subprocess.check_output(cmd, stderr=subprocess.STDOUT) + lines = o.splitlines() + for line in lines: + m = re.match("^([0-9]*) warnings generated", line) + if m: + return int(m.group(1)) + return None + +# Fills in the .warnings field of all Package objects +def add_check_package_warnings(packages): + for name, pkg in packages.iteritems(): + pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path)) + +RELEASE_MONITORING_API = "http://release-monitoring.org/api" + +def get_latest_version_by_distro(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + if len(data['versions']) > 0: + return (True, data['versions'][0], data['id']) + else: + return (True, None, data['id']) + +def get_latest_version_by_guess(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + for p in data['projects']: + if p['name'] == package and len(p['versions']) > 0: + return (False, p['versions'][0], p['id']) + return (False, None, None) + +def get_latest_version(package): + try: + # We first try by using the "Buildroot" distribution on + # release-monitoring.org, if it has a mapping for the current + # package name. + return get_latest_version_by_distro(package) + except urllib2.HTTPError, e: + # If that fails because there is no mapping, we try to search + # in all packages for a package of this name. + if e.code == 404: + return get_latest_version_by_guess(package) + else: + return (False, None, None) + +def get_version_worker(q): + while True: + name, pkg = q.get() + pkg.latest_version = get_latest_version(name) + print " [%04d] %s => %s" % (q.qsize(), name, str(pkg.latest_version)) + q.task_done() + +# Fills in the .latest_version field of all Package objects # - -echo " +# This field has a special format: +# (mapping, version, id) +# with: +# - mapping: boolean that indicates whether release-monitoring.org +# has a mapping for this package name in the Buildroot distribution +# or not +# - version: string containing the latest version known by +# release-monitoring.org for this package +# - id: string containing the id of the project corresponding to this +# package, as known by release-monitoring.org +def add_latest_version_info(packages): + q = Queue() + for name, pkg in packages.iteritems(): + q.put((name, pkg)) + # Since release-monitoring.org is rather slow, we create 8 threads + # that do HTTP requests to the site. + for i in range(8): + t = Thread(target=get_version_worker, args=[q]) + t.daemon = True + t.start() + q.join() + +def calculate_stats(packages): + stats = defaultdict(int) + for name, pkg in packages.iteritems(): + # If packages have multiple infra, take the first one. For the + # vast majority of packages, the target and host infra are the + # same. There are very few packages that use a different infra + # for the host and target variants. + if len(pkg.infras) > 0: + infra = pkg.infras[0][1] + stats["infra-%s" % infra] += 1 + else: + stats["infra-unknown"] += 1 + if pkg.has_license: + stats["license"] += 1 + else: + stats["no-license"] += 1 + if pkg.has_license_files: + stats["license-files"] += 1 + else: + stats["no-license-files"] += 1 + if pkg.has_hash: + stats["hash"] += 1 + else: + stats["no-hash"] += 1 + if pkg.latest_version[0]: + stats["rmo-mapping"] += 1 + else: + stats["rmo-no-mapping"] += 1 + if not pkg.latest_version[1]: + stats["version-unknown"] += 1 + elif pkg.latest_version[1] == pkg.current_version: + stats["version-uptodate"] += 1 + else: + stats["version-not-uptodate"] += 1 + stats["patches"] += pkg.patch_count + return stats + +html_header = """ + Statistics of Buildroot packages @@ -61,395 +362,226 @@ td.lotsofpatches { Results

+""" +html_footer = """ + + + + +""" + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + +def dump_html_pkg(f, pkg): + f.write(" ") + f.write(" %s" % pkg.path[2:]) + + # Patch count + td_class = ["centered"] + if pkg.patch_count == 0: + td_class.append("nopatches") + elif pkg.patch_count < 5: + td_class.append("somepatches") + else: + td_class.append("lotsofpatches") + f.write(" %s" % + (" ".join(td_class), str(pkg.patch_count))) + + # Infrastructure + infra = infra_str(pkg.infras) + td_class = ["centered"] + if infra == "Unknown": + td_class.append("wrong") + else: + td_class.append("correct") + f.write(" %s" % \ + (" ".join(td_class), infra_str(pkg.infras))) + + # License + td_class = ["centered"] + if pkg.has_license: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_license))) + + # License files + td_class = ["centered"] + if pkg.has_license_files: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_license_files))) + + # Hash + td_class = ["centered"] + if pkg.has_hash: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s" % \ + (" ".join(td_class), boolean_str(pkg.has_hash))) + + # Current version + f.write(" %s" % pkg.current_version) + + # Latest version + if pkg.latest_version[1] == None: + td_class.append("version-unknown") + elif pkg.latest_version[1] != pkg.current_version: + td_class.append("version-needs-update") + else: + td_class.append("version-good") + + if pkg.latest_version[1] is None: + latest_version_text = "Unknown" + else: + latest_version_text = "%s" % str(pkg.latest_version[1]) + + latest_version_text += "
" + + if pkg.latest_version[2]: + latest_version_text += "link, " % pkg.latest_version[2] + else: + latest_version_text += "no link, " + + if pkg.latest_version[0]: + latest_version_text += "has mapping" + else: + latest_version_text += "has no mapping" + + f.write(" %s" % \ + (" ".join(td_class), latest_version_text)) + + # Warnings + td_class = ["centered"] + if pkg.warnings == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %d" % \ + (" ".join(td_class), pkg.warnings)) + + f.write(" ") + +def dump_html_all_pkgs(f, packages): + f.write(""" - + + - -" - -autotools_packages=0 -cmake_packages=0 -kconfig_packages=0 -luarocks_package=0 -perl_packages=0 -python_packages=0 -rebar_packages=0 -virtual_packages=0 -generic_packages=0 -waf_packages=0 -manual_packages=0 -packages_with_licence=0 -packages_without_licence=0 -packages_with_license_files=0 -packages_without_license_files=0 -packages_with_hash_file=0 -packages_without_hash_file=0 -total_patch_count=0 -cnt=0 - -for i in $(find boot/ linux/ package/ toolchain/ -name '*.mk' | sort) ; do - - if test \ - $i = "boot/common.mk" -o \ - $i = "linux/linux-ext-ev3dev-linux-drivers.mk" -o \ - $i = "linux/linux-ext-fbtft.mk" -o \ - $i = "linux/linux-ext-xenomai.mk" -o \ - $i = "linux/linux-ext-rtai.mk" -o \ - $i = "package/freescale-imx/freescale-imx.mk" -o \ - $i = "package/gcc/gcc.mk" -o \ - $i = "package/gstreamer/gstreamer.mk" -o \ - $i = "package/gstreamer1/gstreamer1.mk" -o \ - $i = "package/gtk2-themes/gtk2-themes.mk" -o \ - $i = "package/matchbox/matchbox.mk" -o \ - $i = "package/opengl/opengl.mk" -o \ - $i = "package/qt5/qt5.mk" -o \ - $i = "package/x11r7/x11r7.mk" -o \ - $i = "package/doc-asciidoc.mk" -o \ - $i = "package/pkg-autotools.mk" -o \ - $i = "package/pkg-cmake.mk" -o \ - $i = "package/pkg-kconfig.mk" -o \ - $i = "package/pkg-luarocks.mk" -o \ - $i = "package/pkg-perl.mk" -o \ - $i = "package/pkg-python.mk" -o \ - $i = "package/pkg-rebar.mk" -o \ - $i = "package/pkg-virtual.mk" -o \ - $i = "package/pkg-download.mk" -o \ - $i = "package/pkg-generic.mk" -o \ - $i = "package/pkg-waf.mk" -o \ - $i = "package/pkg-kernel-module.mk" -o \ - $i = "package/pkg-utils.mk" -o \ - $i = "package/nvidia-tegra23/nvidia-tegra23.mk" -o \ - $i = "toolchain/toolchain-external/pkg-toolchain-external.mk" -o \ - $i = "toolchain/toolchain-external/toolchain-external.mk" -o \ - $i = "toolchain/toolchain.mk" -o \ - $i = "toolchain/helpers.mk" -o \ - $i = "toolchain/toolchain-wrapper.mk" ; then - echo "skipping $i" 1>&2 - continue - fi - - cnt=$((cnt+1)) - - hashost=0 - hastarget=0 - infratype="" - - # Determine package infrastructure - if grep -E "\(host-autotools-package\)" $i > /dev/null ; then - infratype="autotools" - hashost=1 - fi - - if grep -E "\(autotools-package\)" $i > /dev/null ; then - infratype="autotools" - hastarget=1 - fi - - if grep -E "\(kconfig-package\)" $i > /dev/null ; then - infratype="kconfig" - hastarget=1 - fi - - if grep -E "\(host-luarocks-package\)" $i > /dev/null ; then - infratype="luarocks" - hashost=1 - fi - - if grep -E "\(luarocks-package\)" $i > /dev/null ; then - infratype="luarocks" - hastarget=1 - fi - - if grep -E "\(host-perl-package\)" $i > /dev/null ; then - infratype="perl" - hashost=1 - fi - - if grep -E "\(perl-package\)" $i > /dev/null ; then - infratype="perl" - hastarget=1 - fi - - if grep -E "\(host-python-package\)" $i > /dev/null ; then - infratype="python" - hashost=1 - fi - - if grep -E "\(python-package\)" $i > /dev/null ; then - infratype="python" - hastarget=1 - fi - - if grep -E "\(host-rebar-package\)" $i > /dev/null ; then - infratype="rebar" - hashost=1 - fi - - if grep -E "\(rebar-package\)" $i > /dev/null ; then - infratype="rebar" - hastarget=1 - fi - - if grep -E "\(host-virtual-package\)" $i > /dev/null ; then - infratype="virtual" - hashost=1 - fi - - if grep -E "\(virtual-package\)" $i > /dev/null ; then - infratype="virtual" - hastarget=1 - fi - - if grep -E "\(host-generic-package\)" $i > /dev/null ; then - infratype="generic" - hashost=1 - fi - - if grep -E "\(generic-package\)" $i > /dev/null ; then - infratype="generic" - hastarget=1 - fi - - if grep -E "\(host-cmake-package\)" $i > /dev/null ; then - infratype="cmake" - hashost=1 - fi - - if grep -E "\(cmake-package\)" $i > /dev/null ; then - infratype="cmake" - hastarget=1 - fi - - if grep -E "\(toolchain-external-package\)" $i > /dev/null ; then - infratype="toolchain-external" - hastarget=1 - fi - - if grep -E "\(waf-package\)" $i > /dev/null ; then - infratype="waf" - hastarget=1 - fi - - pkg=$(basename $i) - dir=$(dirname $i) - pkg=${pkg%.mk} - pkgvariable=$(echo ${pkg} | tr "a-z-" "A-Z_") - - - # Count packages per infrastructure - if [ -z ${infratype} ] ; then - infratype="manual" - manual_packages=$(($manual_packages+1)) - elif [ ${infratype} = "autotools" ]; then - autotools_packages=$(($autotools_packages+1)) - elif [ ${infratype} = "cmake" ]; then - cmake_packages=$(($cmake_packages+1)) - elif [ ${infratype} = "kconfig" ]; then - kconfig_packages=$(($kconfig_packages+1)) - elif [ ${infratype} = "luarocks" ]; then - luarocks_packages=$(($luarocks_packages+1)) - elif [ ${infratype} = "perl" ]; then - perl_packages=$(($perl_packages+1)) - elif [ ${infratype} = "python" ]; then - python_packages=$(($python_packages+1)) - elif [ ${infratype} = "rebar" ]; then - rebar_packages=$(($rebar_packages+1)) - elif [ ${infratype} = "virtual" ]; then - virtual_packages=$(($virtual_packages+1)) - elif [ ${infratype} = "generic" ]; then - generic_packages=$(($generic_packages+1)) - elif [ ${infratype} = "waf" ]; then - waf_packages=$(($waf_packages+1)) - fi - - if grep -qE "^${pkgvariable}_LICENSE[ ]*=" $i ; then - packages_with_license=$(($packages_with_license+1)) - license=1 - else - packages_without_license=$(($packages_without_license+1)) - license=0 - fi - - if grep -qE "^${pkgvariable}_LICENSE_FILES[ ]*=" $i ; then - packages_with_license_files=$(($packages_with_license_files+1)) - license_files=1 - else - packages_without_license_files=$(($packages_without_license_files+1)) - license_files=0 - fi - - if test -f ${dir}/${pkg}.hash; then - packages_with_hash_file=$(($packages_with_hash_file+1)) - hash_file=1 - else - packages_without_hash_file=$(($packages_without_hash_file+1)) - hash_file=0 - fi - - echo "" - - echo "" - echo "" - - package_dir=$(dirname $i) - patch_count=$(find ${package_dir} -name '*.patch' | wc -l) - total_patch_count=$(($total_patch_count+$patch_count)) - - if test $patch_count -lt 1 ; then - patch_count_class="nopatches" - elif test $patch_count -lt 5 ; then - patch_count_class="somepatches" - else - patch_count_class="lotsofpatches" - fi - - echo "" - - if [ ${infratype} = "manual" ] ; then - echo "" - else - echo "" - fi - - if [ ${license} -eq 0 ] ; then - echo "" - else - echo "" - fi - - if [ ${license_files} -eq 0 ] ; then - echo "" - else - echo "" - fi - - if [ ${hash_file} -eq 0 ] ; then - echo "" - else - echo "" - fi - - file_list=$(find ${package_dir} -name '*.mk' -o -name '*.in*' -o -name '*.hash') - nwarnings=$(./utils/check-package ${file_list} 2>&1 | sed '/\([0-9]*\) warnings generated/!d; s//\1/') - if [ ${nwarnings} -eq 0 ] ; then - echo "" - else - echo "" - fi - - echo "" - -done -echo "
Id Package Patch count Infrastructure License License files Hash fileCurrent versionLatest version Warnings
$cnt$i" - echo "$patch_count" - echo "manual" - echo "${infratype}
" - if [ ${hashost} -eq 1 -a ${hastarget} -eq 1 ]; then - echo "target + host" - elif [ ${hashost} -eq 1 ]; then - echo "host" - else - echo "target" - fi - echo "		NoYesNoYesNoYes${nwarnings}${nwarnings}
" - -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "
Packages using the generic infrastructure$generic_packages
Packages using the cmake infrastructure$cmake_packages
Packages using the autotools infrastructure$autotools_packages
Packages using the luarocks infrastructure$luarocks_packages
Packages using the kconfig infrastructure$kconfig_packages
Packages using the perl infrastructure$perl_packages
Packages using the python infrastructure$python_packages
Packages using the rebar infrastructure$rebar_packages
Packages using the virtual infrastructure$virtual_packages
Packages using the waf infrastructure$waf_packages
Packages not using any infrastructure$manual_packages
Packages having license information$packages_with_license
Packages not having licence information$packages_without_license
Packages having license files information$packages_with_license_files
Packages not having licence files information$packages_without_license_files
Packages having hash file$packages_with_hash_file
Packages not having hash file$packages_without_hash_file
Number of patches in all packages$total_patch_count
TOTAL$cnt
" - -echo "
" -echo "Updated on $(LANG=C date), Git commit $(git log master -n 1 --pretty=format:%H)" -echo "" - -echo " -" -echo "" +""") + for name, pkg in sorted(packages.iteritems()): + dump_html_pkg(f, pkg) + f.write("") + +def dump_html_stats(f, stats): + f.write("") + infras = [ infra[6:] for infra in stats.keys() if infra.startswith("infra-") ] + for infra in infras: + f.write("" % \ + (infra, stats["infra-%s" % infra])) + f.write("" % + stats["license"]) + f.write("" % + stats["no-license"]) + f.write("" % + stats["license-files"]) + f.write("" % + stats["no-license-files"]) + f.write("" % + stats["hash"]) + f.write("" % + stats["no-hash"]) + f.write("" % + stats["patches"]) + f.write("" % + stats["rmo-mapping"]) + f.write("" % + stats["rmo-no-mapping"]) + f.write("" % + stats["version-uptodate"]) + f.write("" % + stats["version-not-uptodate"]) + f.write("" % + stats["version-unknown"]) + f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
Packages having a mapping on release-monitoring.org%s
Packages lacking a mapping on release-monitoring.org%s
Packages that are up-to-date%s
Packages that are not up-to-date%s
Packages with no known upstream version%s
") + +def dump_html(packages, stats, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_stats(f, stats) + f.write(html_footer) + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-o', dest='output', action='store', required=True, + help='HTML output file') + parser.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + parser.add_argument('-p', dest='packages', action='store', + help='List of packages') + return parser.parse_args() + +def __main__(): + args = parse_args() + if args.npackages and args.packages: + print "ERROR: -n and -p are mutually exclusive" + sys.exit(1) + if args.packages: + package_list = args.packages.split(",") + else: + package_list = None + print "Build package list ..." + packages = get_pkglist(args.npackages, package_list) + print "Get package infra ..." + add_infra_info(packages) + print "Get make info ..." + add_pkg_make_info(packages) + print "Get hash info ..." + add_hash_info(packages) + print "Get patch count ..." + add_patch_count(packages) + print "Get package warnings ..." + add_check_package_warnings(packages) + print "Get latest version ..." + add_latest_version_info(packages) + print "Calculate stats" + stats = calculate_stats(packages) + print "Write HTML" + dump_html(packages, stats, args.output) + +__main__() diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new deleted file mode 100755 index 0d682656df..0000000000 --- a/support/scripts/pkg-stats-new +++ /dev/null @@ -1,587 +0,0 @@ -#!/usr/bin/env python - -import argparse -import fnmatch -import os -from collections import defaultdict -import re -import subprocess -import sys -import json -import urllib2 -from Queue import Queue -from threading import Thread - -class Package: - def __init__(self, name): - self.name = name - self.path = None - self.infras = None - self.has_license = False - self.has_license_files = False - self.has_hash = False - self.patch_count = 0 - self.warnings = 0 - self.current_version = None - self.latest_version = None - def __str__(self): - return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) - -# -# Builds the list of Buildroot packages, returning a dict, where the -# key is the package name, and the value is an instance of the Package -# object. Only the .name and .path fields of the Package object are -# initialized. -# -# npackages: limit to N packages -# package_list: limit to those packages in this list -# -def get_pkglist(npackages, package_list): - WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] - WALK_EXCLUDES = [ "boot/common.mk", - "linux/linux-ext-.*.mk", - "package/freescale-imx/freescale-imx.mk", - "package/gcc/gcc.mk", - "package/gstreamer/gstreamer.mk", - "package/gstreamer1/gstreamer1.mk", - "package/gtk2-themes/gtk2-themes.mk", - "package/matchbox/matchbox.mk", - "package/opengl/opengl.mk", - "package/qt5/qt5.mk", - "package/x11r7/x11r7.mk", - "package/doc-asciidoc.mk", - "package/pkg-.*.mk", - "package/nvidia-tegra23/nvidia-tegra23.mk", - "toolchain/toolchain-external/pkg-toolchain-external.mk", - "toolchain/toolchain-external/toolchain-external.mk", - "toolchain/toolchain.mk", - "toolchain/helpers.mk", - "toolchain/toolchain-wrapper.mk" ] - packages = dict() - count = 0 - for root, dirs, files in os.walk("."): - rootdir = root.split("/") - if len(rootdir) < 2: - continue - if rootdir[1] not in WALK_USEFUL_SUBDIRS: - continue - for f in files: - if not f.endswith(".mk"): - continue - # Strip ending ".mk" - pkgname = f[:-3] - if package_list and pkgname not in package_list: - continue - pkgpath = os.path.join(root, f) - skip = False - for exclude in WALK_EXCLUDES: - # pkgpath[2:] strips the initial './' - if re.match(exclude, pkgpath[2:]): - skip = True - continue - if skip: - continue - p = Package(pkgname) - p.path = pkgpath - packages[pkgname] = p - count += 1 - if npackages and count == npackages: - return packages - return packages - -INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") - -def get_pkg_infra_info(pkgpath): - infras = list() - with open(pkgpath, 'r') as f: - lines = f.readlines() - for l in lines: - match = INFRA_RE.match(l) - if not match: - continue - infra = match.group(1) - if infra.startswith("host-"): - infras.append(("host", infra[5:])) - else: - infras.append(("target", infra)) - return infras - -# Fills in the .infras field of all Package objects -def add_infra_info(packages): - for name, pkg in packages.iteritems(): - pkg.infras = get_pkg_infra_info(pkg.path) - -def pkgname_to_pkgvar(pkgname): - return pkgname.upper().replace("-", "_") - -# Fills in the .has_license and .has_license_files fields of all -# Package objects -def add_pkg_make_info(packages): - licenses = list() - license_files = list() - versions = dict() - - # Licenses - o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - # Strip _LICENSE - pkgvar = pkgvar[:-8] - - # If value is "unknown", no license details available - if value == "unknown": - continue - licenses.append(pkgvar) - - # License files - o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_LICENSE_FILES"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): - continue - - # Strip _LICENSE_FILES - pkgvar = pkgvar[:-14] - - license_files.append(pkgvar) - - # Version - o = subprocess.check_output(["make", "-s", "printvars", "VARS=%_VERSION"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_DL_VERSION"): - continue - - # Strip _VERSION - pkgvar = pkgvar[:-8] - - versions[pkgvar] = value - - for name, pkg in packages.iteritems(): - var = pkgname_to_pkgvar(name) - if var in licenses: - pkg.has_license = True - if var in license_files: - pkg.has_license_files = True - if versions.has_key(var): - pkg.current_version = versions[var] - -# Fills in the .has_hash field of all Package objects -def add_hash_info(packages): - for name, pkg in packages.iteritems(): - hashpath = pkg.path.replace(".mk", ".hash") - pkg.has_hash = os.path.exists(hashpath) - -# Fills in the .patch_count field of all Package objects -def add_patch_count(packages): - for name, pkg in packages.iteritems(): - pkgdir = os.path.dirname(pkg.path) - pkg.patch_count = len(fnmatch.filter(os.listdir(pkgdir), '*.patch')) - -def get_check_package_warnings(pkgdir): - cmd = ["./utils/check-package"] - for root, dirs, files in os.walk(pkgdir): - for f in files: - if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": - cmd.append(f) - o = subprocess.check_output(cmd, stderr=subprocess.STDOUT) - lines = o.splitlines() - for line in lines: - m = re.match("^([0-9]*) warnings generated", line) - if m: - return int(m.group(1)) - return None - -# Fills in the .warnings field of all Package objects -def add_check_package_warnings(packages): - for name, pkg in packages.iteritems(): - pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path)) - -RELEASE_MONITORING_API = "http://release-monitoring.org/api" - -def get_latest_version_by_distro(package): - req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package)) - f = urllib2.urlopen(req) - data = json.loads(f.read()) - if len(data['versions']) > 0: - return (True, data['versions'][0], data['id']) - else: - return (True, None, data['id']) - -def get_latest_version_by_guess(package): - req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package)) - f = urllib2.urlopen(req) - data = json.loads(f.read()) - for p in data['projects']: - if p['name'] == package and len(p['versions']) > 0: - return (False, p['versions'][0], p['id']) - return (False, None, None) - -def get_latest_version(package): - try: - # We first try by using the "Buildroot" distribution on - # release-monitoring.org, if it has a mapping for the current - # package name. - return get_latest_version_by_distro(package) - except urllib2.HTTPError, e: - # If that fails because there is no mapping, we try to search - # in all packages for a package of this name. - if e.code == 404: - return get_latest_version_by_guess(package) - else: - return (False, None, None) - -def get_version_worker(q): - while True: - name, pkg = q.get() - pkg.latest_version = get_latest_version(name) - print " [%04d] %s => %s" % (q.qsize(), name, str(pkg.latest_version)) - q.task_done() - -# Fills in the .latest_version field of all Package objects -# -# This field has a special format: -# (mapping, version, id) -# with: -# - mapping: boolean that indicates whether release-monitoring.org -# has a mapping for this package name in the Buildroot distribution -# or not -# - version: string containing the latest version known by -# release-monitoring.org for this package -# - id: string containing the id of the project corresponding to this -# package, as known by release-monitoring.org -def add_latest_version_info(packages): - q = Queue() - for name, pkg in packages.iteritems(): - q.put((name, pkg)) - # Since release-monitoring.org is rather slow, we create 8 threads - # that do HTTP requests to the site. - for i in range(8): - t = Thread(target=get_version_worker, args=[q]) - t.daemon = True - t.start() - q.join() - -def calculate_stats(packages): - stats = defaultdict(int) - for name, pkg in packages.iteritems(): - # If packages have multiple infra, take the first one. For the - # vast majority of packages, the target and host infra are the - # same. There are very few packages that use a different infra - # for the host and target variants. - if len(pkg.infras) > 0: - infra = pkg.infras[0][1] - stats["infra-%s" % infra] += 1 - else: - stats["infra-unknown"] += 1 - if pkg.has_license: - stats["license"] += 1 - else: - stats["no-license"] += 1 - if pkg.has_license_files: - stats["license-files"] += 1 - else: - stats["no-license-files"] += 1 - if pkg.has_hash: - stats["hash"] += 1 - else: - stats["no-hash"] += 1 - if pkg.latest_version[0]: - stats["rmo-mapping"] += 1 - else: - stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: - stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: - stats["version-uptodate"] += 1 - else: - stats["version-not-uptodate"] += 1 - stats["patches"] += pkg.patch_count - return stats - -html_header = """ - - - -Statistics of Buildroot packages - - -Results
- -

-""" - -html_footer = """ - - - - -""" - -def infra_str(infra_list): - if not infra_list: - return "Unknown" - elif len(infra_list) == 1: - return "%s
%s" % (infra_list[0][1], infra_list[0][0]) - elif infra_list[0][1] == infra_list[1][1]: - return "%s
%s + %s" % \ - (infra_list[0][1], infra_list[0][0], infra_list[1][0]) - else: - return "%s (%s)
%s (%s)" % \ - (infra_list[0][1], infra_list[0][0], - infra_list[1][1], infra_list[1][0]) - -def boolean_str(b): - if b: - return "Yes" - else: - return "No" - -def dump_html_pkg(f, pkg): - f.write(" ") - f.write(" %s" % pkg.path[2:]) - - # Patch count - td_class = ["centered"] - if pkg.patch_count == 0: - td_class.append("nopatches") - elif pkg.patch_count < 5: - td_class.append("somepatches") - else: - td_class.append("lotsofpatches") - f.write(" %s" % - (" ".join(td_class), str(pkg.patch_count))) - - # Infrastructure - infra = infra_str(pkg.infras) - td_class = ["centered"] - if infra == "Unknown": - td_class.append("wrong") - else: - td_class.append("correct") - f.write(" %s" % \ - (" ".join(td_class), infra_str(pkg.infras))) - - # License - td_class = ["centered"] - if pkg.has_license: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s" % \ - (" ".join(td_class), boolean_str(pkg.has_license))) - - # License files - td_class = ["centered"] - if pkg.has_license_files: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s" % \ - (" ".join(td_class), boolean_str(pkg.has_license_files))) - - # Hash - td_class = ["centered"] - if pkg.has_hash: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s" % \ - (" ".join(td_class), boolean_str(pkg.has_hash))) - - # Current version - f.write(" %s" % pkg.current_version) - - # Latest version - if pkg.latest_version[1] == None: - td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: - td_class.append("version-needs-update") - else: - td_class.append("version-good") - - if pkg.latest_version[1] is None: - latest_version_text = "Unknown" - else: - latest_version_text = "%s" % str(pkg.latest_version[1]) - - latest_version_text += "
" - - if pkg.latest_version[2]: - latest_version_text += "link, " % pkg.latest_version[2] - else: - latest_version_text += "no link, " - - if pkg.latest_version[0]: - latest_version_text += "has mapping" - else: - latest_version_text += "has no mapping" - - f.write(" %s" % \ - (" ".join(td_class), latest_version_text)) - - # Warnings - td_class = ["centered"] - if pkg.warnings == 0: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %d" % \ - (" ".join(td_class), pkg.warnings)) - - f.write(" ") - -def dump_html_all_pkgs(f, packages): - f.write(""" - - - - - - - - - - - -""") - for name, pkg in sorted(packages.iteritems()): - dump_html_pkg(f, pkg) - f.write("
PackagePatch countInfrastructureLicenseLicense filesHash fileCurrent versionLatest versionWarnings
") - -def dump_html_stats(f, stats): - f.write("") - infras = [ infra[6:] for infra in stats.keys() if infra.startswith("infra-") ] - for infra in infras: - f.write("" % \ - (infra, stats["infra-%s" % infra])) - f.write("" % - stats["license"]) - f.write("" % - stats["no-license"]) - f.write("" % - stats["license-files"]) - f.write("" % - stats["no-license-files"]) - f.write("" % - stats["hash"]) - f.write("" % - stats["no-hash"]) - f.write("" % - stats["patches"]) - f.write("" % - stats["rmo-mapping"]) - f.write("" % - stats["rmo-no-mapping"]) - f.write("" % - stats["version-uptodate"]) - f.write("" % - stats["version-not-uptodate"]) - f.write("" % - stats["version-unknown"]) - f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
Packages having a mapping on release-monitoring.org%s
Packages lacking a mapping on release-monitoring.org%s
Packages that are up-to-date%s
Packages that are not up-to-date%s
Packages with no known upstream version%s
") - -def dump_html(packages, stats, output): - with open(output, 'w') as f: - f.write(html_header) - dump_html_all_pkgs(f, packages) - dump_html_stats(f, stats) - f.write(html_footer) - -def parse_args(): - parser = argparse.ArgumentParser() - parser.add_argument('-o', dest='output', action='store', required=True, - help='HTML output file') - parser.add_argument('-n', dest='npackages', type=int, action='store', - help='Number of packages') - parser.add_argument('-p', dest='packages', action='store', - help='List of packages') - return parser.parse_args() - -def __main__(): - args = parse_args() - if args.npackages and args.packages: - print "ERROR: -n and -p are mutually exclusive" - sys.exit(1) - if args.packages: - package_list = args.packages.split(",") - else: - package_list = None - print "Build package list ..." - packages = get_pkglist(args.npackages, package_list) - print "Get package infra ..." - add_infra_info(packages) - print "Get make info ..." - add_pkg_make_info(packages) - print "Get hash info ..." - add_hash_info(packages) - print "Get patch count ..." - add_patch_count(packages) - print "Get package warnings ..." - add_check_package_warnings(packages) - print "Get latest version ..." - add_latest_version_info(packages) - print "Calculate stats" - stats = calculate_stats(packages) - print "Write HTML" - dump_html(packages, stats, args.output) - -__main__()