From patchwork Fri Mar 23 20:54:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 890238 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 407G5M1Hwnz9s0p for ; Sat, 24 Mar 2018 07:55:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 9165789A68; Fri, 23 Mar 2018 20:55:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id phHuDAmazTwQ; Fri, 23 Mar 2018 20:55:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id D46F089A78; Fri, 23 Mar 2018 20:55:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 8D8F61C01DA for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8A96E23616 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 44CKc-C-74F7 for ; Fri, 23 Mar 2018 20:55:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by silver.osuosl.org (Postfix) with ESMTP id 41D71228CF for ; Fri, 23 Mar 2018 20:55:00 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id 394DB20858; Fri, 23 Mar 2018 21:54:58 +0100 (CET) Received: from localhost (LFbn-TOU-1-408-85.w86-206.abo.wanadoo.fr [86.206.234.85]) by mail.bootlin.com (Postfix) with ESMTPSA id 7DFDD206A0; Fri, 23 Mar 2018 21:54:57 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Fri, 23 Mar 2018 21:54:51 +0100 Message-Id: <20180323205455.24789-2-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> References: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH v3 1/5] support/scripts/pkg-stats-new: rewrite in Python X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds a new version of the pkg-stats script, rewritten in Python. It is for now implemented in a separate file called, pkg-stats-new, in order to make the diff easily readable. A future commit will rename it to pkg-stats. Compared to the existing shell-based pkg-stats script, the functionality and output is basically the same. The main difference is that the output no longer goes to stdout, but to the file passed as argument using the -o option. This allows stdout to be used for more debugging related information. The way the script works is that a first function get_pkglist() creates a dict associating package names with an instance of a Package() object, containing basic information about the package. Then a number of other functions (add_infra_info, add_pkg_make_info, add_hash_info, add_patch_count, add_check_package_warnings) will calculate additional information about packages, and fill in fields in the Package objects. calculate_stats() then calculates global statistics (how packages have license information, how packages have a hash file, etc.). Finally, dump_html() produces the HTML output, using a number of sub-functions. One improvement over the shell-based version is that we can use regexps to exclude some .mk files. Thanks to this, we can exclude all linux-ext-*.mk files, avoiding incorrect matches. Signed-off-by: Thomas Petazzoni Reviewed-by: Ricardo Martincoski --- Changes since v2: - Take into account patches in sub-directories in add_patch_count(). Comment from Ricardo. - Fix get_check_package_warnings() which was not appending the full file path to be checked. Comment from Ricardo. - Use subprocess.Popen() instead of subprocess.check_output() in get_check_package_warnings(). Comment from Ricardo. - Move a lot of the logic as methods of the Package() class. Changes since v1: take into account Ricardo's comments: - Added copyright notice - Fix flake8 warnings - Pass package path to Package() constructor - Remove CSS properties not needed in this commit - Fix broken #results link - Remove double - Add newlines in the generated HTML - Fix the sorting of the table by defining __eq__ and __lt__ in Package(), and making 'packages' a list rather than a dict. - Add the build date and git commit in the HTML page footer - Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake having a .config. This allows "printvars" to dump all variables even without a .config. --- support/scripts/pkg-stats-new | 459 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 459 insertions(+) create mode 100755 support/scripts/pkg-stats-new diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new new file mode 100755 index 0000000000..955d3ce990 --- /dev/null +++ b/support/scripts/pkg-stats-new @@ -0,0 +1,459 @@ +#!/usr/bin/env python + +# Copyright (C) 2009 by Thomas Petazzoni +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import argparse +import datetime +import fnmatch +import os +from collections import defaultdict +import re +import subprocess + +INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") + + +class Package: + all_licenses = list() + all_license_files = list() + + def __init__(self, name, path): + self.name = name + self.path = path + self.infras = None + self.has_license = False + self.has_license_files = False + self.has_hash = False + self.patch_count = 0 + self.warnings = 0 + + def pkgvar(self): + return self.name.upper().replace("-", "_") + + def set_infra(self): + """ + Fills in the .infras field + """ + self.infras = list() + with open(self.path, 'r') as f: + lines = f.readlines() + for l in lines: + match = INFRA_RE.match(l) + if not match: + continue + infra = match.group(1) + if infra.startswith("host-"): + self.infras.append(("host", infra[5:])) + else: + self.infras.append(("target", infra)) + + def set_license(self): + """ + Fills in the .has_license and .has_license_files fields + """ + var = self.pkgvar() + if var in self.all_licenses: + self.has_license = True + if var in self.all_license_files: + self.has_license_files = True + + def set_hash_info(self): + """ + Fills in the .has_hash field + """ + hashpath = self.path.replace(".mk", ".hash") + self.has_hash = os.path.exists(hashpath) + + def set_patch_count(self): + """ + Fills in the .patch_count field + """ + self.patch_count = 0 + pkgdir = os.path.dirname(self.path) + for subdir, _, _ in os.walk(pkgdir): + self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + + def set_check_package_warnings(self): + """ + Fills in the .warnings field + """ + cmd = ["./utils/check-package"] + pkgdir = os.path.dirname(self.path) + for root, dirs, files in os.walk(pkgdir): + for f in files: + if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": + cmd.append(os.path.join(root, f)) + o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1] + lines = o.splitlines() + for line in lines: + m = re.match("^([0-9]*) warnings generated", line) + if m: + self.warnings = int(m.group(1)) + return + + def __eq__(self, other): + return self.path == other.path + + def __lt__(self, other): + return self.path < other.path + + def __str__(self): + return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ + (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + + +def get_pkglist(): + """ + Builds the list of Buildroot packages, returning a list of Package + objects. Only the .name and .path fields of the Package object are + initialized. + """ + WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] + WALK_EXCLUDES = ["boot/common.mk", + "linux/linux-ext-.*.mk", + "package/freescale-imx/freescale-imx.mk", + "package/gcc/gcc.mk", + "package/gstreamer/gstreamer.mk", + "package/gstreamer1/gstreamer1.mk", + "package/gtk2-themes/gtk2-themes.mk", + "package/matchbox/matchbox.mk", + "package/opengl/opengl.mk", + "package/qt5/qt5.mk", + "package/x11r7/x11r7.mk", + "package/doc-asciidoc.mk", + "package/pkg-.*.mk", + "package/nvidia-tegra23/nvidia-tegra23.mk", + "toolchain/toolchain-external/pkg-toolchain-external.mk", + "toolchain/toolchain-external/toolchain-external.mk", + "toolchain/toolchain.mk", + "toolchain/helpers.mk", + "toolchain/toolchain-wrapper.mk"] + packages = list() + for root, dirs, files in os.walk("."): + rootdir = root.split("/") + if len(rootdir) < 2: + continue + if rootdir[1] not in WALK_USEFUL_SUBDIRS: + continue + for f in files: + if not f.endswith(".mk"): + continue + # Strip ending ".mk" + pkgname = f[:-3] + pkgpath = os.path.join(root, f) + skip = False + for exclude in WALK_EXCLUDES: + # pkgpath[2:] strips the initial './' + if re.match(exclude, pkgpath[2:]): + skip = True + continue + if skip: + continue + p = Package(pkgname, pkgpath) + packages.append(p) + return packages + + +def package_init_make_info(): + # Licenses + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_LICENSE"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + # Strip _LICENSE + pkgvar = pkgvar[:-8] + + # If value is "unknown", no license details available + if value == "unknown": + continue + Package.all_licenses.append(pkgvar) + + # License files + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_LICENSE_FILES"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): + continue + + # Strip _LICENSE_FILES + pkgvar = pkgvar[:-14] + + Package.all_license_files.append(pkgvar) + + +def calculate_stats(packages): + stats = defaultdict(int) + for pkg in packages: + # If packages have multiple infra, take the first one. For the + # vast majority of packages, the target and host infra are the + # same. There are very few packages that use a different infra + # for the host and target variants. + if len(pkg.infras) > 0: + infra = pkg.infras[0][1] + stats["infra-%s" % infra] += 1 + else: + stats["infra-unknown"] += 1 + if pkg.has_license: + stats["license"] += 1 + else: + stats["no-license"] += 1 + if pkg.has_license_files: + stats["license-files"] += 1 + else: + stats["no-license-files"] += 1 + if pkg.has_hash: + stats["hash"] += 1 + else: + stats["no-hash"] += 1 + stats["patches"] += pkg.patch_count + return stats + + +html_header = """ + + + +Statistics of Buildroot packages + + +Results
+ +

+""" + + +html_footer = """ + + + +""" + + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + + +def dump_html_pkg(f, pkg): + f.write(" \n") + f.write(" %s\n" % pkg.path[2:]) + + # Patch count + td_class = ["centered"] + if pkg.patch_count == 0: + td_class.append("nopatches") + elif pkg.patch_count < 5: + td_class.append("somepatches") + else: + td_class.append("lotsofpatches") + f.write(" %s\n" % + (" ".join(td_class), str(pkg.patch_count))) + + # Infrastructure + infra = infra_str(pkg.infras) + td_class = ["centered"] + if infra == "Unknown": + td_class.append("wrong") + else: + td_class.append("correct") + f.write(" %s\n" % + (" ".join(td_class), infra_str(pkg.infras))) + + # License + td_class = ["centered"] + if pkg.has_license: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_license))) + + # License files + td_class = ["centered"] + if pkg.has_license_files: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_license_files))) + + # Hash + td_class = ["centered"] + if pkg.has_hash: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_hash))) + + # Warnings + td_class = ["centered"] + if pkg.warnings == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %d\n" % + (" ".join(td_class), pkg.warnings)) + + f.write(" \n") + + +def dump_html_all_pkgs(f, packages): + f.write(""" + + + + + + + + + + +""") + for pkg in sorted(packages): + dump_html_pkg(f, pkg) + f.write("
PackagePatch countInfrastructureLicenseLicense filesHash fileWarnings
") + + +def dump_html_stats(f, stats): + f.write("\n") + f.write("\n") + infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")] + for infra in infras: + f.write(" \n" % + (infra, stats["infra-%s" % infra])) + f.write(" \n" % + stats["license"]) + f.write(" \n" % + stats["no-license"]) + f.write(" \n" % + stats["license-files"]) + f.write(" \n" % + stats["no-license-files"]) + f.write(" \n" % + stats["hash"]) + f.write(" \n" % + stats["no-hash"]) + f.write(" \n" % + stats["patches"]) + f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
\n") + + +def dump_gen_info(f): + # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032 + o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"]) + git_commit = o.splitlines()[0] + f.write("

Updated on %s, git commit %s

\n" % + (str(datetime.datetime.utcnow()), git_commit)) + + +def dump_html(packages, stats, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_stats(f, stats) + dump_gen_info(f) + f.write(html_footer) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-o', dest='output', action='store', required=True, + help='HTML output file') + return parser.parse_args() + + +def __main__(): + args = parse_args() + print "Build package list ..." + packages = get_pkglist() + print "Getting package make info ..." + package_init_make_info() + print "Getting package details ..." + for pkg in packages: + pkg.set_infra() + pkg.set_license() + pkg.set_hash_info() + pkg.set_patch_count() + pkg.set_check_package_warnings() + print "Calculate stats" + stats = calculate_stats(packages) + print "Write HTML" + dump_html(packages, stats, args.output) + + +__main__() From patchwork Fri Mar 23 20:54:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 890235 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 407G5B2rGBz9s0p for ; Sat, 24 Mar 2018 07:55:10 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 724DC8A604; Fri, 23 Mar 2018 20:55:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MM4zfjwDrle2; Fri, 23 Mar 2018 20:55:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 8A0238A23E; Fri, 23 Mar 2018 20:55:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 987CF1C06F2 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 91C0B8895F for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5k6Ri1GBD2cP for ; Fri, 23 Mar 2018 20:55:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by fraxinus.osuosl.org (Postfix) with ESMTP id E226D88942 for ; Fri, 23 Mar 2018 20:55:00 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id 3D0052084D; Fri, 23 Mar 2018 21:54:58 +0100 (CET) Received: from localhost (LFbn-TOU-1-408-85.w86-206.abo.wanadoo.fr [86.206.234.85]) by mail.bootlin.com (Postfix) with ESMTPSA id CEA832046F; Fri, 23 Mar 2018 21:54:57 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Fri, 23 Mar 2018 21:54:52 +0100 Message-Id: <20180323205455.24789-3-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> References: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH v3 2/5] support/scripts/pkg-stats-new: add -n and -p options X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds the following options to the pkg-stats-new script: -n, to specify a number of packages to parse instead of all packages -p, to specify a list of packages (comma-separated) to parse instead of all packages These options are basically only useful when debugging/developing this script, but they are very useful, because the script is rather slow to run completely with all 2000+ packages, especially once upstream versions will be fetched from release-monitoring.org. Signed-off-by: Thomas Petazzoni Reviewed-by: Ricardo Martincoski --- Changes since v2: - Indicate in help text that the list of packages is comma-separated. Comment from Ricardo. - Fix typo in commit log. Comment from Ricardo. - Added Reviewed-by from Ricardo. Changes since v1: - Fix flake8 warnings --- support/scripts/pkg-stats-new | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index 955d3ce990..5dc70f1671 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -23,6 +23,7 @@ import os from collections import defaultdict import re import subprocess +import sys INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") @@ -116,11 +117,14 @@ class Package: (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) -def get_pkglist(): +def get_pkglist(npackages, package_list): """ Builds the list of Buildroot packages, returning a list of Package objects. Only the .name and .path fields of the Package object are initialized. + + npackages: limit to N packages + package_list: limit to those packages in this list """ WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] WALK_EXCLUDES = ["boot/common.mk", @@ -143,6 +147,7 @@ def get_pkglist(): "toolchain/helpers.mk", "toolchain/toolchain-wrapper.mk"] packages = list() + count = 0 for root, dirs, files in os.walk("."): rootdir = root.split("/") if len(rootdir) < 2: @@ -154,6 +159,8 @@ def get_pkglist(): continue # Strip ending ".mk" pkgname = f[:-3] + if package_list and pkgname not in package_list: + continue pkgpath = os.path.join(root, f) skip = False for exclude in WALK_EXCLUDES: @@ -165,6 +172,9 @@ def get_pkglist(): continue p = Package(pkgname, pkgpath) packages.append(p) + count += 1 + if npackages and count == npackages: + return packages return packages @@ -434,13 +444,24 @@ def parse_args(): parser = argparse.ArgumentParser() parser.add_argument('-o', dest='output', action='store', required=True, help='HTML output file') + parser.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + parser.add_argument('-p', dest='packages', action='store', + help='List of packages (comma separated)') return parser.parse_args() def __main__(): args = parse_args() + if args.npackages and args.packages: + print "ERROR: -n and -p are mutually exclusive" + sys.exit(1) + if args.packages: + package_list = args.packages.split(",") + else: + package_list = None print "Build package list ..." - packages = get_pkglist() + packages = get_pkglist(args.npackages, package_list) print "Getting package make info ..." package_init_make_info() print "Getting package details ..." From patchwork Fri Mar 23 20:54:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 890237 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 407G5G45Tdz9s0p for ; Sat, 24 Mar 2018 07:55:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 40E1289A57; Fri, 23 Mar 2018 20:55:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xzqIV8bDw0Ts; Fri, 23 Mar 2018 20:55:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 22AFD89A64; Fri, 23 Mar 2018 20:55:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 578C71C01DA for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 553C48A281 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HEvrAbDTexW8 for ; Fri, 23 Mar 2018 20:55:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by hemlock.osuosl.org (Postfix) with ESMTP id AB89281B82 for ; Fri, 23 Mar 2018 20:55:00 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id 80F0120877; Fri, 23 Mar 2018 21:54:58 +0100 (CET) Received: from localhost (LFbn-TOU-1-408-85.w86-206.abo.wanadoo.fr [86.206.234.85]) by mail.bootlin.com (Postfix) with ESMTPSA id 21A332082C; Fri, 23 Mar 2018 21:54:58 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Fri, 23 Mar 2018 21:54:53 +0100 Message-Id: <20180323205455.24789-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> References: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH v3 3/5] support/scripts/pkg-stats-new: add current version information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds a new column in the HTML output containing the current version of a package in Buildroot. As such, it isn't terribly useful, but combined with the latest upstream version added in a follow-up commit, it will become very useful. Signed-off-by: Thomas Petazzoni Reviewed-by: Ricardo Martincoski --- Changes since v2: - Address Ricardo's concern about printvars variable sorting affecting the version reported: we now parse all HOST_*_VERSION variables first, and then all *_VERSION variables, ensuring that the target version of a package wins. - Limit length of version string to 20 characters, as suggested by Ricardo. - Move a lot of the logic as methods of the Package() class. Changes since v1: - Fix flake8 warnings - Pass BR2_HAVE_DOT_CONFIG=y when calling make, in order to fake having a .config. This allows "printvars" to dump all variables even without a .config. - Add missing newline in HTML code --- support/scripts/pkg-stats-new | 46 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index 5dc70f1671..43f7e8d543 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -31,6 +31,7 @@ INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") class Package: all_licenses = list() all_license_files = list() + all_versions = dict() def __init__(self, name, path): self.name = name @@ -41,6 +42,7 @@ class Package: self.has_hash = False self.patch_count = 0 self.warnings = 0 + self.current_version = None def pkgvar(self): return self.name.upper().replace("-", "_") @@ -88,6 +90,14 @@ class Package: for subdir, _, _ in os.walk(pkgdir): self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + def set_current_version(self): + """ + Fills in the .current_version field + """ + var = self.pkgvar() + if var in self.all_versions: + self.current_version = self.all_versions[var] + def set_check_package_warnings(self): """ Fills in the .warnings field @@ -217,6 +227,33 @@ def package_init_make_info(): Package.all_license_files.append(pkgvar) + # Version + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_VERSION"]) + + # We process first the host package VERSION, and then the target + # package VERSION. This means that if a package exists in both + # target and host variants, with different version numbers + # (unlikely), we'll report the target version number. + version_list = o.splitlines() + version_list = [x for x in version_list if x.startswith("HOST_")] + \ + [x for x in version_list if not x.startswith("HOST_")] + for l in version_list: + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_DL_VERSION"): + continue + + # Strip _VERSION + pkgvar = pkgvar[:-8] + + Package.all_versions[pkgvar] = value + def calculate_stats(packages): stats = defaultdict(int) @@ -369,6 +406,13 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % (" ".join(td_class), boolean_str(pkg.has_hash))) + # Current version + if len(pkg.current_version) > 20: + current_version = pkg.current_version[:20] + "..." + else: + current_version = pkg.current_version + f.write(" %s\n" % current_version) + # Warnings td_class = ["centered"] if pkg.warnings == 0: @@ -391,6 +435,7 @@ def dump_html_all_pkgs(f, packages): License License files Hash file +Current version Warnings """) @@ -471,6 +516,7 @@ def __main__(): pkg.set_hash_info() pkg.set_patch_count() pkg.set_check_package_warnings() + pkg.set_current_version() print "Calculate stats" stats = calculate_stats(packages) print "Write HTML" From patchwork Fri Mar 23 20:54:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 890236 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 407G5F23F0z9s0p for ; Sat, 24 Mar 2018 07:55:13 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9F812304A7; Fri, 23 Mar 2018 20:55:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zlEjxm8iEqdZ; Fri, 23 Mar 2018 20:55:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 3E7DA23616; Fri, 23 Mar 2018 20:55:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 9EFAD1CF0A3 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 9CC2181B82 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y+vvEKBPl-zY for ; Fri, 23 Mar 2018 20:55:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by hemlock.osuosl.org (Postfix) with ESMTP id B40958A23E for ; Fri, 23 Mar 2018 20:55:00 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id DF97020879; Fri, 23 Mar 2018 21:54:58 +0100 (CET) Received: from localhost (LFbn-TOU-1-408-85.w86-206.abo.wanadoo.fr [86.206.234.85]) by mail.bootlin.com (Postfix) with ESMTPSA id 741B82046F; Fri, 23 Mar 2018 21:54:58 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Fri, 23 Mar 2018 21:54:54 +0100 Message-Id: <20180323205455.24789-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> References: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH v3 4/5] support/scripts/pkg-stats-new: add latest upstream version information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. If there is no result, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Since release-monitoring.org is a bit slow, we have 8 threads that fetch information in parallel. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni --- Changes since v2: - Use the "timeout" argument of urllib2.urlopen() in order to make sure that the requests terminate at some point, even if release-monitoring.org is stuck. - Move a lot of the logic as methods of the Package() class. Changes since v1: - Fix flake8 warnings - Add missing newline in HTML --- support/scripts/pkg-stats-new | 138 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index 43f7e8d543..830040a485 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -24,8 +24,13 @@ from collections import defaultdict import re import subprocess import sys +import json +import urllib2 +from Queue import Queue +from threading import Thread INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") +RELEASE_MONITORING_API = "http://release-monitoring.org/api" class Package: @@ -43,6 +48,7 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.latest_version = None def pkgvar(self): return self.name.upper().replace("-", "_") @@ -116,6 +122,43 @@ class Package: self.warnings = int(m.group(1)) return + def get_latest_version_by_distro(self): + try: + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", self.name)) + f = urllib2.urlopen(req, timeout=15) + except: + # Exceptions can typically be a timeout, or a 404 error if not project + return (False, None, None) + data = json.loads(f.read()) + if len(data['versions']) > 0: + return (True, data['versions'][0], data['id']) + else: + return (True, None, data['id']) + + def get_latest_version_by_guess(self): + try: + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % self.name)) + f = urllib2.urlopen(req, timeout=15) + except: + # Exceptions can typically be a timeout, or a 404 error if not project + return (False, None, None) + data = json.loads(f.read()) + for p in data['projects']: + if p['name'] == self.name and len(p['versions']) > 0: + return (False, p['versions'][0], p['id']) + return (False, None, None) + + def set_latest_version(self): + # We first try by using the "Buildroot" distribution on + # release-monitoring.org, if it has a mapping for the current + # package name. + self.latest_version = self.get_latest_version_by_distro() + if self.latest_version == (False, None, None): + # If that fails because there is no mapping or because we had a + # request timeout, we try to search in all packages for a package + # of this name. + self.latest_version = self.get_latest_version_by_guess() + def __eq__(self, other): return self.path == other.path @@ -255,6 +298,41 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value +def set_version_worker(q): + while True: + pkg = q.get() + pkg.set_latest_version() + print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version)) + q.task_done() + + +def add_latest_version_info(packages): + """ + Fills in the .latest_version field of all Package objects + + This field has a special format: + (mapping, version, id) + with: + - mapping: boolean that indicates whether release-monitoring.org + has a mapping for this package name in the Buildroot distribution + or not + - version: string containing the latest version known by + release-monitoring.org for this package + - id: string containing the id of the project corresponding to this + package, as known by release-monitoring.org + """ + q = Queue() + for pkg in packages: + q.put(pkg) + # Since release-monitoring.org is rather slow, we create 8 threads + # that do HTTP requests to the site. + for i in range(8): + t = Thread(target=set_version_worker, args=[q]) + t.daemon = True + t.start() + q.join() + + def calculate_stats(packages): stats = defaultdict(int) for pkg in packages: @@ -279,6 +357,16 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.latest_version[0]: + stats["rmo-mapping"] += 1 + else: + stats["rmo-no-mapping"] += 1 + if not pkg.latest_version[1]: + stats["version-unknown"] += 1 + elif pkg.latest_version[1] == pkg.current_version: + stats["version-uptodate"] += 1 + else: + stats["version-not-uptodate"] += 1 stats["patches"] += pkg.patch_count return stats @@ -311,6 +399,15 @@ td.somepatches { td.lotsofpatches { background: #ff9a69; } +td.version-good { + background: #d2ffc4; +} +td.version-needs-update { + background: #ff9a69; +} +td.version-unknown { + background: #ffd870; +} Statistics of Buildroot packages @@ -413,6 +510,34 @@ def dump_html_pkg(f, pkg): current_version = pkg.current_version f.write(" %s\n" % current_version) + # Latest version + if pkg.latest_version[1] is None: + td_class.append("version-unknown") + elif pkg.latest_version[1] != pkg.current_version: + td_class.append("version-needs-update") + else: + td_class.append("version-good") + + if pkg.latest_version[1] is None: + latest_version_text = "Unknown" + else: + latest_version_text = "%s" % str(pkg.latest_version[1]) + + latest_version_text += "
" + + if pkg.latest_version[2]: + latest_version_text += "link, " % pkg.latest_version[2] + else: + latest_version_text += "no link, " + + if pkg.latest_version[0]: + latest_version_text += "has mapping" + else: + latest_version_text += "has no mapping" + + f.write(" %s\n" % + (" ".join(td_class), latest_version_text)) + # Warnings td_class = ["centered"] if pkg.warnings == 0: @@ -436,6 +561,7 @@ def dump_html_all_pkgs(f, packages): License files Hash file Current version +Latest version Warnings """) @@ -465,6 +591,16 @@ def dump_html_stats(f, stats): stats["no-hash"]) f.write(" Total number of patches%s\n" % stats["patches"]) + f.write("Packages having a mapping on release-monitoring.org%s\n" % + stats["rmo-mapping"]) + f.write("Packages lacking a mapping on release-monitoring.org%s\n" % + stats["rmo-no-mapping"]) + f.write("Packages that are up-to-date%s\n" % + stats["version-uptodate"]) + f.write("Packages that are not up-to-date%s\n" % + stats["version-not-uptodate"]) + f.write("Packages with no known upstream version%s\n" % + stats["version-unknown"]) f.write("\n") @@ -517,6 +653,8 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() + print "Getting latest versions ..." + add_latest_version_info(packages) print "Calculate stats" stats = calculate_stats(packages) print "Write HTML" From patchwork Fri Mar 23 20:54:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 890239 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 407G5f2DM0z9s0p for ; Sat, 24 Mar 2018 07:55:34 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6D56189A65; Fri, 23 Mar 2018 20:55:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AvvaCkVLzdZI; Fri, 23 Mar 2018 20:55:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id EFC6F89A80; Fri, 23 Mar 2018 20:55:12 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id E8FD61C01DA for ; Fri, 23 Mar 2018 20:55:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DC1C988959 for ; Fri, 23 Mar 2018 20:55:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HoAiLzp1PeH2 for ; Fri, 23 Mar 2018 20:55:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by fraxinus.osuosl.org (Postfix) with ESMTP id 717E988944 for ; Fri, 23 Mar 2018 20:55:01 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id 9DB822082C; Fri, 23 Mar 2018 21:54:59 +0100 (CET) Received: from localhost (LFbn-TOU-1-408-85.w86-206.abo.wanadoo.fr [86.206.234.85]) by mail.bootlin.com (Postfix) with ESMTPSA id C13C5206A0; Fri, 23 Mar 2018 21:54:58 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Fri, 23 Mar 2018 21:54:55 +0100 Message-Id: <20180323205455.24789-6-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> References: <20180323205455.24789-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH v3 5/5] support/scripts/pkg-stats: replace with new Python version X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Rename pkg-stats-new to pkg-stats. Signed-off-by: Thomas Petazzoni --- support/scripts/pkg-stats | 997 +++++++++++++++++++++++++----------------- support/scripts/pkg-stats-new | 664 ---------------------------- 2 files changed, 603 insertions(+), 1058 deletions(-) delete mode 100755 support/scripts/pkg-stats-new diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 48a2cc29a1..830040a485 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/usr/bin/env python # Copyright (C) 2009 by Thomas Petazzoni # @@ -16,16 +16,363 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -# This script generates an HTML file that contains a report about all -# Buildroot packages, their usage of the different package -# infrastructure and possible cleanup actions -# -# Run the script from the Buildroot toplevel directory: -# -# ./support/scripts/pkg-stats > /tmp/pkg.html -# - -echo " +import argparse +import datetime +import fnmatch +import os +from collections import defaultdict +import re +import subprocess +import sys +import json +import urllib2 +from Queue import Queue +from threading import Thread + +INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") +RELEASE_MONITORING_API = "http://release-monitoring.org/api" + + +class Package: + all_licenses = list() + all_license_files = list() + all_versions = dict() + + def __init__(self, name, path): + self.name = name + self.path = path + self.infras = None + self.has_license = False + self.has_license_files = False + self.has_hash = False + self.patch_count = 0 + self.warnings = 0 + self.current_version = None + self.latest_version = None + + def pkgvar(self): + return self.name.upper().replace("-", "_") + + def set_infra(self): + """ + Fills in the .infras field + """ + self.infras = list() + with open(self.path, 'r') as f: + lines = f.readlines() + for l in lines: + match = INFRA_RE.match(l) + if not match: + continue + infra = match.group(1) + if infra.startswith("host-"): + self.infras.append(("host", infra[5:])) + else: + self.infras.append(("target", infra)) + + def set_license(self): + """ + Fills in the .has_license and .has_license_files fields + """ + var = self.pkgvar() + if var in self.all_licenses: + self.has_license = True + if var in self.all_license_files: + self.has_license_files = True + + def set_hash_info(self): + """ + Fills in the .has_hash field + """ + hashpath = self.path.replace(".mk", ".hash") + self.has_hash = os.path.exists(hashpath) + + def set_patch_count(self): + """ + Fills in the .patch_count field + """ + self.patch_count = 0 + pkgdir = os.path.dirname(self.path) + for subdir, _, _ in os.walk(pkgdir): + self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + + def set_current_version(self): + """ + Fills in the .current_version field + """ + var = self.pkgvar() + if var in self.all_versions: + self.current_version = self.all_versions[var] + + def set_check_package_warnings(self): + """ + Fills in the .warnings field + """ + cmd = ["./utils/check-package"] + pkgdir = os.path.dirname(self.path) + for root, dirs, files in os.walk(pkgdir): + for f in files: + if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": + cmd.append(os.path.join(root, f)) + o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1] + lines = o.splitlines() + for line in lines: + m = re.match("^([0-9]*) warnings generated", line) + if m: + self.warnings = int(m.group(1)) + return + + def get_latest_version_by_distro(self): + try: + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", self.name)) + f = urllib2.urlopen(req, timeout=15) + except: + # Exceptions can typically be a timeout, or a 404 error if not project + return (False, None, None) + data = json.loads(f.read()) + if len(data['versions']) > 0: + return (True, data['versions'][0], data['id']) + else: + return (True, None, data['id']) + + def get_latest_version_by_guess(self): + try: + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % self.name)) + f = urllib2.urlopen(req, timeout=15) + except: + # Exceptions can typically be a timeout, or a 404 error if not project + return (False, None, None) + data = json.loads(f.read()) + for p in data['projects']: + if p['name'] == self.name and len(p['versions']) > 0: + return (False, p['versions'][0], p['id']) + return (False, None, None) + + def set_latest_version(self): + # We first try by using the "Buildroot" distribution on + # release-monitoring.org, if it has a mapping for the current + # package name. + self.latest_version = self.get_latest_version_by_distro() + if self.latest_version == (False, None, None): + # If that fails because there is no mapping or because we had a + # request timeout, we try to search in all packages for a package + # of this name. + self.latest_version = self.get_latest_version_by_guess() + + def __eq__(self, other): + return self.path == other.path + + def __lt__(self, other): + return self.path < other.path + + def __str__(self): + return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ + (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + + +def get_pkglist(npackages, package_list): + """ + Builds the list of Buildroot packages, returning a list of Package + objects. Only the .name and .path fields of the Package object are + initialized. + + npackages: limit to N packages + package_list: limit to those packages in this list + """ + WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] + WALK_EXCLUDES = ["boot/common.mk", + "linux/linux-ext-.*.mk", + "package/freescale-imx/freescale-imx.mk", + "package/gcc/gcc.mk", + "package/gstreamer/gstreamer.mk", + "package/gstreamer1/gstreamer1.mk", + "package/gtk2-themes/gtk2-themes.mk", + "package/matchbox/matchbox.mk", + "package/opengl/opengl.mk", + "package/qt5/qt5.mk", + "package/x11r7/x11r7.mk", + "package/doc-asciidoc.mk", + "package/pkg-.*.mk", + "package/nvidia-tegra23/nvidia-tegra23.mk", + "toolchain/toolchain-external/pkg-toolchain-external.mk", + "toolchain/toolchain-external/toolchain-external.mk", + "toolchain/toolchain.mk", + "toolchain/helpers.mk", + "toolchain/toolchain-wrapper.mk"] + packages = list() + count = 0 + for root, dirs, files in os.walk("."): + rootdir = root.split("/") + if len(rootdir) < 2: + continue + if rootdir[1] not in WALK_USEFUL_SUBDIRS: + continue + for f in files: + if not f.endswith(".mk"): + continue + # Strip ending ".mk" + pkgname = f[:-3] + if package_list and pkgname not in package_list: + continue + pkgpath = os.path.join(root, f) + skip = False + for exclude in WALK_EXCLUDES: + # pkgpath[2:] strips the initial './' + if re.match(exclude, pkgpath[2:]): + skip = True + continue + if skip: + continue + p = Package(pkgname, pkgpath) + packages.append(p) + count += 1 + if npackages and count == npackages: + return packages + return packages + + +def package_init_make_info(): + # Licenses + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_LICENSE"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + # Strip _LICENSE + pkgvar = pkgvar[:-8] + + # If value is "unknown", no license details available + if value == "unknown": + continue + Package.all_licenses.append(pkgvar) + + # License files + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_LICENSE_FILES"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): + continue + + # Strip _LICENSE_FILES + pkgvar = pkgvar[:-14] + + Package.all_license_files.append(pkgvar) + + # Version + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_VERSION"]) + + # We process first the host package VERSION, and then the target + # package VERSION. This means that if a package exists in both + # target and host variants, with different version numbers + # (unlikely), we'll report the target version number. + version_list = o.splitlines() + version_list = [x for x in version_list if x.startswith("HOST_")] + \ + [x for x in version_list if not x.startswith("HOST_")] + for l in version_list: + # Get variable name and value + pkgvar, value = l.split("=") + + # If present, strip HOST_ from variable name + if pkgvar.startswith("HOST_"): + pkgvar = pkgvar[5:] + + if pkgvar.endswith("_DL_VERSION"): + continue + + # Strip _VERSION + pkgvar = pkgvar[:-8] + + Package.all_versions[pkgvar] = value + + +def set_version_worker(q): + while True: + pkg = q.get() + pkg.set_latest_version() + print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version)) + q.task_done() + + +def add_latest_version_info(packages): + """ + Fills in the .latest_version field of all Package objects + + This field has a special format: + (mapping, version, id) + with: + - mapping: boolean that indicates whether release-monitoring.org + has a mapping for this package name in the Buildroot distribution + or not + - version: string containing the latest version known by + release-monitoring.org for this package + - id: string containing the id of the project corresponding to this + package, as known by release-monitoring.org + """ + q = Queue() + for pkg in packages: + q.put(pkg) + # Since release-monitoring.org is rather slow, we create 8 threads + # that do HTTP requests to the site. + for i in range(8): + t = Thread(target=set_version_worker, args=[q]) + t.daemon = True + t.start() + q.join() + + +def calculate_stats(packages): + stats = defaultdict(int) + for pkg in packages: + # If packages have multiple infra, take the first one. For the + # vast majority of packages, the target and host infra are the + # same. There are very few packages that use a different infra + # for the host and target variants. + if len(pkg.infras) > 0: + infra = pkg.infras[0][1] + stats["infra-%s" % infra] += 1 + else: + stats["infra-unknown"] += 1 + if pkg.has_license: + stats["license"] += 1 + else: + stats["no-license"] += 1 + if pkg.has_license_files: + stats["license-files"] += 1 + else: + stats["no-license-files"] += 1 + if pkg.has_hash: + stats["hash"] += 1 + else: + stats["no-hash"] += 1 + if pkg.latest_version[0]: + stats["rmo-mapping"] += 1 + else: + stats["rmo-no-mapping"] += 1 + if not pkg.latest_version[1]: + stats["version-unknown"] += 1 + elif pkg.latest_version[1] == pkg.current_version: + stats["version-uptodate"] += 1 + else: + stats["version-not-uptodate"] += 1 + stats["patches"] += pkg.patch_count + return stats + + +html_header = """ + Statistics of Buildroot packages @@ -61,395 +415,250 @@ td.lotsofpatches { Results

+""" + +html_footer = """ + + + +""" + + +def infra_str(infra_list): + if not infra_list: + return "Unknown" + elif len(infra_list) == 1: + return "%s
%s" % (infra_list[0][1], infra_list[0][0]) + elif infra_list[0][1] == infra_list[1][1]: + return "%s
%s + %s" % \ + (infra_list[0][1], infra_list[0][0], infra_list[1][0]) + else: + return "%s (%s)
%s (%s)" % \ + (infra_list[0][1], infra_list[0][0], + infra_list[1][1], infra_list[1][0]) + + +def boolean_str(b): + if b: + return "Yes" + else: + return "No" + + +def dump_html_pkg(f, pkg): + f.write(" \n") + f.write(" %s\n" % pkg.path[2:]) + + # Patch count + td_class = ["centered"] + if pkg.patch_count == 0: + td_class.append("nopatches") + elif pkg.patch_count < 5: + td_class.append("somepatches") + else: + td_class.append("lotsofpatches") + f.write(" %s\n" % + (" ".join(td_class), str(pkg.patch_count))) + + # Infrastructure + infra = infra_str(pkg.infras) + td_class = ["centered"] + if infra == "Unknown": + td_class.append("wrong") + else: + td_class.append("correct") + f.write(" %s\n" % + (" ".join(td_class), infra_str(pkg.infras))) + + # License + td_class = ["centered"] + if pkg.has_license: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_license))) + + # License files + td_class = ["centered"] + if pkg.has_license_files: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_license_files))) + + # Hash + td_class = ["centered"] + if pkg.has_hash: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %s\n" % + (" ".join(td_class), boolean_str(pkg.has_hash))) + + # Current version + if len(pkg.current_version) > 20: + current_version = pkg.current_version[:20] + "..." + else: + current_version = pkg.current_version + f.write(" %s\n" % current_version) + + # Latest version + if pkg.latest_version[1] is None: + td_class.append("version-unknown") + elif pkg.latest_version[1] != pkg.current_version: + td_class.append("version-needs-update") + else: + td_class.append("version-good") + + if pkg.latest_version[1] is None: + latest_version_text = "Unknown" + else: + latest_version_text = "%s" % str(pkg.latest_version[1]) + + latest_version_text += "
" + + if pkg.latest_version[2]: + latest_version_text += "link, " % pkg.latest_version[2] + else: + latest_version_text += "no link, " + + if pkg.latest_version[0]: + latest_version_text += "has mapping" + else: + latest_version_text += "has no mapping" + + f.write(" %s\n" % + (" ".join(td_class), latest_version_text)) + + # Warnings + td_class = ["centered"] + if pkg.warnings == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" %d\n" % + (" ".join(td_class), pkg.warnings)) + + f.write(" \n") + + +def dump_html_all_pkgs(f, packages): + f.write(""" - + + -" - -autotools_packages=0 -cmake_packages=0 -kconfig_packages=0 -luarocks_package=0 -perl_packages=0 -python_packages=0 -rebar_packages=0 -virtual_packages=0 -generic_packages=0 -waf_packages=0 -manual_packages=0 -packages_with_licence=0 -packages_without_licence=0 -packages_with_license_files=0 -packages_without_license_files=0 -packages_with_hash_file=0 -packages_without_hash_file=0 -total_patch_count=0 -cnt=0 - -for i in $(find boot/ linux/ package/ toolchain/ -name '*.mk' | sort) ; do - - if test \ - $i = "boot/common.mk" -o \ - $i = "linux/linux-ext-ev3dev-linux-drivers.mk" -o \ - $i = "linux/linux-ext-fbtft.mk" -o \ - $i = "linux/linux-ext-xenomai.mk" -o \ - $i = "linux/linux-ext-rtai.mk" -o \ - $i = "package/freescale-imx/freescale-imx.mk" -o \ - $i = "package/gcc/gcc.mk" -o \ - $i = "package/gstreamer/gstreamer.mk" -o \ - $i = "package/gstreamer1/gstreamer1.mk" -o \ - $i = "package/gtk2-themes/gtk2-themes.mk" -o \ - $i = "package/matchbox/matchbox.mk" -o \ - $i = "package/opengl/opengl.mk" -o \ - $i = "package/qt5/qt5.mk" -o \ - $i = "package/x11r7/x11r7.mk" -o \ - $i = "package/doc-asciidoc.mk" -o \ - $i = "package/pkg-autotools.mk" -o \ - $i = "package/pkg-cmake.mk" -o \ - $i = "package/pkg-kconfig.mk" -o \ - $i = "package/pkg-luarocks.mk" -o \ - $i = "package/pkg-perl.mk" -o \ - $i = "package/pkg-python.mk" -o \ - $i = "package/pkg-rebar.mk" -o \ - $i = "package/pkg-virtual.mk" -o \ - $i = "package/pkg-download.mk" -o \ - $i = "package/pkg-generic.mk" -o \ - $i = "package/pkg-waf.mk" -o \ - $i = "package/pkg-kernel-module.mk" -o \ - $i = "package/pkg-utils.mk" -o \ - $i = "package/nvidia-tegra23/nvidia-tegra23.mk" -o \ - $i = "toolchain/toolchain-external/pkg-toolchain-external.mk" -o \ - $i = "toolchain/toolchain-external/toolchain-external.mk" -o \ - $i = "toolchain/toolchain.mk" -o \ - $i = "toolchain/helpers.mk" -o \ - $i = "toolchain/toolchain-wrapper.mk" ; then - echo "skipping $i" 1>&2 - continue - fi - - cnt=$((cnt+1)) - - hashost=0 - hastarget=0 - infratype="" - - # Determine package infrastructure - if grep -E "\(host-autotools-package\)" $i > /dev/null ; then - infratype="autotools" - hashost=1 - fi - - if grep -E "\(autotools-package\)" $i > /dev/null ; then - infratype="autotools" - hastarget=1 - fi - - if grep -E "\(kconfig-package\)" $i > /dev/null ; then - infratype="kconfig" - hastarget=1 - fi - - if grep -E "\(host-luarocks-package\)" $i > /dev/null ; then - infratype="luarocks" - hashost=1 - fi - - if grep -E "\(luarocks-package\)" $i > /dev/null ; then - infratype="luarocks" - hastarget=1 - fi - - if grep -E "\(host-perl-package\)" $i > /dev/null ; then - infratype="perl" - hashost=1 - fi - - if grep -E "\(perl-package\)" $i > /dev/null ; then - infratype="perl" - hastarget=1 - fi - - if grep -E "\(host-python-package\)" $i > /dev/null ; then - infratype="python" - hashost=1 - fi - - if grep -E "\(python-package\)" $i > /dev/null ; then - infratype="python" - hastarget=1 - fi - - if grep -E "\(host-rebar-package\)" $i > /dev/null ; then - infratype="rebar" - hashost=1 - fi - - if grep -E "\(rebar-package\)" $i > /dev/null ; then - infratype="rebar" - hastarget=1 - fi - - if grep -E "\(host-virtual-package\)" $i > /dev/null ; then - infratype="virtual" - hashost=1 - fi - - if grep -E "\(virtual-package\)" $i > /dev/null ; then - infratype="virtual" - hastarget=1 - fi - - if grep -E "\(host-generic-package\)" $i > /dev/null ; then - infratype="generic" - hashost=1 - fi - - if grep -E "\(generic-package\)" $i > /dev/null ; then - infratype="generic" - hastarget=1 - fi - - if grep -E "\(host-cmake-package\)" $i > /dev/null ; then - infratype="cmake" - hashost=1 - fi - - if grep -E "\(cmake-package\)" $i > /dev/null ; then - infratype="cmake" - hastarget=1 - fi - - if grep -E "\(toolchain-external-package\)" $i > /dev/null ; then - infratype="toolchain-external" - hastarget=1 - fi - - if grep -E "\(waf-package\)" $i > /dev/null ; then - infratype="waf" - hastarget=1 - fi - - pkg=$(basename $i) - dir=$(dirname $i) - pkg=${pkg%.mk} - pkgvariable=$(echo ${pkg} | tr "a-z-" "A-Z_") - - - # Count packages per infrastructure - if [ -z ${infratype} ] ; then - infratype="manual" - manual_packages=$(($manual_packages+1)) - elif [ ${infratype} = "autotools" ]; then - autotools_packages=$(($autotools_packages+1)) - elif [ ${infratype} = "cmake" ]; then - cmake_packages=$(($cmake_packages+1)) - elif [ ${infratype} = "kconfig" ]; then - kconfig_packages=$(($kconfig_packages+1)) - elif [ ${infratype} = "luarocks" ]; then - luarocks_packages=$(($luarocks_packages+1)) - elif [ ${infratype} = "perl" ]; then - perl_packages=$(($perl_packages+1)) - elif [ ${infratype} = "python" ]; then - python_packages=$(($python_packages+1)) - elif [ ${infratype} = "rebar" ]; then - rebar_packages=$(($rebar_packages+1)) - elif [ ${infratype} = "virtual" ]; then - virtual_packages=$(($virtual_packages+1)) - elif [ ${infratype} = "generic" ]; then - generic_packages=$(($generic_packages+1)) - elif [ ${infratype} = "waf" ]; then - waf_packages=$(($waf_packages+1)) - fi - - if grep -qE "^${pkgvariable}_LICENSE[ ]*=" $i ; then - packages_with_license=$(($packages_with_license+1)) - license=1 - else - packages_without_license=$(($packages_without_license+1)) - license=0 - fi - - if grep -qE "^${pkgvariable}_LICENSE_FILES[ ]*=" $i ; then - packages_with_license_files=$(($packages_with_license_files+1)) - license_files=1 - else - packages_without_license_files=$(($packages_without_license_files+1)) - license_files=0 - fi - - if test -f ${dir}/${pkg}.hash; then - packages_with_hash_file=$(($packages_with_hash_file+1)) - hash_file=1 - else - packages_without_hash_file=$(($packages_without_hash_file+1)) - hash_file=0 - fi - - echo "" - - echo "" - echo "" - - package_dir=$(dirname $i) - patch_count=$(find ${package_dir} -name '*.patch' | wc -l) - total_patch_count=$(($total_patch_count+$patch_count)) - - if test $patch_count -lt 1 ; then - patch_count_class="nopatches" - elif test $patch_count -lt 5 ; then - patch_count_class="somepatches" - else - patch_count_class="lotsofpatches" - fi - - echo "" - - if [ ${infratype} = "manual" ] ; then - echo "" - else - echo "" - fi - - if [ ${license} -eq 0 ] ; then - echo "" - else - echo "" - fi - - if [ ${license_files} -eq 0 ] ; then - echo "" - else - echo "" - fi - - if [ ${hash_file} -eq 0 ] ; then - echo "" - else - echo "" - fi - - file_list=$(find ${package_dir} -name '*.mk' -o -name '*.in*' -o -name '*.hash') - nwarnings=$(./utils/check-package ${file_list} 2>&1 | sed '/\([0-9]*\) warnings generated/!d; s//\1/') - if [ ${nwarnings} -eq 0 ] ; then - echo "" - else - echo "" - fi - - echo "" - -done -echo "
Id Package Patch count Infrastructure License License files Hash fileCurrent versionLatest version Warnings
$cnt$i" - echo "$patch_count" - echo "manual" - echo "${infratype}
" - if [ ${hashost} -eq 1 -a ${hastarget} -eq 1 ]; then - echo "target + host" - elif [ ${hashost} -eq 1 ]; then - echo "host" - else - echo "target" - fi - echo "		NoYesNoYesNoYes${nwarnings}${nwarnings}
" - -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "" -echo "
Packages using the generic infrastructure$generic_packages
Packages using the cmake infrastructure$cmake_packages
Packages using the autotools infrastructure$autotools_packages
Packages using the luarocks infrastructure$luarocks_packages
Packages using the kconfig infrastructure$kconfig_packages
Packages using the perl infrastructure$perl_packages
Packages using the python infrastructure$python_packages
Packages using the rebar infrastructure$rebar_packages
Packages using the virtual infrastructure$virtual_packages
Packages using the waf infrastructure$waf_packages
Packages not using any infrastructure$manual_packages
Packages having license information$packages_with_license
Packages not having licence information$packages_without_license
Packages having license files information$packages_with_license_files
Packages not having licence files information$packages_without_license_files
Packages having hash file$packages_with_hash_file
Packages not having hash file$packages_without_hash_file
Number of patches in all packages$total_patch_count
TOTAL$cnt
" - -echo "
" -echo "Updated on $(LANG=C date), Git commit $(git log master -n 1 --pretty=format:%H)" -echo "" - -echo " -" -echo "" +""") + for pkg in sorted(packages): + dump_html_pkg(f, pkg) + f.write("") + + +def dump_html_stats(f, stats): + f.write("\n") + f.write("\n") + infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")] + for infra in infras: + f.write(" \n" % + (infra, stats["infra-%s" % infra])) + f.write(" \n" % + stats["license"]) + f.write(" \n" % + stats["no-license"]) + f.write(" \n" % + stats["license-files"]) + f.write(" \n" % + stats["no-license-files"]) + f.write(" \n" % + stats["hash"]) + f.write(" \n" % + stats["no-hash"]) + f.write(" \n" % + stats["patches"]) + f.write("\n" % + stats["rmo-mapping"]) + f.write("\n" % + stats["rmo-no-mapping"]) + f.write("\n" % + stats["version-uptodate"]) + f.write("\n" % + stats["version-not-uptodate"]) + f.write("\n" % + stats["version-unknown"]) + f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
Packages having a mapping on release-monitoring.org%s
Packages lacking a mapping on release-monitoring.org%s
Packages that are up-to-date%s
Packages that are not up-to-date%s
Packages with no known upstream version%s
\n") + + +def dump_gen_info(f): + # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032 + o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"]) + git_commit = o.splitlines()[0] + f.write("

Updated on %s, git commit %s

\n" % + (str(datetime.datetime.utcnow()), git_commit)) + + +def dump_html(packages, stats, output): + with open(output, 'w') as f: + f.write(html_header) + dump_html_all_pkgs(f, packages) + dump_html_stats(f, stats) + dump_gen_info(f) + f.write(html_footer) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-o', dest='output', action='store', required=True, + help='HTML output file') + parser.add_argument('-n', dest='npackages', type=int, action='store', + help='Number of packages') + parser.add_argument('-p', dest='packages', action='store', + help='List of packages (comma separated)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + if args.npackages and args.packages: + print "ERROR: -n and -p are mutually exclusive" + sys.exit(1) + if args.packages: + package_list = args.packages.split(",") + else: + package_list = None + print "Build package list ..." + packages = get_pkglist(args.npackages, package_list) + print "Getting package make info ..." + package_init_make_info() + print "Getting package details ..." + for pkg in packages: + pkg.set_infra() + pkg.set_license() + pkg.set_hash_info() + pkg.set_patch_count() + pkg.set_check_package_warnings() + pkg.set_current_version() + print "Getting latest versions ..." + add_latest_version_info(packages) + print "Calculate stats" + stats = calculate_stats(packages) + print "Write HTML" + dump_html(packages, stats, args.output) + + +__main__() diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new deleted file mode 100755 index 830040a485..0000000000 --- a/support/scripts/pkg-stats-new +++ /dev/null @@ -1,664 +0,0 @@ -#!/usr/bin/env python - -# Copyright (C) 2009 by Thomas Petazzoni -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -import argparse -import datetime -import fnmatch -import os -from collections import defaultdict -import re -import subprocess -import sys -import json -import urllib2 -from Queue import Queue -from threading import Thread - -INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)") -RELEASE_MONITORING_API = "http://release-monitoring.org/api" - - -class Package: - all_licenses = list() - all_license_files = list() - all_versions = dict() - - def __init__(self, name, path): - self.name = name - self.path = path - self.infras = None - self.has_license = False - self.has_license_files = False - self.has_hash = False - self.patch_count = 0 - self.warnings = 0 - self.current_version = None - self.latest_version = None - - def pkgvar(self): - return self.name.upper().replace("-", "_") - - def set_infra(self): - """ - Fills in the .infras field - """ - self.infras = list() - with open(self.path, 'r') as f: - lines = f.readlines() - for l in lines: - match = INFRA_RE.match(l) - if not match: - continue - infra = match.group(1) - if infra.startswith("host-"): - self.infras.append(("host", infra[5:])) - else: - self.infras.append(("target", infra)) - - def set_license(self): - """ - Fills in the .has_license and .has_license_files fields - """ - var = self.pkgvar() - if var in self.all_licenses: - self.has_license = True - if var in self.all_license_files: - self.has_license_files = True - - def set_hash_info(self): - """ - Fills in the .has_hash field - """ - hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) - - def set_patch_count(self): - """ - Fills in the .patch_count field - """ - self.patch_count = 0 - pkgdir = os.path.dirname(self.path) - for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) - - def set_current_version(self): - """ - Fills in the .current_version field - """ - var = self.pkgvar() - if var in self.all_versions: - self.current_version = self.all_versions[var] - - def set_check_package_warnings(self): - """ - Fills in the .warnings field - """ - cmd = ["./utils/check-package"] - pkgdir = os.path.dirname(self.path) - for root, dirs, files in os.walk(pkgdir): - for f in files: - if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": - cmd.append(os.path.join(root, f)) - o = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[1] - lines = o.splitlines() - for line in lines: - m = re.match("^([0-9]*) warnings generated", line) - if m: - self.warnings = int(m.group(1)) - return - - def get_latest_version_by_distro(self): - try: - req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", self.name)) - f = urllib2.urlopen(req, timeout=15) - except: - # Exceptions can typically be a timeout, or a 404 error if not project - return (False, None, None) - data = json.loads(f.read()) - if len(data['versions']) > 0: - return (True, data['versions'][0], data['id']) - else: - return (True, None, data['id']) - - def get_latest_version_by_guess(self): - try: - req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % self.name)) - f = urllib2.urlopen(req, timeout=15) - except: - # Exceptions can typically be a timeout, or a 404 error if not project - return (False, None, None) - data = json.loads(f.read()) - for p in data['projects']: - if p['name'] == self.name and len(p['versions']) > 0: - return (False, p['versions'][0], p['id']) - return (False, None, None) - - def set_latest_version(self): - # We first try by using the "Buildroot" distribution on - # release-monitoring.org, if it has a mapping for the current - # package name. - self.latest_version = self.get_latest_version_by_distro() - if self.latest_version == (False, None, None): - # If that fails because there is no mapping or because we had a - # request timeout, we try to search in all packages for a package - # of this name. - self.latest_version = self.get_latest_version_by_guess() - - def __eq__(self, other): - return self.path == other.path - - def __lt__(self, other): - return self.path < other.path - - def __str__(self): - return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) - - -def get_pkglist(npackages, package_list): - """ - Builds the list of Buildroot packages, returning a list of Package - objects. Only the .name and .path fields of the Package object are - initialized. - - npackages: limit to N packages - package_list: limit to those packages in this list - """ - WALK_USEFUL_SUBDIRS = ["boot", "linux", "package", "toolchain"] - WALK_EXCLUDES = ["boot/common.mk", - "linux/linux-ext-.*.mk", - "package/freescale-imx/freescale-imx.mk", - "package/gcc/gcc.mk", - "package/gstreamer/gstreamer.mk", - "package/gstreamer1/gstreamer1.mk", - "package/gtk2-themes/gtk2-themes.mk", - "package/matchbox/matchbox.mk", - "package/opengl/opengl.mk", - "package/qt5/qt5.mk", - "package/x11r7/x11r7.mk", - "package/doc-asciidoc.mk", - "package/pkg-.*.mk", - "package/nvidia-tegra23/nvidia-tegra23.mk", - "toolchain/toolchain-external/pkg-toolchain-external.mk", - "toolchain/toolchain-external/toolchain-external.mk", - "toolchain/toolchain.mk", - "toolchain/helpers.mk", - "toolchain/toolchain-wrapper.mk"] - packages = list() - count = 0 - for root, dirs, files in os.walk("."): - rootdir = root.split("/") - if len(rootdir) < 2: - continue - if rootdir[1] not in WALK_USEFUL_SUBDIRS: - continue - for f in files: - if not f.endswith(".mk"): - continue - # Strip ending ".mk" - pkgname = f[:-3] - if package_list and pkgname not in package_list: - continue - pkgpath = os.path.join(root, f) - skip = False - for exclude in WALK_EXCLUDES: - # pkgpath[2:] strips the initial './' - if re.match(exclude, pkgpath[2:]): - skip = True - continue - if skip: - continue - p = Package(pkgname, pkgpath) - packages.append(p) - count += 1 - if npackages and count == npackages: - return packages - return packages - - -def package_init_make_info(): - # Licenses - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_LICENSE"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - # Strip _LICENSE - pkgvar = pkgvar[:-8] - - # If value is "unknown", no license details available - if value == "unknown": - continue - Package.all_licenses.append(pkgvar) - - # License files - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_LICENSE_FILES"]) - for l in o.splitlines(): - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): - continue - - # Strip _LICENSE_FILES - pkgvar = pkgvar[:-14] - - Package.all_license_files.append(pkgvar) - - # Version - o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", - "-s", "printvars", "VARS=%_VERSION"]) - - # We process first the host package VERSION, and then the target - # package VERSION. This means that if a package exists in both - # target and host variants, with different version numbers - # (unlikely), we'll report the target version number. - version_list = o.splitlines() - version_list = [x for x in version_list if x.startswith("HOST_")] + \ - [x for x in version_list if not x.startswith("HOST_")] - for l in version_list: - # Get variable name and value - pkgvar, value = l.split("=") - - # If present, strip HOST_ from variable name - if pkgvar.startswith("HOST_"): - pkgvar = pkgvar[5:] - - if pkgvar.endswith("_DL_VERSION"): - continue - - # Strip _VERSION - pkgvar = pkgvar[:-8] - - Package.all_versions[pkgvar] = value - - -def set_version_worker(q): - while True: - pkg = q.get() - pkg.set_latest_version() - print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version)) - q.task_done() - - -def add_latest_version_info(packages): - """ - Fills in the .latest_version field of all Package objects - - This field has a special format: - (mapping, version, id) - with: - - mapping: boolean that indicates whether release-monitoring.org - has a mapping for this package name in the Buildroot distribution - or not - - version: string containing the latest version known by - release-monitoring.org for this package - - id: string containing the id of the project corresponding to this - package, as known by release-monitoring.org - """ - q = Queue() - for pkg in packages: - q.put(pkg) - # Since release-monitoring.org is rather slow, we create 8 threads - # that do HTTP requests to the site. - for i in range(8): - t = Thread(target=set_version_worker, args=[q]) - t.daemon = True - t.start() - q.join() - - -def calculate_stats(packages): - stats = defaultdict(int) - for pkg in packages: - # If packages have multiple infra, take the first one. For the - # vast majority of packages, the target and host infra are the - # same. There are very few packages that use a different infra - # for the host and target variants. - if len(pkg.infras) > 0: - infra = pkg.infras[0][1] - stats["infra-%s" % infra] += 1 - else: - stats["infra-unknown"] += 1 - if pkg.has_license: - stats["license"] += 1 - else: - stats["no-license"] += 1 - if pkg.has_license_files: - stats["license-files"] += 1 - else: - stats["no-license-files"] += 1 - if pkg.has_hash: - stats["hash"] += 1 - else: - stats["no-hash"] += 1 - if pkg.latest_version[0]: - stats["rmo-mapping"] += 1 - else: - stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: - stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: - stats["version-uptodate"] += 1 - else: - stats["version-not-uptodate"] += 1 - stats["patches"] += pkg.patch_count - return stats - - -html_header = """ - - - -Statistics of Buildroot packages - - -Results
- -

-""" - - -html_footer = """ - - - -""" - - -def infra_str(infra_list): - if not infra_list: - return "Unknown" - elif len(infra_list) == 1: - return "%s
%s" % (infra_list[0][1], infra_list[0][0]) - elif infra_list[0][1] == infra_list[1][1]: - return "%s
%s + %s" % \ - (infra_list[0][1], infra_list[0][0], infra_list[1][0]) - else: - return "%s (%s)
%s (%s)" % \ - (infra_list[0][1], infra_list[0][0], - infra_list[1][1], infra_list[1][0]) - - -def boolean_str(b): - if b: - return "Yes" - else: - return "No" - - -def dump_html_pkg(f, pkg): - f.write(" \n") - f.write(" %s\n" % pkg.path[2:]) - - # Patch count - td_class = ["centered"] - if pkg.patch_count == 0: - td_class.append("nopatches") - elif pkg.patch_count < 5: - td_class.append("somepatches") - else: - td_class.append("lotsofpatches") - f.write(" %s\n" % - (" ".join(td_class), str(pkg.patch_count))) - - # Infrastructure - infra = infra_str(pkg.infras) - td_class = ["centered"] - if infra == "Unknown": - td_class.append("wrong") - else: - td_class.append("correct") - f.write(" %s\n" % - (" ".join(td_class), infra_str(pkg.infras))) - - # License - td_class = ["centered"] - if pkg.has_license: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) - - # License files - td_class = ["centered"] - if pkg.has_license_files: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) - - # Hash - td_class = ["centered"] - if pkg.has_hash: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) - - # Current version - if len(pkg.current_version) > 20: - current_version = pkg.current_version[:20] + "..." - else: - current_version = pkg.current_version - f.write(" %s\n" % current_version) - - # Latest version - if pkg.latest_version[1] is None: - td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: - td_class.append("version-needs-update") - else: - td_class.append("version-good") - - if pkg.latest_version[1] is None: - latest_version_text = "Unknown" - else: - latest_version_text = "%s" % str(pkg.latest_version[1]) - - latest_version_text += "
" - - if pkg.latest_version[2]: - latest_version_text += "link, " % pkg.latest_version[2] - else: - latest_version_text += "no link, " - - if pkg.latest_version[0]: - latest_version_text += "has mapping" - else: - latest_version_text += "has no mapping" - - f.write(" %s\n" % - (" ".join(td_class), latest_version_text)) - - # Warnings - td_class = ["centered"] - if pkg.warnings == 0: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" %d\n" % - (" ".join(td_class), pkg.warnings)) - - f.write(" \n") - - -def dump_html_all_pkgs(f, packages): - f.write(""" - - - - - - - - - - - - -""") - for pkg in sorted(packages): - dump_html_pkg(f, pkg) - f.write("
PackagePatch countInfrastructureLicenseLicense filesHash fileCurrent versionLatest versionWarnings
") - - -def dump_html_stats(f, stats): - f.write("\n") - f.write("\n") - infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")] - for infra in infras: - f.write(" \n" % - (infra, stats["infra-%s" % infra])) - f.write(" \n" % - stats["license"]) - f.write(" \n" % - stats["no-license"]) - f.write(" \n" % - stats["license-files"]) - f.write(" \n" % - stats["no-license-files"]) - f.write(" \n" % - stats["hash"]) - f.write(" \n" % - stats["no-hash"]) - f.write(" \n" % - stats["patches"]) - f.write("\n" % - stats["rmo-mapping"]) - f.write("\n" % - stats["rmo-no-mapping"]) - f.write("\n" % - stats["version-uptodate"]) - f.write("\n" % - stats["version-not-uptodate"]) - f.write("\n" % - stats["version-unknown"]) - f.write("
Packages using the %s infrastructure%s
Packages having license information%s
Packages not having license information%s
Packages having license files information%s
Packages not having license files information%s
Packages having a hash file%s
Packages not having a hash file%s
Total number of patches%s
Packages having a mapping on release-monitoring.org%s
Packages lacking a mapping on release-monitoring.org%s
Packages that are up-to-date%s
Packages that are not up-to-date%s
Packages with no known upstream version%s
\n") - - -def dump_gen_info(f): - # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032 - o = subprocess.check_output(["git", "log", "master", "-n", "1", "--pretty=format:%H"]) - git_commit = o.splitlines()[0] - f.write("

Updated on %s, git commit %s

\n" % - (str(datetime.datetime.utcnow()), git_commit)) - - -def dump_html(packages, stats, output): - with open(output, 'w') as f: - f.write(html_header) - dump_html_all_pkgs(f, packages) - dump_html_stats(f, stats) - dump_gen_info(f) - f.write(html_footer) - - -def parse_args(): - parser = argparse.ArgumentParser() - parser.add_argument('-o', dest='output', action='store', required=True, - help='HTML output file') - parser.add_argument('-n', dest='npackages', type=int, action='store', - help='Number of packages') - parser.add_argument('-p', dest='packages', action='store', - help='List of packages (comma separated)') - return parser.parse_args() - - -def __main__(): - args = parse_args() - if args.npackages and args.packages: - print "ERROR: -n and -p are mutually exclusive" - sys.exit(1) - if args.packages: - package_list = args.packages.split(",") - else: - package_list = None - print "Build package list ..." - packages = get_pkglist(args.npackages, package_list) - print "Getting package make info ..." - package_init_make_info() - print "Getting package details ..." - for pkg in packages: - pkg.set_infra() - pkg.set_license() - pkg.set_hash_info() - pkg.set_patch_count() - pkg.set_check_package_warnings() - pkg.set_current_version() - print "Getting latest versions ..." - add_latest_version_info(packages) - print "Calculate stats" - stats = calculate_stats(packages) - print "Write HTML" - dump_html(packages, stats, args.output) - - -__main__()