From patchwork Wed Feb 21 22:13:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 876402 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmsH42N6kz9sWG for ; Thu, 22 Feb 2018 09:14:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CC52B84C16; Wed, 21 Feb 2018 22:14:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tfq0wJCScOoG; Wed, 21 Feb 2018 22:14:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id AEA6884CD9; Wed, 21 Feb 2018 22:14:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 245661C23BE for ; Wed, 21 Feb 2018 22:14:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 20F5484CD9 for ; Wed, 21 Feb 2018 22:14:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y3gsNXe4_hMi for ; Wed, 21 Feb 2018 22:14:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by fraxinus.osuosl.org (Postfix) with ESMTP id 05F0B84C16 for ; Wed, 21 Feb 2018 22:14:51 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id BB398207BB; Wed, 21 Feb 2018 23:14:49 +0100 (CET) Received: from localhost (LFbn-1-2142-168.w90-76.abo.wanadoo.fr [90.76.200.168]) by mail.free-electrons.com (Postfix) with ESMTPSA id 6F5C120736; Wed, 21 Feb 2018 23:14:49 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , Ricardo Martincoski Date: Wed, 21 Feb 2018 23:13:41 +0100 Message-Id: <20180221221342.15683-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180221221342.15683-1-thomas.petazzoni@bootlin.com> References: <20180221221342.15683-1-thomas.petazzoni@bootlin.com> Subject: [Buildroot] [PATCH next v2 4/5] support/scripts/pkg-stats-new: add latest upstream version information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. If there is no result, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Since release-monitoring.org is a bit slow, we have 8 threads that fetch information in parallel. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni --- Changes since v1: - Fix flake8 warnings - Add missing newline in HTML --- support/scripts/pkg-stats-new | 142 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) diff --git a/support/scripts/pkg-stats-new b/support/scripts/pkg-stats-new index c4174877aa..b27f4df9a8 100755 --- a/support/scripts/pkg-stats-new +++ b/support/scripts/pkg-stats-new @@ -24,6 +24,10 @@ from collections import defaultdict import re import subprocess import sys +import json +import urllib2 +from Queue import Queue +from threading import Thread class Package: @@ -37,6 +41,7 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.latest_version = None def __eq__(self, other): return self.path == other.path @@ -259,6 +264,83 @@ def add_check_package_warnings(packages): pkg.warnings = get_check_package_warnings(os.path.dirname(pkg.path)) +RELEASE_MONITORING_API = "http://release-monitoring.org/api" + + +def get_latest_version_by_distro(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "project", "Buildroot", package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + if len(data['versions']) > 0: + return (True, data['versions'][0], data['id']) + else: + return (True, None, data['id']) + + +def get_latest_version_by_guess(package): + req = urllib2.Request(os.path.join(RELEASE_MONITORING_API, "projects", "?pattern=%s" % package)) + f = urllib2.urlopen(req) + data = json.loads(f.read()) + for p in data['projects']: + if p['name'] == package and len(p['versions']) > 0: + return (False, p['versions'][0], p['id']) + return (False, None, None) + + +def get_latest_version(package): + try: + # We first try by using the "Buildroot" distribution on + # release-monitoring.org, if it has a mapping for the current + # package name. + return get_latest_version_by_distro(package) + except urllib2.HTTPError, e: + # If that fails because there is no mapping, we try to search + # in all packages for a package of this name. + if e.code == 404: + return get_latest_version_by_guess(package) + else: + return (False, None, None) + + +def get_version_worker(q): + while True: + pkg = q.get() + try: + pkg.latest_version = get_latest_version(pkg.name) + print " [%04d] %s => %s" % (q.qsize(), pkg.name, str(pkg.latest_version)) + except ValueError: + pkg.latest_version = (False, None, None) + print " [%04d] %s => Value Error" % (q.qsize(), pkg.name) + q.task_done() + + +def add_latest_version_info(packages): + """ + Fills in the .latest_version field of all Package objects + + This field has a special format: + (mapping, version, id) + with: + - mapping: boolean that indicates whether release-monitoring.org + has a mapping for this package name in the Buildroot distribution + or not + - version: string containing the latest version known by + release-monitoring.org for this package + - id: string containing the id of the project corresponding to this + package, as known by release-monitoring.org + """ + q = Queue() + for pkg in packages: + q.put(pkg) + # Since release-monitoring.org is rather slow, we create 8 threads + # that do HTTP requests to the site. + for i in range(8): + t = Thread(target=get_version_worker, args=[q]) + t.daemon = True + t.start() + q.join() + + def calculate_stats(packages): stats = defaultdict(int) for pkg in packages: @@ -283,6 +365,16 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.latest_version[0]: + stats["rmo-mapping"] += 1 + else: + stats["rmo-no-mapping"] += 1 + if not pkg.latest_version[1]: + stats["version-unknown"] += 1 + elif pkg.latest_version[1] == pkg.current_version: + stats["version-uptodate"] += 1 + else: + stats["version-not-uptodate"] += 1 stats["patches"] += pkg.patch_count return stats @@ -315,6 +407,15 @@ td.somepatches { td.lotsofpatches { background: #ff9a69; } +td.version-good { + background: #d2ffc4; +} +td.version-needs-update { + background: #ff9a69; +} +td.version-unknown { + background: #ffd870; +} Statistics of Buildroot packages @@ -413,6 +514,34 @@ def dump_html_pkg(f, pkg): # Current version f.write(" %s\n" % pkg.current_version) + # Latest version + if pkg.latest_version[1] is None: + td_class.append("version-unknown") + elif pkg.latest_version[1] != pkg.current_version: + td_class.append("version-needs-update") + else: + td_class.append("version-good") + + if pkg.latest_version[1] is None: + latest_version_text = "Unknown" + else: + latest_version_text = "%s" % str(pkg.latest_version[1]) + + latest_version_text += "
" + + if pkg.latest_version[2]: + latest_version_text += "link, " % pkg.latest_version[2] + else: + latest_version_text += "no link, " + + if pkg.latest_version[0]: + latest_version_text += "has mapping" + else: + latest_version_text += "has no mapping" + + f.write(" %s\n" % + (" ".join(td_class), latest_version_text)) + # Warnings td_class = ["centered"] if pkg.warnings == 0: @@ -436,6 +565,7 @@ def dump_html_all_pkgs(f, packages): License files Hash file Current version +Latest version Warnings """) @@ -465,6 +595,16 @@ def dump_html_stats(f, stats): stats["no-hash"]) f.write(" Total number of patches%s\n" % stats["patches"]) + f.write("Packages having a mapping on release-monitoring.org%s\n" % + stats["rmo-mapping"]) + f.write("Packages lacking a mapping on release-monitoring.org%s\n" % + stats["rmo-no-mapping"]) + f.write("Packages that are up-to-date%s\n" % + stats["version-uptodate"]) + f.write("Packages that are not up-to-date%s\n" % + stats["version-not-uptodate"]) + f.write("Packages with no known upstream version%s\n" % + stats["version-unknown"]) f.write("\n") @@ -517,6 +657,8 @@ def __main__(): add_patch_count(packages) print "Get package warnings ..." add_check_package_warnings(packages) + print "Get latest version ..." + add_latest_version_info(packages) print "Calculate stats" stats = calculate_stats(packages) print "Write HTML"