From patchwork Mon Mar 25 23:28:06 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 231024 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ozlabs.org (Postfix) with ESMTP id 6E03A2C00A0 for ; Tue, 26 Mar 2013 10:31:11 +1100 (EST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4C0B0A01CD; Mon, 25 Mar 2013 23:31:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NLM7k1KGt3Hz; Mon, 25 Mar 2013 23:30:45 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id ADCFFA020E; Mon, 25 Mar 2013 23:29:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id D9FA28F753 for ; Mon, 25 Mar 2013 23:29:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2FBBD87DF6 for ; Mon, 25 Mar 2013 23:29:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References" Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p4GzTOhE2H1r for ; Mon, 25 Mar 2013 23:28:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-we0-f173.google.com (mail-we0-f173.google.com [74.125.82.173]) by whitealder.osuosl.org (Postfix) with ESMTPS id 167A98B7AD for ; Mon, 25 Mar 2013 23:28:40 +0000 (UTC) Received: by mail-we0-f173.google.com with SMTP id t57so1426692wey.18 for ; Mon, 25 Mar 2013 16:28:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references:in-reply-to:references; bh=OAaAqvtzp6CPf5SBVf5okNTbPXSLIillUN3BmnpFdgw=; b=s8bvjzvPJa5KHMZgAMC6Z17CZlFuqaUBH1hAQzvj8HcYzMOTyLCQ9V/0hhw+k4G1KE EyZ7f4cMU6g9eKY3e7G4o73w+pPGrR4CgrLA/0L55b9oyG0pE/QLHqKPmGI3uK9FDj0e y19XIE7HRTVP+L4sCJIr982FZuJmUVT5ToSFyyf2fdel3iYa1xgKEef25ZUnpgzsFfWC 1OZGSTa9ulSnU4xxqN1WwWQwDc/8hFP7SOubt3LNiNEMbbCU8yjaFqvusthYZ9X4NEMm KSMjNpZbjplyAd6U9l6xe21Nrabujf/wHLDpWKErRBabvPJ2T/hGB1d92sZwMRuJQdTh JQRg== X-Received: by 10.180.98.198 with SMTP id ek6mr2169893wib.7.1364254119356; Mon, 25 Mar 2013 16:28:39 -0700 (PDT) Received: from localhost.localdomain (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPS id fg6sm8768wib.10.2013.03.25.16.28.37 (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 25 Mar 2013 16:28:38 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@busybox.net Date: Tue, 26 Mar 2013 00:28:06 +0100 Message-Id: <318477b8af17091558da91d4932337b25ce4b942.1364253936.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 1.7.2.5 In-Reply-To: References: In-Reply-To: References: Cc: "Yann E. MORIN" Subject: [Buildroot] [PATCH 11/15] support/scripts: add gen-manual-lists.py X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: buildroot-bounces@busybox.net From: Samuel Martin Script generating the target and host package tables, and the deprecated stuff list as well. These tables and lists are generated parsing the Config.in files. Signed-off-by: Samuel Martin [yann.morin.1998@free.fr: no leading dot, no menu path for host-utils] Signed-off-by: "Yann E. MORIN" --- Changes since v5: * fix shebang (Yann) * generate files in $(O) (Thomas P.) * minor fixes Changes since v4: * remove date in the generated lists Chagnes since v3: * Re-introduce try/except on the kconfiglib import. This is mandatory because the kconfiglib module is no longer embedded in the Buildroot source tree because of undefined license (the author wants to keep the authorship of his work of course, but he does not seem inclined to define any license on his project which would allow 3rd parties to re-use this project without any legal issues). Changes since v1: * many improvements, fixes, refactoring and cleanup (Arnout) --- support/scripts/gen-manual-lists.py | 379 +++++++++++++++++++++++++++++++++++ 1 files changed, 379 insertions(+), 0 deletions(-) create mode 100755 support/scripts/gen-manual-lists.py diff --git a/support/scripts/gen-manual-lists.py b/support/scripts/gen-manual-lists.py new file mode 100755 index 0000000..ae1d4ff --- /dev/null +++ b/support/scripts/gen-manual-lists.py @@ -0,0 +1,379 @@ +#!/usr/bin/env python +## +## gen-manual-lists.py +## +## This script generates the following Buildroot manual appendices: +## - the package tables (one for the target, the other for host tools); +## - the deprecated items. +## +## Author(s): +## - Samuel Martin +## +## Copyright (C) 2013 Samuel Martin +## +## This program is free software; you can redistribute it and/or modify +## it under the terms of the GNU General Public License as published by +## the Free Software Foundation; either version 2 of the License, or +## (at your option) any later version. +## +## This program is distributed in the hope that it will be useful, +## but WITHOUT ANY WARRANTY; without even the implied warranty of +## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +## GNU General Public License for more details. +## +## You should have received a copy of the GNU General Public License +## along with this program; if not, write to the Free Software +## Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +## + +## Note about python2. +## +## This script can currently only be run using python2 interpreter due to +## its kconfiglib dependency (which is not yet python3 friendly). + +from __future__ import print_function +from __future__ import unicode_literals + +import os +import re +import sys +import datetime +from argparse import ArgumentParser + +try: + import kconfiglib +except ImportError: + message = """ +Could not find the module 'kconfiglib' in the PYTHONPATH: +""" + message += "\n".join([" {0}".format(path) for path in sys.path]) + message += """ + +Make sure the Kconfiglib directory is in the PYTHONPATH, then relaunch the +script. + +You can get kconfiglib from: + https://github.com/ulfalizer/Kconfiglib + + +""" + sys.stderr.write(message) + raise + + +def get_symbol_subset(root, filter_func): + """ Return a generator of kconfig items. + + :param root_item: Root item of the generated subset of items + :param filter_func: Filter function + + """ + if hasattr(root, "get_items"): + get_items = root.get_items + elif hasattr(root, "get_top_level_items"): + get_items = root.get_top_level_items + else: + message = "The symbol does not contain any subset of symbols" + raise Exception(message) + for item in get_items(): + if item.is_symbol(): + if not item.prompts: + continue + if not filter_func(item): + continue + yield item + elif item.is_menu() or item.is_choice(): + for i in get_symbol_subset(item, filter_func): + yield i + + +def get_symbol_parents(item, root=None, enable_choice=False): + """ Return the list of the item's parents. The lasst item of the list is + the closest parent, the first the furthest. + + :param item: Item from which the the parent list is generated + :param root: Root item stopping the search (not included in the + parent list) + :param enable_choice: Flag enabling choices to appear in the parent list + + """ + parent = item.get_parent() + parents = [] + while parent and parent != root: + if parent.is_menu(): + parents.append(parent.get_title()) + elif enable_choice and parent.is_choice(): + parents.append(parent.prompts[0][0]) + parent = parent.get_parent() + if isinstance(root, kconfiglib.Menu) or \ + (enable_choice and isinstance(root, kconfiglib.Choice)): + parents.append("") # Dummy empty parrent to get a leading arrow -> + parents.reverse() + return parents + + +def format_asciidoc_table(root, get_label_func, filter_func=lambda x: True, + enable_choice=False, sorted=True, sub_menu=True, + item_label=None): + """ Return the asciidoc formatted table of the items and their location. + + :param root: Root item of the item subset + :param get_label_func: Item's label getter function + :param filter_func: Filter function to apply on the item subset + :param enable_choice: Enable choices to appear as part of the item's + location + :param sorted: Flag to alphabetically sort the table + :param sub_menu: Output the column with the sub-menu path + + """ + def _format_entry(label, parents, sub_menu): + """ Format an asciidoc table entry. + + """ + if sub_menu: + return "| {0:<40} <| {1}\n".format(label, " -> ".join(parents)) + else: + return "| {0:<40}\n".format(label) + + lines = [] + for item in get_symbol_subset(root, filter_func): + if not item.is_symbol() or not item.prompts: + continue + loc = get_symbol_parents(item, root, enable_choice=enable_choice) + lines.append(_format_entry(get_label_func(item), loc, sub_menu)) + if sorted: + lines.sort(key=lambda x: x.lower()) + if hasattr(root, "get_title"): + loc_label = get_symbol_parents(root, None, enable_choice=enable_choice) + loc_label += [root.get_title(), "..."] + else: + loc_label = ["Location"] + if not item_label: + item_label = "Items" + table = ":halign: center\n\n" + if sub_menu: + width = "100%" + columns = "^1,4" + else: + width = "30%" + columns = "^1" + table = "[width=\"{0}\",cols=\"{1}\",options=\"header\"]\n".format(width, columns) + table += "|===================================================\n" + table += _format_entry(item_label, loc_label, sub_menu) + table += "\n" + "".join(lines) + "\n" + table += "|===================================================\n" + return table + + +class Buildroot: + """ Buildroot configuration object. + + """ + root_config = "Config.in" + package_dirname = "package" + package_prefixes = ["BR2_PACKAGE_", "BR2_PACKAGE_HOST_"] + re_pkg_prefix = re.compile(r"^(" + "|".join(package_prefixes) + ").*") + deprecated_symbol = "BR2_DEPRECATED" + list_in = """\ +// +// Automatically generated list for Buildroot manual. +// + +{table} +""" + + list_info = { + 'target-packages': { + 'filename': "package-list", + 'root_menu': "Package Selection for the target", + 'filter': "_is_package", + 'sorted': True, + 'sub_menu': True, + }, + 'host-packages': { + 'filename': "host-package-list", + 'root_menu': "Host utilities", + 'filter': "_is_package", + 'sorted': True, + 'sub_menu': False, + }, + 'deprecated': { + 'filename': "deprecated-list", + 'root_menu': None, + 'filter': "_is_deprecated", + 'sorted': False, + 'sub_menu': True, + }, + } + + def __init__(self): + self.base_dir = os.environ.get("TOPDIR") + self.output_dir = os.environ.get("O") + self.package_dir = os.path.join(self.base_dir, self.package_dirname) + # The kconfiglib requires an environment variable named "srctree" to + # load the configuration, so set it. + os.environ.update({'srctree': self.base_dir}) + self.config = kconfiglib.Config(os.path.join(self.base_dir, + self.root_config)) + self._deprecated = self.config.get_symbol(self.deprecated_symbol) + + self.gen_date = datetime.datetime.utcnow() + self.br_version_full = os.environ.get("BR2_VERSION_FULL") + if self.br_version_full and self.br_version_full.endswith("-git"): + self.br_version_full = self.br_version_full[:-4] + if not self.br_version_full: + self.br_version_full = "undefined" + + def _get_package_symbols(self, package_name): + """ Return a tuple containing the target and host package symbol. + + """ + symbols = re.sub("[-+.]", "_", package_name) + symbols = symbols.upper() + symbols = tuple([prefix + symbols for prefix in self.package_prefixes]) + return symbols + + def _is_deprecated(self, symbol): + """ Return True if the symbol is marked as deprecated, otherwise False. + + """ + return self._deprecated in symbol.get_referenced_symbols() + + def _is_package(self, symbol): + """ Return True if the symbol is a package or a host package, otherwise + False. + + """ + if not self.re_pkg_prefix.match(symbol.get_name()): + return False + pkg_name = re.sub("BR2_PACKAGE_(HOST_)?(.*)", r"\2", symbol.get_name()) + + pattern = "^(HOST_)?" + pkg_name + "$" + pattern = re.sub("_", ".", pattern) + pattern = re.compile(pattern, re.IGNORECASE) + # Here, we cannot just check for the location of the Config.in because + # of the "virtual" package. + # + # So, to check that a symbol is a package (not a package option or + # anything else), we check for the existence of the package *.mk file. + # + # By the way, to actually check for a package, we should grep all *.mk + # files for the following regex: + # "\$\(eval \$\((host-)?(generic|autotools|cmake)-package\)\)" + # + # Implementation details: + # + # * The package list is generated from the *.mk file existence, the + # first time this function is called. Despite the memory consumtion, + # this list is stored because the execution time of this script is + # noticebly shorter than re-scannig the package sub-tree for each + # symbol. + if not hasattr(self, "_package_list"): + pkg_list = [] + for _, _, files in os.walk(self.package_dir): + for file_ in (f for f in files if f.endswith(".mk")): + pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_)) + setattr(self, "_package_list", pkg_list) + for pkg in getattr(self, "_package_list"): + if pattern.match(pkg): + return True + return False + + def _get_symbol_label(self, symbol, mark_deprecated=True): + """ Return the label (a.k.a. prompt text) of the symbol. + + :param symbol: The symbol + :param mark_deprecated: Append a 'deprecated' to the label + + """ + label = symbol.prompts[0][0] + if self._is_deprecated(symbol) and mark_deprecated: + label += " *(deprecated)*" + return label + + def print_list(self, list_type, enable_choice=True, enable_deprecated=True, + dry_run=False, output=None): + """ Print the requested list. If not dry run, then the list is + automatically written in its own file. + + :param list_type: The list type to be generated + :param enable_choice: Flag enabling choices to appear in the list + :param enable_deprecated: Flag enabling deprecated items to appear in + the package lists + :param dry_run: Dry run (print the list in stdout instead of + writing the list file + + """ + def _get_menu(title): + """ Return the first symbol menu matching the given title. + + """ + menus = self.config.get_menus() + menu = [m for m in menus if m.get_title().lower() == title.lower()] + if not menu: + message = "No such menu: '{0}'".format(title) + raise Exception(message) + return menu[0] + + list_config = self.list_info[list_type] + root_title = list_config.get('root_menu') + if root_title: + root_item = _get_menu(root_title) + else: + root_item = self.config + filter_ = getattr(self, list_config.get('filter')) + filter_func = lambda x: filter_(x) + if not enable_deprecated and list_type != "deprecated": + filter_func = lambda x: filter_(x) and not self._is_deprecated(x) + mark_depr = list_type != "deprecated" + get_label = lambda x: self._get_symbol_label(x, mark_depr) + item_label = "Features" if list_type == "deprecated" else "Packages" + + table = format_asciidoc_table(root_item, get_label, + filter_func=filter_func, + enable_choice=enable_choice, + sorted=list_config.get('sorted'), + sub_menu=list_config.get('sub_menu'), + item_label=item_label) + + content = self.list_in.format(table=table) + + if dry_run: + print(content) + return + + if not output: + output_dir = self.output_dir + if not output_dir: + print("Warning: Undefined output directory.") + print("\tUse source directory as output location.") + output_dir = self.base_dir + output = os.path.join(output_dir, + list_config.get('filename') + ".txt") + if not os.path.exists(os.path.dirname(output)): + os.makedirs(os.path.dirname(output)) + print("Writing the {0} list in:\n\t{1}".format(list_type, output)) + with open(output, 'w') as fout: + fout.write(content) + + +if __name__ == '__main__': + list_types = ['target-packages', 'host-packages', 'deprecated'] + parser = ArgumentParser() + parser.add_argument("list_type", nargs="?", choices=list_types, + help="""\ +Generate the given list (generate all lists if unspecified)""") + parser.add_argument("-n", "--dry-run", dest="dry_run", action='store_true', + help="Output the generated list to stdout") + parser.add_argument("--output-target", dest="output_target", + help="Output target package file") + parser.add_argument("--output-host", dest="output_host", + help="Output host package file") + parser.add_argument("--output-deprecated", dest="output_deprecated", + help="Output deprecated file") + args = parser.parse_args() + lists = [args.list_type] if args.list_type else list_types + buildroot = Buildroot() + for list_name in lists: + output = getattr(args, "output_" + list_name.split("-", 1)[0]) + buildroot.print_list(list_name, dry_run=args.dry_run, output=output)