[v2,3/6] manual: add deprecated-list.txt generation support

The whole deprecated-list.txt file is generated by the deprecated.py
script by searching for symbols depending on BR2_DEPRECATED in the
Buildroot code base.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
---

Change since v1:

* rename the script deprecated.py -> gen-deprecated-list.py (to keep consistency)

---
 docs/manual/manual.mk                  |   3 +
 support/scripts/gen-deprecated-list.py | 144 +++++++++++++++++++++++++++++++++
 2 files changed, 147 insertions(+)
 create mode 100755 support/scripts/gen-deprecated-list.py

On 29/11/12 08:47, Samuel Martin wrote:
> The whole deprecated-list.txt file is generated by the deprecated.py
> script by searching for symbols depending on BR2_DEPRECATED in the
> Buildroot code base.
>
> Signed-off-by: Samuel Martin<s.martin49@gmail.com>
> ---
>
> Change since v1:
>
> * rename the script deprecated.py ->  gen-deprecated-list.py (to keep consistency)
>
> ---
>   docs/manual/manual.mk                  |   3 +
>   support/scripts/gen-deprecated-list.py | 144 +++++++++++++++++++++++++++++++++
>   2 files changed, 147 insertions(+)
>   create mode 100755 support/scripts/gen-deprecated-list.py
>
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index d8437ba..4cd5839 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -27,6 +27,9 @@ endef
>   $(TOPDIR)/docs/manual/package-list.txt:
>   	$(TOPDIR)/support/scripts/gen-manual-pkg-list.sh>  $@
>
> +$(TOPDIR)/docs/manual/deprecated-list.txt:
> +	python2 $(TOPDIR)/support/scripts/gen-deprecated-list.py>  $@

  We don't have python2 in dependencies.sh, and anyway some things
don't currently work if python is python3.  So I'd just call the
script without the python2.  Anyway, does it not work with python3?


> +
>   ################################################################################
>   # GENDOC -- generates the make targets needed to build asciidoc documentation.
>   #
> diff --git a/support/scripts/gen-deprecated-list.py b/support/scripts/gen-deprecated-list.py
> new file mode 100755
> index 0000000..cd8c288
> --- /dev/null
> +++ b/support/scripts/gen-deprecated-list.py
> @@ -0,0 +1,144 @@
> +#!/usr/bin/env python
> +##
> +## deprecated-packages.py
> +##
> +## Author(s):
> +##  - Samuel MARTIN<s.martin49@gmail.com>
> +##
> +## Copyright (C) 2012 Samuel MARTIN
> +##
> +
> +# Python 2.7 script searching for kconfig symbols depending on 'BR2_DEPRECATED'
> +# and generating (printing to the standard output) the manual file in asciidoc.
> +
> +import os
> +import re
> +import sys
> +
> +
> +ASCIIDOC_HEADER = """\
> +//
> +// Autogenerated file
> +//
> +
> +[[deprecated]]
> +Deprecated list
> +---------------
> +
> +The following stuff are marked as _deprecated_ in Buildroot due to
> +their status either too old or unmaintained.

  Better:

The following packages are marked as _deprecated_ because they are
either too old or unmaintained in buildroot.

> +
> +// Please check and sort by grepping the source running:
> +//
> +// $ git grep -EB4 'depends on BR2_DEPRECATED'
> +//
> +// and:
> +//
> +// $ git grep -EB4 'depends on BR2_DEPRECATED' | \\
> +//     grep -Eo '(:|-).*?(config|comment) BR2_.*'

  This is redundant now.  Instead, mention the make target
after 'Autogenerated'.

> +"""
> +
> +NOT_SEARCHED = (".git", "board", "configs", "docs", "output", "support")
> +
> +# Relative path to category data map
> +#   key:   relative path from buildroot topdir to the search location
> +#   value: (rank_in_menuconfig, category_name, recursive_search)
> +#       rank_in_menuconfig is used for ordering the diplaying
> +#       category_name is used in the diplaying
> +#       recursive_search when searching for deprecated stuff
> +CAT_DIR2DATA = {
> +    'arch'      : (0, "target architecture",  True),
> +    '.'         : (1, "build options",        False),
> +    'toolchain' : (2, "toolchain",            True),
> +    'system'    : (3, "system configuration", True),
> +    'package'   : (4, "package selection",    True),
> +    'fs'        : (5, "filesystem images",    True),
> +    'boot'      : (6, "bootloaders",          True),
> +    'linux'     : (7, "kernel",               True),
> +    }
> +
> +DEPR_SYMBOL = "BR2_DEPRECATED"
> +
> +_REGEX = r"config BR2_(.*?)\n" + \
> +    "((.*?(?!config)(prompt|bool|string|int) \"(.*?)\".*?|[^\n]+)\n)*" + \
> +    "(.*?(?!config )" + DEPR_SYMBOL + ".*?)\n" + \
> +    "((.*?(?!config)(prompt|bool|string|int) \"(.*?)\".*?|[^\n]+)\n)*"

  This regex warrants some documentation :-) perhaps with re.VERBOSE

  Why is the !config needed in the prompt lines?

  Is there a reason to use the non-greedy .*? (except between the quotes)?

  I would also name the "interesting" parts with (?P<...>) and use finditer()
and m.group() to retrieve them.  But that's a matter of taste, I guess.

> +
> +REGEX = re.compile(_REGEX, flags=re.MULTILINE)
> +
> +
> +def _get_buildroot_topdir():
> +    topdir = os.path.join(os.path.dirname(__file__), "..", "..")
> +    topdir = os.path.abspath(topdir)
> +    return topdir

  I would just use the current directory as topdir.  All the rest of
buildroot anyway assumes it's being run from the topdir.

> +
> +def get_dir_list():
> +    root = _get_buildroot_topdir()
> +    dirs = ['.']
> +    for dir_ in os.listdir(root):
> +        if dir_ in NOT_SEARCHED:
> +            continue
> +        dir__ = os.path.join(root, dir_)
> +        if not os.path.isdir(dir__):
> +            continue
> +        dirs += [dir_]
> +    return dirs
> +
> +def find_deprecated(root, recursive):
> +    deprecated = list()
> +    for root_, _, files_ in os.walk(root):
> +        if not recursive and root_ != root:
> +            break
> +        for file_ in files_:
> +            if not file_.startswith("Config.in"):
> +                continue
> +            with open(os.path.join(root_, file_), "r") as f:
> +                content = f.read()
> +            if not DEPR_SYMBOL in content:
> +                continue
> +            found = REGEX.findall(content)
> +            if found:
> +                deprecated += found
> +    return deprecated

  It's quite a big change, but: why not use the source statements
in Config.in to do the recursion?  The regex it easy enough:
r'^\s*source "(?<path>.*?)"$'  This would be more future-safe:
when new directories are added, they will be handled correctly.

  It would also be good if you could add a warning when
len(found) != len(re.findall(DEPR_SYMBOL, content))

> +
> +
> +class Category():
> +
> +    def __init__(self, directory):
> +        self.path = os.path.join(_get_buildroot_topdir(), directory)
> +        rank, name, rec = CAT_DIR2DATA.get(directory, (99, directory, True))
> +        self.name = name
> +        self.rank = rank

  You can directly do
self.rank, self.name, rec = CAT_DIR2DATA.get(directory, (99, directory, True))

> +        self.depr_items = find_deprecated(self.path, rec)
> +
> +    def __str__(self):
> +        items_ = list()
> +        for item in self.depr_items:
> +            name_ = item[0].lower().replace("_", " ")
> +            name_ = re.sub("^package ", "", name_)
> +            vers = re.sub(".*?(version )?([0-9].*)", r'\2', name_)
> +            if vers:
> +                vers = re.sub(" ", ".", vers)
> +                name_ = re.sub("(version )?([0-9].*)", vers, name_)
> +            symbol = item[4]
> +            if not symbol:
> +                symbol = item[9]
> +            items_ += ["** %-25s +[%s]+" % (name_, symbol)]

  I don't think the name_ part adds much value.  In most cases, symbol
is either the same or more descriptive.  There are a few exceptions,
but these could be fixed by changing the Config.in.  These need to be
modified: gcc target, squashfs3 rootfs, uboot 11.06 and 11.09.

  BTW, symbol is not really symbol but rather 'description' or something
similar.

> +        items_.sort()
> +        output = "\n* %s:\n\n" % self.name.capitalize()
> +        output += "\n".join(items_)
> +        output += "\n"
> +        return output
> +
> +def main():
> +    categories = [Category(directory) for directory in get_dir_list()]
> +    categories = [category for category in categories if category.depr_items]
> +    categories.sort(cmp=lambda x, y: x.rank - y.rank if x.rank != y.rank \
> +                        else cmp(x.name, y.name))

     categories.sort(key=lambda x: (x.rank, x.name))

> +    output = ASCIIDOC_HEADER
> +    for category in categories:
> +        output += str(category)
> +    print output

  Don't concat the output, just print directly.

     print ASCIIDOC_HEADER
     for category in categories:
         print category


  Regards,
  Arnout


> +
> +if __name__ == "__main__":
> +    main()