[U-Boot,01/93] Add a simple script to remove boards

This script attempts to create a git commit which removes a single board.
It is quite fallible and everything it does needs checking. But it can
help speed up the process.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 tools/rmboard.py | 145 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 145 insertions(+)
 create mode 100755 tools/rmboard.py

On Mon, Nov 19, 2018 at 9:54 AM Simon Glass <sjg@chromium.org> wrote:
>
> This script attempts to create a git commit which removes a single board.
> It is quite fallible and everything it does needs checking. But it can
> help speed up the process.
>

This patch looks like it goes through the configs, finds boards and
blindly deletes them.  What happens in the instance where multiple
defconfig files use the same include/configs file or C source?  If one
of the defconfigs is missing something, does it then delete the
others?  It seems like it should make sure that all variations don't
comply before blowing away the source and include/configs.

adam

> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
>  tools/rmboard.py | 145 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 145 insertions(+)
>  create mode 100755 tools/rmboard.py
>
> diff --git a/tools/rmboard.py b/tools/rmboard.py
> new file mode 100755
> index 00000000000..c6be434c9ed
> --- /dev/null
> +++ b/tools/rmboard.py
> @@ -0,0 +1,145 @@
> +#! /usr/bin/python
> +
> +'''
> +Script to remove boards
> +
> +Usage:
> +   rmboard.py <board_name>...
> +
> +A single commit is created for each board removed.
> +
> +Some boards may depend on files provided by another and this will cause
> +problems.
> +
> +This script works by:
> +    - Looking through the MAINTAINERS files which mention a board to find out
> +        what files the board uses
> +    - Looking through the Kconfig files which mention a board to find one that
> +        needs to have material removed
> +'''
> +
> +import glob
> +import os
> +import re
> +import sys
> +
> +# Bring in the patman libraries
> +our_path = os.path.dirname(os.path.realpath(__file__))
> +sys.path.append(os.path.join(our_path, '../patman'))
> +
> +import command
> +
> +def rm_kconfig_include(path):
> +    """Remove a path from Kconfig files
> +
> +    This function finds the given path in a 'source' statement in a Kconfig
> +    file and removes that line from the file. This is needed because the path
> +    is going to be removed, so any reference to it will cause a problem with
> +    Kconfig parsing.
> +
> +    The changes are made locally and then added to the git staging area.
> +
> +    Args:
> +        path: Path to search for and remove
> +    """
> +    print 'path', path
> +    cmd = ['git', 'grep', path]
> +    stdout = command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
> +    if not stdout:
> +        return
> +    fname = stdout.split(':')[0]
> +
> +    print "Fixing up '%s' to remove reference to '%s'" % (fname, path)
> +    cmd = ['sed', '-i', '\|%s|d' % path, fname]
> +    stdout = command.RunPipe([cmd], capture=True).stdout
> +
> +    cmd = ['git', 'add', fname]
> +    stdout = command.RunPipe([cmd], capture=True).stdout
> +
> +def rm_board(board):
> +    """Handle creating a commit which removes a single board
> +
> +    Args:
> +        board: Board name to remove
> +    """
> +
> +    # Find all MAINTAINERS and Kconfig files which mention the board
> +    cmd = ['git', 'grep', '-l', board]
> +    stdout = command.RunPipe([cmd], capture=True).stdout
> +    maintain = []
> +    kconfig = []
> +    for line in stdout.splitlines():
> +        line = line.strip()
> +        if 'MAINTAINERS' in line:
> +            if line not in maintain:
> +                maintain.append(line)
> +        elif 'Kconfig' in line:
> +            kconfig.append(line)
> +    paths = []
> +    cc = []
> +    print 'maintain', maintain
> +
> +    # Look through the MAINTAINERS file to find things to remove
> +    for fname in maintain:
> +        with open(fname) as fd:
> +            for line in fd:
> +                line = line.strip()
> +                fields = re.split('[ \t]', line, 1)
> +                print fields
> +                if len(fields) == 2:
> +                    if fields[0] == 'M:':
> +                        cc.append(fields[1])
> +                    elif fields[0] == 'F:':
> +                        paths.append(fields[1].strip())
> +    print 'paths', paths
> +
> +    # Expannd any wildcards in the MAINTAINRERS file
> +    real = []
> +    for path in paths:
> +        if path[-1] == '/':
> +            path = path[:-1]
> +        if '*' in path:
> +            globbed = glob.glob(path)
> +            print "Expanded '%s' to '%s'" % (path, globbed)
> +            real += globbed
> +        else:
> +            real.append(path)
> +    print 'real', real
> +
> +    # Search for Kconfig files in the resulting list. Remove any 'source' lines
> +    # which referenced Kconfig files we want to remove
> +    for path in real:
> +        cmd = ['find', path]
> +        stdout = (command.RunPipe([cmd], capture=True, raise_on_error=False).
> +                  stdout)
> +        for fname in stdout.splitlines():
> +            if fname.endswith('Kconfig'):
> +                rm_kconfig_include(fname)
> +
> +    # Remove unwanted files
> +    cmd = ['git', 'rm', '-r'] + real
> +    stdout = command.RunPipe([cmd], capture=True).stdout
> +
> +    # Change the messages as needed
> +    msg = '''arm: Remove %s board
> +
> +This board has not been converted to CONFIG_DM_BLK by the deadline.
> +Remove it.
> +
> +''' % board
> +    for name in cc:
> +        msg += 'Patch-cc: %s\n' % name
> +
> +    # Create the commit
> +    cmd = ['git', 'commit', '-s', '-m', msg]
> +    stdout = command.RunPipe([cmd], capture=True).stdout
> +    print kconfig
> +
> +    # Check if the board is mentioned anywhere else. The user will need to deal
> +    # with this
> +    cmd = ['git', 'grep', '-il', board]
> +    print command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
> +    print ' '.join(cmd)
> +
> +for board in sys.argv[1:]:
> +    rm_board(board)
> --
> 2.19.1.1215.g8438c0b245-goog
>

On Mon, 19 Nov 2018 at 11:19, Adam Ford <aford173@gmail.com> wrote:
>
> On Mon, Nov 19, 2018 at 9:54 AM Simon Glass <sjg@chromium.org> wrote:
> >
> > This script attempts to create a git commit which removes a single board.
> > It is quite fallible and everything it does needs checking. But it can
> > help speed up the process.
> >
>
> This patch looks like it goes through the configs, finds boards and
> blindly deletes them.  What happens in the instance where multiple
> defconfig files use the same include/configs file or C source?  If one
> of the defconfigs is missing something, does it then delete the
> others?  It seems like it should make sure that all variations don't
> comply before blowing away the source and include/configs.
>
> adam
>
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> >  tools/rmboard.py | 145 +++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 145 insertions(+)
> >  create mode 100755 tools/rmboard.py
> >
> > diff --git a/tools/rmboard.py b/tools/rmboard.py
> > new file mode 100755
> > index 00000000000..c6be434c9ed
> > --- /dev/null
> > +++ b/tools/rmboard.py
> > @@ -0,0 +1,145 @@
> > +#! /usr/bin/python
> > +
> > +'''
> > +Script to remove boards
> > +
> > +Usage:
> > +   rmboard.py <board_name>...
> > +
> > +A single commit is created for each board removed.
> > +
> > +Some boards may depend on files provided by another and this will cause
> > +problems.
> > +
> > +This script works by:
> > +    - Looking through the MAINTAINERS files which mention a board to find out
> > +        what files the board uses
> > +    - Looking through the Kconfig files which mention a board to find one that
> > +        needs to have material removed
> > +'''
> > +
> > +import glob
> > +import os
> > +import re
> > +import sys
> > +
> > +# Bring in the patman libraries
> > +our_path = os.path.dirname(os.path.realpath(__file__))
> > +sys.path.append(os.path.join(our_path, '../patman'))
> > +
> > +import command
> > +
> > +def rm_kconfig_include(path):
> > +    """Remove a path from Kconfig files
> > +
> > +    This function finds the given path in a 'source' statement in a Kconfig
> > +    file and removes that line from the file. This is needed because the path
> > +    is going to be removed, so any reference to it will cause a problem with
> > +    Kconfig parsing.
> > +
> > +    The changes are made locally and then added to the git staging area.
> > +
> > +    Args:
> > +        path: Path to search for and remove
> > +    """
> > +    print 'path', path
> > +    cmd = ['git', 'grep', path]
> > +    stdout = command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
> > +    if not stdout:
> > +        return
> > +    fname = stdout.split(':')[0]
> > +
> > +    print "Fixing up '%s' to remove reference to '%s'" % (fname, path)
> > +    cmd = ['sed', '-i', '\|%s|d' % path, fname]
> > +    stdout = command.RunPipe([cmd], capture=True).stdout
> > +
> > +    cmd = ['git', 'add', fname]
> > +    stdout = command.RunPipe([cmd], capture=True).stdout
> > +
> > +def rm_board(board):
> > +    """Handle creating a commit which removes a single board
> > +
> > +    Args:
> > +        board: Board name to remove
> > +    """
> > +
> > +    # Find all MAINTAINERS and Kconfig files which mention the board
> > +    cmd = ['git', 'grep', '-l', board]
> > +    stdout = command.RunPipe([cmd], capture=True).stdout
> > +    maintain = []
> > +    kconfig = []
> > +    for line in stdout.splitlines():
> > +        line = line.strip()
> > +        if 'MAINTAINERS' in line:
> > +            if line not in maintain:
> > +                maintain.append(line)
> > +        elif 'Kconfig' in line:
> > +            kconfig.append(line)
> > +    paths = []
> > +    cc = []
> > +    print 'maintain', maintain
> > +
> > +    # Look through the MAINTAINERS file to find things to remove
> > +    for fname in maintain:
> > +        with open(fname) as fd:
> > +            for line in fd:
> > +                line = line.strip()
> > +                fields = re.split('[ \t]', line, 1)
> > +                print fields
> > +                if len(fields) == 2:
> > +                    if fields[0] == 'M:':
> > +                        cc.append(fields[1])
> > +                    elif fields[0] == 'F:':
> > +                        paths.append(fields[1].strip())
> > +    print 'paths', paths
> > +
> > +    # Expannd any wildcards in the MAINTAINRERS file
> > +    real = []
> > +    for path in paths:
> > +        if path[-1] == '/':
> > +            path = path[:-1]
> > +        if '*' in path:
> > +            globbed = glob.glob(path)
> > +            print "Expanded '%s' to '%s'" % (path, globbed)
> > +            real += globbed
> > +        else:
> > +            real.append(path)
> > +    print 'real', real
> > +
> > +    # Search for Kconfig files in the resulting list. Remove any 'source' lines
> > +    # which referenced Kconfig files we want to remove
> > +    for path in real:
> > +        cmd = ['find', path]
> > +        stdout = (command.RunPipe([cmd], capture=True, raise_on_error=False).
> > +                  stdout)
> > +        for fname in stdout.splitlines():
> > +            if fname.endswith('Kconfig'):
> > +                rm_kconfig_include(fname)
> > +
> > +    # Remove unwanted files
> > +    cmd = ['git', 'rm', '-r'] + real
> > +    stdout = command.RunPipe([cmd], capture=True).stdout
> > +
> > +    # Change the messages as needed
> > +    msg = '''arm: Remove %s board
> > +
> > +This board has not been converted to CONFIG_DM_BLK by the deadline.
> > +Remove it.
> > +
> > +''' % board
> > +    for name in cc:
> > +        msg += 'Patch-cc: %s\n' % name
> > +
> > +    # Create the commit
> > +    cmd = ['git', 'commit', '-s', '-m', msg]
> > +    stdout = command.RunPipe([cmd], capture=True).stdout
> > +    print kconfig
> > +
> > +    # Check if the board is mentioned anywhere else. The user will need to deal
> > +    # with this
> > +    cmd = ['git', 'grep', '-il', board]
> > +    print command.RunPipe([cmd], capture=True, raise_on_error=False).stdout
> > +    print ' '.join(cmd)
> > +
> > +for board in sys.argv[1:]:
> > +    rm_board(board)
> > --
> > 2.19.1.1215.g8438c0b245-goog
> >

Hi Adam,

On Mon, 19 Nov 2018 at 11:19, Adam Ford <aford173@gmail.com> wrote:
>
> On Mon, Nov 19, 2018 at 9:54 AM Simon Glass <sjg@chromium.org> wrote:
> >
> > This script attempts to create a git commit which removes a single board.
> > It is quite fallible and everything it does needs checking. But it can
> > help speed up the process.
> >
>
> This patch looks like it goes through the configs, finds boards and
> blindly deletes them.  What happens in the instance where multiple
> defconfig files use the same include/configs file or C source?  If one
> of the defconfigs is missing something, does it then delete the
> others?  It seems like it should make sure that all variations don't
> comply before blowing away the source and include/configs.

Yes that is one of the failure modes :-) I am not sure how to do better. Regards

Regards,
Simon