From patchwork Mon Jan 19 12:12:22 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 430422 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from theia.denx.de (theia.denx.de [85.214.87.163]) by ozlabs.org (Postfix) with ESMTP id 33A64140119 for ; Mon, 19 Jan 2015 23:12:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id 45FF44B716; Mon, 19 Jan 2015 13:12:39 +0100 (CET) Received: from theia.denx.de ([127.0.0.1]) by localhost (theia.denx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nIawVywqK0+x; Mon, 19 Jan 2015 13:12:39 +0100 (CET) Received: from theia.denx.de (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id E7E034B725; Mon, 19 Jan 2015 13:12:38 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id 881B14B725 for ; Mon, 19 Jan 2015 13:12:34 +0100 (CET) Received: from theia.denx.de ([127.0.0.1]) by localhost (theia.denx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D7SY6r7L-5wn for ; Mon, 19 Jan 2015 13:12:34 +0100 (CET) X-policyd-weight: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 (only DNSBL check requested) Received: from smtp.mei.co.jp (smtp.mei.co.jp [133.183.100.20]) by theia.denx.de (Postfix) with ESMTP id 8C7DC4B716 for ; Mon, 19 Jan 2015 13:12:30 +0100 (CET) Received: from mail-gw.jp.panasonic.com ([157.8.1.157]) by smtp.mei.co.jp (8.12.11.20060614/3.7W/kc-maile13) with ESMTP id t0JCCQE8007531; Mon, 19 Jan 2015 21:12:26 +0900 (JST) Received: from epochmail.jp.panasonic.com ([157.8.1.130]) by mail.jp.panasonic.com (8.11.6p2/3.7W/kc-maili14) with ESMTP id t0JCCPL04317; Mon, 19 Jan 2015 21:12:25 +0900 Received: by epochmail.jp.panasonic.com (8.12.11.20060308/3.7W/lomi14) id t0JCCPHM020282; Mon, 19 Jan 2015 21:12:25 +0900 Received: from poodle by lomi14.jp.panasonic.com (8.12.11.20060308/3.7W) with ESMTP id t0JCCPfC020266; Mon, 19 Jan 2015 21:12:25 +0900 Received: from beagle.diag.org (beagle.diag.org [10.184.179.16]) by poodle (Postfix) with ESMTP id 5BFD72743A5C; Mon, 19 Jan 2015 21:12:25 +0900 (JST) From: Masahiro Yamada To: u-boot@lists.denx.de Date: Mon, 19 Jan 2015 21:12:22 +0900 Message-Id: <1421669542-7534-1-git-send-email-yamada.m@jp.panasonic.com> X-Mailer: git-send-email 1.9.1 Cc: Alexey Brodkin Subject: [U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.13 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: u-boot-bounces@lists.denx.de Errors-To: u-boot-bounces@lists.denx.de This tool can move CONFIG macros from C headers (include/configs/*.h) to defconfigs (configs/*_defconfig) all over the boards. There are tons of CONFIGs in U-boot and moving them by hand is absolutely tool painful. I wrote this script for my local use, so this patch might not clean enough to be applied to the code base. It might contain some bugs. Please do not apply this patch! This patch is here because Simon requested me to share it. See the discussion in here: http://lists.denx.de/pipermail/u-boot/2015-January/201556.html Usage: [1] Please run "git status" and make sure your git repository is clean. [2] Describe the CONFIG option you want move in the file "~/.moveconfig" Say, you want to move CONFIG_CMD_NAND, for example. In this case, the content of "~/.moveconfig" should be like this: $ cat .moveconfig CONFIG_CMD_NAND bool n y - The first field is the name of the CONFIG. - The second field is the type of the CONFIG such as "bool", "int", "hex", etc. - The third value is the default value. The value that is same as the default is omitted in each defconfig. If the type of the CONFIG is bool, the default value must be either 'y' or 'n'. - The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD. For example, CONFIG_CMD_* makes sense only on the main-image. (depends on !SPL_BUILD) In this case, specify 'y' in the forth fields. If the CONFIG should appear also on SPL, specify 'n' here. [4] Run "tools/moveconfig.py" The log will be displayed like this $ tools/moveconfig.py Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ... (jobs: 8) ms7750se : (default) davinci_sonata : (default) tk71 : y db-mv784mp-gp : (default) cm_t3517 : y P2041RDB_SDCARD : y ... The left-hand side field is the board[defconfig] name and the colon is followed by the value of the CONFIG. At the last stage, you will be asked like this: Clean up CONFIG_CMD_NAND in headers? [y/n]: If you say 'y' here, the defines in C headers will be removed. Enjoy! Signed-off-by: Masahiro Yamada Cc: Simon Glass Cc: Alexey Brodkin --- tools/moveconfig.py | 439 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100755 tools/moveconfig.py diff --git a/tools/moveconfig.py b/tools/moveconfig.py new file mode 100755 index 0000000..cb7f7ea --- /dev/null +++ b/tools/moveconfig.py @@ -0,0 +1,439 @@ +#!/usr/bin/env python +# +# Author: Masahiro Yamada +# +# SPDX-License-Identifier: GPL-2.0+ +# + +""" +Move the specified config option to Kconfig +""" + +import errno +import fnmatch +import multiprocessing +import optparse +import os +import re +import shutil +import subprocess +import sys +import tempfile +import time + +SHOW_GNU_MAKE = 'scripts/show-gnu-make' +SLEEP_TIME=0.03 + +CROSS_COMPILE = { + 'aarch64': 'aarch64-linux-', + 'arm': 'arm-unknown-linux-gnueabi-', + 'avr32': 'avr32-linux-', + 'blackfin': 'bfin-elf-', + 'm68k': 'm68k-linux-', + 'microblaze': 'microblaze-linux-', + 'mips': 'mips-linux-', + 'nds32': 'nds32le-linux-', + 'nios2': 'nios2-linux-', + 'openrisc': 'or32-linux-', + 'powerpc': 'powerpc-linux-', + 'sh': 'sh4-gentoo-linux-gnu-', + 'sparc': 'sparc-linux-', + 'x86': 'i386-linux-' +} + +STATE_IDLE = 0 +STATE_DEFCONFIG = 1 +STATE_SILENTOLDCONFIG = 2 + +### helper functions ### +def get_devnull(): + """Get the file object of '/dev/null' device.""" + try: + devnull = subprocess.DEVNULL # py3k + except AttributeError: + devnull = open(os.devnull, 'wb') + return devnull + +def check_top_directory(): + """Exit if we are not at the top of source directory.""" + for f in ('README', 'Licenses'): + if not os.path.exists(f): + sys.exit('Please run at the top of source directory.') + +def get_make_cmd(): + """Get the command name of GNU Make.""" + process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) + ret = process.communicate() + if process.returncode: + sys.exit('GNU Make not found') + return ret[0].rstrip() + +def cleanup_header(header_path, patterns): + with open(header_path) as f: + lines = f.readlines() + + matched = [] + for i, line in enumerate(lines): + for pattern in patterns: + m = pattern.search(line) + if m: + print '%s: %s: %s' % (header_path, i + 1, line), + matched.append(i) + break + + if not matched: + return + + with open(header_path, 'w') as f: + for i, line in enumerate(lines): + if not i in matched: + f.write(line) + +def cleanup_headers(config): + while True: + choice = raw_input('Clean up %s in headers? [y/n]: ' % config).lower() + print choice + if choice == 'y' or choice == 'n': + break + + if choice != 'y': + return + + patterns = [] + patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) + patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) + + for (dirpath, dirnames, filenames) in os.walk('include'): + for filename in filenames: + if not fnmatch.fnmatch(filename, '*~'): + cleanup_header(os.path.join(dirpath, filename), patterns) + +### classes ### +class KconfigParser: + + """A parser of .config file. + + Each line of the output should have the form of: + Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers + Most of them are extracted from .config file. + MAINTAINERS files are also consulted for Status and Maintainers fields. + """ + + re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') + re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') + PREFIX = { '.': '+', 'spl': 'S', 'tpl': 'T' } + + def __init__(self, build_dir, config_attr): + """Create a new .config perser. + + Arguments: + build_dir: Build directory where .config is located + output: File object which the result is written to + """ + self.build_dir = build_dir + self.config = config_attr['config'] + self.type = config_attr['type'] + self.default = config_attr['default'] + if config_attr['no_spl_support'] == 'y': + self.no_spl_support = True + else: + self.no_spl_support = False + self.re = re.compile(r'%s=(.*)' % self.config) + + def get_cross_compile(self): + """Parse .config file and return CROSS_COMPILE + + Arguments: + defconfig: Board (defconfig) name + """ + arch = '' + cpu = '' + dotconfig = os.path.join(self.build_dir, '.config') + for line in open(dotconfig): + m = self.re_arch.match(line) + if m: + arch = m.group(1) + continue + m = self.re_cpu.match(line) + if m: + cpu = m.group(1) + + assert arch, 'Error: arch is not defined in %s' % defconfig + + # fix-up for aarch64 + if arch == 'arm' and cpu == 'armv8': + arch = 'aarch64' + + return CROSS_COMPILE.get(arch, '') + + def update_defconfig(self, defconfig): + values = [] + output_lines = [] + prefixes = {} + + if self.no_spl_support: + images = ('.') + else: + images = ('.', 'spl', 'tpl') + + + for img in images: + autoconf = os.path.join(self.build_dir, img, 'include', + 'autoconf.mk') + + if not os.path.exists(autoconf): + values.append('') + continue + + if self.type == 'bool': + value = 'n' + else: + value = '(undef)' + + for line in open(autoconf): + m = self.re.match(line) + if m: + value = m.group(1) + break + + if value == self.default: + value = '(default)' + + values.append(value) + os.remove(autoconf) + + if value == '(undef)' or value == '(default)': + continue + + if self.type == 'bool' and value == 'n': + output_line = '# %s is not set' % (self.config) + else: + output_line = '%s=%s' % (self.config, value) + if output_line in output_lines: + prefixes[output_line] += self.PREFIX[img] + else: + output_lines.append(output_line) + prefixes[output_line] = self.PREFIX[img] + + output = defconfig[:-len('_defconfig')].ljust(37) + ': ' + for value in values: + output += value.ljust(12) + + print output.strip() + + with open(os.path.join('configs', defconfig), 'a') as f: + for line in output_lines: + if prefixes[line] != '+': + line = prefixes[line] + ':' + line + f.write(line + '\n') + +class Slot: + + """A slot to store a subprocess. + + Each instance of this class handles one subprocess. + This class is useful to control multiple processes + for faster processing. + """ + + def __init__(self, config_attr, devnull, make_cmd): + """Create a new slot. + + Arguments: + output: File object which the result is written to + """ + self.build_dir = tempfile.mkdtemp() + self.devnull = devnull + self.make_cmd = (make_cmd, 'O=' + self.build_dir) + self.parser = KconfigParser(self.build_dir, config_attr) + self.state = STATE_IDLE + + def __del__(self): + """Delete the working directory""" + if self.state != STATE_IDLE: + while self.ps.poll() == None: + pass + shutil.rmtree(self.build_dir) + + def add(self, defconfig): + """Add a new subprocess to the slot. + + Fails if the slot is occupied, that is, the current subprocess + is still running. + + Arguments: + defconfig: Board (defconfig) name + + Returns: + Return True on success or False on fail + """ + if self.state != STATE_IDLE: + return False + cmd = list(self.make_cmd) + cmd.append(defconfig) + self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.defconfig = defconfig + self.state = STATE_DEFCONFIG + return True + + def poll(self): + """Check if the subprocess is running and invoke the .config + parser if the subprocess is terminated. + + Returns: + Return True if the subprocess is terminated, False otherwise + """ + if self.state == STATE_IDLE: + return True + + if self.ps.poll() == None: + return False + + if self.ps.poll() != 0: + sys.exit("failed to process '%s'" % self.defconfig) + + if self.state == STATE_SILENTOLDCONFIG: + self.parser.update_defconfig(self.defconfig) + self.state = STATE_IDLE + return True + + cross_compile = self.parser.get_cross_compile() + cmd = list(self.make_cmd) + if cross_compile: + cmd.append('CROSS_COMPILE=%s' % cross_compile) + cmd.append('silentoldconfig') + self.ps = subprocess.Popen(cmd, stdout=self.devnull) + self.state = STATE_SILENTOLDCONFIG + return False + +class Slots: + + """Controller of the array of subprocess slots.""" + + def __init__(self, config_attr, jobs): + """Create a new slots controller. + + Arguments: + jobs: A number of slots to instantiate + """ + self.slots = [] + devnull = get_devnull() + make_cmd = get_make_cmd() + for i in range(jobs): + self.slots.append(Slot(config_attr, devnull, make_cmd)) + + def add(self, defconfig): + """Add a new subprocess if a vacant slot is available. + + Arguments: + defconfig: Board (defconfig) name + + Returns: + Return True on success or False on fail + """ + for slot in self.slots: + if slot.add(defconfig): + return True + return False + + def available(self): + """Check if there is a vacant slot. + + Returns: + Return True if a vacant slot is found, False if all slots are full + """ + for slot in self.slots: + if slot.poll(): + return True + return False + + def empty(self): + """Check if all slots are vacant. + + Returns: + Return True if all slots are vacant, False if at least one slot + is running + """ + ret = True + for slot in self.slots: + if not slot.poll(): + ret = False + return ret + +def move_config(config_attr, jobs=1): + check_top_directory() + print 'Moving %s (type: %s, default: %s, no_spl: %s) ... (jobs: %d)' % ( + config_attr['config'], + config_attr['type'], + config_attr['default'], + config_attr['no_spl_support'], + jobs) + + # All the defconfig files to be processed + defconfigs = [] + for (dirpath, dirnames, filenames) in os.walk('configs'): + dirpath = dirpath[len('configs') + 1:] + for filename in fnmatch.filter(filenames, '*_defconfig'): + if fnmatch.fnmatch(filename, '.*'): + continue + defconfigs.append(os.path.join(dirpath, filename)) + + slots = Slots(config_attr, jobs) + + # Main loop to process defconfig files: + # Add a new subprocess into a vacant slot. + # Sleep if there is no available slot. + for defconfig in defconfigs: + while not slots.add(defconfig): + while not slots.available(): + # No available slot: sleep for a while + time.sleep(SLEEP_TIME) + + # wait until all the subprocesses finish + while not slots.empty(): + time.sleep(SLEEP_TIME) + + cleanup_headers(config_attr['config']) + +def main(): + try: + cpu_count = multiprocessing.cpu_count() + except NotImplementedError: + cpu_count = 1 + + parser = optparse.OptionParser() + # Add options here + parser.add_option('-j', '--jobs', type='int', default=cpu_count, + help='the number of jobs to run simultaneously') + parser.usage += ' config type default no_spl_support' + (options, args) = parser.parse_args() + + args_key = ('config', 'type', 'default', 'no_spl_support') + config_attr = {} + + if len(args) >= len(args_key): + saved_attr = '' + for i, key in enumerate(args_key): + config_attr[key] = args[i] + saved_attr = saved_attr + ' %s' % args[i] + with open('.moveconfig', 'w') as f: + f.write(saved_attr) + elif os.path.exists('.moveconfig'): + f = open('.moveconfig') + try: + saved_attr = f.readline().split() + for i, key in enumerate(args_key): + config_attr[key] = saved_attr[i] + except: + sys.exit('%s: broken format' % '.moveconfig') + else: + parser.print_usage() + sys.exit(1) + + if not config_attr['config'].startswith('CONFIG_'): + config_attr['config'] = 'CONFIG_' + config_attr['config'] + + move_config(config_attr, options.jobs) + +if __name__ == '__main__': + main()