From patchwork Fri Jun 2 01:39:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 770031 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.denx.de (dione.denx.de [81.169.180.215]) by ozlabs.org (Postfix) with ESMTP id 3wf6RG18Rcz9s72 for ; Fri, 2 Jun 2017 11:42:50 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="i3gNr3c+"; dkim-atps=neutral Received: by lists.denx.de (Postfix, from userid 105) id E3C51C21C72; Fri, 2 Jun 2017 01:40:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lists.denx.de X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.denx.de (localhost [IPv6:::1]) by lists.denx.de (Postfix) with ESMTP id 86629C21C61; Fri, 2 Jun 2017 01:39:30 +0000 (UTC) Received: by lists.denx.de (Postfix, from userid 105) id A7B52C21C72; Fri, 2 Jun 2017 01:39:20 +0000 (UTC) Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com [209.85.218.47]) by lists.denx.de (Postfix) with ESMTPS id 58A1CC21C5B for ; Fri, 2 Jun 2017 01:39:16 +0000 (UTC) Received: by mail-oi0-f47.google.com with SMTP id l18so75308536oig.2 for ; Thu, 01 Jun 2017 18:39:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=xwKu2JmAbTjpsSeQGVJPg+ZpyWZos+KKlA9x2zUfD50=; b=i3gNr3c+GR80osChOle9E6lGav2NDwPRaSGQ4ZDrZMmMJ/3YnysOf5iV6SSxJ9R0IH BHsGbFKW5jwa2VKNhpdo5GzoDLewBMYnphg9szurNvpffbdp+eAvCiZXp7NxSkAZRujg kEsW14+HmmRTM6T+6cEOrgFmQG7QS6p8lsc7M/wBj7TaZbQKGcwJmR3eVye9P9JMIeRj BJKY0tJaFBEMMmFoVIzE9OnW7oOBaH0aNtn0Sp0jBIEMujT0nBjEbMfdY+JPH+/OC3Ak P19+Qu0wiuQ2id8wEXPLE36TV7dRwIaejhZCmXeS1iqhNvhYT6eDGk6tgZKNtGEeVy96 rVlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=xwKu2JmAbTjpsSeQGVJPg+ZpyWZos+KKlA9x2zUfD50=; b=C7SIH3dvUuDO75R2cSI5kpFm2Av/t7ul056Y1t/pjdqkXlaoaNGuDBU9RguLY6ipE/ IbofnMSAiW+rCFsrjl1iYmRGhaMYE+VPgnqNBkNBquvEMbHs1uip/XQqzhzrYmmEL2eH 0akhC9HEBGdzYjgaNpnNblfMszzisbzFUmwOwjukr/Tsw7S6iTOLUK6NMyfA0ijpSqTA wGAh8Gc/huH9NCz6MF9gwMMNYI4KEme8Rg5ib2dpBD5DZopT1Zx1Qw964lbhKzs/vhCq NhLu4HQ5h5v7LdpdhKHg4K4KCcF88AFTgG7+CeiCwJCF6tIu980IfW3AuM03zuuYE7NV c/Ag== X-Gm-Message-State: AODbwcANLB25MxIKZp9oef+5K/GeefToHIgL0K0/DZVkFd4N90IYz/2F R11jgkfvdwml8ejp7K6Yqw== X-Received: by 10.202.168.140 with SMTP id r134mr2624294oie.57.1496367554965; Thu, 01 Jun 2017 18:39:14 -0700 (PDT) Received: from kaki.bld.corp.google.com ([2620:0:10f2:12:9dd6:90dc:9def:746]) by smtp.gmail.com with ESMTPSA id o70sm284309oik.30.2017.06.01.18.39.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Jun 2017 18:39:14 -0700 (PDT) Received: by kaki.bld.corp.google.com (Postfix, from userid 121222) id 9D7B64013C; Thu, 1 Jun 2017 19:39:13 -0600 (MDT) From: Simon Glass To: U-Boot Mailing List Date: Thu, 1 Jun 2017 19:39:03 -0600 Message-Id: <20170602013903.28731-7-sjg@chromium.org> X-Mailer: git-send-email 2.13.0.506.g27d5fe0cd-goog In-Reply-To: <20170602013903.28731-1-sjg@chromium.org> References: <20170602013903.28731-1-sjg@chromium.org> Cc: Joe Hershberger , Tom Rini , Chris Packham Subject: [U-Boot] [PATCH v2 6/6] moveconfig: Support looking for implied CONFIG options X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.18 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" Some CONFIG options can be implied by others and this can help to reduce the size of the defconfig files. For example, CONFIG_X86 implies CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to each of the x86 defconfig files. Add a -i option which searches for such options. Signed-off-by: Simon Glass Reviewed-by: Heiko Schocher Signed-off-by: Simon Glass Reviewed-by: Heiko Schocher --- Changes in v2: - Update cover letter to express future hopes tools/moveconfig.py | 215 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 214 insertions(+), 1 deletion(-) diff --git a/tools/moveconfig.py b/tools/moveconfig.py index fab588f239..09c5a7e5cd 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -132,6 +132,69 @@ To process CONFIG_CMD_FPGAD only for a subset of configs based on path match: ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d - +Finding implied CONFIGs +----------------------- + +Some CONFIG options can be implied by others and this can help to reduce +the size of the defconfig files. For example, CONFIG_X86 implies +CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and +all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to +each of the x86 defconfig files. + +This tool can help find such configs. To use it, first build a database: + + ./tools/moveconfig.py -b + +Then try to query it: + + ./tools/moveconfig.py -i CONFIG_CMD_IRQ + CONFIG_CMD_IRQ found in 311/2384 defconfigs + 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769 + 41 : CONFIG_SYS_FSL_ERRATUM_A007075 + 31 : CONFIG_SYS_FSL_DDR_VER_44 + 28 : CONFIG_ARCH_P1010 + 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549 + 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571 + 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399 + 25 : CONFIG_SYS_FSL_ERRATUM_A008044 + 22 : CONFIG_ARCH_P1020 + 21 : CONFIG_SYS_FSL_DDR_VER_46 + 20 : CONFIG_MAX_PIRQ_LINKS + 20 : CONFIG_HPET_ADDRESS + 20 : CONFIG_X86 + 20 : CONFIG_PCIE_ECAM_SIZE + 20 : CONFIG_IRQ_SLOT_COUNT + 20 : CONFIG_I8259_PIC + 20 : CONFIG_CPU_ADDR_BITS + 20 : CONFIG_RAMBASE + 20 : CONFIG_SYS_FSL_ERRATUM_A005871 + 20 : CONFIG_PCIE_ECAM_BASE + 20 : CONFIG_X86_TSC_TIMER + 20 : CONFIG_I8254_TIMER + 20 : CONFIG_CMD_GETTIME + 19 : CONFIG_SYS_FSL_ERRATUM_A005812 + 18 : CONFIG_X86_RUN_32BIT + 17 : CONFIG_CMD_CHIP_CONFIG + ... + +This shows a list of config options which might imply CONFIG_CMD_EEPROM along +with how many defconfigs they cover. From this you can see that CONFIG_X86 +implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to +the defconfig of every x86 board, you could add a single imply line to the +Kconfig file: + + config X86 + bool "x86 architecture" + ... + imply CMD_EEPROM + +That will cover 20 defconfigs. Many of the options listed are not suitable as +they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply +CMD_EEPROM. + +Using this search you can reduce the size of moveconfig patches. + + Available options ----------------- @@ -195,6 +258,7 @@ To see the complete list of supported options, run """ +import collections import copy import difflib import filecmp @@ -1400,6 +1464,148 @@ def move_config(configs, options, db_queue): slots.show_failed_boards() slots.show_suspicious_boards() +def imply_config(config_list, find_superset=False): + """Find CONFIG options which imply those in the list + + Some CONFIG options can be implied by others and this can help to reduce + the size of the defconfig files. For example, CONFIG_X86 implies + CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and + all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to + each of the x86 defconfig files. + + This function uses the moveconfig database to find such options. It + displays a list of things that could possibly imply those in the list. + The algorithm ignores any that start with CONFIG_TARGET since these + typically refer to only a few defconfigs (often one). It also does not + display a config with less than 5 defconfigs. + + The algorithm works using sets. For each target config in config_list: + - Get the set 'defconfigs' which use that target config + - For each config (from a list of all configs): + - Get the set 'imply_defconfig' of defconfigs which use that config + - + - If imply_defconfigs contains anything not in defconfigs then + this config does not imply the target config + + Params: + config_list: List of CONFIG options to check (each a string) + find_superset: True to look for configs which are a superset of those + already found. So for example if CONFIG_EXYNOS5 implies an option, + but CONFIG_EXYNOS covers a larger set of defconfigs and also + implies that option, this will drop the former in favour of the + latter. In practice this option has not proved very used. + + Note the terminoloy: + config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM') + defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig') + """ + # key is defconfig name, value is dict of (CONFIG_xxx, value) + config_db = {} + + # Holds a dict containing the set of defconfigs that contain each config + # key is config, value is set of defconfigs using that config + defconfig_db = collections.defaultdict(set) + + # Set of all config options we have seen + all_configs = set() + + # Set of all defconfigs we have seen + all_defconfigs = set() + + # Read in the database + configs = {} + with open(CONFIG_DATABASE) as fd: + for line in fd.readlines(): + line = line.rstrip() + if not line: # Separator between defconfigs + config_db[defconfig] = configs + all_defconfigs.add(defconfig) + configs = {} + elif line[0] == ' ': # CONFIG line + config, value = line.strip().split('=', 1) + configs[config] = value + defconfig_db[config].add(defconfig) + all_configs.add(config) + else: # New defconfig + defconfig = line + + # Work through each target config option in tern, independently + for config in config_list: + defconfigs = defconfig_db.get(config) + if not defconfigs: + print '%s not found in any defconfig' % config + continue + + # Get the set of defconfigs without this one (since a config cannot + # imply itself) + non_defconfigs = all_defconfigs - defconfigs + num_defconfigs = len(defconfigs) + print '%s found in %d/%d defconfigs' % (config, num_defconfigs, + len(all_configs)) + + # This will hold the results: key=config, value=defconfigs containing it + imply_configs = {} + rest_configs = all_configs - set([config]) + + # Look at every possible config, except the target one + for imply_config in rest_configs: + if 'CONFIG_TARGET' in imply_config: + continue + + # Find set of defconfigs that have this config + imply_defconfig = defconfig_db[imply_config] + + # Get the intersection of this with defconfigs containing the + # target config + common_defconfigs = imply_defconfig & defconfigs + + # Get the set of defconfigs containing this config which DO NOT + # also contain the taret config. If this set is non-empty it means + # that this config affects other defconfigs as well as (possibly) + # the ones affected by the target config. This means it implies + # things we don't want to imply. + not_common_defconfigs = imply_defconfig & non_defconfigs + if not_common_defconfigs: + continue + + # If there are common defconfigs, imply_config may be useful + if common_defconfigs: + skip = False + if find_superset: + for prev in imply_configs.keys(): + prev_count = len(imply_configs[prev]) + count = len(common_defconfigs) + if (prev_count > count and + (imply_configs[prev] & common_defconfigs == + common_defconfigs)): + # skip imply_config because prev is a superset + skip = True + break + elif count > prev_count: + # delete prev because imply_config is a superset + del imply_configs[prev] + if not skip: + imply_configs[imply_config] = common_defconfigs + + # Now we have a dict imply_configs of configs which imply each config + # The value of each dict item is the set of defconfigs containing that + # config. Rank them so that we print the configs that imply the largest + # number of defconfigs first. + ranked_configs = sorted(imply_configs, + key=lambda k: len(imply_configs[k]), reverse=True) + for config in ranked_configs: + num_common = len(imply_configs[config]) + + # Don't bother if there are less than 5 defconfigs affected. + if num_common < 5: + continue + missing = defconfigs - imply_configs[config] + missing_str = ', '.join(missing) if missing else 'all' + missing_str = '' + print ' %d : %-30s%s' % (num_common, config.ljust(30), + missing_str) + + def main(): try: cpu_count = multiprocessing.cpu_count() @@ -1418,6 +1624,8 @@ def main(): help='a file containing a list of defconfigs to move, ' "one per line (for example 'snow_defconfig') " "or '-' to read from stdin") + parser.add_option('-i', '--imply', action='store_true', default=False, + help='find options which imply others') parser.add_option('-n', '--dry-run', action='store_true', default=False, help='perform a trial run (show log with no changes)') parser.add_option('-e', '--exit-on-error', action='store_true', @@ -1442,7 +1650,8 @@ def main(): (options, configs) = parser.parse_args() - if len(configs) == 0 and not any((options.force_sync, options.build_db)): + if len(configs) == 0 and not any((options.force_sync, options.build_db, + options.imply)): parser.print_usage() sys.exit(1) @@ -1452,6 +1661,10 @@ def main(): check_top_directory() + if options.imply: + imply_config(configs) + return + config_db = {} db_queue = Queue.Queue() t = DatabaseThread(config_db, db_queue)