From patchwork Fri May 25 04:57:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Kiernan X-Patchwork-Id: 920240 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.denx.de (client-ip=81.169.180.215; helo=lists.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="GvjwOt8m"; dkim-atps=neutral Received: from lists.denx.de (dione.denx.de [81.169.180.215]) by ozlabs.org (Postfix) with ESMTP id 40sYsw5Kfwz9s15 for ; Fri, 25 May 2018 14:58:12 +1000 (AEST) Received: by lists.denx.de (Postfix, from userid 105) id ECB3BC21E08; Fri, 25 May 2018 04:57:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lists.denx.de X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED, T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.denx.de (localhost [IPv6:::1]) by lists.denx.de (Postfix) with ESMTP id D16A1C21DB3; Fri, 25 May 2018 04:57:15 +0000 (UTC) Received: by lists.denx.de (Postfix, from userid 105) id F0E29C21BE5; Fri, 25 May 2018 04:57:12 +0000 (UTC) Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by lists.denx.de (Postfix) with ESMTPS id 22CD9C21BE5 for ; Fri, 25 May 2018 04:57:10 +0000 (UTC) Received: by mail-wr0-f195.google.com with SMTP id l41-v6so6863475wre.7 for ; Thu, 24 May 2018 21:57:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=selQTtAq26iYrRw0mRe1FewaQZ87cgwA9CBZbMqEdBM=; b=GvjwOt8m7CoJhfTBzGe6zuj85hGNvyLfou+YTbEB188RDwVBOaCeVEBapE/ijMIbvU Ayy3mJY3pfgpSec4FV5ovnos6/Cu3VXJS01Z+o8S8i3Q+jUoxC9uVz2zkelRBmh+/jhj QBCZWJTp00vR/Gkr3WLti/eIdO4wlPtlk2agemXcjUQ3f9h8MTIQFD4xG2KoCjlpt1Nv acQ99K0B0VoZDWh8pCM4EOTa9MuF2fNZ597U45mdjjXhBfRTqlLN9OHma3AxSEz1xMDs X4meWG7P//qXsbcil+k/X9o13YSnqqJD3oG427uBNw+79bbjyna9teBez0/opWZTzp4E QSdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=selQTtAq26iYrRw0mRe1FewaQZ87cgwA9CBZbMqEdBM=; b=BtZtD1HNJnzN3RMUZqAR696PV5kz7TN301CJaazLHBf0WpmkC7F66h83k6Ja2yxyki GjAPp1Sq3FhEIH7QeybAyvAEkZmwt0LO4O1TQptWJDMTsocnpRoSRTwQs5IS8y7zI469 n9XohdKH9n3OS68aokKxD8o805x2/AdnSY7zl2k7dKBG46vohFzMH33upzeLj2aWJO/p pCRxe+LQAatjUL3xXV8YrxAxlg3hVGxyNQ7vR0lMeI3/pzocZTUMGPZC1egqQwVfcBS7 SI73IpPGwmA3IA6qheVc8AaIQncgdHg1NQOIRKNzgmZLboT8HUHFjPBbaipSKA0S+QLl sQyA== X-Gm-Message-State: ALKqPwejog1dmfla5VL6pOho+ea7u2fp8Actb43zRCPbZ+skdYtGoSaf PMOP7zDDW7k1kzcxLPtx8H/B1IFYEvM= X-Google-Smtp-Source: AB8JxZqRGnPFCArRR4oxUe3XnTEQsuEE4NTiVQZWO5iG7DrAq9kUF0zdikPUwE5h4lvMGLS+oEiAMw== X-Received: by 2002:adf:988c:: with SMTP id w12-v6mr570779wrb.215.1527224229573; Thu, 24 May 2018 21:57:09 -0700 (PDT) Received: from localhost.localdomain (cust246-dsl91-135-6.idnet.net. [91.135.6.246]) by smtp.gmail.com with ESMTPSA id k82-v6sm5818111wmf.17.2018.05.24.21.57.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 21:57:09 -0700 (PDT) From: Alex Kiernan To: u-boot@lists.denx.de Date: Fri, 25 May 2018 04:57:03 +0000 Message-Id: <20180525045704.21658-1-alex.kiernan@gmail.com> X-Mailer: git-send-email 2.17.0 Subject: [U-Boot] [PATCH v1 1/2] buildman: Extract environment as part of each build X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.18 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" As we're building the boards, extract the default U-Boot environment to uboot.env so we can interrogate it later. Signed-off-by: Alex Kiernan --- tools/buildman/builderthread.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index 0efe80d945..688322ced2 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -351,6 +351,16 @@ class BuilderThread(threading.Thread): lines.append(size_result.stdout.splitlines()[1] + ' ' + rodata_size) + # Extract the environment from U-Boot and dump it out + cmd = ['%sobjcopy' % self.toolchain.cross, '-O', 'binary', + '-j', '.rodata.default_environment', + 'env/built-in.o', 'uboot.env'] + command.RunPipe([cmd], capture=False, + capture_stderr=False, cwd=result.out_dir, + raise_on_error=False, env=env) + ubootenv = os.path.join(result.out_dir, 'uboot.env') + self.CopyFiles(result.out_dir, build_dir, '', ['uboot.env']) + # Write out the image sizes file. This is similar to the output # of binutil's 'size' utility, but it omits the header line and # adds an additional hex value at the end of each line for the From patchwork Fri May 25 04:57:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Kiernan X-Patchwork-Id: 920239 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.denx.de (client-ip=81.169.180.215; helo=lists.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FqY6mKOt"; dkim-atps=neutral Received: from lists.denx.de (dione.denx.de [81.169.180.215]) by ozlabs.org (Postfix) with ESMTP id 40sYs16h2qz9s15 for ; Fri, 25 May 2018 14:57:24 +1000 (AEST) Received: by lists.denx.de (Postfix, from userid 105) id 1C9E7C21DFA; Fri, 25 May 2018 04:57:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lists.denx.de X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED, RCVD_IN_MSPIKE_H2, T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.denx.de (localhost [IPv6:::1]) by lists.denx.de (Postfix) with ESMTP id 125D4C21C27; Fri, 25 May 2018 04:57:15 +0000 (UTC) Received: by lists.denx.de (Postfix, from userid 105) id E6B11C21C2C; Fri, 25 May 2018 04:57:12 +0000 (UTC) Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by lists.denx.de (Postfix) with ESMTPS id 53C46C21C27 for ; Fri, 25 May 2018 04:57:11 +0000 (UTC) Received: by mail-wr0-f194.google.com with SMTP id l41-v6so6863523wre.7 for ; Thu, 24 May 2018 21:57:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+j2ClQrsIrHVl5InHZSG8jtdnpuCMw+khC8HKG+SM3M=; b=FqY6mKOtudhRugfNUQpwOj89dghEuBJJbBMpgA0MbeX3lW0fc7N2zQfe7Ez5r8U/82 pQWsg+IOw6Bs8HWGCTdWfQ/hhTvyOKcI1nVBOOHf9pUC+hUd8PUYY3KY6tabfY+JseBE asCIMg9oor5wzRd/I1gSSpvBfrkP1VO45BAlOV8lk2IfECXfalH90967dINTIR5Adn8D DaLLg9uW7yqfjNICtYoPLPzqpHZobcDigYV86pDVUFGD62kveX1ZxJKoo0yK9E9/ot0l S08og6bsprv4ly9u92NrUOjReeX1HLQ86P2keX43VolJrNQ9rewb8LrG4m13Bfwn39rr Xq5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+j2ClQrsIrHVl5InHZSG8jtdnpuCMw+khC8HKG+SM3M=; b=ch5/+R344Q4eganFbN1wc8mH9ECDVoeC9zI++lVClOt7HZ+WMNOm3i9lJ0vuInWFww OeIJ4cqUI1Ei981dKLaYYwMzyBL7ufZzGz2cWfyz+aB8I5jK5s5A/QtNCx86c91L77WY jdVg9yGiOcfMbK/dLxCr7yCiNRMWoY0YAiQH7fXnW049I68J4fkuPLCoPh6PKOkTuBte zvKxyqM0AoL+tHMR0q/VzKI+FzzrAR3oHVrtZHK9ZIQZOYyzI9TntN4eToa/M9YN95I8 1noResdQRZq7QSK/AfYIU0gqRNEnCgkcmMRkuXax1X5SwVxtXd0LYfJcae+FkTqBtkEE slWg== X-Gm-Message-State: ALKqPweuVJNkdgSXxOzmWaQbTMtN2tQzZwBxx6rYnB4fKhfoPdx+9hBq 1wyd2Xr2F3eYjUX580cG991qTfIHDJM= X-Google-Smtp-Source: AB8JxZpZZRIqlmvV9QPj9Ou0vU4F0rydyingpEQa599OIxVNHszp6BZJTw/80ZASgF0ZkNXy97wOWg== X-Received: by 2002:a5d:4045:: with SMTP id w5-v6mr709753wrp.96.1527224230682; Thu, 24 May 2018 21:57:10 -0700 (PDT) Received: from localhost.localdomain (cust246-dsl91-135-6.idnet.net. [91.135.6.246]) by smtp.gmail.com with ESMTPSA id k82-v6sm5818111wmf.17.2018.05.24.21.57.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 21:57:10 -0700 (PDT) From: Alex Kiernan To: u-boot@lists.denx.de Date: Fri, 25 May 2018 04:57:04 +0000 Message-Id: <20180525045704.21658-2-alex.kiernan@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180525045704.21658-1-alex.kiernan@gmail.com> References: <20180525045704.21658-1-alex.kiernan@gmail.com> Subject: [U-Boot] [PATCH v1 2/2] buildman: Add support for environment delta in summary X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.18 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" When summarising the builds, add the -U option to emit delta lines for the default environment built into U-Boot at each commit. Signed-off-by: Alex Kiernan Reviewed-by: Simon Glass --- tools/buildman/builder.py | 128 +++++++++++++++++++++++++++++++++----- tools/buildman/cmdline.py | 2 + tools/buildman/control.py | 3 +- 3 files changed, 117 insertions(+), 16 deletions(-) diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index 2ccdee02a3..a5a2ffdfdf 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -127,6 +127,15 @@ class Config: val = val ^ hash(key) & hash(value) return val +class Environment: + """Holds information about environment variables for a board.""" + def __init__(self, target): + self.target = target + self.environment = {} + + def Add(self, key, value): + self.environment[key] = value + class Builder: """Class for building U-Boot for a particular commit. @@ -199,13 +208,17 @@ class Builder: value is itself a dictionary: key: config name value: config value + environment: Dictionary keyed by environment variable, Each + value is the value of environment variable. """ - def __init__(self, rc, err_lines, sizes, func_sizes, config): + def __init__(self, rc, err_lines, sizes, func_sizes, config, + environment): self.rc = rc self.err_lines = err_lines self.sizes = sizes self.func_sizes = func_sizes self.config = config + self.environment = environment def __init__(self, toolchains, base_dir, git_dir, num_threads, num_jobs, gnu_make='make', checkout=True, show_unknown=True, step=1, @@ -310,7 +323,8 @@ class Builder: def SetDisplayOptions(self, show_errors=False, show_sizes=False, show_detail=False, show_bloat=False, - list_error_boards=False, show_config=False): + list_error_boards=False, show_config=False, + show_environment=False): """Setup display options for the builder. show_errors: True to show summarised error/warning info @@ -319,6 +333,7 @@ class Builder: show_bloat: Show detail for each function list_error_boards: Show the boards which caused each error/warning show_config: Show config deltas + show_environment: Show environment deltas """ self._show_errors = show_errors self._show_sizes = show_sizes @@ -326,6 +341,7 @@ class Builder: self._show_bloat = show_bloat self._list_error_boards = list_error_boards self._show_config = show_config + self._show_environment = show_environment def _AddTimestamp(self): """Add a new timestamp to the list and record the build period. @@ -609,8 +625,33 @@ class Builder: config[key] = value return config + def _ProcessEnvironment(self, fname): + """Read in a uboot.env file + + This function reads in environment variables from a file. + + Args: + fname: Filename to read + + Returns: + Dictionary: + key: environment variable (e.g. bootlimit) + value: value of environment variable (e.g. 1) + """ + environment = {} + if os.path.exists(fname): + with open(fname) as fd: + for line in fd.read().split('\0'): + try: + key, value = line.split('=', 1) + environment[key] = value + except ValueError: + # ignore lines we can't parse + pass + return environment + def GetBuildOutcome(self, commit_upto, target, read_func_sizes, - read_config): + read_config, read_environment): """Work out the outcome of a build. Args: @@ -618,6 +659,7 @@ class Builder: target: Target board to check read_func_sizes: True to read function size information read_config: True to read .config and autoconf.h files + read_environment: True to read uboot.env files Returns: Outcome object @@ -627,6 +669,7 @@ class Builder: sizes = {} func_sizes = {} config = {} + environment = {} if os.path.exists(done_file): with open(done_file, 'r') as fd: return_code = int(fd.readline()) @@ -676,12 +719,18 @@ class Builder: fname = os.path.join(output_dir, name) config[name] = self._ProcessConfig(fname) - return Builder.Outcome(rc, err_lines, sizes, func_sizes, config) + if read_environment: + output_dir = self.GetBuildDir(commit_upto, target) + fname = os.path.join(output_dir, 'uboot.env') + environment = self._ProcessEnvironment(fname) + + return Builder.Outcome(rc, err_lines, sizes, func_sizes, config, + environment) - return Builder.Outcome(OUTCOME_UNKNOWN, [], {}, {}, {}) + return Builder.Outcome(OUTCOME_UNKNOWN, [], {}, {}, {}, {}) def GetResultSummary(self, boards_selected, commit_upto, read_func_sizes, - read_config): + read_config, read_environment): """Calculate a summary of the results of building a commit. Args: @@ -689,6 +738,7 @@ class Builder: commit_upto: Commit number to summarize (0..self.count-1) read_func_sizes: True to read function size information read_config: True to read .config and autoconf.h files + read_environment: True to read uboot.env files Returns: Tuple: @@ -705,6 +755,9 @@ class Builder: value is itself a dictionary: key: config name value: config value + Dictionary keyed by board.target. Each value is a dictionary: + key: environment variable + value: value of environment variable """ def AddLine(lines_summary, lines_boards, line, board): line = line.rstrip() @@ -720,10 +773,12 @@ class Builder: warn_lines_summary = [] warn_lines_boards = {} config = {} + environment = {} for board in boards_selected.itervalues(): outcome = self.GetBuildOutcome(commit_upto, board.target, - read_func_sizes, read_config) + read_func_sizes, read_config, + read_environment) board_dict[board.target] = outcome last_func = None last_was_warning = False @@ -756,8 +811,14 @@ class Builder: tconfig.Add(fname, key, value) config[board.target] = tconfig + tenvironment = Environment(board.target) + if outcome.environment: + for key, value in outcome.environment.iteritems(): + tenvironment.Add(key, value) + environment[board.target] = tenvironment + return (board_dict, err_lines_summary, err_lines_boards, - warn_lines_summary, warn_lines_boards, config) + warn_lines_summary, warn_lines_boards, config, environment) def AddOutcome(self, board_dict, arch_list, changes, char, color): """Add an output to our list of outcomes for each architecture @@ -810,12 +871,14 @@ class Builder: """ self._base_board_dict = {} for board in board_selected: - self._base_board_dict[board] = Builder.Outcome(0, [], [], {}, {}) + self._base_board_dict[board] = Builder.Outcome(0, [], [], {}, {}, + {}) self._base_err_lines = [] self._base_warn_lines = [] self._base_err_line_boards = {} self._base_warn_line_boards = {} self._base_config = None + self._base_environment = None def PrintFuncSizeDetail(self, fname, old, new): grow, shrink, add, remove, up, down = 0, 0, 0, 0, 0, 0 @@ -1010,8 +1073,8 @@ class Builder: def PrintResultSummary(self, board_selected, board_dict, err_lines, err_line_boards, warn_lines, warn_line_boards, - config, show_sizes, show_detail, show_bloat, - show_config): + config, environment, show_sizes, show_detail, + show_bloat, show_config, show_environment): """Compare results with the base results and display delta. Only boards mentioned in board_selected will be considered. This @@ -1036,10 +1099,13 @@ class Builder: value is itself a dictionary: key: config name value: config value + environment: Dictionary keyed by environment variable, Each + value is the value of environment variable. show_sizes: Show image size deltas show_detail: Show detail for each board show_bloat: Show detail for each function show_config: Show config changes + show_environment: Show environment changes """ def _BoardList(line, line_boards): """Helper function to get a line of boards containing a line @@ -1188,6 +1254,36 @@ class Builder: self.PrintSizeSummary(board_selected, board_dict, show_detail, show_bloat) + if show_environment and self._base_environment: + lines = [] + + for target in board_dict: + if target not in board_selected: + continue + + tbase = self._base_environment[target] + tenvironment = environment[target] + environment_plus = {} + environment_minus = {} + environment_change = {} + base = tbase.environment + for key, value in tenvironment.environment.iteritems(): + if key not in base: + environment_plus[key] = value + for key, value in base.iteritems(): + if key not in tenvironment.environment: + environment_minus[key] = value + for key, value in base.iteritems(): + new_value = tenvironment.environment.get(key) + if new_value and value != new_value: + desc = '%s -> %s' % (value, new_value) + environment_change[key] = desc + + _AddConfig(lines, target, environment_plus, environment_minus, + environment_change) + + _OutputConfigInfo(lines) + if show_config and self._base_config: summary = {} arch_config_plus = {} @@ -1294,6 +1390,7 @@ class Builder: self._base_err_line_boards = err_line_boards self._base_warn_line_boards = warn_line_boards self._base_config = config + self._base_environment = environment # Get a list of boards that did not get built, if needed not_built = [] @@ -1306,10 +1403,11 @@ class Builder: def ProduceResultSummary(self, commit_upto, commits, board_selected): (board_dict, err_lines, err_line_boards, warn_lines, - warn_line_boards, config) = self.GetResultSummary( + warn_line_boards, config, environment) = self.GetResultSummary( board_selected, commit_upto, read_func_sizes=self._show_bloat, - read_config=self._show_config) + read_config=self._show_config, + read_environment=self._show_environment) if commits: msg = '%02d: %s' % (commit_upto + 1, commits[commit_upto].subject) @@ -1317,8 +1415,8 @@ class Builder: self.PrintResultSummary(board_selected, board_dict, err_lines if self._show_errors else [], err_line_boards, warn_lines if self._show_errors else [], warn_line_boards, - config, self._show_sizes, self._show_detail, - self._show_bloat, self._show_config) + config, environment, self._show_sizes, self._show_detail, + self._show_bloat, self._show_config, self._show_environment) def ShowSummary(self, commits, board_selected): """Show a build summary for U-Boot for a given board list. diff --git a/tools/buildman/cmdline.py b/tools/buildman/cmdline.py index b8ddd4795c..e493b1ac4a 100644 --- a/tools/buildman/cmdline.py +++ b/tools/buildman/cmdline.py @@ -92,6 +92,8 @@ def ParseArgs(): default=None, help='Number of builder threads to use') parser.add_option('-u', '--show_unknown', action='store_true', default=False, help='Show boards with unknown build result') + parser.add_option('-U', '--show-environment', action='store_true', + default=False, help='Show environment changes in summary') parser.add_option('-v', '--verbose', action='store_true', default=False, help='Show build results while the build progresses') parser.add_option('-V', '--verbose-build', action='store_true', diff --git a/tools/buildman/control.py b/tools/buildman/control.py index 4ac4386db6..bc0819784f 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -319,7 +319,8 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, builder.SetDisplayOptions(options.show_errors, options.show_sizes, options.show_detail, options.show_bloat, options.list_error_boards, - options.show_config) + options.show_config, + options.show_environment) if options.summary: builder.ShowSummary(commits, board_selected) else: