From patchwork Fri Dec 12 20:04:47 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 420663 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ozlabs.org (Postfix) with ESMTP id 6886314009B for ; Sat, 13 Dec 2014 07:06:00 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 1959292A92; Fri, 12 Dec 2014 20:05:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x19xQexy-t5r; Fri, 12 Dec 2014 20:05:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 04829929E6; Fri, 12 Dec 2014 20:05:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (whitealder.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id A9BC51C2187 for ; Fri, 12 Dec 2014 20:05:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id A437692977 for ; Fri, 12 Dec 2014 20:05:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EfU4dV2UGMgC for ; Fri, 12 Dec 2014 20:05:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by whitealder.osuosl.org (Postfix) with ESMTPS id 7408D92914 for ; Fri, 12 Dec 2014 20:05:33 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id em10so3686801wid.5 for ; Fri, 12 Dec 2014 12:05:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gxh6f87CBU625TXqiB1BpiI7GWnjkuj0zvbjnR9liDQ=; b=Otkcrt8w250248JtW5Sxf8PBcQLR0+zA4PRGxVR2kRLcEaZsHaVSZ7wUAqz/qqE+/B kZPfap3fGSjMINKPATRrFGvgCWSYw7f8l+he2gb1aOEgA7V1IBh1XPNNU3766IlnVhlv y80EsHnvUZfzDLlX00bquOmlxvW10ewjQYwmJ5s4ui3/F1hwD7Nv9yEwjQDXPgZTeRAG s4YDm43QteB3THGzf9QWasnZTgD/T9Z/XRFGsilfozf74/O7oXASz6gG1k1Bd8Vb6rKn IMWYALdj9We5pD+zAgyFjBnqQ6cP6MKnEcZC8ST0JkV0C6fJlgjZsQFajYuQ0kzJhzRA YncA== X-Received: by 10.180.93.37 with SMTP id cr5mr10828028wib.76.1418414732134; Fri, 12 Dec 2014 12:05:32 -0800 (PST) Received: from localhost.localdomain (alc112.alcatel.be. [195.207.101.112]) by mx.google.com with ESMTPSA id p1sm3054831wjy.22.2014.12.12.12.05.30 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 12 Dec 2014 12:05:31 -0800 (PST) From: Thomas De Schampheleire To: thomas.petazzoni@free-electrons.com, buildroot@buildroot.org Date: Fri, 12 Dec 2014 21:04:47 +0100 Message-Id: <1418414696-32584-3-git-send-email-patrickdepinguin@gmail.com> X-Mailer: git-send-email 1.8.5.1 In-Reply-To: <1418414696-32584-1-git-send-email-patrickdepinguin@gmail.com> References: <1418414696-32584-1-git-send-email-patrickdepinguin@gmail.com> Cc: Thomas De Schampheleire Subject: [Buildroot] [PATCH v5 02/11] autobuild-run: use docopt for argument parsing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Using docopt, argument parsing becomes trivial. Adding a new argument is a matter of updating the usage text. This commit removes the original cumbersome argument handling and uses docopt instead. A method is added to import the settings from the configuration file in a similar dictionary as the one created by docopt, so that both can be merged (giving priority to the configuration file, as before). With these changes, each option can be passed as argument and in the configuration file. This means that http-user and http-password can now also be added as argument (even though passing the password on the command-line is not recommended). Signed-off-by: Thomas De Schampheleire --- scripts/autobuild-run | 134 +++++++++++++++++++++++++++----------------------- 1 file changed, 72 insertions(+), 62 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 4d9ef90..b686a78 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -57,6 +57,37 @@ # BR2_PACKAGE_CLASSPATH=y, improve the script to detect whether the # necessary host machine requirements are there to build classpath. +"""autobuild-run - run Buildroot autobuilder + +Usage: autobuild-run [options] + +Options: + -h, --help show this help message and exit + -V, --version show version + -n, --ninstances NINSTANCES number of parallel instances [default: 1] + -j, --njobs NJOBS number of parallel jobs [default: 1] + -s, --submitter SUBMITTER name/machine of submitter [default: N/A] + --http-login LOGIN username to send results with + --http-password PASSWORD password to send results with (for security + reasons it is recommended to define this in the + config file instead, with user-read permissions + only) + -c, --config CONFIG path to configuration file + +Format of the configuration file: + + All arguments can also be specified in the configuration file specified with + --config, using 'key = value' format (not including the leading -- + characters). For example: + + [main] + ninstances = + njobs = + http-login = + http-password = + submitter = +""" + import urllib2 import csv from random import randint @@ -70,6 +101,7 @@ import sys import hashlib import argparse import ConfigParser +import docopt MAX_DURATION = 60 * 60 * 4 VERSION = 1 @@ -522,81 +554,59 @@ def run_instance(instance, njobs, http_login, http_password, submitter, sysinfo) ret = do_build(instance, njobs, instance_log) send_results(instance, http_login, http_password, submitter, instance_log, ret) -def config_get(): - """Get configuration parameters, either from the command line or the config file""" +# args / config file merging inspired by: +# https://github.com/docopt/docopt/blob/master/examples/config_file_example.py - epilog_text = """ -Format of the configuration file: +def load_ini_config(configfile): + """Load configuration from file, returning a docopt-like dictionary""" - [main] - ninstances = - njobs = - http-login = - http-password = - submitter = -""" + if not os.path.exists(configfile): + print "ERROR: configuration file %s does not exist" % configfile + sys.exit(1) + + config = ConfigParser.RawConfigParser() + if not config.read(configfile): + print "ERROR: cannot parse configuration file %s" % configfile + sys.exit(1) + + # Prepend '--' to options specified in the config file, so they can be + # merged with those given on the command-line + return dict(('--%s' % key, value) for key, value in config.items('main')) - parser = argparse.ArgumentParser(description='Run Buildroot autobuilder', - formatter_class=argparse.RawDescriptionHelpFormatter, - epilog=epilog_text) - parser.add_argument("--ninstances", '-n', metavar="NINSTANCES", - help="Number of parallel instances", default=None) - parser.add_argument("--njobs", '-j', metavar="NJOBS", - help="Number of parallel jobs", default=None) - parser.add_argument("--submitter", '-s', metavar="SUBMITTER", - help="Name/machine of submitter") - parser.add_argument("--config", '-c', metavar="CONFIG", - help="Path to configuration file") - args = parser.parse_args() - - ninstances = 1 - njobs = 1 - http_login = None - http_password = None - submitter = "N/A" - - if args.config: - if not os.path.exists(args.config): - print "ERROR: configuration file %s does not exist" % args.config - sys.exit(1) - parser = ConfigParser.RawConfigParser() - if not parser.read([args.config]): - print "ERROR: cannot parse configuration file %s" % args.config - sys.exit(1) - if parser.has_option('main', 'ninstances'): - ninstances = parser.getint('main', 'ninstances') - if parser.has_option('main', 'njobs'): - njobs = parser.getint('main', 'njobs') - if parser.has_option('main', 'http-login'): - http_login = parser.get('main', 'http-login') - if parser.has_option('main', 'http-password'): - http_password = parser.get('main', 'http-password') - if parser.has_option('main', 'submitter'): - submitter = parser.get('main', 'submitter') - - if args.njobs: - njobs = int(args.njobs) - if args.ninstances: - ninstances = int(args.ninstances) - if args.submitter: - submitter = args.submitter - - return (ninstances, njobs, http_login, http_password, submitter) + +def merge(dict_1, dict_2): + """Merge two dictionaries. + + Values that evaluate to true take priority over falsy values. + `dict_1` takes priority over `dict_2`. + + """ + return dict((str(key), dict_1.get(key) or dict_2.get(key)) + for key in set(dict_2) | set(dict_1)) def main(): check_version() sysinfo = SystemInfo() - (ninstances, njobs, http_login, http_password, submitter) = config_get() - check_requirements(http_login, http_password) - if http_login is None or http_password is None: + + args = docopt.docopt(__doc__, version=VERSION) + + if args['--config']: + ini_config = load_ini_config(args['--config']) + # merge config/args, priority given to config + args = merge(ini_config, args) + + check_requirements(args['--http-login'], args['--http-password']) + if args['--http-login'] is None or args['--http-password'] is None: print "WARN: due to the lack of http login/password details, results will not be submitted" print "WARN: tarballs of results will be kept locally only" def sigterm_handler(signum, frame): os.killpg(os.getpgid(os.getpid()), signal.SIGTERM) sys.exit(1) processes = [] - for i in range(0, ninstances): - p = Process(target=run_instance, args=(i, njobs, http_login, http_password, submitter, sysinfo)) + for i in range(0, int(args['--ninstances'])): + p = Process(target=run_instance, args=(i, int(args['--njobs']), + args['--http-login'], args['--http-password'], + args['--submitter'], sysinfo)) p.start() processes.append(p) signal.signal(signal.SIGTERM, sigterm_handler)