From patchwork Thu Oct 16 19:15:59 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 400356 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from fraxinus.osuosl.org (fraxinus.osuosl.org [140.211.166.137]) by ozlabs.org (Postfix) with ESMTP id D9A7114010F for ; Fri, 17 Oct 2014 06:16:31 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 29B1EA24FF; Thu, 16 Oct 2014 19:16:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VJBlbSHdDnfJ; Thu, 16 Oct 2014 19:16:28 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E5735A250F; Thu, 16 Oct 2014 19:16:24 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2BFCC1C235D for ; Thu, 16 Oct 2014 19:16:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 23FC73393A for ; Thu, 16 Oct 2014 19:16:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zUlbWkB-nOPn for ; Thu, 16 Oct 2014 19:16:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by silver.osuosl.org (Postfix) with ESMTPS id D512833931 for ; Thu, 16 Oct 2014 19:16:20 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id l4so3346875lbv.24 for ; Thu, 16 Oct 2014 12:16:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=03p0Eg74eJVPIieSfPZBoa0i4O3OQT65nMBm5gLPzqU=; b=yGtZ28ufd85mkavYFeR5vOlxUUbWQf8s1Ci2B7TcUzggqfwIw/QqV7oXYIsC3n4oi2 8bdRqpH6zcSnUQK2SNrlzka+WYSSc6GhXfjMK40hIcryvb0U7Uk08zRFrmqfjZguQeBW 93YwPedICxE6Yc8SAN4/erSoAQ4TFkf6kL5rdVSO4bym7KwDPuH8/LwRoKqBmz9qpudg nREUchn/npGmKa4ZD9JtDydAsM73y8sabdoe1sOeBYomlRJ5EvPDxh0zWsws1HMFVUvm dkkHXq/L2cl3c/Oq7b5gb419SLLEtFyT55IgvoLsG4D1KkSouWhHJY83rDlwrCqpiX8u Hr4g== X-Received: by 10.153.8.164 with SMTP id dl4mr3748441lad.29.1413486979073; Thu, 16 Oct 2014 12:16:19 -0700 (PDT) Received: from localhost.localdomain (d54c62eeb.access.telenet.be. [84.198.46.235]) by mx.google.com with ESMTPSA id tl8sm8044772lbb.47.2014.10.16.12.16.18 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Oct 2014 12:16:18 -0700 (PDT) From: Thomas De Schampheleire To: thomas.petazzoni@free-electrons.com Date: Thu, 16 Oct 2014 21:15:59 +0200 Message-Id: <1413486964-5183-3-git-send-email-patrickdepinguin@gmail.com> X-Mailer: git-send-email 1.8.5.1 In-Reply-To: <1413486964-5183-1-git-send-email-patrickdepinguin@gmail.com> References: <1413486964-5183-1-git-send-email-patrickdepinguin@gmail.com> Cc: Thomas De Schampheleire , buildroot@buildroot.org Subject: [Buildroot] [PATCH buildroot-test 3/8] autobuild-run: use docopt for argument parsing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Using docopt, argument parsing becomes trivial. Adding a new argument is a matter of updating the usage text. This commit removes the original cumbersome argument handling and uses docopt instead. A method is added to import the settings from the configuration file in a similar dictionary as the one created by docopt, so that both can be merged (giving priority to the configuration file, as before). With these changes, each option can be passed as argument and in the configuration file. This means that http-user and http-password can now also be added as argument (even though passing the password on the command-line is not recommended). Signed-off-by: Thomas De Schampheleire --- scripts/autobuild-run | 131 ++++++++++++++++++++++++++------------------------ 1 file changed, 69 insertions(+), 62 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 2ead1f2..8c1918c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -57,6 +57,37 @@ # BR2_PACKAGE_CLASSPATH=y, improve the script to detect whether the # necessary host machine requirements are there to build classpath. +"""autobuild-run - run Buildroot autobuilder + +Usage: autobuild-run [options] + +Options: + -h, --help show this help message and exit + -V, --version show version + -n, --ninstances NINSTANCES number of parallel instances [default: 1] + -j, --njobs NJOBS number of parallel jobs [default: 1] + -s, --submitter SUBMITTER name/machine of submitter [default: N/A] + --http-login LOGIN username to send results with + --http-password PASSWORD password to send results with (for security + reasons it is recommended to define this in the + config file instead, with user-read permissions + only) + -c, --config CONFIG path to configuration file + +Format of the configuration file: + + All arguments can also be specified in the configuration file specified with + --config, using 'key = value' format (not including the leading -- + characters). For example: + + [main] + ninstances = + njobs = + http-login = + http-password = + submitter = +""" + import urllib2 import csv from random import randint @@ -70,6 +101,7 @@ import sys import hashlib import argparse import ConfigParser +from docopt import docopt MAX_DURATION = 60 * 60 * 4 VERSION = 1 @@ -488,72 +520,45 @@ def run_instance(instance, njobs, http_login, http_password, submitter, sysinfo) ret = do_build(instance, njobs, instance_log) send_results(instance, http_login, http_password, submitter, instance_log, ret) -# Function to get the configuration parameters, either from the -# command line, or through a configuration file. -def config_get(): - epilog_text = """ -Format of the configuration file: +# args / config file merging inspired by: +# https://github.com/docopt/docopt/blob/master/examples/config_file_example.py +def load_ini_config(configfile): + if not os.path.exists(configfile): + print "ERROR: configuration file %s does not exist" % configfile + sys.exit(1) - [main] - ninstances = - njobs = - http-login = - http-password = - submitter = -""" + config = ConfigParser.RawConfigParser() + if not config.read(configfile): + print "ERROR: cannot parse configuration file %s" % configfile + sys.exit(1) - parser = argparse.ArgumentParser(description='Run Buildroot autobuilder', - formatter_class=argparse.RawDescriptionHelpFormatter, - epilog=epilog_text) - parser.add_argument("--ninstances", '-n', metavar="NINSTANCES", - help="Number of parallel instances", default=None) - parser.add_argument("--njobs", '-j', metavar="NJOBS", - help="Number of parallel jobs", default=None) - parser.add_argument("--submitter", '-s', metavar="SUBMITTER", - help="Name/machine of submitter") - parser.add_argument("--config", '-c', metavar="CONFIG", - help="Path to configuration file") - args = parser.parse_args() - - ninstances = 1 - njobs = 1 - http_login = None - http_password = None - submitter = "N/A" - - if args.config: - if not os.path.exists(args.config): - print "ERROR: configuration file %s does not exist" % args.config - sys.exit(1) - parser = ConfigParser.RawConfigParser() - if not parser.read([args.config]): - print "ERROR: cannot parse configuration file %s" % args.config - sys.exit(1) - if parser.has_option('main', 'ninstances'): - ninstances = parser.getint('main', 'ninstances') - if parser.has_option('main', 'njobs'): - njobs = parser.getint('main', 'njobs') - if parser.has_option('main', 'http-login'): - http_login = parser.get('main', 'http-login') - if parser.has_option('main', 'http-password'): - http_password = parser.get('main', 'http-password') - if parser.has_option('main', 'submitter'): - submitter = parser.get('main', 'submitter') - - if args.njobs: - njobs = int(args.njobs) - if args.ninstances: - ninstances = int(args.ninstances) - if args.submitter: - submitter = args.submitter - - return (ninstances, njobs, http_login, http_password, submitter) + # Prepend '--' to options specified in the config file, so they can be + # merged with those given on the command-line + return dict(('--%s' % key, value) for key, value in config.items('main')) + + +def merge(dict_1, dict_2): + """Merge two dictionaries. + + Values that evaluate to true take priority over falsy values. + `dict_1` takes priority over `dict_2`. + + """ + return dict((str(key), dict_1.get(key) or dict_2.get(key)) + for key in set(dict_2) | set(dict_1)) if __name__ == '__main__': check_version() sysinfo = SystemInfo() - (ninstances, njobs, http_login, http_password, submitter) = config_get() - do_send_results = http_login and http_password + + args = docopt.docopt(__doc__, version=VERSION) + + if args['--config']: + ini_config = load_ini_config(args['--config']) + # merge config/args, priority given to config + args = merge(ini_config, args) + + do_send_results = args['--http-login'] and args['--http-password'] check_requirements(do_send_results) if not do_send_results: print "WARN: due to the lack of http login/password details, results will not be submitted" @@ -562,8 +567,10 @@ if __name__ == '__main__': os.killpg(os.getpgid(os.getpid()), signal.SIGTERM) sys.exit(1) processes = [] - for i in range(0, ninstances): - p = Process(target=run_instance, args=(i, njobs, http_login, http_password, submitter, sysinfo)) + for i in range(0, int(args['--ninstances'])): + p = Process(target=run_instance, args=(i, int(args['--njobs']), + args['--http-login'], args['--http-password'], + args['--submitter'], sysinfo)) p.start() processes.append(p) signal.signal(signal.SIGTERM, sigterm_handler)