From patchwork Sun Oct 28 23:58:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 990025 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F2TQkebQ"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42jvpq5pFFz9s89 for ; Mon, 29 Oct 2018 10:59:31 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 02EB185774; Sun, 28 Oct 2018 23:59:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nHOGiYSdry9b; Sun, 28 Oct 2018 23:59:29 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 207AE856E9; Sun, 28 Oct 2018 23:59:29 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id D83181BF301 for ; Sun, 28 Oct 2018 23:59:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D5953873F0 for ; Sun, 28 Oct 2018 23:59:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vycAo13zRTLb for ; Sun, 28 Oct 2018 23:59:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id 18772873ED for ; Sun, 28 Oct 2018 23:59:23 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id e4so3904924qkh.6 for ; Sun, 28 Oct 2018 16:59:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EC9Ur2EtkxX5qF4OIOIDznkmomyBGuF9hZdS/7DIe94=; b=F2TQkebQV9KOSW1CGf7joTVIbkRRVfTkWTiRcL8sq0A26r5pv1QCO6q+8ikrLVoRx4 FP4r90oZVWvSi/aYJ54AI7ZWTEhnpR/tYw6Jihu8k63Mr7FL03LZJ4sC28pFjCZ/vA55 YWEbY3NbmnhGr579L6FUGtGRZtagKTORarqFY9us1PCH4yxhIH2w7md9u32puxxNUQgK dQMEG9AJyN8o5e4TrMMborE0DIRCj+5Kq95lVO4QXxMCYAhFXguA/H7EPiHiQgp8H5cW 3cqpEeLDbF6zJkPxmdJGnGKp6vqosK7mvBJz9ZEc4Kc2pcbpnJBPdi2naReOTkexF7h1 OCNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EC9Ur2EtkxX5qF4OIOIDznkmomyBGuF9hZdS/7DIe94=; b=jU0SHkMFk/nW+lFlJESnS2Vt5bAxjj7ggN8MS9hutC8CHGxgBKLHrPCGTZGB0UWQjg Fwf+DRFIhDigBqgFC6pF/nUv7X8Rr/dGifJ+vPaTapLyDJMWwBgfoHWExxsEmzL+K3B5 +/v2TUEiBOiTiO9X5EJV/oCAj/dZYMgTkZ8g63oKbC4fYz3WLvDZDJXlTnO5e6oBG74z jWIE9LNgScfEjQE/L07rge5NT3Fm2o5D3WrFLqE8jjDnGE0XMMM831PIQJUKCzZGXUg2 2ywxwZIqx0Cln7ciw/DabUQzg9cYVwUdhBcw62VmUG6kj9Z946oisfH8JFd2vz+Yw6+k qB2g== X-Gm-Message-State: AGRZ1gK75xEJvTKGvDyKjXHS2bHfIMjYbEp4H2G51q39m7wXxU5drZ9E VeOtOBd4YlQD2Z61t+ZxuTnkoExX X-Google-Smtp-Source: AJdET5fnURq/CKF4sAgxZEQPBNMlBSl+SycdJN4m2DtzD+67WW3tTp5XNvIHp8JkULPz1UACsRIuCA== X-Received: by 2002:a37:1909:: with SMTP id k9mr10555431qkh.61.1540771161887; Sun, 28 Oct 2018 16:59:21 -0700 (PDT) Received: from ultri5.home ([189.114.100.41]) by smtp.gmail.com with ESMTPSA id l14-v6sm12536479qtk.97.2018.10.28.16.59.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 28 Oct 2018 16:59:21 -0700 (PDT) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sun, 28 Oct 2018 20:58:38 -0300 Message-Id: <20181028235839.22472-3-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181028235839.22472-1-ricardo.martincoski@gmail.com> References: <20181028235839.22472-1-ricardo.martincoski@gmail.com> Subject: [Buildroot] [PATCH 2/3] Makefile: offload .gitlab-ci.yml generation X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: | only: | - /-$CI_JOB_NAME$/ | - /-\$CI_JOB_NAME$/ | - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: | only: | refs: | - branches | - tags | variables: | - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: |.runtime_test: &runtime_test | only: | - tags |tests.package.test_python_txaio.TestPythonPy2Txaio: | <<: *runtime_test | only: | - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle --- NOTE: Example of alternative script in Python: https://gitlab.com/RicardoMartincoski/buildroot/commit/a39b7da5e8b6585eb86ce1102dd5a020c9fdec42 The downside is that it would add a dependency, so when someone runs 'make .gitlab-ci.yml' it would fail like this if pyyaml is not installed in the host machine: https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/112928495 --- Makefile | 4 +--- support/scripts/generate-gitlab-ci-yml | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 3 deletions(-) create mode 100755 support/scripts/generate-gitlab-ci-yml diff --git a/Makefile b/Makefile index a69af3b645..6a80809ad2 100644 --- a/Makefile +++ b/Makefile @@ -1155,9 +1155,7 @@ check-package: .PHONY: .gitlab-ci.yml .gitlab-ci.yml: .gitlab-ci.yml.in - cp $< $@ - (cd configs; LC_ALL=C ls -1 *_defconfig) | sed 's/$$/: *defconfig/' >> $@ - set -o pipefail; ./support/testing/run-tests -l 2>&1 | sed -r -e '/^test_run \((.*)\).*/!d; s//\1: *runtime_test/' | LC_ALL=C sort >> $@ + ./support/scripts/generate-gitlab-ci-yml $< $@ include docs/manual/manual.mk -include $(foreach dir,$(BR2_EXTERNAL_DIRS),$(sort $(wildcard $(dir)/docs/*/*.mk))) diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml new file mode 100755 index 0000000000..d824f669b5 --- /dev/null +++ b/support/scripts/generate-gitlab-ci-yml @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -e +set -o pipefail + +input="${1}" +output="${2}" + +cp "${input}" "${output}" + +( + cd configs + LC_ALL=C ls -1 *_defconfig +) \ + | sed 's/$/: *defconfig/' \ + >> "${output}" + +./support/testing/run-tests -l 2>&1 \ + | sed -r -e '/^test_run \((.*)\).*/!d; s//\1: *runtime_test/' \ + | LC_ALL=C sort \ + >> "${output}"