From patchwork Wed Apr 15 16:48:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Preston X-Patchwork-Id: 1271338 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=permerror header.from=codethink.co.uk Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 492SxQ1RXNz9s71 for ; Thu, 16 Apr 2020 02:49:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id AC1CB8620B; Wed, 15 Apr 2020 16:49:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i+JILK6gnmXx; Wed, 15 Apr 2020 16:49:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id CC432861E7; Wed, 15 Apr 2020 16:49:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 534881BF9B5 for ; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4EF7E83F0F for ; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wNJUAB5I2cTP for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from imap3.hz.codethink.co.uk (imap3.hz.codethink.co.uk [176.9.8.87]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 201D482DDE for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from [167.98.27.226] (helo=ts007-build.ts005.codethink.co.uk) by imap3.hz.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1jOlDX-0005f3-QD; Wed, 15 Apr 2020 17:48:52 +0100 From: Thomas Preston To: buildroot@buildroot.org Date: Wed, 15 Apr 2020 17:48:42 +0100 Message-Id: <20200415164846.122126-2-thomas.preston@codethink.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200415164846.122126-1-thomas.preston@codethink.co.uk> References: <20200415164846.122126-1-thomas.preston@codethink.co.uk> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/5] support/docker: Update to debian:buster-20200327 X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: unixmania@gmail.com, michael.drake@codethink.co.uk, yann.morin.1998@free.fr, Thomas Preston Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Update the test container to debian:buster-20200327, because it contains the openssh-server package v7.9, which provides an updated ssh-keygen (since v7.6). Specifically, the host key generation function in openssh-server v7.6 onwards now accepts a flag that allows us to specify a host tree prefix, instead of trying to create SSH host keys in /. This allows us to create SSH host keys and therefore run an SSH server as a non-root user. For example, to create host keys in `prefix/host/etc/ssh`: ssh-keygen -A -f prefix/host Signed-off-by: Thomas Preston --- support/docker/Dockerfile | 2 +- support/docker/apt-sources.list | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile index 03acde85d2..be795a18b4 100644 --- a/support/docker/Dockerfile +++ b/support/docker/Dockerfile @@ -6,7 +6,7 @@ # We use a specific tag for the base image *and* the corresponding date # for the repository., so do not forget to update the apt-sources.list # file that is shipped next to this Dockerfile. -FROM debian:stretch-20171210 +FROM debian:buster-20200327 LABEL maintainer="Buildroot mailing list " \ vendor="Buildroot" \ diff --git a/support/docker/apt-sources.list b/support/docker/apt-sources.list index 789fb8fc17..c13c4d03c4 100644 --- a/support/docker/apt-sources.list +++ b/support/docker/apt-sources.list @@ -1,4 +1,4 @@ -# Latest just before 20171210T000000Z: -deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch main -deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20171209T220346Z/ stretch-updates main -deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20171209T224618Z/ stretch/updates main +# Latest just before 20200327T000000Z: +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20200326T221821Z/ buster main +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20200326T221821Z/ buster-updates main +deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20200316T220151Z/ buster/updates main From patchwork Wed Apr 15 16:48:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Preston X-Patchwork-Id: 1271335 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=permerror header.from=codethink.co.uk Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 492Sx959Xlz9s71 for ; Thu, 16 Apr 2020 02:49:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0889E204FC; Wed, 15 Apr 2020 16:48:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qzTaoKbH-dEC; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 1E24F20407; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id F23F21BF346 for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id EF0A587824 for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IAXwFKOWGh83 for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from imap3.hz.codethink.co.uk (imap3.hz.codethink.co.uk [176.9.8.87]) by hemlock.osuosl.org (Postfix) with ESMTPS id 29E6E8739A for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from [167.98.27.226] (helo=ts007-build.ts005.codethink.co.uk) by imap3.hz.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1jOlDY-0005f3-5q; Wed, 15 Apr 2020 17:48:52 +0100 From: Thomas Preston To: buildroot@buildroot.org Date: Wed, 15 Apr 2020 17:48:43 +0100 Message-Id: <20200415164846.122126-3-thomas.preston@codethink.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200415164846.122126-1-thomas.preston@codethink.co.uk> References: <20200415164846.122126-1-thomas.preston@codethink.co.uk> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 2/5] support/docker: Add openssh-server X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: unixmania@gmail.com, michael.drake@codethink.co.uk, yann.morin.1998@free.fr, Thomas Preston Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Install the openssh-server package into the test container. This package, as well as its dependency openssh-client, is required to test SCP and SFTP download methods on the localhost, as if these tools were already installed on the user's machine. Signed-off-by: Thomas Preston --- support/docker/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile index be795a18b4..f44be230e6 100644 --- a/support/docker/Dockerfile +++ b/support/docker/Dockerfile @@ -36,6 +36,7 @@ RUN apt-get install -y --no-install-recommends \ libncurses5-dev \ locales \ mercurial \ + openssh-server \ python-flake8 \ python-nose2 \ python-pexpect \ From patchwork Wed Apr 15 16:48:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Preston X-Patchwork-Id: 1271337 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=permerror header.from=codethink.co.uk Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 492SxK04r3z9sP7 for ; Thu, 16 Apr 2020 02:49:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 6021F2094B; Wed, 15 Apr 2020 16:49:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KDlP5TopTtcF; Wed, 15 Apr 2020 16:48:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id F32B02151F; Wed, 15 Apr 2020 16:48:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 088251BF9B5 for ; Wed, 15 Apr 2020 16:48:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 00817842FF for ; Wed, 15 Apr 2020 16:48:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rm25TgePaJMA for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from imap3.hz.codethink.co.uk (imap3.hz.codethink.co.uk [176.9.8.87]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2022A83F0F for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from [167.98.27.226] (helo=ts007-build.ts005.codethink.co.uk) by imap3.hz.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1jOlDY-0005f3-Hl; Wed, 15 Apr 2020 17:48:52 +0100 From: Thomas Preston To: buildroot@buildroot.org Date: Wed, 15 Apr 2020 17:48:44 +0100 Message-Id: <20200415164846.122126-4-thomas.preston@codethink.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200415164846.122126-1-thomas.preston@codethink.co.uk> References: <20200415164846.122126-1-thomas.preston@codethink.co.uk> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/5] testing/infra: Add docstrings to base test classes X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: unixmania@gmail.com, michael.drake@codethink.co.uk, yann.morin.1998@free.fr, Thomas Preston Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" It wasn't immediately obvious to me what the two Buildroot base test classes were for, so add docstrings to explain the differences between BRConfigTest and BRTest. Signed-off-by: Thomas Preston --- support/testing/infra/basetest.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py index 5dc6034447..dec0ab7013 100644 --- a/support/testing/infra/basetest.py +++ b/support/testing/infra/basetest.py @@ -29,6 +29,7 @@ MINIMAL_CONFIG = \ class BRConfigTest(unittest.TestCase): + """Test up to the configure stage.""" config = None br2_external = list() downloaddir = None @@ -66,6 +67,7 @@ class BRConfigTest(unittest.TestCase): class BRTest(BRConfigTest): + """Test up to the build stage and instantiate an emulator.""" def __init__(self, names): super(BRTest, self).__init__(names) self.emulator = None From patchwork Wed Apr 15 16:48:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Preston X-Patchwork-Id: 1271336 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=permerror header.from=codethink.co.uk Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 492SxF2S8Rz9s71 for ; Thu, 16 Apr 2020 02:49:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CDBF784483; Wed, 15 Apr 2020 16:49:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sVSrCXTNBc-e; Wed, 15 Apr 2020 16:49:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8DBBD83F0F; Wed, 15 Apr 2020 16:49:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 3259D1BF9BA for ; Wed, 15 Apr 2020 16:48:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2F87987364 for ; Wed, 15 Apr 2020 16:48:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QOWq41DwK6pD for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from imap3.hz.codethink.co.uk (imap3.hz.codethink.co.uk [176.9.8.87]) by hemlock.osuosl.org (Postfix) with ESMTPS id 2ABED8739E for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from [167.98.27.226] (helo=ts007-build.ts005.codethink.co.uk) by imap3.hz.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1jOlDY-0005f3-TT; Wed, 15 Apr 2020 17:48:53 +0100 From: Thomas Preston To: buildroot@buildroot.org Date: Wed, 15 Apr 2020 17:48:45 +0100 Message-Id: <20200415164846.122126-5-thomas.preston@codethink.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200415164846.122126-1-thomas.preston@codethink.co.uk> References: <20200415164846.122126-1-thomas.preston@codethink.co.uk> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 4/5] download: Add SFTP support (not FTPS) X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: unixmania@gmail.com, michael.drake@codethink.co.uk, yann.morin.1998@free.fr, Thomas Preston Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Add Secure File Transfer Program (SFTP) support using a simple wrapper. SFTP is a common protocol used to transfer files securely between enterprises, but it is not currently supported in Buildroot because all of the packages are usually available via HTTP, git or some other download method. SFTP is similar to FTP but it preforms all operations over an encrypted SSH transport. Signed-off-by: Thomas Preston Signed-off-by: Michael Drake --- Config.in | 4 ++++ package/pkg-download.mk | 1 + support/download/dl-wrapper | 2 +- support/download/sftp | 37 +++++++++++++++++++++++++++++++++++++ 4 files changed, 43 insertions(+), 1 deletion(-) create mode 100755 support/download/sftp diff --git a/Config.in b/Config.in index 48fd36ff98..2050dd7a03 100644 --- a/Config.in +++ b/Config.in @@ -114,6 +114,10 @@ config BR2_SCP string "Secure copy (scp) command" default "scp" +config BR2_SFTP + string "Secure file transfer (sftp) command" + default "sftp" + config BR2_HG string "Mercurial (hg) command" default "hg" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index de619ba90a..88790fe46e 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) +export SFTP := $(call qstrip,$(BR2_SFTP)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..6cf0b89cba 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -88,7 +88,7 @@ main() { backend_urlencode="${uri%%+*}" backend="${backend_urlencode%|*}" case "${backend}" in - git|svn|cvs|bzr|file|scp|hg) ;; + git|svn|cvs|bzr|file|scp|hg|sftp) ;; *) backend="wget" ;; esac uri=${uri#*+} diff --git a/support/download/sftp b/support/download/sftp new file mode 100755 index 0000000000..8aeb91e0e8 --- /dev/null +++ b/support/download/sftp @@ -0,0 +1,37 @@ +#!/usr/bin/env bash + +# We want to catch any unexpected failure, and exit immediately +set -e + +# Download helper for sftp, to be called from the download wrapper script +# +# Options: +# -q Be quiet. +# -o FILE Copy to local file FILE. +# -f FILE Copy from remote file FILE. +# -u URI Download file at URI. +# +# Environment: +# SFTP : the sftp command to call + +verbose= +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do + case "${OPT}" in + q) verbose=-q;; + o) output="${OPTARG}";; + f) filename="${OPTARG}";; + u) uri="${OPTARG}";; + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; + \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; + esac +done + +shift $((OPTIND-1)) # Get rid of our options + +# Caller needs to single-quote its arguments to prevent them from +# being expanded a second time (in case there are spaces in them) +_sftp() { + eval ${SFTP} "${@}" +} + +_sftp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'" From patchwork Wed Apr 15 16:48:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Preston X-Patchwork-Id: 1271340 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=permerror header.from=codethink.co.uk Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 492Sxk0Qtpz9sP7 for ; Thu, 16 Apr 2020 02:49:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5448A2151F; Wed, 15 Apr 2020 16:49:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SAdc2lnOeKQf; Wed, 15 Apr 2020 16:49:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 2422F21541; Wed, 15 Apr 2020 16:49:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 4E5791BF346 for ; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4B2E487364 for ; Wed, 15 Apr 2020 16:48:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l8eYa1gVd5H3 for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from imap3.hz.codethink.co.uk (imap3.hz.codethink.co.uk [176.9.8.87]) by hemlock.osuosl.org (Postfix) with ESMTPS id 66BD987356 for ; Wed, 15 Apr 2020 16:48:55 +0000 (UTC) Received: from [167.98.27.226] (helo=ts007-build.ts005.codethink.co.uk) by imap3.hz.codethink.co.uk with esmtpsa (Exim 4.92 #3 (Debian)) id 1jOlDZ-0005f3-8u; Wed, 15 Apr 2020 17:48:53 +0100 From: Thomas Preston To: buildroot@buildroot.org Date: Wed, 15 Apr 2020 17:48:46 +0100 Message-Id: <20200415164846.122126-6-thomas.preston@codethink.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200415164846.122126-1-thomas.preston@codethink.co.uk> References: <20200415164846.122126-1-thomas.preston@codethink.co.uk> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 5/5] support/testing: Add download tests for SCP/SFTP X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: unixmania@gmail.com, michael.drake@codethink.co.uk, yann.morin.1998@free.fr, Thomas Preston Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Add download test infrastructure which starts an OpenSSH server using the sshd binary installed on the Buildroot host. This server can then be used to test the expected usage of the SCP and SFTP download methods. The test creates new SSH keys for the server and client, so that the server can be run as a non-root user. A new test module has been added called `tests.download.sshd` which contains helper methods to create the SSH keys and a class called `OpenSSHDaemon` which handles the sshd server component. The tests download example packages in the br2-external project `ssh`. They check the following conditions for both SCP and SFTP download methods: - Correct hash. - Incorrect hash. - No hash file. The SSH download test infrastructure is based on test_git.py. Signed-off-by: Thomas Preston --- .gitlab-ci.yml | 2 + .../tests/download/br2-external/scp/Config.in | 0 .../download/br2-external/scp/external.desc | 1 + .../download/br2-external/scp/external.mk | 4 + .../scp/package/scp-bad/scp-bad.hash | 1 + .../scp/package/scp-bad/scp-bad.mk | 17 +++ .../scp/package/scp-good/scp-good.hash | 1 + .../scp/package/scp-good/scp-good.mk | 17 +++ .../scp/package/scp-nohash/scp-nohash.mk | 17 +++ .../download/br2-external/sftp/Config.in | 0 .../download/br2-external/sftp/external.desc | 1 + .../download/br2-external/sftp/external.mk | 4 + .../sftp/package/sftp-bad/sftp-bad.hash | 1 + .../sftp/package/sftp-bad/sftp-bad.mk | 17 +++ .../sftp/package/sftp-good/sftp-good.hash | 1 + .../sftp/package/sftp-good/sftp-good.mk | 17 +++ .../sftp/package/sftp-nohash/sftp-nohash.mk | 17 +++ support/testing/tests/download/sshd.py | 129 ++++++++++++++++++ .../tests/download/sshd/ssh-test-1.0.tar.xz | Bin 0 -> 232 bytes support/testing/tests/download/test_ssh.py | 66 +++++++++ 20 files changed, 313 insertions(+) create mode 100644 support/testing/tests/download/br2-external/scp/Config.in create mode 100644 support/testing/tests/download/br2-external/scp/external.desc create mode 100644 support/testing/tests/download/br2-external/scp/external.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.hash create mode 100644 support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.hash create mode 100644 support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.mk create mode 100644 support/testing/tests/download/br2-external/scp/package/scp-nohash/scp-nohash.mk create mode 100644 support/testing/tests/download/br2-external/sftp/Config.in create mode 100644 support/testing/tests/download/br2-external/sftp/external.desc create mode 100644 support/testing/tests/download/br2-external/sftp/external.mk create mode 100644 support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.hash create mode 100644 support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.mk create mode 100644 support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.hash create mode 100644 support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.mk create mode 100644 support/testing/tests/download/br2-external/sftp/package/sftp-nohash/sftp-nohash.mk create mode 100644 support/testing/tests/download/sshd.py create mode 100644 support/testing/tests/download/sshd/ssh-test-1.0.tar.xz create mode 100755 support/testing/tests/download/test_ssh.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4b84a5b709..8061b50526 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -349,6 +349,8 @@ tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: { extends: .runtime tests.core.test_timezone.TestNoTimezone: { extends: .runtime_test } tests.download.test_git.TestGitHash: { extends: .runtime_test } tests.download.test_git.TestGitRefs: { extends: .runtime_test } +tests.download.test_ssh.TestSCP: { extends: .runtime_test } +tests.download.test_ssh.TestSFTP: { extends: .runtime_test } tests.fs.test_ext.TestExt2: { extends: .runtime_test } tests.fs.test_ext.TestExt2r1: { extends: .runtime_test } tests.fs.test_ext.TestExt3: { extends: .runtime_test } diff --git a/support/testing/tests/download/br2-external/scp/Config.in b/support/testing/tests/download/br2-external/scp/Config.in new file mode 100644 index 0000000000..e69de29bb2 diff --git a/support/testing/tests/download/br2-external/scp/external.desc b/support/testing/tests/download/br2-external/scp/external.desc new file mode 100644 index 0000000000..0ca0389a32 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/external.desc @@ -0,0 +1 @@ +name: SCP diff --git a/support/testing/tests/download/br2-external/scp/external.mk b/support/testing/tests/download/br2-external/scp/external.mk new file mode 100644 index 0000000000..49408f2f22 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/external.mk @@ -0,0 +1,4 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_SCP_PATH)/package/*/*.mk)) + +# Get the openssh-server port number from the test infra +SSHD_PORT_NUMBER ?= 2222 diff --git a/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.hash b/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.hash new file mode 100644 index 0000000000..90b15b30c2 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.hash @@ -0,0 +1 @@ +sha256 0000000000000000000000000000000000000000000000000000000000000000 ssh-test-1.0.tar.xz diff --git a/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.mk b/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.mk new file mode 100644 index 0000000000..5f6d55d940 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/package/scp-bad/scp-bad.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# scp-bad +# +################################################################################ + +SCP_BAD_VERSION = 1.0 +SCP_BAD_SOURCE = ssh-test-$(SCP_BAD_VERSION).tar.xz +SCP_BAD_SITE = scp://localhost:$(SSHD_TEST_DIR) +SCP_BAD_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.hash b/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.hash new file mode 100644 index 0000000000..31353a88ba --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.hash @@ -0,0 +1 @@ +sha256 b457c1a37ba7405e8806b93f3d5cc82165db0b0cad25d203f112e32c7a30c0be ssh-test-1.0.tar.xz diff --git a/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.mk b/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.mk new file mode 100644 index 0000000000..eff455a470 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/package/scp-good/scp-good.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# scp-good +# +################################################################################ + +SCP_GOOD_VERSION = 1.0 +SCP_GOOD_SOURCE = ssh-test-$(SCP_GOOD_VERSION).tar.xz +SCP_GOOD_SITE = scp://localhost:$(SSHD_TEST_DIR) +SCP_GOOD_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" \ + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/br2-external/scp/package/scp-nohash/scp-nohash.mk b/support/testing/tests/download/br2-external/scp/package/scp-nohash/scp-nohash.mk new file mode 100644 index 0000000000..f44c45fda3 --- /dev/null +++ b/support/testing/tests/download/br2-external/scp/package/scp-nohash/scp-nohash.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# scp-nohash +# +################################################################################ + +SCP_NOHASH_VERSION = 1.0 +SCP_NOHASH_SOURCE = ssh-test-$(SCP_NOHASH_VERSION).tar.xz +SCP_NOHASH_SITE = scp://localhost:$(SSHD_TEST_DIR) +SCP_NOHASH_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" \ + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/br2-external/sftp/Config.in b/support/testing/tests/download/br2-external/sftp/Config.in new file mode 100644 index 0000000000..e69de29bb2 diff --git a/support/testing/tests/download/br2-external/sftp/external.desc b/support/testing/tests/download/br2-external/sftp/external.desc new file mode 100644 index 0000000000..ae3a2860cd --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/external.desc @@ -0,0 +1 @@ +name: SFTP diff --git a/support/testing/tests/download/br2-external/sftp/external.mk b/support/testing/tests/download/br2-external/sftp/external.mk new file mode 100644 index 0000000000..cd4a7c839e --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/external.mk @@ -0,0 +1,4 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_SFTP_PATH)/package/*/*.mk)) + +# Get the openssh-server port number from the test infra +SSHD_PORT_NUMBER ?= 2222 diff --git a/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.hash b/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.hash new file mode 100644 index 0000000000..90b15b30c2 --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.hash @@ -0,0 +1 @@ +sha256 0000000000000000000000000000000000000000000000000000000000000000 ssh-test-1.0.tar.xz diff --git a/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.mk b/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.mk new file mode 100644 index 0000000000..fa83c2cd68 --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/package/sftp-bad/sftp-bad.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# sftp-bad +# +################################################################################ + +SFTP_BAD_VERSION = 1.0 +SFTP_BAD_SOURCE = ssh-test-$(SFTP_BAD_VERSION).tar.xz +SFTP_BAD_SITE = sftp://localhost/$(SSHD_TEST_DIR) +SFTP_BAD_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" \ + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.hash b/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.hash new file mode 100644 index 0000000000..31353a88ba --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.hash @@ -0,0 +1 @@ +sha256 b457c1a37ba7405e8806b93f3d5cc82165db0b0cad25d203f112e32c7a30c0be ssh-test-1.0.tar.xz diff --git a/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.mk b/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.mk new file mode 100644 index 0000000000..1cb5f22dbc --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/package/sftp-good/sftp-good.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# sftp-good +# +################################################################################ + +SFTP_GOOD_VERSION = 1.0 +SFTP_GOOD_SOURCE = ssh-test-$(SFTP_GOOD_VERSION).tar.xz +SFTP_GOOD_SITE = sftp://localhost/$(SSHD_TEST_DIR) +SFTP_GOOD_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" \ + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/br2-external/sftp/package/sftp-nohash/sftp-nohash.mk b/support/testing/tests/download/br2-external/sftp/package/sftp-nohash/sftp-nohash.mk new file mode 100644 index 0000000000..62a2c3d864 --- /dev/null +++ b/support/testing/tests/download/br2-external/sftp/package/sftp-nohash/sftp-nohash.mk @@ -0,0 +1,17 @@ +################################################################################ +# +# sftp-nohash +# +################################################################################ + +SFTP_NOHASH_VERSION = 1.0 +SFTP_NOHASH_SOURCE = ssh-test-$(SFTP_NOHASH_VERSION).tar.xz +SFTP_NOHASH_SITE = sftp://localhost/$(SSHD_TEST_DIR) +SFTP_NOHASH_DL_OPTS = \ + -P $(SSHD_PORT_NUMBER) \ + -i $(SSH_IDENTITY) \ + -o "UserKnownHostsFile=/dev/null" \ + -o "StrictHostKeyChecking=no" \ + -o "CheckHostIP=no" \ + +$(eval $(generic-package)) diff --git a/support/testing/tests/download/sshd.py b/support/testing/tests/download/sshd.py new file mode 100644 index 0000000000..ba5c2e053f --- /dev/null +++ b/support/testing/tests/download/sshd.py @@ -0,0 +1,129 @@ +import os +import shutil +import subprocess + +# subprocess does not kill the child daemon when a test case fails by raising +# an exception. So use pexpect instead. +import pexpect + +import infra + + +SSHD_PORT_INITIAL = 2222 +SSHD_PORT_LAST = SSHD_PORT_INITIAL + 99 +SSHD_PATH = "/usr/sbin/sshd" +SSHD_HOST_DIR = "host" + +# SSHD_KEY_DIR is where the /etc/ss/ssh_host_*_key files go +SSHD_KEY_DIR = os.path.join(SSHD_HOST_DIR, "etc/ssh") +SSHD_KEY = os.path.join(SSHD_KEY_DIR, "ssh_host_ed25519_key") + +# SSH_CLIENT_KEY_DIR is where the client id_rsa key and authorized_keys files go +SSH_CLIENT_KEY_DIR = os.path.join(SSHD_HOST_DIR, "home/br-user/ssh") +SSH_CLIENT_KEY = os.path.join(SSH_CLIENT_KEY_DIR, "id_rsa") +SSH_AUTH_KEYS_FILE = os.path.join(SSH_CLIENT_KEY_DIR, "authorized_keys") + + +class OpenSSHDaemon(): + + def __init__(self, builddir, logtofile): + """ + Start an OpenSSH SSH Daemon + + In order to support test cases in parallel, select the port the + server will listen to in runtime. Since there is no reliable way + to allocate the port prior to starting the server (another + process in the host machine can use the port between it is + selected from a list and it is really allocated to the server) + try to start the server in a port and in the case it is already + in use, try the next one in the allowed range. + """ + self.daemon = None + self.port = None + + self.logfile = infra.open_log_file(builddir, "sshd", logtofile) + + server_keyfile = os.path.join(builddir, SSHD_KEY) + auth_keys_file = os.path.join(builddir, SSH_AUTH_KEYS_FILE) + daemon_cmd = [SSHD_PATH, + "-D", # or use -ddd to debug + "-e", + "-h", server_keyfile, + "-o", "AuthorizedKeysFile={}".format(auth_keys_file)] + for port in range(SSHD_PORT_INITIAL, SSHD_PORT_LAST + 1): + cmd = daemon_cmd + ["-p", "{}".format(port)] + self.logfile.write( + "> starting sshd with '{}'\n".format(" ".join(cmd))) + self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile, + encoding='utf-8') + ret = self.daemon.expect([ + # Success + "Server listening on :: port {}.".format(port), + # Failure + "Cannot bind any address."]) + if ret == 0: + self.port = port + return + raise SystemError("Could not find a free port to run sshd") + + def stop(self): + if self.daemon is None: + return + self.daemon.terminate(force=True) + + +def generate_keys_server(builddir, logfile): + """Generate keys required to run an OpenSSH Daemon.""" + keyfile = os.path.join(builddir, SSHD_KEY) + if os.path.exists(keyfile): + logfile.write("> SSH server key already exists '{}'".format(keyfile)) + return + + hostdir = os.path.join(builddir, SSHD_HOST_DIR) + keydir = os.path.join(builddir, SSHD_KEY_DIR) + infra.run_cmd_on_host(builddir, ["mkdir", "-p", hostdir, keydir]) + + cmd = ["ssh-keygen", "-A", "-f", hostdir] + logfile.write( + "> generating SSH server keys with '{}'\n".format(" ".join(cmd))) + # When ssh-keygen fails to create an SSH server key it doesn't return a + # useful error code. So use subprocess.getoutput to check for an error + # message instead. + out = subprocess.getoutput(" ".join(cmd)) + logfile.write(out) + if "Could not save your public key" in out: + raise SystemError("Could not generate SSH server keys") + + +def generate_keys_client(builddir, logfile): + """Generate keys required to log into an OpenSSH Daemon via SCP or SFTP.""" + keyfile = os.path.join(builddir, SSH_CLIENT_KEY) + if os.path.exists(keyfile): + logfile.write("> SSH client key already exists '{}'".format(keyfile)) + return + + keydir = os.path.join(builddir, SSH_CLIENT_KEY_DIR) + infra.run_cmd_on_host(builddir, ["mkdir", "-p", keydir]) + + cmd = ["ssh-keygen", + "-f", keyfile, + "-b", "2048", + "-t", "rsa", + "-N", "", + "-q"] + logfile.write( + "> generating SSH client keys with '{}'\n".format(" ".join(cmd))) + ret = subprocess.call(cmd, stdout=logfile, stderr=logfile) + if ret > 0: + raise SystemError("Could not generate SSH client keys") + + # Allow key-based login for this user (so that we can fetch from localhost) + pubkeyfile = os.path.join(keydir, "{}.pub".format(keyfile)) + authfile = os.path.join(keydir, "authorized_keys") + shutil.copy(pubkeyfile, authfile) + + +def generate_keys(builddir, logtofile): + logfile = infra.open_log_file(builddir, "ssh-keygen", logtofile) + generate_keys_server(builddir, logfile) + generate_keys_client(builddir, logfile) diff --git a/support/testing/tests/download/sshd/ssh-test-1.0.tar.xz b/support/testing/tests/download/sshd/ssh-test-1.0.tar.xz new file mode 100644 index 0000000000000000000000000000000000000000..bd83d0aff57569181d48e3c62ed24cf413349857 GIT binary patch literal 232 zcmVv@ZoLY1;=#rcYU!vo{ zY*d180V}vZPgp1s8F;=V2;UVkreiQmd@~S!C7CzGo}p<~MWy zLlsZzOd$tuKG<#wIxTdaXt!(>CWqO6(9;!ET?+(Xp1W5kWc%A