From patchwork Wed Feb 19 16:35:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240883 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=me9bIlso; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3rn5W20z9sPk for ; Thu, 20 Feb 2020 04:00:57 +1100 (AEDT) Received: from localhost ([::1]:56648 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SiV-0005J8-JQ for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:00:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41876) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SO5-0002Z0-C5 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SO4-0007vk-Et for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:49 -0500 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]:40166) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SO4-0007vI-8e for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:48 -0500 Received: by mail-pf1-x431.google.com with SMTP id b185so321717pfb.7 for ; Wed, 19 Feb 2020 08:39:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pmyPbYDPgMfj7iewXRlwREPC4GaTkOAO8bGCy51JlgY=; b=me9bIlsoI9mi9isAShNfpwuA3sKkAuaDhcHsLJfUCKQx43EUxvuiDzlM8C4NBbaqCe pWcWjDd+5nfbN4PS7XtSC1W6iwJ0+c+u0UpjJPwgNVOa8zt+x9gAR9w6SvKjbsxkHrH+ JOeHw6dUOJZJKbG9hoAAEYyHfzRWi2Bj9/GLmQPFD4Enw7m2UkzM33V3RmWdbYw6gPtD aH/0jWNX0NOyO6SHYsYkIJNHkCgvBHpDI5cdm8dqUhpz12VNsxWMZxRP4l2IlKGp+T5E IXDDDShhQf6qYsHIhznckgFc390aA4V2hRrFQdOUs4MRC44GgBTFUU/tJ/wdJL7d1KSF 3cUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pmyPbYDPgMfj7iewXRlwREPC4GaTkOAO8bGCy51JlgY=; b=KZCFzHY/owSeCIYQ3lUVoZc4iZ3MVdZvVtv5iS+d+Z7LVRDxZwGnmafWk0z3gj02c6 HzTneNbVx1knXb0J77G1VtDzLiQ41gTxS+AvPhqUH6QrHbLKCTXYmQ96Yo9gCa8pP4sH 3jRPiUreh0Aqhz9+0enASHXUxkQxoo4+tTLTvoRDo+3FdPnYaXuXVKPJ24WBBuJPT7mp RDqdkM/PcHAhoqH/Gh8sTrfVaCfsmxkBX5LiXmWjxj4Fqu9fggn5KBBM3WWFWYNpNQ2E 4WQIIj1sVwJrFDGVRCRTMvQ1tf47IUghnWNjrbs+7DBAf9GF/Rnr03Xu+cZkaTDF2MEC im/w== X-Gm-Message-State: APjAAAWG8HBViVIa/atdswFo7udEOFUld9sygJ0GmnlLPAAWJjh6AaTi DaMfbL4k7t8fFUf7YKnDNCxzTlkWLKo= X-Google-Smtp-Source: APXvYqzwXSINC6ZBEIjCZB79xAeW7IjB3zJj95Ws1yUKeTUUZ0no53AqJNm56W1NISBHsKTr5IPMog== X-Received: by 2002:a63:7453:: with SMTP id e19mr1077790pgn.50.1582130387017; Wed, 19 Feb 2020 08:39:47 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:46 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 01/14] tests/vm: use $(PYTHON) consistently Date: Wed, 19 Feb 2020 11:35:24 -0500 Message-Id: <20200219163537.22098-2-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::431 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Change Makefile.include to use $(PYTHON) so for vm-boot-ssh to be consistent with other cases like vm-build. Signed-off-by: Robert Foley Reviewed-by: Alex Bennée Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 9e7c46a473..778e506755 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -80,7 +80,7 @@ vm-boot-serial-%: $(IMAGES_DIR)/%.img vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ - $(SRC_PATH)/tests/vm/$* \ + $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ --image "$<" \ --interactive \ From patchwork Wed Feb 19 16:35:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240878 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=GaYQurh2; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3ll0Y5mz9sPk for ; Thu, 20 Feb 2020 03:56:35 +1100 (AEDT) Received: from localhost ([::1]:56524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SeG-0005Pg-Sr for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:56:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41897) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SO7-0002eW-A9 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SO6-0007xG-BN for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:51 -0500 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]:37067) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SO6-0007wc-5c for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:50 -0500 Received: by mail-pf1-x432.google.com with SMTP id p14so329535pfn.4 for ; Wed, 19 Feb 2020 08:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=65Z4dVpY1gcxOALaGKWcd1F123zs7S6VnJl6E5T0NNc=; b=GaYQurh2Pbai5CmUe8nJnrigPOexEmspPBW9Vvhv/61ZKiAtSs1NFM/UrgjVVEnULP JjA550fgVdtnZXr1M+iug2JpLmj30Q6/9TX42IoUz1CHeENELRhgDdfG6rKSeV38YJCB S/0ztD8PuNRkexDJTcg5/N79p/TtIihjFZPIVMhM0XFfJ4Z/6AbkQKYrqPnSlHDsxklw mc0nyAmkWBa/FPdr/avrShGiBzSA8pq/tVDT9O58OuZeWswTEduJ8tG2dHT0Fw0T0LaI Y2fJcgbbIw08VAY77PfRSvYUBOneXIiQsdUGn/BseZ19jlBdevutjBhuu4VL5H27uVug gDyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=65Z4dVpY1gcxOALaGKWcd1F123zs7S6VnJl6E5T0NNc=; b=pvsJaeh/eIm8em2KLZOR1eyPC1MA9jv3ftoNHa7phUXbVe1kkE0D3kLdC2B/9UlnUZ A/mogS0FjpYlcikju+yNMuH2GgDth63PVk2WII2y4xJYs4GXYyrwidh8C4JccbKE3c8N ytGgWIX839FYat2oL6rHCeS7ZNqF7OccR2rUmQx+wTp2+Xsp+96ZvTvvnybzcK0EmOiu ve3tzJJy5WG+Sa1lh/j1uQWTUFtgv23XMPhBU4uY5vPBAmj/HplbY+QL7lWYHLCndmAE 73nLocI1icfWpIjNzJPwS5DnWzoydd9vSNY9nAzIwWekg48m8N4t4CDGevI/qGabyeyk iOMw== X-Gm-Message-State: APjAAAUYdkJMtSHARLvDT5spd4xzO93WlW3WVeGnNz/pPOmoYKpN2c+j fQNvqtTLOFzKRhdkten0mjt2Z7V/bgI= X-Google-Smtp-Source: APXvYqzv1TqfqMfATAwC6Pua0a0ePXkOhxT94Fzr67DVh8DsK/7N/hBOLMkw0Bxk3DvUL/p/94n50w== X-Received: by 2002:a63:9251:: with SMTP id s17mr28329581pgn.127.1582130388959; Wed, 19 Feb 2020 08:39:48 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:48 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 02/14] tests/vm: Debug mode shows ssh output. Date: Wed, 19 Feb 2020 11:35:25 -0500 Message-Id: <20200219163537.22098-3-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::432 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add changes to tests/vm/basevm.py so that during debug mode we show ssh output. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov Reviewed-by: Alex Bennée --- tests/vm/basevm.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 4dee6647e6..c99725b8c0 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -120,11 +120,16 @@ class BaseVM(object): return fname def _ssh_do(self, user, cmd, check): - ssh_cmd = ["ssh", "-q", "-t", + ssh_cmd = ["ssh", + "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, "-o", "ConnectTimeout=1", "-p", self.ssh_port, "-i", self._ssh_key_file] + # If not in debug mode, set ssh to quiet mode to + # avoid printing the results of commands. + if not self.debug: + ssh_cmd.append("-q") for var in self.envvars: ssh_cmd += ['-o', "SendEnv=%s" % var ] assert not isinstance(cmd, str) From patchwork Wed Feb 19 16:35:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240885 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=WSZ1+EZ6; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3vN592lz9sRf for ; Thu, 20 Feb 2020 04:03:12 +1100 (AEDT) Received: from localhost ([::1]:56804 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4Skg-0000vD-GJ for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:03:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41913) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SO9-0002kh-7L for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SO8-0007yi-8Y for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:53 -0500 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]:36643) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SO8-0007y5-2W for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:52 -0500 Received: by mail-pl1-x636.google.com with SMTP id a6so290648plm.3 for ; Wed, 19 Feb 2020 08:39:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AK6voT7klf41ScUCFY9SIwj/Ejz8/BionSSI8Z7kz7w=; b=WSZ1+EZ6MPm26Ma07Mp/WtlRLng2BoKnjv+/9/CU8ZcDnyugbylW08wRLLv3i50JmG pt6TivRN+1KAgyXAPKi65lbhx1xQVCtnpqGAF2YTl/D97/mM02TOIBk+ppI9BAjSAsB1 NzA8vO6IijahNWuDJ3svPFjI0HxKN4CH5ijRjcSHyjVk6Fmt03RKXH1fwLjXAq+Y+mBy cftEI7O32BZZkAjKi1TzwzUcujJIO40HSyOJGWsT5ww/yHYYmgkyQprNq8FNE4yzo0Bf wqakNJkfyBdz66DKKBLHPa/139Jn2sFop2FyeNoK6ccClWRRCOrPYeEtD5bOWeHaTwL0 COHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AK6voT7klf41ScUCFY9SIwj/Ejz8/BionSSI8Z7kz7w=; b=l6q1ftwRa8HDqigCfGhMPJI9xWe7uUIVBKEF3AxhCvgRXhgi+DtF0tqpVU10o76YK0 PKkJypPjnO3xGt8+ux0htUNdVvPDUqkdMpm2djUY/nuLnUMUZDNrjvi4YQ09uLhGYAzd 7qA+nAlw8yGCM/+Dkop0nOYrYXk+EMPjS28HLiqqvEaUH0ka1b3PZGYf4w/UmxVdRLJg 0xFYlRuwC1dDn8Qw95J0GdymCii0cRaco2lixS6MCVoRGkf8AucimzR/IpHQdl3I0GzL IyBX0qGqmR5LJsVy7mhry8aNdAh6Prx8vvP3FrE1cUIO+GD1IoDzztKWF+3WjCprdjEE vcGg== X-Gm-Message-State: APjAAAXPkK6UX/i4+FVLR/zO6ycrw0/+PZXidraosHKmn/Oop6gvlAFR E2g2UKBsOl3eLMmJ4xmmShZETXz3XvU= X-Google-Smtp-Source: APXvYqxZ/XT1iY7SidHMP1LZ0t93S0p60BqvBf/cn9IER2LvDlIVVcp5Gt7Amd4Z2yafvMBTfHIviA== X-Received: by 2002:a17:90a:7345:: with SMTP id j5mr9337899pjs.69.1582130390725; Wed, 19 Feb 2020 08:39:50 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:50 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 03/14] tests/vm: increased max timeout for vm boot. Date: Wed, 19 Feb 2020 11:35:26 -0500 Message-Id: <20200219163537.22098-4-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::636 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add change to increase timeout waiting for VM to boot. Needed for some emulation cases where it can take longer than 5 minutes to boot. Signed-off-by: Robert Foley Reviewed-by: Alex Bennée Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index c99725b8c0..5ca445e29a 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -57,6 +57,10 @@ class BaseVM(object): poweroff = "poweroff" # enable IPv6 networking ipv6 = True + # Scale up some timeouts under TCG. + # 4 is arbitrary, but greater than 2, + # since we found we need to wait more than twice as long. + tcg_ssh_timeout_multiplier = 4 def __init__(self, debug=False, vcpus=None): self._guest = None self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", @@ -309,6 +313,9 @@ class BaseVM(object): sys.stderr.write("### %s ...\n" % text) def wait_ssh(self, seconds=300): + # Allow more time for VM to boot under TCG. + if not kvm_available(self.arch): + seconds *= self.tcg_ssh_timeout_multiplier starttime = datetime.datetime.now() endtime = starttime + datetime.timedelta(seconds=seconds) guest_up = False From patchwork Wed Feb 19 16:35:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240887 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=bzmPi3jZ; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3xK0KWcz9sRf for ; Thu, 20 Feb 2020 04:04:53 +1100 (AEDT) Received: from localhost ([::1]:56866 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SmI-00043r-W8 for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:04:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41930) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOA-0002pw-UT for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SO9-0007zx-Tp for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:54 -0500 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]:38207) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SO9-0007zO-Nn for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:53 -0500 Received: by mail-pl1-x634.google.com with SMTP id t6so285333plj.5 for ; Wed, 19 Feb 2020 08:39:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yQ3ZiDck9TzFgYxsZULoxW3XRdOrweY+xSvjOe2by5c=; b=bzmPi3jZ+1BBc3nQpP0QhJiCHk9wFOLMkO8ci6dKpKTXJGTKPFADfqYQrm+OoEeSiZ te1KGDTyktKWiojvFAJAiAam9VC9Z0X2rXuuPrt2zmY3aHFlNNoXEhhDqj77UBF7uyCw 7urHk92yyp5MdAvjMzvOfrDaAozfD/KyXOJ57Hu8GeofDvBDIvbfsGHdLLP/Wk5H1m66 6okFgAZsEc2ynK1Ag2Hxk6aB+f50phn9XekZB43Md5webK0lPLytA1Xo0urDie+fd+OL x/wc1I00JSxS4M+Y1EPvWFgqauw/PltqCaPHSEirwS9HHXzqZJkXQj3tCACsg1uLFfv8 ZmLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yQ3ZiDck9TzFgYxsZULoxW3XRdOrweY+xSvjOe2by5c=; b=reP/EWr2KDXFO+awzrzWtlnHvgHBs8RYz7mbO6cwGmGwYKZ/zINLoo2T3Shq4ZTD14 pdI1eJ/FMUzN03bLRi8d7xlaaf12+pb9sF5N9+Am3QMjjCNOBXLvCBgKkLaIPOD5DjA4 Lmbm06/jeL5I80M4umwjTpNsdYnStv7HECpmn5gIeWb7LfplRJ7z3tK6doSlJdCl6DR6 gv1391HGmhX9VxPUIe/z+Fd24EWESr/nt2bZC1+g1xW4TPNXdYWhtDTxEEOVhdbh+NoF cqovqEOMRgJ/PH32ZyJhnafIYODFal3ijVU1uaTjY6ANMDK5/vaL6l4vDKGwebdyQOsa LWww== X-Gm-Message-State: APjAAAUdr58xmGQuGWjwJWXJqPzcPBl25BkekJ2VMapN4HA349oxR0x3 ypu9mhyvxDdpX6A7TPD1GgMfrmsFtN0= X-Google-Smtp-Source: APXvYqwqQDE2gZLRa5M5tjYOZ0UCQxV3q3kLyJ8gU2Hi21Db1t4rdFOSeoYF5J6G2LN5fSbrA/9wKQ== X-Received: by 2002:a17:90a:8545:: with SMTP id a5mr9525559pjw.3.1582130392452; Wed, 19 Feb 2020 08:39:52 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:51 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 04/14] tests/vm: give wait_ssh() option to wait for root Date: Wed, 19 Feb 2020 11:35:27 -0500 Message-Id: <20200219163537.22098-5-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::634 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Allow wait_ssh to wait for root user to be ready. This solves the issue where we perform a wait_ssh() successfully, but the root user is not yet ready to be logged in. Signed-off-by: Robert Foley Reviewed-by: Alex Bennée Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 5ca445e29a..7f26892268 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -312,7 +312,7 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) - def wait_ssh(self, seconds=300): + def wait_ssh(self, wait_root=False, seconds=300): # Allow more time for VM to boot under TCG. if not kvm_available(self.arch): seconds *= self.tcg_ssh_timeout_multiplier @@ -320,7 +320,10 @@ class BaseVM(object): endtime = starttime + datetime.timedelta(seconds=seconds) guest_up = False while datetime.datetime.now() < endtime: - if self.ssh("exit 0") == 0: + if wait_root and self.ssh_root("exit 0") == 0: + guest_up = True + break + elif self.ssh("exit 0") == 0: guest_up = True break seconds = (endtime - datetime.datetime.now()).total_seconds() From patchwork Wed Feb 19 16:35:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240857 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=gCkN7ke8; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3Z405tDz9sSN for ; Thu, 20 Feb 2020 03:48:12 +1100 (AEDT) Received: from localhost ([::1]:56274 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SW9-0003Fb-Sp for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:48:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41945) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOD-0002xK-Kw for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOB-00081R-W6 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:57 -0500 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]:43959) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOB-00080y-OC for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:55 -0500 Received: by mail-pf1-x431.google.com with SMTP id s1so313524pfh.10 for ; Wed, 19 Feb 2020 08:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ebGq32l0nFHilBZHXJSmVFGGc6KeoYEINrsNMAqI2qk=; b=gCkN7ke8S9LuLDuubL2fV4vxQC2ve0J+uVGkTjMCy2x45hMnq2zNxVPNsR9Er8t2OE e7RrpNdKEUSLvF6g2uGFC97KeMNKQAt5G/Sc0k/0tvu5SqvMdhlEA34quhpqD+1sl0PN AJ1gTzcZ2QEsn+VtfJwtKMN+ovcwqcHzsKOrTuZIPgkpR6ezAcSFJ11HP6kCuzc6kt6f ZuS92caSN/JNchV0vhhWMi4rtOCeiDIuCaAdJV9BgxcjOAMeU9BjL9cflZPAVpHzY3u0 KC7PN3YpJAY8VbxDNoyiW2eo0ni3/e2+7jfkmMIPdwHgqRRq4iwBNSBmRm56nHMvQY9D RqsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ebGq32l0nFHilBZHXJSmVFGGc6KeoYEINrsNMAqI2qk=; b=gYp+F5ywygxzdQNFuDFPsSEPEShq8BOs8t4Lx3Ca7tHm4A07mPY1xPxnRPI0WT2YOQ LLSQSy2hKagLW2fwZd6HYa+WRzaA4TWe4dSipW0DTR8uTdhCDEo7niwdIltNxBAkWuJ1 bRvuJvGG+HopKdaTlES/mSYW/xCxr62DsNHCOzPgMZoxi/mbps6vB0jiQKXi9GQpRfts dLIdDKtnYb/4EQdbx2tWym/RRxMYkL0sbqFZe1tlLb6ngoJ+RDZm7gbtZ4Okcc6Q+y5c Lzmdbxl07PCQLePtMuk0A9evGi6MGP+HChIWagH0b/L+ph3LMJ8nG+gwIXQ6kE2Rc1Yj 5uJQ== X-Gm-Message-State: APjAAAX05x7X0zZT7MtEij9OL+U9j7N6cg4mL1yQmlpxP21oue63IT1y El5fwgqZfXj9tpFIsANTRdBBi5EUyNE= X-Google-Smtp-Source: APXvYqz1M2QtFpW+KYR68gLmFhTl5o1yKAtGdzXMM7ImKgTmZKBgbG06+a+wWLpm40126SvUxCbb4A== X-Received: by 2002:a65:4d0d:: with SMTP id i13mr28300362pgt.346.1582130394277; Wed, 19 Feb 2020 08:39:54 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:53 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 05/14] tests/vm: Added gen_cloud_init_iso() to basevm.py Date: Wed, 19 Feb 2020 11:35:28 -0500 Message-Id: <20200219163537.22098-6-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::431 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This method was located in both centos and ubuntu.i386. Signed-off-by: Robert Foley Reviewed-by: Alex Bennée Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 40 ++++++++++++++++++++++++++++++++++++++++ tests/vm/centos | 33 +-------------------------------- tests/vm/ubuntu.i386 | 37 +------------------------------------ 3 files changed, 42 insertions(+), 68 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 7f26892268..8400b0e07f 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -345,6 +345,46 @@ class BaseVM(object): def qmp(self, *args, **kwargs): return self._guest.qmp(*args, **kwargs) + def gen_cloud_init_iso(self): + cidir = self._tmpdir + mdata = open(os.path.join(cidir, "meta-data"), "w") + name = self.name.replace(".","-") + mdata.writelines(["instance-id: {}-vm-0\n".format(name), + "local-hostname: {}-guest\n".format(name)]) + mdata.close() + udata = open(os.path.join(cidir, "user-data"), "w") + print("guest user:pw {}:{}".format(self._config['guest_user'], + self._config['guest_pass'])) + udata.writelines(["#cloud-config\n", + "chpasswd:\n", + " list: |\n", + " root:%s\n" % self._config['root_pass'], + " %s:%s\n" % (self._config['guest_user'], + self._config['guest_pass']), + " expire: False\n", + "users:\n", + " - name: %s\n" % self._config['guest_user'], + " sudo: ALL=(ALL) NOPASSWD:ALL\n", + " ssh-authorized-keys:\n", + " - %s\n" % self._config['ssh_pub_key'], + " - name: root\n", + " ssh-authorized-keys:\n", + " - %s\n" % self._config['ssh_pub_key'], + "locale: en_US.UTF-8\n"]) + proxy = os.environ.get("http_proxy") + if not proxy is None: + udata.writelines(["apt:\n", + " proxy: %s" % proxy]) + udata.close() + subprocess.check_call(["genisoimage", "-output", "cloud-init.iso", + "-volid", "cidata", "-joliet", "-rock", + "user-data", "meta-data"], + cwd=cidir, + stdin=self._devnull, stdout=self._stdout, + stderr=self._stdout) + + return os.path.join(cidir, "cloud-init.iso") + def parse_args(vmcls): def get_default_jobs(): diff --git a/tests/vm/centos b/tests/vm/centos index a41ff109eb..0ad4ecf419 100755 --- a/tests/vm/centos +++ b/tests/vm/centos @@ -31,37 +31,6 @@ class CentosVM(basevm.BaseVM): make docker-test-mingw@fedora {verbose} J={jobs} NETWORK=1; """ - def _gen_cloud_init_iso(self): - cidir = self._tmpdir - mdata = open(os.path.join(cidir, "meta-data"), "w") - mdata.writelines(["instance-id: centos-vm-0\n", - "local-hostname: centos-guest\n"]) - mdata.close() - udata = open(os.path.join(cidir, "user-data"), "w") - udata.writelines(["#cloud-config\n", - "chpasswd:\n", - " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS), - " expire: False\n", - "users:\n", - " - name: %s\n" % self.GUEST_USER, - " sudo: ALL=(ALL) NOPASSWD:ALL\n", - " ssh-authorized-keys:\n", - " - %s\n" % basevm.SSH_PUB_KEY, - " - name: root\n", - " ssh-authorized-keys:\n", - " - %s\n" % basevm.SSH_PUB_KEY, - "locale: en_US.UTF-8\n"]) - udata.close() - subprocess.check_call(["genisoimage", "-output", "cloud-init.iso", - "-volid", "cidata", "-joliet", "-rock", - "user-data", "meta-data"], - cwd=cidir, - stdin=self._devnull, stdout=self._stdout, - stderr=self._stdout) - return os.path.join(cidir, "cloud-init.iso") - def build_image(self, img): cimg = self._download_with_cache("https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1802.qcow2.xz") img_tmp = img + ".tmp" @@ -69,7 +38,7 @@ class CentosVM(basevm.BaseVM): subprocess.check_call(["ln", "-f", cimg, img_tmp + ".xz"]) subprocess.check_call(["xz", "--keep", "-dvf", img_tmp + ".xz"]) self.exec_qemu_img("resize", img_tmp, "50G") - self.boot(img_tmp, extra_args = ["-cdrom", self._gen_cloud_init_iso()]) + self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()]) self.wait_ssh() self.ssh_root_check("touch /etc/cloud/cloud-init.disabled") self.ssh_root_check("yum update -y") diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386 index 48e9cb1ad3..3266038fbd 100755 --- a/tests/vm/ubuntu.i386 +++ b/tests/vm/ubuntu.i386 @@ -29,41 +29,6 @@ class UbuntuX86VM(basevm.BaseVM): make --output-sync {target} -j{jobs} {verbose}; """ - def _gen_cloud_init_iso(self): - cidir = self._tmpdir - mdata = open(os.path.join(cidir, "meta-data"), "w") - mdata.writelines(["instance-id: ubuntu-vm-0\n", - "local-hostname: ubuntu-guest\n"]) - mdata.close() - udata = open(os.path.join(cidir, "user-data"), "w") - udata.writelines(["#cloud-config\n", - "chpasswd:\n", - " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS), - " expire: False\n", - "users:\n", - " - name: %s\n" % self.GUEST_USER, - " sudo: ALL=(ALL) NOPASSWD:ALL\n", - " ssh-authorized-keys:\n", - " - %s\n" % basevm.SSH_PUB_KEY, - " - name: root\n", - " ssh-authorized-keys:\n", - " - %s\n" % basevm.SSH_PUB_KEY, - "locale: en_US.UTF-8\n"]) - proxy = os.environ.get("http_proxy") - if not proxy is None: - udata.writelines(["apt:\n", - " proxy: %s" % proxy]) - udata.close() - subprocess.check_call(["genisoimage", "-output", "cloud-init.iso", - "-volid", "cidata", "-joliet", "-rock", - "user-data", "meta-data"], - cwd=cidir, - stdin=self._devnull, stdout=self._stdout, - stderr=self._stdout) - return os.path.join(cidir, "cloud-init.iso") - def build_image(self, img): cimg = self._download_with_cache( "https://cloud-images.ubuntu.com/releases/bionic/release-20191114/ubuntu-18.04-server-cloudimg-i386.img", @@ -71,7 +36,7 @@ class UbuntuX86VM(basevm.BaseVM): img_tmp = img + ".tmp" subprocess.check_call(["cp", "-f", cimg, img_tmp]) self.exec_qemu_img("resize", img_tmp, "50G") - self.boot(img_tmp, extra_args = ["-cdrom", self._gen_cloud_init_iso()]) + self.boot(img_tmp, extra_args = ["-cdrom", self.gen_cloud_init_iso()]) self.wait_ssh() self.ssh_root_check("touch /etc/cloud/cloud-init.disabled") self.ssh_root_check("apt-get update") From patchwork Wed Feb 19 16:35:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240889 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=heNDNZA3; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3ys4Ryyz9sPk for ; Thu, 20 Feb 2020 04:06:13 +1100 (AEDT) Received: from localhost ([::1]:56906 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4Snb-0006hn-1u for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:06:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41963) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOG-00034y-BN for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOE-00082i-66 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:00 -0500 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]:38148) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOD-00082I-TW for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:58 -0500 Received: by mail-pg1-x52f.google.com with SMTP id d6so364935pgn.5 for ; Wed, 19 Feb 2020 08:39:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3Rcexh5kqSFADdUujEVkhgsAb5ir6oyYtsJro9Fnu0Q=; b=heNDNZA3hcSKTgtctVsDu/zjP/4GhzGO7wcMU6LGxq4XXUHi2PLcAlh2fTTjfzM+XZ zC/x0WuFeLWSXS+OVHfY3hI6a1vmJJPTlGyJMSHImbigKor2j8t00l0vJTKU6x8Tnu3z 5tPhJxTAbSmYncQruKJI+q34dwtV6TJudwbG+2lpHyH9M3pxzZiDAdjtBJ5aDahZwEii 0lBzKdvtUD5iimIAU83k3wIonYcSsoWoGyZrxFrqp+YYUn1dKlc/Gf6q40Rju4mTsvdy zndLgQeKaugKxyPIf6qZYy/JW3BahDTEiV+nhs3rha2cHWUJb2vhEM5Dn3LF06XzlAYo RWHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3Rcexh5kqSFADdUujEVkhgsAb5ir6oyYtsJro9Fnu0Q=; b=UkEaPJcypXLvdnGZy0dy5zCnSziqh3FVFTeuqjbICXHfmUWOj/0yvtGS50PZe9G4Ta e/EtGhoKOMvwyqfOc6hAZ/P5HYxvXrTDcowonl9577+RRAzRxbcRAc2PvqqVkC8I/LZT lSrwTWsEAJqYsgorC8bE3cGiJRNQ5p23P++Qc01/rnFTH8bG3cj4fxUe6qjtf2papziG k1Mhyw/sHUOv+ApGNdeqdLBt4/O8ZXrtlTt3rJ0xj8/qhX2q5BCINRoy/l49l2BEMFaa 3RilDhRT6YM+oM3YdU5mjURIvn7myjO6ua5u+WUlO9FDe70CncXR1Psec52CqLFMTF+e 0ifA== X-Gm-Message-State: APjAAAVvLpRmUMX+nBVmOXpALK4e/75Hh0GnZXGrlKYVi6YwzHpz24tO Tu5wHutAamxjywOT8nbyozvATBGTnAY= X-Google-Smtp-Source: APXvYqzHj78rZRXm5CiKOmfnxh8TZQ26VVi1XMmPQhcHhEwPBieBB/DbAKKsTliu0YZwFklpqf/taA== X-Received: by 2002:a63:5903:: with SMTP id n3mr29788938pgb.25.1582130396079; Wed, 19 Feb 2020 08:39:56 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:55 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 06/14] tests/vm: Add workaround to consume console Date: Wed, 19 Feb 2020 11:35:29 -0500 Message-Id: <20200219163537.22098-7-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52f X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The ConsoleSocket object provides a socket interface which will consume all arriving characters on the socket, but will provide those chars via recv() as would a regular socket. This is a workaround we found was needed since there is a known issue where QEMU will hang waiting for console characters to be consumed. We also add the option of logging the console to a file. Signed-off-by: Robert Foley --- python/qemu/console_socket.py | 162 ++++++++++++++++++++++++++++++++++ python/qemu/machine.py | 12 ++- tests/vm/Makefile.include | 4 + tests/vm/basevm.py | 24 ++++- 4 files changed, 194 insertions(+), 8 deletions(-) create mode 100644 python/qemu/console_socket.py diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py new file mode 100644 index 0000000000..a1f74e60ac --- /dev/null +++ b/python/qemu/console_socket.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# +# This python module implements a ConsoleSocket object which is +# designed always drain the socket itself, and place +# the bytes into a in memory buffer for later processing. +# +# Optionally a file path can be passed in and we will also +# dump the characters to this file for debug. +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import asyncore +import socket +import threading +import io +import os +import sys +from collections import deque +import time +import traceback + +class ConsoleSocket(asyncore.dispatcher): + + def __init__(self, address, file=None): + self._recv_timeout_sec = 300 + self._buffer = deque() + self._asyncore_thread = None + self._sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self._sock.connect(address) + self._logfile = None + if file: + self._logfile = open(file, "w") + asyncore.dispatcher.__init__(self, sock=self._sock) + self._thread_start() + self._open = True + + def _thread_start(self): + """Kick off a thread to wait on the asyncore.loop""" + if self._asyncore_thread is not None: + return + self._asyncore_thread = threading.Thread(target=asyncore.loop, + kwargs={'timeout':1}) + self._asyncore_thread.daemon = True + self._asyncore_thread.start() + + def handle_close(self): + """redirect close to base class""" + # Call the base class close, but not self.close() since + # handle_close() occurs in the context of the thread which + # self.close() attempts to join. + asyncore.dispatcher.close(self) + + def close(self): + """Close the base object and wait for the thread to terminate""" + if self._open: + self._open = False + asyncore.dispatcher.close(self) + if self._asyncore_thread is not None: + thread, self._asyncore_thread = self._asyncore_thread, None + thread.join() + if self._logfile: + self._logfile.close() + self._logfile = None + + def handle_read(self): + """process arriving characters into in memory _buffer""" + try: + data = asyncore.dispatcher.recv(self, 1) + # latin1 is needed since there are some chars + # we are receiving that cannot be encoded to utf-8 + # such as 0xe2, 0x80, 0xA6. + string = data.decode("latin1") + except: + print("Exception seen.") + traceback.print_exc() + return + if self._logfile: + self._logfile.write("{}".format(string)) + self._logfile.flush() + for c in string: + self._buffer.append(c) + + def recv(self, n=1): + """Return chars from in memory buffer""" + start_time = time.time() + while len(self._buffer) < n: + time.sleep(0.1) + elapsed_sec = time.time() - start_time + if elapsed_sec > self._recv_timeout_sec: + raise socket.timeout + chars = ''.join([self._buffer.popleft() for i in range(n)]) + # We choose to use latin1 to remain consistent with + # handle_read() and give back the same data as the user would + # receive if they were reading directly from the + # socket w/o our intervention. + return chars.encode("latin1") + + def set_blocking(self): + """Maintain compatibility with socket API""" + pass + + def settimeout(self, seconds): + """Set current timeout on recv""" + self._recv_timeout_sec = seconds + +class ByteBuffer(deque): + """Simple in memory buffer with read/write interface""" + def write(self, bytes): + for i in bytes: + self.append(i) + def read(self, n): + return ''.join([self.popleft() for i in range(n)]) + +if __name__ == '__main__': + # Brief test to exercise the above code. + # The ConsoleSocket will ship some data to the server, + # the server will echo it back and the client will echo what it received. + + # First remove the socket. + address = "./test_console_socket" + if os.path.exists(address): + os.unlink(address) + + # Create the server side. + server_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server_socket.bind(address) + server_socket.listen(1) + + # Create the object we are trying to test. + console_socket = ConsoleSocket(address, file="./logfile.txt") + + # Generate some data and ship it over the socket. + send_data = "" + for i in range(10): + send_data += "this is a test message {}\n".format(i) + console_socket.send(send_data.encode('latin1')) + connection, client_address = server_socket.accept() + + # Process the data on the server and ship it back. + data = connection.recv(len(send_data)) + print("server received: {}".format(data)) + print("server: sending data back to the client") + connection.sendall(data) + + # Client receives teh bytes and displays them. + print("client: receiving bytes") + bytes = console_socket.recv(len(data)) + recv_data = bytes.decode('latin1') + print("client received: {}".format(recv_data)) + assert(recv_data == send_data) + # Close console connection first, then close server. + console_socket.close() + connection.close() + server_socket.close() + print("test successful.") + diff --git a/python/qemu/machine.py b/python/qemu/machine.py index 183d8f3d38..6a4a5c2845 100644 --- a/python/qemu/machine.py +++ b/python/qemu/machine.py @@ -24,6 +24,7 @@ import subprocess import shutil import socket import tempfile +from qemu.console_socket import ConsoleSocket from . import qmp @@ -71,7 +72,8 @@ class QEMUMachine(object): def __init__(self, binary, args=None, wrapper=None, name=None, test_dir="/var/tmp", monitor_address=None, - socket_scm_helper=None, sock_dir=None): + socket_scm_helper=None, sock_dir=None, + console_log=None): ''' Initialize a QEMUMachine @@ -82,6 +84,8 @@ class QEMUMachine(object): @param test_dir: where to create socket and log file @param monitor_address: address for QMP monitor @param socket_scm_helper: helper program, required for send_fd_scm() + @param sock_dir: where to create socket (overrides test_dir for sock) + @param console_log: (optional) path to console log file @note: Qemu process is not started until launch() is used. ''' if args is None: @@ -118,6 +122,7 @@ class QEMUMachine(object): self._console_address = None self._console_socket = None self._remove_files = [] + self._console_log_path = console_log # just in case logging wasn't configured by the main script: logging.basicConfig() @@ -566,7 +571,6 @@ class QEMUMachine(object): Returns a socket connected to the console """ if self._console_socket is None: - self._console_socket = socket.socket(socket.AF_UNIX, - socket.SOCK_STREAM) - self._console_socket.connect(self._console_address) + self._console_socket = ConsoleSocket(self._console_address, + file=self._console_log_path) return self._console_socket diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 778e506755..76a1bd7fe8 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -32,6 +32,7 @@ vm-help vm-test: @echo ' EXTRA_CONFIGURE_OPTS="..."' @echo " J=[0..9]* - Override the -jN parameter for make commands" @echo " DEBUG=1 - Enable verbose output on host and interactive debugging" + @echo " LOG_CONSOLE=1 - Log console to file in: ~/.cache/qemu-vm " @echo " V=1 - Enable verbose ouput on host and guest commands" @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool" @@ -48,6 +49,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(call quiet-command, \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ + $(if $(LOG_CONSOLE),--log-console) \ --image "$@" \ --force \ --build-image $@, \ @@ -62,6 +64,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(DEBUG), --interactive) \ $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ + $(if $(LOG_CONSOLE),--log-console) \ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -82,6 +85,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ + $(if $(LOG_CONSOLE),--log-console) \ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 8400b0e07f..a22b59b23c 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -57,11 +57,14 @@ class BaseVM(object): poweroff = "poweroff" # enable IPv6 networking ipv6 = True + # This is the timeout on the wait for console bytes. + socket_timeout = 120 # Scale up some timeouts under TCG. # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 - def __init__(self, debug=False, vcpus=None): + def __init__(self, debug=False, vcpus=None, + log_console=False): self._guest = None self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", @@ -75,6 +78,11 @@ class BaseVM(object): self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + self._console_log_path = None + if log_console: + self._console_log_path = \ + os.path.join(os.path.expanduser("~/.cache/qemu-vm"), + "{}.install.log".format(self.name)) self.debug = debug self._stderr = sys.stderr self._devnull = open(os.devnull, "w") @@ -185,7 +193,8 @@ class BaseVM(object): args += self._data_args + extra_args logging.debug("QEMU args: %s", " ".join(args)) qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch) - guest = QEMUMachine(binary=qemu_bin, args=args) + guest = QEMUMachine(binary=qemu_bin, args=args, + console_log=self._console_log_path) guest.set_machine('pc') guest.set_console() try: @@ -199,6 +208,8 @@ class BaseVM(object): raise atexit.register(self.shutdown) self._guest = guest + # Init console so we can start consuming the chars. + self.console_init() usernet_info = guest.qmp("human-monitor-command", command_line="info usernet") self.ssh_port = None @@ -210,7 +221,9 @@ class BaseVM(object): raise Exception("Cannot find ssh port from 'info usernet':\n%s" % \ usernet_info) - def console_init(self, timeout = 120): + def console_init(self, timeout = None): + if timeout == None: + timeout = self.socket_timeout vm = self._guest vm.console_socket.settimeout(timeout) @@ -419,6 +432,8 @@ def parse_args(vmcls): help="Interactively run command") parser.add_option("--snapshot", "-s", action="store_true", help="run tests with a snapshot") + parser.add_option("--log-console", action="store_true", + help="Log console to file.") parser.disable_interspersed_args() return parser.parse_args() @@ -430,7 +445,8 @@ def main(vmcls): return 1 logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) - vm = vmcls(debug=args.debug, vcpus=args.jobs) + vm = vmcls(debug=args.debug, vcpus=args.jobs, + log_console=args.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.image, From patchwork Wed Feb 19 16:35:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240861 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=AA8aXCXt; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3bx724Nz9sR4 for ; Thu, 20 Feb 2020 03:49:49 +1100 (AEDT) Received: from localhost ([::1]:56316 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SXi-0006lT-FI for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:49:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41972) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOH-00037m-DC for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOF-00083t-MV for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:01 -0500 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]:33699) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOF-00083K-Dl for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:39:59 -0500 Received: by mail-pf1-x42b.google.com with SMTP id n7so341497pfn.0 for ; Wed, 19 Feb 2020 08:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=36Q8JSf0N/o+sezmkSBG9jdKzoyNhca6OBIv/UTtZJ4=; b=AA8aXCXtJGNvVvQcMNARMLwDKoF7WNKiyp5laffKcMfqz7AlPPsLg4Zn0I7LMB0xOj QAnDAqttd/JdhrFrd4RWUcdgqxRCBZFEZ+fUOgOc4zq35Tfp/zkC1oiJ+7sLwjReyk1K alzBQ3UjvhySvbTWvhXezRGwprqxOgWJZdFDe4qL1g3Zpv4ntWRcfpa9DxmBGFbF+WgA 2NMUK5+uuKuG2hb7qI+SvBON9LvZikvJ445tDJBC2v4EuJRyJppxDdczCztgzn7jJTW/ JSsT0emWW3CP6EvjB4gCXHNhIe0NrsMeompRItWCu3KU3FqShsmqdwicH+GNadGcT/Ry Hqvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=36Q8JSf0N/o+sezmkSBG9jdKzoyNhca6OBIv/UTtZJ4=; b=aXnf8ekBJa9LDHuy/mevXzwBEYoyoe5mhnJ5D6PclyMBdyQyiw2y1k4O/Igypj9tFa fpA0/fd9K/6qTwa+f9zdj7YkiWSAZKWNAbByPKJBKCuQN7DTpWjbk8hHrptG/Ip5H7Eg fGH2I/wjpnzgGj3Ftywf/HhnTjfiKKEXLXZbElJ5DnjYREnWN5Ss6ki9GYDcn/eK085P nkcZIWfMujROZPoX0JZ1/weyDRe0H1KOyvk84ooUGJiG90DWmmXayTH0UYh5WJrEpfFg SBnfB+tRdm9R40IRk/i93Dj7Kff0c2BWsIBmDkvr/EAQg/Cjqe5HLro9pWZk7CnpZh6s D7WA== X-Gm-Message-State: APjAAAVNpvxuZ6o0YB+Ec6o8rexrqxwXRsoyQnFavTBtPBx1lM1Xzy0d VB5YbzR7i+a8rvqpm6zGKqn1dSgMozg= X-Google-Smtp-Source: APXvYqyTTarftHIdBCC+plxvIQh3Vr81ShWTGLcEF3ba5ZGUpqJL85Wa3nLd1qRJT9j2qS8qkuKLtw== X-Received: by 2002:a63:d94d:: with SMTP id e13mr28064012pgj.240.1582130397876; Wed, 19 Feb 2020 08:39:57 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:57 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 07/14] tests/vm: Add configuration to basevm.py Date: Wed, 19 Feb 2020 11:35:30 -0500 Message-Id: <20200219163537.22098-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42b X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being created or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 143 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 112 insertions(+), 31 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index a22b59b23c..ed1b56bcea 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -30,15 +30,39 @@ import shutil import multiprocessing import traceback -SSH_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE = { + 'block' : "-drive file={},if=none,id=drive0,cache=writeback "\ + "-device virtio-blk,drive=drive0,bootindex=0", + 'scsi' : "-device virtio-scsi-device,id=scsi "\ + "-drive file={},format=raw,if=none,id=hd0 "\ + "-device scsi-hd,drive=hd0,bootindex=0", +} class BaseVM(object): - GUEST_USER = "qemu" - GUEST_PASS = "qemupass" - ROOT_PASS = "qemupass" envvars = [ "https_proxy", @@ -63,20 +87,31 @@ class BaseVM(object): # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 - def __init__(self, debug=False, vcpus=None, + def __init__(self, debug=False, vcpus=None, config=None, log_console=False): self._guest = None + # Allow input config to override defaults. + self._config = DEFAULT_CONFIG.copy() + if config != None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", dir=".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file = os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] = \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] = \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file = os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) self._console_log_path = None if log_console: @@ -90,11 +125,14 @@ class BaseVM(object): self._stdout = sys.stdout else: self._stdout = self._devnull + netdev = "user,id=vnet,hostfwd=:127.0.0.1:{}-:22" self._args = [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + - (",ipv6=no" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=no" if not self.ipv6 else "") + + (",dns=" + self._config['dns'] if self._config['dns'] else ""), "-device", "virtio-net-pci,netdev=vnet", "-vnc", "127.0.0.1:0,to=20"] if vcpus and vcpus > 1: @@ -105,6 +143,45 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args = [] + if self._config['qemu_args'] != None: + qemu_args = self._config['qemu_args'] + qemu_args = qemu_args.replace('\n',' ').replace('\r','') + # Remove any empty strings from list. + self._config['extra_args'] = [x for x in qemu_args.split(' ') if x] + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=None): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string = self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout = self.socket_timeout + if not kvm_available(self.arch): + timeout *= 8 + self.console_init(timeout=timeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=None, sha512sum=None): def check_sha256sum(fname): if not sha256sum: @@ -136,8 +213,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, - "-o", "ConnectTimeout=1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout={}".format(self._config["ssh_timeout"]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -186,16 +264,16 @@ class BaseVM(object): "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)] def boot(self, img, extra_args=[]): - args = self._args + [ - "-device", "VGA", - "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img, - "-device", "virtio-blk,drive=drive0,bootindex=0"] - args += self._data_args + extra_args + boot_dev = BOOT_DEVICE[self._config['boot_dev_type']] + boot_params = boot_dev.format(img) + args = self._args + boot_params.split(' ') + args += self._data_args + extra_args + self._config['extra_args'] + args += ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch) guest = QEMUMachine(binary=qemu_bin, args=args, console_log=self._console_log_path) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -307,7 +385,8 @@ class BaseVM(object): self.console_send(command) def console_ssh_init(self, prompt, user, pw): - sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.rstrip() + sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -437,15 +516,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() -def main(vmcls): +def main(vmcls, config=None): try: + if config == None: + config = {} args, argv = parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) - vm = vmcls(debug=args.debug, vcpus=args.jobs, + vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config, log_console=args.log_console) if args.build_image: if os.path.exists(args.image) and not args.force: From patchwork Wed Feb 19 16:35:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240890 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=uxrEMRPD; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N40g3MmFz9sPk for ; Thu, 20 Feb 2020 04:07:47 +1100 (AEDT) Received: from localhost ([::1]:56930 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4Sp7-0008Tv-DB for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:07:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41990) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOJ-0003CF-7E for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOH-00085D-Eh for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:03 -0500 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]:33963) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOH-00084f-6a for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:01 -0500 Received: by mail-pj1-x1044.google.com with SMTP id f2so258273pjq.1 for ; Wed, 19 Feb 2020 08:40:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7RppIHsUBx+sMUATckkjKS/fMgtzYfrRf6rL6C92oB4=; b=uxrEMRPDI4OXMJQZdOigni0/MA+4Xdv2uZiAysEpP0kA0+6JpNw+XmQaoK7TPjLNs0 wP5zZJZNbZTuKfAQx1VRmOeGvu0TI2pVB57i3ubYpiut+2isqnYMJjaJdF2xS4EThn36 XeB5YtQv7xJkSkU5d3yfemMTqIVWcHoAXNPiynyrlqUwHcGN+Cxduswb3bWr8EL4FAiD 7rPzXHDL2hKBHaSwtQNyD7BYExxhroRdQ6u1LbpZ8pCkpD4NUJ76oLKfk44Uns2WJz4c 21r9CsiqXyQwb/vPUgif3JDnlUS6Ykgz5ZJaNQ3EQpAV6Mg/y2D6jNCEwXG2ej/JNv4c Tfaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7RppIHsUBx+sMUATckkjKS/fMgtzYfrRf6rL6C92oB4=; b=qXvEE13bYzHJJoVvraBGKfS382l8nHXcf+Dhb1shTeyGAYHxwm5stGEHGqfZqD/+Hy LybPEBkeXW1fX2NQ6l8cocOwgFQrcwB5hxDFjWb59CsMiY1xzxr6WKHmks8k6QLtez7G 5yK6BCL21k2yCIkSdY0kRgWV6JOdb+QepzZkc90wO9Ht7xBzr14J9GXbSAYgz5seJvv2 W9bbyoi6c+XD7Hc5CRdow1ASVO9jqaGlna8qEXse3ftAN8B1ITP/QHrI4qL5KusXOmco a9zrBwKwGUG+/T6zPcGRZRDDa56s4QF55wEUAj8b9mEJ7j52I5S8gFGeYqU0vIZaxedO XWHg== X-Gm-Message-State: APjAAAUwtd1EjbM/zTemNJn7VsdpI607kUPOobJjoj6WbZk1E3i/hiSL x94bxsU4Dd9wy5h665mu4bYijwH/XXU= X-Google-Smtp-Source: APXvYqzXKW/487/6schOn0OS0C7pXXKaHXKL1vcuyhZnmkcq+1PvlF+zqRStFrDliyJhDRhzqOhQGw== X-Received: by 2002:a17:902:fe10:: with SMTP id g16mr26507709plj.93.1582130399673; Wed, 19 Feb 2020 08:39:59 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:39:59 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 08/14] tests/vm: Added configuration file support Date: Wed, 19 Feb 2020 11:35:31 -0500 Message-Id: <20200219163537.22098-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1044 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example conf_example_aarch64.yml and conf_example_x86.yml. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 2 ++ tests/vm/basevm.py | 29 +++++++++++++++++- tests/vm/conf_example_aarch64.yml | 51 +++++++++++++++++++++++++++++++ tests/vm/conf_example_x86.yml | 50 ++++++++++++++++++++++++++++++ 4 files changed, 131 insertions(+), 1 deletion(-) create mode 100644 tests/vm/conf_example_aarch64.yml create mode 100644 tests/vm/conf_example_x86.yml diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 76a1bd7fe8..e19072f303 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -36,6 +36,8 @@ vm-help vm-test: @echo " V=1 - Enable verbose ouput on host and guest commands" @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool" + @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM configuration .yml file." + @echo " See conf_example_*.yml for file format details." vm-build-all: $(addprefix vm-build-, $(IMAGES)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index ed1b56bcea..a24ce090c7 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -29,6 +29,7 @@ import tempfile import shutil import multiprocessing import traceback +import yaml SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), "..", "keys", "id_rsa") @@ -474,9 +475,31 @@ class BaseVM(object): cwd=cidir, stdin=self._devnull, stdout=self._stdout, stderr=self._stdout) - return os.path.join(cidir, "cloud-init.iso") +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file = args.config + elif 'QEMU_CONFIG' in os.environ: + config_file = os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file)) + with open(config_file) as f: + yaml_dict = yaml.safe_load(f) + + if 'qemu-conf' in yaml_dict: + config.update(yaml_dict['qemu-conf']) + else: + raise Exception("config file {} is not valid"\ + " missing qemu-conf".format(config_file)) + return config + def parse_args(vmcls): def get_default_jobs(): @@ -513,6 +536,9 @@ def parse_args(vmcls): help="run tests with a snapshot") parser.add_option("--log-console", action="store_true", help="Log console to file.") + parser.add_option("--config", "-c", default=None, + help="Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() @@ -524,6 +550,7 @@ def main(vmcls, config=None): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config = parse_config(config, args) logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config, diff --git a/tests/vm/conf_example_aarch64.yml b/tests/vm/conf_example_aarch64.yml new file mode 100644 index 0000000000..9d44ae356f --- /dev/null +++ b/tests/vm/conf_example_aarch64.yml @@ -0,0 +1,51 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defaults. + + # Login username and password(has to be sudo enabled) + guest_user: qemu + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic-version=max + memory: 16G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=16,sockets=2,cores=8 + -numa node,cpus=0-3,nodeid=0 -numa node,cpus=4-7,nodeid=1 + -numa node,cpus=8-11,nodeid=2 -numa node,cpus=12-15,nodeid=3 + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15 + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20 + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + + # Or to skip the install entirely, just provide "" + #install_cmds: "" diff --git a/tests/vm/conf_example_x86.yml b/tests/vm/conf_example_x86.yml new file mode 100644 index 0000000000..78d3f5830f --- /dev/null +++ b/tests/vm/conf_example_x86.yml @@ -0,0 +1,50 @@ +# +# Example yaml for use by any of the x86 based scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +qemu-conf: + + # If any of the below are not provided, we will just use the qemu defaults. + + # Login username and password(has to be sudo enabled) + guest_user: "qemu" + guest_pass: "qemupass" + + # Password for root user can be different from guest. + root_pass: "qemupass" + + # Provide default ssh keys of current user. + # You need to edit the below for your user. + #ssh_key_file: /home//.ssh/id_rsa + #ssh_pub_key_file: /home//.ssh/id_rsa.pub + + cpu: max + machine: pc + memory: 8G + + # The below is a example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=8,sockets=2,cores=4 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node0 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node1 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node2 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node3 + -numa node,cpus=0-1,nodeid=0 -numa node,cpus=2-3,nodeid=1 + -numa node,cpus=4-5,nodeid=2 -numa node,cpus=6-7,nodeid=3 + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15 + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20 + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: "1.234.567.89" + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + boot_dev_type: "block" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + ssh_port: 5555 From patchwork Wed Feb 19 16:35:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240881 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=GwQslVBx; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3pL6ljZz9sPk for ; Thu, 20 Feb 2020 03:58:50 +1100 (AEDT) Received: from localhost ([::1]:56570 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SgS-0001MD-SI for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:58:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42004) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOK-0003Es-76 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOJ-00086I-1f for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:04 -0500 Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]:36881) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOI-00085o-Pv for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:02 -0500 Received: by mail-pj1-x102f.google.com with SMTP id m13so295733pjb.2 for ; Wed, 19 Feb 2020 08:40:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NVtYYcA86qI8H+IkdoM3WPRii1ViHqB0RBWen0zzsK8=; b=GwQslVBxRCVMfrcvu8lCsen1zU+6Lg4ioTdLGsf4+4WJvKeN7yvl+bPlQqlTJ4g6dC 3cpG7Ok4+QipnhY5B7Fucek3+UTdTNaXNeqn9GQ+z+6p01GdepsafN0ZvR1noSSbrXew ZWGMw2cJA8caR0F0rnSR5ziOnY82cgTUNZ20XIvT8eQW+oFFb4L7F5QGcYNW3rPQvREe 7vd4NpXbgQDLJyABCAFqr8Y4T5QIDSmrE+BLwto5FJoFTSamvoXVf8AhUeQ3V5YIS5G5 M3X3ThTb6cfaG55Ku8HWrwrfNenSiq2rd6s1q5MtYFx170BtiphkrQy3NBqFLSwblHU0 sozA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NVtYYcA86qI8H+IkdoM3WPRii1ViHqB0RBWen0zzsK8=; b=BfjxaJTjG6TaP1ZQA8pxmOb+3O6wfnwopeHNumJkSJk6P0EBFtujnCK2RjiA5Z8HBA 50RPXtiW+s9k54RUhW6bO8rwl17nb/onoJQbZ7derGr65yIXcfZSv5SY91CsaApaxPkd jFyXH+RyrU7UMvVyKnVoU30co/8yZUh8ALfcpOUNXXtZDEdyupMbSFsLVtX3/G1ilDf+ 2bRKir0b6xocAMQ/fCgz3Erty54xo0UUFEtPXZ3TrJTI0kWruNHMPYMkx+ZV2rZIq46S xR6fHD3coIAY3S/hJJrz5GkgfV4EQ+krUKmQf8wyC3KPwxrZt2N7r7jJLYEAH+5NxorQ 3Qmw== X-Gm-Message-State: APjAAAUzotjc8cCc95e9TuOHUAMpUYQwkfUMvy2lQHNgA1Kx10He5TJT 4vTOe4rlLrry5sv/VzL2nxZnV8w/RmI= X-Google-Smtp-Source: APXvYqzw0U9+DbhzEMsD8562Nk+2QYMAfHN6MlZ2hwXaJOPwm0Yrtt6DnCLc52Vx8H7sQQLgdHetDA== X-Received: by 2002:a17:902:82cc:: with SMTP id u12mr24677029plz.342.1582130401447; Wed, 19 Feb 2020 08:40:01 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.39.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:00 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 09/14] tests/vm: add --boot-console switch Date: Wed, 19 Feb 2020 11:35:32 -0500 Message-Id: <20200219163537.22098-10-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::102f X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Added ability to view console during boot via --boot-console switch to basevm.py. This helps debug issues that occur during the boot sequence. Also added a new special variable to vm-build: BOOT_CONSOLE=1 will cause this new --boot-console switch to be set. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 11 +++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index e19072f303..d1444c99f5 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -32,6 +32,7 @@ vm-help vm-test: @echo ' EXTRA_CONFIGURE_OPTS="..."' @echo " J=[0..9]* - Override the -jN parameter for make commands" @echo " DEBUG=1 - Enable verbose output on host and interactive debugging" + @echo " BOOT_CONSOLE=1 - Show the console output at boot time. " @echo " LOG_CONSOLE=1 - Log console to file in: ~/.cache/qemu-vm " @echo " V=1 - Enable verbose ouput on host and guest commands" @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @@ -52,6 +53,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(PYTHON) $< \ $(if $(V)$(DEBUG), --debug) \ $(if $(LOG_CONSOLE),--log-console) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$@" \ --force \ --build-image $@, \ @@ -67,6 +69,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(V),--verbose) \ $(if $(LOG_CONSOLE),--log-console) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -88,6 +91,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ $(if $(LOG_CONSOLE),--log-console) \ + $(if $(BOOT_CONSOLE),--boot-console) \ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index a24ce090c7..dc94d1988e 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -89,8 +89,10 @@ class BaseVM(object): # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 def __init__(self, debug=False, vcpus=None, config=None, - log_console=False): + log_console=False, + boot_console=None): self._guest = None + self._boot_console = boot_console # Allow input config to override defaults. self._config = DEFAULT_CONFIG.copy() if config != None: @@ -539,6 +541,8 @@ def parse_args(vmcls): parser.add_option("--config", "-c", default=None, help="Provide config yaml for configuration. "\ "See config_example.yaml for example.") + parser.add_option("--boot-console", action="store_true", + help="Show console during boot. ") parser.disable_interspersed_args() return parser.parse_args() @@ -554,7 +558,8 @@ def main(vmcls, config=None): logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config, - log_console=args.log_console) + log_console=args.log_console, + boot_console=args.boot_console) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.image, @@ -574,6 +579,8 @@ def main(vmcls, config=None): if args.snapshot: img += ",snapshot=on" vm.boot(img) + if vm._boot_console: + vm.wait_boot() vm.wait_ssh() except Exception as e: if isinstance(e, SystemExit) and e.code == 0: From patchwork Wed Feb 19 16:35:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240865 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=ciKTGZ4Q; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3dr1rZKz9sRf for ; Thu, 20 Feb 2020 03:51:28 +1100 (AEDT) Received: from localhost ([::1]:56366 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SZK-0002HI-4u for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:51:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42028) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOM-0003KU-A0 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOK-00088R-Ry for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:06 -0500 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]:38153) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOK-000876-Lo for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:04 -0500 Received: by mail-pg1-x534.google.com with SMTP id d6so365064pgn.5 for ; Wed, 19 Feb 2020 08:40:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BQyiXXZMIOFZUP1qaPFL0Wmx1PB9lf9yghpcYgHV7To=; b=ciKTGZ4Q8GDYZbdxsLSo0G1nzkLy0TzfmU2+ZlrNU3KMIC6fhqV0ISpu7hMIci0H1t CPnWz/bNpmckN613SPebD0GNY1EEfm6Vn/VcDMeyQ5slRKTIVTqHk9Gxr9Goc6GbeLSn 9phoZUoLY84yEjthWFCncSHCMdaCnI1paVh3+Hzm9ShkOt2GSPCm0SieCCDPyvo0GkRi xZtDGRrHvszXaPWFn1oQ2xxW6nE4tQSl4JtWCh05WybWh8oVoQXR0cbAKZRKR+w3esUR uXrZLBUiDSZnxBmWS7ljb/5F5bvnhYkBi6sz06DUo3rN0bR3RWkNqdkp1pORXE3f0QNw P9Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BQyiXXZMIOFZUP1qaPFL0Wmx1PB9lf9yghpcYgHV7To=; b=lc9h7ivNm/JyZN0zp1h3Vp6vIea/FF+dbdXIzdHRv++FezSEf9gmw/tUnRbg5VHSJS WS4aB9FHnHgXsZfD/7uLkMakxIpeR5UbPLtFC2eB8ABLgBIn9dapN3oEEo9t4PCrlB00 ZJ/rHdshfewHP4wfLFA3q8GQ4JfW0gcxOQkpiaDmk0Z0bjP5F6nBfuK8R7+MgQe0VHG3 OwTJ84IIwA6Tu28gYP2V/7k/a3mBHqt236ImTboYuIGE4LYxNXvGs6KBlrvZpbiio5vi bVU2Dx29PoTeWrY+/P1EeJ0fHDpodZ/dfN2ou0r87nRJ7YeQmreI0+ME3KcqwyFTBqI2 p4Vw== X-Gm-Message-State: APjAAAWf/k81gRJg3bfEsmcW+U8PH0jp/A/VEG58lVIEv2Um7vDii0te RKBQjrvPEhWqFBwe1UoiBAc2rG8Doag= X-Google-Smtp-Source: APXvYqwmqvacRdSBjzWzSGFY8B6wU1o9ViJi1fLpjefEc+Tr4L/8w5UfIOX8zIGiNCH3VKrKLk8zig== X-Received: by 2002:aa7:82c9:: with SMTP id f9mr28096221pfn.168.1582130403244; Wed, 19 Feb 2020 08:40:03 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.40.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:02 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 10/14] tests/vm: Add ability to select QEMU from current build. Date: Wed, 19 Feb 2020 11:35:33 -0500 Message-Id: <20200219163537.22098-11-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::534 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Added a new special variable QEMU_LOCAL=1, which will indicate to take the QEMU binary from the current build. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 4 ++++ tests/vm/basevm.py | 29 ++++++++++++++++++++++++----- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index d1444c99f5..7557c154a1 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -35,6 +35,7 @@ vm-help vm-test: @echo " BOOT_CONSOLE=1 - Show the console output at boot time. " @echo " LOG_CONSOLE=1 - Log console to file in: ~/.cache/qemu-vm " @echo " V=1 - Enable verbose ouput on host and guest commands" + @echo " QEMU_LOCAL=1 - Use QEMU binary local to this build." @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool" @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM configuration .yml file." @@ -54,6 +55,7 @@ $(IMAGES_DIR)/%.img: $(SRC_PATH)/tests/vm/% \ $(if $(V)$(DEBUG), --debug) \ $(if $(LOG_CONSOLE),--log-console) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$@" \ --force \ --build-image $@, \ @@ -70,6 +72,7 @@ vm-build-%: $(IMAGES_DIR)/%.img $(if $(V),--verbose) \ $(if $(LOG_CONSOLE),--log-console) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ $(if $(BUILD_TARGET),--build-target $(BUILD_TARGET)) \ --snapshot \ @@ -92,6 +95,7 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(if $(J),--jobs $(J)) \ $(if $(LOG_CONSOLE),--log-console) \ $(if $(BOOT_CONSOLE),--boot-console) \ + --build-path $(BUILD_DIR)\ --image "$<" \ --interactive \ false, \ diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index dc94d1988e..a30a641a4a 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -89,9 +89,9 @@ class BaseVM(object): # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 def __init__(self, debug=False, vcpus=None, config=None, - log_console=False, - boot_console=None): + log_console=False, boot_console=False, build_path=None): self._guest = None + self._build_path = build_path self._boot_console = boot_console # Allow input config to override defaults. self._config = DEFAULT_CONFIG.copy() @@ -273,8 +273,8 @@ class BaseVM(object): args += self._data_args + extra_args + self._config['extra_args'] args += ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) - qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch) - guest = QEMUMachine(binary=qemu_bin, args=args, + qemu_path = get_qemu_path(self.arch, self._build_path) + guest = QEMUMachine(binary=qemu_path, args=args, console_log=self._console_log_path) guest.set_machine(self._config['machine']) guest.set_console() @@ -479,6 +479,22 @@ class BaseVM(object): stderr=self._stdout) return os.path.join(cidir, "cloud-init.iso") +def get_qemu_path(arch, build_path=None): + """Fetch the path to the qemu binary.""" + qemu_local = os.environ.get("QEMU_LOCAL", 0) + # If QEMU environment variable set, it takes precedence + if "QEMU" in os.environ: + qemu_path = os.environ["QEMU"] + elif qemu_local: + if not build_path: + raise Exception("--build-path option required with QEMU_LOCAL") + qemu_path = os.path.join(build_path, arch + "-softmmu") + qemu_path = os.path.join(qemu_path, "qemu-system-" + arch) + else: + # Default is to use system path for qemu. + qemu_path = "qemu-system-" + arch + return qemu_path + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the @@ -543,6 +559,8 @@ def parse_args(vmcls): "See config_example.yaml for example.") parser.add_option("--boot-console", action="store_true", help="Show console during boot. ") + parser.add_option("--build-path", default=None, + help="Path of build directory. ") parser.disable_interspersed_args() return parser.parse_args() @@ -559,7 +577,8 @@ def main(vmcls, config=None): else logging.WARN)) vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config, log_console=args.log_console, - boot_console=args.boot_console) + boot_console=args.boot_console, + build_path=args.build_path) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.image, From patchwork Wed Feb 19 16:35:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240891 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=HwD26yBe; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N4253Dbkz9sPk for ; Thu, 20 Feb 2020 04:09:01 +1100 (AEDT) Received: from localhost ([::1]:56968 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SqJ-0001vR-81 for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:08:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42041) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SON-0003Nk-Ij for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOM-00089b-Hi for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:07 -0500 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]:43952) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOM-000894-Bt for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:06 -0500 Received: by mail-pf1-x429.google.com with SMTP id s1so313726pfh.10 for ; Wed, 19 Feb 2020 08:40:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=A8rtmh1NdudL2G95OwMBtm9/xjxDOXwkQuwmUBhd1bQ=; b=HwD26yBelM7lMNjn4n1K7sKadWa+ltco+M001VIZjD2iIpBG/cFRYvaOe1iD6yJT4v NP83vgBDMLGC6Jx8V0W+nNaH7YMb5Dt3Tx9C9DJP95BjToA0M3hmP3qdVMu5vweJkYYf s+MC9Z3N5TfR0ipxiRXJGrSDiQd1WnVCOD0rS6yRdSE7+DfEyCL/2HAY3wlmLqV5Ood1 NP0TsYT/sxDP/1RS+FCbu1LG2dYDuqy9dj0GyD1lREFV5qoHxQQYNLYhZxxLqVnQ91ox V0zJJuL2vUgMwxfNcCxNQESuL6qLd5tGjHHoU6RMjm9hDNY0ar7gcbY/bWZaK8wzvN2r aX7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=A8rtmh1NdudL2G95OwMBtm9/xjxDOXwkQuwmUBhd1bQ=; b=MbI5qSJUNENMA41Eb5rEpz8lsXuRpXTlgRVTpHs1v/XcNY2NXsFrpJE2WUN14rk5so VCk7hZ3QykQS9kKspLzS8YO2FKxn7/s43u/9aBJZnJQbtlMP2LxcXylziyOQYlEW6Vgb YPjmYtUyJCXw/Qj0hxb88SfKqaZfLtxLwgv/c3ZmA+a4qL/O6xej+lucizOuvornlMI0 NqnkLEmVYxqDqvNUmkLpOjPvFxMW5QFe4Qge18DUn2VDhEl4eeKukeBnBvuYaGsm11C2 TYq1LTCP7aL9E7KSIc+AXxaQdufjgSaP0ZWS3qw6PYqXwDNuM2FDnNto5qCELpiqTVgn 5xPA== X-Gm-Message-State: APjAAAXZuVGhr2ausaNdthr/BLvSsZjGKR7aQDxAWPXRlTt2pWp/yuHm g2QBS06vz7qhNHsqY6bVY8dX9Hij8+c= X-Google-Smtp-Source: APXvYqynVgvgxaGOLSmHP/LoB81y2ayYh5aaI/+qlE50FO/oIXD2E893rFhhNFJcBuC4SFc6n26BXA== X-Received: by 2002:aa7:9a8b:: with SMTP id w11mr28685182pfi.38.1582130405106; Wed, 19 Feb 2020 08:40:05 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.40.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:04 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 11/14] tests/vm: allow wait_ssh() to specify command Date: Wed, 19 Feb 2020 11:35:34 -0500 Message-Id: <20200219163537.22098-12-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::429 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This allows for waiting for completion of arbitrary commands. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index a30a641a4a..792e4a3fb2 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -407,24 +407,24 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) - def wait_ssh(self, wait_root=False, seconds=300): + def wait_ssh(self, wait_root=False, seconds=300, cmd="exit 0"): # Allow more time for VM to boot under TCG. if not kvm_available(self.arch): seconds *= self.tcg_ssh_timeout_multiplier starttime = datetime.datetime.now() endtime = starttime + datetime.timedelta(seconds=seconds) - guest_up = False + cmd_success = False while datetime.datetime.now() < endtime: - if wait_root and self.ssh_root("exit 0") == 0: - guest_up = True + if wait_root and self.ssh_root(cmd) == 0: + cmd_success = True break - elif self.ssh("exit 0") == 0: - guest_up = True + elif self.ssh(cmd) == 0: + cmd_success = True break seconds = (endtime - datetime.datetime.now()).total_seconds() logging.debug("%ds before timeout", seconds) time.sleep(1) - if not guest_up: + if not cmd_success: raise Exception("Timeout while waiting for guest ssh") def shutdown(self): From patchwork Wed Feb 19 16:35:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240870 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=fpF6qa9n; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3gj2lH9z9sR4 for ; Thu, 20 Feb 2020 03:53:05 +1100 (AEDT) Received: from localhost ([::1]:56402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4Sat-0005in-2j for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:53:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42068) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOQ-0003Ut-R1 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOO-0008BY-Mw for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:10 -0500 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:35901) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOO-0008B7-Dr for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:08 -0500 Received: by mail-pf1-x42a.google.com with SMTP id 185so332156pfv.3 for ; Wed, 19 Feb 2020 08:40:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NX9h8pD+23hYiUkVbRsk8kJJ4NVy28DePqm6/XUwIAo=; b=fpF6qa9neTsqXj8bEMartarTfACRvdZyqKUH26PtlvUrvgL/npoSKrsOcPt7Gtm3Tz EGNdn4nUScf7ahCtVNvsVeVH3RPdLYEZY91VmcojOuGzYwbfjtE0zSqPByTuqk0qbklF 6x4aPJa4TaytZ3c3fy0IaiLUv5j1IF6LqBbzFXHoaxBq06IN1VPo2p/Q7BNVJbgmOhxA UzXe9PsdBqJfdgx4yocwdbLRvk8ZCLnFBQiONfpMr2hOkFyg7CBmOLeTESx6OdJOeWSO F5nFQJDQndZnHznVfOYLZ0Z5yZRD7VVlddVz4YKkgTAyl+ObWLZuevj7zF5xZAkKTwD7 BVgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NX9h8pD+23hYiUkVbRsk8kJJ4NVy28DePqm6/XUwIAo=; b=FJa0p0SMHtP0drwpj0Y93LMXgALFkgG6jwt1SPqDcYZfOdPJnod93ffQBSPhU3/Q1g FbdcddoIkWDgB8vPusiC5KZ2Yr93s1mbpr+29sqHC4YPfMUCNUHWBOSgdZBEDYy5LfSl EHoH55Wuz+6aLtL0YxfWEwb2jH8wBXLnA+tXodwGCqMzuFnS/LOM60/LWU/Q8EgwJvUc +DJWcnrvc09mfZOZmOALgL2QZkbDIARSXBX8v63p3+2my5btpFFp1ZLtqttj+8mnTwoy l0cX60FvjVYud4uVppKPemXm3CITp0mGBsGNeIxfsxGOeVJegLRylrHUFYOcZomv2rnb MGMg== X-Gm-Message-State: APjAAAWhSRs39ClYEEHzix5Xy8fyuXrANI/ubb3G8h3HEU1mR3TlYwyF z0BJ+1E2G1Z1vaaLc9KKvvQuSSPX9nI= X-Google-Smtp-Source: APXvYqyCw2LWtfckRKv8ZVgMs/qhm0Ia96KXNsJEQgf7TQmU47nQQe3S0m8PqcrSerNuNaftgwPeaQ== X-Received: by 2002:a63:7453:: with SMTP id e19mr1078954pgn.50.1582130406869; Wed, 19 Feb 2020 08:40:06 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.40.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:06 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 12/14] tests/vm: Added a new script for ubuntu.aarch64. Date: Wed, 19 Feb 2020 11:35:35 -0500 Message-Id: <20200219163537.22098-13-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42a X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM. Another new file is also added aarch64vm.py, which is a module with common methods used by aarch64 VMs, such as how to create the flash images. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 3 +- tests/vm/aarch64vm.py | 100 ++++++++++++++++++++++++++++++++ tests/vm/basevm.py | 8 +++ tests/vm/ubuntu.aarch64 | 117 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 tests/vm/aarch64vm.py create mode 100755 tests/vm/ubuntu.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 7557c154a1..9dfa517936 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -2,7 +2,7 @@ .PHONY: vm-build-all vm-clean-all -IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora +IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 IMAGES_DIR := $(HOME)/.cache/qemu-vm/images IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES)) @@ -18,6 +18,7 @@ vm-help vm-test: @echo " vm-build-openbsd - Build QEMU in OpenBSD VM" @echo " vm-build-centos - Build QEMU in CentOS VM, with Docker" @echo " vm-build-fedora - Build QEMU in Fedora VM" + @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 VM" @echo "" @echo " vm-build-all - Build QEMU in all VMs" @echo " vm-clean-all - Clean up VM images" diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py new file mode 100644 index 0000000000..fd48735b3a --- /dev/null +++ b/tests/vm/aarch64vm.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +# +# VM testing aarch64 library +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available + +# This is the config needed for current version of QEMU. +# This works for both kvm and tcg. +CURRENT_CONFIG = { + 'cpu' : "max", + 'machine' : "virt,gic-version=max", +} + +# The minimum minor version of QEMU we will support with aarch64 VMs is 3. +# QEMU versions less than 3 have various issues running these VMs. +QEMU_AARCH64_MIN_VERSION = 3 + +# The DEFAULT_CONFIG will default to a version of +# parameters that works for backwards compatibility. +DEFAULT_CONFIG = {'kvm' : {'cpu' : "host", + 'machine' : "virt,gic-version=host"}, + 'tcg' : {'cpu' : "cortex-a57", + 'machine' : "virt"}, +} + +def get_config_defaults(vmcls, default_config): + """Fetch the configuration defaults for this VM, + taking into consideration the defaults for + aarch64 first, followed by the defaults for this VM.""" + config = default_config + config.update(aarch_get_config_defaults(vmcls)) + return config + +def aarch_get_config_defaults(vmcls): + # Set the defaults for current version of QEMU. + config = CURRENT_CONFIG + args, argv = basevm.parse_args(vmcls) + qemu_path = basevm.get_qemu_path(vmcls.arch, args.build_path) + qemu_version = basevm.get_qemu_version(qemu_path) + if qemu_version < QEMU_AARCH64_MIN_VERSION: + error = "\nThis major version of QEMU {} is to old for aarch64 VMs.\n"\ + "The major version must be at least {}.\n"\ + "To continue with the current build of QEMU, "\ + "please restart with QEMU_LOCAL=1 .\n" + print(error.format(qemu_version, QEMU_AARCH64_MIN_VERSION)) + exit(1) + if qemu_version == QEMU_AARCH64_MIN_VERSION: + # We have an older version of QEMU, + # set the config values for backwards compatibility. + if kvm_available('aarch64'): + config.update(DEFAULT_CONFIG['kvm']) + else: + config.update(DEFAULT_CONFIG['tcg']) + return config + +def create_flash_images(flash_dir="./"): + """Creates the appropriate pflash files + for an aarch64 VM.""" + flash0_path = get_flash_path(flash_dir, "flash0") + flash1_path = get_flash_path(flash_dir, "flash1") + subprocess.check_call(["dd", "if=/dev/zero", "of={}".format(flash0_path), + "bs=1M", "count=64"]) + # A reliable way to get the QEMU EFI image is via an installed package. + efi_img = "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" + if not os.path.exists(efi_img): + sys.stderr.write("*** {} is missing\n".format(efi_img)) + sys.stderr.write("*** please install qemu-efi-aarch64 package\n") + exit(3) + subprocess.check_call(["dd", "if={}".format(efi_img), + "of={}".format(flash0_path), + "conv=notrunc"]) + subprocess.check_call(["dd", "if=/dev/zero", + "of={}".format(flash1_path), + "bs=1M", "count=64"]) + +def get_pflash_args(flash_dir="./"): + """Returns a string that can be used to + boot qemu using the appropriate pflash files + for aarch64.""" + flash0_path = get_flash_path(flash_dir, "flash0") + flash1_path = get_flash_path(flash_dir, "flash1") + pflash_args_str = "-drive file={},format=raw,if=pflash "\ + "-drive file={},format=raw,if=pflash" + pflash_args = pflash_args_str.format(flash0_path, flash1_path) + return pflash_args.split(" ") + +def get_flash_path(flash_dir, name): + return os.path.join(flash_dir, "{}.img".format(name)) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 792e4a3fb2..6ed04a2a4b 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -495,6 +495,14 @@ def get_qemu_path(arch, build_path=None): qemu_path = "qemu-system-" + arch return qemu_path +def get_qemu_version(qemu_path): + """Get the version number from the current QEMU, + and return the major number.""" + output = subprocess.check_output([qemu_path, '--version']) + version_line = output.decode("utf-8") + version_num = re.split(' |\(', version_line)[3].split('.')[0] + return int(version_num) + def parse_config(config, args): """ Parse yaml config and populate our config structure. The yaml config allows the user to override the diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64 new file mode 100755 index 0000000000..0413971d33 --- /dev/null +++ b/tests/vm/ubuntu.aarch64 @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +# +# Ubuntu aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.i386 Fam Zheng +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +from qemu.accel import kvm_available +import time +import aarch64vm + +DEFAULT_CONFIG = { + 'cpu' : "cortex-a57", + 'machine' : "virt,gic-version=3", + 'install_cmds' : "apt-get update,"\ + "apt-get build-dep -y qemu,"\ + "apt-get install -y libfdt-dev flex bison", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM + # especially using softmmu. + 'ssh_timeout' : 60, +} + +class UbuntuAarch64VM(basevm.BaseVM): + name = "ubuntu.aarch64" + arch = "aarch64" + image_name = "ubuntu-18.04-server-cloudimg-arm64.img" + image_link = "https://cloud-images.ubuntu.com/releases/18.04/release/" + image_name + login_prompt = "ubuntu-aarch64-guest login:" + BUILD_SCRIPT = """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def boot(self, img, extra_args=None): + aarch64vm.create_flash_images(self._tmpdir) + default_args = aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args = default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to change it. + extra_args.extend(["-smp", "8"]) + + # We have overridden boot() since aarch64 has additional parameters. + # Call down to the base class method. + super(UbuntuAarch64VM, self).boot(img, extra_args=extra_args) + + def build_image(self, img): + os_img = self._download_with_cache(self.image_link) + img_tmp = img + ".tmp" + subprocess.check_call(["cp", "-f", os_img, img_tmp]) + subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"]) + ci_img = self.gen_cloud_init_iso() + + self.boot(img_tmp, extra_args = ["-cdrom", ci_img]) + if self._boot_console: + self.wait_boot() + # First command we issue is fix for slow ssh login. + self.wait_ssh(wait_root=True, + cmd="chmod -x /etc/update-motd.d/*") + # Wait for cloud init to finish + self.wait_ssh(wait_root=True, + cmd="ls /var/lib/cloud/instance/boot-finished") + self.ssh_root("touch /etc/cloud/cloud-init.disabled") + # Disable auto upgrades. + # We want to keep the VM system state stable. + self.ssh_root('sed -ie \'s/"1"/"0"/g\' /etc/apt/apt.conf.d/20auto-upgrades') + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] != "": + self.ssh_root("sync") + # Shutdown and then boot it again. + # Allows us to know for sure it is booting (not shutting down) + # before we call wait_ssh(). + self.graceful_shutdown() + self.boot(img_tmp) + if self._boot_console: + self.wait_boot() + self.wait_ssh(wait_root=True) + self.wait_ssh(wait_root=True, cmd="locale") + # The previous update sometimes doesn't survive a reboot, so do it again + self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g /etc/apt/sources.list") + + # Issue the install commands. + # This can be overriden by the user in the config .yml. + install_cmds = self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.graceful_shutdown() + self.wait() + os.rename(img_tmp, img) + return 0 + +if __name__ == "__main__": + defaults = aarch64vm.get_config_defaults(UbuntuAarch64VM, DEFAULT_CONFIG) + sys.exit(basevm.main(UbuntuAarch64VM, defaults)) From patchwork Wed Feb 19 16:35:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240888 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=mcTDaPA/; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3xR090Wz9sRf for ; Thu, 20 Feb 2020 04:04:57 +1100 (AEDT) Received: from localhost ([::1]:56874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SmM-0004Do-PM for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 12:04:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42084) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOS-0003ZU-W9 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOQ-0008Cl-Lm for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:12 -0500 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]:38144) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOQ-0008CG-DQ for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:10 -0500 Received: by mail-pg1-x52a.google.com with SMTP id d6so365188pgn.5 for ; Wed, 19 Feb 2020 08:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=U8AhPED6ldfYVbeLlC+RLh2FFh/ALLPiS0HEVCLcfeY=; b=mcTDaPA/mHo4iF8sV4E2MJ56/fJosZdt+1qjCoOG2YgapwbOFkQNkaRW5E/rYgGgu2 VmfOKJXmsvGTIX9Z7jPvoVm7tmevVBmt+/Dl4d/16VcD7kBBnpqA0sWHbxQ0sS0KDgOj wLtOy8nawFAlaiJSPT3GLnHWmLUqAMzD1JqJYzphNDmF74PChYAiqYKGYk7JSYOCZRwg NSYhNvHBt6cF1xHTc7TXJg85Cp3RB97TTfdp2eDLhpg5DkHCEsvmbMzq2NwzTuedxCiJ wP5Dg54P5WO1V13zbnKEygGYdFVcDbWF0msM2Db2i+ihJ72Uknny6ohTIWyG0pPkTC6A LqQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=U8AhPED6ldfYVbeLlC+RLh2FFh/ALLPiS0HEVCLcfeY=; b=kU8pLgNcLCzMd1ZC7m78du5jPzScvihQXm1tryn+BSDCSzX2pKbigN3EKbzP43LPId ZQ93z54yJncxkcWQfYybD8xZCyhljP8TiygWWnRq9EcWHCGNRmhtTJL/F23esHR1WM9z V5Qa7IavjlHJfBDUyjhC9MHaACYNa5xDtUErcAXUqufPGDj/FNwCf5RvpIO6p4EpNXem XKOpHhZAA7emdnuavyNX0o5EE0Ze7pepvHtG9RQTAvEa6hmiv7bRKj+e2jyPoXEG1NHo feZMkSjaIhI5Cv0DOdGKeXW0AlXeudaTnQ6AoGFzcA4p+oEoq75oFRUXboilLM+vCYS8 orcg== X-Gm-Message-State: APjAAAU5hoey/LVKHNLRhe27sV2tHFCBMZEcDa1B46+o/NZdtA5a0wHM 82Xfk2RommbJ9q+9uwBf2gSCKlCoN+Q= X-Google-Smtp-Source: APXvYqyKdIsuZNj8ufRAQoSrOs2cu+Wxd9xirckU/WVfsHnVjf82CO2UmCSDMhHFRdh4URLnZ044hw== X-Received: by 2002:a62:cfc1:: with SMTP id b184mr27278036pfg.55.1582130408674; Wed, 19 Feb 2020 08:40:08 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.40.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:08 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 13/14] tests/vm: Added a new script for centos.aarch64. Date: Wed, 19 Feb 2020 11:35:36 -0500 Message-Id: <20200219163537.22098-14-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52a X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" centos.aarch64 creates a CentOS 8 image. Also added a new kickstart script used to build the centos.aarch64 image. Signed-off-by: Robert Foley --- tests/vm/Makefile.include | 3 +- tests/vm/centos-8-aarch64.ks | 51 ++++++++ tests/vm/centos.aarch64 | 224 +++++++++++++++++++++++++++++++++++ 3 files changed, 277 insertions(+), 1 deletion(-) create mode 100644 tests/vm/centos-8-aarch64.ks create mode 100755 tests/vm/centos.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 9dfa517936..bb41bc255f 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -2,7 +2,7 @@ .PHONY: vm-build-all vm-clean-all -IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 +IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 centos.aarch64 IMAGES_DIR := $(HOME)/.cache/qemu-vm/images IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES)) @@ -19,6 +19,7 @@ vm-help vm-test: @echo " vm-build-centos - Build QEMU in CentOS VM, with Docker" @echo " vm-build-fedora - Build QEMU in Fedora VM" @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 VM" + @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 VM" @echo "" @echo " vm-build-all - Build QEMU in all VMs" @echo " vm-clean-all - Clean up VM images" diff --git a/tests/vm/centos-8-aarch64.ks b/tests/vm/centos-8-aarch64.ks new file mode 100644 index 0000000000..fd6ebe4d49 --- /dev/null +++ b/tests/vm/centos-8-aarch64.ks @@ -0,0 +1,51 @@ +# CentOS aarch64 image kickstart file. +# This file is used by the CentOS installer to +# script the generation of the image. +# +# Copyright 2020 Linaro +# +ignoredisk --only-use=vda +# System bootloader configuration +bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda +autopart --type=plain +# Partition clearing information +clearpart --linux --initlabel --drives=vda +# Use text mode install +text +repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream +# Use CDROM installation media +cdrom +# Keyboard layouts +keyboard --vckeymap=us --xlayouts='' +# System language +lang en_US.UTF-8 + +# Network information +network --bootproto=dhcp --device=enp0s1 --onboot=off --ipv6=auto --no-activate +network --hostname=localhost.localdomain +# Run the Setup Agent on first boot +firstboot --enable +# Do not configure the X Window System +skipx +# System services +services --enabled="chronyd" +# System timezone +timezone America/New_York --isUtc + +# Shutdown after installation is complete. +shutdown + +%packages +@^server-product-environment +kexec-tools + +%end + +%addon com_redhat_kdump --enable --reserve-mb='auto' + +%end +%anaconda +pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty +pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok +pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty +%end diff --git a/tests/vm/centos.aarch64 b/tests/vm/centos.aarch64 new file mode 100755 index 0000000000..c4a020dc4c --- /dev/null +++ b/tests/vm/centos.aarch64 @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +# +# Centos aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.aarch64 +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import traceback +import aarch64vm + +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : "virt,gic-version=max", + 'install_cmds' : "yum install -y make git python3 gcc gcc-c++ flex bison, "\ + "yum install -y glib2-devel pixman-devel zlib-devel, "\ + "yum install -y perl-Test-Harness, "\ + "sudo dnf config-manager "\ + "--add-repo=https://download.docker.com/linux/centos/docker-ce.repo,"\ + "sudo dnf install --nobest -y docker-ce.aarch64,"\ + "systemctl enable docker", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM. + 'ssh_timeout' : 60, +} + +class CentosAarch64VM(basevm.BaseVM): + name = "centos.aarch64" + arch = "aarch64" + login_prompt = "localhost login:" + prompt = '[root@localhost ~]#' + image_name = "CentOS-8-aarch64-1905-dvd1.iso" + image_link = "http://mirrors.usc.edu/pub/linux/distributions/centos/8.0.1905/isos/aarch64/" + image_link += image_name + BUILD_SCRIPT = """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def set_key_perm(self): + """Set permissions properly on certain files to allow + ssh access.""" + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v /root/.ssh\n") + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v "\ + "/home/{}/.ssh\n".format(self._config["guest_user"])) + + def create_kickstart(self): + """Generate the kickstart file used to generate the centos image.""" + # Start with the template for the kickstart. + ks_file = "../tests/vm/centos-8-aarch64.ks" + subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=True) + # Append the ssh keys to the kickstart file + # as the post processing phase of installation. + with open("ks.cfg", "a") as f: + # Add in the root pw and guest user. + rootpw = "rootpw --plaintext {}\n" + f.write(rootpw.format(self._config["root_pass"])) + add_user = "user --groups=wheel --name={} "\ + "--password={} --plaintext\n" + f.write(add_user.format(self._config["guest_user"], + self._config["guest_pass"])) + # Add the ssh keys. + f.write("%post --log=/root/ks-post.log\n") + f.write("mkdir -p /root/.ssh\n") + addkey = 'echo "{}" >> /root/.ssh/authorized_keys\n' + addkey_cmd = addkey.format(self._config["ssh_pub_key"]) + f.write(addkey_cmd) + f.write('mkdir -p /home/{}/.ssh\n'.format(self._config["guest_user"])) + addkey = 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' + addkey_cmd = addkey.format(self._config["ssh_pub_key"], + self._config["guest_user"]) + f.write(addkey_cmd) + f.write("%end\n") + # Take our kickstart file and create an .iso from it. + # The .iso will be provided to qemu as we boot + # from the install dvd. + # Anaconda will recognize the label "OEMDRV" and will + # start the automated installation. + gen_iso_img = 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' + subprocess.check_call(gen_iso_img, shell=True) + + def wait_for_shutdown(self): + """We wait for qemu to shutdown the VM and exit. + While this happens we display the console view + for easier debugging.""" + # The image creation is essentially done, + # so whether or not the wait is successful we want to + # wait for qemu to exit (the self.wait()) before we return. + try: + self.console_wait("reboot: Power down") + except Exception as e: + sys.stderr.write("Exception hit\n") + if isinstance(e, SystemExit) and e.code == 0: + return 0 + traceback.print_exc() + finally: + self.wait() + + def build_base_image(self, dest_img): + """Run through the centos installer to create + a base image with name dest_img.""" + # We create the temp image, and only rename + # to destination when we are done. + img = dest_img + ".tmp" + # Create an empty image. + # We will provide this as the install destination. + qemu_img_create = "qemu-img create {} 50G".format(img) + subprocess.check_call(qemu_img_create, shell=True) + + # Create our kickstart file to be fed to the installer. + self.create_kickstart() + # Boot the install dvd with the params as our ks.iso + os_img = self._download_with_cache(self.image_link) + dvd_iso = "centos-8-dvd.iso" + subprocess.check_call(["cp", "-f", os_img, dvd_iso]) + extra_args = "-cdrom ks.iso" + extra_args += " -drive file={},if=none,id=drive1,cache=writeback" + extra_args += " -device virtio-blk,drive=drive1,bootindex=1" + extra_args = extra_args.format(dvd_iso).split(" ") + self.boot(img, extra_args=extra_args) + self.console_wait_send("change the selection", "\n") + # We seem to need to hit esc (chr(27)) twice to abort the + # media check, which takes a long time. + # Waiting a bit seems to be more reliable before hitting esc. + self.console_wait("Checking") + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + print("Found Checking") + self.wait_for_shutdown() + os.rename(img, dest_img) + print("Done with base image build: {}".format(dest_img)) + + def check_create_base_img(self, img_base, img_dest): + """Create a base image using the installer. + We will use the base image if it exists. + This helps cut down on install time in case we + need to restart image creation, + since the base image creation can take a long time.""" + if not os.path.exists(img_base): + print("Generate new base image: {}".format(img_base)) + self.build_base_image(img_base); + else: + print("Use existing base image: {}".format(img_base)) + # Save a copy of the base image and copy it to dest. + # which we will use going forward. + subprocess.check_call(["cp", img_base, img_dest]) + + def boot(self, img, extra_args=None): + aarch64vm.create_flash_images(self._tmpdir) + default_args = aarch64vm.get_pflash_args(self._tmpdir) + if extra_args: + extra_args.extend(default_args) + else: + extra_args = default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to change it. + extra_args.extend(["-smp", "8"]) + # We have overridden boot() since aarch64 has additional parameters. + # Call down to the base class method. + super(CentosAarch64VM, self).boot(img, extra_args=extra_args) + + def build_image(self, img): + img_tmp = img + ".tmp" + self.check_create_base_img(img + ".base", img_tmp) + + # Boot the new image for the first time to finish installation. + self.boot(img_tmp) + self.console_init() + self.console_wait_send(self.login_prompt, "root\n") + self.console_wait_send("Password:", + "{}\n".format(self._config["root_pass"])) + + self.set_key_perm() + self.console_wait_send(self.prompt, "rpm -q centos-release\n") + enable_adapter = "sed -i 's/ONBOOT=no/ONBOOT=yes/g'" \ + " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" + self.console_wait_send(self.prompt, enable_adapter) + self.console_wait_send(self.prompt, "ifup enp0s1\n") + self.console_wait_send(self.prompt, + 'echo "qemu ALL=(ALL) NOPASSWD:ALL" | '\ + 'sudo tee /etc/sudoers.d/qemu\n') + self.console_wait(self.prompt) + + # Rest of the commands we issue through ssh. + self.wait_ssh(wait_root=True) + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] != "": + install_cmds = self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.ssh_root("poweroff") + self.wait_for_shutdown() + os.rename(img_tmp, img) + print("image creation complete: {}".format(img)) + return 0 + +if __name__ == "__main__": + defaults = aarch64vm.get_config_defaults(CentosAarch64VM, DEFAULT_CONFIG) + sys.exit(basevm.main(CentosAarch64VM, defaults)) From patchwork Wed Feb 19 16:35:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 1240874 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=EL+vRzOx; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48N3jQ5gy4z9sR4 for ; Thu, 20 Feb 2020 03:54:34 +1100 (AEDT) Received: from localhost ([::1]:56466 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4ScK-0000rZ-LE for incoming@patchwork.ozlabs.org; Wed, 19 Feb 2020 11:54:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42102) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j4SOU-0003d5-Cm for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j4SOS-0008Df-9C for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:14 -0500 Received: from mail-pj1-x1043.google.com ([2607:f8b0:4864:20::1043]:54808) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1j4SOS-0008DJ-11 for qemu-devel@nongnu.org; Wed, 19 Feb 2020 11:40:12 -0500 Received: by mail-pj1-x1043.google.com with SMTP id dw13so292731pjb.4 for ; Wed, 19 Feb 2020 08:40:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3orctIoixlh5zMRLtfbiaWjCyd2poMVCy6JL4hja3+4=; b=EL+vRzOxwE9gWOY+DLLo+81dkJ0kke98FH2J1yK26LA+l1GwNavFm6e88MnGklSdx8 F70jtCaX4eTkH9Zh4wvZK065KNh0WozAXVfICMeuEHiF7Q7aQDdvmaAIR+k+mhwKTFlg 7tzzxE2rHMlsoIH+uN3QG968LC/ubvWEdSTZEh/Z4jAS9jVTvUdKTcx8eZf2C9B0+JZD 7Mb9Bq5to4eqqlzwEQ6gB/xL4Lmyi5oteLPe3lbkg2wCPo9x1Y+0RAbiaicWa4TQ9VUm pduKV17ppqRgmjy0rEW/nRZHkrxDtWudMbVIFsfugECwYza5ftY56MJGcKDtFXVt63G1 GZZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3orctIoixlh5zMRLtfbiaWjCyd2poMVCy6JL4hja3+4=; b=Lkvpl+OGHgooAOSAx/k9Z8ffUjTSmn035Yh2Y66uA+CJ+dP9GawKkTbNJS3rsi2Mka EteNCo+MrHKETPohaqjf3YoeBwHUQhUnPwifMsGwfYOzPAW5Mty8aTClVuoEjLxGH4UO 5OX/unF/GUFjCeCSFVMV+Ule/vet+MSCLRrDDSxUXwq1x8M3qO8yAsNxjyLOcTBUXG2o /mAaxa9CxOCCDrqBO8jMrNs/AZ+eQamUEEFaWqNTJOcKMOp3AfKlV2+MQzQruB3RYy1E 7zl3lClJpqLRLIvgoQivOfbLBmJc7B5fB1tKo1hDFF56WFkJ5P35Yq9ewXq7hrbtnvh/ ekHw== X-Gm-Message-State: APjAAAUdLUzP7jlYS+KAK8ztdx4wvnxrAo2yvKT4BQ5rEdZhfE3TV5YE /wljFR0WcUWZVe+vAuu/eGc4p6lTpxM= X-Google-Smtp-Source: APXvYqzhBpEhlsO5j5ysL65hCOS+dMS4PLmBUw3UDfrvrKCDm+SEPyc6WDAwQV5fTAl1UpIYEonhlg== X-Received: by 2002:a17:90a:8b08:: with SMTP id y8mr9146605pjn.87.1582130410474; Wed, 19 Feb 2020 08:40:10 -0800 (PST) Received: from Rfoley-MA01.hsd1.ma.comcast.net ([2601:199:4480:60c0:3cc2:5eb7:e248:ad23]) by smtp.gmail.com with ESMTPSA id a69sm112362pfa.129.2020.02.19.08.40.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 08:40:09 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH v2 14/14] tests/vm: change scripts to use self._config Date: Wed, 19 Feb 2020 11:35:37 -0500 Message-Id: <20200219163537.22098-15-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200219163537.22098-1-robert.foley@linaro.org> References: <20200219163537.22098-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1043 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This change converts existing scripts to using for example self.ROOT_PASS, to self._config['root_pass']. We made similar changes for GUEST_USER, and GUEST_PASS. This allows us also to remove the change in basevm.py, which adds __getattr__ for backwards compatibility. Signed-off-by: Robert Foley --- tests/vm/basevm.py | 11 ++--------- tests/vm/fedora | 17 +++++++++-------- tests/vm/freebsd | 16 ++++++++-------- tests/vm/netbsd | 19 ++++++++++--------- tests/vm/openbsd | 17 +++++++++-------- 5 files changed, 38 insertions(+), 42 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 6ed04a2a4b..ad04789bcf 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -178,13 +178,6 @@ class BaseVM(object): self.console_init(timeout=timeout) self.console_wait(wait_string) - def __getattr__(self, name): - # Support direct access to config by key. - # for example, access self._config['cpu'] by self.cpu - if name.lower() in self._config.keys(): - return self._config[name.lower()] - return object.__getattribute__(self, name) - def _download_with_cache(self, url, sha256sum=None, sha512sum=None): def check_sha256sum(fname): if not sha256sum: @@ -234,13 +227,13 @@ class BaseVM(object): return r def ssh(self, *cmd): - return self._ssh_do(self.GUEST_USER, cmd, False) + return self._ssh_do(self._config["guest_user"], cmd, False) def ssh_root(self, *cmd): return self._ssh_do("root", cmd, False) def ssh_check(self, *cmd): - self._ssh_do(self.GUEST_USER, cmd, True) + self._ssh_do(self._config["guest_user"], cmd, True) def ssh_root_check(self, *cmd): self._ssh_do("root", cmd, True) diff --git a/tests/vm/fedora b/tests/vm/fedora index 4d7d6049f4..60195ddc46 100755 --- a/tests/vm/fedora +++ b/tests/vm/fedora @@ -105,20 +105,20 @@ class FedoraVM(basevm.BaseVM): self.console_wait_send("7) [!] Root password", "7\n") self.console_wait("Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("8) [ ] User creation", "8\n") self.console_wait_send("1) [ ] Create user", "1\n") self.console_wait_send("3) User name", "3\n") - self.console_wait_send("ENTER:", "%s\n" % self.GUEST_USER) + self.console_wait_send("ENTER:", "%s\n" % self._config["guest_user"]) self.console_wait_send("4) [ ] Use password", "4\n") self.console_wait_send("5) Password", "5\n") self.console_wait("Password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password (confirm):") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("7) Groups", "c\n") while True: @@ -136,7 +136,7 @@ class FedoraVM(basevm.BaseVM): if good: break time.sleep(10) - self.console_send("r\n" % self.GUEST_PASS) + self.console_send("r\n" % self._config["guest_pass"]) self.console_wait_send("'b' to begin install", "b\n") @@ -147,12 +147,13 @@ class FedoraVM(basevm.BaseVM): # setup qemu user prompt = " ~]$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") # setup root user prompt = " ~]#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/freebsd b/tests/vm/freebsd index fb54334696..735fb5f794 100755 --- a/tests/vm/freebsd +++ b/tests/vm/freebsd @@ -110,9 +110,9 @@ class FreeBSDVM(basevm.BaseVM): # post-install configuration self.console_wait("New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype New Password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("Network Configuration", "\n") self.console_wait_send("IPv4", "y") @@ -131,9 +131,9 @@ class FreeBSDVM(basevm.BaseVM): # qemu user self.console_wait_send("Add User Accounts", "y") self.console_wait("Username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait_send("Uid", "\n") self.console_wait_send("Login group", "\n") self.console_wait_send("Login group", "\n") @@ -145,9 +145,9 @@ class FreeBSDVM(basevm.BaseVM): self.console_wait_send("Use an empty password", "\n") self.console_wait_send("Use a random password", "\n") self.console_wait("Enter password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Enter password again:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("Lock out", "\n") self.console_wait_send("OK", "yes\n") self.console_wait_send("Add another user", "no\n") @@ -161,12 +161,12 @@ class FreeBSDVM(basevm.BaseVM): # setup qemu user prompt = "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") # setup root user prompt = "root@freebsd:~ #" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) # setup serial console diff --git a/tests/vm/netbsd b/tests/vm/netbsd index c5069a45f4..d4ad3525e3 100755 --- a/tests/vm/netbsd +++ b/tests/vm/netbsd @@ -128,24 +128,24 @@ class NetBSDVM(basevm.BaseVM): self.console_wait_send("d: Change root password", "d\n") self.console_wait_send("a: Yes", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("o: Add a user", "o\n") self.console_wait("username") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("to group wheel") self.console_wait_send("a: Yes", "a\n") self.console_wait_send("a: /bin/sh", "a\n") self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("New password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Retype new password:") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("a: Configure network", "a\n") self.console_wait_send("a: vioif0", "a\n") @@ -178,12 +178,13 @@ class NetBSDVM(basevm.BaseVM): # setup qemu user prompt = "localhost$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") # setup root user prompt = "localhost#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) # setup virtio-blk #1 (tarfile) diff --git a/tests/vm/openbsd b/tests/vm/openbsd index 22cd9513dd..dbef216853 100755 --- a/tests/vm/openbsd +++ b/tests/vm/openbsd @@ -95,9 +95,9 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which network interface", "done\n") self.console_wait_send("DNS domain name", "localnet\n") self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait("Password for root account") - self.console_send("%s\n" % self.ROOT_PASS) + self.console_send("%s\n" % self._config["root_pass"]) self.console_wait_send("Start sshd(8)", "yes\n") self.console_wait_send("X Window System", "\n") self.console_wait_send("xenodm", "\n") @@ -105,13 +105,13 @@ class OpenBSDVM(basevm.BaseVM): self.console_wait_send("Which speed", "\n") self.console_wait("Setup a user") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Full name") - self.console_send("%s\n" % self.GUEST_USER) + self.console_send("%s\n" % self._config["guest_user"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait("Password") - self.console_send("%s\n" % self.GUEST_PASS) + self.console_send("%s\n" % self._config["guest_pass"]) self.console_wait_send("Allow root ssh login", "yes\n") self.console_wait_send("timezone", "UTC\n") @@ -132,12 +132,13 @@ class OpenBSDVM(basevm.BaseVM): # setup qemu user prompt = "$" - self.console_ssh_init(prompt, self.GUEST_USER, self.GUEST_PASS) + self.console_ssh_init(prompt, self._config["guest_user"], + self._config["guest_pass"]) self.console_wait_send(prompt, "exit\n") # setup root user prompt = "openbsd#" - self.console_ssh_init(prompt, "root", self.ROOT_PASS) + self.console_ssh_init(prompt, "root", self._config["root_pass"]) self.console_sshd_config(prompt) # setup virtio-blk #1 (tarfile)