[{"id":3417985,"web_url":"http://patchwork.ozlabs.org/comment/3417985/","msgid":"<87v7wgletr.fsf@draig.linaro.org>","list_archive_url":null,"date":"2024-11-21T16:41:20","subject":"Re: [PATCH v2 00/18] test/functional: improve functional test\n debugging & fix tuxrun","submitter":{"id":39532,"url":"http://patchwork.ozlabs.org/api/people/39532/","name":"Alex Bennée","email":"alex.bennee@linaro.org"},"content":"Daniel P. Berrangé writes:\n\n> This started out as a series to get rid of the many GBs of temp\n> files the functional tests leave behind. Then it expanded into\n> improving the functional test debugging by ensuring we preserve\n> the QEMU stdout/stderr log file created by the QEMUMachine class.\n> In the course of doing that I encountered some other minor points\n> worth fixing, and then got side tracked into looking at the tuxrun\n> hangs with aarch64be. Investigating the latter exposed some further\n> holes in the debugging story prompting yet more patches, as well as\n> a final solution for tuxrun. So this series does:\n>\n> * Purge all scratch files created by tests\n> * Preserve the stdout/stderr log file\n> * Capture debug log messages on QEMUMachine\n> * Provide a QMP backdoor for debugging stuck QEMUs\n> * Enhance console handling for partial line matches\n> * Fix the tuxrun tests by eliminating sleeps\n>\n> There's quite alot of code here, but at the same time it feels like\n> the kind of stuff that'll be valuable either in the 9.2 release, or\n> in the soon to exist 9.2 stable branch.\n>\n> NB, with this series applied Thomas' tuxrun conversion to functional\n> testing survives 200 iterations on my machine, whereas it would\n> reliably hang in < 20, and often in < 10, before.\n\nQueued to testing/next, thanks.\n\nI'll combine with plugins/next and some other misc fixes and post a\npre-PR for next week.\n\n>\n> Changed in v2:\n>\n> - Changed console interaction to forbid 'failure_message'\n> without 'success_message'\n> - Reword console interaction log messages\n> - Avoid stack trace when seeing early failure\n> - Rewrote comment in acpi bits test\n> - Avoid duplicate os.environ access\n>\n> Daniel P. Berrangé (18):\n> tests/functional: fix mips64el test to honour workdir\n> tests/functional: automatically clean up scratch files after tests\n> tests/functional: remove \"AVOCADO\" from env variable name\n> tests/functional: remove todo wrt avocado.utils.wait_for\n> tests/functional: remove leftover :avocado: tags\n> tests/functional: remove obsolete reference to avocado bug\n> tests/functional: remove comments talking about avocado\n> tests/functional: honour self.workdir in ACPI bits tests\n> tests/functional: put QEMUMachine logs in testcase log directory\n> tests/functional: honour requested test VM name in QEMUMachine\n> tests/functional: enable debug logging for QEMUMachine\n> tests/functional: logs details of console interaction operations\n> tests/functional: don't try to wait for the empty string\n> tests/functional: require non-NULL success_message for console wait\n> tests/functional: rewrite console handling to be bytewise\n> tests/functional: remove time.sleep usage from tuxrun tests\n> tests/functional: add a QMP backdoor for debugging stalled tests\n> tests/functional: avoid accessing log_filename on earlier failures\n>\n> docs/devel/testing/functional.rst | 16 +++++\n> tests/functional/qemu_test/cmd.py | 89 +++++++++++++++++++-----\n> tests/functional/qemu_test/testcase.py | 43 +++++++++---\n> tests/functional/qemu_test/tuxruntest.py | 17 ++---\n> tests/functional/test_acpi_bits.py | 57 +++++----------\n> tests/functional/test_arm_bpim2u.py | 20 ------\n> tests/functional/test_arm_orangepi.py | 27 -------\n> tests/functional/test_m68k_nextcube.py | 3 +-\n> tests/functional/test_mips64el_malta.py | 4 +-\n> tests/functional/test_virtio_gpu.py | 3 +-\n> 10 files changed, 150 insertions(+), 129 deletions(-)","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256\n header.s=google header.b=eJct1/Jc;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4XvPC21Cjfz1xyG\n\tfor ; Fri, 22 Nov 2024 03:41:50 +1100 (AEDT)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1tEAFH-00067k-KK; Thu, 21 Nov 2024 11:41:31 -0500","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1tEAFC-000671-S8\n for qemu-devel@nongnu.org; Thu, 21 Nov 2024 11:41:26 -0500","from mail-wm1-x332.google.com ([2a00:1450:4864:20::332])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)\n (Exim 4.90_1) (envelope-from )\n id 1tEAFA-0005ns-R5\n for qemu-devel@nongnu.org; Thu, 21 Nov 2024 11:41:26 -0500","by mail-wm1-x332.google.com with SMTP id\n 5b1f17b1804b1-431616c23b5so6552675e9.0\n for ; Thu, 21 Nov 2024 08:41:24 -0800 (PST)","from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id\n 5b1f17b1804b1-433b4616843sm60798385e9.20.2024.11.21.08.41.21\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Thu, 21 Nov 2024 08:41:21 -0800 (PST)","from draig (localhost [IPv6:::1])\n by draig.lan (Postfix) with ESMTP id 69E2B5F77B;\n Thu, 21 Nov 2024 16:41:20 +0000 (GMT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=linaro.org; s=google; t=1732207282; x=1732812082; darn=nongnu.org;\n h=content-transfer-encoding:mime-version:message-id:date:user-agent\n :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date\n :message-id:reply-to;\n bh=zRSHpS5mArv737GoSpxyuAlftPzlSnF23/QwowOdgac=;\n b=eJct1/Jc778eNMfisS/Ile4cBF7sVgfjuduKufjcv9Ukg8faX6P38JyWAeFUfVAltm\n ZvD/C0Mpk4SFoYQLDv2iArqIP3NMsgwLEOmO9T35CeuhqvblHCWaGuRRLZOxuk5UyERK\n 8GvOHI37R/HWf0NU6KaZRtflgGPvcekKFOe7BcafaMaI2Xny+GBMY8YzGufwkTXx23I8\n GmQAhoyHizMD7+PZWSll8m7dmk/89KElPlGYakKNJwmCa8g1pV8himVFZPVsoj/WhMMM\n kB15rSg3b7lkN59Sg5AOgFgOkxFTkTGq6sB/XxMbtYml788GjS1QMQ3Co1k4fCDWxhgy\n xUCg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1732207282; x=1732812082;\n h=content-transfer-encoding:mime-version:message-id:date:user-agent\n :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from\n :to:cc:subject:date:message-id:reply-to;\n bh=zRSHpS5mArv737GoSpxyuAlftPzlSnF23/QwowOdgac=;\n b=tj3H5jjaeDfJOe+OqhRd3UVkDvnX+8J6b52vHi9OYb9yw3X/BBp9Q4cYK9QPE+QWMN\n cO/ckynR5MxgdAj9735cZ763Jz5ykn/zJEFmjPoYivUuuNIAQq3JRZ45Z46mf6N2OJpO\n 4A9eIPyiKzQxnayK3ZcXPOvRIXDi2AYAI1cNZHgMlr7yoqySyR7DqgWaUEq0p5AO0rBH\n IPKr9uSZLnFfWs1EJzGo6HF9Cp+FNLvfsa2kIkWGB+jcIqhTZYsEBk3RbGj9YDk9YUVa\n /CvzQxy6Zf2G8vzHwNpYJ0ElBwJYkuM8lgiSqqtiwGdMxgbiuByyf87umo1aZajil71Y\n JTVQ==","X-Gm-Message-State":"AOJu0Yz47D2YSjCQ5bRAs8NqUu8+YoqGP7NfhRwsKh03xuNvf6asGutN\n 1MS4RWGPc6q1e1DWm3EYjNVxJLmMTQsVo/vMJ7JVqegqrEjtW1ZXvxbUkauWXy8=","X-Gm-Gg":"ASbGncvotzKTr0wPzYLLjXRD6GZzp4im2UrQuIkhnU4+sdUfmRtf5qOCIlNariIEcOV\n 9Bvc1ZFWixO9JMjt/4pEdZ2z4ReADK85Zpm8j5WEWdios1Qpu1Yd/UtYmC/D4tLcnH62V78g8fB\n 0sWBs7y5joMT112vbF6Tq8kShL2AhhAXbd/pc7NkbtmqG9QMENwgF71I/dQBdFdwufJiq2GhrSV\n 6LhnpeOv1zoftpuKFh/pgmS1hLglsMIgOB2yvidalmFwbcS","X-Google-Smtp-Source":"\n AGHT+IHo6kAGICz4cq5dWnn6Eg7daJ18wu9cabEhpGawBzV1fhdt9y5gsDd2NZd43QqlNTnIan2pzg==","X-Received":"by 2002:a05:600c:a401:b0:433:c76d:d56b with SMTP id\n 5b1f17b1804b1-433c76dd67fmr25099375e9.12.1732207282081;\n Thu, 21 Nov 2024 08:41:22 -0800 (PST)","From":"=?utf-8?q?Alex_Benn=C3=A9e?= ","To":"Daniel P. =?utf-8?b?QmVycmFuZ8Op?= ","Cc":"qemu-devel@nongnu.org, Thomas Huth ,\n Ani Sinha , Peter Maydell ,\n Philippe =?utf-8?q?Mathieu-Daud=C3=A9?= ","Subject":"Re: [PATCH v2 00/18] test/functional: improve functional test\n debugging & fix tuxrun","In-Reply-To":"<20241121154218.1423005-1-berrange@redhat.com> (\"Daniel P.\n\t=?utf-8?b?QmVycmFuZ8OpIidz?= message of \"Thu, 21 Nov 2024 15:42:00 +0000\")","References":"<20241121154218.1423005-1-berrange@redhat.com>","User-Agent":"mu4e 1.12.7; emacs 29.4","Date":"Thu, 21 Nov 2024 16:41:20 +0000","Message-ID":"<87v7wgletr.fsf@draig.linaro.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"quoted-printable","Received-SPF":"pass client-ip=2a00:1450:4864:20::332;\n envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x332.google.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}}]