From patchwork Thu Sep 23 00:49:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1531487 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OZ26D3Qj; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4HFGmy1mDkz9sRf for ; Thu, 23 Sep 2021 10:50:56 +1000 (AEST) Received: from localhost ([::1]:41570 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mTCwu-0000ym-R8 for incoming@patchwork.ozlabs.org; Wed, 22 Sep 2021 20:50:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55408) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTCvt-0000wc-UP for qemu-devel@nongnu.org; Wed, 22 Sep 2021 20:49:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36457) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTCvq-0006E1-5S for qemu-devel@nongnu.org; Wed, 22 Sep 2021 20:49:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632358184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UF49pZQr/AL6S9DjWsyYbeciSPyHuti7anXxnQwO3Co=; b=OZ26D3QjB2++XjdH+E5ns8ZHBv4RoXfD11bogahgEXQB6o6KkYzAu8GSRUrwoeasnHHU3I cxgfpp+2KqIpireyBheQpxRiQKjI53mxwZNPkz+l/2elK2QToS+uDDr7e7D0+3Mucd0VGl Ay+b7bVJjRMRsy1UJOStFqjU3DyqjKk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-65-cT9BfFGYPbGK8fQVqNR0xA-1; Wed, 22 Sep 2021 20:49:43 -0400 X-MC-Unique: cT9BfFGYPbGK8fQVqNR0xA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 105AE1084687; Thu, 23 Sep 2021 00:49:42 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A30160BF1; Thu, 23 Sep 2021 00:49:39 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH v2 00/17] Switch iotests to using Async QMP Date: Wed, 22 Sep 2021 20:49:21 -0400 Message-Id: <20210923004938.3999963-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -23 X-Spam_score: -2.4 X-Spam_bar: -- X-Spam_report: (-2.4 / 5.0 requ) DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , Eduardo Habkost , qemu-block@nongnu.org, Hanna Reitz , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Based-on: <20210915162955.333025-1-jsnow@redhat.com> [PATCH v4 00/27] python: introduce Asynchronous QMP package GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-aqmp-iotest-wrapper CI: https://gitlab.com/jsnow/qemu/-/pipelines/375637927 Hiya, This series continues where the first AQMP series left off and adds a synchronous 'legacy' wrapper around the new AQMP interface, then drops it straight into iotests to prove that AQMP is functional and totally cool and fine. The disruption and churn to iotests is extremely minimal. (There's actually a net negative SLOC in tests/qemu-iotests.) In the event that a regression happens and I am not physically proximate to inflict damage upon, one may set the QEMU_PYTHON_LEGACY_QMP variable to any non-empty string as it pleases you to engage the QMP machinery you are used to. I'd like to try and get this committed early in the 6.2 development cycle to give ample time to smooth over any possible regressions. I've tested it locally and via gitlab CI, across Python versions 3.6 through 3.10, and "worksforme". If something bad happens, we can revert the actual switch-flip very trivially. Layout: Patches 1-7: ./python/qemu/aqmp changes that serve as pre-requisites. Patches 8-12: other ./python changes that ease the transition. Patches 13-14: iotest changes to support the new QMP backend. Patches 15-17: Make the switch. V2: 001/17:[----] [--] 'python/aqmp: add greeting property to QMPClient' 002/17:[----] [--] 'python/aqmp: add .empty() method to EventListener' 003/17:[----] [--] 'python/aqmp: Return cleared events from EventListener.clear()' 004/17:[0007] [FC] 'python/aqmp: add send_fd_scm' 005/17:[down] 'python/aqmp: Add dict conversion method to Greeting object' 006/17:[down] 'python/aqmp: Reduce severity of EOFError-caused loop terminations' 007/17:[down] 'python/aqmp: Disable logging messages by default' 008/17:[0002] [FC] 'python/qmp: clear events on get_events() call' 009/17:[----] [--] 'python/qmp: add send_fd_scm directly to QEMUMonitorProtocol' 010/17:[----] [--] 'python, iotests: remove socket_scm_helper' 011/17:[0013] [FC] 'python/machine: remove has_quit argument' 012/17:[down] 'python/machine: Handle QMP errors on close more meticulously' 013/17:[0009] [FC] 'iotests: Accommodate async QMP Exception classes' 014/17:[down] 'iotests: Conditionally silence certain AQMP errors' 015/17:[0016] [FC] 'python/aqmp: Create sync QMP wrapper for iotests' 016/17:[0002] [FC] 'python/aqmp: Remove scary message' 017/17:[----] [--] 'python, iotests: replace qmp with aqmp' - Rebased on jsnow/python, which was recently rebased on origin/master. - Make aqmp's send_fd_scm method bark if the socket isn't AF_UNIX (Hanna) - Uh... modify send_fd_scm so it doesn't break when Python 3.11 comes out ... See the commit message for more detail. - Drop the "python/aqmp: Create MessageModel and StandaloneModel class" patch and replace with a far simpler method that just adds an _asdict() method. - Add patches 06 and 07 to change how the AQMP library handles logging. - Adjust docstring in patch 08 (Hanna) - Rename "_has_quit" attribute to "_quid_issued" (Hanna) - Renamed patch 12, simplified the logic in _soft_shutdown a tiny bit. - Fixed bad exception handling logic in 13 (Hanna) - Introduce a helper in patch 14 to silence log output when it's unwanted. - Small addition of _get_greeting() helper in patch 15, coinciding with the new patch 05 here. - Contextual changes in 16. John Snow (17): python/aqmp: add greeting property to QMPClient python/aqmp: add .empty() method to EventListener python/aqmp: Return cleared events from EventListener.clear() python/aqmp: add send_fd_scm python/aqmp: Add dict conversion method to Greeting object python/aqmp: Reduce severity of EOFError-caused loop terminations python/aqmp: Disable logging messages by default python/qmp: clear events on get_events() call python/qmp: add send_fd_scm directly to QEMUMonitorProtocol python, iotests: remove socket_scm_helper python/machine: remove has_quit argument python/machine: Handle QMP errors on close more meticulously iotests: Accommodate async QMP Exception classes iotests: Conditionally silence certain AQMP errors python/aqmp: Create sync QMP wrapper for iotests python/aqmp: Remove scary message python, iotests: replace qmp with aqmp tests/qemu-iotests/socket_scm_helper.c | 136 ---------------------- python/qemu/aqmp/__init__.py | 14 +-- python/qemu/aqmp/events.py | 15 ++- python/qemu/aqmp/legacy.py | 135 +++++++++++++++++++++ python/qemu/aqmp/models.py | 13 +++ python/qemu/aqmp/protocol.py | 7 +- python/qemu/aqmp/qmp_client.py | 27 +++++ python/qemu/machine/machine.py | 133 +++++++++++---------- python/qemu/machine/qtest.py | 2 - python/qemu/qmp/__init__.py | 27 +++-- python/qemu/qmp/qmp_shell.py | 1 - scripts/simplebench/bench_block_job.py | 3 +- tests/Makefile.include | 1 - tests/meson.build | 4 - tests/qemu-iotests/040 | 7 +- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/255 | 2 +- tests/qemu-iotests/iotests.py | 23 +++- tests/qemu-iotests/meson.build | 5 - tests/qemu-iotests/testenv.py | 8 +- tests/qemu-iotests/tests/mirror-top-perms | 12 +- 21 files changed, 315 insertions(+), 262 deletions(-) delete mode 100644 tests/qemu-iotests/socket_scm_helper.c create mode 100644 python/qemu/aqmp/legacy.py delete mode 100644 tests/qemu-iotests/meson.build Reviewed-by: Paolo Bonzini