From patchwork Thu Jul 1 04:12:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1499213 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=eAh860al; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GFlG36Y1zz9sW8 for ; Thu, 1 Jul 2021 14:14:02 +1000 (AEST) Received: from localhost ([::1]:36730 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lyo5O-0008DZ-AN for incoming@patchwork.ozlabs.org; Thu, 01 Jul 2021 00:13:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42232) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4w-0008C8-2x for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:30 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:33579) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lyo4t-0000ZW-0H for qemu-devel@nongnu.org; Thu, 01 Jul 2021 00:13:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625112804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cL7i5NZP5Udpu6OSOAfTtxf6q96tQZUE6D8xT1Aw7gA=; b=eAh860alSmV+5R3FZo19gqcxTD2+dMs1mZVbziDT5A6wbyztBojiBYOjbi7CX7iNETINm8 Dny2LxoKilLh4A7a10xAW0PHwmeSc1cxI4POiM3878i6jsInF7j7CCZyxleVIxVSBXVMgk 27KGNJwXRiQWacs27dQo7+m0hhu8wOA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-509-dPA2qvbqOxaMs4kS41rrHA-1; Thu, 01 Jul 2021 00:13:22 -0400 X-MC-Unique: dPA2qvbqOxaMs4kS41rrHA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A04D280415A; Thu, 1 Jul 2021 04:13:21 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-71.rdu2.redhat.com [10.10.118.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2B712604CC; Thu, 1 Jul 2021 04:13:14 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 00/20] python: introduce Asynchronous QMP package Date: Thu, 1 Jul 2021 00:12:53 -0400 Message-Id: <20210701041313.1696009-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.435, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Eric Blake , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-async-qmp-aqmp CI: https://gitlab.com/jsnow/qemu/-/pipelines/330003554 Docs: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html Based-on: <20210701020921.1679468-1-jsnow@redhat.com> [PULL 00/15] Python patches Hi! This patch series adds an Asynchronous QMP package to the Python library. It offers a few improvements over the previous library: - out-of-band support - true asynchronous event support - avoids undocumented interfaces abusing non-blocking sockets This library serves as the basis for a new qmp-shell program that will offer improved reconnection support, true asynchronous display of events, VM and job status update notifiers, and so on. My intent is to eventually publish this library directly to PyPI as a standalone package. I would like to phase out our usage of the old QMP library over time; eventually replacing it entirely with this one. This series looks big by line count, but it's *mostly* docstrings. Seriously! This package has *no* external dependencies whatsoever. Notes & Design ============== Here are some notes on the design of how the library works, to serve as a primer for review; however I also **highly recommend** browsing the generated Sphinx documentation for this series. Here's that link again: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html The core machinery is split between the AsyncProtocol and QMP classes. AsyncProtocol provides the generic machinery, while QMP provides the QMP-specific details. The design uses two independent coroutines that act as the "bottom half", a writer task and a reader task. These tasks run for the duration of the connection and independently send and receive messages, respectively. A third task, disconnect, is scheduled asynchronously whenever an unrecoverable error occurs and facilitates coalescing of the other two tasks. This diagram for how execute() operates may be helpful for understanding how AsyncProtocol is laid out. The arrows indicate the direction of a QMP message; the long horizontal dash indicates the separation between the upper and lower half of the event loop. The queue mechanisms between both dashes serve as the intermediaries between the upper and lower half. +---------+ | caller | +---------+ ^ | | v +---------+ +---------------> |execute()| -----------+ | +---------+ | | | [-----------------------------------------------------------] | | | v +----+------+ +----------------+ +------+-------+ | ExecQueue | | EventListeners | |Outbound Queue| +----+------+ +----+-----------+ +------+-------+ ^ ^ | | | | [-----------------------------------------------------------] | | | | | v +--+----------------+---+ +-----------+-----------+ | Reader Task/Coroutine | | Writer Task/Coroutine | +-----------+-----------+ +-----------+-----------+ ^ | | v +-----+------+ +-----+------+ |StreamReader| |StreamWriter| +------------+ +------------+ The caller will invoke execute(), which in turn will deposit a message in the outbound send queue. This will wake up the writer task, which well send the message over the wire. The execute() method will then yield to wait for a reply delivered to an execution queue created solely for that execute statement. When a message arrives, the Reader task will unblock and route the message either to the EventListener subsystem, or place it in the appropriate pending execution queue. Once a message is placed in the pending execution queue, execute() will unblock and the execution will conclude, returning the result of the RPC call to the caller. Ugly Bits ========= - MultiException is ... wonky. I am still working out how to avoid needing it. See patch 04/20 for details here, or see https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.error.html Patch 06/20 also goes into details of the ugliness; see AsyncProtocol._results or view the same information here: https://people.redhat.com/~jsnow/sphinx/html/_modules/qemu/aqmp/protocol.html#AsyncProtocol._results - There are quite a few lingering questions I have over the design of the EventListener subsystem; I wrote about those ugly bits in excruciating detail in patch 14/20. You can view them formatted nicely here: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.events.html#experimental-interfaces-design-issues Patch Layout ============ Patches 1-2 are tiny pylint configuration changes. Patches 3-5 begin to check in Async QMP components, they are small. Patches 6-11 add a generic async message-based protocol class, AsyncProto. They are split as small as I could reasonably split them. Patches 12-14 check in more QMP-specific components. Patches 15-18 add qmp_protocol.py, the new 'QMP' class. They're split as far down as I could, I hope they're easy to review. Patches 19-20 add a few finishing touches, they are small patches. Future Work =========== These items are in progress: - A Synchronous QMP wrapper that allows this library to be easily used from non-async code; this will also allow me to prove it works well by demoing its replacement throughout iotests. - A QMP server class; to facilitate writing of unit tests. - Unit tests. Real unit tests. If you made it this far in the cover letter, congrats :) John Snow (20): python/pylint: Add exception for TypeVar names ('T') python/pylint: disable too-many-function-args python/aqmp: add asynchronous QMP (AQMP) subpackage python/aqmp: add error classes python/aqmp: add asyncio compatibility wrappers python/aqmp: add generic async message-based protocol support python/aqmp: add runstate state machine to AsyncProtocol python/aqmp: add logging to AsyncProtocol python/aqmp: add AsyncProtocol.accept() method python/aqmp: add _cb_inbound and _cb_inbound logging hooks python/aqmp: add AsyncProtocol._readline() method python/aqmp: add QMP Message format python/aqmp: add well-known QMP object models python/aqmp: add QMP event support python/aqmp: add QMP protocol support python/aqmp: Add message routing to QMP protocol python/aqmp: add execute() interfaces python/aqmp: add _raw() execution interface python/aqmp: add asyncio_run compatibility wrapper python/aqmp: add scary message python/qemu/aqmp/__init__.py | 61 +++ python/qemu/aqmp/error.py | 97 ++++ python/qemu/aqmp/events.py | 878 +++++++++++++++++++++++++++++++ python/qemu/aqmp/message.py | 207 ++++++++ python/qemu/aqmp/models.py | 133 +++++ python/qemu/aqmp/protocol.py | 851 ++++++++++++++++++++++++++++++ python/qemu/aqmp/py.typed | 0 python/qemu/aqmp/qmp_protocol.py | 565 ++++++++++++++++++++ python/qemu/aqmp/util.py | 183 +++++++ python/setup.cfg | 4 +- 10 files changed, 2978 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/__init__.py create mode 100644 python/qemu/aqmp/error.py create mode 100644 python/qemu/aqmp/events.py create mode 100644 python/qemu/aqmp/message.py create mode 100644 python/qemu/aqmp/models.py create mode 100644 python/qemu/aqmp/protocol.py create mode 100644 python/qemu/aqmp/py.typed create mode 100644 python/qemu/aqmp/qmp_protocol.py create mode 100644 python/qemu/aqmp/util.py Acked-by: Stefan Hajnoczi