From patchwork Tue Apr 13 15:55:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465841 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aD0IgotG; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVb96dBNz9sTD for ; Wed, 14 Apr 2021 01:56:37 +1000 (AEST) Received: from localhost ([::1]:39950 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLP1-0001Uu-GN for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 11:56:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38406) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOb-0001RF-FW for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:58774) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOX-00009E-4c for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lkwW4qd7cXMVTa9AkNsmmBjla4GICLiyPDDcma8u0a4=; b=aD0IgotGpM+NIY5WdJ9QdvWNcwhlawfk3BanwMNjiyuey3/rl7/3g3iNkRSNi1YNP7TTuQ RXuoK/TIpINi8es7epWF0rzmuSBGXnZt1GcPy+LPUBh2XtuZ+nu2z/hKbVvLbedXrhXnh5 WsPermGeYmGI8sYx7PpYGE8IC5ouL38= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-157-9ulr9YSiPf-13uA4TPR50g-1; Tue, 13 Apr 2021 11:56:00 -0400 X-MC-Unique: 9ulr9YSiPf-13uA4TPR50g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 74F1B79EC6 for ; Tue, 13 Apr 2021 15:55:59 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 990886A034; Tue, 13 Apr 2021 15:55:58 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 1/7] util: asyncio-related helpers Date: Tue, 13 Apr 2021 11:55:47 -0400 Message-Id: <20210413155553.2660523-2-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Nothing too interesting design-wise here; mostly asyncio-related helpers designed to make writing Python 3.6-compliant code a little nicer to read. Signed-off-by: John Snow --- util.py | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 util.py diff --git a/util.py b/util.py new file mode 100644 index 0000000..2640f82 --- /dev/null +++ b/util.py @@ -0,0 +1,87 @@ +""" +Misc. utils and helper functions +""" + +import asyncio +import traceback +import sys +from typing import ( + Any, + Coroutine, + TypeVar, +) + + +T = TypeVar('T') + + +def create_task(coro: Coroutine[Any, Any, T]) -> 'asyncio.Future[T]': + """ + Python 3.6-compatible create_task() wrapper. + """ + if hasattr(asyncio, 'create_task'): + # Python 3.7+ + return asyncio.create_task(coro) + + # Python 3.6 + return asyncio.ensure_future(coro) + + +async def wait_closed(writer: asyncio.StreamWriter) -> None: + """ + Python 3.6-compatible StreamWriter.wait_closed() wrapper. + """ + if hasattr(writer, 'wait_closed'): + # Python 3.7+ + await writer.wait_closed() + else: + # Python 3.6 + transport = writer.transport + assert isinstance(transport, asyncio.WriteTransport) + + while not transport.is_closing(): + await asyncio.sleep(0.0) + while transport.get_write_buffer_size() > 0: + await asyncio.sleep(0.0) + + +def asyncio_run(coro: Coroutine[Any, Any, T]) -> T: + """ + Python 3.6-compatible asyncio.run() wrapper. + """ + # Python 3.7+ + if hasattr(asyncio, 'run'): + return asyncio.run(coro) + + # Python 3.6 + loop = asyncio.get_event_loop() + ret = loop.run_until_complete(coro) + loop.close() + + return ret + + +def pretty_traceback() -> str: + """ + Print the current traceback, but indented to provide visual distinction. + + This is useful for printing a traceback within a traceback for + debugging purposes when encapsulating errors to deliver them up the + stack; when those errors are printed, this helps provide a nice + visual grouping to quickly identify the parts of the error that + belong to the inner exception. + + :returns: A string, formatted something like the following:: + + | Traceback (most recent call last): + | File "foobar.py", line 42, in arbitrary_example + | foo.baz() + | ArbitraryError: [Errno 42] Something bad happened! + + """ + exc_lines = [] + for chunk in traceback.format_exception(*sys.exc_info()): + for line in chunk.split("\n"): + if line: + exc_lines.append(f" | {line}") + return "\n".join(exc_lines) From patchwork Tue Apr 13 15:55:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465843 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OuDMJE8f; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVfQ2KcRz9sTD for ; Wed, 14 Apr 2021 01:59:26 +1000 (AEST) Received: from localhost ([::1]:48306 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLRk-0004u5-21 for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 11:59:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38428) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOc-0001Sg-0C for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46809) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOX-0000BB-F6 for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SfeXp++ZNmfkKdeGgfGhy8Bkbe50OiQTtJtWuExrEss=; b=OuDMJE8fP0P/0eB+ZICAwP2KfiMuZefD7eW2JnOQbrZPt/v9bBhuumhhyluivHpyYzAFDg grhkqQSJCRy1DbOk4mY0Gi/Ka8nvRo0XWtIAXxCIxmMqFH5qPxlyZoPvDnF0m8BVnzVucc P+sGyuXxzEE2DNXnvh1Dux9byxWJgb0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-197-ifCkLrEGMwmVxv8rBOIjhQ-1; Tue, 13 Apr 2021 11:56:01 -0400 X-MC-Unique: ifCkLrEGMwmVxv8rBOIjhQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 40CDC189C440 for ; Tue, 13 Apr 2021 15:56:00 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E2636A034; Tue, 13 Apr 2021 15:55:59 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 2/7] error: Error classes and so on. Date: Tue, 13 Apr 2021 11:55:48 -0400 Message-Id: <20210413155553.2660523-3-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" May be somewhat hard to make sense of until you see how these classes are used later on. Notably, although I have split QMP's functionality into a "protocol" class and a "QMP" class, representing a separation of the loop mechanisms and the QMP protocol itself, this file was written prior to that split and contains both "generic" and "QMP-specific" error classes. It will have to be split out later, but for the purposes of an RFC where I wanted a quick eyeball on design, I thought it wasn't necessary to clean that up just yet. The MultiException class might warrant a closer inspection, it's the "weirdest" thing here. It's intended to be used internally by the module, but as with all best laid plans, there is always the ability it will somehow leak out into the caller's space through some unforseen mechanism. Signed-off-by: John Snow --- error.py | 163 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 error.py diff --git a/error.py b/error.py new file mode 100644 index 0000000..f19f8e0 --- /dev/null +++ b/error.py @@ -0,0 +1,163 @@ +"""Generic error classes. + +This module seeks to provide semantic error classes that are intended to +be used directly by clients when they would like to handle particular +semantic failures (e.g. "failed to connect") without needing to know the +enumeration of possible reasons for that failure. + +AQMPError serves as the ancestor for almost all exceptions raised by +this package, and is suitable for use in handling semantic errors from +this library. In most cases, individual methods will attempt to catch +and re-encapsulate various exceptions to provide a semantic +error-handling interface, though this is not necessarily true of +internal interfaces. + +Some errors are not defined here in this module, but exist alongside +more specific error domains in other modules. They are listed here for +convenience anyway. + +The error inheritance tree is as follows:: + + MultiException + AQMPError + ProtocolError + RawProtocolError + DeserializationError + UnexpectedTypeError + GreetingError + NegotiationError + MsgProtocolError (message.py) + ObjectTypeError (message.py) + OrphanedError (message.py) + ServerParseError (message.py) + ConnectError + DisconnectedError + StateError + +The only exception that is not an `AQMPError` is `MultiException`. It is +special, and used to encapsulate one-or-more exceptions of an arbitrary +kind; this exception MAY be raised on disconnect() when there are two or +more exceptions from the AQMP event loop to report back to the caller. + +(The bottom half is designed in such a way that exceptions are attempted +to be handled internally, but in cases of catastrophic failure, it may +still occur.) + +See `MultiException` and `AsyncProtocol.disconnect()` for more details. + +""" + +from typing import Iterable, Iterator + + +class AQMPError(Exception): + # Don't use this directly: create a subclass. + """Base failure for all errors raised by AQMP.""" + + +class ProtocolError(AQMPError): + """Abstract error class for protocol failures.""" + def __init__(self, error_message: str): + super().__init__() + self.error_message = error_message + + def __str__(self) -> str: + return f"QMP protocol error: {self.error_message}" + + +class RawProtocolError(ProtocolError): + """ + Abstract error class for low-level parsing failures. + """ + def __init__(self, error_message: str, raw: bytes): + super().__init__(error_message) + self.raw = raw + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" raw bytes were: {str(self.raw)}", + ]) + + +class DeserializationError(RawProtocolError): + """Incoming message was not understood as JSON.""" + + +class UnexpectedTypeError(RawProtocolError): + """Incoming message was JSON, but not a JSON object.""" + + +class ConnectError(AQMPError): + """ + Initial connection process failed. + Always wraps a "root cause" exception that can be interrogated for info. + """ + + +class GreetingError(ProtocolError): + """An exception occurred during the Greeting phase.""" + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + self.exc = exc + + def __str__(self) -> str: + return ( + f"QMP protocol error: {self.error_message}\n" + f" Cause: {self.exc!s}\n" + ) + + +class NegotiationError(ProtocolError): + """An exception occurred during the Negotiation phase.""" + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + self.exc = exc + + def __str__(self) -> str: + return ( + f"QMP protocol error: {self.error_message}\n" + f" Cause: {self.exc!s}\n" + ) + + +class DisconnectedError(AQMPError): + """ + Command was not able to be completed; we have been Disconnected. + + This error is raised in response to a pending execution when the + back-end is unable to process responses any more. + """ + + +class StateError(AQMPError): + """ + An API command (connect, execute, etc) was issued at an inappropriate time. + + (e.g. execute() while disconnected; connect() while connected; etc.) + """ + + +class MultiException(Exception): + """ + Used for multiplexing exceptions. + + This exception is used in the case that errors were encountered in both the + Reader and Writer tasks, and we must raise more than one. + """ + def __init__(self, exceptions: Iterable[BaseException]): + super().__init__(exceptions) + self.exceptions = list(exceptions) + + def __str__(self) -> str: + ret = "------------------------------\n" + ret += "Multiple Exceptions occurred:\n" + ret += "\n" + for i, exc in enumerate(self.exceptions): + ret += f"{i}) {str(exc)}\n" + ret += "\n" + ret += "-----------------------------\n" + return ret + + def __iter__(self) -> Iterator[BaseException]: + return iter(self.exceptions) From patchwork Tue Apr 13 15:55:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465842 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Mw9GLia6; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVbH1b9hz9sVv for ; Wed, 14 Apr 2021 01:56:43 +1000 (AEST) Received: from localhost ([::1]:40106 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLP6-0001ZI-PB for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 11:56:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38420) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOb-0001Rz-LT for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56824) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOY-0000BE-5m for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9L5+AX3XAUrSmVo3+XVpFA7y5ivJFNxgS3CusV1ivAg=; b=Mw9GLia6I4qhlTPSixH/9iYZermLzBQBea2ZWOsTQk9+NdmRbSbq7NgRxsFrGNntBnppTz MGctx40xQaGUekYbsX8CQNXQ4zaqPqX/QPHBd4IszV6cKRY9ArvtK7WQuxpz6YI717tFLM K0000IpL6gz/q/qCIUd4ud+8gzgFZaY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-422-BbLq7ErRPNWSHURKIyUz3Q-1; Tue, 13 Apr 2021 11:56:02 -0400 X-MC-Unique: BbLq7ErRPNWSHURKIyUz3Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8E27079EC2 for ; Tue, 13 Apr 2021 15:56:01 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AC406A039; Tue, 13 Apr 2021 15:56:00 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 3/7] protocol: generic async message-based protocol loop Date: Tue, 13 Apr 2021 11:55:49 -0400 Message-Id: <20210413155553.2660523-4-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This module provides the protocol-agnostic framework upon which QMP will be built. I also have (not included in this series) a qtest implementation that uses this same framework, which is why it is split into two portions like this. The design uses two independent tasks in the "bottol half", a writer and a reader. These tasks run for the duration of the connection and independently send and receive messages, respectively. A third task, disconnect, is scheduled whenever an error occurs and facilitates coalescing of the other two tasks. MultiException is used in this case if *both* tasks should have Exceptions that need to be reported, though at the time of writing, I think this circumstance might only be a theoretical concern. The generic model here does not provide execute(), but the model for QMP is informative for how this class is laid out. Below, QMP's execute() function deposits a message into the outbound queue. The writer task wakes up to process the queue and deposits information in the write buffer, where the message is finally dispatched. Meanwhile, the execute() call is expected to block on an RPC mailbox waiting for a reply from the server. On the return trip, the reader wakes up when data arrives in the buffer. The message is deserialized and handed off to the protocol layer to route accordingly. QMP will route this message into either the Event queue or one of the pending RPC mailboxes. Upon this message being routed to the correct RPC mailbox, execute() will be woken up and allowed to process the reply and deliver it back to the caller. The reason for separating the inbound and outbound tasks to such an extreme degree is to allow for designs and extensions where this asynchronous loop may be launched in a separate thread. In this model, it is possible to use a synchronous, thread-safe function to deposit new messages into the outbound queue; this was seen as a viable way to offer solid synchronous bindings while still allowing events to be processed truly asynchronously. Separating it this way also allows us to fairly easily support Out-of-band executions with little additional effort; essentially all commands are treated as out-of-band. The execute graph: +---------+ | caller | +---------+ | v +---------+ +---------------- |execute()| <----------+ | +---------+ | | | ----------------------------------------------------------- v | +----+----+ +-----------+ +------+-------+ |Mailboxes| |Event Queue| |Outbound Queue| +----+----+ +------+----+ +------+-------+ | | ^ v v | +--+----------------+---+ +-----------+-----------+ | Reader Task/Coroutine | | Writer Task/Coroutine | +-----------+-----------+ +-----------+-----------+ | ^ v | +-----+------+ +-----+------+ |StreamReader| |StreamWriter| +------------+ +------------+ Signed-off-by: John Snow --- protocol.py | 704 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 704 insertions(+) create mode 100644 protocol.py diff --git a/protocol.py b/protocol.py new file mode 100644 index 0000000..27d1558 --- /dev/null +++ b/protocol.py @@ -0,0 +1,704 @@ +""" +Async message-based protocol support. + +This module provides a generic framework for sending and receiving +messages over an asyncio stream. + +`AsyncProtocol` is an abstract class that implements the core mechanisms +of a simple send/receive protocol, and is designed to be extended. + +`AsyncTasks` provides a container class that aggregates tasks that make +up the loop used by `AsyncProtocol`. +""" + +import asyncio +from asyncio import StreamReader, StreamWriter +import logging +from ssl import SSLContext +from typing import ( + Any, + Awaitable, + Callable, + Coroutine, + Iterator, + List, + Generic, + Optional, + Tuple, + TypeVar, + Union, +) + +from error import ( + ConnectError, + MultiException, + StateError, +) +from util import create_task, pretty_traceback, wait_closed + + +T = TypeVar('T') +_TaskFN = Callable[[], Awaitable[None]] # aka ``async def func() -> None`` +_FutureT = TypeVar('_FutureT', bound=Optional['asyncio.Future[Any]']) +_GatherRet = List[Optional[BaseException]] + + +class AsyncTasks: + """ + AsyncTasks is a collection of bottom half tasks designed to run forever. + + This is a convenience wrapper to make calls from `AsyncProtocol` simpler to + follow by behaving as a simple aggregate of two or more tasks, such that + a higher-level connection manager can simply refer to "the bottom half" + as one coherent entity instead of several. + + The general flow is: + + 1. ``tasks = AsyncTasks(logger_for_my_client)`` + 2. ``tasks.start(my_reader, my_writer)`` + 3. ``...`` + 4. ``await tasks.cancel()`` + 5. ``tasks.result()`` + + :param logger: A logger to use for debugging messages. Useful to + associate messages with a particular server context. + """ + + logger = logging.getLogger(__name__) + + def __init__(self, logger: Optional[logging.Logger] = None): + if logger is not None: + self.logger = logger + + # Named tasks + self.reader: Optional['asyncio.Future[None]'] = None + self.writer: Optional['asyncio.Future[None]'] = None + + # Internal aggregate of all of the above tasks. + self._all: Optional['asyncio.Future[_GatherRet]'] = None + + def _all_tasks(self) -> Iterator[Optional['asyncio.Future[None]']]: + """Yields all tasks, defined or not, in ideal cancellation order.""" + yield self.writer + yield self.reader + + def __iter__(self) -> Iterator['asyncio.Future[None]']: + """Yields all defined tasks, in ideal cancellation order.""" + for task in self._all_tasks(): + if task is not None: + yield task + + @property + def _all_tasks_defined(self) -> bool: + """Returns True if all tasks are defined.""" + return all(map(lambda task: task is not None, self._all_tasks())) + + @property + def _some_tasks_done(self) -> bool: + """Returns True if any defined tasks are done executing.""" + return any(map(lambda task: task.done(), iter(self))) + + def __bool__(self) -> bool: + """Returns True when any tasks are defined at all.""" + return bool(tuple(iter(self))) + + @property + def running(self) -> bool: + """Returns True if all tasks are defined and still running.""" + return self._all_tasks_defined and not self._some_tasks_done + + def start(self, + reader_coro: Coroutine[Any, Any, None], + writer_coro: Coroutine[Any, Any, None]) -> None: + """ + Starts executing tasks in the current async context. + + :param reader_coro: Coroutine, message reader task. + :param writer_coro: Coroutine, message writer task. + """ + self.reader = create_task(reader_coro) + self.writer = create_task(writer_coro) + + # Uses extensible self-iterator. + self._all = asyncio.gather(*iter(self), return_exceptions=True) + + async def cancel(self) -> None: + """ + Cancels all tasks and awaits full cancellation. + + Exceptions, if any, can be obtained by calling `result()`. + """ + for task in self: + if task and not task.done(): + self.logger.debug("cancelling task %s", str(task)) + task.cancel() + + if self._all: + self.logger.debug("Awaiting all tasks to finish ...") + await self._all + + def _cleanup(self) -> None: + """ + Erase all task handles; asserts that no tasks are running. + """ + def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]: + assert (task is None) or task.done() + return None if (task and task.done()) else task + + self.reader = _paranoid_task_erase(self.reader) + self.writer = _paranoid_task_erase(self.writer) + self._all = _paranoid_task_erase(self._all) + + def result(self) -> None: + """ + Raises exception(s) from the finished tasks, if any. + + Called to fully quiesce this task group. asyncio.CancelledError is + never raised; in the event of an intentional cancellation this + function will not raise any errors. + + If an exception in one bottom half caused an unscheduled disconnect, + that exception will be raised. + + :raise: `Exception` Arbitrary exceptions re-raised on behalf of + the bottom half. + :raise: `MultiException` Iterable Exception used to multiplex multiple + exceptions when multiple threads failed. + """ + exceptions: List[BaseException] = [] + results = self._all.result() if self._all else () + self._cleanup() + + for result in results: + if result is None: + continue + if not isinstance(result, asyncio.CancelledError): + exceptions.append(result) + + if len(exceptions) == 1: + raise exceptions.pop() + if len(exceptions) > 1: + raise MultiException(exceptions) + + +class AsyncProtocol(Generic[T]): + """AsyncProtocol implements a generic async message-based protocol. + + This protocol assumes the basic unit of information transfer between + client and server is a "message", the details of which are left up + to the implementation. It assumes the sending and receiving of these + messages is full-duplex and not necessarily correlated; i.e. it + supports asynchronous inbound messages. + + It is designed to be extended by a specific protocol which provides + the implementations for how to read and send messages. These must be + defined in `_do_recv()` and `_do_send()`, respectively. + + Other callbacks that have a default implemention, but may be + extended or overridden: + - _on_connect: Actions performed prior to loop start. + - _on_start: Actions performed immediately after loop start. + - _on_message: Actions performed when a message is received. + The default implementation does nothing at all. + - _cb_outbound: Log/Filter outgoing messages. + - _cb_inbound: Log/Filter incoming messages. + + :param name: Name used for logging messages, if any. + """ + #: Logger object for debugging messages + logger = logging.getLogger(__name__) + + # ------------------------- + # Section: Public interface + # ------------------------- + + def __init__(self, name: Optional[str] = None) -> None: + self.name = name + if self.name is not None: + self.logger = self.logger.getChild(self.name) + + # stream I/O + self._reader: Optional[StreamReader] = None + self._writer: Optional[StreamWriter] = None + + # I/O queues + self._outgoing: asyncio.Queue[T] = asyncio.Queue() + + # I/O tasks (message reader, message writer) + self._tasks = AsyncTasks(self.logger) + + # Disconnect task; separate from the core loop. + self._dc_task: Optional[asyncio.Future[None]] = None + + @property + def running(self) -> bool: + """ + Return True when the loop is currently connected and running. + """ + if self.disconnecting: + return False + return self._tasks.running + + @property + def disconnecting(self) -> bool: + """ + Return True when the loop is disconnecting, or disconnected. + """ + return bool(self._dc_task) + + @property + def unconnected(self) -> bool: + """ + Return True when the loop is fully idle and quiesced. + + Returns True specifically when the loop is neither `running` + nor `disconnecting`. A call to `disconnect()` is required + to transition from `disconnecting` to `unconnected`. + """ + return not (self.running or self.disconnecting) + + async def accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Accept a connection and begin processing message queues. + + :param address: Address to connect to; + UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise: `StateError` (loop is running or disconnecting.) + :raise: `ConnectError` (Connection was not successful.) + """ + if self.disconnecting: + raise StateError("Client is disconnecting/disconnected." + " Call disconnect() to fully disconnect.") + if self.running: + raise StateError("Client is already connected and running.") + assert self.unconnected + + try: + await self._new_session(self._do_accept(address, ssl)) + except Exception as err: + emsg = "Failed to accept incoming connection" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise ConnectError(f"{emsg}: {err!s}") from err + + async def connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Connect to the server and begin processing message queues. + + :param address: Address to connect to; + UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise: `StateError` (loop is running or disconnecting.) + :raise: `ConnectError` (Connection was not successful.) + """ + if self.disconnecting: + raise StateError("Client is disconnecting/disconnected." + " Call disconnect() to fully disconnect.") + if self.running: + raise StateError("Client is already connected and running.") + assert self.unconnected + + try: + await self._new_session(self._do_connect(address, ssl)) + except Exception as err: + emsg = "Failed to connect to server" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise ConnectError(f"{emsg}: {err!s}") from err + + async def disconnect(self) -> None: + """ + Disconnect and wait for all tasks to fully stop. + + If there were exceptions that caused the bottom half to terminate + prematurely, they will be raised here. + + :raise: `Exception` Arbitrary exceptions re-raised on behalf of + the bottom half. + :raise: `MultiException` Iterable Exception used to multiplex multiple + exceptions when multiple tasks failed. + """ + self._schedule_disconnect() + await self._wait_disconnect() + + # ----------------------------- + # Section: Connection machinery + # ----------------------------- + + async def _register_streams(self, + reader: asyncio.StreamReader, + writer: asyncio.StreamWriter) -> None: + """Register the Reader/Writer streams.""" + self._reader = reader + self._writer = writer + + async def _new_session(self, coro: Awaitable[None]) -> None: + """ + Create a new session. + + This is called for both `accept()` and `connect()` pathways. + + :param coro: An awaitable that will perform either connect or accept. + """ + assert self._reader is None + assert self._writer is None + + # NB: If a previous session had stale messages, they are dropped here. + self._outgoing = asyncio.Queue() + + # Connect / Await Connection + await coro + assert self._reader is not None + assert self._writer is not None + + await self._on_connect() + + reader_coro = self._bh_loop_forever(self._bh_recv_message, 'Reader') + writer_coro = self._bh_loop_forever(self._bh_send_message, 'Writer') + self._tasks.start(reader_coro, writer_coro) + + await self._on_start() + + async def _do_accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Acting as the protocol server, accept a single connection. + + Used as the awaitable callback to `_new_session()`. + """ + self.logger.debug("Awaiting connection ...") + connected = asyncio.Event() + server: Optional[asyncio.AbstractServer] = None + + async def _client_connected_cb(reader: asyncio.StreamReader, + writer: asyncio.StreamWriter) -> None: + """Used to accept a single incoming connection, see below.""" + nonlocal server + nonlocal connected + + # A connection has been accepted; stop listening for new ones. + assert server is not None + server.close() + await server.wait_closed() + server = None + + # Register this client as being connected + await self._register_streams(reader, writer) + + # Signal back: We've accepted a client! + connected.set() + + if isinstance(address, tuple): + coro = asyncio.start_server( + _client_connected_cb, + host=address[0], + port=address[1], + ssl=ssl, + backlog=1, + ) + else: + coro = asyncio.start_unix_server( + _client_connected_cb, + path=address, + ssl=ssl, + backlog=1, + ) + + server = await coro # Starts listening + await connected.wait() # Waits for the callback to fire (and finish) + assert server is None + + self.logger.debug("Connection accepted") + + async def _do_connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + self.logger.debug("Connecting ...") + + if isinstance(address, tuple): + connect = asyncio.open_connection(address[0], address[1], ssl=ssl) + else: + connect = asyncio.open_unix_connection(path=address, ssl=ssl) + reader, writer = await(connect) + await self._register_streams(reader, writer) + + self.logger.debug("Connected") + + async def _on_connect(self) -> None: + """ + Async callback invoked after connection, but prior to loop start. + + This callback is invoked after the stream is opened, but prior to + starting the reader/writer tasks. Use this callback to handle + handshakes, greetings, &c to avoid having special edge cases in the + generic message handler. + """ + # Nothing to do in the general case. + + async def _on_start(self) -> None: + """ + Async callback invoked after connection and loop start. + + This callback is invoked after the stream is opened AND after + the reader/writer tasks have been started. Use this callback to + auto-perform certain tasks during the connect() call. + """ + # Nothing to do in the general case. + + def _schedule_disconnect(self) -> None: + """ + Initiate a disconnect; idempotent. + + This is called by the reader/writer tasks upon exceptions, + or directly by a user call to `disconnect()`. + """ + if not self._dc_task: + self._dc_task = create_task(self._bh_disconnect()) + + async def _wait_disconnect(self) -> None: + """ + _wait_disconnect waits for a scheduled disconnect to finish. + + This function will gather any bottom half exceptions and re-raise them; + so it is intended to be used in the upper half call chain. + + If a single exception is encountered, it will be re-raised faithfully. + If multiple are found, they will be multiplexed into a MultiException. + + :raise: `Exception` Many kinds; anything the bottom half raises. + :raise: `MultiException` When the Reader/Writer both have exceptions. + """ + assert self._dc_task + await self._dc_task + self._dc_task = None + + try: + self._tasks.result() + finally: + self._cleanup() + + def _cleanup(self) -> None: + """ + Fully reset this object to a clean state. + """ + assert not self.running + assert self._dc_task is None + # _tasks.result() called in _wait_disconnect does _tasks cleanup, so: + assert not self._tasks + + self._reader = None + self._writer = None + + # ------------------------------ + # Section: Bottom Half functions + # ------------------------------ + + async def _bh_disconnect(self) -> None: + """ + Disconnect and cancel all outstanding tasks. + + It is designed to be called from its task context, self._dc_task. + """ + # RFC: Maybe I shot myself in the foot by trying too hard to + # group the tasks together as one unit. I suspect the ideal + # cancellation order here is actually: MessageWriter, + # StreamWriter, MessageReader + + # What I have here instead is MessageWriter, MessageReader, + # StreamWriter + + # Cancel the the message reader/writer. + await self._tasks.cancel() + + # Handle the stream writer itself, now. + if self._writer: + if not self._writer.is_closing(): + self.logger.debug("Writer is open; draining") + await self._writer.drain() + self.logger.debug("Closing writer") + self._writer.close() + self.logger.debug("Awaiting writer to fully close") + await wait_closed(self._writer) + self.logger.debug("Fully closed.") + + # TODO: Add a hook for higher-level protocol cancellations here? + # (Otherwise, the disconnected logging event happens too soon.) + + self.logger.debug("Protocol Disconnected.") + + async def _bh_loop_forever(self, async_fn: _TaskFN, name: str) -> None: + """ + Run one of the bottom-half functions in a loop forever. + + If the bottom half ever raises any exception, schedule a disconnect. + """ + try: + while True: + await async_fn() + except asyncio.CancelledError as err: + # We are cancelled (by _bh_disconnect), so no need to call it. + self.logger.debug("Task.%s: cancelled: %s.", + name, type(err).__name__) + raise + except: + self.logger.error("Task.%s: failure:\n%s\n", name, + pretty_traceback()) + self.logger.debug("Task.%s: scheduling disconnect.", name) + self._schedule_disconnect() + raise + finally: + self.logger.debug("Task.%s: exiting.", name) + + async def _bh_send_message(self) -> None: + """ + Wait for an outgoing message, then send it. + """ + self.logger.log(5, "Waiting for message in outgoing queue to send ...") + msg = await self._outgoing.get() + try: + self.logger.log(5, "Got outgoing message, sending ...") + await self._send(msg) + finally: + self._outgoing.task_done() + self.logger.log(5, "Outgoing message sent.") + + async def _bh_recv_message(self) -> None: + """ + Wait for an incoming message and call `_on_message` to route it. + + Exceptions seen may be from `_recv` or from `_on_message`. + """ + self.logger.log(5, "Waiting to receive incoming message ...") + msg = await self._recv() + self.logger.log(5, "Routing message ...") + await self._on_message(msg) + self.logger.log(5, "Message routed.") + + # --------------------- + # Section: Datagram I/O + # --------------------- + + def _cb_outbound(self, msg: T) -> T: + """ + Callback: outbound message hook. + + This is intended for subclasses to be able to add arbitrary hooks to + filter or manipulate outgoing messages. The base implementation + does nothing but log the message without any manipulation of the + message. It is designed for you to invoke super() at the tail of + any overridden method. + + :param msg: raw outbound message + :return: final outbound message + """ + self.logger.debug("--> %s", str(msg)) + return msg + + def _cb_inbound(self, msg: T) -> T: + """ + Callback: inbound message hook. + + This is intended for subclasses to be able to add arbitrary hooks to + filter or manipulate incoming messages. The base implementation + does nothing but log the message without any manipulation of the + message. It is designed for you to invoke super() at the head of + any overridden method. + + This method does not "handle" incoming messages; it is a filter. + The actual "endpoint" for incoming messages is `_on_message()`. + + :param msg: raw inbound message + :return: processed inbound message + """ + self.logger.debug("<-- %s", str(msg)) + return msg + + async def _readline(self) -> bytes: + """ + Wait for a newline from the incoming reader. + + This method is provided as a convenience for upper-layer + protocols, as many will be line-based. + + This function *may* return a sequence of bytes without a + trailing newline if EOF occurs, but *some* bytes were + received. In this case, the next call will raise EOF. + + :raise OSError: Stream-related errors. + :raise EOFError: If the reader stream is at EOF and there + are no bytes to return. + """ + assert self._reader is not None + msg_bytes = await self._reader.readline() + self.logger.log(5, "Read %d bytes", len(msg_bytes)) + + if not msg_bytes: + if self._reader.at_eof(): + self.logger.debug("EOF") + raise EOFError() + + return msg_bytes + + async def _do_recv(self) -> T: + """ + Abstract: Read from the stream and return a message. + + Very low-level; intended to only be called by `_recv()`. + """ + raise NotImplementedError + + async def _recv(self) -> T: + """ + Read an arbitrary protocol message. (WARNING: Extremely low-level.) + + This function is intended primarily for _bh_recv_message to use + in an asynchronous task loop. Using it outside of this loop will + "steal" messages from the normal routing mechanism. It is safe to + use during `_on_connect()`, but should not be used otherwise. + + This function uses `_do_recv()` to retrieve the raw message, and + then transforms it using `_cb_inbound()`. + + Errors raised may be any of those from either method implementation. + + :return: A single (filtered, processed) protocol message. + """ + message = await self._do_recv() + return self._cb_inbound(message) + + def _do_send(self, msg: T) -> None: + """ + Abstract: Write a message to the stream. + + Very low-level; intended to only be called by `_send()`. + """ + raise NotImplementedError + + async def _send(self, msg: T) -> None: + """ + Send an arbitrary protocol message. (WARNING: Low-level.) + + Like `_read()`, this function is intended to be called by the writer + task loop that processes outgoing messages. This function will + transform any outgoing messages according to `_cb_outbound()`. + + :raise: OSError - Various stream errors. + """ + assert self._writer is not None + msg = self._cb_outbound(msg) + self._do_send(msg) + + async def _on_message(self, msg: T) -> None: + """ + Called when a new message is received. + + Executed from within the reader loop BH, so be advised that waiting + on other asynchronous tasks may be risky, depending. Additionally, + any errors raised here will directly cause the loop to halt; limit + error checking to what is strictly necessary for message routing. + + :param msg: The incoming message, already logged/filtered. + """ + # Nothing to do in the abstract case. From patchwork Tue Apr 13 15:55:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465849 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=gk0Javkq; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVl700dpz9sTD for ; Wed, 14 Apr 2021 02:03:30 +1000 (AEST) Received: from localhost ([::1]:56214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLVg-0000Bg-KL for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 12:03:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38436) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOc-0001TY-F8 for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:24859) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOY-0000BT-7p for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329365; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XufJ2CLgjzIAubGvoiYCX7qy1kGVUEO3CHR9Ppm6GRY=; b=gk0JavkqhuQHwSWAFl/tOfIL0oZtdRiSa+ZPheVghXnh4PTKzMrjj2akjWVH/5/8uIkjC2 CppRyw3YPzED41hCTrGrFi9GD17tXGo41uvNTr2CiqfKRaaRQPIjcK1YIrvbBpSNOXHjN0 pNcQoMb8QSk7VgchKyNib1FSdJzCbhc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-312-f9L2cPjUPwS_iyt-XlwkqA-1; Tue, 13 Apr 2021 11:56:03 -0400 X-MC-Unique: f9L2cPjUPwS_iyt-XlwkqA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7393B79EC5 for ; Tue, 13 Apr 2021 15:56:02 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id B4EB06A034; Tue, 13 Apr 2021 15:56:01 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 4/7] message: add QMP Message type Date: Tue, 13 Apr 2021 11:55:50 -0400 Message-Id: <20210413155553.2660523-5-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This is an abstraction that represents a single message either sent to or received from the server. It is used to subclass the AsyncProtocol(Generic[T]) type. It was written such that it can be populated by either raw data or by a dict, with the other form being generated on-demand, as-needed. It behaves almost exactly like a dict, but has some extra methods and a special constructor. (It should quack fairly convincingly.) Signed-off-by: John Snow --- message.py | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 message.py diff --git a/message.py b/message.py new file mode 100644 index 0000000..5c7e828 --- /dev/null +++ b/message.py @@ -0,0 +1,196 @@ +""" +QMP Message format and errors. + +This module provides the `Message` class, which represents a single QMP +message sent to or from the server. Several error-classes that depend on +knowing the format of this message are also included here. +""" + +import json +from json import JSONDecodeError +from typing import ( + Dict, + ItemsView, + Iterable, + KeysView, + Optional, + Union, + ValuesView, +) + +from error import ( + DeserializationError, + ProtocolError, + UnexpectedTypeError, +) + + +class Message: + """ + Represents a single QMP protocol message. + + QMP uses JSON objects as its basic communicative unit; so this + object behaves like a MutableMapping. It may be instantiated from + either another mapping (like a dict), or from raw bytes that still + need to be deserialized. + + :param value: Initial value, if any. + :param eager: When true, attempt to serialize (or deserialize) the + initial value immediately, such that conversion exceptions + are raised during the call to the initialization method. + """ + # TODO: make Message properly a MutableMapping so it can be typed as such? + def __init__(self, + value: Union[bytes, Dict[str, object]] = b'', *, + eager: bool = True): + self._data: Optional[bytes] = None + self._obj: Optional[Dict[str, object]] = None + + if isinstance(value, bytes): + self._data = value + if eager: + self._obj = self._deserialize(self._data) + else: + self._obj = value + if eager: + self._data = self._serialize(self._obj) + + @classmethod + def _serialize(cls, value: object) -> bytes: + """ + Serialize a JSON object as bytes. + + :raises: ValueError, TypeError from the json library. + """ + return json.dumps(value, separators=(',', ':')).encode('utf-8') + + @classmethod + def _deserialize(cls, data: bytes) -> Dict[str, object]: + """ + Deserialize JSON bytes into a native python dict. + + :raises: DeserializationError if JSON deserialization + fails for any reason. + :raises: UnexpectedTypeError if data does not represent + a JSON object. + """ + try: + obj = json.loads(data) + except JSONDecodeError as err: + emsg = "Failed to deserialize QMP message." + raise DeserializationError(emsg, data) from err + if not isinstance(obj, dict): + raise UnexpectedTypeError( + "Incoming QMP message is not a JSON object.", + data + ) + return obj + + @property + def data(self) -> bytes: + """ + bytes representing this QMP message. + + Generated on-demand if required. + """ + if self._data is None: + self._data = self._serialize(self._obj or {}) + return self._data + + @property + def _object(self) -> Dict[str, object]: + """ + dict representing this QMP message. + + Generated on-demand if required; Private because it returns an + object that could be used to validate the internal state of the + Message object. + """ + if self._obj is None: + self._obj = self._deserialize(self._data or b'') + return self._obj + + def __str__(self) -> str: + """Pretty-printed representation of this QMP message.""" + return json.dumps(self._object, indent=2) + + def __bytes__(self) -> bytes: + return self.data + + def __contains__(self, item: str) -> bool: # Container, Collection + return item in self._object + + def __iter__(self) -> Iterable[str]: # Iterable, Collection, Mapping + return iter(self._object) + + def __len__(self) -> int: # Sized, Collection, Mapping + return len(self._object) + + def __getitem__(self, key: str) -> object: # Mapping + return self._object[key] + + def __setitem__(self, key: str, value: object) -> None: # MutableMapping + self._object[key] = value + self._data = None + + def __delitem__(self, key: str) -> None: # MutableMapping + del self._object[key] + self._data = None + + def keys(self) -> KeysView[str]: + """Return a KeysView object containing all field names.""" + return self._object.keys() + + def items(self) -> ItemsView[str, object]: + """Return an ItemsView object containing all key:value pairs.""" + return self._object.items() + + def values(self) -> ValuesView[object]: + """Return a ValuesView object containing all field values.""" + return self._object.values() + + def get(self, key: str, + default: Optional[object] = None) -> Optional[object]: + """Get the value for a single key.""" + return self._object.get(key, default) + + +class MsgProtocolError(ProtocolError): + """Abstract error class for protocol errors that have a JSON object.""" + def __init__(self, error_message: str, msg: Message): + super().__init__(error_message) + self.msg = msg + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" Message was: {str(self.msg)}\n", + ]) + + +class ObjectTypeError(MsgProtocolError): + """ + Incoming message was a JSON object, but has an unexpected data shape. + + e.g.: A malformed greeting may cause this error. + """ + + +# FIXME: Remove this? Current draft simply trashes these replies. + +# class OrphanedError(MsgProtocolError): +# """ +# Received message, but had no queue to deliver it to. +# +# e.g.: A reply arrives from the server, but the ID does not match any +# pending execution requests we are aware of. +# """ + + +class ServerParseError(MsgProtocolError): + """ + Server sent a `ParsingError` message. + + e.g. A reply arrives from the server, but it is missing the "ID" + field, which indicates a parsing error on behalf of the server. + """ From patchwork Tue Apr 13 15:55:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465848 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bgak2QpU; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVky3qbyz9sTD for ; Wed, 14 Apr 2021 02:03:22 +1000 (AEST) Received: from localhost ([::1]:55704 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLVX-0008PY-Jk for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 12:03:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOd-0001WN-Rt for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27788) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOZ-0000Ba-KD for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aIfCrRGZsQcxyjJLMuy+Tn71Ybe8bopsExGO/vtKykY=; b=bgak2QpUS2wL8uOaAP+svWD4Eb93oYsxeEfWHgxXMIZFrJee4wW+qxwfYmQMlzSzzrK/Ko eUE8AzwUv1JaPJd5JIqIVyqUtoI/SY7ZTZjRv4Mys9vNGL3LsyLZRTke4C/pUhLTPrbF2B MLUFizn9SoU4Dvdb/7CVIvznu8vwVao= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-470-r7QJUgH3NouA1pfwAbTTag-1; Tue, 13 Apr 2021 11:56:04 -0400 X-MC-Unique: r7QJUgH3NouA1pfwAbTTag-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 683E28710F4 for ; Tue, 13 Apr 2021 15:56:03 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9F2BD6A034; Tue, 13 Apr 2021 15:56:02 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 5/7] models: Add well-known QMP objects Date: Tue, 13 Apr 2021 11:55:51 -0400 Message-Id: <20210413155553.2660523-6-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This uses the third-party pydantic library to provide grammatical validation of various JSON objects used in the QMP protocol, along with documentation that references where these objects are defined. This is done both to ensure that objects conform to the standard set forth in the QMP specification, and to provide a strict type-safe interface that can be used to access information sent by the server in a type-safe way. If you've not run into pydantic before, you define objects by creating classes that inherit from BaseModel. Then, similar to Python's own @dataclass format, you declare the fields (and their types) that you expect to see in this object. Pydantic will then automatically generate a parser/validator for this object, and the end result is a strictly typed, native Python object that is guaranteed to have the fields specified. NOTE: Pydantic does not, by default, ensure that *extra* fields are not present in the model. This is intentional, as it allows backwards compatibility if new fields should be added to the specification in the future. This strictness feature, however, *can* be added. A debug/strict mode could be added (but is not present in this RFC) to enable that strictness on-demand, but for a general-purpose client it's likely best to leave that disabled. Signed-off-by: John Snow --- models.py | 177 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 models.py diff --git a/models.py b/models.py new file mode 100644 index 0000000..7c42d47 --- /dev/null +++ b/models.py @@ -0,0 +1,177 @@ +""" +QMP message models. + +This module provides definitions for several well-defined JSON object +types that are seen in the QMP wire protocol. Using pydantic, these +models also handle the parsing and validation of these objects in order +to provide strict typing guarantees elsewhere in the library. + +Notably, it provides these object models: + +- `Greeting`: the standard QMP greeting message (and nested children) +- Three types of server RPC response messages: + - `ErrorResponse`: A failed-execution reply. (Application-level failure) + - `SuccessResponse`: A successful execution reply. + - `ParsingError`: A reply indicating the RPC message was not understood. + (Library-level failure, or worse.) +- A special pydantic form of the above three; `ServerResponse`, + used to parse incoming messages. +- `AsynchronousEvent`: A generic event message. +""" + +from typing import ( + Any, + Dict, + List, + Type, + TypeVar, + Union, +) + +from pydantic import BaseModel, Field, root_validator, ValidationError + + +from message import Message, ObjectTypeError + + +class MessageBase(BaseModel): + """ + An abstract pydantic model that represents any QMP object. + + It does not define any fields, so it isn't very useful as a type. + However, it provides a strictly typed parsing helper that allows + us to convert from a QMP `Message` object into a specific model, + so long as that model inherits from this class. + """ + _T = TypeVar('_T', bound='MessageBase') + + @classmethod + def parse_msg(cls: Type[_T], obj: Message) -> _T: + """ + Convert a `Message` into a strictly typed Python object. + + For Messages that do not pass validation, pydantic validation + errors are encapsulated using the `ValidationError` class. + + :raises: ValidationError when the given Message cannot be + validated (and converted into) as an instance of this class. + """ + try: + return cls.parse_obj(obj) + except ValidationError as err: + raise ObjectTypeError("Message failed validation.", obj) from err + + +class VersionTriple(BaseModel): + """ + Mirrors qapi/control.json VersionTriple structure. + """ + major: int + minor: int + micro: int + + +class VersionInfo(BaseModel): + """ + Mirrors qapi/control.json VersionInfo structure. + """ + qemu: VersionTriple + package: str + + +class QMPGreeting(BaseModel): + """ + 'QMP' subsection of the protocol greeting. + + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + """ + version: VersionInfo + capabilities: List[str] + + +class Greeting(MessageBase): + """ + QMP protocol greeting message. + + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + """ + QMP: QMPGreeting + + +class ErrorInfo(BaseModel): + """ + Error field inside of an error response. + + Defined in qmp-spec.txt, section 2.4.2, "error". + """ + class_: str = Field(None, alias='class') + desc: str + + +class ParsingError(MessageBase): + """ + Parsing error from QMP that omits ID due to failure. + + Implicitly defined in qmp-spec.txt, section 2.4.2, "error". + """ + error: ErrorInfo + + +class SuccessResponse(MessageBase): + """ + Successful execution response. + + Defined in qmp-spec.txt, section 2.4.1, "success". + """ + return_: Any = Field(None, alias='return') + id: str # NB: The spec allows ANY object here. AQMP does not! + + @root_validator(pre=True) + @classmethod + def check_return_value(cls, + values: Dict[str, object]) -> Dict[str, object]: + """Enforce that the 'return' key is present, even if it is None.""" + # To pydantic, 'Any' means 'Optional'; force its presence: + if 'return' not in values: + raise TypeError("'return' key not present in object.") + return values + + +class ErrorResponse(MessageBase): + """ + Unsuccessful execution response. + + Defined in qmp-spec.txt, section 2.4.2, "error". + """ + error: ErrorInfo + id: str # NB: The spec allows ANY object here. AQMP does not! + + +class ServerResponse(MessageBase): + """ + Union type: This object can be any one of the component messages. + + Implicitly defined in qmp-spec.txt, section 2.4, "Commands Responses". + """ + __root__: Union[SuccessResponse, ErrorResponse, ParsingError] + + +class EventTimestamp(BaseModel): + """ + Timestamp field of QMP event, see `AsynchronousEvent`. + + Defined in qmp-spec.txt, section 2.5, "Asynchronous events". + """ + seconds: int + microseconds: int + + +class AsynchronousEvent(BaseModel): + """ + Asynchronous event message. + + Defined in qmp-spec.txt, section 2.5, "Asynchronous events". + """ + event: str + data: Union[List[Any], Dict[str, Any], str, int, float] + timestamp: EventTimestamp From patchwork Tue Apr 13 15:55:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465845 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HiGY5hfd; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVh26YtPz9sTD for ; Wed, 14 Apr 2021 02:00:50 +1000 (AEST) Received: from localhost ([::1]:51202 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLT6-00064r-GC for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 12:00:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38474) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOe-0001Yd-Rt for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28303) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOa-0000CX-MM for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yLeqvIncFtCrJirz22C15TIsYE2EQq93tbemMjV8Hxo=; b=HiGY5hfdsPTzq8B167stI13SgQS3KSx+cwDzV/ouhIdFHtc57S+nmwUTz1W0cJdZuocRSV RszQ5QUGggWSncnUICMxhfE4O9yxFF7Trrwr71Um+/ORg1QNxYNll7KanbXrL6ioBQIf+K 59IskCz9cxXbsUGvm/tP2c793K2N198= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-188-yG1g1YauN76KAzHK4Um-yg-1; Tue, 13 Apr 2021 11:56:05 -0400 X-MC-Unique: yG1g1YauN76KAzHK4Um-yg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C904B1020C23 for ; Tue, 13 Apr 2021 15:56:04 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F30B50D15; Tue, 13 Apr 2021 15:56:03 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 6/7] qmp_protocol: add QMP client implementation Date: Tue, 13 Apr 2021 11:55:52 -0400 Message-Id: <20210413155553.2660523-7-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Using everything added so far, add the QMP client itself. So far, this QMP object cannot actually pretend to be a server; it only implements the client logic (receiving events and sending commands.) Future work may involve implementing the ability to send events and receive RPC commands, so that we can create a QMP test server for unit test purposes. (It can, however, both connect to or receive a connection from QEMU so that it can be used to instrument iotests.) Note: the event handling is a total hack; I need to figure out the most delightful way to create an interface to consume these easily, as I think it's one of the biggest shortcomings of the synchronous library so far. Consider that part very much a work-in-progress. Signed-off-by: John Snow --- qmp_protocol.py | 420 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 420 insertions(+) create mode 100644 qmp_protocol.py diff --git a/qmp_protocol.py b/qmp_protocol.py new file mode 100644 index 0000000..6e6ac25 --- /dev/null +++ b/qmp_protocol.py @@ -0,0 +1,420 @@ +""" +QMP Client Implementation + +This module provides the QMP class, which can be used to connect and +send commands to a QMP server such as QEMU. The QMP class can be used to +either connect to a listening server, or used to listen and accept an +incoming connection from the server. +""" + +import asyncio +import logging +from typing import ( + Awaitable, + Callable, + Dict, + List, + Mapping, + Optional, + Tuple, + cast, +) + +from error import ( + AQMPError, + DisconnectedError, + DeserializationError, + GreetingError, + NegotiationError, + StateError, + UnexpectedTypeError, +) +from message import ( + Message, + ObjectTypeError, + ServerParseError, +) +from models import ( + ErrorInfo, + ErrorResponse, + Greeting, + ParsingError, + ServerResponse, + SuccessResponse, +) +from protocol import AsyncProtocol +from util import create_task, pretty_traceback + + +class ExecuteError(AQMPError): + """Execution statement returned failure.""" + def __init__(self, + sent: Message, + received: Message, + error: ErrorInfo): + super().__init__() + self.sent = sent + self.received = received + self.error = error + + def __str__(self) -> str: + return self.error.desc + + +_EventCallbackFn = Callable[['QMP', Message], Awaitable[None]] + + +class QMP(AsyncProtocol[Message]): + """ + Implements a QMP connection to/from the server. + + Basic usage looks like this:: + + qmp = QMP('my_virtual_machine_name') + await qmp.connect(('127.0.0.1', 1234)) + ... + res = await qmp.execute('block-query') + ... + await qmp.disconnect() + + :param name: Optional nickname for the connection, used for logging. + """ + #: Logger object for debugging messages + logger = logging.getLogger(__name__) + + def __init__(self, name: Optional[str] = None) -> None: + super().__init__(name) + + # Greeting + self.await_greeting = True + self._greeting: Optional[Greeting] = None + self.greeting_timeout = 5 # (In seconds) + + # RFC: Do I even want to use any timeouts internally? They're + # not defined in the protocol itself. Theoretically, a client + # could simply use asyncio.wait_for(qmp.connect(...), timeout=5) + # and then I don't have to support this interface at all. + # + # We don't need to support any timeouts so long as we never initiate + # any long-term wait that wasn't in direct response to a user action. + + # Command ID counter + self._execute_id = 0 + + # Event handling + self._event_queue: asyncio.Queue[Message] = asyncio.Queue() + self._event_callbacks: List[_EventCallbackFn] = [] + + # Incoming RPC reply messages + self._pending: Dict[str, Tuple[ + asyncio.Future[object], + asyncio.Queue[Message]]] = {} + + def on_event(self, func: _EventCallbackFn) -> _EventCallbackFn: + """ + FIXME: Quick hack: decorator to register event handlers. + + Use it like this:: + + @qmp.on_event + async def my_event_handler(qmp, event: Message) -> None: + print(f"Received event: {event['event']}") + + RFC: What kind of event handler would be the most useful in + practical terms? In tests, we are usually waiting for an + event with some criteria to occur; maybe it would be useful + to allow "coroutine" style functions where we can block + until a certain event shows up? + """ + if func not in self._event_callbacks: + self._event_callbacks.append(func) + return func + + async def _new_session(self, coro: Awaitable[None]) -> None: + self._event_queue = asyncio.Queue() + await super()._new_session(coro) + + async def _on_connect(self) -> None: + """ + Wait for the QMP greeting prior to the engagement of the full loop. + + :raise: GreetingError when the greeting is not understood. + """ + if self.await_greeting: + self._greeting = await self._get_greeting() + + async def _on_start(self) -> None: + """ + Perform QMP negotiation right after the loop starts. + + Negotiation is performed afterwards so that the implementation + can simply use `execute()`, which relies on the loop machinery + to be running. + + :raise: NegotiationError if the negotiation fails in some way. + """ + await self._negotiate() + + async def _get_greeting(self) -> Greeting: + """ + :raise: GreetingError (Many causes.) + """ + self.logger.debug("Awaiting greeting ...") + try: + msg = await asyncio.wait_for(self._recv(), self.greeting_timeout) + return Greeting.parse_msg(msg) + except Exception as err: + if isinstance(err, (asyncio.TimeoutError, OSError, EOFError)): + emsg = "Failed to receive Greeting" + elif isinstance(err, (DeserializationError, UnexpectedTypeError)): + emsg = "Failed to understand Greeting" + elif isinstance(err, ObjectTypeError): + emsg = "Failed to validate Greeting" + else: + emsg = "Unknown failure acquiring Greeting" + + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise GreetingError(emsg, err) from err + + async def _negotiate(self) -> None: + """ + :raise: NegotiationError (Many causes.) + """ + self.logger.debug("Negotiating capabilities ...") + arguments: Dict[str, List[str]] = {'enable': []} + if self._greeting and 'oob' in self._greeting.QMP.capabilities: + arguments['enable'].append('oob') + try: + await self.execute('qmp_capabilities', arguments=arguments) + except Exception as err: + # FIXME: what exceptions do we actually expect execute to raise? + emsg = "Failure negotiating capabilities" + self.logger.error("%s:\n%s\n", emsg, pretty_traceback()) + raise NegotiationError(emsg, err) from err + + async def _bh_disconnect(self) -> None: + # See AsyncProtocol._bh_disconnect(). + await super()._bh_disconnect() + + if self._pending: + self.logger.debug("Cancelling pending executions") + for key in self._pending: + self.logger.debug("Cancelling execution %s", key) + # NB: This signals cancellation, but doesn't fully quiesce; + # it merely requests the cancellation; it will be thrown into + # that tasks's context on the next event loop cycle. + # + # This task is being awaited on by `_execute()`, which will + # exist in the user's callstack in the upper-half. Since + # we're here, we know it isn't running! It won't have a + # chance to run again except to receive a cancellation. + # + # NB: Python 3.9 adds a msg= parameter to cancel that would + # be useful for debugging the 'cause' of cancellations. + self._pending[key][0].cancel() + + self.logger.debug("QMP Disconnected.") + + async def _on_message(self, msg: Message) -> None: + """ + Add an incoming message to the appropriate queue/handler. + + :raise: RawProtocolError (`_recv` via `Message._deserialize`) + :raise: ServerParseError (Message has no 'event' nor 'id' field) + """ + # Incoming messages are not fully parsed/validated here; + # do only light peeking to know how to route the messages. + + if 'event' in msg: + await self._event_queue.put(msg) + # FIXME: quick hack; event queue handling. + for func in self._event_callbacks: + await func(self, msg) + return + + # Below, we assume everything left is an execute/exec-oob response. + + if 'id' in msg: + exec_id = str(msg['id']) + if exec_id not in self._pending: + # qmp-spec.txt, section 2.4: + # 'Clients should drop all the responses + # that have an unknown "id" field.' + self.logger.warning("Unknown ID '%s', response dropped.", + exec_id) + return + else: + # This is a server parsing error; + # It inherently does not "belong" to any pending execution. + # Instead of performing clever recovery, just terminate. + raise ServerParseError( + "Server sent a message without an ID," + " indicating parse failure.", msg) + + _, queue = self._pending[exec_id] + await queue.put(msg) + + async def _do_recv(self) -> Message: + """ + :raise: OSError (Stream errors) + :raise: `EOFError` (When the stream is at EOF) + :raise: `RawProtocolError` (via `Message._deserialize`) + + :return: A single QMP `Message`. + """ + msg_bytes = await self._readline() + msg = Message(msg_bytes, eager=True) + return msg + + def _do_send(self, msg: Message) -> None: + """ + :raise: ValueError (JSON serialization failure) + :raise: TypeError (JSON serialization failure) + :raise: OSError (Stream errors) + """ + assert self._writer is not None + self._writer.write(bytes(msg)) + + def _cleanup(self) -> None: + super()._cleanup() + self._greeting = None + assert self._pending == {} + self._event_queue = asyncio.Queue() + + @classmethod + def make_execute_msg(cls, cmd: str, + arguments: Optional[Mapping[str, object]] = None, + oob: bool = False) -> Message: + """ + Create an executable message to be sent by `execute_msg` later. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If true, execute "out of band". + + :return: An executable QMP message. + """ + msg = Message({'exec-oob' if oob else 'execute': cmd}) + if arguments is not None: + msg['arguments'] = arguments + return msg + + async def _bh_execute(self, msg: Message, + queue: 'asyncio.Queue[Message]') -> object: + """ + Execute a QMP Message and wait for the result. + + :param msg: Message to execute. + :param queue: The queue we should expect to see a reply delivered to. + + :return: Execution result from the server. + The type depends on the command sent. + """ + if not self.running: + raise StateError("QMP is not running.") + assert self._outgoing + + self._outgoing.put_nowait(msg) + reply_msg = await queue.get() + + # May raise ObjectTypeError (Unlikely - only if it has missing keys.) + reply = ServerResponse.parse_msg(reply_msg).__root__ + assert not isinstance(reply, ParsingError) # Handled by BH + + if isinstance(reply, ErrorResponse): + # Server indicated execution failure. + raise ExecuteError(msg, reply_msg, reply.error) + + assert isinstance(reply, SuccessResponse) + return reply.return_ + + async def _execute(self, msg: Message) -> object: + """ + The same as `execute_msg()`, but without safety mechanisms. + + Does not assign an execution ID and does not check that the form + of the message being sent is valid. + + This method *Requires* an 'id' parameter to be set on the + message, it will not set one for you like `execute()` or + `execute_msg()`. + + Do not use "__aqmp#00000" style IDs, use something else to avoid + potential clashes. If this ID clashes with an ID presently + in-use or otherwise clashes with the auto-generated IDs, the + response routing mechanisms in _on_message may very well fail + loudly enough to cause the entire loop to crash. + + The ID should be a str; or at least something JSON + serializable. It *must* be hashable. + """ + exec_id = cast(str, msg['id']) + self.logger.debug("Execute(%s): '%s'", exec_id, + msg.get('execute', msg.get('exec-oob'))) + + queue: asyncio.Queue[Message] = asyncio.Queue(maxsize=1) + task = create_task(self._bh_execute(msg, queue)) + self._pending[exec_id] = (task, queue) + + try: + result = await task + except asyncio.CancelledError as err: + raise DisconnectedError("Disconnected") from err + finally: + del self._pending[exec_id] + + return result + + async def execute_msg(self, msg: Message) -> object: + """ + Execute a QMP message and return the response. + + :param msg: The QMP `Message` to execute. + :raises: ValueError if the QMP `Message` does not have either the + 'execute' or 'exec-oob' fields set. + :raises: ExecuteError if the server returns an error response. + :raises: DisconnectedError if the connection was terminated early. + + :return: Execution response from the server. The type of object depends + on the command that was issued, though most return a dict. + """ + if not ('execute' in msg or 'exec-oob' in msg): + raise ValueError("Requires 'execute' or 'exec-oob' message") + if self.disconnecting: + raise StateError("QMP is disconnecting/disconnected." + " Call disconnect() to fully disconnect.") + + # FIXME: Copy the message here, to avoid leaking the ID back out. + + exec_id = f"__aqmp#{self._execute_id:05d}" + msg['id'] = exec_id + self._execute_id += 1 + + return await self._execute(msg) + + async def execute(self, cmd: str, + arguments: Optional[Mapping[str, object]] = None, + oob: bool = False) -> object: + """ + Execute a QMP command and return the response. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If true, execute "out of band". + + :raise: ExecuteError if the server returns an error response. + :raise: DisconnectedError if the connection was terminated early. + + :return: Execution response from the server. The type of object depends + on the command that was issued, though most return a dict. + """ + # Note: I designed arguments to be its own argument instead of + # kwparams so that we are able to add other modifiers that + # change execution parameters later on. A theoretical + # higher-level API that is generated against a particular QAPI + # Schema should generate function signatures the way we want at + # that point; modifying those commands to behave differently + # could be performed using context managers that alter the QMP + # loop for any commands that occur within that block. + msg = self.make_execute_msg(cmd, arguments, oob=oob) + return await self.execute_msg(msg) From patchwork Tue Apr 13 15:55:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 1465853 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=NLpuk+/O; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FKVsz3s5Hz9sVv for ; Wed, 14 Apr 2021 02:09:27 +1000 (AEST) Received: from localhost ([::1]:39940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lWLbR-0006Sr-5Z for incoming@patchwork.ozlabs.org; Tue, 13 Apr 2021 12:09:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38478) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOg-0001e6-Bo for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:45321) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lWLOb-0000DF-CI for qemu-devel@nongnu.org; Tue, 13 Apr 2021 11:56:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618329368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/r6tK6tIqWuYWrO6uBNX/wB27wd2jnvURmGy+vaA5G8=; b=NLpuk+/OxWNvpEX2JtwgDZ6rLysGR7MRVOt+fXbYPE0N1oHnErXjbyHNoH3o6peJdiAcnL JQBfGV0mVATLPP3/avTwAHRI6irWhkPAUzMu5mlBBet+GzEqIMGAXZ6DEwVfSjeYKYPg9w M8JAoa5TN5of/bxt/ZAeBlfY0HMHzhs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-264-xmGx03g7MKCdMUKl1AxORg-1; Tue, 13 Apr 2021 11:56:06 -0400 X-MC-Unique: xmGx03g7MKCdMUKl1AxORg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B8CFC189C449 for ; Tue, 13 Apr 2021 15:56:05 +0000 (UTC) Received: from scv.redhat.com (ovpn-117-61.rdu2.redhat.com [10.10.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id F16AF6A039; Tue, 13 Apr 2021 15:56:04 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH RFC 7/7] linter config Date: Tue, 13 Apr 2021 11:55:53 -0400 Message-Id: <20210413155553.2660523-8-jsnow@redhat.com> In-Reply-To: <20210413155553.2660523-1-jsnow@redhat.com> References: <20210413155553.2660523-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: crosa@redhat.com, John Snow , ehabkost@redhat.com, stefanha@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Everything in this series should pass with flake8, pylint, and mypy; but there are a few bits of dust swept under the rug with these config files. Signed-off-by: John Snow --- .flake8 | 2 ++ pylintrc | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 .flake8 create mode 100644 pylintrc diff --git a/.flake8 b/.flake8 new file mode 100644 index 0000000..45d8146 --- /dev/null +++ b/.flake8 @@ -0,0 +1,2 @@ +[flake8] +extend-ignore = E722 # Pylint handles this, but smarter. \ No newline at end of file diff --git a/pylintrc b/pylintrc new file mode 100644 index 0000000..7cf16c0 --- /dev/null +++ b/pylintrc @@ -0,0 +1,53 @@ +[MASTER] + +extension-pkg-allow-list=pydantic + +[MESSAGES CONTROL] + +# disable= + +[REPORTS] + +[REFACTORING] + +[MISCELLANEOUS] + +[LOGGING] + +[BASIC] + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _, + fd, + c, + ns, + rc, + T, + +[VARIABLES] + +[STRING] + +[SPELLING] + +[FORMAT] + +[SIMILARITIES] + +# Ignore imports when computing similarities. +ignore-imports=yes + +[TYPECHECK] + +[CLASSES] + +[IMPORTS] + +[DESIGN] + +[EXCEPTIONS]