[{"id":1768517,"web_url":"http://patchwork.ozlabs.org/comment/1768517/","msgid":"","list_archive_url":null,"date":"2017-09-14T11:15:09","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6442,"url":"http://patchwork.ozlabs.org/api/people/6442/","name":"Marc-André Lureau","email":"marcandre.lureau@gmail.com"},"content":"Hi\n\nOn Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> This series was born from this one:\n>\n> https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>\n> The design comes from Markus, and also the whole-bunch-of discussions\n> in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> Stefan, etc. on discussing the topic (...again!), providing shiny\n> ideas and suggestions. Finally we got such a solution that seems to\n> satisfy everyone.\n>\n> I re-started the versioning since this series is totally different\n> from previous one. Now it's version 1.\n>\n> In case new reviewers come along the way without reading previous\n> discussions, I will try to do a summary on what this is all about.\n>\n> What is OOB execution?\n> ======================\n>\n> It's the shortcut of Out-Of-Band execution, its name is given by\n> Markus. It's a way to quickly execute a QMP request. Say, originally\n> QMP is going throw these steps:\n>\n> JSON Parser --> QMP Dispatcher --> Respond\n> /|\\ (2) (3) |\n> (1) | \\|/ (4)\n> +--------- main thread --------+\n>\n> The requests are executed by the so-called QMP-dispatcher after the\n> JSON is parsed. If OOB is on, we run the command directly in the\n> parser and quickly returns.\n\nAll commands should have the \"id\" field mandatory in this case, else\nthe client will not distinguish the replies coming from the last/oob\nand the previous commands.\n\nThis should probably be enforced upfront by client capability checks,\nmore below.\n\n> Yeah I know in current code the parser calls dispatcher directly\n> (please see handle_qmp_command()). However it's not true again after\n> this series (parser will has its own IO thread, and dispatcher will\n> still be run in main thread). So this OOB does brings something\n> different.\n>\n> There are more details on why OOB and the difference/relationship\n> between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> slightly out of topic (and believe me, it's not easy for me to\n> summarize that). For more information, please refers to [1].\n>\n> Summary ends here.\n>\n> Some Implementation Details\n> ===========================\n>\n> Again, I mentioned that the old QMP workflow is this:\n>\n> JSON Parser --> QMP Dispatcher --> Respond\n> /|\\ (2) (3) |\n> (1) | \\|/ (4)\n> +--------- main thread --------+\n>\n> What this series does is, firstly:\n>\n> JSON Parser QMP Dispatcher --> Respond\n> /|\\ | /|\\ (4) |\n> | | (2) | (3) | (5)\n> (1) | +-----> | \\|/\n> +--------- main thread <-------+\n>\n> And further:\n>\n> queue/kick\n> JSON Parser ======> QMP Dispatcher --> Respond\n> /|\\ | (3) /|\\ (4) |\n> (1) | | (2) | | (5)\n> | \\|/ | \\|/\n> IO thread main thread <-------+\n\nIs the queue per monitor or per client? And is the dispatching going\nto be processed even if the client is disconnected, and are new\nclients going to receive the replies from previous clients commands? I\nbelieve there should be a per-client context, so there won't be \"id\"\nrequest conflicts.\n\n>\n> Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> parser.\n\nFrom a protocol point of view, I find that \"run-oob\" distinction per\ncommand a bit pointless. It helps with legacy client that wouldn't\nexpect out-of-order replies if qemu were to run oob commands oob by\ndefault though. Clients shouldn't care about how/where a command is\nbeing queued or not. If they send a command, they want it processed as\nquickly as possible. However, it can be interesting to know if the\nimplementation of the command will be able to deliver oob, so that\ndata in the introspection could be useful.\n\nI would rather propose a client/server capability in qmp_capabilities,\ncall it \"oob\":\n\nThis capability indicates oob commands support.\n\nAn oob command is a regular client message request with the \"id\"\nmember mandatory, but the reply may be delivered\nout of order by the server if the client supports\nit too.\n\nIf both the server and the client have the \"oob\" capability, the\nserver can handle new client requests while previous requests are being\nprocessed.\n\nIf the client doesn't have the \"oob\" capability, it may still call\nan oob command, and make multiple outstanding calls. In this case,\nthe commands are processed in order, so the replies will also be in\norder. The \"id\" member isn't mandatory in this case.\n\nThe client should match the replies with the \"id\" member associated\nwith the requests.\n\nWhen a client is disconnected, the pending commands are not\nnecessarily cancelled. But the future clients will not get replies from\ncommands they didn't make (they might, however, receive side-effects\nevents).\n\nNote that without \"oob\" support, a client may still receive\n messages (or events) from the server between the time a\nrequest is handled by the server and the reply is received. It must\nthus be prepared to handle dispatching both events and reply after\nsending a request.\n\n\n(see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)\n\n\n> The last patch enables this for \"migrate-incoming\" command.\n>\n> Please review. Thanks.\n>\n> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>\n> Peter Xu (15):\n> char-io: fix possible race on IOWatchPoll\n> qobject: allow NULL for qstring_get_str()\n> qobject: introduce qobject_to_str()\n> monitor: move skip_flush into monitor_data_init\n> qjson: add \"opaque\" field to JSONMessageParser\n> monitor: move the cur_mon hack deeper for QMP\n> monitor: unify global init\n> monitor: create IO thread\n> monitor: allow to use IO thread for parsing\n> monitor: introduce monitor_qmp_respond()\n> monitor: separate QMP parser and dispatcher\n\nThere should be a limit in the number of requests the thread can\nqueue. Before the patch, the limit was enforced by system socket\nbuffering I think. Now, should oob commands still be processed even if\nthe queue is full? If so, the thread can't be suspended.\n\n> monitor: enable IO thread for (qmp & !mux) typed\n> qapi: introduce new cmd option \"allow-oob\"\n> qmp: support out-of-band (oob) execution\n> qmp: let migrate-incoming allow out-of-band\n>\n> chardev/char-io.c | 15 ++-\n> docs/devel/qapi-code-gen.txt | 51 ++++++-\n> include/monitor/monitor.h | 2 +-\n> include/qapi/qmp/dispatch.h | 2 +\n> include/qapi/qmp/json-streamer.h | 8 +-\n> include/qapi/qmp/qstring.h | 1 +\n> monitor.c | 283 +++++++++++++++++++++++++++++++--------\n> qapi/introspect.json | 6 +-\n> qapi/migration.json | 3 +-\n> qapi/qmp-dispatch.c | 34 +++++\n> qga/main.c | 5 +-\n> qobject/json-streamer.c | 7 +-\n> qobject/qjson.c | 5 +-\n> qobject/qstring.c | 13 +-\n> scripts/qapi-commands.py | 19 ++-\n> scripts/qapi-introspect.py | 10 +-\n> scripts/qapi.py | 15 ++-\n> scripts/qapi2texi.py | 2 +-\n> tests/libqtest.c | 5 +-\n> tests/qapi-schema/test-qapi.py | 2 +-\n> trace-events | 2 +\n> vl.c | 3 +-\n> 22 files changed, 398 insertions(+), 95 deletions(-)\n>\n> --\n> 2.7.4\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"JZ6eDrWu\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtGDH0Hldz9rvt\n\tfor ;\n\tThu, 14 Sep 2017 21:15:43 +1000 (AEST)","from localhost ([::1]:46975 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsS7R-0002UK-4o\n\tfor incoming@patchwork.ozlabs.org; Thu, 14 Sep 2017 07:15:41 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:41097)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsS73-0002Tk-JY\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 07:15:24 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsS6w-0000lk-Vu\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 07:15:17 -0400","from mail-io0-x230.google.com ([2607:f8b0:4001:c06::230]:46146)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1dsS6w-0000kB-PF\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 07:15:10 -0400","by mail-io0-x230.google.com with SMTP id d16so17681033ioj.3\n\tfor ; Thu, 14 Sep 2017 04:15:10 -0700 (PDT)","by 10.58.69.77 with HTTP; Thu, 14 Sep 2017 04:15:09 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=KhxH5z44urNDHwWtS1jcqF4V8SInp6HNhaxUst8iHrI=;\n\tb=JZ6eDrWuCpdQo4fwOle8XQ4GIVTkl/VxpUSKZNsQ1o9u/C3ite114qzFnTijYrsflh\n\tdmQ34OfPej/j0HHIhAx/NDUYdWe8WM6hmHucqQatSPOGMfAjdLB//w1gfQpxGHplqMEl\n\tBhthseUIR1eIPeHQhiq6NWgXXMbTmSWIknjrj/Xg23jO/+EhkMuqDrSqydbgUiOtVYve\n\t4BlZlfKTCuE4/t5DlSpDqUEgR3sMf8otfhpai3CCaK+0Ld1O9Pljkd7VuwRP1pPS61Mm\n\tOrLhlViUONqdwJ0DRDUl+NGIGcyTnXICJiStO4ZTCEbvKnQOXZjF3X9UJpL9mzonb6hk\n\tfiPw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=KhxH5z44urNDHwWtS1jcqF4V8SInp6HNhaxUst8iHrI=;\n\tb=iVliNeGfIkzXV3Pt0xXfaxO8H4Ro4JN4il4F3swsl6cHaElbeQTdHFu7J22dxUDz7n\n\tWLNNgqCOlm6pHQ9mpG3fZOFoJG2+50khrvlWIagl5mDC1bGm7Q8eH0BTvH7r/1oBSQXc\n\tOZ4UykiZu4WEEzOGrD+OBmiijjNBVOtEhFNtxjLLrRRmFym7l5fG1XFqEWKDZrfebbXf\n\tV9Du12IjRz5e8hPKBBzf3ocXNobAqUZWUBdnLADjCPivm3h0XTkGT5bES1hYfQ6D+FDD\n\tsx+QPVav4dPI9wsOZBPHklhSJX6outDa1ow1vtVkLr/du9Qz4H+/XNXxtlWFkBtGVxYU\n\tzqWQ==","X-Gm-Message-State":"AHPjjUgcUACK/t0xGAEnhJgTCcsh/3MzUDV0RPT+fLGNzEAaR/R7Pm7B\n\tVPvsSVZBbhcKCJ9KwsvjeI30QBc1PMgBJoqjK5o=","X-Google-Smtp-Source":"AOwi7QAj0IItCDgUPvZZAklvBf/pCBTzUr2sOYdv18MFHbuxp1oE2++pEGaWVeF09kkRz0vB4A+hiP3xfzhbzoBnd1c=","X-Received":"by 10.202.77.201 with SMTP id a192mr1960445oib.311.1505387709743;\n\tThu, 14 Sep 2017 04:15:09 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<1505375436-28439-1-git-send-email-peterx@redhat.com>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>","From":"=?utf-8?q?Marc-Andr=C3=A9_Lureau?= ","Date":"Thu, 14 Sep 2017 13:15:09 +0200","Message-ID":"","To":"Peter Xu ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4001:c06::230","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tQEMU , Stefan Hajnoczi , \n\tPaolo Bonzini ,\n\t\"Dr . David Alan Gilbert\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768666,"web_url":"http://patchwork.ozlabs.org/comment/1768666/","msgid":"<20170914151911.GB7370@stefanha-x1.localdomain>","list_archive_url":null,"date":"2017-09-14T15:19:11","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":17227,"url":"http://patchwork.ozlabs.org/api/people/17227/","name":"Stefan Hajnoczi","email":"stefanha@redhat.com"},"content":"On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> There should be a limit in the number of requests the thread can\n> queue. Before the patch, the limit was enforced by system socket\n> buffering I think. Now, should oob commands still be processed even if\n> the queue is full? If so, the thread can't be suspended.\n\nI agree.\n\nMemory usage must be bounded. The number of requests is less important\nthan the amount of memory consumed by them.\n\nExisting QMP clients that send multiple QMP commands without waiting for\nreplies need to rethink their strategy because OOB commands cannot be\nprocessed if queued non-OOB commands consume too much memory.\n\nStefan","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=stefanha@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtMds6kZtz9sPk\n\tfor ;\n\tFri, 15 Sep 2017 01:19:44 +1000 (AEST)","from localhost ([::1]:48371 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsVva-0001sd-3y\n\tfor incoming@patchwork.ozlabs.org; Thu, 14 Sep 2017 11:19:42 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:40963)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsVvH-0001sC-W7\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 11:19:24 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsVvF-0002T3-BI\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 11:19:23 -0400","from mx1.redhat.com ([209.132.183.28]:51772)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsVvF-0002LJ-4d\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 11:19:21 -0400","from smtp.corp.redhat.com\n\t(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 3829C7E42E;\n\tThu, 14 Sep 2017 15:19:19 +0000 (UTC)","from localhost (ovpn-117-58.ams2.redhat.com [10.36.117.58])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id CC7A4784A5;\n\tThu, 14 Sep 2017 15:19:12 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 3829C7E42E","Date":"Thu, 14 Sep 2017 16:19:11 +0100","From":"Stefan Hajnoczi ","To":"=?iso-8859-1?q?Marc-Andr=E9?= Lureau ","Message-ID":"<20170914151911.GB7370@stefanha-x1.localdomain>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.12","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tThu, 14 Sep 2017 15:19:19 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini , \n\t\"Dr . David Alan Gilbert\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768782,"web_url":"http://patchwork.ozlabs.org/comment/1768782/","msgid":"<20170914185314.GA3280@work-vm>","list_archive_url":null,"date":"2017-09-14T18:53:15","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> Hi\n> \n> On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> > This series was born from this one:\n> >\n> > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >\n> > The design comes from Markus, and also the whole-bunch-of discussions\n> > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> > Stefan, etc. on discussing the topic (...again!), providing shiny\n> > ideas and suggestions. Finally we got such a solution that seems to\n> > satisfy everyone.\n> >\n> > I re-started the versioning since this series is totally different\n> > from previous one. Now it's version 1.\n> >\n> > In case new reviewers come along the way without reading previous\n> > discussions, I will try to do a summary on what this is all about.\n> >\n> > What is OOB execution?\n> > ======================\n> >\n> > It's the shortcut of Out-Of-Band execution, its name is given by\n> > Markus. It's a way to quickly execute a QMP request. Say, originally\n> > QMP is going throw these steps:\n> >\n> > JSON Parser --> QMP Dispatcher --> Respond\n> > /|\\ (2) (3) |\n> > (1) | \\|/ (4)\n> > +--------- main thread --------+\n> >\n> > The requests are executed by the so-called QMP-dispatcher after the\n> > JSON is parsed. If OOB is on, we run the command directly in the\n> > parser and quickly returns.\n> \n> All commands should have the \"id\" field mandatory in this case, else\n> the client will not distinguish the replies coming from the last/oob\n> and the previous commands.\n> \n> This should probably be enforced upfront by client capability checks,\n> more below.\n> \n> > Yeah I know in current code the parser calls dispatcher directly\n> > (please see handle_qmp_command()). However it's not true again after\n> > this series (parser will has its own IO thread, and dispatcher will\n> > still be run in main thread). So this OOB does brings something\n> > different.\n> >\n> > There are more details on why OOB and the difference/relationship\n> > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> > slightly out of topic (and believe me, it's not easy for me to\n> > summarize that). For more information, please refers to [1].\n> >\n> > Summary ends here.\n> >\n> > Some Implementation Details\n> > ===========================\n> >\n> > Again, I mentioned that the old QMP workflow is this:\n> >\n> > JSON Parser --> QMP Dispatcher --> Respond\n> > /|\\ (2) (3) |\n> > (1) | \\|/ (4)\n> > +--------- main thread --------+\n> >\n> > What this series does is, firstly:\n> >\n> > JSON Parser QMP Dispatcher --> Respond\n> > /|\\ | /|\\ (4) |\n> > | | (2) | (3) | (5)\n> > (1) | +-----> | \\|/\n> > +--------- main thread <-------+\n> >\n> > And further:\n> >\n> > queue/kick\n> > JSON Parser ======> QMP Dispatcher --> Respond\n> > /|\\ | (3) /|\\ (4) |\n> > (1) | | (2) | | (5)\n> > | \\|/ | \\|/\n> > IO thread main thread <-------+\n> \n> Is the queue per monitor or per client? And is the dispatching going\n> to be processed even if the client is disconnected, and are new\n> clients going to receive the replies from previous clients commands? I\n> believe there should be a per-client context, so there won't be \"id\"\n> request conflicts.\n> \n> >\n> > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> > parser.\n> \n> From a protocol point of view, I find that \"run-oob\" distinction per\n> command a bit pointless. It helps with legacy client that wouldn't\n> expect out-of-order replies if qemu were to run oob commands oob by\n> default though. Clients shouldn't care about how/where a command is\n> being queued or not. If they send a command, they want it processed as\n> quickly as possible. However, it can be interesting to know if the\n> implementation of the command will be able to deliver oob, so that\n> data in the introspection could be useful.\n> \n> I would rather propose a client/server capability in qmp_capabilities,\n> call it \"oob\":\n> \n> This capability indicates oob commands support.\n\nThe problem is indicating which commands support oob as opposed to\nindicating whether oob is present at all. Future versions will\nprobably make more commands oob-able and a client will want to know\nwhether it can rely on a particular command being non-blocking.\n\n> An oob command is a regular client message request with the \"id\"\n> member mandatory, but the reply may be delivered\n> out of order by the server if the client supports\n> it too.\n> \n> If both the server and the client have the \"oob\" capability, the\n> server can handle new client requests while previous requests are being\n> processed.\n> \n> If the client doesn't have the \"oob\" capability, it may still call\n> an oob command, and make multiple outstanding calls. In this case,\n> the commands are processed in order, so the replies will also be in\n> order. The \"id\" member isn't mandatory in this case.\n> \n> The client should match the replies with the \"id\" member associated\n> with the requests.\n> \n> When a client is disconnected, the pending commands are not\n> necessarily cancelled. But the future clients will not get replies from\n> commands they didn't make (they might, however, receive side-effects\n> events).\n\nWhat's the behaviour on the current monitor?\n\n\n> Note that without \"oob\" support, a client may still receive\n> messages (or events) from the server between the time a\n> request is handled by the server and the reply is received. It must\n> thus be prepared to handle dispatching both events and reply after\n> sending a request.\n> \n> \n> (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)\n> \n> \n> > The last patch enables this for \"migrate-incoming\" command.\n> >\n> > Please review. Thanks.\n> >\n> > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >\n> > Peter Xu (15):\n> > char-io: fix possible race on IOWatchPoll\n> > qobject: allow NULL for qstring_get_str()\n> > qobject: introduce qobject_to_str()\n> > monitor: move skip_flush into monitor_data_init\n> > qjson: add \"opaque\" field to JSONMessageParser\n> > monitor: move the cur_mon hack deeper for QMP\n> > monitor: unify global init\n> > monitor: create IO thread\n> > monitor: allow to use IO thread for parsing\n> > monitor: introduce monitor_qmp_respond()\n> > monitor: separate QMP parser and dispatcher\n> \n> There should be a limit in the number of requests the thread can\n> queue. Before the patch, the limit was enforced by system socket\n> buffering I think. Now, should oob commands still be processed even if\n> the queue is full? If so, the thread can't be suspended.\n\nI think the previous discussion was expecting a pair of queues\nper client and perhaps a pair of central queues; each pair being\nfor normal command and oob commands.\n(I'm not expecting these queues to be deep; IMHO '1' is the\nright size for this type of queue in both cases).\n\nDave\n\n> > monitor: enable IO thread for (qmp & !mux) typed\n> > qapi: introduce new cmd option \"allow-oob\"\n> > qmp: support out-of-band (oob) execution\n> > qmp: let migrate-incoming allow out-of-band\n> >\n> > chardev/char-io.c | 15 ++-\n> > docs/devel/qapi-code-gen.txt | 51 ++++++-\n> > include/monitor/monitor.h | 2 +-\n> > include/qapi/qmp/dispatch.h | 2 +\n> > include/qapi/qmp/json-streamer.h | 8 +-\n> > include/qapi/qmp/qstring.h | 1 +\n> > monitor.c | 283 +++++++++++++++++++++++++++++++--------\n> > qapi/introspect.json | 6 +-\n> > qapi/migration.json | 3 +-\n> > qapi/qmp-dispatch.c | 34 +++++\n> > qga/main.c | 5 +-\n> > qobject/json-streamer.c | 7 +-\n> > qobject/qjson.c | 5 +-\n> > qobject/qstring.c | 13 +-\n> > scripts/qapi-commands.py | 19 ++-\n> > scripts/qapi-introspect.py | 10 +-\n> > scripts/qapi.py | 15 ++-\n> > scripts/qapi2texi.py | 2 +-\n> > tests/libqtest.c | 5 +-\n> > tests/qapi-schema/test-qapi.py | 2 +-\n> > trace-events | 2 +\n> > vl.c | 3 +-\n> > 22 files changed, 398 insertions(+), 95 deletions(-)\n> >\n> > --\n> > 2.7.4\n> >\n> \n> \n> \n> -- \n> Marc-André Lureau\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtSP96QMYz9rvt\n\tfor ;\n\tFri, 15 Sep 2017 04:54:03 +1000 (AEST)","from localhost ([::1]:49528 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsZGy-0004El-SL\n\tfor incoming@patchwork.ozlabs.org; Thu, 14 Sep 2017 14:54:00 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:33288)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsZGa-0004EM-Pn\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:53:38 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsZGX-0001e7-JW\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:53:36 -0400","from mx1.redhat.com ([209.132.183.28]:59612)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsZGX-0001dA-99\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:53:33 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id D308AC04B310;\n\tThu, 14 Sep 2017 18:53:27 +0000 (UTC)","from work-vm (ovpn-116-29.ams2.redhat.com [10.36.116.29])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 762DF6F11A;\n\tThu, 14 Sep 2017 18:53:17 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com D308AC04B310","Date":"Thu, 14 Sep 2017 19:53:15 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"=?iso-8859-1?q?Marc-Andr=E9?= Lureau ","Message-ID":"<20170914185314.GA3280@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tThu, 14 Sep 2017 18:53:27 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768783,"web_url":"http://patchwork.ozlabs.org/comment/1768783/","msgid":"<20170914185603.GB3280@work-vm>","list_archive_url":null,"date":"2017-09-14T18:56:04","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Peter Xu (peterx@redhat.com) wrote:\n> This series was born from this one:\n> \n> https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n\nAre patches 1..6 separable and mergable without the rest ?\n\nDave\n\n> The design comes from Markus, and also the whole-bunch-of discussions\n> in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> Stefan, etc. on discussing the topic (...again!), providing shiny\n> ideas and suggestions. Finally we got such a solution that seems to\n> satisfy everyone.\n> \n> I re-started the versioning since this series is totally different\n> from previous one. Now it's version 1.\n> \n> In case new reviewers come along the way without reading previous\n> discussions, I will try to do a summary on what this is all about.\n> \n> What is OOB execution?\n> ======================\n> \n> It's the shortcut of Out-Of-Band execution, its name is given by\n> Markus. It's a way to quickly execute a QMP request. Say, originally\n> QMP is going throw these steps:\n> \n> JSON Parser --> QMP Dispatcher --> Respond\n> /|\\ (2) (3) |\n> (1) | \\|/ (4)\n> +--------- main thread --------+\n> \n> The requests are executed by the so-called QMP-dispatcher after the\n> JSON is parsed. If OOB is on, we run the command directly in the\n> parser and quickly returns.\n> \n> Yeah I know in current code the parser calls dispatcher directly\n> (please see handle_qmp_command()). However it's not true again after\n> this series (parser will has its own IO thread, and dispatcher will\n> still be run in main thread). So this OOB does brings something\n> different.\n> \n> There are more details on why OOB and the difference/relationship\n> between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> slightly out of topic (and believe me, it's not easy for me to\n> summarize that). For more information, please refers to [1].\n> \n> Summary ends here.\n> \n> Some Implementation Details\n> ===========================\n> \n> Again, I mentioned that the old QMP workflow is this:\n> \n> JSON Parser --> QMP Dispatcher --> Respond\n> /|\\ (2) (3) |\n> (1) | \\|/ (4)\n> +--------- main thread --------+\n> \n> What this series does is, firstly:\n> \n> JSON Parser QMP Dispatcher --> Respond\n> /|\\ | /|\\ (4) |\n> | | (2) | (3) | (5)\n> (1) | +-----> | \\|/\n> +--------- main thread <-------+\n> \n> And further:\n> \n> queue/kick\n> JSON Parser ======> QMP Dispatcher --> Respond\n> /|\\ | (3) /|\\ (4) |\n> (1) | | (2) | | (5)\n> | \\|/ | \\|/\n> IO thread main thread <-------+\n> \n> Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> parser.\n> \n> The last patch enables this for \"migrate-incoming\" command.\n> \n> Please review. Thanks.\n> \n> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> \n> Peter Xu (15):\n> char-io: fix possible race on IOWatchPoll\n> qobject: allow NULL for qstring_get_str()\n> qobject: introduce qobject_to_str()\n> monitor: move skip_flush into monitor_data_init\n> qjson: add \"opaque\" field to JSONMessageParser\n> monitor: move the cur_mon hack deeper for QMP\n> monitor: unify global init\n> monitor: create IO thread\n> monitor: allow to use IO thread for parsing\n> monitor: introduce monitor_qmp_respond()\n> monitor: separate QMP parser and dispatcher\n> monitor: enable IO thread for (qmp & !mux) typed\n> qapi: introduce new cmd option \"allow-oob\"\n> qmp: support out-of-band (oob) execution\n> qmp: let migrate-incoming allow out-of-band\n> \n> chardev/char-io.c | 15 ++-\n> docs/devel/qapi-code-gen.txt | 51 ++++++-\n> include/monitor/monitor.h | 2 +-\n> include/qapi/qmp/dispatch.h | 2 +\n> include/qapi/qmp/json-streamer.h | 8 +-\n> include/qapi/qmp/qstring.h | 1 +\n> monitor.c | 283 +++++++++++++++++++++++++++++++--------\n> qapi/introspect.json | 6 +-\n> qapi/migration.json | 3 +-\n> qapi/qmp-dispatch.c | 34 +++++\n> qga/main.c | 5 +-\n> qobject/json-streamer.c | 7 +-\n> qobject/qjson.c | 5 +-\n> qobject/qstring.c | 13 +-\n> scripts/qapi-commands.py | 19 ++-\n> scripts/qapi-introspect.py | 10 +-\n> scripts/qapi.py | 15 ++-\n> scripts/qapi2texi.py | 2 +-\n> tests/libqtest.c | 5 +-\n> tests/qapi-schema/test-qapi.py | 2 +-\n> trace-events | 2 +\n> vl.c | 3 +-\n> 22 files changed, 398 insertions(+), 95 deletions(-)\n> \n> -- \n> 2.7.4\n> \n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx04.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx04.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtSSL1cV3z9s7h\n\tfor ;\n\tFri, 15 Sep 2017 04:56:50 +1000 (AEST)","from localhost ([::1]:49537 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsZJg-00054r-Bo\n\tfor incoming@patchwork.ozlabs.org; Thu, 14 Sep 2017 14:56:48 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:34607)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsZJA-00053A-In\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:56:19 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsZJ9-00037T-7A\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:56:16 -0400","from mx1.redhat.com ([209.132.183.28]:34038)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsZJ8-00036Q-V8\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 14:56:15 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id EFAA985546;\n\tThu, 14 Sep 2017 18:56:13 +0000 (UTC)","from work-vm (ovpn-116-29.ams2.redhat.com [10.36.116.29])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id EA16760841;\n\tThu, 14 Sep 2017 18:56:06 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com EFAA985546","Date":"Thu, 14 Sep 2017 19:56:04 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"Peter Xu ","Message-ID":"<20170914185603.GB3280@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<1505375436-28439-1-git-send-email-peterx@redhat.com>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.28]);\n\tThu, 14 Sep 2017 18:56:14 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tmdroth@linux.vnet.ibm.com, Juan Quintela ,\n\tMarkus Armbruster , qemu-devel@nongnu.org, Stefan\n\tHajnoczi , =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768935,"web_url":"http://patchwork.ozlabs.org/comment/1768935/","msgid":"<20170915035057.GM3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-15T03:50:57","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > There should be a limit in the number of requests the thread can\n> > queue. Before the patch, the limit was enforced by system socket\n> > buffering I think. Now, should oob commands still be processed even if\n> > the queue is full? If so, the thread can't be suspended.\n> \n> I agree.\n> \n> Memory usage must be bounded. The number of requests is less important\n> than the amount of memory consumed by them.\n> \n> Existing QMP clients that send multiple QMP commands without waiting for\n> replies need to rethink their strategy because OOB commands cannot be\n> processed if queued non-OOB commands consume too much memory.\n\nThanks for pointing out this. Yes the memory usage problem is valid,\nas Markus pointed out as well in previous discussions (in \"Flow\nControl\" section of that long reply). Hopefully this series basically\ncan work from design prospective, then I'll add this flow control in\nnext version.\n\nRegarding to what we should do if the limit is reached: Markus\nprovided a few options, but the one I prefer most is that we don't\nrespond, but send an event showing that a command is dropped.\nHowever, I would like it not queued, but a direct reply (after all,\nit's an event, and we should not need to care much on ordering of it).\nThen we can get rid of the babysitting of those \"to be failed\"\nrequests asap, meanwhile we don't lose anything IMHO.\n\nI think I also missed at least a unit test for this new interface.\nAgain, I'll add it after the whole idea is proved solid. Thanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xthKT3fjTz9t2f\n\tfor ;\n\tFri, 15 Sep 2017 13:51:40 +1000 (AEST)","from localhost ([::1]:50986 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dshfF-0006DV-TE\n\tfor incoming@patchwork.ozlabs.org; Thu, 14 Sep 2017 23:51:37 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:53957)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dshep-0006C6-WF\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:51:12 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dshem-0002wF-U6\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:51:12 -0400","from mx1.redhat.com ([209.132.183.28]:44672)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dshem-0002vw-Nm\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:51:08 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 5C8B64E4CB;\n\tFri, 15 Sep 2017 03:51:07 +0000 (UTC)","from pxdev.xzpeter.org (ovpn-12-86.pek2.redhat.com [10.72.12.86])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id B564974442;\n\tFri, 15 Sep 2017 03:50:59 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 5C8B64E4CB","Date":"Fri, 15 Sep 2017 11:50:57 +0800","From":"Peter Xu ","To":"Stefan Hajnoczi ","Message-ID":"<20170915035057.GM3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170914151911.GB7370@stefanha-x1.localdomain>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.38]);\n\tFri, 15 Sep 2017 03:51:07 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth , Juan Quintela\n\t, Markus Armbruster , QEMU\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , \"Dr . David Alan Gilbert\"\n\t, Paolo Bonzini ,\n\tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768936,"web_url":"http://patchwork.ozlabs.org/comment/1768936/","msgid":"<20170915035817.GN3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-15T03:58:17","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, Sep 14, 2017 at 07:56:04PM +0100, Dr. David Alan Gilbert wrote:\n> * Peter Xu (peterx@redhat.com) wrote:\n> > This series was born from this one:\n> > \n> > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> \n> Are patches 1..6 separable and mergable without the rest ?\n\nYes I think so.\n\n(I was always trying to put pre-requisite patches like these ones at\n the front of any of my series rather than separating them into more\n series, since I thought it is convenient for me to manage them (or\n add new ones when respin), and also easier for reviewers (so people\n don't need to try to find the dependencies). And since I put them at\n the head, we can easily merge them without rebasing issue when they\n are good while the rest may still need further work. Hopefully this\n is the right thing to do.)","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xthcr2vK7z9t2Z\n\tfor ;\n\tFri, 15 Sep 2017 14:05:00 +1000 (AEST)","from localhost ([::1]:51059 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dshsA-0007hy-Ie\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 00:04:58 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:56983)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dshlu-0001q3-Qb\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:58:31 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dshlr-0000rg-QF\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:58:30 -0400","from mx1.redhat.com ([209.132.183.28]:51348)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dshlr-0000pq-Km\n\tfor qemu-devel@nongnu.org; Thu, 14 Sep 2017 23:58:27 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 8F0014E4C2;\n\tFri, 15 Sep 2017 03:58:26 +0000 (UTC)","from pxdev.xzpeter.org (ovpn-12-86.pek2.redhat.com [10.72.12.86])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 47BD678419;\n\tFri, 15 Sep 2017 03:58:20 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 8F0014E4C2","Date":"Fri, 15 Sep 2017 11:58:17 +0800","From":"Peter Xu ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170915035817.GN3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t<20170914185603.GB3280@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170914185603.GB3280@work-vm>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.38]);\n\tFri, 15 Sep 2017 03:58:26 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tmdroth@linux.vnet.ibm.com, Juan Quintela ,\n\tMarkus Armbruster , qemu-devel@nongnu.org, Stefan\n\tHajnoczi , =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1768939,"web_url":"http://patchwork.ozlabs.org/comment/1768939/","msgid":"<20170915044622.GO3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-15T04:46:22","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> > Hi\n> > \n> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> > > This series was born from this one:\n> > >\n> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> > >\n> > > The design comes from Markus, and also the whole-bunch-of discussions\n> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n> > > ideas and suggestions. Finally we got such a solution that seems to\n> > > satisfy everyone.\n> > >\n> > > I re-started the versioning since this series is totally different\n> > > from previous one. Now it's version 1.\n> > >\n> > > In case new reviewers come along the way without reading previous\n> > > discussions, I will try to do a summary on what this is all about.\n> > >\n> > > What is OOB execution?\n> > > ======================\n> > >\n> > > It's the shortcut of Out-Of-Band execution, its name is given by\n> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n> > > QMP is going throw these steps:\n> > >\n> > > JSON Parser --> QMP Dispatcher --> Respond\n> > > /|\\ (2) (3) |\n> > > (1) | \\|/ (4)\n> > > +--------- main thread --------+\n> > >\n> > > The requests are executed by the so-called QMP-dispatcher after the\n> > > JSON is parsed. If OOB is on, we run the command directly in the\n> > > parser and quickly returns.\n> > \n> > All commands should have the \"id\" field mandatory in this case, else\n> > the client will not distinguish the replies coming from the last/oob\n> > and the previous commands.\n> > \n> > This should probably be enforced upfront by client capability checks,\n> > more below.\n\nHmm yes since the oob commands are actually running in async way,\nrequest ID should be needed here. However I'm not sure whether\nenabling the whole \"request ID\" thing is too big for this \"try to be\nsmall\" oob change... And IMHO it suites better to be part of the whole\nasync work (no matter which implementation we'll use).\n\nHow about this: we make \"id\" mandatory for \"run-oob\" requests only.\nFor oob commands, they will always have ID then no ordering issue, and\nwe can do it async; for the rest of non-oob commands, we still allow\nthem to go without ID, and since they are not oob, they'll always be\ndone in order as well. Would this work?\n\n> > \n> > > Yeah I know in current code the parser calls dispatcher directly\n> > > (please see handle_qmp_command()). However it's not true again after\n> > > this series (parser will has its own IO thread, and dispatcher will\n> > > still be run in main thread). So this OOB does brings something\n> > > different.\n> > >\n> > > There are more details on why OOB and the difference/relationship\n> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> > > slightly out of topic (and believe me, it's not easy for me to\n> > > summarize that). For more information, please refers to [1].\n> > >\n> > > Summary ends here.\n> > >\n> > > Some Implementation Details\n> > > ===========================\n> > >\n> > > Again, I mentioned that the old QMP workflow is this:\n> > >\n> > > JSON Parser --> QMP Dispatcher --> Respond\n> > > /|\\ (2) (3) |\n> > > (1) | \\|/ (4)\n> > > +--------- main thread --------+\n> > >\n> > > What this series does is, firstly:\n> > >\n> > > JSON Parser QMP Dispatcher --> Respond\n> > > /|\\ | /|\\ (4) |\n> > > | | (2) | (3) | (5)\n> > > (1) | +-----> | \\|/\n> > > +--------- main thread <-------+\n> > >\n> > > And further:\n> > >\n> > > queue/kick\n> > > JSON Parser ======> QMP Dispatcher --> Respond\n> > > /|\\ | (3) /|\\ (4) |\n> > > (1) | | (2) | | (5)\n> > > | \\|/ | \\|/\n> > > IO thread main thread <-------+\n> > \n> > Is the queue per monitor or per client?\n\nThe queue is currently global. I think yes maybe at least we can do it\nper monitor, but I am not sure whether that is urgent or can be\npostponed. After all now QMPRequest (please refer to patch 11) is\ndefined as (mon, id, req) tuple, so at least \"id\" namespace is\nper-monitor.\n\n> > And is the dispatching going\n> > to be processed even if the client is disconnected, and are new\n> > clients going to receive the replies from previous clients\n> > commands?\n\n[1]\n\n(will discuss together below)\n\n> > I\n> > believe there should be a per-client context, so there won't be \"id\"\n> > request conflicts.\n\nI'd say I am not familiar with this \"client\" idea, since after all\nIMHO one monitor is currently designed to mostly work with a single\nclient. Say, unix sockets, telnet, all these backends are only single\nchanneled, and one monitor instance can only work with one client at a\ntime. Then do we really need to add this client layer upon it? IMHO\nthe user can just provide more monitors if they wants more clients\n(and at least these clients should know the existance of the others or\nthere might be problem, otherwise user2 will fail a migration, finally\nnoticed that user1 has already triggered one), and the user should\nmanage them well.\n\n> > \n> > >\n> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> > > parser.\n> > \n> > From a protocol point of view, I find that \"run-oob\" distinction per\n> > command a bit pointless. It helps with legacy client that wouldn't\n> > expect out-of-order replies if qemu were to run oob commands oob by\n> > default though.\n\nAfter all oob somehow breaks existing rules or sync execution. I\nthought the more important goal was at least to keep the legacy\nbehaviors when adding new things, no?\n\n> > Clients shouldn't care about how/where a command is\n> > being queued or not. If they send a command, they want it processed as\n> > quickly as possible. However, it can be interesting to know if the\n> > implementation of the command will be able to deliver oob, so that\n> > data in the introspection could be useful.\n> > \n> > I would rather propose a client/server capability in qmp_capabilities,\n> > call it \"oob\":\n> > \n> > This capability indicates oob commands support.\n> \n> The problem is indicating which commands support oob as opposed to\n> indicating whether oob is present at all. Future versions will\n> probably make more commands oob-able and a client will want to know\n> whether it can rely on a particular command being non-blocking.\n\nYes.\n\nAnd IMHO we don't urgently need that \"whether the server globally\nsupports oob\" thing. Client can just know that from query-qmp-schema\nalready - there will always be the \"allow-oob\" new field for command\ntyped entries. IMHO that's a solid hint.\n\nBut I don't object to return it as well in qmp_capabilities.\n\n> \n> > An oob command is a regular client message request with the \"id\"\n> > member mandatory, but the reply may be delivered\n> > out of order by the server if the client supports\n> > it too.\n> > \n> > If both the server and the client have the \"oob\" capability, the\n> > server can handle new client requests while previous requests are being\n> > processed.\n> > \n> > If the client doesn't have the \"oob\" capability, it may still call\n> > an oob command, and make multiple outstanding calls. In this case,\n> > the commands are processed in order, so the replies will also be in\n> > order. The \"id\" member isn't mandatory in this case.\n> > \n> > The client should match the replies with the \"id\" member associated\n> > with the requests.\n> > \n> > When a client is disconnected, the pending commands are not\n> > necessarily cancelled. But the future clients will not get replies from\n> > commands they didn't make (they might, however, receive side-effects\n> > events).\n> \n> What's the behaviour on the current monitor?\n\nYeah I want to ask the same question, along with questioning about\nabove [1].\n\nIMHO this series will not change the behaviors of these, so IMHO the\nbehaviors will be the same before/after this series. E.g., when client\ndropped right after the command is executed, I think we will still\nexecute the command, though we should encounter something odd in\nmonitor_json_emitter() somewhere when we want to respond. And it will\nhappen the same after this series.\n\n> \n> \n> > Note that without \"oob\" support, a client may still receive\n> > messages (or events) from the server between the time a\n> > request is handled by the server and the reply is received. It must\n> > thus be prepared to handle dispatching both events and reply after\n> > sending a request.\n> > \n> > \n> > (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)\n> > \n> > \n> > > The last patch enables this for \"migrate-incoming\" command.\n> > >\n> > > Please review. Thanks.\n> > >\n> > > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> > >\n> > > Peter Xu (15):\n> > > char-io: fix possible race on IOWatchPoll\n> > > qobject: allow NULL for qstring_get_str()\n> > > qobject: introduce qobject_to_str()\n> > > monitor: move skip_flush into monitor_data_init\n> > > qjson: add \"opaque\" field to JSONMessageParser\n> > > monitor: move the cur_mon hack deeper for QMP\n> > > monitor: unify global init\n> > > monitor: create IO thread\n> > > monitor: allow to use IO thread for parsing\n> > > monitor: introduce monitor_qmp_respond()\n> > > monitor: separate QMP parser and dispatcher\n> > \n> > There should be a limit in the number of requests the thread can\n> > queue. Before the patch, the limit was enforced by system socket\n> > buffering I think. Now, should oob commands still be processed even if\n> > the queue is full? If so, the thread can't be suspended.\n> \n> I think the previous discussion was expecting a pair of queues\n> per client and perhaps a pair of central queues; each pair being\n> for normal command and oob commands.\n> (I'm not expecting these queues to be deep; IMHO '1' is the\n> right size for this type of queue in both cases).\n\nYes. One thing to mention is that, if we see the graph above, I\ndidn't really introduce two queues (input/output), but only one input\nqueue. The response is still handled in dispatcher for now, since I\nthink that's quite enough at least for OOB, and I didn't see much\nbenefit now to split that 2nd queue.\n\nSo I am thinking whether we can just quickly respond an event when the\nqueue is full, as I proposed in the other reply.\n\nRegarding to queue size: I am afraid max_size=1 may not suffice?\nOtherwise a simple batch of:\n\n{\"execute\": \"query-status\"} {\"execute\": \"query-status\"}\n\nWill trigger the failure. But I definitely agree it should not be\nsomething very large. The total memory will be this:\n\n json limit * queue length limit * monitor count limit\n (X) (Y) (Z)\n\nNow we have (X) already (in form of a few tunables for JSON token\ncounts, etc.), we don't have (Z), and we definitely need (Y).\n\nHow about we add limits on Y=16 and Z=8?\n\nWe can do some math if we want some more exact number though.\n\n> \n> Dave\n> \n> > > monitor: enable IO thread for (qmp & !mux) typed\n> > > qapi: introduce new cmd option \"allow-oob\"\n> > > qmp: support out-of-band (oob) execution\n> > > qmp: let migrate-incoming allow out-of-band\n> > >\n> > > chardev/char-io.c | 15 ++-\n> > > docs/devel/qapi-code-gen.txt | 51 ++++++-\n> > > include/monitor/monitor.h | 2 +-\n> > > include/qapi/qmp/dispatch.h | 2 +\n> > > include/qapi/qmp/json-streamer.h | 8 +-\n> > > include/qapi/qmp/qstring.h | 1 +\n> > > monitor.c | 283 +++++++++++++++++++++++++++++++--------\n> > > qapi/introspect.json | 6 +-\n> > > qapi/migration.json | 3 +-\n> > > qapi/qmp-dispatch.c | 34 +++++\n> > > qga/main.c | 5 +-\n> > > qobject/json-streamer.c | 7 +-\n> > > qobject/qjson.c | 5 +-\n> > > qobject/qstring.c | 13 +-\n> > > scripts/qapi-commands.py | 19 ++-\n> > > scripts/qapi-introspect.py | 10 +-\n> > > scripts/qapi.py | 15 ++-\n> > > scripts/qapi2texi.py | 2 +-\n> > > tests/libqtest.c | 5 +-\n> > > tests/qapi-schema/test-qapi.py | 2 +-\n> > > trace-events | 2 +\n> > > vl.c | 3 +-\n> > > 22 files changed, 398 insertions(+), 95 deletions(-)\n> > >\n> > > --\n> > > 2.7.4\n> > >\n> > \n> > \n> > \n> > -- \n> > Marc-André Lureau\n> --\n> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtjYZ516dz9sBZ\n\tfor ;\n\tFri, 15 Sep 2017 14:47:14 +1000 (AEST)","from localhost ([::1]:51230 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsiX2-00028g-NT\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 00:47:12 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:38755)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsiWT-00022c-Ht\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 00:46:39 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsiWQ-0002F3-Ap\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 00:46:37 -0400","from mx1.redhat.com ([209.132.183.28]:51178)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsiWQ-0002E9-0f\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 00:46:34 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id F3273C0587C4;\n\tFri, 15 Sep 2017 04:46:31 +0000 (UTC)","from pxdev.xzpeter.org (ovpn-12-86.pek2.redhat.com [10.72.12.86])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 583B66F139;\n\tFri, 15 Sep 2017 04:46:25 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com F3273C0587C4","Date":"Fri, 15 Sep 2017 12:46:22 +0800","From":"Peter Xu ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170915044622.GO3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170914185314.GA3280@work-vm>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.32]);\n\tFri, 15 Sep 2017 04:46:32 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth , Juan Quintela\n\t, Markus Armbruster , QEMU\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Paolo Bonzini ,\n\tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769102,"web_url":"http://patchwork.ozlabs.org/comment/1769102/","msgid":"<20170915104926.GA14994@stefanha-x1.localdomain>","list_archive_url":null,"date":"2017-09-15T10:49:26","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":17227,"url":"http://patchwork.ozlabs.org/api/people/17227/","name":"Stefan Hajnoczi","email":"stefanha@redhat.com"},"content":"On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > There should be a limit in the number of requests the thread can\n> > > queue. Before the patch, the limit was enforced by system socket\n> > > buffering I think. Now, should oob commands still be processed even if\n> > > the queue is full? If so, the thread can't be suspended.\n> > \n> > I agree.\n> > \n> > Memory usage must be bounded. The number of requests is less important\n> > than the amount of memory consumed by them.\n> > \n> > Existing QMP clients that send multiple QMP commands without waiting for\n> > replies need to rethink their strategy because OOB commands cannot be\n> > processed if queued non-OOB commands consume too much memory.\n> \n> Thanks for pointing out this. Yes the memory usage problem is valid,\n> as Markus pointed out as well in previous discussions (in \"Flow\n> Control\" section of that long reply). Hopefully this series basically\n> can work from design prospective, then I'll add this flow control in\n> next version.\n> \n> Regarding to what we should do if the limit is reached: Markus\n> provided a few options, but the one I prefer most is that we don't\n> respond, but send an event showing that a command is dropped.\n> However, I would like it not queued, but a direct reply (after all,\n> it's an event, and we should not need to care much on ordering of it).\n> Then we can get rid of the babysitting of those \"to be failed\"\n> requests asap, meanwhile we don't lose anything IMHO.\n> \n> I think I also missed at least a unit test for this new interface.\n> Again, I'll add it after the whole idea is proved solid. Thanks,\n\nAnother solution: the server reports available receive buffer space to\nthe client. The server only guarantees immediate OOB processing when\nthe client stays within the receive buffer size.\n\nClients wishing to take advantage of OOB must query the receive buffer\nsize and make sure to leave enough room.\n\nThe advantage of this approach is that the semantics are backwards\ncompatible (existing clients may continue to queue as many commands as\nthey wish) and it requires no new behavior in the client (no new QMP\nevent code path).\n\nStefan","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=stefanha@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtscM5rPtz9t2V\n\tfor ;\n\tFri, 15 Sep 2017 20:50:10 +1000 (AEST)","from localhost ([::1]:52468 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsoCB-000705-3o\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 06:50:03 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:47013)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsoBl-0006zv-QT\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 06:49:38 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsoBi-00005k-OY\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 06:49:37 -0400","from mx1.redhat.com ([209.132.183.28]:44506)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsoBi-000058-FO\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 06:49:34 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id B003081DFC;\n\tFri, 15 Sep 2017 10:49:32 +0000 (UTC)","from localhost (ovpn-117-224.ams2.redhat.com [10.36.117.224])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id F10C560841;\n\tFri, 15 Sep 2017 10:49:27 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com B003081DFC","Date":"Fri, 15 Sep 2017 11:49:26 +0100","From":"Stefan Hajnoczi ","To":"Peter Xu ","Message-ID":"<20170915104926.GA14994@stefanha-x1.localdomain>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915035057.GM3617@pxdev.xzpeter.org>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tFri, 15 Sep 2017 10:49:32 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth , Juan Quintela\n\t, Markus Armbruster , QEMU\n\t, =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , \"Dr . David Alan Gilbert\"\n\t, Paolo Bonzini ,\n\tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769109,"web_url":"http://patchwork.ozlabs.org/comment/1769109/","msgid":"","list_archive_url":null,"date":"2017-09-15T11:14:47","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6442,"url":"http://patchwork.ozlabs.org/api/people/6442/","name":"Marc-André Lureau","email":"marcandre.lureau@gmail.com"},"content":"Hi\n\nOn Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n> On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n>> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> > Hi\n>> >\n>> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n>> > > This series was born from this one:\n>> > >\n>> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>> > >\n>> > > The design comes from Markus, and also the whole-bunch-of discussions\n>> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n>> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n>> > > ideas and suggestions. Finally we got such a solution that seems to\n>> > > satisfy everyone.\n>> > >\n>> > > I re-started the versioning since this series is totally different\n>> > > from previous one. Now it's version 1.\n>> > >\n>> > > In case new reviewers come along the way without reading previous\n>> > > discussions, I will try to do a summary on what this is all about.\n>> > >\n>> > > What is OOB execution?\n>> > > ======================\n>> > >\n>> > > It's the shortcut of Out-Of-Band execution, its name is given by\n>> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n>> > > QMP is going throw these steps:\n>> > >\n>> > > JSON Parser --> QMP Dispatcher --> Respond\n>> > > /|\\ (2) (3) |\n>> > > (1) | \\|/ (4)\n>> > > +--------- main thread --------+\n>> > >\n>> > > The requests are executed by the so-called QMP-dispatcher after the\n>> > > JSON is parsed. If OOB is on, we run the command directly in the\n>> > > parser and quickly returns.\n>> >\n>> > All commands should have the \"id\" field mandatory in this case, else\n>> > the client will not distinguish the replies coming from the last/oob\n>> > and the previous commands.\n>> >\n>> > This should probably be enforced upfront by client capability checks,\n>> > more below.\n>\n> Hmm yes since the oob commands are actually running in async way,\n> request ID should be needed here. However I'm not sure whether\n> enabling the whole \"request ID\" thing is too big for this \"try to be\n> small\" oob change... And IMHO it suites better to be part of the whole\n> async work (no matter which implementation we'll use).\n>\n> How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n> For oob commands, they will always have ID then no ordering issue, and\n> we can do it async; for the rest of non-oob commands, we still allow\n> them to go without ID, and since they are not oob, they'll always be\n> done in order as well. Would this work?\n\nThis mixed-mode is imho more complicated to deal with than having the\nprotocol enforced one way or the other, but that should work.\n\n>\n>> >\n>> > > Yeah I know in current code the parser calls dispatcher directly\n>> > > (please see handle_qmp_command()). However it's not true again after\n>> > > this series (parser will has its own IO thread, and dispatcher will\n>> > > still be run in main thread). So this OOB does brings something\n>> > > different.\n>> > >\n>> > > There are more details on why OOB and the difference/relationship\n>> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n>> > > slightly out of topic (and believe me, it's not easy for me to\n>> > > summarize that). For more information, please refers to [1].\n>> > >\n>> > > Summary ends here.\n>> > >\n>> > > Some Implementation Details\n>> > > ===========================\n>> > >\n>> > > Again, I mentioned that the old QMP workflow is this:\n>> > >\n>> > > JSON Parser --> QMP Dispatcher --> Respond\n>> > > /|\\ (2) (3) |\n>> > > (1) | \\|/ (4)\n>> > > +--------- main thread --------+\n>> > >\n>> > > What this series does is, firstly:\n>> > >\n>> > > JSON Parser QMP Dispatcher --> Respond\n>> > > /|\\ | /|\\ (4) |\n>> > > | | (2) | (3) | (5)\n>> > > (1) | +-----> | \\|/\n>> > > +--------- main thread <-------+\n>> > >\n>> > > And further:\n>> > >\n>> > > queue/kick\n>> > > JSON Parser ======> QMP Dispatcher --> Respond\n>> > > /|\\ | (3) /|\\ (4) |\n>> > > (1) | | (2) | | (5)\n>> > > | \\|/ | \\|/\n>> > > IO thread main thread <-------+\n>> >\n>> > Is the queue per monitor or per client?\n>\n> The queue is currently global. I think yes maybe at least we can do it\n> per monitor, but I am not sure whether that is urgent or can be\n> postponed. After all now QMPRequest (please refer to patch 11) is\n> defined as (mon, id, req) tuple, so at least \"id\" namespace is\n> per-monitor.\n>\n>> > And is the dispatching going\n>> > to be processed even if the client is disconnected, and are new\n>> > clients going to receive the replies from previous clients\n>> > commands?\n>\n> [1]\n>\n> (will discuss together below)\n>\n>> > I\n>> > believe there should be a per-client context, so there won't be \"id\"\n>> > request conflicts.\n>\n> I'd say I am not familiar with this \"client\" idea, since after all\n> IMHO one monitor is currently designed to mostly work with a single\n> client. Say, unix sockets, telnet, all these backends are only single\n> channeled, and one monitor instance can only work with one client at a\n> time. Then do we really need to add this client layer upon it? IMHO\n> the user can just provide more monitors if they wants more clients\n> (and at least these clients should know the existance of the others or\n> there might be problem, otherwise user2 will fail a migration, finally\n> noticed that user1 has already triggered one), and the user should\n> manage them well.\n\nqemu should support a management layer / libvirt restart/reconnect.\nAfaik, it mostly work today. There might be a cases where libvirt can\nbe confused if it receives a reply from a previous connection command,\nbut due to the sync processing of the chardev, I am not sure you can\nget in this situation. By adding \"oob\" commands and queuing, the\nclient will have to remember which was the last \"id\" used, or it will\ncreate more conflict after a reconnect.\n\nImho we should introduce the client/connection concept to avoid this\nconfusion (unexpected reply & per client id space).\n\n>\n>> >\n>> > >\n>> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n>> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n>> > > parser.\n>> >\n>> > From a protocol point of view, I find that \"run-oob\" distinction per\n>> > command a bit pointless. It helps with legacy client that wouldn't\n>> > expect out-of-order replies if qemu were to run oob commands oob by\n>> > default though.\n>\n> After all oob somehow breaks existing rules or sync execution. I\n> thought the more important goal was at least to keep the legacy\n> behaviors when adding new things, no?\n\nOf course we have to keep compatibily. What do you mean by \"oob\nsomehow breaks existing rules or sync execution\"? oob means queuing\nand unordered reply support, so clearly this is breaking the current\n\"mostly ordered\" behaviour (mostly because events may still come any\ntime..., and the reconnect issue discussed above).\n\n>> > Clients shouldn't care about how/where a command is\n>> > being queued or not. If they send a command, they want it processed as\n>> > quickly as possible. However, it can be interesting to know if the\n>> > implementation of the command will be able to deliver oob, so that\n>> > data in the introspection could be useful.\n>> >\n>> > I would rather propose a client/server capability in qmp_capabilities,\n>> > call it \"oob\":\n>> >\n>> > This capability indicates oob commands support.\n>>\n>> The problem is indicating which commands support oob as opposed to\n>> indicating whether oob is present at all. Future versions will\n>> probably make more commands oob-able and a client will want to know\n>> whether it can rely on a particular command being non-blocking.\n>\n> Yes.\n>\n> And IMHO we don't urgently need that \"whether the server globally\n> supports oob\" thing. Client can just know that from query-qmp-schema\n> already - there will always be the \"allow-oob\" new field for command\n> typed entries. IMHO that's a solid hint.\n>\n> But I don't object to return it as well in qmp_capabilities.\n\nDoes it feel right that the client can specify how the command are\nprocessed / queued ? Isn't it preferable to leave that to the server\nto decide? Why would a client specify that? And should the server be\nexpected to behave differently? What the client needs to be able is to\nmatch the unordered replies, and that can be stated during cap\nnegotiation / qmp_capabilties. The server is expected to do a best\neffort to handle commands and their priorities. If the client needs\nseveral command queue, it is simpler to open several connection rather\nthan trying to fit that weird priority logic in the protocol imho.\n\n>\n>>\n>> > An oob command is a regular client message request with the \"id\"\n>> > member mandatory, but the reply may be delivered\n>> > out of order by the server if the client supports\n>> > it too.\n>> >\n>> > If both the server and the client have the \"oob\" capability, the\n>> > server can handle new client requests while previous requests are being\n>> > processed.\n>> >\n>> > If the client doesn't have the \"oob\" capability, it may still call\n>> > an oob command, and make multiple outstanding calls. In this case,\n>> > the commands are processed in order, so the replies will also be in\n>> > order. The \"id\" member isn't mandatory in this case.\n>> >\n>> > The client should match the replies with the \"id\" member associated\n>> > with the requests.\n>> >\n>> > When a client is disconnected, the pending commands are not\n>> > necessarily cancelled. But the future clients will not get replies from\n>> > commands they didn't make (they might, however, receive side-effects\n>> > events).\n>>\n>> What's the behaviour on the current monitor?\n>\n> Yeah I want to ask the same question, along with questioning about\n> above [1].\n>\n> IMHO this series will not change the behaviors of these, so IMHO the\n> behaviors will be the same before/after this series. E.g., when client\n> dropped right after the command is executed, I think we will still\n> execute the command, though we should encounter something odd in\n> monitor_json_emitter() somewhere when we want to respond. And it will\n> happen the same after this series.\n\nI think it can get worse after your series, because you queue the\ncommands, so clearly a new client can get replies from an old client\ncommands. As said above, I am not convinced you can get in that\nsituation with current code.\n\n>\n>>\n>>\n>> > Note that without \"oob\" support, a client may still receive\n>> > messages (or events) from the server between the time a\n>> > request is handled by the server and the reply is received. It must\n>> > thus be prepared to handle dispatching both events and reply after\n>> > sending a request.\n>> >\n>> >\n>> > (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)\n>> >\n>> >\n>> > > The last patch enables this for \"migrate-incoming\" command.\n>> > >\n>> > > Please review. Thanks.\n>> > >\n>> > > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>> > >\n>> > > Peter Xu (15):\n>> > > char-io: fix possible race on IOWatchPoll\n>> > > qobject: allow NULL for qstring_get_str()\n>> > > qobject: introduce qobject_to_str()\n>> > > monitor: move skip_flush into monitor_data_init\n>> > > qjson: add \"opaque\" field to JSONMessageParser\n>> > > monitor: move the cur_mon hack deeper for QMP\n>> > > monitor: unify global init\n>> > > monitor: create IO thread\n>> > > monitor: allow to use IO thread for parsing\n>> > > monitor: introduce monitor_qmp_respond()\n>> > > monitor: separate QMP parser and dispatcher\n>> >\n>> > There should be a limit in the number of requests the thread can\n>> > queue. Before the patch, the limit was enforced by system socket\n>> > buffering I think. Now, should oob commands still be processed even if\n>> > the queue is full? If so, the thread can't be suspended.\n>>\n>> I think the previous discussion was expecting a pair of queues\n>> per client and perhaps a pair of central queues; each pair being\n>> for normal command and oob commands.\n>> (I'm not expecting these queues to be deep; IMHO '1' is the\n>> right size for this type of queue in both cases).\n>\n> Yes. One thing to mention is that, if we see the graph above, I\n> didn't really introduce two queues (input/output), but only one input\n> queue. The response is still handled in dispatcher for now, since I\n> think that's quite enough at least for OOB, and I didn't see much\n> benefit now to split that 2nd queue.\n>\n> So I am thinking whether we can just quickly respond an event when the\n> queue is full, as I proposed in the other reply.\n>\n> Regarding to queue size: I am afraid max_size=1 may not suffice?\n> Otherwise a simple batch of:\n>\n> {\"execute\": \"query-status\"} {\"execute\": \"query-status\"}\n>\n> Will trigger the failure. But I definitely agree it should not be\n> something very large. The total memory will be this:\n>\n> json limit * queue length limit * monitor count limit\n> (X) (Y) (Z)\n>\n> Now we have (X) already (in form of a few tunables for JSON token\n> counts, etc.), we don't have (Z), and we definitely need (Y).\n>\n> How about we add limits on Y=16 and Z=8?\n>\n> We can do some math if we want some more exact number though.\n>\n>>\n>> Dave\n>>\n>> > > monitor: enable IO thread for (qmp & !mux) typed\n>> > > qapi: introduce new cmd option \"allow-oob\"\n>> > > qmp: support out-of-band (oob) execution\n>> > > qmp: let migrate-incoming allow out-of-band\n>> > >\n>> > > chardev/char-io.c | 15 ++-\n>> > > docs/devel/qapi-code-gen.txt | 51 ++++++-\n>> > > include/monitor/monitor.h | 2 +-\n>> > > include/qapi/qmp/dispatch.h | 2 +\n>> > > include/qapi/qmp/json-streamer.h | 8 +-\n>> > > include/qapi/qmp/qstring.h | 1 +\n>> > > monitor.c | 283 +++++++++++++++++++++++++++++++--------\n>> > > qapi/introspect.json | 6 +-\n>> > > qapi/migration.json | 3 +-\n>> > > qapi/qmp-dispatch.c | 34 +++++\n>> > > qga/main.c | 5 +-\n>> > > qobject/json-streamer.c | 7 +-\n>> > > qobject/qjson.c | 5 +-\n>> > > qobject/qstring.c | 13 +-\n>> > > scripts/qapi-commands.py | 19 ++-\n>> > > scripts/qapi-introspect.py | 10 +-\n>> > > scripts/qapi.py | 15 ++-\n>> > > scripts/qapi2texi.py | 2 +-\n>> > > tests/libqtest.c | 5 +-\n>> > > tests/qapi-schema/test-qapi.py | 2 +-\n>> > > trace-events | 2 +\n>> > > vl.c | 3 +-\n>> > > 22 files changed, 398 insertions(+), 95 deletions(-)\n>> > >\n>> > > --\n>> > > 2.7.4\n>> > >\n>> >\n>> >\n>> >\n>> > --\n>> > Marc-André Lureau\n>> --\n>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK\n>\n> --\n> Peter Xu","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"tJ7VgiSs\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtt9m0jqJz9sBZ\n\tfor ;\n\tFri, 15 Sep 2017 21:15:40 +1000 (AEST)","from localhost ([::1]:52549 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsoaw-0000SP-5f\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 07:15:38 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:55572)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsoaD-0000P2-Do\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:14:55 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsoaA-0003Qj-QR\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:14:53 -0400","from mail-io0-x233.google.com ([2607:f8b0:4001:c06::233]:55341)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1dsoaA-0003PH-IN\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:14:50 -0400","by mail-io0-x233.google.com with SMTP id z187so7474406ioz.12\n\tfor ; Fri, 15 Sep 2017 04:14:49 -0700 (PDT)","by 10.58.69.69 with HTTP; Fri, 15 Sep 2017 04:14:47 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=a1bDlKeG03CnpIlIG5A4Ozb4vzVoy467U2u7nqrZW3s=;\n\tb=tJ7VgiSsDpwgCIDwbz7/ZmvXpppgm/60bNrSXrWfGBRTWdKHFMoxwooh50+SK8E9sU\n\tGfEcKPxColtlABB1COPDHRqe0feKRCsqzVTWeiwO98LaavRwtmlWEAsQ0xfUTNetQH4C\n\tTme648inZgTrg5O1na5MMlt1tHfWXw+8rLVXp7d5sxLGa3S1DgyLvkJM1PXeV/04TDpB\n\tQC2j8OxZGRaCV7vD82fywJ6v5vEAcLJ9wYKwgnIcsWl1CGD224xQTwAXOQR6KlghOPKi\n\tv6UCBUFYx8fGXUNhptP16Win39OarftECDOH6cPYrZvkIQF7afKtiVoW7sccWH8si2bp\n\tNajA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=a1bDlKeG03CnpIlIG5A4Ozb4vzVoy467U2u7nqrZW3s=;\n\tb=fMWBiyfvvZATwaoQs8i7+GozvxsuOALuB7Kqe8Umnk93zu12zC3Y5gTzFXkSW2ymGS\n\t5GemhXHWw1b6dACKS9ZwYtmk2eA5hNO4FbXFyRLiepeg59oT9YY59QNeVQp1a0yI31Pl\n\tC9tHYTfHv9YTv3rDeHpFEF0RnBuxml/aVA1FMGmzoeM2d+wygA8PPsm9autsBGRydCFN\n\tql8xcOLs1BQBKFSFQsqaSvyWsVrqy1SIYkWtfcUVqVEzFM/xZOsm2rzBwjLcEcWmCIru\n\tby9nkFnyBId3pJ0sDmnODbw7J3NVBEYqjKPSKobNIDrzyQi1aYAbtL1jBH4B+FamZMvQ\n\tlZwQ==","X-Gm-Message-State":"AHPjjUhscoVg9j0a7Ev6c2Y9669pCnSCGBmRjl6ea3s9c180wuLq05hV\n\tbCcjfcB7gRmSWtAIM+omeA/PHxT+z/ZSWfoOKrk=","X-Google-Smtp-Source":"AOwi7QDxFDDL4uifrsiXM2scmty2aC9yWQvOaftjlwqiZayR71QpA1I8DLHWAz7febhreTFegGESP5Fuu4AVxDPRMQ0=","X-Received":"by 10.202.77.201 with SMTP id a192mr5644011oib.311.1505474088354;\n\tFri, 15 Sep 2017 04:14:48 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170915044622.GO3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>","From":"=?utf-8?q?Marc-Andr=C3=A9_Lureau?= ","Date":"Fri, 15 Sep 2017 13:14:47 +0200","Message-ID":"","To":"Peter Xu ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4001:c06::233","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela , QEMU ,\n\tMarkus Armbruster ,\n\t\"Dr. David Alan Gilbert\" ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769114,"web_url":"http://patchwork.ozlabs.org/comment/1769114/","msgid":"<20170915113428.GF13610@redhat.com>","list_archive_url":null,"date":"2017-09-15T11:34:28","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > There should be a limit in the number of requests the thread can\n> > > > queue. Before the patch, the limit was enforced by system socket\n> > > > buffering I think. Now, should oob commands still be processed even if\n> > > > the queue is full? If so, the thread can't be suspended.\n> > > \n> > > I agree.\n> > > \n> > > Memory usage must be bounded. The number of requests is less important\n> > > than the amount of memory consumed by them.\n> > > \n> > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > replies need to rethink their strategy because OOB commands cannot be\n> > > processed if queued non-OOB commands consume too much memory.\n> > \n> > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > as Markus pointed out as well in previous discussions (in \"Flow\n> > Control\" section of that long reply). Hopefully this series basically\n> > can work from design prospective, then I'll add this flow control in\n> > next version.\n> > \n> > Regarding to what we should do if the limit is reached: Markus\n> > provided a few options, but the one I prefer most is that we don't\n> > respond, but send an event showing that a command is dropped.\n> > However, I would like it not queued, but a direct reply (after all,\n> > it's an event, and we should not need to care much on ordering of it).\n> > Then we can get rid of the babysitting of those \"to be failed\"\n> > requests asap, meanwhile we don't lose anything IMHO.\n> > \n> > I think I also missed at least a unit test for this new interface.\n> > Again, I'll add it after the whole idea is proved solid. Thanks,\n> \n> Another solution: the server reports available receive buffer space to\n> the client. The server only guarantees immediate OOB processing when\n> the client stays within the receive buffer size.\n> \n> Clients wishing to take advantage of OOB must query the receive buffer\n> size and make sure to leave enough room.\n\nI don't think having to query it ahead of time is particularly nice,\nand of course it is inherantly racy.\n\nI would just have QEMU emit an event when it pausing processing of the\nincoming commands due to a full queue. If the event includes the ID\nof the last queued command, the client will know which (if any) of\nits outstanding commands are delayed. Another even can be sent when\nit restarts reading.\n\nRegards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=berrange@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xttcS5Mjnz9t2Z\n\tfor ;\n\tFri, 15 Sep 2017 21:35:19 +1000 (AEST)","from localhost ([::1]:52598 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsotw-00016j-Vt\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 07:35:17 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:60490)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsotU-00014p-5G\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:34:49 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsotQ-0004Om-S3\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:34:48 -0400","from mx1.redhat.com ([209.132.183.28]:55500)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsotQ-0004OL-I1\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 07:34:44 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 3103187649;\n\tFri, 15 Sep 2017 11:34:43 +0000 (UTC)","from redhat.com (ovpn-116-81.ams2.redhat.com [10.36.116.81])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id A367671C5D;\n\tFri, 15 Sep 2017 11:34:31 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 3103187649","Date":"Fri, 15 Sep 2017 12:34:28 +0100","From":"\"Daniel P. Berrange\" ","To":"Stefan Hajnoczi ","Message-ID":"<20170915113428.GF13610@redhat.com>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170915104926.GA14994@stefanha-x1.localdomain>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.26]);\n\tFri, 15 Sep 2017 11:34:43 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"\"Daniel P. Berrange\" ","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau ,\n\tPaolo Bonzini , \"Dr . David Alan Gilbert\"\n\t","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769126,"web_url":"http://patchwork.ozlabs.org/comment/1769126/","msgid":"<20170915120643.GN2170@work-vm>","list_archive_url":null,"date":"2017-09-15T12:06:44","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Daniel P. Berrange (berrange@redhat.com) wrote:\n> On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > There should be a limit in the number of requests the thread can\n> > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > the queue is full? If so, the thread can't be suspended.\n> > > > \n> > > > I agree.\n> > > > \n> > > > Memory usage must be bounded. The number of requests is less important\n> > > > than the amount of memory consumed by them.\n> > > > \n> > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > processed if queued non-OOB commands consume too much memory.\n> > > \n> > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > Control\" section of that long reply). Hopefully this series basically\n> > > can work from design prospective, then I'll add this flow control in\n> > > next version.\n> > > \n> > > Regarding to what we should do if the limit is reached: Markus\n> > > provided a few options, but the one I prefer most is that we don't\n> > > respond, but send an event showing that a command is dropped.\n> > > However, I would like it not queued, but a direct reply (after all,\n> > > it's an event, and we should not need to care much on ordering of it).\n> > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > requests asap, meanwhile we don't lose anything IMHO.\n> > > \n> > > I think I also missed at least a unit test for this new interface.\n> > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > \n> > Another solution: the server reports available receive buffer space to\n> > the client. The server only guarantees immediate OOB processing when\n> > the client stays within the receive buffer size.\n> > \n> > Clients wishing to take advantage of OOB must query the receive buffer\n> > size and make sure to leave enough room.\n> \n> I don't think having to query it ahead of time is particularly nice,\n> and of course it is inherantly racy.\n> \n> I would just have QEMU emit an event when it pausing processing of the\n> incoming commands due to a full queue. If the event includes the ID\n> of the last queued command, the client will know which (if any) of\n> its outstanding commands are delayed. Another even can be sent when\n> it restarts reading.\n\nHmm and now we're implementing flow control!\n\na) What exactly is the current semantics/buffer sizes?\nb) When do clients send multiple QMP commands on one channel without\nwaiting for the response to the previous command?\nc) Would one queue entry for each class of commands/channel work\n (Where a class of commands is currently 'normal' and 'oob')\n\nDave\n\n> Regards,\n> Daniel\n> -- \n> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|\n> |: https://libvirt.org -o- https://fstop138.berrange.com :|\n> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtvKW4grTz9s7C\n\tfor ;\n\tFri, 15 Sep 2017 22:07:27 +1000 (AEST)","from localhost ([::1]:53061 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dspP3-0000lt-Ok\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 08:07:25 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:48701)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dspOg-0000eX-8d\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:07:08 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dspOc-0005b4-6C\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:07:02 -0400","from mx1.redhat.com ([209.132.183.28]:37788)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dspOb-0005af-Ty\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:06:58 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id E756C2C973A;\n\tFri, 15 Sep 2017 12:06:56 +0000 (UTC)","from work-vm (ovpn-117-180.ams2.redhat.com [10.36.117.180])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 9F565189A9;\n\tFri, 15 Sep 2017 12:06:46 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com E756C2C973A","Date":"Fri, 15 Sep 2017 13:06:44 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"\"Daniel P. Berrange\" ","Message-ID":"<20170915120643.GN2170@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915113428.GF13610@redhat.com>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.29]);\n\tFri, 15 Sep 2017 12:06:57 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769131,"web_url":"http://patchwork.ozlabs.org/comment/1769131/","msgid":"<20170915121433.GI13610@redhat.com>","list_archive_url":null,"date":"2017-09-15T12:14:33","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > There should be a limit in the number of requests the thread can\n> > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > \n> > > > > I agree.\n> > > > > \n> > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > than the amount of memory consumed by them.\n> > > > > \n> > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > processed if queued non-OOB commands consume too much memory.\n> > > > \n> > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > Control\" section of that long reply). Hopefully this series basically\n> > > > can work from design prospective, then I'll add this flow control in\n> > > > next version.\n> > > > \n> > > > Regarding to what we should do if the limit is reached: Markus\n> > > > provided a few options, but the one I prefer most is that we don't\n> > > > respond, but send an event showing that a command is dropped.\n> > > > However, I would like it not queued, but a direct reply (after all,\n> > > > it's an event, and we should not need to care much on ordering of it).\n> > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > \n> > > > I think I also missed at least a unit test for this new interface.\n> > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > \n> > > Another solution: the server reports available receive buffer space to\n> > > the client. The server only guarantees immediate OOB processing when\n> > > the client stays within the receive buffer size.\n> > > \n> > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > size and make sure to leave enough room.\n> > \n> > I don't think having to query it ahead of time is particularly nice,\n> > and of course it is inherantly racy.\n> > \n> > I would just have QEMU emit an event when it pausing processing of the\n> > incoming commands due to a full queue. If the event includes the ID\n> > of the last queued command, the client will know which (if any) of\n> > its outstanding commands are delayed. Another even can be sent when\n> > it restarts reading.\n> \n> Hmm and now we're implementing flow control!\n> \n> a) What exactly is the current semantics/buffer sizes?\n> b) When do clients send multiple QMP commands on one channel without\n> waiting for the response to the previous command?\n> c) Would one queue entry for each class of commands/channel work\n> (Where a class of commands is currently 'normal' and 'oob')\n\nI do wonder if we need to worry about request limiting at all from the\nclient side. For non-OOB commands clients will wait for a reply before\nsending a 2nd non-OOB command, so you'll never get a deep queue for.\n\nOOB commands are supposed to be things which can be handled quickly\nwithout blocking, so even if a client sent several commands at once\nwithout waiting for replies, they're going to be processed quickly,\nso whether we temporarily block reading off the wire is a minor\ndetail.\n\nIOW, I think we could just have a fixed 10 command queue and apps just\npretend that there's an infinite queue and nothing bad would happen from\nthe app's POV.\n\nRegards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=berrange@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtvVT0rmLz9s7C\n\tfor ;\n\tFri, 15 Sep 2017 22:15:13 +1000 (AEST)","from localhost ([::1]:53102 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dspWZ-0004ib-95\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 08:15:11 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:52417)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dspWG-0004iT-IC\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:14:53 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dspWC-0003l8-HD\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:14:52 -0400","from mx1.redhat.com ([209.132.183.28]:52786)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dspWC-0003kY-8K\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:14:48 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 643F0C047B92;\n\tFri, 15 Sep 2017 12:14:47 +0000 (UTC)","from redhat.com (ovpn-116-81.ams2.redhat.com [10.36.116.81])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 0FEC4709A5;\n\tFri, 15 Sep 2017 12:14:36 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 643F0C047B92","Date":"Fri, 15 Sep 2017 13:14:33 +0100","From":"\"Daniel P. Berrange\" ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170915121433.GI13610@redhat.com>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170915120643.GN2170@work-vm>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tFri, 15 Sep 2017 12:14:47 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"\"Daniel P. Berrange\" ","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769136,"web_url":"http://patchwork.ozlabs.org/comment/1769136/","msgid":"<20170915121956.GO2170@work-vm>","list_archive_url":null,"date":"2017-09-15T12:19:56","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Daniel P. Berrange (berrange@redhat.com) wrote:\n> On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > \n> > > > > > I agree.\n> > > > > > \n> > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > than the amount of memory consumed by them.\n> > > > > > \n> > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > \n> > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > can work from design prospective, then I'll add this flow control in\n> > > > > next version.\n> > > > > \n> > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > respond, but send an event showing that a command is dropped.\n> > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > \n> > > > > I think I also missed at least a unit test for this new interface.\n> > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > \n> > > > Another solution: the server reports available receive buffer space to\n> > > > the client. The server only guarantees immediate OOB processing when\n> > > > the client stays within the receive buffer size.\n> > > > \n> > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > size and make sure to leave enough room.\n> > > \n> > > I don't think having to query it ahead of time is particularly nice,\n> > > and of course it is inherantly racy.\n> > > \n> > > I would just have QEMU emit an event when it pausing processing of the\n> > > incoming commands due to a full queue. If the event includes the ID\n> > > of the last queued command, the client will know which (if any) of\n> > > its outstanding commands are delayed. Another even can be sent when\n> > > it restarts reading.\n> > \n> > Hmm and now we're implementing flow control!\n> > \n> > a) What exactly is the current semantics/buffer sizes?\n> > b) When do clients send multiple QMP commands on one channel without\n> > waiting for the response to the previous command?\n> > c) Would one queue entry for each class of commands/channel work\n> > (Where a class of commands is currently 'normal' and 'oob')\n> \n> I do wonder if we need to worry about request limiting at all from the\n> client side. For non-OOB commands clients will wait for a reply before\n> sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> \n> OOB commands are supposed to be things which can be handled quickly\n> without blocking, so even if a client sent several commands at once\n> without waiting for replies, they're going to be processed quickly,\n> so whether we temporarily block reading off the wire is a minor\n> detail.\n\nLets just define it so that it can't - you send an OOB command and wait\nfor it's response before sending another on that channel.\n\n> IOW, I think we could just have a fixed 10 command queue and apps just\n> pretend that there's an infinite queue and nothing bad would happen from\n> the app's POV.\n\nCan you justify 10 as opposed to 1?\n\nDave\n\n> Regards,\n> Daniel\n> -- \n> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|\n> |: https://libvirt.org -o- https://fstop138.berrange.com :|\n> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtvcv09Hzz9s3w\n\tfor ;\n\tFri, 15 Sep 2017 22:20:47 +1000 (AEST)","from localhost ([::1]:53126 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dspbx-00084f-5v\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 08:20:45 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:54170)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dspba-00084W-PS\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:20:24 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dspbW-0003R2-OK\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:20:22 -0400","from mx1.redhat.com ([209.132.183.28]:40498)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dspbW-0003QW-FG\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:20:18 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 7C2A45D686;\n\tFri, 15 Sep 2017 12:20:17 +0000 (UTC)","from work-vm (ovpn-117-180.ams2.redhat.com [10.36.117.180])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 9581A70468;\n\tFri, 15 Sep 2017 12:19:59 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 7C2A45D686","Date":"Fri, 15 Sep 2017 13:19:56 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"\"Daniel P. Berrange\" ","Message-ID":"<20170915121956.GO2170@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915121433.GI13610@redhat.com>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.39]);\n\tFri, 15 Sep 2017 12:20:17 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769144,"web_url":"http://patchwork.ozlabs.org/comment/1769144/","msgid":"<20170915122913.GJ13610@redhat.com>","list_archive_url":null,"date":"2017-09-15T12:29:13","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > \n> > > > > > > I agree.\n> > > > > > > \n> > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > than the amount of memory consumed by them.\n> > > > > > > \n> > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > \n> > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > next version.\n> > > > > > \n> > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > \n> > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > \n> > > > > Another solution: the server reports available receive buffer space to\n> > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > the client stays within the receive buffer size.\n> > > > > \n> > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > size and make sure to leave enough room.\n> > > > \n> > > > I don't think having to query it ahead of time is particularly nice,\n> > > > and of course it is inherantly racy.\n> > > > \n> > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > incoming commands due to a full queue. If the event includes the ID\n> > > > of the last queued command, the client will know which (if any) of\n> > > > its outstanding commands are delayed. Another even can be sent when\n> > > > it restarts reading.\n> > > \n> > > Hmm and now we're implementing flow control!\n> > > \n> > > a) What exactly is the current semantics/buffer sizes?\n> > > b) When do clients send multiple QMP commands on one channel without\n> > > waiting for the response to the previous command?\n> > > c) Would one queue entry for each class of commands/channel work\n> > > (Where a class of commands is currently 'normal' and 'oob')\n> > \n> > I do wonder if we need to worry about request limiting at all from the\n> > client side. For non-OOB commands clients will wait for a reply before\n> > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > \n> > OOB commands are supposed to be things which can be handled quickly\n> > without blocking, so even if a client sent several commands at once\n> > without waiting for replies, they're going to be processed quickly,\n> > so whether we temporarily block reading off the wire is a minor\n> > detail.\n> \n> Lets just define it so that it can't - you send an OOB command and wait\n> for it's response before sending another on that channel.\n> \n> > IOW, I think we could just have a fixed 10 command queue and apps just\n> > pretend that there's an infinite queue and nothing bad would happen from\n> > the app's POV.\n> \n> Can you justify 10 as opposed to 1?\n\nSemantically I don't think it makes a difference if the OOB commands are\nbeing processed sequentially by their thread. A >1 length queue would only\nmatter for non-OOB commands if an app was filling the pipeline with non-OOB\nrequests, as then that could block reading of OOB commands. \n\n\nRegards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=berrange@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtvqS6rVjz9s3w\n\tfor ;\n\tFri, 15 Sep 2017 22:29:55 +1000 (AEST)","from localhost ([::1]:53184 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dspkn-0003gH-Cg\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 08:29:53 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:57389)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dspkU-0003fg-GG\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:29:35 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dspkQ-0000Y0-GK\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:29:34 -0400","from mx1.redhat.com ([209.132.183.28]:57594)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dspkQ-0000XQ-80\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 08:29:30 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 2923B4E4C2;\n\tFri, 15 Sep 2017 12:29:29 +0000 (UTC)","from redhat.com (ovpn-116-81.ams2.redhat.com [10.36.116.81])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 3E21A7612C;\n\tFri, 15 Sep 2017 12:29:16 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 2923B4E4C2","Date":"Fri, 15 Sep 2017 13:29:13 +0100","From":"\"Daniel P. Berrange\" ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170915122913.GJ13610@redhat.com>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170915121956.GO2170@work-vm>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.38]);\n\tFri, 15 Sep 2017 12:29:29 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"\"Daniel P. Berrange\" ","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769219,"web_url":"http://patchwork.ozlabs.org/comment/1769219/","msgid":"<20170915142945.GP2170@work-vm>","list_archive_url":null,"date":"2017-09-15T14:29:46","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Daniel P. Berrange (berrange@redhat.com) wrote:\n> On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > \n> > > > > > > > I agree.\n> > > > > > > > \n> > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > \n> > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > \n> > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > next version.\n> > > > > > > \n> > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > \n> > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > \n> > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > the client stays within the receive buffer size.\n> > > > > > \n> > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > size and make sure to leave enough room.\n> > > > > \n> > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > and of course it is inherantly racy.\n> > > > > \n> > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > of the last queued command, the client will know which (if any) of\n> > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > it restarts reading.\n> > > > \n> > > > Hmm and now we're implementing flow control!\n> > > > \n> > > > a) What exactly is the current semantics/buffer sizes?\n> > > > b) When do clients send multiple QMP commands on one channel without\n> > > > waiting for the response to the previous command?\n> > > > c) Would one queue entry for each class of commands/channel work\n> > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > \n> > > I do wonder if we need to worry about request limiting at all from the\n> > > client side. For non-OOB commands clients will wait for a reply before\n> > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > \n> > > OOB commands are supposed to be things which can be handled quickly\n> > > without blocking, so even if a client sent several commands at once\n> > > without waiting for replies, they're going to be processed quickly,\n> > > so whether we temporarily block reading off the wire is a minor\n> > > detail.\n> > \n> > Lets just define it so that it can't - you send an OOB command and wait\n> > for it's response before sending another on that channel.\n> > \n> > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > pretend that there's an infinite queue and nothing bad would happen from\n> > > the app's POV.\n> > \n> > Can you justify 10 as opposed to 1?\n> \n> Semantically I don't think it makes a difference if the OOB commands are\n> being processed sequentially by their thread. A >1 length queue would only\n> matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> requests, as then that could block reading of OOB commands. \n\nBut can't we just tell the app not to?\n\nDave\n\n> \n> Regards,\n> Daniel\n> -- \n> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|\n> |: https://libvirt.org -o- https://fstop138.berrange.com :|\n> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtyVc3gVlz9s0Z\n\tfor ;\n\tSat, 16 Sep 2017 00:30:31 +1000 (AEST)","from localhost ([::1]:53585 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsrdT-0005px-Cf\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 10:30:27 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:33282)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsrd6-0005oC-9q\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:30:06 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsrd2-0001HQ-8r\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:30:04 -0400","from mx1.redhat.com ([209.132.183.28]:49770)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsrd1-0001H6-VY\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:30:00 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id A5C0481E16;\n\tFri, 15 Sep 2017 14:29:58 +0000 (UTC)","from work-vm (ovpn-117-180.ams2.redhat.com [10.36.117.180])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 07AD660176;\n\tFri, 15 Sep 2017 14:29:48 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com A5C0481E16","Date":"Fri, 15 Sep 2017 15:29:46 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"\"Daniel P. Berrange\" ","Message-ID":"<20170915142945.GP2170@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915122913.GJ13610@redhat.com>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tFri, 15 Sep 2017 14:29:58 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769221,"web_url":"http://patchwork.ozlabs.org/comment/1769221/","msgid":"<20170915143239.GL13610@redhat.com>","list_archive_url":null,"date":"2017-09-15T14:32:39","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Sep 15, 2017 at 03:29:46PM +0100, Dr. David Alan Gilbert wrote:\n> * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > \n> > > > > > > > > I agree.\n> > > > > > > > > \n> > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > \n> > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > \n> > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > next version.\n> > > > > > > > \n> > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > \n> > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > \n> > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > the client stays within the receive buffer size.\n> > > > > > > \n> > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > size and make sure to leave enough room.\n> > > > > > \n> > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > and of course it is inherantly racy.\n> > > > > > \n> > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > it restarts reading.\n> > > > > \n> > > > > Hmm and now we're implementing flow control!\n> > > > > \n> > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > waiting for the response to the previous command?\n> > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > \n> > > > I do wonder if we need to worry about request limiting at all from the\n> > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > \n> > > > OOB commands are supposed to be things which can be handled quickly\n> > > > without blocking, so even if a client sent several commands at once\n> > > > without waiting for replies, they're going to be processed quickly,\n> > > > so whether we temporarily block reading off the wire is a minor\n> > > > detail.\n> > > \n> > > Lets just define it so that it can't - you send an OOB command and wait\n> > > for it's response before sending another on that channel.\n> > > \n> > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > the app's POV.\n> > > \n> > > Can you justify 10 as opposed to 1?\n> > \n> > Semantically I don't think it makes a difference if the OOB commands are\n> > being processed sequentially by their thread. A >1 length queue would only\n> > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > requests, as then that could block reading of OOB commands. \n> \n> But can't we just tell the app not to?\n\nYes, a sensible app would not do that. So this feels like mostly a documentation\nproblem.\n\nRegards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=berrange@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtyYn63TGz9s7h\n\tfor ;\n\tSat, 16 Sep 2017 00:33:17 +1000 (AEST)","from localhost ([::1]:53587 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dsrgC-0007ii-0m\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 10:33:16 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:34176)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dsrft-0007iN-6W\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:32:58 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dsrfq-0004Sx-0S\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:32:57 -0400","from mx1.redhat.com ([209.132.183.28]:53520)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dsrfp-0004SW-OR\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:32:53 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 1214461472;\n\tFri, 15 Sep 2017 14:32:52 +0000 (UTC)","from redhat.com (ovpn-116-81.ams2.redhat.com [10.36.116.81])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 00F275C88D;\n\tFri, 15 Sep 2017 14:32:41 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 1214461472","Date":"Fri, 15 Sep 2017 15:32:39 +0100","From":"\"Daniel P. Berrange\" ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170915143239.GL13610@redhat.com>","References":"\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com> <20170915142945.GP2170@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170915142945.GP2170@work-vm>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.39]);\n\tFri, 15 Sep 2017 14:32:52 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"\"Daniel P. Berrange\" ","Cc":"Laurent Vivier , Fam Zheng , Juan\n\tQuintela , Markus Armbruster , \n\tQEMU , Peter Xu , \n\tMichael Roth , Stefan Hajnoczi\n\t, =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769235,"web_url":"http://patchwork.ozlabs.org/comment/1769235/","msgid":"<20170915145632.GD18767@stefanha-x1.localdomain>","list_archive_url":null,"date":"2017-09-15T14:56:32","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":17227,"url":"http://patchwork.ozlabs.org/api/people/17227/","name":"Stefan Hajnoczi","email":"stefanha@redhat.com"},"content":"On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > \n> > > > > > > > I agree.\n> > > > > > > > \n> > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > \n> > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > \n> > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > next version.\n> > > > > > > \n> > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > \n> > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > \n> > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > the client stays within the receive buffer size.\n> > > > > > \n> > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > size and make sure to leave enough room.\n> > > > > \n> > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > and of course it is inherantly racy.\n> > > > > \n> > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > of the last queued command, the client will know which (if any) of\n> > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > it restarts reading.\n> > > > \n> > > > Hmm and now we're implementing flow control!\n> > > > \n> > > > a) What exactly is the current semantics/buffer sizes?\n> > > > b) When do clients send multiple QMP commands on one channel without\n> > > > waiting for the response to the previous command?\n> > > > c) Would one queue entry for each class of commands/channel work\n> > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > \n> > > I do wonder if we need to worry about request limiting at all from the\n> > > client side. For non-OOB commands clients will wait for a reply before\n> > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > \n> > > OOB commands are supposed to be things which can be handled quickly\n> > > without blocking, so even if a client sent several commands at once\n> > > without waiting for replies, they're going to be processed quickly,\n> > > so whether we temporarily block reading off the wire is a minor\n> > > detail.\n> > \n> > Lets just define it so that it can't - you send an OOB command and wait\n> > for it's response before sending another on that channel.\n> > \n> > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > pretend that there's an infinite queue and nothing bad would happen from\n> > > the app's POV.\n> > \n> > Can you justify 10 as opposed to 1?\n> \n> Semantically I don't think it makes a difference if the OOB commands are\n> being processed sequentially by their thread. A >1 length queue would only\n> matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> requests, as then that could block reading of OOB commands. \n\nTo summarize:\n\nThe QMP server has a lookahead of 1 command so it can dispatch\nout-of-band commands. If 2 or more non-OOB commands are queued at the\nsame time then OOB processing will not occur.\n\nIs that right?\n\nStefan","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=stefanha@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtz7163dkz9sBZ\n\tfor ;\n\tSat, 16 Sep 2017 00:58:36 +1000 (AEST)","from localhost ([::1]:53716 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dss4g-0000ob-CL\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 10:58:34 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:41167)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dss2v-0006w6-LB\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:56:47 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dss2s-0006MO-Jr\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:56:45 -0400","from mx1.redhat.com ([209.132.183.28]:43382)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dss2s-0006Lu-Aw\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 10:56:42 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 1CDCEC058EA1;\n\tFri, 15 Sep 2017 14:56:41 +0000 (UTC)","from localhost (ovpn-117-224.ams2.redhat.com [10.36.117.224])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 315C76942B;\n\tFri, 15 Sep 2017 14:56:33 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 1CDCEC058EA1","Date":"Fri, 15 Sep 2017 15:56:32 +0100","From":"Stefan Hajnoczi ","To":"\"Daniel P. Berrange\" ","Message-ID":"<20170915145632.GD18767@stefanha-x1.localdomain>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915122913.GJ13610@redhat.com>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.32]);\n\tFri, 15 Sep 2017 14:56:41 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth , Juan Quintela\n\t, Markus Armbruster , \"Dr. David\n\tAlan Gilbert\" , Peter Xu ,\n\tQEMU , \tStefan Hajnoczi ,\n\t=?iso-8859-1?q?Marc-Andr=E9?= Lureau ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769248,"web_url":"http://patchwork.ozlabs.org/comment/1769248/","msgid":"<20170915151706.GQ2170@work-vm>","list_archive_url":null,"date":"2017-09-15T15:17:07","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Stefan Hajnoczi (stefanha@redhat.com) wrote:\n> On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > \n> > > > > > > > > I agree.\n> > > > > > > > > \n> > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > \n> > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > \n> > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > next version.\n> > > > > > > > \n> > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > \n> > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > \n> > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > the client stays within the receive buffer size.\n> > > > > > > \n> > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > size and make sure to leave enough room.\n> > > > > > \n> > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > and of course it is inherantly racy.\n> > > > > > \n> > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > it restarts reading.\n> > > > > \n> > > > > Hmm and now we're implementing flow control!\n> > > > > \n> > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > waiting for the response to the previous command?\n> > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > \n> > > > I do wonder if we need to worry about request limiting at all from the\n> > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > \n> > > > OOB commands are supposed to be things which can be handled quickly\n> > > > without blocking, so even if a client sent several commands at once\n> > > > without waiting for replies, they're going to be processed quickly,\n> > > > so whether we temporarily block reading off the wire is a minor\n> > > > detail.\n> > > \n> > > Lets just define it so that it can't - you send an OOB command and wait\n> > > for it's response before sending another on that channel.\n> > > \n> > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > the app's POV.\n> > > \n> > > Can you justify 10 as opposed to 1?\n> > \n> > Semantically I don't think it makes a difference if the OOB commands are\n> > being processed sequentially by their thread. A >1 length queue would only\n> > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > requests, as then that could block reading of OOB commands. \n> \n> To summarize:\n> \n> The QMP server has a lookahead of 1 command so it can dispatch\n> out-of-band commands. If 2 or more non-OOB commands are queued at the\n> same time then OOB processing will not occur.\n> \n> Is that right?\n\nI think my view is slightly more complex;\n a) There's a pair of queues for each channel\n b) There's a central pair of queues on the QMP server\n one for OOB commands and one for normal commands.\n c) Each queue is only really guaranteed to be one deep.\n\n That means that each one of the channels can send a non-OOB\ncommand without getting in the way of a channel that wants\nto send one.\n\nDave\n\n> Stefan\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xtzY9454nz9sPs\n\tfor ;\n\tSat, 16 Sep 2017 01:17:49 +1000 (AEST)","from localhost ([::1]:53785 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dssNH-0004x7-Md\n\tfor incoming@patchwork.ozlabs.org; Fri, 15 Sep 2017 11:17:47 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:48551)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dssMy-0004wp-M6\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 11:17:30 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dssMs-0005Kd-Ar\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 11:17:28 -0400","from mx1.redhat.com ([209.132.183.28]:36774)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dssMs-0005JE-1T\n\tfor qemu-devel@nongnu.org; Fri, 15 Sep 2017 11:17:22 -0400","from smtp.corp.redhat.com\n\t(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id CA3C57E43E;\n\tFri, 15 Sep 2017 15:17:20 +0000 (UTC)","from work-vm (ovpn-117-180.ams2.redhat.com [10.36.117.180])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id CB4E57B595;\n\tFri, 15 Sep 2017 15:17:09 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com CA3C57E43E","Date":"Fri, 15 Sep 2017 16:17:07 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"Stefan Hajnoczi ","Message-ID":"<20170915151706.GQ2170@work-vm>","References":"\n\t<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>\n\t<20170915145632.GD18767@stefanha-x1.localdomain>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170915145632.GD18767@stefanha-x1.localdomain>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.12","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tFri, 15 Sep 2017 15:17:21 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Juan Quintela ,\n\tMarkus Armbruster , QEMU ,\n\tPeter Xu , Michael Roth , \n\t=?iso-8859-1?q?Marc-Andr=E9?= Lureau ,\n\tPaolo Bonzini , Fam Zheng , \n\tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1769983,"web_url":"http://patchwork.ozlabs.org/comment/1769983/","msgid":"<20170918083737.GD3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-18T08:37:37","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n> Hi\n> \n> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> > Hi\n> >> >\n> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> >> > > This series was born from this one:\n> >> > >\n> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >> > >\n> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n> >> > > ideas and suggestions. Finally we got such a solution that seems to\n> >> > > satisfy everyone.\n> >> > >\n> >> > > I re-started the versioning since this series is totally different\n> >> > > from previous one. Now it's version 1.\n> >> > >\n> >> > > In case new reviewers come along the way without reading previous\n> >> > > discussions, I will try to do a summary on what this is all about.\n> >> > >\n> >> > > What is OOB execution?\n> >> > > ======================\n> >> > >\n> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n> >> > > QMP is going throw these steps:\n> >> > >\n> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> > > /|\\ (2) (3) |\n> >> > > (1) | \\|/ (4)\n> >> > > +--------- main thread --------+\n> >> > >\n> >> > > The requests are executed by the so-called QMP-dispatcher after the\n> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n> >> > > parser and quickly returns.\n> >> >\n> >> > All commands should have the \"id\" field mandatory in this case, else\n> >> > the client will not distinguish the replies coming from the last/oob\n> >> > and the previous commands.\n> >> >\n> >> > This should probably be enforced upfront by client capability checks,\n> >> > more below.\n> >\n> > Hmm yes since the oob commands are actually running in async way,\n> > request ID should be needed here. However I'm not sure whether\n> > enabling the whole \"request ID\" thing is too big for this \"try to be\n> > small\" oob change... And IMHO it suites better to be part of the whole\n> > async work (no matter which implementation we'll use).\n> >\n> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n> > For oob commands, they will always have ID then no ordering issue, and\n> > we can do it async; for the rest of non-oob commands, we still allow\n> > them to go without ID, and since they are not oob, they'll always be\n> > done in order as well. Would this work?\n> \n> This mixed-mode is imho more complicated to deal with than having the\n> protocol enforced one way or the other, but that should work.\n> \n> >\n> >> >\n> >> > > Yeah I know in current code the parser calls dispatcher directly\n> >> > > (please see handle_qmp_command()). However it's not true again after\n> >> > > this series (parser will has its own IO thread, and dispatcher will\n> >> > > still be run in main thread). So this OOB does brings something\n> >> > > different.\n> >> > >\n> >> > > There are more details on why OOB and the difference/relationship\n> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> >> > > slightly out of topic (and believe me, it's not easy for me to\n> >> > > summarize that). For more information, please refers to [1].\n> >> > >\n> >> > > Summary ends here.\n> >> > >\n> >> > > Some Implementation Details\n> >> > > ===========================\n> >> > >\n> >> > > Again, I mentioned that the old QMP workflow is this:\n> >> > >\n> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> > > /|\\ (2) (3) |\n> >> > > (1) | \\|/ (4)\n> >> > > +--------- main thread --------+\n> >> > >\n> >> > > What this series does is, firstly:\n> >> > >\n> >> > > JSON Parser QMP Dispatcher --> Respond\n> >> > > /|\\ | /|\\ (4) |\n> >> > > | | (2) | (3) | (5)\n> >> > > (1) | +-----> | \\|/\n> >> > > +--------- main thread <-------+\n> >> > >\n> >> > > And further:\n> >> > >\n> >> > > queue/kick\n> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n> >> > > /|\\ | (3) /|\\ (4) |\n> >> > > (1) | | (2) | | (5)\n> >> > > | \\|/ | \\|/\n> >> > > IO thread main thread <-------+\n> >> >\n> >> > Is the queue per monitor or per client?\n> >\n> > The queue is currently global. I think yes maybe at least we can do it\n> > per monitor, but I am not sure whether that is urgent or can be\n> > postponed. After all now QMPRequest (please refer to patch 11) is\n> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n> > per-monitor.\n> >\n> >> > And is the dispatching going\n> >> > to be processed even if the client is disconnected, and are new\n> >> > clients going to receive the replies from previous clients\n> >> > commands?\n> >\n> > [1]\n> >\n> > (will discuss together below)\n> >\n> >> > I\n> >> > believe there should be a per-client context, so there won't be \"id\"\n> >> > request conflicts.\n> >\n> > I'd say I am not familiar with this \"client\" idea, since after all\n> > IMHO one monitor is currently designed to mostly work with a single\n> > client. Say, unix sockets, telnet, all these backends are only single\n> > channeled, and one monitor instance can only work with one client at a\n> > time. Then do we really need to add this client layer upon it? IMHO\n> > the user can just provide more monitors if they wants more clients\n> > (and at least these clients should know the existance of the others or\n> > there might be problem, otherwise user2 will fail a migration, finally\n> > noticed that user1 has already triggered one), and the user should\n> > manage them well.\n> \n> qemu should support a management layer / libvirt restart/reconnect.\n> Afaik, it mostly work today. There might be a cases where libvirt can\n> be confused if it receives a reply from a previous connection command,\n> but due to the sync processing of the chardev, I am not sure you can\n> get in this situation. By adding \"oob\" commands and queuing, the\n> client will have to remember which was the last \"id\" used, or it will\n> create more conflict after a reconnect.\n> \n> Imho we should introduce the client/connection concept to avoid this\n> confusion (unexpected reply & per client id space).\n\nHmm I agree that the reconnect feature would be nice, but if so IMHO\ninstead of throwing responses away when client disconnect, we should\nreally keep them, and when the client reconnects, we queue the\nresponses again.\n\nI think we have other quite simple ways to solve the \"unexpected\nreply\" and \"per-client-id duplication\" issues you have mentioned.\n\nFirstly, when client gets unexpected replies (\"id\" field not in its\nown request queue), the client should just ignore that reply, which\nseems natural to me.\n\nThen, if client disconnected and reconnected, it should not have the\nproblem to generate duplicated id for request, since it should know\nwhat requests it has sent already. A simplest case I can think of is,\nthe ID should contains the following tuple:\n\n (client name, client unique ID, request ID)\n\nHere \"client name\" can be something like \"libvirt\", which is the name\nof client application;\n\n\"client unique ID\" can be anything generated when client starts, it\nidentifies a single client session, maybe a UUID.\n\n\"request ID\" can be a unsigned integer starts from zero, and increases\neach time the client sends one request.\n\nI believe current libvirt is using \"client name\" + \"request ID\". It's\nsomething similar (after all I think we don't normally have >1 libvirt\nto manage single QEMU, so I think it should be good enough).\n\nThen even if client disconnect and reconnect, request ID won't lose,\nand no duplication would happen IMHO.\n\n> \n> >\n> >> >\n> >> > >\n> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> >> > > parser.\n> >> >\n> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n> >> > command a bit pointless. It helps with legacy client that wouldn't\n> >> > expect out-of-order replies if qemu were to run oob commands oob by\n> >> > default though.\n> >\n> > After all oob somehow breaks existing rules or sync execution. I\n> > thought the more important goal was at least to keep the legacy\n> > behaviors when adding new things, no?\n> \n> Of course we have to keep compatibily. What do you mean by \"oob\n> somehow breaks existing rules or sync execution\"? oob means queuing\n> and unordered reply support, so clearly this is breaking the current\n> \"mostly ordered\" behaviour (mostly because events may still come any\n> time..., and the reconnect issue discussed above).\n\nYes. That's what I mean, it breaks the synchronous scemantic. But\nI should definitely not call it a \"break\" though since old clients\nwill work perfectly fine with it. Sorry for the bad wording.\n\n> \n> >> > Clients shouldn't care about how/where a command is\n> >> > being queued or not. If they send a command, they want it processed as\n> >> > quickly as possible. However, it can be interesting to know if the\n> >> > implementation of the command will be able to deliver oob, so that\n> >> > data in the introspection could be useful.\n> >> >\n> >> > I would rather propose a client/server capability in qmp_capabilities,\n> >> > call it \"oob\":\n> >> >\n> >> > This capability indicates oob commands support.\n> >>\n> >> The problem is indicating which commands support oob as opposed to\n> >> indicating whether oob is present at all. Future versions will\n> >> probably make more commands oob-able and a client will want to know\n> >> whether it can rely on a particular command being non-blocking.\n> >\n> > Yes.\n> >\n> > And IMHO we don't urgently need that \"whether the server globally\n> > supports oob\" thing. Client can just know that from query-qmp-schema\n> > already - there will always be the \"allow-oob\" new field for command\n> > typed entries. IMHO that's a solid hint.\n> >\n> > But I don't object to return it as well in qmp_capabilities.\n> \n> Does it feel right that the client can specify how the command are\n> processed / queued ? Isn't it preferable to leave that to the server\n> to decide? Why would a client specify that? And should the server be\n> expected to behave differently? What the client needs to be able is to\n> match the unordered replies, and that can be stated during cap\n> negotiation / qmp_capabilties. The server is expected to do a best\n> effort to handle commands and their priorities. If the client needs\n> several command queue, it is simpler to open several connection rather\n> than trying to fit that weird priority logic in the protocol imho.\n\nSorry I may have missed the point here. We were discussing about a\nglobal hint for \"oob\" support, am I right? Then, could I ask what's\nthe \"weird priority logic\" you mentioned?\n\n> \n> >\n> >>\n> >> > An oob command is a regular client message request with the \"id\"\n> >> > member mandatory, but the reply may be delivered\n> >> > out of order by the server if the client supports\n> >> > it too.\n> >> >\n> >> > If both the server and the client have the \"oob\" capability, the\n> >> > server can handle new client requests while previous requests are being\n> >> > processed.\n> >> >\n> >> > If the client doesn't have the \"oob\" capability, it may still call\n> >> > an oob command, and make multiple outstanding calls. In this case,\n> >> > the commands are processed in order, so the replies will also be in\n> >> > order. The \"id\" member isn't mandatory in this case.\n> >> >\n> >> > The client should match the replies with the \"id\" member associated\n> >> > with the requests.\n> >> >\n> >> > When a client is disconnected, the pending commands are not\n> >> > necessarily cancelled. But the future clients will not get replies from\n> >> > commands they didn't make (they might, however, receive side-effects\n> >> > events).\n> >>\n> >> What's the behaviour on the current monitor?\n> >\n> > Yeah I want to ask the same question, along with questioning about\n> > above [1].\n> >\n> > IMHO this series will not change the behaviors of these, so IMHO the\n> > behaviors will be the same before/after this series. E.g., when client\n> > dropped right after the command is executed, I think we will still\n> > execute the command, though we should encounter something odd in\n> > monitor_json_emitter() somewhere when we want to respond. And it will\n> > happen the same after this series.\n> \n> I think it can get worse after your series, because you queue the\n> commands, so clearly a new client can get replies from an old client\n> commands. As said above, I am not convinced you can get in that\n> situation with current code.\n\nHmm, seems so. But would this a big problem?\n\nI really think the new client should just throw that response away if\nit does not really know that response (from peeking at \"id\" field),\njust like my opinion above.\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwfY146Twz9rxm\n\tfor ;\n\tMon, 18 Sep 2017 18:38:29 +1000 (AEST)","from localhost ([::1]:35248 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dtrZT-0001dy-Lj\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 04:38:27 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:41691)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtrYu-0001cM-1J\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 04:37:54 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtrYq-0005BP-KP\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 04:37:51 -0400","from mx1.redhat.com ([209.132.183.28]:49804)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dtrYq-00050B-9E\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 04:37:48 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id DE3F6C04B30E;\n\tMon, 18 Sep 2017 08:37:46 +0000 (UTC)","from pxdev.xzpeter.org (dhcp-15-224.nay.redhat.com [10.66.15.224])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id B45D2660C0;\n\tMon, 18 Sep 2017 08:37:39 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com DE3F6C04B30E","Date":"Mon, 18 Sep 2017 16:37:37 +0800","From":"Peter Xu ","To":"=?utf-8?q?Marc-Andr=C3=A9?= Lureau ","Message-ID":"<20170918083737.GD3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tMon, 18 Sep 2017 08:37:47 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela , QEMU ,\n\tMarkus Armbruster ,\n\t\"Dr. David Alan Gilbert\" ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770007,"web_url":"http://patchwork.ozlabs.org/comment/1770007/","msgid":"<20170918092625.GE3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-18T09:26:25","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:\n> * Stefan Hajnoczi (stefanha@redhat.com) wrote:\n> > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > > \n> > > > > > > > > > I agree.\n> > > > > > > > > > \n> > > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > > \n> > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > > \n> > > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > > next version.\n> > > > > > > > > \n> > > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > > \n> > > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > > \n> > > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > > the client stays within the receive buffer size.\n> > > > > > > > \n> > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > > size and make sure to leave enough room.\n> > > > > > > \n> > > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > > and of course it is inherantly racy.\n> > > > > > > \n> > > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > > it restarts reading.\n> > > > > > \n> > > > > > Hmm and now we're implementing flow control!\n> > > > > > \n> > > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > > waiting for the response to the previous command?\n> > > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > > \n> > > > > I do wonder if we need to worry about request limiting at all from the\n> > > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > > \n> > > > > OOB commands are supposed to be things which can be handled quickly\n> > > > > without blocking, so even if a client sent several commands at once\n> > > > > without waiting for replies, they're going to be processed quickly,\n> > > > > so whether we temporarily block reading off the wire is a minor\n> > > > > detail.\n> > > > \n> > > > Lets just define it so that it can't - you send an OOB command and wait\n> > > > for it's response before sending another on that channel.\n> > > > \n> > > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > > the app's POV.\n> > > > \n> > > > Can you justify 10 as opposed to 1?\n> > > \n> > > Semantically I don't think it makes a difference if the OOB commands are\n> > > being processed sequentially by their thread. A >1 length queue would only\n> > > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > > requests, as then that could block reading of OOB commands. \n> > \n> > To summarize:\n> > \n> > The QMP server has a lookahead of 1 command so it can dispatch\n> > out-of-band commands. If 2 or more non-OOB commands are queued at the\n> > same time then OOB processing will not occur.\n> > \n> > Is that right?\n> \n> I think my view is slightly more complex;\n> a) There's a pair of queues for each channel\n> b) There's a central pair of queues on the QMP server\n> one for OOB commands and one for normal commands.\n> c) Each queue is only really guaranteed to be one deep.\n> \n> That means that each one of the channels can send a non-OOB\n> command without getting in the way of a channel that wants\n> to send one.\n\nBut current version should not be that complex:\n\nFirstly, parser thread will only be enabled for QMP+NO_MIXED monitors.\n\nThen, we only have a single global queue for QMP non-oob commands, and\nwe don't have response queue yet. We do respond just like before in a\nsynchronous way (I explained why - for OOB we don't need that\ncomplexity IMHO).\n\nWhen we parse commands, we execute it directly if OOB, otherwise we\nput it onto request queue. Request queue handling is done by a main\nthread QEMUBH. That's all.\n\nWould this \"simple version\" suffice to implement this whole OOB idea?\n\n(Again, I really don't think we need to specify queue length to 1,\n though we can make it small)","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwgdH2Sj7z9s72\n\tfor ;\n\tMon, 18 Sep 2017 19:27:15 +1000 (AEST)","from localhost ([::1]:35403 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dtsKf-0006M7-Ej\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 05:27:13 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:54582)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtsK8-0006BH-1N\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 05:26:41 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtsK4-0006V4-1b\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 05:26:40 -0400","from mx1.redhat.com ([209.132.183.28]:37512)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dtsK3-0006Ul-NU\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 05:26:35 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id B1FF713A98;\n\tMon, 18 Sep 2017 09:26:34 +0000 (UTC)","from pxdev.xzpeter.org (dhcp-15-224.nay.redhat.com [10.66.15.224])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 317195C3FA;\n\tMon, 18 Sep 2017 09:26:26 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com B1FF713A98","Date":"Mon, 18 Sep 2017 17:26:25 +0800","From":"Peter Xu ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170918092625.GE3617@pxdev.xzpeter.org>","References":"<20170914151911.GB7370@stefanha-x1.localdomain>\n\t<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>\n\t<20170915145632.GD18767@stefanha-x1.localdomain>\n\t<20170915151706.GQ2170@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170915151706.GQ2170@work-vm>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.29]);\n\tMon, 18 Sep 2017 09:26:34 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Juan Quintela ,\n\tMarkus Armbruster , QEMU ,\n\tMichael Roth , =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ,\n\tFam Zheng , \tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770059,"web_url":"http://patchwork.ozlabs.org/comment/1770059/","msgid":"","list_archive_url":null,"date":"2017-09-18T10:20:19","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6442,"url":"http://patchwork.ozlabs.org/api/people/6442/","name":"Marc-André Lureau","email":"marcandre.lureau@gmail.com"},"content":"Hi\n\nOn Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n> On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n>> Hi\n>>\n>> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n>> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n>> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> >> > Hi\n>> >> >\n>> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n>> >> > > This series was born from this one:\n>> >> > >\n>> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>> >> > >\n>> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n>> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n>> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n>> >> > > ideas and suggestions. Finally we got such a solution that seems to\n>> >> > > satisfy everyone.\n>> >> > >\n>> >> > > I re-started the versioning since this series is totally different\n>> >> > > from previous one. Now it's version 1.\n>> >> > >\n>> >> > > In case new reviewers come along the way without reading previous\n>> >> > > discussions, I will try to do a summary on what this is all about.\n>> >> > >\n>> >> > > What is OOB execution?\n>> >> > > ======================\n>> >> > >\n>> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n>> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n>> >> > > QMP is going throw these steps:\n>> >> > >\n>> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> > > /|\\ (2) (3) |\n>> >> > > (1) | \\|/ (4)\n>> >> > > +--------- main thread --------+\n>> >> > >\n>> >> > > The requests are executed by the so-called QMP-dispatcher after the\n>> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n>> >> > > parser and quickly returns.\n>> >> >\n>> >> > All commands should have the \"id\" field mandatory in this case, else\n>> >> > the client will not distinguish the replies coming from the last/oob\n>> >> > and the previous commands.\n>> >> >\n>> >> > This should probably be enforced upfront by client capability checks,\n>> >> > more below.\n>> >\n>> > Hmm yes since the oob commands are actually running in async way,\n>> > request ID should be needed here. However I'm not sure whether\n>> > enabling the whole \"request ID\" thing is too big for this \"try to be\n>> > small\" oob change... And IMHO it suites better to be part of the whole\n>> > async work (no matter which implementation we'll use).\n>> >\n>> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n>> > For oob commands, they will always have ID then no ordering issue, and\n>> > we can do it async; for the rest of non-oob commands, we still allow\n>> > them to go without ID, and since they are not oob, they'll always be\n>> > done in order as well. Would this work?\n>>\n>> This mixed-mode is imho more complicated to deal with than having the\n>> protocol enforced one way or the other, but that should work.\n>>\n>> >\n>> >> >\n>> >> > > Yeah I know in current code the parser calls dispatcher directly\n>> >> > > (please see handle_qmp_command()). However it's not true again after\n>> >> > > this series (parser will has its own IO thread, and dispatcher will\n>> >> > > still be run in main thread). So this OOB does brings something\n>> >> > > different.\n>> >> > >\n>> >> > > There are more details on why OOB and the difference/relationship\n>> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n>> >> > > slightly out of topic (and believe me, it's not easy for me to\n>> >> > > summarize that). For more information, please refers to [1].\n>> >> > >\n>> >> > > Summary ends here.\n>> >> > >\n>> >> > > Some Implementation Details\n>> >> > > ===========================\n>> >> > >\n>> >> > > Again, I mentioned that the old QMP workflow is this:\n>> >> > >\n>> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> > > /|\\ (2) (3) |\n>> >> > > (1) | \\|/ (4)\n>> >> > > +--------- main thread --------+\n>> >> > >\n>> >> > > What this series does is, firstly:\n>> >> > >\n>> >> > > JSON Parser QMP Dispatcher --> Respond\n>> >> > > /|\\ | /|\\ (4) |\n>> >> > > | | (2) | (3) | (5)\n>> >> > > (1) | +-----> | \\|/\n>> >> > > +--------- main thread <-------+\n>> >> > >\n>> >> > > And further:\n>> >> > >\n>> >> > > queue/kick\n>> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n>> >> > > /|\\ | (3) /|\\ (4) |\n>> >> > > (1) | | (2) | | (5)\n>> >> > > | \\|/ | \\|/\n>> >> > > IO thread main thread <-------+\n>> >> >\n>> >> > Is the queue per monitor or per client?\n>> >\n>> > The queue is currently global. I think yes maybe at least we can do it\n>> > per monitor, but I am not sure whether that is urgent or can be\n>> > postponed. After all now QMPRequest (please refer to patch 11) is\n>> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n>> > per-monitor.\n>> >\n>> >> > And is the dispatching going\n>> >> > to be processed even if the client is disconnected, and are new\n>> >> > clients going to receive the replies from previous clients\n>> >> > commands?\n>> >\n>> > [1]\n>> >\n>> > (will discuss together below)\n>> >\n>> >> > I\n>> >> > believe there should be a per-client context, so there won't be \"id\"\n>> >> > request conflicts.\n>> >\n>> > I'd say I am not familiar with this \"client\" idea, since after all\n>> > IMHO one monitor is currently designed to mostly work with a single\n>> > client. Say, unix sockets, telnet, all these backends are only single\n>> > channeled, and one monitor instance can only work with one client at a\n>> > time. Then do we really need to add this client layer upon it? IMHO\n>> > the user can just provide more monitors if they wants more clients\n>> > (and at least these clients should know the existance of the others or\n>> > there might be problem, otherwise user2 will fail a migration, finally\n>> > noticed that user1 has already triggered one), and the user should\n>> > manage them well.\n>>\n>> qemu should support a management layer / libvirt restart/reconnect.\n>> Afaik, it mostly work today. There might be a cases where libvirt can\n>> be confused if it receives a reply from a previous connection command,\n>> but due to the sync processing of the chardev, I am not sure you can\n>> get in this situation. By adding \"oob\" commands and queuing, the\n>> client will have to remember which was the last \"id\" used, or it will\n>> create more conflict after a reconnect.\n>>\n>> Imho we should introduce the client/connection concept to avoid this\n>> confusion (unexpected reply & per client id space).\n>\n> Hmm I agree that the reconnect feature would be nice, but if so IMHO\n> instead of throwing responses away when client disconnect, we should\n> really keep them, and when the client reconnects, we queue the\n> responses again.\n>\n> I think we have other quite simple ways to solve the \"unexpected\n> reply\" and \"per-client-id duplication\" issues you have mentioned.\n>\n> Firstly, when client gets unexpected replies (\"id\" field not in its\n> own request queue), the client should just ignore that reply, which\n> seems natural to me.\n\nThe trouble is that it may legitimately use the same \"id\" value for\nnew requests. And I don't see a simple way to handle that without\nraces.\n\n>\n> Then, if client disconnected and reconnected, it should not have the\n> problem to generate duplicated id for request, since it should know\n> what requests it has sent already. A simplest case I can think of is,\n> the ID should contains the following tuple:\n\nIf you assume the \"same\" client will recover its state, yes.\n\n>\n> (client name, client unique ID, request ID)\n>\n> Here \"client name\" can be something like \"libvirt\", which is the name\n> of client application;\n>\n> \"client unique ID\" can be anything generated when client starts, it\n> identifies a single client session, maybe a UUID.\n>\n> \"request ID\" can be a unsigned integer starts from zero, and increases\n> each time the client sends one request.\n\nThis is introducing session handling, and can be done in server side\nonly without changes in the protocol I believe.\n\n>\n> I believe current libvirt is using \"client name\" + \"request ID\". It's\n> something similar (after all I think we don't normally have >1 libvirt\n> to manage single QEMU, so I think it should be good enough).\n\nI am not sure we should base our protocol usage assumptions based on\nlibvirt only, but rather on what is possible today (like queuing\nrequests in the socket etc..).\n\n> Then even if client disconnect and reconnect, request ID won't lose,\n> and no duplication would happen IMHO.\n>\n>>\n>> >\n>> >> >\n>> >> > >\n>> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n>> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n>> >> > > parser.\n>> >> >\n>> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n>> >> > command a bit pointless. It helps with legacy client that wouldn't\n>> >> > expect out-of-order replies if qemu were to run oob commands oob by\n>> >> > default though.\n>> >\n>> > After all oob somehow breaks existing rules or sync execution. I\n>> > thought the more important goal was at least to keep the legacy\n>> > behaviors when adding new things, no?\n>>\n>> Of course we have to keep compatibily. What do you mean by \"oob\n>> somehow breaks existing rules or sync execution\"? oob means queuing\n>> and unordered reply support, so clearly this is breaking the current\n>> \"mostly ordered\" behaviour (mostly because events may still come any\n>> time..., and the reconnect issue discussed above).\n>\n> Yes. That's what I mean, it breaks the synchronous scemantic. But\n> I should definitely not call it a \"break\" though since old clients\n> will work perfectly fine with it. Sorry for the bad wording.\n>\n>>\n>> >> > Clients shouldn't care about how/where a command is\n>> >> > being queued or not. If they send a command, they want it processed as\n>> >> > quickly as possible. However, it can be interesting to know if the\n>> >> > implementation of the command will be able to deliver oob, so that\n>> >> > data in the introspection could be useful.\n>> >> >\n>> >> > I would rather propose a client/server capability in qmp_capabilities,\n>> >> > call it \"oob\":\n>> >> >\n>> >> > This capability indicates oob commands support.\n>> >>\n>> >> The problem is indicating which commands support oob as opposed to\n>> >> indicating whether oob is present at all. Future versions will\n>> >> probably make more commands oob-able and a client will want to know\n>> >> whether it can rely on a particular command being non-blocking.\n>> >\n>> > Yes.\n>> >\n>> > And IMHO we don't urgently need that \"whether the server globally\n>> > supports oob\" thing. Client can just know that from query-qmp-schema\n>> > already - there will always be the \"allow-oob\" new field for command\n>> > typed entries. IMHO that's a solid hint.\n>> >\n>> > But I don't object to return it as well in qmp_capabilities.\n>>\n>> Does it feel right that the client can specify how the command are\n>> processed / queued ? Isn't it preferable to leave that to the server\n>> to decide? Why would a client specify that? And should the server be\n>> expected to behave differently? What the client needs to be able is to\n>> match the unordered replies, and that can be stated during cap\n>> negotiation / qmp_capabilties. The server is expected to do a best\n>> effort to handle commands and their priorities. If the client needs\n>> several command queue, it is simpler to open several connection rather\n>> than trying to fit that weird priority logic in the protocol imho.\n>\n> Sorry I may have missed the point here. We were discussing about a\n> global hint for \"oob\" support, am I right? Then, could I ask what's\n> the \"weird priority logic\" you mentioned?\n\nI call per-message oob hint a kind of priority logic, since you can\nmake the same request without oob in the same session and in parallel.\n\n>>\n>> >\n>> >>\n>> >> > An oob command is a regular client message request with the \"id\"\n>> >> > member mandatory, but the reply may be delivered\n>> >> > out of order by the server if the client supports\n>> >> > it too.\n>> >> >\n>> >> > If both the server and the client have the \"oob\" capability, the\n>> >> > server can handle new client requests while previous requests are being\n>> >> > processed.\n>> >> >\n>> >> > If the client doesn't have the \"oob\" capability, it may still call\n>> >> > an oob command, and make multiple outstanding calls. In this case,\n>> >> > the commands are processed in order, so the replies will also be in\n>> >> > order. The \"id\" member isn't mandatory in this case.\n>> >> >\n>> >> > The client should match the replies with the \"id\" member associated\n>> >> > with the requests.\n>> >> >\n>> >> > When a client is disconnected, the pending commands are not\n>> >> > necessarily cancelled. But the future clients will not get replies from\n>> >> > commands they didn't make (they might, however, receive side-effects\n>> >> > events).\n>> >>\n>> >> What's the behaviour on the current monitor?\n>> >\n>> > Yeah I want to ask the same question, along with questioning about\n>> > above [1].\n>> >\n>> > IMHO this series will not change the behaviors of these, so IMHO the\n>> > behaviors will be the same before/after this series. E.g., when client\n>> > dropped right after the command is executed, I think we will still\n>> > execute the command, though we should encounter something odd in\n>> > monitor_json_emitter() somewhere when we want to respond. And it will\n>> > happen the same after this series.\n>>\n>> I think it can get worse after your series, because you queue the\n>> commands, so clearly a new client can get replies from an old client\n>> commands. As said above, I am not convinced you can get in that\n>> situation with current code.\n>\n> Hmm, seems so. But would this a big problem?\n>\n> I really think the new client should just throw that response away if\n> it does not really know that response (from peeking at \"id\" field),\n> just like my opinion above.\n\nThis is a high expectation.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"VG7maQcj\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwhws0Rcsz9s78\n\tfor ;\n\tMon, 18 Sep 2017 20:25:49 +1000 (AEST)","from localhost ([::1]:35626 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dttFL-0003ww-3t\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 06:25:47 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:43112)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dttA9-0008GM-Bx\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:20:27 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dttA7-0005OS-4l\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:20:25 -0400","from mail-io0-x22f.google.com ([2607:f8b0:4001:c06::22f]:47790)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1dttA6-0005NT-Tm\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:20:23 -0400","by mail-io0-x22f.google.com with SMTP id e189so895507ioa.4\n\tfor ; Mon, 18 Sep 2017 03:20:21 -0700 (PDT)","by 10.58.69.69 with HTTP; Mon, 18 Sep 2017 03:20:19 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=Y64v7Ng7jr7xi88Ga+OwBSxdTr6b1ht/H6AOkiAa8fQ=;\n\tb=VG7maQcjzsmBnjoGS3YFZ/3cSmc9BOeKenaW+mvnthctfpUICrq1vAy2cvLcRd+eU7\n\tz46RSRfq80M2YtCYhVvROIUlt1Dyg4vltBmQEJreWo1dfIAlsOnAhJ1BviLNiuVpiL1C\n\tRtp0hSiO0gA+Rhbph4j3cnj1T41G1GuYCKSHQtHNyDg5Fw5CB7hkNkJCmTASxg0ikcbn\n\tLlDoInN6eqhjY1RVj8CJcLBQ+ekF9lUzhcQkTxR4vi04cd21MMDeBiLGVvGiWgDc3omb\n\tCUqYgRMs8SltzlB4/1m650mUzcFMtD5t9wm0wE9T3QJ3orkNOr/XubCv0CcqMr5UeMpp\n\tk02g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=Y64v7Ng7jr7xi88Ga+OwBSxdTr6b1ht/H6AOkiAa8fQ=;\n\tb=BHHrNsZB8L65zGvVeDauJrB6ai7EDcuJHSC9vKFrgm8ReYBkI9GPBeQMxl0Xiy3/e6\n\tb/kudwOQ/Xx7mghCYV0fbVa8wQAORobNBwuafeIT2MGarsHHOfCkEC+XC9FPGilNT29K\n\tZ8n0vb6lESvJfQZvjk60TF6D05GCuXRCRJ/a9QuyX44mywV+nJInXEYfz/CAB5TZWVI8\n\tDkbcEz+bFYDYC6fSHlFurnHWMEMHyyrBCPrSjdgz7AllZgN7hF+BqPoeak7qq7shOOed\n\thZcuzEb+SwqqnyduK83n8bci0MF2zCdxiIIVMI51GpOq/iaktgxlByIa6Wx30ubLmDz9\n\taWJA==","X-Gm-Message-State":"AHPjjUipHOYsAygj1EgC0V0dErqJGXWluLufypABuzQaLYu1ZYVVxP/v\n\tyh8wWpIAkV6Kz6wirIptKAQpVOes6X3PnGw7R0E=","X-Google-Smtp-Source":"AOwi7QBK7KzGc6sy5v0biETtMAZYhyq6Vywy2t+xG6obXGdVvFUwKHGvfl7WvOSV0QR9hzE3DOjju24zOnm8kOZVq1c=","X-Received":"by 10.202.96.131 with SMTP id\n\tu125mr15270197oib.304.1505730020660; \n\tMon, 18 Sep 2017 03:20:20 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170918083737.GD3617@pxdev.xzpeter.org>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>","From":"=?utf-8?q?Marc-Andr=C3=A9_Lureau?= ","Date":"Mon, 18 Sep 2017 12:20:19 +0200","Message-ID":"","To":"Peter Xu ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4001:c06::22f","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela , QEMU ,\n\tMarkus Armbruster ,\n\t\"Dr. David Alan Gilbert\" ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770071,"web_url":"http://patchwork.ozlabs.org/comment/1770071/","msgid":"<20170918104039.GC2581@work-vm>","list_archive_url":null,"date":"2017-09-18T10:40:40","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Peter Xu (peterx@redhat.com) wrote:\n> On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:\n> > * Stefan Hajnoczi (stefanha@redhat.com) wrote:\n> > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > > > \n> > > > > > > > > > > I agree.\n> > > > > > > > > > > \n> > > > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > > > \n> > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > > > \n> > > > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > > > next version.\n> > > > > > > > > > \n> > > > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > > > \n> > > > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > > > \n> > > > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > > > the client stays within the receive buffer size.\n> > > > > > > > > \n> > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > > > size and make sure to leave enough room.\n> > > > > > > > \n> > > > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > > > and of course it is inherantly racy.\n> > > > > > > > \n> > > > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > > > it restarts reading.\n> > > > > > > \n> > > > > > > Hmm and now we're implementing flow control!\n> > > > > > > \n> > > > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > > > waiting for the response to the previous command?\n> > > > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > > > \n> > > > > > I do wonder if we need to worry about request limiting at all from the\n> > > > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > > > \n> > > > > > OOB commands are supposed to be things which can be handled quickly\n> > > > > > without blocking, so even if a client sent several commands at once\n> > > > > > without waiting for replies, they're going to be processed quickly,\n> > > > > > so whether we temporarily block reading off the wire is a minor\n> > > > > > detail.\n> > > > > \n> > > > > Lets just define it so that it can't - you send an OOB command and wait\n> > > > > for it's response before sending another on that channel.\n> > > > > \n> > > > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > > > the app's POV.\n> > > > > \n> > > > > Can you justify 10 as opposed to 1?\n> > > > \n> > > > Semantically I don't think it makes a difference if the OOB commands are\n> > > > being processed sequentially by their thread. A >1 length queue would only\n> > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > > > requests, as then that could block reading of OOB commands. \n> > > \n> > > To summarize:\n> > > \n> > > The QMP server has a lookahead of 1 command so it can dispatch\n> > > out-of-band commands. If 2 or more non-OOB commands are queued at the\n> > > same time then OOB processing will not occur.\n> > > \n> > > Is that right?\n> > \n> > I think my view is slightly more complex;\n> > a) There's a pair of queues for each channel\n> > b) There's a central pair of queues on the QMP server\n> > one for OOB commands and one for normal commands.\n> > c) Each queue is only really guaranteed to be one deep.\n> > \n> > That means that each one of the channels can send a non-OOB\n> > command without getting in the way of a channel that wants\n> > to send one.\n> \n> But current version should not be that complex:\n> \n> Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.\n> \n> Then, we only have a single global queue for QMP non-oob commands, and\n> we don't have response queue yet. We do respond just like before in a\n> synchronous way (I explained why - for OOB we don't need that\n> complexity IMHO).\n\nI think the discussion started because of two related comments:\n Marc-André said :\n 'There should be a limit in the number of requests the thread can\nqueue'\nand Stefan said :\n 'Memory usage must be bounded.'\n\nactually neither of those cases really worried me (because they only\nhappen if the client keeps pumping commands, and that seems it's fault).\n\nHowever, once you start adding a limit, you've got to be careful - if\nyou just added a limit to the central queue, then what happens if that\nqueue is filled by non-OOB commands?\n\nDave\n\n> When we parse commands, we execute it directly if OOB, otherwise we\n> put it onto request queue. Request queue handling is done by a main\n> thread QEMUBH. That's all.\n> \n> Would this \"simple version\" suffice to implement this whole OOB idea?\n> \n> (Again, I really don't think we need to specify queue length to 1,\n> though we can make it small)\n> \n> -- \n> Peter Xu\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwjGn0wgQz9s7F\n\tfor ;\n\tMon, 18 Sep 2017 20:41:19 +1000 (AEST)","from localhost ([::1]:35708 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dttUK-0004U0-IL\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 06:41:16 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:51686)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dttU0-0004Tl-SE\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:40:58 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dttTw-0005zg-T6\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:40:56 -0400","from mx1.redhat.com ([209.132.183.28]:45206)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dttTw-0005zT-Hw\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:40:52 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 7B62F149BD;\n\tMon, 18 Sep 2017 10:40:51 +0000 (UTC)","from work-vm (ovpn-117-229.ams2.redhat.com [10.36.117.229])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 7D17D5D6A6;\n\tMon, 18 Sep 2017 10:40:42 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 7B62F149BD","Date":"Mon, 18 Sep 2017 11:40:40 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"Peter Xu ","Message-ID":"<20170918104039.GC2581@work-vm>","References":"<20170915035057.GM3617@pxdev.xzpeter.org>\n\t<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>\n\t<20170915145632.GD18767@stefanha-x1.localdomain>\n\t<20170915151706.GQ2170@work-vm>\n\t<20170918092625.GE3617@pxdev.xzpeter.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170918092625.GE3617@pxdev.xzpeter.org>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.39]);\n\tMon, 18 Sep 2017 10:40:51 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Juan Quintela ,\n\tMarkus Armbruster , QEMU ,\n\tMichael Roth , =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ,\n\tFam Zheng , \tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770076,"web_url":"http://patchwork.ozlabs.org/comment/1770076/","msgid":"<20170918105516.GD2581@work-vm>","list_archive_url":null,"date":"2017-09-18T10:55:17","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> Hi\n> \n> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n> >> Hi\n> >>\n> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> >> > Hi\n> >> >> >\n> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> >> >> > > This series was born from this one:\n> >> >> > >\n> >> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >> >> > >\n> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n> >> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n> >> >> > > ideas and suggestions. Finally we got such a solution that seems to\n> >> >> > > satisfy everyone.\n> >> >> > >\n> >> >> > > I re-started the versioning since this series is totally different\n> >> >> > > from previous one. Now it's version 1.\n> >> >> > >\n> >> >> > > In case new reviewers come along the way without reading previous\n> >> >> > > discussions, I will try to do a summary on what this is all about.\n> >> >> > >\n> >> >> > > What is OOB execution?\n> >> >> > > ======================\n> >> >> > >\n> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n> >> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n> >> >> > > QMP is going throw these steps:\n> >> >> > >\n> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> > > /|\\ (2) (3) |\n> >> >> > > (1) | \\|/ (4)\n> >> >> > > +--------- main thread --------+\n> >> >> > >\n> >> >> > > The requests are executed by the so-called QMP-dispatcher after the\n> >> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n> >> >> > > parser and quickly returns.\n> >> >> >\n> >> >> > All commands should have the \"id\" field mandatory in this case, else\n> >> >> > the client will not distinguish the replies coming from the last/oob\n> >> >> > and the previous commands.\n> >> >> >\n> >> >> > This should probably be enforced upfront by client capability checks,\n> >> >> > more below.\n> >> >\n> >> > Hmm yes since the oob commands are actually running in async way,\n> >> > request ID should be needed here. However I'm not sure whether\n> >> > enabling the whole \"request ID\" thing is too big for this \"try to be\n> >> > small\" oob change... And IMHO it suites better to be part of the whole\n> >> > async work (no matter which implementation we'll use).\n> >> >\n> >> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n> >> > For oob commands, they will always have ID then no ordering issue, and\n> >> > we can do it async; for the rest of non-oob commands, we still allow\n> >> > them to go without ID, and since they are not oob, they'll always be\n> >> > done in order as well. Would this work?\n> >>\n> >> This mixed-mode is imho more complicated to deal with than having the\n> >> protocol enforced one way or the other, but that should work.\n> >>\n> >> >\n> >> >> >\n> >> >> > > Yeah I know in current code the parser calls dispatcher directly\n> >> >> > > (please see handle_qmp_command()). However it's not true again after\n> >> >> > > this series (parser will has its own IO thread, and dispatcher will\n> >> >> > > still be run in main thread). So this OOB does brings something\n> >> >> > > different.\n> >> >> > >\n> >> >> > > There are more details on why OOB and the difference/relationship\n> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> >> >> > > slightly out of topic (and believe me, it's not easy for me to\n> >> >> > > summarize that). For more information, please refers to [1].\n> >> >> > >\n> >> >> > > Summary ends here.\n> >> >> > >\n> >> >> > > Some Implementation Details\n> >> >> > > ===========================\n> >> >> > >\n> >> >> > > Again, I mentioned that the old QMP workflow is this:\n> >> >> > >\n> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> > > /|\\ (2) (3) |\n> >> >> > > (1) | \\|/ (4)\n> >> >> > > +--------- main thread --------+\n> >> >> > >\n> >> >> > > What this series does is, firstly:\n> >> >> > >\n> >> >> > > JSON Parser QMP Dispatcher --> Respond\n> >> >> > > /|\\ | /|\\ (4) |\n> >> >> > > | | (2) | (3) | (5)\n> >> >> > > (1) | +-----> | \\|/\n> >> >> > > +--------- main thread <-------+\n> >> >> > >\n> >> >> > > And further:\n> >> >> > >\n> >> >> > > queue/kick\n> >> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n> >> >> > > /|\\ | (3) /|\\ (4) |\n> >> >> > > (1) | | (2) | | (5)\n> >> >> > > | \\|/ | \\|/\n> >> >> > > IO thread main thread <-------+\n> >> >> >\n> >> >> > Is the queue per monitor or per client?\n> >> >\n> >> > The queue is currently global. I think yes maybe at least we can do it\n> >> > per monitor, but I am not sure whether that is urgent or can be\n> >> > postponed. After all now QMPRequest (please refer to patch 11) is\n> >> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n> >> > per-monitor.\n> >> >\n> >> >> > And is the dispatching going\n> >> >> > to be processed even if the client is disconnected, and are new\n> >> >> > clients going to receive the replies from previous clients\n> >> >> > commands?\n> >> >\n> >> > [1]\n> >> >\n> >> > (will discuss together below)\n> >> >\n> >> >> > I\n> >> >> > believe there should be a per-client context, so there won't be \"id\"\n> >> >> > request conflicts.\n> >> >\n> >> > I'd say I am not familiar with this \"client\" idea, since after all\n> >> > IMHO one monitor is currently designed to mostly work with a single\n> >> > client. Say, unix sockets, telnet, all these backends are only single\n> >> > channeled, and one monitor instance can only work with one client at a\n> >> > time. Then do we really need to add this client layer upon it? IMHO\n> >> > the user can just provide more monitors if they wants more clients\n> >> > (and at least these clients should know the existance of the others or\n> >> > there might be problem, otherwise user2 will fail a migration, finally\n> >> > noticed that user1 has already triggered one), and the user should\n> >> > manage them well.\n> >>\n> >> qemu should support a management layer / libvirt restart/reconnect.\n> >> Afaik, it mostly work today. There might be a cases where libvirt can\n> >> be confused if it receives a reply from a previous connection command,\n> >> but due to the sync processing of the chardev, I am not sure you can\n> >> get in this situation. By adding \"oob\" commands and queuing, the\n> >> client will have to remember which was the last \"id\" used, or it will\n> >> create more conflict after a reconnect.\n> >>\n> >> Imho we should introduce the client/connection concept to avoid this\n> >> confusion (unexpected reply & per client id space).\n> >\n> > Hmm I agree that the reconnect feature would be nice, but if so IMHO\n> > instead of throwing responses away when client disconnect, we should\n> > really keep them, and when the client reconnects, we queue the\n> > responses again.\n> >\n> > I think we have other quite simple ways to solve the \"unexpected\n> > reply\" and \"per-client-id duplication\" issues you have mentioned.\n> >\n> > Firstly, when client gets unexpected replies (\"id\" field not in its\n> > own request queue), the client should just ignore that reply, which\n> > seems natural to me.\n> \n> The trouble is that it may legitimately use the same \"id\" value for\n> new requests. And I don't see a simple way to handle that without\n> races.\n\nUnder what circumstances can it reuse the same ID for new requests?\nCan't we simply tell it not to?\n\nDave\n\n> >\n> > Then, if client disconnected and reconnected, it should not have the\n> > problem to generate duplicated id for request, since it should know\n> > what requests it has sent already. A simplest case I can think of is,\n> > the ID should contains the following tuple:\n> \n> If you assume the \"same\" client will recover its state, yes.\n> \n> >\n> > (client name, client unique ID, request ID)\n> >\n> > Here \"client name\" can be something like \"libvirt\", which is the name\n> > of client application;\n> >\n> > \"client unique ID\" can be anything generated when client starts, it\n> > identifies a single client session, maybe a UUID.\n> >\n> > \"request ID\" can be a unsigned integer starts from zero, and increases\n> > each time the client sends one request.\n> \n> This is introducing session handling, and can be done in server side\n> only without changes in the protocol I believe.\n> \n> >\n> > I believe current libvirt is using \"client name\" + \"request ID\". It's\n> > something similar (after all I think we don't normally have >1 libvirt\n> > to manage single QEMU, so I think it should be good enough).\n> \n> I am not sure we should base our protocol usage assumptions based on\n> libvirt only, but rather on what is possible today (like queuing\n> requests in the socket etc..).\n> \n> > Then even if client disconnect and reconnect, request ID won't lose,\n> > and no duplication would happen IMHO.\n> >\n> >>\n> >> >\n> >> >> >\n> >> >> > >\n> >> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> >> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> >> >> > > parser.\n> >> >> >\n> >> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n> >> >> > command a bit pointless. It helps with legacy client that wouldn't\n> >> >> > expect out-of-order replies if qemu were to run oob commands oob by\n> >> >> > default though.\n> >> >\n> >> > After all oob somehow breaks existing rules or sync execution. I\n> >> > thought the more important goal was at least to keep the legacy\n> >> > behaviors when adding new things, no?\n> >>\n> >> Of course we have to keep compatibily. What do you mean by \"oob\n> >> somehow breaks existing rules or sync execution\"? oob means queuing\n> >> and unordered reply support, so clearly this is breaking the current\n> >> \"mostly ordered\" behaviour (mostly because events may still come any\n> >> time..., and the reconnect issue discussed above).\n> >\n> > Yes. That's what I mean, it breaks the synchronous scemantic. But\n> > I should definitely not call it a \"break\" though since old clients\n> > will work perfectly fine with it. Sorry for the bad wording.\n> >\n> >>\n> >> >> > Clients shouldn't care about how/where a command is\n> >> >> > being queued or not. If they send a command, they want it processed as\n> >> >> > quickly as possible. However, it can be interesting to know if the\n> >> >> > implementation of the command will be able to deliver oob, so that\n> >> >> > data in the introspection could be useful.\n> >> >> >\n> >> >> > I would rather propose a client/server capability in qmp_capabilities,\n> >> >> > call it \"oob\":\n> >> >> >\n> >> >> > This capability indicates oob commands support.\n> >> >>\n> >> >> The problem is indicating which commands support oob as opposed to\n> >> >> indicating whether oob is present at all. Future versions will\n> >> >> probably make more commands oob-able and a client will want to know\n> >> >> whether it can rely on a particular command being non-blocking.\n> >> >\n> >> > Yes.\n> >> >\n> >> > And IMHO we don't urgently need that \"whether the server globally\n> >> > supports oob\" thing. Client can just know that from query-qmp-schema\n> >> > already - there will always be the \"allow-oob\" new field for command\n> >> > typed entries. IMHO that's a solid hint.\n> >> >\n> >> > But I don't object to return it as well in qmp_capabilities.\n> >>\n> >> Does it feel right that the client can specify how the command are\n> >> processed / queued ? Isn't it preferable to leave that to the server\n> >> to decide? Why would a client specify that? And should the server be\n> >> expected to behave differently? What the client needs to be able is to\n> >> match the unordered replies, and that can be stated during cap\n> >> negotiation / qmp_capabilties. The server is expected to do a best\n> >> effort to handle commands and their priorities. If the client needs\n> >> several command queue, it is simpler to open several connection rather\n> >> than trying to fit that weird priority logic in the protocol imho.\n> >\n> > Sorry I may have missed the point here. We were discussing about a\n> > global hint for \"oob\" support, am I right? Then, could I ask what's\n> > the \"weird priority logic\" you mentioned?\n> \n> I call per-message oob hint a kind of priority logic, since you can\n> make the same request without oob in the same session and in parallel.\n> \n> >>\n> >> >\n> >> >>\n> >> >> > An oob command is a regular client message request with the \"id\"\n> >> >> > member mandatory, but the reply may be delivered\n> >> >> > out of order by the server if the client supports\n> >> >> > it too.\n> >> >> >\n> >> >> > If both the server and the client have the \"oob\" capability, the\n> >> >> > server can handle new client requests while previous requests are being\n> >> >> > processed.\n> >> >> >\n> >> >> > If the client doesn't have the \"oob\" capability, it may still call\n> >> >> > an oob command, and make multiple outstanding calls. In this case,\n> >> >> > the commands are processed in order, so the replies will also be in\n> >> >> > order. The \"id\" member isn't mandatory in this case.\n> >> >> >\n> >> >> > The client should match the replies with the \"id\" member associated\n> >> >> > with the requests.\n> >> >> >\n> >> >> > When a client is disconnected, the pending commands are not\n> >> >> > necessarily cancelled. But the future clients will not get replies from\n> >> >> > commands they didn't make (they might, however, receive side-effects\n> >> >> > events).\n> >> >>\n> >> >> What's the behaviour on the current monitor?\n> >> >\n> >> > Yeah I want to ask the same question, along with questioning about\n> >> > above [1].\n> >> >\n> >> > IMHO this series will not change the behaviors of these, so IMHO the\n> >> > behaviors will be the same before/after this series. E.g., when client\n> >> > dropped right after the command is executed, I think we will still\n> >> > execute the command, though we should encounter something odd in\n> >> > monitor_json_emitter() somewhere when we want to respond. And it will\n> >> > happen the same after this series.\n> >>\n> >> I think it can get worse after your series, because you queue the\n> >> commands, so clearly a new client can get replies from an old client\n> >> commands. As said above, I am not convinced you can get in that\n> >> situation with current code.\n> >\n> > Hmm, seems so. But would this a big problem?\n> >\n> > I really think the new client should just throw that response away if\n> > it does not really know that response (from peeking at \"id\" field),\n> > just like my opinion above.\n> \n> This is a high expectation.\n> \n> \n> -- \n> Marc-André Lureau\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwjbh0mSSz9s7M\n\tfor ;\n\tMon, 18 Sep 2017 20:55:58 +1000 (AEST)","from localhost ([::1]:35760 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dttiV-0008Tr-JI\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 06:55:55 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:56877)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtti6-0008TM-BI\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:55:33 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtti3-0003OR-49\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:55:30 -0400","from mx1.redhat.com ([209.132.183.28]:45596)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dtti2-0003OA-Rh\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 06:55:27 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id C64D0883BA;\n\tMon, 18 Sep 2017 10:55:25 +0000 (UTC)","from work-vm (ovpn-117-229.ams2.redhat.com [10.36.117.229])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 929F960468;\n\tMon, 18 Sep 2017 10:55:19 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com C64D0883BA","Date":"Mon, 18 Sep 2017 11:55:17 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"=?iso-8859-1?q?Marc-Andr=E9?= Lureau ","Message-ID":"<20170918105516.GD2581@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.26]);\n\tMon, 18 Sep 2017 10:55:25 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770084,"web_url":"http://patchwork.ozlabs.org/comment/1770084/","msgid":"","list_archive_url":null,"date":"2017-09-18T11:13:14","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6442,"url":"http://patchwork.ozlabs.org/api/people/6442/","name":"Marc-André Lureau","email":"marcandre.lureau@gmail.com"},"content":"Hi\n\nOn Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert\n wrote:\n> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> Hi\n>>\n>> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n>> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n>> >> Hi\n>> >>\n>> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n>> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n>> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> >> >> > Hi\n>> >> >> >\n>> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n>> >> >> > > This series was born from this one:\n>> >> >> > >\n>> >> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>> >> >> > >\n>> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n>> >> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n>> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n>> >> >> > > ideas and suggestions. Finally we got such a solution that seems to\n>> >> >> > > satisfy everyone.\n>> >> >> > >\n>> >> >> > > I re-started the versioning since this series is totally different\n>> >> >> > > from previous one. Now it's version 1.\n>> >> >> > >\n>> >> >> > > In case new reviewers come along the way without reading previous\n>> >> >> > > discussions, I will try to do a summary on what this is all about.\n>> >> >> > >\n>> >> >> > > What is OOB execution?\n>> >> >> > > ======================\n>> >> >> > >\n>> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n>> >> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n>> >> >> > > QMP is going throw these steps:\n>> >> >> > >\n>> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> >> > > /|\\ (2) (3) |\n>> >> >> > > (1) | \\|/ (4)\n>> >> >> > > +--------- main thread --------+\n>> >> >> > >\n>> >> >> > > The requests are executed by the so-called QMP-dispatcher after the\n>> >> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n>> >> >> > > parser and quickly returns.\n>> >> >> >\n>> >> >> > All commands should have the \"id\" field mandatory in this case, else\n>> >> >> > the client will not distinguish the replies coming from the last/oob\n>> >> >> > and the previous commands.\n>> >> >> >\n>> >> >> > This should probably be enforced upfront by client capability checks,\n>> >> >> > more below.\n>> >> >\n>> >> > Hmm yes since the oob commands are actually running in async way,\n>> >> > request ID should be needed here. However I'm not sure whether\n>> >> > enabling the whole \"request ID\" thing is too big for this \"try to be\n>> >> > small\" oob change... And IMHO it suites better to be part of the whole\n>> >> > async work (no matter which implementation we'll use).\n>> >> >\n>> >> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n>> >> > For oob commands, they will always have ID then no ordering issue, and\n>> >> > we can do it async; for the rest of non-oob commands, we still allow\n>> >> > them to go without ID, and since they are not oob, they'll always be\n>> >> > done in order as well. Would this work?\n>> >>\n>> >> This mixed-mode is imho more complicated to deal with than having the\n>> >> protocol enforced one way or the other, but that should work.\n>> >>\n>> >> >\n>> >> >> >\n>> >> >> > > Yeah I know in current code the parser calls dispatcher directly\n>> >> >> > > (please see handle_qmp_command()). However it's not true again after\n>> >> >> > > this series (parser will has its own IO thread, and dispatcher will\n>> >> >> > > still be run in main thread). So this OOB does brings something\n>> >> >> > > different.\n>> >> >> > >\n>> >> >> > > There are more details on why OOB and the difference/relationship\n>> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n>> >> >> > > slightly out of topic (and believe me, it's not easy for me to\n>> >> >> > > summarize that). For more information, please refers to [1].\n>> >> >> > >\n>> >> >> > > Summary ends here.\n>> >> >> > >\n>> >> >> > > Some Implementation Details\n>> >> >> > > ===========================\n>> >> >> > >\n>> >> >> > > Again, I mentioned that the old QMP workflow is this:\n>> >> >> > >\n>> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> >> > > /|\\ (2) (3) |\n>> >> >> > > (1) | \\|/ (4)\n>> >> >> > > +--------- main thread --------+\n>> >> >> > >\n>> >> >> > > What this series does is, firstly:\n>> >> >> > >\n>> >> >> > > JSON Parser QMP Dispatcher --> Respond\n>> >> >> > > /|\\ | /|\\ (4) |\n>> >> >> > > | | (2) | (3) | (5)\n>> >> >> > > (1) | +-----> | \\|/\n>> >> >> > > +--------- main thread <-------+\n>> >> >> > >\n>> >> >> > > And further:\n>> >> >> > >\n>> >> >> > > queue/kick\n>> >> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n>> >> >> > > /|\\ | (3) /|\\ (4) |\n>> >> >> > > (1) | | (2) | | (5)\n>> >> >> > > | \\|/ | \\|/\n>> >> >> > > IO thread main thread <-------+\n>> >> >> >\n>> >> >> > Is the queue per monitor or per client?\n>> >> >\n>> >> > The queue is currently global. I think yes maybe at least we can do it\n>> >> > per monitor, but I am not sure whether that is urgent or can be\n>> >> > postponed. After all now QMPRequest (please refer to patch 11) is\n>> >> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n>> >> > per-monitor.\n>> >> >\n>> >> >> > And is the dispatching going\n>> >> >> > to be processed even if the client is disconnected, and are new\n>> >> >> > clients going to receive the replies from previous clients\n>> >> >> > commands?\n>> >> >\n>> >> > [1]\n>> >> >\n>> >> > (will discuss together below)\n>> >> >\n>> >> >> > I\n>> >> >> > believe there should be a per-client context, so there won't be \"id\"\n>> >> >> > request conflicts.\n>> >> >\n>> >> > I'd say I am not familiar with this \"client\" idea, since after all\n>> >> > IMHO one monitor is currently designed to mostly work with a single\n>> >> > client. Say, unix sockets, telnet, all these backends are only single\n>> >> > channeled, and one monitor instance can only work with one client at a\n>> >> > time. Then do we really need to add this client layer upon it? IMHO\n>> >> > the user can just provide more monitors if they wants more clients\n>> >> > (and at least these clients should know the existance of the others or\n>> >> > there might be problem, otherwise user2 will fail a migration, finally\n>> >> > noticed that user1 has already triggered one), and the user should\n>> >> > manage them well.\n>> >>\n>> >> qemu should support a management layer / libvirt restart/reconnect.\n>> >> Afaik, it mostly work today. There might be a cases where libvirt can\n>> >> be confused if it receives a reply from a previous connection command,\n>> >> but due to the sync processing of the chardev, I am not sure you can\n>> >> get in this situation. By adding \"oob\" commands and queuing, the\n>> >> client will have to remember which was the last \"id\" used, or it will\n>> >> create more conflict after a reconnect.\n>> >>\n>> >> Imho we should introduce the client/connection concept to avoid this\n>> >> confusion (unexpected reply & per client id space).\n>> >\n>> > Hmm I agree that the reconnect feature would be nice, but if so IMHO\n>> > instead of throwing responses away when client disconnect, we should\n>> > really keep them, and when the client reconnects, we queue the\n>> > responses again.\n>> >\n>> > I think we have other quite simple ways to solve the \"unexpected\n>> > reply\" and \"per-client-id duplication\" issues you have mentioned.\n>> >\n>> > Firstly, when client gets unexpected replies (\"id\" field not in its\n>> > own request queue), the client should just ignore that reply, which\n>> > seems natural to me.\n>>\n>> The trouble is that it may legitimately use the same \"id\" value for\n>> new requests. And I don't see a simple way to handle that without\n>> races.\n>\n> Under what circumstances can it reuse the same ID for new requests?\n> Can't we simply tell it not to?\n\nI don't see any restriction today in the protocol in connecting with a\nnew client that may not know anything from a previous client.\n\nHow would you tell it not to use old IDs? Just by writing an unwritten\nrule, because we don't want to fix the per connection client session\nhandling in qemu?\n\n>\n> Dave\n>\n>> >\n>> > Then, if client disconnected and reconnected, it should not have the\n>> > problem to generate duplicated id for request, since it should know\n>> > what requests it has sent already. A simplest case I can think of is,\n>> > the ID should contains the following tuple:\n>>\n>> If you assume the \"same\" client will recover its state, yes.\n>>\n>> >\n>> > (client name, client unique ID, request ID)\n>> >\n>> > Here \"client name\" can be something like \"libvirt\", which is the name\n>> > of client application;\n>> >\n>> > \"client unique ID\" can be anything generated when client starts, it\n>> > identifies a single client session, maybe a UUID.\n>> >\n>> > \"request ID\" can be a unsigned integer starts from zero, and increases\n>> > each time the client sends one request.\n>>\n>> This is introducing session handling, and can be done in server side\n>> only without changes in the protocol I believe.\n>>\n>> >\n>> > I believe current libvirt is using \"client name\" + \"request ID\". It's\n>> > something similar (after all I think we don't normally have >1 libvirt\n>> > to manage single QEMU, so I think it should be good enough).\n>>\n>> I am not sure we should base our protocol usage assumptions based on\n>> libvirt only, but rather on what is possible today (like queuing\n>> requests in the socket etc..).\n>>\n>> > Then even if client disconnect and reconnect, request ID won't lose,\n>> > and no duplication would happen IMHO.\n>> >\n>> >>\n>> >> >\n>> >> >> >\n>> >> >> > >\n>> >> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n>> >> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n>> >> >> > > parser.\n>> >> >> >\n>> >> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n>> >> >> > command a bit pointless. It helps with legacy client that wouldn't\n>> >> >> > expect out-of-order replies if qemu were to run oob commands oob by\n>> >> >> > default though.\n>> >> >\n>> >> > After all oob somehow breaks existing rules or sync execution. I\n>> >> > thought the more important goal was at least to keep the legacy\n>> >> > behaviors when adding new things, no?\n>> >>\n>> >> Of course we have to keep compatibily. What do you mean by \"oob\n>> >> somehow breaks existing rules or sync execution\"? oob means queuing\n>> >> and unordered reply support, so clearly this is breaking the current\n>> >> \"mostly ordered\" behaviour (mostly because events may still come any\n>> >> time..., and the reconnect issue discussed above).\n>> >\n>> > Yes. That's what I mean, it breaks the synchronous scemantic. But\n>> > I should definitely not call it a \"break\" though since old clients\n>> > will work perfectly fine with it. Sorry for the bad wording.\n>> >\n>> >>\n>> >> >> > Clients shouldn't care about how/where a command is\n>> >> >> > being queued or not. If they send a command, they want it processed as\n>> >> >> > quickly as possible. However, it can be interesting to know if the\n>> >> >> > implementation of the command will be able to deliver oob, so that\n>> >> >> > data in the introspection could be useful.\n>> >> >> >\n>> >> >> > I would rather propose a client/server capability in qmp_capabilities,\n>> >> >> > call it \"oob\":\n>> >> >> >\n>> >> >> > This capability indicates oob commands support.\n>> >> >>\n>> >> >> The problem is indicating which commands support oob as opposed to\n>> >> >> indicating whether oob is present at all. Future versions will\n>> >> >> probably make more commands oob-able and a client will want to know\n>> >> >> whether it can rely on a particular command being non-blocking.\n>> >> >\n>> >> > Yes.\n>> >> >\n>> >> > And IMHO we don't urgently need that \"whether the server globally\n>> >> > supports oob\" thing. Client can just know that from query-qmp-schema\n>> >> > already - there will always be the \"allow-oob\" new field for command\n>> >> > typed entries. IMHO that's a solid hint.\n>> >> >\n>> >> > But I don't object to return it as well in qmp_capabilities.\n>> >>\n>> >> Does it feel right that the client can specify how the command are\n>> >> processed / queued ? Isn't it preferable to leave that to the server\n>> >> to decide? Why would a client specify that? And should the server be\n>> >> expected to behave differently? What the client needs to be able is to\n>> >> match the unordered replies, and that can be stated during cap\n>> >> negotiation / qmp_capabilties. The server is expected to do a best\n>> >> effort to handle commands and their priorities. If the client needs\n>> >> several command queue, it is simpler to open several connection rather\n>> >> than trying to fit that weird priority logic in the protocol imho.\n>> >\n>> > Sorry I may have missed the point here. We were discussing about a\n>> > global hint for \"oob\" support, am I right? Then, could I ask what's\n>> > the \"weird priority logic\" you mentioned?\n>>\n>> I call per-message oob hint a kind of priority logic, since you can\n>> make the same request without oob in the same session and in parallel.\n>>\n>> >>\n>> >> >\n>> >> >>\n>> >> >> > An oob command is a regular client message request with the \"id\"\n>> >> >> > member mandatory, but the reply may be delivered\n>> >> >> > out of order by the server if the client supports\n>> >> >> > it too.\n>> >> >> >\n>> >> >> > If both the server and the client have the \"oob\" capability, the\n>> >> >> > server can handle new client requests while previous requests are being\n>> >> >> > processed.\n>> >> >> >\n>> >> >> > If the client doesn't have the \"oob\" capability, it may still call\n>> >> >> > an oob command, and make multiple outstanding calls. In this case,\n>> >> >> > the commands are processed in order, so the replies will also be in\n>> >> >> > order. The \"id\" member isn't mandatory in this case.\n>> >> >> >\n>> >> >> > The client should match the replies with the \"id\" member associated\n>> >> >> > with the requests.\n>> >> >> >\n>> >> >> > When a client is disconnected, the pending commands are not\n>> >> >> > necessarily cancelled. But the future clients will not get replies from\n>> >> >> > commands they didn't make (they might, however, receive side-effects\n>> >> >> > events).\n>> >> >>\n>> >> >> What's the behaviour on the current monitor?\n>> >> >\n>> >> > Yeah I want to ask the same question, along with questioning about\n>> >> > above [1].\n>> >> >\n>> >> > IMHO this series will not change the behaviors of these, so IMHO the\n>> >> > behaviors will be the same before/after this series. E.g., when client\n>> >> > dropped right after the command is executed, I think we will still\n>> >> > execute the command, though we should encounter something odd in\n>> >> > monitor_json_emitter() somewhere when we want to respond. And it will\n>> >> > happen the same after this series.\n>> >>\n>> >> I think it can get worse after your series, because you queue the\n>> >> commands, so clearly a new client can get replies from an old client\n>> >> commands. As said above, I am not convinced you can get in that\n>> >> situation with current code.\n>> >\n>> > Hmm, seems so. But would this a big problem?\n>> >\n>> > I really think the new client should just throw that response away if\n>> > it does not really know that response (from peeking at \"id\" field),\n>> > just like my opinion above.\n>>\n>> This is a high expectation.\n>>\n>>\n>> --\n>> Marc-André Lureau\n> --\n> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"f0Ejq3K1\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwk0G1zfWz9s7G\n\tfor ;\n\tMon, 18 Sep 2017 21:13:48 +1000 (AEST)","from localhost ([::1]:35795 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dttzl-0003i3-Jx\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 07:13:45 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:34079)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dttzM-0003hh-Go\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:13:23 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dttzH-0002nP-W1\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:13:20 -0400","from mail-io0-x235.google.com ([2607:f8b0:4001:c06::235]:55425)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1dttzH-0002n8-Oc\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:13:15 -0400","by mail-io0-x235.google.com with SMTP id z187so1131880ioz.12\n\tfor ; Mon, 18 Sep 2017 04:13:15 -0700 (PDT)","by 10.58.69.69 with HTTP; Mon, 18 Sep 2017 04:13:14 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=QeBuNrwsCtzHrwY1CBKNWcyJdZaKbVngc3RAQ2ztK9E=;\n\tb=f0Ejq3K1+NqXJgN/QwqrXTjxfaFuI4lU9TPf6Xt2V6SC+jXC98eP0vBDdX65xOtlso\n\tC1/7pOsAZCr2Fwo1geiEBie6JUnpLx4Bgbj5id2ynbkFW3murtzVWl2pM5uJ+cpiNMP+\n\tJHe5fJwRSSyOHt/COUnXy6VHevBW/jKsW+3UFuGrmMkcKeNH6i+E7NqThKR/grAbxrsq\n\tYU94acpFxdjXxGXsfsF8bC3IZkyhpxr9nfIIiDfz+xqjCHvpaeYM5dJYAonlnrzeFLRR\n\tI//THDDqAyD0QQN845/LSZ58g+dubvLy6O/NXEA2XVKiPyymEsQfiNH4ViKvuvkd9IXV\n\tGCHA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=QeBuNrwsCtzHrwY1CBKNWcyJdZaKbVngc3RAQ2ztK9E=;\n\tb=G18FT+z4uWwmwRBpPpIqD3mC8BFBlRM34HKEd9FkxVr6lQv+nWZwWE8MGQGDVhW83d\n\t11fSuE3nQl9O38NXgFaMM/1VtVL7a3aYcpdLQn6L/pJ3t1u1ZBZjkrtaF6lRthkFKXcS\n\t3KhKyLLRCtQ8U9pBr+twV3LtXEyvLbAeUFIVr1ZCMfLWayrc3UKey9MuyJAxiJnShKnw\n\tYiCmnZf5fB6r6eof0+XZ3ffktwnuCTMkQbEJuQnlHZaRIqu/h1FdV/75BEU/PCoFQEeo\n\tfJtXdMRb4vIII27oDnC2cQfZVzqjTaTewfqOBaFfp2cVclfG6Qn2vlBlbUqvYAHqobIi\n\tNQCA==","X-Gm-Message-State":"AHPjjUhzkC9CdXNsyzfoPkgF6R3EwdHbmeGIAw3CMX1H5Ohpf7Nqoh2l\n\trHb1mmdLn4wdGh7qhOZljZlsIM+v3jKqqgMPbjc=","X-Google-Smtp-Source":"AOwi7QC7/f99WvQwXP8ytnRnLxbk53eNt9n2+oVY0KdLdeLA4Gg/7iTIUZ9TvJa6t3IoCn/RrYEalthOgzwNVu0FYrc=","X-Received":"by 10.202.84.15 with SMTP id i15mr8586759oib.255.1505733194690; \n\tMon, 18 Sep 2017 04:13:14 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170918105516.GD2581@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t\n\t<20170918105516.GD2581@work-vm>","From":"=?utf-8?q?Marc-Andr=C3=A9_Lureau?= ","Date":"Mon, 18 Sep 2017 13:13:14 +0200","Message-ID":"","To":"\"Dr. David Alan Gilbert\" ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4001:c06::235","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770089,"web_url":"http://patchwork.ozlabs.org/comment/1770089/","msgid":"<20170918112618.GF2581@work-vm>","list_archive_url":null,"date":"2017-09-18T11:26:19","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> Hi\n> \n> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert\n> wrote:\n> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> Hi\n> >>\n> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n> >> >> Hi\n> >> >>\n> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> >> >> > Hi\n> >> >> >> >\n> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> >> >> >> > > This series was born from this one:\n> >> >> >> > >\n> >> >> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >> >> >> > >\n> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n> >> >> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n> >> >> >> > > ideas and suggestions. Finally we got such a solution that seems to\n> >> >> >> > > satisfy everyone.\n> >> >> >> > >\n> >> >> >> > > I re-started the versioning since this series is totally different\n> >> >> >> > > from previous one. Now it's version 1.\n> >> >> >> > >\n> >> >> >> > > In case new reviewers come along the way without reading previous\n> >> >> >> > > discussions, I will try to do a summary on what this is all about.\n> >> >> >> > >\n> >> >> >> > > What is OOB execution?\n> >> >> >> > > ======================\n> >> >> >> > >\n> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n> >> >> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n> >> >> >> > > QMP is going throw these steps:\n> >> >> >> > >\n> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> >> > > /|\\ (2) (3) |\n> >> >> >> > > (1) | \\|/ (4)\n> >> >> >> > > +--------- main thread --------+\n> >> >> >> > >\n> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the\n> >> >> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n> >> >> >> > > parser and quickly returns.\n> >> >> >> >\n> >> >> >> > All commands should have the \"id\" field mandatory in this case, else\n> >> >> >> > the client will not distinguish the replies coming from the last/oob\n> >> >> >> > and the previous commands.\n> >> >> >> >\n> >> >> >> > This should probably be enforced upfront by client capability checks,\n> >> >> >> > more below.\n> >> >> >\n> >> >> > Hmm yes since the oob commands are actually running in async way,\n> >> >> > request ID should be needed here. However I'm not sure whether\n> >> >> > enabling the whole \"request ID\" thing is too big for this \"try to be\n> >> >> > small\" oob change... And IMHO it suites better to be part of the whole\n> >> >> > async work (no matter which implementation we'll use).\n> >> >> >\n> >> >> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n> >> >> > For oob commands, they will always have ID then no ordering issue, and\n> >> >> > we can do it async; for the rest of non-oob commands, we still allow\n> >> >> > them to go without ID, and since they are not oob, they'll always be\n> >> >> > done in order as well. Would this work?\n> >> >>\n> >> >> This mixed-mode is imho more complicated to deal with than having the\n> >> >> protocol enforced one way or the other, but that should work.\n> >> >>\n> >> >> >\n> >> >> >> >\n> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly\n> >> >> >> > > (please see handle_qmp_command()). However it's not true again after\n> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will\n> >> >> >> > > still be run in main thread). So this OOB does brings something\n> >> >> >> > > different.\n> >> >> >> > >\n> >> >> >> > > There are more details on why OOB and the difference/relationship\n> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to\n> >> >> >> > > summarize that). For more information, please refers to [1].\n> >> >> >> > >\n> >> >> >> > > Summary ends here.\n> >> >> >> > >\n> >> >> >> > > Some Implementation Details\n> >> >> >> > > ===========================\n> >> >> >> > >\n> >> >> >> > > Again, I mentioned that the old QMP workflow is this:\n> >> >> >> > >\n> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> >> > > /|\\ (2) (3) |\n> >> >> >> > > (1) | \\|/ (4)\n> >> >> >> > > +--------- main thread --------+\n> >> >> >> > >\n> >> >> >> > > What this series does is, firstly:\n> >> >> >> > >\n> >> >> >> > > JSON Parser QMP Dispatcher --> Respond\n> >> >> >> > > /|\\ | /|\\ (4) |\n> >> >> >> > > | | (2) | (3) | (5)\n> >> >> >> > > (1) | +-----> | \\|/\n> >> >> >> > > +--------- main thread <-------+\n> >> >> >> > >\n> >> >> >> > > And further:\n> >> >> >> > >\n> >> >> >> > > queue/kick\n> >> >> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n> >> >> >> > > /|\\ | (3) /|\\ (4) |\n> >> >> >> > > (1) | | (2) | | (5)\n> >> >> >> > > | \\|/ | \\|/\n> >> >> >> > > IO thread main thread <-------+\n> >> >> >> >\n> >> >> >> > Is the queue per monitor or per client?\n> >> >> >\n> >> >> > The queue is currently global. I think yes maybe at least we can do it\n> >> >> > per monitor, but I am not sure whether that is urgent or can be\n> >> >> > postponed. After all now QMPRequest (please refer to patch 11) is\n> >> >> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n> >> >> > per-monitor.\n> >> >> >\n> >> >> >> > And is the dispatching going\n> >> >> >> > to be processed even if the client is disconnected, and are new\n> >> >> >> > clients going to receive the replies from previous clients\n> >> >> >> > commands?\n> >> >> >\n> >> >> > [1]\n> >> >> >\n> >> >> > (will discuss together below)\n> >> >> >\n> >> >> >> > I\n> >> >> >> > believe there should be a per-client context, so there won't be \"id\"\n> >> >> >> > request conflicts.\n> >> >> >\n> >> >> > I'd say I am not familiar with this \"client\" idea, since after all\n> >> >> > IMHO one monitor is currently designed to mostly work with a single\n> >> >> > client. Say, unix sockets, telnet, all these backends are only single\n> >> >> > channeled, and one monitor instance can only work with one client at a\n> >> >> > time. Then do we really need to add this client layer upon it? IMHO\n> >> >> > the user can just provide more monitors if they wants more clients\n> >> >> > (and at least these clients should know the existance of the others or\n> >> >> > there might be problem, otherwise user2 will fail a migration, finally\n> >> >> > noticed that user1 has already triggered one), and the user should\n> >> >> > manage them well.\n> >> >>\n> >> >> qemu should support a management layer / libvirt restart/reconnect.\n> >> >> Afaik, it mostly work today. There might be a cases where libvirt can\n> >> >> be confused if it receives a reply from a previous connection command,\n> >> >> but due to the sync processing of the chardev, I am not sure you can\n> >> >> get in this situation. By adding \"oob\" commands and queuing, the\n> >> >> client will have to remember which was the last \"id\" used, or it will\n> >> >> create more conflict after a reconnect.\n> >> >>\n> >> >> Imho we should introduce the client/connection concept to avoid this\n> >> >> confusion (unexpected reply & per client id space).\n> >> >\n> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO\n> >> > instead of throwing responses away when client disconnect, we should\n> >> > really keep them, and when the client reconnects, we queue the\n> >> > responses again.\n> >> >\n> >> > I think we have other quite simple ways to solve the \"unexpected\n> >> > reply\" and \"per-client-id duplication\" issues you have mentioned.\n> >> >\n> >> > Firstly, when client gets unexpected replies (\"id\" field not in its\n> >> > own request queue), the client should just ignore that reply, which\n> >> > seems natural to me.\n> >>\n> >> The trouble is that it may legitimately use the same \"id\" value for\n> >> new requests. And I don't see a simple way to handle that without\n> >> races.\n> >\n> > Under what circumstances can it reuse the same ID for new requests?\n> > Can't we simply tell it not to?\n> \n> I don't see any restriction today in the protocol in connecting with a\n> new client that may not know anything from a previous client.\n\nWell, it knows it's doing a reconnection.\n\n> How would you tell it not to use old IDs? Just by writing an unwritten\n> rule, because we don't want to fix the per connection client session\n> handling in qemu?\n\nBY writing a written rule! This out of order stuff we're adding here\nis a change to the interface and we can define what we require of the\nclient. As long as what we expect is reasonable then we might end\nup with something that's simpler for both the client and qemu.\nAnd I worry this series keeps getting more and more complex for weird\nedge cases.\n\nDave\n\n> >\n> > Dave\n> >\n> >> >\n> >> > Then, if client disconnected and reconnected, it should not have the\n> >> > problem to generate duplicated id for request, since it should know\n> >> > what requests it has sent already. A simplest case I can think of is,\n> >> > the ID should contains the following tuple:\n> >>\n> >> If you assume the \"same\" client will recover its state, yes.\n> >>\n> >> >\n> >> > (client name, client unique ID, request ID)\n> >> >\n> >> > Here \"client name\" can be something like \"libvirt\", which is the name\n> >> > of client application;\n> >> >\n> >> > \"client unique ID\" can be anything generated when client starts, it\n> >> > identifies a single client session, maybe a UUID.\n> >> >\n> >> > \"request ID\" can be a unsigned integer starts from zero, and increases\n> >> > each time the client sends one request.\n> >>\n> >> This is introducing session handling, and can be done in server side\n> >> only without changes in the protocol I believe.\n> >>\n> >> >\n> >> > I believe current libvirt is using \"client name\" + \"request ID\". It's\n> >> > something similar (after all I think we don't normally have >1 libvirt\n> >> > to manage single QEMU, so I think it should be good enough).\n> >>\n> >> I am not sure we should base our protocol usage assumptions based on\n> >> libvirt only, but rather on what is possible today (like queuing\n> >> requests in the socket etc..).\n> >>\n> >> > Then even if client disconnect and reconnect, request ID won't lose,\n> >> > and no duplication would happen IMHO.\n> >> >\n> >> >>\n> >> >> >\n> >> >> >> >\n> >> >> >> > >\n> >> >> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n> >> >> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n> >> >> >> > > parser.\n> >> >> >> >\n> >> >> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n> >> >> >> > command a bit pointless. It helps with legacy client that wouldn't\n> >> >> >> > expect out-of-order replies if qemu were to run oob commands oob by\n> >> >> >> > default though.\n> >> >> >\n> >> >> > After all oob somehow breaks existing rules or sync execution. I\n> >> >> > thought the more important goal was at least to keep the legacy\n> >> >> > behaviors when adding new things, no?\n> >> >>\n> >> >> Of course we have to keep compatibily. What do you mean by \"oob\n> >> >> somehow breaks existing rules or sync execution\"? oob means queuing\n> >> >> and unordered reply support, so clearly this is breaking the current\n> >> >> \"mostly ordered\" behaviour (mostly because events may still come any\n> >> >> time..., and the reconnect issue discussed above).\n> >> >\n> >> > Yes. That's what I mean, it breaks the synchronous scemantic. But\n> >> > I should definitely not call it a \"break\" though since old clients\n> >> > will work perfectly fine with it. Sorry for the bad wording.\n> >> >\n> >> >>\n> >> >> >> > Clients shouldn't care about how/where a command is\n> >> >> >> > being queued or not. If they send a command, they want it processed as\n> >> >> >> > quickly as possible. However, it can be interesting to know if the\n> >> >> >> > implementation of the command will be able to deliver oob, so that\n> >> >> >> > data in the introspection could be useful.\n> >> >> >> >\n> >> >> >> > I would rather propose a client/server capability in qmp_capabilities,\n> >> >> >> > call it \"oob\":\n> >> >> >> >\n> >> >> >> > This capability indicates oob commands support.\n> >> >> >>\n> >> >> >> The problem is indicating which commands support oob as opposed to\n> >> >> >> indicating whether oob is present at all. Future versions will\n> >> >> >> probably make more commands oob-able and a client will want to know\n> >> >> >> whether it can rely on a particular command being non-blocking.\n> >> >> >\n> >> >> > Yes.\n> >> >> >\n> >> >> > And IMHO we don't urgently need that \"whether the server globally\n> >> >> > supports oob\" thing. Client can just know that from query-qmp-schema\n> >> >> > already - there will always be the \"allow-oob\" new field for command\n> >> >> > typed entries. IMHO that's a solid hint.\n> >> >> >\n> >> >> > But I don't object to return it as well in qmp_capabilities.\n> >> >>\n> >> >> Does it feel right that the client can specify how the command are\n> >> >> processed / queued ? Isn't it preferable to leave that to the server\n> >> >> to decide? Why would a client specify that? And should the server be\n> >> >> expected to behave differently? What the client needs to be able is to\n> >> >> match the unordered replies, and that can be stated during cap\n> >> >> negotiation / qmp_capabilties. The server is expected to do a best\n> >> >> effort to handle commands and their priorities. If the client needs\n> >> >> several command queue, it is simpler to open several connection rather\n> >> >> than trying to fit that weird priority logic in the protocol imho.\n> >> >\n> >> > Sorry I may have missed the point here. We were discussing about a\n> >> > global hint for \"oob\" support, am I right? Then, could I ask what's\n> >> > the \"weird priority logic\" you mentioned?\n> >>\n> >> I call per-message oob hint a kind of priority logic, since you can\n> >> make the same request without oob in the same session and in parallel.\n> >>\n> >> >>\n> >> >> >\n> >> >> >>\n> >> >> >> > An oob command is a regular client message request with the \"id\"\n> >> >> >> > member mandatory, but the reply may be delivered\n> >> >> >> > out of order by the server if the client supports\n> >> >> >> > it too.\n> >> >> >> >\n> >> >> >> > If both the server and the client have the \"oob\" capability, the\n> >> >> >> > server can handle new client requests while previous requests are being\n> >> >> >> > processed.\n> >> >> >> >\n> >> >> >> > If the client doesn't have the \"oob\" capability, it may still call\n> >> >> >> > an oob command, and make multiple outstanding calls. In this case,\n> >> >> >> > the commands are processed in order, so the replies will also be in\n> >> >> >> > order. The \"id\" member isn't mandatory in this case.\n> >> >> >> >\n> >> >> >> > The client should match the replies with the \"id\" member associated\n> >> >> >> > with the requests.\n> >> >> >> >\n> >> >> >> > When a client is disconnected, the pending commands are not\n> >> >> >> > necessarily cancelled. But the future clients will not get replies from\n> >> >> >> > commands they didn't make (they might, however, receive side-effects\n> >> >> >> > events).\n> >> >> >>\n> >> >> >> What's the behaviour on the current monitor?\n> >> >> >\n> >> >> > Yeah I want to ask the same question, along with questioning about\n> >> >> > above [1].\n> >> >> >\n> >> >> > IMHO this series will not change the behaviors of these, so IMHO the\n> >> >> > behaviors will be the same before/after this series. E.g., when client\n> >> >> > dropped right after the command is executed, I think we will still\n> >> >> > execute the command, though we should encounter something odd in\n> >> >> > monitor_json_emitter() somewhere when we want to respond. And it will\n> >> >> > happen the same after this series.\n> >> >>\n> >> >> I think it can get worse after your series, because you queue the\n> >> >> commands, so clearly a new client can get replies from an old client\n> >> >> commands. As said above, I am not convinced you can get in that\n> >> >> situation with current code.\n> >> >\n> >> > Hmm, seems so. But would this a big problem?\n> >> >\n> >> > I really think the new client should just throw that response away if\n> >> > it does not really know that response (from peeking at \"id\" field),\n> >> > just like my opinion above.\n> >>\n> >> This is a high expectation.\n> >>\n> >>\n> >> --\n> >> Marc-André Lureau\n> > --\n> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK\n> \n> \n> \n> -- \n> Marc-André Lureau\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwkHZ3k5mz9s7G\n\tfor ;\n\tMon, 18 Sep 2017 21:27:05 +1000 (AEST)","from localhost ([::1]:35832 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dtuCc-0007tP-Qk\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 07:27:02 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:37486)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtuCB-0007t6-QJ\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:26:38 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtuC7-0007jP-Ls\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:26:35 -0400","from mx1.redhat.com ([209.132.183.28]:40002)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dtuC7-0007iw-Bg\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 07:26:31 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id DD7DA5F7AC;\n\tMon, 18 Sep 2017 11:26:29 +0000 (UTC)","from work-vm (ovpn-117-229.ams2.redhat.com [10.36.117.229])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id BDFFA1851B;\n\tMon, 18 Sep 2017 11:26:21 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com DD7DA5F7AC","Date":"Mon, 18 Sep 2017 12:26:19 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"=?iso-8859-1?q?Marc-Andr=E9?= Lureau ","Message-ID":"<20170918112618.GF2581@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t\n\t<20170918105516.GD2581@work-vm>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.39]);\n\tMon, 18 Sep 2017 11:26:30 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770236,"web_url":"http://patchwork.ozlabs.org/comment/1770236/","msgid":"<5e0abb57-0c5b-50b1-38ea-6b699bc21205@redhat.com>","list_archive_url":null,"date":"2017-09-18T15:08:18","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6591,"url":"http://patchwork.ozlabs.org/api/people/6591/","name":"Eric Blake","email":"eblake@redhat.com"},"content":"On 09/18/2017 05:55 AM, Dr. David Alan Gilbert wrote:\n\n>>> I think we have other quite simple ways to solve the \"unexpected\n>>> reply\" and \"per-client-id duplication\" issues you have mentioned.\n>>>\n>>> Firstly, when client gets unexpected replies (\"id\" field not in its\n>>> own request queue), the client should just ignore that reply, which\n>>> seems natural to me.\n\nThat's probably reasonable, if we document it.\n\n>>\n>> The trouble is that it may legitimately use the same \"id\" value for\n>> new requests. And I don't see a simple way to handle that without\n>> races.\n> \n> Under what circumstances can it reuse the same ID for new requests?\n\nLibvirt uses distinct id's for every message on a single connection, but\nthere is always the possibility that it will use id 'libvirt-0' on one\nconnection, then restart libvirtd, then use id 'libvirt-0' on the second\nconnection (there's nothing that I saw in libvirt code that saves the\ncurrent 'mon->nextSerial' value in XML to survive libvirtd restarts).\n\n> Can't we simply tell it not to?\n\nSince use of OOB handling will require client opt-in, yes, we can make\npart of the opt-in process be a contract that the client has to do a\nbetter job of avoiding duplicate id's across reconnects, if we think\nthat is easier to maintain.\n\n>>>\n>>> Then, if client disconnected and reconnected, it should not have the\n>>> problem to generate duplicated id for request, since it should know\n>>> what requests it has sent already. A simplest case I can think of is,\n>>> the ID should contains the following tuple:\n>>\n>> If you assume the \"same\" client will recover its state, yes.\n>>\n>>>\n>>> (client name, client unique ID, request ID)\n>>>\n>>> Here \"client name\" can be something like \"libvirt\", which is the name\n>>> of client application;\n>>>\n>>> \"client unique ID\" can be anything generated when client starts, it\n>>> identifies a single client session, maybe a UUID.\n>>>\n>>> \"request ID\" can be a unsigned integer starts from zero, and increases\n>>> each time the client sends one request.\n>>\n>> This is introducing session handling, and can be done in server side\n>> only without changes in the protocol I believe.\n\nThe 'id' field can be _any_ JSON object - libvirt currently sends a\nstring, but could just as easily send a dict, and then libvirt could\nsupply whatever it wanted in the dict, including uuids, to ensure that\nfuture reconnects don't reuse the id of a previous connection. But\nright now, 'id' is opaque to qemu (and should stay that way) - if qemu\nis going to do any work to ensure that it doesn't send a stale reply to\na new connection, then that has to be tracked externally from whatever\n'id' is passed in; or we just document that clients wanting to use OOB\nhandling have to be careful of their choice of 'id' (and leave it up to\nthe client to avoid confusion, because qemu doesn't care about it).\n\nI'm also fine with requiring that clients that opt-in to OOB handling be\ndocumented as having to ignore unknown 'id' responses, since we already\ndocument that clients must be able to ignore unknown 'event' messages.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx08.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=eblake@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xwqCq5gL8z9s4s\n\tfor ;\n\tTue, 19 Sep 2017 01:09:11 +1000 (AEST)","from localhost ([::1]:37154 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dtxfX-0005f7-N6\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 11:09:09 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:40396)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtxes-0005bZ-Sc\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 11:08:33 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtxer-0005uE-Ea\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 11:08:26 -0400","from mx1.redhat.com ([209.132.183.28]:51194)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dtxer-0005t6-5j\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 11:08:25 -0400","from smtp.corp.redhat.com\n\t(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 50BB3C0587D9;\n\tMon, 18 Sep 2017 15:08:24 +0000 (UTC)","from [10.10.124.97] (ovpn-124-97.rdu2.redhat.com [10.10.124.97])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 13BB2784B1;\n\tMon, 18 Sep 2017 15:08:19 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 50BB3C0587D9","To":"\"Dr. David Alan Gilbert\" , =?utf-8?q?Marc-Andr?=\n\t=?utf-8?b?w6kgTHVyZWF1?= ","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t\n\t<20170918105516.GD2581@work-vm>","From":"Eric Blake ","Openpgp":"url=http://people.redhat.com/eblake/eblake.gpg","Organization":"Red Hat, Inc.","Message-ID":"<5e0abb57-0c5b-50b1-38ea-6b699bc21205@redhat.com>","Date":"Mon, 18 Sep 2017 10:08:18 -0500","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170918105516.GD2581@work-vm>","Content-Type":"multipart/signed; micalg=pgp-sha256;\n\tprotocol=\"application/pgp-signature\";\n\tboundary=\"mRo2qTqWhXD2PdLMnhQuPQDTbpOFLSHxh\"","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.12","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.32]);\n\tMon, 18 Sep 2017 15:08:24 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","X-Content-Filtered-By":"Mailman/MimeDel 2.1.21","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770296,"web_url":"http://patchwork.ozlabs.org/comment/1770296/","msgid":"","list_archive_url":null,"date":"2017-09-18T16:09:29","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":6442,"url":"http://patchwork.ozlabs.org/api/people/6442/","name":"Marc-André Lureau","email":"marcandre.lureau@gmail.com"},"content":"On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert\n wrote:\n> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> Hi\n>>\n>> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert\n>> wrote:\n>> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> >> Hi\n>> >>\n>> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n>> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n>> >> >> Hi\n>> >> >>\n>> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n>> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n>> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n>> >> >> >> > Hi\n>> >> >> >> >\n>> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n>> >> >> >> > > This series was born from this one:\n>> >> >> >> > >\n>> >> >> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n>> >> >> >> > >\n>> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n>> >> >> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n>> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n>> >> >> >> > > ideas and suggestions. Finally we got such a solution that seems to\n>> >> >> >> > > satisfy everyone.\n>> >> >> >> > >\n>> >> >> >> > > I re-started the versioning since this series is totally different\n>> >> >> >> > > from previous one. Now it's version 1.\n>> >> >> >> > >\n>> >> >> >> > > In case new reviewers come along the way without reading previous\n>> >> >> >> > > discussions, I will try to do a summary on what this is all about.\n>> >> >> >> > >\n>> >> >> >> > > What is OOB execution?\n>> >> >> >> > > ======================\n>> >> >> >> > >\n>> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n>> >> >> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n>> >> >> >> > > QMP is going throw these steps:\n>> >> >> >> > >\n>> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> >> >> > > /|\\ (2) (3) |\n>> >> >> >> > > (1) | \\|/ (4)\n>> >> >> >> > > +--------- main thread --------+\n>> >> >> >> > >\n>> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the\n>> >> >> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n>> >> >> >> > > parser and quickly returns.\n>> >> >> >> >\n>> >> >> >> > All commands should have the \"id\" field mandatory in this case, else\n>> >> >> >> > the client will not distinguish the replies coming from the last/oob\n>> >> >> >> > and the previous commands.\n>> >> >> >> >\n>> >> >> >> > This should probably be enforced upfront by client capability checks,\n>> >> >> >> > more below.\n>> >> >> >\n>> >> >> > Hmm yes since the oob commands are actually running in async way,\n>> >> >> > request ID should be needed here. However I'm not sure whether\n>> >> >> > enabling the whole \"request ID\" thing is too big for this \"try to be\n>> >> >> > small\" oob change... And IMHO it suites better to be part of the whole\n>> >> >> > async work (no matter which implementation we'll use).\n>> >> >> >\n>> >> >> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n>> >> >> > For oob commands, they will always have ID then no ordering issue, and\n>> >> >> > we can do it async; for the rest of non-oob commands, we still allow\n>> >> >> > them to go without ID, and since they are not oob, they'll always be\n>> >> >> > done in order as well. Would this work?\n>> >> >>\n>> >> >> This mixed-mode is imho more complicated to deal with than having the\n>> >> >> protocol enforced one way or the other, but that should work.\n>> >> >>\n>> >> >> >\n>> >> >> >> >\n>> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly\n>> >> >> >> > > (please see handle_qmp_command()). However it's not true again after\n>> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will\n>> >> >> >> > > still be run in main thread). So this OOB does brings something\n>> >> >> >> > > different.\n>> >> >> >> > >\n>> >> >> >> > > There are more details on why OOB and the difference/relationship\n>> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n>> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to\n>> >> >> >> > > summarize that). For more information, please refers to [1].\n>> >> >> >> > >\n>> >> >> >> > > Summary ends here.\n>> >> >> >> > >\n>> >> >> >> > > Some Implementation Details\n>> >> >> >> > > ===========================\n>> >> >> >> > >\n>> >> >> >> > > Again, I mentioned that the old QMP workflow is this:\n>> >> >> >> > >\n>> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n>> >> >> >> > > /|\\ (2) (3) |\n>> >> >> >> > > (1) | \\|/ (4)\n>> >> >> >> > > +--------- main thread --------+\n>> >> >> >> > >\n>> >> >> >> > > What this series does is, firstly:\n>> >> >> >> > >\n>> >> >> >> > > JSON Parser QMP Dispatcher --> Respond\n>> >> >> >> > > /|\\ | /|\\ (4) |\n>> >> >> >> > > | | (2) | (3) | (5)\n>> >> >> >> > > (1) | +-----> | \\|/\n>> >> >> >> > > +--------- main thread <-------+\n>> >> >> >> > >\n>> >> >> >> > > And further:\n>> >> >> >> > >\n>> >> >> >> > > queue/kick\n>> >> >> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n>> >> >> >> > > /|\\ | (3) /|\\ (4) |\n>> >> >> >> > > (1) | | (2) | | (5)\n>> >> >> >> > > | \\|/ | \\|/\n>> >> >> >> > > IO thread main thread <-------+\n>> >> >> >> >\n>> >> >> >> > Is the queue per monitor or per client?\n>> >> >> >\n>> >> >> > The queue is currently global. I think yes maybe at least we can do it\n>> >> >> > per monitor, but I am not sure whether that is urgent or can be\n>> >> >> > postponed. After all now QMPRequest (please refer to patch 11) is\n>> >> >> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n>> >> >> > per-monitor.\n>> >> >> >\n>> >> >> >> > And is the dispatching going\n>> >> >> >> > to be processed even if the client is disconnected, and are new\n>> >> >> >> > clients going to receive the replies from previous clients\n>> >> >> >> > commands?\n>> >> >> >\n>> >> >> > [1]\n>> >> >> >\n>> >> >> > (will discuss together below)\n>> >> >> >\n>> >> >> >> > I\n>> >> >> >> > believe there should be a per-client context, so there won't be \"id\"\n>> >> >> >> > request conflicts.\n>> >> >> >\n>> >> >> > I'd say I am not familiar with this \"client\" idea, since after all\n>> >> >> > IMHO one monitor is currently designed to mostly work with a single\n>> >> >> > client. Say, unix sockets, telnet, all these backends are only single\n>> >> >> > channeled, and one monitor instance can only work with one client at a\n>> >> >> > time. Then do we really need to add this client layer upon it? IMHO\n>> >> >> > the user can just provide more monitors if they wants more clients\n>> >> >> > (and at least these clients should know the existance of the others or\n>> >> >> > there might be problem, otherwise user2 will fail a migration, finally\n>> >> >> > noticed that user1 has already triggered one), and the user should\n>> >> >> > manage them well.\n>> >> >>\n>> >> >> qemu should support a management layer / libvirt restart/reconnect.\n>> >> >> Afaik, it mostly work today. There might be a cases where libvirt can\n>> >> >> be confused if it receives a reply from a previous connection command,\n>> >> >> but due to the sync processing of the chardev, I am not sure you can\n>> >> >> get in this situation. By adding \"oob\" commands and queuing, the\n>> >> >> client will have to remember which was the last \"id\" used, or it will\n>> >> >> create more conflict after a reconnect.\n>> >> >>\n>> >> >> Imho we should introduce the client/connection concept to avoid this\n>> >> >> confusion (unexpected reply & per client id space).\n>> >> >\n>> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO\n>> >> > instead of throwing responses away when client disconnect, we should\n>> >> > really keep them, and when the client reconnects, we queue the\n>> >> > responses again.\n>> >> >\n>> >> > I think we have other quite simple ways to solve the \"unexpected\n>> >> > reply\" and \"per-client-id duplication\" issues you have mentioned.\n>> >> >\n>> >> > Firstly, when client gets unexpected replies (\"id\" field not in its\n>> >> > own request queue), the client should just ignore that reply, which\n>> >> > seems natural to me.\n>> >>\n>> >> The trouble is that it may legitimately use the same \"id\" value for\n>> >> new requests. And I don't see a simple way to handle that without\n>> >> races.\n>> >\n>> > Under what circumstances can it reuse the same ID for new requests?\n>> > Can't we simply tell it not to?\n>>\n>> I don't see any restriction today in the protocol in connecting with a\n>> new client that may not know anything from a previous client.\n>\n> Well, it knows it's doing a reconnection.\n\nIf you assume the \"same client\" reconnects to the monitor, I agree.\nBut this is a restriction of monitor usage.\n\n>\n>> How would you tell it not to use old IDs? Just by writing an unwritten\n>> rule, because we don't want to fix the per connection client session\n>> handling in qemu?\n>\n> BY writing a written rule! This out of order stuff we're adding here\n> is a change to the interface and we can define what we require of the\n> client. As long as what we expect is reasonable then we might end\n> up with something that's simpler for both the client and qemu.\n\nAs long as we don't break existing qmp clients.\n\n> And I worry this series keeps getting more and more complex for weird\n> edge cases.\n\nThat's an interesting point-of-view. I see the point in fixing weird\nedge cases in qemu RPC code. More than other code we develop with\nweird edge cases in mind & tests, like the parsing/checking of the\njson schema for ex, in a similar area with the same maintainer.\n\n> Dave\n>\n>> >\n>> > Dave\n>> >\n>> >> >\n>> >> > Then, if client disconnected and reconnected, it should not have the\n>> >> > problem to generate duplicated id for request, since it should know\n>> >> > what requests it has sent already. A simplest case I can think of is,\n>> >> > the ID should contains the following tuple:\n>> >>\n>> >> If you assume the \"same\" client will recover its state, yes.\n>> >>\n>> >> >\n>> >> > (client name, client unique ID, request ID)\n>> >> >\n>> >> > Here \"client name\" can be something like \"libvirt\", which is the name\n>> >> > of client application;\n>> >> >\n>> >> > \"client unique ID\" can be anything generated when client starts, it\n>> >> > identifies a single client session, maybe a UUID.\n>> >> >\n>> >> > \"request ID\" can be a unsigned integer starts from zero, and increases\n>> >> > each time the client sends one request.\n>> >>\n>> >> This is introducing session handling, and can be done in server side\n>> >> only without changes in the protocol I believe.\n>> >>\n>> >> >\n>> >> > I believe current libvirt is using \"client name\" + \"request ID\". It's\n>> >> > something similar (after all I think we don't normally have >1 libvirt\n>> >> > to manage single QEMU, so I think it should be good enough).\n>> >>\n>> >> I am not sure we should base our protocol usage assumptions based on\n>> >> libvirt only, but rather on what is possible today (like queuing\n>> >> requests in the socket etc..).\n>> >>\n>> >> > Then even if client disconnect and reconnect, request ID won't lose,\n>> >> > and no duplication would happen IMHO.\n>> >> >\n>> >> >>\n>> >> >> >\n>> >> >> >> >\n>> >> >> >> > >\n>> >> >> >> > > Then it introduced the \"allow-oob\" parameter in QAPI schema to define\n>> >> >> >> > > commands, and \"run-oob\" flag to let oob-allowed command to run in the\n>> >> >> >> > > parser.\n>> >> >> >> >\n>> >> >> >> > From a protocol point of view, I find that \"run-oob\" distinction per\n>> >> >> >> > command a bit pointless. It helps with legacy client that wouldn't\n>> >> >> >> > expect out-of-order replies if qemu were to run oob commands oob by\n>> >> >> >> > default though.\n>> >> >> >\n>> >> >> > After all oob somehow breaks existing rules or sync execution. I\n>> >> >> > thought the more important goal was at least to keep the legacy\n>> >> >> > behaviors when adding new things, no?\n>> >> >>\n>> >> >> Of course we have to keep compatibily. What do you mean by \"oob\n>> >> >> somehow breaks existing rules or sync execution\"? oob means queuing\n>> >> >> and unordered reply support, so clearly this is breaking the current\n>> >> >> \"mostly ordered\" behaviour (mostly because events may still come any\n>> >> >> time..., and the reconnect issue discussed above).\n>> >> >\n>> >> > Yes. That's what I mean, it breaks the synchronous scemantic. But\n>> >> > I should definitely not call it a \"break\" though since old clients\n>> >> > will work perfectly fine with it. Sorry for the bad wording.\n>> >> >\n>> >> >>\n>> >> >> >> > Clients shouldn't care about how/where a command is\n>> >> >> >> > being queued or not. If they send a command, they want it processed as\n>> >> >> >> > quickly as possible. However, it can be interesting to know if the\n>> >> >> >> > implementation of the command will be able to deliver oob, so that\n>> >> >> >> > data in the introspection could be useful.\n>> >> >> >> >\n>> >> >> >> > I would rather propose a client/server capability in qmp_capabilities,\n>> >> >> >> > call it \"oob\":\n>> >> >> >> >\n>> >> >> >> > This capability indicates oob commands support.\n>> >> >> >>\n>> >> >> >> The problem is indicating which commands support oob as opposed to\n>> >> >> >> indicating whether oob is present at all. Future versions will\n>> >> >> >> probably make more commands oob-able and a client will want to know\n>> >> >> >> whether it can rely on a particular command being non-blocking.\n>> >> >> >\n>> >> >> > Yes.\n>> >> >> >\n>> >> >> > And IMHO we don't urgently need that \"whether the server globally\n>> >> >> > supports oob\" thing. Client can just know that from query-qmp-schema\n>> >> >> > already - there will always be the \"allow-oob\" new field for command\n>> >> >> > typed entries. IMHO that's a solid hint.\n>> >> >> >\n>> >> >> > But I don't object to return it as well in qmp_capabilities.\n>> >> >>\n>> >> >> Does it feel right that the client can specify how the command are\n>> >> >> processed / queued ? Isn't it preferable to leave that to the server\n>> >> >> to decide? Why would a client specify that? And should the server be\n>> >> >> expected to behave differently? What the client needs to be able is to\n>> >> >> match the unordered replies, and that can be stated during cap\n>> >> >> negotiation / qmp_capabilties. The server is expected to do a best\n>> >> >> effort to handle commands and their priorities. If the client needs\n>> >> >> several command queue, it is simpler to open several connection rather\n>> >> >> than trying to fit that weird priority logic in the protocol imho.\n>> >> >\n>> >> > Sorry I may have missed the point here. We were discussing about a\n>> >> > global hint for \"oob\" support, am I right? Then, could I ask what's\n>> >> > the \"weird priority logic\" you mentioned?\n>> >>\n>> >> I call per-message oob hint a kind of priority logic, since you can\n>> >> make the same request without oob in the same session and in parallel.\n>> >>\n>> >> >>\n>> >> >> >\n>> >> >> >>\n>> >> >> >> > An oob command is a regular client message request with the \"id\"\n>> >> >> >> > member mandatory, but the reply may be delivered\n>> >> >> >> > out of order by the server if the client supports\n>> >> >> >> > it too.\n>> >> >> >> >\n>> >> >> >> > If both the server and the client have the \"oob\" capability, the\n>> >> >> >> > server can handle new client requests while previous requests are being\n>> >> >> >> > processed.\n>> >> >> >> >\n>> >> >> >> > If the client doesn't have the \"oob\" capability, it may still call\n>> >> >> >> > an oob command, and make multiple outstanding calls. In this case,\n>> >> >> >> > the commands are processed in order, so the replies will also be in\n>> >> >> >> > order. The \"id\" member isn't mandatory in this case.\n>> >> >> >> >\n>> >> >> >> > The client should match the replies with the \"id\" member associated\n>> >> >> >> > with the requests.\n>> >> >> >> >\n>> >> >> >> > When a client is disconnected, the pending commands are not\n>> >> >> >> > necessarily cancelled. But the future clients will not get replies from\n>> >> >> >> > commands they didn't make (they might, however, receive side-effects\n>> >> >> >> > events).\n>> >> >> >>\n>> >> >> >> What's the behaviour on the current monitor?\n>> >> >> >\n>> >> >> > Yeah I want to ask the same question, along with questioning about\n>> >> >> > above [1].\n>> >> >> >\n>> >> >> > IMHO this series will not change the behaviors of these, so IMHO the\n>> >> >> > behaviors will be the same before/after this series. E.g., when client\n>> >> >> > dropped right after the command is executed, I think we will still\n>> >> >> > execute the command, though we should encounter something odd in\n>> >> >> > monitor_json_emitter() somewhere when we want to respond. And it will\n>> >> >> > happen the same after this series.\n>> >> >>\n>> >> >> I think it can get worse after your series, because you queue the\n>> >> >> commands, so clearly a new client can get replies from an old client\n>> >> >> commands. As said above, I am not convinced you can get in that\n>> >> >> situation with current code.\n>> >> >\n>> >> > Hmm, seems so. But would this a big problem?\n>> >> >\n>> >> > I really think the new client should just throw that response away if\n>> >> > it does not really know that response (from peeking at \"id\" field),\n>> >> > just like my opinion above.\n>> >>\n>> >> This is a high expectation.\n>> >>\n>> >>\n>> >> --\n>> >> Marc-André Lureau\n>> > --\n>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK\n>>\n>>\n>>\n>> --\n>> Marc-André Lureau\n> --\n> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ozlabs.org;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"l3tXD9dd\"; dkim-atps=neutral"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xws0S3Vhnz9s78\n\tfor ;\n\tTue, 19 Sep 2017 02:29:28 +1000 (AEST)","from localhost ([::1]:37727 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dtyvG-0006a5-HT\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 12:29:26 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:40940)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dtyc2-0005yv-Jf\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 12:09:37 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dtybz-0003jx-V6\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 12:09:34 -0400","from mail-io0-x22e.google.com ([2607:f8b0:4001:c06::22e]:46843)\n\tby eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)\n\t(Exim 4.71) (envelope-from )\n\tid 1dtybz-0003jm-Nh\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 12:09:31 -0400","by mail-io0-x22e.google.com with SMTP id d16so3331099ioj.3\n\tfor ; Mon, 18 Sep 2017 09:09:31 -0700 (PDT)","by 10.58.69.69 with HTTP; Mon, 18 Sep 2017 09:09:29 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc:content-transfer-encoding;\n\tbh=rFv7MG5bU0Iaa+VTVpiLwEQHkUmFhtP6ySp1e4aPIUA=;\n\tb=l3tXD9ddmrWZEG0sP2AnUM0R0Z4KXrK7lJf50g9b1HIUIyaiT98vvdMpv1fKTOrnM0\n\t21NPhrl4A0QfgoFsoeh5M5nKpOJE7sDMTZpBA/5BITQ7aC1+oqBRSnUDtXrA9Zm2yDIU\n\ttuoSWHENrDpzeO6gG1tUk1vYaJ5LapnybnhAaXSzlovB9Pdztdh5j1UuL6tv07CWZxLS\n\tBKNCqRjnUZnwRTI/rmHZiQHrdGDmUVWCKRox1cOswLsYz+iZgpPxpYoKcOyXC95z7Xah\n\typcULM1kyF4gtj7lnFRCweouvezIwz8TJYYcCM2o4VcRerRo+ysnhStgoqZvx97bHuVy\n\tcS2g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc:content-transfer-encoding;\n\tbh=rFv7MG5bU0Iaa+VTVpiLwEQHkUmFhtP6ySp1e4aPIUA=;\n\tb=P7rAy/0NRItmlxVZ51u5vKzNBlJGw26qwubfwT1hpPOp3QsflMgdvBZxqfhRZwuOcW\n\tooRm2qryPyDCmI0sHobqFAAjdntemk/VsP0Ra3rlcpJ4QLc9q6LWVYMKPfTU7veMdX0L\n\t1L+jWKIKnGk5KZXk/lruFTpRT/3lRKfGwm/zKkJqYUZpwxcq68eGNF9Xianp7yVoJBbl\n\tVL032xJ+2qeVSxtAFVIR9jO1OziL7cPM3MhF+uEwEF7vulssApyZDN6/0Ru1AuP1Dsf1\n\tK3YrhWn7yR10HrbTJg5Ue1z9qs8ttrZvZ/B0hTyQvlWg9Kt3imVEWzKt5A/gAgnKxI6i\n\tLMig==","X-Gm-Message-State":"AHPjjUgQMxYf3ZYMvVCQMEraIeGmNFAhOG1V3PWCWjU3Q3nOvQ7hH5bP\n\tYLsdTyMi6y4xIiQLO11hLwSZQRYjD9LaWVIFwH8=","X-Google-Smtp-Source":"AOwi7QAi9457Fu/zSP8bED4cYK+YZWHCoFGGnoWEbhqfWeSiWmlJnRESeBoijwG8moEc80ey0Bn1OT6AcGWjukgi6KE=","X-Received":"by 10.202.85.202 with SMTP id\n\tj193mr21336222oib.122.1505750970589; \n\tMon, 18 Sep 2017 09:09:30 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170918112618.GF2581@work-vm>","References":"<1505375436-28439-1-git-send-email-peterx@redhat.com>\n\t\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t\n\t<20170918105516.GD2581@work-vm>\n\t\n\t<20170918112618.GF2581@work-vm>","From":"=?utf-8?q?Marc-Andr=C3=A9_Lureau?= ","Date":"Mon, 18 Sep 2017 18:09:29 +0200","Message-ID":"","To":"\"Dr. David Alan Gilbert\" ","Content-Type":"text/plain; charset=\"UTF-8\"","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: Genre and OS details not\n\trecognized.","X-Received-From":"2607:f8b0:4001:c06::22e","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela ,\n\tMarkus Armbruster , \n\tPeter Xu , QEMU ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770616,"web_url":"http://patchwork.ozlabs.org/comment/1770616/","msgid":"<20170919022312.GG3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-19T02:23:12","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Mon, Sep 18, 2017 at 11:40:40AM +0100, Dr. David Alan Gilbert wrote:\n> * Peter Xu (peterx@redhat.com) wrote:\n> > On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:\n> > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:\n> > > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> > > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > > > > \n> > > > > > > > > > > > I agree.\n> > > > > > > > > > > > \n> > > > > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > > > > \n> > > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > > > > \n> > > > > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > > > > next version.\n> > > > > > > > > > > \n> > > > > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > > > > \n> > > > > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > > > > \n> > > > > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > > > > the client stays within the receive buffer size.\n> > > > > > > > > > \n> > > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > > > > size and make sure to leave enough room.\n> > > > > > > > > \n> > > > > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > > > > and of course it is inherantly racy.\n> > > > > > > > > \n> > > > > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > > > > it restarts reading.\n> > > > > > > > \n> > > > > > > > Hmm and now we're implementing flow control!\n> > > > > > > > \n> > > > > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > > > > waiting for the response to the previous command?\n> > > > > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > > > > \n> > > > > > > I do wonder if we need to worry about request limiting at all from the\n> > > > > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > > > > \n> > > > > > > OOB commands are supposed to be things which can be handled quickly\n> > > > > > > without blocking, so even if a client sent several commands at once\n> > > > > > > without waiting for replies, they're going to be processed quickly,\n> > > > > > > so whether we temporarily block reading off the wire is a minor\n> > > > > > > detail.\n> > > > > > \n> > > > > > Lets just define it so that it can't - you send an OOB command and wait\n> > > > > > for it's response before sending another on that channel.\n> > > > > > \n> > > > > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > > > > the app's POV.\n> > > > > > \n> > > > > > Can you justify 10 as opposed to 1?\n> > > > > \n> > > > > Semantically I don't think it makes a difference if the OOB commands are\n> > > > > being processed sequentially by their thread. A >1 length queue would only\n> > > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > > > > requests, as then that could block reading of OOB commands. \n> > > > \n> > > > To summarize:\n> > > > \n> > > > The QMP server has a lookahead of 1 command so it can dispatch\n> > > > out-of-band commands. If 2 or more non-OOB commands are queued at the\n> > > > same time then OOB processing will not occur.\n> > > > \n> > > > Is that right?\n> > > \n> > > I think my view is slightly more complex;\n> > > a) There's a pair of queues for each channel\n> > > b) There's a central pair of queues on the QMP server\n> > > one for OOB commands and one for normal commands.\n> > > c) Each queue is only really guaranteed to be one deep.\n> > > \n> > > That means that each one of the channels can send a non-OOB\n> > > command without getting in the way of a channel that wants\n> > > to send one.\n> > \n> > But current version should not be that complex:\n> > \n> > Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.\n> > \n> > Then, we only have a single global queue for QMP non-oob commands, and\n> > we don't have response queue yet. We do respond just like before in a\n> > synchronous way (I explained why - for OOB we don't need that\n> > complexity IMHO).\n> \n> I think the discussion started because of two related comments:\n> Marc-André said :\n> 'There should be a limit in the number of requests the thread can\n> queue'\n> and Stefan said :\n> 'Memory usage must be bounded.'\n> \n> actually neither of those cases really worried me (because they only\n> happen if the client keeps pumping commands, and that seems it's fault).\n> \n> However, once you start adding a limit, you've got to be careful - if\n> you just added a limit to the central queue, then what happens if that\n> queue is filled by non-OOB commands?\n\nAh! So I misunderstood \"a pair of queues for each channel\". I\nthought it means the input and output of a single monitor, while I\nthink it actually means \"OOB channel\" and \"non-OOB channel\".\n\nMy plan (or say, this version) starts from only one global queue for\nnon-OOB commands. There is no queue for OOB commands at all. As\ndiscussed below [1], if we receive one OOB command, we execute it\ndirectly and reply to client. And here the \"request queue\" will only\nqueue non-OOB commands. Maybe the name \"request queue\" sounds\nconfusing here.\n\nIf so, we should not have above problem, right? Since even if the\nqueue is full (of course there will only be non-OOB commands in the\nqueue), the parsing is still working, and we will still be able to\nhandle OOB ones:\n\n req = parse(stream);\n\n if (is_oob(req)) {\n execute(req);\n return;\n }\n\n if (queue_full(req_queue)) {\n emit_full_event(req);\n return;\n }\n\n enqueue(req_queue, req);\n\nSo again, this version is a simplified version from previous\ndiscussion (no oob-queue but only non-oob-queue, no respond queue but\nonly request queue, etc...), but hope it can work.\n\nThanks,\n\n> \n> Dave\n> \n> > When we parse commands, we execute it directly if OOB, otherwise we\n> > put it onto request queue. Request queue handling is done by a main\n> > thread QEMUBH. That's all.\n\n[1]\n\n> > \n> > Would this \"simple version\" suffice to implement this whole OOB idea?\n> > \n> > (Again, I really don't think we need to specify queue length to 1,\n> > though we can make it small)\n> > \n> > -- \n> > Peter Xu\n> --\n> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx02.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xx6BW275Cz9s4s\n\tfor ;\n\tTue, 19 Sep 2017 12:24:03 +1000 (AEST)","from localhost ([::1]:39754 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1du8Cf-0008Lo-Dy\n\tfor incoming@patchwork.ozlabs.org; Mon, 18 Sep 2017 22:24:01 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:35678)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1du8C9-0008Jv-2E\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 22:23:30 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1du8C4-0003dJ-S2\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 22:23:29 -0400","from mx1.redhat.com ([209.132.183.28]:34972)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1du8C4-0003ch-J3\n\tfor qemu-devel@nongnu.org; Mon, 18 Sep 2017 22:23:24 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 66449883D7;\n\tTue, 19 Sep 2017 02:23:23 +0000 (UTC)","from pxdev.xzpeter.org (ovpn-12-66.pek2.redhat.com [10.72.12.66])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 3C8F26061B;\n\tTue, 19 Sep 2017 02:23:14 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 66449883D7","Date":"Tue, 19 Sep 2017 10:23:12 +0800","From":"Peter Xu ","To":"\"Dr. David Alan Gilbert\" ","Message-ID":"<20170919022312.GG3617@pxdev.xzpeter.org>","References":"<20170915104926.GA14994@stefanha-x1.localdomain>\n\t<20170915113428.GF13610@redhat.com> <20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>\n\t<20170915145632.GD18767@stefanha-x1.localdomain>\n\t<20170915151706.GQ2170@work-vm>\n\t<20170918092625.GE3617@pxdev.xzpeter.org>\n\t<20170918104039.GC2581@work-vm>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<20170918104039.GC2581@work-vm>","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.26]);\n\tTue, 19 Sep 2017 02:23:23 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Juan Quintela ,\n\tMarkus Armbruster , QEMU ,\n\tMichael Roth , =?utf-8?q?Marc-Andr=C3=A9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ,\n\tFam Zheng , \tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770678,"web_url":"http://patchwork.ozlabs.org/comment/1770678/","msgid":"<20170919062903.GH3617@pxdev.xzpeter.org>","list_archive_url":null,"date":"2017-09-19T06:29:03","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Mon, Sep 18, 2017 at 06:09:29PM +0200, Marc-André Lureau wrote:\n> On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert\n> wrote:\n> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> Hi\n> >>\n> >> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert\n> >> wrote:\n> >> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> >> Hi\n> >> >>\n> >> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu wrote:\n> >> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:\n> >> >> >> Hi\n> >> >> >>\n> >> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu wrote:\n> >> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:\n> >> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:\n> >> >> >> >> > Hi\n> >> >> >> >> >\n> >> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu wrote:\n> >> >> >> >> > > This series was born from this one:\n> >> >> >> >> > >\n> >> >> >> >> > > https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html\n> >> >> >> >> > >\n> >> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions\n> >> >> >> >> > > in previous thread. My heartful thanks to Markus, Daniel, Dave,\n> >> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny\n> >> >> >> >> > > ideas and suggestions. Finally we got such a solution that seems to\n> >> >> >> >> > > satisfy everyone.\n> >> >> >> >> > >\n> >> >> >> >> > > I re-started the versioning since this series is totally different\n> >> >> >> >> > > from previous one. Now it's version 1.\n> >> >> >> >> > >\n> >> >> >> >> > > In case new reviewers come along the way without reading previous\n> >> >> >> >> > > discussions, I will try to do a summary on what this is all about.\n> >> >> >> >> > >\n> >> >> >> >> > > What is OOB execution?\n> >> >> >> >> > > ======================\n> >> >> >> >> > >\n> >> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by\n> >> >> >> >> > > Markus. It's a way to quickly execute a QMP request. Say, originally\n> >> >> >> >> > > QMP is going throw these steps:\n> >> >> >> >> > >\n> >> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> >> >> > > /|\\ (2) (3) |\n> >> >> >> >> > > (1) | \\|/ (4)\n> >> >> >> >> > > +--------- main thread --------+\n> >> >> >> >> > >\n> >> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the\n> >> >> >> >> > > JSON is parsed. If OOB is on, we run the command directly in the\n> >> >> >> >> > > parser and quickly returns.\n> >> >> >> >> >\n> >> >> >> >> > All commands should have the \"id\" field mandatory in this case, else\n> >> >> >> >> > the client will not distinguish the replies coming from the last/oob\n> >> >> >> >> > and the previous commands.\n> >> >> >> >> >\n> >> >> >> >> > This should probably be enforced upfront by client capability checks,\n> >> >> >> >> > more below.\n> >> >> >> >\n> >> >> >> > Hmm yes since the oob commands are actually running in async way,\n> >> >> >> > request ID should be needed here. However I'm not sure whether\n> >> >> >> > enabling the whole \"request ID\" thing is too big for this \"try to be\n> >> >> >> > small\" oob change... And IMHO it suites better to be part of the whole\n> >> >> >> > async work (no matter which implementation we'll use).\n> >> >> >> >\n> >> >> >> > How about this: we make \"id\" mandatory for \"run-oob\" requests only.\n> >> >> >> > For oob commands, they will always have ID then no ordering issue, and\n> >> >> >> > we can do it async; for the rest of non-oob commands, we still allow\n> >> >> >> > them to go without ID, and since they are not oob, they'll always be\n> >> >> >> > done in order as well. Would this work?\n> >> >> >>\n> >> >> >> This mixed-mode is imho more complicated to deal with than having the\n> >> >> >> protocol enforced one way or the other, but that should work.\n> >> >> >>\n> >> >> >> >\n> >> >> >> >> >\n> >> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly\n> >> >> >> >> > > (please see handle_qmp_command()). However it's not true again after\n> >> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will\n> >> >> >> >> > > still be run in main thread). So this OOB does brings something\n> >> >> >> >> > > different.\n> >> >> >> >> > >\n> >> >> >> >> > > There are more details on why OOB and the difference/relationship\n> >> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's\n> >> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to\n> >> >> >> >> > > summarize that). For more information, please refers to [1].\n> >> >> >> >> > >\n> >> >> >> >> > > Summary ends here.\n> >> >> >> >> > >\n> >> >> >> >> > > Some Implementation Details\n> >> >> >> >> > > ===========================\n> >> >> >> >> > >\n> >> >> >> >> > > Again, I mentioned that the old QMP workflow is this:\n> >> >> >> >> > >\n> >> >> >> >> > > JSON Parser --> QMP Dispatcher --> Respond\n> >> >> >> >> > > /|\\ (2) (3) |\n> >> >> >> >> > > (1) | \\|/ (4)\n> >> >> >> >> > > +--------- main thread --------+\n> >> >> >> >> > >\n> >> >> >> >> > > What this series does is, firstly:\n> >> >> >> >> > >\n> >> >> >> >> > > JSON Parser QMP Dispatcher --> Respond\n> >> >> >> >> > > /|\\ | /|\\ (4) |\n> >> >> >> >> > > | | (2) | (3) | (5)\n> >> >> >> >> > > (1) | +-----> | \\|/\n> >> >> >> >> > > +--------- main thread <-------+\n> >> >> >> >> > >\n> >> >> >> >> > > And further:\n> >> >> >> >> > >\n> >> >> >> >> > > queue/kick\n> >> >> >> >> > > JSON Parser ======> QMP Dispatcher --> Respond\n> >> >> >> >> > > /|\\ | (3) /|\\ (4) |\n> >> >> >> >> > > (1) | | (2) | | (5)\n> >> >> >> >> > > | \\|/ | \\|/\n> >> >> >> >> > > IO thread main thread <-------+\n> >> >> >> >> >\n> >> >> >> >> > Is the queue per monitor or per client?\n> >> >> >> >\n> >> >> >> > The queue is currently global. I think yes maybe at least we can do it\n> >> >> >> > per monitor, but I am not sure whether that is urgent or can be\n> >> >> >> > postponed. After all now QMPRequest (please refer to patch 11) is\n> >> >> >> > defined as (mon, id, req) tuple, so at least \"id\" namespace is\n> >> >> >> > per-monitor.\n> >> >> >> >\n> >> >> >> >> > And is the dispatching going\n> >> >> >> >> > to be processed even if the client is disconnected, and are new\n> >> >> >> >> > clients going to receive the replies from previous clients\n> >> >> >> >> > commands?\n> >> >> >> >\n> >> >> >> > [1]\n> >> >> >> >\n> >> >> >> > (will discuss together below)\n> >> >> >> >\n> >> >> >> >> > I\n> >> >> >> >> > believe there should be a per-client context, so there won't be \"id\"\n> >> >> >> >> > request conflicts.\n> >> >> >> >\n> >> >> >> > I'd say I am not familiar with this \"client\" idea, since after all\n> >> >> >> > IMHO one monitor is currently designed to mostly work with a single\n> >> >> >> > client. Say, unix sockets, telnet, all these backends are only single\n> >> >> >> > channeled, and one monitor instance can only work with one client at a\n> >> >> >> > time. Then do we really need to add this client layer upon it? IMHO\n> >> >> >> > the user can just provide more monitors if they wants more clients\n> >> >> >> > (and at least these clients should know the existance of the others or\n> >> >> >> > there might be problem, otherwise user2 will fail a migration, finally\n> >> >> >> > noticed that user1 has already triggered one), and the user should\n> >> >> >> > manage them well.\n> >> >> >>\n> >> >> >> qemu should support a management layer / libvirt restart/reconnect.\n> >> >> >> Afaik, it mostly work today. There might be a cases where libvirt can\n> >> >> >> be confused if it receives a reply from a previous connection command,\n> >> >> >> but due to the sync processing of the chardev, I am not sure you can\n> >> >> >> get in this situation. By adding \"oob\" commands and queuing, the\n> >> >> >> client will have to remember which was the last \"id\" used, or it will\n> >> >> >> create more conflict after a reconnect.\n> >> >> >>\n> >> >> >> Imho we should introduce the client/connection concept to avoid this\n> >> >> >> confusion (unexpected reply & per client id space).\n> >> >> >\n> >> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO\n> >> >> > instead of throwing responses away when client disconnect, we should\n> >> >> > really keep them, and when the client reconnects, we queue the\n> >> >> > responses again.\n> >> >> >\n> >> >> > I think we have other quite simple ways to solve the \"unexpected\n> >> >> > reply\" and \"per-client-id duplication\" issues you have mentioned.\n> >> >> >\n> >> >> > Firstly, when client gets unexpected replies (\"id\" field not in its\n> >> >> > own request queue), the client should just ignore that reply, which\n> >> >> > seems natural to me.\n> >> >>\n> >> >> The trouble is that it may legitimately use the same \"id\" value for\n> >> >> new requests. And I don't see a simple way to handle that without\n> >> >> races.\n> >> >\n> >> > Under what circumstances can it reuse the same ID for new requests?\n> >> > Can't we simply tell it not to?\n> >>\n> >> I don't see any restriction today in the protocol in connecting with a\n> >> new client that may not know anything from a previous client.\n> >\n> > Well, it knows it's doing a reconnection.\n> \n> If you assume the \"same client\" reconnects to the monitor, I agree.\n> But this is a restriction of monitor usage.\n\nIn monitor_qmp_event(), we can empty the request queue when got\nCHR_EVENT_CLOSED. Would that be a solution?","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xxCfF0CQ6z9s7h\n\tfor ;\n\tTue, 19 Sep 2017 16:29:54 +1000 (AEST)","from localhost ([::1]:40270 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1duC2Y-00058l-Pt\n\tfor incoming@patchwork.ozlabs.org; Tue, 19 Sep 2017 02:29:50 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:52518)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1duC22-00058M-VL\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 02:29:20 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1duC1z-00071L-KR\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 02:29:18 -0400","from mx1.redhat.com ([209.132.183.28]:36366)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1duC1z-00070e-BI\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 02:29:15 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 3C55981DE7;\n\tTue, 19 Sep 2017 06:29:13 +0000 (UTC)","from pxdev.xzpeter.org (ovpn-12-66.pek2.redhat.com [10.72.12.66])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id A57655D6A8;\n\tTue, 19 Sep 2017 06:29:06 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 3C55981DE7","Date":"Tue, 19 Sep 2017 14:29:03 +0800","From":"Peter Xu ","To":"=?utf-8?q?Marc-Andr=C3=A9?= Lureau ","Message-ID":"<20170919062903.GH3617@pxdev.xzpeter.org>","References":"\n\t<20170914185314.GA3280@work-vm>\n\t<20170915044622.GO3617@pxdev.xzpeter.org>\n\t\n\t<20170918083737.GD3617@pxdev.xzpeter.org>\n\t\n\t<20170918105516.GD2581@work-vm>\n\t\n\t<20170918112618.GF2581@work-vm>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/1.5.24 (2015-08-30)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tTue, 19 Sep 2017 06:29:13 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Fam Zheng ,\n\tMichael Roth ,\n\tJuan Quintela , QEMU ,\n\tMarkus Armbruster ,\n\t\"Dr. David Alan Gilbert\" ,\n\tStefan Hajnoczi ,\n\tPaolo Bonzini ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}},{"id":1770778,"web_url":"http://patchwork.ozlabs.org/comment/1770778/","msgid":"<20170919091352.GC2107@work-vm>","list_archive_url":null,"date":"2017-09-19T09:13:52","subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","submitter":{"id":48102,"url":"http://patchwork.ozlabs.org/api/people/48102/","name":"Dr. David Alan Gilbert","email":"dgilbert@redhat.com"},"content":"* Peter Xu (peterx@redhat.com) wrote:\n> On Mon, Sep 18, 2017 at 11:40:40AM +0100, Dr. David Alan Gilbert wrote:\n> > * Peter Xu (peterx@redhat.com) wrote:\n> > > On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:\n> > > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:\n> > > > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:\n> > > > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:\n> > > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:\n> > > > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:\n> > > > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:\n> > > > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:\n> > > > > > > > > > > > > > There should be a limit in the number of requests the thread can\n> > > > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket\n> > > > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if\n> > > > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.\n> > > > > > > > > > > > > \n> > > > > > > > > > > > > I agree.\n> > > > > > > > > > > > > \n> > > > > > > > > > > > > Memory usage must be bounded. The number of requests is less important\n> > > > > > > > > > > > > than the amount of memory consumed by them.\n> > > > > > > > > > > > > \n> > > > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for\n> > > > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be\n> > > > > > > > > > > > > processed if queued non-OOB commands consume too much memory.\n> > > > > > > > > > > > \n> > > > > > > > > > > > Thanks for pointing out this. Yes the memory usage problem is valid,\n> > > > > > > > > > > > as Markus pointed out as well in previous discussions (in \"Flow\n> > > > > > > > > > > > Control\" section of that long reply). Hopefully this series basically\n> > > > > > > > > > > > can work from design prospective, then I'll add this flow control in\n> > > > > > > > > > > > next version.\n> > > > > > > > > > > > \n> > > > > > > > > > > > Regarding to what we should do if the limit is reached: Markus\n> > > > > > > > > > > > provided a few options, but the one I prefer most is that we don't\n> > > > > > > > > > > > respond, but send an event showing that a command is dropped.\n> > > > > > > > > > > > However, I would like it not queued, but a direct reply (after all,\n> > > > > > > > > > > > it's an event, and we should not need to care much on ordering of it).\n> > > > > > > > > > > > Then we can get rid of the babysitting of those \"to be failed\"\n> > > > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.\n> > > > > > > > > > > > \n> > > > > > > > > > > > I think I also missed at least a unit test for this new interface.\n> > > > > > > > > > > > Again, I'll add it after the whole idea is proved solid. Thanks,\n> > > > > > > > > > > \n> > > > > > > > > > > Another solution: the server reports available receive buffer space to\n> > > > > > > > > > > the client. The server only guarantees immediate OOB processing when\n> > > > > > > > > > > the client stays within the receive buffer size.\n> > > > > > > > > > > \n> > > > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer\n> > > > > > > > > > > size and make sure to leave enough room.\n> > > > > > > > > > \n> > > > > > > > > > I don't think having to query it ahead of time is particularly nice,\n> > > > > > > > > > and of course it is inherantly racy.\n> > > > > > > > > > \n> > > > > > > > > > I would just have QEMU emit an event when it pausing processing of the\n> > > > > > > > > > incoming commands due to a full queue. If the event includes the ID\n> > > > > > > > > > of the last queued command, the client will know which (if any) of\n> > > > > > > > > > its outstanding commands are delayed. Another even can be sent when\n> > > > > > > > > > it restarts reading.\n> > > > > > > > > \n> > > > > > > > > Hmm and now we're implementing flow control!\n> > > > > > > > > \n> > > > > > > > > a) What exactly is the current semantics/buffer sizes?\n> > > > > > > > > b) When do clients send multiple QMP commands on one channel without\n> > > > > > > > > waiting for the response to the previous command?\n> > > > > > > > > c) Would one queue entry for each class of commands/channel work\n> > > > > > > > > (Where a class of commands is currently 'normal' and 'oob')\n> > > > > > > > \n> > > > > > > > I do wonder if we need to worry about request limiting at all from the\n> > > > > > > > client side. For non-OOB commands clients will wait for a reply before\n> > > > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.\n> > > > > > > > \n> > > > > > > > OOB commands are supposed to be things which can be handled quickly\n> > > > > > > > without blocking, so even if a client sent several commands at once\n> > > > > > > > without waiting for replies, they're going to be processed quickly,\n> > > > > > > > so whether we temporarily block reading off the wire is a minor\n> > > > > > > > detail.\n> > > > > > > \n> > > > > > > Lets just define it so that it can't - you send an OOB command and wait\n> > > > > > > for it's response before sending another on that channel.\n> > > > > > > \n> > > > > > > > IOW, I think we could just have a fixed 10 command queue and apps just\n> > > > > > > > pretend that there's an infinite queue and nothing bad would happen from\n> > > > > > > > the app's POV.\n> > > > > > > \n> > > > > > > Can you justify 10 as opposed to 1?\n> > > > > > \n> > > > > > Semantically I don't think it makes a difference if the OOB commands are\n> > > > > > being processed sequentially by their thread. A >1 length queue would only\n> > > > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB\n> > > > > > requests, as then that could block reading of OOB commands. \n> > > > > \n> > > > > To summarize:\n> > > > > \n> > > > > The QMP server has a lookahead of 1 command so it can dispatch\n> > > > > out-of-band commands. If 2 or more non-OOB commands are queued at the\n> > > > > same time then OOB processing will not occur.\n> > > > > \n> > > > > Is that right?\n> > > > \n> > > > I think my view is slightly more complex;\n> > > > a) There's a pair of queues for each channel\n> > > > b) There's a central pair of queues on the QMP server\n> > > > one for OOB commands and one for normal commands.\n> > > > c) Each queue is only really guaranteed to be one deep.\n> > > > \n> > > > That means that each one of the channels can send a non-OOB\n> > > > command without getting in the way of a channel that wants\n> > > > to send one.\n> > > \n> > > But current version should not be that complex:\n> > > \n> > > Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.\n> > > \n> > > Then, we only have a single global queue for QMP non-oob commands, and\n> > > we don't have response queue yet. We do respond just like before in a\n> > > synchronous way (I explained why - for OOB we don't need that\n> > > complexity IMHO).\n> > \n> > I think the discussion started because of two related comments:\n> > Marc-André said :\n> > 'There should be a limit in the number of requests the thread can\n> > queue'\n> > and Stefan said :\n> > 'Memory usage must be bounded.'\n> > \n> > actually neither of those cases really worried me (because they only\n> > happen if the client keeps pumping commands, and that seems it's fault).\n> > \n> > However, once you start adding a limit, you've got to be careful - if\n> > you just added a limit to the central queue, then what happens if that\n> > queue is filled by non-OOB commands?\n> \n> Ah! So I misunderstood \"a pair of queues for each channel\". I\n> thought it means the input and output of a single monitor, while I\n> think it actually means \"OOB channel\" and \"non-OOB channel\".\n> \n> My plan (or say, this version) starts from only one global queue for\n> non-OOB commands. There is no queue for OOB commands at all. As\n> discussed below [1], if we receive one OOB command, we execute it\n> directly and reply to client. And here the \"request queue\" will only\n> queue non-OOB commands. Maybe the name \"request queue\" sounds\n> confusing here.\n> \n> If so, we should not have above problem, right? Since even if the\n> queue is full (of course there will only be non-OOB commands in the\n> queue), the parsing is still working, and we will still be able to\n> handle OOB ones:\n> \n> req = parse(stream);\n> \n> if (is_oob(req)) {\n> execute(req);\n> return;\n> }\n> \n> if (queue_full(req_queue)) {\n> emit_full_event(req);\n> return;\n> }\n> \n> enqueue(req_queue, req);\n> \n> So again, this version is a simplified version from previous\n> discussion (no oob-queue but only non-oob-queue, no respond queue but\n> only request queue, etc...), but hope it can work.\n\nThat might work. You have to be really careful about allowing\nOOB commands to be parsed, even if the non-OOB queue is full.\n\nOne problem is that one client could fill up that shared queue,\nthen another client would be surprised to find the queue is full\nwhen it tries to send just one command - hence why I thought a separate\nqueue per client would solve that.\n\nDave\n\n> Thanks,\n> \n> > \n> > Dave\n> > \n> > > When we parse commands, we execute it directly if OOB, otherwise we\n> > > put it onto request queue. Request queue handling is done by a main\n> > > thread QEMUBH. That's all.\n> \n> [1]\n> \n> > > \n> > > Would this \"simple version\" suffice to implement this whole OOB idea?\n> > > \n> > > (Again, I really don't think we need to specify queue length to 1,\n> > > though we can make it small)\n> > > \n> > > -- \n> > > Peter Xu\n> > --\n> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK\n> \n> -- \n> Peter Xu\n--\nDr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=dgilbert@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xxHJQ0N67z9sBZ\n\tfor ;\n\tTue, 19 Sep 2017 19:14:46 +1000 (AEST)","from localhost ([::1]:40940 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1duEc8-0003SB-5S\n\tfor incoming@patchwork.ozlabs.org; Tue, 19 Sep 2017 05:14:44 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:48614)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1duEbd-0003O9-Jn\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 05:14:16 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1duEbY-0008Bd-Id\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 05:14:13 -0400","from mx1.redhat.com ([209.132.183.28]:43128)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1duEbY-0008BO-8p\n\tfor qemu-devel@nongnu.org; Tue, 19 Sep 2017 05:14:08 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 38728C047B6A;\n\tTue, 19 Sep 2017 09:14:07 +0000 (UTC)","from work-vm (ovpn-117-120.ams2.redhat.com [10.36.117.120])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 3900F6060A;\n\tTue, 19 Sep 2017 09:13:55 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 38728C047B6A","Date":"Tue, 19 Sep 2017 10:13:52 +0100","From":"\"Dr. David Alan Gilbert\" ","To":"Peter Xu ","Message-ID":"<20170919091352.GC2107@work-vm>","References":"<20170915113428.GF13610@redhat.com>\n\t<20170915120643.GN2170@work-vm>\n\t<20170915121433.GI13610@redhat.com> <20170915121956.GO2170@work-vm>\n\t<20170915122913.GJ13610@redhat.com>\n\t<20170915145632.GD18767@stefanha-x1.localdomain>\n\t<20170915151706.GQ2170@work-vm>\n\t<20170918092625.GE3617@pxdev.xzpeter.org>\n\t<20170918104039.GC2581@work-vm>\n\t<20170919022312.GG3617@pxdev.xzpeter.org>","MIME-Version":"1.0","Content-Type":"text/plain; charset=iso-8859-1","Content-Disposition":"inline","In-Reply-To":"<20170919022312.GG3617@pxdev.xzpeter.org>","User-Agent":"Mutt/1.8.3 (2017-05-23)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tTue, 19 Sep 2017 09:14:07 +0000 (UTC)","Content-Transfer-Encoding":"quoted-printable","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution\n\tsupport","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier , Juan Quintela ,\n\tMarkus Armbruster , QEMU ,\n\tMichael Roth , =?iso-8859-1?q?Marc-Andr=E9?=\n\tLureau , Stefan Hajnoczi\n\t, Paolo Bonzini ,\n\tFam Zheng , \tStefan Hajnoczi ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"}}]