[ovs-dev,v7,2/6] python: Add global option for JSON output to Python tools.

From: Jakob Meng <code@jakobmeng.de>

This patch introduces support for different output formats to the
Python code, as did the previous commit for ovs-xxx tools like
'ovs-appctl --format json dpif/show'.
In particular, tests/appctl.py gains a global option '-f,--format'
which allows users to request JSON instead of plain-text for humans.

This patch does not yet pass the choosen output format to commands.
Doing so requires changes to all command_register() calls and all
command callbacks. To improve readibility those changes have been
split out into a follow up patch. Respectively, whenever an output
format other than 'text' is choosen for tests/appctl.py, the script
will fail.

Reported-at: https://bugzilla.redhat.com/1824861
Signed-off-by: Jakob Meng <code@jakobmeng.de>
---
 NEWS                           |  2 ++
 python/ovs/unixctl/__init__.py | 39 +++++++++++++++++++++++++++++-----
 python/ovs/unixctl/server.py   | 37 +++++++++++++++++++++++++++++++-
 python/ovs/util.py             |  8 +++++++
 tests/appctl.py                | 17 +++++++++++++++
 tests/unixctl-py.at            |  1 +
 6 files changed, 98 insertions(+), 6 deletions(-)

> From: Jakob Meng <code@jakobmeng.de>
>
> This patch introduces support for different output formats to the
> Python code, as did the previous commit for ovs-xxx tools like
> 'ovs-appctl --format json dpif/show'.
> In particular, tests/appctl.py gains a global option '-f,--format'
> which allows users to request JSON instead of plain-text for humans.
>
> This patch does not yet pass the choosen output format to commands.
> Doing so requires changes to all command_register() calls and all
> command callbacks. To improve readibility those changes have been
> split out into a follow up patch. Respectively, whenever an output
> format other than 'text' is choosen for tests/appctl.py, the script
> will fail.
>
> Reported-at: https://bugzilla.redhat.com/1824861
> Signed-off-by: Jakob Meng <code@jakobmeng.de>

Hi Jakob,

See some comments below.

//Eelco

> ---
>  NEWS                           |  2 ++
>  python/ovs/unixctl/__init__.py | 39 +++++++++++++++++++++++++++++-----
>  python/ovs/unixctl/server.py   | 37 +++++++++++++++++++++++++++++++-
>  python/ovs/util.py             |  8 +++++++
>  tests/appctl.py                | 17 +++++++++++++++
>  tests/unixctl-py.at            |  1 +
>  6 files changed, 98 insertions(+), 6 deletions(-)
>
> diff --git a/NEWS b/NEWS
> index 8631ea45e..94a347246 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -35,6 +35,8 @@ v3.3.0 - xx xxx xxxx
>         on mark and labels.
>       * Added new option [-f|--format] to choose the output format, e.g. 'json'
>         or 'text' (by default).
> +   - Python:
> +     * Added support for choosing the output format, e.g. 'json' or 'text'.
>     - ovs-vsctl:
>       * New commands 'set-zone-limit', 'del-zone-limit' and 'list-zone-limits'
>         to manage the maximum number of connections in conntrack zones via
> diff --git a/python/ovs/unixctl/__init__.py b/python/ovs/unixctl/__init__.py
> index 8ee312943..6d9bd5715 100644
> --- a/python/ovs/unixctl/__init__.py
> +++ b/python/ovs/unixctl/__init__.py
> @@ -20,10 +20,14 @@ commands = {}
>
>
>  class _UnixctlCommand(object):
> -    def __init__(self, usage, min_args, max_args, callback, aux):
> +    # FIXME: Output format will be passed as 'output_fmts' to the command in
> +    #        later patch.
> +    def __init__(self, usage, min_args, max_args,  # FIXME: output_fmts,
> +                 callback, aux):
>          self.usage = usage
>          self.min_args = min_args
>          self.max_args = max_args
> +        # FIXME: self.output_fmts = output_fmts
>          self.callback = callback
>          self.aux = aux
>
> @@ -42,10 +46,13 @@ def _unixctl_help(conn, unused_argv, unused_aux):
>      conn.reply(reply)
>
>
> -def command_register(name, usage, min_args, max_args, callback, aux):
> +def command_register_fmt(name, usage, min_args, max_args, output_fmts,
> +                         callback, aux):
>      """ Registers a command with the given 'name' to be exposed by the
>      UnixctlServer. 'usage' describes the arguments to the command; it is used
> -    only for presentation to the user in "help" output.
> +    only for presentation to the user in "help" output.  'output_fmts' is a
> +    bitmap that defines what output formats a command supports, e.g.
> +    OVS_OUTPUT_FMT_TEXT | OVS_OUTPUT_FMT_JSON.
>
>      'callback' is called when the command is received.  It is passed a
>      UnixctlConnection object, the list of arguments as unicode strings, and
> @@ -63,8 +70,30 @@ def command_register(name, usage, min_args, max_args, callback, aux):
>      assert callable(callback)
>
>      if name not in commands:
> -        commands[name] = _UnixctlCommand(usage, min_args, max_args, callback,
> -                                         aux)
> +        commands[name] = _UnixctlCommand(usage, min_args, max_args,
> +                                         # FIXME: output_fmts,
> +                                         callback, aux)
> +
> +
> +# FIXME: command_register() will be replaced with command_register_fmt() in a
> +#        later patch of this series. It is is kept temporarily to reduce the
> +#        amount of changes in this patch.
> +def command_register(name, usage, min_args, max_args, callback, aux):
> +    """ Registers a command with the given 'name' to be exposed by the
> +    UnixctlServer. 'usage' describes the arguments to the command; it is used
> +    only for presentation to the user in "help" output.
> +
> +    'callback' is called when the command is received.  It is passed a
> +    UnixctlConnection object, the list of arguments as unicode strings, and
> +    'aux'.  Normally 'callback' should reply by calling
> +    UnixctlConnection.reply() or UnixctlConnection.reply_error() before it
> +    returns, but if the command cannot be handled immediately, then it can
> +    defer the reply until later.  A given connection can only process a single
> +    request at a time, so a reply must be made eventually to avoid blocking
> +    that connection."""
> +
> +    command_register_fmt(name, usage, min_args, max_args,
> +                         ovs.util.OutputFormat.TEXT, callback, aux)
>
>
>  def socket_name_from_target(target):
> diff --git a/python/ovs/unixctl/server.py b/python/ovs/unixctl/server.py
> index b9cb52fad..1d7f3337d 100644
> --- a/python/ovs/unixctl/server.py
> +++ b/python/ovs/unixctl/server.py
> @@ -12,6 +12,7 @@
>  # See the License for the specific language governing permissions and
>  # limitations under the License.
>
> +import argparse
>  import copy
>  import errno
>  import os
> @@ -35,6 +36,7 @@ class UnixctlConnection(object):
>          assert isinstance(rpc, ovs.jsonrpc.Connection)
>          self._rpc = rpc
>          self._request_id = None
> +        self._fmt = ovs.util.OutputFormat.TEXT
>
>      def run(self):
>          self._rpc.run()
> @@ -116,6 +118,17 @@ class UnixctlConnection(object):
>          elif len(params) > command.max_args:
>              error = '"%s" command takes at most %d arguments' \
>                      % (method, command.max_args)
> +        # FIXME: Uncomment when command.output_fmts is available
> +        # elif self._fmt not in command.output_fmts:
> +        #     error = '"%s" command does not support output format' \
> +        #             ' "%s" (supported: %d, requested: %d)' \
> +        #             % (method, self._fmt.name.lower(), command.output_fmts,
> +        #                self._fmt)
> +        # FIXME: Remove this check once output format will be passed to the
> +        #        command handler below.
> +        elif self._fmt != ovs.util.OutputFormat.TEXT:
> +            error = 'output format "%s" has not been implemented yet' \
> +                    % self._fmt.name.lower()
>          else:
>              for param in params:
>                  if not isinstance(param, str):
> @@ -124,7 +137,8 @@ class UnixctlConnection(object):
>
>              if error is None:
>                  unicode_params = [str(p) for p in params]
> -                command.callback(self, unicode_params, command.aux)
> +                command.callback(self, unicode_params,  # FIXME: self._fmt,
> +                                 command.aux)
>
>          if error:
>              self.reply_error(error)
> @@ -136,6 +150,24 @@ def _unixctl_version(conn, unused_argv, version):
>      conn.reply(version)
>
>
> +def _unixctl_set_options(conn, argv, unused_aux):
> +    assert isinstance(conn, UnixctlConnection)
> +
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument("--format", default="text",
> +                        choices=[fmt.name.lower()
> +                                 for fmt in ovs.util.OutputFormat])
> +
> +    try:
> +        args = parser.parse_args(args=argv)
> +    except argparse.ArgumentError as e:
> +        conn.reply_error(str(e))
> +        return
> +
> +    conn._fmt = ovs.util.OutputFormat[args.format.upper()]
> +    conn.reply(None)
> +
> +
>  class UnixctlServer(object):
>      def __init__(self, listener):
>          assert isinstance(listener, ovs.stream.PassiveStream)
> @@ -210,4 +242,7 @@ class UnixctlServer(object):
>          ovs.unixctl.command_register("version", "", 0, 0, _unixctl_version,
>                                       version)
>
> +        ovs.unixctl.command_register("set-options", "[--format text|json]", 2,
> +                                     2, _unixctl_set_options, None)
> +
>          return 0, UnixctlServer(listener)
> diff --git a/python/ovs/util.py b/python/ovs/util.py
> index 3dba022f8..272ca683d 100644
> --- a/python/ovs/util.py
> +++ b/python/ovs/util.py
> @@ -15,11 +15,19 @@
>  import os
>  import os.path
>  import sys
> +import enum
>
>  PROGRAM_NAME = os.path.basename(sys.argv[0])
>  EOF = -1
>
>
> +@enum.unique
> +# FIXME: Use @enum.verify(enum.NAMED_FLAGS) from Python 3.11 when available.
> +class OutputFormat(enum.IntFlag):
> +    TEXT = 1 << 0
> +    JSON = 1 << 1
> +
> +
>  def abs_file_name(dir_, file_name):
>      """If 'file_name' starts with '/', returns a copy of 'file_name'.
>      Otherwise, returns an absolute path to 'file_name' considering it relative
> diff --git a/tests/appctl.py b/tests/appctl.py
> index b85b364fa..4e8c4adc0 100644
> --- a/tests/appctl.py
> +++ b/tests/appctl.py
> @@ -49,13 +49,30 @@ def main():
>                          help="Arguments to the command.")
>      parser.add_argument("-T", "--timeout", metavar="SECS",
>                          help="wait at most SECS seconds for a response")
> +    parser.add_argument("-f", "--format", metavar="FMT",
> +                        help="Output format.", default="text",
> +                        choices=[fmt.name.lower()
> +                                 for fmt in ovs.util.OutputFormat])
>      args = parser.parse_args()
>
>      signal_alarm(int(args.timeout) if args.timeout else None)
>
>      ovs.vlog.Vlog.init()
>      target = args.target
> +    format = ovs.util.OutputFormat[args.format.upper()]
>      client = connect_to_target(target)
> +
> +    if format != ovs.util.OutputFormat.TEXT:
> +        err_no, error, _ = client.transact(
> +            "set-options", ["--format", args.format])
> +
> +        if err_no:
> +            ovs.util.ovs_fatal(err_no, "%s: transaction error" % target)
> +        elif error is not None:

I see this is a copy from below, but is the 'elif' ok here? Meaning can we not
get both errors at the same time? The C code catches both.

Also 'if err_no' we will still try the second transaction for the command.
I think we should exit even in this case.

> +            sys.stderr.write(error)
> +            ovs.util.ovs_error(0, "%s: server returned an error" % target)
> +            sys.exit(2)
> +
>      err_no, error, result = client.transact(args.command, args.argv)
>      client.close()
>
> diff --git a/tests/unixctl-py.at b/tests/unixctl-py.at
> index 724006118..26c137047 100644
> --- a/tests/unixctl-py.at
> +++ b/tests/unixctl-py.at
> @@ -100,6 +100,7 @@ The available commands are:
>    exit
>    help
>    log                     [[arg ...]]
> +  set-options             [[--format text|json]]
>    version
>    vlog/close
>    vlog/list
> -- 
> 2.39.2

On 15.03.24 11:16, Eelco Chaudron wrote:
> Hi Jakob,
>
> See some comments below.
>
> //Eelco

Thanks and comments below again 😄

Cheers,
Jakob

>> ...
>> --- a/tests/appctl.py
>> +++ b/tests/appctl.py
>> @@ -49,13 +49,30 @@ def main():
>>                          help="Arguments to the command.")
>>      parser.add_argument("-T", "--timeout", metavar="SECS",
>>                          help="wait at most SECS seconds for a response")
>> +    parser.add_argument("-f", "--format", metavar="FMT",
>> +                        help="Output format.", default="text",
>> +                        choices=[fmt.name.lower()
>> +                                 for fmt in ovs.util.OutputFormat])
>>      args = parser.parse_args()
>>
>>      signal_alarm(int(args.timeout) if args.timeout else None)
>>
>>      ovs.vlog.Vlog.init()
>>      target = args.target
>> +    format = ovs.util.OutputFormat[args.format.upper()]
>>      client = connect_to_target(target)
>> +
>> +    if format != ovs.util.OutputFormat.TEXT:
>> +        err_no, error, _ = client.transact(
>> +            "set-options", ["--format", args.format])
>> +
>> +        if err_no:
>> +            ovs.util.ovs_fatal(err_no, "%s: transaction error" % target)
>> +        elif error is not None:
> I see this is a copy from below, but is the 'elif' ok here? Meaning can we not
> get both errors at the same time? The C code catches both.
>
> Also 'if err_no' we will still try the second transaction for the command.
> I think we should exit even in this case.

When we hit a transaction error ('if err_no'), the ovs.util.ovs_fatal() call will sys.exit(1), so the elif branch will not be executed nor the second command transaction?!

What did I miss?🙈

>> +            sys.stderr.write(error)
>> +            ovs.util.ovs_error(0, "%s: server returned an error" % target)
>> +            sys.exit(2)
>> +
>>      err_no, error, result = client.transact(args.command, args.argv)
>>      client.close()
>>

On 19 Mar 2024, at 11:24, Jakob Meng wrote:

> On 15.03.24 11:16, Eelco Chaudron wrote:
>> Hi Jakob,
>>
>> See some comments below.
>>
>> //Eelco
>
> Thanks and comments below again 😄
>
> Cheers,
> Jakob
>
>>> ...
>>> --- a/tests/appctl.py
>>> +++ b/tests/appctl.py
>>> @@ -49,13 +49,30 @@ def main():
>>>                          help="Arguments to the command.")
>>>      parser.add_argument("-T", "--timeout", metavar="SECS",
>>>                          help="wait at most SECS seconds for a response")
>>> +    parser.add_argument("-f", "--format", metavar="FMT",
>>> +                        help="Output format.", default="text",
>>> +                        choices=[fmt.name.lower()
>>> +                                 for fmt in ovs.util.OutputFormat])
>>>      args = parser.parse_args()
>>>
>>>      signal_alarm(int(args.timeout) if args.timeout else None)
>>>
>>>      ovs.vlog.Vlog.init()
>>>      target = args.target
>>> +    format = ovs.util.OutputFormat[args.format.upper()]
>>>      client = connect_to_target(target)
>>> +
>>> +    if format != ovs.util.OutputFormat.TEXT:
>>> +        err_no, error, _ = client.transact(
>>> +            "set-options", ["--format", args.format])
>>> +
>>> +        if err_no:
>>> +            ovs.util.ovs_fatal(err_no, "%s: transaction error" % target)
>>> +        elif error is not None:
>> I see this is a copy from below, but is the 'elif' ok here? Meaning can we not
>> get both errors at the same time? The C code catches both.
>>
>> Also 'if err_no' we will still try the second transaction for the command.
>> I think we should exit even in this case.
>
> When we hit a transaction error ('if err_no'), the ovs.util.ovs_fatal() call will sys.exit(1), so the elif branch will not be executed nor the second command transaction?!
>
> What did I miss?🙈

You missed nothing :) It was me missign the fatal part :(

Acked-by: Eelco Chaudron <echaudro@redhat.com>

>>> +            sys.stderr.write(error)
>>> +            ovs.util.ovs_error(0, "%s: server returned an error" % target)
>>> +            sys.exit(2)
>>> +
>>>      err_no, error, result = client.transact(args.command, args.argv)
>>>      client.close()
>>>