From patchwork Fri Dec 1 19:14:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Adri=C3=A1n_Moreno?= X-Patchwork-Id: 1870794 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSb3Pc3z1ySh for ; Sat, 2 Dec 2023 06:15:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id CAB4C42443; Fri, 1 Dec 2023 19:15:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org CAB4C42443 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KXVjbd9-G6ux; Fri, 1 Dec 2023 19:15:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 671134226C; Fri, 1 Dec 2023 19:15:15 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 671134226C Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 530EBC0DEC; Fri, 1 Dec 2023 19:15:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id EB5D9C0DDD for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B490C60D78 for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B490C60D78 Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O5pjYLb2pPEd for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3A6A360BAE for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 3A6A360BAE DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ohyxtl1kJj4/RctmVmprtuQXN3Uf2L2CbwCL7nV5Lqk=; b=WDoEnvparulJL9mnxyofd7cwCHrrYtTcwKxuH+6c+TjUJ+qiXOnF24HQ/sfqMKIT4ZrrD+ OXYPxK3qoQvHUd0nYWH+JH7tbw9MsIXTvL7CkHnZ1SaLlRzr1jlMCbkMNxkI1L9Ho/+RbA E8IiFCB5+ogzOYle0qOGo5VmBs6jowM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-434-FqNNHoACNUitfX1hUVKC2g-1; Fri, 01 Dec 2023 14:14:56 -0500 X-MC-Unique: FqNNHoACNUitfX1hUVKC2g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8F906101A590 for ; Fri, 1 Dec 2023 19:14:56 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 078D51121307; Fri, 1 Dec 2023 19:14:55 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:38 +0100 Message-ID: <20231201191449.2386134-3-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 02/10] python: ovs: flowviz: Add file processing infra. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" process.py contains a useful base class that processes files odp.py and ofp.py: contain datapath and openflow subcommand definitions as well as the first formatting option: json. Also, this patch adds basic filtering support. Examples: $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow json $ ovs-ofctl dump-flows br-int > flows.txt && ovs-flowviz -i flows.txt openflow json $ ovs-ofctl appctl dpctl/dump-flows | ovs-flowviz -f 'ct' datapath json $ ovs-ofctl appctl dpctl/dump-flows > flows.txt && ovs-flowviz -i flows.txt -f 'drop' datapath json Signed-off-by: Adrian Moreno --- python/automake.mk | 5 +- python/ovs/flowviz/__init__.py | 2 + python/ovs/flowviz/main.py | 103 +++++++++++++++++- python/ovs/flowviz/odp/cli.py | 42 ++++++++ python/ovs/flowviz/ofp/cli.py | 42 ++++++++ python/ovs/flowviz/process.py | 192 +++++++++++++++++++++++++++++++++ 6 files changed, 384 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/odp/cli.py create mode 100644 python/ovs/flowviz/ofp/cli.py create mode 100644 python/ovs/flowviz/process.py diff --git a/python/automake.mk b/python/automake.mk index 4302f0136..4845565b8 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -67,8 +67,11 @@ ovs_flowviz = \ python/ovs/flowviz/__init__.py \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ + python/ovs/flowviz/odp/cli.py \ python/ovs/flowviz/ofp/__init__.py \ - python/ovs/flowviz/ovs-flowviz + python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ovs-flowviz \ + python/ovs/flowviz/process.py # These python files are used at build time but not runtime, diff --git a/python/ovs/flowviz/__init__.py b/python/ovs/flowviz/__init__.py index e69de29bb..898dba522 100644 --- a/python/ovs/flowviz/__init__.py +++ b/python/ovs/flowviz/__init__.py @@ -0,0 +1,2 @@ +import ovs.flowviz.ofp.cli # noqa: F401 +import ovs.flowviz.odp.cli # noqa: F401 diff --git a/python/ovs/flowviz/main.py b/python/ovs/flowviz/main.py index a2d5ca1fa..a45c06e48 100644 --- a/python/ovs/flowviz/main.py +++ b/python/ovs/flowviz/main.py @@ -12,19 +12,67 @@ # See the License for the specific language governing permissions and # limitations under the License. +import os + import click +from ovs.flow.filter import OFFilter + class Options(dict): """Options dictionary""" +def validate_input(ctx, param, value): + """Validate the "-i" option""" + result = list() + for input_str in value: + parts = input_str.strip().split(",") + if len(parts) == 2: + file_parts = tuple(parts) + elif len(parts) == 1: + file_parts = tuple(["Filename: " + parts[0], parts[0]]) + else: + raise click.BadParameter( + "input filename should have the following format: " + "[alias,]FILENAME" + ) + + if not os.path.isfile(file_parts[1]): + raise click.BadParameter( + "input filename %s does not exist" % file_parts[1] + ) + result.append(file_parts) + return result + + @click.group( subcommand_metavar="TYPE", context_settings=dict(help_option_names=["-h", "--help"]), ) +@click.option( + "-i", + "--input", + "filename", + help="Read flows from specified filepath. If not provided, flows will be" + " read from stdin. This option can be specified multiple times." + " Format [alias,]FILENAME. Where alias is a name that shall be used to" + " refer to this FILENAME", + multiple=True, + type=click.Path(), + callback=validate_input, +) +@click.option( + "-f", + "--filter", + help="Filter flows that match the filter expression." + "Run 'ovs-flowviz filter' for a detailed description of the filtering " + "syntax", + type=str, + show_default=False, +) @click.pass_context -def maincli(ctx): +def maincli(ctx, filename, filter): """ OpenvSwitch flow visualization utility. @@ -32,6 +80,59 @@ def maincli(ctx): (such as the output of ovs-ofctl dump-flows or ovs-appctl dpctl/dump-flows) and prints them in different formats. """ + ctx.obj = Options() + ctx.obj["filename"] = filename or None + if filter: + try: + ctx.obj["filter"] = OFFilter(filter) + except Exception as e: + raise click.BadParameter("Wrong filter syntax: {}".format(e)) + + +@maincli.command(hidden=True) +@click.pass_context +def filter(ctx): + """ + \b + Filter Syntax + ************* + + [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ... + + \b + Comparison operators are: + = equality + < less than + > more than + ~= masking (valid for IP and Ethernet fields) + + \b + Logical operators are: + !{expr}: NOT + {expr} && {expr}: AND + {expr} || {expr}: OR + + \b + Matches and flow metadata: + To compare against a match or info field, use the field directly, e.g: + priority=100 + n_bytes>10 + Use simple keywords for flags: + tcp and ip_src=192.168.1.1 + \b + Actions: + Actions values might be dictionaries, use subkeys to access individual + values, e.g: + output.port=3 + Use simple keywords for flags + drop + + \b + Examples of valid filters. + nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443) + arp=true && !arp_tsa=192.168.1.1 + n_bytes>0 && drop=true""" + click.echo(ctx.command.get_help(ctx)) def main(): diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py new file mode 100644 index 000000000..ed2f82065 --- /dev/null +++ b/python/ovs/flowviz/odp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + DatapathFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def datapath(opts): + """Process Datapath Flows.""" + pass + + +class JSONPrint(DatapathFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@datapath.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py new file mode 100644 index 000000000..b9a2a8aad --- /dev/null +++ b/python/ovs/flowviz/ofp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + OpenFlowFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def openflow(opts): + """Process OpenFlow Flows.""" + pass + + +class JSONPrint(OpenFlowFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@openflow.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/process.py b/python/ovs/flowviz/process.py new file mode 100644 index 000000000..413506bf2 --- /dev/null +++ b/python/ovs/flowviz/process.py @@ -0,0 +1,192 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +import json +import click + +from ovs.flow.decoders import FlowEncoder +from ovs.flow.odp import ODPFlow +from ovs.flow.ofp import OFPFlow + + +class FileProcessor(object): + """Base class for file-based Flow processing. It is able to create flows + from strings found in a file (or stdin). + + The process of parsing the flows is extendable in many ways by deriving + this class. + + When process() is called, the base class will: + - call self.start_file() for each new file that get's processed + - call self.create_flow() for each flow line + - apply the filter defined in opts if provided (can be optionally + disabled) + - call self.process_flow() for after the flow has been filtered + - call self.stop_file() after the file has been processed entirely + + In the case of stdin, the filename and file alias is 'stdin'. + + Child classes must at least implement create_flow() and process_flow() + functions. + + Args: + opts (dict): Options dictionary + """ + + def __init__(self, opts): + self.opts = opts + + # Methods that must be implemented by derived classes + def init(self): + """Called before the flow processing begins.""" + pass + + def start_file(self, alias, filename): + """Called before the processing of a file begins. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def create_flow(self, line, idx): + """Called for each line in the file. + Args: + line(str): The flow line + idx(int): The line index + + Returns a Flow. + Must be implemented by child classes. + """ + raise NotImplementedError + + def process_flow(self, flow, name): + """Called for built flow (after filtering). + Args: + flow(Flow): The flow created by create_flow + name(str): The name of the file from which the flow comes + """ + raise NotImplementedError + + def stop_file(self, alias, filename): + """Called after the processing of a file ends. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def end(self): + """Called after the processing ends.""" + pass + + def process(self, do_filter=True): + idx = 0 + filenames = self.opts.get("filename") + filt = self.opts.get("filter") if do_filter else None + self.init() + if filenames: + for alias, filename in filenames: + try: + with open(filename) as f: + self.start_file(alias, filename) + for line in f: + flow = self.create_flow(line, idx) + idx += 1 + if not flow or (filt and not filt.evaluate(flow)): + continue + self.process_flow(flow, alias) + self.stop_file(alias, filename) + except IOError as e: + raise click.BadParameter( + "Failed to read from file {} ({}): {}".format( + filename, e.errno, e.strerror + ) + ) + else: + data = sys.stdin.read() + self.start_file("stdin", "stdin") + for line in data.split("\n"): + line = line.strip() + if line: + flow = self.create_flow(line, idx) + idx += 1 + if ( + not flow + or not getattr(flow, "_sections", None) + or (filt and not filt.evaluate(flow)) + ): + continue + self.process_flow(flow, "stdin") + self.stop_file("stdin", "stdin") + self.end() + + +class DatapathFactory(): + """A mixin class that creates OpenFlow flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in Datapath flow dumps. + if any(s in line for s in [ + "flow-dump from the main thread", + "flow-dump from pmd on core", + ]): + return None + + return ODPFlow(line, idx) + + +class OpenFlowFactory(): + """A mixin class that creates Datapath flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in OpenFlow flow dumps. + if " reply " in line: + return None + + return OFPFlow(line, idx) + + +class JSONProcessor(FileProcessor): + """A FileProcessor that prints flows in JSON format.""" + + def __init__(self, opts): + super().__init__(opts) + self.flows = dict() + + def start_file(self, name, filename): + self.flows_list = list() + + def stop_file(self, name, filename): + self.flows[name] = self.flows_list + + def process_flow(self, flow, name): + self.flows_list.append(flow) + + def json_string(self): + if len(self.flows.keys()) > 1: + return json.dumps( + [ + {"name": name, "flows": [flow.dict() for flow in flows]} + for name, flows in self.flows.items() + ], + indent=4, + cls=FlowEncoder, + ) + return json.dumps( + [flow.dict() for flow in self.flows_list], + indent=4, + cls=FlowEncoder, + )