From patchwork Fri Dec 1 19:14:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870792 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AioT9QPR; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSR5sVSz1ySh for ; Sat, 2 Dec 2023 06:15:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 549506FB0F; Fri, 1 Dec 2023 19:15:15 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 549506FB0F Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AioT9QPR X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A-vJHgvN3k14; Fri, 1 Dec 2023 19:15:13 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id E1F4A6FB1D; Fri, 1 Dec 2023 19:15:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org E1F4A6FB1D Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11273C0DE2; Fri, 1 Dec 2023 19:15:08 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2A014C0DDA for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id EBC4743758 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org EBC4743758 Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AioT9QPR X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SIQ_4_P1nmUD for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6C97B4370E for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6C97B4370E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iEWPHeCnnrHlEknywFSaTGr+dQ4sEfsPeaZqQTbzKds=; b=AioT9QPRDv78HK7uk6Ac0IUrn2ZZJFMGvdxXf5epffkYmdDdmqgGca/Zyd6daXxnY39/o3 EAPKklMgPIM63HjablZv9SsR9DJW0I+4DGUhybNiaoKONTKrvNmztahl77uo+xZMmOH3A5 OOoveo24sFi0S4CR5JAVfrBRAkzuX3g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-489-ES96Ob8UODGhOw-QdnA5pw-1; Fri, 01 Dec 2023 14:14:56 -0500 X-MC-Unique: ES96Ob8UODGhOw-QdnA5pw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BA94385A5B5 for ; Fri, 1 Dec 2023 19:14:55 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 345921121307; Fri, 1 Dec 2023 19:14:55 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:37 +0100 Message-ID: <20231201191449.2386134-2-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 01/10] python: ovs: Add flowviz scheleton. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new python package (just the scheleton for now) to hold a flow visualization tool based on the flow parsing library. Signed-off-by: Adrian Moreno --- python/automake.mk | 12 +++++++-- python/ovs/flowviz/__init__.py | 0 python/ovs/flowviz/main.py | 41 ++++++++++++++++++++++++++++++ python/ovs/flowviz/odp/__init__.py | 0 python/ovs/flowviz/ofp/__init__.py | 0 python/ovs/flowviz/ovs-flowviz | 20 +++++++++++++++ python/setup.py | 11 +++++--- 7 files changed, 79 insertions(+), 5 deletions(-) create mode 100644 python/ovs/flowviz/__init__.py create mode 100644 python/ovs/flowviz/main.py create mode 100644 python/ovs/flowviz/odp/__init__.py create mode 100644 python/ovs/flowviz/ofp/__init__.py create mode 100755 python/ovs/flowviz/ovs-flowviz diff --git a/python/automake.mk b/python/automake.mk index 84cf2eab5..4302f0136 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -63,6 +63,14 @@ ovs_pytests = \ python/ovs/tests/test_odp.py \ python/ovs/tests/test_ofp.py +ovs_flowviz = \ + python/ovs/flowviz/__init__.py \ + python/ovs/flowviz/main.py \ + python/ovs/flowviz/odp/__init__.py \ + python/ovs/flowviz/ofp/__init__.py \ + python/ovs/flowviz/ovs-flowviz + + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ @@ -81,7 +89,7 @@ EXTRA_DIST += \ # C extension support. EXTRA_DIST += python/ovs/_json.c -PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) $(ovs_pytests) +PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) $(ovs_pytests) $(ovs_flowviz) EXTRA_DIST += $(PYFILES) PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) @@ -95,7 +103,7 @@ FLAKE8_PYFILES += \ python/ovs/dirs.py.template \ python/setup.py -nobase_pkgdata_DATA = $(ovs_pyfiles) $(ovstest_pyfiles) +nobase_pkgdata_DATA = $(ovs_pyfiles) $(ovstest_pyfiles) $(ovs_flowviz) ovs-install-data-local: $(MKDIR_P) python/ovs sed \ diff --git a/python/ovs/flowviz/__init__.py b/python/ovs/flowviz/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flowviz/main.py b/python/ovs/flowviz/main.py new file mode 100644 index 000000000..a2d5ca1fa --- /dev/null +++ b/python/ovs/flowviz/main.py @@ -0,0 +1,41 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + + +class Options(dict): + """Options dictionary""" + + +@click.group( + subcommand_metavar="TYPE", + context_settings=dict(help_option_names=["-h", "--help"]), +) +@click.pass_context +def maincli(ctx): + """ + OpenvSwitch flow visualization utility. + + It reads openflow and datapath flows + (such as the output of ovs-ofctl dump-flows or ovs-appctl dpctl/dump-flows) + and prints them in different formats. + """ + + +def main(): + """ + Main Function + """ + maincli() diff --git a/python/ovs/flowviz/odp/__init__.py b/python/ovs/flowviz/odp/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flowviz/ofp/__init__.py b/python/ovs/flowviz/ofp/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flowviz/ovs-flowviz b/python/ovs/flowviz/ovs-flowviz new file mode 100755 index 000000000..9d0959812 --- /dev/null +++ b/python/ovs/flowviz/ovs-flowviz @@ -0,0 +1,20 @@ +#!/usr/bin/env python3 +# +# Copyright (c) 2022,2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovs.flowviz import main + +if __name__ == '__main__': + main.main() diff --git a/python/setup.py b/python/setup.py index bcf832ce9..559538575 100644 --- a/python/setup.py +++ b/python/setup.py @@ -80,6 +80,7 @@ else: extra_cflags = os.environ.get('extra_cflags', '').split() extra_libs = os.environ.get('extra_libs', '').split() +flow_extras_require = ['netaddr', 'pyparsing'] setup_args = dict( name='ovs', @@ -89,7 +90,8 @@ setup_args = dict( author='Open vSwitch', author_email='dev@openvswitch.org', packages=['ovs', 'ovs.compat', 'ovs.compat.sortedcontainers', - 'ovs.db', 'ovs.unixctl', 'ovs.flow'], + 'ovs.db', 'ovs.unixctl', 'ovs.flow', 'ovs.flowviz', + 'ovs.flowviz.ofp', 'ovs.flowviz.odp'], keywords=['openvswitch', 'ovs', 'OVSDB'], license='Apache 2.0', classifiers=[ @@ -109,8 +111,11 @@ setup_args = dict( cmdclass={'build_ext': try_build_ext}, install_requires=['sortedcontainers'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0'], - 'flow': ['netaddr', 'pyparsing'], - 'dns': ['unbound']}, + 'flow': flow_extras_require, + 'dns': ['unbound'], + 'flowviz': [*flow_extras_require, 'click'], + }, + scripts=["ovs/flowviz/ovs-flowviz"], ) try: From patchwork Fri Dec 1 19:14:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870794 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSb3Pc3z1ySh for ; Sat, 2 Dec 2023 06:15:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id CAB4C42443; Fri, 1 Dec 2023 19:15:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org CAB4C42443 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KXVjbd9-G6ux; Fri, 1 Dec 2023 19:15:19 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 671134226C; Fri, 1 Dec 2023 19:15:15 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 671134226C Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 530EBC0DEC; Fri, 1 Dec 2023 19:15:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id EB5D9C0DDD for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B490C60D78 for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B490C60D78 Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WDoEnvpa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O5pjYLb2pPEd for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3A6A360BAE for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 3A6A360BAE DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ohyxtl1kJj4/RctmVmprtuQXN3Uf2L2CbwCL7nV5Lqk=; b=WDoEnvparulJL9mnxyofd7cwCHrrYtTcwKxuH+6c+TjUJ+qiXOnF24HQ/sfqMKIT4ZrrD+ OXYPxK3qoQvHUd0nYWH+JH7tbw9MsIXTvL7CkHnZ1SaLlRzr1jlMCbkMNxkI1L9Ho/+RbA E8IiFCB5+ogzOYle0qOGo5VmBs6jowM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-434-FqNNHoACNUitfX1hUVKC2g-1; Fri, 01 Dec 2023 14:14:56 -0500 X-MC-Unique: FqNNHoACNUitfX1hUVKC2g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8F906101A590 for ; Fri, 1 Dec 2023 19:14:56 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 078D51121307; Fri, 1 Dec 2023 19:14:55 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:38 +0100 Message-ID: <20231201191449.2386134-3-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 02/10] python: ovs: flowviz: Add file processing infra. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" process.py contains a useful base class that processes files odp.py and ofp.py: contain datapath and openflow subcommand definitions as well as the first formatting option: json. Also, this patch adds basic filtering support. Examples: $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow json $ ovs-ofctl dump-flows br-int > flows.txt && ovs-flowviz -i flows.txt openflow json $ ovs-ofctl appctl dpctl/dump-flows | ovs-flowviz -f 'ct' datapath json $ ovs-ofctl appctl dpctl/dump-flows > flows.txt && ovs-flowviz -i flows.txt -f 'drop' datapath json Signed-off-by: Adrian Moreno --- python/automake.mk | 5 +- python/ovs/flowviz/__init__.py | 2 + python/ovs/flowviz/main.py | 103 +++++++++++++++++- python/ovs/flowviz/odp/cli.py | 42 ++++++++ python/ovs/flowviz/ofp/cli.py | 42 ++++++++ python/ovs/flowviz/process.py | 192 +++++++++++++++++++++++++++++++++ 6 files changed, 384 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/odp/cli.py create mode 100644 python/ovs/flowviz/ofp/cli.py create mode 100644 python/ovs/flowviz/process.py diff --git a/python/automake.mk b/python/automake.mk index 4302f0136..4845565b8 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -67,8 +67,11 @@ ovs_flowviz = \ python/ovs/flowviz/__init__.py \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ + python/ovs/flowviz/odp/cli.py \ python/ovs/flowviz/ofp/__init__.py \ - python/ovs/flowviz/ovs-flowviz + python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ovs-flowviz \ + python/ovs/flowviz/process.py # These python files are used at build time but not runtime, diff --git a/python/ovs/flowviz/__init__.py b/python/ovs/flowviz/__init__.py index e69de29bb..898dba522 100644 --- a/python/ovs/flowviz/__init__.py +++ b/python/ovs/flowviz/__init__.py @@ -0,0 +1,2 @@ +import ovs.flowviz.ofp.cli # noqa: F401 +import ovs.flowviz.odp.cli # noqa: F401 diff --git a/python/ovs/flowviz/main.py b/python/ovs/flowviz/main.py index a2d5ca1fa..a45c06e48 100644 --- a/python/ovs/flowviz/main.py +++ b/python/ovs/flowviz/main.py @@ -12,19 +12,67 @@ # See the License for the specific language governing permissions and # limitations under the License. +import os + import click +from ovs.flow.filter import OFFilter + class Options(dict): """Options dictionary""" +def validate_input(ctx, param, value): + """Validate the "-i" option""" + result = list() + for input_str in value: + parts = input_str.strip().split(",") + if len(parts) == 2: + file_parts = tuple(parts) + elif len(parts) == 1: + file_parts = tuple(["Filename: " + parts[0], parts[0]]) + else: + raise click.BadParameter( + "input filename should have the following format: " + "[alias,]FILENAME" + ) + + if not os.path.isfile(file_parts[1]): + raise click.BadParameter( + "input filename %s does not exist" % file_parts[1] + ) + result.append(file_parts) + return result + + @click.group( subcommand_metavar="TYPE", context_settings=dict(help_option_names=["-h", "--help"]), ) +@click.option( + "-i", + "--input", + "filename", + help="Read flows from specified filepath. If not provided, flows will be" + " read from stdin. This option can be specified multiple times." + " Format [alias,]FILENAME. Where alias is a name that shall be used to" + " refer to this FILENAME", + multiple=True, + type=click.Path(), + callback=validate_input, +) +@click.option( + "-f", + "--filter", + help="Filter flows that match the filter expression." + "Run 'ovs-flowviz filter' for a detailed description of the filtering " + "syntax", + type=str, + show_default=False, +) @click.pass_context -def maincli(ctx): +def maincli(ctx, filename, filter): """ OpenvSwitch flow visualization utility. @@ -32,6 +80,59 @@ def maincli(ctx): (such as the output of ovs-ofctl dump-flows or ovs-appctl dpctl/dump-flows) and prints them in different formats. """ + ctx.obj = Options() + ctx.obj["filename"] = filename or None + if filter: + try: + ctx.obj["filter"] = OFFilter(filter) + except Exception as e: + raise click.BadParameter("Wrong filter syntax: {}".format(e)) + + +@maincli.command(hidden=True) +@click.pass_context +def filter(ctx): + """ + \b + Filter Syntax + ************* + + [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ... + + \b + Comparison operators are: + = equality + < less than + > more than + ~= masking (valid for IP and Ethernet fields) + + \b + Logical operators are: + !{expr}: NOT + {expr} && {expr}: AND + {expr} || {expr}: OR + + \b + Matches and flow metadata: + To compare against a match or info field, use the field directly, e.g: + priority=100 + n_bytes>10 + Use simple keywords for flags: + tcp and ip_src=192.168.1.1 + \b + Actions: + Actions values might be dictionaries, use subkeys to access individual + values, e.g: + output.port=3 + Use simple keywords for flags + drop + + \b + Examples of valid filters. + nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443) + arp=true && !arp_tsa=192.168.1.1 + n_bytes>0 && drop=true""" + click.echo(ctx.command.get_help(ctx)) def main(): diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py new file mode 100644 index 000000000..ed2f82065 --- /dev/null +++ b/python/ovs/flowviz/odp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + DatapathFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def datapath(opts): + """Process Datapath Flows.""" + pass + + +class JSONPrint(DatapathFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@datapath.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py new file mode 100644 index 000000000..b9a2a8aad --- /dev/null +++ b/python/ovs/flowviz/ofp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + OpenFlowFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def openflow(opts): + """Process OpenFlow Flows.""" + pass + + +class JSONPrint(OpenFlowFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@openflow.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/process.py b/python/ovs/flowviz/process.py new file mode 100644 index 000000000..413506bf2 --- /dev/null +++ b/python/ovs/flowviz/process.py @@ -0,0 +1,192 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +import json +import click + +from ovs.flow.decoders import FlowEncoder +from ovs.flow.odp import ODPFlow +from ovs.flow.ofp import OFPFlow + + +class FileProcessor(object): + """Base class for file-based Flow processing. It is able to create flows + from strings found in a file (or stdin). + + The process of parsing the flows is extendable in many ways by deriving + this class. + + When process() is called, the base class will: + - call self.start_file() for each new file that get's processed + - call self.create_flow() for each flow line + - apply the filter defined in opts if provided (can be optionally + disabled) + - call self.process_flow() for after the flow has been filtered + - call self.stop_file() after the file has been processed entirely + + In the case of stdin, the filename and file alias is 'stdin'. + + Child classes must at least implement create_flow() and process_flow() + functions. + + Args: + opts (dict): Options dictionary + """ + + def __init__(self, opts): + self.opts = opts + + # Methods that must be implemented by derived classes + def init(self): + """Called before the flow processing begins.""" + pass + + def start_file(self, alias, filename): + """Called before the processing of a file begins. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def create_flow(self, line, idx): + """Called for each line in the file. + Args: + line(str): The flow line + idx(int): The line index + + Returns a Flow. + Must be implemented by child classes. + """ + raise NotImplementedError + + def process_flow(self, flow, name): + """Called for built flow (after filtering). + Args: + flow(Flow): The flow created by create_flow + name(str): The name of the file from which the flow comes + """ + raise NotImplementedError + + def stop_file(self, alias, filename): + """Called after the processing of a file ends. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def end(self): + """Called after the processing ends.""" + pass + + def process(self, do_filter=True): + idx = 0 + filenames = self.opts.get("filename") + filt = self.opts.get("filter") if do_filter else None + self.init() + if filenames: + for alias, filename in filenames: + try: + with open(filename) as f: + self.start_file(alias, filename) + for line in f: + flow = self.create_flow(line, idx) + idx += 1 + if not flow or (filt and not filt.evaluate(flow)): + continue + self.process_flow(flow, alias) + self.stop_file(alias, filename) + except IOError as e: + raise click.BadParameter( + "Failed to read from file {} ({}): {}".format( + filename, e.errno, e.strerror + ) + ) + else: + data = sys.stdin.read() + self.start_file("stdin", "stdin") + for line in data.split("\n"): + line = line.strip() + if line: + flow = self.create_flow(line, idx) + idx += 1 + if ( + not flow + or not getattr(flow, "_sections", None) + or (filt and not filt.evaluate(flow)) + ): + continue + self.process_flow(flow, "stdin") + self.stop_file("stdin", "stdin") + self.end() + + +class DatapathFactory(): + """A mixin class that creates OpenFlow flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in Datapath flow dumps. + if any(s in line for s in [ + "flow-dump from the main thread", + "flow-dump from pmd on core", + ]): + return None + + return ODPFlow(line, idx) + + +class OpenFlowFactory(): + """A mixin class that creates Datapath flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in OpenFlow flow dumps. + if " reply " in line: + return None + + return OFPFlow(line, idx) + + +class JSONProcessor(FileProcessor): + """A FileProcessor that prints flows in JSON format.""" + + def __init__(self, opts): + super().__init__(opts) + self.flows = dict() + + def start_file(self, name, filename): + self.flows_list = list() + + def stop_file(self, name, filename): + self.flows[name] = self.flows_list + + def process_flow(self, flow, name): + self.flows_list.append(flow) + + def json_string(self): + if len(self.flows.keys()) > 1: + return json.dumps( + [ + {"name": name, "flows": [flow.dict() for flow in flows]} + for name, flows in self.flows.items() + ], + indent=4, + cls=FlowEncoder, + ) + return json.dumps( + [flow.dict() for flow in self.flows_list], + indent=4, + cls=FlowEncoder, + ) From patchwork Fri Dec 1 19:14:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870791 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MQ636Ybt; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSP2GDDz23ng for ; Sat, 2 Dec 2023 06:15:17 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 6ECCF6FB2B; Fri, 1 Dec 2023 19:15:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 6ECCF6FB2B Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MQ636Ybt X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bYB90FGKi3ui; Fri, 1 Dec 2023 19:15:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 230986FB0F; Fri, 1 Dec 2023 19:15:10 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 230986FB0F Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4AE7DC0DD9; Fri, 1 Dec 2023 19:15:07 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1796FC0077 for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E788484938 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org E788484938 Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MQ636Ybt X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lVzflHMr5CfK for ; Fri, 1 Dec 2023 19:15:02 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id A1BF384914 for ; Fri, 1 Dec 2023 19:15:02 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A1BF384914 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Acnm1pMkUSFdUEwoXZbiU+C1DvZn8sXW77MKcEak3U=; b=MQ636YbtUV9fxTD7XGm1Ic6SNK5m2dR7YXgKp3oLXcE78X6BL8qNDmnstbqMULIypGTFo7 VLSrHMcIy5BmlaVPTZqhVVwzNMxA1qRjidziDmMiKCcfad6kr0oasQTilhEck5a0lHC/VO aq40pyUktHe04dlSCPSUrcoS0aM+OvU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-427-A-q0jI4YPxSuiyaVsauzzg-1; Fri, 01 Dec 2023 14:14:57 -0500 X-MC-Unique: A-q0jI4YPxSuiyaVsauzzg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 89D3A185A787 for ; Fri, 1 Dec 2023 19:14:57 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id D60C41121308; Fri, 1 Dec 2023 19:14:56 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:39 +0100 Message-ID: <20231201191449.2386134-4-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 03/10] python: ovs: flowviz: Add console formatting. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a flow formatting framework and one implementation for console printing using rich. The flow formatting framework is a simple set of classes that can be used to write different flow formatting implementations. It supports styles to be described by any class, highlighting and config-file based style definition. The first flow formatting implementation is also introduced: the ConsoleFormatter. It uses the an advanced rich-text printing library [1]. The console printing supports: - Heatmap: printing the packet/byte statistics of each flow in a color that represents its relative size: blue (low) -> red (high). - Printing a banner with the file name and alias. - Extensive style definition via config file. This console format is added to both OpenFlow and Datapath flows. Examples: - Highlight drops in datapath flows: $ ovs-flowviz -i flows.txt --highlight "drop" datapath console - Quickly detect where most packets are going using heatmap and paginated output: $ ovs-ofctl dump-flows br-int | ovs-flowviz -p openflow console -h [1] https://rich.readthedocs.io/en/stable/introduction.html Signed-off-by: Adrian Moreno --- python/automake.mk | 2 + python/ovs/flowviz/console.py | 174 ++++++++++++++++ python/ovs/flowviz/format.py | 372 ++++++++++++++++++++++++++++++++++ python/ovs/flowviz/main.py | 57 +++++- python/ovs/flowviz/odp/cli.py | 25 +++ python/ovs/flowviz/ofp/cli.py | 26 +++ python/ovs/flowviz/process.py | 87 +++++++- python/setup.py | 5 +- 8 files changed, 738 insertions(+), 10 deletions(-) create mode 100644 python/ovs/flowviz/console.py create mode 100644 python/ovs/flowviz/format.py diff --git a/python/automake.mk b/python/automake.mk index 4845565b8..8d82cb4a8 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -65,6 +65,8 @@ ovs_pytests = \ ovs_flowviz = \ python/ovs/flowviz/__init__.py \ + python/ovs/flowviz/console.py \ + python/ovs/flowviz/format.py \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ python/ovs/flowviz/odp/cli.py \ diff --git a/python/ovs/flowviz/console.py b/python/ovs/flowviz/console.py new file mode 100644 index 000000000..5b4b047c2 --- /dev/null +++ b/python/ovs/flowviz/console.py @@ -0,0 +1,174 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import colorsys +from rich.console import Console +from rich.text import Text +from rich.style import Style +from rich.color import Color +from rich.panel import Panel +from rich.emoji import Emoji + +from ovs.flowviz.format import FlowFormatter, FlowBuffer, FlowStyle + + +def file_header(name): + return Panel( + Text( + Emoji.replace(":scroll:") + + " " + + name + + " " + + Emoji.replace(":scroll:"), + style="bold", + justify="center", + ) + ) + + +class ConsoleBuffer(FlowBuffer): + """ConsoleBuffer implements FlowBuffer to provide console-based text + formatting based on rich.Text. + + Append functions accept a rich.Style. + + Args: + rtext(rich.Text): Optional; text instance to reuse + """ + + def __init__(self, rtext): + self._text = rtext or Text() + + @property + def text(self): + return self._text + + def _append(self, string, style): + """Append to internal text.""" + return self._text.append(string, style) + + def append_key(self, kv, style): + """Append a key. + Args: + kv (KeyValue): the KeyValue instance to append + style (rich.Style): the style to use + """ + return self._append(kv.meta.kstring, style) + + def append_delim(self, kv, style): + """Append a delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (rich.Style): the style to use + """ + return self._append(kv.meta.delim, style) + + def append_end_delim(self, kv, style): + """Append an end delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (rich.Style): the style to use + """ + return self._append(kv.meta.end_delim, style) + + def append_value(self, kv, style): + """Append a value. + Args: + kv (KeyValue): the KeyValue instance to append + style (rich.Style): the style to use + """ + return self._append(kv.meta.vstring, style) + + def append_extra(self, extra, style): + """Append extra string. + Args: + kv (KeyValue): the KeyValue instance to append + style (rich.Style): the style to use + """ + return self._append(extra, style) + + +class ConsoleFormatter(FlowFormatter): + """ConsoleFormatter is a FlowFormatter that formats flows into the console + using rich.Console. + + Args: + console (rich.Console): Optional, an existing console to use + max_value_len (int): Optional; max length of the printed values + kwargs (dict): Optional; Extra arguments to be passed down to + rich.console.Console() + """ + + def __init__(self, opts=None, console=None, **kwargs): + super(ConsoleFormatter, self).__init__() + style = self.style_from_opts(opts) + self.console = console or Console(color_system="256", **kwargs) + self.style = style or FlowStyle() + + def style_from_opts(self, opts): + return self._style_from_opts(opts, "console", Style) + + def print_flow(self, flow, highlighted=None): + """Prints a flow to the console. + + Args: + flow (ovs_dbg.OFPFlow): the flow to print + style (dict): Optional; style dictionary to use + highlighted (list): Optional; list of KeyValues to highlight + """ + + buf = ConsoleBuffer(Text()) + self.format_flow(buf, flow, highlighted) + self.console.print(buf.text) + + def format_flow(self, buf, flow, highlighted=None): + """Formats the flow into the provided buffer as a rich.Text. + + Args: + buf (FlowBuffer): the flow buffer to append to + flow (ovs_dbg.OFPFlow): the flow to format + style (FlowStyle): Optional; style object to use + highlighted (list): Optional; list of KeyValues to highlight + """ + return super(ConsoleFormatter, self).format_flow( + buf, flow, self.style, highlighted + ) + + +def heat_pallete(min_value, max_value): + """Generates a color pallete based on the 5-color heat pallete so that + for each value between min and max a color is returned that represents it's + relative size. + Args: + min_value (int): minimum value + max_value (int) maximum value + """ + h_min = 0 # red + h_max = 220 / 360 # blue + + def heat(value): + if max_value == min_value: + r, g, b = colorsys.hsv_to_rgb(h_max / 2, 1.0, 1.0) + else: + normalized = (int(value) - min_value) / (max_value - min_value) + hue = ((1 - normalized) + h_min) * (h_max - h_min) + r, g, b = colorsys.hsv_to_rgb(hue, 1.0, 1.0) + return Style(color=Color.from_rgb(r * 255, g * 255, b * 255)) + + return heat + + +def default_highlight(): + """Generates a default style for highlights.""" + return Style(underline=True) diff --git a/python/ovs/flowviz/format.py b/python/ovs/flowviz/format.py new file mode 100644 index 000000000..bc773168a --- /dev/null +++ b/python/ovs/flowviz/format.py @@ -0,0 +1,372 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Flow formatting framework. + +This file defines a simple flow formatting framework. It's comprised of 3 +classes: FlowStyle, FlowFormatter and FlowBuffer. + +The FlowStyle arranges opaque style objects in a dictionary that can be queried +to determine what style a particular key-value should be formatted with. +That way, a particular implementation can represent its style using their own +object. + +The FlowBuffer is an abstract class and must be derived by particular +implementations. It should know how to append parts of a flow using a style. +Only here the type of the style is relevant. + +When asked to format a flow, the FlowFormatter will determine which style +the flow must be formatted with and call FlowBuffer functions with each part +of the flow and their corresponding style. +""" + + +class FlowStyle: + """A FlowStyle determines the KVStyle to use for each key value in a flow. + + Styles are internally represented by a dictionary. + In order to determine the style for a "key", the following items in the + dictionary are fetched: + - key.highlighted.{key} (if key is found in hightlighted) + - key.highlighted (if key is found in hightlighted) + - key.{key} + - key + - default + + In order to determine the style for a "value", the following items in the + dictionary are fetched: + - value.highlighted.{key} (if key is found in hightlighted) + - value.highlighted.type{value.__class__.__name__} + - value.highlighted + (if key is found in hightlighted) + - value.{key} + - value.type.{value.__class__.__name__} + - value + - default + + The actual type of the style object stored for each item above is opaque + to this class and it depends on the particular FlowFormatter child class + that will handle them. Even callables can be stored, if so they will be + called with the value of the field that is to be formatted and the return + object will be used as style. + + Additionally, the following style items can be defined: + - delim: for delimiters + - delim.highlighted: for delimiters of highlighted key-values + """ + + def __init__(self, initial=None): + self._styles = initial if initial is not None else dict() + + def __len__(self): + return len(self._styles) + + def set_flag_style(self, kvstyle): + self._styles["flag"] = kvstyle + + def set_delim_style(self, kvstyle, highlighted=False): + if highlighted: + self._styles["delim.highlighted"] = kvstyle + else: + self._styles["delim"] = kvstyle + + def set_default_key_style(self, kvstyle, highlighted=False): + if highlighted: + self._styles["key.highlighted"] = kvstyle + else: + self._styles["key"] = kvstyle + + def set_default_value_style(self, kvstyle, highlighted=False): + if highlighted: + self._styles["value.highlighted"] = kvstyle + else: + self._styles["value"] = kvstyle + + def set_key_style(self, key, kvstyle, highlighted=False): + if highlighted: + self._styles["key.highlighted.{}".format(key)] = kvstyle + else: + self._styles["key.{}".format(key)] = kvstyle + + def set_value_style(self, key, kvstyle, highlighted=None): + if highlighted: + self._styles["value.highlighted.{}".format(key)] = kvstyle + else: + self._styles["value.{}".format(key)] = kvstyle + + def set_value_type_style(self, name, kvstyle, highlighted=None): + if highlighted: + self._styles["value.highlighted.type.{}".format(name)] = kvstyle + else: + self._styles["value.type.{}".format(name)] = kvstyle + + def get(self, key): + return self._styles.get(key) + + def get_delim_style(self, highlighted=False): + delim_style_lookup = ["delim.highlighted"] if highlighted else [] + delim_style_lookup.extend(["delim", "default"]) + return next( + ( + self._styles.get(s) + for s in delim_style_lookup + if self._styles.get(s) + ), + None, + ) + + def get_flag_style(self): + return self._styles.get("flag") or self._styles.get("default") + + def get_key_style(self, kv, highlighted=False): + key = kv.meta.kstring + + key_style_lookup = ( + ["key.highlighted.%s" % key, "key.highlighted"] + if highlighted + else [] + ) + key_style_lookup.extend(["key.%s" % key, "key", "default"]) + + style = next( + ( + self._styles.get(s) + for s in key_style_lookup + if self._styles.get(s) + ), + None, + ) + if callable(style): + return style(kv.meta.kstring) + return style + + def get_value_style(self, kv, highlighted=False): + key = kv.meta.kstring + value_type = kv.value.__class__.__name__.lower() + value_style_lookup = ( + [ + "value.highlighted.%s" % key, + "value.highlighted.type.%s" % value_type, + "value.highlighted", + ] + if highlighted + else [] + ) + value_style_lookup.extend( + [ + "value.%s" % key, + "value.type.%s" % value_type, + "value", + "default", + ] + ) + + style = next( + ( + self._styles.get(s) + for s in value_style_lookup + if self._styles.get(s) + ), + None, + ) + if callable(style): + return style(kv.meta.vstring) + return style + + +class FlowFormatter: + """FlowFormatter is a base class for Flow Formatters.""" + + def __init__(self): + self._highlighted = list() + + def _style_from_opts(self, opts, opts_key, style_constructor): + """Create style object from options. + + Args: + opts (dict): Options dictionary + opts_key (str): The options style key to extract + (e.g: console or html) + style_constructor(callable): A callable that creates a derived + style object + """ + if not opts or not opts.get("style"): + return None + + section_name = ".".join(["styles", opts.get("style")]) + if section_name not in opts.get("config").sections(): + return None + + config = opts.get("config")[section_name] + style = {} + for key in config: + (_, console, style_full_key) = key.partition(opts_key + ".") + if not console: + continue + + (style_key, _, prop) = style_full_key.rpartition(".") + if not prop or not style_key: + raise Exception("malformed style config: {}".format(key)) + + if not style.get(style_key): + style[style_key] = {} + style[style_key][prop] = config[key] + + return FlowStyle({k: style_constructor(**v) for k, v in style.items()}) + + def format_flow(self, buf, flow, style_obj=None, highlighted=None): + """Formats the flow into the provided buffer. + + Args: + buf (FlowBuffer): the flow buffer to append to + flow (ovs_dbg.OFPFlow): the flow to format + style_obj (FlowStyle): Optional; style to use + highlighted (list): Optional; list of KeyValues to highlight + """ + last_printed_pos = 0 + + if style_obj: + style_obj = style_obj or FlowStyle() + for section in sorted(flow.sections, key=lambda x: x.pos): + buf.append_extra( + flow.orig[last_printed_pos : section.pos], + style=style_obj.get("default"), + ) + self.format_kv_list( + buf, section.data, section.string, style_obj, highlighted + ) + last_printed_pos = section.pos + len(section.string) + else: + # Don't pay the cost of formatting each section one by one. + buf.append_extra(flow.orig.strip(), None) + + def format_kv_list(self, buf, kv_list, full_str, style_obj, highlighted): + """Format a KeyValue List. + + Args: + buf (FlowBuffer): a FlowBuffer to append formatted KeyValues to + kv_list (list[KeyValue]: the KeyValue list to format + full_str (str): the full string containing all k-v + style_obj (FlowStyle): a FlowStyle object to use + highlighted (list): Optional; list of KeyValues to highlight + """ + for i, kv in enumerate(kv_list): + written = self.format_kv( + buf, kv, style_obj=style_obj, highlighted=highlighted + ) + + end = ( + kv_list[i + 1].meta.kpos + if i < (len(kv_list) - 1) + else len(full_str) + ) + + buf.append_extra( + full_str[(kv.meta.kpos + written) : end].rstrip("\n\r"), + style=style_obj.get("default"), + ) + + def format_kv(self, buf, kv, style_obj, highlighted=None): + """Format a KeyValue + + A formatted keyvalue has the following parts: + {key}{delim}{value}[{delim}] + + Args: + buf (FlowBuffer): buffer to append the KeyValue to + kv (KeyValue): The KeyValue to print + style_obj (FlowStyle): The style object to use + highlighted (list): Optional; list of KeyValues to highlight + + Returns the number of printed characters. + """ + ret = 0 + key = kv.meta.kstring + is_highlighted = ( + key in [k.key for k in highlighted] if highlighted else False + ) + + key_style = style_obj.get_key_style(kv, is_highlighted) + buf.append_key(kv, key_style) # format value + ret += len(key) + + if not kv.meta.vstring: + return ret + + if kv.meta.delim not in ("\n", "\t", "\r", ""): + buf.append_delim(kv, style_obj.get_delim_style(is_highlighted)) + ret += len(kv.meta.delim) + + value_style = style_obj.get_value_style(kv, is_highlighted) + + buf.append_value(kv, value_style) # format value + ret += len(kv.meta.vstring) + + if kv.meta.end_delim: + buf.append_end_delim(kv, style_obj.get_delim_style(is_highlighted)) + ret += len(kv.meta.end_delim) + + return ret + + +class FlowBuffer: + """A FlowBuffer is a base class for format buffers. + + Childs must implement the following methods: + append_key(self, kv, style) + append_value(self, kv, style) + append_delim(self, delim, style) + append_end_delim(self, delim, style) + append_extra(self, extra, style) + """ + + def append_key(self, kv, style): + """Append a key. + Args: + kv (KeyValue): the KeyValue instance to append + style (Any): the style to use + """ + raise NotImplementedError + + def append_delim(self, kv, style): + """Append a delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (Any): the style to use + """ + raise NotImplementedError + + def append_end_delim(self, kv, style): + """Append an end delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (Any): the style to use + """ + raise NotImplementedError + + def append_value(self, kv, style): + """Append a value. + Args: + kv (KeyValue): the KeyValue instance to append + style (Any): the style to use + """ + raise NotImplementedError + + def append_extra(self, extra, style): + """Append extra string. + Args: + kv (KeyValue): the KeyValue instance to append + style (Any): the style to use + """ + raise NotImplementedError diff --git a/python/ovs/flowviz/main.py b/python/ovs/flowviz/main.py index a45c06e48..fab4d8557 100644 --- a/python/ovs/flowviz/main.py +++ b/python/ovs/flowviz/main.py @@ -12,11 +12,31 @@ # See the License for the specific language governing permissions and # limitations under the License. +import configparser import os import click from ovs.flow.filter import OFFilter +from ovs.dirs import PKGDATADIR + +_default_config_file = "ovs-flowviz.conf" +_default_config_path = next( + ( + p + for p in [ + os.path.join( + os.getenv("HOME"), ".config", "ovs", _default_config_file + ), + os.path.join(PKGDATADIR, _default_config_file), + os.path.abspath( + os.path.join(os.path.dirname(__file__), _default_config_file) + ), + ] + if os.path.exists(p) + ), + "", +) class Options(dict): @@ -50,6 +70,20 @@ def validate_input(ctx, param, value): subcommand_metavar="TYPE", context_settings=dict(help_option_names=["-h", "--help"]), ) +@click.option( + "-c", + "--config", + help="Use config file", + type=click.Path(), + default=_default_config_path, + show_default=True, +) +@click.option( + "--style", + help="Select style (defined in config file)", + default=None, + show_default=True, +) @click.option( "-i", "--input", @@ -71,8 +105,16 @@ def validate_input(ctx, param, value): type=str, show_default=False, ) +@click.option( + "-l", + "--highlight", + help="Highlight flows that match the filter expression." + " Run 'ofparse filter' for a detailed description of the filtering syntax", + type=str, + show_default=False, +) @click.pass_context -def maincli(ctx, filename, filter): +def maincli(ctx, config, style, filename, filter, highlight): """ OpenvSwitch flow visualization utility. @@ -88,6 +130,19 @@ def maincli(ctx, filename, filter): except Exception as e: raise click.BadParameter("Wrong filter syntax: {}".format(e)) + if highlight: + try: + ctx.obj["highlight"] = OFFilter(highlight) + except Exception as e: + raise click.BadParameter("Wrong filter syntax: {}".format(e)) + + config_file = config or _default_config_path + parser = configparser.ConfigParser() + parser.read(config_file) + + ctx.obj["config"] = parser + ctx.obj["style"] = style + @maincli.command(hidden=True) @click.pass_context diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py index ed2f82065..78f5cfff4 100644 --- a/python/ovs/flowviz/odp/cli.py +++ b/python/ovs/flowviz/odp/cli.py @@ -18,6 +18,7 @@ from ovs.flowviz.main import maincli from ovs.flowviz.process import ( DatapathFactory, JSONProcessor, + ConsoleProcessor, ) @@ -40,3 +41,27 @@ def json(opts): proc = JSONPrint(opts) proc.process() print(proc.json_string()) + + +class DPConsoleProcessor(DatapathFactory, ConsoleProcessor): + def __init__(self, opts, heat_map): + super().__init__(opts, heat_map) + + +@datapath.command() +@click.option( + "-h", + "--heat-map", + is_flag=True, + default=False, + show_default=True, + help="Create heat-map with packet and byte counters", +) +@click.pass_obj +def console(opts, heat_map): + """Print the flows in the console with some style.""" + proc = DPConsoleProcessor( + opts, heat_map=["packets", "bytes"] if heat_map else [] + ) + proc.process() + proc.print() diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py index b9a2a8aad..a28e489ac 100644 --- a/python/ovs/flowviz/ofp/cli.py +++ b/python/ovs/flowviz/ofp/cli.py @@ -18,6 +18,7 @@ from ovs.flowviz.main import maincli from ovs.flowviz.process import ( OpenFlowFactory, JSONProcessor, + ConsoleProcessor, ) @@ -40,3 +41,28 @@ def json(opts): proc = JSONPrint(opts) proc.process() print(proc.json_string()) + + +class OFConsoleProcessor(OpenFlowFactory, ConsoleProcessor): + def __init__(self, opts, heat_map): + super().__init__(opts, heat_map) + + +@openflow.command() +@click.option( + "-h", + "--heat-map", + is_flag=True, + default=False, + show_default=True, + help="Create heat-map with packet and byte counters", +) +@click.pass_obj +def console(opts, heat_map): + """Print the flows in the console with some style.""" + proc = OFConsoleProcessor( + opts, + heat_map=["n_packets", "n_bytes"] if heat_map else [], + ) + proc.process() + proc.print() diff --git a/python/ovs/flowviz/process.py b/python/ovs/flowviz/process.py index 413506bf2..1d5befdd1 100644 --- a/python/ovs/flowviz/process.py +++ b/python/ovs/flowviz/process.py @@ -20,6 +20,13 @@ from ovs.flow.decoders import FlowEncoder from ovs.flow.odp import ODPFlow from ovs.flow.ofp import OFPFlow +from ovs.flowviz.console import ( + ConsoleFormatter, + default_highlight, + heat_pallete, + file_header, +) + class FileProcessor(object): """Base class for file-based Flow processing. It is able to create flows @@ -134,22 +141,25 @@ class FileProcessor(object): self.end() -class DatapathFactory(): - """A mixin class that creates OpenFlow flows.""" +class DatapathFactory: + """A mixin class that creates Datapath flows.""" def create_flow(self, line, idx): # Skip strings commonly found in Datapath flow dumps. - if any(s in line for s in [ - "flow-dump from the main thread", - "flow-dump from pmd on core", - ]): + if any( + s in line + for s in [ + "flow-dump from the main thread", + "flow-dump from pmd on core", + ] + ): return None return ODPFlow(line, idx) -class OpenFlowFactory(): - """A mixin class that creates Datapath flows.""" +class OpenFlowFactory: + """A mixin class that creates OpenFlow flows.""" def create_flow(self, line, idx): # Skip strings commonly found in OpenFlow flow dumps. @@ -190,3 +200,64 @@ class JSONProcessor(FileProcessor): indent=4, cls=FlowEncoder, ) + + +class ConsoleProcessor(FileProcessor): + """A generic Console Processor that prints flows into the console""" + + def __init__(self, opts, heat_map=[]): + super().__init__(opts) + self.heat_map = heat_map + self.console = ConsoleFormatter(opts) + if len(self.console.style) == 0 and self.opts.get("highlight"): + # Add some style to highlights or else they won't be seen. + self.console.style.set_default_value_style( + default_highlight(), True + ) + self.console.style.set_default_key_style(default_highlight(), True) + + self.flows = dict() # Dictionary of flow-lists, one per file. + self.min_max = dict() # Used for heat-map calculation + + def start_file(self, name, filename): + self.flows_list = list() + if len(self.heat_map) > 0: + self.min = [-1] * len(self.heat_map) + self.max = [0] * len(self.heat_map) + + def stop_file(self, name, filename): + self.flows[name] = self.flows_list + if len(self.heat_map) > 0: + self.min_max[name] = (self.min, self.max) + + def process_flow(self, flow, name): + # Running calculation of min and max values for all the fields that + # take place in the heatmap. + for i, field in enumerate(self.heat_map): + val = flow.info.get(field) + if self.min[i] == -1 or val < self.min[i]: + self.min[i] = val + if val > self.max[i]: + self.max[i] = val + + self.flows_list.append(flow) + + def print(self): + for name, flows in self.flows.items(): + self.console.console.print("\n") + self.console.console.print(file_header(name)) + + if len(self.heat_map) > 0 and len(self.flows) > 0: + for i, field in enumerate(self.heat_map): + (min_val, max_val) = self.min_max[name][i] + self.console.style.set_value_style( + field, heat_pallete(min_val, max_val) + ) + + for flow in flows: + high = None + if self.opts.get("highlight"): + result = self.opts.get("highlight").evaluate(flow) + if result: + high = result.kv + self.console.print_flow(flow, high) diff --git a/python/setup.py b/python/setup.py index 559538575..38e72ab62 100644 --- a/python/setup.py +++ b/python/setup.py @@ -113,9 +113,12 @@ setup_args = dict( extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0'], 'flow': flow_extras_require, 'dns': ['unbound'], - 'flowviz': [*flow_extras_require, 'click'], + 'flowviz': + [*flow_extras_require, 'click', 'rich'], }, scripts=["ovs/flowviz/ovs-flowviz"], + data_files=["ovs/flowviz/ovs-flowviz.conf"], + include_package_data=True, ) try: From patchwork Fri Dec 1 19:14:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870796 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=KpIEEkGU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSd602Sz1ySh for ; Sat, 2 Dec 2023 06:15:29 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id F38406FB28; Fri, 1 Dec 2023 19:15:27 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org F38406FB28 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=KpIEEkGU X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3j5npyKU0mdt; Fri, 1 Dec 2023 19:15:25 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9AB946FB31; Fri, 1 Dec 2023 19:15:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 9AB946FB31 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8303CC0DF5; Fri, 1 Dec 2023 19:15:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 830C6C0DDA for ; Fri, 1 Dec 2023 19:15:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 4EA3543767 for ; Fri, 1 Dec 2023 19:15:07 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 4EA3543767 Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=KpIEEkGU X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WiTto6_7zMmM for ; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id D51FE43758 for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D51FE43758 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458104; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l70ENXEvWpHjQAjrFimElWR3XPxyIWSxIZBALt3zM/U=; b=KpIEEkGUU3AW3asOBU1kGGxSRSbpQSiYdqtKwuyKhke4IE6bDLk/kVPuRorcxvtgiIVPnv c8raBkXOj16ILJGG5xsxT94VNjibJ3yF98/aTT7PvEcDmHV6MJUqpjI2HEAKkkMGWJSpYi scOUkuV/6h5w0EDBuiBC601A6tmH+5E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-573-TRqCJQ58N4m3PezxSGJaQw-1; Fri, 01 Dec 2023 14:14:58 -0500 X-MC-Unique: TRqCJQ58N4m3PezxSGJaQw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5A8C7811E7E for ; Fri, 1 Dec 2023 19:14:58 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id C86FD1121308; Fri, 1 Dec 2023 19:14:57 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:40 +0100 Message-ID: <20231201191449.2386134-5-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 04/10] python: ovs: flowviz: Add default config file. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It has two basic styles defined: "dark" and "light" intended for dark and light terminals. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 5 +- python/ovs/flowviz/ovs-flowviz.conf | 94 +++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 python/ovs/flowviz/ovs-flowviz.conf diff --git a/python/automake.mk b/python/automake.mk index 8d82cb4a8..cf8b71659 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -89,7 +89,8 @@ EXTRA_DIST += \ python/ovs/compat/sortedcontainers/LICENSE \ python/README.rst \ python/setup.py \ - python/test_requirements.txt + python/test_requirements.txt \ + python/ovs/flowviz/ovs-flowviz.conf # C extension support. EXTRA_DIST += python/ovs/_json.c @@ -109,6 +110,8 @@ FLAKE8_PYFILES += \ python/setup.py nobase_pkgdata_DATA = $(ovs_pyfiles) $(ovstest_pyfiles) $(ovs_flowviz) +nobase_pkgdata_DATA += python/ovs/flowviz/ovs-flowviz.conf + ovs-install-data-local: $(MKDIR_P) python/ovs sed \ diff --git a/python/ovs/flowviz/ovs-flowviz.conf b/python/ovs/flowviz/ovs-flowviz.conf new file mode 100644 index 000000000..3acd0a29e --- /dev/null +++ b/python/ovs/flowviz/ovs-flowviz.conf @@ -0,0 +1,94 @@ +# Create any number of styles.{style_name} sections with a defined style. +# +# Syntax: +# +# [FORMAT].[PORTION].[SELECTOR].[ELEMENT] = [VALUE] +# +# * FORMAT: console +# * PORTION: The portion of the flow that the style applies to +# - key: Selects how to print the key of a KeyValue pair +# - key: Selects how to print the value of a KeyValue pair +# - flag: Selects how to print the a flag +# - delim: Selects how to print the delimiters around key and values +# +# * SELECTOR: +# - highlighted: to apply when the key is highlighted +# - type.{TYPE}: to apply when the value matches a type +# (special types such as IPAddress or EthMask can be used) +# (only aplicable to 'value') +# - {key_name}: to apply when the key matches the key_name +# +# Console Styles +# ============== +# * ELEMENT: +# - color: defines the color in hex or a color rich starndard ones [1] +# - underline: if set to "true", the selected portion will be underlined +# +#[1] https://rich.readthedocs.io/en/stable/appendix/colors.html#standard-colors + +[styles.dark] + +# defaults for key-values +console.key.color = #5D86BA +console.value.color= #B0C4DE +console.delim.color= #B0C4DE +console.default.color= #FFFFFF + +# defaults for special types +console.value.type.IPAddress.color = #008700 +console.value.type.IPMask.color = #008700 +console.value.type.EthMask.color = #008700 + +# dim some long arguments +console.value.ct.color = grey66 +console.value.ufid.color = grey66 +console.value.clone.color = grey66 +console.value.controller.color = grey66 + +# highlight flags +console.flag.color = #875fff + +# show drop and recirculations +console.key.drop.color = red +console.key.resubmit.color = #00d700 +console.key.output.color = #00d700 +console.value.output.color = #00d700 + +# highlights +console.key.highlighted.color = red +console.key.highlighted.underline = true +console.value.highlighted.underline = true +console.delim.highlighted.underline = true + + +[styles.light] +# If a color is omitted, the default terminal color will be used +# highlight keys +console.key.color = blue + +# special types +console.value.type.IPAddress.color = #008700 +console.value.type.IPMask.color = #008700 +console.value.type.EthMask.color = #008700 + +# dim long arguments +console.value.ct.color = bright_black +console.value.ufid.color = #870000 +console.value.clone.color = bright_black +console.value.controller.color = bright_black + +# highlight flags +console.flag.color = #00005F + +# show drop and recirculations +console.key.drop.color = red +console.key.resubmit.color = #00d700 +console.key.output.color = #005f00 +console.value.output.color = #00d700 + +# highlights +console.key.highlighted.color = #f20905 +console.value.highlighted.color = #f20905 +console.key.highlighted.underline = true +console.value.highlighted.underline = true +console.delim.highlighted.underline = true From patchwork Fri Dec 1 19:14:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870789 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aIXzkDHH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSM0F70z23ng for ; Sat, 2 Dec 2023 06:15:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1960D42230; Fri, 1 Dec 2023 19:15:12 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 1960D42230 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aIXzkDHH X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l1irQz-FWKGw; Fri, 1 Dec 2023 19:15:08 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id D82AB42234; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D82AB42234 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 73DEDC0DDB; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id A07BBC0DD3 for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 69B9B400F6 for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 69B9B400F6 Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aIXzkDHH X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1xTTtZ85qURn for ; Fri, 1 Dec 2023 19:15:01 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8F3B4436E4 for ; Fri, 1 Dec 2023 19:15:01 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 8F3B4436E4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458100; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PB718PIEG60Ozwus+fRIE7coxd4hcQJFwHsc8hsLG4M=; b=aIXzkDHHrd+VH6LJTs21sHv9at74YGkj+BZaENSdFvprB6TihIiYSwc5jb5r6tdKL4Eu0D ipp05ugqEfEgRCcac6almkIn1OAYqbERgvLuo7/6igYoocqyoqNd5uBRcu+q7FKKN6g/5x 5mcLQssI8WjrYWa6iOUwA2JHiAmjGlI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-593-qF5GzQtfO-2_G50JhCJinQ-1; Fri, 01 Dec 2023 14:14:59 -0500 X-MC-Unique: qF5GzQtfO-2_G50JhCJinQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2CF50918563 for ; Fri, 1 Dec 2023 19:14:59 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9AC961121307; Fri, 1 Dec 2023 19:14:58 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:41 +0100 Message-ID: <20231201191449.2386134-6-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 05/10] python: ovs: flowviz: Add html formatting. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a HTML Formatter and use it to print OpenFlow flows in an HTML list with table links. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 3 +- python/ovs/flowviz/html_format.py | 136 ++++++++++++++++++++++++++++ python/ovs/flowviz/ofp/cli.py | 10 ++ python/ovs/flowviz/ofp/html.py | 80 ++++++++++++++++ python/ovs/flowviz/ovs-flowviz.conf | 16 +++- 5 files changed, 243 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/html_format.py create mode 100644 python/ovs/flowviz/ofp/html.py diff --git a/python/automake.mk b/python/automake.mk index cf8b71659..b4c1f84be 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -67,15 +67,16 @@ ovs_flowviz = \ python/ovs/flowviz/__init__.py \ python/ovs/flowviz/console.py \ python/ovs/flowviz/format.py \ + python/ovs/flowviz/html_format.py \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ python/ovs/flowviz/odp/cli.py \ python/ovs/flowviz/ofp/__init__.py \ python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ofp/html.py \ python/ovs/flowviz/ovs-flowviz \ python/ovs/flowviz/process.py - # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ diff --git a/python/ovs/flowviz/html_format.py b/python/ovs/flowviz/html_format.py new file mode 100644 index 000000000..ebfa65c34 --- /dev/null +++ b/python/ovs/flowviz/html_format.py @@ -0,0 +1,136 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovs.flowviz.format import FlowFormatter, FlowBuffer, FlowStyle + + +class HTMLStyle: + """HTMLStyle defines a style for html-formatted flows. + + Args: + color(str): Optional; a string representing the CSS color to use + anchor_gen(callable): Optional; a callable to be used to generate the + href + """ + + def __init__(self, color=None, anchor_gen=None): + self.color = color + self.anchor_gen = anchor_gen + + +class HTMLBuffer(FlowBuffer): + """HTMLBuffer implementes FlowBuffer to provide html-based flow formatting. + + Each flow gets formatted as: +
...
+ """ + + def __init__(self): + self._text = "" + + @property + def text(self): + return self._text + + def _append(self, string, color, href): + """Append a key a string""" + style = ' style="color:{}"'.format(color) if color else "" + self._text += "".format(style) + if href: + self._text += "".format(href) + self._text += string + if href: + self._text += "" + self._text += "" + + def append_key(self, kv, style): + """Append a key. + Args: + kv (KeyValue): the KeyValue instance to append + style (HTMLStyle): the style to use + """ + href = style.anchor_gen(kv) if (style and style.anchor_gen) else "" + return self._append( + kv.meta.kstring, style.color if style else "", href + ) + + def append_delim(self, kv, style): + """Append a delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (HTMLStyle): the style to use + """ + href = style.anchor_gen(kv) if (style and style.anchor_gen) else "" + return self._append(kv.meta.delim, style.color if style else "", href) + + def append_end_delim(self, kv, style): + """Append an end delimiter. + Args: + kv (KeyValue): the KeyValue instance to append + style (HTMLStyle): the style to use + """ + href = style.anchor_gen(kv) if (style and style.anchor_gen) else "" + return self._append( + kv.meta.end_delim, style.color if style else "", href + ) + + def append_value(self, kv, style): + """Append a value. + Args: + kv (KeyValue): the KeyValue instance to append + style (HTMLStyle): the style to use + """ + href = style.anchor_gen(kv) if (style and style.anchor_gen) else "" + return self._append( + kv.meta.vstring, style.color if style else "", href + ) + + def append_extra(self, extra, style): + """Append extra string. + Args: + kv (KeyValue): the KeyValue instance to append + style (HTMLStyle): the style to use + """ + return self._append(extra, style.color if style else "", "") + + +class HTMLFormatter(FlowFormatter): + """Formts a flow in HTML Format.""" + + default_style_obj = FlowStyle( + { + "value.resubmit": HTMLStyle( + anchor_gen=lambda x: "#table_{}".format(x.value["table"]) + ), + "default": HTMLStyle(), + } + ) + + def __init__(self, opts=None): + super(HTMLFormatter, self).__init__() + self.style = ( + self._style_from_opts(opts, "html", HTMLStyle) or FlowStyle() + ) + + def format_flow(self, buf, flow, highlighted=None): + """Formats the flow into the provided buffer as a html object. + + Args: + buf (FlowBuffer): the flow buffer to append to + flow (ovs_dbg.OFPFlow): the flow to format + highlighted (list): Optional; list of KeyValues to highlight + """ + return super(HTMLFormatter, self).format_flow( + buf, flow, self.style, highlighted + ) diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py index a28e489ac..5917a6bf0 100644 --- a/python/ovs/flowviz/ofp/cli.py +++ b/python/ovs/flowviz/ofp/cli.py @@ -15,6 +15,7 @@ import click from ovs.flowviz.main import maincli +from ovs.flowviz.ofp.html import HTMLProcessor from ovs.flowviz.process import ( OpenFlowFactory, JSONProcessor, @@ -66,3 +67,12 @@ def console(opts, heat_map): ) proc.process() proc.print() + + +@openflow.command() +@click.pass_obj +def html(opts): + """Print the flows in an linked HTML list arranged by tables.""" + processor = HTMLProcessor(opts) + processor.process() + print(processor.html()) diff --git a/python/ovs/flowviz/ofp/html.py b/python/ovs/flowviz/ofp/html.py new file mode 100644 index 000000000..a66f5fe8e --- /dev/null +++ b/python/ovs/flowviz/ofp/html.py @@ -0,0 +1,80 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovs.flowviz.html_format import HTMLBuffer, HTMLFormatter, HTMLStyle +from ovs.flowviz.process import ( + OpenFlowFactory, + FileProcessor, +) + + +class HTMLProcessor(OpenFlowFactory, FileProcessor): + """File processor that prints Openflow tables in HTML.""" + + def __init__(self, opts): + super().__init__(opts) + self.data = dict() + + def start_file(self, name, filename): + self.tables = dict() + + def stop_file(self, name, filename): + self.data[name] = self.tables + + def process_flow(self, flow, name): + table = flow.info.get("table") or 0 + if not self.tables.get(table): + self.tables[table] = list() + self.tables[table].append(flow) + + def html(self): + html_obj = "" + for name, tables in self.data.items(): + name = name.replace(" ", "_") + html_obj += "

{}

".format(name) + html_obj += "
" + for table, flows in tables.items(): + formatter = HTMLFormatter(self.opts) + + def anchor(x): + return "#table_%s_%s" % (name, x.value["table"]) + + formatter.style.set_value_style( + "resubmit", + HTMLStyle( + formatter.style.get("value.resubmit"), + anchor_gen=anchor, + ), + ) + html_obj += ( + "

Table {table}

".format( + name=name, table=table + ) + ) + html_obj += "
    ".format(table) + for flow in flows: + html_obj += "
  • ".format(flow.id) + highlighted = None + if self.opts.get("highlight"): + result = self.opts.get("highlight").evaluate(flow) + if result: + highlighted = result.kv + buf = HTMLBuffer() + formatter.format_flow(buf, flow, highlighted) + html_obj += buf.text + html_obj += "
    • " + html_obj += "
" + html_obj += "
" + + return html_obj diff --git a/python/ovs/flowviz/ovs-flowviz.conf b/python/ovs/flowviz/ovs-flowviz.conf index 3acd0a29e..165c453ec 100644 --- a/python/ovs/flowviz/ovs-flowviz.conf +++ b/python/ovs/flowviz/ovs-flowviz.conf @@ -4,7 +4,7 @@ # # [FORMAT].[PORTION].[SELECTOR].[ELEMENT] = [VALUE] # -# * FORMAT: console +# * FORMAT: console or html # * PORTION: The portion of the flow that the style applies to # - key: Selects how to print the key of a KeyValue pair # - key: Selects how to print the value of a KeyValue pair @@ -25,6 +25,11 @@ # - underline: if set to "true", the selected portion will be underlined # #[1] https://rich.readthedocs.io/en/stable/appendix/colors.html#standard-colors +# +# HTML Styles +# ============== +# * ELEMENT: +# - color: defines the color in hex format [styles.dark] @@ -92,3 +97,12 @@ console.value.highlighted.color = #f20905 console.key.highlighted.underline = true console.value.highlighted.underline = true console.delim.highlighted.underline = true + +# html +html.key.color = #00005f +html.value.color = #870000 +html.key.resubmit.color = #00d700 +html.key.output.color = #005f00 +html.value.output.color = #00d700 +html.key.highlighted.color = #FF00FF +html.value.highlighted.color = #FF00FF From patchwork Fri Dec 1 19:14:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870788 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LmzC8Ek8; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSL2rzpz1ySh for ; Sat, 2 Dec 2023 06:15:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5BD8943780; Fri, 1 Dec 2023 19:15:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 5BD8943780 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LmzC8Ek8 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eqdbLqaE_p_s; Fri, 1 Dec 2023 19:15:09 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5D22D4376E; Fri, 1 Dec 2023 19:15:08 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 5D22D4376E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 31F10C0DE0; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id A3064C0DDA for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7F45084938 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 7F45084938 Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LmzC8Ek8 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AVpSsnWSF7ml for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id ECE1284939 for ; Fri, 1 Dec 2023 19:15:02 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org ECE1284939 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BpzQ7W/yxJ93Ve/alsFg1YuNP2h3llVr5OsRh74x99g=; b=LmzC8Ek8GtNUdbUSzqaN5vLUjmdNovA7L1mqS+3ri2BfMOUnHvBtgw5hQizn/W6oC3kwqr xH4f/Ko3VDxyh/FeDuUW9v0QTsHezavX8ImQmVfDQpLDuji0XCfjcJfY7LJ8Xcxb7U/jHm OSN4BzFFgHZUGXTeKwroMhoutfRbYsA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-649-LvSAY8uKOw2MQvqk0p-aEg-1; Fri, 01 Dec 2023 14:15:00 -0500 X-MC-Unique: LvSAY8uKOw2MQvqk0p-aEg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F3EF91C05EB5 for ; Fri, 1 Dec 2023 19:14:59 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6DCC21121307; Fri, 1 Dec 2023 19:14:59 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:42 +0100 Message-ID: <20231201191449.2386134-7-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 06/10] python: ovs: flowviz: Add datapath tree format. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Datapath flows can be arranged into a "tree"-like structure based on recirculation ids, e.g: recirc(0),eth(...),ipv4(...) actions=ct,recirc(0x42) \-> recirc(42),ct_state(0/0),eth(...),ipv4(...) actions=1 \-> recirc(42),ct_state(1/0),eth(...),ipv4(...) actions=userspace(...) This patch adds support for building such logical datapath trees in a format-agnostic way and adds support for console-based formatting supporting: - head-maps formatting of statistics - hash-based pallete of recirculation ids: each recirculation id is assigned a unique color to easily follow the sequence of related actions. - full-tree filtering: if a user specifies a filter, an entire subtree is filtered out if none of its branches satisfy it. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flowviz/console.py | 22 +++ python/ovs/flowviz/odp/cli.py | 21 ++- python/ovs/flowviz/odp/tree.py | 290 +++++++++++++++++++++++++++++++++ 4 files changed, 332 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/odp/tree.py diff --git a/python/automake.mk b/python/automake.mk index b4c1f84be..5050089e9 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -71,6 +71,7 @@ ovs_flowviz = \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ python/ovs/flowviz/odp/cli.py \ + python/ovs/flowviz/odp/tree.py \ python/ovs/flowviz/ofp/__init__.py \ python/ovs/flowviz/ofp/cli.py \ python/ovs/flowviz/ofp/html.py \ diff --git a/python/ovs/flowviz/console.py b/python/ovs/flowviz/console.py index 5b4b047c2..2d65f9bb6 100644 --- a/python/ovs/flowviz/console.py +++ b/python/ovs/flowviz/console.py @@ -13,6 +13,9 @@ # limitations under the License. import colorsys +import itertools +import zlib + from rich.console import Console from rich.text import Text from rich.style import Style @@ -169,6 +172,25 @@ def heat_pallete(min_value, max_value): return heat +def hash_pallete(hue, saturation, value): + """Generates a color pallete with the cartesian product + of the hsv values provided and returns a callable that assigns a color for + each value hash + """ + HSV_tuples = itertools.product(hue, saturation, value) + RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples) + styles = [ + Style(color=Color.from_rgb(r * 255, g * 255, b * 255)) + for r, g, b in RGB_tuples + ] + + def get_style(string): + hash_val = zlib.crc32(bytes(str(string), "utf-8")) + return styles[hash_val % len(styles)] + + return get_style + + def default_highlight(): """Generates a default style for highlights.""" return Style(underline=True) diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py index 78f5cfff4..4740e753e 100644 --- a/python/ovs/flowviz/odp/cli.py +++ b/python/ovs/flowviz/odp/cli.py @@ -13,12 +13,12 @@ # limitations under the License. import click - from ovs.flowviz.main import maincli +from ovs.flowviz.odp.tree import ConsoleTreeProcessor from ovs.flowviz.process import ( DatapathFactory, - JSONProcessor, ConsoleProcessor, + JSONProcessor, ) @@ -65,3 +65,20 @@ def console(opts, heat_map): ) proc.process() proc.print() + + +@datapath.command() +@click.option( + "-h", + "--heat-map", + is_flag=True, + default=False, + show_default=True, + help="Create heat-map with packet and byte counters", +) +@click.pass_obj +def tree(opts, heat_map): + """Print the flows in a tree based on the 'recirc_id'.""" + processor = ConsoleTreeProcessor(opts) + processor.process() + processor.print(heat_map) diff --git a/python/ovs/flowviz/odp/tree.py b/python/ovs/flowviz/odp/tree.py new file mode 100644 index 000000000..cfddb162e --- /dev/null +++ b/python/ovs/flowviz/odp/tree.py @@ -0,0 +1,290 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from rich.style import Style +from rich.text import Text +from rich.tree import Tree + +from ovs.flowviz.console import ( + ConsoleFormatter, + ConsoleBuffer, + hash_pallete, + heat_pallete, + file_header, +) +from ovs.flowviz.process import ( + DatapathFactory, + FileProcessor, +) + + +class TreeElem: + """Element in the tree. + Args: + children (list[TreeElem]): Optional, list of children + is_root (bool): Optional; whether this is the root elemen + """ + + def __init__(self, children=None, is_root=False): + self.children = children or list() + self.is_root = is_root + + def append(self, child): + self.children.append(child) + + +class FlowElem(TreeElem): + """An element that contains a flow. + Args: + flow (Flow): The flow that this element contains + children (list[TreeElem]): Optional, list of children + is_root (bool): Optional; whether this is the root elemen + """ + + def __init__(self, flow, children=None, is_root=False): + self.flow = flow + super(FlowElem, self).__init__(children, is_root) + + def evaluate_any(self, filter): + """Evaluate the filter on the element and all its children. + Args: + filter(OFFilter): the filter to evaluate + + Returns: + True if ANY of the flows (including self and children) evaluates + true + """ + if filter.evaluate(self.flow): + return True + + return any([child.evaluate_any(filter) for child in self.children]) + + +class FlowTree: + """A Flow tree is a a class that processes datapath flows into a tree based + on recirculation ids. + + Args: + flows (list[ODPFlow]): Optional, initial list of flows + root (TreeElem): Optional, root of the tree. + """ + + def __init__(self, flows=None, root=TreeElem(is_root=True)): + self._flows = {} + self.root = root + if flows: + for flow in flows: + self.add(flow) + + def add(self, flow): + """Add a flow""" + rid = flow.match.get("recirc_id") or 0 + if not self._flows.get(rid): + self._flows[rid] = list() + self._flows[rid].append(flow) + + def build(self): + """Build the flow tree.""" + self._build(self.root, 0) + + def traverse(self, callback): + """Traverses the tree calling callback on each element. + + callback: callable that accepts two TreeElem, the current one being + traversed and its parent + func callback(elem, parent): + ... + Note that "parent" can be None if it's the first element. + """ + self._traverse(self.root, None, callback) + + def _traverse(self, elem, parent, callback): + callback(elem, parent) + + for child in elem.children: + self._traverse(child, elem, callback) + + def _build(self, parent, recirc): + """Build the subtree starting at a specific recirc_id. Recursive function. + + Args: + parent (TreeElem): parent of the (sub)tree + recirc(int): the recirc_id subtree to build + """ + flows = self._flows.get(recirc) + if not flows: + return + for flow in sorted( + flows, key=lambda x: x.info.get("packets") or 0, reverse=True + ): + next_recircs = self._get_next_recirc(flow) + + elem = self._new_elem(flow, parent) + parent.append(elem) + + for next_recirc in next_recircs: + self._build(elem, next_recirc) + + def _get_next_recirc(self, flow): + """Get the next recirc_ids from a Flow. + + The recirc_id is obtained from actions such as recirc, but also + complex actions such as check_pkt_len and sample + Args: + flow (ODPFlow): flow to get the recirc_id from. + Returns: + set of next recirculation ids. + """ + + # Helper function to find a recirc in a dictionary of actions. + def find_in_list(actions_list): + recircs = [] + for item in actions_list: + (action, value) = next(iter(item.items())) + if action == "recirc": + recircs.append(value) + elif action == "check_pkt_len": + recircs.extend(find_in_list(value.get("gt"))) + recircs.extend(find_in_list(value.get("le"))) + elif action == "clone": + recircs.extend(find_in_list(value)) + elif action == "sample": + recircs.extend(find_in_list(value.get("actions"))) + return recircs + + recircs = [] + recircs.extend(find_in_list(flow.actions)) + + return set(recircs) + + def _new_elem(self, flow, _): + """Creates a new TreeElem. + + Default implementation is to create a FlowElem. Derived classes can + override this method to return any derived TreeElem + """ + return FlowElem(flow) + + def filter(self, filter): + """Removes the first level subtrees if none of its sub-elements match + the filter. + + Args: + filter(OFFilter): filter to apply + """ + to_remove = list() + for l0 in self.root.children: + passes = l0.evaluate_any(filter) + if not passes: + to_remove.append(l0) + for elem in to_remove: + self.root.children.remove(elem) + + +class ConsoleTreeProcessor(DatapathFactory, FileProcessor): + def __init__(self, opts): + super().__init__(opts) + self.data = dict() + self.ofconsole = ConsoleFormatter(self.opts) + + # Generate a color pallete for cookies + recirc_style_gen = hash_pallete( + hue=[x / 50 for x in range(0, 50)], saturation=[0.7], value=[0.8] + ) + + style = self.ofconsole.style + style.set_default_value_style(Style(color="grey66")) + style.set_key_style("output", Style(color="green")) + style.set_value_style("output", Style(color="green")) + style.set_value_style("recirc", recirc_style_gen) + style.set_value_style("recirc_id", recirc_style_gen) + + def start_file(self, name, filename): + self.tree = ConsoleTree(self.ofconsole, self.opts) + + def process_flow(self, flow, name): + self.tree.add(flow) + + def process(self): + super().process(False) + + def stop_file(self, name, filename): + self.data[name] = self.tree + + def print(self, heat_map): + for name, tree in self.data.items(): + self.ofconsole.console.print("\n") + self.ofconsole.console.print(file_header(name)) + tree.build() + if self.opts.get("filter"): + tree.filter(self.opts.get("filter")) + tree.print(heat_map) + + +class ConsoleTree(FlowTree): + """ConsoleTree is a FlowTree that prints the tree in the console. + + Args: + console (ConsoleFormatter): console to use for printing + opts (dict): Options dictionary + """ + + class ConsoleElem(FlowElem): + def __init__(self, flow=None, is_root=False): + self.tree = None + super(ConsoleTree.ConsoleElem, self).__init__( + flow, is_root=is_root + ) + + def __init__(self, console, opts): + self.console = console + self.opts = opts + super(ConsoleTree, self).__init__(root=self.ConsoleElem(is_root=True)) + + def _new_elem(self, flow, _): + """Override _new_elem to provide ConsoleElems""" + return self.ConsoleElem(flow) + + def _append_to_tree(self, elem, parent): + """Callback to be used for FlowTree._build + Appends the flow to the rich.Tree + """ + if elem.is_root: + elem.tree = Tree("Datapath Flows (logical)") + return + + buf = ConsoleBuffer(Text()) + highlighted = None + if self.opts.get("highlight"): + result = self.opts.get("highlight").evaluate(elem.flow) + if result: + highlighted = result.kv + self.console.format_flow(buf, elem.flow, highlighted) + elem.tree = parent.tree.add(buf.text) + + def print(self, heat=False): + """Print the Flow Tree. + Args: + heat (bool): Optional; whether heat-map style shall be applied + """ + if heat: + for field in ["packets", "bytes"]: + values = [] + for flow_list in self._flows.values(): + values.extend([f.info.get(field) or 0 for f in flow_list]) + self.console.style.set_value_style( + field, heat_pallete(min(values), max(values)) + ) + self.traverse(self._append_to_tree) + self.console.console.print(self.root.tree) From patchwork Fri Dec 1 19:14:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870795 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ii9FzSGi; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSc4d9bz1ySh for ; Sat, 2 Dec 2023 06:15:28 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B3891437E0; Fri, 1 Dec 2023 19:15:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B3891437E0 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ii9FzSGi X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XaaV-4DRXvIg; Fri, 1 Dec 2023 19:15:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5E95343781; Fri, 1 Dec 2023 19:15:17 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 5E95343781 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5EADDC0DE0; Fri, 1 Dec 2023 19:15:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3120FC0DDF for ; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id AB79D42225 for ; Fri, 1 Dec 2023 19:15:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org AB79D42225 Authentication-Results: smtp4.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ii9FzSGi X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nFJvGWai_u69 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 95FEF4220E for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 95FEF4220E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=slCjH33BKQTgyNJTWduDZ1KTqSjXxSraNpa/yNEX/3o=; b=ii9FzSGiJMCpgpJFR/uhaMomUq+pyql1VJvgbQz2AzYt7FKpG7c72FdSANSvpCMg1GR1qi /TCTiHXLw2HvgN1O0N/06bs+oDhdVRdXcB5952zEvaQI8LLgHVh0fqzGn8C5UaYNbtNPKy 6WY0MC7DuCdNFGdS2HV7WMGwLLo9Iao= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-127-3_UjbdlgNPiLkWTULr2_rA-1; Fri, 01 Dec 2023 14:15:01 -0500 X-MC-Unique: 3_UjbdlgNPiLkWTULr2_rA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CB70985A58A for ; Fri, 1 Dec 2023 19:15:00 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4169B1121307; Fri, 1 Dec 2023 19:15:00 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:43 +0100 Message-ID: <20231201191449.2386134-8-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 07/10] python: ovs: flowviz: Add OpenFlow logical view. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This view is interesting for debugging the logical pipeline. It arranges the flows in "logical" groups (not to be confused with OVN's Logical_Flows). A logical group of flows is a set of flows that: - Have the same table number and priority - Match on the same fields (regardless of the value they match against) - Have the same actions, regardless of the arguments for those actions, except for output and recirc, for which arguments do care. Optionally, the cookie can also be force to be unique for the logical group. By doing so, we can extend the information we show by querying an external OVN database and running "ovn-detrace" on each cookie. The result is a compact list of flow groups with interlieved OVN information. Furthermore, if connected to an OVN database, we can apply an OVN regexp filter. Examples: $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h $ export OVN_NB_DB=... $ export OVN_SB_DB=... $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d --ovn-filter="acl.*icmp4" Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 4 +- python/ovs/flowviz/ofp/cli.py | 113 ++++++++++++ python/ovs/flowviz/ofp/logic.py | 303 ++++++++++++++++++++++++++++++++ 3 files changed, 418 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/ofp/logic.py diff --git a/python/automake.mk b/python/automake.mk index 5050089e9..fdffafbc5 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -74,12 +74,12 @@ ovs_flowviz = \ python/ovs/flowviz/odp/tree.py \ python/ovs/flowviz/ofp/__init__.py \ python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ofp/logic.py \ python/ovs/flowviz/ofp/html.py \ python/ovs/flowviz/ovs-flowviz \ python/ovs/flowviz/process.py -# These python files are used at build time but not runtime, -# so they are not installed. +# These python files are used at build time but not runtime, so they are not installed. EXTRA_DIST += \ python/ovs_build_helpers/__init__.py \ python/ovs_build_helpers/extract_ofp_fields.py \ diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py index 5917a6bf0..6b1435ea1 100644 --- a/python/ovs/flowviz/ofp/cli.py +++ b/python/ovs/flowviz/ofp/cli.py @@ -12,10 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +import os + import click from ovs.flowviz.main import maincli from ovs.flowviz.ofp.html import HTMLProcessor +from ovs.flowviz.ofp.logic import LogicFlowProcessor from ovs.flowviz.process import ( OpenFlowFactory, JSONProcessor, @@ -69,6 +72,116 @@ def console(opts, heat_map): proc.print() +def ovn_detrace_callback(ctx, param, value): + """click callback to add detrace information to config object and + set general ovn-detrace flag to True + """ + ctx.obj[param.name] = value + if value != param.default: + ctx.obj["ovn_detrace_flag"] = True + return value + + +@openflow.command() +@click.option( + "-d", + "--ovn-detrace", + "ovn_detrace_flag", + is_flag=True, + show_default=True, + help="Use ovn-detrace to extract cookie information (implies '-c')", +) +@click.option( + "--ovn-detrace-path", + default="/usr/bin", + type=click.Path(), + help="Use an alternative path to where ovn_detrace.py is located. " + "Instead of using this option you can just set PYTHONPATH accordingly.", + show_default=True, + callback=ovn_detrace_callback, +) +@click.option( + "--ovnnb-db", + default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "--ovnsb-db", + default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "-o", + "--ovn-filter", + help="Specify a filter to be run on ovn-detrace information (implied -d). " + "Format: python regular expression " + "(see https://docs.python.org/3/library/re.html)", + callback=ovn_detrace_callback, +) +@click.option( + "-s", + "--show-flows", + is_flag=True, + default=False, + show_default=True, + help="Show the full flows under each logical flow", +) +@click.option( + "-c", + "--cookie", + "cookie_flag", + is_flag=True, + default=False, + show_default=True, + help="Consider the cookie in the logical flow", +) +@click.option( + "-h", + "--heat-map", + is_flag=True, + default=False, + show_default=True, + help="Create heat-map with packet and byte counters (when -s is used)", +) +@click.pass_obj +def logic( + opts, + ovn_detrace_flag, + ovn_detrace_path, + ovnnb_db, + ovnsb_db, + ovn_filter, + show_flows, + cookie_flag, + heat_map, +): + """ + Print the logical structure of the flows. + + First, sorts the flows based on tables and priorities. + Then, deduplicates logically equivalent flows: these a flows that match + on the same set of fields (regardless of the values they match against), + have the same priority, and actions (regardless of action arguments, + except in the case of output and recirculate). + Optionally, the cookie can also be considered to be part of the logical + flow. + """ + if ovn_detrace_flag: + opts["ovn_detrace_flag"] = True + if opts.get("ovn_detrace_flag"): + cookie_flag = True + + processor = LogicFlowProcessor(opts, cookie_flag, heat_map) + processor.process() + processor.print(show_flows) + + @openflow.command() @click.pass_obj def html(opts): diff --git a/python/ovs/flowviz/ofp/logic.py b/python/ovs/flowviz/ofp/logic.py new file mode 100644 index 000000000..cb4568cf1 --- /dev/null +++ b/python/ovs/flowviz/ofp/logic.py @@ -0,0 +1,303 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +import io +import re + +from rich.tree import Tree +from rich.text import Text + +from ovs.flowviz.process import FileProcessor, OpenFlowFactory +from ovs.flowviz.console import ( + ConsoleFormatter, + ConsoleBuffer, + hash_pallete, + file_header, + heat_pallete, +) + + +class LFlow: + """A Logical Flow represents the scheleton of a flow. + + Two logical flows have the same logical representation if they match + against same fields (regardless of the matching value) and have the same + set of actions (regardless of the actions' arguments, except for those + in the exact_actions list). + + Attributes: + flow (OFPFlow): The flow + exact_actions(list): Optional; list of action keys that are + considered unique if the value is also the same. + match_cookie (bool): Optional; if cookies are part of the logical + flow + """ + + def __init__(self, flow, exact_actions=[], match_cookie=False): + self.cookie = flow.info.get("cookie") or 0 if match_cookie else None + self.priority = flow.match.get("priority") or 0 + self.match_keys = tuple([kv.key for kv in flow.match_kv]) + + self.action_keys = tuple( + [ + kv.key + for kv in flow.actions_kv + if kv.key not in exact_actions + ] + ) + self.match_action_kvs = [ + kv for kv in flow.actions_kv if kv.key in exact_actions + ] + + def __eq__(self, other): + return ( + (self.cookie == other.cookie if self.cookie else True) + and self.priority == other.priority + and self.action_keys == other.action_keys + and self.equal_match_action_kvs(other) + and self.match_keys == other.match_keys + ) + + def equal_match_action_kvs(self, other): + """ Compares the logical flow's match action key-values with the + others. + + Args: + other (LFlow): The other LFlow to compare against + + Returns true if both LFlow have the same action k-v. + """ + if len(other.match_action_kvs) != len(self.match_action_kvs): + return False + + for kv in self.match_action_kvs: + found = False + for other_kv in other.match_action_kvs: + if self.match_kv(kv, other_kv): + found = True + break + if not found: + return False + return True + + def match_kv(self, one, other): + """Compares a KeyValue. + Args: + one, other (KeyValue): The objects to compare + + Returns true if both KeyValue objects have the same key and value + """ + return one.key == other.key and one.value == other.value + + def __hash__(self): + hash_data = [ + self.cookie, + self.priority, + self.action_keys, + tuple((kv.key, str(kv.value)) for kv in self.match_action_kvs), + self.match_keys, + ] + if self.cookie: + hash_data.append(self.cookie) + return tuple(hash_data).__hash__() + + def format(self, buf, formatter): + """Format the Logical Flow into a Buffer.""" + if self.cookie: + buf.append_extra( + "cookie={} ".format(hex(self.cookie)).ljust(18), + style=cookie_style_gen(str(self.cookie)), + ) + + buf.append_extra( + "priority={} ".format(self.priority), style="steel_blue" + ) + buf.append_extra(",".join(self.match_keys), style="steel_blue") + buf.append_extra(" ---> ", style="bold magenta") + buf.append_extra(",".join(self.action_keys), style="steel_blue") + + if len(self.match_action_kvs) > 0: + buf.append_extra(" ", style=None) + + for kv in self.match_action_kvs: + formatter.format_kv(buf, kv, formatter.style) + buf.append_extra(",", style=None) + + +class LogicFlowProcessor(OpenFlowFactory, FileProcessor): + def __init__(self, opts, match_cookie, heat_map): + super().__init__(opts) + self.data = dict() + self.min_max = dict() + self.match_cookie = match_cookie + self.heat_map = ["n_packets", "n_bytes"] if heat_map else [] + self.ovn_detrace = ( + OVNDetrace(opts) if opts.get("ovn_detrace_flag") else None + ) + + def start_file(self, name, filename): + if len(self.heat_map) > 0: + self.min = [-1] * len(self.heat_map) + self.max = [0] * len(self.heat_map) + self.tables = dict() + + def stop_file(self, name, filename): + if len(self.heat_map) > 0: + self.min_max[name] = (self.min, self.max) + self.data[name] = self.tables + + def process_flow(self, flow, name): + """Sort the flows by table and logical flow.""" + # Running calculation of min and max values for all the fields that + # take place in the heatmap. + for i, field in enumerate(self.heat_map): + val = flow.info.get(field) + if self.min[i] == -1 or val < self.min[i]: + self.min[i] = val + if val > self.max[i]: + self.max[i] = val + + table = flow.info.get("table") or 0 + if not self.tables.get(table): + self.tables[table] = dict() + + # Group flows by logical hash + lflow = LFlow( + flow, + exact_actions=["output", "resubmit", "drop"], + match_cookie=self.match_cookie, + ) + + if not self.tables[table].get(lflow): + self.tables[table][lflow] = list() + + self.tables[table][lflow].append(flow) + + def print(self, show_flows): + formatter = ConsoleFormatter(opts=self.opts) + console = formatter.console + for name, tables in self.data.items(): + console.print("\n") + console.print(file_header(name)) + tree = Tree("Ofproto Flows (logical)") + + for table_num in sorted(tables.keys()): + table = tables[table_num] + table_tree = tree.add("** TABLE {} **".format(table_num)) + + if len(self.heat_map) > 0 and len(table.values()) > 0: + for i, field in enumerate(self.heat_map): + (min_val, max_val) = self.min_max[name][i] + self.console.style.set_value_style( + field, heat_pallete(min_val, max_val) + ) + + for lflow in sorted( + table.keys(), + key=(lambda x: x.priority), + reverse=True, + ): + flows = table[lflow] + ovn_info = None + if self.ovn_detrace: + ovn_info = self.ovn_detrace.get_ovn_info(lflow.cookie) + if self.opts.get("ovn_filter"): + ovn_regexp = re.compile( + self.opts.get("ovn_filter") + ) + if not ovn_regexp.search(ovn_info): + continue + + buf = ConsoleBuffer(Text()) + + lflow.format(buf, formatter) + buf.append_extra( + " ( x {} )".format(len(flows)), + style="dark_olive_green3", + ) + lflow_tree = table_tree.add(buf.text) + + if ovn_info: + ovn = lflow_tree.add("OVN Info") + for part in ovn_info.split("\n"): + if part.strip(): + ovn.add(part.strip()) + + if show_flows: + for flow in flows: + buf = ConsoleBuffer(Text()) + highlighted = None + if self.opts.get("highlight"): + result = self.opts.get("highlight").evaluate( + flow + ) + if result: + highlighted = result.kv + formatter.format_flow(buf, flow, highlighted) + lflow_tree.add(buf.text) + + console.print(tree) + + +class OVNDetrace(object): + def __init__(self, opts): + if not opts.get("ovn_detrace_flag"): + raise Exception("Cannot initialize OVN Detrace connection") + + if opts.get("ovn_detrace_path"): + sys.path.append(opts.get("ovn_detrace_path")) + + import ovn_detrace + + class FakePrinter(ovn_detrace.Printer): + def __init__(self): + self.buff = io.StringIO() + + def print_p(self, msg): + print(" * ", msg, file=self.buff) + + def print_h(self, msg): + print(" * ", msg, file=self.buff) + + def clear(self): + self.buff = io.StringIO() + + self.ovn_detrace = ovn_detrace + self.ovnnb_conn = ovn_detrace.OVSDB( + opts.get("ovnnb_db"), "OVN_Northbound" + ) + self.ovnsb_conn = ovn_detrace.OVSDB( + opts.get("ovnsb_db"), "OVN_Southbound" + ) + self.ovn_printer = FakePrinter() + self.cookie_handlers = ovn_detrace.get_cookie_handlers( + self.ovnnb_conn, self.ovnsb_conn, self.ovn_printer + ) + + def get_ovn_info(self, cookie): + self.ovn_printer.clear() + self.ovn_detrace.print_record_from_cookie( + self.ovnsb_conn, self.cookie_handlers, "{:x}".format(cookie) + ) + return self.ovn_printer.buff.getvalue() + + +# Try to make it easy to spot same cookies by printing them in different +# colors +cookie_style_gen = hash_pallete( + hue=[x / 10 for x in range(0, 10)], + saturation=[0.5], + value=[0.5 + x / 10 * (0.85 - 0.5) for x in range(0, 10)], +) From patchwork Fri Dec 1 19:14:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870793 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IfYFukK0; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSY1sTQz1ySh for ; Sat, 2 Dec 2023 06:15:25 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5141A43793; Fri, 1 Dec 2023 19:15:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 5141A43793 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IfYFukK0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vYyU65-gd3KW; Fri, 1 Dec 2023 19:15:21 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 81992400F6; Fri, 1 Dec 2023 19:15:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 81992400F6 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E5D5EC0DF1; Fri, 1 Dec 2023 19:15:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5599AC0DE1 for ; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 0E5194375B for ; Fri, 1 Dec 2023 19:15:06 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0E5194375B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m2FqiJqYeIH5 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7F3B643753 for ; Fri, 1 Dec 2023 19:15:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 7F3B643753 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQFQVWwlPTKzohxiR2o4NOSJRU9EUfp8zj/yGXO2Rn8=; b=IfYFukK0U2vjD2w9YNYDuBZVJIGDoQjLBrUD91A6WZC2HD3gp7BLkL8tnddAHHCisb1Lkv ba3g2kLtyDKskwOVzjtfRBTkwMAtnIxT5DYkiRNiTVJ6aArQ1eGw1uMDHxv/lq9Dg1wYw/ lGzUhK9f20OE81DTEaL3tbSaloIaHZE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-578-zSIFgGvmPc2XXBlnHD_QXw-1; Fri, 01 Dec 2023 14:15:01 -0500 X-MC-Unique: zSIFgGvmPc2XXBlnHD_QXw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9E43385A5B5 for ; Fri, 1 Dec 2023 19:15:01 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id 184581121307; Fri, 1 Dec 2023 19:15:00 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:44 +0100 Message-ID: <20231201191449.2386134-9-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 08/10] python: ovs: flowviz: Add Openflow cookie format. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When anaylizing OVN issues, it might be useful to see what OpenFlow flows were generated from each logical flow. In order to make it simpler to visualize this, add a cookie format that simply sorts the flows first by cookie, then by table. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/ovs/flowviz/ofp/cli.py | 57 ++++++++++++++++++++++++++++- python/ovs/flowviz/ofp/logic.py | 63 ++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 2 deletions(-) diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py index 6b1435ea1..9658d00d3 100644 --- a/python/ovs/flowviz/ofp/cli.py +++ b/python/ovs/flowviz/ofp/cli.py @@ -18,7 +18,7 @@ import click from ovs.flowviz.main import maincli from ovs.flowviz.ofp.html import HTMLProcessor -from ovs.flowviz.ofp.logic import LogicFlowProcessor +from ovs.flowviz.ofp.logic import CookieProcessor, LogicFlowProcessor from ovs.flowviz.process import ( OpenFlowFactory, JSONProcessor, @@ -182,6 +182,61 @@ def logic( processor.print(show_flows) +@openflow.command() +@click.option( + "-d", + "--ovn-detrace", + "ovn_detrace_flag", + is_flag=True, + show_default=True, + help="Use ovn-detrace to extract cookie information", +) +@click.option( + "--ovn-detrace-path", + default="/usr/bin", + type=click.Path(), + help="Use an alternative path to where ovn_detrace.py is located. " + "Instead of using this option you can just set PYTHONPATH accordingly", + show_default=True, + callback=ovn_detrace_callback, +) +@click.option( + "--ovnnb-db", + default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "--ovnsb-db", + default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "-o", + "--ovn-filter", + help="Specify a filter to be run on ovn-detrace information (implied -d). " + "Format: python regular expression " + "(see https://docs.python.org/3/library/re.html)", + callback=ovn_detrace_callback, +) +@click.pass_obj +def cookie( + opts, ovn_detrace_flag, ovn_detrace_path, ovnnb_db, ovnsb_db, ovn_filter +): + """Print the flow tables sorted by cookie.""" + if ovn_detrace_flag: + opts["ovn_detrace_flag"] = True + + processor = CookieProcessor(opts) + processor.process() + processor.print() + + @openflow.command() @click.pass_obj def html(opts): diff --git a/python/ovs/flowviz/ofp/logic.py b/python/ovs/flowviz/ofp/logic.py index cb4568cf1..9d244d137 100644 --- a/python/ovs/flowviz/ofp/logic.py +++ b/python/ovs/flowviz/ofp/logic.py @@ -200,7 +200,7 @@ class LogicFlowProcessor(OpenFlowFactory, FileProcessor): if len(self.heat_map) > 0 and len(table.values()) > 0: for i, field in enumerate(self.heat_map): (min_val, max_val) = self.min_max[name][i] - self.console.style.set_value_style( + formatter.style.set_value_style( field, heat_pallete(min_val, max_val) ) @@ -301,3 +301,64 @@ cookie_style_gen = hash_pallete( saturation=[0.5], value=[0.5 + x / 10 * (0.85 - 0.5) for x in range(0, 10)], ) + + +class CookieProcessor(OpenFlowFactory, FileProcessor): + """Processor that sorts flows into cookies and tables.""" + + def __init__(self, opts): + super().__init__(opts) + self.data = dict() + self.ovn_detrace = ( + OVNDetrace(opts) if opts.get("ovn_detrace_flag") else None + ) + + def start_file(self, name, filename): + self.cookies = dict() + + def stop_file(self, name, filename): + self.data[name] = self.cookies + + def process_flow(self, flow, name): + """Sort the flows by table and logical flow.""" + cookie = flow.info.get("cookie") or 0 + if not self.cookies.get(cookie): + self.cookies[cookie] = dict() + + table = flow.info.get("table") or 0 + if not self.cookies[cookie].get(table): + self.cookies[cookie][table] = list() + self.cookies[cookie][table].append(flow) + + def print(self): + ofconsole = ConsoleFormatter(opts=self.opts) + console = ofconsole.console + for name, cookies in self.data.items(): + console.print("\n") + console.print(file_header(name)) + tree = Tree("Ofproto Cookie Tree") + + for cookie, tables in cookies.items(): + ovn_info = None + if self.ovn_detrace: + ovn_info = self.ovn_detrace.get_ovn_info(cookie) + if self.opts.get("ovn_filter"): + ovn_regexp = re.compile(self.opts.get("ovn_filter")) + if not ovn_regexp.search(ovn_info): + continue + + cookie_tree = tree.add("** Cookie {} **".format(hex(cookie))) + if ovn_info: + ovn = cookie_tree.add("OVN Info") + for part in ovn_info.split("\n"): + if part.strip(): + ovn.add(part.strip()) + + tables_tree = cookie_tree.add("Tables") + for table, flows in tables.items(): + table_tree = tables_tree.add("* Table {} * ".format(table)) + for flow in flows: + buf = ConsoleBuffer(Text()) + ofconsole.format_flow(buf, flow) + table_tree.add(buf.text) + console.print(tree) From patchwork Fri Dec 1 19:14:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870797 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=jUIfovvK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjSj3FPkz1ySh for ; Sat, 2 Dec 2023 06:15:33 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 439264381B; Fri, 1 Dec 2023 19:15:31 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 439264381B Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=jUIfovvK X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eG_tTx92bAyA; Fri, 1 Dec 2023 19:15:28 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 29403437EC; Fri, 1 Dec 2023 19:15:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 29403437EC Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5B7CBC0DE1; Fri, 1 Dec 2023 19:15:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2A721C0DFA for ; Fri, 1 Dec 2023 19:15:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 8DF2F84977 for ; Fri, 1 Dec 2023 19:15:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 8DF2F84977 Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=jUIfovvK X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T8Q99tnxw1du for ; Fri, 1 Dec 2023 19:15:10 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1D8E084982 for ; Fri, 1 Dec 2023 19:15:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 1D8E084982 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mRf1GRgga8srs2h0UrIR3gA5JhXPrp4nr3vQ64pzZik=; b=jUIfovvKHywwkCgd9qxGf0LtLZLO32DZy8ahlLHkg8xrbLtRi4dKG6hZjEKVs4kyKvIVwi qQ6dv/Gi4fO18icIltELzaJU8DDmy6QLAVF208+jqja/8Wm/0WZi83RDryv4kMAklH89tT idRhWQ7DUKTdx99edm8fh/nwKVi1q7s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-196-jRD3sOaZOGKJQpMrq1nxNQ-1; Fri, 01 Dec 2023 14:15:02 -0500 X-MC-Unique: jRD3sOaZOGKJQpMrq1nxNQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6EF0085A58A for ; Fri, 1 Dec 2023 19:15:02 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD4841121307; Fri, 1 Dec 2023 19:15:01 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:45 +0100 Message-ID: <20231201191449.2386134-10-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 09/10] python: ovs: flowviz: Add datapath html format. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Using the existing FlowTree and HTMLFormatter, create an HTML tree visualization that also supports collapsing and expanding entire flow trees and subtrees. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flowviz/odp/cli.py | 10 ++ python/ovs/flowviz/odp/html.py | 259 +++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+) create mode 100644 python/ovs/flowviz/odp/html.py diff --git a/python/automake.mk b/python/automake.mk index fdffafbc5..b772cbcd6 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -71,6 +71,7 @@ ovs_flowviz = \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ python/ovs/flowviz/odp/cli.py \ + python/ovs/flowviz/odp/html.py \ python/ovs/flowviz/odp/tree.py \ python/ovs/flowviz/ofp/__init__.py \ python/ovs/flowviz/ofp/cli.py \ diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py index 4740e753e..059dd708e 100644 --- a/python/ovs/flowviz/odp/cli.py +++ b/python/ovs/flowviz/odp/cli.py @@ -14,6 +14,7 @@ import click from ovs.flowviz.main import maincli +from ovs.flowviz.odp.html import HTMLTreeProcessor from ovs.flowviz.odp.tree import ConsoleTreeProcessor from ovs.flowviz.process import ( DatapathFactory, @@ -82,3 +83,12 @@ def tree(opts, heat_map): processor = ConsoleTreeProcessor(opts) processor.process() processor.print(heat_map) + + +@datapath.command() +@click.pass_obj +def html(opts): + """Print the flows in an HTML list sorted by recirc_id.""" + processor = HTMLTreeProcessor(opts) + processor.process() + processor.print() diff --git a/python/ovs/flowviz/odp/html.py b/python/ovs/flowviz/odp/html.py new file mode 100644 index 000000000..4aa08dc70 --- /dev/null +++ b/python/ovs/flowviz/odp/html.py @@ -0,0 +1,259 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ovs.flowviz.html_format import HTMLBuffer, HTMLFormatter +from ovs.flowviz.odp.tree import FlowElem, FlowTree +from ovs.flowviz.process import DatapathFactory, FileProcessor + + +class HTMLTreeProcessor(DatapathFactory, FileProcessor): + def __init__(self, opts): + super().__init__(opts) + self.data = dict() + + def start_file(self, name, filename): + self.tree = HTMLTree(name, self.opts) + + def process_flow(self, flow, name): + self.tree.add(flow) + + def process(self): + super().process(False) + + def stop_file(self, name, filename): + self.data[name] = self.tree + + def print(self): + html_obj = "" + for name, tree in self.data.items(): + html_obj += "
" + html_obj += "

{}

".format(name) + tree.build() + if self.opts.get("filter"): + tree.filter(self.opts.get("filter")) + html_obj += tree.render() + html_obj += "
" + print(html_obj) + + +class HTMLTree(FlowTree): + """HTMLTree is a Flowtree that prints the tree in html format. + + Args: + opts(dict): Options dictionary + flows(dict[int, list[DPFlow]): Optional; initial flows + """ + + html_header = """ + + + + """ # noqa: E501 + + class HTMLTreeElem(FlowElem): + """An element within the HTML Tree. + + It is composed of a flow and its subflows that can be added by calling + append() + """ + + def __init__(self, parent_name, flow=None, opts=None): + self._parent_name = parent_name + self._formatter = HTMLFormatter(opts) + self._opts = opts + super(HTMLTree.HTMLTreeElem, self).__init__(flow) + + def render(self, item=0): + """Render the HTML Element. + Args: + item (int): the item id + + Returns: + (html_obj, items) tuple where html_obj is the html string and + items is the number of subitems rendered in total + """ + parent_name = self._parent_name.replace(" ", "_") + html_obj = "
" + if self.flow: + html_text = """ + + + """ # noqa: E501 + html_obj += html_text.format( + item=item, id=self.flow.id, name=parent_name + ) + + html_text = '
' # noqa: E501 + html_obj += html_text.format(id=self.flow.id) + buf = HTMLBuffer() + highlighted = None + if self._opts.get("highlight"): + result = self._opts.get("highlight").evaluate(self.flow) + if result: + highlighted = result.kv + self._formatter.format_flow(buf, self.flow, highlighted) + html_obj += buf.text + html_obj += "
" + if self.children: + html_obj += "
" + html_obj += "
    " + for sf in self.children: + item += 1 + html_obj += "
  • " + (html_elem, items) = sf.render(item) + html_obj += html_elem + item += items + html_obj += "
    • " + html_obj += "
" + html_obj += "
" + html_obj += "
" + return html_obj, item + + def __init__(self, name, opts, flows=None): + self.opts = opts + self.name = name + super(HTMLTree, self).__init__( + flows, self.HTMLTreeElem("", flow=None, opts=self.opts) + ) + + def _new_elem(self, flow, _): + """Override _new_elem to provide HTMLTreeElems.""" + return self.HTMLTreeElem(self.name, flow, self.opts) + + def render(self): + """Render the Tree in HTML. + + Returns: + an html string representing the element + """ + name = self.name.replace(" ", "_") + + html_text = """ +""" # noqa: E501 + html_obj = self.html_header + html_text.format(name=name) + + html_obj += "
".format(name=name) + (html_elem, _) = self.root.render() + html_obj += html_elem + html_obj += "
" + return html_obj From patchwork Fri Dec 1 19:14:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1870798 X-Patchwork-Delegate: horms@verge.net.au Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=RlS4HaGZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4ShjT16PQtz1ySh for ; Sat, 2 Dec 2023 06:15:49 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D5A7942666; Fri, 1 Dec 2023 19:15:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D5A7942666 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=RlS4HaGZ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PwFA-5cN-EVG; Fri, 1 Dec 2023 19:15:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id F1BFA42499; Fri, 1 Dec 2023 19:15:28 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org F1BFA42499 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 393BEC008E; Fri, 1 Dec 2023 19:15:26 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 09723C008E for ; Fri, 1 Dec 2023 19:15:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id DF5D2423F8 for ; Fri, 1 Dec 2023 19:15:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org DF5D2423F8 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dBmUiagSNwxN for ; Fri, 1 Dec 2023 19:15:17 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id E280642245 for ; Fri, 1 Dec 2023 19:15:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org E280642245 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701458113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZqLdweFrczP8UkM2mCW66CNBRTz0QkOUvdN3kErQ0Lc=; b=RlS4HaGZAj5gouasn5bD5GparxoeyBaHRd5E+jjwN1d2m9tktYtMlmVDHrrhJKAvitYY0K 2w3d10hPDk21LaPhQurvq5wlKnmkdsqBLNz57R+JYusy40A1koqvBVfZ197nO9CuKZTT5B 9P/RDOrIm/6pwl3TeXoQAn1LKbiDhKk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-597-SpeQDDNlMnmNhp7AD2NSKA-1; Fri, 01 Dec 2023 14:15:06 -0500 X-MC-Unique: SpeQDDNlMnmNhp7AD2NSKA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48EAD101A54C for ; Fri, 1 Dec 2023 19:15:03 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.193.138]) by smtp.corp.redhat.com (Postfix) with ESMTP id B016B1121307; Fri, 1 Dec 2023 19:15:02 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 1 Dec 2023 20:14:46 +0100 Message-ID: <20231201191449.2386134-11-amorenoz@redhat.com> In-Reply-To: <20231201191449.2386134-1-amorenoz@redhat.com> References: <20231201191449.2386134-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [RFC PATCH 10/10] python: ovs: flowviz: Add datapath graph format. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Graph view leverages the tree format (specially the tree-based filtering) and uses graphviz library to build a visual graph of the datapath in graphviz format. Conntrack zones are shown in random colors to help visualize connection tracking interdependencies. An html flag builds an HTML page with both the html flows and the graph (in svg) that enables navegation. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flowviz/odp/cli.py | 22 ++ python/ovs/flowviz/odp/graph.py | 418 ++++++++++++++++++++++++++++++++ python/ovs/flowviz/odp/tree.py | 18 +- python/setup.py | 2 +- 5 files changed, 457 insertions(+), 4 deletions(-) create mode 100644 python/ovs/flowviz/odp/graph.py diff --git a/python/automake.mk b/python/automake.mk index b772cbcd6..e75b34e45 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -71,6 +71,7 @@ ovs_flowviz = \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ python/ovs/flowviz/odp/cli.py \ + python/ovs/flowviz/odp/graph.py \ python/ovs/flowviz/odp/html.py \ python/ovs/flowviz/odp/tree.py \ python/ovs/flowviz/ofp/__init__.py \ diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py index 059dd708e..3e470ff1e 100644 --- a/python/ovs/flowviz/odp/cli.py +++ b/python/ovs/flowviz/odp/cli.py @@ -14,6 +14,7 @@ import click from ovs.flowviz.main import maincli +from ovs.flowviz.odp.graph import GraphProcessor from ovs.flowviz.odp.html import HTMLTreeProcessor from ovs.flowviz.odp.tree import ConsoleTreeProcessor from ovs.flowviz.process import ( @@ -92,3 +93,24 @@ def html(opts): processor = HTMLTreeProcessor(opts) processor.process() processor.print() + + +@datapath.command() +@click.option( + "-h", + "--html", + is_flag=True, + default=False, + show_default=True, + help="Output an html file containing the graph", +) +@click.pass_obj +def graph(opts, html): + """Print the flows in an graphviz (.dot) format showing the relationship + of recirc_ids.""" + if len(opts.get("filename")) > 1: + raise click.BadParameter("Graph format only supports one input file") + + processor = GraphProcessor(opts) + processor.process() + processor.print(html) diff --git a/python/ovs/flowviz/odp/graph.py b/python/ovs/flowviz/odp/graph.py new file mode 100644 index 000000000..b26551e67 --- /dev/null +++ b/python/ovs/flowviz/odp/graph.py @@ -0,0 +1,418 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" Defines a Datapath Graph using graphviz. """ +import colorsys +import graphviz +import random + +from ovs.flowviz.odp.html import HTMLTree +from ovs.flowviz.odp.tree import FlowTree +from ovs.flowviz.process import DatapathFactory, FileProcessor + + +class GraphProcessor(DatapathFactory, FileProcessor): + def __init__(self, opts): + super().__init__(opts) + + def start_file(self, name, filename): + self.tree = FlowTree() + + def process_flow(self, flow, name): + self.tree.add(flow) + + def process(self): + super().process(False) + + def print(self, html): + flows = {} + + # Tree traverse callback + def add_flow(elem, _): + if elem.is_root: + return + rid = elem.flow.match.get("recirc_id") or 0 + if not flows.get(rid): + flows[rid] = set() + flows[rid].add(elem.flow) + + self.tree.build() + if self.opts.get("filter"): + self.tree.filter(self.opts.get("filter")) + self.tree.traverse(add_flow) + + if len(flows) == 0: + return + + dpg = DatapathGraph(flows) + if not html: + print(dpg.source()) + return + + html_obj = "" + html_obj += "

Flow Graph

" + html_obj += "
" + svg = dpg.pipe(format="svg") + html_obj += svg.decode("utf-8") + html_obj += "
" + html_tree = HTMLTree("graph", self.opts, flows) + html_tree.build() + html_obj += html_tree.render() + + print(html_obj) + + +class DatapathGraph: + """A DatapathGraph is a class that renders a set of datapath flows into + graphviz graphs. + + Args: + flows(dict[int, list(Flow)]): Dictionary of lists of flows indexed by + recirc_id + """ + + ct_styles = {} + node_styles = { + "default": { + "style": {}, + "desc": "Default", + }, + "action_and_match": { + "style": {"color": "#ff00ff"}, + "desc": "Flow uses CT as match and action", + }, + "match": { + "style": {"color": "#0000ff"}, + "desc": "Flow uses CT only to match", + }, + "action": { + "style": {"color": "#ff0000"}, + "desc": "Flow uses CT only as action", + }, + } + + def __init__(self, flows): + self._flows = flows + + self._output_nodes = [] + self._graph = graphviz.Digraph( + "DP flows", node_attr={"shape": "rectangle"} + ) + self._graph.attr(compound="true") + self._graph.attr(rankdir="LR") + self._graph.attr(ranksep="3") + + self._populate_graph() + + def source(self): + """Return the graphviz source representation of the graph.""" + return self._graph.source + + def pipe(self, *args, **kwargs): + """Output the graph based on arguments given to graphviz.pipe.""" + return self._graph.pipe(*args, **kwargs) + + @classmethod + def recirc_cluster_name(cls, recirc_id): + """Name of the recirculation cluster.""" + return "cluster_recirc_{}".format(hex(recirc_id)) + + @classmethod + def inport_cluster_name(cls, inport): + """Name of the input port cluster.""" + return "cluster_inport_{}".format(inport) + + @classmethod + def invis_node_name(cls, cluster_name): + """Name of the invisible node.""" + return "invis_{}".format(cluster_name) + + @classmethod + def output_node_name(cls, port): + """Name of the ouput node.""" + return "output_{}".format(port) + + def _flow_node(self, flow, name): + """Returns the dictionary of attributes of a graphviz node that + represents the flow with a given name. + """ + summary = "Line: {} \n".format(flow.id) + summary += "\n".join( + [ + flow.section("info").string, + ",".join(flow.match.keys()), + "actions: " + + ",".join(list(a.keys())[0] for a in flow.actions), + ] + ) + + has_ct_match = flow.match.get("ct_state", "0/0") != "0/0" + has_ct_action = bool( + next( + filter(lambda x: x.key in ["ct", "ct_clear"], flow.actions_kv), + None, + ) + ) + + if has_ct_action: + if has_ct_match: + style = "action_and_match" + else: + style = "action" + elif has_ct_match: + style = "match" + else: + style = "default" + + style = self.node_styles.get(style, {}) + + return { + "name": name, + "label": summary, + "tooltip": flow.orig, + "_attributes": style.get("style", {}), + "fontsize": "10", + "nojustify": "true", + "URL": "#flow_{}".format(flow.id), + } + + def _create_recirc_cluster(self, recirc): + """Process a recirculation id, creating its cluster.""" + cluster_name = self.recirc_cluster_name(recirc) + label = "recirc x0{:0x}".format(recirc) + + cluster = self._graph.subgraph(name=cluster_name, comment=label) + with cluster as sg: + sg.attr(rankdir="TB") + sg.attr(ranksep="0.02") + sg.attr(label=label) + sg.attr(margin="5") + self._add_flows_to_graph(sg, self._flows[recirc]) + + self.processed_recircs.append(recirc) + + def _add_flows_to_graph(self, graph, flows): + # Create an invisible node and an edge to the first flow so that + # it ends up at the top of the cluster. + invis = self.invis_node_name(graph.name) + graph.node(invis) + graph.node( + invis, + color="white", + len="0", + shape="point", + width="0", + height="0", + ) + first = True + for flow in flows: + name = "Flow_{}".format(flow.id) + graph.node(**self._flow_node(flow, name)) + if first: + with graph.subgraph() as c: + c.attr(rank="same") + c.edge(name, invis, style="invis") + first = False + # determine next hop based on actions + self._set_next_node_from_actions(name, flow.actions) + + def set_next_node_from_actions(self, name, actions): + """Determine the next nodes based on action list and add edges to + them. + """ + if not self._set_next_node_from_actions(self, name, actions): + # Add to a generic "End" if no other action was detected + self._graph.edge(name, "end") + + def _set_next_node_from_actions(self, name, actions): + created = False + for action in actions: + key, value = next(iter(action.items())) + if key == "check_pkt_len": + created |= self._set_next_node_from_actions( + name, value.get("gt") + ) + created |= self._set_next_node_from_actions( + name, value.get("le") + ) + elif key == "sample": + created |= self._set_next_node_from_actions( + name, value.get("actions") + ) + elif key == "clone": + created |= self._set_next_node_from_actions( + name, value.get("actions") + ) + else: + created |= self._set_next_node_action(name, key, value) + return created + + def _set_next_node_action(self, name, action_name, action_obj): + """Based on the action object, set the next node.""" + if action_name == "recirc": + # If the targer recirculation cluster has not yet been created, + # do it now. + if action_obj not in self.processed_recircs: + self._create_recirc_cluster(action_obj) + + cname = self.recirc_cluster_name(action_obj) + self._graph.edge( + name, + self.invis_node_name(cname), + lhead=cname, + _attributes={"weight": "20"}, + ) + return True + elif action_name == "output": + port = action_obj.get("port") + if port not in self._output_nodes: + self._output_nodes.append(port) + self._graph.edge( + name, self.output_node_name(port), _attributes={"weight": "1"} + ) + return True + elif action_name in ["drop", "userspace", "controller"]: + if action_name not in self._output_nodes: + self._output_nodes.append(action_name) + self._graph.edge(name, action_name, _attributes={"weight": "1"}) + return True + elif action_name == "ct": + zone = action_obj.get("zone", 0) + node_name = "CT zone {}".format(action_obj.get("zone", "default")) + if zone not in self.ct_styles: + # Pick a random (highly saturated) color. + (r, g, b) = colorsys.hsv_to_rgb(random.random(), 1, 1) + color = "#%02x%02x%02x" % ( + int(r * 255), + int(g * 255), + int(b * 255), + ) + self.ct_styles[zone] = color + self._graph.node(node_name, color=color) + + color = self.ct_styles[zone] + self._graph.edge(name, node_name, style="dashed", color=color) + # test + name = node_name + return True + return False + + def _populate_graph(self): + """Populate the the internal graph.""" + self.processed_recircs = [] + + # Create a subcluster for each input port and one for flows that don't + # have in_port() for which we create a dummy inport. + flows_per_inport = {} + free_flows = [] + + for flow in self._flows.get(0): + port = flow.match.get("in_port") + if port: + if not flows_per_inport.get(port): + flows_per_inport[port] = list() + flows_per_inport[port].append(flow) + else: + free_flows.append(flow) + + # It's rare to find flows without input_port match but let's add them + # nevertheless. + if free_flows: + self._graph.edge( + "start", + self.invis_node_name(self.recirc_cluster_name(0)), + lhead=self.recirc_cluster_name(0), + ) + self._graph.node("no_port", shape="Mdiamond") + + # Recirc_clusters are created recursively when an edge is found to + # them. + # Process recirc(0) which is split by input port. + for inport, flows in flows_per_inport.items(): + # Build a subgraph per input port + cluster_name = self.inport_cluster_name(inport) + label = "recirc 0; input port: {}".format(inport) + + with self._graph.subgraph( + name=cluster_name, comment=label + ) as per_port: + per_port.attr(rankdir="TB") + per_port.attr(ranksep="0.02") + per_port.attr(margin="5") + per_port.attr(label=label) + self._add_flows_to_graph(per_port, flows_per_inport[inport]) + + # Create an input node that points to each input subgraph + # They are all inside an anonymous subgraph so that they can be + # alligned. + with self._graph.subgraph() as s: + s.attr(rank="same") + for inport in flows_per_inport: + # Make an Input node point to each subgraph + node_name = "input_{}".format(inport) + cluster_name = self.inport_cluster_name(inport) + s.node( + node_name, + shape="Mdiamond", + label="input port {}".format(inport), + ) + self._graph.edge( + node_name, + self.invis_node_name(cluster_name), + lhead=cluster_name, + _attributes={"weight": "20"}, + ) + + # Create the output nodes in a subgraph so that they are alligned + with self._graph.subgraph() as s: + for port in self._output_nodes: + s.attr(rank="same") + if port == "drop": + s.node( + "drop", + shape="Msquare", + color="red", + label="DROP", + rank="sink", + ) + elif port == "controller": + s.node( + "controller", + shape="Msquare", + color="blue", + label="CONTROLLER", + rank="sink", + ) + elif port == "userspace": + s.node( + "userspace", + shape="Msquare", + color="blue", + label="CONTROLLER", + rank="sink", + ) + else: + s.node( + self.output_node_name(port), + shape="Msquare", + color="green", + label="Port {}".format(port), + rank="sink", + ) + + # Print style legend + with self._graph.subgraph(name="cluster_legend") as s: + s.attr(label="Legend") + for style in self.node_styles.values(): + s.node(name=style.get("desc"), _attributes=style.get("style")) diff --git a/python/ovs/flowviz/odp/tree.py b/python/ovs/flowviz/odp/tree.py index cfddb162e..5439d6499 100644 --- a/python/ovs/flowviz/odp/tree.py +++ b/python/ovs/flowviz/odp/tree.py @@ -76,7 +76,8 @@ class FlowTree: on recirculation ids. Args: - flows (list[ODPFlow]): Optional, initial list of flows + flows (list[ODPFlow]): Optional, initial list of flows or dictionary of + flows indexed by recirc_id root (TreeElem): Optional, root of the tree. """ @@ -84,8 +85,15 @@ class FlowTree: self._flows = {} self.root = root if flows: - for flow in flows: - self.add(flow) + if isinstance(flows, dict): + self._flows = flows + elif isinstance(flows, list): + for flow in flows: + self.add(flow) + else: + raise Exception( + "flows in wrong format: {}".format(type(flows)) + ) def add(self, flow): """Add a flow""" @@ -191,6 +199,10 @@ class FlowTree: for elem in to_remove: self.root.children.remove(elem) + def all(self): + """Return all the flows in a dictionary by recirc_id.""" + return self._flows + class ConsoleTreeProcessor(DatapathFactory, FileProcessor): def __init__(self, opts): diff --git a/python/setup.py b/python/setup.py index 38e72ab62..5e26ffc17 100644 --- a/python/setup.py +++ b/python/setup.py @@ -114,7 +114,7 @@ setup_args = dict( 'flow': flow_extras_require, 'dns': ['unbound'], 'flowviz': - [*flow_extras_require, 'click', 'rich'], + [*flow_extras_require, 'click', 'rich', 'graphviz'], }, scripts=["ovs/flowviz/ovs-flowviz"], data_files=["ovs/flowviz/ovs-flowviz.conf"],