| Message ID | 20231201191449.2386134-8-amorenoz@redhat.com |
|---|---|
| State | RFC |
| Delegated to: | Simon Horman |
| Headers | show |
| Series | Add flow visualization utility | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/intel-ovs-compilation | fail | test: fail |
| ovsrobot/github-robot-_Build_and_Test | fail | github build: failed |
On 1 Dec 2023, at 20:14, Adrian Moreno wrote: > Datapath flows can be arranged into a "tree"-like structure based on > recirculation ids, e.g: > > recirc(0),eth(...),ipv4(...) actions=ct,recirc(0x42) > \-> recirc(42),ct_state(0/0),eth(...),ipv4(...) actions=1 > \-> recirc(42),ct_state(1/0),eth(...),ipv4(...) actions=userspace(...) > > This patch adds support for building such logical datapath trees in a > format-agnostic way and adds support for console-based formatting > supporting: > - head-maps formatting of statistics > - hash-based pallete of recirculation ids: each recirculation id is > assigned a unique color to easily follow the sequence of related > actions. > - full-tree filtering: if a user specifies a filter, an entire subtree > is filtered out if none of its branches satisfy it. > > Signed-off-by: Adrian Moreno <amorenoz@redhat.com> This patch looks good to me. One small nit on a comment not ending with a dot. Acked-by: Eelco Chaudron <echaudro@redhat.com> > --- > python/automake.mk | 1 + > python/ovs/flowviz/console.py | 22 +++ > python/ovs/flowviz/odp/cli.py | 21 ++- > python/ovs/flowviz/odp/tree.py | 290 +++++++++++++++++++++++++++++++++ > 4 files changed, 332 insertions(+), 2 deletions(-) > create mode 100644 python/ovs/flowviz/odp/tree.py > > diff --git a/python/automake.mk b/python/automake.mk > index b4c1f84be..5050089e9 100644 > --- a/python/automake.mk > +++ b/python/automake.mk > @@ -71,6 +71,7 @@ ovs_flowviz = \ > python/ovs/flowviz/main.py \ > python/ovs/flowviz/odp/__init__.py \ > python/ovs/flowviz/odp/cli.py \ > + python/ovs/flowviz/odp/tree.py \ > python/ovs/flowviz/ofp/__init__.py \ > python/ovs/flowviz/ofp/cli.py \ > python/ovs/flowviz/ofp/html.py \ > diff --git a/python/ovs/flowviz/console.py b/python/ovs/flowviz/console.py > index 5b4b047c2..2d65f9bb6 100644 > --- a/python/ovs/flowviz/console.py > +++ b/python/ovs/flowviz/console.py > @@ -13,6 +13,9 @@ > # limitations under the License. > > import colorsys > +import itertools > +import zlib > + > from rich.console import Console > from rich.text import Text > from rich.style import Style > @@ -169,6 +172,25 @@ def heat_pallete(min_value, max_value): > return heat > > > +def hash_pallete(hue, saturation, value): > + """Generates a color pallete with the cartesian product > + of the hsv values provided and returns a callable that assigns a color for > + each value hash > + """ > + HSV_tuples = itertools.product(hue, saturation, value) > + RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples) > + styles = [ > + Style(color=Color.from_rgb(r * 255, g * 255, b * 255)) > + for r, g, b in RGB_tuples > + ] > + > + def get_style(string): > + hash_val = zlib.crc32(bytes(str(string), "utf-8")) > + return styles[hash_val % len(styles)] > + > + return get_style > + > + > def default_highlight(): > """Generates a default style for highlights.""" > return Style(underline=True) > diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py > index 78f5cfff4..4740e753e 100644 > --- a/python/ovs/flowviz/odp/cli.py > +++ b/python/ovs/flowviz/odp/cli.py > @@ -13,12 +13,12 @@ > # limitations under the License. > > import click > - > from ovs.flowviz.main import maincli > +from ovs.flowviz.odp.tree import ConsoleTreeProcessor > from ovs.flowviz.process import ( > DatapathFactory, > - JSONProcessor, > ConsoleProcessor, > + JSONProcessor, > ) > > > @@ -65,3 +65,20 @@ def console(opts, heat_map): > ) > proc.process() > proc.print() > + > + > +@datapath.command() > +@click.option( > + "-h", > + "--heat-map", > + is_flag=True, > + default=False, > + show_default=True, > + help="Create heat-map with packet and byte counters", > +) > +@click.pass_obj > +def tree(opts, heat_map): > + """Print the flows in a tree based on the 'recirc_id'.""" > + processor = ConsoleTreeProcessor(opts) > + processor.process() > + processor.print(heat_map) > diff --git a/python/ovs/flowviz/odp/tree.py b/python/ovs/flowviz/odp/tree.py > new file mode 100644 > index 000000000..cfddb162e > --- /dev/null > +++ b/python/ovs/flowviz/odp/tree.py > @@ -0,0 +1,290 @@ > +# Copyright (c) 2023 Red Hat, Inc. > +# > +# Licensed under the Apache License, Version 2.0 (the "License"); > +# you may not use this file except in compliance with the License. > +# You may obtain a copy of the License at: > +# > +# http://www.apache.org/licenses/LICENSE-2.0 > +# > +# Unless required by applicable law or agreed to in writing, software > +# distributed under the License is distributed on an "AS IS" BASIS, > +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > +# See the License for the specific language governing permissions and > +# limitations under the License. > + > +from rich.style import Style > +from rich.text import Text > +from rich.tree import Tree > + > +from ovs.flowviz.console import ( > + ConsoleFormatter, > + ConsoleBuffer, > + hash_pallete, > + heat_pallete, > + file_header, > +) > +from ovs.flowviz.process import ( > + DatapathFactory, > + FileProcessor, > +) > + > + > +class TreeElem: > + """Element in the tree. > + Args: > + children (list[TreeElem]): Optional, list of children > + is_root (bool): Optional; whether this is the root elemen > + """ > + > + def __init__(self, children=None, is_root=False): > + self.children = children or list() > + self.is_root = is_root > + > + def append(self, child): > + self.children.append(child) > + > + > +class FlowElem(TreeElem): > + """An element that contains a flow. > + Args: > + flow (Flow): The flow that this element contains > + children (list[TreeElem]): Optional, list of children > + is_root (bool): Optional; whether this is the root elemen > + """ > + > + def __init__(self, flow, children=None, is_root=False): > + self.flow = flow > + super(FlowElem, self).__init__(children, is_root) > + > + def evaluate_any(self, filter): > + """Evaluate the filter on the element and all its children. > + Args: > + filter(OFFilter): the filter to evaluate > + > + Returns: > + True if ANY of the flows (including self and children) evaluates > + true > + """ > + if filter.evaluate(self.flow): > + return True > + > + return any([child.evaluate_any(filter) for child in self.children]) > + > + > +class FlowTree: > + """A Flow tree is a a class that processes datapath flows into a tree based > + on recirculation ids. > + > + Args: > + flows (list[ODPFlow]): Optional, initial list of flows > + root (TreeElem): Optional, root of the tree. > + """ > + > + def __init__(self, flows=None, root=TreeElem(is_root=True)): > + self._flows = {} > + self.root = root > + if flows: > + for flow in flows: > + self.add(flow) > + > + def add(self, flow): > + """Add a flow""" > + rid = flow.match.get("recirc_id") or 0 > + if not self._flows.get(rid): > + self._flows[rid] = list() > + self._flows[rid].append(flow) > + > + def build(self): > + """Build the flow tree.""" > + self._build(self.root, 0) > + > + def traverse(self, callback): > + """Traverses the tree calling callback on each element. > + > + callback: callable that accepts two TreeElem, the current one being > + traversed and its parent > + func callback(elem, parent): > + ... > + Note that "parent" can be None if it's the first element. > + """ > + self._traverse(self.root, None, callback) > + > + def _traverse(self, elem, parent, callback): > + callback(elem, parent) > + > + for child in elem.children: > + self._traverse(child, elem, callback) > + > + def _build(self, parent, recirc): > + """Build the subtree starting at a specific recirc_id. Recursive function. > + > + Args: > + parent (TreeElem): parent of the (sub)tree > + recirc(int): the recirc_id subtree to build > + """ > + flows = self._flows.get(recirc) > + if not flows: > + return > + for flow in sorted( > + flows, key=lambda x: x.info.get("packets") or 0, reverse=True > + ): > + next_recircs = self._get_next_recirc(flow) > + > + elem = self._new_elem(flow, parent) > + parent.append(elem) > + > + for next_recirc in next_recircs: > + self._build(elem, next_recirc) > + > + def _get_next_recirc(self, flow): > + """Get the next recirc_ids from a Flow. > + > + The recirc_id is obtained from actions such as recirc, but also > + complex actions such as check_pkt_len and sample > + Args: > + flow (ODPFlow): flow to get the recirc_id from. > + Returns: > + set of next recirculation ids. > + """ > + > + # Helper function to find a recirc in a dictionary of actions. > + def find_in_list(actions_list): > + recircs = [] > + for item in actions_list: > + (action, value) = next(iter(item.items())) > + if action == "recirc": > + recircs.append(value) > + elif action == "check_pkt_len": > + recircs.extend(find_in_list(value.get("gt"))) > + recircs.extend(find_in_list(value.get("le"))) > + elif action == "clone": > + recircs.extend(find_in_list(value)) > + elif action == "sample": > + recircs.extend(find_in_list(value.get("actions"))) > + return recircs > + > + recircs = [] > + recircs.extend(find_in_list(flow.actions)) > + > + return set(recircs) > + > + def _new_elem(self, flow, _): > + """Creates a new TreeElem. > + > + Default implementation is to create a FlowElem. Derived classes can > + override this method to return any derived TreeElem > + """ > + return FlowElem(flow) > + > + def filter(self, filter): > + """Removes the first level subtrees if none of its sub-elements match > + the filter. > + > + Args: > + filter(OFFilter): filter to apply > + """ > + to_remove = list() > + for l0 in self.root.children: > + passes = l0.evaluate_any(filter) > + if not passes: > + to_remove.append(l0) > + for elem in to_remove: > + self.root.children.remove(elem) > + > + > +class ConsoleTreeProcessor(DatapathFactory, FileProcessor): > + def __init__(self, opts): > + super().__init__(opts) > + self.data = dict() > + self.ofconsole = ConsoleFormatter(self.opts) > + > + # Generate a color pallete for cookies Ending comment line with a dot? > + recirc_style_gen = hash_pallete( > + hue=[x / 50 for x in range(0, 50)], saturation=[0.7], value=[0.8] > + ) > + > + style = self.ofconsole.style > + style.set_default_value_style(Style(color="grey66")) > + style.set_key_style("output", Style(color="green")) > + style.set_value_style("output", Style(color="green")) > + style.set_value_style("recirc", recirc_style_gen) > + style.set_value_style("recirc_id", recirc_style_gen) > + > + def start_file(self, name, filename): > + self.tree = ConsoleTree(self.ofconsole, self.opts) > + > + def process_flow(self, flow, name): > + self.tree.add(flow) > + > + def process(self): > + super().process(False) > + > + def stop_file(self, name, filename): > + self.data[name] = self.tree > + > + def print(self, heat_map): > + for name, tree in self.data.items(): > + self.ofconsole.console.print("\n") > + self.ofconsole.console.print(file_header(name)) > + tree.build() > + if self.opts.get("filter"): > + tree.filter(self.opts.get("filter")) > + tree.print(heat_map) > + > + > +class ConsoleTree(FlowTree): > + """ConsoleTree is a FlowTree that prints the tree in the console. > + > + Args: > + console (ConsoleFormatter): console to use for printing > + opts (dict): Options dictionary > + """ > + > + class ConsoleElem(FlowElem): > + def __init__(self, flow=None, is_root=False): > + self.tree = None > + super(ConsoleTree.ConsoleElem, self).__init__( > + flow, is_root=is_root > + ) > + > + def __init__(self, console, opts): > + self.console = console > + self.opts = opts > + super(ConsoleTree, self).__init__(root=self.ConsoleElem(is_root=True)) > + > + def _new_elem(self, flow, _): > + """Override _new_elem to provide ConsoleElems""" > + return self.ConsoleElem(flow) > + > + def _append_to_tree(self, elem, parent): > + """Callback to be used for FlowTree._build > + Appends the flow to the rich.Tree > + """ > + if elem.is_root: > + elem.tree = Tree("Datapath Flows (logical)") > + return > + > + buf = ConsoleBuffer(Text()) > + highlighted = None > + if self.opts.get("highlight"): > + result = self.opts.get("highlight").evaluate(elem.flow) > + if result: > + highlighted = result.kv > + self.console.format_flow(buf, elem.flow, highlighted) > + elem.tree = parent.tree.add(buf.text) > + > + def print(self, heat=False): > + """Print the Flow Tree. > + Args: > + heat (bool): Optional; whether heat-map style shall be applied > + """ > + if heat: > + for field in ["packets", "bytes"]: > + values = [] > + for flow_list in self._flows.values(): > + values.extend([f.info.get(field) or 0 for f in flow_list]) > + self.console.style.set_value_style( > + field, heat_pallete(min(values), max(values)) > + ) > + self.traverse(self._append_to_tree) > + self.console.console.print(self.root.tree) > -- > 2.43.0 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev <
On 30 Jan 2024, at 16:52, Eelco Chaudron wrote: > On 1 Dec 2023, at 20:14, Adrian Moreno wrote: > >> Datapath flows can be arranged into a "tree"-like structure based on >> recirculation ids, e.g: >> >> recirc(0),eth(...),ipv4(...) actions=ct,recirc(0x42) >> \-> recirc(42),ct_state(0/0),eth(...),ipv4(...) actions=1 >> \-> recirc(42),ct_state(1/0),eth(...),ipv4(...) actions=userspace(...) >> >> This patch adds support for building such logical datapath trees in a >> format-agnostic way and adds support for console-based formatting >> supporting: >> - head-maps formatting of statistics >> - hash-based pallete of recirculation ids: each recirculation id is >> assigned a unique color to easily follow the sequence of related >> actions. >> - full-tree filtering: if a user specifies a filter, an entire subtree >> is filtered out if none of its branches satisfy it. >> >> Signed-off-by: Adrian Moreno <amorenoz@redhat.com> > > This patch looks good to me. One small nit on a comment not ending with a dot. > > Acked-by: Eelco Chaudron <echaudro@redhat.com> > Ignore this email, it was supposed to be a reply to patch 6 :(
On 1 Dec 2023, at 20:14, Adrian Moreno wrote: > This view is interesting for debugging the logical pipeline. It arranges > the flows in "logical" groups (not to be confused with OVN's > Logical_Flows). A logical group of flows is a set of flows that: > - Have the same table number and priority > - Match on the same fields (regardless of the value they match against) > - Have the same actions, regardless of the arguments for tose actions, > except for output and recirc, for which arguments do care. > > Optionally, the cookie can also be force to be unique for the logical > group. By doing so, we can extend the information we show by querying an > external OVN database and running "ovn-detrace" on each cookie. The > result is a compact list of flow groups with interlieved OVN > information. > > Furthermore, if connected to an OVN database, we can apply an OVN > regexp filter. > > Examples: > $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic > $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h > $ export OVN_NB_DB=... > $ export OVN_SB_DB=... > $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d > $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d > --ovn-filter="acl.*icmp4" When trying some random trace I got the following: $ ovs-flowviz -i ~/ofctl.txt -l "drop" --style=dark openflow logic -h ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ 📜 Filename: /home/echaudron/ofctl.txt 📜 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ Traceback (most recent call last): File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/bin/ovs-flowviz", line 20, in <module> main.main() File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/main.py", line 196, in main maincli() File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/decorators.py", line 45, in new_func return f(get_current_context().obj, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/cli.py", line 182, in logic processor.print(show_flows) File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/logic.py", line 203, in print self.console.style.set_value_style( ^^^^^^^^^^^^ AttributeError: 'LogicFlowProcessor' object has no attribute 'console' This gets solved if I apply the next patch, see the comment in the next patch. The code itself looks good, I did not test with the actual ovs-detrace tool. > Signed-off-by: Adrian Moreno <amorenoz@redhat.com> > --- > python/automake.mk | 4 +- > python/ovs/flowviz/ofp/cli.py | 113 ++++++++++++ > python/ovs/flowviz/ofp/logic.py | 303 ++++++++++++++++++++++++++++++++ > 3 files changed, 418 insertions(+), 2 deletions(-) > create mode 100644 python/ovs/flowviz/ofp/logic.py > > diff --git a/python/automake.mk b/python/automake.mk > index 5050089e9..fdffafbc5 100644 > --- a/python/automake.mk > +++ b/python/automake.mk > @@ -74,12 +74,12 @@ ovs_flowviz = \ > python/ovs/flowviz/odp/tree.py \ > python/ovs/flowviz/ofp/__init__.py \ > python/ovs/flowviz/ofp/cli.py \ > + python/ovs/flowviz/ofp/logic.py \ > python/ovs/flowviz/ofp/html.py \ > python/ovs/flowviz/ovs-flowviz \ > python/ovs/flowviz/process.py > > -# These python files are used at build time but not runtime, > -# so they are not installed. > +# These python files are used at build time but not runtime, so they are not installed. > EXTRA_DIST += \ > python/ovs_build_helpers/__init__.py \ > python/ovs_build_helpers/extract_ofp_fields.py \ > diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py > index 5917a6bf0..6b1435ea1 100644 > --- a/python/ovs/flowviz/ofp/cli.py > +++ b/python/ovs/flowviz/ofp/cli.py > @@ -12,10 +12,13 @@ > # See the License for the specific language governing permissions and > # limitations under the License. > > +import os > + > import click > > from ovs.flowviz.main import maincli > from ovs.flowviz.ofp.html import HTMLProcessor > +from ovs.flowviz.ofp.logic import LogicFlowProcessor > from ovs.flowviz.process import ( > OpenFlowFactory, > JSONProcessor, > @@ -69,6 +72,116 @@ def console(opts, heat_map): > proc.print() > > > +def ovn_detrace_callback(ctx, param, value): > + """click callback to add detrace information to config object and > + set general ovn-detrace flag to True > + """ > + ctx.obj[param.name] = value > + if value != param.default: > + ctx.obj["ovn_detrace_flag"] = True > + return value > + > + > +@openflow.command() > +@click.option( > + "-d", > + "--ovn-detrace", > + "ovn_detrace_flag", > + is_flag=True, > + show_default=True, > + help="Use ovn-detrace to extract cookie information (implies '-c')", > +) > +@click.option( > + "--ovn-detrace-path", > + default="/usr/bin", > + type=click.Path(), > + help="Use an alternative path to where ovn_detrace.py is located. " > + "Instead of using this option you can just set PYTHONPATH accordingly.", > + show_default=True, > + callback=ovn_detrace_callback, > +) > +@click.option( > + "--ovnnb-db", > + default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock", > + help="Specify the OVN NB database string (implies -d). " > + "If the OVN_NB_DB environment variable is set, it's used as default. " > + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", > + callback=ovn_detrace_callback, > +) > +@click.option( > + "--ovnsb-db", > + default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock", > + help="Specify the OVN NB database string (implies -d). " > + "If the OVN_NB_DB environment variable is set, it's used as default. " > + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", > + callback=ovn_detrace_callback, > +) > +@click.option( > + "-o", > + "--ovn-filter", > + help="Specify a filter to be run on ovn-detrace information (implied -d). " > + "Format: python regular expression " > + "(see https://docs.python.org/3/library/re.html)", > + callback=ovn_detrace_callback, > +) > +@click.option( > + "-s", > + "--show-flows", > + is_flag=True, > + default=False, > + show_default=True, > + help="Show the full flows under each logical flow", > +) > +@click.option( > + "-c", > + "--cookie", > + "cookie_flag", > + is_flag=True, > + default=False, > + show_default=True, > + help="Consider the cookie in the logical flow", > +) > +@click.option( > + "-h", > + "--heat-map", > + is_flag=True, > + default=False, > + show_default=True, > + help="Create heat-map with packet and byte counters (when -s is used)", > +) > +@click.pass_obj > +def logic( > + opts, > + ovn_detrace_flag, > + ovn_detrace_path, > + ovnnb_db, > + ovnsb_db, > + ovn_filter, > + show_flows, > + cookie_flag, > + heat_map, > +): > + """ > + Print the logical structure of the flows. > + > + First, sorts the flows based on tables and priorities. > + Then, deduplicates logically equivalent flows: these a flows that match > + on the same set of fields (regardless of the values they match against), > + have the same priority, and actions (regardless of action arguments, > + except in the case of output and recirculate). > + Optionally, the cookie can also be considered to be part of the logical > + flow. > + """ > + if ovn_detrace_flag: > + opts["ovn_detrace_flag"] = True > + if opts.get("ovn_detrace_flag"): > + cookie_flag = True > + > + processor = LogicFlowProcessor(opts, cookie_flag, heat_map) > + processor.process() > + processor.print(show_flows) > + > + > @openflow.command() > @click.pass_obj > def html(opts): > diff --git a/python/ovs/flowviz/ofp/logic.py b/python/ovs/flowviz/ofp/logic.py > new file mode 100644 > index 000000000..cb4568cf1 > --- /dev/null > +++ b/python/ovs/flowviz/ofp/logic.py > @@ -0,0 +1,303 @@ > +# Copyright (c) 2023 Red Hat, Inc. > +# > +# Licensed under the Apache License, Version 2.0 (the "License"); > +# you may not use this file except in compliance with the License. > +# You may obtain a copy of the License at: > +# > +# http://www.apache.org/licenses/LICENSE-2.0 > +# > +# Unless required by applicable law or agreed to in writing, software > +# distributed under the License is distributed on an "AS IS" BASIS, > +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > +# See the License for the specific language governing permissions and > +# limitations under the License. > + > +import sys > +import io > +import re > + > +from rich.tree import Tree > +from rich.text import Text > + > +from ovs.flowviz.process import FileProcessor, OpenFlowFactory > +from ovs.flowviz.console import ( > + ConsoleFormatter, > + ConsoleBuffer, > + hash_pallete, > + file_header, > + heat_pallete, > +) > + > + > +class LFlow: > + """A Logical Flow represents the scheleton of a flow. > + > + Two logical flows have the same logical representation if they match > + against same fields (regardless of the matching value) and have the same > + set of actions (regardless of the actions' arguments, except for those > + in the exact_actions list). > + > + Attributes: > + flow (OFPFlow): The flow > + exact_actions(list): Optional; list of action keys that are > + considered unique if the value is also the same. > + match_cookie (bool): Optional; if cookies are part of the logical > + flow > + """ > + > + def __init__(self, flow, exact_actions=[], match_cookie=False): > + self.cookie = flow.info.get("cookie") or 0 if match_cookie else None > + self.priority = flow.match.get("priority") or 0 > + self.match_keys = tuple([kv.key for kv in flow.match_kv]) > + > + self.action_keys = tuple( > + [ > + kv.key > + for kv in flow.actions_kv > + if kv.key not in exact_actions > + ] > + ) > + self.match_action_kvs = [ > + kv for kv in flow.actions_kv if kv.key in exact_actions > + ] > + > + def __eq__(self, other): > + return ( > + (self.cookie == other.cookie if self.cookie else True) > + and self.priority == other.priority > + and self.action_keys == other.action_keys > + and self.equal_match_action_kvs(other) > + and self.match_keys == other.match_keys > + ) > + > + def equal_match_action_kvs(self, other): > + """ Compares the logical flow's match action key-values with the > + others. > + > + Args: > + other (LFlow): The other LFlow to compare against > + > + Returns true if both LFlow have the same action k-v. > + """ > + if len(other.match_action_kvs) != len(self.match_action_kvs): > + return False > + > + for kv in self.match_action_kvs: > + found = False > + for other_kv in other.match_action_kvs: > + if self.match_kv(kv, other_kv): > + found = True > + break > + if not found: > + return False > + return True > + > + def match_kv(self, one, other): > + """Compares a KeyValue. > + Args: > + one, other (KeyValue): The objects to compare > + > + Returns true if both KeyValue objects have the same key and value > + """ > + return one.key == other.key and one.value == other.value > + > + def __hash__(self): > + hash_data = [ > + self.cookie, > + self.priority, > + self.action_keys, > + tuple((kv.key, str(kv.value)) for kv in self.match_action_kvs), > + self.match_keys, > + ] > + if self.cookie: > + hash_data.append(self.cookie) > + return tuple(hash_data).__hash__() > + > + def format(self, buf, formatter): > + """Format the Logical Flow into a Buffer.""" > + if self.cookie: > + buf.append_extra( > + "cookie={} ".format(hex(self.cookie)).ljust(18), > + style=cookie_style_gen(str(self.cookie)), > + ) > + > + buf.append_extra( > + "priority={} ".format(self.priority), style="steel_blue" > + ) > + buf.append_extra(",".join(self.match_keys), style="steel_blue") > + buf.append_extra(" ---> ", style="bold magenta") > + buf.append_extra(",".join(self.action_keys), style="steel_blue") > + > + if len(self.match_action_kvs) > 0: > + buf.append_extra(" ", style=None) > + > + for kv in self.match_action_kvs: > + formatter.format_kv(buf, kv, formatter.style) > + buf.append_extra(",", style=None) > + > + > +class LogicFlowProcessor(OpenFlowFactory, FileProcessor): > + def __init__(self, opts, match_cookie, heat_map): > + super().__init__(opts) > + self.data = dict() > + self.min_max = dict() > + self.match_cookie = match_cookie > + self.heat_map = ["n_packets", "n_bytes"] if heat_map else [] > + self.ovn_detrace = ( > + OVNDetrace(opts) if opts.get("ovn_detrace_flag") else None > + ) > + > + def start_file(self, name, filename): > + if len(self.heat_map) > 0: > + self.min = [-1] * len(self.heat_map) > + self.max = [0] * len(self.heat_map) > + self.tables = dict() > + > + def stop_file(self, name, filename): > + if len(self.heat_map) > 0: > + self.min_max[name] = (self.min, self.max) > + self.data[name] = self.tables > + > + def process_flow(self, flow, name): > + """Sort the flows by table and logical flow.""" > + # Running calculation of min and max values for all the fields that > + # take place in the heatmap. > + for i, field in enumerate(self.heat_map): > + val = flow.info.get(field) > + if self.min[i] == -1 or val < self.min[i]: > + self.min[i] = val > + if val > self.max[i]: > + self.max[i] = val > + > + table = flow.info.get("table") or 0 > + if not self.tables.get(table): > + self.tables[table] = dict() > + > + # Group flows by logical hash > + lflow = LFlow( > + flow, > + exact_actions=["output", "resubmit", "drop"], > + match_cookie=self.match_cookie, > + ) > + > + if not self.tables[table].get(lflow): > + self.tables[table][lflow] = list() > + > + self.tables[table][lflow].append(flow) > + > + def print(self, show_flows): > + formatter = ConsoleFormatter(opts=self.opts) > + console = formatter.console > + for name, tables in self.data.items(): > + console.print("\n") > + console.print(file_header(name)) > + tree = Tree("Ofproto Flows (logical)") > + > + for table_num in sorted(tables.keys()): > + table = tables[table_num] > + table_tree = tree.add("** TABLE {} **".format(table_num)) > + > + if len(self.heat_map) > 0 and len(table.values()) > 0: > + for i, field in enumerate(self.heat_map): > + (min_val, max_val) = self.min_max[name][i] > + self.console.style.set_value_style( > + field, heat_pallete(min_val, max_val) > + ) > + > + for lflow in sorted( > + table.keys(), > + key=(lambda x: x.priority), > + reverse=True, > + ): > + flows = table[lflow] > + ovn_info = None > + if self.ovn_detrace: > + ovn_info = self.ovn_detrace.get_ovn_info(lflow.cookie) > + if self.opts.get("ovn_filter"): > + ovn_regexp = re.compile( > + self.opts.get("ovn_filter") > + ) > + if not ovn_regexp.search(ovn_info): > + continue > + > + buf = ConsoleBuffer(Text()) > + > + lflow.format(buf, formatter) > + buf.append_extra( > + " ( x {} )".format(len(flows)), > + style="dark_olive_green3", > + ) > + lflow_tree = table_tree.add(buf.text) > + > + if ovn_info: > + ovn = lflow_tree.add("OVN Info") > + for part in ovn_info.split("\n"): > + if part.strip(): > + ovn.add(part.strip()) > + > + if show_flows: > + for flow in flows: > + buf = ConsoleBuffer(Text()) > + highlighted = None > + if self.opts.get("highlight"): > + result = self.opts.get("highlight").evaluate( > + flow > + ) > + if result: > + highlighted = result.kv > + formatter.format_flow(buf, flow, highlighted) > + lflow_tree.add(buf.text) > + > + console.print(tree) > + > + > +class OVNDetrace(object): > + def __init__(self, opts): > + if not opts.get("ovn_detrace_flag"): > + raise Exception("Cannot initialize OVN Detrace connection") > + > + if opts.get("ovn_detrace_path"): > + sys.path.append(opts.get("ovn_detrace_path")) > + > + import ovn_detrace > + > + class FakePrinter(ovn_detrace.Printer): > + def __init__(self): > + self.buff = io.StringIO() > + > + def print_p(self, msg): > + print(" * ", msg, file=self.buff) > + > + def print_h(self, msg): > + print(" * ", msg, file=self.buff) > + > + def clear(self): > + self.buff = io.StringIO() > + > + self.ovn_detrace = ovn_detrace > + self.ovnnb_conn = ovn_detrace.OVSDB( > + opts.get("ovnnb_db"), "OVN_Northbound" > + ) > + self.ovnsb_conn = ovn_detrace.OVSDB( > + opts.get("ovnsb_db"), "OVN_Southbound" > + ) > + self.ovn_printer = FakePrinter() > + self.cookie_handlers = ovn_detrace.get_cookie_handlers( > + self.ovnnb_conn, self.ovnsb_conn, self.ovn_printer > + ) > + > + def get_ovn_info(self, cookie): > + self.ovn_printer.clear() > + self.ovn_detrace.print_record_from_cookie( > + self.ovnsb_conn, self.cookie_handlers, "{:x}".format(cookie) > + ) > + return self.ovn_printer.buff.getvalue() > + > + > +# Try to make it easy to spot same cookies by printing them in different > +# colors > +cookie_style_gen = hash_pallete( > + hue=[x / 10 for x in range(0, 10)], > + saturation=[0.5], > + value=[0.5 + x / 10 * (0.85 - 0.5) for x in range(0, 10)], > +) > -- > 2.43.0 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On 1/30/24 16:54, Eelco Chaudron wrote: > On 1 Dec 2023, at 20:14, Adrian Moreno wrote: > >> This view is interesting for debugging the logical pipeline. It arranges >> the flows in "logical" groups (not to be confused with OVN's >> Logical_Flows). A logical group of flows is a set of flows that: >> - Have the same table number and priority >> - Match on the same fields (regardless of the value they match against) >> - Have the same actions, regardless of the arguments for tose actions, >> except for output and recirc, for which arguments do care. >> >> Optionally, the cookie can also be force to be unique for the logical >> group. By doing so, we can extend the information we show by querying an >> external OVN database and running "ovn-detrace" on each cookie. The >> result is a compact list of flow groups with interlieved OVN >> information. >> >> Furthermore, if connected to an OVN database, we can apply an OVN >> regexp filter. >> >> Examples: >> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic >> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h >> $ export OVN_NB_DB=... >> $ export OVN_SB_DB=... >> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d >> $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d >> --ovn-filter="acl.*icmp4" > > When trying some random trace I got the following: > > $ ovs-flowviz -i ~/ofctl.txt -l "drop" --style=dark openflow logic -h > > ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ > │ 📜 Filename: /home/echaudron/ofctl.txt 📜 │ > ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ > Traceback (most recent call last): > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/bin/ovs-flowviz", line 20, in <module> > main.main() > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/main.py", line 196, in main > maincli() > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ > return self.main(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main > rv = self.invoke(ctx) > ^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke > return _process_result(sub_ctx.command.invoke(sub_ctx)) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke > return _process_result(sub_ctx.command.invoke(sub_ctx)) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke > return ctx.invoke(self.callback, **ctx.params) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke > return __callback(*args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/click/decorators.py", line 45, in new_func > return f(get_current_context().obj, *args, **kwargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/cli.py", line 182, in logic > processor.print(show_flows) > File "/home/echaudron/Documents/review/ovs_adrian_tools/python-venv/lib64/python3.11/site-packages/ovs/flowviz/ofp/logic.py", line 203, in print > self.console.style.set_value_style( > ^^^^^^^^^^^^ > AttributeError: 'LogicFlowProcessor' object has no attribute 'console' > > This gets solved if I apply the next patch, see the comment in the next patch. > Thanks for doing per-patch testing. I missed that and, since the patch organization is artificial (meaning this was developed over many patches in my private repo and then artificially squashed) I might have more of those kind of errors. > The code itself looks good, I did not test with the actual ovs-detrace tool. > >> Signed-off-by: Adrian Moreno <amorenoz@redhat.com> >> --- >> python/automake.mk | 4 +- >> python/ovs/flowviz/ofp/cli.py | 113 ++++++++++++ >> python/ovs/flowviz/ofp/logic.py | 303 ++++++++++++++++++++++++++++++++ >> 3 files changed, 418 insertions(+), 2 deletions(-) >> create mode 100644 python/ovs/flowviz/ofp/logic.py >> >> diff --git a/python/automake.mk b/python/automake.mk >> index 5050089e9..fdffafbc5 100644 >> --- a/python/automake.mk >> +++ b/python/automake.mk >> @@ -74,12 +74,12 @@ ovs_flowviz = \ >> python/ovs/flowviz/odp/tree.py \ >> python/ovs/flowviz/ofp/__init__.py \ >> python/ovs/flowviz/ofp/cli.py \ >> + python/ovs/flowviz/ofp/logic.py \ >> python/ovs/flowviz/ofp/html.py \ >> python/ovs/flowviz/ovs-flowviz \ >> python/ovs/flowviz/process.py >> >> -# These python files are used at build time but not runtime, >> -# so they are not installed. >> +# These python files are used at build time but not runtime, so they are not installed. >> EXTRA_DIST += \ >> python/ovs_build_helpers/__init__.py \ >> python/ovs_build_helpers/extract_ofp_fields.py \ >> diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py >> index 5917a6bf0..6b1435ea1 100644 >> --- a/python/ovs/flowviz/ofp/cli.py >> +++ b/python/ovs/flowviz/ofp/cli.py >> @@ -12,10 +12,13 @@ >> # See the License for the specific language governing permissions and >> # limitations under the License. >> >> +import os >> + >> import click >> >> from ovs.flowviz.main import maincli >> from ovs.flowviz.ofp.html import HTMLProcessor >> +from ovs.flowviz.ofp.logic import LogicFlowProcessor >> from ovs.flowviz.process import ( >> OpenFlowFactory, >> JSONProcessor, >> @@ -69,6 +72,116 @@ def console(opts, heat_map): >> proc.print() >> >> >> +def ovn_detrace_callback(ctx, param, value): >> + """click callback to add detrace information to config object and >> + set general ovn-detrace flag to True >> + """ >> + ctx.obj[param.name] = value >> + if value != param.default: >> + ctx.obj["ovn_detrace_flag"] = True >> + return value >> + >> + >> +@openflow.command() >> +@click.option( >> + "-d", >> + "--ovn-detrace", >> + "ovn_detrace_flag", >> + is_flag=True, >> + show_default=True, >> + help="Use ovn-detrace to extract cookie information (implies '-c')", >> +) >> +@click.option( >> + "--ovn-detrace-path", >> + default="/usr/bin", >> + type=click.Path(), >> + help="Use an alternative path to where ovn_detrace.py is located. " >> + "Instead of using this option you can just set PYTHONPATH accordingly.", >> + show_default=True, >> + callback=ovn_detrace_callback, >> +) >> +@click.option( >> + "--ovnnb-db", >> + default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock", >> + help="Specify the OVN NB database string (implies -d). " >> + "If the OVN_NB_DB environment variable is set, it's used as default. " >> + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", >> + callback=ovn_detrace_callback, >> +) >> +@click.option( >> + "--ovnsb-db", >> + default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock", >> + help="Specify the OVN NB database string (implies -d). " >> + "If the OVN_NB_DB environment variable is set, it's used as default. " >> + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", >> + callback=ovn_detrace_callback, >> +) >> +@click.option( >> + "-o", >> + "--ovn-filter", >> + help="Specify a filter to be run on ovn-detrace information (implied -d). " >> + "Format: python regular expression " >> + "(see https://docs.python.org/3/library/re.html)", >> + callback=ovn_detrace_callback, >> +) >> +@click.option( >> + "-s", >> + "--show-flows", >> + is_flag=True, >> + default=False, >> + show_default=True, >> + help="Show the full flows under each logical flow", >> +) >> +@click.option( >> + "-c", >> + "--cookie", >> + "cookie_flag", >> + is_flag=True, >> + default=False, >> + show_default=True, >> + help="Consider the cookie in the logical flow", >> +) >> +@click.option( >> + "-h", >> + "--heat-map", >> + is_flag=True, >> + default=False, >> + show_default=True, >> + help="Create heat-map with packet and byte counters (when -s is used)", >> +) >> +@click.pass_obj >> +def logic( >> + opts, >> + ovn_detrace_flag, >> + ovn_detrace_path, >> + ovnnb_db, >> + ovnsb_db, >> + ovn_filter, >> + show_flows, >> + cookie_flag, >> + heat_map, >> +): >> + """ >> + Print the logical structure of the flows. >> + >> + First, sorts the flows based on tables and priorities. >> + Then, deduplicates logically equivalent flows: these a flows that match >> + on the same set of fields (regardless of the values they match against), >> + have the same priority, and actions (regardless of action arguments, >> + except in the case of output and recirculate). >> + Optionally, the cookie can also be considered to be part of the logical >> + flow. >> + """ >> + if ovn_detrace_flag: >> + opts["ovn_detrace_flag"] = True >> + if opts.get("ovn_detrace_flag"): >> + cookie_flag = True >> + >> + processor = LogicFlowProcessor(opts, cookie_flag, heat_map) >> + processor.process() >> + processor.print(show_flows) >> + >> + >> @openflow.command() >> @click.pass_obj >> def html(opts): >> diff --git a/python/ovs/flowviz/ofp/logic.py b/python/ovs/flowviz/ofp/logic.py >> new file mode 100644 >> index 000000000..cb4568cf1 >> --- /dev/null >> +++ b/python/ovs/flowviz/ofp/logic.py >> @@ -0,0 +1,303 @@ >> +# Copyright (c) 2023 Red Hat, Inc. >> +# >> +# Licensed under the Apache License, Version 2.0 (the "License"); >> +# you may not use this file except in compliance with the License. >> +# You may obtain a copy of the License at: >> +# >> +# http://www.apache.org/licenses/LICENSE-2.0 >> +# >> +# Unless required by applicable law or agreed to in writing, software >> +# distributed under the License is distributed on an "AS IS" BASIS, >> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> +# See the License for the specific language governing permissions and >> +# limitations under the License. >> + >> +import sys >> +import io >> +import re >> + >> +from rich.tree import Tree >> +from rich.text import Text >> + >> +from ovs.flowviz.process import FileProcessor, OpenFlowFactory >> +from ovs.flowviz.console import ( >> + ConsoleFormatter, >> + ConsoleBuffer, >> + hash_pallete, >> + file_header, >> + heat_pallete, >> +) >> + >> + >> +class LFlow: >> + """A Logical Flow represents the scheleton of a flow. >> + >> + Two logical flows have the same logical representation if they match >> + against same fields (regardless of the matching value) and have the same >> + set of actions (regardless of the actions' arguments, except for those >> + in the exact_actions list). >> + >> + Attributes: >> + flow (OFPFlow): The flow >> + exact_actions(list): Optional; list of action keys that are >> + considered unique if the value is also the same. >> + match_cookie (bool): Optional; if cookies are part of the logical >> + flow >> + """ >> + >> + def __init__(self, flow, exact_actions=[], match_cookie=False): >> + self.cookie = flow.info.get("cookie") or 0 if match_cookie else None >> + self.priority = flow.match.get("priority") or 0 >> + self.match_keys = tuple([kv.key for kv in flow.match_kv]) >> + >> + self.action_keys = tuple( >> + [ >> + kv.key >> + for kv in flow.actions_kv >> + if kv.key not in exact_actions >> + ] >> + ) >> + self.match_action_kvs = [ >> + kv for kv in flow.actions_kv if kv.key in exact_actions >> + ] >> + >> + def __eq__(self, other): >> + return ( >> + (self.cookie == other.cookie if self.cookie else True) >> + and self.priority == other.priority >> + and self.action_keys == other.action_keys >> + and self.equal_match_action_kvs(other) >> + and self.match_keys == other.match_keys >> + ) >> + >> + def equal_match_action_kvs(self, other): >> + """ Compares the logical flow's match action key-values with the >> + others. >> + >> + Args: >> + other (LFlow): The other LFlow to compare against >> + >> + Returns true if both LFlow have the same action k-v. >> + """ >> + if len(other.match_action_kvs) != len(self.match_action_kvs): >> + return False >> + >> + for kv in self.match_action_kvs: >> + found = False >> + for other_kv in other.match_action_kvs: >> + if self.match_kv(kv, other_kv): >> + found = True >> + break >> + if not found: >> + return False >> + return True >> + >> + def match_kv(self, one, other): >> + """Compares a KeyValue. >> + Args: >> + one, other (KeyValue): The objects to compare >> + >> + Returns true if both KeyValue objects have the same key and value >> + """ >> + return one.key == other.key and one.value == other.value >> + >> + def __hash__(self): >> + hash_data = [ >> + self.cookie, >> + self.priority, >> + self.action_keys, >> + tuple((kv.key, str(kv.value)) for kv in self.match_action_kvs), >> + self.match_keys, >> + ] >> + if self.cookie: >> + hash_data.append(self.cookie) >> + return tuple(hash_data).__hash__() >> + >> + def format(self, buf, formatter): >> + """Format the Logical Flow into a Buffer.""" >> + if self.cookie: >> + buf.append_extra( >> + "cookie={} ".format(hex(self.cookie)).ljust(18), >> + style=cookie_style_gen(str(self.cookie)), >> + ) >> + >> + buf.append_extra( >> + "priority={} ".format(self.priority), style="steel_blue" >> + ) >> + buf.append_extra(",".join(self.match_keys), style="steel_blue") >> + buf.append_extra(" ---> ", style="bold magenta") >> + buf.append_extra(",".join(self.action_keys), style="steel_blue") >> + >> + if len(self.match_action_kvs) > 0: >> + buf.append_extra(" ", style=None) >> + >> + for kv in self.match_action_kvs: >> + formatter.format_kv(buf, kv, formatter.style) >> + buf.append_extra(",", style=None) >> + >> + >> +class LogicFlowProcessor(OpenFlowFactory, FileProcessor): >> + def __init__(self, opts, match_cookie, heat_map): >> + super().__init__(opts) >> + self.data = dict() >> + self.min_max = dict() >> + self.match_cookie = match_cookie >> + self.heat_map = ["n_packets", "n_bytes"] if heat_map else [] >> + self.ovn_detrace = ( >> + OVNDetrace(opts) if opts.get("ovn_detrace_flag") else None >> + ) >> + >> + def start_file(self, name, filename): >> + if len(self.heat_map) > 0: >> + self.min = [-1] * len(self.heat_map) >> + self.max = [0] * len(self.heat_map) >> + self.tables = dict() >> + >> + def stop_file(self, name, filename): >> + if len(self.heat_map) > 0: >> + self.min_max[name] = (self.min, self.max) >> + self.data[name] = self.tables >> + >> + def process_flow(self, flow, name): >> + """Sort the flows by table and logical flow.""" >> + # Running calculation of min and max values for all the fields that >> + # take place in the heatmap. >> + for i, field in enumerate(self.heat_map): >> + val = flow.info.get(field) >> + if self.min[i] == -1 or val < self.min[i]: >> + self.min[i] = val >> + if val > self.max[i]: >> + self.max[i] = val >> + >> + table = flow.info.get("table") or 0 >> + if not self.tables.get(table): >> + self.tables[table] = dict() >> + >> + # Group flows by logical hash >> + lflow = LFlow( >> + flow, >> + exact_actions=["output", "resubmit", "drop"], >> + match_cookie=self.match_cookie, >> + ) >> + >> + if not self.tables[table].get(lflow): >> + self.tables[table][lflow] = list() >> + >> + self.tables[table][lflow].append(flow) >> + >> + def print(self, show_flows): >> + formatter = ConsoleFormatter(opts=self.opts) >> + console = formatter.console >> + for name, tables in self.data.items(): >> + console.print("\n") >> + console.print(file_header(name)) >> + tree = Tree("Ofproto Flows (logical)") >> + >> + for table_num in sorted(tables.keys()): >> + table = tables[table_num] >> + table_tree = tree.add("** TABLE {} **".format(table_num)) >> + >> + if len(self.heat_map) > 0 and len(table.values()) > 0: >> + for i, field in enumerate(self.heat_map): >> + (min_val, max_val) = self.min_max[name][i] >> + self.console.style.set_value_style( >> + field, heat_pallete(min_val, max_val) >> + ) >> + >> + for lflow in sorted( >> + table.keys(), >> + key=(lambda x: x.priority), >> + reverse=True, >> + ): >> + flows = table[lflow] >> + ovn_info = None >> + if self.ovn_detrace: >> + ovn_info = self.ovn_detrace.get_ovn_info(lflow.cookie) >> + if self.opts.get("ovn_filter"): >> + ovn_regexp = re.compile( >> + self.opts.get("ovn_filter") >> + ) >> + if not ovn_regexp.search(ovn_info): >> + continue >> + >> + buf = ConsoleBuffer(Text()) >> + >> + lflow.format(buf, formatter) >> + buf.append_extra( >> + " ( x {} )".format(len(flows)), >> + style="dark_olive_green3", >> + ) >> + lflow_tree = table_tree.add(buf.text) >> + >> + if ovn_info: >> + ovn = lflow_tree.add("OVN Info") >> + for part in ovn_info.split("\n"): >> + if part.strip(): >> + ovn.add(part.strip()) >> + >> + if show_flows: >> + for flow in flows: >> + buf = ConsoleBuffer(Text()) >> + highlighted = None >> + if self.opts.get("highlight"): >> + result = self.opts.get("highlight").evaluate( >> + flow >> + ) >> + if result: >> + highlighted = result.kv >> + formatter.format_flow(buf, flow, highlighted) >> + lflow_tree.add(buf.text) >> + >> + console.print(tree) >> + >> + >> +class OVNDetrace(object): >> + def __init__(self, opts): >> + if not opts.get("ovn_detrace_flag"): >> + raise Exception("Cannot initialize OVN Detrace connection") >> + >> + if opts.get("ovn_detrace_path"): >> + sys.path.append(opts.get("ovn_detrace_path")) >> + >> + import ovn_detrace >> + >> + class FakePrinter(ovn_detrace.Printer): >> + def __init__(self): >> + self.buff = io.StringIO() >> + >> + def print_p(self, msg): >> + print(" * ", msg, file=self.buff) >> + >> + def print_h(self, msg): >> + print(" * ", msg, file=self.buff) >> + >> + def clear(self): >> + self.buff = io.StringIO() >> + >> + self.ovn_detrace = ovn_detrace >> + self.ovnnb_conn = ovn_detrace.OVSDB( >> + opts.get("ovnnb_db"), "OVN_Northbound" >> + ) >> + self.ovnsb_conn = ovn_detrace.OVSDB( >> + opts.get("ovnsb_db"), "OVN_Southbound" >> + ) >> + self.ovn_printer = FakePrinter() >> + self.cookie_handlers = ovn_detrace.get_cookie_handlers( >> + self.ovnnb_conn, self.ovnsb_conn, self.ovn_printer >> + ) >> + >> + def get_ovn_info(self, cookie): >> + self.ovn_printer.clear() >> + self.ovn_detrace.print_record_from_cookie( >> + self.ovnsb_conn, self.cookie_handlers, "{:x}".format(cookie) >> + ) >> + return self.ovn_printer.buff.getvalue() >> + >> + >> +# Try to make it easy to spot same cookies by printing them in different >> +# colors >> +cookie_style_gen = hash_pallete( >> + hue=[x / 10 for x in range(0, 10)], >> + saturation=[0.5], >> + value=[0.5 + x / 10 * (0.85 - 0.5) for x in range(0, 10)], >> +) >> -- >> 2.43.0 >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
diff --git a/python/automake.mk b/python/automake.mk index 5050089e9..fdffafbc5 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -74,12 +74,12 @@ ovs_flowviz = \ python/ovs/flowviz/odp/tree.py \ python/ovs/flowviz/ofp/__init__.py \ python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ofp/logic.py \ python/ovs/flowviz/ofp/html.py \ python/ovs/flowviz/ovs-flowviz \ python/ovs/flowviz/process.py -# These python files are used at build time but not runtime, -# so they are not installed. +# These python files are used at build time but not runtime, so they are not installed. EXTRA_DIST += \ python/ovs_build_helpers/__init__.py \ python/ovs_build_helpers/extract_ofp_fields.py \ diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py index 5917a6bf0..6b1435ea1 100644 --- a/python/ovs/flowviz/ofp/cli.py +++ b/python/ovs/flowviz/ofp/cli.py @@ -12,10 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +import os + import click from ovs.flowviz.main import maincli from ovs.flowviz.ofp.html import HTMLProcessor +from ovs.flowviz.ofp.logic import LogicFlowProcessor from ovs.flowviz.process import ( OpenFlowFactory, JSONProcessor, @@ -69,6 +72,116 @@ def console(opts, heat_map): proc.print() +def ovn_detrace_callback(ctx, param, value): + """click callback to add detrace information to config object and + set general ovn-detrace flag to True + """ + ctx.obj[param.name] = value + if value != param.default: + ctx.obj["ovn_detrace_flag"] = True + return value + + +@openflow.command() +@click.option( + "-d", + "--ovn-detrace", + "ovn_detrace_flag", + is_flag=True, + show_default=True, + help="Use ovn-detrace to extract cookie information (implies '-c')", +) +@click.option( + "--ovn-detrace-path", + default="/usr/bin", + type=click.Path(), + help="Use an alternative path to where ovn_detrace.py is located. " + "Instead of using this option you can just set PYTHONPATH accordingly.", + show_default=True, + callback=ovn_detrace_callback, +) +@click.option( + "--ovnnb-db", + default=os.getenv("OVN_NB_DB") or "unix:/var/run/ovn/ovnnb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "--ovnsb-db", + default=os.getenv("OVN_SB_DB") or "unix:/var/run/ovn/ovnsb_db.sock", + help="Specify the OVN NB database string (implies -d). " + "If the OVN_NB_DB environment variable is set, it's used as default. " + "Otherwise, the default is unix:/var/run/ovn/ovnnb_db.sock", + callback=ovn_detrace_callback, +) +@click.option( + "-o", + "--ovn-filter", + help="Specify a filter to be run on ovn-detrace information (implied -d). " + "Format: python regular expression " + "(see https://docs.python.org/3/library/re.html)", + callback=ovn_detrace_callback, +) +@click.option( + "-s", + "--show-flows", + is_flag=True, + default=False, + show_default=True, + help="Show the full flows under each logical flow", +) +@click.option( + "-c", + "--cookie", + "cookie_flag", + is_flag=True, + default=False, + show_default=True, + help="Consider the cookie in the logical flow", +) +@click.option( + "-h", + "--heat-map", + is_flag=True, + default=False, + show_default=True, + help="Create heat-map with packet and byte counters (when -s is used)", +) +@click.pass_obj +def logic( + opts, + ovn_detrace_flag, + ovn_detrace_path, + ovnnb_db, + ovnsb_db, + ovn_filter, + show_flows, + cookie_flag, + heat_map, +): + """ + Print the logical structure of the flows. + + First, sorts the flows based on tables and priorities. + Then, deduplicates logically equivalent flows: these a flows that match + on the same set of fields (regardless of the values they match against), + have the same priority, and actions (regardless of action arguments, + except in the case of output and recirculate). + Optionally, the cookie can also be considered to be part of the logical + flow. + """ + if ovn_detrace_flag: + opts["ovn_detrace_flag"] = True + if opts.get("ovn_detrace_flag"): + cookie_flag = True + + processor = LogicFlowProcessor(opts, cookie_flag, heat_map) + processor.process() + processor.print(show_flows) + + @openflow.command() @click.pass_obj def html(opts): diff --git a/python/ovs/flowviz/ofp/logic.py b/python/ovs/flowviz/ofp/logic.py new file mode 100644 index 000000000..cb4568cf1 --- /dev/null +++ b/python/ovs/flowviz/ofp/logic.py @@ -0,0 +1,303 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +import io +import re + +from rich.tree import Tree +from rich.text import Text + +from ovs.flowviz.process import FileProcessor, OpenFlowFactory +from ovs.flowviz.console import ( + ConsoleFormatter, + ConsoleBuffer, + hash_pallete, + file_header, + heat_pallete, +) + + +class LFlow: + """A Logical Flow represents the scheleton of a flow. + + Two logical flows have the same logical representation if they match + against same fields (regardless of the matching value) and have the same + set of actions (regardless of the actions' arguments, except for those + in the exact_actions list). + + Attributes: + flow (OFPFlow): The flow + exact_actions(list): Optional; list of action keys that are + considered unique if the value is also the same. + match_cookie (bool): Optional; if cookies are part of the logical + flow + """ + + def __init__(self, flow, exact_actions=[], match_cookie=False): + self.cookie = flow.info.get("cookie") or 0 if match_cookie else None + self.priority = flow.match.get("priority") or 0 + self.match_keys = tuple([kv.key for kv in flow.match_kv]) + + self.action_keys = tuple( + [ + kv.key + for kv in flow.actions_kv + if kv.key not in exact_actions + ] + ) + self.match_action_kvs = [ + kv for kv in flow.actions_kv if kv.key in exact_actions + ] + + def __eq__(self, other): + return ( + (self.cookie == other.cookie if self.cookie else True) + and self.priority == other.priority + and self.action_keys == other.action_keys + and self.equal_match_action_kvs(other) + and self.match_keys == other.match_keys + ) + + def equal_match_action_kvs(self, other): + """ Compares the logical flow's match action key-values with the + others. + + Args: + other (LFlow): The other LFlow to compare against + + Returns true if both LFlow have the same action k-v. + """ + if len(other.match_action_kvs) != len(self.match_action_kvs): + return False + + for kv in self.match_action_kvs: + found = False + for other_kv in other.match_action_kvs: + if self.match_kv(kv, other_kv): + found = True + break + if not found: + return False + return True + + def match_kv(self, one, other): + """Compares a KeyValue. + Args: + one, other (KeyValue): The objects to compare + + Returns true if both KeyValue objects have the same key and value + """ + return one.key == other.key and one.value == other.value + + def __hash__(self): + hash_data = [ + self.cookie, + self.priority, + self.action_keys, + tuple((kv.key, str(kv.value)) for kv in self.match_action_kvs), + self.match_keys, + ] + if self.cookie: + hash_data.append(self.cookie) + return tuple(hash_data).__hash__() + + def format(self, buf, formatter): + """Format the Logical Flow into a Buffer.""" + if self.cookie: + buf.append_extra( + "cookie={} ".format(hex(self.cookie)).ljust(18), + style=cookie_style_gen(str(self.cookie)), + ) + + buf.append_extra( + "priority={} ".format(self.priority), style="steel_blue" + ) + buf.append_extra(",".join(self.match_keys), style="steel_blue") + buf.append_extra(" ---> ", style="bold magenta") + buf.append_extra(",".join(self.action_keys), style="steel_blue") + + if len(self.match_action_kvs) > 0: + buf.append_extra(" ", style=None) + + for kv in self.match_action_kvs: + formatter.format_kv(buf, kv, formatter.style) + buf.append_extra(",", style=None) + + +class LogicFlowProcessor(OpenFlowFactory, FileProcessor): + def __init__(self, opts, match_cookie, heat_map): + super().__init__(opts) + self.data = dict() + self.min_max = dict() + self.match_cookie = match_cookie + self.heat_map = ["n_packets", "n_bytes"] if heat_map else [] + self.ovn_detrace = ( + OVNDetrace(opts) if opts.get("ovn_detrace_flag") else None + ) + + def start_file(self, name, filename): + if len(self.heat_map) > 0: + self.min = [-1] * len(self.heat_map) + self.max = [0] * len(self.heat_map) + self.tables = dict() + + def stop_file(self, name, filename): + if len(self.heat_map) > 0: + self.min_max[name] = (self.min, self.max) + self.data[name] = self.tables + + def process_flow(self, flow, name): + """Sort the flows by table and logical flow.""" + # Running calculation of min and max values for all the fields that + # take place in the heatmap. + for i, field in enumerate(self.heat_map): + val = flow.info.get(field) + if self.min[i] == -1 or val < self.min[i]: + self.min[i] = val + if val > self.max[i]: + self.max[i] = val + + table = flow.info.get("table") or 0 + if not self.tables.get(table): + self.tables[table] = dict() + + # Group flows by logical hash + lflow = LFlow( + flow, + exact_actions=["output", "resubmit", "drop"], + match_cookie=self.match_cookie, + ) + + if not self.tables[table].get(lflow): + self.tables[table][lflow] = list() + + self.tables[table][lflow].append(flow) + + def print(self, show_flows): + formatter = ConsoleFormatter(opts=self.opts) + console = formatter.console + for name, tables in self.data.items(): + console.print("\n") + console.print(file_header(name)) + tree = Tree("Ofproto Flows (logical)") + + for table_num in sorted(tables.keys()): + table = tables[table_num] + table_tree = tree.add("** TABLE {} **".format(table_num)) + + if len(self.heat_map) > 0 and len(table.values()) > 0: + for i, field in enumerate(self.heat_map): + (min_val, max_val) = self.min_max[name][i] + self.console.style.set_value_style( + field, heat_pallete(min_val, max_val) + ) + + for lflow in sorted( + table.keys(), + key=(lambda x: x.priority), + reverse=True, + ): + flows = table[lflow] + ovn_info = None + if self.ovn_detrace: + ovn_info = self.ovn_detrace.get_ovn_info(lflow.cookie) + if self.opts.get("ovn_filter"): + ovn_regexp = re.compile( + self.opts.get("ovn_filter") + ) + if not ovn_regexp.search(ovn_info): + continue + + buf = ConsoleBuffer(Text()) + + lflow.format(buf, formatter) + buf.append_extra( + " ( x {} )".format(len(flows)), + style="dark_olive_green3", + ) + lflow_tree = table_tree.add(buf.text) + + if ovn_info: + ovn = lflow_tree.add("OVN Info") + for part in ovn_info.split("\n"): + if part.strip(): + ovn.add(part.strip()) + + if show_flows: + for flow in flows: + buf = ConsoleBuffer(Text()) + highlighted = None + if self.opts.get("highlight"): + result = self.opts.get("highlight").evaluate( + flow + ) + if result: + highlighted = result.kv + formatter.format_flow(buf, flow, highlighted) + lflow_tree.add(buf.text) + + console.print(tree) + + +class OVNDetrace(object): + def __init__(self, opts): + if not opts.get("ovn_detrace_flag"): + raise Exception("Cannot initialize OVN Detrace connection") + + if opts.get("ovn_detrace_path"): + sys.path.append(opts.get("ovn_detrace_path")) + + import ovn_detrace + + class FakePrinter(ovn_detrace.Printer): + def __init__(self): + self.buff = io.StringIO() + + def print_p(self, msg): + print(" * ", msg, file=self.buff) + + def print_h(self, msg): + print(" * ", msg, file=self.buff) + + def clear(self): + self.buff = io.StringIO() + + self.ovn_detrace = ovn_detrace + self.ovnnb_conn = ovn_detrace.OVSDB( + opts.get("ovnnb_db"), "OVN_Northbound" + ) + self.ovnsb_conn = ovn_detrace.OVSDB( + opts.get("ovnsb_db"), "OVN_Southbound" + ) + self.ovn_printer = FakePrinter() + self.cookie_handlers = ovn_detrace.get_cookie_handlers( + self.ovnnb_conn, self.ovnsb_conn, self.ovn_printer + ) + + def get_ovn_info(self, cookie): + self.ovn_printer.clear() + self.ovn_detrace.print_record_from_cookie( + self.ovnsb_conn, self.cookie_handlers, "{:x}".format(cookie) + ) + return self.ovn_printer.buff.getvalue() + + +# Try to make it easy to spot same cookies by printing them in different +# colors +cookie_style_gen = hash_pallete( + hue=[x / 10 for x in range(0, 10)], + saturation=[0.5], + value=[0.5 + x / 10 * (0.85 - 0.5) for x in range(0, 10)], +)
This view is interesting for debugging the logical pipeline. It arranges the flows in "logical" groups (not to be confused with OVN's Logical_Flows). A logical group of flows is a set of flows that: - Have the same table number and priority - Match on the same fields (regardless of the value they match against) - Have the same actions, regardless of the arguments for those actions, except for output and recirc, for which arguments do care. Optionally, the cookie can also be force to be unique for the logical group. By doing so, we can extend the information we show by querying an external OVN database and running "ovn-detrace" on each cookie. The result is a compact list of flow groups with interlieved OVN information. Furthermore, if connected to an OVN database, we can apply an OVN regexp filter. Examples: $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -s -h $ export OVN_NB_DB=... $ export OVN_SB_DB=... $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow logic -d --ovn-filter="acl.*icmp4" Signed-off-by: Adrian Moreno <amorenoz@redhat.com> --- python/automake.mk | 4 +- python/ovs/flowviz/ofp/cli.py | 113 ++++++++++++ python/ovs/flowviz/ofp/logic.py | 303 ++++++++++++++++++++++++++++++++ 3 files changed, 418 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/ofp/logic.py