From patchwork Thu Jun 16 06:32:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644137 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OUo6rWU5; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsn90nJVz9sFw for ; Thu, 16 Jun 2022 16:33:17 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C4EA840C96; Thu, 16 Jun 2022 06:33:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 441Cp3KqFwHj; Thu, 16 Jun 2022 06:33:13 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 290B340BD9; Thu, 16 Jun 2022 06:33:12 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E9AA4C0032; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6F1B7C002D for ; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 645FD61111 for ; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ymDKXI3EMITG for ; Thu, 16 Jun 2022 06:33:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id E9ADD61112 for ; Thu, 16 Jun 2022 06:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361188; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w+7MxrEVUVvSnLlsYCWPVboZhWAW544H824fahWa3U0=; b=OUo6rWU5JEfypoDwPknwYLW+pT9cRjmhjj3lMLb4MHDX0VUHVcVpTCgYEU7St774Z1Jg8R nMaNRXzZz35TWO9FCMUNOSrKQVjwQ2yKBahSlGwsp0fvNJjoMT3XJl0f0hjQE3oAXerHce ohRLt1gITSMD51m8fSEJtcGNxsxDGD0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-217-NgCx3u-GMuCm9lM1U120Hg-1; Thu, 16 Jun 2022 02:33:05 -0400 X-MC-Unique: NgCx3u-GMuCm9lM1U120Hg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 227A18525AA; Thu, 16 Jun 2022 06:33:05 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id D199A492C3B; Thu, 16 Jun 2022 06:33:03 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:31 +0200 Message-Id: <20220616063247.517147-2-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 01/17] python: add generic Key-Value parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Most of ofproto and dpif flows are based on key-value pairs. These key-value pairs can be represented in several ways, eg: key:value, key=value, key(value). Add the following classes that allow parsing of key-value strings: * KeyValue: holds a key-value pair * KeyMetadata: holds some metadata associated with a KeyValue such as the original key and value strings and their position in the global string * KVParser: is able to parse a string and extract it's key-value pairs as KeyValue instances. Before creating the KeyValue instance it tries to decode the value via the KVDecoders * KVDecoders holds a number of decoders that KVParser can use to decode key-value pairs. It accepts a dictionary of keys and callables to allow users to specify what decoder (i.e: callable) to use for each key Also, flake8 seems to be incorrectly reporting an error (E203) in: "slice[index + offset : index + offset]" which is PEP8 compliant. So, ignore this error. Acked-by: Terry Wilson Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- Makefile.am | 3 +- python/automake.mk | 6 +- python/ovs/flow/__init__.py | 0 python/ovs/flow/decoders.py | 18 +++ python/ovs/flow/kv.py | 314 ++++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 6 files changed, 340 insertions(+), 3 deletions(-) create mode 100644 python/ovs/flow/__init__.py create mode 100644 python/ovs/flow/decoders.py create mode 100644 python/ovs/flow/kv.py diff --git a/Makefile.am b/Makefile.am index cb8076433..4f51d225e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -391,6 +391,7 @@ ALL_LOCAL += flake8-check # E128 continuation line under-indented for visual indent # E129 visually indented line with same indent as next logical line # E131 continuation line unaligned for hanging indent +# E203 whitespace before ':' # E722 do not use bare except, specify exception instead # W503 line break before binary operator # W504 line break after binary operator @@ -403,7 +404,7 @@ ALL_LOCAL += flake8-check # H233 Python 3.x incompatible use of print operator # H238 old style class declaration, use new style (inherit from `object`) FLAKE8_SELECT = H231,H232,H233,H238 -FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E722,W503,W504,F811,D,H,I +FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E203,E722,W503,W504,F811,D,H,I flake8-check: $(FLAKE8_PYFILES) $(FLAKE8_WERROR)$(AM_V_GEN) \ src='$^' && \ diff --git a/python/automake.mk b/python/automake.mk index 767512f17..e3da6bdbe 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -16,7 +16,6 @@ ovs_pyfiles = \ python/ovs/compat/sortedcontainers/sorteddict.py \ python/ovs/compat/sortedcontainers/sortedset.py \ python/ovs/daemon.py \ - python/ovs/fcntl_win.py \ python/ovs/db/__init__.py \ python/ovs/db/custom_index.py \ python/ovs/db/data.py \ @@ -26,6 +25,10 @@ ovs_pyfiles = \ python/ovs/db/schema.py \ python/ovs/db/types.py \ python/ovs/fatal_signal.py \ + python/ovs/fcntl_win.py \ + python/ovs/flow/__init__.py \ + python/ovs/flow/decoders.py \ + python/ovs/flow/kv.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ @@ -42,6 +45,7 @@ ovs_pyfiles = \ python/ovs/version.py \ python/ovs/vlog.py \ python/ovs/winutils.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ diff --git a/python/ovs/flow/__init__.py b/python/ovs/flow/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flow/decoders.py b/python/ovs/flow/decoders.py new file mode 100644 index 000000000..0c2259c76 --- /dev/null +++ b/python/ovs/flow/decoders.py @@ -0,0 +1,18 @@ +"""Defines helpful decoders that can be used to decode information from the +flows. + +A decoder is generally a callable that accepts a string and returns the value +object. +""" + + +def decode_default(value): + """Default decoder. + + It tries to convert into an integer value and, if it fails, just + returns the string. + """ + try: + return int(value, 0) + except ValueError: + return value diff --git a/python/ovs/flow/kv.py b/python/ovs/flow/kv.py new file mode 100644 index 000000000..cceb95e43 --- /dev/null +++ b/python/ovs/flow/kv.py @@ -0,0 +1,314 @@ +"""Common helper classes for flow Key-Value parsing.""" + +import functools +import re + +from ovs.flow.decoders import decode_default + + +class ParseError(RuntimeError): + """Exception raised when an error occurs during parsing.""" + + pass + + +class KeyMetadata(object): + """Class for keeping key metadata. + + Attributes: + kpos (int): The position of the keyword in the parent string. + vpos (int): The position of the value in the parent string. + kstring (string): The keyword string as found in the flow string. + vstring (string): The value as found in the flow string. + delim (string): Optional, the string used as delimiter between the key + and the value. + end_delim (string): Optional, the string used as end delimiter + """ + + def __init__(self, kpos, vpos, kstring, vstring, delim="", end_delim=""): + """Constructor.""" + self.kpos = kpos + self.vpos = vpos + self.kstring = kstring + self.vstring = vstring + self.delim = delim + self.end_delim = end_delim + + def __str__(self): + return "key: [{},{}), val:[{}, {})".format( + self.kpos, + self.kpos + len(self.kstring), + self.vpos, + self.vpos + len(self.vstring), + ) + + def __repr__(self): + return "{}('{}')".format(self.__class__.__name__, self) + + +class KeyValue(object): + """Class for keeping key-value data. + + Attributes: + key (str): The key string. + value (any): The value data. + meta (KeyMetadata): The key metadata. + """ + + def __init__(self, key, value, meta=None): + """Constructor.""" + self.key = key + self.value = value + self.meta = meta + + def __str__(self): + return "{}: {} ({})".format(self.key, str(self.value), str(self.meta)) + + def __repr__(self): + return "{}('{}')".format(self.__class__.__name__, self) + + +class KVDecoders(object): + """KVDecoders class is used by KVParser to select how to decode the value + of a specific keyword. + + A decoder is simply a function that accepts a value string and returns + the value objects to be stored. + The returned value may be of any type. + + Decoders may return a KeyValue instance to indicate that the keyword should + also be modified to match the one provided in the returned KeyValue. + + The decoder to be used will be selected using the key as an index. If not + found, the default decoder will be used. If free keys are found (i.e: + keys without a value), the default_free decoder will be used. For that + reason, the default_free decoder, must return both the key and value to be + stored. + + Args: + decoders (dict): Optional; A dictionary of decoders indexed by keyword. + default (callable): Optional; A decoder used if a match is not found in + configured decoders. If not provided, the default behavior is to + try to decode the value into an integer and, if that fails, + just return the string as-is. + default_free (callable): Optional; The decoder used if a match is not + found in configured decoders and it's a free value (e.g: + a value without a key) Defaults to returning the free value as + keyword and "True" as value. + The callable must accept a string and return a key-value pair. + """ + + def __init__(self, decoders=None, default=None, default_free=None): + self._decoders = decoders or dict() + self._default = default or decode_default + self._default_free = default_free or self._default_free_decoder + + def decode(self, keyword, value_str): + """Decode a keyword and value. + + Args: + keyword (str): The keyword whose value is to be decoded. + value_str (str): The value string. + + Returns: + The key (str) and value(any) to be stored. + """ + + decoder = self._decoders.get(keyword) + if decoder: + result = decoder(value_str) + if isinstance(result, KeyValue): + keyword = result.key + value = result.value + else: + value = result + + return keyword, value + else: + if value_str: + return keyword, self._default(value_str) + else: + return self._default_free(keyword) + + @staticmethod + def _default_free_decoder(key): + """Default decoder for free keywords.""" + return key, True + + +delim_pattern = re.compile(r"(\(|=|:|,|\n|\r|\t)") +parenthesis = re.compile(r"(\(|\))") +end_pattern = re.compile(r"( |,|\n|\r|\t)") +separators = (" ", ",") +end_of_string = (",", "\n", "\t", "\r", "") + + +class KVParser(object): + """KVParser parses a string looking for key-value pairs. + + Args: + string (str): The string to parse. + decoders (KVDecoders): Optional; the KVDecoders instance to use. + """ + + def __init__(self, string, decoders=None): + """Constructor.""" + self._decoders = decoders or KVDecoders() + self._keyval = list() + self._string = string + + def keys(self): + return list(kv.key for kv in self._keyval) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the key-value pairs in string. + + The input string is assumed to contain a list of comma (or space) + separated key-value pairs. + + Key-values pairs can have multiple different delimiters, eg: + "key1:value1,key2=value2,key3(value3)". + + Also, we can stumble upon a "free" keywords, e.g: + "key1=value1,key2=value2,free_keyword". + We consider this as keys without a value. + + So, to parse the string we do the following until the end of the + string is found: + + 1 - Skip any leading comma's or spaces. + 2 - Find the next delimiter (or end_of_string character). + 3 - Depending on the delimiter, obtain the key and the value. + For instance, if the delimiter is "(", find the next matching + ")". + 4 - Use the KVDecoders to decode the key-value. + 5 - Store the KeyValue object with the corresponding metadata. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + keyword = "" + delimiter = "" + rest = "" + + # 1. Skip separator characters. + if self._string[kpos] in separators: + kpos += 1 + continue + + # 2. Find the next delimiter or end of string character. + try: + keyword, delimiter, rest = delim_pattern.split( + self._string[kpos:], 1 + ) + except ValueError: + keyword = self._string[kpos:] # Free keyword + + # 3. Extract the value from the rest of the string. + value_str = "" + vpos = kpos + len(keyword) + 1 + end_delimiter = "" + + if delimiter in ("=", ":"): + # If the delimiter is ':' or '=', the end of the value is the + # end of the string or a ', '. + value_parts = end_pattern.split(rest, 1) + value_str = value_parts[0] + next_kpos = vpos + len(value_str) + + elif delimiter == "(": + # Find matching ")". + level = 1 + index = 0 + value_parts = parenthesis.split(rest) + for val in value_parts: + if val == "(": + level += 1 + elif val == ")": + level -= 1 + index += len(val) + if level == 0: + break + + if level != 0: + raise ParseError( + "Error parsing string {}: " + "Failed to find matching ')' in {}".format( + self._string, rest + ) + ) + + value_str = rest[: index - 1] + next_kpos = vpos + len(value_str) + 1 + end_delimiter = ")" + + # Exceptionally, if after the () we find -> {}, do not treat + # the content of the parenthesis as the value, consider + # ({})->{} as the string value. + if index < len(rest) - 2 and rest[index : index + 2] == "->": + extra_val = rest[index + 2 :].split(",")[0] + value_str = "({})->{}".format(value_str, extra_val) + # remove the first "(". + vpos -= 1 + next_kpos = vpos + len(value_str) + end_delimiter = "" + + elif delimiter in end_of_string: + # Key without a value. + next_kpos = kpos + len(keyword) + vpos = -1 + + # 4. Use KVDecoders to decode the key-value. + try: + key, val = self._decoders.decode(keyword, value_str) + except Exception as e: + raise ParseError( + "Error parsing key-value ({}, {})".format( + keyword, value_str + ) + ) from e + + # Store the KeyValue object with the corresponding metadata. + meta = KeyMetadata( + kpos=kpos, + vpos=vpos, + kstring=keyword, + vstring=value_str, + delim=delimiter, + end_delim=end_delimiter, + ) + + self._keyval.append(KeyValue(key, val, meta)) + + kpos = next_kpos + + +def decode_nested_kv(decoders, value): + """A key-value decoder that extracts nested key-value pairs and returns + them in a dictionary. + + Args: + decoders (KVDecoders): The KVDecoders to use. + value (str): The value string to decode. + """ + if not value: + # Mark as flag + return True + + parser = KVParser(value, decoders) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_kv_decoder(decoders=None): + """Helper function that creates a nested kv decoder with given + KVDecoders.""" + return functools.partial(decode_nested_kv, decoders) diff --git a/python/setup.py b/python/setup.py index cfe01763f..7ac3c3662 100644 --- a/python/setup.py +++ b/python/setup.py @@ -71,7 +71,7 @@ setup_args = dict( author='Open vSwitch', author_email='dev@openvswitch.org', packages=['ovs', 'ovs.compat', 'ovs.compat.sortedcontainers', - 'ovs.db', 'ovs.unixctl'], + 'ovs.db', 'ovs.unixctl', 'ovs.flow'], keywords=['openvswitch', 'ovs', 'OVSDB'], license='Apache 2.0', classifiers=[ From patchwork Thu Jun 16 06:32:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644154 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IW0B0JQS; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNspj0wy1z9sG2 for ; Thu, 16 Jun 2022 16:34:37 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0DCEA8416C; Thu, 16 Jun 2022 06:34:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EpvInyj7MmvR; Thu, 16 Jun 2022 06:34:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id C418584134; Thu, 16 Jun 2022 06:34:05 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 40599C008C; Thu, 16 Jun 2022 06:34:04 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 08785C002D for ; Thu, 16 Jun 2022 06:34:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 484C4611A8 for ; Thu, 16 Jun 2022 06:33:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sBz0JGRmNJDE for ; Thu, 16 Jun 2022 06:33:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 792FA61137 for ; Thu, 16 Jun 2022 06:33:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SO9Lzw1XtbHLkvcJQ4wndzQr2KJfzaqOzT1wZPLFcR8=; b=IW0B0JQSedJozRRKtnAfyfxLh+w1eQ/C0drEwFlrI3O75oaIBg3irMYbtYOIkM2feUbzfQ LxELI6iHWjAdFJWFPcGW0FfKiHzuFVTxn1mBVH3EpRd3Ti/e5si+nCXDhdXacG0q7C4RMY SFcwi62pt+g3aLQrAOA+mChph21FJ3U= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-101-VFeTp2RsP8GpHnfOEefH7Q-1; Thu, 16 Jun 2022 02:33:07 -0400 X-MC-Unique: VFeTp2RsP8GpHnfOEefH7Q-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AA2842999B4F; Thu, 16 Jun 2022 06:33:06 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 86240492C3B; Thu, 16 Jun 2022 06:33:05 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:32 +0200 Message-Id: <20220616063247.517147-3-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 02/17] python: add mask, ip and eth decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add more decoders that can be used by KVParser. For IPv4 and IPv6 addresses, create a new class that wraps netaddr.IPAddress. For Ethernet addresses, create a new class that wraps netaddr.EUI. For Integers, create a new class that performs basic bitwise mask comparisons Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/ovs/flow/decoders.py | 398 ++++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 2 files changed, 399 insertions(+), 1 deletion(-) diff --git a/python/ovs/flow/decoders.py b/python/ovs/flow/decoders.py index 0c2259c76..883e61acf 100644 --- a/python/ovs/flow/decoders.py +++ b/python/ovs/flow/decoders.py @@ -5,6 +5,15 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import netaddr + + +class Decoder(object): + """Base class for all decoder classes.""" + + def to_json(self): + raise NotImplementedError() + def decode_default(value): """Default decoder. @@ -16,3 +25,392 @@ def decode_default(value): return int(value, 0) except ValueError: return value + + +def decode_flag(value): + """Decode a flag. It's existence is just flagged by returning True.""" + return True + + +def decode_int(value): + """Integer decoder. + + Both base10 and base16 integers are supported. + + Used for fields such as: + n_bytes=34 + metadata=0x4 + """ + return int(value, 0) + + +def decode_time(value): + """Time decoder. + + Used for fields such as: + duration=1234.123s + """ + if value == "never": + return value + + time_str = value.rstrip("s") + return float(time_str) + + +class IntMask(Decoder): + """Base class for Integer Mask decoder classes. + + It supports decoding a value/mask pair. The class has to be derived, + and the size attribute must be set. + """ + + size = None # Size in bits. + + def __init__(self, string): + if not self.size: + raise NotImplementedError( + "IntMask should be derived and size should be fixed" + ) + + parts = string.split("/") + if len(parts) > 1: + self._value = int(parts[0], 0) + self._mask = int(parts[1], 0) + if self._mask.bit_length() > self.size: + raise ValueError( + "Integer mask {} is bigger than size {}".format( + self._mask, self.size + ) + ) + else: + self._value = int(parts[0], 0) + self._mask = self.max_mask() + + if self._value.bit_length() > self.size: + raise ValueError( + "Integer value {} is bigger than size {}".format( + self._value, self.size + ) + ) + + @property + def value(self): + return self._value + + @property + def mask(self): + return self._mask + + def max_mask(self): + return 2 ** self.size - 1 + + def fully(self): + """Returns True if it's fully masked.""" + return self._mask == self.max_mask() + + def __str__(self): + if self.fully(): + return str(self._value) + else: + return "{}/{}".format(hex(self._value), hex(self._mask)) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def __eq__(self, other): + """Equality operator. + + Both value and mask must be the same for the comparison to result True. + This can be used to implement filters that expect a specific mask, + e.g: ct.state = 0x1/0xff. + + Args: + other (IntMask): Another IntMask to compare against. + + Returns: + True if the other IntMask is the same as this one. + """ + if isinstance(other, IntMask): + return self.value == other.value and self.mask == other.mask + elif isinstance(other, int): + return self.value == other and self.mask == self.max_mask() + else: + raise ValueError("Cannot compare against ", other) + + def __contains__(self, other): + """Contains operator. + + Args: + other (int or IntMask): Another integer or fully-masked IntMask + to compare against. + + Returns: + True if the other integer or fully-masked IntMask is + contained in this IntMask. + + Example: + 0x1 in IntMask("0xf1/0xff"): True + 0x1 in IntMask("0xf1/0x0f"): True + 0x1 in IntMask("0xf1/0xf0"): False + """ + if isinstance(other, IntMask): + if other.fully(): + return other.value in self + else: + raise ValueError( + "Comparing non fully-masked IntMasks is not supported" + ) + else: + return other & self._mask == self._value & self._mask + + def dict(self): + return {"value": self._value, "mask": self._mask} + + def to_json(self): + return self.dict() + + +class Mask8(IntMask): + size = 8 + + +class Mask16(IntMask): + size = 16 + + +class Mask32(IntMask): + size = 32 + + +class Mask64(IntMask): + size = 64 + + +class Mask128(IntMask): + size = 128 + + +class Mask992(IntMask): + size = 992 + + +def decode_mask(mask_size): + """Value/Mask decoder for values of specific size (bits). + + Used for fields such as: + reg0=0x248/0xff + """ + + class Mask(IntMask): + size = mask_size + __name__ = "Mask{}".format(size) + + return Mask + + +class EthMask(Decoder): + """EthMask represents an Ethernet address with optional mask. + + It uses netaddr.EUI. + + Attributes: + eth (netaddr.EUI): The Ethernet address. + mask (netaddr.EUI): Optional, the Ethernet address mask. + + Args: + string (str): A string representing the masked Ethernet address + e.g: 00.11:22:33:44:55 or 01:00:22:00:33:00/01:00:00:00:00:00 + """ + + def __init__(self, string): + mask_parts = string.split("/") + self._eth = netaddr.EUI(mask_parts[0]) + if len(mask_parts) == 2: + self._mask = netaddr.EUI(mask_parts[1]) + else: + self._mask = None + + @property + def eth(self): + """The Ethernet address.""" + return self._eth + + @property + def mask(self): + """The Ethernet address mask.""" + return self._mask + + def __eq__(self, other): + """Equality operator. + + Both the Ethernet address and the mask are compared. This can be used + to implement filters where we expect a specific mask to be present, + e.g: dl_dst=01:00:00:00:00:00/01:00:00:00:00:00. + + Args: + other (EthMask): Another EthMask to compare against. + + Returns: + True if this EthMask is the same as the other. + """ + return self._mask == other._mask and self._eth == other._eth + + def __contains__(self, other): + """Contains operator. + + Args: + other (netaddr.EUI or EthMask): An Ethernet address. + + Returns: + True if the other netaddr.EUI or fully-masked EthMask is + contained in this EthMask's address range. + """ + if isinstance(other, EthMask): + if other._mask: + raise ValueError( + "Comparing non fully-masked EthMask is not supported" + ) + return other._eth in self + + if self._mask: + return (other.value & self._mask.value) == ( + self._eth.value & self._mask.value + ) + else: + return other == self._eth + + def __str__(self): + if self._mask: + return "/".join( + [ + self._eth.format(netaddr.mac_unix), + self._mask.format(netaddr.mac_unix), + ] + ) + else: + return self._eth.format(netaddr.mac_unix) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) + + +class IPMask(Decoder): + """IPMask stores an IPv6 or IPv4 and a mask. + + It uses netaddr.IPAddress. + + IPMasks can represent valid CIDRs or randomly masked IP Addresses. + + Args: + string (str): A string representing the ip/mask. + """ + + def __init__(self, string): + self._ipnet = None + self._ip = None + self._mask = None + try: + self._ipnet = netaddr.IPNetwork(string) + except netaddr.AddrFormatError: + pass + + if not self._ipnet: + # It's not a valid CIDR. Store ip and mask independently. + parts = string.split("/") + if len(parts) != 2: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) + try: + self._ip = netaddr.IPAddress(parts[0]) + self._mask = netaddr.IPAddress(parts[1]) + except netaddr.AddrFormatError as exc: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) from exc + + def __eq__(self, other): + """Equality operator. + + Both the IPAddress and the mask are compared. This can be used + to implement filters where a specific mask is expected, e.g: + nw_src=192.168.1.0/24. + + Args: + other (IPMask or netaddr.IPNetwork or netaddr.IPAddress): + Another IPAddress or IPNetwork to compare against. + + Returns: + True if this IPMask is the same as the other. + """ + if isinstance(other, netaddr.IPNetwork): + return self._ipnet and self._ipnet == other + if isinstance(other, netaddr.IPAddress): + return self._ipnet and self._ipnet.ip == other + elif isinstance(other, IPMask): + if self._ipnet: + return self._ipnet == other._ipnet + + return self._ip == other._ip and self._mask == other._mask + else: + return False + + def __contains__(self, other): + """Contains operator. + + Only comparing valid CIDRs is supported. + + Args: + other (netaddr.IPAddress or IPMask): An IP address. + + Returns: + True if the other IPAddress is contained in this IPMask's address + range. + """ + if isinstance(other, IPMask): + if not other._ipnet: + raise ValueError("Only comparing valid CIDRs is supported") + + return ( + netaddr.IPAddress(other._ipnet.first) in self + and netaddr.IPAddress(other._ipnet.last) in self + ) + + elif isinstance(other, netaddr.IPAddress): + if self._ipnet: + return other in self._ipnet + return (other & self._mask) == (self._ip & self._mask) + + def cidr(self): + """ + Returns True if the IPMask is a valid CIDR. + """ + return self._ipnet is not None + + @property + def ip(self): + """The IP address.""" + if self._ipnet: + return self._ipnet.ip + return self._ip + + @property + def mask(self): + """The IP mask.""" + if self._ipnet: + return self._ipnet.netmask + return self._mask + + def __str__(self): + if self._ipnet: + return str(self._ipnet) + return "/".join([str(self._ip), str(self._mask)]) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) diff --git a/python/setup.py b/python/setup.py index 7ac3c3662..350ac6056 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers'], + install_requires=['sortedcontainers', 'netaddr'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Thu Jun 16 06:32:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644139 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=O+R0pAs8; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnG2GKyz9sFw for ; Thu, 16 Jun 2022 16:33:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 28F2761128; Thu, 16 Jun 2022 06:33:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id btiEbI4H30Ia; Thu, 16 Jun 2022 06:33:17 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6452161118; Thu, 16 Jun 2022 06:33:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4D6E2C0081; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 019B9C0083 for ; Thu, 16 Jun 2022 06:33:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id DA24B61111 for ; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vvuyXU0N03Dc for ; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0272761109 for ; Thu, 16 Jun 2022 06:33:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t1kevR6+vhEErcAcklv6WemI9c3+7MeouJ4giLFhFFw=; b=O+R0pAs8bmxQlWrx5uZn9lSnJaret7L0buondE5ayFGSmCwzlnIvmxjSKk2QU+H3ISH9FU XLAty+k1p8RTfFJ6OTKfNDfHAf5M7gaB5Rney1NFhqxrl3Ae+Gcz7eOgPgXZp7AQPSAF+3 MjFVrricQ6++R0eZG/lLVwQWDy8uGqU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-347-iYO1tra3MMqCSftzR-xGCw-1; Thu, 16 Jun 2022 02:33:08 -0400 X-MC-Unique: iYO1tra3MMqCSftzR-xGCw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3FF71811766; Thu, 16 Jun 2022 06:33:08 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24F4D492C3B; Thu, 16 Jun 2022 06:33:06 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:33 +0200 Message-Id: <20220616063247.517147-4-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 03/17] python: add list parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Some openflow or dpif flows encode their arguments in lists, eg: "some_action(arg1,arg2,arg3)". In order to decode this in a way that can be then stored and queried, add ListParser and ListDecoders classes that parse lists into KeyValue instances. The ListParser / ListDecoders mechanism is quite similar to KVParser and KVDecoders. Since the "key" of the different KeyValue objects is now ommited, it has to be provided by ListDecoders. For example, take the openflow action "resubmit" that can be written as: resubmit([port],[table][,ct]) Can be decoded by creating a ListDecoders instance such as: ListDecoders([ ("port", decode_default), ("table", decode_int), ("ct", decode_flag), ]) Naturally, the order of the decoders must be kept. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flow/list.py | 121 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 python/ovs/flow/list.py diff --git a/python/automake.mk b/python/automake.mk index e3da6bdbe..4c07004b3 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -29,6 +29,7 @@ ovs_pyfiles = \ python/ovs/flow/__init__.py \ python/ovs/flow/decoders.py \ python/ovs/flow/kv.py \ + python/ovs/flow/list.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flow/list.py b/python/ovs/flow/list.py new file mode 100644 index 000000000..b1e9e3fca --- /dev/null +++ b/python/ovs/flow/list.py @@ -0,0 +1,121 @@ +import re + +from ovs.flow.kv import KeyValue, KeyMetadata, ParseError +from ovs.flow.decoders import decode_default + + +class ListDecoders(object): + """ListDecoders is used by ListParser to decode the elements in the list. + + A decoder is a function that accepts a value and returns its decoded + object. + + ListDecoders is initialized with a list of tuples that contains the + keyword and the decoding function associated with each position in the + list. The order is, therefore, important. + + Args: + decoders (list of tuples): Optional; a list of tuples. + The first element in the tuple is the keyword associated with the + value. The second element in the tuple is the decoder function. + """ + + def __init__(self, decoders=None): + self._decoders = decoders or list() + + def decode(self, index, value_str): + """Decode the index'th element of the list. + + Args: + index (int): The position in the list of the element to decode. + value_str (str): The value string to decode. + """ + if index < 0 or index >= len(self._decoders): + return self._default_decoder(index, value_str) + + try: + key = self._decoders[index][0] + value = self._decoders[index][1](value_str) + return key, value + except Exception as e: + raise ParseError( + "Failed to decode value_str {}: {}".format(value_str, str(e)) + ) + + @staticmethod + def _default_decoder(index, value): + key = "elem_{}".format(index) + return key, decode_default(value) + + +class ListParser(object): + """ListParser parses a list of values and stores them as key-value pairs. + + It uses a ListDecoders instance to decode each element in the list. + + Args: + string (str): The string to parse. + decoders (ListDecoders): Optional, the decoders to use. + delims (list): Optional, list of delimiters of the list. Defaults to + [',']. + """ + def __init__(self, string, decoders=None, delims=[","]): + self._string = string + self._decoders = decoders or ListDecoders() + self._keyval = list() + self._regexp = r"({})".format("|".join(delims)) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the list in string. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + index = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + split_parts = re.split(self._regexp, self._string[kpos:], 1) + value_str = split_parts[0] + + key, value = self._decoders.decode(index, value_str) + + meta = KeyMetadata( + kpos=kpos, + vpos=kpos, + kstring=value_str, + vstring=value_str, + ) + self._keyval.append(KeyValue(key, value, meta)) + + kpos += len(value_str) + 1 + index += 1 + + +def decode_nested_list(decoders, value, delims=[","]): + """Decodes a string value that contains a list of elements and returns + them in a dictionary. + + Args: + decoders (ListDecoders): The ListDecoders to use. + value (str): The value string to decode. + delims (list(str)): Optional, the list of delimiters to use. + """ + parser = ListParser(value, decoders, delims) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_list_decoder(decoders=None, delims=[","]): + """Helper function that creates a nested list decoder with given + ListDecoders and delimiters. + """ + def decoder(value): + return decode_nested_list(decoders, value, delims) + + return decoder From patchwork Thu Jun 16 06:32:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644142 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XV9aSCq6; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnP5XSRz9sFw for ; Thu, 16 Jun 2022 16:33:29 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id BFDE741A69; Thu, 16 Jun 2022 06:33:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7zUfZP9Ih5YU; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 21F4B4094B; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 70881C0088; Thu, 16 Jun 2022 06:33:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 806E4C007A for ; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2F7BB83FFF for ; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=fail (1024-bit key) reason="fail (body has been altered)" header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z4dKZOLZax65 for ; Thu, 16 Jun 2022 06:33:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 8CB8883FAE for ; Thu, 16 Jun 2022 06:33:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n3KiJRkMXeMkkEfRSawqaHzJX1jRZ136sORn/L3aFZk=; b=XV9aSCq6dbfNr9nLWbihvMD/cnLWvR3PaHZB5F42IdtXs3qoN8MSji6VftdP1m2K0AfLh+ 3upGCdS/vpU4LTDNhoXL2lJxql3gnvJQHPFP1mVqXr1MqXk/bqPfHuCPFlKsco26/dS2zP bHNjPoRXLeeI5yoquf7MEy2vS4jnEQE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-259-yaqqBJp4MImCIuIVaeN9oA-1; Thu, 16 Jun 2022 02:33:10 -0400 X-MC-Unique: yaqqBJp4MImCIuIVaeN9oA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB7AB8525A3; Thu, 16 Jun 2022 06:33:09 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8EBA6492C3B; Thu, 16 Jun 2022 06:33:08 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:34 +0200 Message-Id: <20220616063247.517147-5-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 04/17] build-aux: split extract-ofp-fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to be able to reuse the core extraction logic, split the command in two parts. The core extraction logic is moved to python/build while the command that writes the different files out of the extracted field info is kept in build-aux. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- build-aux/extract-ofp-fields | 706 ++++++++--------------------- python/automake.mk | 7 +- python/build/extract_ofp_fields.py | 421 +++++++++++++++++ 3 files changed, 618 insertions(+), 516 deletions(-) create mode 100644 python/build/extract_ofp_fields.py diff --git a/build-aux/extract-ofp-fields b/build-aux/extract-ofp-fields index 8766995d9..efec59c25 100755 --- a/build-aux/extract-ofp-fields +++ b/build-aux/extract-ofp-fields @@ -3,85 +3,23 @@ import getopt import sys import os.path -import re import xml.dom.minidom import build.nroff -line = "" - -# Maps from user-friendly version number to its protocol encoding. -VERSION = {"1.0": 0x01, - "1.1": 0x02, - "1.2": 0x03, - "1.3": 0x04, - "1.4": 0x05, - "1.5": 0x06} -VERSION_REVERSE = dict((v,k) for k, v in VERSION.items()) - -TYPES = {"u8": (1, False), - "be16": (2, False), - "be32": (4, False), - "MAC": (6, False), - "be64": (8, False), - "be128": (16, False), - "tunnelMD": (124, True)} - -FORMATTING = {"decimal": ("MFS_DECIMAL", 1, 8), - "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), - "ct state": ("MFS_CT_STATE", 4, 4), - "Ethernet": ("MFS_ETHERNET", 6, 6), - "IPv4": ("MFS_IPV4", 4, 4), - "IPv6": ("MFS_IPV6", 16, 16), - "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), - "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), - "frag": ("MFS_FRAG", 1, 1), - "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), - "TCP flags": ("MFS_TCP_FLAGS", 2, 2), - "packet type": ("MFS_PACKET_TYPE", 4, 4)} - -PREREQS = {"none": "MFP_NONE", - "Ethernet": "MFP_ETHERNET", - "ARP": "MFP_ARP", - "VLAN VID": "MFP_VLAN_VID", - "IPv4": "MFP_IPV4", - "IPv6": "MFP_IPV6", - "IPv4/IPv6": "MFP_IP_ANY", - "NSH": "MFP_NSH", - "CT": "MFP_CT_VALID", - "MPLS": "MFP_MPLS", - "TCP": "MFP_TCP", - "UDP": "MFP_UDP", - "SCTP": "MFP_SCTP", - "ICMPv4": "MFP_ICMPV4", - "ICMPv6": "MFP_ICMPV6", - "ND": "MFP_ND", - "ND solicit": "MFP_ND_SOLICIT", - "ND advert": "MFP_ND_ADVERT"} - -# Maps a name prefix into an (experimenter ID, class) pair, so: -# -# - Standard OXM classes are written as (0, ) -# -# - Experimenter OXM classes are written as (, 0xffff) -# -# If a name matches more than one prefix, the longest one is used. -OXM_CLASSES = {"NXM_OF_": (0, 0x0000, 'extension'), - "NXM_NX_": (0, 0x0001, 'extension'), - "NXOXM_NSH_": (0x005ad650, 0xffff, 'extension'), - "OXM_OF_": (0, 0x8000, 'standard'), - "OXM_OF_PKT_REG": (0, 0x8001, 'standard'), - "ONFOXM_ET_": (0x4f4e4600, 0xffff, 'standard'), - "ERICOXM_OF_": (0, 0x1000, 'extension'), - - # This is the experimenter OXM class for Nicira, which is the - # one that OVS would be using instead of NXM_OF_ and NXM_NX_ - # if OVS didn't have those grandfathered in. It is currently - # used only to test support for experimenter OXM, since there - # are barely any real uses of experimenter OXM in the wild. - "NXOXM_ET_": (0x00002320, 0xffff, 'extension')} +from build.extract_ofp_fields import ( + extract_ofp_fields, + PREREQS, + OXM_CLASSES, + VERSION, + fatal, + n_errors, +) + +VERSION_REVERSE = dict((v, k) for k, v in VERSION.items()) + def oxm_name_to_class(name): - prefix = '' + prefix = "" class_ = None for p, c in OXM_CLASSES.items(): if name.startswith(p) and len(p) > len(prefix): @@ -92,267 +30,76 @@ def oxm_name_to_class(name): def is_standard_oxm(name): oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) - return oxm_class_type == 'standard' - - -def decode_version_range(range): - if range in VERSION: - return (VERSION[range], VERSION[range]) - elif range.endswith('+'): - return (VERSION[range[:-1]], max(VERSION.values())) - else: - a, b = re.match(r'^([^-]+)-([^-]+)$', range).groups() - return (VERSION[a], VERSION[b]) - - -def get_line(): - global line - global line_number - line = input_file.readline() - line_number += 1 - if line == "": - fatal("unexpected end of input") - - -n_errors = 0 - - -def error(msg): - global n_errors - sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) - n_errors += 1 - - -def fatal(msg): - error(msg) - sys.exit(1) + return oxm_class_type == "standard" def usage(): argv0 = os.path.basename(sys.argv[0]) - print('''\ + print( + """\ %(argv0)s, for extracting OpenFlow field properties from meta-flow.h usage: %(argv0)s INPUT [--meta-flow | --nx-match] where INPUT points to lib/meta-flow.h in the source directory. Depending on the option given, the output written to stdout is intended to be saved either as lib/meta-flow.inc or lib/nx-match.inc for the respective C file to #include.\ -''' % {"argv0": argv0}) +""" + % {"argv0": argv0} + ) sys.exit(0) -def make_sizeof(s): - m = re.match(r'(.*) up to (.*)', s) - if m: - struct, member = m.groups() - return "offsetof(%s, %s)" % (struct, member) - else: - return "sizeof(%s)" % s - - -def parse_oxms(s, prefix, n_bytes): - if s == 'none': - return () - - return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(',')) - - -match_types = dict() - - -def parse_oxm(s, prefix, n_bytes): - global match_types - - m = re.match('([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$', s) - if not m: - fatal("%s: syntax error parsing %s" % (s, prefix)) - - name, oxm_type, of_version, ovs_version = m.groups() - - class_ = oxm_name_to_class(name) - if class_ is None: - fatal("unknown OXM class for %s" % name) - oxm_vendor, oxm_class, oxm_class_type = class_ - - if class_ in match_types: - if oxm_type in match_types[class_]: - fatal("duplicate match type for %s (conflicts with %s)" % - (name, match_types[class_][oxm_type])) - else: - match_types[class_] = dict() - match_types[class_][oxm_type] = name - - # Normally the oxm_length is the size of the field, but for experimenter - # OXMs oxm_length also includes the 4-byte experimenter ID. - oxm_length = n_bytes - if oxm_class == 0xffff: - oxm_length += 4 - - header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) - - if of_version: - if oxm_class_type == 'extension': - fatal("%s: OXM extension can't have OpenFlow version" % name) - if of_version not in VERSION: - fatal("%s: unknown OpenFlow version %s" % (name, of_version)) - of_version_nr = VERSION[of_version] - if of_version_nr < VERSION['1.2']: - fatal("%s: claimed version %s predates OXM" % (name, of_version)) - else: - if oxm_class_type == 'standard': - fatal("%s: missing OpenFlow version number" % name) - of_version_nr = 0 - - return (header, name, of_version_nr, ovs_version) - - -def parse_field(mff, comment): - f = {'mff': mff} - - # First line of comment is the field name. - m = re.match(r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0]) - if not m: - fatal("%s lacks field name" % mff) - f['name'], f['extra_name'] = m.groups() - - # Find the last blank line the comment. The field definitions - # start after that. - blank = None - for i in range(len(comment)): - if not comment[i]: - blank = i - if not blank: - fatal("%s: missing blank line in comment" % mff) - - d = {} - for key in ("Type", "Maskable", "Formatting", "Prerequisites", - "Access", "Prefix lookup member", - "OXM", "NXM", "OF1.0", "OF1.1"): - d[key] = None - for fline in comment[blank + 1:]: - m = re.match(r'([^:]+):\s+(.*)\.$', fline) - if not m: - fatal("%s: syntax error parsing key-value pair as part of %s" - % (fline, mff)) - key, value = m.groups() - if key not in d: - fatal("%s: unknown key" % key) - elif key == 'Code point': - d[key] += [value] - elif d[key] is not None: - fatal("%s: duplicate key" % key) - d[key] = value - for key, value in d.items(): - if not value and key not in ("OF1.0", "OF1.1", - "Prefix lookup member", "Notes"): - fatal("%s: missing %s" % (mff, key)) - - m = re.match(r'([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$', d['Type']) - if not m: - fatal("%s: syntax error in type" % mff) - type_ = m.group(1) - if type_ not in TYPES: - fatal("%s: unknown type %s" % (mff, d['Type'])) - - f['n_bytes'] = TYPES[type_][0] - if m.group(2): - f['n_bits'] = int(m.group(2)) - if f['n_bits'] > f['n_bytes'] * 8: - fatal("%s: more bits (%d) than field size (%d)" - % (mff, f['n_bits'], 8 * f['n_bytes'])) - else: - f['n_bits'] = 8 * f['n_bytes'] - f['variable'] = TYPES[type_][1] - - if d['Maskable'] == 'no': - f['mask'] = 'MFM_NONE' - elif d['Maskable'] == 'bitwise': - f['mask'] = 'MFM_FULLY' - else: - fatal("%s: unknown maskable %s" % (mff, d['Maskable'])) - - fmt = FORMATTING.get(d['Formatting']) - if not fmt: - fatal("%s: unknown format %s" % (mff, d['Formatting'])) - f['formatting'] = d['Formatting'] - if f['n_bytes'] < fmt[1] or f['n_bytes'] > fmt[2]: - fatal("%s: %d-byte field can't be formatted as %s" - % (mff, f['n_bytes'], d['Formatting'])) - f['string'] = fmt[0] - - f['prereqs'] = d['Prerequisites'] - if f['prereqs'] not in PREREQS: - fatal("%s: unknown prerequisites %s" % (mff, d['Prerequisites'])) - - if d['Access'] == 'read-only': - f['writable'] = False - elif d['Access'] == 'read/write': - f['writable'] = True - else: - fatal("%s: unknown access %s" % (mff, d['Access'])) - - f['OF1.0'] = d['OF1.0'] - if not d['OF1.0'] in (None, 'exact match', 'CIDR mask'): - fatal("%s: unknown OF1.0 match type %s" % (mff, d['OF1.0'])) - - f['OF1.1'] = d['OF1.1'] - if not d['OF1.1'] in (None, 'exact match', 'bitwise mask'): - fatal("%s: unknown OF1.1 match type %s" % (mff, d['OF1.1'])) - - f['OXM'] = (parse_oxms(d['OXM'], 'OXM', f['n_bytes']) + - parse_oxms(d['NXM'], 'NXM', f['n_bytes'])) - - f['prefix'] = d["Prefix lookup member"] - - return f - - def protocols_to_c(protocols): - if protocols == set(['of10', 'of11', 'oxm']): - return 'OFPUTIL_P_ANY' - elif protocols == set(['of11', 'oxm']): - return 'OFPUTIL_P_NXM_OF11_UP' - elif protocols == set(['oxm']): - return 'OFPUTIL_P_NXM_OXM_ANY' + if protocols == set(["of10", "of11", "oxm"]): + return "OFPUTIL_P_ANY" + elif protocols == set(["of11", "oxm"]): + return "OFPUTIL_P_NXM_OF11_UP" + elif protocols == set(["oxm"]): + return "OFPUTIL_P_NXM_OXM_ANY" elif protocols == set([]): - return 'OFPUTIL_P_NONE' + return "OFPUTIL_P_NONE" else: assert False def autogen_c_comment(): return [ -"/* Generated automatically; do not modify! -*- buffer-read-only: t -*- */", -""] + "/* Generated automatically; do not modify! " + "-*- buffer-read-only: t -*- */", + "", + ] + def make_meta_flow(meta_flow_h): fields = extract_ofp_fields(meta_flow_h) output = autogen_c_comment() for f in fields: output += ["{"] - output += [" %s," % f['mff']] - if f['extra_name']: - output += [" \"%s\", \"%s\"," % (f['name'], f['extra_name'])] + output += [" %s," % f["mff"]] + if f["extra_name"]: + output += [' "%s", "%s",' % (f["name"], f["extra_name"])] else: - output += [" \"%s\", NULL," % f['name']] + output += [' "%s", NULL,' % f["name"]] - if f['variable']: - variable = 'true' + if f["variable"]: + variable = "true" else: - variable = 'false' - output += [" %d, %d, %s," % (f['n_bytes'], f['n_bits'], variable)] + variable = "false" + output += [" %d, %d, %s," % (f["n_bytes"], f["n_bits"], variable)] - if f['writable']: - rw = 'true' + if f["writable"]: + rw = "true" else: - rw = 'false' - output += [" %s, %s, %s, %s, false," - % (f['mask'], f['string'], PREREQS[f['prereqs']], rw)] - - oxm = f['OXM'] - of10 = f['OF1.0'] - of11 = f['OF1.1'] - if f['mff'] in ('MFF_DL_VLAN', 'MFF_DL_VLAN_PCP'): + rw = "false" + output += [ + " %s, %s, %s, %s, false," + % (f["mask"], f["string"], PREREQS[f["prereqs"]], rw) + ] + + oxm = f["OXM"] + of10 = f["OF1.0"] + of11 = f["OF1.1"] + if f["mff"] in ("MFF_DL_VLAN", "MFF_DL_VLAN_PCP"): # MFF_DL_VLAN and MFF_DL_VLAN_PCP don't exactly correspond to # OF1.1, nor do they have NXM or OXM assignments, but their # meanings can be expressed in every protocol, which is the goal of @@ -367,25 +114,25 @@ def make_meta_flow(meta_flow_h): if oxm: protocols |= set(["oxm"]) - if f['mask'] == 'MFM_FULLY': + if f["mask"] == "MFM_FULLY": cidr_protocols = protocols.copy() bitwise_protocols = protocols.copy() - if of10 == 'exact match': - bitwise_protocols -= set(['of10']) - cidr_protocols -= set(['of10']) - elif of10 == 'CIDR mask': - bitwise_protocols -= set(['of10']) + if of10 == "exact match": + bitwise_protocols -= set(["of10"]) + cidr_protocols -= set(["of10"]) + elif of10 == "CIDR mask": + bitwise_protocols -= set(["of10"]) else: assert of10 is None - if of11 == 'exact match': - bitwise_protocols -= set(['of11']) - cidr_protocols -= set(['of11']) + if of11 == "exact match": + bitwise_protocols -= set(["of11"]) + cidr_protocols -= set(["of11"]) else: - assert of11 in (None, 'bitwise mask') + assert of11 in (None, "bitwise mask") else: - assert f['mask'] == 'MFM_NONE' + assert f["mask"] == "MFM_NONE" cidr_protocols = set([]) bitwise_protocols = set([]) @@ -393,8 +140,8 @@ def make_meta_flow(meta_flow_h): output += [" %s," % protocols_to_c(cidr_protocols)] output += [" %s," % protocols_to_c(bitwise_protocols)] - if f['prefix']: - output += [" FLOW_U32OFS(%s)," % f['prefix']] + if f["prefix"]: + output += [" FLOW_U32OFS(%s)," % f["prefix"]] else: output += [" -1, /* not usable for prefix lookup */"] @@ -409,147 +156,37 @@ def make_nx_match(meta_flow_h): print("static struct nxm_field_index all_nxm_fields[] = {") for f in fields: # Sort by OpenFlow version number (nx-match.c depends on this). - for oxm in sorted(f['OXM'], key=lambda x: x[2]): - header = ("NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0]) - print("""{ .nf = { %s, %d, "%s", %s } },""" % ( - header, oxm[2], oxm[1], f['mff'])) + for oxm in sorted(f["OXM"], key=lambda x: x[2]): + header = "NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0] + print( + """{ .nf = { %s, %d, "%s", %s } },""" + % (header, oxm[2], oxm[1], f["mff"]) + ) print("};") for oline in output: print(oline) -def extract_ofp_fields(fn): - global file_name - global input_file - global line_number - global line - - file_name = fn - input_file = open(file_name) - line_number = 0 - - fields = [] - - while True: - get_line() - if re.match('enum.*mf_field_id', line): - break - - while True: - get_line() - first_line_number = line_number - here = '%s:%d' % (file_name, line_number) - if (line.startswith('/*') - or line.startswith(' *') - or line.startswith('#') - or not line - or line.isspace()): - continue - elif re.match('}', line) or re.match('\s+MFF_N_IDS', line): - break - - # Parse the comment preceding an MFF_ constant into 'comment', - # one line to an array element. - line = line.strip() - if not line.startswith('/*'): - fatal("unexpected syntax between fields") - line = line[1:] - comment = [] - end = False - while not end: - line = line.strip() - if line.startswith('*/'): - get_line() - break - if not line.startswith('*'): - fatal("unexpected syntax within field") - - line = line[1:] - if line.startswith(' '): - line = line[1:] - if line.startswith(' ') and comment: - continuation = True - line = line.lstrip() - else: - continuation = False - - if line.endswith('*/'): - line = line[:-2].rstrip() - end = True - else: - end = False - - if continuation: - comment[-1] += " " + line - else: - comment += [line] - get_line() - - # Drop blank lines at each end of comment. - while comment and not comment[0]: - comment = comment[1:] - while comment and not comment[-1]: - comment = comment[:-1] - - # Parse the MFF_ constant(s). - mffs = [] - while True: - m = re.match('\s+(MFF_[A-Z0-9_]+),?\s?$', line) - if not m: - break - mffs += [m.group(1)] - get_line() - if not mffs: - fatal("unexpected syntax looking for MFF_ constants") - - if len(mffs) > 1 or '' in comment[0]: - for mff in mffs: - # Extract trailing integer. - m = re.match('.*[^0-9]([0-9]+)$', mff) - if not m: - fatal("%s lacks numeric suffix in register group" % mff) - n = m.group(1) - - # Search-and-replace within the comment, - # and drop lines that have for x != n. - instance = [] - for x in comment: - y = x.replace('', n) - if re.search('<[0-9]+>', y): - if ('<%s>' % n) not in y: - continue - y = re.sub('<[0-9]+>', '', y) - instance += [y.strip()] - fields += [parse_field(mff, instance)] - else: - fields += [parse_field(mffs[0], comment)] - continue - - input_file.close() - - if n_errors: - sys.exit(1) - - return fields - ## ------------------------ ## ## Documentation Generation ## ## ------------------------ ## + def field_to_xml(field_node, f, body, summary): f["used"] = True # Summary. - if field_node.hasAttribute('internal'): + if field_node.hasAttribute("internal"): return min_of_version = None min_ovs_version = None - for header, name, of_version_nr, ovs_version_s in f['OXM']: - if (is_standard_oxm(name) - and (min_ovs_version is None or of_version_nr < min_of_version)): + for header, name, of_version_nr, ovs_version_s in f["OXM"]: + if is_standard_oxm(name) and ( + min_ovs_version is None or of_version_nr < min_of_version + ): min_of_version = of_version_nr - ovs_version = [int(x) for x in ovs_version_s.split('.')] + ovs_version = [int(x) for x in ovs_version_s.split(".")] if min_ovs_version is None or ovs_version < min_ovs_version: min_ovs_version = ovs_version summary += ["\\fB%s\\fR" % f["name"]] @@ -565,124 +202,152 @@ def field_to_xml(field_node, f, body, summary): if min_of_version is not None: support += ["OF %s+" % VERSION_REVERSE[min_of_version]] if min_ovs_version is not None: - support += ["OVS %s+" % '.'.join([str(x) for x in min_ovs_version])] - summary += ' and '.join(support) + support += ["OVS %s+" % ".".join([str(x) for x in min_ovs_version])] + summary += " and ".join(support) summary += ["\n"] # Full description. - if field_node.hasAttribute('hidden'): + if field_node.hasAttribute("hidden"): return - title = field_node.attributes['title'].nodeValue + title = field_node.attributes["title"].nodeValue - body += [""".PP + body += [ + """.PP \\fB%s Field\\fR .TS tab(;); l lx. -""" % title] +""" + % title + ] body += ["Name:;\\fB%s\\fR" % f["name"]] if f["extra_name"]: body += [" (aka \\fB%s\\fR)" % f["extra_name"]] - body += ['\n'] + body += ["\n"] body += ["Width:;"] if f["n_bits"] != 8 * f["n_bytes"]: - body += ["%d bits (only the least-significant %d bits " - "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"])] + body += [ + "%d bits (only the least-significant %d bits " + "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"]) + ] elif f["n_bits"] <= 128: body += ["%d bits" % f["n_bits"]] else: body += ["%d bits (%d bytes)" % (f["n_bits"], f["n_bits"] / 8)] - body += ['\n'] + body += ["\n"] body += ["Format:;%s\n" % f["formatting"]] - masks = {"MFM_NONE": "not maskable", - "MFM_FULLY": "arbitrary bitwise masks"} + masks = { + "MFM_NONE": "not maskable", + "MFM_FULLY": "arbitrary bitwise masks", + } body += ["Masking:;%s\n" % masks[f["mask"]]] body += ["Prerequisites:;%s\n" % f["prereqs"]] - access = {True: "read/write", - False: "read-only"}[f["writable"]] + access = {True: "read/write", False: "read-only"}[f["writable"]] body += ["Access:;%s\n" % access] - of10 = {None: "not supported", - "exact match": "yes (exact match only)", - "CIDR mask": "yes (CIDR match only)"} + of10 = { + None: "not supported", + "exact match": "yes (exact match only)", + "CIDR mask": "yes (CIDR match only)", + } body += ["OpenFlow 1.0:;%s\n" % of10[f["OF1.0"]]] - of11 = {None: "not supported", - "exact match": "yes (exact match only)", - "bitwise mask": "yes"} + of11 = { + None: "not supported", + "exact match": "yes (exact match only)", + "bitwise mask": "yes", + } body += ["OpenFlow 1.1:;%s\n" % of11[f["OF1.1"]]] oxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if is_standard_oxm(x[1])]: + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if is_standard_oxm(x[1]) + ]: of_version = VERSION_REVERSE[of_version_nr] - oxms += [r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" % (name, header[2], of_version, ovs_version)] + oxms += [ + r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" + % (name, header[2], of_version, ovs_version) + ] if not oxms: - oxms = ['none'] - body += ['OXM:;T{\n%s\nT}\n' % r'\[char59] '.join(oxms)] + oxms = ["none"] + body += ["OXM:;T{\n%s\nT}\n" % r"\[char59] ".join(oxms)] nxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if not is_standard_oxm(x[1])]: - nxms += [r"\fB%s\fR (%d) since Open vSwitch %s" % (name, header[2], ovs_version)] + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if not is_standard_oxm(x[1]) + ]: + nxms += [ + r"\fB%s\fR (%d) since Open vSwitch %s" + % (name, header[2], ovs_version) + ] if not nxms: - nxms = ['none'] - body += ['NXM:;T{\n%s\nT}\n' % r'\[char59] '.join(nxms)] + nxms = ["none"] + body += ["NXM:;T{\n%s\nT}\n" % r"\[char59] ".join(nxms)] body += [".TE\n"] - body += ['.PP\n'] + body += [".PP\n"] body += [build.nroff.block_xml_to_nroff(field_node.childNodes)] + def group_xml_to_nroff(group_node, fields): - title = group_node.attributes['title'].nodeValue + title = group_node.attributes["title"].nodeValue summary = [] body = [] for node in group_node.childNodes: - if node.nodeType == node.ELEMENT_NODE and node.tagName == 'field': - id_ = node.attributes['id'].nodeValue + if node.nodeType == node.ELEMENT_NODE and node.tagName == "field": + id_ = node.attributes["id"].nodeValue field_to_xml(node, fields[id_], body, summary) else: body += [build.nroff.block_xml_to_nroff([node])] content = [ - '.bp\n', - '.SH \"%s\"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), + ".bp\n", + '.SH "%s"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), '.SS "Summary:"\n', - '.TS\n', - 'tab(;);\n', - 'l l l l l l l.\n', - 'Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n', - '\_;\_;\_;\_;\_;\_\n'] + ".TS\n", + "tab(;);\n", + "l l l l l l l.\n", + "Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n", + "\_;\_;\_;\_;\_;\_\n", + ] content += summary - content += ['.TE\n'] + content += [".TE\n"] content += body - return ''.join(content) + return "".join(content) + def make_oxm_classes_xml(document): - s = '''tab(;); + s = """tab(;); l l l. Prefix;Vendor;Class \_;\_;\_ -''' +""" for key in sorted(OXM_CLASSES, key=OXM_CLASSES.get): vendor, class_, class_type = OXM_CLASSES.get(key) - s += r"\fB%s\fR;" % key.rstrip('_') + s += r"\fB%s\fR;" % key.rstrip("_") if vendor: s += r"\fL0x%08x\fR;" % vendor else: s += "(none);" s += r"\fL0x%04x\fR;" % class_ s += "\n" - e = document.createElement('tbl') + e = document.createElement("tbl") e.appendChild(document.createTextNode(s)) return e + def recursively_replace(node, name, replacement): for child in node.childNodes: if child.nodeType == node.ELEMENT_NODE: @@ -691,11 +356,12 @@ def recursively_replace(node, name, replacement): else: recursively_replace(child, name, replacement) + def make_ovs_fields(meta_flow_h, meta_flow_xml): fields = extract_ofp_fields(meta_flow_h) fields_map = {} for f in fields: - fields_map[f['mff']] = f + fields_map[f["mff"]] = f document = xml.dom.minidom.parse(meta_flow_xml) doc = document.documentElement @@ -704,7 +370,8 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): if version == None: version = "UNKNOWN" - print('''\ + print( + """\ '\\" tp .\\" -*- mode: troff; coding: utf-8 -*- .TH "ovs\-fields" 7 "%s" "Open vSwitch" "Open vSwitch Manual" @@ -740,11 +407,13 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch . .PP -''' % version) +""" + % version + ) - recursively_replace(doc, 'oxm_classes', make_oxm_classes_xml(document)) + recursively_replace(doc, "oxm_classes", make_oxm_classes_xml(document)) - s = '' + s = "" for node in doc.childNodes: if node.nodeType == node.ELEMENT_NODE and node.tagName == "group": s += group_xml_to_nroff(node, fields_map) @@ -757,9 +426,10 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch for f in fields: if "used" not in f: - fatal("%s: field not documented " - "(please add documentation in lib/meta-flow.xml)" - % f["mff"]) + fatal( + "%s: field not documented " + "(please add documentation in lib/meta-flow.xml)" % f["mff"] + ) if n_errors: sys.exit(1) @@ -769,26 +439,27 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch # Life is easier with nroff if we don't try to feed it Unicode. # Fortunately, we only use a few characters outside the ASCII range. - oline = oline.replace(u'\u2208', r'\[mo]') - oline = oline.replace(u'\u2260', r'\[!=]') - oline = oline.replace(u'\u2264', r'\[<=]') - oline = oline.replace(u'\u2265', r'\[>=]') - oline = oline.replace(u'\u00d7', r'\[mu]') + oline = oline.replace(u"\u2208", r"\[mo]") + oline = oline.replace(u"\u2260", r"\[!=]") + oline = oline.replace(u"\u2264", r"\[<=]") + oline = oline.replace(u"\u2265", r"\[>=]") + oline = oline.replace(u"\u00d7", r"\[mu]") if len(oline): output += [oline] # nroff tends to ignore .bp requests if they come after .PP requests, # so remove .PPs that precede .bp. for i in range(len(output)): - if output[i] == '.bp': + if output[i] == ".bp": j = i - 1 - while j >= 0 and output[j] == '.PP': + while j >= 0 and output[j] == ".PP": output[j] = None j -= 1 for i in range(len(output)): if output[i] is not None: print(output[i]) - + + ## ------------ ## ## Main Program ## ## ------------ ## @@ -796,8 +467,9 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch if __name__ == "__main__": argv0 = sys.argv[0] try: - options, args = getopt.gnu_getopt(sys.argv[1:], 'h', - ['help', 'ovs-version=']) + options, args = getopt.gnu_getopt( + sys.argv[1:], "h", ["help", "ovs-version="] + ) except getopt.GetoptError as geo: sys.stderr.write("%s: %s\n" % (argv0, geo.msg)) sys.exit(1) @@ -805,32 +477,38 @@ if __name__ == "__main__": global version version = None for key, value in options: - if key in ['-h', '--help']: + if key in ["-h", "--help"]: usage() - elif key == '--ovs-version': + elif key == "--ovs-version": version = value else: sys.exit(0) if not args: - sys.stderr.write("%s: missing command argument " - "(use --help for help)\n" % argv0) + sys.stderr.write( + "%s: missing command argument " "(use --help for help)\n" % argv0 + ) sys.exit(1) - commands = {"meta-flow": (make_meta_flow, 1), - "nx-match": (make_nx_match, 1), - "ovs-fields": (make_ovs_fields, 2)} + commands = { + "meta-flow": (make_meta_flow, 1), + "nx-match": (make_nx_match, 1), + "ovs-fields": (make_ovs_fields, 2), + } if not args[0] in commands: - sys.stderr.write("%s: unknown command \"%s\" " - "(use --help for help)\n" % (argv0, args[0])) + sys.stderr.write( + '%s: unknown command "%s" ' + "(use --help for help)\n" % (argv0, args[0]) + ) sys.exit(1) func, n_args = commands[args[0]] if len(args) - 1 != n_args: - sys.stderr.write("%s: \"%s\" requires %d arguments but %d " - "provided\n" - % (argv0, args[0], n_args, len(args) - 1)) + sys.stderr.write( + '%s: "%s" requires %d arguments but %d ' + "provided\n" % (argv0, args[0], n_args, len(args) - 1) + ) sys.exit(1) func(*args[1:]) diff --git a/python/automake.mk b/python/automake.mk index 4c07004b3..eea8567d3 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -51,6 +51,7 @@ ovs_pyfiles = \ # so they are not installed. EXTRA_DIST += \ python/build/__init__.py \ + python/build/extract_ofp_fields.py \ python/build/nroff.py \ python/build/soutil.py @@ -69,10 +70,12 @@ PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) FLAKE8_PYFILES += \ $(filter-out python/ovs/compat/% python/ovs/dirs.py,$(PYFILES)) \ - python/setup.py \ python/build/__init__.py \ + python/build/extract_ofp_fields.py \ python/build/nroff.py \ - python/ovs/dirs.py.template + python/build/soutil.py \ + python/ovs/dirs.py.template \ + python/setup.py nobase_pkgdata_DATA = $(ovs_pyfiles) $(ovstest_pyfiles) ovs-install-data-local: diff --git a/python/build/extract_ofp_fields.py b/python/build/extract_ofp_fields.py new file mode 100644 index 000000000..3fe62634a --- /dev/null +++ b/python/build/extract_ofp_fields.py @@ -0,0 +1,421 @@ +import sys +import re + +line = "" + +# Maps from user-friendly version number to its protocol encoding. +VERSION = { + "1.0": 0x01, + "1.1": 0x02, + "1.2": 0x03, + "1.3": 0x04, + "1.4": 0x05, + "1.5": 0x06, +} +VERSION_REVERSE = dict((v, k) for k, v in VERSION.items()) + +TYPES = { + "u8": (1, False), + "be16": (2, False), + "be32": (4, False), + "MAC": (6, False), + "be64": (8, False), + "be128": (16, False), + "tunnelMD": (124, True), +} + +FORMATTING = { + "decimal": ("MFS_DECIMAL", 1, 8), + "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), + "ct state": ("MFS_CT_STATE", 4, 4), + "Ethernet": ("MFS_ETHERNET", 6, 6), + "IPv4": ("MFS_IPV4", 4, 4), + "IPv6": ("MFS_IPV6", 16, 16), + "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), + "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), + "frag": ("MFS_FRAG", 1, 1), + "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), + "TCP flags": ("MFS_TCP_FLAGS", 2, 2), + "packet type": ("MFS_PACKET_TYPE", 4, 4), +} + +PREREQS = { + "none": "MFP_NONE", + "Ethernet": "MFP_ETHERNET", + "ARP": "MFP_ARP", + "VLAN VID": "MFP_VLAN_VID", + "IPv4": "MFP_IPV4", + "IPv6": "MFP_IPV6", + "IPv4/IPv6": "MFP_IP_ANY", + "NSH": "MFP_NSH", + "CT": "MFP_CT_VALID", + "MPLS": "MFP_MPLS", + "TCP": "MFP_TCP", + "UDP": "MFP_UDP", + "SCTP": "MFP_SCTP", + "ICMPv4": "MFP_ICMPV4", + "ICMPv6": "MFP_ICMPV6", + "ND": "MFP_ND", + "ND solicit": "MFP_ND_SOLICIT", + "ND advert": "MFP_ND_ADVERT", +} + +# Maps a name prefix into an (experimenter ID, class) pair, so: +# +# - Standard OXM classes are written as (0, ) +# +# - Experimenter OXM classes are written as (, 0xffff) +# +# If a name matches more than one prefix, the longest one is used. +OXM_CLASSES = { + "NXM_OF_": (0, 0x0000, "extension"), + "NXM_NX_": (0, 0x0001, "extension"), + "NXOXM_NSH_": (0x005AD650, 0xFFFF, "extension"), + "OXM_OF_": (0, 0x8000, "standard"), + "OXM_OF_PKT_REG": (0, 0x8001, "standard"), + "ONFOXM_ET_": (0x4F4E4600, 0xFFFF, "standard"), + "ERICOXM_OF_": (0, 0x1000, "extension"), + # This is the experimenter OXM class for Nicira, which is the + # one that OVS would be using instead of NXM_OF_ and NXM_NX_ + # if OVS didn't have those grandfathered in. It is currently + # used only to test support for experimenter OXM, since there + # are barely any real uses of experimenter OXM in the wild. + "NXOXM_ET_": (0x00002320, 0xFFFF, "extension"), +} + + +def oxm_name_to_class(name): + prefix = "" + class_ = None + for p, c in OXM_CLASSES.items(): + if name.startswith(p) and len(p) > len(prefix): + prefix = p + class_ = c + return class_ + + +def is_standard_oxm(name): + oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) + return oxm_class_type == "standard" + + +def get_line(): + global line + global line_number + line = input_file.readline() + line_number += 1 + if line == "": + fatal("unexpected end of input") + + +n_errors = 0 + + +def error(msg): + global n_errors + sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) + n_errors += 1 + + +def fatal(msg): + error(msg) + sys.exit(1) + + +def parse_oxms(s, prefix, n_bytes): + if s == "none": + return () + + return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(",")) + + +match_types = dict() + + +def parse_oxm(s, prefix, n_bytes): + global match_types + + m = re.match( + r"([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$", # noqa: E501 + s, + ) + if not m: + fatal("%s: syntax error parsing %s" % (s, prefix)) + + name, oxm_type, of_version, ovs_version = m.groups() + + class_ = oxm_name_to_class(name) + if class_ is None: + fatal("unknown OXM class for %s" % name) + oxm_vendor, oxm_class, oxm_class_type = class_ + + if class_ in match_types: + if oxm_type in match_types[class_]: + fatal( + "duplicate match type for %s (conflicts with %s)" + % (name, match_types[class_][oxm_type]) + ) + else: + match_types[class_] = dict() + match_types[class_][oxm_type] = name + + # Normally the oxm_length is the size of the field, but for experimenter + # OXMs oxm_length also includes the 4-byte experimenter ID. + oxm_length = n_bytes + if oxm_class == 0xFFFF: + oxm_length += 4 + + header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) + + if of_version: + if oxm_class_type == "extension": + fatal("%s: OXM extension can't have OpenFlow version" % name) + if of_version not in VERSION: + fatal("%s: unknown OpenFlow version %s" % (name, of_version)) + of_version_nr = VERSION[of_version] + if of_version_nr < VERSION["1.2"]: + fatal("%s: claimed version %s predates OXM" % (name, of_version)) + else: + if oxm_class_type == "standard": + fatal("%s: missing OpenFlow version number" % name) + of_version_nr = 0 + + return (header, name, of_version_nr, ovs_version) + + +def parse_field(mff, comment): + f = {"mff": mff} + + # First line of comment is the field name. + m = re.match( + r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0] + ) + if not m: + fatal("%s lacks field name" % mff) + f["name"], f["extra_name"] = m.groups() + + # Find the last blank line the comment. The field definitions + # start after that. + blank = None + for i in range(len(comment)): + if not comment[i]: + blank = i + if not blank: + fatal("%s: missing blank line in comment" % mff) + + d = {} + for key in ( + "Type", + "Maskable", + "Formatting", + "Prerequisites", + "Access", + "Prefix lookup member", + "OXM", + "NXM", + "OF1.0", + "OF1.1", + ): + d[key] = None + for fline in comment[blank + 1 :]: + m = re.match(r"([^:]+):\s+(.*)\.$", fline) + if not m: + fatal( + "%s: syntax error parsing key-value pair as part of %s" + % (fline, mff) + ) + key, value = m.groups() + if key not in d: + fatal("%s: unknown key" % key) + elif key == "Code point": + d[key] += [value] + elif d[key] is not None: + fatal("%s: duplicate key" % key) + d[key] = value + for key, value in d.items(): + if not value and key not in ( + "OF1.0", + "OF1.1", + "Prefix lookup member", + "Notes", + ): + fatal("%s: missing %s" % (mff, key)) + + m = re.match(r"([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$", d["Type"]) + if not m: + fatal("%s: syntax error in type" % mff) + type_ = m.group(1) + if type_ not in TYPES: + fatal("%s: unknown type %s" % (mff, d["Type"])) + + f["n_bytes"] = TYPES[type_][0] + if m.group(2): + f["n_bits"] = int(m.group(2)) + if f["n_bits"] > f["n_bytes"] * 8: + fatal( + "%s: more bits (%d) than field size (%d)" + % (mff, f["n_bits"], 8 * f["n_bytes"]) + ) + else: + f["n_bits"] = 8 * f["n_bytes"] + f["variable"] = TYPES[type_][1] + + if d["Maskable"] == "no": + f["mask"] = "MFM_NONE" + elif d["Maskable"] == "bitwise": + f["mask"] = "MFM_FULLY" + else: + fatal("%s: unknown maskable %s" % (mff, d["Maskable"])) + + fmt = FORMATTING.get(d["Formatting"]) + if not fmt: + fatal("%s: unknown format %s" % (mff, d["Formatting"])) + f["formatting"] = d["Formatting"] + if f["n_bytes"] < fmt[1] or f["n_bytes"] > fmt[2]: + fatal( + "%s: %d-byte field can't be formatted as %s" + % (mff, f["n_bytes"], d["Formatting"]) + ) + f["string"] = fmt[0] + + f["prereqs"] = d["Prerequisites"] + if f["prereqs"] not in PREREQS: + fatal("%s: unknown prerequisites %s" % (mff, d["Prerequisites"])) + + if d["Access"] == "read-only": + f["writable"] = False + elif d["Access"] == "read/write": + f["writable"] = True + else: + fatal("%s: unknown access %s" % (mff, d["Access"])) + + f["OF1.0"] = d["OF1.0"] + if not d["OF1.0"] in (None, "exact match", "CIDR mask"): + fatal("%s: unknown OF1.0 match type %s" % (mff, d["OF1.0"])) + + f["OF1.1"] = d["OF1.1"] + if not d["OF1.1"] in (None, "exact match", "bitwise mask"): + fatal("%s: unknown OF1.1 match type %s" % (mff, d["OF1.1"])) + + f["OXM"] = parse_oxms(d["OXM"], "OXM", f["n_bytes"]) + parse_oxms( + d["NXM"], "NXM", f["n_bytes"] + ) + + f["prefix"] = d["Prefix lookup member"] + + return f + + +def extract_ofp_fields(fn): + global file_name + global input_file + global line_number + global line + + file_name = fn + input_file = open(file_name) + line_number = 0 + + fields = [] + + while True: + get_line() + if re.match("enum.*mf_field_id", line): + break + + while True: + get_line() + if ( + line.startswith("/*") + or line.startswith(" *") + or line.startswith("#") + or not line + or line.isspace() + ): + continue + elif re.match(r"}", line) or re.match(r"\s+MFF_N_IDS", line): + break + + # Parse the comment preceding an MFF_ constant into 'comment', + # one line to an array element. + line = line.strip() + if not line.startswith("/*"): + fatal("unexpected syntax between fields") + line = line[1:] + comment = [] + end = False + while not end: + line = line.strip() + if line.startswith("*/"): + get_line() + break + if not line.startswith("*"): + fatal("unexpected syntax within field") + + line = line[1:] + if line.startswith(" "): + line = line[1:] + if line.startswith(" ") and comment: + continuation = True + line = line.lstrip() + else: + continuation = False + + if line.endswith("*/"): + line = line[:-2].rstrip() + end = True + else: + end = False + + if continuation: + comment[-1] += " " + line + else: + comment += [line] + get_line() + + # Drop blank lines at each end of comment. + while comment and not comment[0]: + comment = comment[1:] + while comment and not comment[-1]: + comment = comment[:-1] + + # Parse the MFF_ constant(s). + mffs = [] + while True: + m = re.match(r"\s+(MFF_[A-Z0-9_]+),?\s?$", line) + if not m: + break + mffs += [m.group(1)] + get_line() + if not mffs: + fatal("unexpected syntax looking for MFF_ constants") + + if len(mffs) > 1 or "" in comment[0]: + for mff in mffs: + # Extract trailing integer. + m = re.match(".*[^0-9]([0-9]+)$", mff) + if not m: + fatal("%s lacks numeric suffix in register group" % mff) + n = m.group(1) + + # Search-and-replace within the comment, + # and drop lines that have for x != n. + instance = [] + for x in comment: + y = x.replace("", n) + if re.search("<[0-9]+>", y): + if ("<%s>" % n) not in y: + continue + y = re.sub("<[0-9]+>", "", y) + instance += [y.strip()] + fields += [parse_field(mff, instance)] + else: + fields += [parse_field(mffs[0], comment)] + continue + + input_file.close() + + if n_errors: + sys.exit(1) + + return fields From patchwork Thu Jun 16 06:32:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644140 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=J1uuCfuo; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnJ71QKz9sFw for ; Thu, 16 Jun 2022 16:33:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B789961135; Thu, 16 Jun 2022 06:33:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uRABJhcEESL5; Thu, 16 Jun 2022 06:33:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id B1DF161127; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 42EA0C008C; Thu, 16 Jun 2022 06:33:17 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 73F84C0086 for ; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 409E283FB1 for ; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PLPyAXKdE6eB for ; Thu, 16 Jun 2022 06:33:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 2101683FAD for ; Thu, 16 Jun 2022 06:33:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361192; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5Y2NaYOjD9AlLGXBiKgvGuUIbVz2DtzQ9g7czK0ts3Y=; b=J1uuCfuopL+7VM3xtQ5Sz736yJFqMSZ6Sh6fl2UNsP1Yn8NX/TDhhThmdqgIyARahk6bzf B9C+l6ROB4NLL6sf7td/tcUuZeJHS8NjVGelBMQtwnb3V8lx53sQEpNE6/h01nIjfEoaco t7QUiQcjh5i5o0F/ahrkSxtDk9j605A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-14-wzZhopnYP-OnZOg6JRoIyA-1; Thu, 16 Jun 2022 02:33:11 -0400 X-MC-Unique: wzZhopnYP-OnZOg6JRoIyA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5330F3C0ED43; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 280F8492C3B; Thu, 16 Jun 2022 06:33:10 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:35 +0200 Message-Id: <20220616063247.517147-6-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 05/17] build-aux: generate ofp field decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on meta-field information extracted by extract_ofp_fields, autogenerate the right decoder to be used. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- build-aux/automake.mk | 6 ++- build-aux/gen_ofp_field_decoders | 69 ++++++++++++++++++++++++++++++++ python/.gitignore | 1 + python/automake.mk | 7 ++++ 4 files changed, 81 insertions(+), 2 deletions(-) create mode 100755 build-aux/gen_ofp_field_decoders diff --git a/build-aux/automake.mk b/build-aux/automake.mk index 6267ccd7c..b9a77a51c 100644 --- a/build-aux/automake.mk +++ b/build-aux/automake.mk @@ -5,6 +5,7 @@ EXTRA_DIST += \ build-aux/dist-docs \ build-aux/dpdkstrip.py \ build-aux/generate-dhparams-c \ + build-aux/gen_ofp_field_decoders \ build-aux/initial-tab-allowed-files \ build-aux/sodepends.py \ build-aux/soexpand.py \ @@ -12,7 +13,8 @@ EXTRA_DIST += \ build-aux/xml2nroff FLAKE8_PYFILES += \ - $(srcdir)/build-aux/xml2nroff \ build-aux/dpdkstrip.py \ + build-aux/gen_ofp_field_decoders \ build-aux/sodepends.py \ - build-aux/soexpand.py + build-aux/soexpand.py \ + build-aux/xml2nroff diff --git a/build-aux/gen_ofp_field_decoders b/build-aux/gen_ofp_field_decoders new file mode 100755 index 000000000..71364aeb4 --- /dev/null +++ b/build-aux/gen_ofp_field_decoders @@ -0,0 +1,69 @@ +#!/bin/env python + +import argparse + +import build.extract_ofp_fields as extract_fields + + +def main(): + parser = argparse.ArgumentParser( + description="Tool to generate python ofproto field decoders from" + "meta-flow information" + ) + parser.add_argument( + "metaflow", + metavar="FILE", + type=str, + help="Read meta-flow info from file", + ) + + args = parser.parse_args() + + fields = extract_fields.extract_ofp_fields(args.metaflow) + + field_decoders = {} + for field in fields: + decoder = get_decoder(field) + field_decoders[field.get("name")] = decoder + if field.get("extra_name"): + field_decoders[field.get("extra_name")] = decoder + + code = """ +# This file is auto-generated. Do not edit! + +import functools +from ovs.flow import decoders + +field_decoders = {{ +{decoders} +}} +""".format( + decoders="\n".join( + [ + " '{name}': {decoder},".format(name=name, decoder=decoder) + for name, decoder in field_decoders.items() + ] + ) + ) + print(code) + + +def get_decoder(field): + formatting = field.get("formatting") + if formatting in ["decimal", "hexadecimal"]: + if field.get("mask") == "MFM_NONE": + return "decoders.decode_int" + else: + if field.get("n_bits") in [8, 16, 32, 64, 128, 992]: + return "decoders.Mask{}".format(field.get("n_bits")) + return "decoders.decode_mask({})".format(field.get("n_bits")) + elif formatting in ["IPv4", "IPv6"]: + return "decoders.IPMask" + elif formatting == "Ethernet": + return "decoders.EthMask" + else: + return "decoders.decode_default" + + +if __name__ == "__main__": + main() diff --git a/python/.gitignore b/python/.gitignore index 60ace6f05..38e1f8b90 100644 --- a/python/.gitignore +++ b/python/.gitignore @@ -1,2 +1,3 @@ dist/ *.egg-info +ovs/flow/ofp_fields.py diff --git a/python/automake.mk b/python/automake.mk index eea8567d3..c9a1e7c67 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -125,3 +125,10 @@ $(srcdir)/python/ovs/dirs.py: python/ovs/dirs.py.template mv $@.tmp $@ EXTRA_DIST += python/ovs/dirs.py.template CLEANFILES += python/ovs/dirs.py + +$(srcdir)/python/ovs/flow/ofp_fields.py: $(srcdir)/build-aux/gen_ofp_field_decoders include/openvswitch/meta-flow.h + $(AM_V_GEN)$(run_python) $< $(srcdir)/include/openvswitch/meta-flow.h > $@.tmp + $(AM_V_at)mv $@.tmp $@ +EXTRA_DIST += python/ovs/flow/ofp_fields.py +CLEANFILES += python/ovs/flow/ofp_fields.py + From patchwork Thu Jun 16 06:32:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644141 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=F1BkpLVQ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnN0N9nz9sFw for ; Thu, 16 Jun 2022 16:33:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0089561123; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Tc1bq3kYvA4; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6A42561118; Thu, 16 Jun 2022 06:33:20 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5ED28C007A; Thu, 16 Jun 2022 06:33:19 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5B94FC0082 for ; Thu, 16 Jun 2022 06:33:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3B4286111D for ; Thu, 16 Jun 2022 06:33:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1wwRQEph-8l4 for ; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4276F61118 for ; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lzxghs6oX/SyGpAnatvR5ern8wu4Fsvk0o8SUd3lBDE=; b=F1BkpLVQTh470xQ0gopBXcWxm16qiFGjjDs+607FvmYkQvpOM4LA+X5rCqTbXgzZAuEPQa 9Q7EWolV3kMiUbNXP9xUfznUlUC/urM6CuYnMnHozWK1D0atnP83r/mPEBXt6HfR8e7n5X neb5eEVXvIL+MegSMDqRo2H3U+87Wh0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-214-bW62ijtMMDKiEljkdDPJKg-1; Thu, 16 Jun 2022 02:33:12 -0400 X-MC-Unique: bW62ijtMMDKiEljkdDPJKg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7FF523802B95; Thu, 16 Jun 2022 06:33:12 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA3BF492C3B; Thu, 16 Jun 2022 06:33:11 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:36 +0200 Message-Id: <20220616063247.517147-7-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 06/17] python: add flow base class X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It simplifies the implementation of different types of flows by creating the concept of Section (e.g: match, action) and automatic accessors for all the provided Sections Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flow/flow.py | 125 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 python/ovs/flow/flow.py diff --git a/python/automake.mk b/python/automake.mk index c9a1e7c67..b34a5324b 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flow/__init__.py \ python/ovs/flow/decoders.py \ + python/ovs/flow/flow.py \ python/ovs/flow/kv.py \ python/ovs/flow/list.py \ python/ovs/json.py \ diff --git a/python/ovs/flow/flow.py b/python/ovs/flow/flow.py new file mode 100644 index 000000000..2f053e77d --- /dev/null +++ b/python/ovs/flow/flow.py @@ -0,0 +1,125 @@ +""" Defines the Flow class. +""" + + +class Section(object): + """A flow can be seen as composed of different sections, e.g: + + [info] [match] actions=[actions] + + This class represents each of those sections. + + A section is basically a set of Key-Value pairs. Typically, they can be + expressed as a dictionary, for instance the "match" part of a flow can be + expressed as: + { + "nw_src": "192.168.1.1", + "nw_dst": "192.168.1.2", + } + However, some of them must be expressed as a list which allows for + duplicated keys. For instance, the "actions" section could be: + [ + { + "output": 32 + }, + { + "output": 33 + } + ] + + The is_list flag is used to discriminate this. + + Attributes: + name (str): Name of the section. + pos (int): Position within the overall flow string. + string (str): Section string. + data (list[KeyValue]): Parsed data of the section. + is_list (bool): Whether the key-values shall be expressed as a list + (i.e: it allows repeated keys). + """ + + def __init__(self, name, pos, string, data, is_list=False): + self.name = name + self.pos = pos + self.string = string + self.data = data + self.is_list = is_list + + def __str__(self): + return "{} (at {}): {}".format(self.name, self.pos, self.string) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def dict(self): + return {self.name: self.format_data()} + + def format_data(self): + """Returns the section's key-values formatted in a dictionary or list + depending on the value of is_list flag. + """ + if self.is_list: + return [{item.key: item.value} for item in self.data] + else: + return {item.key: item.value for item in self.data} + + +class Flow(object): + """The Flow class is a base class for other types of concrete flows + (such as OFproto Flows or DPIF Flows). + + A flow is basically comprised of a number of sections. + For each section named {section_name}, the flow object will have the + following attributes: + - {section_name} will return the sections data in a formatted way. + - {section_name}_kv will return the sections data as a list of KeyValues. + + Args: + sections (list[Section]): List of sections that comprise the flow + orig (str): Original flow string. + id (Any): Optional; identifier that clients can use to uniquely + identify this flow. + """ + + def __init__(self, sections, orig="", id=None): + self._sections = sections + self._orig = orig + self._id = id + for section in sections: + setattr( + self, section.name, self.section(section.name).format_data() + ) + setattr( + self, + "{}_kv".format(section.name), + self.section(section.name).data, + ) + + def section(self, name): + """Return the section by name.""" + return next( + (sect for sect in self._sections if sect.name == name), None + ) + + @property + def id(self): + """Return the Flow ID.""" + return self._id + + @property + def sections(self): + """Return the all the sections in a list.""" + return self._sections + + @property + def orig(self): + """Return the original flow string.""" + return self._orig + + def dict(self): + """Returns the Flow information in a dictionary.""" + flow_dict = {"orig": self.orig} + for section in self.sections: + flow_dict.update(section.dict()) + + return flow_dict From patchwork Thu Jun 16 06:32:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644144 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bGiiOEnf; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsng06rXz9sFw for ; Thu, 16 Jun 2022 16:33:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id D13BA84004; Thu, 16 Jun 2022 06:33:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rZwTiRb87wRh; Thu, 16 Jun 2022 06:33:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3C9138402F; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B86A1C0081; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 18B8AC007A for ; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0B15C61130 for ; Thu, 16 Jun 2022 06:33:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xhI3dXQ5qSOI for ; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6689B6111B for ; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7DlRB2Zmy9PCtTq0s3MxqqkhskGt3MfENNImxzymVzc=; b=bGiiOEnfsHqy1CLHmxcLQNN7twFzGgOVY5FfqWGNW5iZWzGEGZyBOizpRu3FCquRn8ef0V YfHUyCSDFAOKQEjd49HdlJLdTAmj0IRzroY0UXBkpyXgddYHrZFH+QFIIctxWsHc6OXxFU UVrnHvgSUa5J2tIxelsMkJ/qJqsQzwE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-60-tkzvXupDN-qPS8Y5v9785Q-1; Thu, 16 Jun 2022 02:33:13 -0400 X-MC-Unique: tkzvXupDN-qPS8Y5v9785Q-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A80B93802B95; Thu, 16 Jun 2022 06:33:13 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id C6862492CA5; Thu, 16 Jun 2022 06:33:12 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:37 +0200 Message-Id: <20220616063247.517147-8-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 07/17] python: introduce OpenFlow Flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Introduce OFPFlow class and all its decoders. Most of the decoders are generic (from decoders.py). Some have special syntax and need a specific implementation. Decoders for nat are moved to the common decoders.py because it's syntax is shared with other types of flows (e.g: dpif flows). Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 2 + python/ovs/flow/decoders.py | 108 +++++++++ python/ovs/flow/ofp.py | 428 ++++++++++++++++++++++++++++++++++++ python/ovs/flow/ofp_act.py | 306 ++++++++++++++++++++++++++ 4 files changed, 844 insertions(+) create mode 100644 python/ovs/flow/ofp.py create mode 100644 python/ovs/flow/ofp_act.py diff --git a/python/automake.mk b/python/automake.mk index b34a5324b..50cf6b298 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,8 @@ ovs_pyfiles = \ python/ovs/flow/flow.py \ python/ovs/flow/kv.py \ python/ovs/flow/list.py \ + python/ovs/flow/ofp.py \ + python/ovs/flow/ofp_act.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flow/decoders.py b/python/ovs/flow/decoders.py index 883e61acf..73d28e057 100644 --- a/python/ovs/flow/decoders.py +++ b/python/ovs/flow/decoders.py @@ -6,6 +6,7 @@ object. """ import netaddr +import re class Decoder(object): @@ -414,3 +415,110 @@ class IPMask(Decoder): def to_json(self): return str(self) + + +def decode_free_output(value): + """The value of the output action can be found free, i.e: without the + 'output' keyword. This decoder decodes its value when found this way.""" + try: + return "output", {"port": int(value)} + except ValueError: + return "output", {"port": value.strip('"')} + + +ipv4 = r"(?:\d{1,3}.?){3}\d{1,3}" +ipv4_capture = r"({ipv4})".format(ipv4=ipv4) +ipv6 = r"[\w:\.]+" +ipv6_capture = r"(?:\[*)?({ipv6})(?:\]*)?".format(ipv6=ipv6) +port_range = r":(\d+)(?:-(\d+))?" +ip_range_regexp = r"{ip_cap}(?:-{ip_cap})?(?:{port_range})?" +ipv4_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv4_capture, port_range=port_range) +) +ipv6_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv6_capture, port_range=port_range) +) + + +def decode_ip_port_range(value): + """ + Decodes an IP and port range: + {ip_start}-{ip-end}:{port_start}-{port_end} + + IPv6 addresses are surrounded by "[" and "]" if port ranges are also + present + + Returns the following dictionary: + { + "addrs": { + "start": {ip_start} + "end": {ip_end} + } + "ports": { + "start": {port_start}, + "end": {port_end} + } + (the "ports" key might be omitted) + """ + if value.count(":") > 1: + match = ipv6_port_regex.match(value) + else: + match = ipv4_port_regex.match(value) + + ip_start = match.group(1) + ip_end = match.group(2) + port_start = match.group(3) + port_end = match.group(4) + + result = { + "addrs": { + "start": netaddr.IPAddress(ip_start), + "end": netaddr.IPAddress(ip_end or ip_start), + } + } + if port_start: + result["ports"] = { + "start": int(port_start), + "end": int(port_end or port_start), + } + + return result + + +def decode_nat(value): + """Decodes the 'nat' keyword of the ct action. + + The format is: + nat + Flag format. + nat(type=addrs[:ports][,flag]...) + Full format where the address-port range has the same format as + the one described in decode_ip_port_range. + + Examples: + nat(src=0.0.0.0) + nat(src=0.0.0.0,persistent) + nat(dst=192.168.1.0-192.168.1.253:4000-5000) + nat(dst=192.168.1.0-192.168.1.253,hash) + nat(dst=[fe80::f150]-[fe80::f15f]:255-300) + """ + if not value: + return True # If flag format, the value is True. + + result = dict() + type_parts = value.split("=") + result["type"] = type_parts[0] + + if len(type_parts) > 1: + value_parts = type_parts[1].split(",") + if len(type_parts) != 2: + raise ValueError("Malformed nat action: %s" % value) + + ip_port_range = decode_ip_port_range(value_parts[0]) + + result = {"type": type_parts[0], **ip_port_range} + + for flag in value_parts[1:]: + result[flag] = True + + return result diff --git a/python/ovs/flow/ofp.py b/python/ovs/flow/ofp.py new file mode 100644 index 000000000..0bc110c57 --- /dev/null +++ b/python/ovs/flow/ofp.py @@ -0,0 +1,428 @@ +"""Defines the parsers needed to parse ofproto flows. +""" + +import functools + +from ovs.flow.kv import KVParser, KVDecoders, nested_kv_decoder +from ovs.flow.ofp_fields import field_decoders +from ovs.flow.flow import Flow, Section +from ovs.flow.list import ListDecoders, nested_list_decoder +from ovs.flow.decoders import ( + decode_default, + decode_flag, + decode_int, + decode_time, + decode_mask, + IPMask, + EthMask, + decode_free_output, + decode_nat, +) +from ovs.flow.ofp_act import ( + decode_output, + decode_field, + decode_controller, + decode_bundle, + decode_bundle_load, + decode_encap, + decode_load_field, + decode_set_field, + decode_move_field, + decode_dec_ttl, + decode_chk_pkt_larger, + decode_zone, + decode_exec, + decode_learn, +) + + +class OFPFlow(Flow): + """OFPFLow represents an OpenFlow Flow. + + Attributes: + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not OFPFlow._info_decoders: + OFPFlow._info_decoders = OFPFlow._gen_info_decoders() + return OFPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not OFPFlow._match_decoders: + OFPFlow._match_decoders = OFPFlow._gen_match_decoders() + return OFPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not OFPFlow._action_decoders: + OFPFlow._action_decoders = OFPFlow._gen_action_decoders() + return OFPFlow._action_decoders + + def __init__(self, ofp_string, id=None): + """Create a OFPFlow from a flow string. + + The string is expected to have the followoing format: + + [flow data] [match] actions=[actions] + + Args: + ofp_string(str): An OpenFlow flow string. + id(Any): Optional; any object used to uniquely identify this flow + from the rest. + + Returns + An OFPFlow with the content of the flow string or None if there is + no flow information but the string is expected to be found in a + flow dump. + + Raises + ValueError if the string is malformed. + ParseError if an error in parsing occurs. + """ + if " reply " in ofp_string: + return None + + sections = list() + parts = ofp_string.split("actions=") + if len(parts) != 2: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + actions = parts[1] + + field_parts = parts[0].rstrip(" ").rpartition(" ") + if len(field_parts) != 3: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + info = field_parts[0] + match = field_parts[2] + + iparser = KVParser(info, OFPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=ofp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, OFPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=ofp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, OFPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=ofp_string.find(actions), + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(OFPFlow, self).__init__(sections, ofp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + args = { + "table": decode_int, + "duration": decode_time, + "n_packet": decode_int, + "n_bytes": decode_int, + "cookie": decode_int, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "hard_age": decode_time, + } + return KVDecoders(args) + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + args = { + **OFPFlow._field_decoder_args(), + **OFPFlow._extra_match_decoder_args(), + } + + return KVDecoders(args) + + @staticmethod + def _extra_match_decoder_args(): + """Returns the extra KVDecoder arguments needed to decode the match + part of a flow (apart from the fields).""" + return { + "priority": decode_int, + } + + @staticmethod + def _field_decoder_args(): + """Returns the KVDecoder arguments needed to decode match fields.""" + shorthands = [ + "eth", + "ip", + "ipv6", + "icmp", + "icmp6", + "tcp", + "tcp6", + "udp", + "udp6", + "sctp", + "arp", + "rarp", + "mpls", + "mplsm", + ] + + fields = {**field_decoders, **{key: decode_flag for key in shorthands}} + + # vlan_vid field is special. Although it is technically 12 bit wide, + # bit 12 is allowed to be set to 1 to indicate that the vlan header is + # present (see section VLAN FIELDS in + # http://www.openvswitch.org/support/dist-docs/ovs-fields.7.txt) + # Therefore, override the generated vlan_vid field size. + fields["vlan_vid"] = decode_mask(13) + return fields + + @staticmethod + def _gen_action_decoders(): + """Generate the actions decoders.""" + + actions = { + **OFPFlow._output_actions_decoders_args(), + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + **OFPFlow._fw_action_decoders_args(), + **OFPFlow._control_action_decoders_args(), + **OFPFlow._other_action_decoders_args(), + } + clone_actions = OFPFlow._clone_actions_decoders_args(actions) + actions.update(clone_actions) + return KVDecoders(actions, default_free=decode_free_output) + + @staticmethod + def _output_actions_decoders_args(): + """Returns the decoder arguments for the output actions.""" + return { + "output": decode_output, + "drop": decode_flag, + "controller": decode_controller, + "enqueue": nested_list_decoder( + ListDecoders([("port", decode_default), ("queue", int)]), + delims=[",", ":"], + ), + "bundle": decode_bundle, + "bundle_load": decode_bundle_load, + "group": decode_default, + } + + @staticmethod + def _encap_actions_decoders_args(): + """Returns the decoders arguments for the encap actions.""" + + return { + "pop_vlan": decode_flag, + "strip_vlan": decode_flag, + "push_vlan": decode_default, + "decap": decode_flag, + "encap": decode_encap, + } + + @staticmethod + def _field_action_decoders_args(): + """Returns the decoders arguments for field-modification actions.""" + # Field modification actions + field_default_decoders = [ + "set_mpls_label", + "set_mpls_tc", + "set_mpls_ttl", + "mod_nw_tos", + "mod_nw_ecn", + "mod_tcp_src", + "mod_tcp_dst", + ] + return { + "load": decode_load_field, + "set_field": functools.partial( + decode_set_field, KVDecoders(OFPFlow._field_decoder_args()) + ), + "move": decode_move_field, + "mod_dl_dst": EthMask, + "mod_dl_src": EthMask, + "mod_nw_dst": IPMask, + "mod_nw_src": IPMask, + "dec_ttl": decode_dec_ttl, + "dec_mpls_ttl": decode_flag, + "dec_nsh_ttl": decode_flag, + "check_pkt_larger": decode_chk_pkt_larger, + **{field: decode_default for field in field_default_decoders}, + } + + @staticmethod + def _meta_action_decoders_args(): + """Returns the decoders arguments for the metadata actions.""" + meta_default_decoders = ["set_tunnel", "set_tunnel64", "set_queue"] + return { + "pop_queue": decode_flag, + **{field: decode_default for field in meta_default_decoders}, + } + + @staticmethod + def _fw_action_decoders_args(): + """Returns the decoders arguments for the firewalling actions.""" + return { + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "zone": decode_zone, + "table": decode_int, + "nat": decode_nat, + "force": decode_flag, + "exec": functools.partial( + decode_exec, + KVDecoders( + { + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + } + ), + ), + "alg": decode_default, + } + ) + ), + "ct_clear": decode_flag, + } + + @staticmethod + def _control_action_decoders_args(): + return { + "resubmit": nested_list_decoder( + ListDecoders( + [ + ("port", decode_default), + ("table", decode_int), + ("ct", decode_flag), + ] + ) + ), + "push": decode_field, + "pop": decode_field, + "exit": decode_flag, + "multipath": nested_list_decoder( + ListDecoders( + [ + ("fields", decode_default), + ("basis", decode_int), + ("algorithm", decode_default), + ("n_links", decode_int), + ("arg", decode_int), + ("dst", decode_field), + ] + ) + ), + } + + @staticmethod + def _clone_actions_decoders_args(action_decoders): + """Generate the decoder arguments for the clone actions. + + Args: + action_decoders (dict): The decoders of the supported nested + actions. + """ + return { + "learn": decode_learn( + { + **action_decoders, + "fin_timeout": nested_kv_decoder( + KVDecoders( + { + "idle_timeout": decode_time, + "hard_timeout": decode_time, + } + ) + ), + } + ), + "clone": functools.partial( + decode_exec, KVDecoders(action_decoders) + ), + } + + @staticmethod + def _other_action_decoders_args(): + """Generate the decoder arguments for other actions + (see man(7) ovs-actions).""" + return { + "conjunction": nested_list_decoder( + ListDecoders( + [("id", decode_int), ("k", decode_int), ("n", decode_int)] + ), + delims=[",", "/"], + ), + "note": decode_default, + "sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_set_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "sampling_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + } diff --git a/python/ovs/flow/ofp_act.py b/python/ovs/flow/ofp_act.py new file mode 100644 index 000000000..acb16cd9a --- /dev/null +++ b/python/ovs/flow/ofp_act.py @@ -0,0 +1,306 @@ +"""Defines decoders for OpenFlow actions. +""" + +import functools + +from ovs.flow.decoders import ( + decode_default, + decode_time, + decode_flag, + decode_int, +) +from ovs.flow.kv import nested_kv_decoder, KVDecoders, KeyValue, KVParser +from ovs.flow.list import nested_list_decoder, ListDecoders +from ovs.flow.ofp_fields import field_decoders + + +def decode_output(value): + """Decodes the output value. + + Does not support field specification. + """ + if len(value.split(",")) > 1: + return nested_kv_decoder()(value) + try: + return {"port": int(value)} + except ValueError: + return {"port": value.strip('"')} + + +def decode_controller(value): + """Decodes the controller action.""" + if not value: + return KeyValue("output", "controller") + else: + # Try controller:max_len + try: + max_len = int(value) + return { + "max_len": max_len, + } + except ValueError: + pass + # controller(key[=val], ...) + return nested_kv_decoder()(value) + + +def decode_bundle_load(value): + return decode_bundle(value, True) + + +def decode_bundle(value, load=False): + """Decode bundle action.""" + result = {} + keys = ["fields", "basis", "algorithm", "ofport"] + if load: + keys.append("dst") + + for key in keys: + parts = value.partition(",") + nvalue = parts[0] + value = parts[2] + if key == "ofport": + continue + result[key] = decode_default(nvalue) + + # Handle members: + mvalues = value.split("members:") + result["members"] = [int(port) for port in mvalues[1].split(",")] + return result + + +def decode_encap(value): + """Decodes encap action. Examples: + encap(ethernet) + encap(nsh(md_type=2,tlv(0x1000,10,0x12345678))) + + The generated dict has the following keys: "header", "props", e.g: + { + "header": "ethernet", + } + { + "header": "nsh", + "props": { + "md_type": 2, + "tlv": { + "class": 0x100, + "type": 10, + "value": 0x123456 + } + } + } + """ + + def free_hdr_decoder(free_val): + if free_val not in ["ethernet", "mpls", "mpls_mc", "nsh"]: + raise ValueError( + "Malformed encap action. Unkown header: {}".format(free_val) + ) + return "header", free_val + + parser = KVParser( + value, + KVDecoders( + { + "nsh": nested_kv_decoder( + KVDecoders( + { + "md_type": decode_default, + "tlv": nested_list_decoder( + ListDecoders( + [ + ("class", decode_int), + ("type", decode_int), + ("value", decode_int), + ] + ) + ), + } + ) + ), + }, + default_free=free_hdr_decoder, + ), + ) + parser.parse() + if len(parser.kv()) > 1: + raise ValueError("Malformed encap action: {}".format(value)) + + result = {} + if parser.kv()[0].key == "header": + result["header"] = parser.kv()[0].value + elif parser.kv()[0].key == "nsh": + result["header"] = "nsh" + result["props"] = parser.kv()[0].value + + return result + + +def decode_field(value): + """Decodes a field as defined in the 'Field Specification' of the actions + man page: + http://www.openvswitch.org/support/dist-docs/ovs-actions.7.txt.""" + parts = value.strip("]\n\r").split("[") + result = { + "field": parts[0], + } + + if len(parts) > 1 and parts[1]: + field_range = parts[1].split("..") + start = field_range[0] + end = field_range[1] if len(field_range) > 1 else start + if start: + result["start"] = int(start) + if end: + result["end"] = int(end) + + return result + + +def decode_load_field(value): + """Decodes LOAD actions such as: 'load:value->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed load action : %s" % value) + + # If the load action is performed within a learn() action, + # The value can be specified as another field. + try: + return {"value": int(parts[0], 0), "dst": decode_field(parts[1])} + except ValueError: + return {"src": decode_field(parts[0]), "dst": decode_field(parts[1])} + + +def decode_set_field(field_decoders, value): + """Decodes SET_FIELD actions such as: 'set_field:value/mask->dst'. + + The value is decoded by field_decoders which is a KVDecoders instance. + Args: + field_decoders(KVDecoders): The KVDecoders to be used to decode the + field. + """ + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed set_field action : %s" % value) + + val = parts[0] + dst = parts[1] + + val_result = field_decoders.decode(dst, val) + + return { + "value": {val_result[0]: val_result[1]}, + "dst": decode_field(dst), + } + + +def decode_move_field(value): + """Decodes MOVE actions such as 'move:src->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed move action : %s" % value) + + return { + "src": decode_field(parts[0]), + "dst": decode_field(parts[1]), + } + + +def decode_dec_ttl(value): + """Decodes dec_ttl and dec_ttl(id, id[2], ...) actions.""" + if not value: + return True + return [int(idx) for idx in value.split(",")] + + +def decode_chk_pkt_larger(value): + """Decodes 'check_pkt_larger(pkt_len)->dst' actions.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed check_pkt_larger action : %s" % value) + + pkt_len = int(parts[0].strip("()")) + dst = decode_field(parts[1]) + return {"pkt_len": pkt_len, "dst": dst} + + +# CT decoders +def decode_zone(value): + """Decodes the value of the 'zone' keyword (part of the ct action).""" + try: + return int(value, 0) + except ValueError: + pass + return decode_field(value) + + +def decode_exec(action_decoders, value): + """Decodes the value of the 'exec' keyword (part of the ct action). + + Args: + decode_actions (KVDecoders): The decoders to be used to decode the + nested exec. + value (string): The string to be decoded. + """ + exec_parser = KVParser(value, action_decoders) + exec_parser.parse() + return [{kv.key: kv.value} for kv in exec_parser.kv()] + + +def decode_learn(action_decoders): + """Create the decoder to be used to decode the 'learn' action. + + The learn action has two added complexities: + 1) It can hold any valid action key-value. Therefore we must take + the precalculated action_decoders and use them. That's why we require + them as argument. + + 2) The way fields can be specified is augmented. Not only we have + 'field=value', but we also have: + - 'field=_src_' (where _src_ is another field name) + - and just 'field' + For this we need to create a wrapper of field_decoders that, for each + "field=X" key-value we check if X is a field_name or if it's actually + a value that we need to send to the appropriate field_decoder to + process. + + Args: + action_decoders (dict): Dictionary of decoders to be used in nested + action decoding. + """ + + def decode_learn_field(decoder, value): + """Generates a decoder to be used for the 'field' argument of the + 'learn' action. + + The field can hold a value that should be decoded, either as a field, + or as a the value (see man(7) ovs-actions). + + Args: + decoder (callable): The decoder. + """ + if value in field_decoders.keys(): + # It's a field + return value + else: + return decoder(value) + + learn_field_decoders = { + field: functools.partial(decode_learn_field, decoder) + for field, decoder in field_decoders.items() + } + learn_decoders = { + **action_decoders, + **learn_field_decoders, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "priority": decode_int, + "cookie": decode_int, + "send_flow_rem": decode_flag, + "table": decode_int, + "delete_learned": decode_flag, + "limit": decode_int, + "result_dst": decode_field, + } + + return functools.partial(decode_exec, KVDecoders(learn_decoders)) From patchwork Thu Jun 16 06:32:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644146 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=GYdRrWPx; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnn2YRnz9sFw for ; Thu, 16 Jun 2022 16:33:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4EFF384017; Thu, 16 Jun 2022 06:33:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r89eRcVypt-G; Thu, 16 Jun 2022 06:33:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id DAED58404D; Thu, 16 Jun 2022 06:33:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2DF5EC008C; Thu, 16 Jun 2022 06:33:26 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 118ADC0082 for ; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 54D4D41A47 for ; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mmEXfyZOGssw for ; Thu, 16 Jun 2022 06:33:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id AD2A3401DF for ; Thu, 16 Jun 2022 06:33:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361198; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rF0UJZBqSc498hNLsLaIS3v/Au/TLmvsBt/qmSrQ9Hw=; b=GYdRrWPx8G1g9emk4o02m+4OWJZUlVrlsoj07ohHsngFShWXTUDHjCDLf3LiqdgwXytJzl KMKAgw2kRY7QitghgyxOVvYjehL9nkAgK3K4JQe1zNmGc0tCzOX9Fw1iEsnw7HJ2tCfKdY Kp69kmit2ZaABa/llm7tv/Io6JBQdYQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-44-vAUwlTSrOUOTrphXdCQIxQ-1; Thu, 16 Jun 2022 02:33:15 -0400 X-MC-Unique: vAUwlTSrOUOTrphXdCQIxQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F2D91101AA4D; Thu, 16 Jun 2022 06:33:14 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id EC4F5492C3B; Thu, 16 Jun 2022 06:33:13 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:38 +0200 Message-Id: <20220616063247.517147-9-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 08/17] python: add ovs datapath flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A ODPFlow is a Flow with the following sections: ufid info (e.g: bytes, packets, dp, etc) match actions Only three datapath actions require special handling: gre: because it has double parenthesis geneve: because it supports many concatenated lists of options nat: we reuse the decoder used for openflow actions Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flow/odp.py | 783 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 784 insertions(+) create mode 100644 python/ovs/flow/odp.py diff --git a/python/automake.mk b/python/automake.mk index 50cf6b298..644c23764 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,7 @@ ovs_pyfiles = \ python/ovs/flow/flow.py \ python/ovs/flow/kv.py \ python/ovs/flow/list.py \ + python/ovs/flow/odp.py \ python/ovs/flow/ofp.py \ python/ovs/flow/ofp_act.py \ python/ovs/json.py \ diff --git a/python/ovs/flow/odp.py b/python/ovs/flow/odp.py new file mode 100644 index 000000000..87a3bae2f --- /dev/null +++ b/python/ovs/flow/odp.py @@ -0,0 +1,783 @@ +""" Defines an Open vSwitch Datapath Flow. +""" +import re +from functools import partial + +from ovs.flow.flow import Flow, Section + +from ovs.flow.kv import ( + KVParser, + KVDecoders, + nested_kv_decoder, + decode_nested_kv, +) +from ovs.flow.decoders import ( + decode_default, + decode_time, + decode_int, + decode_mask, + Mask8, + Mask16, + Mask32, + Mask64, + Mask128, + IPMask, + EthMask, + decode_free_output, + decode_flag, + decode_nat, +) + + +class ODPFlow(Flow): + """ODPFLow represents a Open vSwitch Datapath flow. + + Attributes: + ufid: The UFID section with only one key-value, with keyword "ufid". + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not ODPFlow._info_decoders: + ODPFlow._info_decoders = ODPFlow._gen_info_decoders() + return ODPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not ODPFlow._match_decoders: + ODPFlow._match_decoders = ODPFlow._gen_match_decoders() + return ODPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not ODPFlow._action_decoders: + ODPFlow._action_decoders = ODPFlow._gen_action_decoders() + return ODPFlow._action_decoders + + def __init__(self, odp_string, id=None): + """Parse a odp flow string. + + The string is expected to have the following format: + [ufid], [match] [flow data] actions:[actions] + + Args: + odp_string (str): A datapath flow string. + + Returns: + A ODPFlow instance. + """ + sections = [] + + # If UFID present, parse it and add it to it's own section. + ufid_pos = odp_string.find("ufid:") + if ufid_pos >= 0: + ufid_string = odp_string[ + ufid_pos : (odp_string[ufid_pos:].find(",") + 1) + ] + ufid_parser = KVParser( + ufid_string, KVDecoders({"ufid": decode_default}) + ) + ufid_parser.parse() + if len(ufid_parser.kv()) != 1: + raise ValueError("malformed odp flow: %s" % odp_string) + sections.append( + Section("ufid", ufid_pos, ufid_string, ufid_parser.kv()) + ) + + action_pos = odp_string.find("actions:") + if action_pos < 0: + raise ValueError("malformed odp flow: %s" % odp_string) + + # rest of the string is between ufid and actions + rest = odp_string[ + (ufid_pos + len(ufid_string) if ufid_pos >= 0 else 0) : action_pos + ] + + action_pos += 8 # len("actions:") + actions = odp_string[action_pos:] + + field_parts = rest.lstrip(" ").partition(" ") + + if len(field_parts) != 3: + raise ValueError("malformed odp flow: %s" % odp_string) + + match = field_parts[0] + info = field_parts[2] + + iparser = KVParser(info, ODPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=odp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, ODPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=odp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, ODPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=action_pos, + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(ODPFlow, self).__init__(sections, odp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "ufid: {}".format(self.ufid) if self.ufid else "" + string += "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + return KVDecoders(ODPFlow._info_decoders_args()) + + @staticmethod + def _info_decoders_args(): + """Generate the decoder args for the info KVDecoders.""" + return { + "packets": decode_int, + "bytes": decode_int, + "used": decode_time, + "flags": decode_default, + "dp": decode_default, + } + + @staticmethod + def _gen_action_decoders(): + """Generate the action KVDecoders.""" + return KVDecoders( + ODPFlow._action_decoders_args(), default_free=decode_free_output + ) + + @staticmethod + def _action_decoders_args(): + """Generate the arguments for the action KVDecoders.""" + _decoders = { + "drop": decode_flag, + "lb_output": decode_int, + "trunc": decode_int, + "recirc": decode_int, + "userspace": nested_kv_decoder( + KVDecoders( + { + "pid": decode_int, + "sFlow": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "output": decode_int, + } + ) + ), + "slow_path": decode_default, + "flow_sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_sed_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "output_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + "ipfix": nested_kv_decoder( + KVDecoders( + { + "output_port": decode_default, + } + ) + ), + "controller": nested_kv_decoder( + KVDecoders( + { + "reason": decode_int, + "dont_send": decode_int, + "continuation": decode_int, + "recirc_id": decode_int, + "rule_cookie": decode_int, + "controller_id": decode_int, + "max_len": decode_int, + } + ) + ), + "userdata": decode_default, + "actions": decode_flag, + "tunnel_out_port": decode_default, + "push_eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "type": decode_int, + } + ) + ), + "pop_eth": decode_flag, + } + ) + ), + "set": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + "push_vlan": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "cfi": decode_int, + "tpid": decode_int, + } + ) + ), + "pop_vlan": decode_flag, + "push_nsh": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "ttl": decode_int, + "mdtype": decode_int, + "np": decode_int, + "spi": decode_int, + "si": decode_int, + "c1": decode_int, + "c2": decode_int, + "c3": decode_int, + "c4": decode_int, + "md2": decode_int, + } + ) + ), + "pop_nsh": decode_flag, + "tnl_pop": decode_int, + "ct_clear": decode_flag, + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "force_commit": decode_flag, + "zone": decode_int, + "mark": Mask32, + "label": Mask128, + "helper": decode_default, + "timeout": decode_default, + "nat": decode_nat, + } + ) + ), + **ODPFlow._tnl_action_decoder_args(), + } + + _decoders["clone"] = nested_kv_decoder( + KVDecoders(decoders=_decoders, default_free=decode_free_output) + ) + + return { + **_decoders, + "sample": nested_kv_decoder( + KVDecoders( + { + "sample": (lambda x: float(x.strip("%"))), + "actions": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + "check_pkt_len": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "gt": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + "le": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + } + + @staticmethod + def _tnl_action_decoder_args(): + """Generate the decoder arguments for the tunnel actions.""" + return { + "tnl_push": nested_kv_decoder( + KVDecoders( + { + "tnl_port": decode_default, + "header": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "type": decode_int, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "dl_type": decode_int, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": decode_int, + "tos": decode_int, + "ttl": decode_int, + "frag": decode_int, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_int, + "proto": decode_int, + "tclass": decode_int, + "hlimit": decode_int, + } + ) + ), + "udp": nested_kv_decoder( + KVDecoders( + { + "src": decode_int, + "dst": decode_int, + "dsum": Mask16, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "vni": decode_int, + } + ) + ), + "geneve": nested_kv_decoder( + KVDecoders( + { + "oam": decode_flag, + "crit": decode_flag, + "vni": decode_int, + "options": partial( + decode_geneve, False + ), + } + ) + ), + "gre": decode_tnl_gre, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": decode_int, + "sid": decode_int, + "idx": decode_int, + "dir": decode_int, + "hwid": decode_int, + } + ) + ), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "msgtype": decode_int, + "teid": decode_int, + } + ) + ), + } + ) + ), + "out_port": decode_default, + } + ) + ) + } + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + return KVDecoders(ODPFlow._match_decoders_args()) + + @staticmethod + def _match_decoders_args(): + """Generate the arguments for the match KVDecoders.""" + return { + **ODPFlow._field_decoders_args(), + "encap": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + } + + @staticmethod + def _field_decoders_args(): + """Generate the decoder arguments for the match fields.""" + return { + "skb_priority": Mask32, + "skb_mark": Mask32, + "recirc_id": decode_int, + "dp_hash": Mask32, + "ct_state": decode_default, + "ct_zone": Mask16, + "ct_mark": Mask32, + "ct_label": Mask128, + "ct_tuple4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "ct_tuple6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "tunnel": nested_kv_decoder( + KVDecoders( + { + "tun_id": Mask64, + "src": IPMask, + "dst": IPMask, + "ipv6_src": IPMask, + "ipv6_dst": IPMask, + "tos": Mask8, + "ttl": Mask8, + "tp_src": Mask16, + "tp_dst": Mask16, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": Mask8, + "idx": Mask32, + "sid": decode_int, + "dir": Mask8, + "hwid": Mask8, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "gbp": nested_kv_decoder( + KVDecoders( + { + "id": Mask16, + "flags": Mask8, + } + ) + ) + } + ) + ), + "geneve": partial(decode_geneve, True), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "msgtype": Mask8, + } + ) + ), + "flags": decode_default, + } + ) + ), + "in_port": decode_default, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + } + ) + ), + "vlan": nested_kv_decoder( + KVDecoders( + { + "vid": Mask16, + "pcp": Mask16, + "cfi": Mask16, + } + ) + ), + "eth_type": Mask16, + "mpls": nested_kv_decoder( + KVDecoders( + { + "label": Mask32, + "tc": Mask32, + "ttl": Mask32, + "bos": Mask32, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tos": Mask8, + "ttl": Mask8, + "frag": decode_default, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_mask(20), + "proto": Mask8, + "tclass": Mask8, + "hlimit": Mask8, + "frag": decode_default, + } + ) + ), + "tcp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "tcp_flags": decode_default, + "udp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "sctp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "icmp": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "icmpv6": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "arp": nested_kv_decoder( + KVDecoders( + { + "sip": IPMask, + "tip": IPMask, + "op": Mask16, + "sha": EthMask, + "tha": EthMask, + } + ) + ), + "nd": nested_kv_decoder( + KVDecoders( + { + "target": IPMask, + "sll": EthMask, + "tll": EthMask, + } + ) + ), + "nd_ext": nested_kv_decoder( + KVDecoders( + { + "nd_reserved": Mask32, + "nd_options_type": Mask8, + } + ) + ), + "packet_type": nested_kv_decoder( + KVDecoders( + { + "ns": Mask16, + "id": Mask16, + } + ) + ), + "nsh": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "mdtype": Mask8, + "np": Mask8, + "spi": Mask32, + "si": Mask8, + "c1": Mask32, + "c2": Mask32, + "c3": Mask32, + "c4": Mask32, + } + ) + ), + } + + +def decode_geneve(mask, value): + """Decode geneve options. + Used for both tnl_push(header(geneve(options()))) action and + tunnel(geneve()) match. + + It has the following format: + + {class=0xffff,type=0x80,len=4,0xa} + + Args: + mask (bool): Whether masking is supported. + value (str): The value to decode. + """ + if mask: + decoders = { + "class": Mask16, + "type": Mask8, + "len": Mask8, + } + + def free_decoder(value): + return "data", Mask128(value) + + else: + decoders = { + "class": decode_int, + "type": decode_int, + "len": decode_int, + } + + def free_decoder(value): + return "data", decode_int(value) + + result = [] + for opts in re.findall(r"{.*?}", value): + result.append( + decode_nested_kv( + KVDecoders(decoders=decoders, default_free=free_decoder), + opts.strip("{}"), + ) + ) + return result + + +def decode_tnl_gre(value): + """ + Decode tnl_push(header(gre())) action. + + It has the following format: + + gre((flags=0x2000,proto=0x6558),key=0x1e241)) + + Args: + value (str): The value to decode. + """ + return decode_nested_kv( + KVDecoders( + { + "flags": decode_int, + "proto": decode_int, + "key": decode_int, + "csum": decode_int, + "seq": decode_int, + } + ), + value.replace("(", "").replace(")", ""), + ) From patchwork Thu Jun 16 06:32:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644145 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=E5/Rc8Wy; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnh47Fzz9sFw for ; Thu, 16 Jun 2022 16:33:44 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7C03941058; Thu, 16 Jun 2022 06:33:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v5WldpnqgD-A; Thu, 16 Jun 2022 06:33:39 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id EC80940FC1; Thu, 16 Jun 2022 06:33:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8D2D4C0083; Thu, 16 Jun 2022 06:33:29 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1A5E4C0084 for ; Thu, 16 Jun 2022 06:33:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 373B183FF8 for ; Thu, 16 Jun 2022 06:33:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CIiBGzd1t19R for ; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 31F538400C for ; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IpAWSUHFIHFKUJxt//er6NdvT8hJS3G4Upgzk+RuRRI=; b=E5/Rc8WyhQchXkQkLDjJcRcf66dWqB2e8rrI7MxEY1rpMtlRXnFbcXXOySaA4+vHLl2gVw bXYXigtw5ESW7bgajreYJBk2tC4bm6sEOGMJCwGAzQ3t4bTTIT9cEsd1a9SVhRB2HyiLhc vGXaLppvkHRZk85aJYen+A2MebaBibo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-241-3gu3z3_gM6K140uL52TIzg-1; Thu, 16 Jun 2022 02:33:16 -0400 X-MC-Unique: 3gu3z3_gM6K140uL52TIzg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 366161C05EA6; Thu, 16 Jun 2022 06:33:16 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 42F02492C3B; Thu, 16 Jun 2022 06:33:15 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:39 +0200 Message-Id: <20220616063247.517147-10-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 09/17] python: add flow filtering syntax X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on pyparsing, create a very simple filtering syntax. It supports basic logic statements (and, &, or, ||, not, !), numerical operations (<, >), equality (=, !=), and masking (~=). The latter is only supported in certain fields (IntMask, EthMask, IPMask). Masking operation is semantically equivalent to "includes", therefore: ip_src ~= 192.168.1.1 means that ip_src field is either a host IP address equal to 192.168.1.1 or an IPMask that includes it (e.g: 192.168.1.1/24). Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flow/filter.py | 261 ++++++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 3 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 python/ovs/flow/filter.py diff --git a/python/automake.mk b/python/automake.mk index 644c23764..20d9486e8 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flow/__init__.py \ python/ovs/flow/decoders.py \ + python/ovs/flow/filter.py \ python/ovs/flow/flow.py \ python/ovs/flow/kv.py \ python/ovs/flow/list.py \ diff --git a/python/ovs/flow/filter.py b/python/ovs/flow/filter.py new file mode 100644 index 000000000..f5ba4eae4 --- /dev/null +++ b/python/ovs/flow/filter.py @@ -0,0 +1,261 @@ +""" Defines a Flow Filtering syntax. +""" +import pyparsing as pp +import netaddr +from functools import reduce +from operator import and_, or_ + +from ovs.flow.decoders import ( + decode_default, + decode_int, + Decoder, + IPMask, + EthMask, +) + + +class EvaluationResult(object): + """An EvaluationResult is the result of an evaluation. It contains the + boolean result and the list of key-values that were evaluated. + + Note that since boolean operations (and, not, or) are based only on + __bool__ we use bitwise alternatives (&, ||, ~). + """ + + def __init__(self, result, *kv): + self.result = result + self.kv = kv if kv else list() + + def __and__(self, other): + """Logical and operation.""" + return EvaluationResult( + self.result and other.result, *self.kv, *other.kv + ) + + def __or__(self, other): + """Logical or operation.""" + return EvaluationResult( + self.result or other.result, *self.kv, *other.kv + ) + + def __invert__(self): + """Logical not operation.""" + return EvaluationResult(not self.result, *self.kv) + + def __bool__(self): + """Boolean operation.""" + return self.result + + def __repr__(self): + return "{} [{}]".format(self.result, self.kv) + + +class ClauseExpression(object): + """ A clause expression represents a specific expression in the filter. + + A clause has the following form: + [field] [operator] [value] + + Valid operators are: + = (equality) + != (inequality) + < (arithmetic less-than) + > (arithmetic more-than) + ~= (__contains__) + + When evaluated, the clause finds what relevant part of the flow to use for + evaluation, tries to translate the clause value to the relevant type and + performs the clause operation. + + Attributes: + field (str): The flow field used in the clause. + operator (str): The flow operator used in the clause. + value (str): The value to perform the comparison against. + """ + operators = {} + type_decoders = { + int: decode_int, + netaddr.IPAddress: IPMask, + netaddr.EUI: EthMask, + bool: bool, + } + + def __init__(self, tokens): + self.field = tokens[0] + self.value = "" + self.operator = "" + + if len(tokens) > 1: + self.operator = tokens[1] + self.value = tokens[2] + + def __repr__(self): + return "{}(field: {}, operator: {}, value: {})".format( + self.__class__.__name__, self.field, self.operator, self.value + ) + + def _find_data_in_kv(self, kv_list): + """Find a KeyValue for evaluation in a list of KeyValue. + + Args: + kv_list (list[KeyValue]): list of KeyValue to look into. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + """ + key_parts = self.field.split(".") + field = key_parts[0] + kvs = [kv for kv in kv_list if kv.key == field] + if not kvs: + return None + + for kv in kvs: + if kv.key == self.field: + # exact match + return (kv, kv.value) + if len(key_parts) > 1: + data = kv.value + for subkey in key_parts[1:]: + try: + data = data.get(subkey) + except Exception: + data = None + break + if not data: + break + if data: + return (kv, data) + return None + + def _find_keyval_to_evaluate(self, flow): + """Finds the key-value and data to use for evaluation on a flow. + + Args: + flow(Flow): The flow where the lookup is performed. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + + """ + for section in flow.sections: + data = self._find_data_in_kv(section.data) + if data: + return data + return None + + def evaluate(self, flow): + """Returns whether the clause is satisfied by the flow. + + Args: + flow (Flow): the flow to evaluate. + """ + result = self._find_keyval_to_evaluate(flow) + + if not result: + return EvaluationResult(False) + + keyval, data = result + + if not self.value and not self.operator: + # just asserting the existance of the key + return EvaluationResult(True, keyval) + + # Decode the value based on the type of data + if isinstance(data, Decoder): + decoder = data.__class__ + else: + decoder = self.type_decoders.get(data.__class__) or decode_default + + decoded_value = decoder(self.value) + + if self.operator == "=": + return EvaluationResult(decoded_value == data, keyval) + elif self.operator == "<": + return EvaluationResult(data < decoded_value, keyval) + elif self.operator == ">": + return EvaluationResult(data > decoded_value, keyval) + elif self.operator == "~=": + return EvaluationResult(decoded_value in data, keyval) + + +class BoolNot(object): + def __init__(self, t): + self.op, self.args = t[0] + + def __repr__(self): + return "NOT({})".format(self.args) + + def evaluate(self, flow): + return ~self.args.evaluate(flow) + + +class BoolAnd(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def __repr__(self): + return "AND({})".format(self.args) + + def evaluate(self, flow): + return reduce(and_, [arg.evaluate(flow) for arg in self.args]) + + +class BoolOr(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def evaluate(self, flow): + return reduce(or_, [arg.evaluate(flow) for arg in self.args]) + + def __repr__(self): + return "OR({})".format(self.args) + + +class OFFilter(object): + """OFFilter represents an Open vSwitch Flow Filter. + + It is built with a filter expression string composed of logically-separated + clauses (see ClauseExpression for details on the clause syntax). + + Args: + expr(str): String filter expression. + """ + w = pp.Word(pp.alphanums + "." + ":" + "_" + "/" + "-") + operators = ( + pp.Literal("=") + | pp.Literal("~=") + | pp.Literal("<") + | pp.Literal(">") + | pp.Literal("!=") + ) + + clause = (w + operators + w) | w + clause.setParseAction(ClauseExpression) + + statement = pp.infixNotation( + clause, + [ + ("!", 1, pp.opAssoc.RIGHT, BoolNot), + ("not", 1, pp.opAssoc.RIGHT, BoolNot), + ("&&", 2, pp.opAssoc.LEFT, BoolAnd), + ("and", 2, pp.opAssoc.LEFT, BoolAnd), + ("||", 2, pp.opAssoc.LEFT, BoolOr), + ("or", 2, pp.opAssoc.LEFT, BoolOr), + ], + ) + + def __init__(self, expr): + self._filter = self.statement.parseString(expr) + + def evaluate(self, flow): + """Evaluate whether the flow satisfies the filter. + + Args: + flow(Flow): a openflow or datapath flow. + + Returns: + An EvaluationResult with the result of the evaluation. + """ + return self._filter[0].evaluate(flow) diff --git a/python/setup.py b/python/setup.py index 350ac6056..a2698fa39 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers', 'netaddr'], + install_requires=['sortedcontainers', 'netaddr', 'pyparsing'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Thu Jun 16 06:32:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644147 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Qd/1xRL2; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnq26zxz9sFw for ; Thu, 16 Jun 2022 16:33:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D536B61171; Thu, 16 Jun 2022 06:33:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mhFcOjx_9NGO; Thu, 16 Jun 2022 06:33:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id D65C36118B; Thu, 16 Jun 2022 06:33:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6EEE8C0032; Thu, 16 Jun 2022 06:33:32 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19607C0085 for ; Thu, 16 Jun 2022 06:33:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6CBB684042 for ; Thu, 16 Jun 2022 06:33:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u67te9mI3KLS for ; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 8476A83FB2 for ; Thu, 16 Jun 2022 06:33:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VNzQCX5noR8LHyiXzXwMD6s+sWgqc5B3YgF8YiAoffI=; b=Qd/1xRL28AOgRvUW582lbyg7KfcOb2W881ro4C3lAmTZBK0msZqBK+B32tMdKbSpQFqXm5 2oIlr0FLrlE4R2NtNI7Av/aH6bVyYosapTAGdyqVux+i0L+hNuvX3ufovhASGqfyFsn4RF Uo1R5hqYgT5gqovEfOtIL1SF/Dz0fHs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-324-jIkNkbnyO5m_gXduSvEi-A-1; Thu, 16 Jun 2022 02:33:18 -0400 X-MC-Unique: jIkNkbnyO5m_gXduSvEi-A-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A4D8E3C0ED4A; Thu, 16 Jun 2022 06:33:17 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9695F492CA5; Thu, 16 Jun 2022 06:33:16 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:40 +0200 Message-Id: <20220616063247.517147-11-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 10/17] python: add a json encoder to flow fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The json encoder can be used to convert Flows to json. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/ovs/flow/decoders.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/python/ovs/flow/decoders.py b/python/ovs/flow/decoders.py index 73d28e057..7378d4176 100644 --- a/python/ovs/flow/decoders.py +++ b/python/ovs/flow/decoders.py @@ -5,6 +5,7 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import json import netaddr import re @@ -522,3 +523,16 @@ def decode_nat(value): result[flag] = True return result + + +class FlowEncoder(json.JSONEncoder): + """FlowEncoder is a json.JSONEncoder instance that can be used to + serialize flow fields.""" + + def default(self, obj): + if isinstance(obj, Decoder): + return obj.to_json() + elif isinstance(obj, netaddr.IPAddress): + return str(obj) + + return json.JSONEncoder.default(self, obj) From patchwork Thu Jun 16 06:32:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644148 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HadvPbns; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnv0txhz9sFw for ; Thu, 16 Jun 2022 16:33:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 17CC3410A1; Thu, 16 Jun 2022 06:33:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I6GnAjE4lL1v; Thu, 16 Jun 2022 06:33:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id BBC1541045; Thu, 16 Jun 2022 06:33:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 795C4C0032; Thu, 16 Jun 2022 06:33:39 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1ADFAC0032 for ; Thu, 16 Jun 2022 06:33:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A603E61152 for ; Thu, 16 Jun 2022 06:33:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ddKoxMsGZiBf for ; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id A905E6113E for ; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mYFT19u8/DjGQoBplOVlJ4s1irypo87U7WaSRWz/vD8=; b=HadvPbnsauCm78rwBAaWSIw3BHUYhhRFgxJBtgism7rsJodeZNVKyen9v2FiaEotvhjHqO HEJ7NgvcnT+6xzWPxTS/5V5RVV1Hq3v6Gqe0dJajBZ/d/oxgcdsnCVOFCKkXUhPHzXtfi+ 6RdJAzJTi9urNp93mtgPsviKD7HsNoE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-364-Mei7GJszOOq3pZ-BeN6lgg-1; Thu, 16 Jun 2022 02:33:19 -0400 X-MC-Unique: Mei7GJszOOq3pZ-BeN6lgg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BEC5D858EED; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id EDD55492C3B; Thu, 16 Jun 2022 06:33:17 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:41 +0200 Message-Id: <20220616063247.517147-12-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 11/17] tests: verify flows in ofp-actions are parseable X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Create a small helper script and check that flows used in ofp-actions.at are parseable. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- tests/automake.mk | 2 ++ tests/ofp-actions.at | 18 +++++++++++++++++ tests/ovs-test-ofparse.py | 42 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+) create mode 100755 tests/ovs-test-ofparse.py diff --git a/tests/automake.mk b/tests/automake.mk index 261fbb942..ab23f04ca 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,6 +19,7 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch @@ -525,6 +526,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ tests/test-jsonrpc.py \ diff --git a/tests/ofp-actions.at b/tests/ofp-actions.at index 9d820eba6..c75f3ccb5 100644 --- a/tests/ofp-actions.at +++ b/tests/ofp-actions.at @@ -329,6 +329,7 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow10 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP AT_SETUP([OpenFlow 1.0 "instruction" translations]) @@ -359,6 +360,7 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-instructions OpenFlow10 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP AT_SETUP([OpenFlow 1.1 action translation]) @@ -502,6 +504,7 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow11 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP AT_SETUP([OpenFlow 1.1 instruction translation]) @@ -737,6 +740,7 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow12 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP dnl Our primary goal here is to verify OpenFlow 1.3-specific changes, @@ -798,6 +802,7 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow13 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP dnl Our primary goal here is to verify that OpenFlow 1.5-specific changes, @@ -827,17 +832,20 @@ AT_CAPTURE_FILE([experr]) AT_CHECK( [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow15 < input.txt], [0], [expout], [experr]) +AT_CHECK([cat expout | grep 'actions=' | ovs-test-ofparse.py]) AT_CLEANUP AT_SETUP([ofp-actions - inconsistent MPLS actions]) OVS_VSWITCHD_START dnl OK: Use fin_timeout action on TCP flow AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=fin_timeout(idle_timeout=1)']) +AT_CHECK([echo 'tcp actions=fin_timeout(idle_timeout=1)' | ovs-test-ofparse.py]) dnl Bad: Use fin_timeout action on TCP flow that has been converted to MPLS AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)'], [1], [], [dnl ovs-ofctl: none of the usable flow formats (OpenFlow10,NXM) is among the allowed flow formats (OpenFlow11) ]) +AT_CHECK([echo 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)' | ovs-test-ofparse.py]) OVS_VSWITCHD_STOP AT_CLEANUP @@ -853,6 +861,8 @@ AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: mpls actions=load:0xa->OXM_OF_MPLS_LABEL[[]] ]) +AT_CHECK([echo 'mpls actions=set_field:10->mpls_label' | ovs-test-ofparse.py]) +AT_CHECK([echo 'mpls actions=load:0xa->OXM_OF_MPLS_LABEL[[]]'| ovs-test-ofparse.py]) OVS_VSWITCHD_STOP AT_CLEANUP @@ -862,14 +872,17 @@ OVS_VSWITCHD_START dnl OpenFlow 1.0 has an "enqueue" action. For OpenFlow 1.1+, we translate dnl it to a series of actions that accomplish the same thing. AT_CHECK([ovs-ofctl -O OpenFlow10 add-flow br0 'actions=enqueue(123,456)']) +AT_CHECK([echo 'actions=enqueue(123,456)' | ovs-test-ofparse.py]) AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: actions=enqueue:123:456 ]) +AT_CHECK([echo 'actions=enqueue:123:456' | ovs-test-ofparse.py]) AT_CHECK([ovs-ofctl -O OpenFlow13 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.3): reset_counts actions=set_queue:456,output:123,pop_queue ]) +AT_CHECK([echo 'actions=set_queue:456,output:123,pop_queue' | ovs-test-ofparse.py]) OVS_VSWITCHD_STOP AT_CLEANUP @@ -887,6 +900,8 @@ AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ttl:123 ]) +AT_CHECK([echo 'ip,actions=mod_nw_ttl:123' | ovs-test-ofparse.py]) +AT_CHECK([echo 'ip actions=load:0x7b->NXM_NX_IP_TTL[[]]' | ovs-test-ofparse.py]) OVS_VSWITCHD_STOP AT_CLEANUP @@ -898,10 +913,12 @@ dnl OpenFlow 1.1, but no other version, has a "mod_nw_ecn" action. dnl Check that we translate it properly for OF1.0 and OF1.2. dnl (OF1.3+ should be the same as OF1.2.) AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ecn:2']) +AT_CHECK([echo 'ip,actions=mod_nw_ecn:2' | ovs-test-ofparse.py]) AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: ip actions=load:0x2->NXM_NX_IP_ECN[[]] ]) +AT_CHECK([echo 'ip actions=load:0x2->NXM_NX_IP_ECN[[]]' | ovs-test-ofparse.py]) AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ecn:2 @@ -910,6 +927,7 @@ AT_CHECK([ovs-ofctl -O OpenFlow12 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.2): ip actions=set_field:2->nw_ecn ]) +AT_CHECK([echo 'ip actions=set_field:2->nw_ecn' | ovs-test-ofparse.py]) dnl Check that OF1.2+ set_field to set ECN is translated into the OF1.1 dnl mod_nw_ecn action. diff --git a/tests/ovs-test-ofparse.py b/tests/ovs-test-ofparse.py new file mode 100755 index 000000000..0a3d3e681 --- /dev/null +++ b/tests/ovs-test-ofparse.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ovs-test-ofparse reads flows from stdin and tries to parse them using +the python flow parsing library. +""" + +import fileinput +import sys + +from ovs.flow.ofp import OFPFlow + + +def main(): + for flow in fileinput.input(): + try: + result_flow = OFPFlow(flow) + if flow != str(result_flow): + print("in: {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed") + except Exception as e: + print(e) + return 1 + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Thu Jun 16 06:32:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644143 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Lwj6j7es; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnc4354z9sFw for ; Thu, 16 Jun 2022 16:33:40 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5B04341B72; Thu, 16 Jun 2022 06:33:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w3NJwJOZN9VM; Thu, 16 Jun 2022 06:33:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 2F38E41A55; Thu, 16 Jun 2022 06:33:29 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 60D31C0086; Thu, 16 Jun 2022 06:33:28 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1105DC0082 for ; Thu, 16 Jun 2022 06:33:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id AAF2E4003A for ; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 18LbIKr8_rga for ; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id C7C6041A3E for ; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361201; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kpfrFPq+W3/AxivnYUbZaBcZVLeevjvwtDgpN6X9duA=; b=Lwj6j7eszkIpAe/XJH0Lj18tO6DRxlKvw6QsYb0zJAdSG1yBCBp69gzimFIwnNTRnLVN7W VaHTcuoJHWt1m2CyCwjC3Cyx3V1O1Q1opHOwijAKV3QdOrpKHBPNcVtlm0HrRocAnF+1w2 93jrhtsk/UPeGogqt289GxwYK/XPFOE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-398-p22AHt_uOIiGPnqTzlj69g-1; Thu, 16 Jun 2022 02:33:20 -0400 X-MC-Unique: p22AHt_uOIiGPnqTzlj69g-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DDFD32999B56; Thu, 16 Jun 2022 06:33:19 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0F6E1492C3B; Thu, 16 Jun 2022 06:33:18 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:42 +0200 Message-Id: <20220616063247.517147-13-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 12/17] tests: verify flows in odp.at are parseable X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Create a small helper script and check that flows tested in odp.at are parseable. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- tests/automake.mk | 2 ++ tests/odp.at | 11 ++++++++++ tests/ovs-test-dpparse.py | 42 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 55 insertions(+) create mode 100755 tests/ovs-test-dpparse.py diff --git a/tests/automake.mk b/tests/automake.mk index ab23f04ca..fcb1857b7 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,6 +19,7 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-dpparse.py \ $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch @@ -526,6 +527,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-dpparse.py \ tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ diff --git a/tests/odp.at b/tests/odp.at index 4d08c59ca..abd22e04a 100644 --- a/tests/odp.at +++ b/tests/odp.at @@ -107,6 +107,7 @@ sed -i'back' 's/\(in_port(1)\),\(eth\)/\1,packet_type(ns=0,id=0),\2/' odp-out.tx AT_CHECK_UNQUOTED([ovstest test-odp parse-keys < odp-in.txt], [0], [`cat odp-out.txt` ]) +AT_CHECK_UNQUOTED([cat odp-out.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CLEANUP AT_SETUP([OVS datapath wildcarded key parsing and formatting - valid forms]) @@ -194,6 +195,7 @@ sed -n 's/,frag=no),.*/,frag=later)/p' odp-base.txt AT_CAPTURE_FILE([odp.txt]) AT_CHECK_UNQUOTED([ovstest test-odp parse-wc-keys < odp.txt], [0], [`cat odp.txt` ]) +AT_CHECK_UNQUOTED([cat odp.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CLEANUP AT_SETUP([OVS datapath wildcarded key filtering.]) @@ -241,24 +243,31 @@ in_port(1),eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15),eth_type(0x86dd),ipv ]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x1235' < odp-base.txt], [0], [`cat odp-eth-type.txt` ]) +AT_CHECK_UNQUOTED([cat odp-eth-type.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) +AT_CHECK_UNQUOTED([cat odp-vlan.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99,ip' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) +AT_CHECK_UNQUOTED([cat odp-ipv4.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x0800,nw_src=35.8.2.199,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='icmp,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-icmp.txt` ]) +AT_CHECK_UNQUOTED([cat odp-icmp.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='arp,arp_spa=1.2.3.5' < odp-base.txt], [0], [`cat odp-arp.txt` ]) +AT_CHECK_UNQUOTED([cat odp-arp.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp.txt` ]) +AT_CHECK_UNQUOTED([cat odp-tcp.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp6,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp6.txt` ]) +AT_CHECK_UNQUOTED([cat odp-tcp6.txt | sed 's/^#.*//' | sed 's/$/ actions:drop/' | ovs-test-dpparse.py]) AT_CLEANUP AT_SETUP([OVS datapath actions parsing and formatting - valid forms]) @@ -389,6 +398,7 @@ add_mpls(label=200,tc=7,ttl=64,bos=1,eth_type=0x8847) AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], [`cat actions.txt` ]) +AT_CHECK_UNQUOTED([cat actions.txt | sed 's/^/actions:/' | ovs-test-dpparse.py]) AT_CLEANUP AT_SETUP([OVS datapath actions parsing and formatting - invalid forms]) @@ -434,6 +444,7 @@ odp_actions_from_string: error `cat actions.txt | head -3 | tail -1` odp_actions_from_string: error ]) +AT_CHECK_UNQUOTED([cat actions.txt | sed 's/^/actions:/' | ovs-test-dpparse.py]) AT_CLEANUP AT_SETUP([OVS datapath actions parsing and formatting - actions too long]) diff --git a/tests/ovs-test-dpparse.py b/tests/ovs-test-dpparse.py new file mode 100755 index 000000000..e9489a3c6 --- /dev/null +++ b/tests/ovs-test-dpparse.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ovs-test-dpparse reads flows from stdin and tries to parse them using +the python flow parsing library. +""" + +import fileinput +import sys + +from ovs.flow.odp import ODPFlow + + +def main(): + for flow in fileinput.input(): + try: + result_flow = ODPFlow(flow) + if flow != str(result_flow): + print("in: {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed") + except Exception as e: + print(e) + return 1 + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Thu Jun 16 06:32:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644149 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=CMYo65p7; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsnz0Pk1z9sFw for ; Thu, 16 Jun 2022 16:33:59 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 582FC41B59; Thu, 16 Jun 2022 06:33:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8nBXTtZd33BR; Thu, 16 Jun 2022 06:33:53 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id D26AE41BAF; Thu, 16 Jun 2022 06:33:43 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3781EC008A; Thu, 16 Jun 2022 06:33:41 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1C471C0084 for ; Thu, 16 Jun 2022 06:33:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2D52640CCE for ; Thu, 16 Jun 2022 06:33:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9JIVAw4WGJZq for ; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id B802E40CD6 for ; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8jN3mNQux4ph1FFjXH2t5Gs1jP4inaaVHlfv/7VKX0Y=; b=CMYo65p7kI8KunQ7uJ6N39JFMYlAg3gGdBAieGXJq4QTT8VldGfT0U9g7s3ZvnGkARaBKl YtM0hXaecpRVl83Xji4Or02eMy4o2lSH8yJtn6Hw7Bn5rCIYWFRGyTjyM/gut3d4iYBxww gNT06ly3ABQotKm24Un53OGB/bDDsN8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-482-OOKKbwGVMxK5xA5fK7kSsw-1; Thu, 16 Jun 2022 02:33:21 -0400 X-MC-Unique: OOKKbwGVMxK5xA5fK7kSsw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0F8E93C0ED4B; Thu, 16 Jun 2022 06:33:21 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2845B492C3B; Thu, 16 Jun 2022 06:33:20 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:43 +0200 Message-Id: <20220616063247.517147-14-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 13/17] python: introduce unit tests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Use pytest to run unit tests as part of the standard testsuite. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- .ci/linux-prepare.sh | 3 + Documentation/intro/install/general.rst | 4 ++ python/automake.mk | 9 ++- python/ovs/tests/test_kv.py | 76 +++++++++++++++++++++++++ python/test_requirements.txt | 3 + tests/atlocal.in | 20 +++++++ tests/automake.mk | 1 + tests/pytest.at | 8 +++ tests/testsuite.at | 1 + 9 files changed, 123 insertions(+), 2 deletions(-) create mode 100644 python/ovs/tests/test_kv.py create mode 100644 python/test_requirements.txt create mode 100644 tests/pytest.at diff --git a/.ci/linux-prepare.sh b/.ci/linux-prepare.sh index 1fe890846..e8b25af15 100755 --- a/.ci/linux-prepare.sh +++ b/.ci/linux-prepare.sh @@ -42,6 +42,9 @@ if [ "$M32" ]; then sudo apt-get install -y $pkgs fi +# Install python test dependencies +pip install -r python/test_requirements.txt + # IPv6 is supported by kernel but disabled in TravisCI images: # https://github.com/travis-ci/travis-ci/issues/8891 # Enable it to avoid skipping of IPv6 related tests. diff --git a/Documentation/intro/install/general.rst b/Documentation/intro/install/general.rst index a297aadac..e2c15a620 100644 --- a/Documentation/intro/install/general.rst +++ b/Documentation/intro/install/general.rst @@ -181,6 +181,10 @@ following to obtain better warnings: come from the "hacking" flake8 plugin. If it's not installed, the warnings just won't occur until it's run on a system with "hacking" installed. +- the python packages listed in "python/test_requirements.txt" (compatible + with pip). If they are installed, the pytest-based Python unit tests will + be run. + You may find the ovs-dev script found in ``utilities/ovs-dev.py`` useful. .. _general-install-reqs: diff --git a/python/automake.mk b/python/automake.mk index 20d9486e8..e8ed984ca 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -52,6 +52,9 @@ ovs_pyfiles = \ python/ovs/vlog.py \ python/ovs/winutils.py +ovs_pytests = \ + python/ovs/tests/test_kv.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ @@ -64,12 +67,14 @@ EXTRA_DIST += \ EXTRA_DIST += \ python/ovs/compat/sortedcontainers/LICENSE \ python/README.rst \ - python/setup.py + python/setup.py \ + python/test_requirements.txt # C extension support. EXTRA_DIST += python/ovs/_json.c -PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) +PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) $(ovs_pytests) + EXTRA_DIST += $(PYFILES) PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) diff --git a/python/ovs/tests/test_kv.py b/python/ovs/tests/test_kv.py new file mode 100644 index 000000000..c5b66de88 --- /dev/null +++ b/python/ovs/tests/test_kv.py @@ -0,0 +1,76 @@ +import pytest + +from ovs.flow.kv import KVParser, KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ( + "cookie=0x0, duration=147566.365s, table=0, n_packets=39, n_bytes=2574, idle_age=65534, hard_age=65534", # noqa: E501 + None, + ), + [ + KeyValue("cookie", 0), + KeyValue("duration", "147566.365s"), + KeyValue("table", 0), + KeyValue("n_packets", 39), + KeyValue("n_bytes", 2574), + KeyValue("idle_age", 65534), + KeyValue("hard_age", 65534), + ], + ), + ( + ( + "load:0x4->NXM_NX_REG13[],load:0x9->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:0a:58:a9:fe:00:02,resubmit(,8)", # noqa: E501 + None, + ), + [ + KeyValue("load", "0x4->NXM_NX_REG13[]"), + KeyValue("load", "0x9->NXM_NX_REG11[]"), + KeyValue("load", "0x8->NXM_NX_REG12[]"), + KeyValue("load", "0x1->OXM_OF_METADATA[]"), + KeyValue("load", "0x1->NXM_NX_REG14[]"), + KeyValue("mod_dl_src", "0a:58:a9:fe:00:02"), + KeyValue("resubmit", ",8"), + ], + ), + ( + ("l1(l2(l3(l4())))", None), + [KeyValue("l1", "l2(l3(l4()))")] + ), + ( + ("l1(l2(l3(l4()))),foo:bar", None), + [KeyValue("l1", "l2(l3(l4()))"), KeyValue("foo", "bar")], + ), + ( + ("enqueue:1:2,output=2", None), + [KeyValue("enqueue", "1:2"), KeyValue("output", 2)], + ), + ( + ("value_to_reg(100)->someReg[10],foo:bar", None), + [ + KeyValue("value_to_reg", "(100)->someReg[10]"), + KeyValue("foo", "bar"), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + tparser = KVParser(input_string, decoders) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr diff --git a/python/test_requirements.txt b/python/test_requirements.txt new file mode 100644 index 000000000..6aaee13e3 --- /dev/null +++ b/python/test_requirements.txt @@ -0,0 +1,3 @@ +pytest +netaddr +pyparsing diff --git a/tests/atlocal.in b/tests/atlocal.in index 142ea2090..e02248f6f 100644 --- a/tests/atlocal.in +++ b/tests/atlocal.in @@ -238,3 +238,23 @@ export ASAN_OPTIONS # for the build. UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=true:log_path=ubsan:$UBSAN_OPTIONS export UBSAN_OPTIONS + +# Check whether Python test requirements are available. +REQUIREMENT_PATH=$abs_top_srcdir/python/test_requirements.txt $PYTHON3 -c ' +import os +import pathlib +import pkg_resources +import sys + +with pathlib.Path(os.path.join(os.getenv("REQUIREMENT_PATH"))).open() as reqs: + for req in pkg_resources.parse_requirements(reqs): + try: + pkg_resources.require(str(req)) + except pkg_resources.DistributionNotFound: + sys.exit(2) +' +case $? in + 0) HAVE_PYTEST=yes ;; + 2) HAVE_PYTEST=no ;; + *) echo "$0: unexpected error probing Python unit test requirements" >&2 ;; +esac diff --git a/tests/automake.mk b/tests/automake.mk index fcb1857b7..ad5bf7247 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -102,6 +102,7 @@ TESTSUITE_AT = \ tests/ovsdb-rbac.at \ tests/ovs-vsctl.at \ tests/ovs-xapi-sync.at \ + tests/pytest.at \ tests/stp.at \ tests/rstp.at \ tests/interface-reconfigure.at \ diff --git a/tests/pytest.at b/tests/pytest.at new file mode 100644 index 000000000..0e75da4c4 --- /dev/null +++ b/tests/pytest.at @@ -0,0 +1,8 @@ +AT_BANNER([Python unit tests]) + +# Run pytest unit tests. +AT_SETUP([Pytest unit tests - Python3]) +AT_KEYWORDS([python]) +AT_SKIP_IF([test "$HAVE_PYTEST" = "no"]) +AT_CHECK([$PYTHON3 -m pytest $top_srcdir/python/ovs],[0], [ignore], [ignore]) +AT_CLEANUP() diff --git a/tests/testsuite.at b/tests/testsuite.at index 58adfa09c..14a28b517 100644 --- a/tests/testsuite.at +++ b/tests/testsuite.at @@ -78,3 +78,4 @@ m4_include([tests/mcast-snooping.at]) m4_include([tests/packet-type-aware.at]) m4_include([tests/nsh.at]) m4_include([tests/drop-stats.at]) +m4_include([tests/pytest.at]) From patchwork Thu Jun 16 06:32:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644151 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YkMtS3uW; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNspV3rplz9sFw for ; Thu, 16 Jun 2022 16:34:26 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 973A041C22; Thu, 16 Jun 2022 06:34:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v859IkKjVezn; Thu, 16 Jun 2022 06:34:22 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id AFA2E41BF0; Thu, 16 Jun 2022 06:34:13 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0CEF2C0083; Thu, 16 Jun 2022 06:34:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 26C29C0081 for ; Thu, 16 Jun 2022 06:34:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id CC55841039 for ; Thu, 16 Jun 2022 06:33:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VnhRiO-Km50d for ; Thu, 16 Jun 2022 06:33:32 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 4004440FE9 for ; Thu, 16 Jun 2022 06:33:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361207; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n3WIWXOEnQp0CkaTSdOg7lsMDz95yIPjxIYoPrlKNsk=; b=YkMtS3uWgJO7vdVsX9wZp/DnErjI51bHphQMu1qKvaEjB961ACuCMCfErFR6ioKuK15VmE tf+NWvorH4OiHccsnk77CJRGARE4hyDJxu/ZJOgb/K6l/QTq44NBDGpwfuJqmlb0XhBmAV pNq62KUK06B3ZeCRQ30wInOBwepvvdA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-224-J3hLU1ELPOO-XTJ7ylBGVQ-1; Thu, 16 Jun 2022 02:33:22 -0400 X-MC-Unique: J3hLU1ELPOO-XTJ7ylBGVQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1F8D03802B95; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E895492C3B; Thu, 16 Jun 2022 06:33:21 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:44 +0200 Message-Id: <20220616063247.517147-15-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 14/17] python: add unit tests for ListParser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for ListParser class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 3 +- python/ovs/tests/test_list.py | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 python/ovs/tests/test_list.py diff --git a/python/automake.mk b/python/automake.mk index e8ed984ca..4e41f6934 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -53,7 +53,8 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ - python/ovs/tests/test_kv.py + python/ovs/tests/test_kv.py \ + python/ovs/tests/test_list.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_list.py b/python/ovs/tests/test_list.py new file mode 100644 index 000000000..e5139869f --- /dev/null +++ b/python/ovs/tests/test_list.py @@ -0,0 +1,66 @@ +import pytest + +from ovs.flows.list import ListParser, ListDecoders +from ovs.flows.kv import KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ("field1,field2,3,nested:value", None, [","]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + KeyValue("elem_3", "nested:value"), + ], + ), + ( + ( + "field1,field2,3,nested:value", + ListDecoders( + [ + ("key1", str), + ("key2", str), + ("key3", int), + ("key4", lambda x: x.split(":"), [","]), + ] + ), + [","], + ), + [ + KeyValue("key1", "field1"), + KeyValue("key2", "field2"), + KeyValue("key3", 3), + KeyValue("key4", ["nested", "value"]), + ], + ), + ( + ("field1:field2:3", None, [":"]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + delims = input_data[2] + tparser = ListParser(input_string, decoders, delims) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr From patchwork Thu Jun 16 06:32:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644150 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WONlKzcs; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNspP6VHsz9sFw for ; Thu, 16 Jun 2022 16:34:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 099FA611CE; Thu, 16 Jun 2022 06:34:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CKk8Fl08YD0y; Thu, 16 Jun 2022 06:34:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id C499C6121C; Thu, 16 Jun 2022 06:34:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8370CC0032; Thu, 16 Jun 2022 06:34:01 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 17CB1C0032 for ; Thu, 16 Jun 2022 06:34:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 9FA7184047 for ; Thu, 16 Jun 2022 06:33:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KMFmdRLfk-NO for ; Thu, 16 Jun 2022 06:33:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id C705F84059 for ; Thu, 16 Jun 2022 06:33:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LeWAZwzEcBh3Q32ktrcRs1W2seefL8udxVv9pLk1fys=; b=WONlKzcsOwcdmiUG/wDm2rBy84yK1w4inGUwJjMT+EPG6e2PQ3HamQIqMMQNJhE/PH8SJ7 abazl142MznM/mxuCmgDNvIbg+1ygf/jUtOP7gXMRnO5uVN+9fl1oEwCHYrgb6oAo5Yhve 4fgqr6FnGbPuoTwhU2zAxNlF+xO7TYg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-582-pC6kUJ4NOlS_U2KuF-gE9w-1; Thu, 16 Jun 2022 02:33:23 -0400 X-MC-Unique: pC6kUJ4NOlS_U2KuF-gE9w-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2F9623802B97; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5EBED492C3B; Thu, 16 Jun 2022 06:33:22 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:45 +0200 Message-Id: <20220616063247.517147-16-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 15/17] python: add unit tests for openflow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for OFPFlow class and ip-port range decoder Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 4 +- python/ovs/tests/test_decoders.py | 130 ++++++++ python/ovs/tests/test_list.py | 4 +- python/ovs/tests/test_ofp.py | 534 ++++++++++++++++++++++++++++++ 4 files changed, 669 insertions(+), 3 deletions(-) create mode 100644 python/ovs/tests/test_decoders.py create mode 100644 python/ovs/tests/test_ofp.py diff --git a/python/automake.mk b/python/automake.mk index 4e41f6934..d5955c240 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -53,8 +53,10 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ + python/ovs/tests/test_decoders.py \ python/ovs/tests/test_kv.py \ - python/ovs/tests/test_list.py + python/ovs/tests/test_list.py \ + python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_decoders.py b/python/ovs/tests/test_decoders.py new file mode 100644 index 000000000..cf621d131 --- /dev/null +++ b/python/ovs/tests/test_decoders.py @@ -0,0 +1,130 @@ +from netaddr import IPAddress +import pytest + +from ovs.flow.decoders import decode_ip_port_range + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "192.168.0.0-192.168.0.200:1000-2000", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + }, + ), + ( + "192.168.0.0-192.168.0.200", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + }, + ), + ( + "192.168.0.0-192.168.0.200:2000", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + "ports": { + "start": 2000, + "end": 2000, + }, + }, + ), + ( + "192.168.0.1:1000-2000", + { + "addrs": { + "start": IPAddress("192.168.0.1"), + "end": IPAddress("192.168.0.1"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + }, + ), + ( + "[fe80:0000:0000:0000:0204:61ff:fe9d:f150]-[fe80:0000:0000:0000:0204:61ff:fe9d:f15f]:255", # noqa: E501 + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f15f" + ), + }, + "ports": { + "start": 255, + "end": 255, + }, + }, + ), + ( + "[fe80::204:61ff:254.157.241.86]-[fe80::204:61ff:254.157.241.100]:255-300", # noqa: E501 + { + "addrs": { + "start": IPAddress("fe80::204:61ff:254.157.241.86"), + "end": IPAddress("fe80::204:61ff:254.157.241.100"), + }, + "ports": { + "start": 255, + "end": 300, + }, + }, + ), + ( + "[fe80::f150]-[fe80::f15f]:255-300", + { + "addrs": { + "start": IPAddress("fe80::f150"), + "end": IPAddress("fe80::f15f"), + }, + "ports": { + "start": 255, + "end": 300, + }, + }, + ), + ( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150-fe80:0000:0000:0000:0204:61ff:fe9d:f15f", # noqa: E501 + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f15f" + ), + }, + }, + ), + ( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156", + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156" + ), + }, + }, + ), + ], +) +def test_decode_ip_port_range(input_string, expected): + assert expected == decode_ip_port_range(input_string) diff --git a/python/ovs/tests/test_list.py b/python/ovs/tests/test_list.py index e5139869f..e08ee1e85 100644 --- a/python/ovs/tests/test_list.py +++ b/python/ovs/tests/test_list.py @@ -1,7 +1,7 @@ import pytest -from ovs.flows.list import ListParser, ListDecoders -from ovs.flows.kv import KeyValue +from ovs.flow.list import ListParser, ListDecoders +from ovs.flow.kv import KeyValue @pytest.mark.parametrize( diff --git a/python/ovs/tests/test_ofp.py b/python/ovs/tests/test_ofp.py new file mode 100644 index 000000000..7a93b2fd4 --- /dev/null +++ b/python/ovs/tests/test_ofp.py @@ -0,0 +1,534 @@ +import netaddr +import pytest + +from ovs.flow.ofp import OFPFlow +from ovs.flow.kv import KeyValue +from ovs.flow.decoders import EthMask, IPMask, decode_mask + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions=local,3,4,5,output:foo", + [ + KeyValue("output", {"port": "local"}), + KeyValue("output", {"port": 3}), + KeyValue("output", {"port": 4}), + KeyValue("output", {"port": 5}), + KeyValue("output", {"port": "foo"}), + ], + ), + ( + "actions=controller,controller:200", + [ + KeyValue("output", "controller"), + KeyValue("controller", {"max_len": 200}), + ], + ), + ( + "actions=enqueue(foo,42),enqueue:foo:42,enqueue(bar,4242)", + [ + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "bar", "queue": 4242}), + ], + ), + ( + "actions=bundle(eth_src,0,hrw,ofport,members:4,8)", + [ + KeyValue( + "bundle", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=bundle_load(eth_src,0,hrw,ofport,reg0,members:4,8)", + [ + KeyValue( + "bundle_load", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "dst": "reg0", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=group:3", + [KeyValue("group", 3)], + ), + ( + "actions=strip_vlan", + [KeyValue("strip_vlan", True)], + ), + ( + "actions=pop_vlan", + [KeyValue("pop_vlan", True)], + ), + ( + "actions=push_vlan:0x8100", + [KeyValue("push_vlan", 0x8100)], + ), + ( + "actions=push_mpls:0x8848", + [KeyValue("push_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=encap(nsh(md_type=2,tlv(0x1000,10,0x12345678)))", + [ + KeyValue( + "encap", + { + "header": "nsh", + "props": { + "md_type": 2, + "tlv": { + "class": 0x1000, + "type": 10, + "value": 0x12345678, + }, + }, + }, + ) + ], + ), + ( + "actions=encap(ethernet)", + [ + KeyValue( + "encap", + {"header": "ethernet"}, + ) + ], + ), + ( + "actions=encap(mpls)", + [ + KeyValue( + "encap", + {"header": "mpls"}, + ) + ], + ), + ( + "actions=load:0x001122334455->eth_src", + [ + KeyValue( + "load", + {"value": 0x001122334455, "dst": {"field": "eth_src"}}, + ) + ], + ), + ( + "actions=load:1->eth_src[1]", + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "eth_src", "start": 1, "end": 1}, + }, + ) + ], + ), + ( + "actions=learn(load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[])", + [ + KeyValue( + "learn", + [ + { + "load": { + "src": {"field": "NXM_NX_TUN_ID"}, + "dst": {"field": "NXM_NX_TUN_ID"}, + } + } + ], + ), + ], + ), + ( + "actions=set_field:00:11:22:33:44:55->eth_src", + [ + KeyValue( + "set_field", + { + "value": {"eth_src": EthMask("00:11:22:33:44:55")}, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:01:00:00:00:00:00/01:00:00:00:00:00->eth_src", + [ + KeyValue( + "set_field", + { + "value": { + "eth_src": EthMask( + "01:00:00:00:00:00/01:00:00:00:00:00" + ) + }, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:0x10ff->vlan_vid", + [ + KeyValue( + "set_field", + { + "value": {"vlan_vid": decode_mask(13)("0x10ff")}, + "dst": {"field": "vlan_vid"}, + }, + ) + ], + ), + ( + "actions=move:reg0[0..5]->reg1[16..31]", + [ + KeyValue( + "move", + { + "src": {"field": "reg0", "start": 0, "end": 5}, + "dst": {"field": "reg1", "start": 16, "end": 31}, + }, + ) + ], + ), + ( + "actions=mod_dl_dst:00:11:22:33:44:55", + [KeyValue("mod_dl_dst", EthMask("00:11:22:33:44:55"))], + ), + ( + "actions=mod_nw_dst:192.168.1.1", + [KeyValue("mod_nw_dst", IPMask("192.168.1.1"))], + ), + ( + "actions=mod_nw_dst:fe80::ec17:7bff:fe61:7aac", + [KeyValue("mod_nw_dst", IPMask("fe80::ec17:7bff:fe61:7aac"))], + ), + ( + "actions=dec_ttl,dec_ttl(1,2,3)", + [KeyValue("dec_ttl", True), KeyValue("dec_ttl", [1, 2, 3])], + ), + ( + "actions=set_mpls_label:0x100,set_mpls_tc:2,set_mpls_ttl:10", + [ + KeyValue("set_mpls_label", 0x100), + KeyValue("set_mpls_tc", 2), + KeyValue("set_mpls_ttl", 10), + ], + ), + ( + "actions=check_pkt_larger(100)->reg0[10]", + [ + KeyValue( + "check_pkt_larger", + { + "pkt_len": 100, + "dst": {"field": "reg0", "start": 10, "end": 10}, + }, + ), + ], + ), + ( + "actions=pop_queue,set_tunnel:0x10,set_tunnel64:0x65000,set_queue=3", # noqa: E501 + [ + KeyValue("pop_queue", True), + KeyValue("set_tunnel", 0x10), + KeyValue("set_tunnel64", 0x65000), + KeyValue("set_queue", 3), + ], + ), + ( + "actions=ct(zone=10,table=2,nat(snat=192.168.0.0-192.168.0.200:1000-2000,random))", # noqa: E501 + [ + KeyValue( + "ct", + { + "zone": 10, + "table": 2, + "nat": { + "type": "snat", + "addrs": { + "start": netaddr.IPAddress("192.168.0.0"), + "end": netaddr.IPAddress("192.168.0.200"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + "random": True, + }, + }, + ) + ], + ), + ( + "actions=ct(commit,zone=NXM_NX_REG13[0..15],table=2,exec(load:0->NXM_NX_CT_LABEL[0]))", # noqa: E501 + [ + KeyValue( + "ct", + { + "commit": True, + "zone": { + "field": "NXM_NX_REG13", + "start": 0, + "end": 15, + }, + "table": 2, + "exec": [ + { + "load": { + "value": 0, + "dst": { + "field": "NXM_NX_CT_LABEL", + "start": 0, + "end": 0, + }, + }, + }, + ], + }, + ) + ], + ), + ( + "actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xda6f52b0,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=172.30.204.105,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])", # noqa: E501 + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "NXM_NX_REG10", "start": 7, "end": 7}, + }, + ), + KeyValue( + "learn", + [ + {"table": 69}, + {"delete_learned": True}, + {"cookie": 3664728752}, + {"OXM_OF_METADATA[]": True}, + {"eth_type": 2048}, + {"NXM_OF_IP_SRC[]": True}, + {"ip_dst": IPMask("172.30.204.105/32")}, + {"nw_proto": 6}, + {"NXM_OF_TCP_SRC[]": "NXM_OF_TCP_DST[]"}, + { + "load": { + "value": 1, + "dst": { + "field": "NXM_NX_REG10", + "start": 7, + "end": 7, + }, + } + }, + ], + ), + ], + ), + ( + "actions=resubmit(,8),resubmit:3,resubmit(1,2,ct)", + [ + KeyValue("resubmit", {"port": "", "table": 8}), + KeyValue("resubmit", {"port": 3}), + KeyValue("resubmit", {"port": 1, "table": 2, "ct": True}), + ], + ), + ( + "actions=clone(ct_clear,load:0->NXM_NX_REG11[],load:0->NXM_NX_REG12[],load:0->NXM_NX_REG13[],load:0x1d->NXM_NX_REG13[],load:0x1f->NXM_NX_REG11[],load:0x1c->NXM_NX_REG12[],load:0x11->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0->NXM_NX_REG10[],load:0->NXM_NX_REG15[],load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],load:0->NXM_NX_REG5[],load:0->NXM_NX_REG6[],load:0->NXM_NX_REG7[],load:0->NXM_NX_REG8[],load:0->NXM_NX_REG9[],resubmit(,8))", # noqa: E501 + [ + KeyValue( + "clone", + [ + {"ct_clear": True}, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 29, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 31, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 28, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 17, + "dst": {"field": "OXM_OF_METADATA"}, + } + }, + { + "load": { + "value": 2, + "dst": {"field": "NXM_NX_REG14"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG10"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG15"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG0"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG1"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG2"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG3"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG4"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG5"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG6"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG7"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG8"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG9"}, + } + }, + {"resubmit": {"port": "", "table": 8}}, + ], + ) + ], + ), + ( + "actions=conjunction(1234, 1/2),note:00.00.11.22.33.ff,sample(probability=123,collector_set_id=0x123,obs_domain_id=0x123,obs_point_id=0x123,sampling_port=inport0,ingress)", # noqa: E501 + [ + KeyValue("conjunction", {"id": 1234, "k": 1, "n": 2}), + KeyValue("note", "00.00.11.22.33.ff"), + KeyValue( + "sample", + { + "probability": 123, + "collector_set_id": 0x123, + "obs_domain_id": 0x123, + "obs_point_id": 0x123, + "sampling_port": "inport0", + "ingress": True, + }, + ), + ], + ), + ], +) +def test_act(input_string, expected): + ofp = OFPFlow(input_string) + actions = ofp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = ofp.section("actions").pos + astring = ofp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Thu Jun 16 06:32:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644153 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aHvoNgSY; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNsph68D4z9sFw for ; Thu, 16 Jun 2022 16:34:36 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1322D61233; Thu, 16 Jun 2022 06:34:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pe6wuwhaqIfi; Thu, 16 Jun 2022 06:34:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 74A03611E4; Thu, 16 Jun 2022 06:34:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3862EC008B; Thu, 16 Jun 2022 06:34:14 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1559DC008B for ; Thu, 16 Jun 2022 06:34:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2755E4103C for ; Thu, 16 Jun 2022 06:33:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b5-KlGWPaMUu for ; Thu, 16 Jun 2022 06:33:33 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 09B8D40FEE for ; Thu, 16 Jun 2022 06:33:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361207; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=niPlFF2ycRkXPS8Wd4PplC5S2aYsUP7jVGCssXc4y/o=; b=aHvoNgSY/pgDF0RtkHsVMcWj5H3uOQx7SXcgyXMPxthisf3pcvyUzrPoGJq0Md9SmHfgf8 G/qupPyyvEg+d0EHarRHNYIapRyQ7gsprX7So6X1MLVVTq1ObyGm1h7kLkpy0bGyxChvZW GnsZStPePXwhUvCS6im9L2GyRU7J7Jc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-548-zoGaQ2YRNteC6Dv8VHL_sg-1; Thu, 16 Jun 2022 02:33:24 -0400 X-MC-Unique: zoGaQ2YRNteC6Dv8VHL_sg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7E920811E75; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8CA2A492C3B; Thu, 16 Jun 2022 06:33:23 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:46 +0200 Message-Id: <20220616063247.517147-17-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 16/17] python: add unit tests to datapath parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests to datapath flow parsing. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/tests/test_odp.py | 527 +++++++++++++++++++++++++++++++++++ 2 files changed, 528 insertions(+) create mode 100644 python/ovs/tests/test_odp.py diff --git a/python/automake.mk b/python/automake.mk index d5955c240..fe41b653d 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -56,6 +56,7 @@ ovs_pytests = \ python/ovs/tests/test_decoders.py \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ + python/ovs/tests/test_odp.py \ python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, diff --git a/python/ovs/tests/test_odp.py b/python/ovs/tests/test_odp.py new file mode 100644 index 000000000..715be3869 --- /dev/null +++ b/python/ovs/tests/test_odp.py @@ -0,0 +1,527 @@ +import netaddr +import pytest + +from ovs.flow.odp import ODPFlow +from ovs.flow.kv import KeyValue +from ovs.flow.decoders import ( + EthMask, + IPMask, + Mask32, + Mask16, + Mask8, + Mask128, +) + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "skb_priority(0x123),skb_mark(0x123),recirc_id(0x123),dp_hash(0x123),ct_zone(0x123), actions:", # noqa: E501 + [ + KeyValue("skb_priority", Mask32("0x123")), + KeyValue("skb_mark", Mask32("0x123")), + KeyValue("recirc_id", 0x123), + KeyValue("dp_hash", Mask32("0x123")), + KeyValue("ct_zone", Mask16("0x123")), + ], + ), + ( + "tunnel(tun_id=0x7f10354,src=10.10.10.10,dst=20.20.20.20,ttl=64,flags(csum|key)) actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "tun_id": 0x7F10354, + "src": IPMask("10.10.10.10"), + "dst": IPMask("20.20.20.20"), + "ttl": 64, + "flags": "csum|key", + }, + ) + ], + ), + ( + "tunnel(geneve({class=0,type=0,len=4,0xa/0xff}),vxlan(flags=0x800000,vni=0x1c7),erspan(ver=2,dir=1,hwid=0x1)), actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "geneve": [ + { + "class": Mask16("0"), + "type": Mask8("0"), + "len": Mask8("4"), + "data": Mask128("0xa/0xff"), + } + ], + "vxlan": {"flags": 0x800000, "vni": 0x1C7}, + "erspan": {"ver": 2, "dir": 1, "hwid": 0x1}, + }, + ) + ], + ), + ( + "in_port(2),eth(src=11:22:33:44:55:66,dst=66:55:44:33:22:11) actions:", # noqa: E501 + [ + KeyValue("in_port", 2), + KeyValue( + "eth", + { + "src": EthMask("11:22:33:44:55:66"), + "dst": EthMask("66:55:44:33:22:11"), + }, + ), + ], + ), + ( + "eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0) actions:", # noqa: E501 + [ + KeyValue("eth_type", Mask16("0x800/0x006")), + KeyValue( + "ipv4", + { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xFF"), + "tos": Mask8("0x2/0xF0"), + }, + ), + ], + ), + ( + "encap(eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0)) actions:", # noqa: E501 + [ + KeyValue( + "encap", + { + "eth_type": Mask16("0x800/0x006"), + "ipv4": { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xff"), + "tos": Mask8("0x2/0xf0"), + }, + }, + ), + ], + ), + ], +) +def test_odp_fields(input_string, expected): + odp = ODPFlow(input_string) + match = odp.match_kv + for i in range(len(expected)): + assert expected[i].key == match[i].key + assert expected[i].value == match[i].value + + # Assert positions relative to action string are OK. + mpos = odp.section("match").pos + mstring = odp.section("match").string + + kpos = match[i].meta.kpos + kstr = match[i].meta.kstring + vpos = match[i].meta.vpos + vstr = match[i].meta.vstring + assert mstring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert mstring[vpos : vpos + len(vstr)] == vstr + + # Assert mstring meta is correct. + assert input_string[mpos : mpos + len(mstring)] == mstring + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions:ct" + ",ct(commit)" + ",ct(commit,zone=5)" + ",ct(commit,mark=0xa0a0a0a0/0xfefefefe)" + ",ct(commit,label=0x1234567890abcdef1234567890abcdef/0xf1f2f3f4f5f6f7f8f9f0fafbfcfdfeff)" # noqa: E501 + ",ct(commit,helper=ftp)" + ",ct(commit,helper=tftp)" + ",ct(commit,timeout=ovs_tp_1_tcp4)" + ",ct(nat)", + [ + KeyValue("ct", True), + KeyValue("ct", {"commit": True}), + KeyValue("ct", {"commit": True, "zone": 5}), + KeyValue( + "ct", + {"commit": True, "mark": Mask32("0xA0A0A0A0/0xFEFEFEFE")}, + ), + KeyValue( + "ct", + { + "commit": True, + "label": Mask128( + "0x1234567890ABCDEF1234567890ABCDEF/0xF1F2F3F4F5F6F7F8F9F0FAFBFCFDFEFF" # noqa: E501 + ), + }, + ), + KeyValue("ct", {"commit": True, "helper": "ftp"}), + KeyValue("ct", {"commit": True, "helper": "tftp"}), + KeyValue("ct", {"commit": True, "timeout": "ovs_tp_1_tcp4"}), + KeyValue("ct", {"nat": True}), + ], + ), + ( + "actions:ct(nat)" + ",ct(commit,nat(src))" + ",ct(commit,nat(dst))" + ",ct(commit,nat(src=10.0.0.240,random))" + ",ct(commit,nat(src=10.0.0.240:32768-65535,random))" + ",ct(commit,nat(dst=10.0.0.128-10.0.0.254,hash))" + ",ct(commit,nat(src=10.0.0.240-10.0.0.254:32768-65535,persistent))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:a18b,random))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:1-fe80::20c:29ff:fe88:a18b,random))" # noqa: E501 + ",ct(commit,nat(src=[[fe80::20c:29ff:fe88:1]]-[[fe80::20c:29ff:fe88:a18b]]:255-4096,random))" # noqa: E501 + ",ct(commit,helper=ftp,nat(src=10.1.1.240-10.1.1.255))" + ",ct(force_commit)", + [ + KeyValue("ct", {"nat": True}), + KeyValue("ct", {"commit": True, "nat": {"type": "src"}}), + KeyValue("ct", {"commit": True, "nat": {"type": "dst"}}), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "dst", + "addrs": { + "start": netaddr.IPAddress("10.0.0.128"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "hash": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "persistent": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "ports": { + "start": 255, + "end": 4096, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.1.1.240"), + "end": netaddr.IPAddress("10.1.1.255"), + }, + }, + "helper": "ftp", + }, + ), + KeyValue("ct", {"force_commit": True}), + ], + ), + ( + "actions:set(tunnel(tun_id=0xabcdef1234567890,src=1.1.1.1,dst=2.2.2.2,ttl=64,flags(df|csum|key)))" # noqa: E501 + ",tnl_pop(4)" + ",tnl_push(tnl_port(6),header(size=50,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x0800),ipv4(src=1.1.2.88,dst=1.1.2.92,proto=17,tos=0,ttl=64,frag=0x4000),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))" # noqa: E501 + ",tnl_push(tnl_port(6),header(size=70,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x86dd),ipv6(src=2001:cafe::88,dst=2001:cafe::92,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))", # noqa: E501 + [ + KeyValue( + "set", + { + "tunnel": { + "tun_id": 0xABCDEF1234567890, + "src": IPMask("1.1.1.1"), + "dst": IPMask("2.2.2.2"), + "ttl": 64, + "flags": "df|csum|key", + } + }, + ), + KeyValue("tnl_pop", 4), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 50, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x800, + }, + "ipv4": { + "src": IPMask("1.1.2.88"), + "dst": IPMask("1.1.2.92"), + "proto": 17, + "tos": 0, + "ttl": 64, + "frag": 0x4000, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 70, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x86DD, + }, + "ipv6": { + "src": IPMask("2001:cafe::88"), + "dst": IPMask("2001:cafe::92"), + "label": 0, + "proto": 17, + "tclass": 0x0, + "hlimit": 64, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + ], + ), + ( + "actions:tnl_push(header(geneve(oam,vni=0x1c7)))" + ",tnl_push(header(geneve(crit,vni=0x1c7,options({class=0xffff,type=0x80,len=4,0xa}))))" # noqa: E501 + ",tnl_push(header(gre((flags=0xa000,proto=0x6558),csum=0x0,key=0x1e241)))", # noqa: E501 + [ + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "oam": True, + "vni": 0x1C7, + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "crit": True, + "vni": 0x1C7, + "options": [ + { + "class": 0xFFFF, + "type": 0x80, + "len": 4, + "data": 0xA, + } + ], + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "gre": { + "flags": 0xA000, + "proto": 0x6558, + "key": 0x1E241, + "csum": 0x0, + } + } + }, + ), + ], + ), + ( + "actions:clone(1)" ",clone(clone(push_vlan(vid=12,pcp=0),2),1)", + [ + KeyValue("clone", {"output": {"port": 1}}), + KeyValue( + "clone", + { + "output": {"port": 1}, + "clone": { + "push_vlan": { + "vid": 12, + "pcp": 0, + }, + "output": {"port": 2}, + }, + }, + ), + ], + ), + ( + "actions: check_pkt_len(size=200,gt(4),le(5))" + ",check_pkt_len(size=200,gt(drop),le(5))" + ",check_pkt_len(size=200,gt(ct(nat)),le(drop))", + [ + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"output": {"port": 4}}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"drop": True}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"ct": {"nat": True}}, + "le": {"drop": True}, + }, + ), + ], + ), + ], +) +def test_odp_actions(input_string, expected): + odp = ODPFlow(input_string) + actions = odp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = odp.section("actions").pos + astring = odp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Thu Jun 16 06:32:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1644152 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=BbDC0eI5; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LNspg2xJnz9sFw for ; Thu, 16 Jun 2022 16:34:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2E21140FF7; Thu, 16 Jun 2022 06:34:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rHrCOm5lr9yM; Thu, 16 Jun 2022 06:34:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 4706E4103D; Thu, 16 Jun 2022 06:34:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03FFDC0032; Thu, 16 Jun 2022 06:34:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1D8B9C007A for ; Thu, 16 Jun 2022 06:34:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0C361611C8 for ; Thu, 16 Jun 2022 06:33:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T3j_974dIznZ for ; Thu, 16 Jun 2022 06:33:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id DB90D61179 for ; Thu, 16 Jun 2022 06:33:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655361209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aZioSoP5YWxBrzHFMWhFz9N/T6P1WB0XJIwyThxxZzU=; b=BbDC0eI5IKlOr6ZsdFhYmZcNcYiANNGwYK/tRjNRrFYXNa+uXJD7rABpeQkbi4HsQkAMuk nPIeU1WaBxm+HpGJP029kwYq2ZLxKxr4AGrAoCkbWDPlGQAeerhND8HMGgoPB+uCw2n1ig BCWvqi6duW09CvETORGnr7oy1A7Bvpo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-81-JhgnkueGON6nRggevbOPlQ-1; Thu, 16 Jun 2022 02:33:26 -0400 X-MC-Unique: JhgnkueGON6nRggevbOPlQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 219D81C05EAF; Thu, 16 Jun 2022 06:33:26 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.17]) by smtp.corp.redhat.com (Postfix) with ESMTP id CC775492C3B; Thu, 16 Jun 2022 06:33:24 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 16 Jun 2022 08:32:47 +0200 Message-Id: <20220616063247.517147-18-amorenoz@redhat.com> In-Reply-To: <20220616063247.517147-1-amorenoz@redhat.com> References: <20220616063247.517147-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v4 17/17] python: add unit tests for filtering engine X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit test for OFFilter class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/tests/test_filter.py | 221 ++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 python/ovs/tests/test_filter.py diff --git a/python/automake.mk b/python/automake.mk index fe41b653d..6a0561cbb 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -54,6 +54,7 @@ ovs_pyfiles = \ ovs_pytests = \ python/ovs/tests/test_decoders.py \ + python/ovs/tests/test_filter.py \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ python/ovs/tests/test_odp.py \ diff --git a/python/ovs/tests/test_filter.py b/python/ovs/tests/test_filter.py new file mode 100644 index 000000000..47d9d9420 --- /dev/null +++ b/python/ovs/tests/test_filter.py @@ -0,0 +1,221 @@ +import pytest + +from ovs.flow.filter import OFFilter +from ovs.flow.ofp import OFPFlow +from ovs.flow.odp import ODPFlow + + +@pytest.mark.parametrize( + "expr,flow,expected,match", + [ + ( + "nw_src=192.168.1.1 && tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 || tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1 || tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 && tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + False, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1/30", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "n_bytes=100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>10", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<1000", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes", "drop"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes>10 && !output.port=3", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["n_bytes", "output"], + ), + ( + "dl_src=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + False, + ["dl_src", "tp_dst"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst~=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + True, + ["dl_src", "tp_dst"], + ), + ( + "encap", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ( + "encap.ipv4.src=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + False, + ["encap"], + ), + ( + "encap.ipv4.src~=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ], +) +def test_filter(expr, flow, expected, match): + ffilter = OFFilter(expr) + result = ffilter.evaluate(flow) + if expected: + assert result + else: + assert not result + + assert [kv.key for kv in result.kv] == match