From patchwork Fri Jan 28 16:04:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585752 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=KXNEjfML; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3D0mNgz9t3b for ; Sat, 29 Jan 2022 03:05:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1E2ED60C0E; Fri, 28 Jan 2022 16:05:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wCDfoj_gwa2z; Fri, 28 Jan 2022 16:05:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3630D60BFC; Fri, 28 Jan 2022 16:05:06 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F1064C0011; Fri, 28 Jan 2022 16:05:05 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 46CD2C000B for ; Fri, 28 Jan 2022 16:05:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3AC7D404C5 for ; Fri, 28 Jan 2022 16:05:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NtEINiuAbWKV for ; Fri, 28 Jan 2022 16:05:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 25E9C4049B for ; Fri, 28 Jan 2022 16:05:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385901; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vDNGy5N/Dbgi3HPu5ZUQDhR+IgfZOupmqFzJFbrbmec=; b=KXNEjfMLhIUOr/rrtMBi5C0DupCdw6VQnqW+1U1pDztmW/c93ZVz2DnNwRcRkSxt9JU542 GdRLzHfBJCVKb0gt3pW9yoUVe4kfuuB0ekqyBVCQCm/sYtXYD30dbjAonCLkv5q4Lc95v0 Lj0Wi16yW6Jqeesyk50abw6t34Yb4Wg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-496-zD3AtC2dPIeN6pLpQahRsA-1; Fri, 28 Jan 2022 11:04:58 -0500 X-MC-Unique: zD3AtC2dPIeN6pLpQahRsA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BABB4802927; Fri, 28 Jan 2022 16:04:57 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id C446E84A2A; Fri, 28 Jan 2022 16:04:55 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:24 +0100 Message-Id: <20220128160441.23477-2-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 01/18] python: add generic Key-Value parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Most of ofproto and dpif flows are based on key-value pairs. These key-value pairs can be represented in several ways, eg: key:value, key=value, key(value). Add the following classes that allow parsing of key-value strings: * KeyValue: holds a key-value pair * KeyMetadata: holds some metadata associated with a KeyValue such as the original key and value strings and their position in the global string * KVParser: is able to parse a string and extract it's key-value pairs as KeyValue instances. Before creating the KeyValue instance it tries to decode the value via the KVDecoders * KVDecoders holds a number of decoders that KVParser can use to decode key-value pairs. It accepts a dictionary of keys and callables to allow users to specify what decoder (i.e: callable) to use for each key Also, flake8 seems to be incorrectly reporting an error (E203) in: "slice[index + offset : index + offset]" which is PEP8 compliant. So, ignore this error. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- Makefile.am | 3 +- python/automake.mk | 6 +- python/ovs/flows/__init__.py | 0 python/ovs/flows/decoders.py | 18 ++ python/ovs/flows/kv.py | 320 +++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 6 files changed, 346 insertions(+), 3 deletions(-) create mode 100644 python/ovs/flows/__init__.py create mode 100644 python/ovs/flows/decoders.py create mode 100644 python/ovs/flows/kv.py diff --git a/Makefile.am b/Makefile.am index cb8076433..4f51d225e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -391,6 +391,7 @@ ALL_LOCAL += flake8-check # E128 continuation line under-indented for visual indent # E129 visually indented line with same indent as next logical line # E131 continuation line unaligned for hanging indent +# E203 whitespace before ':' # E722 do not use bare except, specify exception instead # W503 line break before binary operator # W504 line break after binary operator @@ -403,7 +404,7 @@ ALL_LOCAL += flake8-check # H233 Python 3.x incompatible use of print operator # H238 old style class declaration, use new style (inherit from `object`) FLAKE8_SELECT = H231,H232,H233,H238 -FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E722,W503,W504,F811,D,H,I +FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E203,E722,W503,W504,F811,D,H,I flake8-check: $(FLAKE8_PYFILES) $(FLAKE8_WERROR)$(AM_V_GEN) \ src='$^' && \ diff --git a/python/automake.mk b/python/automake.mk index 767512f17..7ce842d66 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -16,7 +16,6 @@ ovs_pyfiles = \ python/ovs/compat/sortedcontainers/sorteddict.py \ python/ovs/compat/sortedcontainers/sortedset.py \ python/ovs/daemon.py \ - python/ovs/fcntl_win.py \ python/ovs/db/__init__.py \ python/ovs/db/custom_index.py \ python/ovs/db/data.py \ @@ -26,6 +25,10 @@ ovs_pyfiles = \ python/ovs/db/schema.py \ python/ovs/db/types.py \ python/ovs/fatal_signal.py \ + python/ovs/fcntl_win.py \ + python/ovs/flows/__init__.py \ + python/ovs/flows/decoders.py \ + python/ovs/flows/kv.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ @@ -42,6 +45,7 @@ ovs_pyfiles = \ python/ovs/version.py \ python/ovs/vlog.py \ python/ovs/winutils.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ diff --git a/python/ovs/flows/__init__.py b/python/ovs/flows/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py new file mode 100644 index 000000000..73d5c0c60 --- /dev/null +++ b/python/ovs/flows/decoders.py @@ -0,0 +1,18 @@ +""" Defines helpful decoders that can be used to decode information from the +flows. + +A decoder is generally a callable that accepts a string and returns the value +object. +""" + + +def decode_default(value): + """Default decoder. + + It tries to convert into an integer value and, if it fails, just + returns the string. + """ + try: + return int(value, 0) + except ValueError: + return value diff --git a/python/ovs/flows/kv.py b/python/ovs/flows/kv.py new file mode 100644 index 000000000..78cfe627e --- /dev/null +++ b/python/ovs/flows/kv.py @@ -0,0 +1,320 @@ +""" Common helper classes for flow Key-Value parsing. +""" + +import functools +import re + +from ovs.flows.decoders import decode_default + + +class ParseError(RuntimeError): + """Exception raised when an error occurs during parsing.""" + pass + + +class KeyMetadata(object): + """Class for keeping key metadata. + + Attributes: + kpos (int): The position of the keyword in the parent string. + vpos (int): The position of the value in the parent string. + kstring (string): The keyword string as found in the flow string. + vstring (string): The value as found in the flow string. + delim (string): Optional, the string use as delimiter between the key + and the value. + end_delim (string): Optional, the string use as end delimiter between + the key and the value. + """ + + def __init__(self, kpos, vpos, kstring, vstring, delim="", end_delim=""): + """Constructor.""" + self.kpos = kpos + self.vpos = vpos + self.kstring = kstring + self.vstring = vstring + self.delim = delim + self.end_delim = end_delim + + def __str__(self): + return "key: [{},{}), val:[{}, {})".format( + self.kpos, + self.kpos + len(self.kstring), + self.vpos, + self.vpos + len(self.vstring), + ) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + +class KeyValue(object): + """Class for keeping key-value data. + + Attributes: + key (str): The key string. + value (any): The value data. + meta (KeyMetadata): The key metadata. + """ + + def __init__(self, key, value, meta=None): + """Constructor.""" + self.key = key + self.value = value + self.meta = meta + + def __str__(self): + return "{}: {} ({})".format(self.key, str(self.value), str(self.meta)) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + +class KVDecoders(object): + """KVDecoders class is used by KVParser to select how to decode the value + of a specific keyword. + + A decoder is simply a function that accepts a value string and returns + the value objects to be stored. + The returned value may be of any type. + + Decoders may return a KeyValue instance to indicate that the keyword should + also be modified to match the one provided in the returned KeyValue. + + The decoder to be used will be selected using the key as an index. If not + found, the default decoder will be used. If free keys are found (i.e: + keys without a value), the default_free decoder will be used. For that + reason, the default_free decoder, must return both the key and value to be + stored. + + Args: + decoders (dict): Optional; A dictionary of decoders indexed by keyword. + default (callable): Optional; A decoder used if a match is not found in + configured decoders. If not provided, the default behavior is to + try to decode the value into an integer and, if that fails, + just return the string as-is. + default_free (callable): Optional; The decoder used if a match is not + found in configured decoders and it's a free value (e.g: + a value without a key) Defaults to returning the free value as + keyword and "True" as value. + The callable must accept a string and return a key-value pair. + """ + + def __init__(self, decoders=None, default=None, default_free=None): + self._decoders = decoders or dict() + self._default = default or decode_default + self._default_free = default_free or self._default_free_decoder + + def decode(self, keyword, value_str): + """Decode a keyword and value. + + Args: + keyword (str): The keyword whose value is to be decoded. + value_str (str): The value string. + + Returns: + The key (str) and value(any) to be stored. + """ + + decoder = self._decoders.get(keyword) + if decoder: + result = decoder(value_str) + if isinstance(result, KeyValue): + keyword = result.key + value = result.value + else: + value = result + + return keyword, value + else: + if value_str: + return keyword, self._default(value_str) + else: + return self._default_free(keyword) + + @staticmethod + def _default_free_decoder(key): + """Default decoder for free kewords.""" + return key, True + + +delim_pattern = re.compile(r"(\(|=|:|,|\n|\r|\t)") +parenthesys_pattern = re.compile(r"(\(|\))") +end_pattern = re.compile(r"( |,|\n|\r|\t)") +separators = (" ", ",") +end_of_string = (",", "\n", "\t", "\r", "") + + +class KVParser(object): + """KVParser parses a string looking for key-value pairs. + + Args: + decoders (KVDecoders): Optional; the KVDecoders instance to use. + string (str): The string to parse. + """ + + def __init__(self, string, decoders=None): + """Constructor.""" + self._decoders = decoders or KVDecoders() + self._keyval = list() + self._string = string + + def keys(self): + return list(kv.key for kv in self._keyval) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the key-value pairs in string. + + The input string is assumed to contain a list of comma (or space) + separated key-value pairs. + + Key-values pairs can have multiple different delimiters, eg: + "key1:value1,key2=value2,key3(value3)". + + Also, we can stumble upon a "free" keywords, e.g: + "key1=value1,key2=value2,free_keyword". + We consider this as keys without a value. + + So, to parse the string we do the following until the end of the + string is found: + + 1 - Skip any leading comma's or spaces. + 2 - Find the next delimiter (or end_of_string character). + 3 - Depending on the delimiter, obtain the key and the value. + For instance, if the delimiter is "(", find the next matching + ")". + 4 - Use the KVDecoders to decode the key-value. + 5 - Store the KeyValue object with the corresponding metadata. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + keyword = "" + delimiter = "" + rest = "" + + # 1. Skip separator characters. + if self._string[kpos] in separators: + kpos += 1 + continue + + # 2. Find the next delimiter or end of string character. + split_parts = delim_pattern.split(self._string[kpos:], 1) + + if len(split_parts) == 0: + break + + keyword = split_parts[0] + if len(split_parts) == 3: + # If not at the end of the string, re.split() should return + # 3 strings: [keyword, delimiter, rest]. + delimiter = split_parts[1] + rest = split_parts[2] + + # Extract the value from the rest of the string. + value_str = "" + vpos = kpos + len(keyword) + 1 + end_delimiter = "" + + if delimiter in ("=", ":"): + # If the delimiter is ':' or '=', the end of the value is the + # end of the string or a ', '. + value_parts = end_pattern.split(rest, 1) + value_str = value_parts[0] if len(value_parts) == 3 else rest + next_kpos = vpos + len(value_str) + + elif delimiter == "(": + # Find matching ")". + level = 1 + index = 0 + value_parts = parenthesys_pattern.split(rest) + for val in value_parts: + if val == "(": + level += 1 + elif val == ")": + level -= 1 + index += len(val) + if level == 0: + break + + if level != 0: + raise ParseError( + "Error parsing string {}: " + "Failed to find matching ')' in {}".format( + self._string, rest + ) + ) + + value_str = rest[: index - 1] + next_kpos = vpos + len(value_str) + 1 + end_delimiter = ")" + + # Exceptionally, if after the () we find -> {}, do not treat + # the content of the parenthesis as the value, consider + # ({})->{} as the string value. + if index < len(rest) - 2 and rest[index : index + 2] == "->": + extra_val = rest[index + 2 :].split(",")[0] + value_str = "({})->{}".format(value_str, extra_val) + # remove the first "(". + vpos -= 1 + next_kpos = vpos + len(value_str) + end_delimiter = "" + + elif delimiter in end_of_string: + # Key without a value. + next_kpos = kpos + len(keyword) + vpos = -1 + + # 4. Use KVDecoders to decode the key-value. + try: + key, val = self._decoders.decode(keyword, value_str) + except Exception as e: + raise ParseError( + "Error parsing key-value ({}, {})".format( + keyword, value_str + ) + ) from e + + # Store the KeyValue object with the corresponding metadata. + meta = KeyMetadata( + kpos=kpos, + vpos=vpos, + kstring=keyword, + vstring=value_str, + delim=delimiter, + end_delim=end_delimiter, + ) + + self._keyval.append(KeyValue(key, val, meta)) + + kpos = next_kpos + + +def decode_nested_kv(decoders, value): + """A key-value decoder that extracts nested key-value pairs and returns + them in a dictionary. + + Args: + decoders (KVDecoders): The KVDecoders to use. + value (str): The value string to decode. + """ + if not value: + # Mark as flag + return True + + parser = KVParser(value, decoders) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_kv_decoder(decoders=None): + """Helper function that creates a nested kv decoder with given + KVDecoders.""" + return functools.partial(decode_nested_kv, decoders) diff --git a/python/setup.py b/python/setup.py index cfe01763f..0e6b0ea39 100644 --- a/python/setup.py +++ b/python/setup.py @@ -71,7 +71,7 @@ setup_args = dict( author='Open vSwitch', author_email='dev@openvswitch.org', packages=['ovs', 'ovs.compat', 'ovs.compat.sortedcontainers', - 'ovs.db', 'ovs.unixctl'], + 'ovs.db', 'ovs.unixctl', 'ovs.flows'], keywords=['openvswitch', 'ovs', 'OVSDB'], license='Apache 2.0', classifiers=[ From patchwork Fri Jan 28 16:04:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585755 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dZDfPyjP; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3M5Wdcz9t3b for ; Sat, 29 Jan 2022 03:05:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 616B740BDC; Fri, 28 Jan 2022 16:05:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I9PxHtMRKIBV; Fri, 28 Jan 2022 16:05:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id D45E2404C5; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DB982C0079; Fri, 28 Jan 2022 16:05:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9D634C0039 for ; Fri, 28 Jan 2022 16:05:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 619B760C02 for ; Fri, 28 Jan 2022 16:05:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id muCic7zSo3kE for ; Fri, 28 Jan 2022 16:05:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id F387E60AB8 for ; Fri, 28 Jan 2022 16:05:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385903; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1V08lsV+wqQvYH04+Z2DHsFHjYk9eOqy427jL7JPOTU=; b=dZDfPyjP8gfEkTFitjYlI/GyCLMkRCuuNDftSQb5Dhvc2sUqi2c8Gc4xAQ2UA/o7jf8C45 kbegJA22giuxLfl1MIXp2JxadFMeYbKoZLmAV4HkI97hWi66eeYl2RewTgA6yYAl54Igz5 15UTAvGUCLKZQmi+IEQJoIimWAXkEYo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-341-Dp0tHc3BO12AdOPkCjB_OQ-1; Fri, 28 Jan 2022 11:05:00 -0500 X-MC-Unique: Dp0tHc3BO12AdOPkCjB_OQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9F1CD802925; Fri, 28 Jan 2022 16:04:59 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C09984A2E; Fri, 28 Jan 2022 16:04:57 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:25 +0100 Message-Id: <20220128160441.23477-3-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 02/18] python: add mask, ip and eth decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add more decoders that can be used by KVParser. For IPv4 and IPv6 addresses, create a new class that wraps netaddr.IPAddress. For Ethernet addresses, create a new class that wraps netaddr.EUI. For Integers, create a new class that performs basic bitwise mask comparisons Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/ovs/flows/decoders.py | 398 +++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 2 files changed, 399 insertions(+), 1 deletion(-) diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 73d5c0c60..2f8e5bd0a 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -5,6 +5,15 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import netaddr + + +class Decoder(object): + """Base class for all decoder classes.""" + + def to_json(self): + raise NotImplementedError() + def decode_default(value): """Default decoder. @@ -16,3 +25,392 @@ def decode_default(value): return int(value, 0) except ValueError: return value + + +def decode_flag(value): + """Decode a flag. It's existance is just flagged by returning True.""" + return True + + +def decode_int(value): + """Integer decoder. + + Both base10 and base16 integers are supported. + + Used for fields such as: + n_bytes=34 + metadata=0x4 + """ + return int(value, 0) + + +def decode_time(value): + """Time decoder. + + Used for fields such as: + duration=1234.123s + """ + if value == "never": + return value + + time_str = value.rstrip("s") + return float(time_str) + + +class IntMask(Decoder): + """Base class for Integer Mask decoder classes. + + It supports decoding a value/mask pair. The class has to be derived, + and the size attribute must be set. + """ + + size = None # Size in bits. + + def __init__(self, string): + if not self.size: + raise NotImplementedError( + "IntMask should be derived and " "size should be fixed" + ) + + parts = string.split("/") + if len(parts) > 1: + self._value = int(parts[0], 0) + self._mask = int(parts[1], 0) + if self._mask.bit_length() > self.size: + raise ValueError( + "Integer mask {} is bigger than size {}".format( + self._mask, self.size + ) + ) + else: + self._value = int(parts[0], 0) + self._mask = self.max_mask() + + if self._value.bit_length() > self.size: + raise ValueError( + "Integer value {} is bigger than size {}".format( + self._value, self.size + ) + ) + + @property + def value(self): + return self._value + + @property + def mask(self): + return self._mask + + def max_mask(self): + return 2 ** self.size - 1 + + def fully(self): + """Returns True if it's fully masked.""" + return self._mask == self.max_mask() + + def __str__(self): + if self.fully(): + return str(self._value) + else: + return "{}/{}".format(hex(self._value), hex(self._mask)) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def __eq__(self, other): + """Equality operator. + + Both value and mask must be the same for the comparison to result True. + This can be used to implement filters that expect a specific mask, + e.g: ct.state = 0x1/0xff. + + Args: + other (IntMask): Another IntMask to compare against. + + Returns: + True if the other IntMask is the same as this one. + """ + if isinstance(other, IntMask): + return self.value == other.value and self.mask == other.mask + elif isinstance(other, int): + return self.value == other and self.mask == self.max_mask() + else: + raise ValueError("Cannot compare against ", other) + + def __contains__(self, other): + """Contains operator. + + Args: + other (int or IntMask): Another integer or fully-masked IntMask + to compare against. + + Returns: + True if the other integer or fully-masked IntMask is + contained in this IntMask. + + Example: + 0x1 in IntMask("0xf1/0xff"): True + 0x1 in IntMask("0xf1/0x0f"): True + 0x1 in IntMask("0xf1/0xf0"): False + """ + if isinstance(other, IntMask): + if other.fully(): + return other.value in self + else: + raise ValueError( + "Comparing non fully-masked IntMasks is not supported" + ) + else: + return other & self._mask == self._value & self._mask + + def dict(self): + return {"value": self._value, "mask": self._mask} + + def to_json(self): + return self.dict() + + +class Mask8(IntMask): + size = 8 + + +class Mask16(IntMask): + size = 16 + + +class Mask32(IntMask): + size = 32 + + +class Mask64(IntMask): + size = 64 + + +class Mask128(IntMask): + size = 128 + + +class Mask992(IntMask): + size = 992 + + +def decode_mask(mask_size): + """Value/Mask decoder for values of specific size (bits). + + Used for fields such as: + reg0=0x248/0xff + """ + + class Mask(IntMask): + size = mask_size + __name__ = "Mask{}".format(size) + + return Mask + + +class EthMask(Decoder): + """EthMask represents an Ethernet address with optional mask. + + It uses netaddr.EUI. + + Attributes: + eth (netaddr.EUI): The ethernet address. + mask (netaddr.EUI): Optional, the ethernet address mask. + + Args: + string (str): A string representing the masked ethernet address + e.g: 00.11:22:33:44:55 or 01:00:22:00:33:00/01:00:00:00:00:00 + """ + + def __init__(self, string): + mask_parts = string.split("/") + self._eth = netaddr.EUI(mask_parts[0]) + if len(mask_parts) == 2: + self._mask = netaddr.EUI(mask_parts[1]) + else: + self._mask = None + + @property + def eth(self): + """The ethernet address.""" + return self._eth + + @property + def mask(self): + """The ethernet address mask.""" + return self._mask + + def __eq__(self, other): + """Equality operator. + + Both the Ethernet address and the mask are compared. This can be used + to implement filters where we expecte a specific mask to be present, + e.g: dl_dst=01:00:00:00:00:00/01:00:00:00:00:00. + + Args: + other (EthMask): Another EthMask to compare against. + + Returns: + True if this EthMask is the same as the other. + """ + return self._mask == other._mask and self._eth == other._eth + + def __contains__(self, other): + """Contains operator. + + Args: + other (netaddr.EUI or EthMask): An Ethernet address. + + Returns: + True if the other netaddr.EUI or fully-maked EthMask is + contained in this EthMask's address range. + """ + if isinstance(other, EthMask): + if other._mask: + raise ValueError( + "Comparing non fully-masked EthMask is not supported" + ) + return other._eth in self + + if self._mask: + return (other.value & self._mask.value) == ( + self._eth.value & self._mask.value + ) + else: + return other == self._eth + + def __str__(self): + if self._mask: + return "/".join( + [ + self._eth.format(netaddr.mac_unix), + self._mask.format(netaddr.mac_unix), + ] + ) + else: + return self._eth.format(netaddr.mac_unix) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) + + +class IPMask(Decoder): + """IPMask stores an IPv6 or IPv4 and a mask. + + It uses netaddr.IPAddress. + + IPMasks can represent valid CIDRs or randomly maked IP Addresses. + + Args: + string (str): A string representing the ip/mask. + """ + + def __init__(self, string): + self._ipnet = None + self._ip = None + self._mask = None + try: + self._ipnet = netaddr.IPNetwork(string) + except netaddr.AddrFormatError: + pass + + if not self._ipnet: + # It's not a valid CIDR. Store ip and mask independently. + parts = string.split("/") + if len(parts) != 2: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) + try: + self._ip = netaddr.IPAddress(parts[0]) + self._mask = netaddr.IPAddress(parts[1]) + except netaddr.AddrFormatError as exc: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) from exc + + def __eq__(self, other): + """Equality operator. + + Both the IPAddress and the mask are compared. This can be used + to implement filters where a specific mask is expected, e.g: + nw_src=192.168.1.0/24. + + Args: + other (IPMask or netaddr.IPNetwork or netaddr.IPAddress): + Another IPAddress or IPNetwork to compare against. + + Returns: + True if this IPMask is the same as the other. + """ + if isinstance(other, netaddr.IPNetwork): + return self._ipnet and self._ipnet == other + if isinstance(other, netaddr.IPAddress): + return self._ipnet and self._ipnet.ip == other + elif isinstance(other, IPMask): + if self._ipnet: + return self._ipnet == other._ipnet + + return self._ip == other._ip and self._mask == other._mask + else: + return False + + def __contains__(self, other): + """Contains operator. + + Only comparing valid CIDRs is supported. + + Args: + other (netaddr.IPAddres or IPMask): An IP address. + + Returns: + True if the other IPAddress is contained in this IPMask's address + range. + """ + if isinstance(other, IPMask): + if not other._ipnet: + raise ValueError("Only comparing valid CIDRs is supported") + + return ( + netaddr.IPAddress(other._ipnet.first) in self + and netaddr.IPAddress(other._ipnet.last) in self + ) + + elif isinstance(other, netaddr.IPAddress): + if self._ipnet: + return other in self._ipnet + return (other & self._mask) == (self._ip & self._mask) + + def cidr(self): + """ + Returns True if the IPMask is a valid CIDR. + """ + return self._ipnet is not None + + @property + def ip(self): + """The IP address.""" + if self._ipnet: + return self._ipnet.ip + return self._ip + + @property + def mask(self): + """The IP mask.""" + if self._ipnet: + return self._ipnet.netmask + return self._mask + + def __str__(self): + if self._ipnet: + return str(self._ipnet) + return "/".join([str(self._ip), str(self._mask)]) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) diff --git a/python/setup.py b/python/setup.py index 0e6b0ea39..b06370bd9 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers'], + install_requires=['sortedcontainers', 'netaddr'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Fri Jan 28 16:04:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585756 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Zekrev21; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3Q63lDz9t3b for ; Sat, 29 Jan 2022 03:05:22 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 62CAE40BD7; Fri, 28 Jan 2022 16:05:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eHVOyemZcKSx; Fri, 28 Jan 2022 16:05:14 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5545340BC5; Fri, 28 Jan 2022 16:05:12 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8C951C007C; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id A1C86C000B for ; Fri, 28 Jan 2022 16:05:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7E09A404C5 for ; Fri, 28 Jan 2022 16:05:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id krUGZNfKgqzl for ; Fri, 28 Jan 2022 16:05:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2DE8C40ABD for ; Fri, 28 Jan 2022 16:05:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385906; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cQQ+aJup1Xsi7FF3lbh1/QNCN8fXe1iGEPyreGRYksM=; b=Zekrev21+2DnD3TNPFQPiEqMlrhmJ/CemPEM9+5IlHxg7pge+YVOs/+5n+3FKsa3nUuCNG oGngCrj+uVOlrvyAMZLKj5y5SnDwQMScNh1Y8qqHW4ye+nd2z1QU0UorJ+xaMJF++oK6XG xyDtVoxKpQWU6KKQo4RJv9WDrfPtwiY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558-FoOAmgKqNv-NIMu-83gmHQ-1; Fri, 28 Jan 2022 11:05:02 -0500 X-MC-Unique: FoOAmgKqNv-NIMu-83gmHQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C4CE6802925; Fri, 28 Jan 2022 16:05:01 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2441E84A2A; Fri, 28 Jan 2022 16:04:59 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:26 +0100 Message-Id: <20220128160441.23477-4-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 03/18] python: add list parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Some openflow or dpif flows encode their arguments in lists, eg: "some_action(arg1,arg2,arg3)". In order to decode this in a way that can be then stored and queried, add ListParser and ListDecoders classes that parse lists into KeyValue instances. The ListParser / ListDecoders mechanism is quite similar to KVParser and KVDecoders. Since the "key" of the different KeyValue objects is now ommited, it has to be provided by ListDecoders. For example, take the openflow action "resubmit" that can be written as: resubmit([port],[table][,ct]) Can be decoded by creating a ListDecoders instance such as: ListDecoders([ ("port", decode_default), ("table", decode_int), ("ct", decode_flag), ]) Naturally, the order of the decoders must be kept. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flows/list.py | 124 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 python/ovs/flows/list.py diff --git a/python/automake.mk b/python/automake.mk index 7ce842d66..73438d615 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -29,6 +29,7 @@ ovs_pyfiles = \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ python/ovs/flows/kv.py \ + python/ovs/flows/list.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py new file mode 100644 index 000000000..cafd23d0a --- /dev/null +++ b/python/ovs/flows/list.py @@ -0,0 +1,124 @@ +import re + +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError +from ovs.flows.decoders import decode_default + + +class ListDecoders(object): + """ListDecoders is used by ListParser to decode the elements in the list. + + A decoder is a function that accepts a value and returns its decoded + object. + + ListDecoders is initialized with a list of of tuples that contains the + keyword and the decoding function associated with each position in the + list. The order is, therefore, important. + + Args: + decoders (list of tuples): Optional; a list of tuples. + The first element in the tuple is the keyword associated with the + value. The second element in the tuple is the decoder function. + """ + + def __init__(self, decoders=None): + self._decoders = decoders or list() + + def decode(self, index, value_str): + """Decode the index'th element of the list. + + Args: + index (int): The position in the list of the element to decode. + value_str (str): The value string to decode. + """ + if index < 0 or index >= len(self._decoders): + return self._default_decoder(index, value_str) + + try: + key = self._decoders[index][0] + value = self._decoders[index][1](value_str) + return key, value + except Exception as e: + raise ParseError( + "Failed to decode value_str {}: {}".format(value_str, str(e)) + ) + + @staticmethod + def _default_decoder(index, value): + key = "elem_{}".format(index) + return key, decode_default(value) + + +class ListParser(object): + """ListParser parses a list of values and stores them as key-value pairs. + + It uses a ListDecoders instance to decode each element in the list. + + Args: + string (str): The string to parse. + decoders (ListDecoders): Optional, the decoders to use. + delims (list): Optional, list of delimiters of the list. Defaults to + [',']. + """ + def __init__(self, string, decoders=None, delims=[","]): + self._string = string + self._decoders = decoders or ListDecoders() + self._keyval = list() + self._regexp = r"({})".format("|".join(delims)) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the list in string. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + index = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + split_parts = re.split(self._regexp, self._string[kpos:], 1) + if len(split_parts) == 0: + break + + value_str = split_parts[0] + + key, value = self._decoders.decode(index, value_str) + + meta = KeyMetadata( + kpos=kpos, + vpos=kpos, + kstring=value_str, + vstring=value_str, + ) + self._keyval.append(KeyValue(key, value, meta)) + + kpos += len(value_str) + 1 + index += 1 + + +def decode_nested_list(decoders, value, delims=[","]): + """Decodes a string value that contains a list of elements and returns + them in a dictionary. + + Args: + decoders (ListDecoders): The ListDecoders to use. + value (str): The value string to decode. + delims (list(str)): Optional, the list of delimiters to use. + """ + parser = ListParser(value, decoders, delims) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_list_decoder(decoders=None, delims=[","]): + """Helper function that creates a nested list decoder with given + ListDecoders and delimiters. + """ + def decoder(value): + return decode_nested_list(decoders, value, delims) + + return decoder From patchwork Fri Jan 28 16:04:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585757 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=RjlgkUrD; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3W0rhPz9t3b for ; Sat, 29 Jan 2022 03:05:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 8B06741D17; Fri, 28 Jan 2022 16:05:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r9WEYFu2SFUS; Fri, 28 Jan 2022 16:05:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id C184F41C77; Fri, 28 Jan 2022 16:05:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 185E2C000B; Fri, 28 Jan 2022 16:05:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 55AF8C007E for ; Fri, 28 Jan 2022 16:05:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 28A00610D9 for ; Fri, 28 Jan 2022 16:05:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=fail (1024-bit key) reason="fail (body has been altered)" header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kVTcLAaf0fFp for ; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6B006610E7 for ; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iSxTLJKok+xEZ+zObsbwGiyzVko466/t3dzky/qjnIc=; b=RjlgkUrDxdc9Oix5XFaYVBa/TLu3MiFpRXiAYhF8gIN1n72Pc/4X/tT7Xd52pFYjLxHHuV tP2ao1bE28SyCP67vlEg7vGbvVelI7718tQAlwYjcc6Q6ISazPu0H9gXCXUoQlEazHFuD3 9ISgjxHX+B4+1Ea0uFgkkh8D9TiY1aQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-644-u-iXqiPgOWyFiUpy4cvlvQ-1; Fri, 28 Jan 2022 11:05:04 -0500 X-MC-Unique: u-iXqiPgOWyFiUpy4cvlvQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D7DFA802C87; Fri, 28 Jan 2022 16:05:03 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 48B8584A2A; Fri, 28 Jan 2022 16:05:01 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:27 +0100 Message-Id: <20220128160441.23477-5-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 04/18] build-aux: split extract-ofp-fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to be able to reuse the core extraction logic, split the command in two parts. The core extraction logic is moved to python/build while the command that writes the different files out of the extracted field info is kept in build-aux. Signed-off-by: Adrian Moreno --- build-aux/extract-ofp-fields | 706 ++++++++--------------------- python/automake.mk | 1 + python/build/extract_ofp_fields.py | 386 ++++++++++++++++ 3 files changed, 579 insertions(+), 514 deletions(-) create mode 100644 python/build/extract_ofp_fields.py diff --git a/build-aux/extract-ofp-fields b/build-aux/extract-ofp-fields index 8766995d9..efec59c25 100755 --- a/build-aux/extract-ofp-fields +++ b/build-aux/extract-ofp-fields @@ -3,85 +3,23 @@ import getopt import sys import os.path -import re import xml.dom.minidom import build.nroff -line = "" - -# Maps from user-friendly version number to its protocol encoding. -VERSION = {"1.0": 0x01, - "1.1": 0x02, - "1.2": 0x03, - "1.3": 0x04, - "1.4": 0x05, - "1.5": 0x06} -VERSION_REVERSE = dict((v,k) for k, v in VERSION.items()) - -TYPES = {"u8": (1, False), - "be16": (2, False), - "be32": (4, False), - "MAC": (6, False), - "be64": (8, False), - "be128": (16, False), - "tunnelMD": (124, True)} - -FORMATTING = {"decimal": ("MFS_DECIMAL", 1, 8), - "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), - "ct state": ("MFS_CT_STATE", 4, 4), - "Ethernet": ("MFS_ETHERNET", 6, 6), - "IPv4": ("MFS_IPV4", 4, 4), - "IPv6": ("MFS_IPV6", 16, 16), - "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), - "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), - "frag": ("MFS_FRAG", 1, 1), - "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), - "TCP flags": ("MFS_TCP_FLAGS", 2, 2), - "packet type": ("MFS_PACKET_TYPE", 4, 4)} - -PREREQS = {"none": "MFP_NONE", - "Ethernet": "MFP_ETHERNET", - "ARP": "MFP_ARP", - "VLAN VID": "MFP_VLAN_VID", - "IPv4": "MFP_IPV4", - "IPv6": "MFP_IPV6", - "IPv4/IPv6": "MFP_IP_ANY", - "NSH": "MFP_NSH", - "CT": "MFP_CT_VALID", - "MPLS": "MFP_MPLS", - "TCP": "MFP_TCP", - "UDP": "MFP_UDP", - "SCTP": "MFP_SCTP", - "ICMPv4": "MFP_ICMPV4", - "ICMPv6": "MFP_ICMPV6", - "ND": "MFP_ND", - "ND solicit": "MFP_ND_SOLICIT", - "ND advert": "MFP_ND_ADVERT"} - -# Maps a name prefix into an (experimenter ID, class) pair, so: -# -# - Standard OXM classes are written as (0, ) -# -# - Experimenter OXM classes are written as (, 0xffff) -# -# If a name matches more than one prefix, the longest one is used. -OXM_CLASSES = {"NXM_OF_": (0, 0x0000, 'extension'), - "NXM_NX_": (0, 0x0001, 'extension'), - "NXOXM_NSH_": (0x005ad650, 0xffff, 'extension'), - "OXM_OF_": (0, 0x8000, 'standard'), - "OXM_OF_PKT_REG": (0, 0x8001, 'standard'), - "ONFOXM_ET_": (0x4f4e4600, 0xffff, 'standard'), - "ERICOXM_OF_": (0, 0x1000, 'extension'), - - # This is the experimenter OXM class for Nicira, which is the - # one that OVS would be using instead of NXM_OF_ and NXM_NX_ - # if OVS didn't have those grandfathered in. It is currently - # used only to test support for experimenter OXM, since there - # are barely any real uses of experimenter OXM in the wild. - "NXOXM_ET_": (0x00002320, 0xffff, 'extension')} +from build.extract_ofp_fields import ( + extract_ofp_fields, + PREREQS, + OXM_CLASSES, + VERSION, + fatal, + n_errors, +) + +VERSION_REVERSE = dict((v, k) for k, v in VERSION.items()) + def oxm_name_to_class(name): - prefix = '' + prefix = "" class_ = None for p, c in OXM_CLASSES.items(): if name.startswith(p) and len(p) > len(prefix): @@ -92,267 +30,76 @@ def oxm_name_to_class(name): def is_standard_oxm(name): oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) - return oxm_class_type == 'standard' - - -def decode_version_range(range): - if range in VERSION: - return (VERSION[range], VERSION[range]) - elif range.endswith('+'): - return (VERSION[range[:-1]], max(VERSION.values())) - else: - a, b = re.match(r'^([^-]+)-([^-]+)$', range).groups() - return (VERSION[a], VERSION[b]) - - -def get_line(): - global line - global line_number - line = input_file.readline() - line_number += 1 - if line == "": - fatal("unexpected end of input") - - -n_errors = 0 - - -def error(msg): - global n_errors - sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) - n_errors += 1 - - -def fatal(msg): - error(msg) - sys.exit(1) + return oxm_class_type == "standard" def usage(): argv0 = os.path.basename(sys.argv[0]) - print('''\ + print( + """\ %(argv0)s, for extracting OpenFlow field properties from meta-flow.h usage: %(argv0)s INPUT [--meta-flow | --nx-match] where INPUT points to lib/meta-flow.h in the source directory. Depending on the option given, the output written to stdout is intended to be saved either as lib/meta-flow.inc or lib/nx-match.inc for the respective C file to #include.\ -''' % {"argv0": argv0}) +""" + % {"argv0": argv0} + ) sys.exit(0) -def make_sizeof(s): - m = re.match(r'(.*) up to (.*)', s) - if m: - struct, member = m.groups() - return "offsetof(%s, %s)" % (struct, member) - else: - return "sizeof(%s)" % s - - -def parse_oxms(s, prefix, n_bytes): - if s == 'none': - return () - - return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(',')) - - -match_types = dict() - - -def parse_oxm(s, prefix, n_bytes): - global match_types - - m = re.match('([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$', s) - if not m: - fatal("%s: syntax error parsing %s" % (s, prefix)) - - name, oxm_type, of_version, ovs_version = m.groups() - - class_ = oxm_name_to_class(name) - if class_ is None: - fatal("unknown OXM class for %s" % name) - oxm_vendor, oxm_class, oxm_class_type = class_ - - if class_ in match_types: - if oxm_type in match_types[class_]: - fatal("duplicate match type for %s (conflicts with %s)" % - (name, match_types[class_][oxm_type])) - else: - match_types[class_] = dict() - match_types[class_][oxm_type] = name - - # Normally the oxm_length is the size of the field, but for experimenter - # OXMs oxm_length also includes the 4-byte experimenter ID. - oxm_length = n_bytes - if oxm_class == 0xffff: - oxm_length += 4 - - header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) - - if of_version: - if oxm_class_type == 'extension': - fatal("%s: OXM extension can't have OpenFlow version" % name) - if of_version not in VERSION: - fatal("%s: unknown OpenFlow version %s" % (name, of_version)) - of_version_nr = VERSION[of_version] - if of_version_nr < VERSION['1.2']: - fatal("%s: claimed version %s predates OXM" % (name, of_version)) - else: - if oxm_class_type == 'standard': - fatal("%s: missing OpenFlow version number" % name) - of_version_nr = 0 - - return (header, name, of_version_nr, ovs_version) - - -def parse_field(mff, comment): - f = {'mff': mff} - - # First line of comment is the field name. - m = re.match(r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0]) - if not m: - fatal("%s lacks field name" % mff) - f['name'], f['extra_name'] = m.groups() - - # Find the last blank line the comment. The field definitions - # start after that. - blank = None - for i in range(len(comment)): - if not comment[i]: - blank = i - if not blank: - fatal("%s: missing blank line in comment" % mff) - - d = {} - for key in ("Type", "Maskable", "Formatting", "Prerequisites", - "Access", "Prefix lookup member", - "OXM", "NXM", "OF1.0", "OF1.1"): - d[key] = None - for fline in comment[blank + 1:]: - m = re.match(r'([^:]+):\s+(.*)\.$', fline) - if not m: - fatal("%s: syntax error parsing key-value pair as part of %s" - % (fline, mff)) - key, value = m.groups() - if key not in d: - fatal("%s: unknown key" % key) - elif key == 'Code point': - d[key] += [value] - elif d[key] is not None: - fatal("%s: duplicate key" % key) - d[key] = value - for key, value in d.items(): - if not value and key not in ("OF1.0", "OF1.1", - "Prefix lookup member", "Notes"): - fatal("%s: missing %s" % (mff, key)) - - m = re.match(r'([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$', d['Type']) - if not m: - fatal("%s: syntax error in type" % mff) - type_ = m.group(1) - if type_ not in TYPES: - fatal("%s: unknown type %s" % (mff, d['Type'])) - - f['n_bytes'] = TYPES[type_][0] - if m.group(2): - f['n_bits'] = int(m.group(2)) - if f['n_bits'] > f['n_bytes'] * 8: - fatal("%s: more bits (%d) than field size (%d)" - % (mff, f['n_bits'], 8 * f['n_bytes'])) - else: - f['n_bits'] = 8 * f['n_bytes'] - f['variable'] = TYPES[type_][1] - - if d['Maskable'] == 'no': - f['mask'] = 'MFM_NONE' - elif d['Maskable'] == 'bitwise': - f['mask'] = 'MFM_FULLY' - else: - fatal("%s: unknown maskable %s" % (mff, d['Maskable'])) - - fmt = FORMATTING.get(d['Formatting']) - if not fmt: - fatal("%s: unknown format %s" % (mff, d['Formatting'])) - f['formatting'] = d['Formatting'] - if f['n_bytes'] < fmt[1] or f['n_bytes'] > fmt[2]: - fatal("%s: %d-byte field can't be formatted as %s" - % (mff, f['n_bytes'], d['Formatting'])) - f['string'] = fmt[0] - - f['prereqs'] = d['Prerequisites'] - if f['prereqs'] not in PREREQS: - fatal("%s: unknown prerequisites %s" % (mff, d['Prerequisites'])) - - if d['Access'] == 'read-only': - f['writable'] = False - elif d['Access'] == 'read/write': - f['writable'] = True - else: - fatal("%s: unknown access %s" % (mff, d['Access'])) - - f['OF1.0'] = d['OF1.0'] - if not d['OF1.0'] in (None, 'exact match', 'CIDR mask'): - fatal("%s: unknown OF1.0 match type %s" % (mff, d['OF1.0'])) - - f['OF1.1'] = d['OF1.1'] - if not d['OF1.1'] in (None, 'exact match', 'bitwise mask'): - fatal("%s: unknown OF1.1 match type %s" % (mff, d['OF1.1'])) - - f['OXM'] = (parse_oxms(d['OXM'], 'OXM', f['n_bytes']) + - parse_oxms(d['NXM'], 'NXM', f['n_bytes'])) - - f['prefix'] = d["Prefix lookup member"] - - return f - - def protocols_to_c(protocols): - if protocols == set(['of10', 'of11', 'oxm']): - return 'OFPUTIL_P_ANY' - elif protocols == set(['of11', 'oxm']): - return 'OFPUTIL_P_NXM_OF11_UP' - elif protocols == set(['oxm']): - return 'OFPUTIL_P_NXM_OXM_ANY' + if protocols == set(["of10", "of11", "oxm"]): + return "OFPUTIL_P_ANY" + elif protocols == set(["of11", "oxm"]): + return "OFPUTIL_P_NXM_OF11_UP" + elif protocols == set(["oxm"]): + return "OFPUTIL_P_NXM_OXM_ANY" elif protocols == set([]): - return 'OFPUTIL_P_NONE' + return "OFPUTIL_P_NONE" else: assert False def autogen_c_comment(): return [ -"/* Generated automatically; do not modify! -*- buffer-read-only: t -*- */", -""] + "/* Generated automatically; do not modify! " + "-*- buffer-read-only: t -*- */", + "", + ] + def make_meta_flow(meta_flow_h): fields = extract_ofp_fields(meta_flow_h) output = autogen_c_comment() for f in fields: output += ["{"] - output += [" %s," % f['mff']] - if f['extra_name']: - output += [" \"%s\", \"%s\"," % (f['name'], f['extra_name'])] + output += [" %s," % f["mff"]] + if f["extra_name"]: + output += [' "%s", "%s",' % (f["name"], f["extra_name"])] else: - output += [" \"%s\", NULL," % f['name']] + output += [' "%s", NULL,' % f["name"]] - if f['variable']: - variable = 'true' + if f["variable"]: + variable = "true" else: - variable = 'false' - output += [" %d, %d, %s," % (f['n_bytes'], f['n_bits'], variable)] + variable = "false" + output += [" %d, %d, %s," % (f["n_bytes"], f["n_bits"], variable)] - if f['writable']: - rw = 'true' + if f["writable"]: + rw = "true" else: - rw = 'false' - output += [" %s, %s, %s, %s, false," - % (f['mask'], f['string'], PREREQS[f['prereqs']], rw)] - - oxm = f['OXM'] - of10 = f['OF1.0'] - of11 = f['OF1.1'] - if f['mff'] in ('MFF_DL_VLAN', 'MFF_DL_VLAN_PCP'): + rw = "false" + output += [ + " %s, %s, %s, %s, false," + % (f["mask"], f["string"], PREREQS[f["prereqs"]], rw) + ] + + oxm = f["OXM"] + of10 = f["OF1.0"] + of11 = f["OF1.1"] + if f["mff"] in ("MFF_DL_VLAN", "MFF_DL_VLAN_PCP"): # MFF_DL_VLAN and MFF_DL_VLAN_PCP don't exactly correspond to # OF1.1, nor do they have NXM or OXM assignments, but their # meanings can be expressed in every protocol, which is the goal of @@ -367,25 +114,25 @@ def make_meta_flow(meta_flow_h): if oxm: protocols |= set(["oxm"]) - if f['mask'] == 'MFM_FULLY': + if f["mask"] == "MFM_FULLY": cidr_protocols = protocols.copy() bitwise_protocols = protocols.copy() - if of10 == 'exact match': - bitwise_protocols -= set(['of10']) - cidr_protocols -= set(['of10']) - elif of10 == 'CIDR mask': - bitwise_protocols -= set(['of10']) + if of10 == "exact match": + bitwise_protocols -= set(["of10"]) + cidr_protocols -= set(["of10"]) + elif of10 == "CIDR mask": + bitwise_protocols -= set(["of10"]) else: assert of10 is None - if of11 == 'exact match': - bitwise_protocols -= set(['of11']) - cidr_protocols -= set(['of11']) + if of11 == "exact match": + bitwise_protocols -= set(["of11"]) + cidr_protocols -= set(["of11"]) else: - assert of11 in (None, 'bitwise mask') + assert of11 in (None, "bitwise mask") else: - assert f['mask'] == 'MFM_NONE' + assert f["mask"] == "MFM_NONE" cidr_protocols = set([]) bitwise_protocols = set([]) @@ -393,8 +140,8 @@ def make_meta_flow(meta_flow_h): output += [" %s," % protocols_to_c(cidr_protocols)] output += [" %s," % protocols_to_c(bitwise_protocols)] - if f['prefix']: - output += [" FLOW_U32OFS(%s)," % f['prefix']] + if f["prefix"]: + output += [" FLOW_U32OFS(%s)," % f["prefix"]] else: output += [" -1, /* not usable for prefix lookup */"] @@ -409,147 +156,37 @@ def make_nx_match(meta_flow_h): print("static struct nxm_field_index all_nxm_fields[] = {") for f in fields: # Sort by OpenFlow version number (nx-match.c depends on this). - for oxm in sorted(f['OXM'], key=lambda x: x[2]): - header = ("NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0]) - print("""{ .nf = { %s, %d, "%s", %s } },""" % ( - header, oxm[2], oxm[1], f['mff'])) + for oxm in sorted(f["OXM"], key=lambda x: x[2]): + header = "NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0] + print( + """{ .nf = { %s, %d, "%s", %s } },""" + % (header, oxm[2], oxm[1], f["mff"]) + ) print("};") for oline in output: print(oline) -def extract_ofp_fields(fn): - global file_name - global input_file - global line_number - global line - - file_name = fn - input_file = open(file_name) - line_number = 0 - - fields = [] - - while True: - get_line() - if re.match('enum.*mf_field_id', line): - break - - while True: - get_line() - first_line_number = line_number - here = '%s:%d' % (file_name, line_number) - if (line.startswith('/*') - or line.startswith(' *') - or line.startswith('#') - or not line - or line.isspace()): - continue - elif re.match('}', line) or re.match('\s+MFF_N_IDS', line): - break - - # Parse the comment preceding an MFF_ constant into 'comment', - # one line to an array element. - line = line.strip() - if not line.startswith('/*'): - fatal("unexpected syntax between fields") - line = line[1:] - comment = [] - end = False - while not end: - line = line.strip() - if line.startswith('*/'): - get_line() - break - if not line.startswith('*'): - fatal("unexpected syntax within field") - - line = line[1:] - if line.startswith(' '): - line = line[1:] - if line.startswith(' ') and comment: - continuation = True - line = line.lstrip() - else: - continuation = False - - if line.endswith('*/'): - line = line[:-2].rstrip() - end = True - else: - end = False - - if continuation: - comment[-1] += " " + line - else: - comment += [line] - get_line() - - # Drop blank lines at each end of comment. - while comment and not comment[0]: - comment = comment[1:] - while comment and not comment[-1]: - comment = comment[:-1] - - # Parse the MFF_ constant(s). - mffs = [] - while True: - m = re.match('\s+(MFF_[A-Z0-9_]+),?\s?$', line) - if not m: - break - mffs += [m.group(1)] - get_line() - if not mffs: - fatal("unexpected syntax looking for MFF_ constants") - - if len(mffs) > 1 or '' in comment[0]: - for mff in mffs: - # Extract trailing integer. - m = re.match('.*[^0-9]([0-9]+)$', mff) - if not m: - fatal("%s lacks numeric suffix in register group" % mff) - n = m.group(1) - - # Search-and-replace within the comment, - # and drop lines that have for x != n. - instance = [] - for x in comment: - y = x.replace('', n) - if re.search('<[0-9]+>', y): - if ('<%s>' % n) not in y: - continue - y = re.sub('<[0-9]+>', '', y) - instance += [y.strip()] - fields += [parse_field(mff, instance)] - else: - fields += [parse_field(mffs[0], comment)] - continue - - input_file.close() - - if n_errors: - sys.exit(1) - - return fields - ## ------------------------ ## ## Documentation Generation ## ## ------------------------ ## + def field_to_xml(field_node, f, body, summary): f["used"] = True # Summary. - if field_node.hasAttribute('internal'): + if field_node.hasAttribute("internal"): return min_of_version = None min_ovs_version = None - for header, name, of_version_nr, ovs_version_s in f['OXM']: - if (is_standard_oxm(name) - and (min_ovs_version is None or of_version_nr < min_of_version)): + for header, name, of_version_nr, ovs_version_s in f["OXM"]: + if is_standard_oxm(name) and ( + min_ovs_version is None or of_version_nr < min_of_version + ): min_of_version = of_version_nr - ovs_version = [int(x) for x in ovs_version_s.split('.')] + ovs_version = [int(x) for x in ovs_version_s.split(".")] if min_ovs_version is None or ovs_version < min_ovs_version: min_ovs_version = ovs_version summary += ["\\fB%s\\fR" % f["name"]] @@ -565,124 +202,152 @@ def field_to_xml(field_node, f, body, summary): if min_of_version is not None: support += ["OF %s+" % VERSION_REVERSE[min_of_version]] if min_ovs_version is not None: - support += ["OVS %s+" % '.'.join([str(x) for x in min_ovs_version])] - summary += ' and '.join(support) + support += ["OVS %s+" % ".".join([str(x) for x in min_ovs_version])] + summary += " and ".join(support) summary += ["\n"] # Full description. - if field_node.hasAttribute('hidden'): + if field_node.hasAttribute("hidden"): return - title = field_node.attributes['title'].nodeValue + title = field_node.attributes["title"].nodeValue - body += [""".PP + body += [ + """.PP \\fB%s Field\\fR .TS tab(;); l lx. -""" % title] +""" + % title + ] body += ["Name:;\\fB%s\\fR" % f["name"]] if f["extra_name"]: body += [" (aka \\fB%s\\fR)" % f["extra_name"]] - body += ['\n'] + body += ["\n"] body += ["Width:;"] if f["n_bits"] != 8 * f["n_bytes"]: - body += ["%d bits (only the least-significant %d bits " - "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"])] + body += [ + "%d bits (only the least-significant %d bits " + "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"]) + ] elif f["n_bits"] <= 128: body += ["%d bits" % f["n_bits"]] else: body += ["%d bits (%d bytes)" % (f["n_bits"], f["n_bits"] / 8)] - body += ['\n'] + body += ["\n"] body += ["Format:;%s\n" % f["formatting"]] - masks = {"MFM_NONE": "not maskable", - "MFM_FULLY": "arbitrary bitwise masks"} + masks = { + "MFM_NONE": "not maskable", + "MFM_FULLY": "arbitrary bitwise masks", + } body += ["Masking:;%s\n" % masks[f["mask"]]] body += ["Prerequisites:;%s\n" % f["prereqs"]] - access = {True: "read/write", - False: "read-only"}[f["writable"]] + access = {True: "read/write", False: "read-only"}[f["writable"]] body += ["Access:;%s\n" % access] - of10 = {None: "not supported", - "exact match": "yes (exact match only)", - "CIDR mask": "yes (CIDR match only)"} + of10 = { + None: "not supported", + "exact match": "yes (exact match only)", + "CIDR mask": "yes (CIDR match only)", + } body += ["OpenFlow 1.0:;%s\n" % of10[f["OF1.0"]]] - of11 = {None: "not supported", - "exact match": "yes (exact match only)", - "bitwise mask": "yes"} + of11 = { + None: "not supported", + "exact match": "yes (exact match only)", + "bitwise mask": "yes", + } body += ["OpenFlow 1.1:;%s\n" % of11[f["OF1.1"]]] oxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if is_standard_oxm(x[1])]: + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if is_standard_oxm(x[1]) + ]: of_version = VERSION_REVERSE[of_version_nr] - oxms += [r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" % (name, header[2], of_version, ovs_version)] + oxms += [ + r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" + % (name, header[2], of_version, ovs_version) + ] if not oxms: - oxms = ['none'] - body += ['OXM:;T{\n%s\nT}\n' % r'\[char59] '.join(oxms)] + oxms = ["none"] + body += ["OXM:;T{\n%s\nT}\n" % r"\[char59] ".join(oxms)] nxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if not is_standard_oxm(x[1])]: - nxms += [r"\fB%s\fR (%d) since Open vSwitch %s" % (name, header[2], ovs_version)] + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if not is_standard_oxm(x[1]) + ]: + nxms += [ + r"\fB%s\fR (%d) since Open vSwitch %s" + % (name, header[2], ovs_version) + ] if not nxms: - nxms = ['none'] - body += ['NXM:;T{\n%s\nT}\n' % r'\[char59] '.join(nxms)] + nxms = ["none"] + body += ["NXM:;T{\n%s\nT}\n" % r"\[char59] ".join(nxms)] body += [".TE\n"] - body += ['.PP\n'] + body += [".PP\n"] body += [build.nroff.block_xml_to_nroff(field_node.childNodes)] + def group_xml_to_nroff(group_node, fields): - title = group_node.attributes['title'].nodeValue + title = group_node.attributes["title"].nodeValue summary = [] body = [] for node in group_node.childNodes: - if node.nodeType == node.ELEMENT_NODE and node.tagName == 'field': - id_ = node.attributes['id'].nodeValue + if node.nodeType == node.ELEMENT_NODE and node.tagName == "field": + id_ = node.attributes["id"].nodeValue field_to_xml(node, fields[id_], body, summary) else: body += [build.nroff.block_xml_to_nroff([node])] content = [ - '.bp\n', - '.SH \"%s\"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), + ".bp\n", + '.SH "%s"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), '.SS "Summary:"\n', - '.TS\n', - 'tab(;);\n', - 'l l l l l l l.\n', - 'Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n', - '\_;\_;\_;\_;\_;\_\n'] + ".TS\n", + "tab(;);\n", + "l l l l l l l.\n", + "Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n", + "\_;\_;\_;\_;\_;\_\n", + ] content += summary - content += ['.TE\n'] + content += [".TE\n"] content += body - return ''.join(content) + return "".join(content) + def make_oxm_classes_xml(document): - s = '''tab(;); + s = """tab(;); l l l. Prefix;Vendor;Class \_;\_;\_ -''' +""" for key in sorted(OXM_CLASSES, key=OXM_CLASSES.get): vendor, class_, class_type = OXM_CLASSES.get(key) - s += r"\fB%s\fR;" % key.rstrip('_') + s += r"\fB%s\fR;" % key.rstrip("_") if vendor: s += r"\fL0x%08x\fR;" % vendor else: s += "(none);" s += r"\fL0x%04x\fR;" % class_ s += "\n" - e = document.createElement('tbl') + e = document.createElement("tbl") e.appendChild(document.createTextNode(s)) return e + def recursively_replace(node, name, replacement): for child in node.childNodes: if child.nodeType == node.ELEMENT_NODE: @@ -691,11 +356,12 @@ def recursively_replace(node, name, replacement): else: recursively_replace(child, name, replacement) + def make_ovs_fields(meta_flow_h, meta_flow_xml): fields = extract_ofp_fields(meta_flow_h) fields_map = {} for f in fields: - fields_map[f['mff']] = f + fields_map[f["mff"]] = f document = xml.dom.minidom.parse(meta_flow_xml) doc = document.documentElement @@ -704,7 +370,8 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): if version == None: version = "UNKNOWN" - print('''\ + print( + """\ '\\" tp .\\" -*- mode: troff; coding: utf-8 -*- .TH "ovs\-fields" 7 "%s" "Open vSwitch" "Open vSwitch Manual" @@ -740,11 +407,13 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch . .PP -''' % version) +""" + % version + ) - recursively_replace(doc, 'oxm_classes', make_oxm_classes_xml(document)) + recursively_replace(doc, "oxm_classes", make_oxm_classes_xml(document)) - s = '' + s = "" for node in doc.childNodes: if node.nodeType == node.ELEMENT_NODE and node.tagName == "group": s += group_xml_to_nroff(node, fields_map) @@ -757,9 +426,10 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch for f in fields: if "used" not in f: - fatal("%s: field not documented " - "(please add documentation in lib/meta-flow.xml)" - % f["mff"]) + fatal( + "%s: field not documented " + "(please add documentation in lib/meta-flow.xml)" % f["mff"] + ) if n_errors: sys.exit(1) @@ -769,26 +439,27 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch # Life is easier with nroff if we don't try to feed it Unicode. # Fortunately, we only use a few characters outside the ASCII range. - oline = oline.replace(u'\u2208', r'\[mo]') - oline = oline.replace(u'\u2260', r'\[!=]') - oline = oline.replace(u'\u2264', r'\[<=]') - oline = oline.replace(u'\u2265', r'\[>=]') - oline = oline.replace(u'\u00d7', r'\[mu]') + oline = oline.replace(u"\u2208", r"\[mo]") + oline = oline.replace(u"\u2260", r"\[!=]") + oline = oline.replace(u"\u2264", r"\[<=]") + oline = oline.replace(u"\u2265", r"\[>=]") + oline = oline.replace(u"\u00d7", r"\[mu]") if len(oline): output += [oline] # nroff tends to ignore .bp requests if they come after .PP requests, # so remove .PPs that precede .bp. for i in range(len(output)): - if output[i] == '.bp': + if output[i] == ".bp": j = i - 1 - while j >= 0 and output[j] == '.PP': + while j >= 0 and output[j] == ".PP": output[j] = None j -= 1 for i in range(len(output)): if output[i] is not None: print(output[i]) - + + ## ------------ ## ## Main Program ## ## ------------ ## @@ -796,8 +467,9 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch if __name__ == "__main__": argv0 = sys.argv[0] try: - options, args = getopt.gnu_getopt(sys.argv[1:], 'h', - ['help', 'ovs-version=']) + options, args = getopt.gnu_getopt( + sys.argv[1:], "h", ["help", "ovs-version="] + ) except getopt.GetoptError as geo: sys.stderr.write("%s: %s\n" % (argv0, geo.msg)) sys.exit(1) @@ -805,32 +477,38 @@ if __name__ == "__main__": global version version = None for key, value in options: - if key in ['-h', '--help']: + if key in ["-h", "--help"]: usage() - elif key == '--ovs-version': + elif key == "--ovs-version": version = value else: sys.exit(0) if not args: - sys.stderr.write("%s: missing command argument " - "(use --help for help)\n" % argv0) + sys.stderr.write( + "%s: missing command argument " "(use --help for help)\n" % argv0 + ) sys.exit(1) - commands = {"meta-flow": (make_meta_flow, 1), - "nx-match": (make_nx_match, 1), - "ovs-fields": (make_ovs_fields, 2)} + commands = { + "meta-flow": (make_meta_flow, 1), + "nx-match": (make_nx_match, 1), + "ovs-fields": (make_ovs_fields, 2), + } if not args[0] in commands: - sys.stderr.write("%s: unknown command \"%s\" " - "(use --help for help)\n" % (argv0, args[0])) + sys.stderr.write( + '%s: unknown command "%s" ' + "(use --help for help)\n" % (argv0, args[0]) + ) sys.exit(1) func, n_args = commands[args[0]] if len(args) - 1 != n_args: - sys.stderr.write("%s: \"%s\" requires %d arguments but %d " - "provided\n" - % (argv0, args[0], n_args, len(args) - 1)) + sys.stderr.write( + '%s: "%s" requires %d arguments but %d ' + "provided\n" % (argv0, args[0], n_args, len(args) - 1) + ) sys.exit(1) func(*args[1:]) diff --git a/python/automake.mk b/python/automake.mk index 73438d615..54c2321a9 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -51,6 +51,7 @@ ovs_pyfiles = \ # so they are not installed. EXTRA_DIST += \ python/build/__init__.py \ + python/build/extract_ofp_fields.py \ python/build/nroff.py \ python/build/soutil.py diff --git a/python/build/extract_ofp_fields.py b/python/build/extract_ofp_fields.py new file mode 100644 index 000000000..f6938b6dd --- /dev/null +++ b/python/build/extract_ofp_fields.py @@ -0,0 +1,386 @@ +import getopt +import sys +import os.path +import re +import xml.dom.minidom +import build.nroff + +line = "" + +# Maps from user-friendly version number to its protocol encoding. +VERSION = {"1.0": 0x01, + "1.1": 0x02, + "1.2": 0x03, + "1.3": 0x04, + "1.4": 0x05, + "1.5": 0x06} +VERSION_REVERSE = dict((v,k) for k, v in VERSION.items()) + +TYPES = {"u8": (1, False), + "be16": (2, False), + "be32": (4, False), + "MAC": (6, False), + "be64": (8, False), + "be128": (16, False), + "tunnelMD": (124, True)} + +FORMATTING = {"decimal": ("MFS_DECIMAL", 1, 8), + "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), + "ct state": ("MFS_CT_STATE", 4, 4), + "Ethernet": ("MFS_ETHERNET", 6, 6), + "IPv4": ("MFS_IPV4", 4, 4), + "IPv6": ("MFS_IPV6", 16, 16), + "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), + "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), + "frag": ("MFS_FRAG", 1, 1), + "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), + "TCP flags": ("MFS_TCP_FLAGS", 2, 2), + "packet type": ("MFS_PACKET_TYPE", 4, 4)} + +PREREQS = {"none": "MFP_NONE", + "Ethernet": "MFP_ETHERNET", + "ARP": "MFP_ARP", + "VLAN VID": "MFP_VLAN_VID", + "IPv4": "MFP_IPV4", + "IPv6": "MFP_IPV6", + "IPv4/IPv6": "MFP_IP_ANY", + "NSH": "MFP_NSH", + "CT": "MFP_CT_VALID", + "MPLS": "MFP_MPLS", + "TCP": "MFP_TCP", + "UDP": "MFP_UDP", + "SCTP": "MFP_SCTP", + "ICMPv4": "MFP_ICMPV4", + "ICMPv6": "MFP_ICMPV6", + "ND": "MFP_ND", + "ND solicit": "MFP_ND_SOLICIT", + "ND advert": "MFP_ND_ADVERT"} + +# Maps a name prefix into an (experimenter ID, class) pair, so: +# +# - Standard OXM classes are written as (0, ) +# +# - Experimenter OXM classes are written as (, 0xffff) +# +# If a name matches more than one prefix, the longest one is used. +OXM_CLASSES = {"NXM_OF_": (0, 0x0000, 'extension'), + "NXM_NX_": (0, 0x0001, 'extension'), + "NXOXM_NSH_": (0x005ad650, 0xffff, 'extension'), + "OXM_OF_": (0, 0x8000, 'standard'), + "OXM_OF_PKT_REG": (0, 0x8001, 'standard'), + "ONFOXM_ET_": (0x4f4e4600, 0xffff, 'standard'), + "ERICOXM_OF_": (0, 0x1000, 'extension'), + + # This is the experimenter OXM class for Nicira, which is the + # one that OVS would be using instead of NXM_OF_ and NXM_NX_ + # if OVS didn't have those grandfathered in. It is currently + # used only to test support for experimenter OXM, since there + # are barely any real uses of experimenter OXM in the wild. + "NXOXM_ET_": (0x00002320, 0xffff, 'extension')} + +def oxm_name_to_class(name): + prefix = '' + class_ = None + for p, c in OXM_CLASSES.items(): + if name.startswith(p) and len(p) > len(prefix): + prefix = p + class_ = c + return class_ + + +def is_standard_oxm(name): + oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) + return oxm_class_type == 'standard' + + +def get_line(): + global line + global line_number + line = input_file.readline() + line_number += 1 + if line == "": + fatal("unexpected end of input") + + +n_errors = 0 + + +def error(msg): + global n_errors + sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) + n_errors += 1 + + +def fatal(msg): + error(msg) + sys.exit(1) + +def parse_oxms(s, prefix, n_bytes): + if s == 'none': + return () + + return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(',')) + + +match_types = dict() + + +def parse_oxm(s, prefix, n_bytes): + global match_types + + m = re.match('([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$', s) + if not m: + fatal("%s: syntax error parsing %s" % (s, prefix)) + + name, oxm_type, of_version, ovs_version = m.groups() + + class_ = oxm_name_to_class(name) + if class_ is None: + fatal("unknown OXM class for %s" % name) + oxm_vendor, oxm_class, oxm_class_type = class_ + + if class_ in match_types: + if oxm_type in match_types[class_]: + fatal("duplicate match type for %s (conflicts with %s)" % + (name, match_types[class_][oxm_type])) + else: + match_types[class_] = dict() + match_types[class_][oxm_type] = name + + # Normally the oxm_length is the size of the field, but for experimenter + # OXMs oxm_length also includes the 4-byte experimenter ID. + oxm_length = n_bytes + if oxm_class == 0xffff: + oxm_length += 4 + + header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) + + if of_version: + if oxm_class_type == 'extension': + fatal("%s: OXM extension can't have OpenFlow version" % name) + if of_version not in VERSION: + fatal("%s: unknown OpenFlow version %s" % (name, of_version)) + of_version_nr = VERSION[of_version] + if of_version_nr < VERSION['1.2']: + fatal("%s: claimed version %s predates OXM" % (name, of_version)) + else: + if oxm_class_type == 'standard': + fatal("%s: missing OpenFlow version number" % name) + of_version_nr = 0 + + return (header, name, of_version_nr, ovs_version) + + +def parse_field(mff, comment): + f = {'mff': mff} + + # First line of comment is the field name. + m = re.match(r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0]) + if not m: + fatal("%s lacks field name" % mff) + f['name'], f['extra_name'] = m.groups() + + # Find the last blank line the comment. The field definitions + # start after that. + blank = None + for i in range(len(comment)): + if not comment[i]: + blank = i + if not blank: + fatal("%s: missing blank line in comment" % mff) + + d = {} + for key in ("Type", "Maskable", "Formatting", "Prerequisites", + "Access", "Prefix lookup member", + "OXM", "NXM", "OF1.0", "OF1.1"): + d[key] = None + for fline in comment[blank + 1:]: + m = re.match(r'([^:]+):\s+(.*)\.$', fline) + if not m: + fatal("%s: syntax error parsing key-value pair as part of %s" + % (fline, mff)) + key, value = m.groups() + if key not in d: + fatal("%s: unknown key" % key) + elif key == 'Code point': + d[key] += [value] + elif d[key] is not None: + fatal("%s: duplicate key" % key) + d[key] = value + for key, value in d.items(): + if not value and key not in ("OF1.0", "OF1.1", + "Prefix lookup member", "Notes"): + fatal("%s: missing %s" % (mff, key)) + + m = re.match(r'([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$', d['Type']) + if not m: + fatal("%s: syntax error in type" % mff) + type_ = m.group(1) + if type_ not in TYPES: + fatal("%s: unknown type %s" % (mff, d['Type'])) + + f['n_bytes'] = TYPES[type_][0] + if m.group(2): + f['n_bits'] = int(m.group(2)) + if f['n_bits'] > f['n_bytes'] * 8: + fatal("%s: more bits (%d) than field size (%d)" + % (mff, f['n_bits'], 8 * f['n_bytes'])) + else: + f['n_bits'] = 8 * f['n_bytes'] + f['variable'] = TYPES[type_][1] + + if d['Maskable'] == 'no': + f['mask'] = 'MFM_NONE' + elif d['Maskable'] == 'bitwise': + f['mask'] = 'MFM_FULLY' + else: + fatal("%s: unknown maskable %s" % (mff, d['Maskable'])) + + fmt = FORMATTING.get(d['Formatting']) + if not fmt: + fatal("%s: unknown format %s" % (mff, d['Formatting'])) + f['formatting'] = d['Formatting'] + if f['n_bytes'] < fmt[1] or f['n_bytes'] > fmt[2]: + fatal("%s: %d-byte field can't be formatted as %s" + % (mff, f['n_bytes'], d['Formatting'])) + f['string'] = fmt[0] + + f['prereqs'] = d['Prerequisites'] + if f['prereqs'] not in PREREQS: + fatal("%s: unknown prerequisites %s" % (mff, d['Prerequisites'])) + + if d['Access'] == 'read-only': + f['writable'] = False + elif d['Access'] == 'read/write': + f['writable'] = True + else: + fatal("%s: unknown access %s" % (mff, d['Access'])) + + f['OF1.0'] = d['OF1.0'] + if not d['OF1.0'] in (None, 'exact match', 'CIDR mask'): + fatal("%s: unknown OF1.0 match type %s" % (mff, d['OF1.0'])) + + f['OF1.1'] = d['OF1.1'] + if not d['OF1.1'] in (None, 'exact match', 'bitwise mask'): + fatal("%s: unknown OF1.1 match type %s" % (mff, d['OF1.1'])) + + f['OXM'] = (parse_oxms(d['OXM'], 'OXM', f['n_bytes']) + + parse_oxms(d['NXM'], 'NXM', f['n_bytes'])) + + f['prefix'] = d["Prefix lookup member"] + + return f + +def extract_ofp_fields(fn): + global file_name + global input_file + global line_number + global line + + file_name = fn + input_file = open(file_name) + line_number = 0 + + fields = [] + + while True: + get_line() + if re.match('enum.*mf_field_id', line): + break + + while True: + get_line() + first_line_number = line_number + here = '%s:%d' % (file_name, line_number) + if (line.startswith('/*') + or line.startswith(' *') + or line.startswith('#') + or not line + or line.isspace()): + continue + elif re.match('}', line) or re.match('\s+MFF_N_IDS', line): + break + + # Parse the comment preceding an MFF_ constant into 'comment', + # one line to an array element. + line = line.strip() + if not line.startswith('/*'): + fatal("unexpected syntax between fields") + line = line[1:] + comment = [] + end = False + while not end: + line = line.strip() + if line.startswith('*/'): + get_line() + break + if not line.startswith('*'): + fatal("unexpected syntax within field") + + line = line[1:] + if line.startswith(' '): + line = line[1:] + if line.startswith(' ') and comment: + continuation = True + line = line.lstrip() + else: + continuation = False + + if line.endswith('*/'): + line = line[:-2].rstrip() + end = True + else: + end = False + + if continuation: + comment[-1] += " " + line + else: + comment += [line] + get_line() + + # Drop blank lines at each end of comment. + while comment and not comment[0]: + comment = comment[1:] + while comment and not comment[-1]: + comment = comment[:-1] + + # Parse the MFF_ constant(s). + mffs = [] + while True: + m = re.match('\s+(MFF_[A-Z0-9_]+),?\s?$', line) + if not m: + break + mffs += [m.group(1)] + get_line() + if not mffs: + fatal("unexpected syntax looking for MFF_ constants") + + if len(mffs) > 1 or '' in comment[0]: + for mff in mffs: + # Extract trailing integer. + m = re.match('.*[^0-9]([0-9]+)$', mff) + if not m: + fatal("%s lacks numeric suffix in register group" % mff) + n = m.group(1) + + # Search-and-replace within the comment, + # and drop lines that have for x != n. + instance = [] + for x in comment: + y = x.replace('', n) + if re.search('<[0-9]+>', y): + if ('<%s>' % n) not in y: + continue + y = re.sub('<[0-9]+>', '', y) + instance += [y.strip()] + fields += [parse_field(mff, instance)] + else: + fields += [parse_field(mffs[0], comment)] + continue + + input_file.close() + + if n_errors: + sys.exit(1) + + return fields From patchwork Fri Jan 28 16:04:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585754 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JzCW0pCN; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3L63F7z9t3b for ; Sat, 29 Jan 2022 03:05:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2BE8184D1B; Fri, 28 Jan 2022 16:05:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hEWGvD3BVyvS; Fri, 28 Jan 2022 16:05:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id F2E7584CED; Fri, 28 Jan 2022 16:05:13 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 60038C002D; Fri, 28 Jan 2022 16:05:12 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id E9155C0080 for ; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C244884C0C for ; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5i8KFuTCTy3E for ; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id A88F284C27 for ; Fri, 28 Jan 2022 16:05:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385908; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QWv3R/K5OQu4mSeNpYCd5Q5meUCcxjhWQB7yH/LpzdE=; b=JzCW0pCNhUqWQmCywHU+4qBVEUEQAATbAuN09VmCJDGVTy6L0BdvqaSHsYdJ2YEF2DQGtu L5p7XU0yLQobAlWIbaq/t4FGkn16GHEd+aT6fHasj8kIY1YTrRgMq3ppt+tMirDjhFykZo apLA2toU43jUcTxAda5GT/9ONWOCdl4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-koULjoQGMbK7DpuZjqnxhQ-1; Fri, 28 Jan 2022 11:05:07 -0500 X-MC-Unique: koULjoQGMbK7DpuZjqnxhQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 11FD8802C87; Fri, 28 Jan 2022 16:05:06 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 37C0D84A34; Fri, 28 Jan 2022 16:05:04 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:28 +0100 Message-Id: <20220128160441.23477-6-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 05/18] build-aux: generate ofp field decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on meta-field information extracted by extract_ofp_fields, autogenerate the right decoder to be used. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- build-aux/automake.mk | 3 +- build-aux/gen_ofp_field_decoders | 69 ++++++++++++++++++++++++++++++++ python/.gitignore | 1 + python/automake.mk | 7 ++++ 4 files changed, 79 insertions(+), 1 deletion(-) create mode 100755 build-aux/gen_ofp_field_decoders diff --git a/build-aux/automake.mk b/build-aux/automake.mk index 6267ccd7c..a8bb0acfd 100644 --- a/build-aux/automake.mk +++ b/build-aux/automake.mk @@ -9,7 +9,8 @@ EXTRA_DIST += \ build-aux/sodepends.py \ build-aux/soexpand.py \ build-aux/text2c \ - build-aux/xml2nroff + build-aux/xml2nroff \ + build-aux/gen_ofp_field_decoders FLAKE8_PYFILES += \ $(srcdir)/build-aux/xml2nroff \ diff --git a/build-aux/gen_ofp_field_decoders b/build-aux/gen_ofp_field_decoders new file mode 100755 index 000000000..3cc480042 --- /dev/null +++ b/build-aux/gen_ofp_field_decoders @@ -0,0 +1,69 @@ +#!/bin/env python + +import argparse + +import build.extract_ofp_fields as extract_fields + + +def main(): + parser = argparse.ArgumentParser( + description="Tool to generate python ofproto field decoders from" + "meta-flow information" + ) + parser.add_argument( + "metaflow", + metavar="FILE", + type=str, + help="Read meta-flow info from file", + ) + + args = parser.parse_args() + + fields = extract_fields.extract_ofp_fields(args.metaflow) + + field_decoders = {} + for field in fields: + decoder = get_decoder(field) + field_decoders[field.get("name")] = decoder + if field.get("extra_name"): + field_decoders[field.get("extra_name")] = decoder + + code = """ +# This file is auto-generated. Do not edit! + +import functools +from ovs.flows import decoders + +field_decoders = {{ +{decoders} +}} +""".format( + decoders="\n".join( + [ + " '{name}': {decoder},".format(name=name, decoder=decoder) + for name, decoder in field_decoders.items() + ] + ) + ) + print(code) + + +def get_decoder(field): + formatting = field.get("formatting") + if formatting in ["decimal", "hexadecimal"]: + if field.get("mask") == "MFM_NONE": + return "decoders.decode_int" + else: + if field.get("n_bits") in [8, 16, 32, 64, 128, 992]: + return "decoders.Mask{}".format(field.get("n_bits")) + return "decoders.decode_mask({})".format(field.get("n_bits")) + elif formatting in ["IPv4", "IPv6"]: + return "decoders.IPMask" + elif formatting == "Ethernet": + return "decoders.EthMask" + else: + return "decoders.decode_default" + + +if __name__ == "__main__": + main() diff --git a/python/.gitignore b/python/.gitignore index 60ace6f05..c8ffd4574 100644 --- a/python/.gitignore +++ b/python/.gitignore @@ -1,2 +1,3 @@ dist/ *.egg-info +ovs/flows/ofp_fields.py diff --git a/python/automake.mk b/python/automake.mk index 54c2321a9..5f7a44518 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -123,3 +123,10 @@ $(srcdir)/python/ovs/dirs.py: python/ovs/dirs.py.template mv $@.tmp $@ EXTRA_DIST += python/ovs/dirs.py.template CLEANFILES += python/ovs/dirs.py + +$(srcdir)/python/ovs/flows/ofp_fields.py: $(srcdir)/build-aux/gen_ofp_field_decoders include/openvswitch/meta-flow.h + $(AM_V_GEN)$(run_python) $< $(srcdir)/include/openvswitch/meta-flow.h > $@.tmp + $(AM_V_at)mv $@.tmp $@ +EXTRA_DIST += python/ovs/flows/ofp_fields.py +CLEANFILES += python/ovs/flows/ofp_fields.py + From patchwork Fri Jan 28 16:04:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585758 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Ic36fP+8; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3Z69STz9t3b for ; Sat, 29 Jan 2022 03:05:30 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 824AD41D2C; Fri, 28 Jan 2022 16:05:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V3jEbxz2U74o; Fri, 28 Jan 2022 16:05:24 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 87DDA41CEA; Fri, 28 Jan 2022 16:05:19 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 57CC2C0011; Fri, 28 Jan 2022 16:05:19 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 14C56C0070 for ; Fri, 28 Jan 2022 16:05:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id DF80341CAE for ; Fri, 28 Jan 2022 16:05:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dAvv81AvK92D for ; Fri, 28 Jan 2022 16:05:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id C469341C75 for ; Fri, 28 Jan 2022 16:05:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385911; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aq0aY9n3if4B5Tt4LS2I91ENg9iVDUa3mh2kAIl8yyM=; b=Ic36fP+8A6RFqR+O/cgsEjYOgCol8Vl+YtJc2/em1n+JKuCK0i3Q9xXw8J2RjayXGPWy7a hsVNqZRfDRpD6PP4pLPOpKIu/2Bqhnku2K9Sr2SygGasEoPVjPORIhJDPtAJk5oBThIqfk GHH/G1jlO21wrcszwMA7cJU7ZtcEf+Y= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-621-B0EaAcoMM9GZe5lJSQ8eaA-1; Fri, 28 Jan 2022 11:05:09 -0500 X-MC-Unique: B0EaAcoMM9GZe5lJSQ8eaA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3D61B2F26; Fri, 28 Jan 2022 16:05:08 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 679BB84A2C; Fri, 28 Jan 2022 16:05:06 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:29 +0100 Message-Id: <20220128160441.23477-7-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 06/18] python: add flow base class X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It simplifies the implementation of different types of flows by creating the concept of Section (e.g: match, action) and automatic accessors for all the provided Sections Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flows/flow.py | 125 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 python/ovs/flows/flow.py diff --git a/python/automake.mk b/python/automake.mk index 5f7a44518..b7debfbd9 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ python/ovs/json.py \ diff --git a/python/ovs/flows/flow.py b/python/ovs/flows/flow.py new file mode 100644 index 000000000..2456d5f87 --- /dev/null +++ b/python/ovs/flows/flow.py @@ -0,0 +1,125 @@ +""" Defines the Flow class. +""" + + +class Section(object): + """A flow can be seen as composed of different sections, e.g: + + [info] [match] actions=[actions] + + This class represents each of those sections. + + A section is basically a set of Key-Value pairs. Typically, they can be + expressed as a dictionary, for instance the "match" part of a flow can be + expressed as: + { + "nw_src": "192.168.1.1", + "nw_dst": "192.168.1.2", + } + However, some of them must be expressed as a list which allows for + duplicated keys. For instance, the "actions" section could be: + [ + { + "output": 32 + }, + { + "output": 33 + } + ] + + The is_list flag is used to discriminate this. + + Attributes: + name (str): Name of the section. + pos (int): Position within the overall flow string. + string (str): Section string. + data (list[KeyValue]): Parsed data of the section. + is_list (bool): Whether the key-values shall be expressed as a list + (i.e: it allows repeated keys). + """ + + def __init__(self, name, pos, string, data, is_list=False): + self.name = name + self.pos = pos + self.string = string + self.data = data + self.is_list = is_list + + def __str__(self): + return "{} (at {}): {}".format(self.name, self.pos, self.string) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def dict(self): + return {self.name: self.format_data()} + + def format_data(self): + """Returns the section's key-values formatted in a dictionary or list + depending on the value of is_list flag. + """ + if self.is_list: + return [{item.key: item.value} for item in self.data] + else: + return {item.key: item.value for item in self.data} + + +class Flow(object): + """The Flow class is a base class for other types of concrete flows + (such as OFproto Flows or DPIF Flows). + + A flow is basically comprised of a number of sections. + For each section named {section_name}, the flow object will have the + following attributes: + - {section_name} will return the sections data in a formatted way. + - {section_name}_kv will return the sections data as a list of KeyValues. + + Args: + sections (list[Section]): List of sections that comprise the flow + orig (str): Original flow string. + id (Any): Optional; identifier that clients can use to uniqely identify + this flow. + """ + + def __init__(self, sections, orig="", id=None): + self._sections = sections + self._orig = orig + self._id = id + for section in sections: + setattr( + self, section.name, self.section(section.name).format_data() + ) + setattr( + self, + "{}_kv".format(section.name), + self.section(section.name).data, + ) + + def section(self, name): + """Return the section by name.""" + return next( + (sect for sect in self._sections if sect.name == name), None + ) + + @property + def id(self): + """Return the Flow ID.""" + return self._id + + @property + def sections(self): + """Return the all the sections in a list.""" + return self._sections + + @property + def orig(self): + """Return the original flow string.""" + return self._orig + + def dict(self): + """Returns the Flow information in a dictionary.""" + flow_dict = {"orig": self.orig} + for section in self.sections: + flow_dict.update(section.dict()) + + return flow_dict From patchwork Fri Jan 28 16:04:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585759 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=VLngV3oS; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj3k0lPBz9t3b for ; Sat, 29 Jan 2022 03:05:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B56FA61102; Fri, 28 Jan 2022 16:05:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CECNmQcDCT7I; Fri, 28 Jan 2022 16:05:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3746461101; Fri, 28 Jan 2022 16:05:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F2672C0011; Fri, 28 Jan 2022 16:05:22 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 117BCC0011 for ; Fri, 28 Jan 2022 16:05:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5376F40C09 for ; Fri, 28 Jan 2022 16:05:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9bBt0rZWw5iv for ; Fri, 28 Jan 2022 16:05:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8ACCC40BE2 for ; Fri, 28 Jan 2022 16:05:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385914; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+45MYiO02hW16TtiNmvtI4nyvfSGjx3wOj/JfIhs+Yw=; b=VLngV3oSq0mETpqRcMwJCP1Xx62XQTIAgWzeujlm0Bh6bZhYZbI3XCclAUkRl31dcWvm3F fo+9aTeI+tcV0WuQf2JHl/UuVxB8RzBKU6w0JdpI/Wez2qJerjKuOWk4CZOoaMxlfojkEh fNdoemBDidQvk0tnJKXb4sGOhaSGYo0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-533--dGNhQ6iNtaIrScZiz0L3g-1; Fri, 28 Jan 2022 11:05:10 -0500 X-MC-Unique: -dGNhQ6iNtaIrScZiz0L3g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EBE0F18B9F08; Fri, 28 Jan 2022 16:05:09 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9FF0884A2A; Fri, 28 Jan 2022 16:05:08 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:30 +0100 Message-Id: <20220128160441.23477-8-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 07/18] python: introduce OpenFlow Flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Introduce OFPFlow class and all its decoders. Most of the decoders are generic (from decoders.py). Some have special syntax and need a specific implementation. Decoders for nat are moved to the common decoders.py because it's syntax is shared with other types of flows (e.g: dpif flows). Signed-off-by: Adrian Moreno --- python/automake.mk | 2 + python/ovs/flows/decoders.py | 108 +++++++++ python/ovs/flows/ofp.py | 453 +++++++++++++++++++++++++++++++++++ python/ovs/flows/ofp_act.py | 243 +++++++++++++++++++ 4 files changed, 806 insertions(+) create mode 100644 python/ovs/flows/ofp.py create mode 100644 python/ovs/flows/ofp_act.py diff --git a/python/automake.mk b/python/automake.mk index b7debfbd9..7b6d6596f 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,8 @@ ovs_pyfiles = \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ + python/ovs/flows/ofp.py \ + python/ovs/flows/ofp_act.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 2f8e5bd0a..1462b0b9d 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -6,6 +6,7 @@ object. """ import netaddr +import re class Decoder(object): @@ -414,3 +415,110 @@ class IPMask(Decoder): def to_json(self): return str(self) + + +def decode_free_output(value): + """The value of the output action can be found free, i.e: without the + 'output' keyword. This decoder decodes its value when found this way.""" + try: + return "output", {"port": int(value)} + except ValueError: + return "output", {"port": value.strip('"')} + + +ipv4 = r"(?:\d{1,3}.?){3}\d{1,3}" +ipv4_capture = r"({ipv4})".format(ipv4=ipv4) +""" +The following IPv6 regexp is a modified version of the one in: +https://community.helpsystems.com/forums/intermapper/miscellaneous-topics/5acc4fcf-fa83-e511-80cf-0050568460e4?_ga=2.113564423.1432958022.1523882681-2146416484.1523557976 + +It matches all these types of ipv6 addresses: +fe80:0000:0000:0000:0204:61ff:fe9d:f156 +fe80:0:0:0:204:61ff:fe9d:f156 +fe80::204:61ff:fe9d:f156 +fe80:0000:0000:0000:0204:61ff:254.157.241.86 +fe80:0:0:0:0204:61ff:254.157.241.86 +fe80::204:61ff:254.157.241.86 +::1 +2001:: +fe80:: +""" +ipv6 = r"(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}(?:[0-9A-Fa-f]{1,4}|:))|(?:(?:[0-9A-Fa-f]{1,4}:){6}(?::[0-9A-Fa-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[0-9A-Fa-f]{1,4}:){5}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,2})|:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[0-9A-Fa-f]{1,4}:){4}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,3})|(?:(?::[0-9A-Fa-f]{1,4})?:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){3}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,4})|(?:(?::[0-9A-Fa-f]{1,4}){0,2}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){2}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,5})|(?:(?::[0-9A-Fa-f]{1,4}){0,3}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){1}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|(?:(?::[0-9A-Fa-f]{1,4}){0,4}:(?:(?:25[0-5]|2[0-4]\ d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?::(?:(?:(?::[0-9A-Fa-f]{1,4}){1,7})|(?:(?::[0-9A-Fa-f]{1,4}){0,5}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))" # noqa: E501 +ipv6_capture = r"(?:\[*)?({ipv6})(?:\]*)?".format(ipv6=ipv6) +port_range = r":(\d+)(?:-(\d+))?" +ip_range_regexp = r"{ip_cap}(?:-{ip_cap})?(?:{port_range})?" +ipv4_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv4_capture, port_range=port_range) +) +ipv6_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv6_capture, port_range=port_range) +) + + +def decode_ip_port_range(value): + """ + Decodes an IP and port range: + {ip_start}-{ip-end}:{port_start}-{port_end} + + IPv6 addresses are surrounded by "[" and "]" if port ranges are also + present + + Returns the following dictionary: + { + "addrs": { + "start": {ip_start} + "end": {ip_end} + } + "ports": { + "start": {port_start}, + "end": {port_end} + } + (the "ports" key might be omitted) + """ + if value.count(":") > 1: + match = ipv6_port_regex.match(value) + else: + match = ipv4_port_regex.match(value) + + ip_start = match.group(1) + ip_end = match.group(2) + port_start = match.group(3) + port_end = match.group(4) + + result = { + "addrs": { + "start": netaddr.IPAddress(ip_start), + "end": netaddr.IPAddress(ip_end or ip_start), + } + } + if port_start: + result["ports"] = { + "start": int(port_start), + "end": int(port_end or port_start), + } + + return result + + +def decode_nat(value): + """Decodes the 'nat' keyword of the ct action""" + if not value: + return True + + result = dict() + type_parts = value.split("=") + result["type"] = type_parts[0] + + if len(type_parts) > 1: + value_parts = type_parts[1].split(",") + if len(type_parts) != 2: + raise ValueError("Malformed nat action: %s" % value) + + ip_port_range = decode_ip_port_range(value_parts[0]) + + result = {"type": type_parts[0], **ip_port_range} + + for flag in value_parts[1:]: + result[flag] = True + + return result diff --git a/python/ovs/flows/ofp.py b/python/ovs/flows/ofp.py new file mode 100644 index 000000000..1af06fa01 --- /dev/null +++ b/python/ovs/flows/ofp.py @@ -0,0 +1,453 @@ +""" Defines the parsers needed to parse ofproto flows. +""" + +import functools + +from ovs.flows.kv import KVParser, KVDecoders, nested_kv_decoder +from ovs.flows.ofp_fields import field_decoders +from ovs.flows.flow import Flow, Section +from ovs.flows.list import ListDecoders, nested_list_decoder +from ovs.flows.decoders import ( + decode_default, + decode_flag, + decode_int, + decode_time, + decode_mask, + IPMask, + EthMask, + decode_free_output, + decode_nat, +) +from ovs.flows.ofp_act import ( + decode_output, + decode_field, + decode_controller, + decode_bundle, + decode_bundle_load, + decode_encap_ethernet, + decode_load_field, + decode_set_field, + decode_move_field, + decode_dec_ttl, + decode_chk_pkt_larger, + decode_zone, + decode_exec, + decode_learn, +) + + +class OFPFlow(Flow): + """OFPFLow represents an OpenFlow Flow. + + Attributes: + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not OFPFlow._info_decoders: + OFPFlow._info_decoders = OFPFlow._gen_info_decoders() + return OFPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not OFPFlow._match_decoders: + OFPFlow._match_decoders = OFPFlow._gen_match_decoders() + return OFPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not OFPFlow._action_decoders: + OFPFlow._action_decoders = OFPFlow._gen_action_decoders() + return OFPFlow._action_decoders + + def __init__(self, ofp_string, id=None): + """Create a OFPFlow from a flow string. + + The string is expected to have the follwoing format: + + [flow data] [match] actions=[actions] + + Args: + ofp_string(str): An OpenFlow flow string. + id(Any): Optional; any object used to uniquely identify this flow + from the rest. + + Returns + An OFPFlow with the content of the flow string or None if there is + no flow information but the string is expected to be found in a a + flow dump. + + Raises + ValueError if the string is malformed. + ParseError if an error in parsing occurs. + """ + if " reply " in ofp_string: + return None + + sections = list() + parts = ofp_string.split("actions=") + if len(parts) != 2: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + actions = parts[1] + + field_parts = parts[0].rstrip(" ").rpartition(" ") + if len(field_parts) != 3: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + info = field_parts[0] + match = field_parts[2] + + iparser = KVParser(info, OFPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=ofp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, OFPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=ofp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, OFPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=ofp_string.find(actions), + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(OFPFlow, self).__init__(sections, ofp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + args = { + "table": decode_int, + "duration": decode_time, + "n_packet": decode_int, + "n_bytes": decode_int, + "cookie": decode_int, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "hard_age": decode_time, + } + return KVDecoders(args) + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + args = { + **OFPFlow._field_decoder_args(), + **OFPFlow._extra_match_decoder_args(), + } + + return KVDecoders(args) + + @staticmethod + def _extra_match_decoder_args(): + """Returns the extra KVDecoder arguments needed to decode the match + part of a flow (apart from the fields).""" + return { + "priority": decode_int, + } + + @staticmethod + def _field_decoder_args(): + """Returns the KVDecoder arguments needed to decode match fields + decoding match fields. + """ + shorthands = [ + "eth", + "ip", + "ipv6", + "icmp", + "icmp6", + "tcp", + "tcp6", + "udp", + "udp6", + "sctp", + "arp", + "rarp", + "mpls", + "mplsm", + ] + + fields = {**field_decoders, **{key: decode_flag for key in shorthands}} + + # vlan_vid field is special. Although it is technically 12 bit wide, + # bit 12 is allowed to be set to 1 to indicate that the vlan header is + # present (see section VLAN FIELDS in + # http://www.openvswitch.org/support/dist-docs/ovs-fields.7.txt) + # Therefore, override the generated vlan_vid field size. + fields["vlan_vid"] = decode_mask(13) + return fields + + @staticmethod + def _gen_action_decoders(): + """Generate the actions decoders.""" + + actions = { + **OFPFlow._output_actions_decoders_args(), + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + **OFPFlow._fw_action_decoders_args(), + **OFPFlow._control_action_decoders_args(), + **OFPFlow._other_action_decoders_args(), + } + clone_actions = OFPFlow._clone_actions_decoders_args(actions) + actions.update(clone_actions) + return KVDecoders(actions, default_free=decode_free_output) + + @staticmethod + def _output_actions_decoders_args(): + """Returns the decoder arguments for the output actions.""" + return { + "output": decode_output, + "drop": decode_flag, + "controller": decode_controller, + "enqueue": nested_list_decoder( + ListDecoders([("port", decode_default), ("queue", int)]), + delims=[",", ":"], + ), + "bundle": decode_bundle, + "bundle_load": decode_bundle_load, + "group": decode_default, + } + + @staticmethod + def _encap_actions_decoders_args(): + """Returns the decoders arguments for the encap actions.""" + + return { + "pop_vlan": decode_flag, + "strip_vlan": decode_flag, + "push_vlan": decode_default, + "decap": decode_flag, + "encap": nested_kv_decoder( + KVDecoders( + { + "nsh": nested_kv_decoder( + KVDecoders( + { + "md_type": decode_default, + "tlv": nested_list_decoder( + ListDecoders( + [ + ("class", decode_int), + ("type", decode_int), + ("value", decode_int), + ] + ) + ), + } + ) + ), + }, + default=None, + default_free=decode_encap_ethernet, + ) + ), + } + + @staticmethod + def _field_action_decoders_args(): + """Returns the decoders arguments for field-modification actions.""" + # Field modification actions + field_default_decoders = [ + "set_mpls_label", + "set_mpls_tc", + "set_mpls_ttl", + "mod_nw_tos", + "mod_nw_ecn", + "mod_tcp_src", + "mod_tcp_dst", + ] + return { + "load": decode_load_field, + "set_field": functools.partial( + decode_set_field, KVDecoders(OFPFlow._field_decoder_args()) + ), + "move": decode_move_field, + "mod_dl_dst": EthMask, + "mod_dl_src": EthMask, + "mod_nw_dst": IPMask, + "mod_nw_src": IPMask, + "dec_ttl": decode_dec_ttl, + "dec_mpls_ttl": decode_flag, + "dec_nsh_ttl": decode_flag, + "check_pkt_larger": decode_chk_pkt_larger, + **{field: decode_default for field in field_default_decoders}, + } + + @staticmethod + def _meta_action_decoders_args(): + """Returns the decoders arguments for the metadata actions.""" + meta_default_decoders = ["set_tunnel", "set_tunnel64", "set_queue"] + return { + "pop_queue": decode_flag, + **{field: decode_default for field in meta_default_decoders}, + } + + @staticmethod + def _fw_action_decoders_args(): + """Returns the decoders arguments for the firewalling actions.""" + return { + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "zone": decode_zone, + "table": decode_int, + "nat": decode_nat, + "force": decode_flag, + "exec": functools.partial( + decode_exec, + KVDecoders( + { + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + } + ), + ), + "alg": decode_default, + } + ) + ), + "ct_clear": decode_flag, + } + + @staticmethod + def _control_action_decoders_args(): + return { + "resubmit": nested_list_decoder( + ListDecoders( + [ + ("port", decode_default), + ("table", decode_int), + ("ct", decode_flag), + ] + ) + ), + "push": decode_field, + "pop": decode_field, + "exit": decode_flag, + "multipath": nested_list_decoder( + ListDecoders( + [ + ("fields", decode_default), + ("basis", decode_int), + ("algorithm", decode_default), + ("n_links", decode_int), + ("arg", decode_int), + ("dst", decode_field), + ] + ) + ), + } + + @staticmethod + def _clone_actions_decoders_args(action_decoders): + """Generate the decoder arguments for the clone actions. + + Args: + action_decoders (dict): The decoders of the supported nested + actions. + """ + return { + "learn": decode_learn( + { + **action_decoders, + "fin_timeout": nested_kv_decoder( + KVDecoders( + { + "idle_timeout": decode_time, + "hard_timeout": decode_time, + } + ) + ), + } + ), + "clone": functools.partial( + decode_exec, KVDecoders(action_decoders) + ), + } + + @staticmethod + def _other_action_decoders_args(): + """Generate the decoder arguments for other actions + (see man(7) ovs-actions).""" + return { + "conjunction": nested_list_decoder( + ListDecoders( + [("id", decode_int), ("k", decode_int), ("n", decode_int)] + ), + delims=[",", "/"], + ), + "note": decode_default, + "sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_set_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "sampling_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + } diff --git a/python/ovs/flows/ofp_act.py b/python/ovs/flows/ofp_act.py new file mode 100644 index 000000000..9e04445ba --- /dev/null +++ b/python/ovs/flows/ofp_act.py @@ -0,0 +1,243 @@ +""" Defines decoders for openflow actions. +""" + +import functools + +from ovs.flows.kv import nested_kv_decoder, KVDecoders, KeyValue, KVParser +from ovs.flows.decoders import ( + decode_default, + decode_time, + decode_flag, + decode_int, +) +from ovs.flows.ofp_fields import field_decoders + + +def decode_output(value): + """Decodes the output value. + + Does not support field specification. + """ + if len(value.split(",")) > 1: + return nested_kv_decoder()(value) + try: + return {"port": int(value)} + except ValueError: + return {"port": value.strip('"')} + + +def decode_controller(value): + """Decodes the controller action.""" + if not value: + return KeyValue("output", "controller") + else: + # Try controller:max_len + try: + max_len = int(value) + return { + "max_len": max_len, + } + except ValueError: + pass + # controller(key[=val], ...) + return nested_kv_decoder()(value) + + +def decode_bundle_load(value): + return decode_bundle(value, True) + + +def decode_bundle(value, load=False): + """Decode bundle action.""" + result = {} + keys = ["fields", "basis", "algorithm", "ofport"] + if load: + keys.append("dst") + + for key in keys: + parts = value.partition(",") + nvalue = parts[0] + value = parts[2] + if key == "ofport": + continue + result[key] = decode_default(nvalue) + + # Handle members: + mvalues = value.split("members:") + result["members"] = [int(port) for port in mvalues[1].split(",")] + return result + + +def decode_encap_ethernet(value): + """Decodes encap ethernet value.""" + return "ethernet", int(value, 0) + + +def decode_field(value): + """Decodes a field as defined in the 'Field Specification' of the actions + man page: + http://www.openvswitch.org/support/dist-docs/ovs-actions.7.txt.""" + parts = value.strip("]\n\r").split("[") + result = { + "field": parts[0], + } + + if len(parts) > 1 and parts[1]: + field_range = parts[1].split("..") + start = field_range[0] + end = field_range[1] if len(field_range) > 1 else start + if start: + result["start"] = int(start) + if end: + result["end"] = int(end) + + return result + + +def decode_load_field(value): + """Decodes LOAD actions such as: 'load:value->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed load action : %s" % value) + + # If the load action is performed within a learn() action, + # The value can be specified as another field. + try: + return {"value": int(parts[0], 0), "dst": decode_field(parts[1])} + except ValueError: + return {"src": decode_field(parts[0]), "dst": decode_field(parts[1])} + + +def decode_set_field(field_decoders, value): + """Decodes SET_FIELD actions such as: 'set_field:value/mask->dst'. + + The value is decoded by field_decoders which is a KVDecoders instance. + Args: + field_decoders(KVDecoders): The KVDecoders to be used to decode the + field. + """ + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed set_field action : %s" % value) + + val = parts[0] + dst = parts[1] + + val_result = field_decoders.decode(dst, val) + + return { + "value": {val_result[0]: val_result[1]}, + "dst": decode_field(dst), + } + + +def decode_move_field(value): + """Decodes MOVE actions such as 'move:src->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed move action : %s" % value) + + return { + "src": decode_field(parts[0]), + "dst": decode_field(parts[1]), + } + + +def decode_dec_ttl(value): + """Decodes dec_ttl and dec_ttl(id, id[2], ...) actions.""" + if not value: + return True + return [int(idx) for idx in value.split(",")] + + +def decode_chk_pkt_larger(value): + """Decodes 'check_pkt_larger(pkt_len)->dst' actions.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed check_pkt_larger action : %s" % value) + + pkt_len = int(parts[0].strip("()")) + dst = decode_field(parts[1]) + return {"pkt_len": pkt_len, "dst": dst} + + +# CT decoders +def decode_zone(value): + """Decodes the value of the 'zone' keyword (part of the ct action).""" + try: + return int(value, 0) + except ValueError: + pass + return decode_field(value) + + +def decode_exec(action_decoders, value): + """Decodes the value of the 'exec' keyword (part of the ct action). + + Args: + decode_actions (KVDecoders): The decoders to be used to decode the + nested exec. + value (string): The string to be decoded. + """ + exec_parser = KVParser(value, action_decoders) + exec_parser.parse() + return [{kv.key: kv.value} for kv in exec_parser.kv()] + + +def decode_learn(action_decoders): + """Create the decoder to be used to decode the 'learn' action. + + The learn action has two added complexities: + 1) It can hold any valid action key-value. Therefore we must take + the precalculated action_decoders and use them. That's why we require + them as argument. + + 2) The way fields can be specified is augmented. Not only we have + 'field=value', but we also have: + - 'field=_src_' (where _src_ is another field name) + - and just 'field' + For this we need to create a wrapper of field_decoders that, for each + "field=X" key-value we check if X is a field_name or if it's acually + a value that we need to send to the appropriate field_decoder to + process. + + Args: + action_decoders (dict): Dictionary of decoders to be used in nested + action decoding. + """ + + def decode_learn_field(decoder, value): + """Generates a decoder to be used for the 'field' argument of the + 'learn' action. + + The field can hold a value that should be decoded, either as a field, + or as a the value (see man(7) ovs-actions). + + Args: + decoder (callable): The decoder. + """ + if value in field_decoders.keys(): + # It's a field + return value + else: + return decoder(value) + + learn_field_decoders = { + field: functools.partial(decode_learn_field, decoder) + for field, decoder in field_decoders.items() + } + learn_decoders = { + **action_decoders, + **learn_field_decoders, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "priority": decode_int, + "cookie": decode_int, + "send_flow_rem": decode_flag, + "table": decode_int, + "delete_learned": decode_flag, + "limit": decode_int, + "result_dst": decode_field, + } + + return functools.partial(decode_exec, KVDecoders(learn_decoders)) From patchwork Fri Jan 28 16:04:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585763 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=d2xJzxDd; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6D22n0z9t3b for ; Sat, 29 Jan 2022 03:07:48 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id BF6FC61150; Fri, 28 Jan 2022 16:07:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7b5_RzV161nn; Fri, 28 Jan 2022 16:07:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id B267161134; Fri, 28 Jan 2022 16:07:41 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id AC353C0021; Fri, 28 Jan 2022 16:07:40 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 23B58C0039 for ; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4AAA284D40 for ; Fri, 28 Jan 2022 16:06:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3sMwzzEppGfR for ; Fri, 28 Jan 2022 16:06:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id E9D5184D12 for ; Fri, 28 Jan 2022 16:06:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385966; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u7PiuaIRgTLyFXeH2FvSuqpulzCr1jKKxcUW13Sz608=; b=d2xJzxDdR79ACRJCy7YrDtNCLaVLaZpaIIEJJlPhMtEDMv2ZU+7cmPi2x314nk6QGY8CSj xJNOYW4S1+MOfJSTWgx43hCh2FEhUpu2Evemq++I2RL/yxgrblNS6V+1B70Vbv8qfJqQiO fyg5+UoKoVVAv8BldkF35Gnp1MKWfgg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-332-64Hjqo2jPTe9wg9_psMPvw-1; Fri, 28 Jan 2022 11:06:03 -0500 X-MC-Unique: 64Hjqo2jPTe9wg9_psMPvw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 65A39835B50; Fri, 28 Jan 2022 16:06:02 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 501FC84A2A; Fri, 28 Jan 2022 16:05:10 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:31 +0100 Message-Id: <20220128160441.23477-9-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 08/18] python: add ovs datapath flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A ODPFlow is a Flow with the following sections: ufid info (e.g: bytes, packets, dp, etc) match actions Use a factory class ODPFlowFactory to cache the decoder objects. Only three datapath actions require special handling: gre: because it has double parenthesis geneve: because it supports many concatenated lists of options nat: we reuse the decoder used for openflow actions Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flows/odp.py | 783 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 784 insertions(+) create mode 100644 python/ovs/flows/odp.py diff --git a/python/automake.mk b/python/automake.mk index 7b6d6596f..aa03a4820 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,7 @@ ovs_pyfiles = \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ + python/ovs/flows/odp.py \ python/ovs/flows/ofp.py \ python/ovs/flows/ofp_act.py \ python/ovs/json.py \ diff --git a/python/ovs/flows/odp.py b/python/ovs/flows/odp.py new file mode 100644 index 000000000..a86a0d6b1 --- /dev/null +++ b/python/ovs/flows/odp.py @@ -0,0 +1,783 @@ +""" Defines an Open vSwitch Datapath Flow. +""" +import re +from functools import partial + +from ovs.flows.flow import Flow, Section + +from ovs.flows.kv import ( + KVParser, + KVDecoders, + nested_kv_decoder, + decode_nested_kv, +) +from ovs.flows.decoders import ( + decode_default, + decode_time, + decode_int, + decode_mask, + Mask8, + Mask16, + Mask32, + Mask64, + Mask128, + IPMask, + EthMask, + decode_free_output, + decode_flag, + decode_nat, +) + + +class ODPFlow(Flow): + """ODPFLow represents a Open vSwitch Datapath flow. + + Attributes: + ufid: The UFID section with only one key-value, with keyword "ufid". + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not ODPFlow._info_decoders: + ODPFlow._info_decoders = ODPFlow._gen_info_decoders() + return ODPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not ODPFlow._match_decoders: + ODPFlow._match_decoders = ODPFlow._gen_match_decoders() + return ODPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not ODPFlow._action_decoders: + ODPFlow._action_decoders = ODPFlow._gen_action_decoders() + return ODPFlow._action_decoders + + def __init__(self, odp_string, id=None): + """Parse a odp flow string. + + The string is expected to have the follwoing format: + [ufid], [match] [flow data] actions:[actions] + + Args: + odp_string (str): A datapath flow string. + + Returns: + A ODPFlow instance. + """ + sections = [] + + # If UFID present, parse it and + ufid_pos = odp_string.find("ufid:") + if ufid_pos >= 0: + ufid_string = odp_string[ + ufid_pos : (odp_string[ufid_pos:].find(",") + 1) + ] + ufid_parser = KVParser( + ufid_string, KVDecoders({"ufid": decode_default}) + ) + ufid_parser.parse() + if len(ufid_parser.kv()) != 1: + raise ValueError("malformed odp flow: %s" % odp_string) + sections.append( + Section("ufid", ufid_pos, ufid_string, ufid_parser.kv()) + ) + + action_pos = odp_string.find("actions:") + if action_pos < 0: + raise ValueError("malformed odp flow: %s" % odp_string) + + # rest of the string is between ufid and actions + rest = odp_string[ + (ufid_pos + len(ufid_string) if ufid_pos >= 0 else 0) : action_pos + ] + + action_pos += 8 # len("actions:") + actions = odp_string[action_pos:] + + field_parts = rest.lstrip(" ").partition(" ") + + if len(field_parts) != 3: + raise ValueError("malformed odp flow: %s" % odp_string) + + match = field_parts[0] + info = field_parts[2] + + iparser = KVParser(info, ODPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=odp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, ODPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=odp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, ODPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=action_pos, + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(ODPFlow, self).__init__(sections, odp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "ufid: {}".format(self.ufid) if self.ufid else "" + string += "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + return KVDecoders(ODPFlow._info_decoders_args()) + + @staticmethod + def _info_decoders_args(): + """Generate the decoder args for the info KVDecoders.""" + return { + "packets": decode_int, + "bytes": decode_int, + "used": decode_time, + "flags": decode_default, + "dp": decode_default, + } + + @staticmethod + def _gen_action_decoders(): + """Generate the action KVDecoders.""" + return KVDecoders( + ODPFlow._action_decoders_args(), default_free=decode_free_output + ) + + @staticmethod + def _action_decoders_args(): + """Generate the arguments for the action KVDecoders.""" + _decoders = { + "drop": decode_flag, + "lb_output": decode_int, + "trunc": decode_int, + "recirc": decode_int, + "userspace": nested_kv_decoder( + KVDecoders( + { + "pid": decode_int, + "sFlow": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "output": decode_int, + } + ) + ), + "slow_path": decode_default, + "flow_sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_sed_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "output_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + "ipfix": nested_kv_decoder( + KVDecoders( + { + "output_port": decode_default, + } + ) + ), + "controller": nested_kv_decoder( + KVDecoders( + { + "reason": decode_int, + "dont_send": decode_int, + "continuation": decode_int, + "recirc_id": decode_int, + "rule_cookie": decode_int, + "controller_id": decode_int, + "max_len": decode_int, + } + ) + ), + "userdata": decode_default, + "actions": decode_flag, + "tunnel_out_port": decode_int, + "push_eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "type": decode_int, + } + ) + ), + "pop_eth": decode_flag, + } + ) + ), + "set": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + "push_vlan": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "cfi": decode_int, + "tpid": decode_int, + } + ) + ), + "pop_vlan": decode_flag, + "push_nsh": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "ttl": decode_int, + "mdtype": decode_int, + "np": decode_int, + "spi": decode_int, + "si": decode_int, + "c1": decode_int, + "c2": decode_int, + "c3": decode_int, + "c4": decode_int, + "md2": decode_int, + } + ) + ), + "pop_nsh": decode_flag, + "tnl_pop": decode_int, + "ct_clear": decode_flag, + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "force_commit": decode_flag, + "zone": decode_int, + "mark": Mask32, + "label": Mask128, + "helper": decode_default, + "timeout": decode_default, + "nat": decode_nat, + } + ) + ), + **ODPFlow._tnl_action_decoder_args(), + } + + _decoders["clone"] = nested_kv_decoder( + KVDecoders(decoders=_decoders, default_free=decode_free_output) + ) + + return { + **_decoders, + "sample": nested_kv_decoder( + KVDecoders( + { + "sample": (lambda x: float(x.strip("%"))), + "actions": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + "check_pkt_len": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "gt": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + "le": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + } + + @staticmethod + def _tnl_action_decoder_args(): + """Generate the decoder arguments for the tunnel actions.""" + return { + "tnl_push": nested_kv_decoder( + KVDecoders( + { + "tnl_port": decode_int, + "header": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "type": decode_int, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "dl_type": decode_int, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": decode_int, + "tos": decode_int, + "ttl": decode_int, + "frag": decode_int, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_int, + "proto": decode_int, + "tclass": decode_int, + "hlimit": decode_int, + } + ) + ), + "udp": nested_kv_decoder( + KVDecoders( + { + "src": decode_int, + "dst": decode_int, + "dsum": Mask16, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "vni": decode_int, + } + ) + ), + "geneve": nested_kv_decoder( + KVDecoders( + { + "oam": decode_flag, + "crit": decode_flag, + "vni": decode_int, + "options": partial( + decode_geneve, False + ), + } + ) + ), + "gre": decode_tnl_gre, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": decode_int, + "sid": decode_int, + "idx": decode_int, + "dir": decode_int, + "hwid": decode_int, + } + ) + ), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "msgtype": decode_int, + "teid": decode_int, + } + ) + ), + } + ) + ), + "out_port": decode_int, + } + ) + ) + } + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + return KVDecoders(ODPFlow._match_decoders_args()) + + @staticmethod + def _match_decoders_args(): + """Generate the arguments for the match KVDecoders.""" + return { + **ODPFlow._field_decoders_args(), + "encap": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + } + + @staticmethod + def _field_decoders_args(): + """Generate the decoder arguments for the match fields.""" + return { + "skb_priority": Mask32, + "skb_mark": Mask32, + "recirc_id": decode_int, + "dp_hash": Mask32, + "ct_state": decode_default, + "ct_zone": Mask16, + "ct_mark": Mask32, + "ct_label": Mask128, + "ct_tuple4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "ct_tuple6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "tunnel": nested_kv_decoder( + KVDecoders( + { + "tun_id": Mask64, + "src": IPMask, + "dst": IPMask, + "ipv6_src": IPMask, + "ipv6_dst": IPMask, + "tos": Mask8, + "ttl": Mask8, + "tp_src": Mask16, + "tp_dst": Mask16, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": Mask8, + "idx": Mask32, + "sid": decode_int, + "dir": Mask8, + "hwid": Mask8, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "gbp": nested_kv_decoder( + KVDecoders( + { + "id": Mask16, + "flags": Mask8, + } + ) + ) + } + ) + ), + "geneve": partial(decode_geneve, True), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "msgtype": Mask8, + } + ) + ), + "flags": decode_default, + } + ) + ), + "in_port": decode_default, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + } + ) + ), + "vlan": nested_kv_decoder( + KVDecoders( + { + "vid": Mask16, + "pcp": Mask16, + "cfi": Mask16, + } + ) + ), + "eth_type": Mask16, + "mpls": nested_kv_decoder( + KVDecoders( + { + "label": Mask32, + "tc": Mask32, + "ttl": Mask32, + "bos": Mask32, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tos": Mask8, + "ttl": Mask8, + "frag": decode_default, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_mask(20), + "proto": Mask8, + "tclass": Mask8, + "hlimit": Mask8, + "frag": decode_default, + } + ) + ), + "tcp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "tcp_flags": decode_default, + "udp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "sctp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "icmp": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "icmpv6": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "arp": nested_kv_decoder( + KVDecoders( + { + "sip": IPMask, + "tip": IPMask, + "op": Mask16, + "sha": EthMask, + "tha": EthMask, + } + ) + ), + "nd": nested_kv_decoder( + KVDecoders( + { + "target": IPMask, + "sll": EthMask, + "tll": EthMask, + } + ) + ), + "nd_ext": nested_kv_decoder( + KVDecoders( + { + "nd_reserved": Mask32, + "nd_options_type": Mask8, + } + ) + ), + "packet_type": nested_kv_decoder( + KVDecoders( + { + "ns": Mask16, + "id": Mask16, + } + ) + ), + "nsh": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "mdtype": Mask8, + "np": Mask8, + "spi": Mask32, + "si": Mask8, + "c1": Mask32, + "c2": Mask32, + "c3": Mask32, + "c4": Mask32, + } + ) + ), + } + + +def decode_geneve(mask, value): + """Decode geneve options. + Used for both tnl_push(header(geneve(options()))) action and + tunnel(geneve()) match. + + It has the following format: + + {class=0xffff,type=0x80,len=4,0xa} + + Args: + mask (bool): Whether masking is supported. + value (str): The value to decode. + """ + if mask: + decoders = { + "class": Mask16, + "type": Mask8, + "len": Mask8, + } + + def free_decoder(value): + return "data", Mask128(value) + + else: + decoders = { + "class": decode_int, + "type": decode_int, + "len": decode_int, + } + + def free_decoder(value): + return "data", decode_int(value) + + result = [] + for opts in re.findall(r"{.*?}", value): + result.append( + decode_nested_kv( + KVDecoders(decoders=decoders, default_free=free_decoder), + opts.strip("{}"), + ) + ) + return result + + +def decode_tnl_gre(value): + """ + Decode tnl_push(header(gre())) action. + + It has the following format: + + gre((flags=0x2000,proto=0x6558),key=0x1e241)) + + Args: + value (str): The value to decode. + """ + return decode_nested_kv( + KVDecoders( + { + "flags": decode_int, + "proto": decode_int, + "key": decode_int, + "csum": decode_int, + "seq": decode_int, + } + ), + value.replace("(", "").replace(")", ""), + ) From patchwork Fri Jan 28 16:04:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585768 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Td0SeVjf; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6k6c6Xz9t3b for ; Sat, 29 Jan 2022 03:08:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 88C3B84F51; Fri, 28 Jan 2022 16:08:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hdz1zqUvC_Z5; Fri, 28 Jan 2022 16:08:10 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 19E0F84F09; Fri, 28 Jan 2022 16:08:08 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BCE60C001A; Fri, 28 Jan 2022 16:08:06 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 22B0DC001A for ; Fri, 28 Jan 2022 16:08:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 02CEB41768 for ; Fri, 28 Jan 2022 16:06:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8C0smQiuDiHu for ; Fri, 28 Jan 2022 16:06:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0B95441CA3 for ; Fri, 28 Jan 2022 16:06:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385976; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BjWlHu3dAlMB6+g70SStrkP6tWhrSQEAdOum5ys3sPk=; b=Td0SeVjfZSiqnE3uHeRkwlPCSt9Kbl/2tWoekv0Yaig99pgkwsKVVfODSysiEzkjihTpiC xefSzyYe+UXLIjAZ2TfPCqjzQyAiouESUFn3sYXquv1hnfntKgBber9mjTXU7erp4GQC8Z OqJALWOWCo/SqykgqlgEieATm7coT1E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-594-Ix2j5_98NN2gDYZmA2HjjQ-1; Fri, 28 Jan 2022 11:06:14 -0500 X-MC-Unique: Ix2j5_98NN2gDYZmA2HjjQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1F4992F4C; Fri, 28 Jan 2022 16:06:13 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id C481884A34; Fri, 28 Jan 2022 16:06:02 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:32 +0100 Message-Id: <20220128160441.23477-10-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 09/18] python: add flow filtering syntax X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on pyparsing, create a very simple filtering syntax. It supports basic logic statements (and, &, or, ||, not, !), numerical operations (<, >), equality (=, !=) and masking (~=). The latter is only supported in certain fields (IntMask, EthMask, IPMask). Masking operation is semantically equivalent to "includes", therefore: ip_src ~= 192.168.1.1 means that ip_src field is either a host IP address equal to 192.168.1.1 or an IPMask that includes it (e.g: 192.168.1.1/24). Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flows/filter.py | 261 +++++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 3 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 python/ovs/flows/filter.py diff --git a/python/automake.mk b/python/automake.mk index aa03a4820..d56ac604b 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/filter.py \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ diff --git a/python/ovs/flows/filter.py b/python/ovs/flows/filter.py new file mode 100644 index 000000000..e520e90dc --- /dev/null +++ b/python/ovs/flows/filter.py @@ -0,0 +1,261 @@ +""" Defines a Flow Filtering syntax. +""" +import pyparsing as pp +import netaddr +from functools import reduce +from operator import and_, or_ + +from ovs.flows.decoders import ( + decode_default, + decode_int, + Decoder, + IPMask, + EthMask, +) + + +class EvaluationResult(object): + """An EvaluationResult is the result of an evaluation. It contains the + boolean result and the list of key-values that were evaluated. + + Note that since boolean operations (and, not, or) are based only on + __bool__ we use bitwise alternatives (&, ||, ~). + """ + + def __init__(self, result, *kv): + self.result = result + self.kv = kv if kv else list() + + def __and__(self, other): + """Logical and operation.""" + return EvaluationResult( + self.result and other.result, *self.kv, *other.kv + ) + + def __or__(self, other): + """Logical or operation.""" + return EvaluationResult( + self.result or other.result, *self.kv, *other.kv + ) + + def __invert__(self): + """Logical not operation.""" + return EvaluationResult(not self.result, *self.kv) + + def __bool__(self): + """Boolean operation.""" + return self.result + + def __repr__(self): + return "{} [{}]".format(self.result, self.kv) + + +class ClauseExpression(object): + """ A clause expression represents a specific expression in the filter. + + A clause has the following form: + [field] [operator] [value] + + Valid operators are: + = (equality) + != (inequality) + < (arithmetic less-than) + > (arithmetic more-than) + ~= (__contains__) + + When evaluated, the clause finds what relevant part of the flow to use for + evaluation, tries to translate the clause value to the relevant type and + performs the clause operation. + + Attributes: + field (str): The flow field used in the clause. + operator (str): The flow operator used in the clause. + value (str): The value to perform the comparison against. + """ + operators = {} + type_decoders = { + int: decode_int, + netaddr.IPAddress: IPMask, + netaddr.EUI: EthMask, + bool: bool, + } + + def __init__(self, tokens): + self.field = tokens[0] + self.value = "" + self.operator = "" + + if len(tokens) > 1: + self.operator = tokens[1] + self.value = tokens[2] + + def __repr__(self): + return "{}(field: {}, operator: {}, value: {})".format( + self.__class__.__name__, self.field, self.operator, self.value + ) + + def _find_data_in_kv(self, kv_list): + """Find a KeyValue for evaluation in a list of KeyValue. + + Args: + kv_list (list[KeyValue]): list of KeyValue to look into. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + """ + key_parts = self.field.split(".") + field = key_parts[0] + kvs = [kv for kv in kv_list if kv.key == field] + if not kvs: + return None + + for kv in kvs: + if kv.key == self.field: + # exact match + return (kv, kv.value) + if len(key_parts) > 1: + data = kv.value + for subkey in key_parts[1:]: + try: + data = data.get(subkey) + except Exception: + data = None + break + if not data: + break + if data: + return (kv, data) + return None + + def _find_keyval_to_evaluate(self, flow): + """Finds the key-value and data to use for evaluation on a flow. + + Args: + flow(Flow): The flow where the lookup is performed. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + + """ + for section in flow.sections: + data = self._find_data_in_kv(section.data) + if data: + return data + return None + + def evaluate(self, flow): + """Returns whether the clause is satisfied by the flow. + + Args: + flow (Flow): the flow to evaluate. + """ + result = self._find_keyval_to_evaluate(flow) + + if not result: + return EvaluationResult(False) + + keyval, data = result + + if not self.value and not self.operator: + # just asserting the existance of the key + return EvaluationResult(True, keyval) + + # Decode the value based on the type of data + if isinstance(data, Decoder): + decoder = data.__class__ + else: + decoder = self.type_decoders.get(data.__class__) or decode_default + + decoded_value = decoder(self.value) + + if self.operator == "=": + return EvaluationResult(decoded_value == data, keyval) + elif self.operator == "<": + return EvaluationResult(data < decoded_value, keyval) + elif self.operator == ">": + return EvaluationResult(data > decoded_value, keyval) + elif self.operator == "~=": + return EvaluationResult(decoded_value in data, keyval) + + +class BoolNot(object): + def __init__(self, t): + self.op, self.args = t[0] + + def __repr__(self): + return "NOT({})".format(self.args) + + def evaluate(self, flow): + return ~self.args.evaluate(flow) + + +class BoolAnd(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def __repr__(self): + return "AND({})".format(self.args) + + def evaluate(self, flow): + return reduce(and_, [arg.evaluate(flow) for arg in self.args]) + + +class BoolOr(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def evaluate(self, flow): + return reduce(or_, [arg.evaluate(flow) for arg in self.args]) + + def __repr__(self): + return "OR({})".format(self.args) + + +class OFFilter(object): + """OFFilter represents an Open vSwitch Flow Filter. + + It is built with a filter expression string composed of logically-separated + clauses (see ClauseExpression for details on the clause syntax). + + Args: + expr(str): String filter expression. + """ + w = pp.Word(pp.alphanums + "." + ":" + "_" + "/" + "-") + operators = ( + pp.Literal("=") + | pp.Literal("~=") + | pp.Literal("<") + | pp.Literal(">") + | pp.Literal("!=") + ) + + clause = (w + operators + w) | w + clause.setParseAction(ClauseExpression) + + statement = pp.infixNotation( + clause, + [ + ("!", 1, pp.opAssoc.RIGHT, BoolNot), + ("not", 1, pp.opAssoc.RIGHT, BoolNot), + ("&&", 2, pp.opAssoc.LEFT, BoolAnd), + ("and", 2, pp.opAssoc.LEFT, BoolAnd), + ("||", 2, pp.opAssoc.LEFT, BoolOr), + ("or", 2, pp.opAssoc.LEFT, BoolOr), + ], + ) + + def __init__(self, expr): + self._filter = self.statement.parseString(expr) + + def evaluate(self, flow): + """Evaluate whether the flow satisfies the filter. + + Args: + flow(Flow): a openflow or datapath flow. + + Returns: + An EvaluationResult with the result of the evaluation. + """ + return self._filter[0].evaluate(flow) diff --git a/python/setup.py b/python/setup.py index b06370bd9..4e8a9761a 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers', 'netaddr'], + install_requires=['sortedcontainers', 'netaddr', 'pyparsing'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Fri Jan 28 16:04:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585761 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HrjlzXPp; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj652vYhz9t3b for ; Sat, 29 Jan 2022 03:07:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 93FDC61118; Fri, 28 Jan 2022 16:07:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lYh02hkeCOg3; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id EB83E60AB8; Fri, 28 Jan 2022 16:07:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BA0BDC0021; Fri, 28 Jan 2022 16:07:37 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 927D9C000B for ; Fri, 28 Jan 2022 16:07:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EA1716112F for ; Fri, 28 Jan 2022 16:06:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5yNZBAZZsr7D for ; Fri, 28 Jan 2022 16:06:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 55F996112B for ; Fri, 28 Jan 2022 16:06:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385977; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ck1F5eJ0vobgxo3QXdqvJb7gVqQvMsxYffxHq3dSFX4=; b=HrjlzXPpZz82SRDuGxQjj4uRWRpTpxjOBSFpFtI545pCQNBJjAuZd+CtY879rmtVG7Wm0S JiUnbP5ze6rHS/FormMAWPooeeegrtZsV+9S3B9TOXzAsOcqYLvYpGieC7X4QF7F6GYU0L GEc5BGdbE3/ooHh5qBTrriqWAUI0dGo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-155-x6PHcx-WPiGHkP1Z-puDwg-1; Fri, 28 Jan 2022 11:06:15 -0500 X-MC-Unique: x6PHcx-WPiGHkP1Z-puDwg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ADFF218B9F00; Fri, 28 Jan 2022 16:06:14 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7216F84A3A; Fri, 28 Jan 2022 16:06:13 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:33 +0100 Message-Id: <20220128160441.23477-11-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 10/18] python: add a json encoder to flow fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The json encoder can be used to convert Flows to json. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/ovs/flows/decoders.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 1462b0b9d..4ef29905b 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -5,6 +5,7 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import json import netaddr import re @@ -522,3 +523,16 @@ def decode_nat(value): result[flag] = True return result + + +class FlowEncoder(json.JSONEncoder): + """FlowEncoder is a json.JSONEncoder instance that can be used to + serialize flow fields.""" + + def default(self, obj): + if isinstance(obj, Decoder): + return obj.to_json() + elif isinstance(obj, netaddr.IPAddress): + return str(obj) + + return json.JSONEncoder.default(self, obj) From patchwork Fri Jan 28 16:04:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585765 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=NLQN9RsZ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6X6Dx6z9t3b for ; Sat, 29 Jan 2022 03:08:04 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C2E1E61196; Fri, 28 Jan 2022 16:08:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vfVhQMZ7z3_P; Fri, 28 Jan 2022 16:07:56 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id CB7796115F; Fri, 28 Jan 2022 16:07:53 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id AB3D2C0039; Fri, 28 Jan 2022 16:07:52 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 18F79C002D for ; Fri, 28 Jan 2022 16:07:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7E03684C08 for ; Fri, 28 Jan 2022 16:06:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hZ6jHTVTy1q1 for ; Fri, 28 Jan 2022 16:06:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6E6A684CFE for ; Fri, 28 Jan 2022 16:06:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+ZB9Eu497AWOOPgM1TuVySQ3Lz9ZFta2d5lIYarine0=; b=NLQN9RsZU1qQVH2dtFfx6nA/GTYORajUX9jNqnPbkHL3dUJpdq6a5OuBwGa5cn4aEdU37N hjR2nOI1FVmOeMzf99WmSUt4x8iK1kOGDB+IWzdo494hGNvH5vqzpGcODyKl73sia4z9yg KDxIbCzHXoGatgSuvTPKKND/PDxzAog= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-343-eanPg613MTKkruecBw1Lqg-1; Fri, 28 Jan 2022 11:06:17 -0500 X-MC-Unique: eanPg613MTKkruecBw1Lqg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A0CEF100CCC0; Fri, 28 Jan 2022 16:06:16 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1451C84A3E; Fri, 28 Jan 2022 16:06:14 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:34 +0100 Message-Id: <20220128160441.23477-12-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 11/18] tests: wrap ovs-ofctl calls to test python parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Some ovs-ofctl commands are used to parse or dump openflow flows, specially in ofp-actions.at Use a wrapper around ovs-ofctl, called ovs-test-ofparse.py that, apart from calling ovs-ofctl, also parses its output (or input, depending on the command) to make sure the python flow parsing library can also parse the flows. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- tests/automake.mk | 3 ++ tests/ofp-actions.at | 46 +++++++++---------- tests/ovs-test-ofparse.py | 94 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 23 deletions(-) create mode 100755 tests/ovs-test-ofparse.py diff --git a/tests/automake.mk b/tests/automake.mk index 43731d097..b53a9093b 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,9 +19,11 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch + COMMON_MACROS_AT = \ tests/ovsdb-macros.at \ tests/ovs-macros.at \ @@ -520,6 +522,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ tests/test-jsonrpc.py \ diff --git a/tests/ofp-actions.at b/tests/ofp-actions.at index ef4898abb..0b79742be 100644 --- a/tests/ofp-actions.at +++ b/tests/ofp-actions.at @@ -327,7 +327,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow10 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow10 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -500,7 +500,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow11 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow11 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -735,7 +735,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow12 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow12 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -788,7 +788,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow13 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow13 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -817,16 +817,16 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow15 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow15 < input.txt], [0], [expout], [experr]) AT_CLEANUP AT_SETUP([ofp-actions - inconsistent MPLS actions]) OVS_VSWITCHD_START dnl OK: Use fin_timeout action on TCP flow -AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=fin_timeout(idle_timeout=1)']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 -vwarn add-flow br0 'tcp actions=fin_timeout(idle_timeout=1)']) dnl Bad: Use fin_timeout action on TCP flow that has been converted to MPLS -AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)'], +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 -vwarn add-flow br0 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)'], [1], [], [dnl ovs-ofctl: none of the usable flow formats (OpenFlow10,NXM) is among the allowed flow formats (OpenFlow11) ]) @@ -840,8 +840,8 @@ dnl In OpenFlow 1.3, set_field always sets all the bits in the field, dnl but when we translate to NXAST_LOAD we need to only set the bits that dnl actually exist (e.g. mpls_label only has 20 bits) otherwise OVS rejects dnl the "load" action as invalid. Check that we do this correctly. -AT_CHECK([ovs-ofctl -O OpenFlow13 add-flow br0 mpls,actions=set_field:10-\>mpls_label]) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow13 add-flow br0 mpls,actions=set_field:10-\>mpls_label]) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: mpls actions=load:0xa->OXM_OF_MPLS_LABEL[[]] ]) @@ -853,12 +853,12 @@ AT_KEYWORDS([ofp-actions]) OVS_VSWITCHD_START dnl OpenFlow 1.0 has an "enqueue" action. For OpenFlow 1.1+, we translate dnl it to a series of actions that accomplish the same thing. -AT_CHECK([ovs-ofctl -O OpenFlow10 add-flow br0 'actions=enqueue(123,456)']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 add-flow br0 'actions=enqueue(123,456)']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: actions=enqueue:123:456 ]) -AT_CHECK([ovs-ofctl -O OpenFlow13 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow13 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.3): reset_counts actions=set_queue:456,output:123,pop_queue ]) @@ -870,12 +870,12 @@ AT_KEYWORDS([ofp-actions]) OVS_VSWITCHD_START dnl OpenFlow 1.1+ have a mod_nw_ttl action. For OpenFlow 1.0, we translate dnl it to an Open vSwitch extension. -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ttl:123']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ttl:123']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: ip actions=load:0x7b->NXM_NX_IP_TTL[[]] ]) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ttl:123 ]) @@ -889,16 +889,16 @@ OVS_VSWITCHD_START dnl OpenFlow 1.1, but no other version, has a "mod_nw_ecn" action. dnl Check that we translate it properly for OF1.0 and OF1.2. dnl (OF1.3+ should be the same as OF1.2.) -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ecn:2']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ecn:2']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: ip actions=load:0x2->NXM_NX_IP_ECN[[]] ]) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ecn:2 ]) -AT_CHECK([ovs-ofctl -O OpenFlow12 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow12 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.2): ip actions=set_field:2->nw_ecn ]) @@ -911,8 +911,8 @@ dnl that anything that comes in as reg_load gets translated back to reg_load dnl on output. Perhaps this is somewhat inconsistent but it's what OVS dnl has done for multiple versions. AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow12 add-flow br0 'ip,actions=set_field:2->ip_ecn']) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow12 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ecn:2 ]) @@ -920,9 +920,9 @@ OFPST_FLOW reply (OF1.1): dnl Check that OF1.2+ set_field to set ECN is translated for earlier OF dnl versions. AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow10 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 add-flow br0 'ip,actions=set_field:2->ip_ecn']) AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=set_field:2->ip_ecn']) OVS_VSWITCHD_STOP AT_CLEANUP diff --git a/tests/ovs-test-ofparse.py b/tests/ovs-test-ofparse.py new file mode 100755 index 000000000..bf578b6d5 --- /dev/null +++ b/tests/ovs-test-ofparse.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 + +""" ovs-test-ofparse is just a wrapper around ovs-ofctl + that also runs the python flow parsing utility to check that flows are + parseable. +""" + +import subprocess +import sys +import re + +from ovs.flows.ofp import OFPFlow + +diff_regexp = re.compile(r"\d{2}: (\d{2}|\(none\)) -> (\d{2}|\(none\))$") + + +def run_ofctl(with_stdin): + cmd = sys.argv + cmd[0] = "ovs-ofctl" + if with_stdin: + p = subprocess.Popen( + cmd, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + input_data = sys.stdin.read() + out, err = p.communicate(input_data.encode("utf-8")) + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + else: + p = subprocess.Popen( + cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE + ) + out, err = p.communicate() + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + + +def main(): + flows = list() + return_code = 0 + + if "parse-actions" in sys.argv: + return_code, out, err = run_ofctl(True) + + out_lines = out.decode("utf-8").split("\n") + for line in out_lines: + if not ( + "bad" in line # skip "bad action at..." + or line.strip() == "" # skip empty lines + or diff_regexp.match(line) # skip differences + ): + flows.append(line) + + elif "add-flow" in sys.argv: + return_code, out, err = run_ofctl(False) + flows.append(sys.argv[-1]) + + elif "dump-flows" in sys.argv: + return_code, out, err = run_ofctl(False) + out_lines = out.decode("utf-8").split("\n") + + for line in out_lines: + if not ( + "reply" in line # skip NXST_FLOW reply: + or line.strip() == "" # skip empty lines + ): + flows.append(line) + else: + print("Unsupported command: {}".format(sys.argv)) + sys.exit(1) + + if return_code == 0: + for flow in flows: + try: + result_flow = OFPFlow(flow) + if flow != str(result_flow): + print("in: {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed") + except Exception as e: + print(e) + return 1 + + return return_code + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Fri Jan 28 16:04:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585766 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ggYPVSNH; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6d20g6z9t3b for ; Sat, 29 Jan 2022 03:08:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EB021611AC; Fri, 28 Jan 2022 16:08:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bVdmctsAznwC; Fri, 28 Jan 2022 16:08:05 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id D5FBE61163; Fri, 28 Jan 2022 16:08:02 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8FAC0C001A; Fri, 28 Jan 2022 16:08:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0FAF1C000B for ; Fri, 28 Jan 2022 16:08:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 10C3184C29 for ; Fri, 28 Jan 2022 16:06:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WrsUDMnI07iA for ; Fri, 28 Jan 2022 16:06:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id C75F184C27 for ; Fri, 28 Jan 2022 16:06:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643385998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qVeLcjpGMYIlEJ34gHvG8g+na7ReZkUnurr/fkqN2zQ=; b=ggYPVSNHQHGq1b/xgo1tTT570BOiT+yAZljCnkrI8faJ8L3GL3cVOIsenYM4uvHwfZtU9A T6AZTlK3Gjv3ic2hIPLYBDsZ7W4rLTB0qAhxdWqltwPT5trrXUDodD0lQQGk/9wYdsIm3g 4u5OyrgzE0zAlQMGnUXb2mQVMmDb4Ls= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-jbYXHqc9O1aaRHSDXwZxWw-1; Fri, 28 Jan 2022 11:06:35 -0500 X-MC-Unique: jbYXHqc9O1aaRHSDXwZxWw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4332B1091DA0; Fri, 28 Jan 2022 16:06:34 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id CC25A84A2A; Fri, 28 Jan 2022 16:06:16 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:35 +0100 Message-Id: <20220128160441.23477-13-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 12/18] tests: Wrap test-odp to also run python parsers X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" test-odp is used to parse datapath flow actions and matches within the odp tests. Wrap calls to this tool in a python script that also parses them using the python flow parsing library. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- tests/automake.mk | 2 ++ tests/odp.at | 36 ++++++++++---------- tests/ovs-test-dpparse.py | 70 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 90 insertions(+), 18 deletions(-) create mode 100755 tests/ovs-test-dpparse.py diff --git a/tests/automake.mk b/tests/automake.mk index b53a9093b..80422679e 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,6 +19,7 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-dpparse.py \ $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch @@ -522,6 +523,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-dpparse.py \ tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ diff --git a/tests/odp.at b/tests/odp.at index 07a5cfe39..69e86c5c1 100644 --- a/tests/odp.at +++ b/tests/odp.at @@ -105,7 +105,7 @@ sed -i'back' 's/\(skb_mark(0)\),\(ct\)/\1,ct_state(0),ct_zone(0),\2/' odp-out.tx sed -i'back' 's/\(skb_mark([[^)]]*)\),\(recirc\)/\1,ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),\2/' odp-out.txt sed -i'back' 's/\(in_port(1)\),\(eth\)/\1,packet_type(ns=0,id=0),\2/' odp-out.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-keys < odp-in.txt], [0], [`cat odp-out.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-keys < odp-in.txt], [0], [`cat odp-out.txt` ]) AT_CLEANUP @@ -192,7 +192,7 @@ sed -n 's/,frag=no),.*/,frag=later)/p' odp-base.txt sed 's/^/skb_priority(0),tunnel(tun_id=0xfedcba9876543210,src=10.0.0.1,dst=10.0.0.2,ttl=128,erspan(ver=2,dir=1,hwid=0x7/0xf),flags(df|key)),skb_mark(0),recirc_id(0),dp_hash(0),/' odp-base.txt ) > odp.txt AT_CAPTURE_FILE([odp.txt]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-wc-keys < odp.txt], [0], [`cat odp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-wc-keys < odp.txt], [0], [`cat odp.txt` ]) AT_CLEANUP @@ -239,25 +239,25 @@ AT_DATA([odp-tcp6.txt], [dnl in_port(1),eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15),eth_type(0x86dd),ipv6(src=::1/::255,dst=::2/::255,label=0/0xf0,proto=10/0xf0,tclass=0x70/0xf0,hlimit=128/0xf0,frag=no) in_port(1),eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15),eth_type(0x86dd),ipv6(src=::1,dst=::2,label=0,proto=6,tclass=0,hlimit=128,frag=no),tcp(src=80/0xff00,dst=8080/0xff) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x1235' < odp-base.txt], [0], [`cat odp-eth-type.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_type=0x1235' < odp-base.txt], [0], [`cat odp-eth-type.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_vlan=99' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99,ip' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_vlan=99,ip' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='ip,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='ip,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x0800,nw_src=35.8.2.199,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_type=0x0800,nw_src=35.8.2.199,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='icmp,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-icmp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='icmp,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-icmp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='arp,arp_spa=1.2.3.5' < odp-base.txt], [0], [`cat odp-arp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='arp,arp_spa=1.2.3.5' < odp-base.txt], [0], [`cat odp-arp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='tcp,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp6,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp6.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='tcp6,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp6.txt` ]) AT_CLEANUP @@ -385,14 +385,14 @@ check_pkt_len(size=200,gt(ct(nat)),le(drop)) check_pkt_len(size=200,gt(set(eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15))),le(set(eth(src=00:01:02:03:04:06,dst=10:11:12:13:14:16)))) lb_output(1) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [`cat actions.txt` ]) AT_CLEANUP AT_SETUP([OVS datapath actions parsing and formatting - invalid forms]) dnl This caused a hang in older versions. -AT_CHECK([echo 'encap_nsh@:{@' | ovstest test-odp parse-actions +AT_CHECK([echo 'encap_nsh@:{@' | ovs-test-dpparse.py ovstest test-odp parse-actions ], [0], [dnl odp_actions_from_string: error ]) @@ -427,7 +427,7 @@ data_invalid=$(printf '%*s' 131018 | tr ' ' "a") echo "userspace(pid=1234567,userdata(${data_valid}),tunnel_out_port=10)" >> actions.txt echo "userspace(pid=1234567,userdata(${data_invalid}),tunnel_out_port=10)" >> actions.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [dnl `cat actions.txt | head -1` odp_actions_from_string: error `cat actions.txt | head -3 | tail -1` @@ -443,7 +443,7 @@ actions=$(printf 'set(encap()),%.0s' $(seq 8190)) echo "${actions}set(encap())" > actions.txt echo "${actions}set(encap()),set(encap())" >> actions.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [dnl `cat actions.txt | head -1` odp_actions_from_string: error ]) @@ -457,7 +457,7 @@ dnl sequence of keys. 'syntax error' indicates oversized list of keys. keys=$(printf 'encap(),%.0s' $(seq 16382)) echo "${keys}encap()" > keys.txt echo "${keys}encap(),encap()" >> keys.txt -AT_CHECK([ovstest test-odp parse-keys < keys.txt | sed 's/encap(),//g'], [0], [dnl +AT_CHECK([ovs-test-dpparse.py ovstest test-odp parse-keys < keys.txt | sed 's/encap(),//g'], [0], [dnl odp_flow_key_to_flow: error (duplicate encap attribute in flow key; the flow key in error is: encap()) odp_flow_from_string: error (syntax error at encap()) ]) @@ -467,7 +467,7 @@ AT_SETUP([OVS datapath keys parsing and formatting - 33 nested encap ]) AT_DATA([odp-in.txt], [dnl encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap())))))))))))))))))))))))))))))))) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-keys < odp-in.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-keys < odp-in.txt], [0], [dnl odp_flow_from_string: error (syntax error at encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap()))))))))))))))))))))))))))))))))) ]) AT_CLEANUP diff --git a/tests/ovs-test-dpparse.py b/tests/ovs-test-dpparse.py new file mode 100755 index 000000000..5ccb426c1 --- /dev/null +++ b/tests/ovs-test-dpparse.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + +""" + ovs-test-dpparse is just a wrapper around ovs-dpctl + that also runs the python flow parsing utility to check that flows are + parseable. +""" +import subprocess +import sys +import re + +from ovs.flows.odp import ODPFlow + +diff_regexp = re.compile(r"\d{2}: (\d{2}|\(none\)) -> (\d{2}|\(none\))$") + + +def run(input_data): + p = subprocess.Popen( + sys.argv[1:], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + out, err = p.communicate(input_data.encode("utf-8")) + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + + +def main(): + return_code = 0 + input_data = sys.stdin.read() + return_code, out, err = run(input_data) + + if return_code == 0: + flows = list() + for line in input_data.split("\n"): + if not ( + "error" in line # skip errors + or line.strip() == "" # skip empty lines + or line.strip()[0] == "#" # skip comments + ): + flows.append(line) + + for flow in flows: + if any( + c in sys.argv + for c in ["parse-keys", "parse-wc-keys", "parse-filter"] + ): + # Add actions=drop so that the flow is properly formatted + flow += " actions:drop" + elif "parse-actions" in sys.argv: + flow = "actions:" + flow + try: + result_flow = ODPFlow(flow) + if flow != str(result_flow): + print("in : {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed!") + + except Exception as e: + print(e) + return 1 + + return return_code + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Fri Jan 28 16:04:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585764 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Q46HAkjL; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6M647hz9t3b for ; Sat, 29 Jan 2022 03:07:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 05B2161175; Fri, 28 Jan 2022 16:07:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yAja9pzGBkJt; Fri, 28 Jan 2022 16:07:52 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1D96D61163; Fri, 28 Jan 2022 16:07:51 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E44A0C001A; Fri, 28 Jan 2022 16:07:50 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 13511C001A for ; Fri, 28 Jan 2022 16:07:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 633EA61143 for ; Fri, 28 Jan 2022 16:06:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NdLv0pO0WbG7 for ; Fri, 28 Jan 2022 16:06:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 510D161103 for ; Fri, 28 Jan 2022 16:06:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fpSZD4nN85ZsESw6R7H/6uQSvc6sPzwUnyp/0o3k/RE=; b=Q46HAkjL8jhDYpOXnWYqxJ02VSqT0A6yO/0OEpUTeD+0GBlz7L6tNCuKxH8soCdKfHuQMK xL2v07Kd5GZZjUCGzhyv+UQzK2pYbszEdbE4+JLuJc5HHPhYOCGfZhEMuEV2MbyJ5Xd4T3 kpIySbq9EJMD+ZLzYMQDvFkEywmRYnI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-235-6s2xI6OHOd6jO4U89gyg4Q-1; Fri, 28 Jan 2022 11:06:37 -0500 X-MC-Unique: 6s2xI6OHOd6jO4U89gyg4Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EDBE318B9F3A; Fri, 28 Jan 2022 16:06:35 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id A21AD84A2A; Fri, 28 Jan 2022 16:06:34 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:36 +0100 Message-Id: <20220128160441.23477-14-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 13/18] python: detect changes in flow formatting code X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to minimize the risk of having the python flow parsing code and the C flow formatting code divert, add a target that checks if the formatting code has been changed since the last revision and warn the developer if it has. The script also makes it easy to update the dependency file so hopefully it will not cause too much trouble for a developer that has modifed the file without changing the flow string format. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- .gitignore | 1 + python/automake.mk | 9 +++ python/build/flow-parse-deps.py | 106 ++++++++++++++++++++++++++++++++ python/ovs/flows/deps.py | 5 ++ 4 files changed, 121 insertions(+) create mode 100755 python/build/flow-parse-deps.py create mode 100644 python/ovs/flows/deps.py diff --git a/.gitignore b/.gitignore index f1cdcf124..e6bca1cd2 100644 --- a/.gitignore +++ b/.gitignore @@ -79,3 +79,4 @@ testsuite.tmp.orig /Documentation/_build /.venv /cxx-check +/flowparse-deps-check diff --git a/python/automake.mk b/python/automake.mk index d56ac604b..0d29e9eb0 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/deps.py \ python/ovs/flows/filter.py \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ @@ -57,6 +58,7 @@ ovs_pyfiles = \ EXTRA_DIST += \ python/build/__init__.py \ python/build/extract_ofp_fields.py \ + python/build/flow-parse-deps.py \ python/build/nroff.py \ python/build/soutil.py @@ -77,6 +79,7 @@ FLAKE8_PYFILES += \ $(filter-out python/ovs/compat/% python/ovs/dirs.py,$(PYFILES)) \ python/setup.py \ python/build/__init__.py \ + python/build/flow-parse-deps.py \ python/build/nroff.py \ python/ovs/dirs.py.template @@ -135,3 +138,9 @@ $(srcdir)/python/ovs/flows/ofp_fields.py: $(srcdir)/build-aux/gen_ofp_field_deco EXTRA_DIST += python/ovs/flows/ofp_fields.py CLEANFILES += python/ovs/flows/ofp_fields.py +ALL_LOCAL += flowparse-deps-check +DEPS = $(shell $(AM_V_GEN)$(run_python) $(srcdir)/python/build/flow-parse-deps.py list) +flowparse-deps-check: $(srcdir)/python/build/flow-parse-deps.py $(DEPS) + $(AM_V_GEN)$(run_python) $(srcdir)/python/build/flow-parse-deps.py check + touch $@ +CLEANFILES += flowparse-deps-check diff --git a/python/build/flow-parse-deps.py b/python/build/flow-parse-deps.py new file mode 100755 index 000000000..848ef6bac --- /dev/null +++ b/python/build/flow-parse-deps.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Breaks lines read from stdin into groups using blank lines as +# group separators, then sorts lines within the groups for +# reproducibility. + + +# ovs-test-ofparse is just a wrapper around ovs-ofctl +# that also runs the python flow parsing utility to check that flows are +# parseable + +import hashlib +import sys +import os + +DEPENDENCIES = ["lib/ofp-actions.c", "lib/odp-util.c"] +DEPENDENCY_FILE = "python/ovs/flows/deps.py" +SRC_DIR = os.path.join(os.path.dirname(__file__), "..", "..") + + +def usage(): + print( + """ +Usage {cmd} [check | update | list] +Tool to verify flow parsing python code is kept in sync with +flow printing C code. + +Commands: + check: check the dependencies are met + update: update the dependencies based on current file content + list: list the dependency files +""".format( + cmd=sys.argv[0] + ) + ) + + +def digest(filename): + with open(os.path.join(SRC_DIR, filename), "rb") as f: + return hashlib.md5(f.read()).hexdigest() + + +def main(): + if len(sys.argv) != 2: + usage() + sys.exit(1) + + if sys.argv[1] == "list": + print(" ".join(DEPENDENCIES)) + elif sys.argv[1] == "update": + dep_str = list() + for dep in DEPENDENCIES: + dep_str.append( + ' "{dep}": "{digest}"'.format(dep=dep, digest=digest(dep)) + ) + + depends = """# File automatically generated. Do not modify manually! +dependencies = {{ +{dependencies_dict} +}}""".format( + dependencies_dict=",\n".join(dep_str) + ) + with open(os.path.join(SRC_DIR, DEPENDENCY_FILE), "w") as f: + print(depends, file=f) + + elif sys.argv[1] == "check": + sys.path.append(os.path.join(SRC_DIR, "python")) + from ovs.flows.deps import dependencies + + for dep in DEPENDENCIES: + expected = dependencies.get(dep) + if not expected or expected != digest(dep): + print( + """ +Dependency file {dep} has changed. +Please verify the flow output format has not changed. +If it has changed, modify the python flow parsing code accordingly. + +Once you're done, update the dependencies by running '{cmd} update'. +After doing so, check-in the new dependency file. +""".format( + dep=dep, + cmd=sys.argv[0], + ) + ) + return 2 + else: + usage() + sys.exit(1) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/python/ovs/flows/deps.py b/python/ovs/flows/deps.py new file mode 100644 index 000000000..83b3f1747 --- /dev/null +++ b/python/ovs/flows/deps.py @@ -0,0 +1,5 @@ +# File automatically generated. Do not modify manually! +dependencies = { + "lib/ofp-actions.c": "f108b3e119f03b3373064589aecdeaf0", + "lib/odp-util.c": "4e2ce0d47bd82ebf50082a0fb2d390c5" +} From patchwork Fri Jan 28 16:04:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585767 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IjGaohCB; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6g61QQz9t3b for ; Sat, 29 Jan 2022 03:08:11 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id CDC6040C5E; Fri, 28 Jan 2022 16:08:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hH6tkhLiAE7o; Fri, 28 Jan 2022 16:08:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 08C8A40C39; Fri, 28 Jan 2022 16:08:06 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B413FC007E; Fri, 28 Jan 2022 16:08:04 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 36BD2C007F for ; Fri, 28 Jan 2022 16:08:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id EC2F284C27 for ; Fri, 28 Jan 2022 16:06:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mCzVlmNruz6m for ; Fri, 28 Jan 2022 16:06:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id CE7B384CB6 for ; Fri, 28 Jan 2022 16:06:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386008; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WODbxI/sDGKaGdTiUu4XsLfKIaofDSci8sZI9RSwbN4=; b=IjGaohCBNv0GE8WkPJ9/UueJNVQTyVF/mFnkN3kHT2HwPBuaFrdMERiYt2INf0mLEJ/Hw2 CBrlPj2YTHx37Bk6X6jBhQuCfch8VjdA1qNjk9CFM6rNZys7I3vP55Zb17YyTscXwL2l0/ Co9sLGknglGiXbUa+PYmSczhgB17yjM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-RraRbGGoNHyE50x1GOFccQ-1; Fri, 28 Jan 2022 11:06:39 -0500 X-MC-Unique: RraRbGGoNHyE50x1GOFccQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EA183100CD19; Fri, 28 Jan 2022 16:06:37 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 58BEF84A3F; Fri, 28 Jan 2022 16:06:36 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:37 +0100 Message-Id: <20220128160441.23477-15-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 14/18] python: introduce unit tests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Use pytest to run unit tests as part of the standard testsuite. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- .github/workflows/build-and-test.yml | 3 + Documentation/intro/install/general.rst | 4 ++ python/automake.mk | 9 ++- python/ovs/tests/test_kv.py | 76 +++++++++++++++++++++++++ python/test_requirements.txt | 3 + tests/atlocal.in | 19 +++++++ tests/automake.mk | 1 + tests/pytest.at | 7 +++ tests/testsuite.at | 1 + 9 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 python/ovs/tests/test_kv.py create mode 100644 python/test_requirements.txt create mode 100644 tests/pytest.at diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml index eac3504e4..44df1c2d5 100644 --- a/.github/workflows/build-and-test.yml +++ b/.github/workflows/build-and-test.yml @@ -123,6 +123,9 @@ jobs: with: python-version: '3.9' + - name: install python dependencies + run: pip install -r python/test_requirements.txt + - name: create ci signature file for the dpdk cache key if: matrix.dpdk != '' || matrix.dpdk_shared != '' # This will collect most of DPDK related lines, so hash will be different diff --git a/Documentation/intro/install/general.rst b/Documentation/intro/install/general.rst index c4300cd53..711fb98a4 100644 --- a/Documentation/intro/install/general.rst +++ b/Documentation/intro/install/general.rst @@ -181,6 +181,10 @@ following to obtain better warnings: come from the "hacking" flake8 plugin. If it's not installed, the warnings just won't occur until it's run on a system with "hacking" installed. +- the python packages listed in "python/test_requirements.txt" (compatible + with pip). If they are installed, the pytest-based Python unit tests will + be run. + You may find the ovs-dev script found in ``utilities/ovs-dev.py`` useful. .. _general-install-reqs: diff --git a/python/automake.mk b/python/automake.mk index 0d29e9eb0..f22dd88a9 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -53,6 +53,9 @@ ovs_pyfiles = \ python/ovs/vlog.py \ python/ovs/winutils.py +ovs_pytests = \ + python/ovs/tests/test_kv.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ @@ -66,12 +69,14 @@ EXTRA_DIST += \ EXTRA_DIST += \ python/ovs/compat/sortedcontainers/LICENSE \ python/README.rst \ - python/setup.py + python/setup.py \ + python/test_requirements.txt # C extension support. EXTRA_DIST += python/ovs/_json.c -PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) +PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) $(ovs_pytests) + EXTRA_DIST += $(PYFILES) PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) diff --git a/python/ovs/tests/test_kv.py b/python/ovs/tests/test_kv.py new file mode 100644 index 000000000..e81804d49 --- /dev/null +++ b/python/ovs/tests/test_kv.py @@ -0,0 +1,76 @@ +import pytest + +from ovs.flows.kv import KVParser, KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ( + "cookie=0x0, duration=147566.365s, table=0, n_packets=39, n_bytes=2574, idle_age=65534, hard_age=65534", # noqa: E501 + None, + ), + [ + KeyValue("cookie", 0), + KeyValue("duration", "147566.365s"), + KeyValue("table", 0), + KeyValue("n_packets", 39), + KeyValue("n_bytes", 2574), + KeyValue("idle_age", 65534), + KeyValue("hard_age", 65534), + ], + ), + ( + ( + "load:0x4->NXM_NX_REG13[],load:0x9->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:0a:58:a9:fe:00:02,resubmit(,8)", # noqa: E501 + None, + ), + [ + KeyValue("load", "0x4->NXM_NX_REG13[]"), + KeyValue("load", "0x9->NXM_NX_REG11[]"), + KeyValue("load", "0x8->NXM_NX_REG12[]"), + KeyValue("load", "0x1->OXM_OF_METADATA[]"), + KeyValue("load", "0x1->NXM_NX_REG14[]"), + KeyValue("mod_dl_src", "0a:58:a9:fe:00:02"), + KeyValue("resubmit", ",8"), + ], + ), + ( + ("l1(l2(l3(l4())))", None), + [KeyValue("l1", "l2(l3(l4()))")] + ), + ( + ("l1(l2(l3(l4()))),foo:bar", None), + [KeyValue("l1", "l2(l3(l4()))"), KeyValue("foo", "bar")], + ), + ( + ("enqueue:1:2,output=2", None), + [KeyValue("enqueue", "1:2"), KeyValue("output", 2)], + ), + ( + ("value_to_reg(100)->someReg[10],foo:bar", None), + [ + KeyValue("value_to_reg", "(100)->someReg[10]"), + KeyValue("foo", "bar"), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + tparser = KVParser(input_string, decoders) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr diff --git a/python/test_requirements.txt b/python/test_requirements.txt new file mode 100644 index 000000000..6aaee13e3 --- /dev/null +++ b/python/test_requirements.txt @@ -0,0 +1,3 @@ +pytest +netaddr +pyparsing diff --git a/tests/atlocal.in b/tests/atlocal.in index a0ad239ec..2426416a0 100644 --- a/tests/atlocal.in +++ b/tests/atlocal.in @@ -222,3 +222,22 @@ export OVS_CTL_TIMEOUT # matter break everything. ASAN_OPTIONS=detect_leaks=0:abort_on_error=true:log_path=asan:$ASAN_OPTIONS export ASAN_OPTIONS + +# Check whether Python test requirements are available. +REQUIREMENT_PATH=$abs_top_srcdir/python/test_requirements.txt $PYTHON3 -c ' +import os +import pathlib +import pkg_resources + +with pathlib.Path(os.path.join(os.getenv("REQUIREMENT_PATH"))).open() as reqs: + for req in pkg_resources.parse_requirements(reqs): + try: + pkg_resources.require(str(req)) + except pkg_resources.DistributionNotFound: + sys.exit(2) +' +case $? in + 0) HAVE_PYTEST=yes ;; + 2) HAVE_PYTEST=no ;; + *) echo "$0: unexpected error probing Python unit test requirements" >&2 ;; +esac diff --git a/tests/automake.mk b/tests/automake.mk index 80422679e..1aefac29d 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -103,6 +103,7 @@ TESTSUITE_AT = \ tests/ovsdb-rbac.at \ tests/ovs-vsctl.at \ tests/ovs-xapi-sync.at \ + tests/pytest.at \ tests/stp.at \ tests/rstp.at \ tests/interface-reconfigure.at \ diff --git a/tests/pytest.at b/tests/pytest.at new file mode 100644 index 000000000..44a88ed98 --- /dev/null +++ b/tests/pytest.at @@ -0,0 +1,7 @@ +AT_BANNER([Python unit tests]) + +# Run pytest unit tests. +AT_SETUP([Pytest unit tests - Python3]) +AT_SKIP_IF([test "$HAVE_PYTEST" = "no"]) +AT_CHECK([python -m pytest $top_srcdir/python/ovs],[0], [ignore], [ignore]) +AT_CLEANUP() diff --git a/tests/testsuite.at b/tests/testsuite.at index 58adfa09c..14a28b517 100644 --- a/tests/testsuite.at +++ b/tests/testsuite.at @@ -78,3 +78,4 @@ m4_include([tests/mcast-snooping.at]) m4_include([tests/packet-type-aware.at]) m4_include([tests/nsh.at]) m4_include([tests/drop-stats.at]) +m4_include([tests/pytest.at]) From patchwork Fri Jan 28 16:04:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585762 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ChjF8mj1; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj684W2Sz9t3b for ; Sat, 29 Jan 2022 03:07:44 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 73EAB41D0F; Fri, 28 Jan 2022 16:07:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fBiuhG9ix4Mb; Fri, 28 Jan 2022 16:07:41 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 6DB4541CE1; Fri, 28 Jan 2022 16:07:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 80E24C0070; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id B9884C000B for ; Fri, 28 Jan 2022 16:07:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E3B1340BBC for ; Fri, 28 Jan 2022 16:06:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kMm5DGvhqEGo for ; Fri, 28 Jan 2022 16:06:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id F27FF40B1F for ; Fri, 28 Jan 2022 16:06:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386004; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n6zQkYNpiRRaHcMEU9XAhkkn6vs3Z2/TjsT3JsEPeyw=; b=ChjF8mj13dUSZ9ZJFwaKtsLYD5m1ohlp9HP9kW9KL4FejkLNBL/molkNIZt4DISIvwWtcv aquLeEDX+WV7AdZqJdZRwCxIbjc2jnl9itB/EKHRcitwdGddPHz9fjwuoaA/5RmhTt6KvT TuhGcX+dllwTMO4E+Oy+NboRflWachU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-136-0X0ZtqFuNuSgVXY6LfbvnA-1; Fri, 28 Jan 2022 11:06:40 -0500 X-MC-Unique: 0X0ZtqFuNuSgVXY6LfbvnA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A6D1C1F2E7; Fri, 28 Jan 2022 16:06:39 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47A6884A2A; Fri, 28 Jan 2022 16:06:38 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:38 +0100 Message-Id: <20220128160441.23477-16-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 15/18] python: add unit tests for ListParser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for ListParser class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 3 +- python/ovs/tests/test_list.py | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 python/ovs/tests/test_list.py diff --git a/python/automake.mk b/python/automake.mk index f22dd88a9..9ae28b875 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -54,7 +54,8 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ - python/ovs/tests/test_kv.py + python/ovs/tests/test_kv.py \ + python/ovs/tests/test_list.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_list.py b/python/ovs/tests/test_list.py new file mode 100644 index 000000000..e5139869f --- /dev/null +++ b/python/ovs/tests/test_list.py @@ -0,0 +1,66 @@ +import pytest + +from ovs.flows.list import ListParser, ListDecoders +from ovs.flows.kv import KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ("field1,field2,3,nested:value", None, [","]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + KeyValue("elem_3", "nested:value"), + ], + ), + ( + ( + "field1,field2,3,nested:value", + ListDecoders( + [ + ("key1", str), + ("key2", str), + ("key3", int), + ("key4", lambda x: x.split(":"), [","]), + ] + ), + [","], + ), + [ + KeyValue("key1", "field1"), + KeyValue("key2", "field2"), + KeyValue("key3", 3), + KeyValue("key4", ["nested", "value"]), + ], + ), + ( + ("field1:field2:3", None, [":"]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + delims = input_data[2] + tparser = ListParser(input_string, decoders, delims) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr From patchwork Fri Jan 28 16:04:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585769 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZWxTPSvQ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6p43sJz9t3b for ; Sat, 29 Jan 2022 03:08:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id D766484F9A; Fri, 28 Jan 2022 16:08:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EcGYlZiSvH1U; Fri, 28 Jan 2022 16:08:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1135F84F15; Fri, 28 Jan 2022 16:08:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C6647C0079; Fri, 28 Jan 2022 16:08:08 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 35255C007B for ; Fri, 28 Jan 2022 16:08:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id EC5E284C16 for ; Fri, 28 Jan 2022 16:07:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CJQY2wFjMbif for ; Fri, 28 Jan 2022 16:07:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id DC6A584D16 for ; Fri, 28 Jan 2022 16:07:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJGjyvB1RND6G3d+aTfpVU5I4A9dP3orhpU8aA5mJfo=; b=ZWxTPSvQijLBCorE1lnxyJ8sL7+zto+jupDqr4nqkLOI25Jylqyi+hSj1VlucVrWEUtxaf bGHP+kg8WbvCASHdg1Md0c/R8wKkuYhclr3X2628ah9TzD9hIfULSfH2vpZ05DnCx/Ax/N BDN0UhaO2cE25IzNEsqeydx6BEQ5v2c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-460-G2FoBpkOOSmZfXWDDt9x9A-1; Fri, 28 Jan 2022 11:07:00 -0500 X-MC-Unique: G2FoBpkOOSmZfXWDDt9x9A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 73C9C100D680; Fri, 28 Jan 2022 16:06:59 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id 03FED84A2A; Fri, 28 Jan 2022 16:06:39 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:39 +0100 Message-Id: <20220128160441.23477-17-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 16/18] python: add unit tests for openflow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for OFPFlow class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 3 +- python/ovs/tests/test_ofp.py | 524 +++++++++++++++++++++++++++++++++++ 2 files changed, 526 insertions(+), 1 deletion(-) create mode 100644 python/ovs/tests/test_ofp.py diff --git a/python/automake.mk b/python/automake.mk index 9ae28b875..43bb660f3 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -55,7 +55,8 @@ ovs_pyfiles = \ ovs_pytests = \ python/ovs/tests/test_kv.py \ - python/ovs/tests/test_list.py + python/ovs/tests/test_list.py \ + python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_ofp.py b/python/ovs/tests/test_ofp.py new file mode 100644 index 000000000..6868696d3 --- /dev/null +++ b/python/ovs/tests/test_ofp.py @@ -0,0 +1,524 @@ +import netaddr +import pytest + +from ovs.flows.ofp import OFPFlow +from ovs.flows.kv import KeyValue +from ovs.flows.decoders import EthMask, IPMask, decode_mask + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions=local,3,4,5,output:foo", + [ + KeyValue("output", {"port": "local"}), + KeyValue("output", {"port": 3}), + KeyValue("output", {"port": 4}), + KeyValue("output", {"port": 5}), + KeyValue("output", {"port": "foo"}), + ], + ), + ( + "actions=controller,controller:200", + [ + KeyValue("output", "controller"), + KeyValue("controller", {"max_len": 200}), + ], + ), + ( + "actions=enqueue(foo,42),enqueue:foo:42,enqueue(bar,4242)", + [ + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "bar", "queue": 4242}), + ], + ), + ( + "actions=bundle(eth_src,0,hrw,ofport,members:4,8)", + [ + KeyValue( + "bundle", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=bundle_load(eth_src,0,hrw,ofport,reg0,members:4,8)", + [ + KeyValue( + "bundle_load", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "dst": "reg0", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=group:3", + [KeyValue("group", 3)], + ), + ( + "actions=strip_vlan", + [KeyValue("strip_vlan", True)], + ), + ( + "actions=pop_vlan", + [KeyValue("pop_vlan", True)], + ), + ( + "actions=push_vlan:0x8100", + [KeyValue("push_vlan", 0x8100)], + ), + ( + "actions=push_mpls:0x8848", + [KeyValue("push_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=encap(nsh(md_type=2,tlv(0x1000,10,0x12345678)))", + [ + KeyValue( + "encap", + { + "nsh": { + "md_type": 2, + "tlv": { + "class": 0x1000, + "type": 10, + "value": 0x12345678, + }, + } + }, + ) + ], + ), + ( + "actions=encap(0x0800)", + [ + KeyValue( + "encap", + {"ethernet": 0x800}, + ) + ], + ), + ( + "actions=load:0x001122334455->eth_src", + [ + KeyValue( + "load", + {"value": 0x001122334455, "dst": {"field": "eth_src"}}, + ) + ], + ), + ( + "actions=load:1->eth_src[1]", + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "eth_src", "start": 1, "end": 1}, + }, + ) + ], + ), + ( + "actions=learn(load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[])", + [ + KeyValue( + "learn", + [ + { + "load": { + "src": {"field": "NXM_NX_TUN_ID"}, + "dst": {"field": "NXM_NX_TUN_ID"}, + } + } + ], + ), + ], + ), + ( + "actions=set_field:00:11:22:33:44:55->eth_src", + [ + KeyValue( + "set_field", + { + "value": {"eth_src": EthMask("00:11:22:33:44:55")}, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:01:00:00:00:00:00/01:00:00:00:00:00->eth_src", + [ + KeyValue( + "set_field", + { + "value": { + "eth_src": EthMask( + "01:00:00:00:00:00/01:00:00:00:00:00" + ) + }, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:0x10ff->vlan_vid", + [ + KeyValue( + "set_field", + { + "value": {"vlan_vid": decode_mask(13)("0x10ff")}, + "dst": {"field": "vlan_vid"}, + }, + ) + ], + ), + ( + "actions=move:reg0[0..5]->reg1[16..31]", + [ + KeyValue( + "move", + { + "src": {"field": "reg0", "start": 0, "end": 5}, + "dst": {"field": "reg1", "start": 16, "end": 31}, + }, + ) + ], + ), + ( + "actions=mod_dl_dst:00:11:22:33:44:55", + [KeyValue("mod_dl_dst", EthMask("00:11:22:33:44:55"))], + ), + ( + "actions=mod_nw_dst:192.168.1.1", + [KeyValue("mod_nw_dst", IPMask("192.168.1.1"))], + ), + ( + "actions=mod_nw_dst:fe80::ec17:7bff:fe61:7aac", + [KeyValue("mod_nw_dst", IPMask("fe80::ec17:7bff:fe61:7aac"))], + ), + ( + "actions=dec_ttl,dec_ttl(1,2,3)", + [KeyValue("dec_ttl", True), KeyValue("dec_ttl", [1, 2, 3])], + ), + ( + "actions=set_mpls_label:0x100,set_mpls_tc:2,set_mpls_ttl:10", + [ + KeyValue("set_mpls_label", 0x100), + KeyValue("set_mpls_tc", 2), + KeyValue("set_mpls_ttl", 10), + ], + ), + ( + "actions=check_pkt_larger(100)->reg0[10]", + [ + KeyValue( + "check_pkt_larger", + { + "pkt_len": 100, + "dst": {"field": "reg0", "start": 10, "end": 10}, + }, + ), + ], + ), + ( + "actions=pop_queue,set_tunnel:0x10,set_tunnel64:0x65000,set_queue=3", # noqa: E501 + [ + KeyValue("pop_queue", True), + KeyValue("set_tunnel", 0x10), + KeyValue("set_tunnel64", 0x65000), + KeyValue("set_queue", 3), + ], + ), + ( + "actions=ct(zone=10,table=2,nat(snat=192.168.0.0-192.168.0.200:1000-2000,random))", # noqa: E501 + [ + KeyValue( + "ct", + { + "zone": 10, + "table": 2, + "nat": { + "type": "snat", + "addrs": { + "start": netaddr.IPAddress("192.168.0.0"), + "end": netaddr.IPAddress("192.168.0.200"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + "random": True, + }, + }, + ) + ], + ), + ( + "actions=ct(commit,zone=NXM_NX_REG13[0..15],table=2,exec(load:0->NXM_NX_CT_LABEL[0]))", # noqa: E501 + [ + KeyValue( + "ct", + { + "commit": True, + "zone": { + "field": "NXM_NX_REG13", + "start": 0, + "end": 15, + }, + "table": 2, + "exec": [ + { + "load": { + "value": 0, + "dst": { + "field": "NXM_NX_CT_LABEL", + "start": 0, + "end": 0, + }, + }, + }, + ], + }, + ) + ], + ), + ( + "actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xda6f52b0,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=172.30.204.105,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])", # noqa: E501 + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "NXM_NX_REG10", "start": 7, "end": 7}, + }, + ), + KeyValue( + "learn", + [ + {"table": 69}, + {"delete_learned": True}, + {"cookie": 3664728752}, + {"OXM_OF_METADATA[]": True}, + {"eth_type": 2048}, + {"NXM_OF_IP_SRC[]": True}, + {"ip_dst": IPMask("172.30.204.105/32")}, + {"nw_proto": 6}, + {"NXM_OF_TCP_SRC[]": "NXM_OF_TCP_DST[]"}, + { + "load": { + "value": 1, + "dst": { + "field": "NXM_NX_REG10", + "start": 7, + "end": 7, + }, + } + }, + ], + ), + ], + ), + ( + "actions=resubmit(,8),resubmit:3,resubmit(1,2,ct)", + [ + KeyValue("resubmit", {"port": "", "table": 8}), + KeyValue("resubmit", {"port": 3}), + KeyValue("resubmit", {"port": 1, "table": 2, "ct": True}), + ], + ), + ( + "actions=clone(ct_clear,load:0->NXM_NX_REG11[],load:0->NXM_NX_REG12[],load:0->NXM_NX_REG13[],load:0x1d->NXM_NX_REG13[],load:0x1f->NXM_NX_REG11[],load:0x1c->NXM_NX_REG12[],load:0x11->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0->NXM_NX_REG10[],load:0->NXM_NX_REG15[],load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],load:0->NXM_NX_REG5[],load:0->NXM_NX_REG6[],load:0->NXM_NX_REG7[],load:0->NXM_NX_REG8[],load:0->NXM_NX_REG9[],resubmit(,8))", # noqa: E501 + [ + KeyValue( + "clone", + [ + {"ct_clear": True}, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 29, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 31, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 28, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 17, + "dst": {"field": "OXM_OF_METADATA"}, + } + }, + { + "load": { + "value": 2, + "dst": {"field": "NXM_NX_REG14"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG10"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG15"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG0"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG1"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG2"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG3"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG4"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG5"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG6"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG7"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG8"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG9"}, + } + }, + {"resubmit": {"port": "", "table": 8}}, + ], + ) + ], + ), + ( + "actions=conjunction(1234, 1/2),note:00.00.11.22.33.ff,sample(probability=123,collector_set_id=0x123,obs_domain_id=0x123,obs_point_id=0x123,sampling_port=inport0,ingress)", # noqa: E501 + [ + KeyValue("conjunction", {"id": 1234, "k": 1, "n": 2}), + KeyValue("note", "00.00.11.22.33.ff"), + KeyValue( + "sample", + { + "probability": 123, + "collector_set_id": 0x123, + "obs_domain_id": 0x123, + "obs_point_id": 0x123, + "sampling_port": "inport0", + "ingress": True, + }, + ), + ], + ), + ], +) +def test_act(input_string, expected): + ofp = OFPFlow(input_string) + actions = ofp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = ofp.section("actions").pos + astring = ofp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Fri Jan 28 16:04:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585771 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ajtVgweb; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj7F3Gw3z9t3b for ; Sat, 29 Jan 2022 03:08:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A9BDC41DE0; Fri, 28 Jan 2022 16:08:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GLlUvhliQL1V; Fri, 28 Jan 2022 16:08:37 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 80ADA41D9E; Fri, 28 Jan 2022 16:08:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 121ADC0021; Fri, 28 Jan 2022 16:08:35 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19C8FC000B for ; Fri, 28 Jan 2022 16:08:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1526941C85 for ; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LhH7gJxxxGL6 for ; Fri, 28 Jan 2022 16:07:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id E06C841C5D for ; Fri, 28 Jan 2022 16:07:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386055; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lSs3ekImNKwLPU8oGDT4euIKp413bCEbflDARVBNYJk=; b=ajtVgweb756gaEvkSKZDXjnrT7pD5NngNOKWT9z35OschAemJLWAEflB3iE+XkDgiAMs50 UUKDcfHX40/Pdjy3Bg2Bk9eLpcqzNPisP2FlYR0zZC8Z4y9Z0S4b5CLancphQUCr4K3LDf N77Xzt35uDwvNR2Xmdqf9Ivn+UAcJXk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-564-MFysSKAiNkmWD26o0hZwcw-1; Fri, 28 Jan 2022 11:07:32 -0500 X-MC-Unique: MFysSKAiNkmWD26o0hZwcw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 589081018724; Fri, 28 Jan 2022 16:07:31 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id B40DD84A2A; Fri, 28 Jan 2022 16:06:59 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:40 +0100 Message-Id: <20220128160441.23477-18-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 17/18] python: add unit tests to datapath parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/tests/test_odp.py | 527 +++++++++++++++++++++++++++++++++++ 2 files changed, 528 insertions(+) create mode 100644 python/ovs/tests/test_odp.py diff --git a/python/automake.mk b/python/automake.mk index 43bb660f3..44c1035a3 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -56,6 +56,7 @@ ovs_pyfiles = \ ovs_pytests = \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ + python/ovs/tests/test_odp.py \ python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, diff --git a/python/ovs/tests/test_odp.py b/python/ovs/tests/test_odp.py new file mode 100644 index 000000000..8ca8f38bf --- /dev/null +++ b/python/ovs/tests/test_odp.py @@ -0,0 +1,527 @@ +import netaddr +import pytest + +from ovs.flows.odp import ODPFlow +from ovs.flows.kv import KeyValue +from ovs.flows.decoders import ( + EthMask, + IPMask, + Mask32, + Mask16, + Mask8, + Mask128, +) + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "skb_priority(0x123),skb_mark(0x123),recirc_id(0x123),dp_hash(0x123),ct_zone(0x123), actions:", # noqa: E501 + [ + KeyValue("skb_priority", Mask32("0x123")), + KeyValue("skb_mark", Mask32("0x123")), + KeyValue("recirc_id", 0x123), + KeyValue("dp_hash", Mask32("0x123")), + KeyValue("ct_zone", Mask16("0x123")), + ], + ), + ( + "tunnel(tun_id=0x7f10354,src=10.10.10.10,dst=20.20.20.20,ttl=64,flags(csum|key)) actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "tun_id": 0x7F10354, + "src": IPMask("10.10.10.10"), + "dst": IPMask("20.20.20.20"), + "ttl": 64, + "flags": "csum|key", + }, + ) + ], + ), + ( + "tunnel(geneve({class=0,type=0,len=4,0xa/0xff}),vxlan(flags=0x800000,vni=0x1c7),erspan(ver=2,dir=1,hwid=0x1)), actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "geneve": [ + { + "class": Mask16("0"), + "type": Mask8("0"), + "len": Mask8("4"), + "data": Mask128("0xa/0xff"), + } + ], + "vxlan": {"flags": 0x800000, "vni": 0x1C7}, + "erspan": {"ver": 2, "dir": 1, "hwid": 0x1}, + }, + ) + ], + ), + ( + "in_port(2),eth(src=11:22:33:44:55:66,dst=66:55:44:33:22:11) actions:", # noqa: E501 + [ + KeyValue("in_port", 2), + KeyValue( + "eth", + { + "src": EthMask("11:22:33:44:55:66"), + "dst": EthMask("66:55:44:33:22:11"), + }, + ), + ], + ), + ( + "eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0) actions:", # noqa: E501 + [ + KeyValue("eth_type", Mask16("0x800/0x006")), + KeyValue( + "ipv4", + { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xFF"), + "tos": Mask8("0x2/0xF0"), + }, + ), + ], + ), + ( + "encap(eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0)) actions:", # noqa: E501 + [ + KeyValue( + "encap", + { + "eth_type": Mask16("0x800/0x006"), + "ipv4": { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xff"), + "tos": Mask8("0x2/0xf0"), + }, + }, + ), + ], + ), + ], +) +def test_odp_fields(input_string, expected): + odp = ODPFlow(input_string) + match = odp.match_kv + for i in range(len(expected)): + assert expected[i].key == match[i].key + assert expected[i].value == match[i].value + + # Assert positions relative to action string are OK. + mpos = odp.section("match").pos + mstring = odp.section("match").string + + kpos = match[i].meta.kpos + kstr = match[i].meta.kstring + vpos = match[i].meta.vpos + vstr = match[i].meta.vstring + assert mstring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert mstring[vpos : vpos + len(vstr)] == vstr + + # Assert mstring meta is correct. + assert input_string[mpos : mpos + len(mstring)] == mstring + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions:ct" + ",ct(commit)" + ",ct(commit,zone=5)" + ",ct(commit,mark=0xa0a0a0a0/0xfefefefe)" + ",ct(commit,label=0x1234567890abcdef1234567890abcdef/0xf1f2f3f4f5f6f7f8f9f0fafbfcfdfeff)" # noqa: E501 + ",ct(commit,helper=ftp)" + ",ct(commit,helper=tftp)" + ",ct(commit,timeout=ovs_tp_1_tcp4)" + ",ct(nat)", + [ + KeyValue("ct", True), + KeyValue("ct", {"commit": True}), + KeyValue("ct", {"commit": True, "zone": 5}), + KeyValue( + "ct", + {"commit": True, "mark": Mask32("0xA0A0A0A0/0xFEFEFEFE")}, + ), + KeyValue( + "ct", + { + "commit": True, + "label": Mask128( + "0x1234567890ABCDEF1234567890ABCDEF/0xF1F2F3F4F5F6F7F8F9F0FAFBFCFDFEFF" # noqa: E501 + ), + }, + ), + KeyValue("ct", {"commit": True, "helper": "ftp"}), + KeyValue("ct", {"commit": True, "helper": "tftp"}), + KeyValue("ct", {"commit": True, "timeout": "ovs_tp_1_tcp4"}), + KeyValue("ct", {"nat": True}), + ], + ), + ( + "actions:ct(nat)" + ",ct(commit,nat(src))" + ",ct(commit,nat(dst))" + ",ct(commit,nat(src=10.0.0.240,random))" + ",ct(commit,nat(src=10.0.0.240:32768-65535,random))" + ",ct(commit,nat(dst=10.0.0.128-10.0.0.254,hash))" + ",ct(commit,nat(src=10.0.0.240-10.0.0.254:32768-65535,persistent))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:a18b,random))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:1-fe80::20c:29ff:fe88:a18b,random))" # noqa: E501 + ",ct(commit,nat(src=[[fe80::20c:29ff:fe88:1]]-[[fe80::20c:29ff:fe88:a18b]]:255-4096,random))" # noqa: E501 + ",ct(commit,helper=ftp,nat(src=10.1.1.240-10.1.1.255))" + ",ct(force_commit)", + [ + KeyValue("ct", {"nat": True}), + KeyValue("ct", {"commit": True, "nat": {"type": "src"}}), + KeyValue("ct", {"commit": True, "nat": {"type": "dst"}}), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "dst", + "addrs": { + "start": netaddr.IPAddress("10.0.0.128"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "hash": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "persistent": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "ports": { + "start": 255, + "end": 4096, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.1.1.240"), + "end": netaddr.IPAddress("10.1.1.255"), + }, + }, + "helper": "ftp", + }, + ), + KeyValue("ct", {"force_commit": True}), + ], + ), + ( + "actions:set(tunnel(tun_id=0xabcdef1234567890,src=1.1.1.1,dst=2.2.2.2,ttl=64,flags(df|csum|key)))" # noqa: E501 + ",tnl_pop(4)" + ",tnl_push(tnl_port(6),header(size=50,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x0800),ipv4(src=1.1.2.88,dst=1.1.2.92,proto=17,tos=0,ttl=64,frag=0x4000),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))" # noqa: E501 + ",tnl_push(tnl_port(6),header(size=70,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x86dd),ipv6(src=2001:cafe::88,dst=2001:cafe::92,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))", # noqa: E501 + [ + KeyValue( + "set", + { + "tunnel": { + "tun_id": 0xABCDEF1234567890, + "src": IPMask("1.1.1.1"), + "dst": IPMask("2.2.2.2"), + "ttl": 64, + "flags": "df|csum|key", + } + }, + ), + KeyValue("tnl_pop", 4), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 50, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x800, + }, + "ipv4": { + "src": IPMask("1.1.2.88"), + "dst": IPMask("1.1.2.92"), + "proto": 17, + "tos": 0, + "ttl": 64, + "frag": 0x4000, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 70, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x86DD, + }, + "ipv6": { + "src": IPMask("2001:cafe::88"), + "dst": IPMask("2001:cafe::92"), + "label": 0, + "proto": 17, + "tclass": 0x0, + "hlimit": 64, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + ], + ), + ( + "actions:tnl_push(header(geneve(oam,vni=0x1c7)))" + ",tnl_push(header(geneve(crit,vni=0x1c7,options({class=0xffff,type=0x80,len=4,0xa}))))" # noqa: E501 + ",tnl_push(header(gre((flags=0xa000,proto=0x6558),csum=0x0,key=0x1e241)))", # noqa: E501 + [ + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "oam": True, + "vni": 0x1C7, + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "crit": True, + "vni": 0x1C7, + "options": [ + { + "class": 0xFFFF, + "type": 0x80, + "len": 4, + "data": 0xA, + } + ], + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "gre": { + "flags": 0xA000, + "proto": 0x6558, + "key": 0x1E241, + "csum": 0x0, + } + } + }, + ), + ], + ), + ( + "actions:clone(1)" ",clone(clone(push_vlan(vid=12,pcp=0),2),1)", + [ + KeyValue("clone", {"output": {"port": 1}}), + KeyValue( + "clone", + { + "output": {"port": 1}, + "clone": { + "push_vlan": { + "vid": 12, + "pcp": 0, + }, + "output": {"port": 2}, + }, + }, + ), + ], + ), + ( + "actions: check_pkt_len(size=200,gt(4),le(5))" + ",check_pkt_len(size=200,gt(drop),le(5))" + ",check_pkt_len(size=200,gt(ct(nat)),le(drop))", + [ + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"output": {"port": 4}}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"drop": True}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"ct": {"nat": True}}, + "le": {"drop": True}, + }, + ), + ], + ), + ], +) +def test_odp_actions(input_string, expected): + odp = ODPFlow(input_string) + actions = odp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = odp.section("actions").pos + astring = odp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Fri Jan 28 16:04:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1585770 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ILaZJvhC; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Jlj6z1mYjz9t3b for ; Sat, 29 Jan 2022 03:08:27 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7223F40BE6; Fri, 28 Jan 2022 16:08:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 22-LKA-66fOM; Fri, 28 Jan 2022 16:08:22 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8F30E40C68; Fri, 28 Jan 2022 16:08:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 51B91C000B; Fri, 28 Jan 2022 16:08:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1CEADC001A for ; Fri, 28 Jan 2022 16:08:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6CA5684CAA for ; Fri, 28 Jan 2022 16:07:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6QlQ25cc-N9u for ; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 7134B84D2D for ; Fri, 28 Jan 2022 16:07:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643386057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Shd7fXHGkqSqaPF2sod2OJJ8BzjF2mNaKONjbneTxsM=; b=ILaZJvhCwrajYyzToaYSANEc/5iGpWDN5A+nWjwBPyQswBkn9dufw/5JA4pqPeB5XaGphC ab36S/5X8sb+7O08aANQm6rM/sjbBhVhmNRnFl0nMFo0gg4WWpmlDzNyNFkDsTLGtgzkOp TqgKRzcxTNfQYnEN1XEf/W5pJObHlE8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-619-FRSQ2ZmUOWSrkBp2Ob1D4Q-1; Fri, 28 Jan 2022 11:07:34 -0500 X-MC-Unique: FRSQ2ZmUOWSrkBp2Ob1D4Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1F59718C89C4; Fri, 28 Jan 2022 16:07:33 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.193.202]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD78384A2A; Fri, 28 Jan 2022 16:07:31 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 28 Jan 2022 17:04:41 +0100 Message-Id: <20220128160441.23477-19-amorenoz@redhat.com> In-Reply-To: <20220128160441.23477-1-amorenoz@redhat.com> References: <20220128160441.23477-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2 18/18] python: add unit tests for filtering engine X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit test for OFFilter class. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/tests/test_filter.py | 221 ++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 python/ovs/tests/test_filter.py diff --git a/python/automake.mk b/python/automake.mk index 44c1035a3..fac32ff2b 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -54,6 +54,7 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ + python/ovs/tests/test_filter.py \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ python/ovs/tests/test_odp.py \ diff --git a/python/ovs/tests/test_filter.py b/python/ovs/tests/test_filter.py new file mode 100644 index 000000000..823ad2b48 --- /dev/null +++ b/python/ovs/tests/test_filter.py @@ -0,0 +1,221 @@ +import pytest + +from ovs.flows.filter import OFFilter +from ovs.flows.ofp import OFPFlow +from ovs.flows.odp import ODPFlow + + +@pytest.mark.parametrize( + "expr,flow,expected,match", + [ + ( + "nw_src=192.168.1.1 && tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 || tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1 || tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 && tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + False, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1/30", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "n_bytes=100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>10", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<1000", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes", "drop"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes>10 && !output.port=3", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["n_bytes", "output"], + ), + ( + "dl_src=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + False, + ["dl_src", "tp_dst"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst~=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + True, + ["dl_src", "tp_dst"], + ), + ( + "encap", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ( + "encap.ipv4.src=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + False, + ["encap"], + ), + ( + "encap.ipv4.src~=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ], +) +def test_filter(expr, flow, expected, match): + ffilter = OFFilter(expr) + result = ffilter.evaluate(flow) + if expected: + assert result + else: + assert not result + + assert [kv.key for kv in result.kv] == match