From patchwork Fri Mar 11 15:21:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604408 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Qvwsfxfh; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV666j2gz9sG3 for ; Sat, 12 Mar 2022 02:22:06 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id BA16F419E2; Fri, 11 Mar 2022 15:22:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UNn2a7NQ10lD; Fri, 11 Mar 2022 15:22:01 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 2BEEE419E0; Fri, 11 Mar 2022 15:22:00 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6EA79C000B; Fri, 11 Mar 2022 15:21:58 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4E2A2C000B for ; Fri, 11 Mar 2022 15:21:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2E3EB841AC for ; Fri, 11 Mar 2022 15:21:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8oGPCZ-95EtU for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id BD527841A6 for ; Fri, 11 Mar 2022 15:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y7i5M5FwIT3UF+fORIZK8IzYhnY+14QE0yTF4dLROOQ=; b=QvwsfxfhNpImd7cS0MxHqGmu3r8UYZN+EQVVIh8pS/4sgDGRz3dQhmMa77ZgCw8J7T6yTv Tj2xCPobsBuywIlQPLKHNVt7C3/kF2Cpm4RhOP4g5uIM8HDc3LST4mJchqWlsjz4ru1dnM 3+pp3Vp8oESkPDQkTAuZztQB7LWvLLw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-202-wB8sxqQkMvOuVt28pXZF4g-1; Fri, 11 Mar 2022 10:21:50 -0500 X-MC-Unique: wB8sxqQkMvOuVt28pXZF4g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1528D1091DA1 for ; Fri, 11 Mar 2022 15:21:48 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9D9537FFF1; Fri, 11 Mar 2022 15:21:46 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:11 +0100 Message-Id: <20220311152128.3988946-2-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 01/18] python: add generic Key-Value parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Most of ofproto and dpif flows are based on key-value pairs. These key-value pairs can be represented in several ways, eg: key:value, key=value, key(value). Add the following classes that allow parsing of key-value strings: * KeyValue: holds a key-value pair * KeyMetadata: holds some metadata associated with a KeyValue such as the original key and value strings and their position in the global string * KVParser: is able to parse a string and extract it's key-value pairs as KeyValue instances. Before creating the KeyValue instance it tries to decode the value via the KVDecoders * KVDecoders holds a number of decoders that KVParser can use to decode key-value pairs. It accepts a dictionary of keys and callables to allow users to specify what decoder (i.e: callable) to use for each key Also, flake8 seems to be incorrectly reporting an error (E203) in: "slice[index + offset : index + offset]" which is PEP8 compliant. So, ignore this error. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron Acked-by: Terry Wilson --- Makefile.am | 3 +- python/automake.mk | 6 +- python/ovs/flows/__init__.py | 0 python/ovs/flows/decoders.py | 18 ++ python/ovs/flows/kv.py | 314 +++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 6 files changed, 340 insertions(+), 3 deletions(-) create mode 100644 python/ovs/flows/__init__.py create mode 100644 python/ovs/flows/decoders.py create mode 100644 python/ovs/flows/kv.py diff --git a/Makefile.am b/Makefile.am index cb8076433..4f51d225e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -391,6 +391,7 @@ ALL_LOCAL += flake8-check # E128 continuation line under-indented for visual indent # E129 visually indented line with same indent as next logical line # E131 continuation line unaligned for hanging indent +# E203 whitespace before ':' # E722 do not use bare except, specify exception instead # W503 line break before binary operator # W504 line break after binary operator @@ -403,7 +404,7 @@ ALL_LOCAL += flake8-check # H233 Python 3.x incompatible use of print operator # H238 old style class declaration, use new style (inherit from `object`) FLAKE8_SELECT = H231,H232,H233,H238 -FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E722,W503,W504,F811,D,H,I +FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E203,E722,W503,W504,F811,D,H,I flake8-check: $(FLAKE8_PYFILES) $(FLAKE8_WERROR)$(AM_V_GEN) \ src='$^' && \ diff --git a/python/automake.mk b/python/automake.mk index 767512f17..7ce842d66 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -16,7 +16,6 @@ ovs_pyfiles = \ python/ovs/compat/sortedcontainers/sorteddict.py \ python/ovs/compat/sortedcontainers/sortedset.py \ python/ovs/daemon.py \ - python/ovs/fcntl_win.py \ python/ovs/db/__init__.py \ python/ovs/db/custom_index.py \ python/ovs/db/data.py \ @@ -26,6 +25,10 @@ ovs_pyfiles = \ python/ovs/db/schema.py \ python/ovs/db/types.py \ python/ovs/fatal_signal.py \ + python/ovs/fcntl_win.py \ + python/ovs/flows/__init__.py \ + python/ovs/flows/decoders.py \ + python/ovs/flows/kv.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ @@ -42,6 +45,7 @@ ovs_pyfiles = \ python/ovs/version.py \ python/ovs/vlog.py \ python/ovs/winutils.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ diff --git a/python/ovs/flows/__init__.py b/python/ovs/flows/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py new file mode 100644 index 000000000..0c2259c76 --- /dev/null +++ b/python/ovs/flows/decoders.py @@ -0,0 +1,18 @@ +"""Defines helpful decoders that can be used to decode information from the +flows. + +A decoder is generally a callable that accepts a string and returns the value +object. +""" + + +def decode_default(value): + """Default decoder. + + It tries to convert into an integer value and, if it fails, just + returns the string. + """ + try: + return int(value, 0) + except ValueError: + return value diff --git a/python/ovs/flows/kv.py b/python/ovs/flows/kv.py new file mode 100644 index 000000000..9fd87c3cc --- /dev/null +++ b/python/ovs/flows/kv.py @@ -0,0 +1,314 @@ +"""Common helper classes for flow Key-Value parsing.""" + +import functools +import re + +from ovs.flows.decoders import decode_default + + +class ParseError(RuntimeError): + """Exception raised when an error occurs during parsing.""" + + pass + + +class KeyMetadata(object): + """Class for keeping key metadata. + + Attributes: + kpos (int): The position of the keyword in the parent string. + vpos (int): The position of the value in the parent string. + kstring (string): The keyword string as found in the flow string. + vstring (string): The value as found in the flow string. + delim (string): Optional, the string used as delimiter between the key + and the value. + end_delim (string): Optional, the string used as end delimiter + """ + + def __init__(self, kpos, vpos, kstring, vstring, delim="", end_delim=""): + """Constructor.""" + self.kpos = kpos + self.vpos = vpos + self.kstring = kstring + self.vstring = vstring + self.delim = delim + self.end_delim = end_delim + + def __str__(self): + return "key: [{},{}), val:[{}, {})".format( + self.kpos, + self.kpos + len(self.kstring), + self.vpos, + self.vpos + len(self.vstring), + ) + + def __repr__(self): + return "{}('{}')".format(self.__class__.__name__, self) + + +class KeyValue(object): + """Class for keeping key-value data. + + Attributes: + key (str): The key string. + value (any): The value data. + meta (KeyMetadata): The key metadata. + """ + + def __init__(self, key, value, meta=None): + """Constructor.""" + self.key = key + self.value = value + self.meta = meta + + def __str__(self): + return "{}: {} ({})".format(self.key, str(self.value), str(self.meta)) + + def __repr__(self): + return "{}('{}')".format(self.__class__.__name__, self) + + +class KVDecoders(object): + """KVDecoders class is used by KVParser to select how to decode the value + of a specific keyword. + + A decoder is simply a function that accepts a value string and returns + the value objects to be stored. + The returned value may be of any type. + + Decoders may return a KeyValue instance to indicate that the keyword should + also be modified to match the one provided in the returned KeyValue. + + The decoder to be used will be selected using the key as an index. If not + found, the default decoder will be used. If free keys are found (i.e: + keys without a value), the default_free decoder will be used. For that + reason, the default_free decoder, must return both the key and value to be + stored. + + Args: + decoders (dict): Optional; A dictionary of decoders indexed by keyword. + default (callable): Optional; A decoder used if a match is not found in + configured decoders. If not provided, the default behavior is to + try to decode the value into an integer and, if that fails, + just return the string as-is. + default_free (callable): Optional; The decoder used if a match is not + found in configured decoders and it's a free value (e.g: + a value without a key) Defaults to returning the free value as + keyword and "True" as value. + The callable must accept a string and return a key-value pair. + """ + + def __init__(self, decoders=None, default=None, default_free=None): + self._decoders = decoders or dict() + self._default = default or decode_default + self._default_free = default_free or self._default_free_decoder + + def decode(self, keyword, value_str): + """Decode a keyword and value. + + Args: + keyword (str): The keyword whose value is to be decoded. + value_str (str): The value string. + + Returns: + The key (str) and value(any) to be stored. + """ + + decoder = self._decoders.get(keyword) + if decoder: + result = decoder(value_str) + if isinstance(result, KeyValue): + keyword = result.key + value = result.value + else: + value = result + + return keyword, value + else: + if value_str: + return keyword, self._default(value_str) + else: + return self._default_free(keyword) + + @staticmethod + def _default_free_decoder(key): + """Default decoder for free keywords.""" + return key, True + + +delim_pattern = re.compile(r"(\(|=|:|,|\n|\r|\t)") +parenthesis = re.compile(r"(\(|\))") +end_pattern = re.compile(r"( |,|\n|\r|\t)") +separators = (" ", ",") +end_of_string = (",", "\n", "\t", "\r", "") + + +class KVParser(object): + """KVParser parses a string looking for key-value pairs. + + Args: + string (str): The string to parse. + decoders (KVDecoders): Optional; the KVDecoders instance to use. + """ + + def __init__(self, string, decoders=None): + """Constructor.""" + self._decoders = decoders or KVDecoders() + self._keyval = list() + self._string = string + + def keys(self): + return list(kv.key for kv in self._keyval) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the key-value pairs in string. + + The input string is assumed to contain a list of comma (or space) + separated key-value pairs. + + Key-values pairs can have multiple different delimiters, eg: + "key1:value1,key2=value2,key3(value3)". + + Also, we can stumble upon a "free" keywords, e.g: + "key1=value1,key2=value2,free_keyword". + We consider this as keys without a value. + + So, to parse the string we do the following until the end of the + string is found: + + 1 - Skip any leading comma's or spaces. + 2 - Find the next delimiter (or end_of_string character). + 3 - Depending on the delimiter, obtain the key and the value. + For instance, if the delimiter is "(", find the next matching + ")". + 4 - Use the KVDecoders to decode the key-value. + 5 - Store the KeyValue object with the corresponding metadata. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + keyword = "" + delimiter = "" + rest = "" + + # 1. Skip separator characters. + if self._string[kpos] in separators: + kpos += 1 + continue + + # 2. Find the next delimiter or end of string character. + try: + keyword, delimiter, rest = delim_pattern.split( + self._string[kpos:], 1 + ) + except ValueError: + keyword = self._string[kpos:] # Free keyword + + # 3. Extract the value from the rest of the string. + value_str = "" + vpos = kpos + len(keyword) + 1 + end_delimiter = "" + + if delimiter in ("=", ":"): + # If the delimiter is ':' or '=', the end of the value is the + # end of the string or a ', '. + value_parts = end_pattern.split(rest, 1) + value_str = value_parts[0] + next_kpos = vpos + len(value_str) + + elif delimiter == "(": + # Find matching ")". + level = 1 + index = 0 + value_parts = parenthesis.split(rest) + for val in value_parts: + if val == "(": + level += 1 + elif val == ")": + level -= 1 + index += len(val) + if level == 0: + break + + if level != 0: + raise ParseError( + "Error parsing string {}: " + "Failed to find matching ')' in {}".format( + self._string, rest + ) + ) + + value_str = rest[: index - 1] + next_kpos = vpos + len(value_str) + 1 + end_delimiter = ")" + + # Exceptionally, if after the () we find -> {}, do not treat + # the content of the parenthesis as the value, consider + # ({})->{} as the string value. + if index < len(rest) - 2 and rest[index : index + 2] == "->": + extra_val = rest[index + 2 :].split(",")[0] + value_str = "({})->{}".format(value_str, extra_val) + # remove the first "(". + vpos -= 1 + next_kpos = vpos + len(value_str) + end_delimiter = "" + + elif delimiter in end_of_string: + # Key without a value. + next_kpos = kpos + len(keyword) + vpos = -1 + + # 4. Use KVDecoders to decode the key-value. + try: + key, val = self._decoders.decode(keyword, value_str) + except Exception as e: + raise ParseError( + "Error parsing key-value ({}, {})".format( + keyword, value_str + ) + ) from e + + # Store the KeyValue object with the corresponding metadata. + meta = KeyMetadata( + kpos=kpos, + vpos=vpos, + kstring=keyword, + vstring=value_str, + delim=delimiter, + end_delim=end_delimiter, + ) + + self._keyval.append(KeyValue(key, val, meta)) + + kpos = next_kpos + + +def decode_nested_kv(decoders, value): + """A key-value decoder that extracts nested key-value pairs and returns + them in a dictionary. + + Args: + decoders (KVDecoders): The KVDecoders to use. + value (str): The value string to decode. + """ + if not value: + # Mark as flag + return True + + parser = KVParser(value, decoders) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_kv_decoder(decoders=None): + """Helper function that creates a nested kv decoder with given + KVDecoders.""" + return functools.partial(decode_nested_kv, decoders) diff --git a/python/setup.py b/python/setup.py index cfe01763f..0e6b0ea39 100644 --- a/python/setup.py +++ b/python/setup.py @@ -71,7 +71,7 @@ setup_args = dict( author='Open vSwitch', author_email='dev@openvswitch.org', packages=['ovs', 'ovs.compat', 'ovs.compat.sortedcontainers', - 'ovs.db', 'ovs.unixctl'], + 'ovs.db', 'ovs.unixctl', 'ovs.flows'], keywords=['openvswitch', 'ovs', 'OVSDB'], license='Apache 2.0', classifiers=[ From patchwork Fri Mar 11 15:21:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604406 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=SAePyG0b; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV5z6dsmz9sG3 for ; Sat, 12 Mar 2022 02:21:59 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D6AD0418EE; Fri, 11 Mar 2022 15:21:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L_RZXVB6avNG; Fri, 11 Mar 2022 15:21:56 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id CBF4641529; Fri, 11 Mar 2022 15:21:55 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A5803C0012; Fri, 11 Mar 2022 15:21:55 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id C168DC000B for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A9FF24159E for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Fm6hoyOKYQ1d for ; Fri, 11 Mar 2022 15:21:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4568941529 for ; Fri, 11 Mar 2022 15:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012111; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m1WYrtpdNM8l3Q4cXPoor508Ep6fRu9afNz3WAGnGD0=; b=SAePyG0bN7ZCOlHczhH2UMbe3zKQ/cfC0LzCvS6OmK8xl3AQzgmMSdkybOfj6SEOIu0nnH Pxdlbyx3Hyp5235Nh5i0cCV6Rk/sBvekR2e+wAcLZf14ITk1jcmoeUDo+2u1uZLK3BEs9l YsTEEtQ9c66jAgkFRY0qEbT8QdNDrB4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-mqXRN99zNRyK_5gykg45WQ-1; Fri, 11 Mar 2022 10:21:50 -0500 X-MC-Unique: mqXRN99zNRyK_5gykg45WQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8FC871091DA3 for ; Fri, 11 Mar 2022 15:21:49 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F6FB7FFF1; Fri, 11 Mar 2022 15:21:48 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:12 +0100 Message-Id: <20220311152128.3988946-3-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 02/18] python: add mask, ip and eth decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add more decoders that can be used by KVParser. For IPv4 and IPv6 addresses, create a new class that wraps netaddr.IPAddress. For Ethernet addresses, create a new class that wraps netaddr.EUI. For Integers, create a new class that performs basic bitwise mask comparisons Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/ovs/flows/decoders.py | 398 +++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 2 files changed, 399 insertions(+), 1 deletion(-) diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 0c2259c76..883e61acf 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -5,6 +5,15 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import netaddr + + +class Decoder(object): + """Base class for all decoder classes.""" + + def to_json(self): + raise NotImplementedError() + def decode_default(value): """Default decoder. @@ -16,3 +25,392 @@ def decode_default(value): return int(value, 0) except ValueError: return value + + +def decode_flag(value): + """Decode a flag. It's existence is just flagged by returning True.""" + return True + + +def decode_int(value): + """Integer decoder. + + Both base10 and base16 integers are supported. + + Used for fields such as: + n_bytes=34 + metadata=0x4 + """ + return int(value, 0) + + +def decode_time(value): + """Time decoder. + + Used for fields such as: + duration=1234.123s + """ + if value == "never": + return value + + time_str = value.rstrip("s") + return float(time_str) + + +class IntMask(Decoder): + """Base class for Integer Mask decoder classes. + + It supports decoding a value/mask pair. The class has to be derived, + and the size attribute must be set. + """ + + size = None # Size in bits. + + def __init__(self, string): + if not self.size: + raise NotImplementedError( + "IntMask should be derived and size should be fixed" + ) + + parts = string.split("/") + if len(parts) > 1: + self._value = int(parts[0], 0) + self._mask = int(parts[1], 0) + if self._mask.bit_length() > self.size: + raise ValueError( + "Integer mask {} is bigger than size {}".format( + self._mask, self.size + ) + ) + else: + self._value = int(parts[0], 0) + self._mask = self.max_mask() + + if self._value.bit_length() > self.size: + raise ValueError( + "Integer value {} is bigger than size {}".format( + self._value, self.size + ) + ) + + @property + def value(self): + return self._value + + @property + def mask(self): + return self._mask + + def max_mask(self): + return 2 ** self.size - 1 + + def fully(self): + """Returns True if it's fully masked.""" + return self._mask == self.max_mask() + + def __str__(self): + if self.fully(): + return str(self._value) + else: + return "{}/{}".format(hex(self._value), hex(self._mask)) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def __eq__(self, other): + """Equality operator. + + Both value and mask must be the same for the comparison to result True. + This can be used to implement filters that expect a specific mask, + e.g: ct.state = 0x1/0xff. + + Args: + other (IntMask): Another IntMask to compare against. + + Returns: + True if the other IntMask is the same as this one. + """ + if isinstance(other, IntMask): + return self.value == other.value and self.mask == other.mask + elif isinstance(other, int): + return self.value == other and self.mask == self.max_mask() + else: + raise ValueError("Cannot compare against ", other) + + def __contains__(self, other): + """Contains operator. + + Args: + other (int or IntMask): Another integer or fully-masked IntMask + to compare against. + + Returns: + True if the other integer or fully-masked IntMask is + contained in this IntMask. + + Example: + 0x1 in IntMask("0xf1/0xff"): True + 0x1 in IntMask("0xf1/0x0f"): True + 0x1 in IntMask("0xf1/0xf0"): False + """ + if isinstance(other, IntMask): + if other.fully(): + return other.value in self + else: + raise ValueError( + "Comparing non fully-masked IntMasks is not supported" + ) + else: + return other & self._mask == self._value & self._mask + + def dict(self): + return {"value": self._value, "mask": self._mask} + + def to_json(self): + return self.dict() + + +class Mask8(IntMask): + size = 8 + + +class Mask16(IntMask): + size = 16 + + +class Mask32(IntMask): + size = 32 + + +class Mask64(IntMask): + size = 64 + + +class Mask128(IntMask): + size = 128 + + +class Mask992(IntMask): + size = 992 + + +def decode_mask(mask_size): + """Value/Mask decoder for values of specific size (bits). + + Used for fields such as: + reg0=0x248/0xff + """ + + class Mask(IntMask): + size = mask_size + __name__ = "Mask{}".format(size) + + return Mask + + +class EthMask(Decoder): + """EthMask represents an Ethernet address with optional mask. + + It uses netaddr.EUI. + + Attributes: + eth (netaddr.EUI): The Ethernet address. + mask (netaddr.EUI): Optional, the Ethernet address mask. + + Args: + string (str): A string representing the masked Ethernet address + e.g: 00.11:22:33:44:55 or 01:00:22:00:33:00/01:00:00:00:00:00 + """ + + def __init__(self, string): + mask_parts = string.split("/") + self._eth = netaddr.EUI(mask_parts[0]) + if len(mask_parts) == 2: + self._mask = netaddr.EUI(mask_parts[1]) + else: + self._mask = None + + @property + def eth(self): + """The Ethernet address.""" + return self._eth + + @property + def mask(self): + """The Ethernet address mask.""" + return self._mask + + def __eq__(self, other): + """Equality operator. + + Both the Ethernet address and the mask are compared. This can be used + to implement filters where we expect a specific mask to be present, + e.g: dl_dst=01:00:00:00:00:00/01:00:00:00:00:00. + + Args: + other (EthMask): Another EthMask to compare against. + + Returns: + True if this EthMask is the same as the other. + """ + return self._mask == other._mask and self._eth == other._eth + + def __contains__(self, other): + """Contains operator. + + Args: + other (netaddr.EUI or EthMask): An Ethernet address. + + Returns: + True if the other netaddr.EUI or fully-masked EthMask is + contained in this EthMask's address range. + """ + if isinstance(other, EthMask): + if other._mask: + raise ValueError( + "Comparing non fully-masked EthMask is not supported" + ) + return other._eth in self + + if self._mask: + return (other.value & self._mask.value) == ( + self._eth.value & self._mask.value + ) + else: + return other == self._eth + + def __str__(self): + if self._mask: + return "/".join( + [ + self._eth.format(netaddr.mac_unix), + self._mask.format(netaddr.mac_unix), + ] + ) + else: + return self._eth.format(netaddr.mac_unix) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) + + +class IPMask(Decoder): + """IPMask stores an IPv6 or IPv4 and a mask. + + It uses netaddr.IPAddress. + + IPMasks can represent valid CIDRs or randomly masked IP Addresses. + + Args: + string (str): A string representing the ip/mask. + """ + + def __init__(self, string): + self._ipnet = None + self._ip = None + self._mask = None + try: + self._ipnet = netaddr.IPNetwork(string) + except netaddr.AddrFormatError: + pass + + if not self._ipnet: + # It's not a valid CIDR. Store ip and mask independently. + parts = string.split("/") + if len(parts) != 2: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) + try: + self._ip = netaddr.IPAddress(parts[0]) + self._mask = netaddr.IPAddress(parts[1]) + except netaddr.AddrFormatError as exc: + raise ValueError( + "value {}: is not an ipv4 or ipv6 address".format(string) + ) from exc + + def __eq__(self, other): + """Equality operator. + + Both the IPAddress and the mask are compared. This can be used + to implement filters where a specific mask is expected, e.g: + nw_src=192.168.1.0/24. + + Args: + other (IPMask or netaddr.IPNetwork or netaddr.IPAddress): + Another IPAddress or IPNetwork to compare against. + + Returns: + True if this IPMask is the same as the other. + """ + if isinstance(other, netaddr.IPNetwork): + return self._ipnet and self._ipnet == other + if isinstance(other, netaddr.IPAddress): + return self._ipnet and self._ipnet.ip == other + elif isinstance(other, IPMask): + if self._ipnet: + return self._ipnet == other._ipnet + + return self._ip == other._ip and self._mask == other._mask + else: + return False + + def __contains__(self, other): + """Contains operator. + + Only comparing valid CIDRs is supported. + + Args: + other (netaddr.IPAddress or IPMask): An IP address. + + Returns: + True if the other IPAddress is contained in this IPMask's address + range. + """ + if isinstance(other, IPMask): + if not other._ipnet: + raise ValueError("Only comparing valid CIDRs is supported") + + return ( + netaddr.IPAddress(other._ipnet.first) in self + and netaddr.IPAddress(other._ipnet.last) in self + ) + + elif isinstance(other, netaddr.IPAddress): + if self._ipnet: + return other in self._ipnet + return (other & self._mask) == (self._ip & self._mask) + + def cidr(self): + """ + Returns True if the IPMask is a valid CIDR. + """ + return self._ipnet is not None + + @property + def ip(self): + """The IP address.""" + if self._ipnet: + return self._ipnet.ip + return self._ip + + @property + def mask(self): + """The IP mask.""" + if self._ipnet: + return self._ipnet.netmask + return self._mask + + def __str__(self): + if self._ipnet: + return str(self._ipnet) + return "/".join([str(self._ip), str(self._mask)]) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def to_json(self): + return str(self) diff --git a/python/setup.py b/python/setup.py index 0e6b0ea39..b06370bd9 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers'], + install_requires=['sortedcontainers', 'netaddr'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Fri Mar 11 15:21:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604409 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LZWzXQ4W; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV694tTTz9sG3 for ; Sat, 12 Mar 2022 02:22:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8650E61317; Fri, 11 Mar 2022 15:22:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q5WI8RLHqj7E; Fri, 11 Mar 2022 15:22:03 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id E4F25612ED; Fri, 11 Mar 2022 15:22:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9FB0CC0088; Fri, 11 Mar 2022 15:21:59 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 49FC9C000B for ; Fri, 11 Mar 2022 15:21:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 04C4940283 for ; Fri, 11 Mar 2022 15:21:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N7FQvRdeYLhu for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id C3CE740207 for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PUvHfeWt3IQpFQV+PDz0U/26OWGtNRyURVjtsXI/LuU=; b=LZWzXQ4W30YNs9QkyPnUwrRbwzbX1ie6VQp95svqQb3diyN/WqGV+ur+grD6sxF3Ag/Fly q2VLG/WeqAzifN9ZmtH+IpJreKorr1OhsGBUjOMchn2iBMD8S/dx3D2VTw6oG9Y29LhKJ6 4hbTzYesCsNI+s7uhkDc5t40u7bVgdA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-TB5bm5syPiiYoaiLT2IOVg-1; Fri, 11 Mar 2022 10:21:52 -0500 X-MC-Unique: TB5bm5syPiiYoaiLT2IOVg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53DED1854E21 for ; Fri, 11 Mar 2022 15:21:51 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id F1C567F0DE; Fri, 11 Mar 2022 15:21:49 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:13 +0100 Message-Id: <20220311152128.3988946-4-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 03/18] python: add list parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Some openflow or dpif flows encode their arguments in lists, eg: "some_action(arg1,arg2,arg3)". In order to decode this in a way that can be then stored and queried, add ListParser and ListDecoders classes that parse lists into KeyValue instances. The ListParser / ListDecoders mechanism is quite similar to KVParser and KVDecoders. Since the "key" of the different KeyValue objects is now ommited, it has to be provided by ListDecoders. For example, take the openflow action "resubmit" that can be written as: resubmit([port],[table][,ct]) Can be decoded by creating a ListDecoders instance such as: ListDecoders([ ("port", decode_default), ("table", decode_int), ("ct", decode_flag), ]) Naturally, the order of the decoders must be kept. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/flows/list.py | 121 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 python/ovs/flows/list.py diff --git a/python/automake.mk b/python/automake.mk index 7ce842d66..73438d615 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -29,6 +29,7 @@ ovs_pyfiles = \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ python/ovs/flows/kv.py \ + python/ovs/flows/list.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py new file mode 100644 index 000000000..57e7c5908 --- /dev/null +++ b/python/ovs/flows/list.py @@ -0,0 +1,121 @@ +import re + +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError +from ovs.flows.decoders import decode_default + + +class ListDecoders(object): + """ListDecoders is used by ListParser to decode the elements in the list. + + A decoder is a function that accepts a value and returns its decoded + object. + + ListDecoders is initialized with a list of tuples that contains the + keyword and the decoding function associated with each position in the + list. The order is, therefore, important. + + Args: + decoders (list of tuples): Optional; a list of tuples. + The first element in the tuple is the keyword associated with the + value. The second element in the tuple is the decoder function. + """ + + def __init__(self, decoders=None): + self._decoders = decoders or list() + + def decode(self, index, value_str): + """Decode the index'th element of the list. + + Args: + index (int): The position in the list of the element to decode. + value_str (str): The value string to decode. + """ + if index < 0 or index >= len(self._decoders): + return self._default_decoder(index, value_str) + + try: + key = self._decoders[index][0] + value = self._decoders[index][1](value_str) + return key, value + except Exception as e: + raise ParseError( + "Failed to decode value_str {}: {}".format(value_str, str(e)) + ) + + @staticmethod + def _default_decoder(index, value): + key = "elem_{}".format(index) + return key, decode_default(value) + + +class ListParser(object): + """ListParser parses a list of values and stores them as key-value pairs. + + It uses a ListDecoders instance to decode each element in the list. + + Args: + string (str): The string to parse. + decoders (ListDecoders): Optional, the decoders to use. + delims (list): Optional, list of delimiters of the list. Defaults to + [',']. + """ + def __init__(self, string, decoders=None, delims=[","]): + self._string = string + self._decoders = decoders or ListDecoders() + self._keyval = list() + self._regexp = r"({})".format("|".join(delims)) + + def kv(self): + return self._keyval + + def __iter__(self): + return iter(self._keyval) + + def parse(self): + """Parse the list in string. + + Raises: + ParseError if any parsing error occurs. + """ + kpos = 0 + index = 0 + while kpos < len(self._string) and self._string[kpos] != "\n": + split_parts = re.split(self._regexp, self._string[kpos:], 1) + value_str = split_parts[0] + + key, value = self._decoders.decode(index, value_str) + + meta = KeyMetadata( + kpos=kpos, + vpos=kpos, + kstring=value_str, + vstring=value_str, + ) + self._keyval.append(KeyValue(key, value, meta)) + + kpos += len(value_str) + 1 + index += 1 + + +def decode_nested_list(decoders, value, delims=[","]): + """Decodes a string value that contains a list of elements and returns + them in a dictionary. + + Args: + decoders (ListDecoders): The ListDecoders to use. + value (str): The value string to decode. + delims (list(str)): Optional, the list of delimiters to use. + """ + parser = ListParser(value, decoders, delims) + parser.parse() + return {kv.key: kv.value for kv in parser.kv()} + + +def nested_list_decoder(decoders=None, delims=[","]): + """Helper function that creates a nested list decoder with given + ListDecoders and delimiters. + """ + def decoder(value): + return decode_nested_list(decoders, value, delims) + + return decoder From patchwork Fri Mar 11 15:21:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604411 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Oext3586; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV6G2QbSz9sG3 for ; Sat, 12 Mar 2022 02:22:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A0C6C40C2F; Fri, 11 Mar 2022 15:22:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3ZZHaii9zbrf; Fri, 11 Mar 2022 15:22:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 15A8A40BA4; Fri, 11 Mar 2022 15:22:06 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 835F3C0033; Fri, 11 Mar 2022 15:22:03 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 65B0DC0089 for ; Fri, 11 Mar 2022 15:22:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5291340B93 for ; Fri, 11 Mar 2022 15:22:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 35mpTVbjEwTo for ; Fri, 11 Mar 2022 15:21:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1B0A340283 for ; Fri, 11 Mar 2022 15:21:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2j5/caXeCfqwiSl+BgiK95I41PxvDmVX+GcjjGkdFDc=; b=Oext3586R0ua3+zn5TJcBjaQ/ScutZpEbnSf1buqgIRCmqHnYxiH3elM4FGoO3eO1jxCmQ 0P81Vcdscp+1wE6eJh+sKKSZW2+69BehzBkLtli7MacqNa5hLGMz+Wacnx+lwLGRtNk3q6 u62P8UlUPRjrXAwGUr04MXuaOM9bXBs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-592-rpOa_QonPcGX1L9NawfYgw-1; Fri, 11 Mar 2022 10:21:53 -0500 X-MC-Unique: rpOa_QonPcGX1L9NawfYgw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1B09F1854E26 for ; Fri, 11 Mar 2022 15:21:53 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id AFBEC7F0D7; Fri, 11 Mar 2022 15:21:51 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:14 +0100 Message-Id: <20220311152128.3988946-5-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 04/18] build-aux: split extract-ofp-fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to be able to reuse the core extraction logic, split the command in two parts. The core extraction logic is moved to python/build while the command that writes the different files out of the extracted field info is kept in build-aux. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- build-aux/extract-ofp-fields | 706 ++++++++--------------------- python/automake.mk | 7 +- python/build/extract_ofp_fields.py | 421 +++++++++++++++++ 3 files changed, 618 insertions(+), 516 deletions(-) create mode 100644 python/build/extract_ofp_fields.py diff --git a/build-aux/extract-ofp-fields b/build-aux/extract-ofp-fields index 8766995d9..efec59c25 100755 --- a/build-aux/extract-ofp-fields +++ b/build-aux/extract-ofp-fields @@ -3,85 +3,23 @@ import getopt import sys import os.path -import re import xml.dom.minidom import build.nroff -line = "" - -# Maps from user-friendly version number to its protocol encoding. -VERSION = {"1.0": 0x01, - "1.1": 0x02, - "1.2": 0x03, - "1.3": 0x04, - "1.4": 0x05, - "1.5": 0x06} -VERSION_REVERSE = dict((v,k) for k, v in VERSION.items()) - -TYPES = {"u8": (1, False), - "be16": (2, False), - "be32": (4, False), - "MAC": (6, False), - "be64": (8, False), - "be128": (16, False), - "tunnelMD": (124, True)} - -FORMATTING = {"decimal": ("MFS_DECIMAL", 1, 8), - "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), - "ct state": ("MFS_CT_STATE", 4, 4), - "Ethernet": ("MFS_ETHERNET", 6, 6), - "IPv4": ("MFS_IPV4", 4, 4), - "IPv6": ("MFS_IPV6", 16, 16), - "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), - "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), - "frag": ("MFS_FRAG", 1, 1), - "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), - "TCP flags": ("MFS_TCP_FLAGS", 2, 2), - "packet type": ("MFS_PACKET_TYPE", 4, 4)} - -PREREQS = {"none": "MFP_NONE", - "Ethernet": "MFP_ETHERNET", - "ARP": "MFP_ARP", - "VLAN VID": "MFP_VLAN_VID", - "IPv4": "MFP_IPV4", - "IPv6": "MFP_IPV6", - "IPv4/IPv6": "MFP_IP_ANY", - "NSH": "MFP_NSH", - "CT": "MFP_CT_VALID", - "MPLS": "MFP_MPLS", - "TCP": "MFP_TCP", - "UDP": "MFP_UDP", - "SCTP": "MFP_SCTP", - "ICMPv4": "MFP_ICMPV4", - "ICMPv6": "MFP_ICMPV6", - "ND": "MFP_ND", - "ND solicit": "MFP_ND_SOLICIT", - "ND advert": "MFP_ND_ADVERT"} - -# Maps a name prefix into an (experimenter ID, class) pair, so: -# -# - Standard OXM classes are written as (0, ) -# -# - Experimenter OXM classes are written as (, 0xffff) -# -# If a name matches more than one prefix, the longest one is used. -OXM_CLASSES = {"NXM_OF_": (0, 0x0000, 'extension'), - "NXM_NX_": (0, 0x0001, 'extension'), - "NXOXM_NSH_": (0x005ad650, 0xffff, 'extension'), - "OXM_OF_": (0, 0x8000, 'standard'), - "OXM_OF_PKT_REG": (0, 0x8001, 'standard'), - "ONFOXM_ET_": (0x4f4e4600, 0xffff, 'standard'), - "ERICOXM_OF_": (0, 0x1000, 'extension'), - - # This is the experimenter OXM class for Nicira, which is the - # one that OVS would be using instead of NXM_OF_ and NXM_NX_ - # if OVS didn't have those grandfathered in. It is currently - # used only to test support for experimenter OXM, since there - # are barely any real uses of experimenter OXM in the wild. - "NXOXM_ET_": (0x00002320, 0xffff, 'extension')} +from build.extract_ofp_fields import ( + extract_ofp_fields, + PREREQS, + OXM_CLASSES, + VERSION, + fatal, + n_errors, +) + +VERSION_REVERSE = dict((v, k) for k, v in VERSION.items()) + def oxm_name_to_class(name): - prefix = '' + prefix = "" class_ = None for p, c in OXM_CLASSES.items(): if name.startswith(p) and len(p) > len(prefix): @@ -92,267 +30,76 @@ def oxm_name_to_class(name): def is_standard_oxm(name): oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) - return oxm_class_type == 'standard' - - -def decode_version_range(range): - if range in VERSION: - return (VERSION[range], VERSION[range]) - elif range.endswith('+'): - return (VERSION[range[:-1]], max(VERSION.values())) - else: - a, b = re.match(r'^([^-]+)-([^-]+)$', range).groups() - return (VERSION[a], VERSION[b]) - - -def get_line(): - global line - global line_number - line = input_file.readline() - line_number += 1 - if line == "": - fatal("unexpected end of input") - - -n_errors = 0 - - -def error(msg): - global n_errors - sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) - n_errors += 1 - - -def fatal(msg): - error(msg) - sys.exit(1) + return oxm_class_type == "standard" def usage(): argv0 = os.path.basename(sys.argv[0]) - print('''\ + print( + """\ %(argv0)s, for extracting OpenFlow field properties from meta-flow.h usage: %(argv0)s INPUT [--meta-flow | --nx-match] where INPUT points to lib/meta-flow.h in the source directory. Depending on the option given, the output written to stdout is intended to be saved either as lib/meta-flow.inc or lib/nx-match.inc for the respective C file to #include.\ -''' % {"argv0": argv0}) +""" + % {"argv0": argv0} + ) sys.exit(0) -def make_sizeof(s): - m = re.match(r'(.*) up to (.*)', s) - if m: - struct, member = m.groups() - return "offsetof(%s, %s)" % (struct, member) - else: - return "sizeof(%s)" % s - - -def parse_oxms(s, prefix, n_bytes): - if s == 'none': - return () - - return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(',')) - - -match_types = dict() - - -def parse_oxm(s, prefix, n_bytes): - global match_types - - m = re.match('([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$', s) - if not m: - fatal("%s: syntax error parsing %s" % (s, prefix)) - - name, oxm_type, of_version, ovs_version = m.groups() - - class_ = oxm_name_to_class(name) - if class_ is None: - fatal("unknown OXM class for %s" % name) - oxm_vendor, oxm_class, oxm_class_type = class_ - - if class_ in match_types: - if oxm_type in match_types[class_]: - fatal("duplicate match type for %s (conflicts with %s)" % - (name, match_types[class_][oxm_type])) - else: - match_types[class_] = dict() - match_types[class_][oxm_type] = name - - # Normally the oxm_length is the size of the field, but for experimenter - # OXMs oxm_length also includes the 4-byte experimenter ID. - oxm_length = n_bytes - if oxm_class == 0xffff: - oxm_length += 4 - - header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) - - if of_version: - if oxm_class_type == 'extension': - fatal("%s: OXM extension can't have OpenFlow version" % name) - if of_version not in VERSION: - fatal("%s: unknown OpenFlow version %s" % (name, of_version)) - of_version_nr = VERSION[of_version] - if of_version_nr < VERSION['1.2']: - fatal("%s: claimed version %s predates OXM" % (name, of_version)) - else: - if oxm_class_type == 'standard': - fatal("%s: missing OpenFlow version number" % name) - of_version_nr = 0 - - return (header, name, of_version_nr, ovs_version) - - -def parse_field(mff, comment): - f = {'mff': mff} - - # First line of comment is the field name. - m = re.match(r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0]) - if not m: - fatal("%s lacks field name" % mff) - f['name'], f['extra_name'] = m.groups() - - # Find the last blank line the comment. The field definitions - # start after that. - blank = None - for i in range(len(comment)): - if not comment[i]: - blank = i - if not blank: - fatal("%s: missing blank line in comment" % mff) - - d = {} - for key in ("Type", "Maskable", "Formatting", "Prerequisites", - "Access", "Prefix lookup member", - "OXM", "NXM", "OF1.0", "OF1.1"): - d[key] = None - for fline in comment[blank + 1:]: - m = re.match(r'([^:]+):\s+(.*)\.$', fline) - if not m: - fatal("%s: syntax error parsing key-value pair as part of %s" - % (fline, mff)) - key, value = m.groups() - if key not in d: - fatal("%s: unknown key" % key) - elif key == 'Code point': - d[key] += [value] - elif d[key] is not None: - fatal("%s: duplicate key" % key) - d[key] = value - for key, value in d.items(): - if not value and key not in ("OF1.0", "OF1.1", - "Prefix lookup member", "Notes"): - fatal("%s: missing %s" % (mff, key)) - - m = re.match(r'([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$', d['Type']) - if not m: - fatal("%s: syntax error in type" % mff) - type_ = m.group(1) - if type_ not in TYPES: - fatal("%s: unknown type %s" % (mff, d['Type'])) - - f['n_bytes'] = TYPES[type_][0] - if m.group(2): - f['n_bits'] = int(m.group(2)) - if f['n_bits'] > f['n_bytes'] * 8: - fatal("%s: more bits (%d) than field size (%d)" - % (mff, f['n_bits'], 8 * f['n_bytes'])) - else: - f['n_bits'] = 8 * f['n_bytes'] - f['variable'] = TYPES[type_][1] - - if d['Maskable'] == 'no': - f['mask'] = 'MFM_NONE' - elif d['Maskable'] == 'bitwise': - f['mask'] = 'MFM_FULLY' - else: - fatal("%s: unknown maskable %s" % (mff, d['Maskable'])) - - fmt = FORMATTING.get(d['Formatting']) - if not fmt: - fatal("%s: unknown format %s" % (mff, d['Formatting'])) - f['formatting'] = d['Formatting'] - if f['n_bytes'] < fmt[1] or f['n_bytes'] > fmt[2]: - fatal("%s: %d-byte field can't be formatted as %s" - % (mff, f['n_bytes'], d['Formatting'])) - f['string'] = fmt[0] - - f['prereqs'] = d['Prerequisites'] - if f['prereqs'] not in PREREQS: - fatal("%s: unknown prerequisites %s" % (mff, d['Prerequisites'])) - - if d['Access'] == 'read-only': - f['writable'] = False - elif d['Access'] == 'read/write': - f['writable'] = True - else: - fatal("%s: unknown access %s" % (mff, d['Access'])) - - f['OF1.0'] = d['OF1.0'] - if not d['OF1.0'] in (None, 'exact match', 'CIDR mask'): - fatal("%s: unknown OF1.0 match type %s" % (mff, d['OF1.0'])) - - f['OF1.1'] = d['OF1.1'] - if not d['OF1.1'] in (None, 'exact match', 'bitwise mask'): - fatal("%s: unknown OF1.1 match type %s" % (mff, d['OF1.1'])) - - f['OXM'] = (parse_oxms(d['OXM'], 'OXM', f['n_bytes']) + - parse_oxms(d['NXM'], 'NXM', f['n_bytes'])) - - f['prefix'] = d["Prefix lookup member"] - - return f - - def protocols_to_c(protocols): - if protocols == set(['of10', 'of11', 'oxm']): - return 'OFPUTIL_P_ANY' - elif protocols == set(['of11', 'oxm']): - return 'OFPUTIL_P_NXM_OF11_UP' - elif protocols == set(['oxm']): - return 'OFPUTIL_P_NXM_OXM_ANY' + if protocols == set(["of10", "of11", "oxm"]): + return "OFPUTIL_P_ANY" + elif protocols == set(["of11", "oxm"]): + return "OFPUTIL_P_NXM_OF11_UP" + elif protocols == set(["oxm"]): + return "OFPUTIL_P_NXM_OXM_ANY" elif protocols == set([]): - return 'OFPUTIL_P_NONE' + return "OFPUTIL_P_NONE" else: assert False def autogen_c_comment(): return [ -"/* Generated automatically; do not modify! -*- buffer-read-only: t -*- */", -""] + "/* Generated automatically; do not modify! " + "-*- buffer-read-only: t -*- */", + "", + ] + def make_meta_flow(meta_flow_h): fields = extract_ofp_fields(meta_flow_h) output = autogen_c_comment() for f in fields: output += ["{"] - output += [" %s," % f['mff']] - if f['extra_name']: - output += [" \"%s\", \"%s\"," % (f['name'], f['extra_name'])] + output += [" %s," % f["mff"]] + if f["extra_name"]: + output += [' "%s", "%s",' % (f["name"], f["extra_name"])] else: - output += [" \"%s\", NULL," % f['name']] + output += [' "%s", NULL,' % f["name"]] - if f['variable']: - variable = 'true' + if f["variable"]: + variable = "true" else: - variable = 'false' - output += [" %d, %d, %s," % (f['n_bytes'], f['n_bits'], variable)] + variable = "false" + output += [" %d, %d, %s," % (f["n_bytes"], f["n_bits"], variable)] - if f['writable']: - rw = 'true' + if f["writable"]: + rw = "true" else: - rw = 'false' - output += [" %s, %s, %s, %s, false," - % (f['mask'], f['string'], PREREQS[f['prereqs']], rw)] - - oxm = f['OXM'] - of10 = f['OF1.0'] - of11 = f['OF1.1'] - if f['mff'] in ('MFF_DL_VLAN', 'MFF_DL_VLAN_PCP'): + rw = "false" + output += [ + " %s, %s, %s, %s, false," + % (f["mask"], f["string"], PREREQS[f["prereqs"]], rw) + ] + + oxm = f["OXM"] + of10 = f["OF1.0"] + of11 = f["OF1.1"] + if f["mff"] in ("MFF_DL_VLAN", "MFF_DL_VLAN_PCP"): # MFF_DL_VLAN and MFF_DL_VLAN_PCP don't exactly correspond to # OF1.1, nor do they have NXM or OXM assignments, but their # meanings can be expressed in every protocol, which is the goal of @@ -367,25 +114,25 @@ def make_meta_flow(meta_flow_h): if oxm: protocols |= set(["oxm"]) - if f['mask'] == 'MFM_FULLY': + if f["mask"] == "MFM_FULLY": cidr_protocols = protocols.copy() bitwise_protocols = protocols.copy() - if of10 == 'exact match': - bitwise_protocols -= set(['of10']) - cidr_protocols -= set(['of10']) - elif of10 == 'CIDR mask': - bitwise_protocols -= set(['of10']) + if of10 == "exact match": + bitwise_protocols -= set(["of10"]) + cidr_protocols -= set(["of10"]) + elif of10 == "CIDR mask": + bitwise_protocols -= set(["of10"]) else: assert of10 is None - if of11 == 'exact match': - bitwise_protocols -= set(['of11']) - cidr_protocols -= set(['of11']) + if of11 == "exact match": + bitwise_protocols -= set(["of11"]) + cidr_protocols -= set(["of11"]) else: - assert of11 in (None, 'bitwise mask') + assert of11 in (None, "bitwise mask") else: - assert f['mask'] == 'MFM_NONE' + assert f["mask"] == "MFM_NONE" cidr_protocols = set([]) bitwise_protocols = set([]) @@ -393,8 +140,8 @@ def make_meta_flow(meta_flow_h): output += [" %s," % protocols_to_c(cidr_protocols)] output += [" %s," % protocols_to_c(bitwise_protocols)] - if f['prefix']: - output += [" FLOW_U32OFS(%s)," % f['prefix']] + if f["prefix"]: + output += [" FLOW_U32OFS(%s)," % f["prefix"]] else: output += [" -1, /* not usable for prefix lookup */"] @@ -409,147 +156,37 @@ def make_nx_match(meta_flow_h): print("static struct nxm_field_index all_nxm_fields[] = {") for f in fields: # Sort by OpenFlow version number (nx-match.c depends on this). - for oxm in sorted(f['OXM'], key=lambda x: x[2]): - header = ("NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0]) - print("""{ .nf = { %s, %d, "%s", %s } },""" % ( - header, oxm[2], oxm[1], f['mff'])) + for oxm in sorted(f["OXM"], key=lambda x: x[2]): + header = "NXM_HEADER(0x%x,0x%x,%s,0,%d)" % oxm[0] + print( + """{ .nf = { %s, %d, "%s", %s } },""" + % (header, oxm[2], oxm[1], f["mff"]) + ) print("};") for oline in output: print(oline) -def extract_ofp_fields(fn): - global file_name - global input_file - global line_number - global line - - file_name = fn - input_file = open(file_name) - line_number = 0 - - fields = [] - - while True: - get_line() - if re.match('enum.*mf_field_id', line): - break - - while True: - get_line() - first_line_number = line_number - here = '%s:%d' % (file_name, line_number) - if (line.startswith('/*') - or line.startswith(' *') - or line.startswith('#') - or not line - or line.isspace()): - continue - elif re.match('}', line) or re.match('\s+MFF_N_IDS', line): - break - - # Parse the comment preceding an MFF_ constant into 'comment', - # one line to an array element. - line = line.strip() - if not line.startswith('/*'): - fatal("unexpected syntax between fields") - line = line[1:] - comment = [] - end = False - while not end: - line = line.strip() - if line.startswith('*/'): - get_line() - break - if not line.startswith('*'): - fatal("unexpected syntax within field") - - line = line[1:] - if line.startswith(' '): - line = line[1:] - if line.startswith(' ') and comment: - continuation = True - line = line.lstrip() - else: - continuation = False - - if line.endswith('*/'): - line = line[:-2].rstrip() - end = True - else: - end = False - - if continuation: - comment[-1] += " " + line - else: - comment += [line] - get_line() - - # Drop blank lines at each end of comment. - while comment and not comment[0]: - comment = comment[1:] - while comment and not comment[-1]: - comment = comment[:-1] - - # Parse the MFF_ constant(s). - mffs = [] - while True: - m = re.match('\s+(MFF_[A-Z0-9_]+),?\s?$', line) - if not m: - break - mffs += [m.group(1)] - get_line() - if not mffs: - fatal("unexpected syntax looking for MFF_ constants") - - if len(mffs) > 1 or '' in comment[0]: - for mff in mffs: - # Extract trailing integer. - m = re.match('.*[^0-9]([0-9]+)$', mff) - if not m: - fatal("%s lacks numeric suffix in register group" % mff) - n = m.group(1) - - # Search-and-replace within the comment, - # and drop lines that have for x != n. - instance = [] - for x in comment: - y = x.replace('', n) - if re.search('<[0-9]+>', y): - if ('<%s>' % n) not in y: - continue - y = re.sub('<[0-9]+>', '', y) - instance += [y.strip()] - fields += [parse_field(mff, instance)] - else: - fields += [parse_field(mffs[0], comment)] - continue - - input_file.close() - - if n_errors: - sys.exit(1) - - return fields - ## ------------------------ ## ## Documentation Generation ## ## ------------------------ ## + def field_to_xml(field_node, f, body, summary): f["used"] = True # Summary. - if field_node.hasAttribute('internal'): + if field_node.hasAttribute("internal"): return min_of_version = None min_ovs_version = None - for header, name, of_version_nr, ovs_version_s in f['OXM']: - if (is_standard_oxm(name) - and (min_ovs_version is None or of_version_nr < min_of_version)): + for header, name, of_version_nr, ovs_version_s in f["OXM"]: + if is_standard_oxm(name) and ( + min_ovs_version is None or of_version_nr < min_of_version + ): min_of_version = of_version_nr - ovs_version = [int(x) for x in ovs_version_s.split('.')] + ovs_version = [int(x) for x in ovs_version_s.split(".")] if min_ovs_version is None or ovs_version < min_ovs_version: min_ovs_version = ovs_version summary += ["\\fB%s\\fR" % f["name"]] @@ -565,124 +202,152 @@ def field_to_xml(field_node, f, body, summary): if min_of_version is not None: support += ["OF %s+" % VERSION_REVERSE[min_of_version]] if min_ovs_version is not None: - support += ["OVS %s+" % '.'.join([str(x) for x in min_ovs_version])] - summary += ' and '.join(support) + support += ["OVS %s+" % ".".join([str(x) for x in min_ovs_version])] + summary += " and ".join(support) summary += ["\n"] # Full description. - if field_node.hasAttribute('hidden'): + if field_node.hasAttribute("hidden"): return - title = field_node.attributes['title'].nodeValue + title = field_node.attributes["title"].nodeValue - body += [""".PP + body += [ + """.PP \\fB%s Field\\fR .TS tab(;); l lx. -""" % title] +""" + % title + ] body += ["Name:;\\fB%s\\fR" % f["name"]] if f["extra_name"]: body += [" (aka \\fB%s\\fR)" % f["extra_name"]] - body += ['\n'] + body += ["\n"] body += ["Width:;"] if f["n_bits"] != 8 * f["n_bytes"]: - body += ["%d bits (only the least-significant %d bits " - "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"])] + body += [ + "%d bits (only the least-significant %d bits " + "may be nonzero)" % (f["n_bytes"] * 8, f["n_bits"]) + ] elif f["n_bits"] <= 128: body += ["%d bits" % f["n_bits"]] else: body += ["%d bits (%d bytes)" % (f["n_bits"], f["n_bits"] / 8)] - body += ['\n'] + body += ["\n"] body += ["Format:;%s\n" % f["formatting"]] - masks = {"MFM_NONE": "not maskable", - "MFM_FULLY": "arbitrary bitwise masks"} + masks = { + "MFM_NONE": "not maskable", + "MFM_FULLY": "arbitrary bitwise masks", + } body += ["Masking:;%s\n" % masks[f["mask"]]] body += ["Prerequisites:;%s\n" % f["prereqs"]] - access = {True: "read/write", - False: "read-only"}[f["writable"]] + access = {True: "read/write", False: "read-only"}[f["writable"]] body += ["Access:;%s\n" % access] - of10 = {None: "not supported", - "exact match": "yes (exact match only)", - "CIDR mask": "yes (CIDR match only)"} + of10 = { + None: "not supported", + "exact match": "yes (exact match only)", + "CIDR mask": "yes (CIDR match only)", + } body += ["OpenFlow 1.0:;%s\n" % of10[f["OF1.0"]]] - of11 = {None: "not supported", - "exact match": "yes (exact match only)", - "bitwise mask": "yes"} + of11 = { + None: "not supported", + "exact match": "yes (exact match only)", + "bitwise mask": "yes", + } body += ["OpenFlow 1.1:;%s\n" % of11[f["OF1.1"]]] oxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if is_standard_oxm(x[1])]: + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if is_standard_oxm(x[1]) + ]: of_version = VERSION_REVERSE[of_version_nr] - oxms += [r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" % (name, header[2], of_version, ovs_version)] + oxms += [ + r"\fB%s\fR (%d) since OpenFlow %s and Open vSwitch %s" + % (name, header[2], of_version, ovs_version) + ] if not oxms: - oxms = ['none'] - body += ['OXM:;T{\n%s\nT}\n' % r'\[char59] '.join(oxms)] + oxms = ["none"] + body += ["OXM:;T{\n%s\nT}\n" % r"\[char59] ".join(oxms)] nxms = [] - for header, name, of_version_nr, ovs_version in [x for x in sorted(f['OXM'], key=lambda x: x[2]) if not is_standard_oxm(x[1])]: - nxms += [r"\fB%s\fR (%d) since Open vSwitch %s" % (name, header[2], ovs_version)] + for header, name, of_version_nr, ovs_version in [ + x + for x in sorted(f["OXM"], key=lambda x: x[2]) + if not is_standard_oxm(x[1]) + ]: + nxms += [ + r"\fB%s\fR (%d) since Open vSwitch %s" + % (name, header[2], ovs_version) + ] if not nxms: - nxms = ['none'] - body += ['NXM:;T{\n%s\nT}\n' % r'\[char59] '.join(nxms)] + nxms = ["none"] + body += ["NXM:;T{\n%s\nT}\n" % r"\[char59] ".join(nxms)] body += [".TE\n"] - body += ['.PP\n'] + body += [".PP\n"] body += [build.nroff.block_xml_to_nroff(field_node.childNodes)] + def group_xml_to_nroff(group_node, fields): - title = group_node.attributes['title'].nodeValue + title = group_node.attributes["title"].nodeValue summary = [] body = [] for node in group_node.childNodes: - if node.nodeType == node.ELEMENT_NODE and node.tagName == 'field': - id_ = node.attributes['id'].nodeValue + if node.nodeType == node.ELEMENT_NODE and node.tagName == "field": + id_ = node.attributes["id"].nodeValue field_to_xml(node, fields[id_], body, summary) else: body += [build.nroff.block_xml_to_nroff([node])] content = [ - '.bp\n', - '.SH \"%s\"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), + ".bp\n", + '.SH "%s"\n' % build.nroff.text_to_nroff(title.upper() + " FIELDS"), '.SS "Summary:"\n', - '.TS\n', - 'tab(;);\n', - 'l l l l l l l.\n', - 'Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n', - '\_;\_;\_;\_;\_;\_\n'] + ".TS\n", + "tab(;);\n", + "l l l l l l l.\n", + "Name;Bytes;Mask;RW?;Prereqs;NXM/OXM Support\n", + "\_;\_;\_;\_;\_;\_\n", + ] content += summary - content += ['.TE\n'] + content += [".TE\n"] content += body - return ''.join(content) + return "".join(content) + def make_oxm_classes_xml(document): - s = '''tab(;); + s = """tab(;); l l l. Prefix;Vendor;Class \_;\_;\_ -''' +""" for key in sorted(OXM_CLASSES, key=OXM_CLASSES.get): vendor, class_, class_type = OXM_CLASSES.get(key) - s += r"\fB%s\fR;" % key.rstrip('_') + s += r"\fB%s\fR;" % key.rstrip("_") if vendor: s += r"\fL0x%08x\fR;" % vendor else: s += "(none);" s += r"\fL0x%04x\fR;" % class_ s += "\n" - e = document.createElement('tbl') + e = document.createElement("tbl") e.appendChild(document.createTextNode(s)) return e + def recursively_replace(node, name, replacement): for child in node.childNodes: if child.nodeType == node.ELEMENT_NODE: @@ -691,11 +356,12 @@ def recursively_replace(node, name, replacement): else: recursively_replace(child, name, replacement) + def make_ovs_fields(meta_flow_h, meta_flow_xml): fields = extract_ofp_fields(meta_flow_h) fields_map = {} for f in fields: - fields_map[f['mff']] = f + fields_map[f["mff"]] = f document = xml.dom.minidom.parse(meta_flow_xml) doc = document.documentElement @@ -704,7 +370,8 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): if version == None: version = "UNKNOWN" - print('''\ + print( + """\ '\\" tp .\\" -*- mode: troff; coding: utf-8 -*- .TH "ovs\-fields" 7 "%s" "Open vSwitch" "Open vSwitch Manual" @@ -740,11 +407,13 @@ def make_ovs_fields(meta_flow_h, meta_flow_xml): ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch . .PP -''' % version) +""" + % version + ) - recursively_replace(doc, 'oxm_classes', make_oxm_classes_xml(document)) + recursively_replace(doc, "oxm_classes", make_oxm_classes_xml(document)) - s = '' + s = "" for node in doc.childNodes: if node.nodeType == node.ELEMENT_NODE and node.tagName == "group": s += group_xml_to_nroff(node, fields_map) @@ -757,9 +426,10 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch for f in fields: if "used" not in f: - fatal("%s: field not documented " - "(please add documentation in lib/meta-flow.xml)" - % f["mff"]) + fatal( + "%s: field not documented " + "(please add documentation in lib/meta-flow.xml)" % f["mff"] + ) if n_errors: sys.exit(1) @@ -769,26 +439,27 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch # Life is easier with nroff if we don't try to feed it Unicode. # Fortunately, we only use a few characters outside the ASCII range. - oline = oline.replace(u'\u2208', r'\[mo]') - oline = oline.replace(u'\u2260', r'\[!=]') - oline = oline.replace(u'\u2264', r'\[<=]') - oline = oline.replace(u'\u2265', r'\[>=]') - oline = oline.replace(u'\u00d7', r'\[mu]') + oline = oline.replace(u"\u2208", r"\[mo]") + oline = oline.replace(u"\u2260", r"\[!=]") + oline = oline.replace(u"\u2264", r"\[<=]") + oline = oline.replace(u"\u2265", r"\[>=]") + oline = oline.replace(u"\u00d7", r"\[mu]") if len(oline): output += [oline] # nroff tends to ignore .bp requests if they come after .PP requests, # so remove .PPs that precede .bp. for i in range(len(output)): - if output[i] == '.bp': + if output[i] == ".bp": j = i - 1 - while j >= 0 and output[j] == '.PP': + while j >= 0 and output[j] == ".PP": output[j] = None j -= 1 for i in range(len(output)): if output[i] is not None: print(output[i]) - + + ## ------------ ## ## Main Program ## ## ------------ ## @@ -796,8 +467,9 @@ ovs\-fields \- protocol header fields in OpenFlow and Open vSwitch if __name__ == "__main__": argv0 = sys.argv[0] try: - options, args = getopt.gnu_getopt(sys.argv[1:], 'h', - ['help', 'ovs-version=']) + options, args = getopt.gnu_getopt( + sys.argv[1:], "h", ["help", "ovs-version="] + ) except getopt.GetoptError as geo: sys.stderr.write("%s: %s\n" % (argv0, geo.msg)) sys.exit(1) @@ -805,32 +477,38 @@ if __name__ == "__main__": global version version = None for key, value in options: - if key in ['-h', '--help']: + if key in ["-h", "--help"]: usage() - elif key == '--ovs-version': + elif key == "--ovs-version": version = value else: sys.exit(0) if not args: - sys.stderr.write("%s: missing command argument " - "(use --help for help)\n" % argv0) + sys.stderr.write( + "%s: missing command argument " "(use --help for help)\n" % argv0 + ) sys.exit(1) - commands = {"meta-flow": (make_meta_flow, 1), - "nx-match": (make_nx_match, 1), - "ovs-fields": (make_ovs_fields, 2)} + commands = { + "meta-flow": (make_meta_flow, 1), + "nx-match": (make_nx_match, 1), + "ovs-fields": (make_ovs_fields, 2), + } if not args[0] in commands: - sys.stderr.write("%s: unknown command \"%s\" " - "(use --help for help)\n" % (argv0, args[0])) + sys.stderr.write( + '%s: unknown command "%s" ' + "(use --help for help)\n" % (argv0, args[0]) + ) sys.exit(1) func, n_args = commands[args[0]] if len(args) - 1 != n_args: - sys.stderr.write("%s: \"%s\" requires %d arguments but %d " - "provided\n" - % (argv0, args[0], n_args, len(args) - 1)) + sys.stderr.write( + '%s: "%s" requires %d arguments but %d ' + "provided\n" % (argv0, args[0], n_args, len(args) - 1) + ) sys.exit(1) func(*args[1:]) diff --git a/python/automake.mk b/python/automake.mk index 73438d615..d7d33928a 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -51,6 +51,7 @@ ovs_pyfiles = \ # so they are not installed. EXTRA_DIST += \ python/build/__init__.py \ + python/build/extract_ofp_fields.py \ python/build/nroff.py \ python/build/soutil.py @@ -69,10 +70,12 @@ PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) FLAKE8_PYFILES += \ $(filter-out python/ovs/compat/% python/ovs/dirs.py,$(PYFILES)) \ - python/setup.py \ python/build/__init__.py \ + python/build/extract_ofp_fields.py \ python/build/nroff.py \ - python/ovs/dirs.py.template + python/build/soutil.py \ + python/ovs/dirs.py.template \ + python/setup.py nobase_pkgdata_DATA = $(ovs_pyfiles) $(ovstest_pyfiles) ovs-install-data-local: diff --git a/python/build/extract_ofp_fields.py b/python/build/extract_ofp_fields.py new file mode 100644 index 000000000..3fe62634a --- /dev/null +++ b/python/build/extract_ofp_fields.py @@ -0,0 +1,421 @@ +import sys +import re + +line = "" + +# Maps from user-friendly version number to its protocol encoding. +VERSION = { + "1.0": 0x01, + "1.1": 0x02, + "1.2": 0x03, + "1.3": 0x04, + "1.4": 0x05, + "1.5": 0x06, +} +VERSION_REVERSE = dict((v, k) for k, v in VERSION.items()) + +TYPES = { + "u8": (1, False), + "be16": (2, False), + "be32": (4, False), + "MAC": (6, False), + "be64": (8, False), + "be128": (16, False), + "tunnelMD": (124, True), +} + +FORMATTING = { + "decimal": ("MFS_DECIMAL", 1, 8), + "hexadecimal": ("MFS_HEXADECIMAL", 1, 127), + "ct state": ("MFS_CT_STATE", 4, 4), + "Ethernet": ("MFS_ETHERNET", 6, 6), + "IPv4": ("MFS_IPV4", 4, 4), + "IPv6": ("MFS_IPV6", 16, 16), + "OpenFlow 1.0 port": ("MFS_OFP_PORT", 2, 2), + "OpenFlow 1.1+ port": ("MFS_OFP_PORT_OXM", 4, 4), + "frag": ("MFS_FRAG", 1, 1), + "tunnel flags": ("MFS_TNL_FLAGS", 2, 2), + "TCP flags": ("MFS_TCP_FLAGS", 2, 2), + "packet type": ("MFS_PACKET_TYPE", 4, 4), +} + +PREREQS = { + "none": "MFP_NONE", + "Ethernet": "MFP_ETHERNET", + "ARP": "MFP_ARP", + "VLAN VID": "MFP_VLAN_VID", + "IPv4": "MFP_IPV4", + "IPv6": "MFP_IPV6", + "IPv4/IPv6": "MFP_IP_ANY", + "NSH": "MFP_NSH", + "CT": "MFP_CT_VALID", + "MPLS": "MFP_MPLS", + "TCP": "MFP_TCP", + "UDP": "MFP_UDP", + "SCTP": "MFP_SCTP", + "ICMPv4": "MFP_ICMPV4", + "ICMPv6": "MFP_ICMPV6", + "ND": "MFP_ND", + "ND solicit": "MFP_ND_SOLICIT", + "ND advert": "MFP_ND_ADVERT", +} + +# Maps a name prefix into an (experimenter ID, class) pair, so: +# +# - Standard OXM classes are written as (0, ) +# +# - Experimenter OXM classes are written as (, 0xffff) +# +# If a name matches more than one prefix, the longest one is used. +OXM_CLASSES = { + "NXM_OF_": (0, 0x0000, "extension"), + "NXM_NX_": (0, 0x0001, "extension"), + "NXOXM_NSH_": (0x005AD650, 0xFFFF, "extension"), + "OXM_OF_": (0, 0x8000, "standard"), + "OXM_OF_PKT_REG": (0, 0x8001, "standard"), + "ONFOXM_ET_": (0x4F4E4600, 0xFFFF, "standard"), + "ERICOXM_OF_": (0, 0x1000, "extension"), + # This is the experimenter OXM class for Nicira, which is the + # one that OVS would be using instead of NXM_OF_ and NXM_NX_ + # if OVS didn't have those grandfathered in. It is currently + # used only to test support for experimenter OXM, since there + # are barely any real uses of experimenter OXM in the wild. + "NXOXM_ET_": (0x00002320, 0xFFFF, "extension"), +} + + +def oxm_name_to_class(name): + prefix = "" + class_ = None + for p, c in OXM_CLASSES.items(): + if name.startswith(p) and len(p) > len(prefix): + prefix = p + class_ = c + return class_ + + +def is_standard_oxm(name): + oxm_vendor, oxm_class, oxm_class_type = oxm_name_to_class(name) + return oxm_class_type == "standard" + + +def get_line(): + global line + global line_number + line = input_file.readline() + line_number += 1 + if line == "": + fatal("unexpected end of input") + + +n_errors = 0 + + +def error(msg): + global n_errors + sys.stderr.write("%s:%d: %s\n" % (file_name, line_number, msg)) + n_errors += 1 + + +def fatal(msg): + error(msg) + sys.exit(1) + + +def parse_oxms(s, prefix, n_bytes): + if s == "none": + return () + + return tuple(parse_oxm(s2.strip(), prefix, n_bytes) for s2 in s.split(",")) + + +match_types = dict() + + +def parse_oxm(s, prefix, n_bytes): + global match_types + + m = re.match( + r"([A-Z0-9_]+)\(([0-9]+)\) since(?: OF(1\.[0-9]+) and)? v([12]\.[0-9]+)$", # noqa: E501 + s, + ) + if not m: + fatal("%s: syntax error parsing %s" % (s, prefix)) + + name, oxm_type, of_version, ovs_version = m.groups() + + class_ = oxm_name_to_class(name) + if class_ is None: + fatal("unknown OXM class for %s" % name) + oxm_vendor, oxm_class, oxm_class_type = class_ + + if class_ in match_types: + if oxm_type in match_types[class_]: + fatal( + "duplicate match type for %s (conflicts with %s)" + % (name, match_types[class_][oxm_type]) + ) + else: + match_types[class_] = dict() + match_types[class_][oxm_type] = name + + # Normally the oxm_length is the size of the field, but for experimenter + # OXMs oxm_length also includes the 4-byte experimenter ID. + oxm_length = n_bytes + if oxm_class == 0xFFFF: + oxm_length += 4 + + header = (oxm_vendor, oxm_class, int(oxm_type), oxm_length) + + if of_version: + if oxm_class_type == "extension": + fatal("%s: OXM extension can't have OpenFlow version" % name) + if of_version not in VERSION: + fatal("%s: unknown OpenFlow version %s" % (name, of_version)) + of_version_nr = VERSION[of_version] + if of_version_nr < VERSION["1.2"]: + fatal("%s: claimed version %s predates OXM" % (name, of_version)) + else: + if oxm_class_type == "standard": + fatal("%s: missing OpenFlow version number" % name) + of_version_nr = 0 + + return (header, name, of_version_nr, ovs_version) + + +def parse_field(mff, comment): + f = {"mff": mff} + + # First line of comment is the field name. + m = re.match( + r'"([^"]+)"(?:\s+\(aka "([^"]+)"\))?(?:\s+\(.*\))?\.', comment[0] + ) + if not m: + fatal("%s lacks field name" % mff) + f["name"], f["extra_name"] = m.groups() + + # Find the last blank line the comment. The field definitions + # start after that. + blank = None + for i in range(len(comment)): + if not comment[i]: + blank = i + if not blank: + fatal("%s: missing blank line in comment" % mff) + + d = {} + for key in ( + "Type", + "Maskable", + "Formatting", + "Prerequisites", + "Access", + "Prefix lookup member", + "OXM", + "NXM", + "OF1.0", + "OF1.1", + ): + d[key] = None + for fline in comment[blank + 1 :]: + m = re.match(r"([^:]+):\s+(.*)\.$", fline) + if not m: + fatal( + "%s: syntax error parsing key-value pair as part of %s" + % (fline, mff) + ) + key, value = m.groups() + if key not in d: + fatal("%s: unknown key" % key) + elif key == "Code point": + d[key] += [value] + elif d[key] is not None: + fatal("%s: duplicate key" % key) + d[key] = value + for key, value in d.items(): + if not value and key not in ( + "OF1.0", + "OF1.1", + "Prefix lookup member", + "Notes", + ): + fatal("%s: missing %s" % (mff, key)) + + m = re.match(r"([a-zA-Z0-9]+)(?: \(low ([0-9]+) bits\))?$", d["Type"]) + if not m: + fatal("%s: syntax error in type" % mff) + type_ = m.group(1) + if type_ not in TYPES: + fatal("%s: unknown type %s" % (mff, d["Type"])) + + f["n_bytes"] = TYPES[type_][0] + if m.group(2): + f["n_bits"] = int(m.group(2)) + if f["n_bits"] > f["n_bytes"] * 8: + fatal( + "%s: more bits (%d) than field size (%d)" + % (mff, f["n_bits"], 8 * f["n_bytes"]) + ) + else: + f["n_bits"] = 8 * f["n_bytes"] + f["variable"] = TYPES[type_][1] + + if d["Maskable"] == "no": + f["mask"] = "MFM_NONE" + elif d["Maskable"] == "bitwise": + f["mask"] = "MFM_FULLY" + else: + fatal("%s: unknown maskable %s" % (mff, d["Maskable"])) + + fmt = FORMATTING.get(d["Formatting"]) + if not fmt: + fatal("%s: unknown format %s" % (mff, d["Formatting"])) + f["formatting"] = d["Formatting"] + if f["n_bytes"] < fmt[1] or f["n_bytes"] > fmt[2]: + fatal( + "%s: %d-byte field can't be formatted as %s" + % (mff, f["n_bytes"], d["Formatting"]) + ) + f["string"] = fmt[0] + + f["prereqs"] = d["Prerequisites"] + if f["prereqs"] not in PREREQS: + fatal("%s: unknown prerequisites %s" % (mff, d["Prerequisites"])) + + if d["Access"] == "read-only": + f["writable"] = False + elif d["Access"] == "read/write": + f["writable"] = True + else: + fatal("%s: unknown access %s" % (mff, d["Access"])) + + f["OF1.0"] = d["OF1.0"] + if not d["OF1.0"] in (None, "exact match", "CIDR mask"): + fatal("%s: unknown OF1.0 match type %s" % (mff, d["OF1.0"])) + + f["OF1.1"] = d["OF1.1"] + if not d["OF1.1"] in (None, "exact match", "bitwise mask"): + fatal("%s: unknown OF1.1 match type %s" % (mff, d["OF1.1"])) + + f["OXM"] = parse_oxms(d["OXM"], "OXM", f["n_bytes"]) + parse_oxms( + d["NXM"], "NXM", f["n_bytes"] + ) + + f["prefix"] = d["Prefix lookup member"] + + return f + + +def extract_ofp_fields(fn): + global file_name + global input_file + global line_number + global line + + file_name = fn + input_file = open(file_name) + line_number = 0 + + fields = [] + + while True: + get_line() + if re.match("enum.*mf_field_id", line): + break + + while True: + get_line() + if ( + line.startswith("/*") + or line.startswith(" *") + or line.startswith("#") + or not line + or line.isspace() + ): + continue + elif re.match(r"}", line) or re.match(r"\s+MFF_N_IDS", line): + break + + # Parse the comment preceding an MFF_ constant into 'comment', + # one line to an array element. + line = line.strip() + if not line.startswith("/*"): + fatal("unexpected syntax between fields") + line = line[1:] + comment = [] + end = False + while not end: + line = line.strip() + if line.startswith("*/"): + get_line() + break + if not line.startswith("*"): + fatal("unexpected syntax within field") + + line = line[1:] + if line.startswith(" "): + line = line[1:] + if line.startswith(" ") and comment: + continuation = True + line = line.lstrip() + else: + continuation = False + + if line.endswith("*/"): + line = line[:-2].rstrip() + end = True + else: + end = False + + if continuation: + comment[-1] += " " + line + else: + comment += [line] + get_line() + + # Drop blank lines at each end of comment. + while comment and not comment[0]: + comment = comment[1:] + while comment and not comment[-1]: + comment = comment[:-1] + + # Parse the MFF_ constant(s). + mffs = [] + while True: + m = re.match(r"\s+(MFF_[A-Z0-9_]+),?\s?$", line) + if not m: + break + mffs += [m.group(1)] + get_line() + if not mffs: + fatal("unexpected syntax looking for MFF_ constants") + + if len(mffs) > 1 or "" in comment[0]: + for mff in mffs: + # Extract trailing integer. + m = re.match(".*[^0-9]([0-9]+)$", mff) + if not m: + fatal("%s lacks numeric suffix in register group" % mff) + n = m.group(1) + + # Search-and-replace within the comment, + # and drop lines that have for x != n. + instance = [] + for x in comment: + y = x.replace("", n) + if re.search("<[0-9]+>", y): + if ("<%s>" % n) not in y: + continue + y = re.sub("<[0-9]+>", "", y) + instance += [y.strip()] + fields += [parse_field(mff, instance)] + else: + fields += [parse_field(mffs[0], comment)] + continue + + input_file.close() + + if n_errors: + sys.exit(1) + + return fields From patchwork Fri Mar 11 15:21:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604410 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HdrTCJxh; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV6F0VJPz9sG3 for ; Sat, 12 Mar 2022 02:22:13 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7531761334; Fri, 11 Mar 2022 15:22:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CgzCaegcsPTk; Fri, 11 Mar 2022 15:22:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 18C4061304; Fri, 11 Mar 2022 15:22:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8AE40C0089; Fri, 11 Mar 2022 15:22:00 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 04762C0012 for ; Fri, 11 Mar 2022 15:21:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id DEB35612E2 for ; Fri, 11 Mar 2022 15:21:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ntg9JtkRxIex for ; Fri, 11 Mar 2022 15:21:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id F053A611EB for ; Fri, 11 Mar 2022 15:21:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FAJFp7ZFOiGOUvEOf1NVo5D5qZ7tUM+kfPyhFOKX6E0=; b=HdrTCJxhg+fpvgud3NpN8GMPLUKw0Nk2FMnFtflPSrneYkS4ErC/mev2xkDQB+odxy7ukx 3RYzDlNbe09+Pyzxhekuw6tB6kDJN1mVNH61E1n/u5krEi64eN6ejTUjhnAfKuXQRAvJRY LUo51tkx9CcQhjrVcPvY5WMwRVz8xXc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-85-qP6VZDzhM0mSOoA0Eqo8zg-1; Fri, 11 Mar 2022 10:21:55 -0500 X-MC-Unique: qP6VZDzhM0mSOoA0Eqo8zg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BE2E85200 for ; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8018D7F0D7; Fri, 11 Mar 2022 15:21:53 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:15 +0100 Message-Id: <20220311152128.3988946-6-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 05/18] build-aux: generate ofp field decoders X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on meta-field information extracted by extract_ofp_fields, autogenerate the right decoder to be used. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- build-aux/automake.mk | 6 ++- build-aux/gen_ofp_field_decoders | 69 ++++++++++++++++++++++++++++++++ python/.gitignore | 1 + python/automake.mk | 7 ++++ 4 files changed, 81 insertions(+), 2 deletions(-) create mode 100755 build-aux/gen_ofp_field_decoders diff --git a/build-aux/automake.mk b/build-aux/automake.mk index 6267ccd7c..b9a77a51c 100644 --- a/build-aux/automake.mk +++ b/build-aux/automake.mk @@ -5,6 +5,7 @@ EXTRA_DIST += \ build-aux/dist-docs \ build-aux/dpdkstrip.py \ build-aux/generate-dhparams-c \ + build-aux/gen_ofp_field_decoders \ build-aux/initial-tab-allowed-files \ build-aux/sodepends.py \ build-aux/soexpand.py \ @@ -12,7 +13,8 @@ EXTRA_DIST += \ build-aux/xml2nroff FLAKE8_PYFILES += \ - $(srcdir)/build-aux/xml2nroff \ build-aux/dpdkstrip.py \ + build-aux/gen_ofp_field_decoders \ build-aux/sodepends.py \ - build-aux/soexpand.py + build-aux/soexpand.py \ + build-aux/xml2nroff diff --git a/build-aux/gen_ofp_field_decoders b/build-aux/gen_ofp_field_decoders new file mode 100755 index 000000000..3cc480042 --- /dev/null +++ b/build-aux/gen_ofp_field_decoders @@ -0,0 +1,69 @@ +#!/bin/env python + +import argparse + +import build.extract_ofp_fields as extract_fields + + +def main(): + parser = argparse.ArgumentParser( + description="Tool to generate python ofproto field decoders from" + "meta-flow information" + ) + parser.add_argument( + "metaflow", + metavar="FILE", + type=str, + help="Read meta-flow info from file", + ) + + args = parser.parse_args() + + fields = extract_fields.extract_ofp_fields(args.metaflow) + + field_decoders = {} + for field in fields: + decoder = get_decoder(field) + field_decoders[field.get("name")] = decoder + if field.get("extra_name"): + field_decoders[field.get("extra_name")] = decoder + + code = """ +# This file is auto-generated. Do not edit! + +import functools +from ovs.flows import decoders + +field_decoders = {{ +{decoders} +}} +""".format( + decoders="\n".join( + [ + " '{name}': {decoder},".format(name=name, decoder=decoder) + for name, decoder in field_decoders.items() + ] + ) + ) + print(code) + + +def get_decoder(field): + formatting = field.get("formatting") + if formatting in ["decimal", "hexadecimal"]: + if field.get("mask") == "MFM_NONE": + return "decoders.decode_int" + else: + if field.get("n_bits") in [8, 16, 32, 64, 128, 992]: + return "decoders.Mask{}".format(field.get("n_bits")) + return "decoders.decode_mask({})".format(field.get("n_bits")) + elif formatting in ["IPv4", "IPv6"]: + return "decoders.IPMask" + elif formatting == "Ethernet": + return "decoders.EthMask" + else: + return "decoders.decode_default" + + +if __name__ == "__main__": + main() diff --git a/python/.gitignore b/python/.gitignore index 60ace6f05..c8ffd4574 100644 --- a/python/.gitignore +++ b/python/.gitignore @@ -1,2 +1,3 @@ dist/ *.egg-info +ovs/flows/ofp_fields.py diff --git a/python/automake.mk b/python/automake.mk index d7d33928a..16d8db98f 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -125,3 +125,10 @@ $(srcdir)/python/ovs/dirs.py: python/ovs/dirs.py.template mv $@.tmp $@ EXTRA_DIST += python/ovs/dirs.py.template CLEANFILES += python/ovs/dirs.py + +$(srcdir)/python/ovs/flows/ofp_fields.py: $(srcdir)/build-aux/gen_ofp_field_decoders include/openvswitch/meta-flow.h + $(AM_V_GEN)$(run_python) $< $(srcdir)/include/openvswitch/meta-flow.h > $@.tmp + $(AM_V_at)mv $@.tmp $@ +EXTRA_DIST += python/ovs/flows/ofp_fields.py +CLEANFILES += python/ovs/flows/ofp_fields.py + From patchwork Fri Mar 11 15:21:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604412 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OZaKXK3Z; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV6j661wz9sG3 for ; Sat, 12 Mar 2022 02:22:37 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 224A584856; Fri, 11 Mar 2022 15:22:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kQSQkploVpXD; Fri, 11 Mar 2022 15:22:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 0CFBD84839; Fri, 11 Mar 2022 15:22:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D3D1EC0012; Fri, 11 Mar 2022 15:22:33 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 31DEFC0033 for ; Fri, 11 Mar 2022 15:22:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7FB1B40B5E for ; Fri, 11 Mar 2022 15:22:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gafJ3ea9DfzE for ; Fri, 11 Mar 2022 15:22:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id A902240C22 for ; Fri, 11 Mar 2022 15:22:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012129; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PHoJE24xBKPdFQb4aAXeLK59PuYAV26dGohOjz0q+HA=; b=OZaKXK3Z89gbQ5UvLFvNyU9Zw6AJqbIy4AQs52GCvXfeEa//S0e/26GFg0748m6Tr5gA1v jqQq9wTR3GLFQHndHZnjSBZGRO74inYfPih0yAkiTB7K2mFV8FhHxUQ2bAsaVljnfIrhTP RoY5krPK+o4eYCZf9FlUF/CY76PeCrM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-224-1NiEckGxPneKx0vVZnaERQ-1; Fri, 11 Mar 2022 10:22:05 -0500 X-MC-Unique: 1NiEckGxPneKx0vVZnaERQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5ED5E1091DA0 for ; Fri, 11 Mar 2022 15:22:04 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F8747F0D7; Fri, 11 Mar 2022 15:21:54 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:16 +0100 Message-Id: <20220311152128.3988946-7-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 06/18] python: add flow base class X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It simplifies the implementation of different types of flows by creating the concept of Section (e.g: match, action) and automatic accessors for all the provided Sections Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flows/flow.py | 125 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 python/ovs/flows/flow.py diff --git a/python/automake.mk b/python/automake.mk index 16d8db98f..ac7dcdd44 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ python/ovs/json.py \ diff --git a/python/ovs/flows/flow.py b/python/ovs/flows/flow.py new file mode 100644 index 000000000..2f053e77d --- /dev/null +++ b/python/ovs/flows/flow.py @@ -0,0 +1,125 @@ +""" Defines the Flow class. +""" + + +class Section(object): + """A flow can be seen as composed of different sections, e.g: + + [info] [match] actions=[actions] + + This class represents each of those sections. + + A section is basically a set of Key-Value pairs. Typically, they can be + expressed as a dictionary, for instance the "match" part of a flow can be + expressed as: + { + "nw_src": "192.168.1.1", + "nw_dst": "192.168.1.2", + } + However, some of them must be expressed as a list which allows for + duplicated keys. For instance, the "actions" section could be: + [ + { + "output": 32 + }, + { + "output": 33 + } + ] + + The is_list flag is used to discriminate this. + + Attributes: + name (str): Name of the section. + pos (int): Position within the overall flow string. + string (str): Section string. + data (list[KeyValue]): Parsed data of the section. + is_list (bool): Whether the key-values shall be expressed as a list + (i.e: it allows repeated keys). + """ + + def __init__(self, name, pos, string, data, is_list=False): + self.name = name + self.pos = pos + self.string = string + self.data = data + self.is_list = is_list + + def __str__(self): + return "{} (at {}): {}".format(self.name, self.pos, self.string) + + def __repr__(self): + return "%s('%s')" % (self.__class__.__name__, self) + + def dict(self): + return {self.name: self.format_data()} + + def format_data(self): + """Returns the section's key-values formatted in a dictionary or list + depending on the value of is_list flag. + """ + if self.is_list: + return [{item.key: item.value} for item in self.data] + else: + return {item.key: item.value for item in self.data} + + +class Flow(object): + """The Flow class is a base class for other types of concrete flows + (such as OFproto Flows or DPIF Flows). + + A flow is basically comprised of a number of sections. + For each section named {section_name}, the flow object will have the + following attributes: + - {section_name} will return the sections data in a formatted way. + - {section_name}_kv will return the sections data as a list of KeyValues. + + Args: + sections (list[Section]): List of sections that comprise the flow + orig (str): Original flow string. + id (Any): Optional; identifier that clients can use to uniquely + identify this flow. + """ + + def __init__(self, sections, orig="", id=None): + self._sections = sections + self._orig = orig + self._id = id + for section in sections: + setattr( + self, section.name, self.section(section.name).format_data() + ) + setattr( + self, + "{}_kv".format(section.name), + self.section(section.name).data, + ) + + def section(self, name): + """Return the section by name.""" + return next( + (sect for sect in self._sections if sect.name == name), None + ) + + @property + def id(self): + """Return the Flow ID.""" + return self._id + + @property + def sections(self): + """Return the all the sections in a list.""" + return self._sections + + @property + def orig(self): + """Return the original flow string.""" + return self._orig + + def dict(self): + """Returns the Flow information in a dictionary.""" + flow_dict = {"orig": self.orig} + for section in self.sections: + flow_dict.update(section.dict()) + + return flow_dict From patchwork Fri Mar 11 15:21:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604414 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=E/z7puZ1; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV733N7dz9sG3 for ; Sat, 12 Mar 2022 02:22:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A230A41A3B; Fri, 11 Mar 2022 15:22:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HzfGTczKnkgN; Fri, 11 Mar 2022 15:22:51 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 89E2841A51; Fri, 11 Mar 2022 15:22:50 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A0283C0083; Fri, 11 Mar 2022 15:22:48 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 43F59C0073 for ; Fri, 11 Mar 2022 15:22:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 452F740BDA for ; Fri, 11 Mar 2022 15:22:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Fck56XLiTylu for ; Fri, 11 Mar 2022 15:22:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8535B40C2B for ; Fri, 11 Mar 2022 15:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEKd4Cgr3JNOJYriglLiH5hM9sd2xLmTvptOi/bMveM=; b=E/z7puZ1dnZm0/bYgOEEn+KeA+1FsMNT309nnb8LsNaczegu+GWdqJAvczAhuJr2oS7bxh 5LmtQQEQGdff/9oJs5mTMKOUBAavhVefxi/0ixBKKdP/FThBVpdnZV01lt2WGydmvxH3mE WH6zUrSTfGn/s9Lf1zBAwOaZ6SwSOHg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-424-qRWa7J4VNCaKreAlgTGMVw-1; Fri, 11 Mar 2022 10:22:07 -0500 X-MC-Unique: qRWa7J4VNCaKreAlgTGMVw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 36309824FAA for ; Fri, 11 Mar 2022 15:22:06 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id D4E8A7F0DC; Fri, 11 Mar 2022 15:22:04 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:17 +0100 Message-Id: <20220311152128.3988946-8-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 07/18] python: introduce OpenFlow Flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Introduce OFPFlow class and all its decoders. Most of the decoders are generic (from decoders.py). Some have special syntax and need a specific implementation. Decoders for nat are moved to the common decoders.py because it's syntax is shared with other types of flows (e.g: dpif flows). Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 2 + python/ovs/flows/decoders.py | 108 +++++++++ python/ovs/flows/ofp.py | 428 +++++++++++++++++++++++++++++++++++ python/ovs/flows/ofp_act.py | 306 +++++++++++++++++++++++++ 4 files changed, 844 insertions(+) create mode 100644 python/ovs/flows/ofp.py create mode 100644 python/ovs/flows/ofp_act.py diff --git a/python/automake.mk b/python/automake.mk index ac7dcdd44..5b0c0d63f 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,8 @@ ovs_pyfiles = \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ + python/ovs/flows/ofp.py \ + python/ovs/flows/ofp_act.py \ python/ovs/json.py \ python/ovs/jsonrpc.py \ python/ovs/ovsuuid.py \ diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 883e61acf..73d28e057 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -6,6 +6,7 @@ object. """ import netaddr +import re class Decoder(object): @@ -414,3 +415,110 @@ class IPMask(Decoder): def to_json(self): return str(self) + + +def decode_free_output(value): + """The value of the output action can be found free, i.e: without the + 'output' keyword. This decoder decodes its value when found this way.""" + try: + return "output", {"port": int(value)} + except ValueError: + return "output", {"port": value.strip('"')} + + +ipv4 = r"(?:\d{1,3}.?){3}\d{1,3}" +ipv4_capture = r"({ipv4})".format(ipv4=ipv4) +ipv6 = r"[\w:\.]+" +ipv6_capture = r"(?:\[*)?({ipv6})(?:\]*)?".format(ipv6=ipv6) +port_range = r":(\d+)(?:-(\d+))?" +ip_range_regexp = r"{ip_cap}(?:-{ip_cap})?(?:{port_range})?" +ipv4_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv4_capture, port_range=port_range) +) +ipv6_port_regex = re.compile( + ip_range_regexp.format(ip_cap=ipv6_capture, port_range=port_range) +) + + +def decode_ip_port_range(value): + """ + Decodes an IP and port range: + {ip_start}-{ip-end}:{port_start}-{port_end} + + IPv6 addresses are surrounded by "[" and "]" if port ranges are also + present + + Returns the following dictionary: + { + "addrs": { + "start": {ip_start} + "end": {ip_end} + } + "ports": { + "start": {port_start}, + "end": {port_end} + } + (the "ports" key might be omitted) + """ + if value.count(":") > 1: + match = ipv6_port_regex.match(value) + else: + match = ipv4_port_regex.match(value) + + ip_start = match.group(1) + ip_end = match.group(2) + port_start = match.group(3) + port_end = match.group(4) + + result = { + "addrs": { + "start": netaddr.IPAddress(ip_start), + "end": netaddr.IPAddress(ip_end or ip_start), + } + } + if port_start: + result["ports"] = { + "start": int(port_start), + "end": int(port_end or port_start), + } + + return result + + +def decode_nat(value): + """Decodes the 'nat' keyword of the ct action. + + The format is: + nat + Flag format. + nat(type=addrs[:ports][,flag]...) + Full format where the address-port range has the same format as + the one described in decode_ip_port_range. + + Examples: + nat(src=0.0.0.0) + nat(src=0.0.0.0,persistent) + nat(dst=192.168.1.0-192.168.1.253:4000-5000) + nat(dst=192.168.1.0-192.168.1.253,hash) + nat(dst=[fe80::f150]-[fe80::f15f]:255-300) + """ + if not value: + return True # If flag format, the value is True. + + result = dict() + type_parts = value.split("=") + result["type"] = type_parts[0] + + if len(type_parts) > 1: + value_parts = type_parts[1].split(",") + if len(type_parts) != 2: + raise ValueError("Malformed nat action: %s" % value) + + ip_port_range = decode_ip_port_range(value_parts[0]) + + result = {"type": type_parts[0], **ip_port_range} + + for flag in value_parts[1:]: + result[flag] = True + + return result diff --git a/python/ovs/flows/ofp.py b/python/ovs/flows/ofp.py new file mode 100644 index 000000000..98902c1e7 --- /dev/null +++ b/python/ovs/flows/ofp.py @@ -0,0 +1,428 @@ +"""Defines the parsers needed to parse ofproto flows. +""" + +import functools + +from ovs.flows.kv import KVParser, KVDecoders, nested_kv_decoder +from ovs.flows.ofp_fields import field_decoders +from ovs.flows.flow import Flow, Section +from ovs.flows.list import ListDecoders, nested_list_decoder +from ovs.flows.decoders import ( + decode_default, + decode_flag, + decode_int, + decode_time, + decode_mask, + IPMask, + EthMask, + decode_free_output, + decode_nat, +) +from ovs.flows.ofp_act import ( + decode_output, + decode_field, + decode_controller, + decode_bundle, + decode_bundle_load, + decode_encap, + decode_load_field, + decode_set_field, + decode_move_field, + decode_dec_ttl, + decode_chk_pkt_larger, + decode_zone, + decode_exec, + decode_learn, +) + + +class OFPFlow(Flow): + """OFPFLow represents an OpenFlow Flow. + + Attributes: + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not OFPFlow._info_decoders: + OFPFlow._info_decoders = OFPFlow._gen_info_decoders() + return OFPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not OFPFlow._match_decoders: + OFPFlow._match_decoders = OFPFlow._gen_match_decoders() + return OFPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not OFPFlow._action_decoders: + OFPFlow._action_decoders = OFPFlow._gen_action_decoders() + return OFPFlow._action_decoders + + def __init__(self, ofp_string, id=None): + """Create a OFPFlow from a flow string. + + The string is expected to have the followoing format: + + [flow data] [match] actions=[actions] + + Args: + ofp_string(str): An OpenFlow flow string. + id(Any): Optional; any object used to uniquely identify this flow + from the rest. + + Returns + An OFPFlow with the content of the flow string or None if there is + no flow information but the string is expected to be found in a + flow dump. + + Raises + ValueError if the string is malformed. + ParseError if an error in parsing occurs. + """ + if " reply " in ofp_string: + return None + + sections = list() + parts = ofp_string.split("actions=") + if len(parts) != 2: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + actions = parts[1] + + field_parts = parts[0].rstrip(" ").rpartition(" ") + if len(field_parts) != 3: + raise ValueError("malformed ofproto flow: %s" % ofp_string) + + info = field_parts[0] + match = field_parts[2] + + iparser = KVParser(info, OFPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=ofp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, OFPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=ofp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, OFPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=ofp_string.find(actions), + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(OFPFlow, self).__init__(sections, ofp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + args = { + "table": decode_int, + "duration": decode_time, + "n_packet": decode_int, + "n_bytes": decode_int, + "cookie": decode_int, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "hard_age": decode_time, + } + return KVDecoders(args) + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + args = { + **OFPFlow._field_decoder_args(), + **OFPFlow._extra_match_decoder_args(), + } + + return KVDecoders(args) + + @staticmethod + def _extra_match_decoder_args(): + """Returns the extra KVDecoder arguments needed to decode the match + part of a flow (apart from the fields).""" + return { + "priority": decode_int, + } + + @staticmethod + def _field_decoder_args(): + """Returns the KVDecoder arguments needed to decode match fields.""" + shorthands = [ + "eth", + "ip", + "ipv6", + "icmp", + "icmp6", + "tcp", + "tcp6", + "udp", + "udp6", + "sctp", + "arp", + "rarp", + "mpls", + "mplsm", + ] + + fields = {**field_decoders, **{key: decode_flag for key in shorthands}} + + # vlan_vid field is special. Although it is technically 12 bit wide, + # bit 12 is allowed to be set to 1 to indicate that the vlan header is + # present (see section VLAN FIELDS in + # http://www.openvswitch.org/support/dist-docs/ovs-fields.7.txt) + # Therefore, override the generated vlan_vid field size. + fields["vlan_vid"] = decode_mask(13) + return fields + + @staticmethod + def _gen_action_decoders(): + """Generate the actions decoders.""" + + actions = { + **OFPFlow._output_actions_decoders_args(), + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + **OFPFlow._fw_action_decoders_args(), + **OFPFlow._control_action_decoders_args(), + **OFPFlow._other_action_decoders_args(), + } + clone_actions = OFPFlow._clone_actions_decoders_args(actions) + actions.update(clone_actions) + return KVDecoders(actions, default_free=decode_free_output) + + @staticmethod + def _output_actions_decoders_args(): + """Returns the decoder arguments for the output actions.""" + return { + "output": decode_output, + "drop": decode_flag, + "controller": decode_controller, + "enqueue": nested_list_decoder( + ListDecoders([("port", decode_default), ("queue", int)]), + delims=[",", ":"], + ), + "bundle": decode_bundle, + "bundle_load": decode_bundle_load, + "group": decode_default, + } + + @staticmethod + def _encap_actions_decoders_args(): + """Returns the decoders arguments for the encap actions.""" + + return { + "pop_vlan": decode_flag, + "strip_vlan": decode_flag, + "push_vlan": decode_default, + "decap": decode_flag, + "encap": decode_encap, + } + + @staticmethod + def _field_action_decoders_args(): + """Returns the decoders arguments for field-modification actions.""" + # Field modification actions + field_default_decoders = [ + "set_mpls_label", + "set_mpls_tc", + "set_mpls_ttl", + "mod_nw_tos", + "mod_nw_ecn", + "mod_tcp_src", + "mod_tcp_dst", + ] + return { + "load": decode_load_field, + "set_field": functools.partial( + decode_set_field, KVDecoders(OFPFlow._field_decoder_args()) + ), + "move": decode_move_field, + "mod_dl_dst": EthMask, + "mod_dl_src": EthMask, + "mod_nw_dst": IPMask, + "mod_nw_src": IPMask, + "dec_ttl": decode_dec_ttl, + "dec_mpls_ttl": decode_flag, + "dec_nsh_ttl": decode_flag, + "check_pkt_larger": decode_chk_pkt_larger, + **{field: decode_default for field in field_default_decoders}, + } + + @staticmethod + def _meta_action_decoders_args(): + """Returns the decoders arguments for the metadata actions.""" + meta_default_decoders = ["set_tunnel", "set_tunnel64", "set_queue"] + return { + "pop_queue": decode_flag, + **{field: decode_default for field in meta_default_decoders}, + } + + @staticmethod + def _fw_action_decoders_args(): + """Returns the decoders arguments for the firewalling actions.""" + return { + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "zone": decode_zone, + "table": decode_int, + "nat": decode_nat, + "force": decode_flag, + "exec": functools.partial( + decode_exec, + KVDecoders( + { + **OFPFlow._encap_actions_decoders_args(), + **OFPFlow._field_action_decoders_args(), + **OFPFlow._meta_action_decoders_args(), + } + ), + ), + "alg": decode_default, + } + ) + ), + "ct_clear": decode_flag, + } + + @staticmethod + def _control_action_decoders_args(): + return { + "resubmit": nested_list_decoder( + ListDecoders( + [ + ("port", decode_default), + ("table", decode_int), + ("ct", decode_flag), + ] + ) + ), + "push": decode_field, + "pop": decode_field, + "exit": decode_flag, + "multipath": nested_list_decoder( + ListDecoders( + [ + ("fields", decode_default), + ("basis", decode_int), + ("algorithm", decode_default), + ("n_links", decode_int), + ("arg", decode_int), + ("dst", decode_field), + ] + ) + ), + } + + @staticmethod + def _clone_actions_decoders_args(action_decoders): + """Generate the decoder arguments for the clone actions. + + Args: + action_decoders (dict): The decoders of the supported nested + actions. + """ + return { + "learn": decode_learn( + { + **action_decoders, + "fin_timeout": nested_kv_decoder( + KVDecoders( + { + "idle_timeout": decode_time, + "hard_timeout": decode_time, + } + ) + ), + } + ), + "clone": functools.partial( + decode_exec, KVDecoders(action_decoders) + ), + } + + @staticmethod + def _other_action_decoders_args(): + """Generate the decoder arguments for other actions + (see man(7) ovs-actions).""" + return { + "conjunction": nested_list_decoder( + ListDecoders( + [("id", decode_int), ("k", decode_int), ("n", decode_int)] + ), + delims=[",", "/"], + ), + "note": decode_default, + "sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_set_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "sampling_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + } diff --git a/python/ovs/flows/ofp_act.py b/python/ovs/flows/ofp_act.py new file mode 100644 index 000000000..f09139d79 --- /dev/null +++ b/python/ovs/flows/ofp_act.py @@ -0,0 +1,306 @@ +"""Defines decoders for OpenFlow actions. +""" + +import functools + +from ovs.flows.decoders import ( + decode_default, + decode_time, + decode_flag, + decode_int, +) +from ovs.flows.kv import nested_kv_decoder, KVDecoders, KeyValue, KVParser +from ovs.flows.list import nested_list_decoder, ListDecoders +from ovs.flows.ofp_fields import field_decoders + + +def decode_output(value): + """Decodes the output value. + + Does not support field specification. + """ + if len(value.split(",")) > 1: + return nested_kv_decoder()(value) + try: + return {"port": int(value)} + except ValueError: + return {"port": value.strip('"')} + + +def decode_controller(value): + """Decodes the controller action.""" + if not value: + return KeyValue("output", "controller") + else: + # Try controller:max_len + try: + max_len = int(value) + return { + "max_len": max_len, + } + except ValueError: + pass + # controller(key[=val], ...) + return nested_kv_decoder()(value) + + +def decode_bundle_load(value): + return decode_bundle(value, True) + + +def decode_bundle(value, load=False): + """Decode bundle action.""" + result = {} + keys = ["fields", "basis", "algorithm", "ofport"] + if load: + keys.append("dst") + + for key in keys: + parts = value.partition(",") + nvalue = parts[0] + value = parts[2] + if key == "ofport": + continue + result[key] = decode_default(nvalue) + + # Handle members: + mvalues = value.split("members:") + result["members"] = [int(port) for port in mvalues[1].split(",")] + return result + + +def decode_encap(value): + """Decodes encap action. Examples: + encap(ethernet) + encap(nsh(md_type=2,tlv(0x1000,10,0x12345678))) + + The generated dict has the following keys: "header", "props", e.g: + { + "header": "ethernet", + } + { + "header": "nsh", + "props": { + "md_type": 2, + "tlv": { + "class": 0x100, + "type": 10, + "value": 0x123456 + } + } + } + """ + + def free_hdr_decoder(free_val): + if free_val not in ["ethernet", "mpls", "mpls_mc", "nsh"]: + raise ValueError( + "Malformed encap action. Unkown header: {}".format(free_val) + ) + return "header", free_val + + parser = KVParser( + value, + KVDecoders( + { + "nsh": nested_kv_decoder( + KVDecoders( + { + "md_type": decode_default, + "tlv": nested_list_decoder( + ListDecoders( + [ + ("class", decode_int), + ("type", decode_int), + ("value", decode_int), + ] + ) + ), + } + ) + ), + }, + default_free=free_hdr_decoder, + ), + ) + parser.parse() + if len(parser.kv()) > 1: + raise ValueError("Malformed encap action: {}".format(value)) + + result = {} + if parser.kv()[0].key == "header": + result["header"] = parser.kv()[0].value + elif parser.kv()[0].key == "nsh": + result["header"] = "nsh" + result["props"] = parser.kv()[0].value + + return result + + +def decode_field(value): + """Decodes a field as defined in the 'Field Specification' of the actions + man page: + http://www.openvswitch.org/support/dist-docs/ovs-actions.7.txt.""" + parts = value.strip("]\n\r").split("[") + result = { + "field": parts[0], + } + + if len(parts) > 1 and parts[1]: + field_range = parts[1].split("..") + start = field_range[0] + end = field_range[1] if len(field_range) > 1 else start + if start: + result["start"] = int(start) + if end: + result["end"] = int(end) + + return result + + +def decode_load_field(value): + """Decodes LOAD actions such as: 'load:value->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed load action : %s" % value) + + # If the load action is performed within a learn() action, + # The value can be specified as another field. + try: + return {"value": int(parts[0], 0), "dst": decode_field(parts[1])} + except ValueError: + return {"src": decode_field(parts[0]), "dst": decode_field(parts[1])} + + +def decode_set_field(field_decoders, value): + """Decodes SET_FIELD actions such as: 'set_field:value/mask->dst'. + + The value is decoded by field_decoders which is a KVDecoders instance. + Args: + field_decoders(KVDecoders): The KVDecoders to be used to decode the + field. + """ + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed set_field action : %s" % value) + + val = parts[0] + dst = parts[1] + + val_result = field_decoders.decode(dst, val) + + return { + "value": {val_result[0]: val_result[1]}, + "dst": decode_field(dst), + } + + +def decode_move_field(value): + """Decodes MOVE actions such as 'move:src->dst'.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed move action : %s" % value) + + return { + "src": decode_field(parts[0]), + "dst": decode_field(parts[1]), + } + + +def decode_dec_ttl(value): + """Decodes dec_ttl and dec_ttl(id, id[2], ...) actions.""" + if not value: + return True + return [int(idx) for idx in value.split(",")] + + +def decode_chk_pkt_larger(value): + """Decodes 'check_pkt_larger(pkt_len)->dst' actions.""" + parts = value.split("->") + if len(parts) != 2: + raise ValueError("Malformed check_pkt_larger action : %s" % value) + + pkt_len = int(parts[0].strip("()")) + dst = decode_field(parts[1]) + return {"pkt_len": pkt_len, "dst": dst} + + +# CT decoders +def decode_zone(value): + """Decodes the value of the 'zone' keyword (part of the ct action).""" + try: + return int(value, 0) + except ValueError: + pass + return decode_field(value) + + +def decode_exec(action_decoders, value): + """Decodes the value of the 'exec' keyword (part of the ct action). + + Args: + decode_actions (KVDecoders): The decoders to be used to decode the + nested exec. + value (string): The string to be decoded. + """ + exec_parser = KVParser(value, action_decoders) + exec_parser.parse() + return [{kv.key: kv.value} for kv in exec_parser.kv()] + + +def decode_learn(action_decoders): + """Create the decoder to be used to decode the 'learn' action. + + The learn action has two added complexities: + 1) It can hold any valid action key-value. Therefore we must take + the precalculated action_decoders and use them. That's why we require + them as argument. + + 2) The way fields can be specified is augmented. Not only we have + 'field=value', but we also have: + - 'field=_src_' (where _src_ is another field name) + - and just 'field' + For this we need to create a wrapper of field_decoders that, for each + "field=X" key-value we check if X is a field_name or if it's actually + a value that we need to send to the appropriate field_decoder to + process. + + Args: + action_decoders (dict): Dictionary of decoders to be used in nested + action decoding. + """ + + def decode_learn_field(decoder, value): + """Generates a decoder to be used for the 'field' argument of the + 'learn' action. + + The field can hold a value that should be decoded, either as a field, + or as a the value (see man(7) ovs-actions). + + Args: + decoder (callable): The decoder. + """ + if value in field_decoders.keys(): + # It's a field + return value + else: + return decoder(value) + + learn_field_decoders = { + field: functools.partial(decode_learn_field, decoder) + for field, decoder in field_decoders.items() + } + learn_decoders = { + **action_decoders, + **learn_field_decoders, + "idle_timeout": decode_time, + "hard_timeout": decode_time, + "priority": decode_int, + "cookie": decode_int, + "send_flow_rem": decode_flag, + "table": decode_int, + "delete_learned": decode_flag, + "limit": decode_int, + "result_dst": decode_field, + } + + return functools.partial(decode_exec, KVDecoders(learn_decoders)) From patchwork Fri Mar 11 15:21:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604420 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WAPKmsJJ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7b5ygTz9sGG for ; Sat, 12 Mar 2022 02:23:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C6CAC613B8; Fri, 11 Mar 2022 15:23:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6Yj6PY1ykg-E; Fri, 11 Mar 2022 15:23:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 91C8861395; Fri, 11 Mar 2022 15:23:15 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4DC62C0012; Fri, 11 Mar 2022 15:23:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1F889C0073 for ; Fri, 11 Mar 2022 15:23:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A3208612ED for ; Fri, 11 Mar 2022 15:22:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u8TbS_XB9LlX for ; Fri, 11 Mar 2022 15:22:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6348561309 for ; Fri, 11 Mar 2022 15:22:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012134; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EbOFyIJLj1tMhKlGJzpez1mMstLRnS9O/2QKBa8vo4c=; b=WAPKmsJJWRTxzJgc03sixe2XAybbd1PhPFvzwoNyMh/4WIUZ0QQcbQVVu0tc4qlwSzeb5T xiMArWzjTKaZ/8gPWJSbEC6+fy8+2crZACP43NAF1DQiuSbtt2r32VFatlaDIZVpSX8RIB /m6cYPHk8btAGJRQKFj6lexQNMejC9U= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-345-cpI7sGW6MHmVNUdsF2jnBA-1; Fri, 11 Mar 2022 10:22:08 -0500 X-MC-Unique: cpI7sGW6MHmVNUdsF2jnBA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CA0B9824FAB for ; Fri, 11 Mar 2022 15:22:07 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8D9217FFF4; Fri, 11 Mar 2022 15:22:06 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:18 +0100 Message-Id: <20220311152128.3988946-9-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 08/18] python: add ovs datapath flow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A ODPFlow is a Flow with the following sections: ufid info (e.g: bytes, packets, dp, etc) match actions Only three datapath actions require special handling: gre: because it has double parenthesis geneve: because it supports many concatenated lists of options nat: we reuse the decoder used for openflow actions Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flows/odp.py | 783 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 784 insertions(+) create mode 100644 python/ovs/flows/odp.py diff --git a/python/automake.mk b/python/automake.mk index 5b0c0d63f..aabbf7db2 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -31,6 +31,7 @@ ovs_pyfiles = \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ + python/ovs/flows/odp.py \ python/ovs/flows/ofp.py \ python/ovs/flows/ofp_act.py \ python/ovs/json.py \ diff --git a/python/ovs/flows/odp.py b/python/ovs/flows/odp.py new file mode 100644 index 000000000..e894f4b96 --- /dev/null +++ b/python/ovs/flows/odp.py @@ -0,0 +1,783 @@ +""" Defines an Open vSwitch Datapath Flow. +""" +import re +from functools import partial + +from ovs.flows.flow import Flow, Section + +from ovs.flows.kv import ( + KVParser, + KVDecoders, + nested_kv_decoder, + decode_nested_kv, +) +from ovs.flows.decoders import ( + decode_default, + decode_time, + decode_int, + decode_mask, + Mask8, + Mask16, + Mask32, + Mask64, + Mask128, + IPMask, + EthMask, + decode_free_output, + decode_flag, + decode_nat, +) + + +class ODPFlow(Flow): + """ODPFLow represents a Open vSwitch Datapath flow. + + Attributes: + ufid: The UFID section with only one key-value, with keyword "ufid". + info: The info section. + match: The match section. + actions: The actions section. + id: The id object given at construction time. + + """ + + """ + These class variables are used to cache the KVDecoders instances. This + will speed up subsequent flow parsings. + """ + _info_decoders = None + _match_decoders = None + _action_decoders = None + + @staticmethod + def info_decoders(): + """Return the KVDecoders instance to parse the info section. + + Uses the cached version if available. + """ + if not ODPFlow._info_decoders: + ODPFlow._info_decoders = ODPFlow._gen_info_decoders() + return ODPFlow._info_decoders + + @staticmethod + def match_decoders(): + """Return the KVDecoders instance to parse the match section. + + Uses the cached version if available. + """ + if not ODPFlow._match_decoders: + ODPFlow._match_decoders = ODPFlow._gen_match_decoders() + return ODPFlow._match_decoders + + @staticmethod + def action_decoders(): + """Return the KVDecoders instance to parse the actions section. + + Uses the cached version if available. + """ + if not ODPFlow._action_decoders: + ODPFlow._action_decoders = ODPFlow._gen_action_decoders() + return ODPFlow._action_decoders + + def __init__(self, odp_string, id=None): + """Parse a odp flow string. + + The string is expected to have the following format: + [ufid], [match] [flow data] actions:[actions] + + Args: + odp_string (str): A datapath flow string. + + Returns: + A ODPFlow instance. + """ + sections = [] + + # If UFID present, parse it and add it to it's own section. + ufid_pos = odp_string.find("ufid:") + if ufid_pos >= 0: + ufid_string = odp_string[ + ufid_pos : (odp_string[ufid_pos:].find(",") + 1) + ] + ufid_parser = KVParser( + ufid_string, KVDecoders({"ufid": decode_default}) + ) + ufid_parser.parse() + if len(ufid_parser.kv()) != 1: + raise ValueError("malformed odp flow: %s" % odp_string) + sections.append( + Section("ufid", ufid_pos, ufid_string, ufid_parser.kv()) + ) + + action_pos = odp_string.find("actions:") + if action_pos < 0: + raise ValueError("malformed odp flow: %s" % odp_string) + + # rest of the string is between ufid and actions + rest = odp_string[ + (ufid_pos + len(ufid_string) if ufid_pos >= 0 else 0) : action_pos + ] + + action_pos += 8 # len("actions:") + actions = odp_string[action_pos:] + + field_parts = rest.lstrip(" ").partition(" ") + + if len(field_parts) != 3: + raise ValueError("malformed odp flow: %s" % odp_string) + + match = field_parts[0] + info = field_parts[2] + + iparser = KVParser(info, ODPFlow.info_decoders()) + iparser.parse() + isection = Section( + name="info", + pos=odp_string.find(info), + string=info, + data=iparser.kv(), + ) + sections.append(isection) + + mparser = KVParser(match, ODPFlow.match_decoders()) + mparser.parse() + msection = Section( + name="match", + pos=odp_string.find(match), + string=match, + data=mparser.kv(), + ) + sections.append(msection) + + aparser = KVParser(actions, ODPFlow.action_decoders()) + aparser.parse() + asection = Section( + name="actions", + pos=action_pos, + string=actions, + data=aparser.kv(), + is_list=True, + ) + sections.append(asection) + + super(ODPFlow, self).__init__(sections, odp_string, id) + + def __str__(self): + if self._orig: + return self._orig + else: + return self.to_string() + + def to_string(self): + """Return a text representation of the flow.""" + string = "ufid: {}".format(self.ufid) if self.ufid else "" + string += "Info: {} | ".format(self.info) + string += "Match : {} | ".format(self.match) + string += "Actions: {}".format(self.actions) + return string + + @staticmethod + def _gen_info_decoders(): + """Generate the info KVDecoders.""" + return KVDecoders(ODPFlow._info_decoders_args()) + + @staticmethod + def _info_decoders_args(): + """Generate the decoder args for the info KVDecoders.""" + return { + "packets": decode_int, + "bytes": decode_int, + "used": decode_time, + "flags": decode_default, + "dp": decode_default, + } + + @staticmethod + def _gen_action_decoders(): + """Generate the action KVDecoders.""" + return KVDecoders( + ODPFlow._action_decoders_args(), default_free=decode_free_output + ) + + @staticmethod + def _action_decoders_args(): + """Generate the arguments for the action KVDecoders.""" + _decoders = { + "drop": decode_flag, + "lb_output": decode_int, + "trunc": decode_int, + "recirc": decode_int, + "userspace": nested_kv_decoder( + KVDecoders( + { + "pid": decode_int, + "sFlow": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "output": decode_int, + } + ) + ), + "slow_path": decode_default, + "flow_sample": nested_kv_decoder( + KVDecoders( + { + "probability": decode_int, + "collector_sed_id": decode_int, + "obs_domain_id": decode_int, + "obs_point_id": decode_int, + "output_port": decode_default, + "ingress": decode_flag, + "egress": decode_flag, + } + ) + ), + "ipfix": nested_kv_decoder( + KVDecoders( + { + "output_port": decode_default, + } + ) + ), + "controller": nested_kv_decoder( + KVDecoders( + { + "reason": decode_int, + "dont_send": decode_int, + "continuation": decode_int, + "recirc_id": decode_int, + "rule_cookie": decode_int, + "controller_id": decode_int, + "max_len": decode_int, + } + ) + ), + "userdata": decode_default, + "actions": decode_flag, + "tunnel_out_port": decode_default, + "push_eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "type": decode_int, + } + ) + ), + "pop_eth": decode_flag, + } + ) + ), + "set": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + "push_vlan": nested_kv_decoder( + KVDecoders( + { + "vid": decode_int, + "pcp": decode_int, + "cfi": decode_int, + "tpid": decode_int, + } + ) + ), + "pop_vlan": decode_flag, + "push_nsh": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "ttl": decode_int, + "mdtype": decode_int, + "np": decode_int, + "spi": decode_int, + "si": decode_int, + "c1": decode_int, + "c2": decode_int, + "c3": decode_int, + "c4": decode_int, + "md2": decode_int, + } + ) + ), + "pop_nsh": decode_flag, + "tnl_pop": decode_int, + "ct_clear": decode_flag, + "ct": nested_kv_decoder( + KVDecoders( + { + "commit": decode_flag, + "force_commit": decode_flag, + "zone": decode_int, + "mark": Mask32, + "label": Mask128, + "helper": decode_default, + "timeout": decode_default, + "nat": decode_nat, + } + ) + ), + **ODPFlow._tnl_action_decoder_args(), + } + + _decoders["clone"] = nested_kv_decoder( + KVDecoders(decoders=_decoders, default_free=decode_free_output) + ) + + return { + **_decoders, + "sample": nested_kv_decoder( + KVDecoders( + { + "sample": (lambda x: float(x.strip("%"))), + "actions": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + "check_pkt_len": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "gt": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + "le": nested_kv_decoder( + KVDecoders( + decoders=_decoders, + default_free=decode_free_output, + ) + ), + } + ) + ), + } + + @staticmethod + def _tnl_action_decoder_args(): + """Generate the decoder arguments for the tunnel actions.""" + return { + "tnl_push": nested_kv_decoder( + KVDecoders( + { + "tnl_port": decode_default, + "header": nested_kv_decoder( + KVDecoders( + { + "size": decode_int, + "type": decode_int, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + "dl_type": decode_int, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": decode_int, + "tos": decode_int, + "ttl": decode_int, + "frag": decode_int, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_int, + "proto": decode_int, + "tclass": decode_int, + "hlimit": decode_int, + } + ) + ), + "udp": nested_kv_decoder( + KVDecoders( + { + "src": decode_int, + "dst": decode_int, + "dsum": Mask16, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "vni": decode_int, + } + ) + ), + "geneve": nested_kv_decoder( + KVDecoders( + { + "oam": decode_flag, + "crit": decode_flag, + "vni": decode_int, + "options": partial( + decode_geneve, False + ), + } + ) + ), + "gre": decode_tnl_gre, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": decode_int, + "sid": decode_int, + "idx": decode_int, + "dir": decode_int, + "hwid": decode_int, + } + ) + ), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": decode_int, + "msgtype": decode_int, + "teid": decode_int, + } + ) + ), + } + ) + ), + "out_port": decode_default, + } + ) + ) + } + + @staticmethod + def _gen_match_decoders(): + """Generate the match KVDecoders.""" + return KVDecoders(ODPFlow._match_decoders_args()) + + @staticmethod + def _match_decoders_args(): + """Generate the arguments for the match KVDecoders.""" + return { + **ODPFlow._field_decoders_args(), + "encap": nested_kv_decoder( + KVDecoders(ODPFlow._field_decoders_args()) + ), + } + + @staticmethod + def _field_decoders_args(): + """Generate the decoder arguments for the match fields.""" + return { + "skb_priority": Mask32, + "skb_mark": Mask32, + "recirc_id": decode_int, + "dp_hash": Mask32, + "ct_state": decode_default, + "ct_zone": Mask16, + "ct_mark": Mask32, + "ct_label": Mask128, + "ct_tuple4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "ct_tuple6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tcp_src": Mask16, + "tcp_dst": Mask16, + } + ) + ), + "tunnel": nested_kv_decoder( + KVDecoders( + { + "tun_id": Mask64, + "src": IPMask, + "dst": IPMask, + "ipv6_src": IPMask, + "ipv6_dst": IPMask, + "tos": Mask8, + "ttl": Mask8, + "tp_src": Mask16, + "tp_dst": Mask16, + "erspan": nested_kv_decoder( + KVDecoders( + { + "ver": Mask8, + "idx": Mask32, + "sid": decode_int, + "dir": Mask8, + "hwid": Mask8, + } + ) + ), + "vxlan": nested_kv_decoder( + KVDecoders( + { + "gbp": nested_kv_decoder( + KVDecoders( + { + "id": Mask16, + "flags": Mask8, + } + ) + ) + } + ) + ), + "geneve": partial(decode_geneve, True), + "gtpu": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "msgtype": Mask8, + } + ) + ), + "flags": decode_default, + } + ) + ), + "in_port": decode_default, + "eth": nested_kv_decoder( + KVDecoders( + { + "src": EthMask, + "dst": EthMask, + } + ) + ), + "vlan": nested_kv_decoder( + KVDecoders( + { + "vid": Mask16, + "pcp": Mask16, + "cfi": Mask16, + } + ) + ), + "eth_type": Mask16, + "mpls": nested_kv_decoder( + KVDecoders( + { + "label": Mask32, + "tc": Mask32, + "ttl": Mask32, + "bos": Mask32, + } + ) + ), + "ipv4": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "proto": Mask8, + "tos": Mask8, + "ttl": Mask8, + "frag": decode_default, + } + ) + ), + "ipv6": nested_kv_decoder( + KVDecoders( + { + "src": IPMask, + "dst": IPMask, + "label": decode_mask(20), + "proto": Mask8, + "tclass": Mask8, + "hlimit": Mask8, + "frag": decode_default, + } + ) + ), + "tcp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "tcp_flags": decode_default, + "udp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "sctp": nested_kv_decoder( + KVDecoders( + { + "src": Mask16, + "dst": Mask16, + } + ) + ), + "icmp": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "icmpv6": nested_kv_decoder( + KVDecoders( + { + "type": Mask8, + "code": Mask8, + } + ) + ), + "arp": nested_kv_decoder( + KVDecoders( + { + "sip": IPMask, + "tip": IPMask, + "op": Mask16, + "sha": EthMask, + "tha": EthMask, + } + ) + ), + "nd": nested_kv_decoder( + KVDecoders( + { + "target": IPMask, + "sll": EthMask, + "tll": EthMask, + } + ) + ), + "nd_ext": nested_kv_decoder( + KVDecoders( + { + "nd_reserved": Mask32, + "nd_options_type": Mask8, + } + ) + ), + "packet_type": nested_kv_decoder( + KVDecoders( + { + "ns": Mask16, + "id": Mask16, + } + ) + ), + "nsh": nested_kv_decoder( + KVDecoders( + { + "flags": Mask8, + "mdtype": Mask8, + "np": Mask8, + "spi": Mask32, + "si": Mask8, + "c1": Mask32, + "c2": Mask32, + "c3": Mask32, + "c4": Mask32, + } + ) + ), + } + + +def decode_geneve(mask, value): + """Decode geneve options. + Used for both tnl_push(header(geneve(options()))) action and + tunnel(geneve()) match. + + It has the following format: + + {class=0xffff,type=0x80,len=4,0xa} + + Args: + mask (bool): Whether masking is supported. + value (str): The value to decode. + """ + if mask: + decoders = { + "class": Mask16, + "type": Mask8, + "len": Mask8, + } + + def free_decoder(value): + return "data", Mask128(value) + + else: + decoders = { + "class": decode_int, + "type": decode_int, + "len": decode_int, + } + + def free_decoder(value): + return "data", decode_int(value) + + result = [] + for opts in re.findall(r"{.*?}", value): + result.append( + decode_nested_kv( + KVDecoders(decoders=decoders, default_free=free_decoder), + opts.strip("{}"), + ) + ) + return result + + +def decode_tnl_gre(value): + """ + Decode tnl_push(header(gre())) action. + + It has the following format: + + gre((flags=0x2000,proto=0x6558),key=0x1e241)) + + Args: + value (str): The value to decode. + """ + return decode_nested_kv( + KVDecoders( + { + "flags": decode_int, + "proto": decode_int, + "key": decode_int, + "csum": decode_int, + "seq": decode_int, + } + ), + value.replace("(", "").replace(")", ""), + ) From patchwork Fri Mar 11 15:21:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604413 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hvZ/xTKs; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV703dggz9sG3 for ; Sat, 12 Mar 2022 02:22:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 264C961360; Fri, 11 Mar 2022 15:22:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qE03FU25BNtF; Fri, 11 Mar 2022 15:22:48 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id E4D7161358; Fri, 11 Mar 2022 15:22:47 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B7096C0012; Fri, 11 Mar 2022 15:22:47 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2B099C0012 for ; Fri, 11 Mar 2022 15:22:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2DF9461315 for ; Fri, 11 Mar 2022 15:22:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hA6DtSCm7Dy3 for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4D6C6612E3 for ; Fri, 11 Mar 2022 15:22:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=33AEoUNu3pjBkAO2wZjfkeRQAnvMqPjBFN/eUzzQ++s=; b=hvZ/xTKsEA0lyguUyRJ/MQLKPXFUYERZHtZwwLoOQsGz3g04YtDhoSUimoux3ZSrg2P6v5 Fb0xPCCOU5OtEf29i64Zd5xpBAwEQpRw8jymNGK2xL2Ug50po3VxaB0MQg0P/kBNhOJCnK vONdxz1fFydxSUZhbDUKdKUFTErbTM4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-251-uu0QXoGHMReroYfLamIE8w-1; Fri, 11 Mar 2022 10:22:10 -0500 X-MC-Unique: uu0QXoGHMReroYfLamIE8w-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D05841854E26 for ; Fri, 11 Mar 2022 15:22:09 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 54B207F0DC; Fri, 11 Mar 2022 15:22:08 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:19 +0100 Message-Id: <20220311152128.3988946-10-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 09/18] python: add flow filtering syntax X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Based on pyparsing, create a very simple filtering syntax. It supports basic logic statements (and, &, or, ||, not, !), numerical operations (<, >), equality (=, !=), and masking (~=). The latter is only supported in certain fields (IntMask, EthMask, IPMask). Masking operation is semantically equivalent to "includes", therefore: ip_src ~= 192.168.1.1 means that ip_src field is either a host IP address equal to 192.168.1.1 or an IPMask that includes it (e.g: 192.168.1.1/24). Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/flows/filter.py | 261 +++++++++++++++++++++++++++++++++++++ python/setup.py | 2 +- 3 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 python/ovs/flows/filter.py diff --git a/python/automake.mk b/python/automake.mk index aabbf7db2..9bd7a9e2f 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/filter.py \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ python/ovs/flows/list.py \ diff --git a/python/ovs/flows/filter.py b/python/ovs/flows/filter.py new file mode 100644 index 000000000..e520e90dc --- /dev/null +++ b/python/ovs/flows/filter.py @@ -0,0 +1,261 @@ +""" Defines a Flow Filtering syntax. +""" +import pyparsing as pp +import netaddr +from functools import reduce +from operator import and_, or_ + +from ovs.flows.decoders import ( + decode_default, + decode_int, + Decoder, + IPMask, + EthMask, +) + + +class EvaluationResult(object): + """An EvaluationResult is the result of an evaluation. It contains the + boolean result and the list of key-values that were evaluated. + + Note that since boolean operations (and, not, or) are based only on + __bool__ we use bitwise alternatives (&, ||, ~). + """ + + def __init__(self, result, *kv): + self.result = result + self.kv = kv if kv else list() + + def __and__(self, other): + """Logical and operation.""" + return EvaluationResult( + self.result and other.result, *self.kv, *other.kv + ) + + def __or__(self, other): + """Logical or operation.""" + return EvaluationResult( + self.result or other.result, *self.kv, *other.kv + ) + + def __invert__(self): + """Logical not operation.""" + return EvaluationResult(not self.result, *self.kv) + + def __bool__(self): + """Boolean operation.""" + return self.result + + def __repr__(self): + return "{} [{}]".format(self.result, self.kv) + + +class ClauseExpression(object): + """ A clause expression represents a specific expression in the filter. + + A clause has the following form: + [field] [operator] [value] + + Valid operators are: + = (equality) + != (inequality) + < (arithmetic less-than) + > (arithmetic more-than) + ~= (__contains__) + + When evaluated, the clause finds what relevant part of the flow to use for + evaluation, tries to translate the clause value to the relevant type and + performs the clause operation. + + Attributes: + field (str): The flow field used in the clause. + operator (str): The flow operator used in the clause. + value (str): The value to perform the comparison against. + """ + operators = {} + type_decoders = { + int: decode_int, + netaddr.IPAddress: IPMask, + netaddr.EUI: EthMask, + bool: bool, + } + + def __init__(self, tokens): + self.field = tokens[0] + self.value = "" + self.operator = "" + + if len(tokens) > 1: + self.operator = tokens[1] + self.value = tokens[2] + + def __repr__(self): + return "{}(field: {}, operator: {}, value: {})".format( + self.__class__.__name__, self.field, self.operator, self.value + ) + + def _find_data_in_kv(self, kv_list): + """Find a KeyValue for evaluation in a list of KeyValue. + + Args: + kv_list (list[KeyValue]): list of KeyValue to look into. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + """ + key_parts = self.field.split(".") + field = key_parts[0] + kvs = [kv for kv in kv_list if kv.key == field] + if not kvs: + return None + + for kv in kvs: + if kv.key == self.field: + # exact match + return (kv, kv.value) + if len(key_parts) > 1: + data = kv.value + for subkey in key_parts[1:]: + try: + data = data.get(subkey) + except Exception: + data = None + break + if not data: + break + if data: + return (kv, data) + return None + + def _find_keyval_to_evaluate(self, flow): + """Finds the key-value and data to use for evaluation on a flow. + + Args: + flow(Flow): The flow where the lookup is performed. + + Returns: + If found, tuple (kv, data) where kv is the KeyValue that matched + and data is the data to be used for evaluation. None if not found. + + """ + for section in flow.sections: + data = self._find_data_in_kv(section.data) + if data: + return data + return None + + def evaluate(self, flow): + """Returns whether the clause is satisfied by the flow. + + Args: + flow (Flow): the flow to evaluate. + """ + result = self._find_keyval_to_evaluate(flow) + + if not result: + return EvaluationResult(False) + + keyval, data = result + + if not self.value and not self.operator: + # just asserting the existance of the key + return EvaluationResult(True, keyval) + + # Decode the value based on the type of data + if isinstance(data, Decoder): + decoder = data.__class__ + else: + decoder = self.type_decoders.get(data.__class__) or decode_default + + decoded_value = decoder(self.value) + + if self.operator == "=": + return EvaluationResult(decoded_value == data, keyval) + elif self.operator == "<": + return EvaluationResult(data < decoded_value, keyval) + elif self.operator == ">": + return EvaluationResult(data > decoded_value, keyval) + elif self.operator == "~=": + return EvaluationResult(decoded_value in data, keyval) + + +class BoolNot(object): + def __init__(self, t): + self.op, self.args = t[0] + + def __repr__(self): + return "NOT({})".format(self.args) + + def evaluate(self, flow): + return ~self.args.evaluate(flow) + + +class BoolAnd(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def __repr__(self): + return "AND({})".format(self.args) + + def evaluate(self, flow): + return reduce(and_, [arg.evaluate(flow) for arg in self.args]) + + +class BoolOr(object): + def __init__(self, pattern): + self.args = pattern[0][0::2] + + def evaluate(self, flow): + return reduce(or_, [arg.evaluate(flow) for arg in self.args]) + + def __repr__(self): + return "OR({})".format(self.args) + + +class OFFilter(object): + """OFFilter represents an Open vSwitch Flow Filter. + + It is built with a filter expression string composed of logically-separated + clauses (see ClauseExpression for details on the clause syntax). + + Args: + expr(str): String filter expression. + """ + w = pp.Word(pp.alphanums + "." + ":" + "_" + "/" + "-") + operators = ( + pp.Literal("=") + | pp.Literal("~=") + | pp.Literal("<") + | pp.Literal(">") + | pp.Literal("!=") + ) + + clause = (w + operators + w) | w + clause.setParseAction(ClauseExpression) + + statement = pp.infixNotation( + clause, + [ + ("!", 1, pp.opAssoc.RIGHT, BoolNot), + ("not", 1, pp.opAssoc.RIGHT, BoolNot), + ("&&", 2, pp.opAssoc.LEFT, BoolAnd), + ("and", 2, pp.opAssoc.LEFT, BoolAnd), + ("||", 2, pp.opAssoc.LEFT, BoolOr), + ("or", 2, pp.opAssoc.LEFT, BoolOr), + ], + ) + + def __init__(self, expr): + self._filter = self.statement.parseString(expr) + + def evaluate(self, flow): + """Evaluate whether the flow satisfies the filter. + + Args: + flow(Flow): a openflow or datapath flow. + + Returns: + An EvaluationResult with the result of the evaluation. + """ + return self._filter[0].evaluate(flow) diff --git a/python/setup.py b/python/setup.py index b06370bd9..4e8a9761a 100644 --- a/python/setup.py +++ b/python/setup.py @@ -87,7 +87,7 @@ setup_args = dict( ext_modules=[setuptools.Extension("ovs._json", sources=["ovs/_json.c"], libraries=['openvswitch'])], cmdclass={'build_ext': try_build_ext}, - install_requires=['sortedcontainers', 'netaddr'], + install_requires=['sortedcontainers', 'netaddr', 'pyparsing'], extras_require={':sys_platform == "win32"': ['pywin32 >= 1.0']}, ) From patchwork Fri Mar 11 15:21:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604416 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Z8Kq5t8T; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7D3gQ8z9sG3 for ; Sat, 12 Mar 2022 02:23:04 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id AF95940C43; Fri, 11 Mar 2022 15:23:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xHH-D7oTVD9L; Fri, 11 Mar 2022 15:23:01 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id BE17640C81; Fri, 11 Mar 2022 15:22:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9DD37C0012; Fri, 11 Mar 2022 15:22:59 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 02C9BC000B for ; Fri, 11 Mar 2022 15:22:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 0DF4F40C3E for ; Fri, 11 Mar 2022 15:22:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HY8WRW6MzYYx for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6912940C03 for ; Fri, 11 Mar 2022 15:22:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j5pGyIfrfaEo9WgxLdy7hioXQetStmBB8Rs6HoS9I+E=; b=Z8Kq5t8TDT4KknAf2DwVK+z73HOKaVsRi/eljK5RVU7g5wM5WvmvqmERRz0Bz+5XBifLU/ HHLx2Vg3QOx1pCxRu8ifVlhhSgAUtIvRklUK+2H8bFSw7/p1OEEAoS4kmqaBx6qcxAiGYW G9FtrVVWvRUTKKHDd9tdHo7gQccKrUM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-509-Ux31NT0vPKCAOmWy-N7cFw-1; Fri, 11 Mar 2022 10:22:12 -0500 X-MC-Unique: Ux31NT0vPKCAOmWy-N7cFw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50033100C611 for ; Fri, 11 Mar 2022 15:22:11 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E3AF7F0DC; Fri, 11 Mar 2022 15:22:10 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:20 +0100 Message-Id: <20220311152128.3988946-11-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 10/18] python: add a json encoder to flow fields X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The json encoder can be used to convert Flows to json. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/ovs/flows/decoders.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/python/ovs/flows/decoders.py b/python/ovs/flows/decoders.py index 73d28e057..7378d4176 100644 --- a/python/ovs/flows/decoders.py +++ b/python/ovs/flows/decoders.py @@ -5,6 +5,7 @@ A decoder is generally a callable that accepts a string and returns the value object. """ +import json import netaddr import re @@ -522,3 +523,16 @@ def decode_nat(value): result[flag] = True return result + + +class FlowEncoder(json.JSONEncoder): + """FlowEncoder is a json.JSONEncoder instance that can be used to + serialize flow fields.""" + + def default(self, obj): + if isinstance(obj, Decoder): + return obj.to_json() + elif isinstance(obj, netaddr.IPAddress): + return str(obj) + + return json.JSONEncoder.default(self, obj) From patchwork Fri Mar 11 15:21:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604419 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=h4V9Zu97; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7b4hN5z9sG3 for ; Sat, 12 Mar 2022 02:23:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 66C8641A4C; Fri, 11 Mar 2022 15:23:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DH8QsGsEgTax; Fri, 11 Mar 2022 15:23:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0B00E41A43; Fri, 11 Mar 2022 15:23:19 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 55355C008B; Fri, 11 Mar 2022 15:23:18 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2AB98C000B for ; Fri, 11 Mar 2022 15:23:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 15F0461349 for ; Fri, 11 Mar 2022 15:22:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2Tc6jahf_xdl for ; Fri, 11 Mar 2022 15:22:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3BEF46135A for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012139; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cKzbjH4UpbJWWd6aN2FlsAwUq3vkjSVhVTE6tRv45Ao=; b=h4V9Zu97UBJE7IWzShxJnqrwtqh3VIEUHQPw475dOnqMC49kFEyV/x9r4KolJZaeNOUTH2 w/i7Yr6tvWWZMdc6kBqIJuthyBysKn0SSVckTgOhvKoQ/8wbOJEwO88rQ8M33KjNcq9gts 3EG1ilMsO7I0KXoY+3UnFhFkY7HHpoM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-329-x7mAZo5sPHSMjYcF3O9RfA-1; Fri, 11 Mar 2022 10:22:14 -0500 X-MC-Unique: x7mAZo5sPHSMjYcF3O9RfA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BD9961091DA3 for ; Fri, 11 Mar 2022 15:22:12 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id ABE4C7FFEE; Fri, 11 Mar 2022 15:22:11 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:21 +0100 Message-Id: <20220311152128.3988946-12-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 11/18] tests: wrap ovs-ofctl calls to test python parser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Some ovs-ofctl commands are used to parse or dump openflow flows, specially in ofp-actions.at Use a wrapper around ovs-ofctl, called ovs-test-ofparse.py that, apart from calling ovs-ofctl, also parses its output (or input, depending on the command) to make sure the python flow parsing library can also parse the flows. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- tests/automake.mk | 3 ++ tests/ofp-actions.at | 46 ++++++++-------- tests/ovs-test-ofparse.py | 107 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 133 insertions(+), 23 deletions(-) create mode 100755 tests/ovs-test-ofparse.py diff --git a/tests/automake.mk b/tests/automake.mk index 8a9151f81..230085236 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,9 +19,11 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch + COMMON_MACROS_AT = \ tests/ovsdb-macros.at \ tests/ovs-macros.at \ @@ -523,6 +525,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ tests/test-jsonrpc.py \ diff --git a/tests/ofp-actions.at b/tests/ofp-actions.at index 9d820eba6..6ee0d4773 100644 --- a/tests/ofp-actions.at +++ b/tests/ofp-actions.at @@ -327,7 +327,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow10 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow10 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -500,7 +500,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow11 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow11 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -735,7 +735,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow12 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow12 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -796,7 +796,7 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow13 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow13 < input.txt], [0], [expout], [experr]) AT_CLEANUP @@ -825,16 +825,16 @@ AT_CAPTURE_FILE([input.txt]) AT_CAPTURE_FILE([expout]) AT_CAPTURE_FILE([experr]) AT_CHECK( - [ovs-ofctl '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow15 < input.txt], + [ovs-test-ofparse.py '-vPATTERN:console:%c|%p|%m' parse-actions OpenFlow15 < input.txt], [0], [expout], [experr]) AT_CLEANUP AT_SETUP([ofp-actions - inconsistent MPLS actions]) OVS_VSWITCHD_START dnl OK: Use fin_timeout action on TCP flow -AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=fin_timeout(idle_timeout=1)']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 -vwarn add-flow br0 'tcp actions=fin_timeout(idle_timeout=1)']) dnl Bad: Use fin_timeout action on TCP flow that has been converted to MPLS -AT_CHECK([ovs-ofctl -O OpenFlow11 -vwarn add-flow br0 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)'], +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 -vwarn add-flow br0 'tcp actions=push_mpls:0x8847,fin_timeout(idle_timeout=1)'], [1], [], [dnl ovs-ofctl: none of the usable flow formats (OpenFlow10,NXM) is among the allowed flow formats (OpenFlow11) ]) @@ -848,8 +848,8 @@ dnl In OpenFlow 1.3, set_field always sets all the bits in the field, dnl but when we translate to NXAST_LOAD we need to only set the bits that dnl actually exist (e.g. mpls_label only has 20 bits) otherwise OVS rejects dnl the "load" action as invalid. Check that we do this correctly. -AT_CHECK([ovs-ofctl -O OpenFlow13 add-flow br0 mpls,actions=set_field:10-\>mpls_label]) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow13 add-flow br0 mpls,actions=set_field:10-\>mpls_label]) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: mpls actions=load:0xa->OXM_OF_MPLS_LABEL[[]] ]) @@ -861,12 +861,12 @@ AT_KEYWORDS([ofp-actions]) OVS_VSWITCHD_START dnl OpenFlow 1.0 has an "enqueue" action. For OpenFlow 1.1+, we translate dnl it to a series of actions that accomplish the same thing. -AT_CHECK([ovs-ofctl -O OpenFlow10 add-flow br0 'actions=enqueue(123,456)']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 add-flow br0 'actions=enqueue(123,456)']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: actions=enqueue:123:456 ]) -AT_CHECK([ovs-ofctl -O OpenFlow13 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow13 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.3): reset_counts actions=set_queue:456,output:123,pop_queue ]) @@ -878,12 +878,12 @@ AT_KEYWORDS([ofp-actions]) OVS_VSWITCHD_START dnl OpenFlow 1.1+ have a mod_nw_ttl action. For OpenFlow 1.0, we translate dnl it to an Open vSwitch extension. -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ttl:123']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ttl:123']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: ip actions=load:0x7b->NXM_NX_IP_TTL[[]] ]) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ttl:123 ]) @@ -897,16 +897,16 @@ OVS_VSWITCHD_START dnl OpenFlow 1.1, but no other version, has a "mod_nw_ecn" action. dnl Check that we translate it properly for OF1.0 and OF1.2. dnl (OF1.3+ should be the same as OF1.2.) -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ecn:2']) -AT_CHECK([ovs-ofctl -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=mod_nw_ecn:2']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 dump-flows br0 | ofctl_strip], [0], [dnl NXST_FLOW reply: ip actions=load:0x2->NXM_NX_IP_ECN[[]] ]) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ecn:2 ]) -AT_CHECK([ovs-ofctl -O OpenFlow12 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow12 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.2): ip actions=set_field:2->nw_ecn ]) @@ -919,8 +919,8 @@ dnl that anything that comes in as reg_load gets translated back to reg_load dnl on output. Perhaps this is somewhat inconsistent but it's what OVS dnl has done for multiple versions. AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow12 add-flow br0 'ip,actions=set_field:2->ip_ecn']) -AT_CHECK([ovs-ofctl -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl +AT_CHECK([ovs-test-ofparse.py -O OpenFlow12 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 dump-flows br0 | ofctl_strip], [0], [dnl OFPST_FLOW reply (OF1.1): ip actions=mod_nw_ecn:2 ]) @@ -928,9 +928,9 @@ OFPST_FLOW reply (OF1.1): dnl Check that OF1.2+ set_field to set ECN is translated for earlier OF dnl versions. AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow10 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow10 add-flow br0 'ip,actions=set_field:2->ip_ecn']) AT_CHECK([ovs-ofctl del-flows br0]) -AT_CHECK([ovs-ofctl -O OpenFlow11 add-flow br0 'ip,actions=set_field:2->ip_ecn']) +AT_CHECK([ovs-test-ofparse.py -O OpenFlow11 add-flow br0 'ip,actions=set_field:2->ip_ecn']) OVS_VSWITCHD_STOP AT_CLEANUP diff --git a/tests/ovs-test-ofparse.py b/tests/ovs-test-ofparse.py new file mode 100755 index 000000000..3598ac04b --- /dev/null +++ b/tests/ovs-test-ofparse.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ovs-test-ofparse is just a wrapper around ovs-ofctl +that also runs the python flow parsing utility to check that flows are +parseable. +""" + +import subprocess +import sys +import re + +from ovs.flows.ofp import OFPFlow + +diff_regexp = re.compile(r"\d{2}: (\d{2}|\(none\)) -> (\d{2}|\(none\))$") + + +def run_ofctl(with_stdin): + cmd = sys.argv + cmd[0] = "ovs-ofctl" + if with_stdin: + p = subprocess.Popen( + cmd, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + input_data = sys.stdin.read() + out, err = p.communicate(input_data.encode("utf-8")) + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + else: + p = subprocess.Popen( + cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE + ) + out, err = p.communicate() + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + + +def main(): + flows = list() + return_code = 0 + + if "parse-actions" in sys.argv: + return_code, out, err = run_ofctl(True) + + out_lines = out.decode("utf-8").split("\n") + for line in out_lines: + if not ( + "bad" in line # skip "bad action at..." + or line.strip() == "" # skip empty lines + or diff_regexp.match(line) # skip differences + ): + flows.append(line) + + elif "add-flow" in sys.argv: + return_code, out, err = run_ofctl(False) + flows.append(sys.argv[-1]) + + elif "dump-flows" in sys.argv: + return_code, out, err = run_ofctl(False) + out_lines = out.decode("utf-8").split("\n") + + for line in out_lines: + if not ( + "reply" in line # skip NXST_FLOW reply: + or line.strip() == "" # skip empty lines + ): + flows.append(line) + else: + print("Unsupported command: {}".format(sys.argv)) + sys.exit(1) + + if return_code == 0: + for flow in flows: + try: + result_flow = OFPFlow(flow) + if flow != str(result_flow): + print("in: {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed") + except Exception as e: + print(e) + return 1 + + return return_code + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Fri Mar 11 15:21:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604423 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Bkn2q/HW; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV842B4Nz9sG3 for ; Sat, 12 Mar 2022 02:23:48 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C5623613CE; Fri, 11 Mar 2022 15:23:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yAJpzv3FoWFX; Fri, 11 Mar 2022 15:23:41 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 96BA4613E2; Fri, 11 Mar 2022 15:23:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5423AC0033; Fri, 11 Mar 2022 15:23:40 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 17D17C0012 for ; Fri, 11 Mar 2022 15:23:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 32B1A41529 for ; Fri, 11 Mar 2022 15:22:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YZaq6GOek2cT for ; Fri, 11 Mar 2022 15:22:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id AB479419DE for ; Fri, 11 Mar 2022 15:22:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012140; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PiYDGwNidMJKeMO94N1VBGyxb+PFEJUMlzFeBlAHX78=; b=Bkn2q/HWhdXBhMIqKrohnocdRJtPWWE2e3PToFFEJTqQLM4i2mYBhLTGlqq9iPvn/y1/Ke 3RZc/ZrOXWyeL2WKuZgN0bkX/UgR2kNgMYadaUw05wNUHz2B+5+8SydcW5pm5QritlLeg7 FZYKoizCSeoMnlvGHzWfa7ir96ZZIQc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-93-L98NrpdiM5WlB67V0PvarQ-1; Fri, 11 Mar 2022 10:22:15 -0500 X-MC-Unique: L98NrpdiM5WlB67V0PvarQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3CE631091DA0 for ; Fri, 11 Mar 2022 15:22:14 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F7967F0D0; Fri, 11 Mar 2022 15:22:12 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:22 +0100 Message-Id: <20220311152128.3988946-13-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 12/18] tests: Wrap test-odp to also run python parsers X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" test-odp is used to parse datapath flow actions and matches within the odp tests. Wrap calls to this tool in a python script that also parses them using the python flow parsing library. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- tests/automake.mk | 2 + tests/odp.at | 36 ++++++++--------- tests/ovs-test-dpparse.py | 82 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 18 deletions(-) create mode 100755 tests/ovs-test-dpparse.py diff --git a/tests/automake.mk b/tests/automake.mk index 230085236..0279585cd 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -19,6 +19,7 @@ EXTRA_DIST += \ $(OVSDB_CLUSTER_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ + $(srcdir)/tests/ovs-test-dpparse.py \ $(srcdir)/tests/ovs-test-ofparse.py \ $(srcdir)/tests/testsuite \ $(srcdir)/tests/testsuite.patch @@ -525,6 +526,7 @@ CHECK_PYFILES = \ tests/flowgen.py \ tests/mfex_fuzzy.py \ tests/ovsdb-monitor-sort.py \ + tests/ovs-test-dpparse.py \ tests/ovs-test-ofparse.py \ tests/test-daemon.py \ tests/test-json.py \ diff --git a/tests/odp.at b/tests/odp.at index 4d08c59ca..00880565d 100644 --- a/tests/odp.at +++ b/tests/odp.at @@ -105,7 +105,7 @@ sed -i'back' 's/\(skb_mark(0)\),\(ct\)/\1,ct_state(0),ct_zone(0),\2/' odp-out.tx sed -i'back' 's/\(skb_mark([[^)]]*)\),\(recirc\)/\1,ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),\2/' odp-out.txt sed -i'back' 's/\(in_port(1)\),\(eth\)/\1,packet_type(ns=0,id=0),\2/' odp-out.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-keys < odp-in.txt], [0], [`cat odp-out.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-keys < odp-in.txt], [0], [`cat odp-out.txt` ]) AT_CLEANUP @@ -192,7 +192,7 @@ sed -n 's/,frag=no),.*/,frag=later)/p' odp-base.txt sed 's/^/skb_priority(0),tunnel(tun_id=0xfedcba9876543210,src=10.0.0.1,dst=10.0.0.2,ttl=128,erspan(ver=2,dir=1,hwid=0x7/0xf),flags(df|key)),skb_mark(0),recirc_id(0),dp_hash(0),/' odp-base.txt ) > odp.txt AT_CAPTURE_FILE([odp.txt]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-wc-keys < odp.txt], [0], [`cat odp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-wc-keys < odp.txt], [0], [`cat odp.txt` ]) AT_CLEANUP @@ -239,25 +239,25 @@ AT_DATA([odp-tcp6.txt], [dnl in_port(1),eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15),eth_type(0x86dd),ipv6(src=::1/::255,dst=::2/::255,label=0/0xf0,proto=10/0xf0,tclass=0x70/0xf0,hlimit=128/0xf0,frag=no) in_port(1),eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15),eth_type(0x86dd),ipv6(src=::1,dst=::2,label=0,proto=6,tclass=0,hlimit=128,frag=no),tcp(src=80/0xff00,dst=8080/0xff) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x1235' < odp-base.txt], [0], [`cat odp-eth-type.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_type=0x1235' < odp-base.txt], [0], [`cat odp-eth-type.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_vlan=99' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_vlan=99,ip' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_vlan=99,ip' < odp-vlan-base.txt], [0], [`cat odp-vlan.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='ip,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='ip,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='ip,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='dl_type=0x0800,nw_src=35.8.2.199,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='dl_type=0x0800,nw_src=35.8.2.199,nw_dst=172.16.0.199' < odp-base.txt], [0], [`cat odp-ipv4.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='icmp,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-icmp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='icmp,nw_src=35.8.2.199' < odp-base.txt], [0], [`cat odp-icmp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='arp,arp_spa=1.2.3.5' < odp-base.txt], [0], [`cat odp-arp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='arp,arp_spa=1.2.3.5' < odp-base.txt], [0], [`cat odp-arp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='tcp,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp.txt` ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-filter filter='tcp6,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp6.txt` +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-filter filter='tcp6,tp_src=90' < odp-base.txt], [0], [`cat odp-tcp6.txt` ]) AT_CLEANUP @@ -386,14 +386,14 @@ check_pkt_len(size=200,gt(set(eth(src=00:01:02:03:04:05,dst=10:11:12:13:14:15))) lb_output(1) add_mpls(label=200,tc=7,ttl=64,bos=1,eth_type=0x8847) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [`cat actions.txt` ]) AT_CLEANUP AT_SETUP([OVS datapath actions parsing and formatting - invalid forms]) dnl This caused a hang in older versions. -AT_CHECK([echo 'encap_nsh@:{@' | ovstest test-odp parse-actions +AT_CHECK([echo 'encap_nsh@:{@' | ovs-test-dpparse.py ovstest test-odp parse-actions ], [0], [dnl odp_actions_from_string: error ]) @@ -428,7 +428,7 @@ data_invalid=$(printf '%*s' 131018 | tr ' ' "a") echo "userspace(pid=1234567,userdata(${data_valid}),tunnel_out_port=10)" >> actions.txt echo "userspace(pid=1234567,userdata(${data_invalid}),tunnel_out_port=10)" >> actions.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [dnl `cat actions.txt | head -1` odp_actions_from_string: error `cat actions.txt | head -3 | tail -1` @@ -444,7 +444,7 @@ actions=$(printf 'set(encap()),%.0s' $(seq 8190)) echo "${actions}set(encap())" > actions.txt echo "${actions}set(encap()),set(encap())" >> actions.txt -AT_CHECK_UNQUOTED([ovstest test-odp parse-actions < actions.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-actions < actions.txt], [0], [dnl `cat actions.txt | head -1` odp_actions_from_string: error ]) @@ -458,7 +458,7 @@ dnl sequence of keys. 'syntax error' indicates oversized list of keys. keys=$(printf 'encap(),%.0s' $(seq 16382)) echo "${keys}encap()" > keys.txt echo "${keys}encap(),encap()" >> keys.txt -AT_CHECK([ovstest test-odp parse-keys < keys.txt | sed 's/encap(),//g'], [0], [dnl +AT_CHECK([ovs-test-dpparse.py ovstest test-odp parse-keys < keys.txt | sed 's/encap(),//g'], [0], [dnl odp_flow_key_to_flow: error (duplicate encap attribute in flow key; the flow key in error is: encap()) odp_flow_from_string: error (syntax error at encap()) ]) @@ -468,7 +468,7 @@ AT_SETUP([OVS datapath keys parsing and formatting - 33 nested encap ]) AT_DATA([odp-in.txt], [dnl encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap())))))))))))))))))))))))))))))))) ]) -AT_CHECK_UNQUOTED([ovstest test-odp parse-keys < odp-in.txt], [0], [dnl +AT_CHECK_UNQUOTED([ovs-test-dpparse.py ovstest test-odp parse-keys < odp-in.txt], [0], [dnl odp_flow_from_string: error (syntax error at encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap(encap()))))))))))))))))))))))))))))))))) ]) AT_CLEANUP diff --git a/tests/ovs-test-dpparse.py b/tests/ovs-test-dpparse.py new file mode 100755 index 000000000..7f7c59fa3 --- /dev/null +++ b/tests/ovs-test-dpparse.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ovs-test-dpparse is just a wrapper around ovs-dpctl +that also runs the python flow parsing utility to check that flows are +parseable. +""" +import subprocess +import sys +import re + +from ovs.flows.odp import ODPFlow + +diff_regexp = re.compile(r"\d{2}: (\d{2}|\(none\)) -> (\d{2}|\(none\))$") + + +def run(input_data): + p = subprocess.Popen( + sys.argv[1:], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + out, err = p.communicate(input_data.encode("utf-8")) + + print(out.decode("utf-8"), file=sys.stdout, end="") + print(err.decode("utf-8"), file=sys.stderr, end="") + return p.returncode, out, err + + +def main(): + return_code = 0 + input_data = sys.stdin.read() + return_code, out, err = run(input_data) + + if return_code == 0: + flows = list() + for line in input_data.split("\n"): + if not ( + "error" in line # skip errors + or line.strip() == "" # skip empty lines + or line.strip()[0] == "#" # skip comments + ): + flows.append(line) + + for flow in flows: + if any( + c in sys.argv + for c in ["parse-keys", "parse-wc-keys", "parse-filter"] + ): + # Add actions=drop so that the flow is properly formatted + flow += " actions:drop" + elif "parse-actions" in sys.argv: + flow = "actions:" + flow + try: + result_flow = ODPFlow(flow) + if flow != str(result_flow): + print("in : {}".format(flow)) + print("out: {}".format(str(result_flow))) + raise ValueError("Flow conversion back to string failed!") + + except Exception as e: + print(e) + return 1 + + return return_code + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Fri Mar 11 15:21:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604415 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=S9FTTCZ3; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV763Brhz9sG3 for ; Sat, 12 Mar 2022 02:22:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D10C441A6A; Fri, 11 Mar 2022 15:22:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wu15YrmjPICe; Fri, 11 Mar 2022 15:22:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id E594641A3E; Fri, 11 Mar 2022 15:22:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id AD071C0012; Fri, 11 Mar 2022 15:22:54 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3733DC0073 for ; Fri, 11 Mar 2022 15:22:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 958D5841FC for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R290z15vE3uI for ; Fri, 11 Mar 2022 15:22:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 11F198483F for ; Fri, 11 Mar 2022 15:22:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012138; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VLvMApgglkpEmE2vvy282KHRy/yPe6QSMXHyNMgQ5jU=; b=S9FTTCZ3fqhnjeV0x8ctwBSg3w+fC7V4u+Zk0ndjzRzJGlo7RofAFRfMZPPmmn4/iw8xOM bUw2b5sAmc0Shmssr8pr6J0ggxLrQ0k/j+/TaMSIGpvbDh7rcAFMUQQF6Yp6hMtWUutKPI MuKlI2cGHBEiHaxdC2vOrQRhAkvm3OA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-544-gWLzXV3VOSW0-_igCVzauQ-1; Fri, 11 Mar 2022 10:22:16 -0500 X-MC-Unique: gWLzXV3VOSW0-_igCVzauQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D3B9D824FA6 for ; Fri, 11 Mar 2022 15:22:15 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9052C7F0DC; Fri, 11 Mar 2022 15:22:14 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:23 +0100 Message-Id: <20220311152128.3988946-14-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 13/18] python: detect changes in flow formatting code X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to minimize the risk of having the python flow parsing code and the C flow formatting code divert, add a target that checks if the formatting code has been changed since the last revision and warn the developer if it has. The script also makes it easy to update the dependency file so hopefully it will not cause too much trouble for a developer that has modifed the file without changing the flow string format. Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- .gitignore | 1 + python/automake.mk | 9 +++ python/build/flow-parse-deps.py | 106 ++++++++++++++++++++++++++++++++ python/ovs/flows/deps.py | 5 ++ 4 files changed, 121 insertions(+) create mode 100755 python/build/flow-parse-deps.py create mode 100644 python/ovs/flows/deps.py diff --git a/.gitignore b/.gitignore index f1cdcf124..e6bca1cd2 100644 --- a/.gitignore +++ b/.gitignore @@ -79,3 +79,4 @@ testsuite.tmp.orig /Documentation/_build /.venv /cxx-check +/flowparse-deps-check diff --git a/python/automake.mk b/python/automake.mk index 9bd7a9e2f..e5501c58e 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -28,6 +28,7 @@ ovs_pyfiles = \ python/ovs/fcntl_win.py \ python/ovs/flows/__init__.py \ python/ovs/flows/decoders.py \ + python/ovs/flows/deps.py \ python/ovs/flows/filter.py \ python/ovs/flows/flow.py \ python/ovs/flows/kv.py \ @@ -57,6 +58,7 @@ ovs_pyfiles = \ EXTRA_DIST += \ python/build/__init__.py \ python/build/extract_ofp_fields.py \ + python/build/flow-parse-deps.py \ python/build/nroff.py \ python/build/soutil.py @@ -77,6 +79,7 @@ FLAKE8_PYFILES += \ $(filter-out python/ovs/compat/% python/ovs/dirs.py,$(PYFILES)) \ python/build/__init__.py \ python/build/extract_ofp_fields.py \ + python/build/flow-parse-deps.py \ python/build/nroff.py \ python/build/soutil.py \ python/ovs/dirs.py.template \ @@ -137,3 +140,9 @@ $(srcdir)/python/ovs/flows/ofp_fields.py: $(srcdir)/build-aux/gen_ofp_field_deco EXTRA_DIST += python/ovs/flows/ofp_fields.py CLEANFILES += python/ovs/flows/ofp_fields.py +ALL_LOCAL += flowparse-deps-check +DEPS = $(shell $(AM_V_GEN)$(run_python) $(srcdir)/python/build/flow-parse-deps.py list) +flowparse-deps-check: $(srcdir)/python/build/flow-parse-deps.py $(DEPS) + $(AM_V_GEN)$(run_python) $(srcdir)/python/build/flow-parse-deps.py check + touch $@ +CLEANFILES += flowparse-deps-check diff --git a/python/build/flow-parse-deps.py b/python/build/flow-parse-deps.py new file mode 100755 index 000000000..848ef6bac --- /dev/null +++ b/python/build/flow-parse-deps.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +# Copyright (c) 2021 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Breaks lines read from stdin into groups using blank lines as +# group separators, then sorts lines within the groups for +# reproducibility. + + +# ovs-test-ofparse is just a wrapper around ovs-ofctl +# that also runs the python flow parsing utility to check that flows are +# parseable + +import hashlib +import sys +import os + +DEPENDENCIES = ["lib/ofp-actions.c", "lib/odp-util.c"] +DEPENDENCY_FILE = "python/ovs/flows/deps.py" +SRC_DIR = os.path.join(os.path.dirname(__file__), "..", "..") + + +def usage(): + print( + """ +Usage {cmd} [check | update | list] +Tool to verify flow parsing python code is kept in sync with +flow printing C code. + +Commands: + check: check the dependencies are met + update: update the dependencies based on current file content + list: list the dependency files +""".format( + cmd=sys.argv[0] + ) + ) + + +def digest(filename): + with open(os.path.join(SRC_DIR, filename), "rb") as f: + return hashlib.md5(f.read()).hexdigest() + + +def main(): + if len(sys.argv) != 2: + usage() + sys.exit(1) + + if sys.argv[1] == "list": + print(" ".join(DEPENDENCIES)) + elif sys.argv[1] == "update": + dep_str = list() + for dep in DEPENDENCIES: + dep_str.append( + ' "{dep}": "{digest}"'.format(dep=dep, digest=digest(dep)) + ) + + depends = """# File automatically generated. Do not modify manually! +dependencies = {{ +{dependencies_dict} +}}""".format( + dependencies_dict=",\n".join(dep_str) + ) + with open(os.path.join(SRC_DIR, DEPENDENCY_FILE), "w") as f: + print(depends, file=f) + + elif sys.argv[1] == "check": + sys.path.append(os.path.join(SRC_DIR, "python")) + from ovs.flows.deps import dependencies + + for dep in DEPENDENCIES: + expected = dependencies.get(dep) + if not expected or expected != digest(dep): + print( + """ +Dependency file {dep} has changed. +Please verify the flow output format has not changed. +If it has changed, modify the python flow parsing code accordingly. + +Once you're done, update the dependencies by running '{cmd} update'. +After doing so, check-in the new dependency file. +""".format( + dep=dep, + cmd=sys.argv[0], + ) + ) + return 2 + else: + usage() + sys.exit(1) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/python/ovs/flows/deps.py b/python/ovs/flows/deps.py new file mode 100644 index 000000000..9a75475b5 --- /dev/null +++ b/python/ovs/flows/deps.py @@ -0,0 +1,5 @@ +# File automatically generated. Do not modify manually! +dependencies = { + "lib/ofp-actions.c": "c839d7d34a6e9ab1dcb1c0961211d3a6", + "lib/odp-util.c": "8fb7c5fa46ceb7e887c44ed4fcc80ee5" +} From patchwork Fri Mar 11 15:21:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604417 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=fzF0k7rp; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7J47qJz9sG3 for ; Sat, 12 Mar 2022 02:23:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 13B3D6138F; Fri, 11 Mar 2022 15:23:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zOEYibg5I2MU; Fri, 11 Mar 2022 15:23:03 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id B4CA16137A; Fri, 11 Mar 2022 15:23:02 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E283AC0082; Fri, 11 Mar 2022 15:23:01 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1234DC0082 for ; Fri, 11 Mar 2022 15:23:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B21AA841A6 for ; Fri, 11 Mar 2022 15:22:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2W6_HHXvurvG for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 897D584854 for ; Fri, 11 Mar 2022 15:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012139; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JYGUWb9tMgzU3oIHGlOmC9cWV80Dq9rPzQk4v7BSs1A=; b=fzF0k7rpalJ7MN5XxYRdaJKn7a0HWFcyki7cUZosxOnDTORrS8k4d2oJaTannX6bs3sz+V PwjesihGwzxrfRUT327IUNGKEubzgvR4E/du8wMvOsGFHGEeM4L4uFawN+yrOnelthbD/W 1PmnP9N8qdetH22zUwctInwLZuuWEls= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-169-WkfY5JeIOaS4flttNkIlJA-1; Fri, 11 Mar 2022 10:22:18 -0500 X-MC-Unique: WkfY5JeIOaS4flttNkIlJA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5654F100C611 for ; Fri, 11 Mar 2022 15:22:17 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 379B47F0D0; Fri, 11 Mar 2022 15:22:16 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:24 +0100 Message-Id: <20220311152128.3988946-15-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 14/18] python: introduce unit tests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Use pytest to run unit tests as part of the standard testsuite. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- .github/workflows/build-and-test.yml | 3 + Documentation/intro/install/general.rst | 4 ++ python/automake.mk | 9 ++- python/ovs/tests/test_kv.py | 76 +++++++++++++++++++++++++ python/test_requirements.txt | 3 + tests/atlocal.in | 19 +++++++ tests/automake.mk | 1 + tests/pytest.at | 7 +++ tests/testsuite.at | 1 + 9 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 python/ovs/tests/test_kv.py create mode 100644 python/test_requirements.txt create mode 100644 tests/pytest.at diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml index eac3504e4..44df1c2d5 100644 --- a/.github/workflows/build-and-test.yml +++ b/.github/workflows/build-and-test.yml @@ -123,6 +123,9 @@ jobs: with: python-version: '3.9' + - name: install python dependencies + run: pip install -r python/test_requirements.txt + - name: create ci signature file for the dpdk cache key if: matrix.dpdk != '' || matrix.dpdk_shared != '' # This will collect most of DPDK related lines, so hash will be different diff --git a/Documentation/intro/install/general.rst b/Documentation/intro/install/general.rst index c4300cd53..711fb98a4 100644 --- a/Documentation/intro/install/general.rst +++ b/Documentation/intro/install/general.rst @@ -181,6 +181,10 @@ following to obtain better warnings: come from the "hacking" flake8 plugin. If it's not installed, the warnings just won't occur until it's run on a system with "hacking" installed. +- the python packages listed in "python/test_requirements.txt" (compatible + with pip). If they are installed, the pytest-based Python unit tests will + be run. + You may find the ovs-dev script found in ``utilities/ovs-dev.py`` useful. .. _general-install-reqs: diff --git a/python/automake.mk b/python/automake.mk index e5501c58e..43a27d3cb 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -53,6 +53,9 @@ ovs_pyfiles = \ python/ovs/vlog.py \ python/ovs/winutils.py +ovs_pytests = \ + python/ovs/tests/test_kv.py + # These python files are used at build time but not runtime, # so they are not installed. EXTRA_DIST += \ @@ -66,12 +69,14 @@ EXTRA_DIST += \ EXTRA_DIST += \ python/ovs/compat/sortedcontainers/LICENSE \ python/README.rst \ - python/setup.py + python/setup.py \ + python/test_requirements.txt # C extension support. EXTRA_DIST += python/ovs/_json.c -PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) +PYFILES = $(ovs_pyfiles) python/ovs/dirs.py $(ovstest_pyfiles) $(ovs_pytests) + EXTRA_DIST += $(PYFILES) PYCOV_CLEAN_FILES += $(PYFILES:.py=.py,cover) diff --git a/python/ovs/tests/test_kv.py b/python/ovs/tests/test_kv.py new file mode 100644 index 000000000..e81804d49 --- /dev/null +++ b/python/ovs/tests/test_kv.py @@ -0,0 +1,76 @@ +import pytest + +from ovs.flows.kv import KVParser, KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ( + "cookie=0x0, duration=147566.365s, table=0, n_packets=39, n_bytes=2574, idle_age=65534, hard_age=65534", # noqa: E501 + None, + ), + [ + KeyValue("cookie", 0), + KeyValue("duration", "147566.365s"), + KeyValue("table", 0), + KeyValue("n_packets", 39), + KeyValue("n_bytes", 2574), + KeyValue("idle_age", 65534), + KeyValue("hard_age", 65534), + ], + ), + ( + ( + "load:0x4->NXM_NX_REG13[],load:0x9->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:0a:58:a9:fe:00:02,resubmit(,8)", # noqa: E501 + None, + ), + [ + KeyValue("load", "0x4->NXM_NX_REG13[]"), + KeyValue("load", "0x9->NXM_NX_REG11[]"), + KeyValue("load", "0x8->NXM_NX_REG12[]"), + KeyValue("load", "0x1->OXM_OF_METADATA[]"), + KeyValue("load", "0x1->NXM_NX_REG14[]"), + KeyValue("mod_dl_src", "0a:58:a9:fe:00:02"), + KeyValue("resubmit", ",8"), + ], + ), + ( + ("l1(l2(l3(l4())))", None), + [KeyValue("l1", "l2(l3(l4()))")] + ), + ( + ("l1(l2(l3(l4()))),foo:bar", None), + [KeyValue("l1", "l2(l3(l4()))"), KeyValue("foo", "bar")], + ), + ( + ("enqueue:1:2,output=2", None), + [KeyValue("enqueue", "1:2"), KeyValue("output", 2)], + ), + ( + ("value_to_reg(100)->someReg[10],foo:bar", None), + [ + KeyValue("value_to_reg", "(100)->someReg[10]"), + KeyValue("foo", "bar"), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + tparser = KVParser(input_string, decoders) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr diff --git a/python/test_requirements.txt b/python/test_requirements.txt new file mode 100644 index 000000000..6aaee13e3 --- /dev/null +++ b/python/test_requirements.txt @@ -0,0 +1,3 @@ +pytest +netaddr +pyparsing diff --git a/tests/atlocal.in b/tests/atlocal.in index a0ad239ec..2426416a0 100644 --- a/tests/atlocal.in +++ b/tests/atlocal.in @@ -222,3 +222,22 @@ export OVS_CTL_TIMEOUT # matter break everything. ASAN_OPTIONS=detect_leaks=0:abort_on_error=true:log_path=asan:$ASAN_OPTIONS export ASAN_OPTIONS + +# Check whether Python test requirements are available. +REQUIREMENT_PATH=$abs_top_srcdir/python/test_requirements.txt $PYTHON3 -c ' +import os +import pathlib +import pkg_resources + +with pathlib.Path(os.path.join(os.getenv("REQUIREMENT_PATH"))).open() as reqs: + for req in pkg_resources.parse_requirements(reqs): + try: + pkg_resources.require(str(req)) + except pkg_resources.DistributionNotFound: + sys.exit(2) +' +case $? in + 0) HAVE_PYTEST=yes ;; + 2) HAVE_PYTEST=no ;; + *) echo "$0: unexpected error probing Python unit test requirements" >&2 ;; +esac diff --git a/tests/automake.mk b/tests/automake.mk index 0279585cd..e7ffc2691 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -103,6 +103,7 @@ TESTSUITE_AT = \ tests/ovsdb-rbac.at \ tests/ovs-vsctl.at \ tests/ovs-xapi-sync.at \ + tests/pytest.at \ tests/stp.at \ tests/rstp.at \ tests/interface-reconfigure.at \ diff --git a/tests/pytest.at b/tests/pytest.at new file mode 100644 index 000000000..44a88ed98 --- /dev/null +++ b/tests/pytest.at @@ -0,0 +1,7 @@ +AT_BANNER([Python unit tests]) + +# Run pytest unit tests. +AT_SETUP([Pytest unit tests - Python3]) +AT_SKIP_IF([test "$HAVE_PYTEST" = "no"]) +AT_CHECK([python -m pytest $top_srcdir/python/ovs],[0], [ignore], [ignore]) +AT_CLEANUP() diff --git a/tests/testsuite.at b/tests/testsuite.at index 58adfa09c..14a28b517 100644 --- a/tests/testsuite.at +++ b/tests/testsuite.at @@ -78,3 +78,4 @@ m4_include([tests/mcast-snooping.at]) m4_include([tests/packet-type-aware.at]) m4_include([tests/nsh.at]) m4_include([tests/drop-stats.at]) +m4_include([tests/pytest.at]) From patchwork Fri Mar 11 15:21:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604418 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PlSEYC5J; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7K3Xmkz9sGG for ; Sat, 12 Mar 2022 02:23:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 21E6040BE4; Fri, 11 Mar 2022 15:23:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rij14V_TqP-e; Fri, 11 Mar 2022 15:23:05 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 164C540BE6; Fri, 11 Mar 2022 15:23:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C8AD6C000B; Fri, 11 Mar 2022 15:23:03 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2031DC008B for ; Fri, 11 Mar 2022 15:23:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id D4B1C8485A for ; Fri, 11 Mar 2022 15:22:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pi9zsLQ27pNh for ; Fri, 11 Mar 2022 15:22:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 32FF084854 for ; Fri, 11 Mar 2022 15:22:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vsif0i1o0xGFq/zA6BKGiss1K2vdL3oHHdWHpqLyMOo=; b=PlSEYC5Jfl24/9OfQWCPTUmIV0qnqJF8N3wMhTlCKHlTZ2MxN6eL4SQqucPbB03d4iBb7r ZlBA5zCg4VCPUCdIC+8+yFQcnc2O35Qnte6Ho+edOW0oAZylcAtyYLDkTVqvtRFBevFUnc 3i9SvZMG3qU64afB8aHqicB9EIex8/0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-452-R7bs4Fy3PCSw2r-YYvcJmA-1; Fri, 11 Mar 2022 10:22:19 -0500 X-MC-Unique: R7bs4Fy3PCSw2r-YYvcJmA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D3C27824FAC for ; Fri, 11 Mar 2022 15:22:18 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id BAA047F0DE; Fri, 11 Mar 2022 15:22:17 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:25 +0100 Message-Id: <20220311152128.3988946-16-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 15/18] python: add unit tests for ListParser X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for ListParser class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 3 +- python/ovs/tests/test_list.py | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 python/ovs/tests/test_list.py diff --git a/python/automake.mk b/python/automake.mk index 43a27d3cb..559ec3019 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -54,7 +54,8 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ - python/ovs/tests/test_kv.py + python/ovs/tests/test_kv.py \ + python/ovs/tests/test_list.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_list.py b/python/ovs/tests/test_list.py new file mode 100644 index 000000000..e5139869f --- /dev/null +++ b/python/ovs/tests/test_list.py @@ -0,0 +1,66 @@ +import pytest + +from ovs.flows.list import ListParser, ListDecoders +from ovs.flows.kv import KeyValue + + +@pytest.mark.parametrize( + "input_data,expected", + [ + ( + ("field1,field2,3,nested:value", None, [","]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + KeyValue("elem_3", "nested:value"), + ], + ), + ( + ( + "field1,field2,3,nested:value", + ListDecoders( + [ + ("key1", str), + ("key2", str), + ("key3", int), + ("key4", lambda x: x.split(":"), [","]), + ] + ), + [","], + ), + [ + KeyValue("key1", "field1"), + KeyValue("key2", "field2"), + KeyValue("key3", 3), + KeyValue("key4", ["nested", "value"]), + ], + ), + ( + ("field1:field2:3", None, [":"]), + [ + KeyValue("elem_0", "field1"), + KeyValue("elem_1", "field2"), + KeyValue("elem_2", 3), + ], + ), + ], +) +def test_kv_parser(input_data, expected): + input_string = input_data[0] + decoders = input_data[1] + delims = input_data[2] + tparser = ListParser(input_string, decoders, delims) + tparser.parse() + result = tparser.kv() + assert len(expected) == len(result) + for i in range(0, len(result)): + assert result[i].key == expected[i].key + assert result[i].value == expected[i].value + kpos = result[i].meta.kpos + kstr = result[i].meta.kstring + vpos = result[i].meta.vpos + vstr = result[i].meta.vstring + assert input_string[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert input_string[vpos : vpos + len(vstr)] == vstr From patchwork Fri Mar 11 15:21:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604422 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=CCaBRbAD; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV810py3z9sG3 for ; Sat, 12 Mar 2022 02:23:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B909A613ED; Fri, 11 Mar 2022 15:23:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FY4dhNBO-F5J; Fri, 11 Mar 2022 15:23:37 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id C3A0E613C4; Fri, 11 Mar 2022 15:23:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8A666C0012; Fri, 11 Mar 2022 15:23:35 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4950EC0073 for ; Fri, 11 Mar 2022 15:23:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 6268241A19 for ; Fri, 11 Mar 2022 15:22:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 11kHPu_yZnat for ; Fri, 11 Mar 2022 15:22:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 1B7B7419F3 for ; Fri, 11 Mar 2022 15:22:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012146; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/8jZknjcYLntsxfxAUjZPdNLXcMtoqgzNZTpkW+GghI=; b=CCaBRbADniW/bsp71MtGEb85ygMWdqlg33udrGA7QYTCncOK+vfaj9m6D8psK8f7hBXnIS xu9/c8vmD5Ri9pYHOvwF3UDOuHdyvE7U3NEdkrgMDxvaD/+vGcxhXwOtNQy5fV+ibYj0C/ cuaknbD4yZl9YIb/a3U9fsk7X5egeVI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-267-BNLJ80QLONKemHEX9qoDNA-1; Fri, 11 Mar 2022 10:22:24 -0500 X-MC-Unique: BNLJ80QLONKemHEX9qoDNA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 66D25800423 for ; Fri, 11 Mar 2022 15:22:23 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 32A027F0D0; Fri, 11 Mar 2022 15:22:19 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:26 +0100 Message-Id: <20220311152128.3988946-17-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 16/18] python: add unit tests for openflow parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit tests for OFPFlow class and ip-port range decoder Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 4 +- python/ovs/tests/test_decoders.py | 130 ++++++++ python/ovs/tests/test_ofp.py | 534 ++++++++++++++++++++++++++++++ 3 files changed, 667 insertions(+), 1 deletion(-) create mode 100644 python/ovs/tests/test_decoders.py create mode 100644 python/ovs/tests/test_ofp.py diff --git a/python/automake.mk b/python/automake.mk index 559ec3019..c9d9d02d1 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -54,8 +54,10 @@ ovs_pyfiles = \ python/ovs/winutils.py ovs_pytests = \ + python/ovs/tests/test_decoders.py \ python/ovs/tests/test_kv.py \ - python/ovs/tests/test_list.py + python/ovs/tests/test_list.py \ + python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, # so they are not installed. diff --git a/python/ovs/tests/test_decoders.py b/python/ovs/tests/test_decoders.py new file mode 100644 index 000000000..e387c5b2a --- /dev/null +++ b/python/ovs/tests/test_decoders.py @@ -0,0 +1,130 @@ +from netaddr import IPAddress +import pytest + +from ovs.flows.decoders import decode_ip_port_range + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "192.168.0.0-192.168.0.200:1000-2000", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + }, + ), + ( + "192.168.0.0-192.168.0.200", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + }, + ), + ( + "192.168.0.0-192.168.0.200:2000", + { + "addrs": { + "start": IPAddress("192.168.0.0"), + "end": IPAddress("192.168.0.200"), + }, + "ports": { + "start": 2000, + "end": 2000, + }, + }, + ), + ( + "192.168.0.1:1000-2000", + { + "addrs": { + "start": IPAddress("192.168.0.1"), + "end": IPAddress("192.168.0.1"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + }, + ), + ( + "[fe80:0000:0000:0000:0204:61ff:fe9d:f150]-[fe80:0000:0000:0000:0204:61ff:fe9d:f15f]:255", # noqa: E501 + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f15f" + ), + }, + "ports": { + "start": 255, + "end": 255, + }, + }, + ), + ( + "[fe80::204:61ff:254.157.241.86]-[fe80::204:61ff:254.157.241.100]:255-300", # noqa: E501 + { + "addrs": { + "start": IPAddress("fe80::204:61ff:254.157.241.86"), + "end": IPAddress("fe80::204:61ff:254.157.241.100"), + }, + "ports": { + "start": 255, + "end": 300, + }, + }, + ), + ( + "[fe80::f150]-[fe80::f15f]:255-300", + { + "addrs": { + "start": IPAddress("fe80::f150"), + "end": IPAddress("fe80::f15f"), + }, + "ports": { + "start": 255, + "end": 300, + }, + }, + ), + ( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150-fe80:0000:0000:0000:0204:61ff:fe9d:f15f", # noqa: E501 + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f150" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f15f" + ), + }, + }, + ), + ( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156", + { + "addrs": { + "start": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156" + ), + "end": IPAddress( + "fe80:0000:0000:0000:0204:61ff:fe9d:f156" + ), + }, + }, + ), + ], +) +def test_decode_ip_port_range(input_string, expected): + assert expected == decode_ip_port_range(input_string) diff --git a/python/ovs/tests/test_ofp.py b/python/ovs/tests/test_ofp.py new file mode 100644 index 000000000..925daca46 --- /dev/null +++ b/python/ovs/tests/test_ofp.py @@ -0,0 +1,534 @@ +import netaddr +import pytest + +from ovs.flows.ofp import OFPFlow +from ovs.flows.kv import KeyValue +from ovs.flows.decoders import EthMask, IPMask, decode_mask + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions=local,3,4,5,output:foo", + [ + KeyValue("output", {"port": "local"}), + KeyValue("output", {"port": 3}), + KeyValue("output", {"port": 4}), + KeyValue("output", {"port": 5}), + KeyValue("output", {"port": "foo"}), + ], + ), + ( + "actions=controller,controller:200", + [ + KeyValue("output", "controller"), + KeyValue("controller", {"max_len": 200}), + ], + ), + ( + "actions=enqueue(foo,42),enqueue:foo:42,enqueue(bar,4242)", + [ + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "foo", "queue": 42}), + KeyValue("enqueue", {"port": "bar", "queue": 4242}), + ], + ), + ( + "actions=bundle(eth_src,0,hrw,ofport,members:4,8)", + [ + KeyValue( + "bundle", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=bundle_load(eth_src,0,hrw,ofport,reg0,members:4,8)", + [ + KeyValue( + "bundle_load", + { + "fields": "eth_src", + "basis": 0, + "algorithm": "hrw", + "dst": "reg0", + "members": [4, 8], + }, + ), + ], + ), + ( + "actions=group:3", + [KeyValue("group", 3)], + ), + ( + "actions=strip_vlan", + [KeyValue("strip_vlan", True)], + ), + ( + "actions=pop_vlan", + [KeyValue("pop_vlan", True)], + ), + ( + "actions=push_vlan:0x8100", + [KeyValue("push_vlan", 0x8100)], + ), + ( + "actions=push_mpls:0x8848", + [KeyValue("push_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=pop_mpls:0x8848", + [KeyValue("pop_mpls", 0x8848)], + ), + ( + "actions=encap(nsh(md_type=2,tlv(0x1000,10,0x12345678)))", + [ + KeyValue( + "encap", + { + "header": "nsh", + "props": { + "md_type": 2, + "tlv": { + "class": 0x1000, + "type": 10, + "value": 0x12345678, + }, + }, + }, + ) + ], + ), + ( + "actions=encap(ethernet)", + [ + KeyValue( + "encap", + {"header": "ethernet"}, + ) + ], + ), + ( + "actions=encap(mpls)", + [ + KeyValue( + "encap", + {"header": "mpls"}, + ) + ], + ), + ( + "actions=load:0x001122334455->eth_src", + [ + KeyValue( + "load", + {"value": 0x001122334455, "dst": {"field": "eth_src"}}, + ) + ], + ), + ( + "actions=load:1->eth_src[1]", + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "eth_src", "start": 1, "end": 1}, + }, + ) + ], + ), + ( + "actions=learn(load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[])", + [ + KeyValue( + "learn", + [ + { + "load": { + "src": {"field": "NXM_NX_TUN_ID"}, + "dst": {"field": "NXM_NX_TUN_ID"}, + } + } + ], + ), + ], + ), + ( + "actions=set_field:00:11:22:33:44:55->eth_src", + [ + KeyValue( + "set_field", + { + "value": {"eth_src": EthMask("00:11:22:33:44:55")}, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:01:00:00:00:00:00/01:00:00:00:00:00->eth_src", + [ + KeyValue( + "set_field", + { + "value": { + "eth_src": EthMask( + "01:00:00:00:00:00/01:00:00:00:00:00" + ) + }, + "dst": {"field": "eth_src"}, + }, + ) + ], + ), + ( + "actions=set_field:0x10ff->vlan_vid", + [ + KeyValue( + "set_field", + { + "value": {"vlan_vid": decode_mask(13)("0x10ff")}, + "dst": {"field": "vlan_vid"}, + }, + ) + ], + ), + ( + "actions=move:reg0[0..5]->reg1[16..31]", + [ + KeyValue( + "move", + { + "src": {"field": "reg0", "start": 0, "end": 5}, + "dst": {"field": "reg1", "start": 16, "end": 31}, + }, + ) + ], + ), + ( + "actions=mod_dl_dst:00:11:22:33:44:55", + [KeyValue("mod_dl_dst", EthMask("00:11:22:33:44:55"))], + ), + ( + "actions=mod_nw_dst:192.168.1.1", + [KeyValue("mod_nw_dst", IPMask("192.168.1.1"))], + ), + ( + "actions=mod_nw_dst:fe80::ec17:7bff:fe61:7aac", + [KeyValue("mod_nw_dst", IPMask("fe80::ec17:7bff:fe61:7aac"))], + ), + ( + "actions=dec_ttl,dec_ttl(1,2,3)", + [KeyValue("dec_ttl", True), KeyValue("dec_ttl", [1, 2, 3])], + ), + ( + "actions=set_mpls_label:0x100,set_mpls_tc:2,set_mpls_ttl:10", + [ + KeyValue("set_mpls_label", 0x100), + KeyValue("set_mpls_tc", 2), + KeyValue("set_mpls_ttl", 10), + ], + ), + ( + "actions=check_pkt_larger(100)->reg0[10]", + [ + KeyValue( + "check_pkt_larger", + { + "pkt_len": 100, + "dst": {"field": "reg0", "start": 10, "end": 10}, + }, + ), + ], + ), + ( + "actions=pop_queue,set_tunnel:0x10,set_tunnel64:0x65000,set_queue=3", # noqa: E501 + [ + KeyValue("pop_queue", True), + KeyValue("set_tunnel", 0x10), + KeyValue("set_tunnel64", 0x65000), + KeyValue("set_queue", 3), + ], + ), + ( + "actions=ct(zone=10,table=2,nat(snat=192.168.0.0-192.168.0.200:1000-2000,random))", # noqa: E501 + [ + KeyValue( + "ct", + { + "zone": 10, + "table": 2, + "nat": { + "type": "snat", + "addrs": { + "start": netaddr.IPAddress("192.168.0.0"), + "end": netaddr.IPAddress("192.168.0.200"), + }, + "ports": { + "start": 1000, + "end": 2000, + }, + "random": True, + }, + }, + ) + ], + ), + ( + "actions=ct(commit,zone=NXM_NX_REG13[0..15],table=2,exec(load:0->NXM_NX_CT_LABEL[0]))", # noqa: E501 + [ + KeyValue( + "ct", + { + "commit": True, + "zone": { + "field": "NXM_NX_REG13", + "start": 0, + "end": 15, + }, + "table": 2, + "exec": [ + { + "load": { + "value": 0, + "dst": { + "field": "NXM_NX_CT_LABEL", + "start": 0, + "end": 0, + }, + }, + }, + ], + }, + ) + ], + ), + ( + "actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xda6f52b0,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=172.30.204.105,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])", # noqa: E501 + [ + KeyValue( + "load", + { + "value": 1, + "dst": {"field": "NXM_NX_REG10", "start": 7, "end": 7}, + }, + ), + KeyValue( + "learn", + [ + {"table": 69}, + {"delete_learned": True}, + {"cookie": 3664728752}, + {"OXM_OF_METADATA[]": True}, + {"eth_type": 2048}, + {"NXM_OF_IP_SRC[]": True}, + {"ip_dst": IPMask("172.30.204.105/32")}, + {"nw_proto": 6}, + {"NXM_OF_TCP_SRC[]": "NXM_OF_TCP_DST[]"}, + { + "load": { + "value": 1, + "dst": { + "field": "NXM_NX_REG10", + "start": 7, + "end": 7, + }, + } + }, + ], + ), + ], + ), + ( + "actions=resubmit(,8),resubmit:3,resubmit(1,2,ct)", + [ + KeyValue("resubmit", {"port": "", "table": 8}), + KeyValue("resubmit", {"port": 3}), + KeyValue("resubmit", {"port": 1, "table": 2, "ct": True}), + ], + ), + ( + "actions=clone(ct_clear,load:0->NXM_NX_REG11[],load:0->NXM_NX_REG12[],load:0->NXM_NX_REG13[],load:0x1d->NXM_NX_REG13[],load:0x1f->NXM_NX_REG11[],load:0x1c->NXM_NX_REG12[],load:0x11->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0->NXM_NX_REG10[],load:0->NXM_NX_REG15[],load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],load:0->NXM_NX_REG5[],load:0->NXM_NX_REG6[],load:0->NXM_NX_REG7[],load:0->NXM_NX_REG8[],load:0->NXM_NX_REG9[],resubmit(,8))", # noqa: E501 + [ + KeyValue( + "clone", + [ + {"ct_clear": True}, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 29, + "dst": {"field": "NXM_NX_REG13"}, + } + }, + { + "load": { + "value": 31, + "dst": {"field": "NXM_NX_REG11"}, + } + }, + { + "load": { + "value": 28, + "dst": {"field": "NXM_NX_REG12"}, + } + }, + { + "load": { + "value": 17, + "dst": {"field": "OXM_OF_METADATA"}, + } + }, + { + "load": { + "value": 2, + "dst": {"field": "NXM_NX_REG14"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG10"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG15"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG0"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG1"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG2"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG3"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG4"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG5"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG6"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG7"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG8"}, + } + }, + { + "load": { + "value": 0, + "dst": {"field": "NXM_NX_REG9"}, + } + }, + {"resubmit": {"port": "", "table": 8}}, + ], + ) + ], + ), + ( + "actions=conjunction(1234, 1/2),note:00.00.11.22.33.ff,sample(probability=123,collector_set_id=0x123,obs_domain_id=0x123,obs_point_id=0x123,sampling_port=inport0,ingress)", # noqa: E501 + [ + KeyValue("conjunction", {"id": 1234, "k": 1, "n": 2}), + KeyValue("note", "00.00.11.22.33.ff"), + KeyValue( + "sample", + { + "probability": 123, + "collector_set_id": 0x123, + "obs_domain_id": 0x123, + "obs_point_id": 0x123, + "sampling_port": "inport0", + "ingress": True, + }, + ), + ], + ), + ], +) +def test_act(input_string, expected): + ofp = OFPFlow(input_string) + actions = ofp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = ofp.section("actions").pos + astring = ofp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Fri Mar 11 15:21:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604421 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LKYvajSz; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV7j0WGPz9sG3 for ; Sat, 12 Mar 2022 02:23:29 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 88B4D40D7C; Fri, 11 Mar 2022 15:23:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gaPHINNzQI1U; Fri, 11 Mar 2022 15:23:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id EBB3A40CE5; Fri, 11 Mar 2022 15:23:20 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4E4FBC0073; Fri, 11 Mar 2022 15:23:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1ADAFC0089 for ; Fri, 11 Mar 2022 15:23:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 80E3B8481D for ; Fri, 11 Mar 2022 15:22:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dK6VNrFmm4LY for ; Fri, 11 Mar 2022 15:22:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 53E2384817 for ; Fri, 11 Mar 2022 15:22:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012147; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YAoHYrRhfBLnPeZ9XbGMfJzek/P4GNd1KMeY041AS7c=; b=LKYvajSzgTxq5MSPyYfW9HVVeWM75j13/LD6ytZmHereDncmvnTUegYN8vG4oUuivMdqE+ sdDfpntpL6K/L2hyVfflxC7zICVysETrIb72HCPHYQq2oVsO9Zb54jXvrFVYz2DcoeKNCY 3CcomXZ9PkEX0kN8bZI+uxLPXU1bCGo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-210-Xn4RA-YjOii2uF0t8z3WLQ-1; Fri, 11 Mar 2022 10:22:25 -0500 X-MC-Unique: Xn4RA-YjOii2uF0t8z3WLQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 06666800422 for ; Fri, 11 Mar 2022 15:22:25 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD10A7F0DC; Fri, 11 Mar 2022 15:22:23 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:27 +0100 Message-Id: <20220311152128.3988946-18-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 17/18] python: add unit tests to datapath parsing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 1 + python/ovs/tests/test_odp.py | 527 +++++++++++++++++++++++++++++++++++ 2 files changed, 528 insertions(+) create mode 100644 python/ovs/tests/test_odp.py diff --git a/python/automake.mk b/python/automake.mk index c9d9d02d1..b2e7f6008 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -57,6 +57,7 @@ ovs_pytests = \ python/ovs/tests/test_decoders.py \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ + python/ovs/tests/test_odp.py \ python/ovs/tests/test_ofp.py # These python files are used at build time but not runtime, diff --git a/python/ovs/tests/test_odp.py b/python/ovs/tests/test_odp.py new file mode 100644 index 000000000..8ca8f38bf --- /dev/null +++ b/python/ovs/tests/test_odp.py @@ -0,0 +1,527 @@ +import netaddr +import pytest + +from ovs.flows.odp import ODPFlow +from ovs.flows.kv import KeyValue +from ovs.flows.decoders import ( + EthMask, + IPMask, + Mask32, + Mask16, + Mask8, + Mask128, +) + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "skb_priority(0x123),skb_mark(0x123),recirc_id(0x123),dp_hash(0x123),ct_zone(0x123), actions:", # noqa: E501 + [ + KeyValue("skb_priority", Mask32("0x123")), + KeyValue("skb_mark", Mask32("0x123")), + KeyValue("recirc_id", 0x123), + KeyValue("dp_hash", Mask32("0x123")), + KeyValue("ct_zone", Mask16("0x123")), + ], + ), + ( + "tunnel(tun_id=0x7f10354,src=10.10.10.10,dst=20.20.20.20,ttl=64,flags(csum|key)) actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "tun_id": 0x7F10354, + "src": IPMask("10.10.10.10"), + "dst": IPMask("20.20.20.20"), + "ttl": 64, + "flags": "csum|key", + }, + ) + ], + ), + ( + "tunnel(geneve({class=0,type=0,len=4,0xa/0xff}),vxlan(flags=0x800000,vni=0x1c7),erspan(ver=2,dir=1,hwid=0x1)), actions:", # noqa: E501 + [ + KeyValue( + "tunnel", + { + "geneve": [ + { + "class": Mask16("0"), + "type": Mask8("0"), + "len": Mask8("4"), + "data": Mask128("0xa/0xff"), + } + ], + "vxlan": {"flags": 0x800000, "vni": 0x1C7}, + "erspan": {"ver": 2, "dir": 1, "hwid": 0x1}, + }, + ) + ], + ), + ( + "in_port(2),eth(src=11:22:33:44:55:66,dst=66:55:44:33:22:11) actions:", # noqa: E501 + [ + KeyValue("in_port", 2), + KeyValue( + "eth", + { + "src": EthMask("11:22:33:44:55:66"), + "dst": EthMask("66:55:44:33:22:11"), + }, + ), + ], + ), + ( + "eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0) actions:", # noqa: E501 + [ + KeyValue("eth_type", Mask16("0x800/0x006")), + KeyValue( + "ipv4", + { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xFF"), + "tos": Mask8("0x2/0xF0"), + }, + ), + ], + ), + ( + "encap(eth_type(0x800/0x006),ipv4(src=192.168.1.1/24,dst=192.168.0.0/16,proto=0x1,tos=0x2/0xf0)) actions:", # noqa: E501 + [ + KeyValue( + "encap", + { + "eth_type": Mask16("0x800/0x006"), + "ipv4": { + "src": IPMask("192.168.1.1/24"), + "dst": IPMask("192.168.0.0/16"), + "proto": Mask8("0x1/0xff"), + "tos": Mask8("0x2/0xf0"), + }, + }, + ), + ], + ), + ], +) +def test_odp_fields(input_string, expected): + odp = ODPFlow(input_string) + match = odp.match_kv + for i in range(len(expected)): + assert expected[i].key == match[i].key + assert expected[i].value == match[i].value + + # Assert positions relative to action string are OK. + mpos = odp.section("match").pos + mstring = odp.section("match").string + + kpos = match[i].meta.kpos + kstr = match[i].meta.kstring + vpos = match[i].meta.vpos + vstr = match[i].meta.vstring + assert mstring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert mstring[vpos : vpos + len(vstr)] == vstr + + # Assert mstring meta is correct. + assert input_string[mpos : mpos + len(mstring)] == mstring + + +@pytest.mark.parametrize( + "input_string,expected", + [ + ( + "actions:ct" + ",ct(commit)" + ",ct(commit,zone=5)" + ",ct(commit,mark=0xa0a0a0a0/0xfefefefe)" + ",ct(commit,label=0x1234567890abcdef1234567890abcdef/0xf1f2f3f4f5f6f7f8f9f0fafbfcfdfeff)" # noqa: E501 + ",ct(commit,helper=ftp)" + ",ct(commit,helper=tftp)" + ",ct(commit,timeout=ovs_tp_1_tcp4)" + ",ct(nat)", + [ + KeyValue("ct", True), + KeyValue("ct", {"commit": True}), + KeyValue("ct", {"commit": True, "zone": 5}), + KeyValue( + "ct", + {"commit": True, "mark": Mask32("0xA0A0A0A0/0xFEFEFEFE")}, + ), + KeyValue( + "ct", + { + "commit": True, + "label": Mask128( + "0x1234567890ABCDEF1234567890ABCDEF/0xF1F2F3F4F5F6F7F8F9F0FAFBFCFDFEFF" # noqa: E501 + ), + }, + ), + KeyValue("ct", {"commit": True, "helper": "ftp"}), + KeyValue("ct", {"commit": True, "helper": "tftp"}), + KeyValue("ct", {"commit": True, "timeout": "ovs_tp_1_tcp4"}), + KeyValue("ct", {"nat": True}), + ], + ), + ( + "actions:ct(nat)" + ",ct(commit,nat(src))" + ",ct(commit,nat(dst))" + ",ct(commit,nat(src=10.0.0.240,random))" + ",ct(commit,nat(src=10.0.0.240:32768-65535,random))" + ",ct(commit,nat(dst=10.0.0.128-10.0.0.254,hash))" + ",ct(commit,nat(src=10.0.0.240-10.0.0.254:32768-65535,persistent))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:a18b,random))" + ",ct(commit,nat(src=fe80::20c:29ff:fe88:1-fe80::20c:29ff:fe88:a18b,random))" # noqa: E501 + ",ct(commit,nat(src=[[fe80::20c:29ff:fe88:1]]-[[fe80::20c:29ff:fe88:a18b]]:255-4096,random))" # noqa: E501 + ",ct(commit,helper=ftp,nat(src=10.1.1.240-10.1.1.255))" + ",ct(force_commit)", + [ + KeyValue("ct", {"nat": True}), + KeyValue("ct", {"commit": True, "nat": {"type": "src"}}), + KeyValue("ct", {"commit": True, "nat": {"type": "dst"}}), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.240"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "dst", + "addrs": { + "start": netaddr.IPAddress("10.0.0.128"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "hash": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.0.0.240"), + "end": netaddr.IPAddress("10.0.0.254"), + }, + "ports": { + "start": 32768, + "end": 65535, + }, + "persistent": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress( + "fe80::20c:29ff:fe88:1" + ), + "end": netaddr.IPAddress( + "fe80::20c:29ff:fe88:a18b" + ), + }, + "ports": { + "start": 255, + "end": 4096, + }, + "random": True, + }, + }, + ), + KeyValue( + "ct", + { + "commit": True, + "nat": { + "type": "src", + "addrs": { + "start": netaddr.IPAddress("10.1.1.240"), + "end": netaddr.IPAddress("10.1.1.255"), + }, + }, + "helper": "ftp", + }, + ), + KeyValue("ct", {"force_commit": True}), + ], + ), + ( + "actions:set(tunnel(tun_id=0xabcdef1234567890,src=1.1.1.1,dst=2.2.2.2,ttl=64,flags(df|csum|key)))" # noqa: E501 + ",tnl_pop(4)" + ",tnl_push(tnl_port(6),header(size=50,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x0800),ipv4(src=1.1.2.88,dst=1.1.2.92,proto=17,tos=0,ttl=64,frag=0x4000),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))" # noqa: E501 + ",tnl_push(tnl_port(6),header(size=70,type=4,eth(dst=f8:bc:12:44:34:b6,src=f8:bc:12:46:58:e0,dl_type=0x86dd),ipv6(src=2001:cafe::88,dst=2001:cafe::92,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x0),vxlan(flags=0x8000000,vni=0x1c7)),out_port(1))", # noqa: E501 + [ + KeyValue( + "set", + { + "tunnel": { + "tun_id": 0xABCDEF1234567890, + "src": IPMask("1.1.1.1"), + "dst": IPMask("2.2.2.2"), + "ttl": 64, + "flags": "df|csum|key", + } + }, + ), + KeyValue("tnl_pop", 4), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 50, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x800, + }, + "ipv4": { + "src": IPMask("1.1.2.88"), + "dst": IPMask("1.1.2.92"), + "proto": 17, + "tos": 0, + "ttl": 64, + "frag": 0x4000, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + KeyValue( + "tnl_push", + { + "tnl_port": 6, + "header": { + "size": 70, + "type": 4, + "eth": { + "dst": EthMask("f8:bc:12:44:34:b6"), + "src": EthMask("f8:bc:12:46:58:e0"), + "dl_type": 0x86DD, + }, + "ipv6": { + "src": IPMask("2001:cafe::88"), + "dst": IPMask("2001:cafe::92"), + "label": 0, + "proto": 17, + "tclass": 0x0, + "hlimit": 64, + }, + "udp": {"src": 0, "dst": 4789, "csum": 0x0}, + "vxlan": { + "flags": 0x8000000, + "vni": 0x1C7, + }, + }, + "out_port": 1, + }, + ), + ], + ), + ( + "actions:tnl_push(header(geneve(oam,vni=0x1c7)))" + ",tnl_push(header(geneve(crit,vni=0x1c7,options({class=0xffff,type=0x80,len=4,0xa}))))" # noqa: E501 + ",tnl_push(header(gre((flags=0xa000,proto=0x6558),csum=0x0,key=0x1e241)))", # noqa: E501 + [ + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "oam": True, + "vni": 0x1C7, + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "geneve": { + "crit": True, + "vni": 0x1C7, + "options": [ + { + "class": 0xFFFF, + "type": 0x80, + "len": 4, + "data": 0xA, + } + ], + } + } + }, + ), + KeyValue( + "tnl_push", + { + "header": { + "gre": { + "flags": 0xA000, + "proto": 0x6558, + "key": 0x1E241, + "csum": 0x0, + } + } + }, + ), + ], + ), + ( + "actions:clone(1)" ",clone(clone(push_vlan(vid=12,pcp=0),2),1)", + [ + KeyValue("clone", {"output": {"port": 1}}), + KeyValue( + "clone", + { + "output": {"port": 1}, + "clone": { + "push_vlan": { + "vid": 12, + "pcp": 0, + }, + "output": {"port": 2}, + }, + }, + ), + ], + ), + ( + "actions: check_pkt_len(size=200,gt(4),le(5))" + ",check_pkt_len(size=200,gt(drop),le(5))" + ",check_pkt_len(size=200,gt(ct(nat)),le(drop))", + [ + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"output": {"port": 4}}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"drop": True}, + "le": {"output": {"port": 5}}, + }, + ), + KeyValue( + "check_pkt_len", + { + "size": 200, + "gt": {"ct": {"nat": True}}, + "le": {"drop": True}, + }, + ), + ], + ), + ], +) +def test_odp_actions(input_string, expected): + odp = ODPFlow(input_string) + actions = odp.actions_kv + for i in range(len(expected)): + assert expected[i].key == actions[i].key + assert expected[i].value == actions[i].value + + # Assert positions relative to action string are OK. + apos = odp.section("actions").pos + astring = odp.section("actions").string + + kpos = actions[i].meta.kpos + kstr = actions[i].meta.kstring + vpos = actions[i].meta.vpos + vstr = actions[i].meta.vstring + assert astring[kpos : kpos + len(kstr)] == kstr + if vpos != -1: + assert astring[vpos : vpos + len(vstr)] == vstr + + # Assert astring meta is correct. + assert input_string[apos : apos + len(astring)] == astring From patchwork Fri Mar 11 15:21:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1604424 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=W+W1D6+g; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KFV8755B4z9sG3 for ; Sat, 12 Mar 2022 02:23:51 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C99F384947; Fri, 11 Mar 2022 15:23:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tbdb5njxYinG; Fri, 11 Mar 2022 15:23:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6923284930; Fri, 11 Mar 2022 15:23:42 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 62E0DC0083; Fri, 11 Mar 2022 15:23:41 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 52CEDC0012 for ; Fri, 11 Mar 2022 15:23:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6A99040C01 for ; Fri, 11 Mar 2022 15:22:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rDFd0I_2Yp2m for ; Fri, 11 Mar 2022 15:22:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 56DC340C1D for ; Fri, 11 Mar 2022 15:22:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1647012149; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YP8ceLY3/CT1MEw/MyWDh/34pAZqc+s+HQL9ae9W/bM=; b=W+W1D6+g79erWQLMOnoApreXeWRUOnO5DpQ8juTdj/JY+LHLIY6dnrFHFJnhBeOwPtnNQk iMcdOPWmiSjG6VcAwvb5k1ktYL9VZyjQPK31PrNBY9yNoCC9f8T7O4YTxfg6uWkF0/xuuh ORbA2q/0/30bPYw50UR0Dizs48ILrSA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-20-skTclQXWNhq965YoaoRlkw-1; Fri, 11 Mar 2022 10:22:27 -0500 X-MC-Unique: skTclQXWNhq965YoaoRlkw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A35111854E2E for ; Fri, 11 Mar 2022 15:22:26 +0000 (UTC) Received: from amorenoz.users.ipa.redhat.com (unknown [10.39.192.121]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6C9EC7F0DC; Fri, 11 Mar 2022 15:22:25 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Fri, 11 Mar 2022 16:21:28 +0100 Message-Id: <20220311152128.3988946-19-amorenoz@redhat.com> In-Reply-To: <20220311152128.3988946-1-amorenoz@redhat.com> References: <20220311152128.3988946-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=amorenoz@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v3 18/18] python: add unit tests for filtering engine X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add unit test for OFFilter class. Acked-by: Eelco Chaudron Signed-off-by: Adrian Moreno --- python/automake.mk | 1 + python/ovs/tests/test_filter.py | 221 ++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 python/ovs/tests/test_filter.py diff --git a/python/automake.mk b/python/automake.mk index b2e7f6008..85ed8e715 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -55,6 +55,7 @@ ovs_pyfiles = \ ovs_pytests = \ python/ovs/tests/test_decoders.py \ + python/ovs/tests/test_filter.py \ python/ovs/tests/test_kv.py \ python/ovs/tests/test_list.py \ python/ovs/tests/test_odp.py \ diff --git a/python/ovs/tests/test_filter.py b/python/ovs/tests/test_filter.py new file mode 100644 index 000000000..823ad2b48 --- /dev/null +++ b/python/ovs/tests/test_filter.py @@ -0,0 +1,221 @@ +import pytest + +from ovs.flows.filter import OFFilter +from ovs.flows.ofp import OFPFlow +from ovs.flows.odp import ODPFlow + + +@pytest.mark.parametrize( + "expr,flow,expected,match", + [ + ( + "nw_src=192.168.1.1 && tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 || tcp_dst=80", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1 || tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + True, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.2 && tcp_dst=90", + OFPFlow( + "nw_src=192.168.1.1,tcp_dst=80 actions=drop" + ), + False, + ["nw_src", "tcp_dst"], + ), + ( + "nw_src=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.1/30", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + True, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "nw_src~=192.168.1.0/16", + OFPFlow( + "nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" + ), + False, + ["nw_src"], + ), + ( + "n_bytes=100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>10", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<100", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes<1000", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=drop" # noqa: E501 + ), + True, + ["n_bytes", "drop"], + ), + ( + "n_bytes>0 && drop=true", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + False, + ["n_bytes"], + ), + ( + "n_bytes>10 && !output.port=3", + OFPFlow( + "n_bytes=100 priority=100,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["n_bytes", "output"], + ), + ( + "dl_src=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:55", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tcp_dst=80 actions=2" # noqa: E501 + ), + True, + ["dl_src"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + False, + ["dl_src", "tp_dst"], + ), + ( + "dl_src~=00:11:22:33:44:66 && tp_dst~=1000", + OFPFlow( + "n_bytes=100 priority=100,dl_src=00:11:22:33:44:55/ff:ff:ff:ff:ff:00,nw_src=192.168.1.0/24,tp_dst=0x03e8/0xfff8 actions=2" # noqa: E501 + ), + True, + ["dl_src", "tp_dst"], + ), + ( + "encap", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ( + "encap.ipv4.src=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + False, + ["encap"], + ), + ( + "encap.ipv4.src~=10.76.23.240", + ODPFlow( + "encap(eth_type(0x0800),ipv4(src=10.76.23.240/255.255.255.248,dst=10.76.23.106,proto=17,tos=0/0,ttl=64,frag=no)) actions:drop" # noqa: E501 + ), + True, + ["encap"], + ), + ], +) +def test_filter(expr, flow, expected, match): + ffilter = OFFilter(expr) + result = ffilter.evaluate(flow) + if expected: + assert result + else: + assert not result + + assert [kv.key for kv in result.kv] == match