[ovs-dev,v2,03/18] python: add list parser

Some openflow or dpif flows encode their arguments in lists, eg:
"some_action(arg1,arg2,arg3)". In order to decode this in a way that can
be then stored and queried, add ListParser and ListDecoders classes
that parse lists into KeyValue instances.

The ListParser / ListDecoders mechanism is quite similar to KVParser and
KVDecoders. Since the "key" of the different KeyValue objects is now
ommited, it has to be provided by ListDecoders.

For example, take the openflow action "resubmit" that can be written as:

    resubmit([port],[table][,ct])

Can be decoded by creating a ListDecoders instance such as:

    ListDecoders([
                  ("port", decode_default),
                  ("table", decode_int),
                  ("ct", decode_flag),
                ])

Naturally, the order of the decoders must be kept.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
---
 python/automake.mk       |   1 +
 python/ovs/flows/list.py | 124 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 125 insertions(+)
 create mode 100644 python/ovs/flows/list.py

Adrian Moreno <amorenoz@redhat.com> writes:

> diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py
> new file mode 100644
> index 000000000..cafd23d0a
> --- /dev/null
> +++ b/python/ovs/flows/list.py
> @@ -0,0 +1,124 @@
> +import re
> +
> +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError
> +from ovs.flows.decoders import decode_default
> +
> +
> +class ListDecoders(object):
> +    """ListDecoders is used by ListParser to decode the elements in the list.
> +
> +    A decoder is a function that accepts a value and returns its decoded
> +    object.
> +
> +    ListDecoders is initialized with a list of of tuples that contains the

s/of of/of/

On 1/28/22 11:04, Adrian Moreno wrote:
> Some openflow or dpif flows encode their arguments in lists, eg:
> "some_action(arg1,arg2,arg3)". In order to decode this in a way that can
> be then stored and queried, add ListParser and ListDecoders classes
> that parse lists into KeyValue instances.
> 
> The ListParser / ListDecoders mechanism is quite similar to KVParser and
> KVDecoders. Since the "key" of the different KeyValue objects is now
> ommited, it has to be provided by ListDecoders.
> 
> For example, take the openflow action "resubmit" that can be written as:
> 
>      resubmit([port],[table][,ct])
> 
> Can be decoded by creating a ListDecoders instance such as:
> 
>      ListDecoders([
>                    ("port", decode_default),
>                    ("table", decode_int),
>                    ("ct", decode_flag),
>                  ])
> 
> Naturally, the order of the decoders must be kept.
> 
> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
> ---
>   python/automake.mk       |   1 +
>   python/ovs/flows/list.py | 124 +++++++++++++++++++++++++++++++++++++++
>   2 files changed, 125 insertions(+)
>   create mode 100644 python/ovs/flows/list.py
> 
> diff --git a/python/automake.mk b/python/automake.mk
> index 7ce842d66..73438d615 100644
> --- a/python/automake.mk
> +++ b/python/automake.mk
> @@ -29,6 +29,7 @@ ovs_pyfiles = \
>   	python/ovs/flows/__init__.py \
>   	python/ovs/flows/decoders.py \
>   	python/ovs/flows/kv.py \
> +	python/ovs/flows/list.py \
>   	python/ovs/json.py \
>   	python/ovs/jsonrpc.py \
>   	python/ovs/ovsuuid.py \
> diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py
> new file mode 100644
> index 000000000..cafd23d0a
> --- /dev/null
> +++ b/python/ovs/flows/list.py
> @@ -0,0 +1,124 @@
> +import re
> +
> +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError
> +from ovs.flows.decoders import decode_default
> +
> +
> +class ListDecoders(object):
> +    """ListDecoders is used by ListParser to decode the elements in the list.
> +
> +    A decoder is a function that accepts a value and returns its decoded
> +    object.
> +
> +    ListDecoders is initialized with a list of of tuples that contains the
> +    keyword and the decoding function associated with each position in the
> +    list. The order is, therefore, important.
> +
> +    Args:
> +        decoders (list of tuples): Optional; a list of tuples.
> +            The first element in the tuple is the keyword associated with the
> +            value. The second element in the tuple is the decoder function.
> +    """
> +
> +    def __init__(self, decoders=None):
> +        self._decoders = decoders or list()
> +
> +    def decode(self, index, value_str):
> +        """Decode the index'th element of the list.
> +
> +        Args:
> +            index (int): The position in the list of the element to decode.
> +            value_str (str): The value string to decode.
> +        """
> +        if index < 0 or index >= len(self._decoders):
> +            return self._default_decoder(index, value_str)
> +
> +        try:
> +            key = self._decoders[index][0]
> +            value = self._decoders[index][1](value_str)
> +            return key, value
> +        except Exception as e:
> +            raise ParseError(
> +                "Failed to decode value_str {}: {}".format(value_str, str(e))
> +            )
> +
> +    @staticmethod
> +    def _default_decoder(index, value):
> +        key = "elem_{}".format(index)
> +        return key, decode_default(value)
> +
> +
> +class ListParser(object):
> +    """ListParser parses a list of values and stores them as key-value pairs.
> +
> +    It uses a ListDecoders instance to decode each element in the list.
> +
> +    Args:
> +        string (str): The string to parse.
> +        decoders (ListDecoders): Optional, the decoders to use.
> +        delims (list): Optional, list of delimiters of the list. Defaults to
> +            [','].
> +    """
> +    def __init__(self, string, decoders=None, delims=[","]):
> +        self._string = string
> +        self._decoders = decoders or ListDecoders()
> +        self._keyval = list()
> +        self._regexp = r"({})".format("|".join(delims))
> +
> +    def kv(self):
> +        return self._keyval
> +
> +    def __iter__(self):
> +        return iter(self._keyval)
> +
> +    def parse(self):
> +        """Parse the list in string.
> +
> +        Raises:
> +            ParseError if any parsing error occurs.
> +        """
> +        kpos = 0
> +        index = 0
> +        while kpos < len(self._string) and self._string[kpos] != "\n":
> +            split_parts = re.split(self._regexp, self._string[kpos:], 1)
> +            if len(split_parts) == 0:
> +                break

Like I commented in patch 1, I don't think len(split_parts) can be 0 here.

> +
> +            value_str = split_parts[0]
> +
> +            key, value = self._decoders.decode(index, value_str)
> +
> +            meta = KeyMetadata(
> +                kpos=kpos,
> +                vpos=kpos,
> +                kstring=value_str,
> +                vstring=value_str,
> +            )
> +            self._keyval.append(KeyValue(key, value, meta))
> +
> +            kpos += len(value_str) + 1
> +            index += 1
> +
> +
> +def decode_nested_list(decoders, value, delims=[","]):
> +    """Decodes a string value that contains a list of elements and returns
> +    them in a dictionary.
> +
> +    Args:
> +        decoders (ListDecoders): The ListDecoders to use.
> +        value (str): The value string to decode.
> +        delims (list(str)): Optional, the list of delimiters to use.
> +    """
> +    parser = ListParser(value, decoders, delims)
> +    parser.parse()
> +    return {kv.key: kv.value for kv in parser.kv()}
> +
> +
> +def nested_list_decoder(decoders=None, delims=[","]):
> +    """Helper function that creates a nested list decoder with given
> +    ListDecoders and delimiters.
> +    """
> +    def decoder(value):
> +        return decode_nested_list(decoders, value, delims)
> +
> +    return decoder
>

On 2/5/22 03:55, Mark Michelson wrote:
> On 1/28/22 11:04, Adrian Moreno wrote:
>> Some openflow or dpif flows encode their arguments in lists, eg:
>> "some_action(arg1,arg2,arg3)". In order to decode this in a way that can
>> be then stored and queried, add ListParser and ListDecoders classes
>> that parse lists into KeyValue instances.
>>
>> The ListParser / ListDecoders mechanism is quite similar to KVParser and
>> KVDecoders. Since the "key" of the different KeyValue objects is now
>> ommited, it has to be provided by ListDecoders.
>>
>> For example, take the openflow action "resubmit" that can be written as:
>>
>>      resubmit([port],[table][,ct])
>>
>> Can be decoded by creating a ListDecoders instance such as:
>>
>>      ListDecoders([
>>                    ("port", decode_default),
>>                    ("table", decode_int),
>>                    ("ct", decode_flag),
>>                  ])
>>
>> Naturally, the order of the decoders must be kept.
>>
>> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
>> ---
>>   python/automake.mk       |   1 +
>>   python/ovs/flows/list.py | 124 +++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 125 insertions(+)
>>   create mode 100644 python/ovs/flows/list.py
>>
>> diff --git a/python/automake.mk b/python/automake.mk
>> index 7ce842d66..73438d615 100644
>> --- a/python/automake.mk
>> +++ b/python/automake.mk
>> @@ -29,6 +29,7 @@ ovs_pyfiles = \
>>       python/ovs/flows/__init__.py \
>>       python/ovs/flows/decoders.py \
>>       python/ovs/flows/kv.py \
>> +    python/ovs/flows/list.py \
>>       python/ovs/json.py \
>>       python/ovs/jsonrpc.py \
>>       python/ovs/ovsuuid.py \
>> diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py
>> new file mode 100644
>> index 000000000..cafd23d0a
>> --- /dev/null
>> +++ b/python/ovs/flows/list.py
>> @@ -0,0 +1,124 @@
>> +import re
>> +
>> +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError
>> +from ovs.flows.decoders import decode_default
>> +
>> +
>> +class ListDecoders(object):
>> +    """ListDecoders is used by ListParser to decode the elements in the list.
>> +
>> +    A decoder is a function that accepts a value and returns its decoded
>> +    object.
>> +
>> +    ListDecoders is initialized with a list of of tuples that contains the
>> +    keyword and the decoding function associated with each position in the
>> +    list. The order is, therefore, important.
>> +
>> +    Args:
>> +        decoders (list of tuples): Optional; a list of tuples.
>> +            The first element in the tuple is the keyword associated with the
>> +            value. The second element in the tuple is the decoder function.
>> +    """
>> +
>> +    def __init__(self, decoders=None):
>> +        self._decoders = decoders or list()
>> +
>> +    def decode(self, index, value_str):
>> +        """Decode the index'th element of the list.
>> +
>> +        Args:
>> +            index (int): The position in the list of the element to decode.
>> +            value_str (str): The value string to decode.
>> +        """
>> +        if index < 0 or index >= len(self._decoders):
>> +            return self._default_decoder(index, value_str)
>> +
>> +        try:
>> +            key = self._decoders[index][0]
>> +            value = self._decoders[index][1](value_str)
>> +            return key, value
>> +        except Exception as e:
>> +            raise ParseError(
>> +                "Failed to decode value_str {}: {}".format(value_str, str(e))
>> +            )
>> +
>> +    @staticmethod
>> +    def _default_decoder(index, value):
>> +        key = "elem_{}".format(index)
>> +        return key, decode_default(value)
>> +
>> +
>> +class ListParser(object):
>> +    """ListParser parses a list of values and stores them as key-value pairs.
>> +
>> +    It uses a ListDecoders instance to decode each element in the list.
>> +
>> +    Args:
>> +        string (str): The string to parse.
>> +        decoders (ListDecoders): Optional, the decoders to use.
>> +        delims (list): Optional, list of delimiters of the list. Defaults to
>> +            [','].
>> +    """
>> +    def __init__(self, string, decoders=None, delims=[","]):
>> +        self._string = string
>> +        self._decoders = decoders or ListDecoders()
>> +        self._keyval = list()
>> +        self._regexp = r"({})".format("|".join(delims))
>> +
>> +    def kv(self):
>> +        return self._keyval
>> +
>> +    def __iter__(self):
>> +        return iter(self._keyval)
>> +
>> +    def parse(self):
>> +        """Parse the list in string.
>> +
>> +        Raises:
>> +            ParseError if any parsing error occurs.
>> +        """
>> +        kpos = 0
>> +        index = 0
>> +        while kpos < len(self._string) and self._string[kpos] != "\n":
>> +            split_parts = re.split(self._regexp, self._string[kpos:], 1)
>> +            if len(split_parts) == 0:
>> +                break
> 
> Like I commented in patch 1, I don't think len(split_parts) can be 0 here.
> 

Agree. I'll remove it in the next version.

>> +
>> +            value_str = split_parts[0]
>> +
>> +            key, value = self._decoders.decode(index, value_str)
>> +
>> +            meta = KeyMetadata(
>> +                kpos=kpos,
>> +                vpos=kpos,
>> +                kstring=value_str,
>> +                vstring=value_str,
>> +            )
>> +            self._keyval.append(KeyValue(key, value, meta))
>> +
>> +            kpos += len(value_str) + 1
>> +            index += 1
>> +
>> +
>> +def decode_nested_list(decoders, value, delims=[","]):
>> +    """Decodes a string value that contains a list of elements and returns
>> +    them in a dictionary.
>> +
>> +    Args:
>> +        decoders (ListDecoders): The ListDecoders to use.
>> +        value (str): The value string to decode.
>> +        delims (list(str)): Optional, the list of delimiters to use.
>> +    """
>> +    parser = ListParser(value, decoders, delims)
>> +    parser.parse()
>> +    return {kv.key: kv.value for kv in parser.kv()}
>> +
>> +
>> +def nested_list_decoder(decoders=None, delims=[","]):
>> +    """Helper function that creates a nested list decoder with given
>> +    ListDecoders and delimiters.
>> +    """
>> +    def decoder(value):
>> +        return decode_nested_list(decoders, value, delims)
>> +
>> +    return decoder
>>
>

On 1/31/22 22:06, James Troup wrote:
> Adrian Moreno <amorenoz@redhat.com> writes:
> 
>> diff --git a/python/ovs/flows/list.py b/python/ovs/flows/list.py
>> new file mode 100644
>> index 000000000..cafd23d0a
>> --- /dev/null
>> +++ b/python/ovs/flows/list.py
>> @@ -0,0 +1,124 @@
>> +import re
>> +
>> +from ovs.flows.kv import KeyValue, KeyMetadata, ParseError
>> +from ovs.flows.decoders import decode_default
>> +
>> +
>> +class ListDecoders(object):
>> +    """ListDecoders is used by ListParser to decode the elements in the list.
>> +
>> +    A decoder is a function that accepts a value and returns its decoded
>> +    object.
>> +
>> +    ListDecoders is initialized with a list of of tuples that contains the
> 
> s/of of/of/
> 

Thanks James. I'll fix it in the next version.

On 28 Jan 2022, at 17:04, Adrian Moreno wrote:

> Some openflow or dpif flows encode their arguments in lists, eg:
> "some_action(arg1,arg2,arg3)". In order to decode this in a way that can
> be then stored and queried, add ListParser and ListDecoders classes
> that parse lists into KeyValue instances.
>
> The ListParser / ListDecoders mechanism is quite similar to KVParser and
> KVDecoders. Since the "key" of the different KeyValue objects is now
> ommited, it has to be provided by ListDecoders.
>
> For example, take the openflow action "resubmit" that can be written as:
>
>     resubmit([port],[table][,ct])
>
> Can be decoded by creating a ListDecoders instance such as:
>
>     ListDecoders([
>                   ("port", decode_default),
>                   ("table", decode_int),
>                   ("ct", decode_flag),
>                 ])
>
> Naturally, the order of the decoders must be kept.
>
> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>


Assuming you will fix the double “to to”, you can add my ack to the next version.

Acked-by: Eelco Chaudron <echaudro@redhat.com>