From patchwork Mon Apr 15 16:20:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Terry Wilson X-Patchwork-Id: 1923798 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=TC4nRgra; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VJC8d3pTpz1yY4 for ; Tue, 16 Apr 2024 02:21:03 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C720140443; Mon, 15 Apr 2024 16:21:00 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id HaUMCDUMjfo9; Mon, 15 Apr 2024 16:20:58 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0895840236 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=TC4nRgra Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 0895840236; Mon, 15 Apr 2024 16:20:58 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CECE6C0077; Mon, 15 Apr 2024 16:20:57 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 51403C0037 for ; Mon, 15 Apr 2024 16:20:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4D420605A0 for ; Mon, 15 Apr 2024 16:20:56 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 3zufvoHTew59 for ; Mon, 15 Apr 2024 16:20:55 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=twilson@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 07D1360585 Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 07D1360585 Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=TC4nRgra Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 07D1360585 for ; Mon, 15 Apr 2024 16:20:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1713198053; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0SIF8zABIzog/mvwYQW2GIlNfTucb9wTDEgcsClT94Y=; b=TC4nRgra9VrHmNGF1ay4vS2TRADueHO5BFU59MYlAdY05KUVLj98dmHQQKxKKxp20TSJjT /ClThkq7F16Wtx72WarVU1FsoHfwXZ+66ubC6tNNA/f32TsxeTFBpTLzOMShYHYjCdYtLM 0PC2R3PA+oE2XB1MU9wzkb6yA3ChGKM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-696-aDl2CoPpO92SRkFstjU0NA-1; Mon, 15 Apr 2024 12:20:52 -0400 X-MC-Unique: aDl2CoPpO92SRkFstjU0NA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E936210499A5 for ; Mon, 15 Apr 2024 16:20:51 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.22.32.137]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD0742166B36; Mon, 15 Apr 2024 16:20:51 +0000 (UTC) From: Terry Wilson To: dev@openvswitch.org Date: Mon, 15 Apr 2024 11:20:49 -0500 Message-Id: <20240415162049.1073623-1-twilson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v2] python: ovsdb-idl: Add custom transaction operations. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It can be useful to be able to send raw transaction operations through the Idl's connection. For example, to clean up MAC_Binding entries for floating IPs without having to monitor the MAC_Binding table which can be quite large. Signed-off-by: Terry Wilson --- NEWS | 2 ++ python/ovs/db/idl.py | 18 ++++++++++++++- tests/ovsdb-idl.at | 27 ++++++++++++++++++++++ tests/test-ovsdb.py | 55 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 101 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index b92cec532..f30b859fd 100644 --- a/NEWS +++ b/NEWS @@ -7,6 +7,8 @@ Post-v3.3.0 - The primary development branch has been renamed from 'master' to 'main'. The OVS tree remains hosted on GitHub. https://github.com/openvswitch/ovs.git + - Python: + * Add custom transaction support to the Idl via add_op(). v3.3.0 - 16 Feb 2024 diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py index a80da84e7..1220e4cc6 100644 --- a/python/ovs/db/idl.py +++ b/python/ovs/db/idl.py @@ -1703,6 +1703,8 @@ class Transaction(object): self._inserted_rows = {} # Map from UUID to _InsertedRow + self._operations = [] + def add_comment(self, comment): """Appends 'comment' to the comments that will be passed to the OVSDB server when this transaction is committed. (The comment will be @@ -1838,7 +1840,7 @@ class Transaction(object): "rows": [rows]}) # Add updates. - any_updates = False + any_updates = bool(self._operations) for row in self._txn_rows.values(): if row._changes is None: if row._table.is_root: @@ -1977,6 +1979,7 @@ class Transaction(object): if self.dry_run: operations.append({"op": "abort"}) + operations += self._operations if not any_updates: self._status = Transaction.UNCHANGED else: @@ -1991,6 +1994,19 @@ class Transaction(object): self.__disassemble() return self._status + def add_op(self, op): + """Add a raw OVSDB operation to the transaction + + This can be useful for re-using the existing Idl connection to take + actions that are difficult or expensive to do with the Idl itself. e.g. + deleting a bunch of rows on the server that you don't want to store + in memory. + + :param op: An "op" for an OVSDB "transact" request (rfc 7047 Sec 5.2) + :type op: dict + """ + self._operations.append(op) + def commit_block(self): """Attempts to commit this transaction, blocking until the commit either succeeds or fails. Returns the final commit status, which may diff --git a/tests/ovsdb-idl.at b/tests/ovsdb-idl.at index fb568dd82..487545a13 100644 --- a/tests/ovsdb-idl.at +++ b/tests/ovsdb-idl.at @@ -2758,6 +2758,33 @@ OVSDB_CHECK_IDL_PERS_UUID_INSERT([simple idl, persistent uuid insert], [['This UUID would duplicate a UUID already present within the table or deleted within the same transaction']]) +OVSDB_CHECK_IDL_PY([simple idl, python, add_op], + [], + [['insert 1, insert 2, insert 3, insert 1' \ + 'add_op {"op": "delete", "table": "simple", "where": [["i", "==", 1]]}' \ + 'add_op {"op": "insert", "table": "simple", "row": {"i": 2}}, delete 3' \ + 'insert 2, add_op {"op": "update", "table": "simple", "row": {"i": 1}, "where": [["i", "==", 2]]}' + ]], + [[000: empty +001: commit, status=success +002: table simple: i=1 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<1> +002: table simple: i=1 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<2> +002: table simple: i=2 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<3> +002: table simple: i=3 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<4> +003: commit, status=success +004: table simple: i=2 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<3> +004: table simple: i=3 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<4> +005: commit, status=success +006: table simple: i=2 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<3> +006: table simple: i=2 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<5> +007: commit, status=success +008: table simple: i=1 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<3> +008: table simple: i=1 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<5> +008: table simple: i=1 r=0 b=false s= u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<6> +009: done +]],[],sort) + + m4_define([OVSDB_CHECK_IDL_CHANGE_AWARE], [AT_SETUP([simple idl, database change aware, online conversion - $1]) AT_KEYWORDS([ovsdb server idl db_change_aware conversion $1]) diff --git a/tests/test-ovsdb.py b/tests/test-ovsdb.py index 48f8ee2d7..58829f520 100644 --- a/tests/test-ovsdb.py +++ b/tests/test-ovsdb.py @@ -36,6 +36,53 @@ vlog.set_levels_from_string("console:dbg") vlog.init(None) +OBJ_STR = "_OBJECT_{}_" + + +def substitute_object_text(data, quotechar='"', obj_chars=("{}", "[]")): + obj_chars = dict(obj_chars) + in_quote = False + in_object = [] + removed_text = [] + output = "" + start = end = 0 + for i, s in enumerate(data): + if not in_object: + if not in_quote and s in obj_chars: + in_object.append(s) + start = i + else: + output += s + if s == quotechar: + in_quote = not in_quote + elif not in_quote: # unquoted object + if s == in_object[0]: + in_object.append(s) + elif s == obj_chars[in_object[0]]: + in_object.pop() + if not in_object: + end = i + 1 + removed_text.append(data[start:end]) + output += OBJ_STR.format(len(removed_text) - 1) + return output, removed_text + + +def recover_object_text_from_list(words, json): + if not json: + return words + results = [] + i = 0 + for word in words: + while json: + if OBJ_STR.format(i) not in word: + results.append(word) + break + replacement = json.pop(0) + results.append(word.replace(OBJ_STR.format(i), replacement)) + i += 1 + return results + + def unbox_json(json): if type(json) is list and len(json) == 1: return json[0] @@ -377,8 +424,10 @@ def idl_set(idl, commands, step): increment = False fetch_cmds = [] events = [] + commands, data = substitute_object_text(commands) for command in commands.split(','): words = command.split() + words = recover_object_text_from_list(words, data) name = words[0] args = words[1:] @@ -437,6 +486,12 @@ def idl_set(idl, commands, step): s = txn.insert(idl.tables["simple"], new_uuid=args[0], persist_uuid=True) s.i = int(args[1]) + elif name == "add_op": + if len(args) != 1: + sys.stderr.write('"add_op" command requires 1 argument\n') + sys.stderr.write(f"args={args}\n") + sys.exit(1) + txn.add_op(ovs.json.from_string(args[0])) elif name == "delete": if len(args) != 1: sys.stderr.write('"delete" command requires 1 argument\n')