From patchwork Thu Feb 22 15:54:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrian Moreno X-Patchwork-Id: 1902839 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bIb0PqPS; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Tgd4R0mJ6z23d2 for ; Fri, 23 Feb 2024 02:54:30 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 31F734193E; Thu, 22 Feb 2024 15:54:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ynrGWjWvzQii; Thu, 22 Feb 2024 15:54:26 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 54CC54029F Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bIb0PqPS Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 54CC54029F; Thu, 22 Feb 2024 15:54:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 320CAC0077; Thu, 22 Feb 2024 15:54:26 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 82DACC0037 for ; Thu, 22 Feb 2024 15:54:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6C93D8308A for ; Thu, 22 Feb 2024 15:54:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bdpuCZG1yQWE for ; Thu, 22 Feb 2024 15:54:23 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amorenoz@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 1B3F782FBE Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 1B3F782FBE Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bIb0PqPS Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1B3F782FBE for ; Thu, 22 Feb 2024 15:54:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708617261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=12doHCYIWuCFslaLRPybmsJoByiYfaFnnlOMcWf0at8=; b=bIb0PqPSGIs4J57M1dzMw3NcpUzsoo+MsKEqXgg+7/mMjlsbBTkI/ggXbWZjHJau/5aita hngUKcX7izWEXMcU4dKzwQ2i1hqEZDEas7+mbGe8ZCl6/8cjGeqJeQAvsF5JEcZqNzo1es 4VOk3Dfiun6NlCMQQe0i0qbYItg1mb4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-571-_IGu0yFzMV-Ko7vIWWzIVg-1; Thu, 22 Feb 2024 10:54:18 -0500 X-MC-Unique: _IGu0yFzMV-Ko7vIWWzIVg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3DF8D85A58B; Thu, 22 Feb 2024 15:54:18 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.192.43]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47230C03380; Thu, 22 Feb 2024 15:54:17 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Thu, 22 Feb 2024 16:54:14 +0100 Message-ID: <20240222155415.273552-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3] bond: Reset stats when deleting post recirc rule. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In order to properly balance bond traffic, ofproto/bond periodically reads usage statistics of the post-recirculation rules (which are added to a hidden internal table). To do that, each "struct bond_entry" (which represents a hash within a bond) stores the last seen statistics for its rule. When a hash is moved to another member (due to a bond rebalance or the previous member going down), the rule is typically just modified, i.e: same match different actions. In this case, statistics are preserved and accounting continues to work. However, if the rule gets completely deleted (e.g: when all bond members go down) and then re-created, the new rule will have 0 tx_bytes but its associated entry will still store a non-zero last-seen value. This situation leads to an overflow of the delta calculation (computed as [current_stats_value - last_seen_value]), which can affect traffic as the hash will be considered to carry a lot of traffic and rebalancing will kick in. In order to fix this situation, reset the value of last seen statistics on rule deletion. Implementation notes: Modifying pr_tx_bytes requires write-locking the global rwlock but a lockless version of update_recirc_rules was being maintained to avoid locking on bon_unref(). Considering the small impact of locking during bond removal, removing the lockless version and relying on clang's thread safety analysis is preferred. Also, folding Ilya's [1], i.e: fixing thread safety annotation in update_recirc_rules() to require holding write-lock. [1] https://patchwork.ozlabs.org/project/openvswitch/patch/20240209161718.1149494-1-i.maximets@ovn.org/ Reported-at: https://github.com/openvswitch/ovs-issues/issues/319 Co-authored-by: Ilya Maximets Signed-off-by: Ilya Maximets Signed-off-by: Adrian Moreno --- ofproto/bond.c | 33 +++++++++++++++------------------ tests/ofproto-dpif.at | 16 ++++++++++++++++ 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/ofproto/bond.c b/ofproto/bond.c index cfdf44f85..84f59335f 100644 --- a/ofproto/bond.c +++ b/ofproto/bond.c @@ -186,7 +186,7 @@ static struct bond_member *choose_output_member(const struct bond *, struct flow_wildcards *, uint16_t vlan) OVS_REQ_RDLOCK(rwlock); -static void update_recirc_rules__(struct bond *); +static void update_recirc_rules(struct bond *) OVS_REQ_WRLOCK(rwlock); static bool bond_may_recirc(const struct bond *); static void bond_update_post_recirc_rules__(struct bond *, bool force) OVS_REQ_WRLOCK(rwlock); @@ -299,7 +299,10 @@ bond_unref(struct bond *bond) } free(bond->hash); bond->hash = NULL; - update_recirc_rules__(bond); + + ovs_rwlock_wrlock(&rwlock); + update_recirc_rules(bond); + ovs_rwlock_unlock(&rwlock); hmap_destroy(&bond->pr_rule_ops); free(bond->primary); @@ -331,17 +334,8 @@ add_pr_rule(struct bond *bond, const struct match *match, hmap_insert(&bond->pr_rule_ops, &pr_op->hmap_node, hash); } -/* This function should almost never be called directly. - * 'update_recirc_rules()' should be called instead. Since - * this function modifies 'bond->pr_rule_ops', it is only - * safe when 'rwlock' is held. - * - * However, when the 'bond' is the only reference in the system, - * calling this function avoid acquiring lock only to satisfy - * lock annotation. Currently, only 'bond_unref()' calls - * this function directly. */ static void -update_recirc_rules__(struct bond *bond) +update_recirc_rules(struct bond *bond) OVS_REQ_WRLOCK(rwlock) { struct match match; struct bond_pr_rule_op *pr_op; @@ -407,6 +401,15 @@ update_recirc_rules__(struct bond *bond) VLOG_ERR("failed to remove post recirculation flow %s", err_s); free(err_s); + } else if (bond->hash) { + /* If the flow deletion failed, a subsecuent call to + * ofproto_dpif_add_internal_flow() would just modify the flow + * preserving its statistics. Therefore, only reset the entry's + * byte counter if it succeeds. */ + uint32_t hash = pr_op->match.flow.dp_hash & BOND_MASK; + struct bond_entry *entry = &bond->hash[hash]; + + entry->pr_tx_bytes = 0; } hmap_remove(&bond->pr_rule_ops, &pr_op->hmap_node); @@ -421,12 +424,6 @@ update_recirc_rules__(struct bond *bond) ofpbuf_uninit(&ofpacts); } -static void -update_recirc_rules(struct bond *bond) - OVS_REQ_RDLOCK(rwlock) -{ - update_recirc_rules__(bond); -} /* Updates 'bond''s overall configuration to 's'. * diff --git a/tests/ofproto-dpif.at b/tests/ofproto-dpif.at index e305e7b9c..71737db8c 100644 --- a/tests/ofproto-dpif.at +++ b/tests/ofproto-dpif.at @@ -547,6 +547,22 @@ ovs-appctl time/warp 1000 100 ovs-appctl bond/show > bond3.txt AT_CHECK([sed -n '/member p2/,/^$/p' bond3.txt | grep 'hash'], [0], [ignore]) +AT_CHECK([ovs-appctl netdev-dummy/set-admin-state p1 down], 0, [OK +]) +AT_CHECK([ovs-appctl netdev-dummy/set-admin-state p2 down], 0, [OK +]) +ovs-appctl time/warp 1000 100 +AT_CHECK([ovs-appctl netdev-dummy/set-admin-state p1 up], 0, [OK +]) +AT_CHECK([ovs-appctl netdev-dummy/set-admin-state p2 up], 0, [OK +]) +ovs-appctl time/warp 1000 100 + +AT_CHECK([SEND_TCP_BOND_PKTS([p5], [5], [65500])]) +# We sent 49125 KB of data total in 3 batches. No hash should have more +# than that amount of load. Just checking that it is within 5 digits. +AT_CHECK([ovs-appctl bond/show | grep -E '[[0-9]]{6}'], [1]) + OVS_VSWITCHD_STOP() AT_CLEANUP