From patchwork Tue Sep 15 19:21:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Flavio Leitner X-Patchwork-Id: 1364613 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=sysclose.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=sysclose.org header.i=@sysclose.org header.a=rsa-sha256 header.s=201903 header.b=bjZ8C6Tw; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BrY4b0wntz9sTH for ; Wed, 16 Sep 2020 05:21:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id DD1812277A; Tue, 15 Sep 2020 19:21:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zBvLisMNyKW7; Tue, 15 Sep 2020 19:21:30 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 4430F2043A; Tue, 15 Sep 2020 19:21:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3A1EBC0864; Tue, 15 Sep 2020 19:21:30 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11F7CC0864 for ; Tue, 15 Sep 2020 19:21:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 01B3B86A35 for ; Tue, 15 Sep 2020 19:21:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BeXYiqrrWK0u for ; Tue, 15 Sep 2020 19:21:27 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sysclose.org (smtp.sysclose.org [69.164.214.230]) by whitealder.osuosl.org (Postfix) with ESMTPS id C347886A6D for ; Tue, 15 Sep 2020 19:21:27 +0000 (UTC) Received: from localhost (unknown [45.71.105.251]) by sysclose.org (Postfix) with ESMTPSA id E8FC830BD; Tue, 15 Sep 2020 19:22:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 sysclose.org E8FC830BD DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sysclose.org; s=201903; t=1600197766; bh=RP0mOcPv1yVOZhJ1KkYiClUK999KWHc4RhPvuw1c3Is=; h=From:To:Cc:Subject:Date:From; b=bjZ8C6TwVn2WbPi62SpysoyPPnU0cz/Ugq6ZaMQTwEvH/ZVAp5IO0VV3r65wmaTLi qhFUpgxpGxN7uew9pOgfy2nzTY+Fg/k3ewXJ/Q5Ytn72IoTMwI1zLul6f8Dy/WBK9w Y0//uu4Ss+hiMJW6Yo+gnhAq8/hJ9cZ2PbJfnM2IjsvcdQ+yjN2TVhhz9kdCNMKJvW JkYjff9GcfkKvD+wjc9i8mFOZMSQnecGcyo01W6VOV5AF6eFmW4R42eyAHUJRa7LJf Q4pPAuk511m5g/TBOHeCSCuC8E5MpG/CnBcbkIiTp+/XkZW0YPK9BXIX5D18qwARMM JFRwdkLzU6CKQ== From: Flavio Leitner To: dev@openvswitch.org Date: Tue, 15 Sep 2020 16:21:20 -0300 Message-Id: <20200915192120.3543861-1-fbl@sysclose.org> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Cc: Flavio Leitner Subject: [ovs-dev] [PATCH] ofproto-dpif-upcall: Log the emergency flow flush. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When the number of flows in the datapath reaches twice the maximum, revalidators will delete all flows as an emergency action to recover. In that case, log a message with values and increase a coverage counter. Signed-off-by: Flavio Leitner Acked-by: Eelco Chaudron Acked-by: Aaron Conole --- ofproto/ofproto-dpif-upcall.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 05a912f57..12e94287a 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -57,6 +57,7 @@ COVERAGE_DEFINE(upcall_ukey_contention); COVERAGE_DEFINE(upcall_ukey_replace); COVERAGE_DEFINE(revalidate_missed_dp_flow); COVERAGE_DEFINE(upcall_flow_limit_hit); +COVERAGE_DEFINE(upcall_flow_limit_kill); /* A thread that reads upcalls from dpif, forwards each upcall's packet, * and possibly sets up a kernel flow as a cache. */ @@ -2622,6 +2623,7 @@ revalidate(struct revalidator *revalidator) long long int max_idle; long long int now; + size_t kill_all_limit; size_t n_dp_flows; bool kill_them_all; @@ -2649,7 +2651,18 @@ revalidate(struct revalidator *revalidator) COVERAGE_INC(upcall_flow_limit_hit); } - kill_them_all = n_dp_flows > flow_limit * 2; + kill_them_all = false; + kill_all_limit = flow_limit * 2; + if (OVS_UNLIKELY(n_dp_flows > kill_all_limit)) { + static struct vlog_rate_limit rlem = VLOG_RATE_LIMIT_INIT(1, 5); + + VLOG_WARN_RL(&rlem, "Emergency: deleting all flows " + "(now: %"PRIuSIZE", max: %"PRIuSIZE")", + n_dp_flows, kill_all_limit); + COVERAGE_INC(upcall_flow_limit_kill); + kill_them_all = true; + } + max_idle = n_dp_flows > flow_limit ? 100 : ofproto_max_idle; udpif->dpif->current_ms = time_msec();