From patchwork Wed Aug 14 08:39:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michele Baldessari X-Patchwork-Id: 1146846 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=acksyn.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=acksyn.org header.i=@acksyn.org header.b="QPVemNx4"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 467jgd6Wwtz9sDB for ; Wed, 14 Aug 2019 18:39:41 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 3B905AD7; Wed, 14 Aug 2019 08:39:39 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 080D2ACD for ; Wed, 14 Aug 2019 08:39:38 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from palahniuk.acksyn.org (palahniuk.acksyn.org [5.9.7.26]) by smtp1.linuxfoundation.org (Postfix) with ESMTP id 8AEDACF for ; Wed, 14 Aug 2019 08:39:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by palahniuk.acksyn.org (Postfix) with ESMTP id 524832E9B1; Wed, 14 Aug 2019 04:39:35 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=acksyn.org; h= content-transfer-encoding:mime-version:x-mailer:message-id:date :date:subject:subject:from:from:received:received; s=2010; t= 1565771974; bh=u2Vk2X/LLjMBNavx8WbhYH2PUAkcC2oT97sHDs8oEzE=; b=Q PVemNx42zDq93lEzRyTkewfOTKW45MXzylBvsYEps3Wno7oStPFJjppU3t9ymkwP 4q35mo4L67MB3WE2ZMTud/sxrh8hqV2UoqCCcdfwgv1x96YARiAthS3QQk90GadW Ab7si+qB6mM6/PaVsS5LBYBEvO5xdxQWYYALam2CIw= Received: from palahniuk.acksyn.org ([127.0.0.1]) by localhost (mail.acksyn.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id kEKoqDJAx908; Wed, 14 Aug 2019 04:39:34 -0400 (EDT) Received: from localhost (unknown [95.233.105.101]) by palahniuk.acksyn.org (Postfix) with ESMTPSA id 6ADF42E6A8; Wed, 14 Aug 2019 04:39:34 -0400 (EDT) From: Michele Baldessari To: dev@openvswitch.org Date: Wed, 14 Aug 2019 10:39:13 +0200 Message-Id: <20190814083913.9273-1-michele@acksyn.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Michele Baldessari Subject: [ovs-dev] [PATCH] Make pidfile_is_running more robust against empty pidfiles X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org In some of our destructive testing of ovn-dbs inside containers managed by pacemaker we reached a situation where /var/run/openvswitch had empty .pid files. The current code does not deal well with them and pidfile_is_running() returns true in such a case and this confuses the OCF resource agent. - Before this change: Inside a container run: killall ovsdb-server; echo -n '' > /var/run/openvswitch/ovnnb_db.pid; echo -n '' > /var/run/openvswitch/ovnsb_db.pid We will observe that the cluster is unable to ever recover because it believes the ovn processes to be running when they really aren't and eventually just fails: podman container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp15/openstack-ovn-northd:pcmklatest] ovn-dbs-bundle-0 (ocf::ovn:ovndb-servers): Master controller-0 ovn-dbs-bundle-1 (ocf::ovn:ovndb-servers): Stopped controller-1 ovn-dbs-bundle-2 (ocf::ovn:ovndb-servers): Slave controller-2 - After this change the cluster is able to recover from this state and correctly start the resource: podman container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp15/openstack-ovn-northd:pcmklatest] ovn-dbs-bundle-0 (ocf::ovn:ovndb-servers): Master controller-0 ovn-dbs-bundle-1 (ocf::ovn:ovndb-servers): Slave controller-1 ovn-dbs-bundle-2 (ocf::ovn:ovndb-servers): Slave controller-2 Signed-off-by: Michele Baldessari --- ovn/utilities/ovn-ctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ovn/utilities/ovn-ctl b/ovn/utilities/ovn-ctl index 7e5cd469c83c..65f03e28ddba 100755 --- a/ovn/utilities/ovn-ctl +++ b/ovn/utilities/ovn-ctl @@ -35,7 +35,7 @@ ovn_northd_db_conf_file="$etcdir/ovn-northd-db-params.conf" pidfile_is_running () { pidfile=$1 - test -e "$pidfile" && pid=`cat "$pidfile"` && pid_exists "$pid" + test -e "$pidfile" && [ -s "$pidfile" ] && pid=`cat "$pidfile"` && pid_exists "$pid" } >/dev/null 2>&1 stop_nb_ovsdb() {