From patchwork Fri Jun 1 19:07:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: aginwala aginwala X-Patchwork-Id: 924734 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Jy5jyH33"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40zRNS4jYyz9s0w for ; Mon, 4 Jun 2018 04:26:29 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 5ABAAE67; Sun, 3 Jun 2018 18:18:00 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id A4B24D80 for ; Sun, 3 Jun 2018 18:24:14 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg0-f52.google.com (mail-pg0-f52.google.com [74.125.83.52]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 0B81F75E for ; Sun, 3 Jun 2018 18:24:13 +0000 (UTC) Received: by mail-pg0-f52.google.com with SMTP id p9-v6so13398234pgc.9 for ; Sun, 03 Jun 2018 11:24:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=Ii8DwXJphOjys3agcS6rGda+ATumgWnpm8pQ6XZiha0=; b=Jy5jyH33QZlrnmtPNyDmujZnPsd0sVyKyeCgEAw1CUvQLfZm0gjOYKfXMgwAtCpxis zx1VhZzpsLOjPuPfgcODPR/rjVgaX034IeUgquMxwljdj1BYNqfgDohmNhTrszCy24Y5 EK4n3vK/eySq7u+UWmGfUoszBiCkw0mDk3EnrU43S5nmPY2cy12vmk9iZFRvIARoQsst UQGr6A/vkTkIWpelpPMoA//8RccPDqgLxiF8aK9DDTjFsdxuLRUTm9blEUNQSszfINd5 JuiQvhE2RoEcLkkbain+zU0GDTIOZBUC82gqbII1GQ4K0qNdR8bJUJRSnfLboMJJkl8k ZxNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=Ii8DwXJphOjys3agcS6rGda+ATumgWnpm8pQ6XZiha0=; b=uXmpsZYN7gQJfEV2Y6l1xLqVwAG93Ut7B/vGzRe9phtLqdeR62eAAfsGAAdHzGalYZ JCQACqviDR5o7GRYU0T/mfM7WuEtSGZ0mgGt+8P/w9U5iMGGyKGmGl/qK/pNbX5dTj3s pMYonm3L5M1kxy5h5tcTxs0/pJnSBmNHK6+6zncv9qKBhfg04Rk8caOS9TzHwBAdJ9l5 7C2wQrCPi3pKI422oknTYxfbHNWuDp7qIbGWhK+0NYEuGHYVMnolXJx8ItHzGXwbZLD4 KeZR8/nIE+wflOfCpzA9seF/mbDqpPTXjP6bkpeQGch8U3OLcOrCI1K6XHvDREtgpal7 T6tQ== X-Gm-Message-State: APt69E2Y/yNFAHJqonU9rUy1L3qT3Jrg8PJPm5GzVrruC4mvpYuoqRKB ubbSjN8hpQnCqhWBep7AScrV2A== X-Google-Smtp-Source: ADUXVKI4EW5AW7U+tpgnylQUphpMMoN/pfEGMreSXGF22YIlpx3XM2lV8/XMZPgU3zjodwn1p3Xbtw== X-Received: by 2002:a62:a104:: with SMTP id b4-v6mr4328442pff.159.1528050253321; Sun, 03 Jun 2018 11:24:13 -0700 (PDT) Received: from ubuntu.hsd1.ca.comcast.net (c-73-162-228-254.hsd1.ca.comcast.net. [73.162.228.254]) by smtp.gmail.com with ESMTPSA id h65-v6sm104614409pfj.54.2018.06.03.11.24.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 03 Jun 2018 11:24:12 -0700 (PDT) From: aginwala X-Google-Original-From: aginwala To: dev@openvswitch.org Date: Fri, 1 Jun 2018 12:07:20 -0700 Message-Id: <1527880040-12555-1-git-send-email-aginwala@ebay.com> X-Mailer: git-send-email 1.9.1 X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_00, DATE_IN_PAST_24_48, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: aginwala Subject: [ovs-dev] [PATCH v4] ovndb-servers.ocf: add LB support for managing ovndb cluster: X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org using pacemaker so that controllers can be placed in different fault domains. More background about the discussions can be found on: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-May/046770.html Signed-off-by: aginwala --- Documentation/topics/integration.rst | 34 +++++++++++++--- ovn/utilities/ovndb-servers.ocf | 76 +++++++++++++++++++++++++++--------- 2 files changed, 86 insertions(+), 24 deletions(-) diff --git a/Documentation/topics/integration.rst b/Documentation/topics/integration.rst index 0447faf..5d2d3e4 100644 --- a/Documentation/topics/integration.rst +++ b/Documentation/topics/integration.rst @@ -243,12 +243,14 @@ node at which the active server is run, it is not efficient to instruct all the ovn-controllers and the ovn-northd to listen to the latest active server's ip-address. -This problem can be solved by using a native ocf resource agent -``ocf:heartbeat:IPaddr2``. The IPAddr2 resource agent is just a resource with -an ip-address. When we colocate this resource with the active server, pacemaker -will enable the active server to be connected with a single ip-address all the -time. This is the ip-address that needs to be given as the parameter while -creating the `ovndb_servers` resource. +This problem can be solved by two ways: + +1. By using a native ocf resource agent ``ocf:heartbeat:IPaddr2``. +The IPAddr2 resource agent is just a resource with an ip-address. When we +colocate this resource with the active server, pacemaker will enable the +active server to be connected with a single ip-address all the time. This is +the ip-address that needs to be given as the parameter while creating the +`ovndb_servers` resource. Use the following command to create the IPAddr2 resource and colocate it with the active server:: @@ -258,3 +260,23 @@ with the active server:: $ pcs constraint order promote ovndb_servers-master then VirtualIP $ pcs constraint colocation add VirtualIP with master ovndb_servers-master \ score=INFINITY + + +2. Using load balancer vip ip as a master_ip. +In order to use this feature, one needs to use listen_on_master_ip_only to no. +Current code for load balancer have been tested to work with tcp protocol +and needs to be tested/enchanced for ssl. Using load balancer, standby nodes +will not listen on nb and sb db ports so that load balancer will always +communicate to the active node and all the traffic will be sent to active node only. +Standby will continue to sync using LB VIP IP in this case. + +Use the following command to create pcs resource using LB VIP IP:: + + $ pcs resource create ovndb_servers ocf:ovn:ovndb-servers \ + master_ip="" \ + listen_on_master_ip_only="no" \ + ovn_ctl= \ + op monitor interval="10s" \ + op monitor role=Master interval="15s" + $ pcs resource master ovndb_servers-master ovndb_servers \ + meta notify="true" diff --git a/ovn/utilities/ovndb-servers.ocf b/ovn/utilities/ovndb-servers.ocf index 23dc700..c60ad4f 100755 --- a/ovn/utilities/ovndb-servers.ocf +++ b/ovn/utilities/ovndb-servers.ocf @@ -9,6 +9,7 @@ : ${SB_MASTER_PROTO_DEFAULT="tcp"} : ${MANAGE_NORTHD_DEFAULT="no"} : ${INACTIVE_PROBE_DEFAULT="5000"} +: ${LISTEN_ON_MASTER_IP_ONLY_DEFAULT="yes"} CRM_MASTER="${HA_SBIN_DIR}/crm_master -l reboot" CRM_ATTR_REPL_INFO="${HA_SBIN_DIR}/crm_attribute --type crm_config --name OVN_REPL_INFO -s ovn_ovsdb_master_server" @@ -21,6 +22,10 @@ SB_MASTER_PROTO=${OCF_RESKEY_sb_master_protocol:-${SB_MASTER_PROTO_DEFAULT}} MANAGE_NORTHD=${OCF_RESKEY_manage_northd:-${MANAGE_NORTHD_DEFAULT}} INACTIVE_PROBE=${OCF_RESKEY_inactive_probe_interval:-${INACTIVE_PROBE_DEFAULT}} +# In order for pacemaker to work with LB, we can set LISTEN_ON_MASTER_IP_ONLY +# to false and pass LB vip IP while creating pcs resource. +LISTEN_ON_MASTER_IP_ONLY=${OCF_RESKEY_listen_on_master_ip_only:-${LISTEN_ON_MASTER_IP_ONLY_DEFAULT}} + # Invalid IP address is an address that can never exist in the network, as # mentioned in rfc-5737. The ovsdb servers connects to this IP address till # a master is promoted and the IPAddr2 resource is started. @@ -117,6 +122,16 @@ ovsdb_server_metadata() { + + + If set to yes, the OVNDBs will listen on master IP. Otherwise, it will + listen on 0.0.0.0. Set to yes when using pacemaker managed vip resource + as MASTER_IP; set to no when using external LB VIP. + + Listen on master IP or 0.0.0.0 + + + @@ -157,22 +172,25 @@ ovsdb_server_notify() { ${OVN_CTL} --ovn-manage-ovsdb=no start_northd fi - conn=`ovn-nbctl get NB_global . connections` - if [ "$conn" == "[]" ] - then - ovn-nbctl -- --id=@conn_uuid create Connection \ + # Not needed while listening on 0.0.0.0 as we do not want to allow + # local binds. However, it is needed if vip ip is binded to nodes. + if [ "x${LISTEN_ON_MASTER_IP_ONLY}" = xyes ]; then + conn=`ovn-nbctl get NB_global . connections` + if [ "$conn" == "[]" ] + then + ovn-nbctl -- --id=@conn_uuid create Connection \ target="p${NB_MASTER_PROTO}\:${NB_MASTER_PORT}\:${MASTER_IP}" \ inactivity_probe=$INACTIVE_PROBE -- set NB_Global . connections=@conn_uuid - fi + fi - conn=`ovn-sbctl get SB_global . connections` - if [ "$conn" == "[]" ] - then - ovn-sbctl -- --id=@conn_uuid create Connection \ + conn=`ovn-sbctl get SB_global . connections` + if [ "$conn" == "[]" ] + then + ovn-sbctl -- --id=@conn_uuid create Connection \ target="p${SB_MASTER_PROTO}\:${SB_MASTER_PORT}\:${MASTER_IP}" \ inactivity_probe=$INACTIVE_PROBE -- set SB_Global . connections=@conn_uuid + fi fi - else if [ "$MANAGE_NORTHD" = "yes" ]; then # Stop ovn-northd service. Set --ovn-manage-ovsdb=no so that @@ -295,15 +313,13 @@ ovsdb_server_start() { set ${OVN_CTL} - set $@ --db-nb-addr=${MASTER_IP} --db-nb-port=${NB_MASTER_PORT} - set $@ --db-sb-addr=${MASTER_IP} --db-sb-port=${SB_MASTER_PORT} - - if [ "x${NB_MASTER_PROTO}" = xtcp ]; then - set $@ --db-nb-create-insecure-remote=yes - fi + if [ "x${LISTEN_ON_MASTER_IP_ONLY}" = xno ]; then + set $@ --db-nb-port=${NB_MASTER_PORT} + set $@ --db-sb-port=${SB_MASTER_PORT} - if [ "x${SB_MASTER_PROTO}" = xtcp ]; then - set $@ --db-sb-create-insecure-remote=yes + else + set $@ --db-nb-addr=${MASTER_IP} --db-nb-port=${NB_MASTER_PORT} + set $@ --db-sb-addr=${MASTER_IP} --db-sb-port=${SB_MASTER_PORT} fi if [ "x${present_master}" = x ]; then @@ -313,15 +329,33 @@ ovsdb_server_start() { # Force all copies to come up as slaves by pointing them into # space and let pacemaker pick one to promote: # + if [ "x${NB_MASTER_PROTO}" = xtcp ]; then + set $@ --db-nb-create-insecure-remote=yes + fi + + if [ "x${SB_MASTER_PROTO}" = xtcp ]; then + set $@ --db-sb-create-insecure-remote=yes + fi set $@ --db-nb-sync-from-addr=${INVALID_IP_ADDRESS} --db-sb-sync-from-addr=${INVALID_IP_ADDRESS} elif [ ${present_master} != ${host_name} ]; then + # TODO: for using LB vip, need to test for ssl. + if [ "x${LISTEN_ON_MASTER_IP_ONLY}" = xyes ]; then + if [ "x${NB_MASTER_PROTO}" = xtcp ]; then + set $@ --db-nb-create-insecure-remote=yes + fi + + if [ "x${SB_MASTER_PROTO}" = xtcp ]; then + set $@ --db-sb-create-insecure-remote=yes + fi + fi # An existing master is active, connect to it set $@ --db-nb-sync-from-addr=${MASTER_IP} --db-sb-sync-from-addr=${MASTER_IP} set $@ --db-nb-sync-from-port=${NB_MASTER_PORT} set $@ --db-nb-sync-from-proto=${NB_MASTER_PROTO} set $@ --db-sb-sync-from-port=${SB_MASTER_PORT} set $@ --db-sb-sync-from-proto=${SB_MASTER_PROTO} + fi $@ start_ovsdb @@ -416,6 +450,12 @@ ovsdb_server_promote() { ;; esac + # Restart ovs so that new master can listen on tcp port. + # TODO make sure to do more testing as unifying ovs restart both LB and + # pacemaker VIP resource do not break existing pacemkaer vip functionality + # for managing ovndbs. + ${OVN_CTL} stop_ovsdb + ovsdb_server_start ${OVN_CTL} promote_ovnnb ${OVN_CTL} promote_ovnsb