From patchwork Fri May 17 00:27:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Sharma X-Patchwork-Id: 1100790 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=nutanix.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=nutanix.com header.i=@nutanix.com header.b="GjyWSJCJ"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 454pyz527hz9s4Y for ; Fri, 17 May 2019 10:27:38 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id DFE01A95; Fri, 17 May 2019 00:27:33 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 2F560949 for ; Fri, 17 May 2019 00:27:33 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx0a-002c1b01.pphosted.com (mx0a-002c1b01.pphosted.com [148.163.151.68]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 24B025E4 for ; Fri, 17 May 2019 00:27:31 +0000 (UTC) Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4H0Jnnu002119; Thu, 16 May 2019 17:27:31 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h=from : to : cc : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=proofpoint20171006; bh=x4DsbI69QOsLuLBAXpajoryai8UnsdTeFK91+Moykj0=; b=GjyWSJCJEfUPw3QVhQLUvcSeldOKjoAq92tDGJo6yCWtKl7/cAbm9r+MVFRhuZsME2WA 98Aa4jRlNcIrIbhArxmKAZP1LtvQgnFb+SZknCZKStDd1upHI8sK3B3LTMR3LOOwRR1V rRZPXWrKYEcDREq6TkMbhFVCQxkO0iPFNOMttwi6EthIMDcYtwcu4552sgkyPKKnBSj2 LWoXUxnzkp6SfeYorxWHgy9agiyf0AemPj9JNo3UJk6JBmmL39vEa2YhvtmierMi8Kzf ZbYK78lYpScp2THC57fF4d52JjJcuZHyay3ScfLysTVE+8omhGu+CAyygSzMf2EreQm2 9g== Received: from nam05-co1-obe.outbound.protection.outlook.com (mail-co1nam05lp2051.outbound.protection.outlook.com [104.47.48.51]) by mx0a-002c1b01.pphosted.com with ESMTP id 2sgw6ca5wk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 16 May 2019 17:27:31 -0700 Received: from MW2PR02MB3899.namprd02.prod.outlook.com (52.132.178.28) by MW2PR02MB3689.namprd02.prod.outlook.com (52.132.177.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1900.16; Fri, 17 May 2019 00:27:29 +0000 Received: from MW2PR02MB3899.namprd02.prod.outlook.com ([fe80::908f:c38c:a9ee:651c]) by MW2PR02MB3899.namprd02.prod.outlook.com ([fe80::908f:c38c:a9ee:651c%2]) with mapi id 15.20.1878.024; Fri, 17 May 2019 00:27:28 +0000 From: Ankur Sharma To: "ovs-dev@openvswitch.org" Thread-Topic: [PATCH v2] OVN: Fix the ovn-controller 100% usage issue with put_mac_bindings Thread-Index: AQHVDEdPeSSRZZUNO0e5BlbKchSICQ== Date: Fri, 17 May 2019 00:27:28 +0000 Message-ID: <1558052945-95192-1-git-send-email-ankur.sharma@nutanix.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR08CA0016.namprd08.prod.outlook.com (2603:10b6:a03:100::29) To MW2PR02MB3899.namprd02.prod.outlook.com (2603:10b6:907:4::28) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 1.8.3.1 x-originating-ip: [192.146.154.1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: eb12178d-1cf3-4769-e393-08d6da5e7155 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(7193020); SRVR:MW2PR02MB3689; x-ms-traffictypediagnostic: MW2PR02MB3689: x-proofpoint-crosstenant: true x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0040126723 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(346002)(366004)(376002)(396003)(136003)(39860400002)(199004)(189003)(7736002)(66556008)(64756008)(66446008)(8676002)(81156014)(66476007)(66946007)(305945005)(102836004)(2616005)(81166006)(476003)(36756003)(66066001)(186003)(6116002)(68736007)(3846002)(386003)(6506007)(4326008)(25786009)(256004)(26005)(486006)(6916009)(44832011)(71200400001)(71190400001)(52116002)(6486002)(6512007)(2501003)(50226002)(99286004)(73956011)(316002)(54906003)(5640700003)(53936002)(6436002)(4720700003)(14454004)(66574012)(5660300002)(8936002)(2351001)(2906002)(86362001)(478600001)(64030200001); DIR:OUT; SFP:1102; SCL:1; SRVR:MW2PR02MB3689; H:MW2PR02MB3899.namprd02.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: nutanix.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: kyL2JsK8WJ20ltORi2t7DGBx3n+V9s47ronCqLU2OZlcqM5uhoM4t2doGk7bUTVAtSmsAn9EkDmWThBK6Pl8gIkud3JUYoyPv3s03Hr3bRShnr23aI5ewEOm+mYcgNNBcz0oNCDFm+j4aH92pT0HyQ1Cu9WQbLQCy0b6YkUSGzlLHt8rYXA2+MgBkm8kgTiZw7WZfvM+Tu6G9wK79TSRYZMjjzU1fOZWD/3djeN8sVmG1m4jfmKhNruS3ApnrHe+6PEQVH9egngcqFN+tdClreGXVXPUKKDaVliFYuO0JXNXJj29zoz1bT3O8nvKeb9lYWgVLNvhhmkqj7s3tKZ0sESS32q0XqSx+7be8RSmvQVqrFou/h/uKXaP4w2+lTXvCBXBxrTd5OpH+NTasBNcmP0wDsvs4sLXhWitEyLw8BA= MIME-Version: 1.0 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: eb12178d-1cf3-4769-e393-08d6da5e7155 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 May 2019 00:27:28.8102 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR02MB3689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-05-16_19:, , signatures=0 X-Proofpoint-Spam-Reason: safe X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2] OVN: Fix the ovn-controller 100% usage issue with put_mac_bindings X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org ISSUE: a. As soon as entries get added to put_mac_bindings in pinctrl.c, ovn-controller CPU consumption reached 100%. b. This happens because in wait_put_mac_bindings, if !hmap_is_empty(&put_mac_bindings) succeeds and calls poll_immediat_wake(). ovn-controller.log: "2019-05-10T19:43:28.035Z|00035|poll_loop|INFO|wakeup due to 0-ms timeout at ovn/controller/pinctrl.c:2520 (99% CPU usage)" ROOT_CAUSE: a. Earlier it used to work fine, because in run_put_mac_bindings all the bindings in put_mac_bindings would be flushed. b. As a part of adding a new thread in pinctrl, this line got replaced with calling buffer_put_mac_bindings. " . . . /* Move the mac bindings from 'put_mac_bindings' hmap to * 'buffered_mac_bindings' and notify the pinctrl_handler. * pinctrl_handler will reinject the buffered packets. */ if (!hmap_is_empty(&put_mac_bindings)) { buffer_put_mac_bindings(); notify_pinctrl_handler(); } " And buffer_put_mac_binding would pop the bindings from put_mac_bindings, thereby emptying it. c. However, 1c24b2f490ba002bbfeb23006965188a7c5b9747 changed the buffer dequeueing logic and in the process removed buffer_put_mac_binding, as a result put_mac_bindings would never get empty. FIX: a. Added call to flush_put_mac_bindings back in run_put_mac_bindings. b. Additionally, updated the documentation in pinctrl.c to reflect the new buffer dequeueing logic added by 1c24b2f490ba002bbfeb23006965188a7c5b9747. Signed-off-by: Ankur Sharma Reported-by: Ankur Sharma CC: Lorenzo Bianconi Fixes: 1c24b2f490ba ("OVN: fix pinctrl ip buffering for gw router port") --- ovn/controller/pinctrl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/ovn/controller/pinctrl.c b/ovn/controller/pinctrl.c index 8ae1f9b..b7bb4c9 100644 --- a/ovn/controller/pinctrl.c +++ b/ovn/controller/pinctrl.c @@ -91,11 +91,13 @@ VLOG_DEFINE_THIS_MODULE(pinctrl); * * - arp/nd_ns - These actions generate an ARP/IPv6 Neighbor solicit * requests. The original packets are buffered and - * injected back when put_arp/put_nd actions are called. + * injected back when put_arp/put_nd resolves + * corresponding ARP/IPv6 Neighbor solicit requests. * When pinctrl_run(), writes the mac bindings from the * 'put_mac_bindings' hmap to the MAC_Binding table in - * SB DB, it moves these mac bindings to another hmap - - * 'buffered_mac_bindings'. + * SB DB, run_buffered_binding will add the buffered + * packets to buffered_mac_bindings and notify + * pinctrl_handler. * * The pinctrl_handler thread calls the function - * send_mac_binding_buffered_pkts(), which uses @@ -2468,6 +2470,7 @@ run_put_mac_bindings(struct ovsdb_idl_txn *ovnsb_idl_txn, sbrec_mac_binding_by_lport_ip, pmb); } + flush_put_mac_bindings(); } static void