From patchwork Wed Feb 17 21:09:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yifeng Sun X-Patchwork-Id: 1441383 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=kmhkY2lj; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Dgr7j52tQz9sCD for ; Thu, 18 Feb 2021 08:09:35 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EDE1A605AA for ; Wed, 17 Feb 2021 21:09:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2URBSXKq0nvE for ; Wed, 17 Feb 2021 21:09:32 +0000 (UTC) Received: by smtp3.osuosl.org (Postfix, from userid 1001) id EED10605C9; Wed, 17 Feb 2021 21:09:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 24D05605A3; Wed, 17 Feb 2021 21:09:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D453BC0174; Wed, 17 Feb 2021 21:09:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 99B04C013A for ; Wed, 17 Feb 2021 21:09:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 804B960598 for ; Wed, 17 Feb 2021 21:09:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DM7-jRik5oPj for ; Wed, 17 Feb 2021 21:09:18 +0000 (UTC) Received: by smtp3.osuosl.org (Postfix, from userid 1001) id 9C8E0605A7; Wed, 17 Feb 2021 21:09:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4456960598 for ; Wed, 17 Feb 2021 21:09:16 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id y25so5956490pfp.5 for ; Wed, 17 Feb 2021 13:09:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=dvx+a+dApswWCN5RwQ8dEXiqIcASuMhriKSg/Cdp9bo=; b=kmhkY2ljmXhusPjbIfJq0c93E+/Ss1xr/w5G5Kpe4cc0jLMctWDBnWgywA19sJf+Sa fQf5l2ra1RpBOJe0dJv/Def4jfsMFVx8hT7AePS4D0Tl7JGCqSUtXJhwGl7pUpuNDzoB VF+vNkqyw5JasMERjcr4qM2v3DltXb5fQkRzaVATosAG0SFeNj7OA7NowDHGt1cbehLQ IPcDfQre8HtqdNh+zqOSPshrCHDnMCQQSGP3wlt1H0J31Wozg2JYH2uGu8+nf2w7u8RJ 4cGcQhiL1rBQ5M8xVE1ILfJiDzV1FRmNlsoXvmxW8nKyD0aWHWLkCkHIUcNeZh0WEMu2 qJPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=dvx+a+dApswWCN5RwQ8dEXiqIcASuMhriKSg/Cdp9bo=; b=qa9wZDUH83AiNxSHQ1FLVmt+2YId5aYAc9NR9ogrSp4esr98XFbp4cA3axOElD7g+5 Nj5IJIUNK72QH3oH+QHE285oZR6/WrA7onmWBQPEJA/lR7TVTzQPrbOgIjNcIxZVtwqt +WcNgx1EnYOoqdoek2u7aR3ZBsJqoUv29iJk4Cx2AJriM48PgFKsODyxqf9OiS1yqufm DToMaNIAVxfxfCHFmWp7fRNdgmA97kplea0cpJgPH2alN7RjjFjrZDB7kPvadLh537F+ wF7ZBu31ugwRi/Ayvk4RVKkrg7FZu9ww/gHW+Tg2f4la92e3L02yCpPFI1BnMnqsEzqS lwFw== X-Gm-Message-State: AOAM532bfDOv506GooA1QRLMv+unwRY0eLMQt3twIEjjsQylRUGycDlV oG0qDhPjMufqJO629eq89lKNC8yeBGwmyA== X-Google-Smtp-Source: ABdhPJxApeiIoinVXoHem7Bc2ZfY/AptHIC1GnLMqwFc7L9fbGoSA94RXgPNokOEs9QDDPtVkDJJfw== X-Received: by 2002:a05:6a00:ad1:b029:1ec:c826:55aa with SMTP id c17-20020a056a000ad1b02901ecc82655aamr997638pfl.26.1613596156315; Wed, 17 Feb 2021 13:09:16 -0800 (PST) Received: from kern417.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id b126sm3218896pfg.192.2021.02.17.13.09.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Feb 2021 13:09:15 -0800 (PST) From: Yifeng Sun To: dev@openvswitch.org Date: Wed, 17 Feb 2021 13:09:05 -0800 Message-Id: <1613596145-3110-1-git-send-email-pkusunyifeng@gmail.com> X-Mailer: git-send-email 2.7.4 Subject: [ovs-dev] [PATCHv2] connmgr: Check nullptr inside ofmonitor_report() X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovs-vswitchd could crash under these circumstances: 1. When one bridge is being destroyed, ofproto_destroy() is called and connmgr pointer of its ofproto struct is nullified. This ofproto struct is deallocated through 'ovsrcu_postpone(ofproto_destroy_defer__, p);'. 2. Before RCU enters quiesce state to actually free this ofproto struct, revalidator thread calls udpif_revalidator(), which could handle a learn flow and calls ofproto_flow_mod_learn(), it later calls ofmonitor_report() and ofproto struct's connmgr pointer is accessed. The crash stack trace is shown below: 0 ofmonitor_report (mgr=0x0, rule=rule@entry=0x7fa4ac067c30, event=event@entry=NXFME_ADDED, reason=reason@entry=OFPRR_IDLE_TIMEOUT, abbrev_ofconn=0x0, abbrev_xid=0, old_actions=old_actions@entry=0x0) at ofproto/connmgr.c:2160 1 0x00007fa4d6803495 in add_flow_finish (ofproto=0x55d9075d4ab0, ofm=, req=req@entry=0x0) at ofproto/ofproto.c:5221 2 0x00007fa4d68036af in modify_flows_finish (req=0x0, ofm=0x7fa4980753f0, ofproto=0x55d9075d4ab0) at ofproto/ofproto.c:5823 3 ofproto_flow_mod_finish (ofproto=0x55d9075d4ab0, ofm=ofm@entry=0x7fa4980753f0, req=req@entry=0x0) at ofproto/ofproto.c:8088 4 0x00007fa4d680372d in ofproto_flow_mod_learn_finish (ofm=ofm@entry=0x7fa4980753f0, orig_ofproto=orig_ofproto@entry=0x0) at ofproto/ofproto.c:5439 5 0x00007fa4d68072f9 in ofproto_flow_mod_learn (ofm=0x7fa4980753f0, keep_ref=keep_ref@entry=true, limit=, below_limitp=below_limitp@entry=0x0) at ofproto/ofproto.c:5499 6 0x00007fa4d6835d33 in xlate_push_stats_entry (entry=0x7fa498012448, stats=stats@entry=0x7fa4d2701a10, offloaded=offloaded@entry=false) at ofproto/ofproto-dpif-xlate-cache.c:127 7 0x00007fa4d6835e3a in xlate_push_stats (xcache=, stats=stats@entry=0x7fa4d2701a10, offloaded=offloaded@entry=false) at ofproto/ofproto-dpif-xlate-cache.c:181 8 0x00007fa4d6822046 in revalidate_ukey (udpif=udpif@entry=0x55d90760b240, ukey=ukey@entry=0x7fa4b0191660, stats=stats@entry=0x7fa4d2705118, odp_actions=odp_actions@entry=0x7fa4d2701b50, reval_seq=reval_seq@entry=5655486242, recircs=recircs@entry=0x7fa4d2701b40, offloaded=false) at ofproto/ofproto-dpif-upcall.c:2294 9 0x00007fa4d6825aee in revalidate (revalidator=0x55d90769dd00) at ofproto/ofproto-dpif-upcall.c:2683 10 0x00007fa4d6825cf3 in udpif_revalidator (arg=0x55d90769dd00) at ofproto/ofproto-dpif-upcall.c:936 11 0x00007fa4d6259c9f in ovsthread_wrapper (aux_=) at lib/ovs-thread.c:423 12 0x00007fa4d582cea5 in start_thread () from /usr/lib64/libpthread.so.0 13 0x00007fa4d504b96d in clone () from /usr/lib64/libc.so.6 At the time of crash, the involved ofproto was already deallocated: (gdb) print *ofproto $1 = ..., name = 0x55d907602820 "nsx-managed", ..., ports = {..., one = 0x0, mask = 63, n = 0}, ..., connmgr = 0x0, ... This patch fixes it. VMware-BZ: #2700626 Signed-off-by: Yifeng Sun Acked-by: William Tu < u9012063@gmail.com> --- v1->v2: Add check for ofmonitor_flush, thanks William. ofproto/connmgr.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/ofproto/connmgr.c b/ofproto/connmgr.c index 9c5c633b4171..fa8f6cd0e83a 100644 --- a/ofproto/connmgr.c +++ b/ofproto/connmgr.c @@ -2140,7 +2140,7 @@ ofmonitor_report(struct connmgr *mgr, struct rule *rule, const struct rule_actions *old_actions) OVS_REQUIRES(ofproto_mutex) { - if (rule_is_hidden(rule)) { + if (!mgr || rule_is_hidden(rule)) { return; } @@ -2244,6 +2244,10 @@ ofmonitor_flush(struct connmgr *mgr) { struct ofconn *ofconn; + if (!mgr) { + return; + } + LIST_FOR_EACH (ofconn, connmgr_node, &mgr->conns) { struct rconn_packet_counter *counter = ofconn->monitor_counter;