From patchwork Thu May 27 10:29:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianbo Liu X-Patchwork-Id: 1484544 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=LKRNeFj2; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FrPFq27G6z9sT6 for ; Thu, 27 May 2021 20:29:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0993D608A9; Thu, 27 May 2021 10:29:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 50sZfhRkE8wr; Thu, 27 May 2021 10:29:48 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 3691A607DE; Thu, 27 May 2021 10:29:47 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F1171C000E; Thu, 27 May 2021 10:29:46 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 57022C0001 for ; Thu, 27 May 2021 10:29:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3D7FB82E9C for ; Thu, 27 May 2021 10:29:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=nvidia.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BESO2QxgTS4u for ; Thu, 27 May 2021 10:29:43 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2076.outbound.protection.outlook.com [40.107.244.76]) by smtp1.osuosl.org (Postfix) with ESMTPS id C56BE82BC5 for ; Thu, 27 May 2021 10:29:43 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nyFmHJr46EJoQq7tuTaTFfYc115dJ5dbn7jvCi8ZinMBnwCdqQhxDRYjd4/9jzkkHjky8tDqSlx9EjW9agZRfg18VyancvAxjINTxQ3g7Yirde8f9XV+KWpbtzEQYaz14FIYNuNKMIFTC0saR8fvrmWYoKhnMfK7AzxorrsCzA+n38qtEQCPVxWRRInh3MMQm8uguerSEO3IlBdWm+rhgj7tEcwB+loi4lda3/Cvle5cpxtXak0WuzDtCwO9BRF9R61G4iiN165uibOXgUAu3e9r09c/nxoq8ljmRwGtJXeRYUUSPhuLmKaG+hkVsq2e685Kvn4dDQj0cbVxj4WXPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iBYfVmeaQXURJTBr8ZS99gpWJKKdRtEVkFtW2M+8LIU=; b=GZRNLDNU1LzrXFxPjOf6sOF0vndEtkVIAeMOqs7vA1Uc173tNu49cBU/cvbGB63TYY5h6hzUNxyoXuGM2yLE3jWESDGCNWE/5gIdjzZf7DilWr5ItWKSpUXnYB3HHS/HVTuEW2iTFY7IcQRegj6gwArzS8RskGkfABYBdyzGMrBeWgVKCjjHt1kWCBXWVnxf+Lk847xBDDEI7123dPUSfhf9I/cmLumaiH81oPXT7nOvP6dEcvDrj+4yYg7uGCIiqxBWJDXPsPPNaY7d7f/EUyBA4YDCr+X9YeipmfvXJKJtTi2abWYI9TaReHupXJSctN8Hh8PfIJoL5EyE5YT1Wg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=netronome.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iBYfVmeaQXURJTBr8ZS99gpWJKKdRtEVkFtW2M+8LIU=; b=LKRNeFj2csslWHCTswxDrFpUdPUs3kP4HD/48WvTiy7u7n3yWLGGv2jXP6aJLOUEfuDNhePiBXgdHgY/VHiVpEYnDQS5/m9Bp3nED6cbN456+2qUGZhUt4pYi0C42gnBmlk57efIoAlmLxRMZOh/x9bcpGREa2n2jXShCK7l9X9p3Yd+QpHz2EB6HvHOfuITuORS4KsVvVDfZKrii+GsROjnyFG2U96vpGvut7xE+Qkzk0PvJmaHwsUMXzPCL/X/MWWn4N9UisA3uD4Fu0Qqxy1JmmjOMd507JA/1exu9Ox+EoU2cwhJqqDwLaJf9dRTcElwyADuPJ2qQ2qrNknW/A== Received: from BN6PR16CA0025.namprd16.prod.outlook.com (2603:10b6:405:14::11) by DM5PR12MB1849.namprd12.prod.outlook.com (2603:10b6:3:107::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4150.25; Thu, 27 May 2021 10:29:42 +0000 Received: from BN8NAM11FT030.eop-nam11.prod.protection.outlook.com (2603:10b6:405:14:cafe::eb) by BN6PR16CA0025.outlook.office365.com (2603:10b6:405:14::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.20 via Frontend Transport; Thu, 27 May 2021 10:29:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; netronome.com; dkim=none (message not signed) header.d=none;netronome.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by BN8NAM11FT030.mail.protection.outlook.com (10.13.177.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4129.25 via Frontend Transport; Thu, 27 May 2021 10:29:41 +0000 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 27 May 2021 03:29:38 -0700 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.18) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 27 May 2021 10:29:36 +0000 From: Jianbo Liu To: Date: Thu, 27 May 2021 10:29:14 +0000 Message-ID: <20210527102914.27175-1-jianbol@nvidia.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5073ba5b-6ee1-44be-1f75-08d920fa56cc X-MS-TrafficTypeDiagnostic: DM5PR12MB1849: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wNVJbsi2INdte3R+qt5gXbK0MDcblFXlewaVVH58hq1zbd337SnyTo0dTGH86oXzT5tO2CFeo/vjXcLM64osF0GjCg2Pze49CLfNEupCU3YEYcUzTowsGWpdvVW33u3PiKW4d8UO2QVshUQ3byLA/C5XRtOLqotoxxf5+qgGhpmtiY34XEoJpxKNhgCw4DKyrqz8Yn1cKFlIlAaA9nU2H/b1VixIXORLZr2d0WxFZAvvVvzQClEuCk7PdXZSYR8vgsmF2mnzFpXfxbkf0yEZfu+/alD4xnJ9k4lA/bUhbaYY9JPLDGxyaD5jVGWi8mDhxYLB5t+HnDtKz91Moib8jtVJErfDWkIsJJnEtQUmiY7yCv6vnWSzqLqjxgV4kKcn+wcczjT/s5Hwfon6BWzoIjDnQbpVcp/dGhacZL+bQOngOjDQTYZISmYGVkQ4uYvYEJXsgrwAkByOZDbLaEEVhjL/MKVi5bA9fYoigU8tcyEG7OMWhlWC5O+DyDpTj9fQWwLsw+4UIKdsytxaWPRXO0zbewU5YYdOiKTnySG677eQuoCyFtqUpfTi4jitmZTe9leXBCYka6W2bZhXcQXP8yvU+dlyu1Yc2ZPGdzR1+DEBERnPWy2vlTUR/tmEjzTq86yqQia/HJJ+U2yytdVbdeVeQzeaibaZOjIPN5vA7nc= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(376002)(136003)(39850400004)(396003)(46966006)(36840700001)(83380400001)(82310400003)(2906002)(26005)(426003)(2616005)(7696005)(82740400003)(336012)(8936002)(86362001)(316002)(356005)(6916009)(8676002)(186003)(5660300002)(4326008)(47076005)(70206006)(6666004)(36756003)(7636003)(36860700001)(54906003)(1076003)(107886003)(478600001)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2021 10:29:41.7987 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5073ba5b-6ee1-44be-1f75-08d920fa56cc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT030.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1849 Cc: simon.horman@netronome.com, Jianbo Liu , i.maximets@ovn.org Subject: [ovs-dev] [PATCH v2] ofproto-dpif-upcall: Fix race condition while purging X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" There is a race condidtion between purger and handler in dpif-netlink. Handler may create new ukey and install it while executing 'ovs-appctl revalidator/purge' command. However, before handler calls transition_ukey() in handle_upcalls(), purger may get this ukey from umap, then evict and delete it. This will trigger ovs_abort in transition_ukey() for handler because it is trying to set state to EVICTED or OPERATIONAL, but ukey is already in DELETED state. To fix this issue, purger must not delete ukey in VISIBLE state. Fixes: 98bb4286970d ("tests: Add command to purge revalidators of flows.") Signed-off-by: Jianbo Liu Reviewed-by: Roi Dayan --- v2: Set PURGE_SOFT mode for dpif-netlink only. ofproto/ofproto-dpif-upcall.c | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index ccf97266c..963e8a17a 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -327,6 +327,12 @@ struct ukey_op { struct dpif_op dop; /* Flow operation. */ }; +enum sweep_type { + PURGE_NONE, + PURGE_SOFT, + PURGE_HARD, +}; + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); static struct ovs_list all_udpifs = OVS_LIST_INITIALIZER(&all_udpifs); @@ -345,7 +351,7 @@ static unsigned long udpif_get_n_flows(struct udpif *); static void revalidate(struct revalidator *); static void revalidator_pause(struct revalidator *); static void revalidator_sweep(struct revalidator *); -static void revalidator_purge(struct revalidator *); +static void revalidator_purge(struct revalidator *, enum sweep_type); static void upcall_unixctl_show(struct unixctl_conn *conn, int argc, const char *argv[], void *aux); static void upcall_unixctl_disable_megaflows(struct unixctl_conn *, int argc, @@ -541,7 +547,7 @@ udpif_stop_threads(struct udpif *udpif, bool delete_flows) if (delete_flows) { for (i = 0; i < udpif->n_revalidators; i++) { - revalidator_purge(&udpif->revalidators[i]); + revalidator_purge(&udpif->revalidators[i], PURGE_HARD); } } @@ -2772,7 +2778,8 @@ revalidator_pause(struct revalidator *revalidator) } static void -revalidator_sweep__(struct revalidator *revalidator, bool purge) +revalidator_sweep__(struct revalidator *revalidator, + enum sweep_type sweep_type) { struct udpif *udpif; uint64_t dump_seq, reval_seq; @@ -2803,13 +2810,13 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) } ukey_state = ukey->state; if (ukey_state == UKEY_OPERATIONAL - || (ukey_state == UKEY_VISIBLE && purge)) { + || (ukey_state == UKEY_VISIBLE && sweep_type == PURGE_HARD)) { struct recirc_refs recircs = RECIRC_REFS_EMPTY_INITIALIZER; bool seq_mismatch = (ukey->dump_seq != dump_seq && ukey->reval_seq != reval_seq); enum reval_result result; - if (purge) { + if (sweep_type > PURGE_NONE) { result = UKEY_DELETE; } else if (!seq_mismatch) { result = UKEY_KEEP; @@ -2856,13 +2863,13 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) static void revalidator_sweep(struct revalidator *revalidator) { - revalidator_sweep__(revalidator, false); + revalidator_sweep__(revalidator, PURGE_NONE); } static void -revalidator_purge(struct revalidator *revalidator) +revalidator_purge(struct revalidator *revalidator, enum sweep_type sweep_type) { - revalidator_sweep__(revalidator, true); + revalidator_sweep__(revalidator, sweep_type); } /* In reaction to dpif purge, purges all 'ukey's with same 'pmd_id'. */ @@ -3053,10 +3060,17 @@ upcall_unixctl_purge(struct unixctl_conn *conn, int argc OVS_UNUSED, struct udpif *udpif; LIST_FOR_EACH (udpif, list_node, &all_udpifs) { + enum sweep_type sweep_type; int n; + if (udpif->dpif->dpif_class == &dpif_netlink_class) { + sweep_type = PURGE_SOFT; + } else { + sweep_type = PURGE_HARD; + } + for (n = 0; n < udpif->n_revalidators; n++) { - revalidator_purge(&udpif->revalidators[n]); + revalidator_purge(&udpif->revalidators[n], sweep_type); } } unixctl_command_reply(conn, "");