From patchwork Tue May 11 08:38:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roi Dayan X-Patchwork-Id: 1476957 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=Ivo2WW1j; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FfWXp1kFNz9sX3 for ; Tue, 11 May 2021 18:38:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4AFCC60D59; Tue, 11 May 2021 08:38:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2weJSSBWY8Po; Tue, 11 May 2021 08:38:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTP id 53D0360D57; Tue, 11 May 2021 08:38:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 312C7C000D; Tue, 11 May 2021 08:38:30 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0EA24C0001 for ; Tue, 11 May 2021 08:38:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id F1AD884418 for ; Tue, 11 May 2021 08:38:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=nvidia.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8SG38YopNgBa for ; Tue, 11 May 2021 08:38:28 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20629.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::629]) by smtp1.osuosl.org (Postfix) with ESMTPS id DB29A83C5B for ; Tue, 11 May 2021 08:38:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mk9MDfvubJbofnftP0Mu1gMaI9OR7Kl2vCELosNQpeu0CDkvJkny1oEAQLqJoSZo9CSLg65v8+ZY0bFdm1qMwFuSmQOsvcRWtjSdJWw2kFVvZsNppqHtIY+jL0MD6nXbtCCazYDySIGWX38DkMMngs6AmZEzW43SVkOXuPCkKgRCDHCl+Hfx/mkjR7ATfbFd5ejltgH2KslOGSGkDh5gSrSSEEE7+Q14m5hpcChsk3UKWqSh/JVGf2fn+xw6CfyMG6/JZzW6AcA+kkMxpMOl84pyTrIwz5V7sSMUx5I47mvmSGR1q9hSCkLlsXc8MoYgTqOtrtHOqFxqwcxLXSY1Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+wMUUQPU17clKXTzlBS9QenqUKz90ilv2TXpJM/mOAI=; b=RyXCeL7HniteePTN/3e6PV4UuaSzPAO5JFUrv0QWFWpt+6DsPJZcMz03/3bkwAJj7H092XnGU6d1DWQuN3vn7WZZEAZxwRveRsCIkJjjS1BipRwyihD6Z2Zct68DKXgFI2ZcAbaB0PlOYw+9NIerHtGjN3IZcE4vZ1s0AZYvKtur8zVg5fPEl9Tfjq6Cs272NPblO6PzORENcpp8sNDgYFWHXq22pOjiV2cTncmdNl6/fvyZsMY/bw9MzhVEKsKZHk/tgafIkGBqarsE7tsY9LnHC7rEPw/5xErjd2nliAhU6lV/TjXMm0Xo0Ltpq1xBvGrnHOfx2BKxQwajSa2j9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=netronome.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+wMUUQPU17clKXTzlBS9QenqUKz90ilv2TXpJM/mOAI=; b=Ivo2WW1jGCioiN7hXdN+e1o25b6hKy9v04hssKuPyebVGezT8m2b5TFv+zPP01wPMNiHzLorficUpnHGOROskm83dTGIJmsjuCFekCXe2VxyHRLkURYFz2HplDq/VuGp59Yz4N0tlrwXwWapx+JYoxTSkd4SPhRQOgSAGxEr+jeSrQymB7Dht8JVPobNOZLGCUFZ8z2weSZtZuEX0JIcmQoZM8SrTY8i+D9IJwQY7223w2dicID/REc7ZI62orNpf1nDCqvwHObSKjTFjKhlCxi5vtHAezaytqsmmJQDmNyq81YVaL/qOEcJ4KjBmT5rZpWkUIgs8xD7seRES3UqxQ== Received: from BN9PR03CA0053.namprd03.prod.outlook.com (2603:10b6:408:fb::28) by MW3PR12MB4540.namprd12.prod.outlook.com (2603:10b6:303:52::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.26; Tue, 11 May 2021 08:38:26 +0000 Received: from BN8NAM11FT044.eop-nam11.prod.protection.outlook.com (2603:10b6:408:fb:cafe::dd) by BN9PR03CA0053.outlook.office365.com (2603:10b6:408:fb::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25 via Frontend Transport; Tue, 11 May 2021 08:38:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; netronome.com; dkim=none (message not signed) header.d=none;netronome.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT044.mail.protection.outlook.com (10.13.177.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4108.25 via Frontend Transport; Tue, 11 May 2021 08:38:25 +0000 Received: from HQMAIL107.nvidia.com (172.20.187.13) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 11 May 2021 08:38:24 +0000 Received: from dev-r-vrt-138.mtr.labs.mlnx (172.20.145.6) by mail.nvidia.com (172.20.187.13) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Tue, 11 May 2021 08:38:22 +0000 From: Roi Dayan To: Date: Tue, 11 May 2021 11:38:21 +0300 Message-ID: <20210511083821.60724-1-roid@nvidia.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 713479dd-b022-46e9-1446-08d9145824a3 X-MS-TrafficTypeDiagnostic: MW3PR12MB4540: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jezPwdu5YRctISTGymDPPXnMTP36IqzqzzOp+T7DhNXk8njYBtrAUtsHdtf4bvfOZIyZz4HEUndeE0TfV9KIa1FBeQR0qjpl7YwVpaqbOaNgtPMwgnyGZgXmQFUddOOjMhjaQnjZkgtdHXsF9pzdC9LrK2i4mxrxWcszXYQZ2YwkovLqfuERx2yMJ4mge9Hs67XUSUV+Y7MwsUNJBkgQG6JLtetIbkQ+X/PmXV9bigd6d02whLgNzG9A8oXX//cTFRAgOxB4f+0Lafch4CAxIDmj1C6ICjZN45SYjvFEI5cX9JgA3XiZB2AaMuLFNUeQr/hZQ4Fnv2GiIYFIXfPmCNmB1tBFz7NO8NHRo9sEolZYETq3pbhRnmNoL+DQyX8mXpMVBIRvWLpIi1d/bJro/OOLAP704BjvYEh8hxF7iuzIdEPO5Fa1nI9my9lgKmOojSdgTprSuUFl5ZeP+MHnMqiReNZUoACuh5XcTXVl4oxEMQvz8sZK0ULKRW5o61k4W4njgNd9K0OERzxpHoiAUR1+PDI3eeUX3KZC8efF4RITMVwRc3JC67Iz6ngU77+GOHwIirm9IObryfXLWTStNQltReJiMbEaaRayBV1XMckmVbn1EMpJ4wVaxEQAAYrkjXB0ZcOpVuNs+AWWN2eE4gN0oLvxFactveTCfzkln+A= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(39860400002)(136003)(396003)(376002)(46966006)(36840700001)(4326008)(70206006)(83380400001)(70586007)(2616005)(82740400003)(26005)(2906002)(1076003)(47076005)(107886003)(336012)(54906003)(6916009)(186003)(36860700001)(316002)(8676002)(7636003)(36756003)(82310400003)(5660300002)(426003)(356005)(86362001)(478600001)(36906005)(8936002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 May 2021 08:38:25.2131 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 713479dd-b022-46e9-1446-08d9145824a3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT044.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4540 Cc: Jianbo Liu , Ilya Maximets , Simon Horman Subject: [ovs-dev] [PATCH] ofproto-dpif-upcall: Fix race condition while purging X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Jianbo Liu There is a race condidtion between purger and handler. Handler may create new ukey and install it while executing 'ovs-appctl revalidator/purge' command. However, before handler calls transition_ukey() in handle_upcalls(), purger may get this ukey from umap, then evict and delete it. This will trigger ovs_abort in transition_ukey() for handler because it is trying to set state to EVICTED or OPERATIONAL, but ukey is already in DELETED state. To fix this issue, purger must not delete ukey in VISIBLE state. Fixes: 98bb4286970d ("tests: Add command to purge revalidators of flows.") Signed-off-by: Jianbo Liu Reviewed-by: Roi Dayan --- ofproto/ofproto-dpif-upcall.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index ccf97266c0b9..3add505ff652 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -327,6 +327,12 @@ struct ukey_op { struct dpif_op dop; /* Flow operation. */ }; +enum sweep_type { + PURGE_NONE, + PURGE_SOFT, + PURGE_HARD, +}; + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); static struct ovs_list all_udpifs = OVS_LIST_INITIALIZER(&all_udpifs); @@ -345,7 +351,7 @@ static unsigned long udpif_get_n_flows(struct udpif *); static void revalidate(struct revalidator *); static void revalidator_pause(struct revalidator *); static void revalidator_sweep(struct revalidator *); -static void revalidator_purge(struct revalidator *); +static void revalidator_purge(struct revalidator *, enum sweep_type); static void upcall_unixctl_show(struct unixctl_conn *conn, int argc, const char *argv[], void *aux); static void upcall_unixctl_disable_megaflows(struct unixctl_conn *, int argc, @@ -541,7 +547,7 @@ udpif_stop_threads(struct udpif *udpif, bool delete_flows) if (delete_flows) { for (i = 0; i < udpif->n_revalidators; i++) { - revalidator_purge(&udpif->revalidators[i]); + revalidator_purge(&udpif->revalidators[i], PURGE_HARD); } } @@ -2772,7 +2778,8 @@ revalidator_pause(struct revalidator *revalidator) } static void -revalidator_sweep__(struct revalidator *revalidator, bool purge) +revalidator_sweep__(struct revalidator *revalidator, + enum sweep_type sweep_type) { struct udpif *udpif; uint64_t dump_seq, reval_seq; @@ -2803,13 +2810,13 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) } ukey_state = ukey->state; if (ukey_state == UKEY_OPERATIONAL - || (ukey_state == UKEY_VISIBLE && purge)) { + || (ukey_state == UKEY_VISIBLE && sweep_type == PURGE_HARD)) { struct recirc_refs recircs = RECIRC_REFS_EMPTY_INITIALIZER; bool seq_mismatch = (ukey->dump_seq != dump_seq && ukey->reval_seq != reval_seq); enum reval_result result; - if (purge) { + if (sweep_type > PURGE_NONE) { result = UKEY_DELETE; } else if (!seq_mismatch) { result = UKEY_KEEP; @@ -2856,13 +2863,13 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) static void revalidator_sweep(struct revalidator *revalidator) { - revalidator_sweep__(revalidator, false); + revalidator_sweep__(revalidator, PURGE_NONE); } static void -revalidator_purge(struct revalidator *revalidator) +revalidator_purge(struct revalidator *revalidator, enum sweep_type sweep_type) { - revalidator_sweep__(revalidator, true); + revalidator_sweep__(revalidator, sweep_type); } /* In reaction to dpif purge, purges all 'ukey's with same 'pmd_id'. */ @@ -3056,7 +3063,7 @@ upcall_unixctl_purge(struct unixctl_conn *conn, int argc OVS_UNUSED, int n; for (n = 0; n < udpif->n_revalidators; n++) { - revalidator_purge(&udpif->revalidators[n]); + revalidator_purge(&udpif->revalidators[n], PURGE_SOFT); } } unixctl_command_reply(conn, "");