From patchwork Thu Jan 9 12:21:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2031838 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4YTP886v2fz1yP7 for ; Thu, 9 Jan 2025 23:23:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7D6884096F; Thu, 9 Jan 2025 12:23:19 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id khCBBFg1pG3d; Thu, 9 Jan 2025 12:23:18 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 100584021A Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 100584021A; Thu, 9 Jan 2025 12:23:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BBE1FC087D; Thu, 9 Jan 2025 12:23:17 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 31BA6C0612 for ; Thu, 9 Jan 2025 12:23:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 212F680AFA for ; Thu, 9 Jan 2025 12:23:17 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ClvJ17jpSwgB for ; Thu, 9 Jan 2025 12:23:16 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.66; helo=mail-wm1-f66.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 0BEF280ABE Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 0BEF280ABE Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by smtp1.osuosl.org (Postfix) with ESMTPS id 0BEF280ABE for ; Thu, 9 Jan 2025 12:23:15 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id 5b1f17b1804b1-4361fe642ddso9541075e9.2 for ; Thu, 09 Jan 2025 04:23:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736425394; x=1737030194; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SrNP4iM+dKjOzSzFjGe+I78I/aSS5rp5dYXJ1rHOADA=; b=FTKLFqKdEQ3CaSJUM5VVBeImN/nbPLH1VxdZ0ot4Y5nP+GCzUJYS19e7smZPGEppgA lntMDTywGs1T1ru1hcOHDi1z8e+D0QjPrQZUrF7NhrV6qLf9IQ809teg/irK+mKKr9Ub RFzVjvAKwWOGofVGfwNWSXMsdBXQ96P0q0uz3Rpl2zjAMphQyXcJoMli6UH19T2AL2Zw 2ZEIK44rk5u5ga0HFOQ3u69ipGVKBraB2hM4hxIS+weBe8kuDUoEbOIuSjADFZBYuo+W nTDHavxa123Yr6rj1P5k/yjY/s6Wu85RbvC8Aa0NM84pAFh1xGbEXApeJwfnrXjj+0pB nbdw== X-Forwarded-Encrypted: i=1; AJvYcCV2lGdC66qRU8GK+2MMd4Z87mI/spfW3VJmFeePt8jv/gyIou2JglVbLMPuGH2HI3Oeyvk=@openvswitch.org X-Gm-Message-State: AOJu0YyLTdYf9v0+2lZHOwg0Qg6UU9hzQKbI8t5vGfEyafRh8mNy/6z0 zZ2eqXizUujS8rNFmDCydUpaVh36vOBT5cPimxoyzOBxMdw47C7f X-Gm-Gg: ASbGncu7ZeU7R6/WkLEwaIEO6/1s0Rvl1f5gMZN9KrVEYTN/ZNkz3PzLCIhXwSrUErY 8E8c/V0g2G9t4ImKZ9VUPcCZ2ObbfnSDj6km35df4cGxqMVG0L7m7n+iBoYjNxQv1RmMe6kAr+G hzCunj57K/Dp4XIdRXuFjAZoSsfqycKQ7ZiTQJz/ZKv4O8wtlD9uQngFqWDYm9YcGyq9n0Wk9Cz T0ScK33Pt9OCtL5HBpp4csi14f1sAtyW1GOvdk0ZClAGg3ec5hMFTHmpNugq2K2tYXWaMlIZmN8 blv65MrQ1q6nR7PGQmQ= X-Google-Smtp-Source: AGHT+IEUp+4FQMuai1vfLoPr02YbfVvdXSu1gq6Z2SFxrOsxv/3on8mNC2W9EOG2nmadrYP5xzO/dg== X-Received: by 2002:a05:6000:1564:b0:38a:8c9f:dd61 with SMTP id ffacd0b85a97d-38a8c9fdeeamr2300174f8f.46.1736425393780; Thu, 09 Jan 2025 04:23:13 -0800 (PST) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e38534csm1672896f8f.43.2025.01.09.04.23.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2025 04:23:13 -0800 (PST) From: Ilya Maximets To: netdev@vger.kernel.org Date: Thu, 9 Jan 2025 13:21:24 +0100 Message-ID: <20250109122225.4034688-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.47.0 MIME-Version: 1.0 Subject: [ovs-dev] [PATCH net] openvswitch: fix lockup on tx to unregistering netdev with carrier X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dev@openvswitch.org, Friedrich Weber , Luca Czesla , linux-kernel@vger.kernel.org, Ilya Maximets , Felix Huettner , Eric Dumazet , Simon Horman , Jakub Kicinski , Paolo Abeni , "David S. Miller" Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Commit in a fixes tag attempted to fix the issue in the following sequence of calls: do_output -> ovs_vport_send -> dev_queue_xmit -> __dev_queue_xmit -> netdev_core_pick_tx -> skb_tx_hash When device is unregistering, the 'dev->real_num_tx_queues' goes to zero and the 'while (unlikely(hash >= qcount))' loop inside the 'skb_tx_hash' becomes infinite, locking up the core forever. But unfortunately, checking just the carrier status is not enough to fix the issue, because some devices may still be in unregistering state while reporting carrier status OK. One example of such device is a net/dummy. It sets carrier ON on start, but it doesn't implement .ndo_stop to set the carrier off. And it makes sense, because dummy doesn't really have a carrier. Therefore, while this device is unregistering, it's still easy to hit the infinite loop in the skb_tx_hash() from the OVS datapath. There might be other drivers that do the same, but dummy by itself is important for the OVS ecosystem, because it is frequently used as a packet sink for tcpdump while debugging OVS deployments. And when the issue is hit, the only way to recover is to reboot. Fix that by also checking if the device is running. The running state is handled by the net core during unregistering, so it covers unregistering case better, and we don't really need to send packets to devices that are not running anyway. While only checking the running state might be enough, the carrier check is preserved. The running and the carrier states seem disjoined throughout the code and different drivers. And other core functions like __dev_direct_xmit() check both before attempting to transmit a packet. So, it seems safer to check both flags in OVS as well. Fixes: 066b86787fa3 ("net: openvswitch: fix race on port output") Reported-by: Friedrich Weber Closes: https://mail.openvswitch.org/pipermail/ovs-discuss/2025-January/053423.html Signed-off-by: Ilya Maximets Tested-by: Friedrich Weber Reviewed-by: Aaron Conole --- net/openvswitch/actions.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index 16e260014684..704c858cf209 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -934,7 +934,9 @@ static void do_output(struct datapath *dp, struct sk_buff *skb, int out_port, { struct vport *vport = ovs_vport_rcu(dp, out_port); - if (likely(vport && netif_carrier_ok(vport->dev))) { + if (likely(vport && + netif_running(vport->dev) && + netif_carrier_ok(vport->dev))) { u16 mru = OVS_CB(skb)->mru; u32 cutlen = OVS_CB(skb)->cutlen;