| Message ID | 178784100453.66359.11703988426858088454@ecanepa-zbook.public |
|---|---|
| Headers | show |
| Series | New 7.0 ubuntu_kselftests_net/net:tun tests flaky (LP: #2158217) | expand |
On 8/27/26 08:47, Edoardo Canepa via kernel-team wrote: > BugLink: https://bugs.launchpad.net/bugs/2158217 > > [Impact] > The tun_vnet_udptnl selftests (introduced in the 7.0 cycle) are flaky > in ubuntu_kselftests_net/net:tun, causing sporadic CI failures. The > send_gso_packet variants fail with zero bytes received on the inner UDP > socket after packets are written into the TUN file descriptor: > > tun.c:947:send_gso_packet:Expected ret (0) == variant->data_size (1423) > tun.c:948:send_gso_packet:Expected r_num_mss (0) == variant->r_num_mss (2) > > The failures are non-deterministic: different variants fail on each run > and all tunnel types (4in4, 4in6, 6in4, 6in6) are affected. Only the > decap direction (TUN fd -> geneve -> socket) is impacted; the encap > direction (socket -> geneve -> TUN fd) always succeeds. > > Two independent root causes were identified: > > 1. The test IPv6 addresses fall in the 2002::/16 range reserved for the > 6to4 transition mechanism (RFC 3056). When the sit module is loaded > the kernel claims this prefix, creating a competing local route that > prevents geneve from decapsulating IPv6-outer packets. > > 2. The geneve fixture binds a UDP encap socket on port 4789 in the > default network namespace. Any host network activity that also > touches that port or the test address ranges (VPN tunnels, container > overlay networks, libvirt bridges, etc.) can silently divert or drop > packets in the geneve decapsulation path. > > This is not a kernel regression; the tests exhibit the same flakiness > when the 7.0 test binary is run on a 6.17 kernel. > > [Fix] > Two patches: > > Patch 1 (upstream cherry-pick ef01724fa235): Replaces the 2002::/16 > IPv6 test addresses with fd00:db8::/32 ULA addresses that carry no > special kernel semantics. > > Patch 2 (SAUCE): Calls unshare(CLONE_NEWNET) at program startup so the > test runs in a pristine, isolated network namespace free of competing > traffic. This follows the same pattern already used by several other > net selftests (so_incoming_cpu, tcp_port_share, icmp_rfc4884, > ipv6_fragmentation, so_netns_cookie, ipsec, tcp_ao). > > [Test] > Built the tun test binary from the resolute kernel source tree and > executed it repeatedly before and after the fix: > > Before: 7 out of 55 runs had at least one failure (~12.7% failure rate), > with 9 individual test failures spread across all tunnel types. > > After: 0 failures across 20 consecutive runs. > > [Where problems could occur] > The unshare(CLONE_NEWNET) call requires CAP_SYS_ADMIN, which the test > already needs to create TUN/TAP devices. Running in an isolated > namespace means the test cannot exercise interactions with pre-existing > host network configuration, but that is not the purpose of these unit > tests. > Please update the LP bug description to add more info about the fix. Also, there is a double cover-letter and missing "[SRU][R]" from the subject of the patches. Acked-by: Rickey Castillo <rickey.castillo.valenzuela@canonical.com>
Rejected for the following reasons: V2 incoming. On 8/27/26 17:47, Edoardo Canepa wrote: > BugLink: https://bugs.launchpad.net/bugs/2158217 > > [Impact] > The tun_vnet_udptnl selftests (introduced in the 7.0 cycle) are flaky > in ubuntu_kselftests_net/net:tun, causing sporadic CI failures. The > send_gso_packet variants fail with zero bytes received on the inner UDP > socket after packets are written into the TUN file descriptor: > > tun.c:947:send_gso_packet:Expected ret (0) == variant->data_size (1423) > tun.c:948:send_gso_packet:Expected r_num_mss (0) == variant->r_num_mss (2) > > The failures are non-deterministic: different variants fail on each run > and all tunnel types (4in4, 4in6, 6in4, 6in6) are affected. Only the > decap direction (TUN fd -> geneve -> socket) is impacted; the encap > direction (socket -> geneve -> TUN fd) always succeeds. > > Two independent root causes were identified: > > 1. The test IPv6 addresses fall in the 2002::/16 range reserved for the > 6to4 transition mechanism (RFC 3056). When the sit module is loaded > the kernel claims this prefix, creating a competing local route that > prevents geneve from decapsulating IPv6-outer packets. > > 2. The geneve fixture binds a UDP encap socket on port 4789 in the > default network namespace. Any host network activity that also > touches that port or the test address ranges (VPN tunnels, container > overlay networks, libvirt bridges, etc.) can silently divert or drop > packets in the geneve decapsulation path. > > This is not a kernel regression; the tests exhibit the same flakiness > when the 7.0 test binary is run on a 6.17 kernel. > > [Fix] > Two patches: > > Patch 1 (upstream cherry-pick ef01724fa235): Replaces the 2002::/16 > IPv6 test addresses with fd00:db8::/32 ULA addresses that carry no > special kernel semantics. > > Patch 2 (SAUCE): Calls unshare(CLONE_NEWNET) at program startup so the > test runs in a pristine, isolated network namespace free of competing > traffic. This follows the same pattern already used by several other > net selftests (so_incoming_cpu, tcp_port_share, icmp_rfc4884, > ipv6_fragmentation, so_netns_cookie, ipsec, tcp_ao). > > [Test] > Built the tun test binary from the resolute kernel source tree and > executed it repeatedly before and after the fix: > > Before: 7 out of 55 runs had at least one failure (~12.7% failure rate), > with 9 individual test failures spread across all tunnel types. > > After: 0 failures across 20 consecutive runs. > > [Where problems could occur] > The unshare(CLONE_NEWNET) call requires CAP_SYS_ADMIN, which the test > already needs to create TUN/TAP devices. Running in an isolated > namespace means the test cannot exercise interactions with pre-existing > host network configuration, but that is not the purpose of these unit > tests.