| Message ID | 20240104032749.802262-1-brad@faucet.nz |
|---|---|
| State | Superseded |
| Delegated to: | Simon Horman |
| Headers | show |
| Series | [ovs-dev] system-tests: Test openflow matching for ct related packets with SNAT. | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | fail | github build: failed |
| ovsrobot/intel-ovs-compilation | fail | test: fail |
On Thu, Jan 04, 2024 at 04:27:49PM +1300, Brad Cowie wrote: > Linux kernel commit ebddb1404900 ("net: move the nat function to > nf_nat_ovs for ovs and tc") introduced a regression into the kernel > datapath which prevented the openvswitch match key from being updated > when nat was undone for packets in the related conntrack state. This > issue caused these packets (usually ICMP/ICMPv6 error packets) to > match the wrong openflow rule. > > This issue was fixed in linux kernel commit e6345d2824a3 ("netfilter: > nf_nat: fix action not being set for all ct states"). > > This test will reproduce the issue and fail for kernel versions > v6.2 to v6.6, and will pass on earlier kernel versions where the issue > wasn't present, or on later kernel versions that have the fix applied. > > Link: https://lore.kernel.org/netdev/20231221224311.130319-1-brad@faucet.nz/ > Suggested-by: Aaron Conole <aconole@redhat.com> > Signed-off-by: Brad Cowie <brad@faucet.nz> Hi Brad, thanks for following-up on this. One question from my side is, given that this is currently broken in many kernels in use today, how we should integrate this. For one thing, applying this patch causes the CI to fail. https://github.com/ovsrobot/ovs/actions/runs/7405341045 It might be nice if we could detect known to be broken kernels. But I'm not sure, there is an easy way to do that, other than running the test itself. Do you have any thoughts on this?
On 1/4/24 11:57, Simon Horman wrote: > On Thu, Jan 04, 2024 at 04:27:49PM +1300, Brad Cowie wrote: >> Linux kernel commit ebddb1404900 ("net: move the nat function to >> nf_nat_ovs for ovs and tc") introduced a regression into the kernel >> datapath which prevented the openvswitch match key from being updated >> when nat was undone for packets in the related conntrack state. This >> issue caused these packets (usually ICMP/ICMPv6 error packets) to >> match the wrong openflow rule. >> >> This issue was fixed in linux kernel commit e6345d2824a3 ("netfilter: >> nf_nat: fix action not being set for all ct states"). >> >> This test will reproduce the issue and fail for kernel versions >> v6.2 to v6.6, and will pass on earlier kernel versions where the issue >> wasn't present, or on later kernel versions that have the fix applied. >> >> Link: https://lore.kernel.org/netdev/20231221224311.130319-1-brad@faucet.nz/ >> Suggested-by: Aaron Conole <aconole@redhat.com> >> Signed-off-by: Brad Cowie <brad@faucet.nz> > > Hi Brad, > > thanks for following-up on this. > > One question from my side is, given that this is currently broken in many > kernels in use today, how we should integrate this. For one thing, > applying this patch causes the CI to fail. > > https://github.com/ovsrobot/ovs/actions/runs/7405341045 > > It might be nice if we could detect known to be broken kernels. > But I'm not sure, there is an easy way to do that, other than > running the test itself. > > Do you have any thoughts on this? One option could be to exclude the kernels below 6.7 with: OVS_CHECK_KERNEL(6, 7) Unfortunately the original issue seems to be backported to some distribution kernels, but all the kernels 6.7+ should be fine. However, I'm not convinced the test failed in CI because of this. They failed due to difference on packet and by counters. In case of offload tests they also have matching packet counters, but different byte counters. I know there were cases where TC counts bytes differently. So, the counters seem to be not a very reliable source of information. Is there any other way we can detect the issue without comparing exact values of the packet/byte counters? Best regards, Ilya Maximets.
On 1/4/24 15:03, Ilya Maximets wrote: > On 1/4/24 11:57, Simon Horman wrote: >> On Thu, Jan 04, 2024 at 04:27:49PM +1300, Brad Cowie wrote: >>> Linux kernel commit ebddb1404900 ("net: move the nat function to >>> nf_nat_ovs for ovs and tc") introduced a regression into the kernel >>> datapath which prevented the openvswitch match key from being updated >>> when nat was undone for packets in the related conntrack state. This >>> issue caused these packets (usually ICMP/ICMPv6 error packets) to >>> match the wrong openflow rule. >>> >>> This issue was fixed in linux kernel commit e6345d2824a3 ("netfilter: >>> nf_nat: fix action not being set for all ct states"). >>> >>> This test will reproduce the issue and fail for kernel versions >>> v6.2 to v6.6, and will pass on earlier kernel versions where the issue >>> wasn't present, or on later kernel versions that have the fix applied. >>> >>> Link: https://lore.kernel.org/netdev/20231221224311.130319-1-brad@faucet.nz/ >>> Suggested-by: Aaron Conole <aconole@redhat.com> >>> Signed-off-by: Brad Cowie <brad@faucet.nz> >> >> Hi Brad, >> >> thanks for following-up on this. >> >> One question from my side is, given that this is currently broken in many >> kernels in use today, how we should integrate this. For one thing, >> applying this patch causes the CI to fail. >> >> https://github.com/ovsrobot/ovs/actions/runs/7405341045 >> >> It might be nice if we could detect known to be broken kernels. >> But I'm not sure, there is an easy way to do that, other than >> running the test itself. >> >> Do you have any thoughts on this? > > One option could be to exclude the kernels below 6.7 with: > > OVS_CHECK_KERNEL(6, 7) > > Unfortunately the original issue seems to be backported to some > distribution kernels, but all the kernels 6.7+ should be fine. > > However, I'm not convinced the test failed in CI because of this. > They failed due to difference on packet and by counters. > In case of offload tests they also have matching packet counters, > but different byte counters. I know there were cases where TC > counts bytes differently. So, the counters seem to be not a > very reliable source of information. Is there any other way > we can detect the issue without comparing exact values of the > packet/byte counters? Stripping out the byte counters might be enough, I guess, in pair with the kernel version check. But if there is a better way to check, it might be better to not rely on packet counters as well. > > Best regards, Ilya Maximets.
Ilya Maximets <i.maximets@ovn.org> writes: > On 1/4/24 15:03, Ilya Maximets wrote: >> On 1/4/24 11:57, Simon Horman wrote: >>> On Thu, Jan 04, 2024 at 04:27:49PM +1300, Brad Cowie wrote: >>>> Linux kernel commit ebddb1404900 ("net: move the nat function to >>>> nf_nat_ovs for ovs and tc") introduced a regression into the kernel >>>> datapath which prevented the openvswitch match key from being updated >>>> when nat was undone for packets in the related conntrack state. This >>>> issue caused these packets (usually ICMP/ICMPv6 error packets) to >>>> match the wrong openflow rule. >>>> >>>> This issue was fixed in linux kernel commit e6345d2824a3 ("netfilter: >>>> nf_nat: fix action not being set for all ct states"). >>>> >>>> This test will reproduce the issue and fail for kernel versions >>>> v6.2 to v6.6, and will pass on earlier kernel versions where the issue >>>> wasn't present, or on later kernel versions that have the fix applied. >>>> >>>> Link: https://lore.kernel.org/netdev/20231221224311.130319-1-brad@faucet.nz/ >>>> Suggested-by: Aaron Conole <aconole@redhat.com> >>>> Signed-off-by: Brad Cowie <brad@faucet.nz> >>> >>> Hi Brad, >>> >>> thanks for following-up on this. >>> >>> One question from my side is, given that this is currently broken in many >>> kernels in use today, how we should integrate this. For one thing, >>> applying this patch causes the CI to fail. >>> >>> https://github.com/ovsrobot/ovs/actions/runs/7405341045 >>> >>> It might be nice if we could detect known to be broken kernels. >>> But I'm not sure, there is an easy way to do that, other than >>> running the test itself. >>> >>> Do you have any thoughts on this? >> >> One option could be to exclude the kernels below 6.7 with: >> >> OVS_CHECK_KERNEL(6, 7) >> >> Unfortunately the original issue seems to be backported to some >> distribution kernels, but all the kernels 6.7+ should be fine. >> >> However, I'm not convinced the test failed in CI because of this. >> They failed due to difference on packet and by counters. >> In case of offload tests they also have matching packet counters, >> but different byte counters. I know there were cases where TC >> counts bytes differently. So, the counters seem to be not a >> very reliable source of information. Is there any other way >> we can detect the issue without comparing exact values of the >> packet/byte counters? > > Stripping out the byte counters might be enough, I guess, in > pair with the kernel version check. But if there is a better > way to check, it might be better to not rely on packet counters > as well. Thanks to everyone for the quick attention. We should consider putting this test into the kernel tree instead of housing it in the OVS tree. We're really using ovs here as a glorified flow generator, but maybe we can just take the flows that it spits out and use them in tools/testing/selftests/net/openvswitch/openvswitch.sh with ovs-dpctl.py and update the test_nat_connect_v4 with this check. That would also have the advantage of keeping the tests with the kernel, since it is a datapath specific bug. I think we could also try stripping counters w/ kernel version check. One reason we sometimes get counters are programs running which do autoconf or other kinds of auto-disocvery on newly created interfaces. This happens for sure on ipv6 enabled stacks which get lots of additional traffic, so we generally can't rely on counters being perfect. Maybe we could do some kind of '>=' test instead of '==' if we really want to compare these counters, but I still think it can be a bit error prone. Actually, the most "accurate" thing to do would be attach some ebpf programs to the action execution pipeline and intercept the packet as it goes through these actions and validate that we are performing the correct actions there. That method is probably way overkill, though. >> >> Best regards, Ilya Maximets.
On Fri, 5 Jan 2024 at 03:03, Ilya Maximets <i.maximets@ovn.org> wrote: > On 1/4/24 11:57, Simon Horman wrote: >> One question from my side is, given that this is currently broken in many >> kernels in use today, how we should integrate this. For one thing, >> applying this patch causes the CI to fail. >> >> https://github.com/ovsrobot/ovs/actions/runs/7405341045 >> >> It might be nice if we could detect known to be broken kernels. >> But I'm not sure, there is an easy way to do that, other than >> running the test itself. >> >> Do you have any thoughts on this? > > One option could be to exclude the kernels below 6.7 with: > > OVS_CHECK_KERNEL(6, 7) > > Unfortunately the original issue seems to be backported to some > distribution kernels, but all the kernels 6.7+ should be fine. > > However, I'm not convinced the test failed in CI because of this. > They failed due to difference on packet and by counters. > In case of offload tests they also have matching packet counters, > but different byte counters. I know there were cases where TC > counts bytes differently. So, the counters seem to be not a > very reliable source of information. Is there any other way > we can detect the issue without comparing exact values of the > packet/byte counters? Thanks all for the quick feedback! I've now submitted a V2 patch which addresses both issues raised: 1. Skip running the test on kernel versions older than 6.7 2. Remove byte counters to fix the offloads system tests On Fri, 5 Jan 2024 at 04:40, Aaron Conole <aconole@redhat.com> wrote: > We should consider putting this test into the kernel tree instead of > housing it in the OVS tree. We're really using ovs here as a glorified > flow generator, but maybe we can just take the flows that it spits out > and use them in tools/testing/selftests/net/openvswitch/openvswitch.sh > with ovs-dpctl.py and update the test_nat_connect_v4 with this check. > That would also have the advantage of keeping the tests with the kernel, > since it is a datapath specific bug. Agree, this should be tested in the kernel to catch any potential future regressions before they are merged. I've attached a quick patch I wrote today which adds a basic kernel self-test using the same ruleset as the ovs test I created. Was interested in getting some early feedback here on this list first, and if positive, I can clean this patch up and submit to the kernel next week. By the way, I'm happy if you guys decide not to merge my ovs patch and you just want to test this in the kernel, however I do wonder if there's some additional benefit to having the test in both the kernel and ovs. This would allow package builders to run the ovs tests to sanity check if the kernel they are building on has working NAT support or not. Especially given Ilya's point that ebddb1404900 has been backported to many different distro kernels. --- .../selftests/net/openvswitch/openvswitch.sh | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/tools/testing/selftests/net/openvswitch/openvswitch.sh b/tools/testing/selftests/net/openvswitch/openvswitch.sh index f8499d4c87f3..a1ba0c64f263 100755 --- a/tools/testing/selftests/net/openvswitch/openvswitch.sh +++ b/tools/testing/selftests/net/openvswitch/openvswitch.sh @@ -17,6 +17,7 @@ tests=" ct_connect_v4 ip4-ct-xon: Basic ipv4 tcp connection using ct connect_v4 ip4-xon: Basic ipv4 ping between two NS nat_connect_v4 ip4-nat-xon: Basic ipv4 tcp connection via NAT + nat_related_v4 ip4-nat-related-xon: ICMP related via SNAT netlink_checks ovsnl: validate netlink attrs and settings upcall_interfaces ovs: test the upcall interfaces drop_reason drop: test drop reasons are emitted" @@ -473,6 +474,54 @@ test_nat_connect_v4 () { return 0 } +# nat_related_v4 test +test_nat_related_v4 () { + which nc >/dev/null 2>/dev/null || return $ksft_skip + + sbx_add "test_nat_related_v4" || return $? + + ovs_add_dp "test_nat_related_v4" natrelated4 || return 1 + info "create namespaces" + for ns in client server; do + ovs_add_netns_and_veths "test_nat_related_v4" "natrelated4" "$ns" \ + "${ns:0:1}0" "${ns:0:1}1" || return 1 + done + + ip netns exec client ip addr add 172.31.110.10/24 dev c1 + ip netns exec client ip link set c1 up + ip netns exec server ip addr add 172.31.110.20/24 dev s1 + ip netns exec server ip link set s1 up + + ip netns exec server ip route add 192.168.0.20/32 via 172.31.110.10 + + ovs_add_flow "test_nat_related_v4" natrelated4 \ + 'in_port(1),eth(),eth_type(0x0806),arp()' '2' || return 1 + ovs_add_flow "test_nat_related_v4" natrelated4 \ + 'in_port(2),eth(),eth_type(0x0806),arp()' '1' || return 1 + ovs_add_flow "test_nat_related_v4" natrelated4 \ + "ct_state(-trk),in_port(1),eth(),eth_type(0x0800),ipv4(dst=172.31.110.20)" \ + "ct(commit,nat(src=192.168.0.20)),recirc(0x1)" + ovs_add_flow "test_nat_related_v4" natrelated4 \ + "ct_state(-trk),in_port(2),eth(),eth_type(0x0800),ipv4()" \ + "ct(commit,nat),recirc(0x2)" + + ovs_add_flow "test_nat_related_v4" natrelated4 \ + "recirc_id(0x1),ct_state(+trk-inv),in_port(1),eth(),eth_type(0x0800),ipv4()" "2" + ovs_add_flow "test_nat_related_v4" natrelated4 \ + "recirc_id(0x2),ct_state(+rel+trk),in_port(2),eth(),eth_type(0x0800),ipv4(src=172.31.110.20,dst=172.31.110.10,proto=1),icmp()" "1" + ovs_add_flow "test_nat_related_v4" natrelated4 \ + "recirc_id(0x2),ct_state(+rel+trk),in_port(2),eth(),eth_type(0x0800),ipv4(dst=192.168.0.20,proto=1),icmp()" "drop" + + # Solicit destination unreachable response from server + ovs_sbx "test_nat_connect_v4" ip netns exec client bash -c "echo a | nc -u -w 1 172.31.110.20 10000" + + # Check to make sure no packets were dropped (matched rule with incorrect IP) + python3 "$ovs_base/ovs-dpctl.py" dump-flows natrelated4 | grep "drop" | grep "packets:0" || return 1 + + info "done..." + return 0 +} + # netlink_validation # - Create a dp # - check no warning with "old version" simulation
diff --git a/tests/system-traffic.at b/tests/system-traffic.at index 3cdd2f125..f4b5b4c00 100644 --- a/tests/system-traffic.at +++ b/tests/system-traffic.at @@ -6473,6 +6473,94 @@ AT_CHECK([tcpdump -n -v "icmp" -r p0.pcap 2>/dev/null | grep -E 'wrong|bad'], [1 OVS_TRAFFIC_VSWITCHD_STOP AT_CLEANUP +AT_SETUP([conntrack - ICMP related with SNAT]) +AT_SKIP_IF([test $HAVE_NC = no]) +AT_SKIP_IF([test $HAVE_TCPDUMP = no]) +CHECK_CONNTRACK() +CHECK_CONNTRACK_NAT() +OVS_TRAFFIC_VSWITCHD_START() + +ADD_NAMESPACES(at_ns0, at_ns1) + +ADD_VETH(p0, at_ns0, br0, "10.1.1.1/24") +NS_CHECK_EXEC([at_ns0], [ip link set dev p0 address 80:88:88:88:88:88]) +ADD_VETH(p1, at_ns1, br0, "10.1.1.2/24") + +dnl Allow IP traffic from ns0->ns1, rewrite source IP with SNAT to 10.1.1.254. +dnl Only allow related ICMP responses back and undo NAT to restore original IP. +AT_DATA([flows.txt], [dnl +ct_state=-trk,ip actions=ct(table=0) +ct_state=+trk,ip,in_port=1 actions=ct(commit,nat(src=10.1.1.254)),2 +ct_state=+rel+trk,icmp,in_port=2,nw_dst=10.1.1.254 actions=ct(commit,table=1,nat) +dnl +dnl Handle ICMP related packets. +dnl These should match first rule with original IPs before SNAT. +dnl The second rule, which matches on the SNAT IP, shouldn't match any packets. +table=1,in_port=2,ct_state=+rel+trk,icmp,nw_src=10.1.1.2,nw_dst=10.1.1.1 action=1 +table=1,in_port=2,ct_state=+rel+trk,icmp,nw_dst=10.1.1.254 action=goto_table:2 +table=1,priority=0,action=drop +dnl +dnl Drop any ICMP related packets that incorrectly reach this table. +table=2,priority=0,action=drop +dnl +dnl ARP +priority=100 arp arp_op=1 action=move:OXM_OF_ARP_TPA[[]]->NXM_NX_REG2[[]],resubmit(,8),goto_table:10 +priority=10 arp action=normal +priority=0,action=drop +dnl +dnl MAC resolution table for IP in reg2, stores mac in OXM_OF_PKT_REG0 +table=8,reg2=0x0a0101f0/0xfffffff0,action=load:0x808888888888->OXM_OF_PKT_REG0[[]] +table=8,priority=0,action=load:0->OXM_OF_PKT_REG0[[]] +dnl ARP responder mac filled in at OXM_OF_PKT_REG0, or 0 for normal action. +dnl TPA IP in reg2. +dnl Swaps the fields of the ARP message to turn a query to a response. +table=10 priority=100 arp xreg0=0 action=normal +table=10 priority=10,arp,arp_op=1,action=load:2->OXM_OF_ARP_OP[[]],move:OXM_OF_ARP_SHA[[]]->OXM_OF_ARP_THA[[]],move:OXM_OF_PKT_REG0[[0..47]]->OXM_OF_ARP_SHA[[]],move:OXM_OF_ARP_SPA[[]]->OXM_OF_ARP_TPA[[]],move:NXM_NX_REG2[[]]->OXM_OF_ARP_SPA[[]],move:NXM_OF_ETH_SRC[[]]->NXM_OF_ETH_DST[[]],move:OXM_OF_PKT_REG0[[0..47]]->NXM_OF_ETH_SRC[[]],move:NXM_OF_IN_PORT[[]]->NXM_NX_REG3[[0..15]],load:0->NXM_OF_IN_PORT[[]],output:NXM_NX_REG3[[0..15]] +table=10 priority=0 action=drop +]) + +AT_CHECK([ovs-ofctl --bundle add-flows br0 flows.txt]) + +rm p0.pcap +OVS_DAEMONIZE([tcpdump -n -U -i ovs-p0 -w p0.pcap], [tcpdump.pid]) +sleep 1 + +dnl UDP packets from ns0->ns1 should solicit "destination unreachable" response. +NS_CHECK_EXEC([at_ns0], [bash -c "echo a | nc $NC_EOF_OPT -u 10.1.1.2 10000"]) + +dnl Flush conntrack state. +dnl To verify related packets are handled exactly the same as before flushing. +AT_CHECK([ovs-appctl dpctl/flush-conntrack], [0]) + +dnl Solicit another "destination unreachable" response. +dnl To verify that after flushing, the same openflow rules are matched. +NS_CHECK_EXEC([at_ns0], [bash -c "echo a | nc $NC_EOF_OPT -u 10.1.1.2 10000"]) + +AT_CHECK([ovs-appctl revalidator/purge], [0]) +AT_CHECK([ovs-ofctl -O OpenFlow15 dump-flows br0 | ofctl_strip | sort | grep -v drop], [0], [dnl + n_packets=1, n_bytes=42, priority=10,arp actions=NORMAL + n_packets=2, n_bytes=144, ct_state=+rel+trk,icmp,in_port=2,nw_dst=10.1.1.254 actions=ct(commit,table=1,nat) + n_packets=2, n_bytes=84, priority=100,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[[]]->NXM_NX_REG2[[]],resubmit(,8),goto_table:10 + n_packets=2, n_bytes=88, ct_state=+trk,ip,in_port=1 actions=ct(commit,nat(src=10.1.1.254)),output:2 + n_packets=4, n_bytes=232, ct_state=-trk,ip actions=ct(table=0) + table=1, ct_state=+rel+trk,icmp,in_port=2,nw_dst=10.1.1.254 actions=goto_table:2 + table=1, n_packets=2, n_bytes=144, ct_state=+rel+trk,icmp,in_port=2,nw_src=10.1.1.2,nw_dst=10.1.1.1 actions=output:1 + table=10, n_packets=1, n_bytes=42, priority=10,arp,arp_op=1 actions=set_field:2->arp_op,move:NXM_NX_ARP_SHA[[]]->NXM_NX_ARP_THA[[]],move:OXM_OF_PKT_REG0[[0..47]]->NXM_NX_ARP_SHA[[]],move:NXM_OF_ARP_SPA[[]]->NXM_OF_ARP_TPA[[]],move:NXM_NX_REG2[[]]->NXM_OF_ARP_SPA[[]],move:NXM_OF_ETH_SRC[[]]->NXM_OF_ETH_DST[[]],move:OXM_OF_PKT_REG0[[0..47]]->NXM_OF_ETH_SRC[[]],move:NXM_OF_IN_PORT[[]]->NXM_NX_REG3[[0..15]],set_field:0->in_port,output:NXM_NX_REG3[[0..15]] + table=10, n_packets=1, n_bytes=42, priority=100,arp,reg0=0,reg1=0 actions=NORMAL + table=8, n_packets=1, n_bytes=42, priority=0 actions=set_field:0->xreg0 + table=8, n_packets=1, n_bytes=42, reg2=0xa0101f0/0xfffffff0 actions=set_field:0x808888888888->xreg0 +OFPST_FLOW reply (OF1.5): +]) + +AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(10.1.1.2) | sed -e 's/dst=10.1.1.2[[45]][[0-9]]/dst=10.1.1.2XX/'], [0], [dnl +udp,orig=(src=10.1.1.1,dst=10.1.1.2,sport=<cleared>,dport=<cleared>),reply=(src=10.1.1.2,dst=10.1.1.2XX,sport=<cleared>,dport=<cleared>) +]) + +AT_CHECK([tcpdump -n -v "icmp" -r p0.pcap 2>/dev/null | grep -E 'wrong|bad'], [1], [ignore-nolog]) + +OVS_TRAFFIC_VSWITCHD_STOP +AT_CLEANUP + dnl CHECK_FTP_NAT(TITLE, IP_ADDR, FLOWS, CT_DUMP) dnl dnl Checks the implementation of conntrack with FTP ALGs in combination with
Linux kernel commit ebddb1404900 ("net: move the nat function to nf_nat_ovs for ovs and tc") introduced a regression into the kernel datapath which prevented the openvswitch match key from being updated when nat was undone for packets in the related conntrack state. This issue caused these packets (usually ICMP/ICMPv6 error packets) to match the wrong openflow rule. This issue was fixed in linux kernel commit e6345d2824a3 ("netfilter: nf_nat: fix action not being set for all ct states"). This test will reproduce the issue and fail for kernel versions v6.2 to v6.6, and will pass on earlier kernel versions where the issue wasn't present, or on later kernel versions that have the fix applied. Link: https://lore.kernel.org/netdev/20231221224311.130319-1-brad@faucet.nz/ Suggested-by: Aaron Conole <aconole@redhat.com> Signed-off-by: Brad Cowie <brad@faucet.nz> --- tests/system-traffic.at | 88 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+)