From patchwork Wed Sep 2 23:44:03 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Pfaff X-Patchwork-Id: 513701 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (li376-54.members.linode.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id A4B0E1401F0 for ; Thu, 3 Sep 2015 09:45:05 +1000 (AEST) Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id A69D2108AD; Wed, 2 Sep 2015 16:44:29 -0700 (PDT) X-Original-To: dev@openvswitch.org Delivered-To: dev@openvswitch.org Received: from mx1e4.cudamail.com (mx1.cudamail.com [69.90.118.67]) by archives.nicira.com (Postfix) with ESMTPS id 6FE611087C for ; Wed, 2 Sep 2015 16:44:28 -0700 (PDT) Received: from bar2.cudamail.com (unknown [192.168.21.12]) by mx1e4.cudamail.com (Postfix) with ESMTPS id DD2ED1E0442 for ; Wed, 2 Sep 2015 17:44:27 -0600 (MDT) X-ASG-Debug-ID: 1441237467-03dc53534098ce0001-byXFYA Received: from mx1-pf1.cudamail.com ([192.168.24.1]) by bar2.cudamail.com with ESMTP id xGGaPYDYPHgMxhvl (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 02 Sep 2015 17:44:27 -0600 (MDT) X-Barracuda-Envelope-From: blp@nicira.com X-Barracuda-RBL-Trusted-Forwarder: 192.168.24.1 Received: from unknown (HELO mail-pa0-f43.google.com) (209.85.220.43) by mx1-pf1.cudamail.com with ESMTPS (RC4-SHA encrypted); 2 Sep 2015 23:44:26 -0000 Received-SPF: unknown (mx1-pf1.cudamail.com: Multiple SPF records returned) X-Barracuda-RBL-Trusted-Forwarder: 209.85.220.43 Received: by padhy1 with SMTP id hy1so26332853pad.1 for ; Wed, 02 Sep 2015 16:44:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PK14bDvNN+5YU2OfdE0LkupD1UWSurkS71x7NtoInjc=; b=D1qD7cL/LnV/ETf0S80GEhtBjhtnI5vCb1NWd2daUYqRZjGN5nsreGTTrsO/BY6xdC 6ciieEnhi6t0OEB+k7FLt4K1AlsUdhs4oSPc9So2axMQnC8RCIsQ0j7jwHflyo8K3DMF 79EdcEW6QDKDfqN4y5NiMitsxb+HSWZgsG1BIo5762mXYmQQawNT7+hfQw26G4T5fHA1 lSSts23VKqq1kvMNYAWYoSddNVj/QtRrFCVO/4qZEYbqzHt+Mv/NUuMhSsAfXePaMQwR kp8z09dS1viwRKoIvityr+z7sJf5ghrEdx/JtDXtcMxKECXHMJIXW+QzQoXDHWx4tFee jdgA== X-Gm-Message-State: ALoCoQnf55kv+Wbo2mANTGOsMfwidrJaQrDU5bXBtGjSkE77IilxMhwIxFIN838XintebRuhw/uV X-Received: by 10.66.144.40 with SMTP id sj8mr60094858pab.55.1441237466075; Wed, 02 Sep 2015 16:44:26 -0700 (PDT) Received: from sigabrt.benpfaff.org ([208.91.2.4]) by smtp.gmail.com with ESMTPSA id f5sm23046151pas.23.2015.09.02.16.44.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 02 Sep 2015 16:44:24 -0700 (PDT) X-CudaMail-Envelope-Sender: blp@nicira.com X-Barracuda-Apparent-Source-IP: 208.91.2.4 From: Ben Pfaff To: dev@openvswitch.org X-CudaMail-Whitelist-To: dev@openvswitch.org X-CudaMail-MID: CM-E1-901091859 X-CudaMail-DTE: 090215 X-CudaMail-Originating-IP: 209.85.220.43 Date: Wed, 2 Sep 2015 16:44:03 -0700 X-ASG-Orig-Subj: [##CM-E1-901091859##][PATCH v2 06/14] ovn: Implement basic end-to-end full mesh test. Message-Id: <1441237451-17940-6-git-send-email-blp@nicira.com> X-Mailer: git-send-email 2.1.3 In-Reply-To: <1441237451-17940-1-git-send-email-blp@nicira.com> References: <1441237451-17940-1-git-send-email-blp@nicira.com> X-Barracuda-Connect: UNKNOWN[192.168.24.1] X-Barracuda-Start-Time: 1441237467 X-Barracuda-Encrypted: DHE-RSA-AES256-SHA X-Barracuda-URL: https://web.cudamail.com:443/cgi-mod/mark.cgi X-ASG-Whitelist: Header =?UTF-8?B?eFwtY3VkYW1haWxcLXdoaXRlbGlzdFwtdG8=?= X-Virus-Scanned: by bsmtpd at cudamail.com X-Barracuda-BRTS-Status: 1 Cc: Ben Pfaff Subject: [ovs-dev] [PATCH v2 06/14] ovn: Implement basic end-to-end full mesh test. X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" This is a really basic test of the OVN features. It verifies that basic L2 connectivity works as expected over a 3-hypervisor setup with 3 VMs per hypervisor and all 9 VMs on a single logical switch. I wanted to test stateless ACLs also but there's no simple way to set them up with ovn-nbctl. I guess that points toward a future direction! The infrastructure added by this patch, which is based on similar code from ovs-sim, should be useful as a basis for later and more advanced OVN end-to-end tests. Signed-off-by: Ben Pfaff --- tests/automake.mk | 2 +- tests/ofproto-macros.at | 167 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/ovn.at | 145 ++++++++++++++++++++++++++++++++++++++++- tests/ovs-macros.at | 21 ++++-- 4 files changed, 328 insertions(+), 7 deletions(-) diff --git a/tests/automake.mk b/tests/automake.mk index 32f757b..f7e84a0 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -109,7 +109,7 @@ SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite DISTCLEANFILES += tests/atconfig tests/atlocal -AUTOTEST_PATH = utilities:vswitchd:ovsdb:vtep:tests:$(PTHREAD_WIN32_DIR_DLL):ovn:ovn/controller-vtep:ovn/northd:ovn/utilities +AUTOTEST_PATH = utilities:vswitchd:ovsdb:vtep:tests:$(PTHREAD_WIN32_DIR_DLL):ovn:ovn/controller-vtep:ovn/northd:ovn/utilities:ovn/controller check-local: tests/atconfig tests/atlocal $(TESTSUITE) $(SHELL) '$(TESTSUITE)' -C tests AUTOTEST_PATH=$(AUTOTEST_PATH) $(TESTSUITEFLAGS) diff --git a/tests/ofproto-macros.at b/tests/ofproto-macros.at index da9990c..29e9710 100644 --- a/tests/ofproto-macros.at +++ b/tests/ofproto-macros.at @@ -47,6 +47,173 @@ s/No error/Success/ parse_listening_port () { sed -n 's/.*0:.*: listening on port \([0-9]*\)$/\1/p' }] + +start_daemon () { + "$@" -vconsole:off --detach --no-chdir --pidfile --log-file + pid=`cat "$OVS_RUNDIR"/$1.pid` + on_exit "kill $pid" +} + +# sim_add SANDBOX +# +# Starts a new simulated Open vSwitch instance named SANDBOX. Files related to +# the instance, such as logs, databases, sockets, and pidfiles, are created in +# a subdirectory of the main test directory also named SANDBOX. Afterward, the +# "as" command (see below) can be used to run Open vSwitch utilities in the +# context of the new sandbox. +# +# The new sandbox starts out without any bridges. Use ovs-vsctl in the context +# of the new sandbox to create a bridge, e.g.: +# +# sim_add hv0 # Create sandbox hv0. +# as hv0 # Set hv0 as default sandbox. +# ovs-vsctl add-br br0 # Add bridge br0 inside hv0. +# +# or: +# +# sim_add hv0 +# as hv0 ovs-vsctl add-br br0 +sims= +sim_add () { + echo "adding simulator '$1'" + + sims="$sims $1" + + # Create sandbox. + local d="$ovs_base"/$1 + mkdir "$d" || return 1 + ovs_setenv $1 + + # Create database and start ovsdb-server. + : > "$d"/.conf.db.~lock~ + as $1 ovsdb-tool create "$d"/conf.db "$abs_top_srcdir"/vswitchd/vswitch.ovsschema || return 1 + as $1 start_daemon ovsdb-server --remote=punix:"$d"/db.sock || return 1 + + # Initialize database. + as $1 ovs-vsctl --no-wait -- init || return 1 + + # Start ovs-vswitchd + as $1 start_daemon ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif +} + +# "as $1" sets the OVS_*DIR environment variables to point to $ovs_base/$1. +# +# "as $1 COMMAND..." sets those variables in a subshell and invokes COMMAND +# there. +as() { + if test "X$1" != X; then + (ovs_setenv $1; shift; $@) + else + ovs_setenv $1 + fi +} + +# ovn_start +# +# Creates and initializes ovn-sb and ovn-nb databases and starts their +# ovsdb-server instance, and starts ovn-northd running against them. +ovn_start () { + for db in ovn-sb ovn-nb; do + echo "creating $db database" + d=$ovs_base/$db + mkdir "$d" || return 1 + : > "$d"/.$db.db.~lock~ + as $db ovsdb-tool create "$d"/$db.db "$abs_top_srcdir"/ovn/$db.ovsschema + as $db start_daemon ovsdb-server --remote=punix:"$d"/$db.sock "$d"/$db.db + done + + OVN_NB_DB=unix:$ovs_base/ovn-nb/ovn-nb.sock; export OVN_NB_DB + + echo "starting ovn-northd" + mkdir "$ovs_base"/northd + as northd start_daemon ovn-northd \ + --ovnnb-db=unix:"$ovs_base"/ovn-nb/ovn-nb.sock \ + --ovnsb-db=unix:"$ovs_base"/ovn-sb/ovn-sb.sock +} + +# Interconnection networks. +# +# When multiple sandboxed Open vSwitch instances exist, one will inevitably +# want to connect them together. These commands allow for that. Conceptually, +# an interconnection network is a switch for which these functions make it easy +# to plug into other switches in other sandboxed Open vSwitch instances. +# Interconnection networks are implemented as bridges in a switch named "main", +# so to use interconnection networks please avoid working with that switch +# directly. + +# net_add NETWORK +# +# Creates a new interconnection network named NETWORK. +net_add () { + test -d "$ovs_base"/main || sim_add main || return 1 + as main ovs-vsctl add-br "$1" +} + +# net_attach NETWORK BRIDGE +# +# Adds a new port to BRIDGE in the default sandbox (as set with as()) and plugs +# it into the NETWORK interconnection network. NETWORK must already have been +# created by a previous invocation of net_add. The default sandbox must not be +# "main". +net_attach () { + local net=$1 bridge=$2 + + local port=${sandbox}_$bridge + as main ovs-vsctl \ + -- add-port $net $port \ + -- set Interface $port options:pstream="punix:$ovs_base/main/$port.sock" options:rxq_pcap="$ovs_base/main/$port-rx.pcap" options:tx_pcap="$ovs_base/main/$port-tx.pcap" \ + || return 1 + + ovs-vsctl \ + -- set Interface $bridge options:tx_pcap="$ovs_base/$sandbox/$bridge-tx.pcap" options:rxq_pcap="$ovs_base/$sandbox/$bridge-rx.pcap" \ + -- add-port $bridge ${bridge}_$net \ + -- set Interface ${bridge}_$net options:stream="unix:$ovs_base/main/$port.sock" options:rxq_pcap="$ovs_base/$sandbox/${bridge}_$net-rx.pcap" options:tx_pcap="$ovs_base/$sandbox/${bridge}_$net-tx.pcap" \ + || return 1 +} + +# ovn_attach NETWORK BRIDGE IP [MASKLEN] +# +# First, this command attaches BRIDGE to interconnection network NETWORK, just +# like "net_attach NETWORK BRIDGE". Second, it configures (simulated) IP +# address IP (with network mask length MASKLEN, which defaults to 24) on +# BRIDGE. Finally, it configures the Open vSwitch database to work with OVN +# and starts ovn-controller. +ovn_attach() { + local net=$1 bridge=$2 ip=$3 masklen=${4-24} + net_attach $net $bridge || return 1 + + mac=`ovs-vsctl get Interface $bridge mac_in_use | sed s/\"//g` + arp_table="$arp_table $sandbox,$bridge,$ip,$mac" + ovs-appctl netdev-dummy/ip4addr $bridge $ip/$masklen >/dev/null || return 1 + ovs-appctl ovs/route/add $ip/$masklen $bridge >/dev/null || return 1 + ovs-vsctl \ + -- set Open_vSwitch . external-ids:system-id=$sandbox \ + -- set Open_vSwitch . external-ids:ovn-remote=unix:$ovs_base/ovn-sb/ovn-sb.sock \ + -- set Open_vSwitch . external-ids:ovn-encap-type=geneve \ + -- set Open_vSwitch . external-ids:ovn-encap-ip=$ip \ + -- add-br br-int \ + -- set bridge br-int fail-mode=secure other-config:disable-in-band=true \ + || return 1 + start_daemon ovn-controller || return 1 +} + +# ovn_populate_arp +# +# This pre-populates the ARP tables of all of the OVN instances that have been +# started with ovn_attach(). That means that packets sent from one hypervisor +# to another never get dropped or delayed by ARP resolution, which makes +# testing easier. +ovn_populate_arp() { + for e1 in $arp_table; do + set `echo $e1 | sed 's/,/ /g'`; sb1=$1 br1=$2 ip=$3 mac=$4 + for e2 in $arp_table; do + set `echo $e2 | sed 's/,/ /g'`; sb2=$1 br2=$2 + if test $sb1,$br1 != $sb2,$br2; then + as $sb2 ovs-appctl tnl/arp/set $br2 $ip $mac + fi + done + done +} m4_divert_pop([PREPARE_TESTS]) m4_define([STRIP_XIDS], [[sed 's/ (xid=0x[0-9a-fA-F]*)//']]) diff --git a/tests/ovn.at b/tests/ovn.at index d1696de..8e442fa 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -1,4 +1,4 @@ -AT_BANNER([OVN]) +AT_BANNER([OVN components]) AT_SETUP([ovn -- lexer]) dnl For lines without =>, input and expected output are identical. @@ -423,3 +423,146 @@ sed 's/ =>.*//' test-cases.txt > input.txt sed 's/.* => //' test-cases.txt > expout AT_CHECK([ovstest test-ovn parse-actions < input.txt], [0], [expout]) AT_CLEANUP + +AT_BANNER([OVN end-to-end tests]) + +AT_SETUP([ovn -- 3 HVs, 3 VIFs/HV, 1 logical switch]) +AT_SKIP_IF([test $HAVE_PYTHON = no]) +ovn_start + +# Create hypervisors hv[123]. +# Add vif1[123] to hv1, vif2[123] to hv2, vif3[123]. +# Add all of the vifs to a single logical switch lsw0. +# Turn on port security on all the vifs except vif[123]1. +# Make vif13, vif2[23], vif3[123] destinations for unknown MACs. +ovn-nbctl lswitch-add lsw0 +net_add n1 +for i in 1 2 3; do + sim_add hv$i + as hv$i + ovs-vsctl add-br br-phys + ovn_attach n1 br-phys 192.168.0.$i + + for j in 1 2 3; do + ovs-vsctl add-port br-int vif$i$j -- set Interface vif$i$j external-ids:iface-id=lp$i$j options:tx_pcap=hv$i/vif$i$j-tx.pcap options:rxq_pcap=hv$i/vif$i$j-rx.pcap ofport-request=$i$j + ovn-nbctl lport-add lsw0 lp$i$j + if test $j = 1; then + ovn-nbctl lport-set-macs lp$i$j f0:00:00:00:00:$i$j unknown + else + ovn-nbctl lport-set-macs lp$i$j f0:00:00:00:00:$i$j + ovn-nbctl lport-set-port-security lp$i$j f0:00:00:00:00:$i$j + fi + done +done + +# Pre-populate the hypervisors' ARP tables so that we don't lose any +# packets for ARP resolution (native tunneling doesn't queue packets +# for ARP resolution). +ovn_populate_arp + +# Allow some time for ovn-northd and ovn-controller to catch up. +# XXX This should be more systematic. +sleep 1 +ovn-sbctl --db=unix:`pwd`/ovn-sb/ovn-sb.sock dump-flows -- list multicast_group + +# test_packet INPORT DST SRC ETHTYPE OUTPORT... +# +# This shell function causes a packet to be received on INPORT. The packet's +# content has Ethernet destination DST and source SRC (each exactly 12 hex +# digits) and Ethernet type ETHTYPE (4 hex digits). The OUTPORTs (zero or +# more) list the VIFs on which the packet should be received. INPORT and the +# OUTPORTs are specified as lport numbers, e.g. 11 for vif11. +trim_zeros() { + sed 's/\(00\)\{1,\}$//' +} +for i in 1 2 3; do + for j in 1 2 3; do + : > $i$j.expected + done +done +test_packet() { + local inport=$1 packet=$2$3$4; shift; shift; shift; shift + hv=hv`echo $inport | sed 's/^\(.\).*/\1/'` + vif=vif$inport + as $hv ovs-appctl netdev-dummy/receive $vif $packet + for outport; do + echo $packet | trim_zeros >> $outport.expected + done +} + +# Send packets between all pairs of source and destination ports: +# +# 1. Unicast packets are delivered to exactly one lport (except that packets +# destined to their input ports are dropped). +# +# 2. Broadcast and multicast are delivered to all lports except the input port. +# +# 3. When port security is turned on, the lswitch drops packets from the wrong +# MAC address. +# +# 4. The lswitch drops all packets with a VLAN tag. +# +# 5. The lswitch drops all packets with a multicast source address. (This only +# affects behavior when port security is turned off, since otherwise port +# security would drop the packet anyway.) +# +# 6. The lswitch delivers packets with an unknown destination to lports with +# "unknown" among their MAC addresses (and port security disabled). +for is in 1 2 3; do + for js in 1 2 3; do + s=$is$js + bcast= + unknown= + for id in 1 2 3; do + for jd in 1 2 3; do + d=$id$jd + impersonate= + if test $d != $s; then + unicast=$d + bcast="$bcast $d" + if test $js = 1; then + impersonate=$d + fi + if test $jd = 1; then + unknown="$unknown $d" + fi + else + unicast= + fi + test_packet $s f000000000$d f000000000$s $s$d $unicast #1 + test_packet $s f000000000$d f00000000055 55$d $impersonate #3 + test_packet $s f000000000$d f00000000055 810000091234 #4 + test_packet $s f000000000$d 0100000000$s $s$d #5 + done + done + + # Broadcast and multicast. + test_packet $s ffffffffffff f000000000$s ${s}ff $bcast #2 + test_packet $s 010000000000 f000000000$s ${s}ff $bcast #2 + if test $js = 1; then + bcast_impersonate=$bcast + else + bcast_impersonate= + fi + test_packet $s 010000000000 f00000000044 44ff $bcast_impersonate #3 + + test_packet $s f0000000ffff f000000000$s ${s}66 $unknown #6 + done +done + +# Allow some time for packet forwarding. +# XXX This can be improved. +sleep 1 + +# Now check the packets actually received against the ones expected. +for i in 1 2 3; do + for j in 1 2 3; do + file=hv$i/vif$i$j-tx.pcap + echo $file + $PYTHON "$top_srcdir/utilities/ovs-pcap.in" $file | trim_zeros > $i$j.packets + cp $i$j.expected expout + AT_CHECK([cat $i$j.packets], [0], [expout]) + echo + done +done +AT_CLEANUP diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at index 058830b..541b042 100644 --- a/tests/ovs-macros.at +++ b/tests/ovs-macros.at @@ -19,11 +19,22 @@ ovs_init() { ovs_base=`pwd` trap '. "$ovs_base/cleanup"' 0 : > cleanup - OVS_RUNDIR=$ovs_base; export OVS_RUNDIR - OVS_LOGDIR=$ovs_base; export OVS_LOGDIR - OVS_DBDIR=$ovs_base; export OVS_DBDIR - OVS_SYSCONFDIR=$ovs_base; export OVS_SYSCONFDIR - OVS_PKGDATADIR=$ovs_base; export OVS_PKGDATADIR + ovs_setenv +} + +# With no parameter or an empty parameter, sets the OVS_*DIR +# environment variables to point to $ovs_base, the base directory in +# which the test is running. +# +# With a parameter, sets them to $ovs_base/$1. +ovs_setenv() { + sandbox=$1 + ovs_dir=$ovs_base${1:+/$1} + OVS_RUNDIR=$ovs_dir; export OVS_RUNDIR + OVS_LOGDIR=$ovs_dir; export OVS_LOGDIR + OVS_DBDIR=$ovs_dir; export OVS_DBDIR + OVS_SYSCONFDIR=$ovs_dir; export OVS_SYSCONFDIR + OVS_PKGDATADIR=$ovs_dir; export OVS_PKGDATADIR } ovs_wait () {