diff mbox series

[ovs-dev,2/2] Add fake multinode system tests.

Message ID 20221201191314.1916813-1-numans@ovn.org
State Changes Requested
Headers show
Series CI: Add ovn-fake-multinode jobs | expand

Checks

Context Check Description
ovsrobot/apply-robot fail apply and check: fail

Commit Message

Numan Siddique Dec. 1, 2022, 7:13 p.m. UTC
From: Numan Siddique <numans@ovn.org>

This patch adds a simple system test using ovn-fake-multinode
setup.  The tests can be run as - 'make check-multinode'

Before running these tests, user should deploy fake_multinode setup
by running 'ovn_cluster.sh start'.

This test suite is also triggered for the newly added fake multinode CI
job.

The fake multinode system tests suite can be enhanced further for new
features and to cover multi node scenarios.

Signed-off-by: Numan Siddique <numans@ovn.org>
---
 .../workflows/ovn-fake-multinode-tests.yml    |  57 ++++++
 tests/automake.mk                             |  28 ++-
 tests/multinode-macros.at                     | 189 ++++++++++++++++++
 tests/multinode-testsuite.at                  |  27 +++
 tests/multinode.at                            |  74 +++++++
 5 files changed, 374 insertions(+), 1 deletion(-)
 create mode 100644 tests/multinode-macros.at
 create mode 100644 tests/multinode-testsuite.at
 create mode 100644 tests/multinode.at

Comments

0-day Robot Dec. 1, 2022, 7:38 p.m. UTC | #1
Bleep bloop.  Greetings Numan Siddique, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


git-am:
error: sha1 information is lacking or useless (.github/workflows/ovn-fake-multinode-tests.yml).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Add fake multinode system tests.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".


Please check this out.  If you feel there has been an error, please email aconole@redhat.com

Thanks,
0-day Robot
Ales Musil Jan. 23, 2023, 4:36 p.m. UTC | #2
Hi Numan,

On Thu, Dec 1, 2022 at 8:13 PM <numans@ovn.org> wrote:

> From: Numan Siddique <numans@ovn.org>
>
> This patch adds a simple system test using ovn-fake-multinode
> setup.  The tests can be run as - 'make check-multinode'
>
> Before running these tests, user should deploy fake_multinode setup
> by running 'ovn_cluster.sh start'.
>
> This test suite is also triggered for the newly added fake multinode CI
> job.
>
> The fake multinode system tests suite can be enhanced further for new
> features and to cover multi node scenarios.
>
> Signed-off-by: Numan Siddique <numans@ovn.org>
> ---
>  .../workflows/ovn-fake-multinode-tests.yml    |  57 ++++++
>  tests/automake.mk                             |  28 ++-
>  tests/multinode-macros.at                     | 189 ++++++++++++++++++
>  tests/multinode-testsuite.at                  |  27 +++
>  tests/multinode.at                            |  74 +++++++
>  5 files changed, 374 insertions(+), 1 deletion(-)
>  create mode 100644 tests/multinode-macros.at
>  create mode 100644 tests/multinode-testsuite.at
>  create mode 100644 tests/multinode.at
>
> diff --git a/.github/workflows/ovn-fake-multinode-tests.yml
> b/.github/workflows/ovn-fake-multinode-tests.yml
> index 3727b9835..c24eb2965 100644
> --- a/.github/workflows/ovn-fake-multinode-tests.yml
> +++ b/.github/workflows/ovn-fake-multinode-tests.yml
> @@ -86,6 +86,12 @@ jobs:
>        OS_IMAGE: "fedora:36"
>        CENTRAL_IMAGE: ${{ matrix.cfg.central_image }}
>        ENABLE_SSL: no
> +      CC: gcc
> +      OPTS: "--disable-ssl"
> +      dependencies: |
> +        automake libtool gcc bc libjemalloc2 libjemalloc-dev    \
> +        libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev  \
> +        selinux-policy-dev ncat python3-scapy isc-dhcp-server
>        # https://github.com/actions/runner-images/issues/6282
>        XDG_RUNTIME_DIR: ''
>
> @@ -145,6 +151,57 @@ jobs:
>          cd ovn-fake-multinode
>          sudo ./.ci/test_basic.sh
>
> +    - name: install required dependencies
> +      run:  sudo apt install -y ${{ env.dependencies }}
>

nit: It would be better to install dependencies before other steps.


> +
> +    - name: install libunbound libunwind
> +      run:  sudo apt install -y libunbound-dev libunwind-dev
> +
> +    - name: update PATH
> +      run:  |
> +        echo "$HOME/bin"        >> $GITHUB_PATH
> +        echo "$HOME/.local/bin" >> $GITHUB_PATH
> +
> +    - name: set up python
> +      uses: actions/setup-python@v4
> +      with:
> +        python-version: '3.x'
> +
> +    - name: Check out ovn
> +      uses: actions/checkout@v3
> +      with:
> +        path: 'ovn'
> +        submodules: recursive
> +
> +    - name: Build OVN and trigger fake-multinode system tests
> +      run: |
> +        set -x
> +        cd ovn
> +        ./.ci/linux-prepare.sh
> +        ./.ci/linux-build.sh
> +        sudo make check-multinode
> +
> +    - name: copy logs on failure
> +      if: failure() || cancelled()
> +      run: |
> +        # upload-artifact@v3 throws exceptions if it tries to upload
> socket
> +        # files and we could have some socket files in testsuite.dir.
> +        # Also, upload-artifact@v3 doesn't work well enough with
> wildcards.
> +        # So, we're just archiving everything here to avoid any issues.
> +        mkdir logs
> +        cp ovn/config.log ./logs/
> +        # multinode tests are run as root, need to adjust permissions.
> +        sudo chmod -R +r ovn/tests/multinode-testsuite.dir.* || true
> +        cp -r ovn/tests/multinode-testsuite.dir.* ./logs/ || true
> +        tar -czvf logs.tgz logs/
> +
> +    - name: upload logs on failure
> +      if: failure() || cancelled()
> +      uses: actions/upload-artifact@v3
> +      with:
> +        name: logs-linux-${{ join(matrix.cfg.*, '-') }}
> +        path: logs.tgz
> +
>      - name: Stop cluster
>        run: |
>          cd ovn-fake-multinode
> diff --git a/tests/automake.mk b/tests/automake.mk
> index dce9c9108..88cda5f20 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -5,10 +5,12 @@ EXTRA_DIST += \
>         $(SYSTEM_KMOD_TESTSUITE_AT) \
>         $(SYSTEM_USERSPACE_TESTSUITE_AT) \
>         $(PERF_TESTSUITE_AT) \
> +       $(MULTINODE_TESTSUITE_AT) \
>         $(TESTSUITE) \
>         $(SYSTEM_KMOD_TESTSUITE) \
>         $(SYSTEM_USERSPACE_TESTSUITE) \
>         $(PERF_TESTSUITE) \
> +       $(MULTINODE_TESTSUITE) \
>         tests/atlocal.in \
>         $(srcdir)/package.m4 \
>         $(srcdir)/tests/testsuite \
> @@ -61,6 +63,11 @@ PERF_TESTSUITE_AT = \
>         tests/perf-testsuite.at \
>         tests/perf-northd.at
>
> +MULTINODE_TESTSUITE_AT = \
> +       tests/multinode-testsuite.at \
> +       tests/multinode-macros.at \
> +       tests/multinode.at
> +
>  check_SCRIPTS += tests/atlocal
>
>  TESTSUITE = $(srcdir)/tests/testsuite
> @@ -72,7 +79,9 @@ PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite
>  PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir
>  PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results
>  DISTCLEANFILES += tests/atconfig tests/atlocal
> -
> +MULTINODE_TESTSUITE = $(srcdir)/tests/multinode-testsuite
> +MULTINODE_TESTSUITE_DIR =
> $(abs_top_builddir)/tests/multinode-testsuite.dir
> +MULTINODE_TESTSUITE_RESULTS = $(MULTINODE_TESTSUITE_DIR)/results
>  AUTOTEST_PATH =
> $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic
>
>  export ovs_srcdir
> @@ -82,6 +91,7 @@ check-local:
>         "$$@" $(TESTSUITEFLAGS) || \
>         (test -z "$$(find $(TESTSUITE_DIR) -name 'sanitizers.*')" && \
>          test X'$(RECHECK)' = Xyes && "$$@" --recheck)
> +
>

nit: Extra empty line?


>
>  # Python Coverage support.
>  # Requires coverage.py http://nedbatchelder.com/code/coverage/.
> @@ -197,6 +207,18 @@ check-perf: all
>         @echo
>         @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)"
>
> +check-multinode: all
> +       @mkdir -p $(MULTINODE_TESTSUITE_DIR)
> +       @echo  > $(MULTINODE_TESTSUITE_RESULTS)
> +       set $(SHELL) '$(MULTINODE_TESTSUITE)' -C tests
> AUTOTEST_PATH='$(AUTOTEST_PATH)'; \
> +       "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@"
> --recheck)
> +       @echo
> +       @echo  '## -------------------- ##'
> +       @echo  '##  Multinode test Results ##'
> +       @echo  '## -------------------- ##'
> +       @cat $(MULTINODE_TESTSUITE_RESULTS)
> +       @echo
> +       @echo "Results can be found in $(MULTINODE_TESTSUITE_RESULTS)"
>
>  AUTOTEST = $(AUTOM4TE) --language=autotest
>
> @@ -223,6 +245,10 @@ $(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT)
> $(COMMON_MACROS_AT)
>         $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
>         $(AM_V_at)mv $@.tmp $@
>
> +$(MULTINODE_TESTSUITE): package.m4 $(MULTINODE_TESTSUITE_AT)
> $(COMMON_MACROS_AT)
> +       $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
> +       $(AM_V_at)mv $@.tmp $@
> +
>  # The `:;' works around a Bash 3.2 bug when the output is not writeable.
>  $(srcdir)/package.m4: $(top_srcdir)/configure.ac
>         $(AM_V_GEN):;{ \
> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
> new file mode 100644
> index 000000000..88a3816eb
> --- /dev/null
> +++ b/tests/multinode-macros.at
> @@ -0,0 +1,189 @@
> +#
> +#
> +
> +# M_NS_EXEC([fake_node], [namespace], [command])
> +#
> +# Execute 'command' in 'namespace'
> +m4_define([M_NS_EXEC],
> +    [podman exec $1 ip netns exec $2 $3])
>

Shouldn't we use -i every time we call "podman exec"?


> +
> +# M_NS_CHECK_EXEC(fake_node], [namespace], [command], other_params...)
>

nit: Missing [ before "fake_node]".


> +#
> +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s
> namespace'.
> +# 'other_params' as passed as they are to AT_CHECK.
> +m4_define([M_NS_CHECK_EXEC],
> +    [ AT_CHECK([M_NS_EXEC([$1], [$2], [$3])],
> m4_shift(m4_shift(m4_shift($@)))) ]
> +)
> +
> +OVS_START_SHELL_HELPERS
> +
> +m_as() {
> +    c=$1
> +    shift
> +    podman exec $c "$@"
> +}
> +
> +m_central_as () {
> +    podman exec ovn-central "$@"
> +}
> +
> +check_fake_multinode_setup() {
> +    check m_as ovn-central ovn-nbctl --wait=hv sync
> +    AT_CHECK([m_as ovn-chassis-1 ovn-appctl -t ovn-controller version],
> [0], [ignore])
> +    AT_CHECK([m_as ovn-chassis-2 ovn-appctl -t ovn-controller version],
> [0], [ignore])
> +    AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0],
> [ignore])
> +}
> +
> +cleanup_multinode_resources() {
> +    m_as ovn-central rm -f /etc/ovn/ovnnb_db.db
> +    m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd
> +    check m_as ovn-central ovn-nbctl --wait=hv sync
> +    for c in ovn-chassis-1 ovn-chassis-2 ovn-gw-1
> +    do
> +        m_as $c ovs-vsctl del-br br-int
> +        m_as $c ip --all netns delete
> +    done
> +}
> +
> +multinode_nbctl () {
> +    m_as ovn-central ovn-nbctl "$@"
> +}
> +
> +# count_rows TABLE [CONDITION...]
> +#
> +# Prints the number of rows in TABLE (that satisfy CONDITION).
> +# Uses the southbound db by default; set DB=nb for the northbound
> database.
> +m_count_rows() {
> +    local db=$(parse_db $1) table=$(parse_table $1); shift
> +    m_central_as ovn-${db}ctl --format=table --no-headings find $table
> "$@" | wc -l
> +}
> +
> +# check_row_count [DATABASE:]TABLE COUNT [CONDITION...]
> +#
> +# Checks that TABLE contains COUNT rows (that satisfy CONDITION).
> +# The default DATABASE is "sb".
> +m_check_row_count() {
> +    local db=$(parse_db $1) table=$(parse_table $1); shift
>
+    local count=$1; shift
> +    local found=$(m_count_rows $c $db:$table "$@")
> +    echo
> +    echo "Checking for $count rows in $db $table${1+ with $*}... found
> $found"
> +    if test "$count" != "$found"; then
> +        m_central_as ovn-${db}ctl list $table
> +        AT_FAIL_IF([:])
> +    fi
> +}
> +
> +# wait_row_count [DATABASE:]TABLE COUNT [CONDITION...]
> +#
> +# Waits until TABLE contains COUNT rows (that satisfy CONDITION).
> +# The default DATABASE is "sb".
> +m_wait_row_count() {
> +    local db=$(parse_db $1) table=$(parse_table $1); shift
> +    local count=$1; shift
> +    local a=$1 b=$2 c=$3 d=$4 e=$5
> +    echo "Waiting until $count rows in $db $table${1+ with $*}..."
> +    OVS_WAIT_UNTIL([test $count = $(m_count_rows $db:$table $a $b $c $d
> $e)],[
> +      echo "$db table $table has the following rows. $(m_count_rows
> $db:$table $a $b $c $d $e) rows match instead of expected $count:"
> +      m_central_as ovn-${db}ctl list $table])
> +}
> +
> +# multinode_wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]]
> +#
> +# Wait until all of the values of COLUMN in the rows of TABLE (that
> +# satisfy CONDITION) equal EXPECTED (ignoring order).
> +#
> +# The default DATABASE is "sb".
> +#
> +# COLUMN defaults to _uuid if unspecified.
> +m_wait_column() {
> +    local expected=$(for d in $1; do echo $d; done | sort)
> +    local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid};
> shift; shift; shift
> +    local a=$1 b=$2 c=$3 d=$4 e=$5
> +
> +    echo
> +    echo "Waiting until $column in $db $table${1+ with $*} is
> $expected..."
> +    OVS_WAIT_UNTIL([
> +      found=$(m_central_as ovn-${db}ctl --bare --columns $column find
> $table $a $b $c $d $e)
> +      found=$(for d in $found; do echo $d; done | sort)
> +      test "$expected" = "$found"
> +    ], [
> +      echo "$column in $db table $table has value $found, from the
> following rows:"
> +      m_central_as ovn-${db}ctl list $table])
> +}
> +
> +# fetch_column [DATABASE:]TABLE COLUMN [CONDITION...]
> +#
> +# Fetches and prints all the values of COLUMN in the rows of TABLE
> +# (that satisfy CONDITION), sorting the results lexicographically.
> +# The default DATABASE is "sb".
> +m_fetch_column() {
> +    local db=$(parse_db $1) table=$(parse_table $1) column=${2-_uuid};
> shift; shift
> +    # Using "echo" removes spaces and newlines.
> +    echo $(m_central_as ovn-${db}ctl --bare --columns $column find $table
> "$@" | sort)
> +}
> +
> +# check_column EXPECTED [DATABASE:]TABLE COLUMN [CONDITION...]
> +#
> +# Fetches all of the values of COLUMN in the rows of TABLE (that
> +# satisfy CONDITION), and compares them against EXPECTED (ignoring
> +# order).
> +#
> +# The default DATABASE is "sb".
> +m_check_column() {
> +    local expected=$1 db=$(parse_db $2) table=$(parse_table $2)
> column=${3-_uuid}; shift; shift; shift
> +    local found=$(m_central_as ovn-${db}ctl --bare --columns $column find
> $table "$@")
> +
> +    # Sort the expected and found values.
> +    local found=$(for d in $found; do echo $d; done | sort)
> +    local expected=$(for d in $expected; do echo $d; done | sort)
> +
> +    echo
> +    echo "Checking values in $db $table${1+ with $*} against $expected...
> found $found"
> +    if test "$found" != "$expected"; then
> +        m_central_as ovn-${db}ctl list $table
> +        AT_FAIL_IF([:])
> +    fi
> +}
> +
> +# wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]]
> +#
> +# Wait until all of the values of COLUMN in the rows of TABLE (that
> +# satisfy CONDITION) equal EXPECTED (ignoring order).
> +#
> +# The default DATABASE is "sb".
> +#
> +# COLUMN defaults to _uuid if unspecified.
> +m_wait_column() {
> +    local expected=$(for d in $1; do echo $d; done | sort)
> +    local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid};
> shift; shift; shift
> +    local a=$1 b=$2 c=$3 d=$4 e=$5
> +
> +    echo
> +    echo "Waiting until $column in $db $table${1+ with $*} is
> $expected..."
> +    OVS_WAIT_UNTIL([
> +      found=$(m_central_as ovn-${db}ctl --bare --columns $column find
> $table $a $b $c $d $e)
> +      found=$(for d in $found; do echo $d; done | sort)
> +      test "$expected" = "$found"
> +    ], [
> +      echo "$column in $db table $table has value $found, from the
> following rows:"
> +      m_central_as ovn-${db}ctl list $table])
> +}
> +
> +# wait_for_ports_up [PORT...]
> +#
> +# With arguments, waits for specified Logical_Switch_Ports to come up.
> +# Without arguments, waits for all "plain" and router
> +# Logical_Switch_Ports to come up.
> +m_wait_for_ports_up() {
> +    if test $# = 0; then
> +        m_wait_row_count nb:Logical_Switch_Port 0 up!=true type='""'
> +        m_wait_row_count nb:Logical_Switch_Port 0 up!=true type=router
> +    else
> +        for port; do
> +            m_wait_row_count nb:Logical_Switch_Port 1 up=true name=$port
> +        done
> +    fi
> +}
> +
> +OVS_END_SHELL_HELPERS
> diff --git a/tests/multinode-testsuite.at b/tests/multinode-testsuite.at
> new file mode 100644
> index 000000000..ea10b0276
> --- /dev/null
> +++ b/tests/multinode-testsuite.at
> @@ -0,0 +1,27 @@
> +AT_INIT
> +
> +AT_COPYRIGHT([Copyright (c) 2022 Red Hat,
> +
> +Licensed under the Apache License, Version 2.0 (the "License");
> +you may not use this file except in compliance with the License.
> +You may obtain a copy of the License at:
> +
> +    http://www.apache.org/licenses/LICENSE-2.0
> +
> +Unless required by applicable law or agreed to in writing, software
> +distributed under the License is distributed on an "AS IS" BASIS,
> +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +See the License for the specific language governing permissions and
> +limitations under the License.])
> +
> +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS])
> +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases])
> +
> +m4_include([tests/ovs-macros.at])
> +m4_include([tests/ovsdb-macros.at])
> +m4_include([tests/ofproto-macros.at])
> +m4_include([tests/ovn-macros.at])
> +m4_include([tests/system-common-macros.at])
> +m4_include([tests/multinode-macros.at])
> +
> +m4_include([tests/multinode.at])
> diff --git a/tests/multinode.at b/tests/multinode.at
> new file mode 100644
> index 000000000..754e488d6
> --- /dev/null
> +++ b/tests/multinode.at
> @@ -0,0 +1,74 @@
> +AT_BANNER([ovn multinode system tests using ovn-fake-multinode])
> +
> +AT_SETUP([ovn multinode basic test])
> +
> +# Check that ovn-fake-multinode setup is up and running
> +check_fake_multinode_setup
> +
> +# Delete the multinode NB and OVS resources before starting the test.
> +cleanup_multinode_resources
> +
> +# Test East-West switching
> +check multinode_nbctl ls-add sw0
> +check multinode_nbctl lsp-add sw0 sw0-port1
> +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03
> 10.0.0.3 1000::3"
> +check multinode_nbctl lsp-add sw0 sw0-port2
> +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04
> 10.0.0.4 1000::4"
> +
> +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1
> 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
> +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2
> 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a
> +
> +m_wait_for_ports_up
> +
> +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2
> 10.0.0.4 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +# Add ACLs to drop all traffic
> +check multinode_nbctl pg-add pg0 sw0-port1 sw0-port2
> +check multinode_nbctl acl-add pg0 to-lport 1001 "outport == @pg0 && ip4"
> drop
> +check multinode_nbctl --wait=hv sync
> +
> +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2
> 10.0.0.4], \
> +[1], [ignore])
> +
> +# Add ACLs to allow icmp traffic
> +check multinode_nbctl acl-add pg0 to-lport 1002 "outport == @pg0 && ip4
> && icmp" allow-related
> +check multinode_nbctl --wait=hv sync
> +
> +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2
> 10.0.0.4 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +
> +# Create the second logical switch with one port
> +check multinode_nbctl ls-add sw1
> +check multinode_nbctl lsp-add sw1 sw1-port1
> +check multinode_nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03
> 20.0.0.3 2000::3"
> +
> +# Create a logical router and attach both logical switches
> +check multinode_nbctl lr-add lr0
> +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
> 1000::a/64
> +check multinode_nbctl lsp-add sw0 sw0-lr0
> +check multinode_nbctl lsp-set-type sw0-lr0 router
> +check multinode_nbctl lsp-set-addresses sw0-lr0 router
> +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0
> +
> +check multinode_nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24
> 2000::a/64
> +check multinode_nbctl lsp-add sw1 sw1-lr0
> +check multinode_nbctl lsp-set-type sw1-lr0 router
> +check multinode_nbctl lsp-set-addresses sw1-lr0 router
> +check multinode_nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
> +
> +m_as ovn-chassis-2 /data/create_fake_vm.sh sw1-port1 sw1p1
> 40:54:00:00:00:03 20.0.0.3 24 20.0.0.1 2000::4/64 1000::a
> +
> +m_wait_for_ports_up sw1-port1
> +
> +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2
> 20.0.0.3 | FORMAT_PING], \
> +[0], [dnl
> +3 packets transmitted, 3 received, 0% packet loss, time 0ms
> +])
> +
> +AT_CLEANUP
> --
> 2.38.1
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
>
Other than that it looks good.

Thanks,
Ales
diff mbox series

Patch

diff --git a/.github/workflows/ovn-fake-multinode-tests.yml b/.github/workflows/ovn-fake-multinode-tests.yml
index 3727b9835..c24eb2965 100644
--- a/.github/workflows/ovn-fake-multinode-tests.yml
+++ b/.github/workflows/ovn-fake-multinode-tests.yml
@@ -86,6 +86,12 @@  jobs:
       OS_IMAGE: "fedora:36"
       CENTRAL_IMAGE: ${{ matrix.cfg.central_image }}
       ENABLE_SSL: no
+      CC: gcc
+      OPTS: "--disable-ssl"
+      dependencies: |
+        automake libtool gcc bc libjemalloc2 libjemalloc-dev    \
+        libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev  \
+        selinux-policy-dev ncat python3-scapy isc-dhcp-server
       # https://github.com/actions/runner-images/issues/6282
       XDG_RUNTIME_DIR: ''
 
@@ -145,6 +151,57 @@  jobs:
         cd ovn-fake-multinode
         sudo ./.ci/test_basic.sh
 
+    - name: install required dependencies
+      run:  sudo apt install -y ${{ env.dependencies }}
+
+    - name: install libunbound libunwind
+      run:  sudo apt install -y libunbound-dev libunwind-dev
+
+    - name: update PATH
+      run:  |
+        echo "$HOME/bin"        >> $GITHUB_PATH
+        echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+    - name: set up python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.x'
+
+    - name: Check out ovn
+      uses: actions/checkout@v3
+      with:
+        path: 'ovn'
+        submodules: recursive
+
+    - name: Build OVN and trigger fake-multinode system tests
+      run: |
+        set -x
+        cd ovn
+        ./.ci/linux-prepare.sh
+        ./.ci/linux-build.sh
+        sudo make check-multinode
+
+    - name: copy logs on failure
+      if: failure() || cancelled()
+      run: |
+        # upload-artifact@v3 throws exceptions if it tries to upload socket
+        # files and we could have some socket files in testsuite.dir.
+        # Also, upload-artifact@v3 doesn't work well enough with wildcards.
+        # So, we're just archiving everything here to avoid any issues.
+        mkdir logs
+        cp ovn/config.log ./logs/
+        # multinode tests are run as root, need to adjust permissions.
+        sudo chmod -R +r ovn/tests/multinode-testsuite.dir.* || true
+        cp -r ovn/tests/multinode-testsuite.dir.* ./logs/ || true
+        tar -czvf logs.tgz logs/
+
+    - name: upload logs on failure
+      if: failure() || cancelled()
+      uses: actions/upload-artifact@v3
+      with:
+        name: logs-linux-${{ join(matrix.cfg.*, '-') }}
+        path: logs.tgz
+
     - name: Stop cluster
       run: |
         cd ovn-fake-multinode
diff --git a/tests/automake.mk b/tests/automake.mk
index dce9c9108..88cda5f20 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -5,10 +5,12 @@  EXTRA_DIST += \
 	$(SYSTEM_KMOD_TESTSUITE_AT) \
 	$(SYSTEM_USERSPACE_TESTSUITE_AT) \
 	$(PERF_TESTSUITE_AT) \
+	$(MULTINODE_TESTSUITE_AT) \
 	$(TESTSUITE) \
 	$(SYSTEM_KMOD_TESTSUITE) \
 	$(SYSTEM_USERSPACE_TESTSUITE) \
 	$(PERF_TESTSUITE) \
+	$(MULTINODE_TESTSUITE) \
 	tests/atlocal.in \
 	$(srcdir)/package.m4 \
 	$(srcdir)/tests/testsuite \
@@ -61,6 +63,11 @@  PERF_TESTSUITE_AT = \
 	tests/perf-testsuite.at \
 	tests/perf-northd.at
 
+MULTINODE_TESTSUITE_AT = \
+	tests/multinode-testsuite.at \
+	tests/multinode-macros.at \
+	tests/multinode.at
+
 check_SCRIPTS += tests/atlocal
 
 TESTSUITE = $(srcdir)/tests/testsuite
@@ -72,7 +79,9 @@  PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite
 PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir
 PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results
 DISTCLEANFILES += tests/atconfig tests/atlocal
-
+MULTINODE_TESTSUITE = $(srcdir)/tests/multinode-testsuite
+MULTINODE_TESTSUITE_DIR = $(abs_top_builddir)/tests/multinode-testsuite.dir
+MULTINODE_TESTSUITE_RESULTS = $(MULTINODE_TESTSUITE_DIR)/results
 AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic
 
 export ovs_srcdir
@@ -82,6 +91,7 @@  check-local:
 	"$$@" $(TESTSUITEFLAGS) || \
 	(test -z "$$(find $(TESTSUITE_DIR) -name 'sanitizers.*')" && \
 	 test X'$(RECHECK)' = Xyes && "$$@" --recheck)
+
 
 # Python Coverage support.
 # Requires coverage.py http://nedbatchelder.com/code/coverage/.
@@ -197,6 +207,18 @@  check-perf: all
 	@echo
 	@echo "Results can be found in $(PERF_TESTSUITE_RESULTS)"
 
+check-multinode: all
+	@mkdir -p $(MULTINODE_TESTSUITE_DIR)
+	@echo  > $(MULTINODE_TESTSUITE_RESULTS)
+	set $(SHELL) '$(MULTINODE_TESTSUITE)' -C tests  AUTOTEST_PATH='$(AUTOTEST_PATH)'; \
+	"$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck)
+	@echo
+	@echo  '## -------------------- ##'
+	@echo  '##  Multinode test Results ##'
+	@echo  '## -------------------- ##'
+	@cat $(MULTINODE_TESTSUITE_RESULTS)
+	@echo
+	@echo "Results can be found in $(MULTINODE_TESTSUITE_RESULTS)"
 
 AUTOTEST = $(AUTOM4TE) --language=autotest
 
@@ -223,6 +245,10 @@  $(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT)
 	$(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
 	$(AM_V_at)mv $@.tmp $@
 
+$(MULTINODE_TESTSUITE): package.m4 $(MULTINODE_TESTSUITE_AT) $(COMMON_MACROS_AT)
+	$(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
+	$(AM_V_at)mv $@.tmp $@
+
 # The `:;' works around a Bash 3.2 bug when the output is not writeable.
 $(srcdir)/package.m4: $(top_srcdir)/configure.ac
 	$(AM_V_GEN):;{ \
diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at
new file mode 100644
index 000000000..88a3816eb
--- /dev/null
+++ b/tests/multinode-macros.at
@@ -0,0 +1,189 @@ 
+#
+#
+
+# M_NS_EXEC([fake_node], [namespace], [command])
+#
+# Execute 'command' in 'namespace'
+m4_define([M_NS_EXEC],
+    [podman exec $1 ip netns exec $2 $3])
+
+# M_NS_CHECK_EXEC(fake_node], [namespace], [command], other_params...)
+#
+# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s namespace'.
+# 'other_params' as passed as they are to AT_CHECK.
+m4_define([M_NS_CHECK_EXEC],
+    [ AT_CHECK([M_NS_EXEC([$1], [$2], [$3])], m4_shift(m4_shift(m4_shift($@)))) ]
+)
+
+OVS_START_SHELL_HELPERS
+
+m_as() {
+    c=$1
+    shift
+    podman exec $c "$@"
+}
+
+m_central_as () {
+    podman exec ovn-central "$@"
+}
+
+check_fake_multinode_setup() {
+    check m_as ovn-central ovn-nbctl --wait=hv sync
+    AT_CHECK([m_as ovn-chassis-1 ovn-appctl -t ovn-controller version], [0], [ignore])
+    AT_CHECK([m_as ovn-chassis-2 ovn-appctl -t ovn-controller version], [0], [ignore])
+    AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0], [ignore])
+}
+
+cleanup_multinode_resources() {
+    m_as ovn-central rm -f /etc/ovn/ovnnb_db.db
+    m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd
+    check m_as ovn-central ovn-nbctl --wait=hv sync
+    for c in ovn-chassis-1 ovn-chassis-2 ovn-gw-1
+    do
+        m_as $c ovs-vsctl del-br br-int
+        m_as $c ip --all netns delete
+    done
+}
+
+multinode_nbctl () {
+    m_as ovn-central ovn-nbctl "$@"
+}
+
+# count_rows TABLE [CONDITION...]
+#
+# Prints the number of rows in TABLE (that satisfy CONDITION).
+# Uses the southbound db by default; set DB=nb for the northbound database.
+m_count_rows() {
+    local db=$(parse_db $1) table=$(parse_table $1); shift
+    m_central_as ovn-${db}ctl --format=table --no-headings find $table "$@" | wc -l
+}
+
+# check_row_count [DATABASE:]TABLE COUNT [CONDITION...]
+#
+# Checks that TABLE contains COUNT rows (that satisfy CONDITION).
+# The default DATABASE is "sb".
+m_check_row_count() {
+    local db=$(parse_db $1) table=$(parse_table $1); shift
+    local count=$1; shift
+    local found=$(m_count_rows $c $db:$table "$@")
+    echo
+    echo "Checking for $count rows in $db $table${1+ with $*}... found $found"
+    if test "$count" != "$found"; then
+        m_central_as ovn-${db}ctl list $table
+        AT_FAIL_IF([:])
+    fi
+}
+
+# wait_row_count [DATABASE:]TABLE COUNT [CONDITION...]
+#
+# Waits until TABLE contains COUNT rows (that satisfy CONDITION).
+# The default DATABASE is "sb".
+m_wait_row_count() {
+    local db=$(parse_db $1) table=$(parse_table $1); shift
+    local count=$1; shift
+    local a=$1 b=$2 c=$3 d=$4 e=$5
+    echo "Waiting until $count rows in $db $table${1+ with $*}..."
+    OVS_WAIT_UNTIL([test $count = $(m_count_rows $db:$table $a $b $c $d $e)],[
+      echo "$db table $table has the following rows. $(m_count_rows $db:$table $a $b $c $d $e) rows match instead of expected $count:"
+      m_central_as ovn-${db}ctl list $table])
+}
+
+# multinode_wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]]
+#
+# Wait until all of the values of COLUMN in the rows of TABLE (that
+# satisfy CONDITION) equal EXPECTED (ignoring order).
+#
+# The default DATABASE is "sb".
+#
+# COLUMN defaults to _uuid if unspecified.
+m_wait_column() {
+    local expected=$(for d in $1; do echo $d; done | sort)
+    local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift
+    local a=$1 b=$2 c=$3 d=$4 e=$5
+
+    echo
+    echo "Waiting until $column in $db $table${1+ with $*} is $expected..."
+    OVS_WAIT_UNTIL([
+      found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e)
+      found=$(for d in $found; do echo $d; done | sort)
+      test "$expected" = "$found"
+    ], [
+      echo "$column in $db table $table has value $found, from the following rows:"
+      m_central_as ovn-${db}ctl list $table])
+}
+
+# fetch_column [DATABASE:]TABLE COLUMN [CONDITION...]
+#
+# Fetches and prints all the values of COLUMN in the rows of TABLE
+# (that satisfy CONDITION), sorting the results lexicographically.
+# The default DATABASE is "sb".
+m_fetch_column() {
+    local db=$(parse_db $1) table=$(parse_table $1) column=${2-_uuid}; shift; shift
+    # Using "echo" removes spaces and newlines.
+    echo $(m_central_as ovn-${db}ctl --bare --columns $column find $table "$@" | sort)
+}
+
+# check_column EXPECTED [DATABASE:]TABLE COLUMN [CONDITION...]
+#
+# Fetches all of the values of COLUMN in the rows of TABLE (that
+# satisfy CONDITION), and compares them against EXPECTED (ignoring
+# order).
+#
+# The default DATABASE is "sb".
+m_check_column() {
+    local expected=$1 db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift
+    local found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table "$@")
+
+    # Sort the expected and found values.
+    local found=$(for d in $found; do echo $d; done | sort)
+    local expected=$(for d in $expected; do echo $d; done | sort)
+
+    echo
+    echo "Checking values in $db $table${1+ with $*} against $expected... found $found"
+    if test "$found" != "$expected"; then
+        m_central_as ovn-${db}ctl list $table
+        AT_FAIL_IF([:])
+    fi
+}
+
+# wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]]
+#
+# Wait until all of the values of COLUMN in the rows of TABLE (that
+# satisfy CONDITION) equal EXPECTED (ignoring order).
+#
+# The default DATABASE is "sb".
+#
+# COLUMN defaults to _uuid if unspecified.
+m_wait_column() {
+    local expected=$(for d in $1; do echo $d; done | sort)
+    local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift
+    local a=$1 b=$2 c=$3 d=$4 e=$5
+
+    echo
+    echo "Waiting until $column in $db $table${1+ with $*} is $expected..."
+    OVS_WAIT_UNTIL([
+      found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e)
+      found=$(for d in $found; do echo $d; done | sort)
+      test "$expected" = "$found"
+    ], [
+      echo "$column in $db table $table has value $found, from the following rows:"
+      m_central_as ovn-${db}ctl list $table])
+}
+
+# wait_for_ports_up [PORT...]
+#
+# With arguments, waits for specified Logical_Switch_Ports to come up.
+# Without arguments, waits for all "plain" and router
+# Logical_Switch_Ports to come up.
+m_wait_for_ports_up() {
+    if test $# = 0; then
+        m_wait_row_count nb:Logical_Switch_Port 0 up!=true type='""'
+        m_wait_row_count nb:Logical_Switch_Port 0 up!=true type=router
+    else
+        for port; do
+            m_wait_row_count nb:Logical_Switch_Port 1 up=true name=$port
+        done
+    fi
+}
+
+OVS_END_SHELL_HELPERS
diff --git a/tests/multinode-testsuite.at b/tests/multinode-testsuite.at
new file mode 100644
index 000000000..ea10b0276
--- /dev/null
+++ b/tests/multinode-testsuite.at
@@ -0,0 +1,27 @@ 
+AT_INIT
+
+AT_COPYRIGHT([Copyright (c) 2022 Red Hat,
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at:
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.])
+
+m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS])
+AT_ARG_OPTION([rebuild], [Do not use cached versions of databases])
+
+m4_include([tests/ovs-macros.at])
+m4_include([tests/ovsdb-macros.at])
+m4_include([tests/ofproto-macros.at])
+m4_include([tests/ovn-macros.at])
+m4_include([tests/system-common-macros.at])
+m4_include([tests/multinode-macros.at])
+
+m4_include([tests/multinode.at])
diff --git a/tests/multinode.at b/tests/multinode.at
new file mode 100644
index 000000000..754e488d6
--- /dev/null
+++ b/tests/multinode.at
@@ -0,0 +1,74 @@ 
+AT_BANNER([ovn multinode system tests using ovn-fake-multinode])
+
+AT_SETUP([ovn multinode basic test])
+
+# Check that ovn-fake-multinode setup is up and running
+check_fake_multinode_setup
+
+# Delete the multinode NB and OVS resources before starting the test.
+cleanup_multinode_resources
+
+# Test East-West switching
+check multinode_nbctl ls-add sw0
+check multinode_nbctl lsp-add sw0 sw0-port1
+check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3 1000::3"
+check multinode_nbctl lsp-add sw0 sw0-port2
+check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4 1000::4"
+
+m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a
+m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a
+
+m_wait_for_ports_up
+
+M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \
+[0], [dnl
+3 packets transmitted, 3 received, 0% packet loss, time 0ms
+])
+
+# Add ACLs to drop all traffic
+check multinode_nbctl pg-add pg0 sw0-port1 sw0-port2
+check multinode_nbctl acl-add pg0 to-lport 1001 "outport == @pg0 && ip4" drop
+check multinode_nbctl --wait=hv sync
+
+M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4], \
+[1], [ignore])
+
+# Add ACLs to allow icmp traffic
+check multinode_nbctl acl-add pg0 to-lport 1002 "outport == @pg0 && ip4 && icmp" allow-related
+check multinode_nbctl --wait=hv sync
+
+M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \
+[0], [dnl
+3 packets transmitted, 3 received, 0% packet loss, time 0ms
+])
+
+
+# Create the second logical switch with one port
+check multinode_nbctl ls-add sw1
+check multinode_nbctl lsp-add sw1 sw1-port1
+check multinode_nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 20.0.0.3 2000::3"
+
+# Create a logical router and attach both logical switches
+check multinode_nbctl lr-add lr0
+check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 1000::a/64
+check multinode_nbctl lsp-add sw0 sw0-lr0
+check multinode_nbctl lsp-set-type sw0-lr0 router
+check multinode_nbctl lsp-set-addresses sw0-lr0 router
+check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0
+
+check multinode_nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24 2000::a/64
+check multinode_nbctl lsp-add sw1 sw1-lr0
+check multinode_nbctl lsp-set-type sw1-lr0 router
+check multinode_nbctl lsp-set-addresses sw1-lr0 router
+check multinode_nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
+
+m_as ovn-chassis-2 /data/create_fake_vm.sh sw1-port1 sw1p1 40:54:00:00:00:03 20.0.0.3 24 20.0.0.1 2000::4/64 1000::a
+
+m_wait_for_ports_up sw1-port1
+
+M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.3 | FORMAT_PING], \
+[0], [dnl
+3 packets transmitted, 3 received, 0% packet loss, time 0ms
+])
+
+AT_CLEANUP