| Message ID | 20210630131502.2381915-3-mark.d.gray@redhat.com |
|---|---|
| State | Superseded, archived |
| Delegated to: | Numan Siddique |
| Headers | show |
| Series | tests: Add check-perf target | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot | fail | github build: failed |
On 6/30/21 3:15 PM, Mark Gray wrote: > Add a suite of micro-benchmarks to aid a developer in understanding the > performance impact of any changes that they are making. They can be used to > help to understand the relative performance between two test runs on the same > test machine, but are not intended to give the absolute performance of OVN. > > To invoke the performance testsuite, run: > > $ make check-perf > > This will run all available performance tests. > > Additional metrics (e.g. memory, coverage, perf counters) may be added > in the future. Additional tests (e.g. additional topologies, ovn-controller > tests) may be added in the future. > > Signed-off-by: Mark Gray <mark.d.gray@redhat.com> > --- > > Notes: > v2: create results directory to fix build error > v3: forgot to commit, create results directory to fix build error > v4: fix 0-day issues > remove `sudo` in Makefile > updated documentation > > Documentation/topics/testing.rst | 50 ++++++++ > tests/.gitignore | 3 + > tests/automake.mk | 27 ++++ > tests/perf-northd.at | 207 +++++++++++++++++++++++++++++++ > tests/perf-testsuite.at | 26 ++++ > 5 files changed, 313 insertions(+) > create mode 100644 tests/perf-northd.at > create mode 100644 tests/perf-testsuite.at > > diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst > index be9e7c57331c..db265344a507 100644 > --- a/Documentation/topics/testing.rst > +++ b/Documentation/topics/testing.rst > @@ -256,3 +256,53 @@ the following:: > All the features documented under `Unit Tests`_ are available for the > datapath testsuites, except that the datapath testsuites do not > support running tests in parallel. > + > +Performance testing > +~~~~~~~~~~~~~~~~~~~ > + > +OVN includes a suite of micro-benchmarks to aid a developer in understanding > +the performance impact of any changes that they are making. They can be used to > +help to understand the relative performance between two test runs on the same > +test machine, but are not intended to give the absolute performance of OVN. > + > +To invoke the performance testsuite, run:: > + > + $ make check-perf > + > +This will run all available performance tests. Some of these tests may be > +long-running as they need to build complex logical network topologies. In order > +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be > +cached. In order to force the tests to rebuild all these objects, run:: > + > + $ make check-perf TESTSUITEFLAGS="--rebuild" > + > +A typical workflow for a developer trying to improve the performance of OVN > +would be the following: > + > +0. Optional: Modify/add a performance test to buld the topology that you are > + benchmarking, if required. > +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached > + databases (and complete a test run). The results of each test run are > + displayed on the screen at the end of the test run but are also saved in the > + file ``tests/perf-testsuite.dir/results``. > + > +.. note:: > + This step may take some time depending on the number of tests that are being > + rebuilt, the complexity of the tests and the performance of the test > + machine. If you are only using one test, you can specify the test to run by > + adding the test number to the ``make`` command. > + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) > + > +2. Run ``make check-perf`` to measure the performance metric that you are > + benchmarking against. If you are only using one test, you can specify the > + test to run by adding the test number to the ``make`` command. > + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) > +3. Modify OVN code to implement the change that you believe will improve the > + performance. > +4. Go to Step 2. to continue making improvements. > + > +If, as a developer, you modify a performance test in a way that may change one > +of these cached objects, be sure to rebuild the test. > + > +The cached objects are stored under the relevant folder in > +``tests/perf-testsuite.dir/cached``. > diff --git a/tests/.gitignore b/tests/.gitignore > index 8479f9bb0f8f..65cb1c6e4fad 100644 > --- a/tests/.gitignore > +++ b/tests/.gitignore > @@ -22,6 +22,9 @@ > /system-offloads-testsuite > /system-offloads-testsuite.dir/ > /system-offloads-testsuite.log > +/perf-testsuite > +/perf-testsuite.dir/ > +/perf-testsuite.log > /test-aes128 > /test-atomic > /test-bundle > diff --git a/tests/automake.mk b/tests/automake.mk > index a8ec64212791..5b890d644eeb 100644 > --- a/tests/automake.mk > +++ b/tests/automake.mk > @@ -4,9 +4,11 @@ EXTRA_DIST += \ > $(SYSTEM_TESTSUITE_AT) \ > $(SYSTEM_KMOD_TESTSUITE_AT) \ > $(SYSTEM_USERSPACE_TESTSUITE_AT) \ > + $(PERF_TESTSUITE_AT) \ > $(TESTSUITE) \ > $(SYSTEM_KMOD_TESTSUITE) \ > $(SYSTEM_USERSPACE_TESTSUITE) \ > + $(PERF_TESTSUITE) \ > tests/atlocal.in \ > $(srcdir)/package.m4 \ > $(srcdir)/tests/testsuite \ > @@ -53,6 +55,10 @@ SYSTEM_TESTSUITE_AT = \ > tests/system-ovn.at \ > tests/system-ovn-kmod.at > > +PERF_TESTSUITE_AT = \ > + tests/perf-testsuite.at \ > + tests/perf-northd.at > + > check_SCRIPTS += tests/atlocal > > TESTSUITE = $(srcdir)/tests/testsuite > @@ -60,6 +66,9 @@ TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch > TESTSUITE_DIR = $(abs_top_builddir)/tests/testsuite.dir > SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite > SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite > +PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite > +PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir > +PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results > DISTCLEANFILES += tests/atconfig tests/atlocal > > AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic > @@ -172,6 +181,20 @@ check-system-userspace: all > > clean-local: > test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean > + > +check-perf: all > + @mkdir -p $(PERF_TESTSUITE_DIR) > + @echo > $(PERF_TESTSUITE_RESULTS) > + set $(SHELL) '$(PERF_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ > + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck) > + @echo > + @echo '## -------------------- ##' > + @echo '## Performance Results ##' > + @echo '## -------------------- ##' > + @cat $(PERF_TESTSUITE_RESULTS) > + @echo > + @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" > + > > AUTOTEST = $(AUTOM4TE) --language=autotest > > @@ -194,6 +217,10 @@ $(SYSTEM_USERSPACE_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT) $(SYSTEM_USERSP > $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > $(AM_V_at)mv $@.tmp $@ > > +$(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) > + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > + $(AM_V_at)mv $@.tmp $@ > + > # The `:;' works around a Bash 3.2 bug when the output is not writeable. > $(srcdir)/package.m4: $(top_srcdir)/configure.ac > $(AM_V_GEN):;{ \ > diff --git a/tests/perf-northd.at b/tests/perf-northd.at > new file mode 100644 > index 000000000000..eb0c391c4650 > --- /dev/null > +++ b/tests/perf-northd.at > @@ -0,0 +1,207 @@ > +AT_BANNER([ovn-northd performance tests]) > + > +# CACHE_NBDB() > +# > +# Save the current northbound database for future test runs. > +# > +m4_define([CACHE_NBDB],[ > + mkdir -p ${at_suite_dir}/cached/${at_group} > + cp -f ${ovs_base}/ovn-nb/ovn-nb.db ${at_suite_dir}/cached/${at_group}/ > +]) > + > +# BUILD_NBDB([COMMANDS]) > +# > +# Configure the northbound database using COMMANDS. > +# > +# BUILD_NBDB() will check if there is a cached nortbound database present > +# for this test group. If present, it will use the cached version. If the > +# testsuite was run using the '--rebuild' flag, it will force a rebuild of the > +# northbound database. > +# > +m4_define([BUILD_NBDB],[ > + if [[ ! -f ${at_suite_dir}/cached/${at_group}/ovn-nb.db ]] || [[ $at_arg_rebuild != false ]]; then > + echo "Rebuild NBDB" > + $1 > + CACHE_NBDB() > + else > + echo "Using cached NBDB" > + rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ > + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound > + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db > + fi > + ovn-nbctl --wait=sb sync > +]) > + > +# PERF_RECORD_BANNER() > +# > +# Append standard banner to performance results. > +# > +m4_define([PERF_RECORD_START],[ > + echo >> ${at_suite_dir}/results > + echo "$at_desc_line" >> ${at_suite_dir}/results > + echo " ---" >> ${at_suite_dir}/results > +]) > + > +# PERF_RECORD_RESULT([KEY], [VALUE]) > +# > +# Append KEY and VALUE to performance results. > +# > +m4_define([PERF_RECORD_RESULT],[ > + echo " $1: $2" >> ${at_suite_dir}/results > +]) > + > +m4_define([PARSE_STOPWATCH], [ > + grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' > +]) > + > +# PERF_RECORD_STOPWATCH([NAME], [METRIC]) > +# > +# Append the value of the OVN stopwatch metric METRIC from stopwatch NAME > +# to performance results. > +# > +m4_define([PERF_RECORD_STOPWATCH], [ > + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) > +]) > + > +# PERF_RECORD() > +# > +# Append a number of metrics to performance results > +# > +m4_define([PERF_RECORD_STOP], [ > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) I think now that we have stopwatches for the whole northd loop (and for ovnsb_db_run) we might as well dump them here too. What do you think? Otherwise, the changes in this patch look good to me. Acked-by: Dumitru Ceara <dceara@redhat.com> Regards, Dumitru
On Wed, Jul 14, 2021 at 9:08 AM Dumitru Ceara <dceara@redhat.com> wrote: > > On 6/30/21 3:15 PM, Mark Gray wrote: > > Add a suite of micro-benchmarks to aid a developer in understanding the > > performance impact of any changes that they are making. They can be used to > > help to understand the relative performance between two test runs on the same > > test machine, but are not intended to give the absolute performance of OVN. > > > > To invoke the performance testsuite, run: > > > > $ make check-perf > > > > This will run all available performance tests. > > > > Additional metrics (e.g. memory, coverage, perf counters) may be added > > in the future. Additional tests (e.g. additional topologies, ovn-controller > > tests) may be added in the future. > > > > Signed-off-by: Mark Gray <mark.d.gray@redhat.com> > > --- > > > > Notes: > > v2: create results directory to fix build error > > v3: forgot to commit, create results directory to fix build error > > v4: fix 0-day issues > > remove `sudo` in Makefile > > updated documentation > > > > Documentation/topics/testing.rst | 50 ++++++++ > > tests/.gitignore | 3 + > > tests/automake.mk | 27 ++++ > > tests/perf-northd.at | 207 +++++++++++++++++++++++++++++++ > > tests/perf-testsuite.at | 26 ++++ > > 5 files changed, 313 insertions(+) > > create mode 100644 tests/perf-northd.at > > create mode 100644 tests/perf-testsuite.at > > > > diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst > > index be9e7c57331c..db265344a507 100644 > > --- a/Documentation/topics/testing.rst > > +++ b/Documentation/topics/testing.rst > > @@ -256,3 +256,53 @@ the following:: > > All the features documented under `Unit Tests`_ are available for the > > datapath testsuites, except that the datapath testsuites do not > > support running tests in parallel. > > + > > +Performance testing > > +~~~~~~~~~~~~~~~~~~~ > > + > > +OVN includes a suite of micro-benchmarks to aid a developer in understanding > > +the performance impact of any changes that they are making. They can be used to > > +help to understand the relative performance between two test runs on the same > > +test machine, but are not intended to give the absolute performance of OVN. > > + > > +To invoke the performance testsuite, run:: > > + > > + $ make check-perf > > + > > +This will run all available performance tests. Some of these tests may be > > +long-running as they need to build complex logical network topologies. In order > > +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be > > +cached. In order to force the tests to rebuild all these objects, run:: > > + > > + $ make check-perf TESTSUITEFLAGS="--rebuild" > > + > > +A typical workflow for a developer trying to improve the performance of OVN > > +would be the following: > > + > > +0. Optional: Modify/add a performance test to buld the topology that you are > > + benchmarking, if required. > > +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached > > + databases (and complete a test run). The results of each test run are > > + displayed on the screen at the end of the test run but are also saved in the > > + file ``tests/perf-testsuite.dir/results``. > > + > > +.. note:: > > + This step may take some time depending on the number of tests that are being > > + rebuilt, the complexity of the tests and the performance of the test > > + machine. If you are only using one test, you can specify the test to run by > > + adding the test number to the ``make`` command. > > + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) > > + > > +2. Run ``make check-perf`` to measure the performance metric that you are > > + benchmarking against. If you are only using one test, you can specify the > > + test to run by adding the test number to the ``make`` command. > > + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) > > +3. Modify OVN code to implement the change that you believe will improve the > > + performance. > > +4. Go to Step 2. to continue making improvements. > > + > > +If, as a developer, you modify a performance test in a way that may change one > > +of these cached objects, be sure to rebuild the test. > > + > > +The cached objects are stored under the relevant folder in > > +``tests/perf-testsuite.dir/cached``. > > diff --git a/tests/.gitignore b/tests/.gitignore > > index 8479f9bb0f8f..65cb1c6e4fad 100644 > > --- a/tests/.gitignore > > +++ b/tests/.gitignore > > @@ -22,6 +22,9 @@ > > /system-offloads-testsuite > > /system-offloads-testsuite.dir/ > > /system-offloads-testsuite.log > > +/perf-testsuite > > +/perf-testsuite.dir/ > > +/perf-testsuite.log > > /test-aes128 > > /test-atomic > > /test-bundle > > diff --git a/tests/automake.mk b/tests/automake.mk > > index a8ec64212791..5b890d644eeb 100644 > > --- a/tests/automake.mk > > +++ b/tests/automake.mk > > @@ -4,9 +4,11 @@ EXTRA_DIST += \ > > $(SYSTEM_TESTSUITE_AT) \ > > $(SYSTEM_KMOD_TESTSUITE_AT) \ > > $(SYSTEM_USERSPACE_TESTSUITE_AT) \ > > + $(PERF_TESTSUITE_AT) \ > > $(TESTSUITE) \ > > $(SYSTEM_KMOD_TESTSUITE) \ > > $(SYSTEM_USERSPACE_TESTSUITE) \ > > + $(PERF_TESTSUITE) \ > > tests/atlocal.in \ > > $(srcdir)/package.m4 \ > > $(srcdir)/tests/testsuite \ > > @@ -53,6 +55,10 @@ SYSTEM_TESTSUITE_AT = \ > > tests/system-ovn.at \ > > tests/system-ovn-kmod.at > > > > +PERF_TESTSUITE_AT = \ > > + tests/perf-testsuite.at \ > > + tests/perf-northd.at > > + > > check_SCRIPTS += tests/atlocal > > > > TESTSUITE = $(srcdir)/tests/testsuite > > @@ -60,6 +66,9 @@ TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch > > TESTSUITE_DIR = $(abs_top_builddir)/tests/testsuite.dir > > SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite > > SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite > > +PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite > > +PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir > > +PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results > > DISTCLEANFILES += tests/atconfig tests/atlocal > > > > AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic > > @@ -172,6 +181,20 @@ check-system-userspace: all > > > > clean-local: > > test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean > > + > > +check-perf: all > > + @mkdir -p $(PERF_TESTSUITE_DIR) > > + @echo > $(PERF_TESTSUITE_RESULTS) > > + set $(SHELL) '$(PERF_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ > > + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck) > > + @echo > > + @echo '## -------------------- ##' > > + @echo '## Performance Results ##' > > + @echo '## -------------------- ##' > > + @cat $(PERF_TESTSUITE_RESULTS) > > + @echo > > + @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" > > + > > > > AUTOTEST = $(AUTOM4TE) --language=autotest > > > > @@ -194,6 +217,10 @@ $(SYSTEM_USERSPACE_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT) $(SYSTEM_USERSP > > $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > > $(AM_V_at)mv $@.tmp $@ > > > > +$(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) > > + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > > + $(AM_V_at)mv $@.tmp $@ > > + > > # The `:;' works around a Bash 3.2 bug when the output is not writeable. > > $(srcdir)/package.m4: $(top_srcdir)/configure.ac > > $(AM_V_GEN):;{ \ > > diff --git a/tests/perf-northd.at b/tests/perf-northd.at > > new file mode 100644 > > index 000000000000..eb0c391c4650 > > --- /dev/null > > +++ b/tests/perf-northd.at > > @@ -0,0 +1,207 @@ > > +AT_BANNER([ovn-northd performance tests]) > > + > > +# CACHE_NBDB() > > +# > > +# Save the current northbound database for future test runs. > > +# > > +m4_define([CACHE_NBDB],[ > > + mkdir -p ${at_suite_dir}/cached/${at_group} > > + cp -f ${ovs_base}/ovn-nb/ovn-nb.db ${at_suite_dir}/cached/${at_group}/ > > +]) > > + > > +# BUILD_NBDB([COMMANDS]) > > +# > > +# Configure the northbound database using COMMANDS. > > +# > > +# BUILD_NBDB() will check if there is a cached nortbound database present > > +# for this test group. If present, it will use the cached version. If the > > +# testsuite was run using the '--rebuild' flag, it will force a rebuild of the > > +# northbound database. > > +# > > +m4_define([BUILD_NBDB],[ > > + if [[ ! -f ${at_suite_dir}/cached/${at_group}/ovn-nb.db ]] || [[ $at_arg_rebuild != false ]]; then > > + echo "Rebuild NBDB" > > + $1 > > + CACHE_NBDB() > > + else > > + echo "Using cached NBDB" > > + rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ > > + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound > > + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db > > + fi > > + ovn-nbctl --wait=sb sync > > +]) > > + > > +# PERF_RECORD_BANNER() > > +# > > +# Append standard banner to performance results. > > +# > > +m4_define([PERF_RECORD_START],[ > > + echo >> ${at_suite_dir}/results > > + echo "$at_desc_line" >> ${at_suite_dir}/results > > + echo " ---" >> ${at_suite_dir}/results > > +]) > > + > > +# PERF_RECORD_RESULT([KEY], [VALUE]) > > +# > > +# Append KEY and VALUE to performance results. > > +# > > +m4_define([PERF_RECORD_RESULT],[ > > + echo " $1: $2" >> ${at_suite_dir}/results > > +]) > > + > > +m4_define([PARSE_STOPWATCH], [ > > + grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' > > +]) > > + > > +# PERF_RECORD_STOPWATCH([NAME], [METRIC]) > > +# > > +# Append the value of the OVN stopwatch metric METRIC from stopwatch NAME > > +# to performance results. > > +# > > +m4_define([PERF_RECORD_STOPWATCH], [ > > + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) > > +]) > > + > > +# PERF_RECORD() > > +# > > +# Append a number of metrics to performance results > > +# > > +m4_define([PERF_RECORD_STOP], [ > > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) > > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) > > I think now that we have stopwatches for the whole northd loop (and for > ovnsb_db_run) we might as well dump them here too. What do you think? > > Otherwise, the changes in this patch look good to me. > > Acked-by: Dumitru Ceara <dceara@redhat.com> Thanks a lot Mark for this patch series and Dumitru for the reviews. I think this will be helpful for developers. I applied both the patches to the main branch with the below changes to the 2nd patch. Please let me know if something is wrong with the changes I've made. I can submit a follow up patch if there are any mistakes. I incorporated the changes suggested by Dumitru as well. -------------------------------------------- diff --git a/tests/perf-northd.at b/tests/perf-northd.at index eb0c391c46..74b69e9d4f 100644 --- a/tests/perf-northd.at +++ b/tests/perf-northd.at @@ -26,8 +26,8 @@ m4_define([BUILD_NBDB],[ else echo "Using cached NBDB" rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ - ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound - ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db + ovn-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound + ovn-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db fi ovn-nbctl --wait=sb sync ]) @@ -51,7 +51,7 @@ m4_define([PERF_RECORD_RESULT],[ ]) m4_define([PARSE_STOPWATCH], [ - grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' + grep $1 | sed 's/[[^0-9.]]*//g' ]) # PERF_RECORD_STOPWATCH([NAME], [METRIC]) @@ -60,7 +60,7 @@ m4_define([PARSE_STOPWATCH], [ # to performance results. # m4_define([PERF_RECORD_STOPWATCH], [ - PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show $1 | PARSE_STOPWATCH($2)`]) ]) # PERF_RECORD() @@ -68,8 +68,12 @@ m4_define([PERF_RECORD_STOPWATCH], [ # Append a number of metrics to performance results # m4_define([PERF_RECORD_STOP], [ - PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) - PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB in msec)]) + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB in msec)]) + PERF_RECORD_STOPWATCH(ovnsb_db_run, ["Maximum"], [Maximum (SB in msec)]) + PERF_RECORD_STOPWATCH(ovnsb_db_run, ["Short term average"], [Average (SB in msec)]) + PERF_RECORD_STOPWATCH(ovn-northd-loop, ["Maximum"], [Maximum (northd-loop in msec)]) + PERF_RECORD_STOPWATCH(ovn-northd-loop, ["Short term average"], [Average (northd-loop in msec)]) ]) # OVN_NBCTL([NBCTL_COMMAND]) -------------------------------------------- Thanks Numan > > Regards, > Dumitru > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
On 23/07/2021 21:18, Numan Siddique wrote: > On Wed, Jul 14, 2021 at 9:08 AM Dumitru Ceara <dceara@redhat.com> wrote: >> >> On 6/30/21 3:15 PM, Mark Gray wrote: >>> Add a suite of micro-benchmarks to aid a developer in understanding the >>> performance impact of any changes that they are making. They can be used to >>> help to understand the relative performance between two test runs on the same >>> test machine, but are not intended to give the absolute performance of OVN. >>> >>> To invoke the performance testsuite, run: >>> >>> $ make check-perf >>> >>> This will run all available performance tests. >>> >>> Additional metrics (e.g. memory, coverage, perf counters) may be added >>> in the future. Additional tests (e.g. additional topologies, ovn-controller >>> tests) may be added in the future. >>> >>> Signed-off-by: Mark Gray <mark.d.gray@redhat.com> >>> --- >>> >>> Notes: >>> v2: create results directory to fix build error >>> v3: forgot to commit, create results directory to fix build error >>> v4: fix 0-day issues >>> remove `sudo` in Makefile >>> updated documentation >>> >>> Documentation/topics/testing.rst | 50 ++++++++ >>> tests/.gitignore | 3 + >>> tests/automake.mk | 27 ++++ >>> tests/perf-northd.at | 207 +++++++++++++++++++++++++++++++ >>> tests/perf-testsuite.at | 26 ++++ >>> 5 files changed, 313 insertions(+) >>> create mode 100644 tests/perf-northd.at >>> create mode 100644 tests/perf-testsuite.at >>> >>> diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst >>> index be9e7c57331c..db265344a507 100644 >>> --- a/Documentation/topics/testing.rst >>> +++ b/Documentation/topics/testing.rst >>> @@ -256,3 +256,53 @@ the following:: >>> All the features documented under `Unit Tests`_ are available for the >>> datapath testsuites, except that the datapath testsuites do not >>> support running tests in parallel. >>> + >>> +Performance testing >>> +~~~~~~~~~~~~~~~~~~~ >>> + >>> +OVN includes a suite of micro-benchmarks to aid a developer in understanding >>> +the performance impact of any changes that they are making. They can be used to >>> +help to understand the relative performance between two test runs on the same >>> +test machine, but are not intended to give the absolute performance of OVN. >>> + >>> +To invoke the performance testsuite, run:: >>> + >>> + $ make check-perf >>> + >>> +This will run all available performance tests. Some of these tests may be >>> +long-running as they need to build complex logical network topologies. In order >>> +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be >>> +cached. In order to force the tests to rebuild all these objects, run:: >>> + >>> + $ make check-perf TESTSUITEFLAGS="--rebuild" >>> + >>> +A typical workflow for a developer trying to improve the performance of OVN >>> +would be the following: >>> + >>> +0. Optional: Modify/add a performance test to buld the topology that you are >>> + benchmarking, if required. >>> +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached >>> + databases (and complete a test run). The results of each test run are >>> + displayed on the screen at the end of the test run but are also saved in the >>> + file ``tests/perf-testsuite.dir/results``. >>> + >>> +.. note:: >>> + This step may take some time depending on the number of tests that are being >>> + rebuilt, the complexity of the tests and the performance of the test >>> + machine. If you are only using one test, you can specify the test to run by >>> + adding the test number to the ``make`` command. >>> + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) >>> + >>> +2. Run ``make check-perf`` to measure the performance metric that you are >>> + benchmarking against. If you are only using one test, you can specify the >>> + test to run by adding the test number to the ``make`` command. >>> + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) >>> +3. Modify OVN code to implement the change that you believe will improve the >>> + performance. >>> +4. Go to Step 2. to continue making improvements. >>> + >>> +If, as a developer, you modify a performance test in a way that may change one >>> +of these cached objects, be sure to rebuild the test. >>> + >>> +The cached objects are stored under the relevant folder in >>> +``tests/perf-testsuite.dir/cached``. >>> diff --git a/tests/.gitignore b/tests/.gitignore >>> index 8479f9bb0f8f..65cb1c6e4fad 100644 >>> --- a/tests/.gitignore >>> +++ b/tests/.gitignore >>> @@ -22,6 +22,9 @@ >>> /system-offloads-testsuite >>> /system-offloads-testsuite.dir/ >>> /system-offloads-testsuite.log >>> +/perf-testsuite >>> +/perf-testsuite.dir/ >>> +/perf-testsuite.log >>> /test-aes128 >>> /test-atomic >>> /test-bundle >>> diff --git a/tests/automake.mk b/tests/automake.mk >>> index a8ec64212791..5b890d644eeb 100644 >>> --- a/tests/automake.mk >>> +++ b/tests/automake.mk >>> @@ -4,9 +4,11 @@ EXTRA_DIST += \ >>> $(SYSTEM_TESTSUITE_AT) \ >>> $(SYSTEM_KMOD_TESTSUITE_AT) \ >>> $(SYSTEM_USERSPACE_TESTSUITE_AT) \ >>> + $(PERF_TESTSUITE_AT) \ >>> $(TESTSUITE) \ >>> $(SYSTEM_KMOD_TESTSUITE) \ >>> $(SYSTEM_USERSPACE_TESTSUITE) \ >>> + $(PERF_TESTSUITE) \ >>> tests/atlocal.in \ >>> $(srcdir)/package.m4 \ >>> $(srcdir)/tests/testsuite \ >>> @@ -53,6 +55,10 @@ SYSTEM_TESTSUITE_AT = \ >>> tests/system-ovn.at \ >>> tests/system-ovn-kmod.at >>> >>> +PERF_TESTSUITE_AT = \ >>> + tests/perf-testsuite.at \ >>> + tests/perf-northd.at >>> + >>> check_SCRIPTS += tests/atlocal >>> >>> TESTSUITE = $(srcdir)/tests/testsuite >>> @@ -60,6 +66,9 @@ TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch >>> TESTSUITE_DIR = $(abs_top_builddir)/tests/testsuite.dir >>> SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite >>> SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite >>> +PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite >>> +PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir >>> +PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results >>> DISTCLEANFILES += tests/atconfig tests/atlocal >>> >>> AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic >>> @@ -172,6 +181,20 @@ check-system-userspace: all >>> >>> clean-local: >>> test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean >>> + >>> +check-perf: all >>> + @mkdir -p $(PERF_TESTSUITE_DIR) >>> + @echo > $(PERF_TESTSUITE_RESULTS) >>> + set $(SHELL) '$(PERF_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ >>> + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck) >>> + @echo >>> + @echo '## -------------------- ##' >>> + @echo '## Performance Results ##' >>> + @echo '## -------------------- ##' >>> + @cat $(PERF_TESTSUITE_RESULTS) >>> + @echo >>> + @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" >>> + >>> >>> AUTOTEST = $(AUTOM4TE) --language=autotest >>> >>> @@ -194,6 +217,10 @@ $(SYSTEM_USERSPACE_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT) $(SYSTEM_USERSP >>> $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at >>> $(AM_V_at)mv $@.tmp $@ >>> >>> +$(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) >>> + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at >>> + $(AM_V_at)mv $@.tmp $@ >>> + >>> # The `:;' works around a Bash 3.2 bug when the output is not writeable. >>> $(srcdir)/package.m4: $(top_srcdir)/configure.ac >>> $(AM_V_GEN):;{ \ >>> diff --git a/tests/perf-northd.at b/tests/perf-northd.at >>> new file mode 100644 >>> index 000000000000..eb0c391c4650 >>> --- /dev/null >>> +++ b/tests/perf-northd.at >>> @@ -0,0 +1,207 @@ >>> +AT_BANNER([ovn-northd performance tests]) >>> + >>> +# CACHE_NBDB() >>> +# >>> +# Save the current northbound database for future test runs. >>> +# >>> +m4_define([CACHE_NBDB],[ >>> + mkdir -p ${at_suite_dir}/cached/${at_group} >>> + cp -f ${ovs_base}/ovn-nb/ovn-nb.db ${at_suite_dir}/cached/${at_group}/ >>> +]) >>> + >>> +# BUILD_NBDB([COMMANDS]) >>> +# >>> +# Configure the northbound database using COMMANDS. >>> +# >>> +# BUILD_NBDB() will check if there is a cached nortbound database present >>> +# for this test group. If present, it will use the cached version. If the >>> +# testsuite was run using the '--rebuild' flag, it will force a rebuild of the >>> +# northbound database. >>> +# >>> +m4_define([BUILD_NBDB],[ >>> + if [[ ! -f ${at_suite_dir}/cached/${at_group}/ovn-nb.db ]] || [[ $at_arg_rebuild != false ]]; then >>> + echo "Rebuild NBDB" >>> + $1 >>> + CACHE_NBDB() >>> + else >>> + echo "Using cached NBDB" >>> + rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ >>> + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound >>> + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db >>> + fi >>> + ovn-nbctl --wait=sb sync >>> +]) >>> + >>> +# PERF_RECORD_BANNER() >>> +# >>> +# Append standard banner to performance results. >>> +# >>> +m4_define([PERF_RECORD_START],[ >>> + echo >> ${at_suite_dir}/results >>> + echo "$at_desc_line" >> ${at_suite_dir}/results >>> + echo " ---" >> ${at_suite_dir}/results >>> +]) >>> + >>> +# PERF_RECORD_RESULT([KEY], [VALUE]) >>> +# >>> +# Append KEY and VALUE to performance results. >>> +# >>> +m4_define([PERF_RECORD_RESULT],[ >>> + echo " $1: $2" >> ${at_suite_dir}/results >>> +]) >>> + >>> +m4_define([PARSE_STOPWATCH], [ >>> + grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' >>> +]) >>> + >>> +# PERF_RECORD_STOPWATCH([NAME], [METRIC]) >>> +# >>> +# Append the value of the OVN stopwatch metric METRIC from stopwatch NAME >>> +# to performance results. >>> +# >>> +m4_define([PERF_RECORD_STOPWATCH], [ >>> + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) >>> +]) >>> + >>> +# PERF_RECORD() >>> +# >>> +# Append a number of metrics to performance results >>> +# >>> +m4_define([PERF_RECORD_STOP], [ >>> + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) >>> + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) >> >> I think now that we have stopwatches for the whole northd loop (and for >> ovnsb_db_run) we might as well dump them here too. What do you think? >> >> Otherwise, the changes in this patch look good to me. >> >> Acked-by: Dumitru Ceara <dceara@redhat.com> > > Thanks a lot Mark for this patch series and Dumitru for the reviews. > I think this will be helpful for developers. I'm not 100% sure of that but lets see. Either way, it can easily be removed or let rot :) Thanks for committing. TBH, I was planning to rework based on Dumitru's comment about northd stopwatches but I see you made them > > I applied both the patches to the main branch with the below changes > to the 2nd patch. Please let me know > if something is wrong with the changes I've made. I can submit a > follow up patch if there are any mistakes. Thanks, that all looks good. > > I incorporated the changes suggested by Dumitru as well. > > -------------------------------------------- > diff --git a/tests/perf-northd.at b/tests/perf-northd.at > index eb0c391c46..74b69e9d4f 100644 > --- a/tests/perf-northd.at > +++ b/tests/perf-northd.at > @@ -26,8 +26,8 @@ m4_define([BUILD_NBDB],[ > else > echo "Using cached NBDB" > rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ > - ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound > - ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db > ${at_suite_dir}/cached/${at_group}/ovn-nb.db > + ovn-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound > + ovn-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db > ${at_suite_dir}/cached/${at_group}/ovn-nb.db > fi > ovn-nbctl --wait=sb sync > ]) > @@ -51,7 +51,7 @@ m4_define([PERF_RECORD_RESULT],[ > ]) > > m4_define([PARSE_STOPWATCH], [ > - grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' > + grep $1 | sed 's/[[^0-9.]]*//g' > ]) > > # PERF_RECORD_STOPWATCH([NAME], [METRIC]) > @@ -60,7 +60,7 @@ m4_define([PARSE_STOPWATCH], [ > # to performance results. > # > m4_define([PERF_RECORD_STOPWATCH], [ > - PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE > stopwatch/show | PARSE_STOPWATCH($1, $2)`]) > + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE > stopwatch/show $1 | PARSE_STOPWATCH($2)`]) > ]) > > # PERF_RECORD() > @@ -68,8 +68,12 @@ m4_define([PERF_RECORD_STOPWATCH], [ > # Append a number of metrics to performance results > # > m4_define([PERF_RECORD_STOP], [ > - PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) > - PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB in msec)]) > + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], > [Average (NB in msec)]) > + PERF_RECORD_STOPWATCH(ovnsb_db_run, ["Maximum"], [Maximum (SB in msec)]) > + PERF_RECORD_STOPWATCH(ovnsb_db_run, ["Short term average"], > [Average (SB in msec)]) > + PERF_RECORD_STOPWATCH(ovn-northd-loop, ["Maximum"], [Maximum > (northd-loop in msec)]) > + PERF_RECORD_STOPWATCH(ovn-northd-loop, ["Short term average"], > [Average (northd-loop in msec)]) > ]) > > # OVN_NBCTL([NBCTL_COMMAND]) > > -------------------------------------------- > > Thanks > Numan > >> >> Regards, >> Dumitru >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> >
diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index be9e7c57331c..db265344a507 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -256,3 +256,53 @@ the following:: All the features documented under `Unit Tests`_ are available for the datapath testsuites, except that the datapath testsuites do not support running tests in parallel. + +Performance testing +~~~~~~~~~~~~~~~~~~~ + +OVN includes a suite of micro-benchmarks to aid a developer in understanding +the performance impact of any changes that they are making. They can be used to +help to understand the relative performance between two test runs on the same +test machine, but are not intended to give the absolute performance of OVN. + +To invoke the performance testsuite, run:: + + $ make check-perf + +This will run all available performance tests. Some of these tests may be +long-running as they need to build complex logical network topologies. In order +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be +cached. In order to force the tests to rebuild all these objects, run:: + + $ make check-perf TESTSUITEFLAGS="--rebuild" + +A typical workflow for a developer trying to improve the performance of OVN +would be the following: + +0. Optional: Modify/add a performance test to buld the topology that you are + benchmarking, if required. +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached + databases (and complete a test run). The results of each test run are + displayed on the screen at the end of the test run but are also saved in the + file ``tests/perf-testsuite.dir/results``. + +.. note:: + This step may take some time depending on the number of tests that are being + rebuilt, the complexity of the tests and the performance of the test + machine. If you are only using one test, you can specify the test to run by + adding the test number to the ``make`` command. + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) + +2. Run ``make check-perf`` to measure the performance metric that you are + benchmarking against. If you are only using one test, you can specify the + test to run by adding the test number to the ``make`` command. + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``) +3. Modify OVN code to implement the change that you believe will improve the + performance. +4. Go to Step 2. to continue making improvements. + +If, as a developer, you modify a performance test in a way that may change one +of these cached objects, be sure to rebuild the test. + +The cached objects are stored under the relevant folder in +``tests/perf-testsuite.dir/cached``. diff --git a/tests/.gitignore b/tests/.gitignore index 8479f9bb0f8f..65cb1c6e4fad 100644 --- a/tests/.gitignore +++ b/tests/.gitignore @@ -22,6 +22,9 @@ /system-offloads-testsuite /system-offloads-testsuite.dir/ /system-offloads-testsuite.log +/perf-testsuite +/perf-testsuite.dir/ +/perf-testsuite.log /test-aes128 /test-atomic /test-bundle diff --git a/tests/automake.mk b/tests/automake.mk index a8ec64212791..5b890d644eeb 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -4,9 +4,11 @@ EXTRA_DIST += \ $(SYSTEM_TESTSUITE_AT) \ $(SYSTEM_KMOD_TESTSUITE_AT) \ $(SYSTEM_USERSPACE_TESTSUITE_AT) \ + $(PERF_TESTSUITE_AT) \ $(TESTSUITE) \ $(SYSTEM_KMOD_TESTSUITE) \ $(SYSTEM_USERSPACE_TESTSUITE) \ + $(PERF_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ $(srcdir)/tests/testsuite \ @@ -53,6 +55,10 @@ SYSTEM_TESTSUITE_AT = \ tests/system-ovn.at \ tests/system-ovn-kmod.at +PERF_TESTSUITE_AT = \ + tests/perf-testsuite.at \ + tests/perf-northd.at + check_SCRIPTS += tests/atlocal TESTSUITE = $(srcdir)/tests/testsuite @@ -60,6 +66,9 @@ TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch TESTSUITE_DIR = $(abs_top_builddir)/tests/testsuite.dir SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite +PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite +PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir +PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results DISTCLEANFILES += tests/atconfig tests/atlocal AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic @@ -172,6 +181,20 @@ check-system-userspace: all clean-local: test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean + +check-perf: all + @mkdir -p $(PERF_TESTSUITE_DIR) + @echo > $(PERF_TESTSUITE_RESULTS) + set $(SHELL) '$(PERF_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck) + @echo + @echo '## -------------------- ##' + @echo '## Performance Results ##' + @echo '## -------------------- ##' + @cat $(PERF_TESTSUITE_RESULTS) + @echo + @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" + AUTOTEST = $(AUTOM4TE) --language=autotest @@ -194,6 +217,10 @@ $(SYSTEM_USERSPACE_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT) $(SYSTEM_USERSP $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at $(AM_V_at)mv $@.tmp $@ +$(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at + $(AM_V_at)mv $@.tmp $@ + # The `:;' works around a Bash 3.2 bug when the output is not writeable. $(srcdir)/package.m4: $(top_srcdir)/configure.ac $(AM_V_GEN):;{ \ diff --git a/tests/perf-northd.at b/tests/perf-northd.at new file mode 100644 index 000000000000..eb0c391c4650 --- /dev/null +++ b/tests/perf-northd.at @@ -0,0 +1,207 @@ +AT_BANNER([ovn-northd performance tests]) + +# CACHE_NBDB() +# +# Save the current northbound database for future test runs. +# +m4_define([CACHE_NBDB],[ + mkdir -p ${at_suite_dir}/cached/${at_group} + cp -f ${ovs_base}/ovn-nb/ovn-nb.db ${at_suite_dir}/cached/${at_group}/ +]) + +# BUILD_NBDB([COMMANDS]) +# +# Configure the northbound database using COMMANDS. +# +# BUILD_NBDB() will check if there is a cached nortbound database present +# for this test group. If present, it will use the cached version. If the +# testsuite was run using the '--rebuild' flag, it will force a rebuild of the +# northbound database. +# +m4_define([BUILD_NBDB],[ + if [[ ! -f ${at_suite_dir}/cached/${at_group}/ovn-nb.db ]] || [[ $at_arg_rebuild != false ]]; then + echo "Rebuild NBDB" + $1 + CACHE_NBDB() + else + echo "Using cached NBDB" + rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db + fi + ovn-nbctl --wait=sb sync +]) + +# PERF_RECORD_BANNER() +# +# Append standard banner to performance results. +# +m4_define([PERF_RECORD_START],[ + echo >> ${at_suite_dir}/results + echo "$at_desc_line" >> ${at_suite_dir}/results + echo " ---" >> ${at_suite_dir}/results +]) + +# PERF_RECORD_RESULT([KEY], [VALUE]) +# +# Append KEY and VALUE to performance results. +# +m4_define([PERF_RECORD_RESULT],[ + echo " $1: $2" >> ${at_suite_dir}/results +]) + +m4_define([PARSE_STOPWATCH], [ + grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' +]) + +# PERF_RECORD_STOPWATCH([NAME], [METRIC]) +# +# Append the value of the OVN stopwatch metric METRIC from stopwatch NAME +# to performance results. +# +m4_define([PERF_RECORD_STOPWATCH], [ + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) +]) + +# PERF_RECORD() +# +# Append a number of metrics to performance results +# +m4_define([PERF_RECORD_STOP], [ + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) +]) + +# OVN_NBCTL([NBCTL_COMMAND]) +# +# Add NBCTL_COMMAND to list of commands to be run by RUN_OVN_NBCTL(). +# +m4_define([OVN_NBCTL], [ + command="${command} -- $1" +]) + +# RUN_OVN_NBCTL() +# +# Execute list of commands built by the OVN_NBCTL() macro. +# +m4_define([RUN_OVN_NBCTL], [ + check ovn-nbctl ${command} + unset command +]) + +OVS_START_SHELL_HELPERS +generate_subnet () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + echo $a.$b.0.0/16 +} +generate_ip () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + local c=$(printf %d $(expr $2 / 256)) + local d=$(printf %d $(expr $2 % 256)) + echo $a.$b.$c.$d +} +generate_router_ip () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + echo $a.$b.255.254 +} +generate_mac () { + local a=$(printf %02x $(expr $1 / 256)) + local b=$(printf %02x $(expr $1 % 256)) + local c=$(printf %02x $(expr $2 / 256)) + local d=$(printf %02x $(expr $2 % 256)) + echo f0:00:$a:$b:$c:$d +} +OVS_END_SHELL_HELPERS + +# OVN_BASIC_SCALE_CONFIG(HYPERVISORS, PORTS) +# +# Configures OVN with HYPERVISORS x logical switches. Each logical +# switch has PORTS x logical ports and is connected to one +# Logical Router. The logical router hosts an snat entry for the subnet hosted +# on each logical switch. This is illustrated below. +# +# . +# . +# . +# . +# . +# . Logical Switch ───┐ +# │ │ +# Logical Port ───────────┤ │ +# │ │ +# Logical Port ───────────┼── Logical Switch ───┤ +# │ │ +# Logical Port ───────────┤ ├──── Logical Router +# │ │ (snat entries) +# . Logical Switch ───┤ +# . │ +# . . │ +# . │ +# . +# +m4_define([OVN_BASIC_SCALE_CONFIG], [ + # In order to speed up building the NBDB, we run `ovn-nbctl` in daemon mode. + # We also coalesce `ovn-nbctl` commands using the OVN_NBCTL() and + # RUN_OVN_NBCTL() macros + # + on_exit 'kill $(cat ovn-nbctl.pid)' + export OVN_NB_DAEMON=$(ovn-nbctl --pidfile --detach) + + for hv in $(seq 1 $1); do + echo "Adding hypervisor ${hv}" + total_hv=$1 + logical_switch=lsw${hv} + logical_router=lrw${hv} + logical_router_port=lr${hv}lrp0 + logical_switch_router_port=lr${hv}lsp0 + logical_router_ip=$(generate_router_ip ${hv}) + logical_router_nat_ip=$(generate_ip $((hv+total_hv)) 1) + logical_router_mac=$(generate_mac ${hv} 1) + logical_switch_subnet=$(generate_subnet ${hv}) + + OVN_NBCTL(ls-add $logical_switch) + OVN_NBCTL(set Logical_Switch $logical_switch other_config:subnet=${logical_switch_subnet}) + OVN_NBCTL(set Logical_Switch $logical_switch other_config:exclude_ips=$logical_router_ip) + + OVN_NBCTL(lr-add $logical_router) + OVN_NBCTL(lrp-add $logical_router $logical_router_port $logical_router_mac ${logical_router_ip}/16) + OVN_NBCTL(lr-nat-add $logical_router snat $logical_router_nat_ip $logical_switch_subnet) + OVN_NBCTL(lsp-add $logical_switch $logical_switch_router_port -- set Logical_Switch_Port $logical_switch_router_port type=router options:router-port=$logical_router_port addresses=dynamic) + + for port in $(seq 1 $2); do + logical_switch_port=lsw${hv}lsp${port} + OVN_NBCTL(lsp-add $logical_switch $logical_switch_port) + OVN_NBCTL(lsp-set-addresses $logical_switch_port dynamic) + done + + RUN_OVN_NBCTL() + + done +]) + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor]) +PERF_RECORD_START() + +ovn_start + +BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200)) + +PERF_RECORD_STOP() +AT_CLEANUP +]) + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor]) +PERF_RECORD_START() + +ovn_start + +BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50)) + +PERF_RECORD_STOP() +AT_CLEANUP +]) \ No newline at end of file diff --git a/tests/perf-testsuite.at b/tests/perf-testsuite.at new file mode 100644 index 000000000000..31cdc850c55f --- /dev/null +++ b/tests/perf-testsuite.at @@ -0,0 +1,26 @@ +AT_INIT + +AT_COPYRIGHT([Copyright (c) 2021 Red Hat, + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.]) + +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS]) +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases]) + +m4_include([tests/ovs-macros.at]) +m4_include([tests/ovsdb-macros.at]) +m4_include([tests/ofproto-macros.at]) +m4_include([tests/ovn-macros.at]) + +m4_include([tests/perf-northd.at]) +
Add a suite of micro-benchmarks to aid a developer in understanding the performance impact of any changes that they are making. They can be used to help to understand the relative performance between two test runs on the same test machine, but are not intended to give the absolute performance of OVN. To invoke the performance testsuite, run: $ make check-perf This will run all available performance tests. Additional metrics (e.g. memory, coverage, perf counters) may be added in the future. Additional tests (e.g. additional topologies, ovn-controller tests) may be added in the future. Signed-off-by: Mark Gray <mark.d.gray@redhat.com> --- Notes: v2: create results directory to fix build error v3: forgot to commit, create results directory to fix build error v4: fix 0-day issues remove `sudo` in Makefile updated documentation Documentation/topics/testing.rst | 50 ++++++++ tests/.gitignore | 3 + tests/automake.mk | 27 ++++ tests/perf-northd.at | 207 +++++++++++++++++++++++++++++++ tests/perf-testsuite.at | 26 ++++ 5 files changed, 313 insertions(+) create mode 100644 tests/perf-northd.at create mode 100644 tests/perf-testsuite.at