From patchwork Tue Dec 13 17:40:40 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stephen Finucane X-Patchwork-Id: 705473 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3tdRqV0fdsz9t2b for ; Wed, 14 Dec 2016 04:42:30 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="key not found in DNS" (0-bit key; unprotected) header.d=that.guru header.i=@that.guru header.b="qivA34tR"; dkim-atps=neutral Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id EF747BDA; Tue, 13 Dec 2016 17:41:00 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 60287BA9 for ; Tue, 13 Dec 2016 17:40:58 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from brown.birch.relay.mailchannels.net (brown.birch.relay.mailchannels.net [23.83.209.23]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 60C561BF for ; Tue, 13 Dec 2016 17:40:56 +0000 (UTC) X-Sender-Id: mxroute|x-authuser|stephen@that.guru Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id A504FE1D75 for ; Tue, 13 Dec 2016 17:40:55 +0000 (UTC) Received: from one.mxroute.com (ip-10-107-69-155.us-west-2.compute.internal [10.107.69.155]) by relay.mailchannels.net (Postfix) with ESMTPA id 07076E1E6A for ; Tue, 13 Dec 2016 17:40:54 +0000 (UTC) X-Sender-Id: mxroute|x-authuser|stephen@that.guru Received: from one.mxroute.com ([TEMPUNAVAIL]. [10.25.23.62]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384) by 0.0.0.0:2500 (trex/5.7.8); Tue, 13 Dec 2016 17:40:55 +0000 X-MC-Relay: Neutral X-MailChannels-SenderId: mxroute|x-authuser|stephen@that.guru X-MailChannels-Auth-Id: mxroute X-MC-Loop-Signature: 1481650855429:1850786986 X-MC-Ingress-Time: 1481650855428 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=that.guru; s=default; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=UOCRu+m9cbUQFLEeaEL04LNiVbtu6uJUcKEl+0naMr8=; b=qivA34tRCbsW3P3RfUSLfXMnS8 a28c95U+OtBMtlsx8lC9OOyiQ+yUu30dVHsCekJFOhIaOoPpLEazcSlPQ8UjRq/0UMDgVhvRhh/dx NWaO+/T+ekv82SUxICwctngxaIGXewiDzXbAy5nW9DaIPFKrXeRbTIfO0GCsWPP1zYwp0uaU2ibtP sGhnwLTFFK6oWgJ+X9CAJ6rE14vLuK71WhqPoR2aXzwZQUvuzjaXr3q9fywftQtqBBjMUnPvjHJio YUu8qD/2RCCod4nCWzBSIg9RpQuohr1oZC2QYDGgKyWtaWsWo6oDdUH/I/QrYV+cjEO5K9cGndfY/ mdtj0Zcw==; From: Stephen Finucane To: dev@openvswitch.org Date: Tue, 13 Dec 2016 17:40:40 +0000 Message-Id: <20161213174042.6098-4-stephen@that.guru> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20161213174042.6098-1-stephen@that.guru> References: <20161213174042.6098-1-stephen@that.guru> X-AuthUser: stephen@that.guru X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 3/5] doc: Move testing to testing section X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This makes more sense here, seeing as it's not exactly installation related. Signed-off-by: Stephen Finucane --- Documentation/intro/install/general.rst | 348 +------------------------------ Documentation/topics/testing.rst | 359 +++++++++++++++++++++++++++++++- 2 files changed, 359 insertions(+), 348 deletions(-) diff --git a/Documentation/intro/install/general.rst b/Documentation/intro/install/general.rst index 89b4e74..371643a 100644 --- a/Documentation/intro/install/general.rst +++ b/Documentation/intro/install/general.rst @@ -436,7 +436,8 @@ and ``vif1.0`` to it:: $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 vif1.0 -Refer to ovs-vsctl(8) for more details. +Refer to ovs-vsctl(8) for more details. You may also wish to refer to +:doc:`/topics/testing` for information on more generic testing of OVS. Upgrading --------- @@ -511,351 +512,6 @@ above, but also replaces the old kernel module with the new one. Open vSwitch startup scripts for Debian, XenServer and RHEL use ovs-ctl's functions and it is recommended that these functions be used for other software platforms too. -.. _general-testing: - -Testing -------- - -This section describe Open vSwitch's built-in support for various test -suites. You must bootstrap, configure and build Open vSwitch (steps are -in "Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD" -above) before you run the tests described here. You do not need to -install Open vSwitch or to build or load the kernel module to run these -test suites. You do not need supervisor privilege to run these test -suites. - -Unit Tests -~~~~~~~~~~ - -Open vSwitch includes a suite of self-tests. Before you submit patches -upstream, we advise that you run the tests and ensure that they pass. If you -add new features to Open vSwitch, then adding tests for those features will -ensure your features don't break as developers modify other areas of Open -vSwitch. - -To run all the unit tests in Open vSwitch, one at a time, run:: - - $ make check - -This takes under 5 minutes on a modern desktop system. - -To run all the unit tests in Open vSwitch in parallel, run:: - - $ make check TESTSUITEFLAGS=-j8 - -You can run up to eight threads. This takes under a minute on a modern 4-core -desktop system. - -To see a list of all the available tests, run: - - $ make check TESTSUITEFLAGS=--list - -To run only a subset of tests, e.g. test 123 and tests 477 through 484, run:: - - $ make check TESTSUITEFLAGS='123 477-484' - -Tests do not have inter-dependencies, so you may run any subset. - -To run tests matching a keyword, e.g. ``ovsdb``, run:: - - $ make check TESTSUITEFLAGS='-k ovsdb' - -To see a complete list of test options, run:: - - $ make check TESTSUITEFLAGS=--help - -The results of a testing run are reported in ``tests/testsuite.log``. Report -report test failures as bugs and include the ``testsuite.log`` in your report. - -.. note:: - Sometimes a few tests may fail on some runs but not others. This is usually a - bug in the testsuite, not a bug in Open vSwitch itself. If you find that a - test fails intermittently, please report it, since the developers may not - have noticed. You can make the testsuite automatically rerun tests that fail, - by adding ``RECHECK=yes`` to the ``make`` command line, e.g.:: - - $ make check TESTSUITEFLAGS=-j8 RECHECK=yes - -Coverage -++++++++ - -If the build was configured with ``--enable-coverage`` and the ``lcov`` utility -is installed, you can run the testsuite and generate a code coverage report by -using the ``check-lcoc`` target:: - - $ make check-lcov - -All the same options are avaiable via TESTSUITEFLAGS. For example:: - - $ make check-lcov TESTSUITEFLAGS=-j8 -k ovn - -Valgrind -++++++++ - -If you have ``valgrind`` installed, you can run the testsuite under -valgrind by using the ``check-valgrind`` target:: - - $ make check-valgrind - -When you do this, the "valgrind" results for test ```` are reported in files -named ``tests/testsuite.dir//valgrind.*``. - -All the same options are available via TESTSUITEFLAGS. - -.. hint:: - You may find that the valgrind results are easier to interpret if you put - ``-q`` in ``~/.valgrindrc``, since that reduces the amount of output. - -.. _general-oftest: - -OFTest -~~~~~~ - -OFTest is an OpenFlow protocol testing suite. Open vSwitch includes a Makefile -target to run OFTest with Open vSwitch in "dummy mode". In this mode of -testing, no packets travel across physical or virtual networks. Instead, Unix -domain sockets stand in as simulated networks. This simulation is imperfect, -but it is much easier to set up, does not require extra physical or virtual -hardware, and does not require supervisor privileges. - -To run OFTest with Open vSwitch, first read and follow the instructions under -**Testing** above. Second, obtain a copy of OFTest and install its -prerequisites. You need a copy of OFTest that includes commit 406614846c5 (make -ovs-dummy platform work again). This commit was merged into the OFTest -repository on Feb 1, 2013, so any copy of OFTest more recent than that should -work. Testing OVS in dummy mode does not require root privilege, so you may -ignore that requirement. - -Optionally, add the top-level OFTest directory (containing the ``oft`` program) -to your ``$PATH``. This slightly simplifies running OFTest later. - -To run OFTest in dummy mode, run the following command from your Open vSwitch -build directory:: - - $ make check-oftest OFT= - -where ```` is the absolute path to the ``oft`` program in OFTest. -If you added "oft" to your $PATH, you may omit the OFT variable -assignment - -By default, ``check-oftest`` passes ``oft`` just enough options to enable dummy -mode. You can use ``OFTFLAGS`` to pass additional options. For example, to run -just the ``basic.Echo`` test instead of all tests (the default) and enable -verbose logging, run:: - - $ make check-oftest OFT= OFTFLAGS='--verbose -T basic.Echo' - -If you use OFTest that does not include commit 4d1f3eb2c792 (oft: change -default port to 6653), merged into the OFTest repository in October 2013, then -you need to add an option to use the IETF-assigned controller port:: - - $ make check-oftest OFT= OFTFLAGS='--port=6653' - -Interpret OFTest results cautiously. Open vSwitch can fail a given test in -OFTest for many reasons, including bugs in Open vSwitch, bugs in OFTest, bugs -in the "dummy mode" integration, and differing interpretations of the OpenFlow -standard and other standards. - -.. note:: - Open vSwitch has not been validated against OFTest. Report test failures that - you believe to represent bugs in Open vSwitch. Include the precise versions - of Open vSwitch and OFTest in your bug report, plus any other information - needed to reproduce the problem. - -Ryu -~~~ - -Ryu is an OpenFlow controller written in Python that includes an extensive -OpenFlow testsuite. Open vSwitch includes a Makefile target to run Ryu in -"dummy mode". See **OFTest** above for an explanation of dummy mode. - -To run Ryu tests with Open vSwitch, first read and follow the instructions -under **Testing** above. Second, obtain a copy of Ryu, install its -prerequisites, and build it. You do not need to install Ryu (some of the tests -do not get installed, so it does not help). - -To run Ryu tests, run the following command from your Open vSwitch build -directory:: - - $ make check-ryu RYUDIR=`` - -where ```` is the absolute path to the root of the Ryu source -distribution. The default ```` is ``$srcdir/../ryu`` -where ``$srcdir`` is your Open vSwitch source directory. If this is correct, -omit ``RYUDIR`` - -.. note:: - Open vSwitch has not been validated against Ryu. Report test failures that - you believe to represent bugs in Open vSwitch. Include the precise versions - of Open vSwitch and Ryu in your bug report, plus any other information - needed to reproduce the problem. - -Datapath testing -~~~~~~~~~~~~~~~~ - -Open vSwitch includes a suite of tests specifically for datapath functionality, -which can be run against the userspace or kernel datapaths. If you are -developing datapath features, it is recommended that you use these tests and -build upon them to verify your implementation. - -The datapath tests make some assumptions about the environment. They must be -run under root privileges on a Linux system with support for network -namespaces. For ease of use, the OVS source tree includes a vagrant box to -invoke these tests. Running the tests inside Vagrant provides kernel isolation, -protecting your development host from kernel panics or configuration conflicts -in the testsuite. If you wish to run the tests without using the vagrant box, -there are further instructions below. - -Vagrant -+++++++ - -.. important:: - - Requires Vagrant (version 1.7.0 or later) and a compatible hypervisor - -.. note:: - You must **Bootstrap** and **Configure** the sources before you run the steps - described here. - -A Vagrantfile is provided allowing to compile and provision the source tree as -found locally in a virtual machine using the following command:: - - $ vagrant up - -This will bring up a Fedora 23 VM by default. If you wish to use a different -box or a vagrant backend not supported by the default box, the ``Vagrantfile`` -can be modified to use a different box as base. - -The VM can be reprovisioned at any time:: - - $ vagrant provision - -OVS out-of-tree compilation environment can be set up with:: - - $ ./boot.sh - $ vagrant provision --provision-with configure_ovs,build_ovs - -This will set up an out-of-tree build environment inside the VM in -``/root/build``. The source code can be found in ``/vagrant``. - -To recompile and reinstall OVS in the VM using RPM:: - - $ ./boot.sh - $ vagrant provision --provision-with configure_ovs,install_rpm - -Two provisioners are included to run system tests with the OVS kernel module or -with a userspace datapath. This tests are different from the self-tests -mentioned above. To run them:: - - $ ./boot.sh - $ vagrant provision --provision-with \ - configure_ovs,test_ovs_kmod,test_ovs_system_userspace - -The results of the testsuite reside in the VM root user's home directory:: - - $ vagrant ssh - $ sudo -s - $ cd /root/build - $ ls tests/system* - -Native -++++++ - -The datapath testsuite as invoked by Vagrant above may also be run manually on -a Linux system with root privileges. These tests may take several minutes to -complete, and cannot be run in parallel. - -Userspace datapath -''''''''''''''''''' - -To invoke the datapath testsuite with the userspace datapath, run:: - - $ make check-system-userspace - -The results of the testsuite are in ``tests/system-userspace-traffic.dir``. - -Kernel datapath -''''''''''''''' - -Make targets are also provided for testing the Linux kernel module. Note that -these tests operate by inserting modules into the running Linux kernel, so if -the tests are able to trigger a bug in the OVS kernel module or in the upstream -kernel then the kernel may panic. - -To run the testsuite against the kernel module which is currently installed on -your system, run:: - - $ make check-kernel - -To install the kernel module from the current build directory and run the -testsuite against that kernel module:: - - $ make check-kmod - -The results of the testsuite are in ``tests/system-kmod-traffic.dir``. - -Continuous Integration with Travis-CI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A .travis.yml file is provided to automatically build Open vSwitch with various -build configurations and run the testsuite using travis-ci. Builds will be -performed with gcc, sparse and clang with the -Werror compiler flag included, -therefore the build will fail if a new warning has been introduced. - -The CI build is triggered via git push (regardless of the specific branch) or -pull request against any Open vSwitch GitHub repository that is linked to -travis-ci. - -Instructions to setup travis-ci for your GitHub repository: - -1. Go to http://travis-ci.org/ and sign in using your GitHub ID. -2. Go to the "Repositories" tab and enable the ovs repository. You may disable - builds for pushes or pull requests. -3. In order to avoid forks sending build failures to the upstream mailing list, - the notification email recipient is encrypted. If you want to receive email - notification for build failures, replace the the encrypted string: - - 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis - 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org - 3. Add/replace the notifications section in .travis.yml and fill in the - secure string as returned by travis encrypt:: - - notifications: - email: - recipients: - - secure: "....." - - .. note:: - You may remove/omit the notifications section to fall back to default - notification behaviour which is to send an email directly to the author and - committer of the failing commit. Note that the email is only sent if the - author/committer have commit rights for the particular GitHub repository. - -4. Pushing a commit to the repository which breaks the build or the - testsuite will now trigger a email sent to mylist@mydomain.org - -Static Code Analysis -~~~~~~~~~~~~~~~~~~~~ - -Static Analysis is a method of debugging Software by examining code rather than -actually executing it. This can be done through 'scan-build' commandline -utility which internally uses clang (or) gcc to compile the code and also -invokes a static analyzer to do the code analysis. At the end of the build, the -reports are aggregated in to a common folder and can later be analyzed using -'scan-view'. - -Open vSwitch includes a Makefile target to trigger static code analysis:: - - $ ./boot.sh - $ ./configure CC=clang # clang - # or - $ ./configure CC=gcc CFLAGS="-std=gnu99" # gcc - $ make clang-analyze - -You should invoke scan-view to view analysis results. The last line of output -from ``clang-analyze`` will list the command (containing results directory) -that you should invoke to view the results on a browser. - Reporting Bugs -------------- diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index 5265ab1..599e570 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -25,8 +25,363 @@ Testing ======= -.. TODO(stephenfin): Flesh this out with information from the general - installation guide, among others. +It is possible to test Open vSwitch using both tooling provided with Open +vSwitch and using a variety of third party tooling. + +Built-in Tooling +---------------- + +Open vSwitch provides a number of different test suites and other tooling for +validating basic functionality of OVS. Before running any of the tests +described here, you must bootstrap, configure and build Open vSwitch as +described in :doc:`/intro/install/general`. You do not need to install Open +vSwitch or to build or load the kernel module to run these test suites. You do +not need supervisor privilege to run these test suites. + +.. _testing-unit-tests: + +Unit Tests +~~~~~~~~~~ + +Open vSwitch includes a suite of self-tests. Before you submit patches +upstream, we advise that you run the tests and ensure that they pass. If you +add new features to Open vSwitch, then adding tests for those features will +ensure your features don't break as developers modify other areas of Open +vSwitch. + +To run all the unit tests in Open vSwitch, one at a time, run:: + + $ make check + +This takes under 5 minutes on a modern desktop system. + +To run all the unit tests in Open vSwitch in parallel, run:: + + $ make check TESTSUITEFLAGS=-j8 + +You can run up to eight threads. This takes under a minute on a modern 4-core +desktop system. + +To see a list of all the available tests, run: + + $ make check TESTSUITEFLAGS=--list + +To run only a subset of tests, e.g. test 123 and tests 477 through 484, run:: + + $ make check TESTSUITEFLAGS='123 477-484' + +Tests do not have inter-dependencies, so you may run any subset. + +To run tests matching a keyword, e.g. ``ovsdb``, run:: + + $ make check TESTSUITEFLAGS='-k ovsdb' + +To see a complete list of test options, run:: + + $ make check TESTSUITEFLAGS=--help + +The results of a testing run are reported in ``tests/testsuite.log``. Report +report test failures as bugs and include the ``testsuite.log`` in your report. + +.. note:: + Sometimes a few tests may fail on some runs but not others. This is usually a + bug in the testsuite, not a bug in Open vSwitch itself. If you find that a + test fails intermittently, please report it, since the developers may not + have noticed. You can make the testsuite automatically rerun tests that fail, + by adding ``RECHECK=yes`` to the ``make`` command line, e.g.:: + + $ make check TESTSUITEFLAGS=-j8 RECHECK=yes + +.. _testing-coverage: + +Coverage +~~~~~~~~ + +If the build was configured with ``--enable-coverage`` and the ``lcov`` utility +is installed, you can run the testsuite and generate a code coverage report by +using the ``check-lcoc`` target:: + + $ make check-lcov + +All the same options are avaiable via TESTSUITEFLAGS. For example:: + + $ make check-lcov TESTSUITEFLAGS=-j8 -k ovn + +.. _testing-valgrind: + +Valgrind +~~~~~~~~ + +If you have ``valgrind`` installed, you can run the testsuite under +valgrind by using the ``check-valgrind`` target:: + + $ make check-valgrind + +When you do this, the "valgrind" results for test ```` are reported in files +named ``tests/testsuite.dir//valgrind.*``. + +All the same options are available via TESTSUITEFLAGS. + +.. hint:: + You may find that the valgrind results are easier to interpret if you put + ``-q`` in ``~/.valgrindrc``, since that reduces the amount of output. + +.. _testing-oftest: + +OFTest +~~~~~~ + +OFTest is an OpenFlow protocol testing suite. Open vSwitch includes a Makefile +target to run OFTest with Open vSwitch in "dummy mode". In this mode of +testing, no packets travel across physical or virtual networks. Instead, Unix +domain sockets stand in as simulated networks. This simulation is imperfect, +but it is much easier to set up, does not require extra physical or virtual +hardware, and does not require supervisor privileges. + +To run OFTest with Open vSwitch, you must obtain a copy of OFTest and install +its prerequisites. You need a copy of OFTest that includes commit 406614846c5 +(make ovs-dummy platform work again). This commit was merged into the OFTest +repository on Feb 1, 2013, so any copy of OFTest more recent than that should +work. Testing OVS in dummy mode does not require root privilege, so you may +ignore that requirement. + +Optionally, add the top-level OFTest directory (containing the ``oft`` program) +to your ``$PATH``. This slightly simplifies running OFTest later. + +To run OFTest in dummy mode, run the following command from your Open vSwitch +build directory:: + + $ make check-oftest OFT= + +where ```` is the absolute path to the ``oft`` program in OFTest. +If you added "oft" to your $PATH, you may omit the OFT variable +assignment + +By default, ``check-oftest`` passes ``oft`` just enough options to enable dummy +mode. You can use ``OFTFLAGS`` to pass additional options. For example, to run +just the ``basic.Echo`` test instead of all tests (the default) and enable +verbose logging, run:: + + $ make check-oftest OFT= OFTFLAGS='--verbose -T basic.Echo' + +If you use OFTest that does not include commit 4d1f3eb2c792 (oft: change +default port to 6653), merged into the OFTest repository in October 2013, then +you need to add an option to use the IETF-assigned controller port:: + + $ make check-oftest OFT= OFTFLAGS='--port=6653' + +Interpret OFTest results cautiously. Open vSwitch can fail a given test in +OFTest for many reasons, including bugs in Open vSwitch, bugs in OFTest, bugs +in the "dummy mode" integration, and differing interpretations of the OpenFlow +standard and other standards. + +.. note:: + Open vSwitch has not been validated against OFTest. Report test failures that + you believe to represent bugs in Open vSwitch. Include the precise versions + of Open vSwitch and OFTest in your bug report, plus any other information + needed to reproduce the problem. + +.. _ryu: + +Ryu +~~~ + +Ryu is an OpenFlow controller written in Python that includes an extensive +OpenFlow testsuite. Open vSwitch includes a Makefile target to run Ryu in +"dummy mode". See :ref:`testing-oftest` above for an explanation of dummy mode. + +To run Ryu tests with Open vSwitch, first read and follow the instructions +under **Testing** above. Second, obtain a copy of Ryu, install its +prerequisites, and build it. You do not need to install Ryu (some of the tests +do not get installed, so it does not help). + +To run Ryu tests, run the following command from your Open vSwitch build +directory:: + + $ make check-ryu RYUDIR=`` + +where ```` is the absolute path to the root of the Ryu source +distribution. The default ```` is ``$srcdir/../ryu`` +where ``$srcdir`` is your Open vSwitch source directory. If this is correct, +omit ``RYUDIR`` + +.. note:: + Open vSwitch has not been validated against Ryu. Report test failures that + you believe to represent bugs in Open vSwitch. Include the precise versions + of Open vSwitch and Ryu in your bug report, plus any other information + needed to reproduce the problem. + +Datapath testing +~~~~~~~~~~~~~~~~ + +Open vSwitch includes a suite of tests specifically for datapath functionality, +which can be run against the userspace or kernel datapaths. If you are +developing datapath features, it is recommended that you use these tests and +build upon them to verify your implementation. + +The datapath tests make some assumptions about the environment. They must be +run under root privileges on a Linux system with support for network +namespaces. For ease of use, the OVS source tree includes a vagrant box to +invoke these tests. Running the tests inside Vagrant provides kernel isolation, +protecting your development host from kernel panics or configuration conflicts +in the testsuite. If you wish to run the tests without using the vagrant box, +there are further instructions below. + +Vagrant ++++++++ + +.. important:: + + Requires Vagrant (version 1.7.0 or later) and a compatible hypervisor + +.. note:: + You must **Bootstrap** and **Configure** the sources before you run the steps + described here. + +A Vagrantfile is provided allowing to compile and provision the source tree as +found locally in a virtual machine using the following command:: + + $ vagrant up + +This will bring up a Fedora 23 VM by default. If you wish to use a different +box or a vagrant backend not supported by the default box, the ``Vagrantfile`` +can be modified to use a different box as base. + +The VM can be reprovisioned at any time:: + + $ vagrant provision + +OVS out-of-tree compilation environment can be set up with:: + + $ ./boot.sh + $ vagrant provision --provision-with configure_ovs,build_ovs + +This will set up an out-of-tree build environment inside the VM in +``/root/build``. The source code can be found in ``/vagrant``. + +To recompile and reinstall OVS in the VM using RPM:: + + $ ./boot.sh + $ vagrant provision --provision-with configure_ovs,install_rpm + +Two provisioners are included to run system tests with the OVS kernel module or +with a userspace datapath. This tests are different from the self-tests +mentioned above. To run them:: + + $ ./boot.sh + $ vagrant provision --provision-with \ + configure_ovs,test_ovs_kmod,test_ovs_system_userspace + +The results of the testsuite reside in the VM root user's home directory:: + + $ vagrant ssh + $ sudo -s + $ cd /root/build + $ ls tests/system* + +Native +++++++ + +The datapath testsuite as invoked by Vagrant above may also be run manually on +a Linux system with root privileges. These tests may take several minutes to +complete, and cannot be run in parallel. + +Userspace datapath +''''''''''''''''''' + +To invoke the datapath testsuite with the userspace datapath, run:: + + $ make check-system-userspace + +The results of the testsuite are in ``tests/system-userspace-traffic.dir``. + +Kernel datapath +''''''''''''''' + +Make targets are also provided for testing the Linux kernel module. Note that +these tests operate by inserting modules into the running Linux kernel, so if +the tests are able to trigger a bug in the OVS kernel module or in the upstream +kernel then the kernel may panic. + +To run the testsuite against the kernel module which is currently installed on +your system, run:: + + $ make check-kernel + +To install the kernel module from the current build directory and run the +testsuite against that kernel module:: + + $ make check-kmod + +The results of the testsuite are in ``tests/system-kmod-traffic.dir``. + +.. _testing-static-analysis: + +Static Code Analysis +~~~~~~~~~~~~~~~~~~~~ + +Static Analysis is a method of debugging Software by examining code rather than +actually executing it. This can be done through 'scan-build' commandline +utility which internally uses clang (or) gcc to compile the code and also +invokes a static analyzer to do the code analysis. At the end of the build, the +reports are aggregated in to a common folder and can later be analyzed using +'scan-view'. + +Open vSwitch includes a Makefile target to trigger static code analysis:: + + $ ./boot.sh + $ ./configure CC=clang # clang + # or + $ ./configure CC=gcc CFLAGS="-std=gnu99" # gcc + $ make clang-analyze + +You should invoke scan-view to view analysis results. The last line of output +from ``clang-analyze`` will list the command (containing results directory) +that you should invoke to view the results on a browser. + +.. _testing-ci: + +Continuous Integration with Travis CI +------------------------------------- + +A .travis.yml file is provided to automatically build Open vSwitch with various +build configurations and run the testsuite using Travis CI. Builds will be +performed with gcc, sparse and clang with the -Werror compiler flag included, +therefore the build will fail if a new warning has been introduced. + +The CI build is triggered via git push (regardless of the specific branch) or +pull request against any Open vSwitch GitHub repository that is linked to +travis-ci. + +Instructions to setup travis-ci for your GitHub repository: + +1. Go to http://travis-ci.org/ and sign in using your GitHub ID. +2. Go to the "Repositories" tab and enable the ovs repository. You may disable + builds for pushes or pull requests. +3. In order to avoid forks sending build failures to the upstream mailing list, + the notification email recipient is encrypted. If you want to receive email + notification for build failures, replace the the encrypted string: + + 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis + 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org + 3. Add/replace the notifications section in .travis.yml and fill in the + secure string as returned by travis encrypt:: + + notifications: + email: + recipients: + - secure: "....." + + .. note:: + You may remove/omit the notifications section to fall back to default + notification behaviour which is to send an email directly to the author and + committer of the failing commit. Note that the email is only sent if the + author/committer have commit rights for the particular GitHub repository. + +4. Pushing a commit to the repository which breaks the build or the + testsuite will now trigger a email sent to mylist@mydomain.org + +.. _testing-vsperf: vsperf ------