| Message ID | 20210628180028.581283-1-aconole@redhat.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [ovs-dev] Testing: Introduce "bring your own lab" as an additional test option | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | warning | apply and check: warning |
| ovsrobot/github-robot | success | github build: passed |
Bleep bloop. Greetings Aaron Conole, I am a robot and I have tried out your patch. Thanks for your contribution. I encountered some error that I wasn't expecting. See the details below. checkpatch: WARNING: Line is 81 characters long (recommended limit is 79) #162 FILE: Documentation/topics/testing.rst:491: on the outcome of the test. This value is case-sensitive. PATCHID should be the WARNING: Line is 82 characters long (recommended limit is 79) #163 FILE: Documentation/topics/testing.rst:492: patch for which the test was executed, and ONE LINE DESCRIPTION should be a simple WARNING: Line is 100 characters long (recommended limit is 79) #174 FILE: Documentation/topics/testing.rst:503: Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore macro definitions of FOR_EACH Lines checked: 214, Warnings: 3, Errors: 0 Please check this out. If you feel there has been an error, please email aconole@redhat.com Thanks, 0-day Robot
> Two big changes are announced here, and the first pertains to > allowing other developers and labs to report patch status in the > ozlabs patchwork UI to assist maintainers making decisions on > patch inclusion. > > To make this as easy as possible, the robot will take a known form > of email sent to the ovs-build@openvswitch.org mailing list and > push the information into patchwork. This information can be browsed > either via the web UI or programmatically via the checks API. > > The second change is to de-emphasize the use of Travis-CI as a project > wide resource. Travis-CI might still be useful for individual developers > testing small patches, but can no longer fulfill the role of project-wide > continuous integration service. Remove the badge from the front page, and > note the other CI services that OVS supports. Rather than completely drop > all references to travis, simply emphasize using GitHub Actions as the > preferred CI solution. > Hi Aaron, thanks for driving this. A few comments below but overall agree with this process. > Signed-off-by: Aaron Conole <aconole@redhat.com> > Refs: https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384040.html > --- > .../contributing/submitting-patches.rst | 8 +- > Documentation/topics/testing.rst | 122 ++++++++++++------ > NEWS | 3 + > README.rst | 2 - > 4 files changed, 90 insertions(+), 45 deletions(-) > > diff --git a/Documentation/internals/contributing/submitting-patches.rst > b/Documentation/internals/contributing/submitting-patches.rst > index 4a6780371d..a6a10f8d89 100644 > --- a/Documentation/internals/contributing/submitting-patches.rst > +++ b/Documentation/internals/contributing/submitting-patches.rst > @@ -68,10 +68,10 @@ Testing is also important: > feature. A bug fix patch should preferably add a test that would > fail if the bug recurs. > > -If you are using GitHub, then you may utilize the travis-ci.org and the GitHub > -Actions CI build systems. They will run some of the above tests automatically > -when you push changes to your repository. See the "Continuous Integration > with > -Travis-CI" in :doc:`/topics/testing` for details on how to set it up. > +If you are using GitHub, then you may utilize the the GitHub Actions CI build > +systems. They will run some of the above tests automatically when you push > +changes to your repository. See the "Continuous Integration" section in > +:doc:`/topics/testing` for details on continuous integration. > > Email Subject > ------------- > diff --git a/Documentation/topics/testing.rst > b/Documentation/topics/testing.rst > index 951fe9e851..7e0ced8855 100644 > --- a/Documentation/topics/testing.rst > +++ b/Documentation/topics/testing.rst > @@ -416,45 +416,89 @@ You should invoke scan-view to view analysis results. > The last line of output > from ``clang-analyze`` will list the command (containing results directory) > that you should invoke to view the results on a browser. > > -Continuous Integration with Travis CI > -------------------------------------- > - > -A .travis.yml file is provided to automatically build Open vSwitch with various > -build configurations and run the testsuite using Travis CI. Builds will be > -performed with gcc, sparse and clang with the -Werror compiler flag included, > -therefore the build will fail if a new warning has been introduced. > - > -The CI build is triggered via git push (regardless of the specific branch) or > -pull request against any Open vSwitch GitHub repository that is linked to > -travis-ci. > - > -Instructions to setup travis-ci for your GitHub repository: > - > -1. Go to https://travis-ci.org/ and sign in using your GitHub ID. > -2. Go to the "Repositories" tab and enable the ovs repository. You may disable > - builds for pushes or pull requests. > -3. In order to avoid forks sending build failures to the upstream mailing list, > - the notification email recipient is encrypted. If you want to receive email > - notification for build failures, replace the encrypted string: > - > - 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis > - 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org > - 3. Add/replace the notifications section in .travis.yml and fill in the > - secure string as returned by travis encrypt:: > - > - notifications: > - email: > - recipients: > - - secure: "....." > - > - .. note:: > - You may remove/omit the notifications section to fall back to default > - notification behaviour which is to send an email directly to the author and > - committer of the failing commit. Note that the email is only sent if the > - author/committer have commit rights for the particular GitHub repository. > - > -4. Pushing a commit to the repository which breaks the build or the > - testsuite will now trigger a email sent to mylist@mydomain.org > +Continuous Integration > +---------------------- > + > +The Open vSwitch project can make use of multiple public CI services to help > +developers ensure their patches don't introduce additional regressions. Each > +service requires different forms of configuration, and for the supported > +services the configuration file(s) to automatically build Open vSwitch with > +various build configurations and run the testsuites is/are available. For > +example, the GitHub Actions builds will be performed with gcc, sparse and > clang > +with the -Werror compiler flag included, therefore the build will fail if a new > +warning has been introduced. > + > +Each ci system uses a different method of enablement, but most are available > +from the GitHub settings page. By default, Open vSwitch includes a GitHub > +actions running configuration, and this will automatically email the repository > +owner. > + > +Currently, Open vSwitch project enables the following public CI services along > +with the appropriate configuration settings:: > + > + - AppVeyor: appveyor.yml > + - Cirrus-CI: .cirrus.yml > + - GitHub Actions: .github/workflows > + - Travis-CI: .travis.yml Is it worth mentioning 'Read the Docs' for documentation builds? > + > +GitHub Actions is available without any additional configuration. > + > +Additionally, as each patch is posted to the mailing list, the public CI > +machinery will run additional tests on the patches and report results. > + > +Bring Your Own Lab > +------------------ > + > +The Open vSwitch project makes use of the ozlabs patchwork instance for > +patch status and management. This patch tracking tool provides information > +on the state of changes proposed for Open vSwitch. One feature that the > +patchwork instance offers is for the concept of 'Checks' to be reported. > +These are sets of test functionality that can be executed and present > +status 'at-a-glance' with the ability to drill deeper into why a particular > +test failed. > + > +To this end, the Open vSwitch "0-day Robot" will accept emails sent to > +ovs-build@openvswitch.org formatted in the right way to be reflected on this > +page. This allows any lab to contribute to the testing and validation of > +patches. > + > +To report a test status to a particular patch, send exactly one email to > +the mailing formatted as such:: > + > + From: your email <address@server.com> > + To: ovs-build@openvswitch.org > + Date: Mon, 28 Jun 2021 00:00:00 +0000 > + Subject: |STATUS| pwPATCHID commit subject > + > + Test-Label: your-robot-or-test-name > + Test-Status: STATUS > + http://patchwork.ozlabs.org/api/patches/PATCHID/ > + > + _ONE LINE DESCRIPTION_ +1 really like the approach above. Looking to set something up in the Intel labs to submit to this ourselves. > + > + Addtional details Is the purpose of additional details to provide info on the Hardware setup, OVS configuration, logs for tests that fail etc? Or do you envision that going elsewhere? > + > +In the above example, the STATUS is one of "success" "warning" "fail" > depending > +on the outcome of the test. This value is case-sensitive. PATCHID should be > the > +patch for which the test was executed, and ONE LINE DESCRIPTION should be > a simple > +one line result description, ex: "robot-test: success" > + > +It is strongly recommended that if the patch is reporting a failure case, one > +of the patch authors is cc'd as well. > + > +Example:: > + > + From: 0-Day Robot <robot@bytheb.org> > + To: ovs-build@openvswitch.org > + Date: Fri, 25 Jun 2021 14:30:37 +0400 > + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore > macro definitions of FOR_EACH > + > + Test-Label: github-robot > + Test-Status: success > + http://patchwork.ozlabs.org/api/patches/1497375/ > + > + _github build: passed_ > + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336 > So this will work for functional testing, I'm wondering if we were to do regression testing for performance how we would approach it? For sure if you failed due to performance there is probably a lot of detail required to understand in order to reproduce. Maybe we're getting ahead of ourselves here and functional testing is a good place to start? Regards Ian > vsperf > ------ > diff --git a/NEWS b/NEWS > index b9cb74e9e6..a771eec1a1 100644 > --- a/NEWS > +++ b/NEWS > @@ -21,6 +21,9 @@ Post-v2.15.0 > - ovsdb-tool: > * New option '--election-timer' to the 'create-cluster' command to set the > leader election timer during cluster creation. > + - Testing: > + * Integrate "Bring your own lab" to the build reports page, allowing any > + testing lab to submit a test report for a patch. > > > v2.15.0 - 15 Feb 2021 > diff --git a/README.rst b/README.rst > index 319f705154..8e64d74aee 100644 > --- a/README.rst > +++ b/README.rst > @@ -8,8 +8,6 @@ Open vSwitch > > .. image:: > https://github.com/openvswitch/ovs/workflows/Build%20and%20Test/badge.s > vg > :target: https://github.com/openvswitch/ovs/actions > -.. image:: https://travis-ci.org/openvswitch/ovs.png > - :target: https://travis-ci.org/openvswitch/ovs > .. image:: > https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch= > master&svg=true&retina=true > :target: https://ci.appveyor.com/project/blp/ovs/history > .. image:: https://api.cirrus-ci.com/github/openvswitch/ovs.svg > -- > 2.31.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
"Stokes, Ian" <ian.stokes@intel.com> writes: >> Two big changes are announced here, and the first pertains to >> allowing other developers and labs to report patch status in the >> ozlabs patchwork UI to assist maintainers making decisions on >> patch inclusion. >> >> To make this as easy as possible, the robot will take a known form >> of email sent to the ovs-build@openvswitch.org mailing list and >> push the information into patchwork. This information can be browsed >> either via the web UI or programmatically via the checks API. >> >> The second change is to de-emphasize the use of Travis-CI as a project >> wide resource. Travis-CI might still be useful for individual developers >> testing small patches, but can no longer fulfill the role of project-wide >> continuous integration service. Remove the badge from the front page, and >> note the other CI services that OVS supports. Rather than completely drop >> all references to travis, simply emphasize using GitHub Actions as the >> preferred CI solution. >> > > Hi Aaron, thanks for driving this. A few comments below but overall agree with this process. > >> Signed-off-by: Aaron Conole <aconole@redhat.com> >> Refs: https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384040.html >> --- >> .../contributing/submitting-patches.rst | 8 +- >> Documentation/topics/testing.rst | 122 ++++++++++++------ >> NEWS | 3 + >> README.rst | 2 - >> 4 files changed, 90 insertions(+), 45 deletions(-) >> >> diff --git a/Documentation/internals/contributing/submitting-patches.rst >> b/Documentation/internals/contributing/submitting-patches.rst >> index 4a6780371d..a6a10f8d89 100644 >> --- a/Documentation/internals/contributing/submitting-patches.rst >> +++ b/Documentation/internals/contributing/submitting-patches.rst >> @@ -68,10 +68,10 @@ Testing is also important: >> feature. A bug fix patch should preferably add a test that would >> fail if the bug recurs. >> >> -If you are using GitHub, then you may utilize the travis-ci.org and the GitHub >> -Actions CI build systems. They will run some of the above tests automatically >> -when you push changes to your repository. See the "Continuous Integration >> with >> -Travis-CI" in :doc:`/topics/testing` for details on how to set it up. >> +If you are using GitHub, then you may utilize the the GitHub Actions CI build >> +systems. They will run some of the above tests automatically when you push >> +changes to your repository. See the "Continuous Integration" section in >> +:doc:`/topics/testing` for details on continuous integration. >> >> Email Subject >> ------------- >> diff --git a/Documentation/topics/testing.rst >> b/Documentation/topics/testing.rst >> index 951fe9e851..7e0ced8855 100644 >> --- a/Documentation/topics/testing.rst >> +++ b/Documentation/topics/testing.rst >> @@ -416,45 +416,89 @@ You should invoke scan-view to view analysis results. >> The last line of output >> from ``clang-analyze`` will list the command (containing results directory) >> that you should invoke to view the results on a browser. >> >> -Continuous Integration with Travis CI >> -------------------------------------- >> - >> -A .travis.yml file is provided to automatically build Open vSwitch with various >> -build configurations and run the testsuite using Travis CI. Builds will be >> -performed with gcc, sparse and clang with the -Werror compiler flag included, >> -therefore the build will fail if a new warning has been introduced. >> - >> -The CI build is triggered via git push (regardless of the specific branch) or >> -pull request against any Open vSwitch GitHub repository that is linked to >> -travis-ci. >> - >> -Instructions to setup travis-ci for your GitHub repository: >> - >> -1. Go to https://travis-ci.org/ and sign in using your GitHub ID. >> -2. Go to the "Repositories" tab and enable the ovs repository. You may disable >> - builds for pushes or pull requests. >> -3. In order to avoid forks sending build failures to the upstream mailing list, >> - the notification email recipient is encrypted. If you want to receive email >> - notification for build failures, replace the encrypted string: >> - >> - 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis >> - 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org >> - 3. Add/replace the notifications section in .travis.yml and fill in the >> - secure string as returned by travis encrypt:: >> - >> - notifications: >> - email: >> - recipients: >> - - secure: "....." >> - >> - .. note:: >> - You may remove/omit the notifications section to fall back to default >> - notification behaviour which is to send an email directly to the author and >> - committer of the failing commit. Note that the email is only sent if the >> - author/committer have commit rights for the particular GitHub repository. >> - >> -4. Pushing a commit to the repository which breaks the build or the >> - testsuite will now trigger a email sent to mylist@mydomain.org >> +Continuous Integration >> +---------------------- >> + >> +The Open vSwitch project can make use of multiple public CI services to help >> +developers ensure their patches don't introduce additional regressions. Each >> +service requires different forms of configuration, and for the supported >> +services the configuration file(s) to automatically build Open vSwitch with >> +various build configurations and run the testsuites is/are available. For >> +example, the GitHub Actions builds will be performed with gcc, sparse and >> clang >> +with the -Werror compiler flag included, therefore the build will fail if a new >> +warning has been introduced. >> + >> +Each ci system uses a different method of enablement, but most are available >> +from the GitHub settings page. By default, Open vSwitch includes a GitHub >> +actions running configuration, and this will automatically email the repository >> +owner. >> + >> +Currently, Open vSwitch project enables the following public CI services along >> +with the appropriate configuration settings:: >> + >> + - AppVeyor: appveyor.yml >> + - Cirrus-CI: .cirrus.yml >> + - GitHub Actions: .github/workflows >> + - Travis-CI: .travis.yml > > Is it worth mentioning 'Read the Docs' for documentation builds? Good point, I'll add it. >> + >> +GitHub Actions is available without any additional configuration. >> + >> +Additionally, as each patch is posted to the mailing list, the public CI >> +machinery will run additional tests on the patches and report results. >> + >> +Bring Your Own Lab >> +------------------ >> + >> +The Open vSwitch project makes use of the ozlabs patchwork instance for >> +patch status and management. This patch tracking tool provides information >> +on the state of changes proposed for Open vSwitch. One feature that the >> +patchwork instance offers is for the concept of 'Checks' to be reported. >> +These are sets of test functionality that can be executed and present >> +status 'at-a-glance' with the ability to drill deeper into why a particular >> +test failed. >> + >> +To this end, the Open vSwitch "0-day Robot" will accept emails sent to >> +ovs-build@openvswitch.org formatted in the right way to be reflected on this >> +page. This allows any lab to contribute to the testing and validation of >> +patches. >> + >> +To report a test status to a particular patch, send exactly one email to >> +the mailing formatted as such:: >> + >> + From: your email <address@server.com> >> + To: ovs-build@openvswitch.org >> + Date: Mon, 28 Jun 2021 00:00:00 +0000 >> + Subject: |STATUS| pwPATCHID commit subject >> + >> + Test-Label: your-robot-or-test-name >> + Test-Status: STATUS >> + http://patchwork.ozlabs.org/api/patches/PATCHID/ >> + >> + _ONE LINE DESCRIPTION_ > > +1 really like the approach above. Looking to set something up in the Intel labs to submit to this ourselves. > >> + >> + Addtional details > > Is the purpose of additional details to provide info on the Hardware setup, OVS configuration, logs for tests that fail etc? Or do you envision that going elsewhere? The idea behind additional details is to provide logs, configuration, etc. that can be used by the developer or maintainer to assess the report (mostly in the case of Test-Status: fail). The system will provide a link to the HTML mailing list archive, so being able to click through from Patchwork is quite useful. I'll clarify it in the doc. >> + >> +In the above example, the STATUS is one of "success" "warning" "fail" >> depending >> +on the outcome of the test. This value is case-sensitive. PATCHID should be >> the >> +patch for which the test was executed, and ONE LINE DESCRIPTION should be >> a simple >> +one line result description, ex: "robot-test: success" >> + >> +It is strongly recommended that if the patch is reporting a failure case, one >> +of the patch authors is cc'd as well. >> + >> +Example:: >> + >> + From: 0-Day Robot <robot@bytheb.org> >> + To: ovs-build@openvswitch.org >> + Date: Fri, 25 Jun 2021 14:30:37 +0400 >> + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore >> macro definitions of FOR_EACH >> + >> + Test-Label: github-robot >> + Test-Status: success >> + http://patchwork.ozlabs.org/api/patches/1497375/ >> + >> + _github build: passed_ >> + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336 >> > > So this will work for functional testing, I'm wondering if we were to do regression testing for performance how we would approach it? > > For sure if you failed due to performance there is probably a lot of detail required to understand in order to reproduce. > > Maybe we're getting ahead of ourselves here and functional testing is a good place to start? There are a few ways to slice performance testing: 1. patch-by-patch (which is quite helpful for pushing into patchwork) 2. Periodic tip-of-tree testing for mainline (which does performance after merges) 3. "regression" for different branches (like dpdk-latest, etc). This might be considered a form of (2) above. The one I think makes most sense in this testing framework is (1), but it misses testing after patches get merged (for example, two patches might "pass" performance testing separately, but fail when merged). This is because we associate 'checks' to a patch. But it might be okay to have a periodic patch that we post to do this (with some kind of special keyword, maybe? Just thinking out loud). Even still, "performance" is a nebulous concept to try and test. Some vendors might not want to expose all the information on setups that they use, etc. As you write, it's probably best to start with functional testing and vet it out. Thanks for the review, Ian! > Regards > Ian > > >> vsperf >> ------ >> diff --git a/NEWS b/NEWS >> index b9cb74e9e6..a771eec1a1 100644 >> --- a/NEWS >> +++ b/NEWS >> @@ -21,6 +21,9 @@ Post-v2.15.0 >> - ovsdb-tool: >> * New option '--election-timer' to the 'create-cluster' command to set the >> leader election timer during cluster creation. >> + - Testing: >> + * Integrate "Bring your own lab" to the build reports page, allowing any >> + testing lab to submit a test report for a patch. >> >> >> v2.15.0 - 15 Feb 2021 >> diff --git a/README.rst b/README.rst >> index 319f705154..8e64d74aee 100644 >> --- a/README.rst >> +++ b/README.rst >> @@ -8,8 +8,6 @@ Open vSwitch >> >> .. image:: >> https://github.com/openvswitch/ovs/workflows/Build%20and%20Test/badge.s >> vg >> :target: https://github.com/openvswitch/ovs/actions >> -.. image:: https://travis-ci.org/openvswitch/ovs.png >> - :target: https://travis-ci.org/openvswitch/ovs >> .. image:: >> https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch= >> master&svg=true&retina=true >> :target: https://ci.appveyor.com/project/blp/ovs/history >> .. image:: https://api.cirrus-ci.com/github/openvswitch/ovs.svg >> -- >> 2.31.1 >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
diff --git a/Documentation/internals/contributing/submitting-patches.rst b/Documentation/internals/contributing/submitting-patches.rst index 4a6780371d..a6a10f8d89 100644 --- a/Documentation/internals/contributing/submitting-patches.rst +++ b/Documentation/internals/contributing/submitting-patches.rst @@ -68,10 +68,10 @@ Testing is also important: feature. A bug fix patch should preferably add a test that would fail if the bug recurs. -If you are using GitHub, then you may utilize the travis-ci.org and the GitHub -Actions CI build systems. They will run some of the above tests automatically -when you push changes to your repository. See the "Continuous Integration with -Travis-CI" in :doc:`/topics/testing` for details on how to set it up. +If you are using GitHub, then you may utilize the the GitHub Actions CI build +systems. They will run some of the above tests automatically when you push +changes to your repository. See the "Continuous Integration" section in +:doc:`/topics/testing` for details on continuous integration. Email Subject ------------- diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index 951fe9e851..7e0ced8855 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -416,45 +416,89 @@ You should invoke scan-view to view analysis results. The last line of output from ``clang-analyze`` will list the command (containing results directory) that you should invoke to view the results on a browser. -Continuous Integration with Travis CI -------------------------------------- - -A .travis.yml file is provided to automatically build Open vSwitch with various -build configurations and run the testsuite using Travis CI. Builds will be -performed with gcc, sparse and clang with the -Werror compiler flag included, -therefore the build will fail if a new warning has been introduced. - -The CI build is triggered via git push (regardless of the specific branch) or -pull request against any Open vSwitch GitHub repository that is linked to -travis-ci. - -Instructions to setup travis-ci for your GitHub repository: - -1. Go to https://travis-ci.org/ and sign in using your GitHub ID. -2. Go to the "Repositories" tab and enable the ovs repository. You may disable - builds for pushes or pull requests. -3. In order to avoid forks sending build failures to the upstream mailing list, - the notification email recipient is encrypted. If you want to receive email - notification for build failures, replace the encrypted string: - - 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis - 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org - 3. Add/replace the notifications section in .travis.yml and fill in the - secure string as returned by travis encrypt:: - - notifications: - email: - recipients: - - secure: "....." - - .. note:: - You may remove/omit the notifications section to fall back to default - notification behaviour which is to send an email directly to the author and - committer of the failing commit. Note that the email is only sent if the - author/committer have commit rights for the particular GitHub repository. - -4. Pushing a commit to the repository which breaks the build or the - testsuite will now trigger a email sent to mylist@mydomain.org +Continuous Integration +---------------------- + +The Open vSwitch project can make use of multiple public CI services to help +developers ensure their patches don't introduce additional regressions. Each +service requires different forms of configuration, and for the supported +services the configuration file(s) to automatically build Open vSwitch with +various build configurations and run the testsuites is/are available. For +example, the GitHub Actions builds will be performed with gcc, sparse and clang +with the -Werror compiler flag included, therefore the build will fail if a new +warning has been introduced. + +Each ci system uses a different method of enablement, but most are available +from the GitHub settings page. By default, Open vSwitch includes a GitHub +actions running configuration, and this will automatically email the repository +owner. + +Currently, Open vSwitch project enables the following public CI services along +with the appropriate configuration settings:: + + - AppVeyor: appveyor.yml + - Cirrus-CI: .cirrus.yml + - GitHub Actions: .github/workflows + - Travis-CI: .travis.yml + +GitHub Actions is available without any additional configuration. + +Additionally, as each patch is posted to the mailing list, the public CI +machinery will run additional tests on the patches and report results. + +Bring Your Own Lab +------------------ + +The Open vSwitch project makes use of the ozlabs patchwork instance for +patch status and management. This patch tracking tool provides information +on the state of changes proposed for Open vSwitch. One feature that the +patchwork instance offers is for the concept of 'Checks' to be reported. +These are sets of test functionality that can be executed and present +status 'at-a-glance' with the ability to drill deeper into why a particular +test failed. + +To this end, the Open vSwitch "0-day Robot" will accept emails sent to +ovs-build@openvswitch.org formatted in the right way to be reflected on this +page. This allows any lab to contribute to the testing and validation of +patches. + +To report a test status to a particular patch, send exactly one email to +the mailing formatted as such:: + + From: your email <address@server.com> + To: ovs-build@openvswitch.org + Date: Mon, 28 Jun 2021 00:00:00 +0000 + Subject: |STATUS| pwPATCHID commit subject + + Test-Label: your-robot-or-test-name + Test-Status: STATUS + http://patchwork.ozlabs.org/api/patches/PATCHID/ + + _ONE LINE DESCRIPTION_ + + Addtional details + +In the above example, the STATUS is one of "success" "warning" "fail" depending +on the outcome of the test. This value is case-sensitive. PATCHID should be the +patch for which the test was executed, and ONE LINE DESCRIPTION should be a simple +one line result description, ex: "robot-test: success" + +It is strongly recommended that if the patch is reporting a failure case, one +of the patch authors is cc'd as well. + +Example:: + + From: 0-Day Robot <robot@bytheb.org> + To: ovs-build@openvswitch.org + Date: Fri, 25 Jun 2021 14:30:37 +0400 + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore macro definitions of FOR_EACH + + Test-Label: github-robot + Test-Status: success + http://patchwork.ozlabs.org/api/patches/1497375/ + + _github build: passed_ + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336 vsperf ------ diff --git a/NEWS b/NEWS index b9cb74e9e6..a771eec1a1 100644 --- a/NEWS +++ b/NEWS @@ -21,6 +21,9 @@ Post-v2.15.0 - ovsdb-tool: * New option '--election-timer' to the 'create-cluster' command to set the leader election timer during cluster creation. + - Testing: + * Integrate "Bring your own lab" to the build reports page, allowing any + testing lab to submit a test report for a patch. v2.15.0 - 15 Feb 2021 diff --git a/README.rst b/README.rst index 319f705154..8e64d74aee 100644 --- a/README.rst +++ b/README.rst @@ -8,8 +8,6 @@ Open vSwitch .. image:: https://github.com/openvswitch/ovs/workflows/Build%20and%20Test/badge.svg :target: https://github.com/openvswitch/ovs/actions -.. image:: https://travis-ci.org/openvswitch/ovs.png - :target: https://travis-ci.org/openvswitch/ovs .. image:: https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch=master&svg=true&retina=true :target: https://ci.appveyor.com/project/blp/ovs/history .. image:: https://api.cirrus-ci.com/github/openvswitch/ovs.svg
Two big changes are announced here, and the first pertains to allowing other developers and labs to report patch status in the ozlabs patchwork UI to assist maintainers making decisions on patch inclusion. To make this as easy as possible, the robot will take a known form of email sent to the ovs-build@openvswitch.org mailing list and push the information into patchwork. This information can be browsed either via the web UI or programmatically via the checks API. The second change is to de-emphasize the use of Travis-CI as a project wide resource. Travis-CI might still be useful for individual developers testing small patches, but can no longer fulfill the role of project-wide continuous integration service. Remove the badge from the front page, and note the other CI services that OVS supports. Rather than completely drop all references to travis, simply emphasize using GitHub Actions as the preferred CI solution. Signed-off-by: Aaron Conole <aconole@redhat.com> Refs: https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384040.html --- .../contributing/submitting-patches.rst | 8 +- Documentation/topics/testing.rst | 122 ++++++++++++------ NEWS | 3 + README.rst | 2 - 4 files changed, 90 insertions(+), 45 deletions(-)