diff mbox series

[ovs-dev,RFC] cirrus: Add Cirrus CI integration for FreeBSD build

Message ID 20181211143417.21234-1-i.maximets@samsung.com
State Accepted
Headers show
Series [ovs-dev,RFC] cirrus: Add Cirrus CI integration for FreeBSD build | expand

Commit Message

Ilya Maximets Dec. 11, 2018, 2:34 p.m. UTC 
CirrusCI [1] is free for open-sorce projects and provides similar
to TravisCI interfaces. One significant difference is ability
to run tasks on FreeBSD instances.

This patch adds simple configuration file to test OVS build
on two FreeBSD releases with gcc and clang.
Unit tests are commented out because they are broken for now.

To enable the automated checks Cirrus CI application from GitHub
Marketplace should be installed. See details in Quick Start guide [2].

[1] https://cirrus-ci.org
[2] https://cirrus-ci.org/guide/quick-start/

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
---

This could be useful to test FreeBSD build automatically to early
detect issues. CirrusCI looks suitable for that purpose and it's
the only public CI system that I managed to find that supports
FreeBSD images. Sending as RFC to start discussion.

Few more differentiation points with Travis. Cirrus allows to
configure number of CPUs and the amount of required memory. It
also supports Linux, OSX and Windows containers. So, maybe, if
we'll like it, we could drop Travis and AppVeyor and use single
CI system for all the tests.

I'm not advertising and not trying to convince to replace our
current CI systems. I found CirrusCI just few days ago by a mail [3]
in qemu-devel list. I just think that it's the only option for
now to test BSD builds with public CI. And it worked fine for me
with below configuration.

Thoughts ?

As a note: I just spotted that TravisCI adds windows support.
Maybe we could use it in the future for windows builds.

[3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html

 .cirrus.yml | 30 ++++++++++++++++++++++++++++++
 Makefile.am |  1 +
 2 files changed, 31 insertions(+)
 create mode 100644 .cirrus.yml

Comments

Aaron Conole Dec. 11, 2018, 6:25 p.m. UTC | #1 
Ilya Maximets <i.maximets@samsung.com> writes:

> CirrusCI [1] is free for open-sorce projects and provides similar
> to TravisCI interfaces. One significant difference is ability
> to run tasks on FreeBSD instances.
>
> This patch adds simple configuration file to test OVS build
> on two FreeBSD releases with gcc and clang.
> Unit tests are commented out because they are broken for now.
>
> To enable the automated checks Cirrus CI application from GitHub
> Marketplace should be installed. See details in Quick Start guide [2].
>
> [1] https://cirrus-ci.org
> [2] https://cirrus-ci.org/guide/quick-start/
>
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
>
> This could be useful to test FreeBSD build automatically to early
> detect issues. CirrusCI looks suitable for that purpose and it's
> the only public CI system that I managed to find that supports
> FreeBSD images. Sending as RFC to start discussion.

Awesome find.  Thanks, Ilya!

> Few more differentiation points with Travis. Cirrus allows to
> configure number of CPUs and the amount of required memory. It
> also supports Linux, OSX and Windows containers. So, maybe, if
> we'll like it, we could drop Travis and AppVeyor and use single
> CI system for all the tests.
>
> I'm not advertising and not trying to convince to replace our
> current CI systems. I found CirrusCI just few days ago by a mail [3]
> in qemu-devel list. I just think that it's the only option for
> now to test BSD builds with public CI. And it worked fine for me
> with below configuration.
>
> Thoughts ?

I'm not married to technology for a reason :)  If it seems better, then
it is worth checking out.  I can play around with it a little bit, but
I'll be AFK until Jan, so it might be better for someone else who has
the recent time to take a look.

One thing I really find painful with Travis is the difficulty running
the alternate test suites (make check-*).  Travis can run them, but we
need to request privileged containers, and that will add to the build
time.  Not sure if there's some kind of custom privilege support from
Cirrus that would let us run the alternate test suites - or even let us
run them by requesting privileges.

Another thing I'm not sure about is the polling.  At least Travis does
have a RESTful API that we can poll with.  I hope the CirrusCI system
would as well.  Currently, I am looking into getting the robot to poll
the Travis system and respond to a series if there is a problem.

> As a note: I just spotted that TravisCI adds windows support.
> Maybe we could use it in the future for windows builds.
>
> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
>
>  .cirrus.yml | 30 ++++++++++++++++++++++++++++++
>  Makefile.am |  1 +
>  2 files changed, 31 insertions(+)
>  create mode 100644 .cirrus.yml
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> new file mode 100644
> index 000000000..4902fd370
> --- /dev/null
> +++ b/.cirrus.yml
> @@ -0,0 +1,30 @@
> +freebsd_build_task:
> +
> +  freebsd_instance:
> +    matrix:
> +      image: freebsd-12-0-release-amd64
> +      image: freebsd-11-2-release-amd64
> +    cpu: 4
> +    memory: 8G
> +
> +  env:
> +    matrix:
> +      COMPILER: gcc
> +      COMPILER: clang
> +
> +  prepare_script:
> +    - pkg install -y automake libtool gmake gcc wget
> +                     python py27-six py27-sphinx
> +
> +  configure_script:
> +    - ./boot.sh
> +    - ./configure CC=$COMPILER MAKE=gmake OVS_CFLAGS='-Wall' --enable-Werror
> +                  || { cat config.log; exit 1; }
> +
> +  build_script:
> +    - gmake -j8
> +
> +#  TODO(i.maximets): Uncomment the test suite execution when it is fixed.
> +#  check_script:
> +#    - gmake -j8 check TESTSUITEFLAGS=-j8 RECHECK=yes
> +#                || { cat ./tests/testsuite.log; exit 1; }
> diff --git a/Makefile.am b/Makefile.am
> index 64725654c..8408509c9 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -76,6 +76,7 @@ EXTRA_DIST = \
>  	MAINTAINERS.rst \
>  	README.rst \
>  	NOTICE \
> +	.cirrus.yml \
>  	.travis.yml \
>  	.travis/linux-build.sh \
>  	.travis/linux-prepare.sh \
Ben Pfaff Dec. 11, 2018, 9:06 p.m. UTC | #2 
On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
> CirrusCI [1] is free for open-sorce projects and provides similar
> to TravisCI interfaces. One significant difference is ability
> to run tasks on FreeBSD instances.
> 
> This patch adds simple configuration file to test OVS build
> on two FreeBSD releases with gcc and clang.
> Unit tests are commented out because they are broken for now.
> 
> To enable the automated checks Cirrus CI application from GitHub
> Marketplace should be installed. See details in Quick Start guide [2].
> 
> [1] https://cirrus-ci.org
> [2] https://cirrus-ci.org/guide/quick-start/
> 
> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> ---
> 
> This could be useful to test FreeBSD build automatically to early
> detect issues. CirrusCI looks suitable for that purpose and it's
> the only public CI system that I managed to find that supports
> FreeBSD images. Sending as RFC to start discussion.
> 
> Few more differentiation points with Travis. Cirrus allows to
> configure number of CPUs and the amount of required memory. It
> also supports Linux, OSX and Windows containers. So, maybe, if
> we'll like it, we could drop Travis and AppVeyor and use single
> CI system for all the tests.
> 
> I'm not advertising and not trying to convince to replace our
> current CI systems. I found CirrusCI just few days ago by a mail [3]
> in qemu-devel list. I just think that it's the only option for
> now to test BSD builds with public CI. And it worked fine for me
> with below configuration.
> 
> Thoughts ?
> 
> As a note: I just spotted that TravisCI adds windows support.
> Maybe we could use it in the future for windows builds.
> 
> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html

Good idea.  It's hard for me to see what harm this would cause, so I
enabled cirrus-ci at github and applied this to master.

The big weakness with appveyor is that its limits are too low to run the
whole OVS testsuite.  I wonder whether cirrus-ci has better limits for
this purpose.
Aaron Conole Dec. 11, 2018, 9:20 p.m. UTC | #3 
On Tue, 11 Dec 2018 13:06:36 -0800, Ben Pfaff <blp@ovn.org> wrote:

> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
>> CirrusCI [1] is free for open-sorce projects and provides similar
>> to TravisCI interfaces. One significant difference is ability
>> to run tasks on FreeBSD instances.
>> 
>> This patch adds simple configuration file to test OVS build
>> on two FreeBSD releases with gcc and clang.
>> Unit tests are commented out because they are broken for now.
>> 
>> To enable the automated checks Cirrus CI application from GitHub
>> Marketplace should be installed. See details in Quick Start guide [2].
>> 
>> [1] https://cirrus-ci.org
>> [2] https://cirrus-ci.org/guide/quick-start/
>> 
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>> 
>> This could be useful to test FreeBSD build automatically to early
>> detect issues. CirrusCI looks suitable for that purpose and it's
>> the only public CI system that I managed to find that supports
>> FreeBSD images. Sending as RFC to start discussion.
>> 
>> Few more differentiation points with Travis. Cirrus allows to
>> configure number of CPUs and the amount of required memory. It
>> also supports Linux, OSX and Windows containers. So, maybe, if
>> we'll like it, we could drop Travis and AppVeyor and use single
>> CI system for all the tests.
>> 
>> I'm not advertising and not trying to convince to replace our
>> current CI systems. I found CirrusCI just few days ago by a mail [3]
>> in qemu-devel list. I just think that it's the only option for
>> now to test BSD builds with public CI. And it worked fine for me
>> with below configuration.
>> 
>> Thoughts ?
>> 
>> As a note: I just spotted that TravisCI adds windows support.
>> Maybe we could use it in the future for windows builds.
>> 
>> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
>
> Good idea.  It's hard for me to see what harm this would cause, so I
> enabled cirrus-ci at github and applied this to master.

FYI, I've also enabled the cirrus-ci under the ovsrobot, in tandem.

> The big weakness with appveyor is that its limits are too low to run the
> whole OVS testsuite.  I wonder whether cirrus-ci has better limits for
> this purpose.
Ilya Maximets Dec. 12, 2018, 4 p.m. UTC | #4 
On 12.12.2018 0:06, Ben Pfaff wrote:
> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
>> CirrusCI [1] is free for open-sorce projects and provides similar
>> to TravisCI interfaces. One significant difference is ability
>> to run tasks on FreeBSD instances.
>>
>> This patch adds simple configuration file to test OVS build
>> on two FreeBSD releases with gcc and clang.
>> Unit tests are commented out because they are broken for now.
>>
>> To enable the automated checks Cirrus CI application from GitHub
>> Marketplace should be installed. See details in Quick Start guide [2].
>>
>> [1] https://cirrus-ci.org
>> [2] https://cirrus-ci.org/guide/quick-start/
>>
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>
>> This could be useful to test FreeBSD build automatically to early
>> detect issues. CirrusCI looks suitable for that purpose and it's
>> the only public CI system that I managed to find that supports
>> FreeBSD images. Sending as RFC to start discussion.
>>
>> Few more differentiation points with Travis. Cirrus allows to
>> configure number of CPUs and the amount of required memory. It
>> also supports Linux, OSX and Windows containers. So, maybe, if
>> we'll like it, we could drop Travis and AppVeyor and use single
>> CI system for all the tests.
>>
>> I'm not advertising and not trying to convince to replace our
>> current CI systems. I found CirrusCI just few days ago by a mail [3]
>> in qemu-devel list. I just think that it's the only option for
>> now to test BSD builds with public CI. And it worked fine for me
>> with below configuration.
>>
>> Thoughts ?
>>
>> As a note: I just spotted that TravisCI adds windows support.
>> Maybe we could use it in the future for windows builds.
>>
>> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
> 
> Good idea.  It's hard for me to see what harm this would cause, so I
> enabled cirrus-ci at github and applied this to master.
> 
> The big weakness with appveyor is that its limits are too low to run the
> whole OVS testsuite.  I wonder whether cirrus-ci has better limits for
> this purpose.

As far as I know, the upper limit for resources is 8CPU + 24GB RAM. Default
timeout for tasks is 60 minutes and it, probably, could be extended using
special config in .yml file.
Ilya Maximets Dec. 12, 2018, 4:03 p.m. UTC | #5 
On 11.12.2018 21:25, Aaron Conole wrote:
> Ilya Maximets <i.maximets@samsung.com> writes:
> 
>> CirrusCI [1] is free for open-sorce projects and provides similar
>> to TravisCI interfaces. One significant difference is ability
>> to run tasks on FreeBSD instances.
>>
>> This patch adds simple configuration file to test OVS build
>> on two FreeBSD releases with gcc and clang.
>> Unit tests are commented out because they are broken for now.
>>
>> To enable the automated checks Cirrus CI application from GitHub
>> Marketplace should be installed. See details in Quick Start guide [2].
>>
>> [1] https://cirrus-ci.org
>> [2] https://cirrus-ci.org/guide/quick-start/
>>
>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>> ---
>>
>> This could be useful to test FreeBSD build automatically to early
>> detect issues. CirrusCI looks suitable for that purpose and it's
>> the only public CI system that I managed to find that supports
>> FreeBSD images. Sending as RFC to start discussion.
> 
> Awesome find.  Thanks, Ilya!
> 
>> Few more differentiation points with Travis. Cirrus allows to
>> configure number of CPUs and the amount of required memory. It
>> also supports Linux, OSX and Windows containers. So, maybe, if
>> we'll like it, we could drop Travis and AppVeyor and use single
>> CI system for all the tests.
>>
>> I'm not advertising and not trying to convince to replace our
>> current CI systems. I found CirrusCI just few days ago by a mail [3]
>> in qemu-devel list. I just think that it's the only option for
>> now to test BSD builds with public CI. And it worked fine for me
>> with below configuration.
>>
>> Thoughts ?
> 
> I'm not married to technology for a reason :)  If it seems better, then
> it is worth checking out.  I can play around with it a little bit, but
> I'll be AFK until Jan, so it might be better for someone else who has
> the recent time to take a look.
> 
> One thing I really find painful with Travis is the difficulty running
> the alternate test suites (make check-*).  Travis can run them, but we
> need to request privileged containers, and that will add to the build
> time.  Not sure if there's some kind of custom privilege support from
> Cirrus that would let us run the alternate test suites - or even let us
> run them by requesting privileges.

Didn't find any information about privileges in CirrusCI docs. Maybe
we need to try and find out the real restrictions.

> 
> Another thing I'm not sure about is the polling.  At least Travis does
> have a RESTful API that we can poll with.  I hope the CirrusCI system
> would as well.  Currently, I am looking into getting the robot to poll
> the Travis system and respond to a series if there is a problem.

Did you thought about using github webhooks for 'Check suite' events?
They will be independent from the exact CI system.

> 
>> As a note: I just spotted that TravisCI adds windows support.
>> Maybe we could use it in the future for windows builds.
>>
>> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
>>
>>  .cirrus.yml | 30 ++++++++++++++++++++++++++++++
>>  Makefile.am |  1 +
>>  2 files changed, 31 insertions(+)
>>  create mode 100644 .cirrus.yml
>>
>> diff --git a/.cirrus.yml b/.cirrus.yml
>> new file mode 100644
>> index 000000000..4902fd370
>> --- /dev/null
>> +++ b/.cirrus.yml
>> @@ -0,0 +1,30 @@
>> +freebsd_build_task:
>> +
>> +  freebsd_instance:
>> +    matrix:
>> +      image: freebsd-12-0-release-amd64
>> +      image: freebsd-11-2-release-amd64
>> +    cpu: 4
>> +    memory: 8G
>> +
>> +  env:
>> +    matrix:
>> +      COMPILER: gcc
>> +      COMPILER: clang
>> +
>> +  prepare_script:
>> +    - pkg install -y automake libtool gmake gcc wget
>> +                     python py27-six py27-sphinx
>> +
>> +  configure_script:
>> +    - ./boot.sh
>> +    - ./configure CC=$COMPILER MAKE=gmake OVS_CFLAGS='-Wall' --enable-Werror
>> +                  || { cat config.log; exit 1; }
>> +
>> +  build_script:
>> +    - gmake -j8
>> +
>> +#  TODO(i.maximets): Uncomment the test suite execution when it is fixed.
>> +#  check_script:
>> +#    - gmake -j8 check TESTSUITEFLAGS=-j8 RECHECK=yes
>> +#                || { cat ./tests/testsuite.log; exit 1; }
>> diff --git a/Makefile.am b/Makefile.am
>> index 64725654c..8408509c9 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -76,6 +76,7 @@ EXTRA_DIST = \
>>  	MAINTAINERS.rst \
>>  	README.rst \
>>  	NOTICE \
>> +	.cirrus.yml \
>>  	.travis.yml \
>>  	.travis/linux-build.sh \
>>  	.travis/linux-prepare.sh \
> 
>
Ben Pfaff Dec. 12, 2018, 4:24 p.m. UTC | #6 
On Wed, Dec 12, 2018 at 07:00:28PM +0300, Ilya Maximets wrote:
> On 12.12.2018 0:06, Ben Pfaff wrote:
> > On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
> >> CirrusCI [1] is free for open-sorce projects and provides similar
> >> to TravisCI interfaces. One significant difference is ability
> >> to run tasks on FreeBSD instances.
> >>
> >> This patch adds simple configuration file to test OVS build
> >> on two FreeBSD releases with gcc and clang.
> >> Unit tests are commented out because they are broken for now.
> >>
> >> To enable the automated checks Cirrus CI application from GitHub
> >> Marketplace should be installed. See details in Quick Start guide [2].
> >>
> >> [1] https://cirrus-ci.org
> >> [2] https://cirrus-ci.org/guide/quick-start/
> >>
> >> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >> ---
> >>
> >> This could be useful to test FreeBSD build automatically to early
> >> detect issues. CirrusCI looks suitable for that purpose and it's
> >> the only public CI system that I managed to find that supports
> >> FreeBSD images. Sending as RFC to start discussion.
> >>
> >> Few more differentiation points with Travis. Cirrus allows to
> >> configure number of CPUs and the amount of required memory. It
> >> also supports Linux, OSX and Windows containers. So, maybe, if
> >> we'll like it, we could drop Travis and AppVeyor and use single
> >> CI system for all the tests.
> >>
> >> I'm not advertising and not trying to convince to replace our
> >> current CI systems. I found CirrusCI just few days ago by a mail [3]
> >> in qemu-devel list. I just think that it's the only option for
> >> now to test BSD builds with public CI. And it worked fine for me
> >> with below configuration.
> >>
> >> Thoughts ?
> >>
> >> As a note: I just spotted that TravisCI adds windows support.
> >> Maybe we could use it in the future for windows builds.
> >>
> >> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
> > 
> > Good idea.  It's hard for me to see what harm this would cause, so I
> > enabled cirrus-ci at github and applied this to master.
> > 
> > The big weakness with appveyor is that its limits are too low to run the
> > whole OVS testsuite.  I wonder whether cirrus-ci has better limits for
> > this purpose.
> 
> As far as I know, the upper limit for resources is 8CPU + 24GB RAM. Default
> timeout for tasks is 60 minutes and it, probably, could be extended using
> special config in .yml file.

I think that the issue with appveyor is the time limit of 60 minutes.
If cirrus-ci can allow for a longer build time, it would probably make a
big difference.
Ilya Maximets Dec. 12, 2018, 4:35 p.m. UTC | #7 
On 12.12.2018 19:24, Ben Pfaff wrote:
> On Wed, Dec 12, 2018 at 07:00:28PM +0300, Ilya Maximets wrote:
>> On 12.12.2018 0:06, Ben Pfaff wrote:
>>> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
>>>> CirrusCI [1] is free for open-sorce projects and provides similar
>>>> to TravisCI interfaces. One significant difference is ability
>>>> to run tasks on FreeBSD instances.
>>>>
>>>> This patch adds simple configuration file to test OVS build
>>>> on two FreeBSD releases with gcc and clang.
>>>> Unit tests are commented out because they are broken for now.
>>>>
>>>> To enable the automated checks Cirrus CI application from GitHub
>>>> Marketplace should be installed. See details in Quick Start guide [2].
>>>>
>>>> [1] https://cirrus-ci.org
>>>> [2] https://cirrus-ci.org/guide/quick-start/
>>>>
>>>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>>>> ---
>>>>
>>>> This could be useful to test FreeBSD build automatically to early
>>>> detect issues. CirrusCI looks suitable for that purpose and it's
>>>> the only public CI system that I managed to find that supports
>>>> FreeBSD images. Sending as RFC to start discussion.
>>>>
>>>> Few more differentiation points with Travis. Cirrus allows to
>>>> configure number of CPUs and the amount of required memory. It
>>>> also supports Linux, OSX and Windows containers. So, maybe, if
>>>> we'll like it, we could drop Travis and AppVeyor and use single
>>>> CI system for all the tests.
>>>>
>>>> I'm not advertising and not trying to convince to replace our
>>>> current CI systems. I found CirrusCI just few days ago by a mail [3]
>>>> in qemu-devel list. I just think that it's the only option for
>>>> now to test BSD builds with public CI. And it worked fine for me
>>>> with below configuration.
>>>>
>>>> Thoughts ?
>>>>
>>>> As a note: I just spotted that TravisCI adds windows support.
>>>> Maybe we could use it in the future for windows builds.
>>>>
>>>> [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00935.html
>>>
>>> Good idea.  It's hard for me to see what harm this would cause, so I
>>> enabled cirrus-ci at github and applied this to master.
>>>
>>> The big weakness with appveyor is that its limits are too low to run the
>>> whole OVS testsuite.  I wonder whether cirrus-ci has better limits for
>>> this purpose.
>>
>> As far as I know, the upper limit for resources is 8CPU + 24GB RAM. Default
>> timeout for tasks is 60 minutes and it, probably, could be extended using
>> special config in .yml file.
> 
> I think that the issue with appveyor is the time limit of 60 minutes.
> If cirrus-ci can allow for a longer build time, it would probably make a
> big difference.

There is a field 'timeout_in' in .cirrus.yml that could be used to extend the
time limit:
    https://cirrus-ci.org/faq/#instance-timed-out

We could try it out.

It'll be better if someone with "OVS on Windows" experience could look at this.
Alin-Gabriel Serdean Dec. 12, 2018, 4:50 p.m. UTC | #8 
> -----Mesaj original-----
> De la: ovs-dev-bounces@openvswitch.org <ovs-dev-
> bounces@openvswitch.org> În numele Ilya Maximets
> Trimis: Wednesday, December 12, 2018 6:36 PM
> Către: Ben Pfaff <blp@ovn.org>
> Cc: ovs-dev@openvswitch.org; Alin Gabriel Serdean <aserdean@ovn.org>
> Subiect: Re: [ovs-dev] [RFC] cirrus: Add Cirrus CI integration for FreeBSD
build
> 
> On 12.12.2018 19:24, Ben Pfaff wrote:
> > On Wed, Dec 12, 2018 at 07:00:28PM +0300, Ilya Maximets wrote:
> >> On 12.12.2018 0:06, Ben Pfaff wrote:
> >>> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
> >>>> CirrusCI [1] is free for open-sorce projects and provides similar
> >>>> to TravisCI interfaces. One significant difference is ability to
> >>>> run tasks on FreeBSD instances.
> >>>>
> >>>> This patch adds simple configuration file to test OVS build on two
> >>>> FreeBSD releases with gcc and clang.
> >>>> Unit tests are commented out because they are broken for now.
> >>>>
> >>>> To enable the automated checks Cirrus CI application from GitHub
> >>>> Marketplace should be installed. See details in Quick Start guide
[2].
> >>>>
> >>>> [1] https://cirrus-ci.org
> >>>> [2] https://cirrus-ci.org/guide/quick-start/
> >>>>
> >>>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >>>> ---
> >>>>
> >>>> This could be useful to test FreeBSD build automatically to early
> >>>> detect issues. CirrusCI looks suitable for that purpose and it's
> >>>> the only public CI system that I managed to find that supports
> >>>> FreeBSD images. Sending as RFC to start discussion.
> >>>>
> >>>> Few more differentiation points with Travis. Cirrus allows to
> >>>> configure number of CPUs and the amount of required memory. It also
> >>>> supports Linux, OSX and Windows containers. So, maybe, if we'll
> >>>> like it, we could drop Travis and AppVeyor and use single CI system
> >>>> for all the tests.
> >>>>
> >>>> I'm not advertising and not trying to convince to replace our
> >>>> current CI systems. I found CirrusCI just few days ago by a mail
> >>>> [3] in qemu-devel list. I just think that it's the only option for
> >>>> now to test BSD builds with public CI. And it worked fine for me
> >>>> with below configuration.
> >>>>
> >>>> Thoughts ?
> >>>>
> >>>> As a note: I just spotted that TravisCI adds windows support.
> >>>> Maybe we could use it in the future for windows builds.
> >>>>
> >>>> [3]
> >>>> https://lists.nongnu.org/archive/html/qemu-devel/2018-
> 12/msg00935.h
> >>>> tml
> >>>
> >>> Good idea.  It's hard for me to see what harm this would cause, so I
> >>> enabled cirrus-ci at github and applied this to master.
> >>>
> >>> The big weakness with appveyor is that its limits are too low to run
> >>> the whole OVS testsuite.  I wonder whether cirrus-ci has better
> >>> limits for this purpose.
> >>
> >> As far as I know, the upper limit for resources is 8CPU + 24GB RAM.
> >> Default timeout for tasks is 60 minutes and it, probably, could be
> >> extended using special config in .yml file.
> >
> > I think that the issue with appveyor is the time limit of 60 minutes.
> > If cirrus-ci can allow for a longer build time, it would probably make
> > a big difference.
> 
> There is a field 'timeout_in' in .cirrus.yml that could be used to extend
the
> time limit:
>     https://cirrus-ci.org/faq/#instance-timed-out
> 
> We could try it out.
> 
> It'll be better if someone with "OVS on Windows" experience could look at
> this.

Sorry for the delay.

I'll play a bit with it tonight and send out updates.
Ilya Maximets Dec. 12, 2018, 4:58 p.m. UTC | #9 
On 12.12.2018 19:50, aserdean@ovn.org wrote:
>> -----Mesaj original-----
>> De la: ovs-dev-bounces@openvswitch.org <ovs-dev-
>> bounces@openvswitch.org> În numele Ilya Maximets
>> Trimis: Wednesday, December 12, 2018 6:36 PM
>> Către: Ben Pfaff <blp@ovn.org>
>> Cc: ovs-dev@openvswitch.org; Alin Gabriel Serdean <aserdean@ovn.org>
>> Subiect: Re: [ovs-dev] [RFC] cirrus: Add Cirrus CI integration for FreeBSD
> build
>>
>> On 12.12.2018 19:24, Ben Pfaff wrote:
>>> On Wed, Dec 12, 2018 at 07:00:28PM +0300, Ilya Maximets wrote:
>>>> On 12.12.2018 0:06, Ben Pfaff wrote:
>>>>> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
>>>>>> CirrusCI [1] is free for open-sorce projects and provides similar
>>>>>> to TravisCI interfaces. One significant difference is ability to
>>>>>> run tasks on FreeBSD instances.
>>>>>>
>>>>>> This patch adds simple configuration file to test OVS build on two
>>>>>> FreeBSD releases with gcc and clang.
>>>>>> Unit tests are commented out because they are broken for now.
>>>>>>
>>>>>> To enable the automated checks Cirrus CI application from GitHub
>>>>>> Marketplace should be installed. See details in Quick Start guide
> [2].
>>>>>>
>>>>>> [1] https://cirrus-ci.org
>>>>>> [2] https://cirrus-ci.org/guide/quick-start/
>>>>>>
>>>>>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
>>>>>> ---
>>>>>>
>>>>>> This could be useful to test FreeBSD build automatically to early
>>>>>> detect issues. CirrusCI looks suitable for that purpose and it's
>>>>>> the only public CI system that I managed to find that supports
>>>>>> FreeBSD images. Sending as RFC to start discussion.
>>>>>>
>>>>>> Few more differentiation points with Travis. Cirrus allows to
>>>>>> configure number of CPUs and the amount of required memory. It also
>>>>>> supports Linux, OSX and Windows containers. So, maybe, if we'll
>>>>>> like it, we could drop Travis and AppVeyor and use single CI system
>>>>>> for all the tests.
>>>>>>
>>>>>> I'm not advertising and not trying to convince to replace our
>>>>>> current CI systems. I found CirrusCI just few days ago by a mail
>>>>>> [3] in qemu-devel list. I just think that it's the only option for
>>>>>> now to test BSD builds with public CI. And it worked fine for me
>>>>>> with below configuration.
>>>>>>
>>>>>> Thoughts ?
>>>>>>
>>>>>> As a note: I just spotted that TravisCI adds windows support.
>>>>>> Maybe we could use it in the future for windows builds.
>>>>>>
>>>>>> [3]
>>>>>> https://lists.nongnu.org/archive/html/qemu-devel/2018-
>> 12/msg00935.h
>>>>>> tml
>>>>>
>>>>> Good idea.  It's hard for me to see what harm this would cause, so I
>>>>> enabled cirrus-ci at github and applied this to master.
>>>>>
>>>>> The big weakness with appveyor is that its limits are too low to run
>>>>> the whole OVS testsuite.  I wonder whether cirrus-ci has better
>>>>> limits for this purpose.
>>>>
>>>> As far as I know, the upper limit for resources is 8CPU + 24GB RAM.
>>>> Default timeout for tasks is 60 minutes and it, probably, could be
>>>> extended using special config in .yml file.
>>>
>>> I think that the issue with appveyor is the time limit of 60 minutes.
>>> If cirrus-ci can allow for a longer build time, it would probably make
>>> a big difference.
>>
>> There is a field 'timeout_in' in .cirrus.yml that could be used to extend
> the
>> time limit:
>>     https://cirrus-ci.org/faq/#instance-timed-out
>>
>> We could try it out.
>>
>> It'll be better if someone with "OVS on Windows" experience could look at
>> this.
> 
> Sorry for the delay.
> 
> I'll play a bit with it tonight and send out updates.

Cool. Thanks.

I just wanted to mention that even without extending the timeout we can just
split the test suite into separate tasks like this:

  env:
    matrix:
      TESTSUITEFLAGS: -1000
      TESTSUITEFLAGS: 1001-2000
      TESTSUITEFLAGS: 2001-

  check_script:
    - make -j8 check TESTSUITEFLAGS="-j8 $TESTSUITEFLAGS"

And each of three tasks will have its own 60 minutes limit.
This could help if the testsuite is the main time consuming issue.
Alin-Gabriel Serdean Dec. 12, 2018, 5:41 p.m. UTC | #10 
> -----Mesaj original-----
> De la: ovs-dev-bounces@openvswitch.org <ovs-dev-
> bounces@openvswitch.org> În numele Ilya Maximets
> Trimis: Wednesday, December 12, 2018 6:59 PM
> Către: aserdean@ovn.org; 'Ben Pfaff' <blp@ovn.org>
> Cc: ovs-dev@openvswitch.org
> Subiect: Re: [ovs-dev] [RFC] cirrus: Add Cirrus CI integration for FreeBSD
build
> 
> On 12.12.2018 19:50, aserdean@ovn.org wrote:
> >> -----Mesaj original-----
> >> De la: ovs-dev-bounces@openvswitch.org <ovs-dev-
> >> bounces@openvswitch.org> În numele Ilya Maximets
> >> Trimis: Wednesday, December 12, 2018 6:36 PM
> >> Către: Ben Pfaff <blp@ovn.org>
> >> Cc: ovs-dev@openvswitch.org; Alin Gabriel Serdean
> <aserdean@ovn.org>
> >> Subiect: Re: [ovs-dev] [RFC] cirrus: Add Cirrus CI integration for
> >> FreeBSD
> > build
> >>
> >> On 12.12.2018 19:24, Ben Pfaff wrote:
> >>> On Wed, Dec 12, 2018 at 07:00:28PM +0300, Ilya Maximets wrote:
> >>>> On 12.12.2018 0:06, Ben Pfaff wrote:
> >>>>> On Tue, Dec 11, 2018 at 05:34:17PM +0300, Ilya Maximets wrote:
> >>>>>> CirrusCI [1] is free for open-sorce projects and provides similar
> >>>>>> to TravisCI interfaces. One significant difference is ability to
> >>>>>> run tasks on FreeBSD instances.
> >>>>>>
> >>>>>> This patch adds simple configuration file to test OVS build on
> >>>>>> two FreeBSD releases with gcc and clang.
> >>>>>> Unit tests are commented out because they are broken for now.
> >>>>>>
> >>>>>> To enable the automated checks Cirrus CI application from GitHub
> >>>>>> Marketplace should be installed. See details in Quick Start guide
> > [2].
> >>>>>>
> >>>>>> [1] https://cirrus-ci.org
> >>>>>> [2] https://cirrus-ci.org/guide/quick-start/
> >>>>>>
> >>>>>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >>>>>> ---
> >>>>>>
> >>>>>> This could be useful to test FreeBSD build automatically to early
> >>>>>> detect issues. CirrusCI looks suitable for that purpose and it's
> >>>>>> the only public CI system that I managed to find that supports
> >>>>>> FreeBSD images. Sending as RFC to start discussion.
> >>>>>>
> >>>>>> Few more differentiation points with Travis. Cirrus allows to
> >>>>>> configure number of CPUs and the amount of required memory. It
> >>>>>> also supports Linux, OSX and Windows containers. So, maybe, if
> >>>>>> we'll like it, we could drop Travis and AppVeyor and use single
> >>>>>> CI system for all the tests.
> >>>>>>
> >>>>>> I'm not advertising and not trying to convince to replace our
> >>>>>> current CI systems. I found CirrusCI just few days ago by a mail
> >>>>>> [3] in qemu-devel list. I just think that it's the only option
> >>>>>> for now to test BSD builds with public CI. And it worked fine for
> >>>>>> me with below configuration.
> >>>>>>
> >>>>>> Thoughts ?
> >>>>>>
> >>>>>> As a note: I just spotted that TravisCI adds windows support.
> >>>>>> Maybe we could use it in the future for windows builds.
> >>>>>>
> >>>>>> [3]
> >>>>>> https://lists.nongnu.org/archive/html/qemu-devel/2018-
> >> 12/msg00935.h
> >>>>>> tml
> >>>>>
> >>>>> Good idea.  It's hard for me to see what harm this would cause, so
> >>>>> I enabled cirrus-ci at github and applied this to master.
> >>>>>
> >>>>> The big weakness with appveyor is that its limits are too low to
> >>>>> run the whole OVS testsuite.  I wonder whether cirrus-ci has
> >>>>> better limits for this purpose.
> >>>>
> >>>> As far as I know, the upper limit for resources is 8CPU + 24GB RAM.
> >>>> Default timeout for tasks is 60 minutes and it, probably, could be
> >>>> extended using special config in .yml file.
> >>>
> >>> I think that the issue with appveyor is the time limit of 60 minutes.
> >>> If cirrus-ci can allow for a longer build time, it would probably
> >>> make a big difference.
> >>
> >> There is a field 'timeout_in' in .cirrus.yml that could be used to
> >> extend
> > the
> >> time limit:
> >>     https://cirrus-ci.org/faq/#instance-timed-out
> >>
> >> We could try it out.
> >>
> >> It'll be better if someone with "OVS on Windows" experience could
> >> look at this.
> >
> > Sorry for the delay.
> >
> > I'll play a bit with it tonight and send out updates.
> 
> Cool. Thanks.
> 
> I just wanted to mention that even without extending the timeout we can
> just split the test suite into separate tasks like this:
> 
>   env:
>     matrix:
>       TESTSUITEFLAGS: -1000
>       TESTSUITEFLAGS: 1001-2000
>       TESTSUITEFLAGS: 2001-
> 
>   check_script:
>     - make -j8 check TESTSUITEFLAGS="-j8 $TESTSUITEFLAGS"
> 
> And each of three tasks will have its own 60 minutes limit.
> This could help if the testsuite is the main time consuming issue.

Thanks a lot for the pointers!
diff mbox series

Patch

diff --git a/.cirrus.yml b/.cirrus.yml
new file mode 100644
index 000000000..4902fd370
--- /dev/null
+++ b/.cirrus.yml
@@ -0,0 +1,30 @@ 
+freebsd_build_task:
+
+  freebsd_instance:
+    matrix:
+      image: freebsd-12-0-release-amd64
+      image: freebsd-11-2-release-amd64
+    cpu: 4
+    memory: 8G
+
+  env:
+    matrix:
+      COMPILER: gcc
+      COMPILER: clang
+
+  prepare_script:
+    - pkg install -y automake libtool gmake gcc wget
+                     python py27-six py27-sphinx
+
+  configure_script:
+    - ./boot.sh
+    - ./configure CC=$COMPILER MAKE=gmake OVS_CFLAGS='-Wall' --enable-Werror
+                  || { cat config.log; exit 1; }
+
+  build_script:
+    - gmake -j8
+
+#  TODO(i.maximets): Uncomment the test suite execution when it is fixed.
+#  check_script:
+#    - gmake -j8 check TESTSUITEFLAGS=-j8 RECHECK=yes
+#                || { cat ./tests/testsuite.log; exit 1; }
diff --git a/Makefile.am b/Makefile.am
index 64725654c..8408509c9 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -76,6 +76,7 @@  EXTRA_DIST = \
 	MAINTAINERS.rst \
 	README.rst \
 	NOTICE \
+	.cirrus.yml \
 	.travis.yml \
 	.travis/linux-build.sh \
 	.travis/linux-prepare.sh \