From patchwork Mon Feb 5 02:48:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ansis Atteka X-Patchwork-Id: 869135 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zZX983LqLz9s72 for ; Mon, 5 Feb 2018 13:48:58 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 252081053; Mon, 5 Feb 2018 02:48:55 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id A8C19103C for ; Mon, 5 Feb 2018 02:48:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 52A241C0 for ; Mon, 5 Feb 2018 02:48:52 +0000 (UTC) X-Originating-IP: 208.91.1.34 Received: from aatteka-Precision-Tower-5810.eng.vmware.com (unknown [208.91.1.34]) (Authenticated sender: aatteka@ovn.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id B147BA80C7; Mon, 5 Feb 2018 03:48:49 +0100 (CET) From: Ansis Atteka To: dev@openvswitch.org Date: Sun, 4 Feb 2018 18:48:53 -0800 Message-Id: <20180205024853.4116-1-aatteka@ovn.org> X-Mailer: git-send-email 2.14.1 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Ansis Atteka Subject: [ovs-dev] [PATCHv2] poc: Introduce Proof of Concepts (Package building) X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Ansis Atteka This patch sets up foundations for Proof of Concepts that simply materialize documentation into Ansible instructions executed in virtualized Vagrant environment. This Proof of Concept allows to easily build: 1. *.deb packages on Ubuntu 16.04; AND 2. *.rpm packages on CentOS 7.4. It also sets up DEB and RPM repository over HTTP that can be used to pull these openvswitch packages with apt-get or yum from another host. This particular Proof of Concept is intended to address following use-cases: 1. for new OVS users to see how debian and rpm packages are built; 2. for developers to easily check for packaging build regressions; 3. for developers to easily share their sandbox builds into QE setups (opposed to manually copying binaries); 4. for developers to add other Proof of Concepts that possibly may require full end-to-end integration with other thirdparty projects (e.g. DPI, libvirt, IPsec) and need Open vSwitch packages. Tested-by: Greg Rose Reviewed-by: Greg Rose Signed-off-by: Ansis Atteka Reviewed-by: Greg Rose --- .gitignore | 2 + Documentation/topics/testing.rst | 56 ++++++++++++++++++++ Makefile.am | 3 ++ poc/builders/Vagrantfile | 33 ++++++++++++ poc/playbook-centos-builder.yml | 108 +++++++++++++++++++++++++++++++++++++++ poc/playbook-ubuntu-builder.yml | 70 +++++++++++++++++++++++++ 6 files changed, 272 insertions(+) create mode 100644 poc/builders/Vagrantfile create mode 100644 poc/playbook-centos-builder.yml create mode 100644 poc/playbook-ubuntu-builder.yml diff --git a/.gitignore b/.gitignore index 8019bee41..81faf270d 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ *.lib *.pdb *.pyc +*.retry *.so *.suo **/*.sym @@ -29,6 +30,7 @@ .dirstamp .libs .tmp_versions +.vagrant .gitattributes /Makefile /Makefile.in diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index a49336b79..253857171 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -389,3 +389,59 @@ validate the suitability of different vSwitch implementations in a telco deployment environment. More information can be found on the `OPNFV wiki`_. .. _OPNFV wiki: https://wiki.opnfv.org/display/vsperf/VSperf+Home + +Proof of Concepts +~~~~~~~~~~~~~~~~~ + +Proof of Concepts are documentation materialized into Ansible recipes +executed in VirtualBox or Libvirt environments orchastrated by Vagrant. +Proof of Concepts allow developers to create small virtualized setups that +demonstrate how certain Open vSwitch features are intended to work avoiding +user introduced errors by overlooking instructions. Proof of Concepts +are also helpful when integrating with thirdparty software, because standard +unit tests with make check are limited. + +Vagrant by default uses VirtualBox provider. However, if Libvirt is your +choice of virtualization technology, then you can use it by installing Libvirt +plugin: + + $ vagrant plugin install vagrant-libvirt + +And then appending --provider=libvirt flag to vagrant commands. + +The host where Vagrant runs does not need to have any special software +installed besides vagrant, virtualbox (or libvirt and libvirt-dev) and +ansible. + +The following Proof of Concepts are supported: + +Builders +++++++++ + +This particular Proof of Concept demonsrtates integration with Debian and RPM +packaging tools: + + $ cd ./poc/builders + + $ vagrant up + +Once that command finished you can get packages from /var/www/html +directory. Since those hosts are also configured as repositories then +you can add them to /etc/apt/sources.list.d or /etc/yum.repos.d +configuration files on another host to retrieve packages with yum or +apt-get. + +hen you have made changes to OVS source code and want to rebuild packages run: + $ git commit -a + $ vagrant rsync && vagrant provision + +Whenever packages are rebuilt the Open vSwitch release number increases +by one and you can simply upgrade Open vSwitch by running yum or apt-get +update commands. + +Once you are done with experimenting you can tear down setup with: + + $ vagrant destroy + +Sometimes deployment of Proof of Concept may fail, if, for example, VMs +don't have network reachability to the Internet. diff --git a/Makefile.am b/Makefile.am index ed4b7fd7c..174a421e6 100644 --- a/Makefile.am +++ b/Makefile.am @@ -90,6 +90,9 @@ EXTRA_DIST = \ build-aux/sodepends.py \ build-aux/soexpand.py \ build-aux/xml2nroff \ + poc/builders/Vagrantfile \ + poc/playbook-centos-builder.yml \ + poc/playbook-ubuntu-builder.yml \ $(MAN_FRAGMENTS) \ $(MAN_ROOTS) \ Vagrantfile \ diff --git a/poc/builders/Vagrantfile b/poc/builders/Vagrantfile new file mode 100644 index 000000000..af2986789 --- /dev/null +++ b/poc/builders/Vagrantfile @@ -0,0 +1,33 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +VAGRANTFILE_API_VERSION = "2" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + config.ssh.insert_key = false + + # Centos-7.4 builder host + config.vm.define "centosbuilder" do |builder| + builder.vm.hostname = "centosbuilder.dev" + builder.vm.box = "centos/7" + builder.vm.synced_folder "../../", "/git/ovs", type: "rsync", + rsync__args: ["--archive", "--delete", "-z"] + builder.vm.provision "builder", type: "ansible" do |ansible| + ansible.playbook = "../playbook-centos-builder.yml" + ansible.sudo = true + end + end + + # Ubuntu-16.04 builder host + config.vm.define "ubuntubuilder" do |builder| + builder.vm.hostname = "ubuntubuilder.dev" + builder.vm.box = "generic/ubuntu1604" + builder.vm.synced_folder "../../", "/git/ovs", type: "rsync", + rsync__args: ["--archive", "--delete", "-z"] + builder.vm.provision "builder", type: "ansible" do |ansible| + ansible.playbook = "../playbook-ubuntu-builder.yml" + ansible.sudo = true + end + end + +end diff --git a/poc/playbook-centos-builder.yml b/poc/playbook-centos-builder.yml new file mode 100644 index 000000000..71f104010 --- /dev/null +++ b/poc/playbook-centos-builder.yml @@ -0,0 +1,108 @@ +--- +- hosts: all + become: true + name: builder + tasks: + + - name: Create Ansible Local Facts Directory + file: path=/etc/ansible/facts.d state=directory + + - name: Initiate Build Numbering + copy: + content: '{ "release":"1" }' + dest: "/etc/ansible/facts.d/builder.fact" + force: no + + - name: Set source directory for building + set_fact: + SOURCE: "/root/rpmbuild/SOURCES" + + - name: Reload Ansible Local Facts + setup: filter=ansible_local + + - name: Install "yum-utils", "rpmdevtools", "createrepo", "httpd", "git" + yum: update_cache=yes name={{item}} state=present + with_items: + - yum-utils + - rpmdevtools + - createrepo + - httpd + - git + + - name: Remove untracked files from Open vSwitch GIT repository + command: chdir=/git/ovs/ git clean -xdf + + - name: Reset Open vSwitch GIT repository to last comitted state + command: chdir=/git/ovs/ git reset --hard + + - name: Generate spec files for easy build dependency retrieval + shell: sed -e 's/@VERSION@/0.0.1/' {{item}}.in > /tmp/{{item}} + args: + chdir: /git/ovs/rhel + with_items: + - openvswitch.spec + - openvswitch-kmod-rhel6.spec + + - name: Install build dependencies specified from spec files + shell: echo "y" | yum-builddep /tmp/{{item}} + with_items: + - openvswitch.spec + - openvswitch-kmod-rhel6.spec + + - name: Create rpm dev tree + command: rpmdev-setuptree + + - name: Run "./boot.sh" + command: chdir=/git/ovs/ ./boot.sh + + - name: Run "./configure" + command: chdir=/git/ovs/ ./configure + + - name: Run "make dist" + command: chdir=/git/ovs/ make dist + + - name: Parse out Open vSwitch version from "configure.ac" + command: chdir=/git/ovs autoconf -t AC_INIT:'$2' + register: version + + - name: Copy source tarball to rpm dev tree + command: cp /git/ovs/openvswitch-{{version.stdout}}.tar.gz {{SOURCE}} + + - name: Unarchive openvswitch source tarball + unarchive: + src: "{{SOURCE}}/openvswitch-{{version.stdout}}.tar.gz" + dest: "{{SOURCE}}" + remote_src: yes + + - name: Update release number in spec files + lineinfile: + path: "{{SOURCE}}/openvswitch-{{version.stdout}}/rhel/{{item}}" + regexp: '^Release:' + line: "Release: {{ ansible_local.builder.release }}" + with_items: + - openvswitch.spec + - openvswitch-kmod-rhel6.spec + + - name: Build Open vSwitch user space rpms + command: rpmbuild -bb --without check rhel/openvswitch.spec + args: + chdir: "{{SOURCE}}/openvswitch-{{version.stdout}}" + + - name: Build Open vSwitch kmod rpms (only for currently loaded kernel) + command: rpmbuild -bb --without check rhel/openvswitch-kmod-rhel6.spec + args: + chdir: "{{SOURCE}}/openvswitch-{{version.stdout}}" + + - name: Copy RPM packages to /var/www/html + command: cp -r /root/rpmbuild/RPMS/ /var/www/html + + - name: Create RPM Package index file for repository + command: chdir=/var/www/html createrepo /var/www/html + + - name: Make sure Apache is running + systemd: state=started name=httpd + + - name: Bump up Build Number + copy: + content: '{ "release":"{{ansible_local.builder.release|int+1}}" }' + dest: "/etc/ansible/facts.d/builder.fact" diff --git a/poc/playbook-ubuntu-builder.yml b/poc/playbook-ubuntu-builder.yml new file mode 100644 index 000000000..5a13c6069 --- /dev/null +++ b/poc/playbook-ubuntu-builder.yml @@ -0,0 +1,70 @@ +--- +- hosts: all + become: true + name: builder + gather_facts: no + pre_tasks: + - name: 'install python2' + raw: sudo apt-get -y install python-simplejson + tasks: + + - name: Create Ansible Local Facts Directory + file: path=/etc/ansible/facts.d state=directory + + - name: Initiate Build Numbering + copy: + content: '{ "release":"1" }' + dest: "/etc/ansible/facts.d/builder.fact" + force: no + + - name: Reload Ansible Local Facts + setup: filter=ansible_local + + - name: Install "devscripts", "equivs", "apache2", "autoconf" + apt: update_cache=yes name={{item}} state=present + with_items: + - devscripts + - equivs + - apache2 + - autoconf + + - name: Remove untracked files from Open vSwitch GIT repository + command: chdir=/git/ovs/ git clean -xdf + + - name: Reset Open vSwitch GIT repository to last comitted state + command: chdir=/git/ovs/ git reset --hard + + - name: Parse out Open vSwitch version from "configure.ac" + command: chdir=/git/ovs autoconf -t AC_INIT:'$2' + register: version + + - name: Concatenate full version + set_fact: + full_version: "{{version.stdout}}-{{ansible_local.builder.release}}" + + - name: Update Open vSwitch version to {{full_version}} + command: chdir=/git/ovs/ dch -b -v {{full_version}} Vagrant Build + + - name: Build debian package with Open vSwitch build dependencies + command: chdir=/git/ovs/ mk-build-deps -B debian/control + + - name: Install Open vSwitch {{full_version}} build dependencies + apt: deb=/git/ovs/openvswitch-build-deps-depends_{{full_version}}_all.deb + + - name: Build Open vSwitch {{full_version}} debian packages + shell: DEB_BUILD_OPTIONS='nocheck' fakeroot debian/rules binary + args: + chdir: /git/ovs/ + + - name: Move debian packages to /var/www/html + shell: mv /git/*.deb /var/www/html/ + + - name: Create Debian Package index file for repository + shell: dpkg-scanpackages . | gzip -9c > Packages.gz + args: + chdir: /var/www/html + + - name: Bump up Build Number + copy: + content: '{ "release":"{{ansible_local.builder.release|int+1}}" }' + dest: "/etc/ansible/facts.d/builder.fact"