[ovs-dev,v3] utilities: Add simple container automation

Add simple script that allows user to run tests and builds
using container. At the same time add example Dockerfile for
Fedora 37.

Basic usage is:
./utilities/containers/ci.sh

This will compile the project with GCC, it expects podman
as container platform with default image being "ovn-org/ovn-tests".

The test parameters can be specified as environment variables.

Signed-off-by: Ales Musil <amusil@redhat.com>
---
v3: Rebase on top of main.
    Address comments from Mark:
    - Fix "ARH" typo.
    - Add python dependencies to a file.
---
 .ci/linux-prepare.sh                     |   2 +-
 utilities/automake.mk                    |   4 +
 utilities/containers/Makefile            |   7 ++
 utilities/containers/ci.sh               | 137 +++++++++++++++++++++++
 utilities/containers/fedora/Dockerfile   |  62 ++++++++++
 utilities/containers/py-requirements.txt |   6 +
 6 files changed, 217 insertions(+), 1 deletion(-)
 create mode 100644 utilities/containers/Makefile
 create mode 100755 utilities/containers/ci.sh
 create mode 100755 utilities/containers/fedora/Dockerfile
 create mode 100644 utilities/containers/py-requirements.txt

Bleep bloop.  Greetings Ales Musil, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


checkpatch:
WARNING: Line is 89 characters long (recommended limit is 79)
#69 FILE: utilities/containers/Makefile:7:
build: ;$(CONTAINER_CMD) build --no-cache --rm -t $(IMAGE_NAME) -f $(DISTRO)/Dockerfile .

Lines checked: 296, Warnings: 1, Errors: 0


Please check this out.  If you feel there has been an error, please email aconole@redhat.com

Thanks,
0-day Robot

Thanks Ales,

Acked-by: Mark Michelson <mmichels@redhat.com>

On 2/14/23 03:33, Ales Musil wrote:
> Add simple script that allows user to run tests and builds
> using container. At the same time add example Dockerfile for
> Fedora 37.
> 
> Basic usage is:
> ./utilities/containers/ci.sh
> 
> This will compile the project with GCC, it expects podman
> as container platform with default image being "ovn-org/ovn-tests".
> 
> The test parameters can be specified as environment variables.
> 
> Signed-off-by: Ales Musil <amusil@redhat.com>
> ---
> v3: Rebase on top of main.
>      Address comments from Mark:
>      - Fix "ARH" typo.
>      - Add python dependencies to a file.
> ---
>   .ci/linux-prepare.sh                     |   2 +-
>   utilities/automake.mk                    |   4 +
>   utilities/containers/Makefile            |   7 ++
>   utilities/containers/ci.sh               | 137 +++++++++++++++++++++++
>   utilities/containers/fedora/Dockerfile   |  62 ++++++++++
>   utilities/containers/py-requirements.txt |   6 +
>   6 files changed, 217 insertions(+), 1 deletion(-)
>   create mode 100644 utilities/containers/Makefile
>   create mode 100755 utilities/containers/ci.sh
>   create mode 100755 utilities/containers/fedora/Dockerfile
>   create mode 100644 utilities/containers/py-requirements.txt
> 
> diff --git a/.ci/linux-prepare.sh b/.ci/linux-prepare.sh
> index e0c528479..6617d0c42 100755
> --- a/.ci/linux-prepare.sh
> +++ b/.ci/linux-prepare.sh
> @@ -18,4 +18,4 @@ cd sparse && make -j4 HAVE_LLVM= HAVE_SQLITE= install && cd ..
>   #     https://github.com/pypa/pip/issues/10655
>   pip3 install --disable-pip-version-check --user wheel
>   pip3 install --disable-pip-version-check --user \
> -    flake8 'hacking>=3.0' sphinx setuptools pyelftools pyOpenSSL
> +-r utilities/containers/py-requirements.txt
> diff --git a/utilities/automake.mk b/utilities/automake.mk
> index 67b04cbff..142fe6810 100644
> --- a/utilities/automake.mk
> +++ b/utilities/automake.mk
> @@ -37,6 +37,10 @@ EXTRA_DIST += \
>       utilities/ovn_detrace.py.in \
>       utilities/ovndb-servers.ocf \
>       utilities/checkpatch.py \
> +    utilities/containers/ci.sh \
> +    utilities/containers/Makefile \
> +    utilities/containers/py-requirements.txt \
> +    utilities/containers/fedora/Dockerfile \
>       utilities/docker/Makefile \
>       utilities/docker/start-ovn \
>       utilities/docker/ovn_default_nb_port \
> diff --git a/utilities/containers/Makefile b/utilities/containers/Makefile
> new file mode 100644
> index 000000000..8b39e3493
> --- /dev/null
> +++ b/utilities/containers/Makefile
> @@ -0,0 +1,7 @@
> +CONTAINER_CMD ?= podman
> +DISTRO ?= fedora
> +IMAGE_NAME ?= "ovn-org/ovn-tests"
> +
> +.PHONY: build
> +
> +build: ;$(CONTAINER_CMD) build --no-cache --rm -t $(IMAGE_NAME) -f $(DISTRO)/Dockerfile .
> diff --git a/utilities/containers/ci.sh b/utilities/containers/ci.sh
> new file mode 100755
> index 000000000..e00971e0c
> --- /dev/null
> +++ b/utilities/containers/ci.sh
> @@ -0,0 +1,137 @@
> +#!/bin/bash -xe
> +# Copyright (c) 2022, Red Hat, Inc.
> +#
> +# Licensed under the Apache License, Version 2.0 (the "License");
> +# you may not use this file except in compliance with the License.
> +# You may obtain a copy of the License at:
> +#
> +#     http://www.apache.org/licenses/LICENSE-2.0
> +#
> +# Unless required by applicable law or agreed to in writing, software
> +# distributed under the License is distributed on an "AS IS" BASIS,
> +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +# See the License for the specific language governing permissions and
> +# limitations under the License.
> +#
> +
> +OVN_PATH=${OVN_PATH:-$PWD}
> +OVS_PATH=${OVS_PATH:-$OVN_PATH/ovs}
> +CONTAINER_CMD=${CONTAINER_CMD:-podman}
> +CONTAINER_WORKSPACE="/workspace"
> +CONTAINER_WORKDIR="/workspace/ovn-tmp"
> +IMAGE_NAME=${IMAGE_NAME:-"ovn-org/ovn-tests"}
> +
> +# Test variables
> +ARCH=${ARCH:-$(uname -i)}
> +CC=${CC:-gcc}
> +
> +
> +test -t 1 && USE_TTY="t"
> +
> +function container_exec() {
> +    ${CONTAINER_CMD} exec "-i$USE_TTY" "$CONTAINER_ID" /bin/bash -c "$1"
> +}
> +
> +function container_shell() {
> +    ${CONTAINER_CMD} exec "-i$USE_TTY" "$CONTAINER_ID" /bin/bash
> +}
> +
> +function remove_container() {
> +    res=$?
> +    [ "$res" -ne 0 ] && echo "*** ERROR: $res ***"
> +    ${CONTAINER_CMD} rm -f "$CONTAINER_ID"
> +}
> +
> +function copy_sources_to_workdir() {
> +    container_exec "
> +        mkdir -p $CONTAINER_WORKDIR \
> +        && \
> +        cp -a $CONTAINER_WORKSPACE/ovn/. $CONTAINER_WORKDIR \
> +        && \
> +        rm -rf $CONTAINER_WORKDIR/ovs \
> +        &&
> +        cp -a $CONTAINER_WORKSPACE/ovs/. $CONTAINER_WORKDIR/ovs
> +    "
> +}
> +
> +function overwrite_jobs() {
> +    container_exec "
> +        sed -i s/-j[0-9]/-j$jobs/ $CONTAINER_WORKDIR/.ci/linux-build.sh
> +    "
> +}
> +
> +function run_tests() {
> +    container_exec "
> +        cd $CONTAINER_WORKDIR \
> +        && \
> +        ARCH=$ARCH CC=$CC LIBS=$LIBS OPTS=$OPTS TESTSUITE=$TESTSUITE \
> +        TEST_RANGE=$TEST_RANGE SANITIZERS=$SANITIZERS \
> +        ./.ci/linux-build.sh
> +    "
> +}
> +
> +options=$(getopt --options "" \
> +    --long help,shell,jobs:,ovn-path:,ovs-path:,image-name:\
> +    -- "${@}")
> +eval set -- "$options"
> +while true; do
> +    case "$1" in
> +    --shell)
> +        shell="1"
> +        ;;
> +    --jobs)
> +        shift
> +        jobs="$1"
> +        ;;
> +    --ovn-path)
> +        shift
> +        OVN_PATH="$1"
> +        ;;
> +    --ovs-path)
> +        shift
> +        OVS_PATH="$1"
> +        ;;
> +    --image-name)
> +        shift
> +        IMAGE_NAME="$1"
> +        ;;
> +    --help)
> +        set +x
> +        printf "$0 [--shell] [--help] [--jobs=<JOBS>] [--ovn-path=<OVN_PATH>]"
> +        printf "[--ovs-path=<OVS_PATH>] [--image-name=<IMAGE_NAME>]\n"
> +        exit
> +        ;;
> +    --)
> +        shift
> +        break
> +        ;;
> +    esac
> +    shift
> +done
> +
> +# Workaround for https://bugzilla.redhat.com/2153359
> +if [ "$ARCH" = "aarch64" ]; then
> +    ASAN_OPTIONS="detect_leaks=0"
> +fi
> +
> +CONTAINER_ID="$($CONTAINER_CMD run --privileged -d \
> +    --pids-limit=-1 \
> +    --env ASAN_OPTIONS=$ASAN_OPTIONS \
> +    -v /lib/modules/$(uname -r):/lib/modules/$(uname -r):ro \
> +    -v $OVN_PATH:$CONTAINER_WORKSPACE/ovn:Z \
> +    -v $OVS_PATH:$CONTAINER_WORKSPACE/ovs:Z \
> +    $IMAGE_NAME)"
> +trap remove_container EXIT
> +
> +copy_sources_to_workdir
> +
> +if [ -n "$jobs" ]; then
> +    overwrite_jobs
> +fi
> +
> +if [ -n "$shell" ];then
> +    container_shell
> +    exit 0
> +fi
> +
> +run_tests
> diff --git a/utilities/containers/fedora/Dockerfile b/utilities/containers/fedora/Dockerfile
> new file mode 100755
> index 000000000..be8cd6ff2
> --- /dev/null
> +++ b/utilities/containers/fedora/Dockerfile
> @@ -0,0 +1,62 @@
> +FROM quay.io/fedora/fedora:37
> +
> +# Update distro, install packages and clean any possible leftovers
> +RUN dnf -y update \
> +    && \
> +    dnf -y install \
> +        autoconf \
> +        automake \
> +        checkpolicy \
> +        clang \
> +        curl \
> +        dhcp-server \
> +        ethtool \
> +        gcc \
> +        gcc-c++ \
> +        git \
> +        glibc-langpack-en \
> +        groff \
> +        iproute \
> +        iproute-tc \
> +        iputils \
> +        kernel-devel \
> +        libcap-ng-devel \
> +        libtool \
> +        net-tools \
> +        nmap-ncat \
> +        openssl \
> +        openssl-devel \
> +        procps-ng \
> +        python3-devel \
> +        python3-pip \
> +        rpmdevtools \
> +        rsync \
> +        selinux-policy-devel \
> +        tcpdump \
> +        unbound \
> +        unbound-devel \
> +        wget \
> +        which \
> +    && \
> +    dnf clean all
> +
> +# Compile sparse from source
> +WORKDIR /workspace/sparse
> +
> +RUN git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \
> +    /workspace/sparse \
> +    && \
> +    make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install
> +
> +WORKDIR /workspace
> +
> +COPY py-requirements.txt /tmp/py-requirements.txt
> +
> +# Update and install pip dependencies
> +RUN python3 -m pip install --upgrade pip \
> +    && \
> +    python3 -m pip install wheel \
> +    && \
> +    python3 -m pip install -r /tmp/py-requirements.txt
> +
> +CMD ["/usr/sbin/init"]
> diff --git a/utilities/containers/py-requirements.txt b/utilities/containers/py-requirements.txt
> new file mode 100644
> index 000000000..d7bd21e0d
> --- /dev/null
> +++ b/utilities/containers/py-requirements.txt
> @@ -0,0 +1,6 @@
> +flake8
> +hacking>=3.0
> +sphinx
> +setuptools
> +pyelftools
> +pyOpenSSL

On 2/16/23 17:43, Mark Michelson wrote:
> Thanks Ales,
> 
> Acked-by: Mark Michelson <mmichels@redhat.com>
> 

Thanks, Ales and Mark!  I applied this to the main branch.  But I do
have one concern: I think we need to start using this in CI upstream to
avoid the script/dockerfile going stale.  Ales, do you have some cycles
to look into that?

Regards,
Dumitru

On Fri, Feb 17, 2023 at 1:04 PM Dumitru Ceara <dceara@redhat.com> wrote:

> On 2/16/23 17:43, Mark Michelson wrote:
> > Thanks Ales,
> >
> > Acked-by: Mark Michelson <mmichels@redhat.com>
> >
>
> Thanks, Ales and Mark!  I applied this to the main branch.  But I do
> have one concern: I think we need to start using this in CI upstream to
> avoid the script/dockerfile going stale.  Ales, do you have some cycles
> to look into that?
>
> Regards,
> Dumitru
>
>
Agreed, I'll look into it.

Thanks,
Ales