Message ID | 1470291041-4554-6-git-send-email-dja@axtens.net |
---|---|
State | Changes Requested |
Headers | show |
On 04/08/16 16:10, Daniel Axtens wrote: > This makes it possible to use Docker and docker-compose for development > as an alternative to Vagrant. > > I quite liked vagrant a couple of years ago, but currently: > > * Trying to install VirtualBox on Ubuntu wants me to disable > Secure Boot, and I don't want to do that. > > * Trying to use the libvirt plugin for vagrant requires I pick > from a very small set of possible images, and requires that I > install the upstream vagrant rather than the vagrant shipped > with Ubuntu 16.04 > > * I find docker containers faster to work with and more transparent. > > So I've done the work to make docker work for Patchwork development. > This doesn't break or in any way interfere with using Vagrant, it just > provides an alternative. > > It includes support for headless selenium tests using Chromium. > > Signed-off-by: Daniel Axtens <dja@axtens.net> Excellent! I don't use either Docker or Vagrant very much these days, but definitely Docker more than Vagrant... Haven't tested this yet and my Docker-fu is pretty rusty, so I'll hold off on my Reviewed-by until I've given it a go, but I didn't see any major issues. Minor comments below. > +# Development Installation using Docker > + > +1. Install Docker and docker-compose. > +2. Clone this repo, as with vagrant. > +3. Build the images. This will download ~200MB from the internet: > + > + $ docker-compose build > + > +3. Run as follows: Should that be 4? > + > + * Regular server: > + > + $ docker-compose up > + > + This will be visible on http://localhost:8000/. > + > + * Shell: > + > + $ docker-compose run --rm web --shell > + > + * Quick test (not including selenium UI interaction tests): > + > + $ docker-compose run --rm web --quick-test > + > + * Full tests, including selenium, run headlessly: > + > + $ docker-compose run --rm web --test > + > + * To reset the database before beginning, add `--reset` to the command line after `web` and before any other arguments. > + > + * If you want to run non-headless tests, you'll need something like this ugly hack: > + > + $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ --link patchwork_db_1:db -p 8000:8000 -v /tmp/.X11-unix:/tmp/.X11-unix -e PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash > + > +With both vagrant and docker, any edits to the project files made locally are immediately visible to the VM/container, and so should be picked up by the Django auto-reloader. > + > # Talks and Presentations > > * [**A New Patchwork**][pdf-fosdem] - FOSDEM 2016 > diff --git a/docker-compose.yml b/docker-compose.yml > new file mode 100644 > index 000000000000..80d2571a2dbc > --- /dev/null > +++ b/docker-compose.yml > @@ -0,0 +1,23 @@ > +# the version of docker-compose shipped in ubuntu 16.04 is > +# 1.5.2, which doesn't support version 2 syntax. Yay! > +# also, v1 doesn't support explicit build args, so if you're not > +# uid 1000, you will either need to manually hack the Dockerfile > +# or upgrade to v2 and use the build-arg to override it. Gross :( > + > +db: > + build: docker/db > + volumes: > + - ./docker/db/data:/var/lib/mysql > +web: > + build: . > + dockerfile: ./docker/Dockerfile > + command: python3 manage.py runserver 0.0.0.0:8000 > + volumes: > + - .:/home/patchwork/patchwork/ > + ports: > + - "8000:8000" > + links: > + - db > + environment: > + - PW_TEST_DB_HOST=db > + - PW_TEST_DB_PORT=3306 > \ No newline at end of file > diff --git a/docker/Dockerfile b/docker/Dockerfile > new file mode 100644 > index 000000000000..b5262cc516a3 > --- /dev/null > +++ b/docker/Dockerfile > @@ -0,0 +1,44 @@ > +FROM ubuntu > + > +ARG UID=1000 > + > +ENV PROJECT_HOME /home/patchwork/patchwork > + > +ENV db_user root > +ENV db_pass password > + > +ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev > +ENV DEBIAN_FRONTEND noninteractive > +ENV PYTHONUNBUFFERED 1 > + > +# System > +RUN apt-get update -qq && \ > + apt-get install -y --no-install-recommends \ > + python-dev python-pip python-setuptools python-wheel \ > + python3-dev python3-pip python3-setuptools python3-wheel \ > + libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \ > + chromium-browser build-essential && \ > + ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/ > + > +# User > +RUN useradd --uid=$UID --create-home patchwork > + > +# Python requirements. > +# If you update requrirements, you should rebuild the container. requirements > +COPY requirements-*.txt /tmp/ > +RUN pip3 install virtualenv tox && \ > + pip3 install -r /tmp/requirements-dev.txt > +# we deliberately leave the requirements files in tmp so we can > +# ping the user if they change it! > + > +COPY docker/bashrc /tmp/bashrc > + > +# we put the code in ~/patchwork rather than ~ so that we > +# can put in these bashrc snippets > +RUN cat /tmp/bashrc >> /home/patchwork/.bashrc > + > +COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh > + > +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] > +USER patchwork > +WORKDIR /home/patchwork/patchwork > diff --git a/docker/bashrc b/docker/bashrc > new file mode 100644 > index 000000000000..eb2ed7dddb85 > --- /dev/null > +++ b/docker/bashrc > @@ -0,0 +1,5 @@ > +# This snippet is appended to ~/.bashrc when the container is created > + > +alias runserver='python3 $PROJECT_HOME/manage.py runserver 0.0.0.0:8000' > +alias createsu='python3 $PROJECT_HOME/manage.py createsuperuser' > + > diff --git a/docker/db/.dockerignore b/docker/db/.dockerignore > new file mode 100644 > index 000000000000..1269488f7fb1 > --- /dev/null > +++ b/docker/db/.dockerignore > @@ -0,0 +1 @@ > +data > diff --git a/docker/db/.gitignore b/docker/db/.gitignore > new file mode 100644 > index 000000000000..60baa9cb833f > --- /dev/null > +++ b/docker/db/.gitignore > @@ -0,0 +1 @@ > +data/* > diff --git a/docker/db/Dockerfile b/docker/db/Dockerfile > new file mode 100644 > index 000000000000..5df9b5acb486 > --- /dev/null > +++ b/docker/db/Dockerfile > @@ -0,0 +1,10 @@ > +FROM mysql:5.7 > + > +ENV MYSQL_ROOT_PASSWORD password > +ENV MYSQL_USER patchwork > +ENV MYSQL_PASSWORD password > + > +# We don't want to use the MYSQL_DATABASE env here because > +# we want to be able to create the database with UTF-8 explictly. > +# We also can't load in the data because it's in XML, yay. > + > diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh > new file mode 100755 > index 000000000000..c6cf18713fd8 > --- /dev/null > +++ b/docker/entrypoint.sh > @@ -0,0 +1,80 @@ > +#!/bin/bash > +set -euo pipefail > + > +# functions > +test_db_connection() { > + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null > +} > + > +reset_data() { > + mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF > +DROP DATABASE IF EXISTS patchwork; > +CREATE DATABASE patchwork CHARACTER SET utf8; > +GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; > +FLUSH PRIVILEGES; > +EOF > + > + # load initial data > + python3 $PROJECT_HOME/manage.py migrate #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_tags #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_states #> /dev/null > + python3 $PROJECT_HOME/manage.py loaddata default_projects #> /dev/null > +} > + > + > +# The script begins! > + > +# check if patchwork is mounted. Checking if we exist is a > +# very good start! > +if [ ! -f ~patchwork/patchwork/docker/entrypoint.sh ]; then > + echo "The patchwork directory doesn't seem to be mounted!" > + echo "Are you using docker-compose?" > + echo "If not, you need -v PATH_TO_PATCHWORK:/home/patchwork/patchwork" > + exit 1 > +fi > + > +# check if we need to rebuild because requirements changed > +for x in /tmp/requirements-*.txt; do > + if ! cmp $x ~/patchwork/$(basename $x); then > + echo "A requirements file has changed." > + echo "Please rebuild the patchwork image:" > + echo " docker-compose build web" > + exit 1 > + fi > +done > + > +# check if mysql is connected > +if ! test_db_connection; then > + echo "MySQL seems not to be connected, or the patchwork user is broken" > + echo "MySQL may still be starting. Waiting 5 seconds." > + sleep 5 > + if ! test_db_connection; then > + echo "Still cannot connect to MySQL." > + echo "Are you using docker-compose? If not, have you set up the link correctly?" > + exit 1 > + fi > +fi > + > +# rebuild mysql db > +# do this on --reset or if the db doesn't exist > +if [[ "$1" == "--reset" ]]; then > + shift > + reset_data > +elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2> /dev/null ); then > + reset_data > +fi > + > +if [ $# -eq 0 ]; then > + # we probably ran with --reset and nothing else > + # just exit cleanly > + exit 0 > +elif [ "$1" == "--shell" ]; then > + exec bash > +elif [ "$1" == "--quick-test" ]; then > + export PW_SKIP_BROWSER_TESTS=yes > + python3 manage.py test > +elif [ "$1" == "--test" ]; then > + xvfb-run --server-args='-screen 0, 1024x768x16' python3 manage.py test > +else # run whatever CMD is set to > + $@ > +fi >
On 04/08/16 16:10, Daniel Axtens wrote: > This makes it possible to use Docker and docker-compose for development > as an alternative to Vagrant. > > I quite liked vagrant a couple of years ago, but currently: > > * Trying to install VirtualBox on Ubuntu wants me to disable > Secure Boot, and I don't want to do that. > > * Trying to use the libvirt plugin for vagrant requires I pick > from a very small set of possible images, and requires that I > install the upstream vagrant rather than the vagrant shipped > with Ubuntu 16.04 > > * I find docker containers faster to work with and more transparent. > > So I've done the work to make docker work for Patchwork development. > This doesn't break or in any way interfere with using Vagrant, it just > provides an alternative. > > It includes support for headless selenium tests using Chromium. > > Signed-off-by: Daniel Axtens <dja@axtens.net> Step 18 : WORKDIR /home/patchwork/patchwork ---> Running in d3058a7597b4 ---> 135d5890866b Removing intermediate container d3058a7597b4 Successfully built 135d5890866b WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`. Creating patchwork_db_1 Creating patchwork_web_1 Attaching to patchwork_db_1, patchwork_web_1 db_1 | Initializing database web_1 | MySQL seems not to be connected, or the patchwork user is broken web_1 | MySQL may still be starting. Waiting 5 seconds. web_1 | Still cannot connect to MySQL. web_1 | Are you using docker-compose? If not, have you set up the link correctly? patchwork_web_1 exited with code 1 db_1 | Database initialized db_1 | MySQL init process in progress... db_1 | Warning: Unable to load '/usr/share/zoneinfo/Factory' as time zone. Skipping it. db_1 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. db_1 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it. db_1 | Warning: Unable to load '/usr/share/zoneinfo/posix/Factory' as time zone. Skipping it. db_1 | Warning: Unable to load '/usr/share/zoneinfo/right/Factory' as time zone. Skipping it. db_1 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. db_1 | mysql: [Warning] Using a password on the command line interface can be insecure. db_1 | mysql: [Warning] Using a password on the command line interface can be insecure. db_1 | db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* db_1 | db_1 | db_1 | MySQL init process done. Ready for start up. db_1 | Perhaps 5 seconds isn't enough?
Andrew Donnellan <andrew.donnellan@au1.ibm.com> writes: > Attaching to patchwork_db_1, patchwork_web_1 > db_1 | Initializing database > web_1 | MySQL seems not to be connected, or the patchwork user is broken > web_1 | MySQL may still be starting. Waiting 5 seconds. > web_1 | Still cannot connect to MySQL. > web_1 | Are you using docker-compose? If not, have you set up the link > correctly? > patchwork_web_1 exited with code 1 > db_1 | Database initialized > db_1 | MySQL init process in progress... ... > > Perhaps 5 seconds isn't enough? > Erk. So it's OK in general, just not if the database is being set up for the first time. I'll make it sleep a bit longer to catch that and I'll test without a DB instance. Regards, Daniel > -- > Andrew Donnellan OzLabs, ADL Canberra > andrew.donnellan@au1.ibm.com IBM Australia Limited
Daniel Axtens <dja@axtens.net> writes: > Erk. So it's OK in general, just not if the database is being set up for > the first time. I'll make it sleep a bit longer to catch that and I'll > test without a DB instance. And apparently that makes my test fail because the patchwork user no longer can create the test db. *sigh* I'll roll the fix for that into v3. Regards, Daniel
diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000000..24473bbe1237 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +.vagrant +docker/db + diff --git a/README.md b/README.md index c83e7d31f1db..6f889c09def0 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ subsystems of the Linux kernel. Although Patchwork has been developed with the kernel workflow in mind, the aim is to be flexible enough to suit the majority of community projects. -# Development Installation +# Development Installation using Vagrant 1. Install [**Vagrant**][ref-vagrant] 2. Clone this repo: @@ -32,6 +32,42 @@ of community projects. $ cd patchwork $ vagrant up +# Development Installation using Docker + +1. Install Docker and docker-compose. +2. Clone this repo, as with vagrant. +3. Build the images. This will download ~200MB from the internet: + + $ docker-compose build + +3. Run as follows: + + * Regular server: + + $ docker-compose up + + This will be visible on http://localhost:8000/. + + * Shell: + + $ docker-compose run --rm web --shell + + * Quick test (not including selenium UI interaction tests): + + $ docker-compose run --rm web --quick-test + + * Full tests, including selenium, run headlessly: + + $ docker-compose run --rm web --test + + * To reset the database before beginning, add `--reset` to the command line after `web` and before any other arguments. + + * If you want to run non-headless tests, you'll need something like this ugly hack: + + $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ --link patchwork_db_1:db -p 8000:8000 -v /tmp/.X11-unix:/tmp/.X11-unix -e PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash + +With both vagrant and docker, any edits to the project files made locally are immediately visible to the VM/container, and so should be picked up by the Django auto-reloader. + # Talks and Presentations * [**A New Patchwork**][pdf-fosdem] - FOSDEM 2016 diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 000000000000..80d2571a2dbc --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,23 @@ +# the version of docker-compose shipped in ubuntu 16.04 is +# 1.5.2, which doesn't support version 2 syntax. Yay! +# also, v1 doesn't support explicit build args, so if you're not +# uid 1000, you will either need to manually hack the Dockerfile +# or upgrade to v2 and use the build-arg to override it. + +db: + build: docker/db + volumes: + - ./docker/db/data:/var/lib/mysql +web: + build: . + dockerfile: ./docker/Dockerfile + command: python3 manage.py runserver 0.0.0.0:8000 + volumes: + - .:/home/patchwork/patchwork/ + ports: + - "8000:8000" + links: + - db + environment: + - PW_TEST_DB_HOST=db + - PW_TEST_DB_PORT=3306 \ No newline at end of file diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 000000000000..b5262cc516a3 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,44 @@ +FROM ubuntu + +ARG UID=1000 + +ENV PROJECT_HOME /home/patchwork/patchwork + +ENV db_user root +ENV db_pass password + +ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev +ENV DEBIAN_FRONTEND noninteractive +ENV PYTHONUNBUFFERED 1 + +# System +RUN apt-get update -qq && \ + apt-get install -y --no-install-recommends \ + python-dev python-pip python-setuptools python-wheel \ + python3-dev python3-pip python3-setuptools python3-wheel \ + libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \ + chromium-browser build-essential && \ + ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/ + +# User +RUN useradd --uid=$UID --create-home patchwork + +# Python requirements. +# If you update requrirements, you should rebuild the container. +COPY requirements-*.txt /tmp/ +RUN pip3 install virtualenv tox && \ + pip3 install -r /tmp/requirements-dev.txt +# we deliberately leave the requirements files in tmp so we can +# ping the user if they change it! + +COPY docker/bashrc /tmp/bashrc + +# we put the code in ~/patchwork rather than ~ so that we +# can put in these bashrc snippets +RUN cat /tmp/bashrc >> /home/patchwork/.bashrc + +COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh + +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] +USER patchwork +WORKDIR /home/patchwork/patchwork diff --git a/docker/bashrc b/docker/bashrc new file mode 100644 index 000000000000..eb2ed7dddb85 --- /dev/null +++ b/docker/bashrc @@ -0,0 +1,5 @@ +# This snippet is appended to ~/.bashrc when the container is created + +alias runserver='python3 $PROJECT_HOME/manage.py runserver 0.0.0.0:8000' +alias createsu='python3 $PROJECT_HOME/manage.py createsuperuser' + diff --git a/docker/db/.dockerignore b/docker/db/.dockerignore new file mode 100644 index 000000000000..1269488f7fb1 --- /dev/null +++ b/docker/db/.dockerignore @@ -0,0 +1 @@ +data diff --git a/docker/db/.gitignore b/docker/db/.gitignore new file mode 100644 index 000000000000..60baa9cb833f --- /dev/null +++ b/docker/db/.gitignore @@ -0,0 +1 @@ +data/* diff --git a/docker/db/Dockerfile b/docker/db/Dockerfile new file mode 100644 index 000000000000..5df9b5acb486 --- /dev/null +++ b/docker/db/Dockerfile @@ -0,0 +1,10 @@ +FROM mysql:5.7 + +ENV MYSQL_ROOT_PASSWORD password +ENV MYSQL_USER patchwork +ENV MYSQL_PASSWORD password + +# We don't want to use the MYSQL_DATABASE env here because +# we want to be able to create the database with UTF-8 explictly. +# We also can't load in the data because it's in XML, yay. + diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh new file mode 100755 index 000000000000..c6cf18713fd8 --- /dev/null +++ b/docker/entrypoint.sh @@ -0,0 +1,80 @@ +#!/bin/bash +set -euo pipefail + +# functions +test_db_connection() { + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null +} + +reset_data() { + mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF +DROP DATABASE IF EXISTS patchwork; +CREATE DATABASE patchwork CHARACTER SET utf8; +GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; +FLUSH PRIVILEGES; +EOF + + # load initial data + python3 $PROJECT_HOME/manage.py migrate #> /dev/null + python3 $PROJECT_HOME/manage.py loaddata default_tags #> /dev/null + python3 $PROJECT_HOME/manage.py loaddata default_states #> /dev/null + python3 $PROJECT_HOME/manage.py loaddata default_projects #> /dev/null +} + + +# The script begins! + +# check if patchwork is mounted. Checking if we exist is a +# very good start! +if [ ! -f ~patchwork/patchwork/docker/entrypoint.sh ]; then + echo "The patchwork directory doesn't seem to be mounted!" + echo "Are you using docker-compose?" + echo "If not, you need -v PATH_TO_PATCHWORK:/home/patchwork/patchwork" + exit 1 +fi + +# check if we need to rebuild because requirements changed +for x in /tmp/requirements-*.txt; do + if ! cmp $x ~/patchwork/$(basename $x); then + echo "A requirements file has changed." + echo "Please rebuild the patchwork image:" + echo " docker-compose build web" + exit 1 + fi +done + +# check if mysql is connected +if ! test_db_connection; then + echo "MySQL seems not to be connected, or the patchwork user is broken" + echo "MySQL may still be starting. Waiting 5 seconds." + sleep 5 + if ! test_db_connection; then + echo "Still cannot connect to MySQL." + echo "Are you using docker-compose? If not, have you set up the link correctly?" + exit 1 + fi +fi + +# rebuild mysql db +# do this on --reset or if the db doesn't exist +if [[ "$1" == "--reset" ]]; then + shift + reset_data +elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2> /dev/null ); then + reset_data +fi + +if [ $# -eq 0 ]; then + # we probably ran with --reset and nothing else + # just exit cleanly + exit 0 +elif [ "$1" == "--shell" ]; then + exec bash +elif [ "$1" == "--quick-test" ]; then + export PW_SKIP_BROWSER_TESTS=yes + python3 manage.py test +elif [ "$1" == "--test" ]; then + xvfb-run --server-args='-screen 0, 1024x768x16' python3 manage.py test +else # run whatever CMD is set to + $@ +fi
This makes it possible to use Docker and docker-compose for development as an alternative to Vagrant. I quite liked vagrant a couple of years ago, but currently: * Trying to install VirtualBox on Ubuntu wants me to disable Secure Boot, and I don't want to do that. * Trying to use the libvirt plugin for vagrant requires I pick from a very small set of possible images, and requires that I install the upstream vagrant rather than the vagrant shipped with Ubuntu 16.04 * I find docker containers faster to work with and more transparent. So I've done the work to make docker work for Patchwork development. This doesn't break or in any way interfere with using Vagrant, it just provides an alternative. It includes support for headless selenium tests using Chromium. Signed-off-by: Daniel Axtens <dja@axtens.net> --- v2: Chrome -> Chromium Minor cleanups --- .dockerignore | 3 ++ README.md | 38 ++++++++++++++++++++++- docker-compose.yml | 23 ++++++++++++++ docker/Dockerfile | 44 +++++++++++++++++++++++++++ docker/bashrc | 5 ++++ docker/db/.dockerignore | 1 + docker/db/.gitignore | 1 + docker/db/Dockerfile | 10 +++++++ docker/entrypoint.sh | 80 +++++++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 204 insertions(+), 1 deletion(-) create mode 100644 .dockerignore create mode 100644 docker-compose.yml create mode 100644 docker/Dockerfile create mode 100644 docker/bashrc create mode 100644 docker/db/.dockerignore create mode 100644 docker/db/.gitignore create mode 100644 docker/db/Dockerfile create mode 100755 docker/entrypoint.sh