From patchwork Sun Sep 3 15:14:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 809320 X-Patchwork-Delegate: stephen.finucane@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xlc4j6vT0z9t2y for ; Mon, 4 Sep 2017 01:16:05 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Wg5tXg27"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xlc4j5j6wzDrnF for ; Mon, 4 Sep 2017 01:16:05 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Wg5tXg27"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xlc3l1L2gzDqjs for ; Mon, 4 Sep 2017 01:15:15 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Wg5tXg27"; dkim-atps=neutral Received: by mail-pf0-x241.google.com with SMTP id g13so2957041pfm.2 for ; Sun, 03 Sep 2017 08:15:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AiTSSHz9Uj9/2O5GrchGAB2eQqjXn9S73Z9TfSrKb6c=; b=Wg5tXg27T+Yg64kYHXG8SUNU6yqC/3a36ltNVmwNT+hP1Z0Z4OEb6Yzz/7tGZJ81CY 1p8CLbrOfyUewAOH12+loc6NA+BQ4sC1dh7ikijQC0+CporegnP9j3rS+T1aU8CsKqS+ 75c/iWlPdne+57xjxNUqUx8i3pOt3prMwCC9Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AiTSSHz9Uj9/2O5GrchGAB2eQqjXn9S73Z9TfSrKb6c=; b=UkWQ1c5w6miRVhF6k/256r09q9PdJ3gQpwnpwPoVgmKbmUR6duNLqTtmRl/NlWM9rz B5R7YKbZhSxl/K686H0KkJZ6QB0FMMRnN2LBJI4pJCWTC4oG7mmlAtm2zm723OjuYv0F Pazs5dxkXtcTHGPPwc3JpHidEik3DYENoh5R+gkAtUgtzNlBW/SYadowu6glP2ei1bcu +o1oU6GJa4apdx61x6T7ypIuEiq6arLDH5I5BOm62GeaUkDxcS9zUe1xAs68pANf88LZ WCFx8gSwBgjYwGbPf1vZxjG3m9juEMPtvzPEqCay3ak2PsgHpFH6hu1qyorQXZEssF9S APHw== X-Gm-Message-State: AHPjjUh0jiRAUClABhqMtr8h7dJIOXlwx6GrMMjOf5UzG9cYEJn23/wD oT89Mhovpfp6XqMCNk2dOg== X-Google-Smtp-Source: ADKCNb73eU2C2JYwz85roqp6O6FT8VIR3S77L7cCLJCf6NOyMJ9CZZMNO0Q+oWNHBoc9qiU7a36zAA== X-Received: by 10.84.129.14 with SMTP id 14mr9908982plb.38.1504451713043; Sun, 03 Sep 2017 08:15:13 -0700 (PDT) Received: from localhost.localdomain (124-171-202-56.dyn.iinet.net.au. [124.171.202.56]) by smtp.gmail.com with ESMTPSA id t10sm7565993pgr.54.2017.09.03.08.15.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 03 Sep 2017 08:15:12 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 3/4] Support testing with PostgreSQL Date: Mon, 4 Sep 2017 01:14:43 +1000 Message-Id: <20170903151444.25660-4-dja@axtens.net> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170903151444.25660-1-dja@axtens.net> References: <20170903151444.25660-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" This allows us to easily test against PostgreSQL using the same tooling we normally use. This is helpful in (for example) shaking out the test failures that were observed on ozlabs.org To use it: docker-compose -f docker-compose-pg.yml (You may find it necessary to do a 'docker-compose down' first, depending on what state the system is in and what command you're running.) Signed-off-by: Daniel Axtens --- docker-compose-pg.yml | 30 ++++++++++++++++++++++++++ requirements-test.txt | 3 ++- tools/docker/Dockerfile | 2 +- tools/docker/entrypoint.sh | 53 +++++++++++++++++++++++++++++++++++++--------- 4 files changed, 76 insertions(+), 12 deletions(-) create mode 100644 docker-compose-pg.yml diff --git a/docker-compose-pg.yml b/docker-compose-pg.yml new file mode 100644 index 000000000000..34900418375e --- /dev/null +++ b/docker-compose-pg.yml @@ -0,0 +1,30 @@ +# the version of docker-compose shipped in ubuntu 16.04 is +# 1.5.2, which doesn't support version 2 syntax. Yay! +# also, v1 doesn't support explicit build args, so if you're not +# uid 1000, you will either need to manually hack the Dockerfile +# or upgrade to v2 and use the build-arg to override it. + +db: + image: postgres + environment: + - POSTGRES_PASSWORD=super_sekrit_postgres_passwd + volumes: + - ./tools/docker/db/postdata:/var/lib/postgresql/data + +web: + build: . + dockerfile: ./tools/docker/Dockerfile + command: python3 manage.py runserver 0.0.0.0:8000 + volumes: + - .:/home/patchwork/patchwork/ + ports: + - "8000:8000" + links: + - db + environment: + - PGPASSWORD=super_sekrit_postgres_passwd + - PW_TEST_DB_HOST=db + - PW_TEST_DB_PORT=5432 + - PW_TEST_DB_TYPE=postgres + - PW_TEST_DB_USER=postgres + - PW_TEST_DB_PASS=super_sekrit_postgres_passwd diff --git a/requirements-test.txt b/requirements-test.txt index cead336c7c77..141cf661663d 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -1,4 +1,5 @@ -mysqlclient>=1.3,<1.4 # replace this with psycopg2 for a PostgreSQL backend +mysqlclient>=1.3,<1.4 +psycopg2>=2.7,<2.8 django-debug-toolbar==1.8 python-dateutil>2.0,<3.0 selenium>=3.0,<4.0 diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index ff05707a6049..946c646188e5 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -21,7 +21,7 @@ RUN apt-get update -qq && \ python3.5-dev python3-pip python3-setuptools python3-wheel \ python3.4-dev findutils=4.4.2-7 \ libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \ - chromium-browser build-essential git && \ + chromium-browser build-essential git postgresql-client && \ ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/ # User diff --git a/tools/docker/entrypoint.sh b/tools/docker/entrypoint.sh index 2f413b02e83e..4b656adaeb35 100755 --- a/tools/docker/entrypoint.sh +++ b/tools/docker/entrypoint.sh @@ -1,13 +1,31 @@ #!/bin/bash -set -euo pipefail +set -eo pipefail + +if [ x${PW_TEST_DB_TYPE} = x ]; then + PW_TEST_DB_TYPE=mysql +fi + +set -u # functions test_db_connection() { - mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null + if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then + echo ';' | psql -h $PW_TEST_DB_HOST -U postgres 2> /dev/null > /dev/null + else + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null + fi } -reset_data() { +test_database() { + if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then + echo ';' | psql -h $PW_TEST_DB_HOST -U postgres patchwork 2> /dev/null + else + echo ';' | mysql -h $PW_TEST_DB_HOST -u patchwork -ppassword patchwork 2> /dev/null + fi +} + +reset_data_mysql() { mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF DROP DATABASE IF EXISTS patchwork; CREATE DATABASE patchwork CHARACTER SET utf8; @@ -15,6 +33,21 @@ GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%'; FLUSH PRIVILEGES; EOF +} + +reset_data_postgres() { + psql -h $PW_TEST_DB_HOST -U postgres < /dev/null @@ -46,13 +79,13 @@ for x in /tmp/requirements-*.txt; do fi done -# check if mysql is connected +# check if db is connected if ! test_db_connection; then - echo "MySQL seems not to be connected, or the patchwork user is broken" - echo "MySQL may still be starting. Waiting 5 seconds." + echo "The database seems not to be connected, or the patchwork user is broken" + echo "MySQL/Postgres may still be starting. Waiting 5 seconds." sleep 5 if ! test_db_connection; then - echo "Still cannot connect to MySQL." + echo "Still cannot connect to database." echo "Maybe you are starting the db for the first time. Waiting up to 60 seconds." for i in {0..9}; do sleep 5 @@ -61,19 +94,19 @@ if ! test_db_connection; then fi done if ! test_db_connection; then - echo "Still cannot connect to MySQL. Giving up." + echo "Still cannot connect to database. Giving up." echo "Are you using docker-compose? If not, have you set up the link correctly?" exit 1 fi fi fi -# rebuild mysql db +# rebuild db # do this on --reset or if the db doesn't exist if [[ "$1" == "--reset" ]]; then shift reset_data -elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2> /dev/null ); then +elif ! test_database; then reset_data fi