From patchwork Wed Dec 8 17:02:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: You-Sheng Yang X-Patchwork-Id: 1565370 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YSsWnBAo; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4J8Nlp3rLMz9ssD for ; Thu, 9 Dec 2021 04:03:18 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4J8Nlp2RjFz3bW6 for ; Thu, 9 Dec 2021 04:03:18 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YSsWnBAo; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1033; helo=mail-pj1-x1033.google.com; envelope-from=vicamo@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YSsWnBAo; dkim-atps=neutral Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4J8NlQ5hHPz3bXP for ; Thu, 9 Dec 2021 04:02:58 +1100 (AEDT) Received: by mail-pj1-x1033.google.com with SMTP id y14-20020a17090a2b4e00b001a5824f4918so4768183pjc.4 for ; Wed, 08 Dec 2021 09:02:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=poAXdVduPqdbZmZN5f7gmkNTGD9AktJCMiPTg0AktyA=; b=YSsWnBAohBFsi82yTfRIODFGoYDqiE/6BKy+UzEzAYUjjHpMUm34TKceVuRHkFuL3Y VDcr8LG5Jv1QSiu5XglyuuNo13t8XdzF8FazNffd2DRoI9+iV7d1PyQOlDWYxanerl5N 50HaVnnql1NXdX8HE1xuuf1xbUa/2IUHeB8fzdA2rJDQWvlK9zrps2Aavwjr+cWALm1O xfMDvawhVROJNRVwmVs1NOPLYhgaoB+lYHPhXICGEISVpyE6lcPve1AdokT5ju/f0eK1 Q1sbSbrK3ngUxcc4//9+vrpqUjtyybU1iKwyDyAYIU0rwraApClT4yM9SlnpPc4P6xcz 0frg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=poAXdVduPqdbZmZN5f7gmkNTGD9AktJCMiPTg0AktyA=; b=RF9kgHbKjFK03q1QT1JiWWh7qxbrWOGDD0ixw6z2ex8A/US0QhnVtH2YeWZxZK5SHH 8JhhUSUnF6KofvUv3F28jX7ukmQOoSH4NDh9WkUzW+jxn22OZWO9CayvOBhFA3SpFsdJ bRxObnOV6pI2abBZhtIS8OFGnkZAfpwNHf6W/oylAd3/yg/C/IRx66Kzn6h0PCI+qaXq HKMhtU4KOUlkWwGP1jJ467PxwwEIRqtxm7eSL9WYVeZ26GB2gHwaBq45iuwJLHPnmu+g jLA0olfrypvLEKMhuZRA+O/l8Lah3tEOu0AczPZ3tr9CWC27HTfNwxDBm+8Cpcd/A6XR zFOg== X-Gm-Message-State: AOAM530l/dHeOOhg2X7AOhlQXXLvCbvLYu3d/3oMafN4r7nPiRFlpS9J fDylZvGCLdxqUYRzrBPek/RZP9J0DNg= X-Google-Smtp-Source: ABdhPJzETw6hMK5lGe9mixhk0Fb+UkVWeKRIuQcAVv+Km0GQyM6N6/lYyugPVOOrJ6Eij8M5ZcyXCQ== X-Received: by 2002:a17:90a:7086:: with SMTP id g6mr8642141pjk.34.1638982974442; Wed, 08 Dec 2021 09:02:54 -0800 (PST) Received: from localhost (218-173-141-247.dynamic-ip.hinet.net. [218.173.141.247]) by smtp.gmail.com with ESMTPSA id c5sm3125363pgi.89.2021.12.08.09.02.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Dec 2021 09:02:54 -0800 (PST) From: You-Sheng Yang To: patchwork@lists.ozlabs.org Subject: [PATCH v3 4/6] docker: drop database maintenance stuff Date: Thu, 9 Dec 2021 01:02:36 +0800 Message-Id: <20211208170238.142017-5-vicamo@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211208170238.142017-1-vicamo@gmail.com> References: <20211208170238.142017-1-vicamo@gmail.com> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Trying to re-create database inside a client docker could be risky and error prone. 1. The previous handling process may fail service startup when an empty database was created. `test_database()` will always succeed as it tests only the existence of the database, not any table inside. Without migrations being called on such empty database, patchwork cannot work normally. 2. Checking the existence of a patchwork specific table is also invalid under the empty database scenario, because migrations might still remain to be done. As a result, we shall test if the database exist and do migrations always, so this change removes database maintenance functions. Signed-off-by: You-Sheng Yang --- docker-compose-pg.yml | 4 +- docker-compose.yml | 3 +- docs/development/installation.rst | 10 ++- ...se-maintenance-stuff-e3317975c1c53ade.yaml | 5 ++ tools/docker/entrypoint.sh | 67 +++---------------- 5 files changed, 26 insertions(+), 63 deletions(-) create mode 100644 releasenotes/notes/docker-drop-database-maintenance-stuff-e3317975c1c53ade.yaml diff --git a/docker-compose-pg.yml b/docker-compose-pg.yml index c99b25c..9129f32 100644 --- a/docker-compose-pg.yml +++ b/docker-compose-pg.yml @@ -5,6 +5,8 @@ services: volumes: - ./tools/docker/db/postdata:/var/lib/postgresql/data environment: + - POSTGRES_DB=patchwork + - POSTGRES_USER=patchwork - POSTGRES_PASSWORD=password web: @@ -27,5 +29,5 @@ services: - DATABASE_HOST=db - DATABASE_PORT=5432 - DATABASE_NAME=patchwork - - DATABASE_USER=postgres + - DATABASE_USER=patchwork - DATABASE_PASSWORD=password diff --git a/docker-compose.yml b/docker-compose.yml index b91b7ef..d824436 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -6,6 +6,7 @@ services: - ./tools/docker/db/data:/var/lib/mysql environment: - MYSQL_ROOT_PASSWORD=root + - MYSQL_DATABASE=patchwork - MYSQL_USER=patchwork - MYSQL_PASSWORD=password @@ -28,6 +29,6 @@ services: # skip DATABASE_TYPE explicitly as mysql should be the default type. - DATABASE_HOST=db - DATABASE_PORT=3306 + - DATABASE_NAME=patchwork - DATABASE_USER=patchwork - DATABASE_PASSWORD=password - - MYSQL_ROOT_PASSWORD=root diff --git a/docs/development/installation.rst b/docs/development/installation.rst index c5c0269..877c2f6 100644 --- a/docs/development/installation.rst +++ b/docs/development/installation.rst @@ -91,6 +91,10 @@ To run unit tests against the system Python packages, run: .. code-block:: shell + # For MySQL database: + $ docker-compose exec -T -- db sh -c \ + "exec mysql -uroot -p\"\${MYSQL_ROOT_PASSWORD}\" -e \"GRANT ALL ON \\\`test\\_\${MYSQL_DATABASE}%\\\`.* to '\${MYSQL_USER}'@'%'; FLUSH PRIVILEGES;\"" + $ docker-compose run --rm web python manage.py test To run unit tests for multiple versions using ``tox``, run: @@ -99,12 +103,12 @@ To run unit tests for multiple versions using ``tox``, run: $ docker-compose run --rm web tox -To reset the database before any of these commands, add ``--reset`` to the -command line after ``web`` and before any other arguments: +To reset the database, stop the db container and purge the database files: .. code-block:: shell - $ docker-compose run --rm web --reset tox + $ docker-compose stop db + $ sudo rm -rf tools/docker/db Any local edits to the project files made locally are immediately visible to the Docker container, and so should be picked up by the Django auto-reloader. diff --git a/releasenotes/notes/docker-drop-database-maintenance-stuff-e3317975c1c53ade.yaml b/releasenotes/notes/docker-drop-database-maintenance-stuff-e3317975c1c53ade.yaml new file mode 100644 index 0000000..9cf0fcb --- /dev/null +++ b/releasenotes/notes/docker-drop-database-maintenance-stuff-e3317975c1c53ade.yaml @@ -0,0 +1,5 @@ +--- +fixes: + - | + No longer perform database recreation inside development container startup. + Users should provide a ready-to-use database, empty or previously populated. diff --git a/tools/docker/entrypoint.sh b/tools/docker/entrypoint.sh index 7e0af5a..d1bad82 100755 --- a/tools/docker/entrypoint.sh +++ b/tools/docker/entrypoint.sh @@ -16,20 +16,11 @@ postgres) *) export DATABASE_TYPE=mysql mysql_args=( ${DATABASE_HOST:+--host=${DATABASE_HOST}} ${DATABASE_PORT:+--port=${DATABASE_PORT}} "--user=${DATABASE_USER}" "--password=${DATABASE_PASSWORD}" ) - mysql_root_args=( ${DATABASE_HOST:+--host=${DATABASE_HOST}} ${DATABASE_PORT:+--port=${DATABASE_PORT}} "--user=root" "--password=${MYSQL_ROOT_PASSWORD:-}" ) ;; esac # functions -test_db_connection() { - if [ ${DATABASE_TYPE} = "postgres" ]; then - echo ';' | psql "${psql_args[@]}" 2> /dev/null > /dev/null - else - mysqladmin "${mysql_root_args[@]}" ping > /dev/null 2> /dev/null - fi -} - test_database() { if [ ${DATABASE_TYPE} = "postgres" ]; then echo ';' | psql "${psql_args[@]}" "${DATABASE_NAME}" 2> /dev/null @@ -38,37 +29,6 @@ test_database() { fi } -reset_data_mysql() { - mysql "${mysql_root_args[@]}" << EOF -DROP DATABASE IF EXISTS ${DATABASE_NAME}; -CREATE DATABASE ${DATABASE_NAME} CHARACTER SET utf8; -GRANT ALL ON ${DATABASE_NAME}.* TO '${DATABASE_USER}' IDENTIFIED BY '${DATABASE_PASSWORD}'; -GRANT ALL ON \`test\\_${DATABASE_NAME}%\`.* to '${DATABASE_USER}'@'%'; -FLUSH PRIVILEGES; -EOF -} - -reset_data_postgres() { - psql "${psql_args[@]}" < /dev/null - python manage.py loaddata default_tags #> /dev/null - python manage.py loaddata default_states #> /dev/null - python manage.py loaddata default_projects #> /dev/null -} - # the script begins! # check if patchwork is mounted. Checking if we exist is a @@ -103,20 +63,20 @@ done set -e # check if db is connected -if ! test_db_connection; then +if ! test_database; then echo "The database seems not to be connected, or the ${DATABASE_USER} user is broken" echo "MySQL/Postgres may still be starting. Waiting 5 seconds." sleep 5 - if ! test_db_connection; then + if ! test_database; then echo "Still cannot connect to database." echo "Maybe you are starting the db for the first time. Waiting up to 60 seconds." for i in {0..9}; do sleep 5 - if test_db_connection; then + if test_database; then break fi done - if ! test_db_connection; then + if ! test_database; then echo "Still cannot connect to database. Giving up." echo "Are you using docker-compose? If not, have you set up the link correctly?" exit 1 @@ -124,19 +84,10 @@ if ! test_db_connection; then fi fi -# rebuild db -# do this on --reset or if the db doesn't exist -if [[ "$1" == "--reset" ]]; then - shift - reset_data -elif ! test_database; then - reset_data -fi - -if [ $# -eq 0 ]; then - # we probably ran with --reset and nothing else - # just exit cleanly - exit 0 -fi +# load initial data +python manage.py migrate #> /dev/null +python manage.py loaddata default_tags #> /dev/null +python manage.py loaddata default_states #> /dev/null +python manage.py loaddata default_projects #> /dev/null exec "$@"