| Message ID | 20190608173622.6711-5-stephen@that.guru |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Integrate tox-docker | expand |
Stephen Finucane <stephen@that.guru> writes: > This eliminates the need to use docker-compose for most use cases. > Instead, we can now do: > > tox -e py27-django111-postgres > > If you're using a locally configured PostgreSQL or MySQL instance, you > simply omit the last factor and things behave as before: > > tox -e py27-django111 > > We removed the 'venv' environment, since it was never actually that > useful and is even less so now (you'd need to have a local DB set up) > and add the 'skip_missing_interpreters' flag for folks on an OS that > doesn't provide all the Python versions under the sun like Fedora does. > > Signed-off-by: Stephen Finucane <stephen@that.guru> > Cc: Daniel Axtens <dja@axtens.net> > --- > v3: > - Bump the mysql container version to 8.0 to work around an issue with > tox-docker (https://github.com/tox-dev/tox-docker/issues/35) > - Use the 'root' user for the mysql container since it simplifies things > greatly (we do this for Travis already) > - Revert changes to requirements.txt which turned out to be unnecessary > and broke Travis > - Remove the venv environment instead of attempting to hack it into > working > --- > docs/development/contributing.rst | 28 +++++++--------------------- > docs/development/installation.rst | 6 ------ > patchwork/settings/dev.py | 12 ++++++++++++ > tox.ini | 25 ++++++++++++++++++++----- > 4 files changed, 39 insertions(+), 32 deletions(-) > > diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst > index 5089bba8..f713f872 100644 > --- a/docs/development/contributing.rst > +++ b/docs/development/contributing.rst > @@ -31,29 +31,15 @@ Testing > ------- > > Patchwork includes a `tox`_ script to automate testing. This requires a > -functional database and some Python requirements like *tox*. Refer to > -:doc:`installation` for information on how to configure these. > - > -You may also need to install *tox*. If so, do this now: > +functional database and some Python requirements like *tox*. These can be > +installed using :command:`pip`: > > .. code-block:: shell > > - $ pip install --user tox > - > -.. tip:: > - > - If you're using Docker, you may not need to install *tox* > - locally. Instead, it will already be installed inside the > - container. For Docker, you can run *tox* like so: > - > - .. code-block:: shell > - > - $ docker-compose run --rm web tox [ARGS...] > - > - For more information, refer to :ref:`installation-docker`. > + $ pip install --user tox tox-docker > > -Assuming these requirements are met, actually testing Patchwork is quite easy > -to do. To start, you can show the default targets like so: > +Once installed, actually testing Patchwork is quite easy to do. To start, you > +can show the default targets like so: > > .. code-block:: shell > > @@ -66,7 +52,7 @@ parameter: > > .. code-block:: shell > > - $ tox -e py27-django18 > + $ tox -e py36-django21-mysql So I'm trying this out (finally!) and it seems to want to install all the dependencies locally before starting a container. I don't have the mysql bits installed, so it fails looking for `mysql_config`. Is this supposed to happen or am I Doing It Wrong? Regards, Daniel > > In the case of the unit tests targets, you can also run specific tests by > passing the fully qualified test name as an additional argument to this > @@ -74,7 +60,7 @@ command: > > .. code-block:: shell > > - $ tox -e py27-django18 patchwork.tests.SubjectCleanUpTest > + $ tox -e py36-django21-mysql patchwork.tests.SubjectCleanUpTest > > Because Patchwork support multiple versions of Django, it's very important that > you test against all supported versions. When run without argument, tox will do > diff --git a/docs/development/installation.rst b/docs/development/installation.rst > index 0ab755f4..433c3a41 100644 > --- a/docs/development/installation.rst > +++ b/docs/development/installation.rst > @@ -86,12 +86,6 @@ To run unit tests against the system Python packages, run: > > $ docker-compose run --rm web python manage.py test > > -To run unit tests for multiple versions using ``tox``, run: > - > -.. code-block:: shell > - > - $ docker-compose run --rm web tox > - > To reset the database before any of these commands, add ``--reset`` to the > command line after ``web`` and before any other arguments: > > diff --git a/patchwork/settings/dev.py b/patchwork/settings/dev.py > index 53fa58f6..cfb9256c 100644 > --- a/patchwork/settings/dev.py > +++ b/patchwork/settings/dev.py > @@ -19,6 +19,18 @@ try: > except ImportError: > debug_toolbar = None > > +# > +# tox-docker settings > +# > + > +if 'POSTGRES_5432_TCP' in os.environ: > + os.environ['PW_TEST_DB_HOST'] = os.environ['POSTGRES_HOST'] > + os.environ['PW_TEST_DB_PORT'] = os.environ['POSTGRES_5432_TCP'] > +elif 'MYSQL_3306_TCP' in os.environ: > + os.environ['PW_TEST_DB_HOST'] = os.environ['MYSQL_HOST'] > + os.environ['PW_TEST_DB_PORT'] = os.environ['MYSQL_3306_TCP'] > + > + > # > # Core settings > # https://docs.djangoproject.com/en/1.11/ref/settings/#core-settings > diff --git a/tox.ini b/tox.ini > index d4c34e1c..3c71dafd 100644 > --- a/tox.ini > +++ b/tox.ini > @@ -1,6 +1,7 @@ > [tox] > minversion = 2.0 > -envlist = pep8,docs,py{27,34}-django111,py{35,36}-django{111,20,21,22} > +envlist = pep8,docs,py{27,34}-django111-{mysql,postgres},py{35,36}-django{111,20,21,22}-{mysql,postgres} > +skip_missing_interpreters = True > skipsdist = True > > [testenv] > @@ -17,6 +18,9 @@ deps = > django22: django>=2.2,<2.3 > django22: djangorestframework>=3.9.2 > django22: django-filter>=2.1,<3.0 > +docker = > + postgres: postgres:9.6 > + mysql: mysql:8.0 > setenv = > DJANGO_SETTINGS_MODULE = patchwork.settings.dev > PYTHONDONTWRITEBYTECODE = 1 > @@ -24,15 +28,25 @@ setenv = > py27: PYTHONWARNINGS = once > py{34,36}:PYTHONWARNINGS = once,ignore::ImportWarning:backports > py35:PYTHONWARNINGS = once,ignore::ResourceWarning:unittest.suite,ignore::ImportWarning:backports > + postgres: PW_TEST_DB_TYPE = postgres > + postgres: PW_TEST_DB_USER = postgres > + postgres: PW_TEST_DB_PASS = password > + mysql: PW_TEST_DB_USER = root > + mysql: PW_TEST_DB_PASS = password > passenv = > http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY > PW_TEST_DB_TYPE PW_TEST_DB_USER PW_TEST_DB_PASS PW_TEST_DB_HOST > PW_TEST_DB_PORT > +dockerenv = > + postgres: POSTGRES_PASSWORD=password > + mysql: MYSQL_ROOT_PASSWORD=password > commands = > - python {toxinidir}/manage.py test --noinput '{posargs:patchwork}' > + python {toxinidir}/manage.py test --noinput {posargs:patchwork} > > [testenv:bashate] > deps = bashate > +docker = > +dockerenv = > whitelist_externals = bash > commands = > bash -c "find {toxinidir} \ > @@ -43,6 +57,8 @@ commands = > [testenv:pep8] > basepython = python2.7 > deps = flake8 > +docker = > +dockerenv = > commands = flake8 {posargs} patchwork patchwork/bin/pwclient > > [flake8] > @@ -57,6 +73,8 @@ exclude = ./patchwork/migrations > [testenv:docs] > deps = > -r{toxinidir}/docs/requirements.txt > +docker = > +dockerenv = > commands = > sphinx-build -E -W -b dirhtml -d docs/_build/doctrees docs docs/_build/html > > @@ -67,9 +85,6 @@ deps = > -r{toxinidir}/requirements-prod.txt > commands = pylint patchwork --rcfile=pylint.rc > > -[testenv:venv] > -commands = {posargs} > - > [testenv:coverage] > basepython = python2.7 > deps = > -- > 2.21.0
>> .. code-block:: shell >> >> - $ tox -e py27-django18 >> + $ tox -e py36-django21-mysql > > So I'm trying this out (finally!) and it seems to want to install all > the dependencies locally before starting a container. I don't have the > mysql bits installed, so it fails looking for `mysql_config`. Is this > supposed to happen or am I Doing It Wrong? > Ok, so on further analysis it looks like this is the designed behaviour: that when running tox, all the python versions and local dependencies would live on my laptop directly rather than in a docker container. If so, I'm not a fan. I am not primarily a web, python, or database developer and I like having all of that stuff live in an isolated docker container. I especially like that it's also consistent for everyone who wants to hack on patchwork - that they can run the full suite of tests across all the supported versions with nothing more than docker and docker-compose. tox-docker provides, afaict, no way to do this. (Also, less universally, I run Ubuntu, not Fedora and getting multiple python versions is a pain, as you can see from the dockerfiles.) What's the main problem that you're trying to solve here? Is it that you have to type 'docker-compose run web --tox -e py36-django21' rather than just 'tox -e py36-django21'? Regards, Daniel
On Thu, 2019-08-22 at 23:52 +1000, Daniel Axtens wrote: > > > .. code-block:: shell > > > > > > - $ tox -e py27-django18 > > > + $ tox -e py36-django21-mysql > > > > So I'm trying this out (finally!) and it seems to want to install all > > the dependencies locally before starting a container. I don't have the > > mysql bits installed, so it fails looking for `mysql_config`. Is this > > supposed to happen or am I Doing It Wrong? > > > > Ok, so on further analysis it looks like this is the designed behaviour: > that when running tox, all the python versions and local dependencies > would live on my laptop directly rather than in a docker container. Correct. > If so, I'm not a fan. I am not primarily a web, python, or database > developer and I like having all of that stuff live in an isolated docker > container. I especially like that it's also consistent for everyone who > wants to hack on patchwork - that they can run the full suite of tests > across all the supported versions with nothing more than docker and > docker-compose. tox-docker provides, afaict, no way to do this. (Also, > less universally, I run Ubuntu, not Fedora and getting multiple python > versions is a pain, as you can see from the dockerfiles.) > > What's the main problem that you're trying to solve here? Is it that you > have to type 'docker-compose run web --tox -e py36-django21' rather than > just 'tox -e py36-django21'? Personally, I'm finding I'm having to jump through a lot of hoops to get docker working as I expect, including things like needing to create the '.env' file so it uses the right UID, and it also takes _forever_ to rebuild (which I need to do rather frequently as I tinker with dependencies). Finally, it doesn't work like I'd expect a Python thing to usually work, meaning whenever I go to tinker with Patchwork after a break, I need to re-learn how to test things. Given that I have an environment that already has most of the dependencies needed to run this, I'm not really getting any of the benefits docker provides but am seeing most of the costs. How about we don't strip the 'web' Dockerfile to the bones and instead add this as an alternate approach to running tests? I get faster tests and you still get full isolation. Alternatively, we can switch to pure Python DB drivers (removing the need for non-Python dependencies) in our development requirements.txt and use pyenv to provide multiple Python versions? That avoids having two ways to do the same thing but does still mean you need a little work on your end, which might not be desirable. Stephen > Regards, > Daniel
Stephen Finucane <stephen@that.guru> writes: > On Thu, 2019-08-22 at 23:52 +1000, Daniel Axtens wrote: >> > > .. code-block:: shell >> > > >> > > - $ tox -e py27-django18 >> > > + $ tox -e py36-django21-mysql >> > >> > So I'm trying this out (finally!) and it seems to want to install all >> > the dependencies locally before starting a container. I don't have the >> > mysql bits installed, so it fails looking for `mysql_config`. Is this >> > supposed to happen or am I Doing It Wrong? >> > >> >> Ok, so on further analysis it looks like this is the designed behaviour: >> that when running tox, all the python versions and local dependencies >> would live on my laptop directly rather than in a docker container. > > Correct. > >> If so, I'm not a fan. I am not primarily a web, python, or database >> developer and I like having all of that stuff live in an isolated docker >> container. I especially like that it's also consistent for everyone who >> wants to hack on patchwork - that they can run the full suite of tests >> across all the supported versions with nothing more than docker and >> docker-compose. tox-docker provides, afaict, no way to do this. (Also, >> less universally, I run Ubuntu, not Fedora and getting multiple python >> versions is a pain, as you can see from the dockerfiles.) >> >> What's the main problem that you're trying to solve here? Is it that you >> have to type 'docker-compose run web --tox -e py36-django21' rather than >> just 'tox -e py36-django21'? > > Personally, I'm finding I'm having to jump through a lot of hoops to > get docker working as I expect, including things like needing to create > the '.env' file so it uses the right UID, and it also takes _forever_ > to rebuild (which I need to do rather frequently as I tinker with > dependencies). Finally, it doesn't work like I'd expect a Python thing > to usually work, meaning whenever I go to tinker with Patchwork after a > break, I need to re-learn how to test things. Given that I have an > environment that already has most of the dependencies needed to run > this, I'm not really getting any of the benefits docker provides but am > seeing most of the costs. Fair enough. How does this sound: We copy the original tox.ini into tools/docker/ We make the main tox file the tox file you suggested, so that on your laptop you can run `tox -e whatever` and things will go well. In entry-point.sh, we intercept `--tox` and use the saved tox.ini file for inside docker (tox -C tools/docker/tox.ini $@) We have to keep the two files in sync, but then we have both systems working as expected, and we can clarify in documentation when to use each of them. If you don't like that I'm happy to explore the pure python driver plus pyenv approach. Regards, Daniel > > How about we don't strip the 'web' Dockerfile to the bones and instead > add this as an alternate approach to running tests? I get faster tests > and you still get full isolation. Alternatively, we can switch to pure > Python DB drivers (removing the need for non-Python dependencies) in > our development requirements.txt and use pyenv to provide multiple > Python versions? That avoids having two ways to do the same thing but > does still mean you need a little work on your end, which might not be > desirable. > > Stephen > >> Regards, >> Daniel
On Mon, 2019-08-26 at 14:25 +1000, Daniel Axtens wrote: > Stephen Finucane <stephen@that.guru> writes: > > > On Thu, 2019-08-22 at 23:52 +1000, Daniel Axtens wrote: > > > > > .. code-block:: shell > > > > > > > > > > - $ tox -e py27-django18 > > > > > + $ tox -e py36-django21-mysql > > > > > > > > So I'm trying this out (finally!) and it seems to want to install all > > > > the dependencies locally before starting a container. I don't have the > > > > mysql bits installed, so it fails looking for `mysql_config`. Is this > > > > supposed to happen or am I Doing It Wrong? > > > > > > > > > > Ok, so on further analysis it looks like this is the designed behaviour: > > > that when running tox, all the python versions and local dependencies > > > would live on my laptop directly rather than in a docker container. > > > > Correct. > > > > > If so, I'm not a fan. I am not primarily a web, python, or database > > > developer and I like having all of that stuff live in an isolated docker > > > container. I especially like that it's also consistent for everyone who > > > wants to hack on patchwork - that they can run the full suite of tests > > > across all the supported versions with nothing more than docker and > > > docker-compose. tox-docker provides, afaict, no way to do this. (Also, > > > less universally, I run Ubuntu, not Fedora and getting multiple python > > > versions is a pain, as you can see from the dockerfiles.) > > > > > > What's the main problem that you're trying to solve here? Is it that you > > > have to type 'docker-compose run web --tox -e py36-django21' rather than > > > just 'tox -e py36-django21'? > > > > Personally, I'm finding I'm having to jump through a lot of hoops to > > get docker working as I expect, including things like needing to create > > the '.env' file so it uses the right UID, and it also takes _forever_ > > to rebuild (which I need to do rather frequently as I tinker with > > dependencies). Finally, it doesn't work like I'd expect a Python thing > > to usually work, meaning whenever I go to tinker with Patchwork after a > > break, I need to re-learn how to test things. Given that I have an > > environment that already has most of the dependencies needed to run > > this, I'm not really getting any of the benefits docker provides but am > > seeing most of the costs. > > Fair enough. How does this sound: > > We copy the original tox.ini into tools/docker/ > > We make the main tox file the tox file you suggested, so that on your > laptop you can run `tox -e whatever` and things will go well. > > In entry-point.sh, we intercept `--tox` and use the saved tox.ini file > for inside docker (tox -C tools/docker/tox.ini $@) > > We have to keep the two files in sync, but then we have both systems > working as expected, and we can clarify in documentation when to use > each of them. I don't think any of that is necessary. As things stand, running e.g. 'tox -e py27-django111' will continue to have the same behavior as previously. The only change will happen if you introduce an additional factor and call something like 'tox -e py27-django111-mysql'. That's not included in any of the default environmnents by default so I don't think it's an issue. Stephen > If you don't like that I'm happy to explore the pure python driver plus > pyenv approach. > > Regards, > Daniel > > > How about we don't strip the 'web' Dockerfile to the bones and instead > > add this as an alternate approach to running tests? I get faster tests > > and you still get full isolation. Alternatively, we can switch to pure > > Python DB drivers (removing the need for non-Python dependencies) in > > our development requirements.txt and use pyenv to provide multiple > > Python versions? That avoids having two ways to do the same thing but > > does still mean you need a little work on your end, which might not be > > desirable. > > > > Stephen > > > > > Regards, > > > Daniel
Stephen Finucane <stephen@that.guru> writes: > On Mon, 2019-08-26 at 14:25 +1000, Daniel Axtens wrote: >> Stephen Finucane <stephen@that.guru> writes: >> >> > On Thu, 2019-08-22 at 23:52 +1000, Daniel Axtens wrote: >> > > > > .. code-block:: shell >> > > > > >> > > > > - $ tox -e py27-django18 >> > > > > + $ tox -e py36-django21-mysql >> > > > >> > > > So I'm trying this out (finally!) and it seems to want to install all >> > > > the dependencies locally before starting a container. I don't have the >> > > > mysql bits installed, so it fails looking for `mysql_config`. Is this >> > > > supposed to happen or am I Doing It Wrong? >> > > > >> > > >> > > Ok, so on further analysis it looks like this is the designed behaviour: >> > > that when running tox, all the python versions and local dependencies >> > > would live on my laptop directly rather than in a docker container. >> > >> > Correct. >> > >> > > If so, I'm not a fan. I am not primarily a web, python, or database >> > > developer and I like having all of that stuff live in an isolated docker >> > > container. I especially like that it's also consistent for everyone who >> > > wants to hack on patchwork - that they can run the full suite of tests >> > > across all the supported versions with nothing more than docker and >> > > docker-compose. tox-docker provides, afaict, no way to do this. (Also, >> > > less universally, I run Ubuntu, not Fedora and getting multiple python >> > > versions is a pain, as you can see from the dockerfiles.) >> > > >> > > What's the main problem that you're trying to solve here? Is it that you >> > > have to type 'docker-compose run web --tox -e py36-django21' rather than >> > > just 'tox -e py36-django21'? >> > >> > Personally, I'm finding I'm having to jump through a lot of hoops to >> > get docker working as I expect, including things like needing to create >> > the '.env' file so it uses the right UID, and it also takes _forever_ >> > to rebuild (which I need to do rather frequently as I tinker with >> > dependencies). Finally, it doesn't work like I'd expect a Python thing >> > to usually work, meaning whenever I go to tinker with Patchwork after a >> > break, I need to re-learn how to test things. Given that I have an >> > environment that already has most of the dependencies needed to run >> > this, I'm not really getting any of the benefits docker provides but am >> > seeing most of the costs. >> >> Fair enough. How does this sound: >> >> We copy the original tox.ini into tools/docker/ >> >> We make the main tox file the tox file you suggested, so that on your >> laptop you can run `tox -e whatever` and things will go well. >> >> In entry-point.sh, we intercept `--tox` and use the saved tox.ini file >> for inside docker (tox -C tools/docker/tox.ini $@) >> >> We have to keep the two files in sync, but then we have both systems >> working as expected, and we can clarify in documentation when to use >> each of them. > > I don't think any of that is necessary. As things stand, running e.g. > 'tox -e py27-django111' will continue to have the same behavior as > previously. The only change will happen if you introduce an additional > factor and call something like 'tox -e py27-django111-mysql'. That's > not included in any of the default environmnents by default so I don't > think it's an issue. I thought you were changing the default environment list - there's a change to envlist in tox.ini: > [tox] > minversion = 2.0 >-envlist = pep8,docs,py{27,34}-django111,py{35,36}-django{111,20,21,22} >+envlist = pep8,docs,py{27,34}-django111-{mysql,postgres},py{35,36}-django{111,20,21,22}-{mysql,postgres} >+skip_missing_interpreters = True > skipsdist = True I'd really like to keep `docker-compose run web --tox` working, but otherwise I'm happy to make whatever changes make things easy for you. Perhaps we could specify default envlist using the TOXENV environment variable in entrypoint.sh? Then you could change the default tox.ini environments with no issues... Regards, Daniel > > Stephen > >> If you don't like that I'm happy to explore the pure python driver plus >> pyenv approach. >> >> Regards, >> Daniel >> >> > How about we don't strip the 'web' Dockerfile to the bones and instead >> > add this as an alternate approach to running tests? I get faster tests >> > and you still get full isolation. Alternatively, we can switch to pure >> > Python DB drivers (removing the need for non-Python dependencies) in >> > our development requirements.txt and use pyenv to provide multiple >> > Python versions? That avoids having two ways to do the same thing but >> > does still mean you need a little work on your end, which might not be >> > desirable. >> > >> > Stephen >> > >> > > Regards, >> > > Daniel
On Mon, 2019-09-02 at 20:52 +1000, Daniel Axtens wrote: > Stephen Finucane <stephen@that.guru> writes: > > > On Mon, 2019-08-26 at 14:25 +1000, Daniel Axtens wrote: > > > Stephen Finucane <stephen@that.guru> writes: > > > > > > > On Thu, 2019-08-22 at 23:52 +1000, Daniel Axtens wrote: > > > > > > > .. code-block:: shell > > > > > > > > > > > > > > - $ tox -e py27-django18 > > > > > > > + $ tox -e py36-django21-mysql > > > > > > > > > > > > So I'm trying this out (finally!) and it seems to want to install all > > > > > > the dependencies locally before starting a container. I don't have the > > > > > > mysql bits installed, so it fails looking for `mysql_config`. Is this > > > > > > supposed to happen or am I Doing It Wrong? > > > > > > > > > > > > > > > > Ok, so on further analysis it looks like this is the designed behaviour: > > > > > that when running tox, all the python versions and local dependencies > > > > > would live on my laptop directly rather than in a docker container. > > > > > > > > Correct. > > > > > > > > > If so, I'm not a fan. I am not primarily a web, python, or database > > > > > developer and I like having all of that stuff live in an isolated docker > > > > > container. I especially like that it's also consistent for everyone who > > > > > wants to hack on patchwork - that they can run the full suite of tests > > > > > across all the supported versions with nothing more than docker and > > > > > docker-compose. tox-docker provides, afaict, no way to do this. (Also, > > > > > less universally, I run Ubuntu, not Fedora and getting multiple python > > > > > versions is a pain, as you can see from the dockerfiles.) > > > > > > > > > > What's the main problem that you're trying to solve here? Is it that you > > > > > have to type 'docker-compose run web --tox -e py36-django21' rather than > > > > > just 'tox -e py36-django21'? > > > > > > > > Personally, I'm finding I'm having to jump through a lot of hoops to > > > > get docker working as I expect, including things like needing to create > > > > the '.env' file so it uses the right UID, and it also takes _forever_ > > > > to rebuild (which I need to do rather frequently as I tinker with > > > > dependencies). Finally, it doesn't work like I'd expect a Python thing > > > > to usually work, meaning whenever I go to tinker with Patchwork after a > > > > break, I need to re-learn how to test things. Given that I have an > > > > environment that already has most of the dependencies needed to run > > > > this, I'm not really getting any of the benefits docker provides but am > > > > seeing most of the costs. > > > > > > Fair enough. How does this sound: > > > > > > We copy the original tox.ini into tools/docker/ > > > > > > We make the main tox file the tox file you suggested, so that on your > > > laptop you can run `tox -e whatever` and things will go well. > > > > > > In entry-point.sh, we intercept `--tox` and use the saved tox.ini file > > > for inside docker (tox -C tools/docker/tox.ini $@) > > > > > > We have to keep the two files in sync, but then we have both systems > > > working as expected, and we can clarify in documentation when to use > > > each of them. > > > > I don't think any of that is necessary. As things stand, running e.g. > > 'tox -e py27-django111' will continue to have the same behavior as > > previously. The only change will happen if you introduce an additional > > factor and call something like 'tox -e py27-django111-mysql'. That's > > not included in any of the default environmnents by default so I don't > > think it's an issue. > > I thought you were changing the default environment list - there's > a change to envlist in tox.ini: Ah damn it. I forgot to send the new version that drops this. Let me do just that. Sorry! > > [tox] > > minversion = 2.0 > > -envlist = pep8,docs,py{27,34}-django111,py{35,36}-django{111,20,21,22} > > +envlist = pep8,docs,py{27,34}-django111-{mysql,postgres},py{35,36}-django{111,20,21,22}-{mysql,postgres} > > +skip_missing_interpreters = True > > skipsdist = True > > I'd really like to keep `docker-compose run web --tox` working, but > otherwise I'm happy to make whatever changes make things easy for > you. Perhaps we could specify default envlist using the TOXENV > environment variable in entrypoint.sh? Then you could change the default > tox.ini environments with no issues... Hopefully this shouldn't be necessary, per above. Stephen > Regards, > Daniel > > > Stephen > > > > > If you don't like that I'm happy to explore the pure python driver plus > > > pyenv approach. > > > > > > Regards, > > > Daniel > > > > > > > How about we don't strip the 'web' Dockerfile to the bones and instead > > > > add this as an alternate approach to running tests? I get faster tests > > > > and you still get full isolation. Alternatively, we can switch to pure > > > > Python DB drivers (removing the need for non-Python dependencies) in > > > > our development requirements.txt and use pyenv to provide multiple > > > > Python versions? That avoids having two ways to do the same thing but > > > > does still mean you need a little work on your end, which might not be > > > > desirable. > > > > > > > > Stephen > > > > > > > > > Regards, > > > > > Daniel
diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst index 5089bba8..f713f872 100644 --- a/docs/development/contributing.rst +++ b/docs/development/contributing.rst @@ -31,29 +31,15 @@ Testing ------- Patchwork includes a `tox`_ script to automate testing. This requires a -functional database and some Python requirements like *tox*. Refer to -:doc:`installation` for information on how to configure these. - -You may also need to install *tox*. If so, do this now: +functional database and some Python requirements like *tox*. These can be +installed using :command:`pip`: .. code-block:: shell - $ pip install --user tox - -.. tip:: - - If you're using Docker, you may not need to install *tox* - locally. Instead, it will already be installed inside the - container. For Docker, you can run *tox* like so: - - .. code-block:: shell - - $ docker-compose run --rm web tox [ARGS...] - - For more information, refer to :ref:`installation-docker`. + $ pip install --user tox tox-docker -Assuming these requirements are met, actually testing Patchwork is quite easy -to do. To start, you can show the default targets like so: +Once installed, actually testing Patchwork is quite easy to do. To start, you +can show the default targets like so: .. code-block:: shell @@ -66,7 +52,7 @@ parameter: .. code-block:: shell - $ tox -e py27-django18 + $ tox -e py36-django21-mysql In the case of the unit tests targets, you can also run specific tests by passing the fully qualified test name as an additional argument to this @@ -74,7 +60,7 @@ command: .. code-block:: shell - $ tox -e py27-django18 patchwork.tests.SubjectCleanUpTest + $ tox -e py36-django21-mysql patchwork.tests.SubjectCleanUpTest Because Patchwork support multiple versions of Django, it's very important that you test against all supported versions. When run without argument, tox will do diff --git a/docs/development/installation.rst b/docs/development/installation.rst index 0ab755f4..433c3a41 100644 --- a/docs/development/installation.rst +++ b/docs/development/installation.rst @@ -86,12 +86,6 @@ To run unit tests against the system Python packages, run: $ docker-compose run --rm web python manage.py test -To run unit tests for multiple versions using ``tox``, run: - -.. code-block:: shell - - $ docker-compose run --rm web tox - To reset the database before any of these commands, add ``--reset`` to the command line after ``web`` and before any other arguments: diff --git a/patchwork/settings/dev.py b/patchwork/settings/dev.py index 53fa58f6..cfb9256c 100644 --- a/patchwork/settings/dev.py +++ b/patchwork/settings/dev.py @@ -19,6 +19,18 @@ try: except ImportError: debug_toolbar = None +# +# tox-docker settings +# + +if 'POSTGRES_5432_TCP' in os.environ: + os.environ['PW_TEST_DB_HOST'] = os.environ['POSTGRES_HOST'] + os.environ['PW_TEST_DB_PORT'] = os.environ['POSTGRES_5432_TCP'] +elif 'MYSQL_3306_TCP' in os.environ: + os.environ['PW_TEST_DB_HOST'] = os.environ['MYSQL_HOST'] + os.environ['PW_TEST_DB_PORT'] = os.environ['MYSQL_3306_TCP'] + + # # Core settings # https://docs.djangoproject.com/en/1.11/ref/settings/#core-settings diff --git a/tox.ini b/tox.ini index d4c34e1c..3c71dafd 100644 --- a/tox.ini +++ b/tox.ini @@ -1,6 +1,7 @@ [tox] minversion = 2.0 -envlist = pep8,docs,py{27,34}-django111,py{35,36}-django{111,20,21,22} +envlist = pep8,docs,py{27,34}-django111-{mysql,postgres},py{35,36}-django{111,20,21,22}-{mysql,postgres} +skip_missing_interpreters = True skipsdist = True [testenv] @@ -17,6 +18,9 @@ deps = django22: django>=2.2,<2.3 django22: djangorestframework>=3.9.2 django22: django-filter>=2.1,<3.0 +docker = + postgres: postgres:9.6 + mysql: mysql:8.0 setenv = DJANGO_SETTINGS_MODULE = patchwork.settings.dev PYTHONDONTWRITEBYTECODE = 1 @@ -24,15 +28,25 @@ setenv = py27: PYTHONWARNINGS = once py{34,36}:PYTHONWARNINGS = once,ignore::ImportWarning:backports py35:PYTHONWARNINGS = once,ignore::ResourceWarning:unittest.suite,ignore::ImportWarning:backports + postgres: PW_TEST_DB_TYPE = postgres + postgres: PW_TEST_DB_USER = postgres + postgres: PW_TEST_DB_PASS = password + mysql: PW_TEST_DB_USER = root + mysql: PW_TEST_DB_PASS = password passenv = http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY PW_TEST_DB_TYPE PW_TEST_DB_USER PW_TEST_DB_PASS PW_TEST_DB_HOST PW_TEST_DB_PORT +dockerenv = + postgres: POSTGRES_PASSWORD=password + mysql: MYSQL_ROOT_PASSWORD=password commands = - python {toxinidir}/manage.py test --noinput '{posargs:patchwork}' + python {toxinidir}/manage.py test --noinput {posargs:patchwork} [testenv:bashate] deps = bashate +docker = +dockerenv = whitelist_externals = bash commands = bash -c "find {toxinidir} \ @@ -43,6 +57,8 @@ commands = [testenv:pep8] basepython = python2.7 deps = flake8 +docker = +dockerenv = commands = flake8 {posargs} patchwork patchwork/bin/pwclient [flake8] @@ -57,6 +73,8 @@ exclude = ./patchwork/migrations [testenv:docs] deps = -r{toxinidir}/docs/requirements.txt +docker = +dockerenv = commands = sphinx-build -E -W -b dirhtml -d docs/_build/doctrees docs docs/_build/html @@ -67,9 +85,6 @@ deps = -r{toxinidir}/requirements-prod.txt commands = pylint patchwork --rcfile=pylint.rc -[testenv:venv] -commands = {posargs} - [testenv:coverage] basepython = python2.7 deps =
This eliminates the need to use docker-compose for most use cases. Instead, we can now do: tox -e py27-django111-postgres If you're using a locally configured PostgreSQL or MySQL instance, you simply omit the last factor and things behave as before: tox -e py27-django111 We removed the 'venv' environment, since it was never actually that useful and is even less so now (you'd need to have a local DB set up) and add the 'skip_missing_interpreters' flag for folks on an OS that doesn't provide all the Python versions under the sun like Fedora does. Signed-off-by: Stephen Finucane <stephen@that.guru> Cc: Daniel Axtens <dja@axtens.net> --- v3: - Bump the mysql container version to 8.0 to work around an issue with tox-docker (https://github.com/tox-dev/tox-docker/issues/35) - Use the 'root' user for the mysql container since it simplifies things greatly (we do this for Travis already) - Revert changes to requirements.txt which turned out to be unnecessary and broke Travis - Remove the venv environment instead of attempting to hack it into working --- docs/development/contributing.rst | 28 +++++++--------------------- docs/development/installation.rst | 6 ------ patchwork/settings/dev.py | 12 ++++++++++++ tox.ini | 25 ++++++++++++++++++++----- 4 files changed, 39 insertions(+), 32 deletions(-)