| Message ID | 20201004111550.236227-1-stephen@that.guru |
|---|---|
| State | Accepted |
| Headers | show |
| Series | Add parallel testing | expand |
On Sun, 2020-10-04 at 12:15 +0100, Stephen Finucane wrote: > This saves us a good deal of time, even allowing for process startup and > general overhead. Before: > > real 6m39.103s > user 1m9.561s > sys 0m2.705s > > After: > > real 4m5.267s > user 1m24.527s > sys 0m3.828s > > This requires N databases, where N is the N of processors on the host. > We need to update the docker configuration for this. > > Signed-off-by: Stephen Finucane <stephen@that.guru> Applied.
Hi Stephen, > diff --git tools/docker/entrypoint.sh tools/docker/entrypoint.sh > index 6314f1b6..8f7ea4f7 100755 > --- tools/docker/entrypoint.sh > +++ tools/docker/entrypoint.sh > @@ -26,7 +26,7 @@ reset_data_mysql() { > DROP DATABASE IF EXISTS patchwork; > CREATE DATABASE patchwork CHARACTER SET utf8; > GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; > -GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%'; > +GRANT ALL ON 'test_patchwork\_%'.* to 'patchwork'@'%'; This breaks for me on the latest git. docker-compose run web --reset prints: ERROR 1064 (42000) at line 4: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''test_patchwork\_%'.* to 'patchwork'@'%'' at line 1 I don't think you can do a grant on a wildcard. Certainly I haven't been able to get it to work - this "big hammer": GRANT ALL ON test_patchwork*.* to 'patchwork'@'%'; fails with a more meaningful error: ERROR 1046 (3D000) at line 4: No database selected I am thinking of doing a little scriptlet that just iterates on ncpus but if you were able to get this to work on your end (or if anyone knows how to actually do what we're going for) I'd be keen to know. Kind regards, Daniel > FLUSH PRIVILEGES; > EOF > } > diff --git tox.ini tox.ini > index 0849b059..28f29f6d 100644 > --- tox.ini > +++ tox.ini > @@ -28,7 +28,7 @@ passenv = > PW_TEST_DB_TYPE PW_TEST_DB_USER PW_TEST_DB_PASS PW_TEST_DB_HOST > PW_TEST_DB_PORT > commands = > - python {toxinidir}/manage.py test --noinput {posargs:patchwork} > + python {toxinidir}/manage.py test --noinput --parallel -- {posargs:patchwork} > > [testenv:bashate] > deps = bashate > -- > 2.25.4 > > _______________________________________________ > Patchwork mailing list > Patchwork@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork
> I am thinking of doing a little scriptlet that just iterates on ncpus > but if you were able to get this to work on your end (or if anyone knows > how to actually do what we're going for) I'd be keen to know. Ah so interestingly the MySQL docs seem to suggest this does work, if you use backticks instead of single quotes: https://dev.mysql.com/doc/refman/8.0/en/grant.html#grant-accounts-passwords I'll give that a go and if it works I'll send a fixup tomorrow and apply it. Kind regards, Daniel > > Kind regards, > Daniel > > >> FLUSH PRIVILEGES; >> EOF >> } >> diff --git tox.ini tox.ini >> index 0849b059..28f29f6d 100644 >> --- tox.ini >> +++ tox.ini >> @@ -28,7 +28,7 @@ passenv = >> PW_TEST_DB_TYPE PW_TEST_DB_USER PW_TEST_DB_PASS PW_TEST_DB_HOST >> PW_TEST_DB_PORT >> commands = >> - python {toxinidir}/manage.py test --noinput {posargs:patchwork} >> + python {toxinidir}/manage.py test --noinput --parallel -- {posargs:patchwork} >> >> [testenv:bashate] >> deps = bashate >> -- >> 2.25.4 >> >> _______________________________________________ >> Patchwork mailing list >> Patchwork@lists.ozlabs.org >> https://lists.ozlabs.org/listinfo/patchwork
diff --git tools/docker/entrypoint.sh tools/docker/entrypoint.sh index 6314f1b6..8f7ea4f7 100755 --- tools/docker/entrypoint.sh +++ tools/docker/entrypoint.sh @@ -26,7 +26,7 @@ reset_data_mysql() { DROP DATABASE IF EXISTS patchwork; CREATE DATABASE patchwork CHARACTER SET utf8; GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%'; +GRANT ALL ON 'test_patchwork\_%'.* to 'patchwork'@'%'; FLUSH PRIVILEGES; EOF } diff --git tox.ini tox.ini index 0849b059..28f29f6d 100644 --- tox.ini +++ tox.ini @@ -28,7 +28,7 @@ passenv = PW_TEST_DB_TYPE PW_TEST_DB_USER PW_TEST_DB_PASS PW_TEST_DB_HOST PW_TEST_DB_PORT commands = - python {toxinidir}/manage.py test --noinput {posargs:patchwork} + python {toxinidir}/manage.py test --noinput --parallel -- {posargs:patchwork} [testenv:bashate] deps = bashate
This saves us a good deal of time, even allowing for process startup and general overhead. Before: real 6m39.103s user 1m9.561s sys 0m2.705s After: real 4m5.267s user 1m24.527s sys 0m3.828s This requires N databases, where N is the N of processors on the host. We need to update the docker configuration for this. Signed-off-by: Stephen Finucane <stephen@that.guru> --- tools/docker/entrypoint.sh | 2 +- tox.ini | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)