[00/13] Start generating API examples from tests
mbox series

Message ID 20181030113153.7855-1-stephen@that.guru
Headers show
Series
  • Start generating API examples from tests
Related show

Message

Stephen Finucane Oct. 30, 2018, 11:31 a.m. UTC
As part of the OpenAPI work, I realized I wanted to be able to show
actual API requests and responses for the documentation. We could
generate these manually but doing so would be tedious and would need to
be redone every time the API changed. There is a better way: generate
these things from the tests. We know that the requests and responses
must be correct because we're actually testing against them. It also
provides an opportunity to examine our test coverage and fill in gaps,
of which there are a few. All of these samples are currently unused but
they will be used once we merge in the OpenAPI work.

Note that some of the changes are bigger than we'd like. That's because
some tests were actually testing multiple things. This reduced the LoC
necessary but meant we didn't have a single request/response to save.

Stephen Finucane (13):
  tests: Add 'store_samples' decorator
  tests: Add 'store_samples' decorator to 'test_bundle'
  tests: Add 'store_samples' decorator to 'test_project'
  tests: Add 'store_samples' decorator to 'test_people'
  tests: Add 'store_samples' decorator to 'test_user'
  tests: Add 'store_samples' decorator to 'test_patch'
  tests: Add 'store_samples' decorator to 'test_cover'
  tests: Add 'store_samples' decorator to 'test_series'
  tests: Add 'store_samples' decorator to 'test_comment'
  tests: Add 'store_samples' decorator to 'test_check'
  tests: Add tests for '/events' resource
  tests: Add 'store_samples' decorator to 'test_event'
  signals: Fix 'series-completed' event

 .gitignore                          |   1 +
 patchwork/signals.py                |  21 +++-
 patchwork/tests/api/test_bundle.py  |  76 +++++++++++--
 patchwork/tests/api/test_check.py   |  54 ++++++---
 patchwork/tests/api/test_comment.py |  57 ++++++----
 patchwork/tests/api/test_cover.py   |  51 ++++++---
 patchwork/tests/api/test_event.py   | 165 ++++++++++++++++++++++++++++
 patchwork/tests/api/test_patch.py   | 145 ++++++++++++++++--------
 patchwork/tests/api/test_person.py  |  37 +++++--
 patchwork/tests/api/test_project.py | 110 ++++++++++++++-----
 patchwork/tests/api/test_series.py  |  65 +++++++----
 patchwork/tests/api/test_user.py    |  65 +++++++++--
 patchwork/tests/api/utils.py        |  93 ++++++++++++++++
 patchwork/tests/test_events.py      |  33 +++++-
 patchwork/tests/utils.py            |  10 ++
 15 files changed, 808 insertions(+), 175 deletions(-)
 create mode 100644 patchwork/tests/api/test_event.py
 create mode 100644 patchwork/tests/api/utils.py

Comments

Daniel Axtens Nov. 11, 2018, 1 a.m. UTC | #1
Hi Stephen,

> As part of the OpenAPI work, I realized I wanted to be able to show
> actual API requests and responses for the documentation. We could
> generate these manually but doing so would be tedious and would need to
> be redone every time the API changed. There is a better way: generate
> these things from the tests. We know that the requests and responses
> must be correct because we're actually testing against them. It also
> provides an opportunity to examine our test coverage and fill in gaps,
> of which there are a few. All of these samples are currently unused but
> they will be used once we merge in the OpenAPI work.
>
> Note that some of the changes are bigger than we'd like. That's because
> some tests were actually testing multiple things. This reduced the LoC
> necessary but meant we didn't have a single request/response to save.

(Sorry, I would test this myself, but what with me being in an airport
and this potentially interacting with other series, I figure it's best
to just ask this time...)

What's the impact on the time to run tests with this? I notice it splits
up a lot of tests; does that make the already painfully slow test suite
any slower?

Regards,
Daniel

>
> Stephen Finucane (13):
>   tests: Add 'store_samples' decorator
>   tests: Add 'store_samples' decorator to 'test_bundle'
>   tests: Add 'store_samples' decorator to 'test_project'
>   tests: Add 'store_samples' decorator to 'test_people'
>   tests: Add 'store_samples' decorator to 'test_user'
>   tests: Add 'store_samples' decorator to 'test_patch'
>   tests: Add 'store_samples' decorator to 'test_cover'
>   tests: Add 'store_samples' decorator to 'test_series'
>   tests: Add 'store_samples' decorator to 'test_comment'
>   tests: Add 'store_samples' decorator to 'test_check'
>   tests: Add tests for '/events' resource
>   tests: Add 'store_samples' decorator to 'test_event'
>   signals: Fix 'series-completed' event
>
>  .gitignore                          |   1 +
>  patchwork/signals.py                |  21 +++-
>  patchwork/tests/api/test_bundle.py  |  76 +++++++++++--
>  patchwork/tests/api/test_check.py   |  54 ++++++---
>  patchwork/tests/api/test_comment.py |  57 ++++++----
>  patchwork/tests/api/test_cover.py   |  51 ++++++---
>  patchwork/tests/api/test_event.py   | 165 ++++++++++++++++++++++++++++
>  patchwork/tests/api/test_patch.py   | 145 ++++++++++++++++--------
>  patchwork/tests/api/test_person.py  |  37 +++++--
>  patchwork/tests/api/test_project.py | 110 ++++++++++++++-----
>  patchwork/tests/api/test_series.py  |  65 +++++++----
>  patchwork/tests/api/test_user.py    |  65 +++++++++--
>  patchwork/tests/api/utils.py        |  93 ++++++++++++++++
>  patchwork/tests/test_events.py      |  33 +++++-
>  patchwork/tests/utils.py            |  10 ++
>  15 files changed, 808 insertions(+), 175 deletions(-)
>  create mode 100644 patchwork/tests/api/test_event.py
>  create mode 100644 patchwork/tests/api/utils.py
>
> -- 
> 2.17.2
>
> _______________________________________________
> Patchwork mailing list
> Patchwork@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork
Stephen Finucane Nov. 12, 2018, 2:52 p.m. UTC | #2
On Sun, 2018-11-11 at 12:00 +1100, Daniel Axtens wrote:
> Hi Stephen,
> 
> > As part of the OpenAPI work, I realized I wanted to be able to show
> > actual API requests and responses for the documentation. We could
> > generate these manually but doing so would be tedious and would need to
> > be redone every time the API changed. There is a better way: generate
> > these things from the tests. We know that the requests and responses
> > must be correct because we're actually testing against them. It also
> > provides an opportunity to examine our test coverage and fill in gaps,
> > of which there are a few. All of these samples are currently unused but
> > they will be used once we merge in the OpenAPI work.
> > 
> > Note that some of the changes are bigger than we'd like. That's because
> > some tests were actually testing multiple things. This reduced the LoC
> > necessary but meant we didn't have a single request/response to save.
> 
> (Sorry, I would test this myself, but what with me being in an airport
> and this potentially interacting with other series, I figure it's best
> to just ask this time...)
> 
> What's the impact on the time to run tests with this? I notice it splits
> up a lot of tests; does that make the already painfully slow test suite
> any slower?

There is an impact but it's very slight.

Before:

   $ docker-compose run --rm web tox -e py27-django111 patchwork.tests.api
   ............................................
   ----------------------------------------------------------------------
   Ran 44 tests in 2.604s

   OK

After:

   $ docker-compose run --rm web tox -e py27-django111 patchwork.tests.api
   .................................................................................................
   ----------------------------------------------------------------------
   Ran 97 tests in 3.980s

   OK

Personally, I think this is acceptable as the samples are helpful and
the tests probably should have been written in this more modular
fashion the first day. As a general rule of thumb too, the REST API
tests are very fast, once you exclude the time taken to set up the
testing database (which is constant and slooooow). The slowest tests
are the ones for views, which require a lot more of Django's machinery.
We should probably try to optimize those.

Stephen

> Regards,
> Daniel
> 
> > Stephen Finucane (13):
> >   tests: Add 'store_samples' decorator
> >   tests: Add 'store_samples' decorator to 'test_bundle'
> >   tests: Add 'store_samples' decorator to 'test_project'
> >   tests: Add 'store_samples' decorator to 'test_people'
> >   tests: Add 'store_samples' decorator to 'test_user'
> >   tests: Add 'store_samples' decorator to 'test_patch'
> >   tests: Add 'store_samples' decorator to 'test_cover'
> >   tests: Add 'store_samples' decorator to 'test_series'
> >   tests: Add 'store_samples' decorator to 'test_comment'
> >   tests: Add 'store_samples' decorator to 'test_check'
> >   tests: Add tests for '/events' resource
> >   tests: Add 'store_samples' decorator to 'test_event'
> >   signals: Fix 'series-completed' event
> > 
> >  .gitignore                          |   1 +
> >  patchwork/signals.py                |  21 +++-
> >  patchwork/tests/api/test_bundle.py  |  76 +++++++++++--
> >  patchwork/tests/api/test_check.py   |  54 ++++++---
> >  patchwork/tests/api/test_comment.py |  57 ++++++----
> >  patchwork/tests/api/test_cover.py   |  51 ++++++---
> >  patchwork/tests/api/test_event.py   | 165 ++++++++++++++++++++++++++++
> >  patchwork/tests/api/test_patch.py   | 145 ++++++++++++++++--------
> >  patchwork/tests/api/test_person.py  |  37 +++++--
> >  patchwork/tests/api/test_project.py | 110 ++++++++++++++-----
> >  patchwork/tests/api/test_series.py  |  65 +++++++----
> >  patchwork/tests/api/test_user.py    |  65 +++++++++--
> >  patchwork/tests/api/utils.py        |  93 ++++++++++++++++
> >  patchwork/tests/test_events.py      |  33 +++++-
> >  patchwork/tests/utils.py            |  10 ++
> >  15 files changed, 808 insertions(+), 175 deletions(-)
> >  create mode 100644 patchwork/tests/api/test_event.py
> >  create mode 100644 patchwork/tests/api/utils.py
> > 
> > -- 
> > 2.17.2
> > 
> > _______________________________________________
> > Patchwork mailing list
> > Patchwork@lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/patchwork