mbox series

[v2,0/3] Use aiohttp in pkg-stats

Message ID 20200804195248.1238754-1-thomas.petazzoni@bootlin.com
Headers show
Series Use aiohttp in pkg-stats | expand

Message

Thomas Petazzoni Aug. 4, 2020, 7:52 p.m. UTC 
Hello,

I started investigating why pkg-stats was sometimes stuck on the
server running it on a daily basis to populate
autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The
subprocesses started by "multiprocessing" to retrieve the latest
upstream version from release-monitoring.org were stuck holding a
lock. Without providing a definitive conclusion, some preliminary
research showed that multiprocessing can be tricky and cause some
issues with locks.

Discussing this with Titouan, he suggested to use aiohttp instead of
multiprocessing. And indeed, it makes a lot of sense to use this
popular asynchronous HTTP library.

This patch series switches the latest version retrieval and the
upstream URL checking to aiohttp, and as a bonus adds some logging to
show the progress of the retrieval, as it can be quite long.

Changes since v1:

 - Pass trust_env=True when creating the aiohttp.ClientSession() so
   that HTTP proxy environment variables are taken into
   account. Suggested by Matt Weber.

 - Fix bogus indentation that was breaking the logic of the latest
   version retrieval.

Thanks,

Thomas

Thomas Petazzoni (3):
  support/scripts/pkg-stats: use aiohttp for latest version retrieval
  support/scripts/pkg-stats: use aiohttp for upstream URL checking
  support/scripts/pkg-stats: show progress of upstream URL and latest
    version

 support/scripts/pkg-stats | 206 ++++++++++++++++++++++----------------
 1 file changed, 122 insertions(+), 84 deletions(-)