mbox series

[0/3] Use aiohttp in pkg-stats

Message ID 20200804124044.1217873-1-thomas.petazzoni@bootlin.com
Headers show
Series Use aiohttp in pkg-stats | expand

Message

Thomas Petazzoni Aug. 4, 2020, 12:40 p.m. UTC
Hello,

I started investigating why pkg-stats was sometimes stuck on the
server running it on a daily basis to populate
autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The
subprocesses started by "multiprocessing" to retrieve the latest
upstream version from release-monitoring.org were stuck holding a
lock. Without providing a definitive conclusion, some preliminary
research showed that multiprocessing can be tricky and cause some
issues with locks.

Discussing this with Titouan, he suggested to use aiohttp instead of
multiprocessing. And indeed, it makes a lot of sense to use this
popular asynchronous HTTP library.

This patch series switches the latest version retrieval and the
upstream URL checking to aiohttp, and as a bonus adds some logging to
show the progress of the retrieval, as it can be quite long.

Thanks in advance for your review and comments.

Thomas

Thomas Petazzoni (3):
  support/scripts/pkg-stats: use aiohttp for latest version retrieval
  support/scripts/pkg-stats: use aiohttp for upstream URL checking
  support/scripts/pkg-stats: show progress of upstream URL and latest
    version

 support/scripts/pkg-stats | 206 ++++++++++++++++++++++----------------
 1 file changed, 122 insertions(+), 84 deletions(-)