[RFC] patchwork: get more than one page's worth of series
diff mbox

Message ID 1461299436-7208-1-git-send-email-andrew.donnellan@au1.ibm.com
State Changes Requested
Headers show

Commit Message

Andrew Donnellan April 22, 2016, 4:30 a.m. UTC
At present, when getting the list of series from Patchwork, we request a
single "page" of results (up to 20 series). This more or less works for
daemon mode up to a certain point - it may cause problems for an extremely
busy Patchwork instance, but for most setups it'll work fine.

However, when we're not running in daemon mode, and we specify --count=200,
we'd like to actually try and get 200 series, rather than testing the first
20 over and over again...

Instead of grabbing a single page, follow the "next" link that Patchwork
returns and request as many pages as possible. Change the query string to
get the maximum of 100 series per page, rather than the default 20.

This approach is rather naive, and won't work very well on Patchwork setups
with a large existing database.

Closes: #17 ("Enable the ability to go to page 2 (and beyond) of Patchwork
results")
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

---

Depends-on: http://patchwork.ozlabs.org/patch/613373/

We need to work out a smarter method than just getting every single series
that Patchwork has ever seen across all projects - I don't think it's as
simple as trying to grab however many series are specified by --count...
Discussion going on at https://github.com/ruscur/snowpatch/issues/17
---
 src/main.rs      |  2 +-
 src/patchwork.rs | 12 +++++++++---
 2 files changed, 10 insertions(+), 4 deletions(-)

Comments

Andrew Donnellan April 22, 2016, 6:49 a.m. UTC | #1
On 22/04/16 14:30, Andrew Donnellan wrote:
> At present, when getting the list of series from Patchwork, we request a
> single "page" of results (up to 20 series). This more or less works for
> daemon mode up to a certain point - it may cause problems for an extremely
> busy Patchwork instance, but for most setups it'll work fine.
>
> However, when we're not running in daemon mode, and we specify --count=200,
> we'd like to actually try and get 200 series, rather than testing the first
> 20 over and over again...
>
> Instead of grabbing a single page, follow the "next" link that Patchwork
> returns and request as many pages as possible. Change the query string to
> get the maximum of 100 series per page, rather than the default 20.
>
> This approach is rather naive, and won't work very well on Patchwork setups
> with a large existing database.
>
> Closes: #17 ("Enable the ability to go to page 2 (and beyond) of Patchwork
> results")
> Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

I'm seeing some issues with this patch that I'll need to look into next 
week.
Andrew Donnellan July 18, 2016, 11:08 p.m. UTC | #2
On 22/04/16 16:49, Andrew Donnellan wrote:
> I'm seeing some issues with this patch that I'll need to look into next
> week.

As it turns out, I still haven't gotten around to fixing this. Oh well, 
we'll just redo it all with the new upstream-patchwork API...

Patch
diff mbox

diff --git a/src/main.rs b/src/main.rs
index d0177af..776a538 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -265,7 +265,7 @@  fn main() {
 
     // Poll patchwork for new series. For each series, get patches, apply and test.
     'daemon: loop {
-        let series_list = patchwork.get_series_query().unwrap().results.unwrap();
+        let series_list = patchwork.get_series_query().unwrap();
         for series in series_list {
             // If it's already been tested, we can skip it
             if series.test_state.is_some() {
diff --git a/src/patchwork.rs b/src/patchwork.rs
index c8b9179..279933f 100644
--- a/src/patchwork.rs
+++ b/src/patchwork.rs
@@ -37,7 +37,7 @@  use utils;
 
 // TODO: more constants.  constants for format strings of URLs and such.
 pub static PATCHWORK_API: &'static str = "/api/1.0";
-pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand";
+pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand&perpage=100";
 
 // /api/1.0/projects/*/series/
 
@@ -176,10 +176,16 @@  impl PatchworkServer {
             .header(Connection::close()).send()
     }
 
-    pub fn get_series_query(&self) -> Result<SeriesList, DecoderError> {
+    pub fn get_series_query(&self) -> Result<Vec<Series>, DecoderError> {
         let url = format!("{}{}/series/{}", &self.url,
                           PATCHWORK_API, PATCHWORK_QUERY);
-        json::decode(&self.get(&url).unwrap())
+        let mut page: SeriesList = try!(json::decode(&self.get(&url).unwrap()));
+        let mut series = page.results.unwrap();
+        while page.next.is_some() {
+            page = try!(json::decode(&self.get(&url).unwrap()));
+            series.extend(page.results.unwrap());
+        }
+        Ok(series)
     }
 
     pub fn get_patch(&self, series: &Series) -> PathBuf {