| Message ID | 1461299436-7208-1-git-send-email-andrew.donnellan@au1.ibm.com |
|---|---|
| State | Changes Requested |
| Headers | show |
On 22/04/16 14:30, Andrew Donnellan wrote: > At present, when getting the list of series from Patchwork, we request a > single "page" of results (up to 20 series). This more or less works for > daemon mode up to a certain point - it may cause problems for an extremely > busy Patchwork instance, but for most setups it'll work fine. > > However, when we're not running in daemon mode, and we specify --count=200, > we'd like to actually try and get 200 series, rather than testing the first > 20 over and over again... > > Instead of grabbing a single page, follow the "next" link that Patchwork > returns and request as many pages as possible. Change the query string to > get the maximum of 100 series per page, rather than the default 20. > > This approach is rather naive, and won't work very well on Patchwork setups > with a large existing database. > > Closes: #17 ("Enable the ability to go to page 2 (and beyond) of Patchwork > results") > Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> I'm seeing some issues with this patch that I'll need to look into next week.
On 22/04/16 16:49, Andrew Donnellan wrote: > I'm seeing some issues with this patch that I'll need to look into next > week. As it turns out, I still haven't gotten around to fixing this. Oh well, we'll just redo it all with the new upstream-patchwork API...
diff --git a/src/main.rs b/src/main.rs index d0177af..776a538 100644 --- a/src/main.rs +++ b/src/main.rs @@ -265,7 +265,7 @@ fn main() { // Poll patchwork for new series. For each series, get patches, apply and test. 'daemon: loop { - let series_list = patchwork.get_series_query().unwrap().results.unwrap(); + let series_list = patchwork.get_series_query().unwrap(); for series in series_list { // If it's already been tested, we can skip it if series.test_state.is_some() { diff --git a/src/patchwork.rs b/src/patchwork.rs index c8b9179..279933f 100644 --- a/src/patchwork.rs +++ b/src/patchwork.rs @@ -37,7 +37,7 @@ use utils; // TODO: more constants. constants for format strings of URLs and such. pub static PATCHWORK_API: &'static str = "/api/1.0"; -pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand"; +pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand&perpage=100"; // /api/1.0/projects/*/series/ @@ -176,10 +176,16 @@ impl PatchworkServer { .header(Connection::close()).send() } - pub fn get_series_query(&self) -> Result<SeriesList, DecoderError> { + pub fn get_series_query(&self) -> Result<Vec<Series>, DecoderError> { let url = format!("{}{}/series/{}", &self.url, PATCHWORK_API, PATCHWORK_QUERY); - json::decode(&self.get(&url).unwrap()) + let mut page: SeriesList = try!(json::decode(&self.get(&url).unwrap())); + let mut series = page.results.unwrap(); + while page.next.is_some() { + page = try!(json::decode(&self.get(&url).unwrap())); + series.extend(page.results.unwrap()); + } + Ok(series) } pub fn get_patch(&self, series: &Series) -> PathBuf {
At present, when getting the list of series from Patchwork, we request a single "page" of results (up to 20 series). This more or less works for daemon mode up to a certain point - it may cause problems for an extremely busy Patchwork instance, but for most setups it'll work fine. However, when we're not running in daemon mode, and we specify --count=200, we'd like to actually try and get 200 series, rather than testing the first 20 over and over again... Instead of grabbing a single page, follow the "next" link that Patchwork returns and request as many pages as possible. Change the query string to get the maximum of 100 series per page, rather than the default 20. This approach is rather naive, and won't work very well on Patchwork setups with a large existing database. Closes: #17 ("Enable the ability to go to page 2 (and beyond) of Patchwork results") Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> --- Depends-on: http://patchwork.ozlabs.org/patch/613373/ We need to work out a smarter method than just getting every single series that Patchwork has ever seen across all projects - I don't think it's as simple as trying to grab however many series are specified by --count... Discussion going on at https://github.com/ruscur/snowpatch/issues/17 --- src/main.rs | 2 +- src/patchwork.rs | 12 +++++++++--- 2 files changed, 10 insertions(+), 4 deletions(-)