Message ID | 20180221141716.10908-5-dja@axtens.net |
---|---|
State | Changes Requested |
Headers | show |
Series | Tools and fixes for parallel parsing | expand |
Context | Check | Description |
---|---|---|
dja/snowpatch-0_1_0 | success | master/apply_patch Successfully applied |
dja/snowpatch-snowpatch_job_snowpatch-patchwork | success | Test snowpatch/job/snowpatch-patchwork on branch master |
On 22/02/18 01:17, Daniel Axtens wrote: > If you have multiple archives, you quickly tire of typing stuff like > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-1 & > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-2 & > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-3 & > python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-4 & > and having to copy and paste it - or retype it! - each time you reset > the database. > > Instead, this patch allows you to do > tools/scripts/parallel_parsearchive.sh --list-id=patchwork.ozlabs.org -- foo-* > > Much easier, especially when you are doing it a dozen times. > > Signed-off-by: Daniel Axtens <dja@axtens.net> The overengineering approach to this would be to make the parsearchive command support -j and work around python's broken threading model and all ;) For the purposes, this solution will work fine though... (Also, who was it that said Greenspun's tenth rule, but with GNU parallel?) One comment below, with that fixed: Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> > --- > tools/scripts/parallel_parsearchive.sh | 55 ++++++++++++++++++++++++++++++++++ > 1 file changed, 55 insertions(+) > create mode 100755 tools/scripts/parallel_parsearchive.sh > > diff --git a/tools/scripts/parallel_parsearchive.sh b/tools/scripts/parallel_parsearchive.sh > new file mode 100755 > index 000000000000..a5197f6c2c5f > --- /dev/null > +++ b/tools/scripts/parallel_parsearchive.sh > @@ -0,0 +1,55 @@ > +#!/bin/bash > +# Patchwork - automated patch tracking system > +# Copyright (C) 2018 Daniel Axtens <dja@axtens.net> > +# > +# This file is part of the Patchwork package. > +# > +# Patchwork is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# Patchwork is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +set -euo pipefail > + > +usage() { > + cat <<EOF > +parallel_parsearchive.sh - load archives in parallel > +Usage: > + parallel_parsearchive.sh [parsearchive options] -- <archives> > + The -- is mandatory. > + As many processes as there are archives will be spun up. > + > +Example: > +EOF Did you mean to put an example here?
diff --git a/tools/scripts/parallel_parsearchive.sh b/tools/scripts/parallel_parsearchive.sh new file mode 100755 index 000000000000..a5197f6c2c5f --- /dev/null +++ b/tools/scripts/parallel_parsearchive.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# Patchwork - automated patch tracking system +# Copyright (C) 2018 Daniel Axtens <dja@axtens.net> +# +# This file is part of the Patchwork package. +# +# Patchwork is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# Patchwork is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +set -euo pipefail + +usage() { + cat <<EOF +parallel_parsearchive.sh - load archives in parallel +Usage: + parallel_parsearchive.sh [parsearchive options] -- <archives> + The -- is mandatory. + As many processes as there are archives will be spun up. + +Example: +EOF + exit 1 +} + +if [ $# -eq 0 ] || [[ $1 == "-h" ]]; then + usage; +fi + +PARSEARCHIVE_OPTIONS="" +while [[ $1 != "--" ]]; do + PARSEARCHIVE_OPTIONS="$PARSEARCHIVE_OPTIONS $1" + shift + if [ $# -eq 0 ]; then + usage; + fi +done +shift + +if [ $# -eq 0 ]; then + usage; +fi + + +for x in "$@"; do + echo "Starting $x" + python3 manage.py parsearchive $PARSEARCHIVE_OPTIONS "$x" & +done +echo "Processes started in the background."
If you have multiple archives, you quickly tire of typing stuff like python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-1 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-2 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-3 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-4 & and having to copy and paste it - or retype it! - each time you reset the database. Instead, this patch allows you to do tools/scripts/parallel_parsearchive.sh --list-id=patchwork.ozlabs.org -- foo-* Much easier, especially when you are doing it a dozen times. Signed-off-by: Daniel Axtens <dja@axtens.net> --- tools/scripts/parallel_parsearchive.sh | 55 ++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100755 tools/scripts/parallel_parsearchive.sh