diff mbox series

[kteam-tools] stable: add jenkins-git-update.sh

Message ID 20180808105001.14164-1-po-hsu.lin@canonical.com
State New
Headers show
Series [kteam-tools] stable: add jenkins-git-update.sh | expand

Commit Message

Po-Hsu Lin Aug. 8, 2018, 10:50 a.m. UTC 
Add a script that can help us to update all SRU related git repositories on
all of our jenkins server, and will capability to kill the test request
handler to allow it respawn with cron jobs.

Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
---
 stable/README                |  8 ++++
 stable/jenkins-git-update.sh | 93 ++++++++++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+)
 create mode 100755 stable/jenkins-git-update.sh

Comments

Andy Whitcroft Aug. 8, 2018, 12:51 p.m. UTC | #1 
On Wed, Aug 08, 2018 at 06:50:01PM +0800, Po-Hsu Lin wrote:
> Add a script that can help us to update all SRU related git repositories on
> all of our jenkins server, and will capability to kill the test request
> handler to allow it respawn with cron jobs.
> 
> Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>

This is a fine idea in principle.  A number of our other kmsgq
based services handle this via an aqpm message, often delivered to
a service specific admin interface.  This allows us to pull the
source repositories on a cron job, and then on demand stop the
current runner at the end of its current instruction.

See the mainline-worker for an example of how we bind the admin port
there; though actually its naming is a bit substandard because it is
host specific and not host and service specific.  I must fix that.

-apw
diff mbox series

Patch

diff --git a/stable/README b/stable/README
index 2a2b5431..c909d99c 100644
--- a/stable/README
+++ b/stable/README
@@ -82,3 +82,11 @@  create-sru-cards.py:
 
     Example:
       create-sru-cards.py 2017.10.09
+
+jenkins-git-update.sh:
+    Update all the SRU related git repo on all of our jenkins servers and their
+    slaves. It can also kill the test request handler (kmsg-jenking) to make new
+    configs take effect if requested.
+
+    Example:
+        jenkins-git-update.sh -r
diff --git a/stable/jenkins-git-update.sh b/stable/jenkins-git-update.sh
new file mode 100755
index 00000000..50636aae
--- /dev/null
+++ b/stable/jenkins-git-update.sh
@@ -0,0 +1,93 @@ 
+#!/bin/bash
+#
+# A small tool for updating SRU git repos on jenkins servers.
+# Note that this script works only with bash 4 and later and
+# you will need to have access to these servers as well.
+#
+#                           Po-Hsu Lin <po-hsu.lin@canonical.com>
+#
+
+declare -A jenkins=( ["bare-metal"]="jenkins@10.246.72.4"
+                     ["live-patching"]="jenkins@10.246.72.5"
+                     ["aws"]="jenkins@10.246.72.47"
+                     ["azure"]="jenkins@10.246.72.46"
+                     ["google"]="jenkins@10.246.72.7" )
+declare -A slaves=( ["bare-metal"]=""
+                    ["live-patching"]=""
+                    ["aws"]="aws@obruchev"
+                    ["azure"]="azure@obruchev"
+                    ["google"]="google@obruchev" )
+repos=("autotest-client-tests" "autotest" "ckct" "kernel-testing")
+respawn=false
+
+
+function help_msg {
+    echo "Usage: $0 [-r]"
+    echo -e "\t-h : Print this help message and exit."
+    echo -e "\t-r : Kill the kmsgq-jenkins and request-handler to make it respawn with cron job."
+}
+
+
+function interrupted {
+    echo "SIGINT, terminating ssh-agent"
+    exit
+}
+
+
+function cleanup {
+    ssh-agent -k
+}
+
+
+while getopts :rh opt; do
+  case $opt in
+    r)
+      echo "Requested to restart kmsgq-jenkins and request-handler"
+      respawn=true
+      ;;
+    h)
+      help_msg
+      exit
+      ;;
+    \?)
+      echo "Invalid option: -$OPTARG"
+      help_msg
+      exit
+      ;;
+  esac
+done
+
+
+trap interrupted SIGINT
+trap cleanup EXIT
+eval `ssh-agent -s` > /dev/null
+ssh-add
+echo "ssh-agent with PID: $SSH_AGENT_PID will be terminated on exit / SIGINT"
+echo ""
+
+for target in "${!jenkins[@]}"
+do
+    echo -e "\033[1;32m == Running on $target jenkins == \033[m"
+    for dir in "${repos[@]}"
+    do
+        echo -e "\033[1;33m Updating $dir: \033[m"
+        ssh "${jenkins[$target]}" "cd $dir; git pull --ff-only"
+    done
+    echo ""
+    # Update slaves
+    if [ "${slaves[$target]}" != "" ]; then
+        echo -e "\033[1;36m -- Running on $target slave node: ${slaves[$target]} -- \033[m"
+        for dir in "${repos[@]}"
+        do
+            echo -e "\033[1;33m Updating $dir: \033[m"
+            ssh "${jenkins[$target]}" "ssh ${slaves[$target]} \"cd $dir; git pull --ff-only\""
+        done
+        echo ""
+    fi
+    if [ $respawn == true ]; then
+        echo "Killing kmsgq-jenkins / requests-handler on $target jenkins"
+        ssh "${jenkins[$target]}" "pkill -f kmsgq-jenkins"
+        ssh "${jenkins[$target]}" "pkill -f requests-handler"
+        echo ""
+    fi
+done