From patchwork Tue Oct 26 13:54:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schrimpf X-Patchwork-Id: 1546467 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20210112 header.b=dOz90xu1; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=KqSqOhPy; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2607:f8b0:4864:20::1039; helo=mail-pj1-x1039.google.com; envelope-from=swupdate+bncbcpkta6p7yerbm4r4cfqmgqekjaw63a@googlegroups.com; receiver=) Received: from mail-pj1-x1039.google.com (mail-pj1-x1039.google.com [IPv6:2607:f8b0:4864:20::1039]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HdtcS1KP6z9sRN for ; Wed, 27 Oct 2021 00:55:02 +1100 (AEDT) Received: by mail-pj1-x1039.google.com with SMTP id r7-20020a17090a454700b001a1ca0191b8sf9197041pjm.4 for ; Tue, 26 Oct 2021 06:55:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1635256499; cv=pass; d=google.com; s=arc-20160816; b=1HqRO6a8OGV2xgH87Jqga0ppfbZP3ZWfbcRjWzYNX2rlqJ/PgqY9FtpnjM6FXQPH3y Atuv9D611bFQYwGLyDHi+A9a3uFtPGJ9eEbSffs+u0OFi3UWdyC3jIe/S0TOcwfa0RLO yFo14HVKdjwJx/GIS2R0qL1oQkd00ChF75gE5Pwy4YptM8SeyEb4vnhBj9cwR/mMkEK8 FebExFeRpdBojZeOtYvqH0Vjk4vGcPqurqp/g+sNHJ0/fC0+yK6U9KiubATPNyZ1Uqvk HegDsfw03LHNKOtwSrgRu56jprH8e3EDuZ0DaKnS6BOH++7XujlA25ghPl/f/0z2M6Q0 iCTA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:to:subject:message-id:date:from :mime-version:sender:dkim-signature:dkim-signature; bh=Kj07iXhTsk5sbB1h/7+39JrMgcfTbTfNEElKfxxxz68=; b=NgFu56Uw+6WGMp9grWG+9bzxNGTOtGkLKg+yGclbKBaD6xmJAyQpN4PKG6Aq5uBY1i 0lmQzbqvsSczASZ+MwAcSBevDAb39lxW1iMdk3p8tFqLyRkPe8IHee1WjN/aPGQmMF2i uP5QMxXoQ+zoaNL/zH6Pp1dd3O6BBxfDGDen4Ti4q7Fzsz9DKAHxf/VqU/xxxi4bosJb p6jo8ckIXhMlamWL5cqmsthE7W/uIwzgJc/Ffv4NHgYUm0Ik+YECRivqnwMKQoroV4lP jhJM5ViHTJLERTOeCczBwkRsbfRG1Td+KpQkZnDg992MoFiUtizd4sVUSMHCo84g0MAV rBCA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=nezL+3h4; spf=pass (google.com: domain of joh.schrimpf@gmail.com designates 2607:f8b0:4864:20::b2f as permitted sender) smtp.mailfrom=joh.schrimpf@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:mime-version:from:date:message-id:subject:to :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=Kj07iXhTsk5sbB1h/7+39JrMgcfTbTfNEElKfxxxz68=; b=dOz90xu1Xkwv6fOP4IgulhTMKj8icdHOZlaLvtH5tUammeSpx2W0sJlGs5fvINjeId T7g1/rHCrLmbIVdg8OYwCHcCc2gQChJMd9uGZrWPtr59OJLfT4werdnlnxiLvh+VgI9R aTGZNaILK8ptWDwDx1u1TUs5mtHPiumj6G9dbBR/o/m7AgVmiokWExki2rB2qpd2mvV+ s1ACCwJ3S8CLyvN53A9kVfnq1xqhBJVFmj5tzLiXTqRRLHmZ+ZzynYiOLERBXL0GJMCf 7w75dKyFbLLGoD9oEGbNvJW6VVze78NmULY1N2tsGUwZGBCofjZNrh8YKjYP0dVPsj5X pR1g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=Kj07iXhTsk5sbB1h/7+39JrMgcfTbTfNEElKfxxxz68=; b=KqSqOhPyK08/V2IULjL7eFr/s+RHHejUHVpYLdvppRJ/VwU70MUklrAUhEVm5fqWlV G0OW+miDRO0klIC3FNUGEhZF9qNwCGBU0+0czM9R/pNlOe+sArLIehsOemeigHWIFSkt r7DnaDOLx/luvhBWJ+p4tOjsZCap1y6BVxsscOIBWKOKL/i3JTq4LNwdvS4oGZyZAKp+ RA47Fjd6UG5/eO4oam1yGHRTt9eIF5Gzev/v29ewU+o7HDLY8TtVPKfqpoD3aMFiYGym Zri010l3xJkq8nn5DVnMBHcLdL2IRZ/AEr5STTaJEiHnQUeE1/Zw2DdU+oKcYHcD2bJm HMqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:mime-version:from:date:message-id:subject :to:x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=Kj07iXhTsk5sbB1h/7+39JrMgcfTbTfNEElKfxxxz68=; b=znS8ILRh60Wnu5NpNf6ZBISNFvQgVOvA2ZQ59/wavRJQ8Zlz8rmwB2SG50rvT+J7sD T3ZdYYM/SqhDexQLfh+jhfLajwjZ1y6lGo+QOYYIYCHqHHWGXjkLsqAG1coNUMoteECJ zEgIupdSO1H7pVTiAiJGicaYn+79rv5GMCWGNA6pmLBBc4zpDgTV2j6As5EYyxWMUKcp O2lncWKtrQP3atVEHVGCjTzhxDj6lgSq7115yJZD34dFHjJ4LjzEFw/bsHi3DFPnYkRq e+pKkXliQ0jzOnduFmVrEXpPVEz02JANQMm39n6XpFA5usHjlnhJaxc3GAmnavjp8mzZ bvOg== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOAM533H7GHUQmfXjnH3eJ4emmC5PFPz8FdYhMBH5In/Pg5Fi/Ia89tR LKDN0bCtN1mrlhGKJ2PQ0RA= X-Google-Smtp-Source: ABdhPJyPbuuqlQYtr4BeVNlp/WDYbfROnoeXpJhKYMPCBlb89BaivIyXIMoA0gpXcD8cmUxQsRn7WA== X-Received: by 2002:a17:90b:3841:: with SMTP id nl1mr8642419pjb.24.1635256499267; Tue, 26 Oct 2021 06:54:59 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a05:6a00:10ce:: with SMTP id d14ls3575373pfu.4.gmail; Tue, 26 Oct 2021 06:54:58 -0700 (PDT) X-Received: by 2002:a62:4e87:0:b0:47b:dbbf:c6f0 with SMTP id c129-20020a624e87000000b0047bdbbfc6f0mr21304043pfb.47.1635256498459; Tue, 26 Oct 2021 06:54:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635256498; cv=none; d=google.com; s=arc-20160816; b=BjYLO2SougAmDP6aXNBNOzm4LGki8NvVNuqt+MrmQts5LhwNGCqaPFbLFIUv+vk1XY 8/zkLJ4J0+0iRghp29sHWhcz9OnRoiKt4tyhxtG+LopqnmNkEh6YbE6zv0yHflpBkoP0 Imm9EWBom1mMSYuVLc6wRUzR0xio/Cg4I/EwSCX9cm3Oxuh8rrognd5mtllOsIT/neii oh2FLLD76igJwUMS5sKce9Mz7ELQp0IETt6d1coruebQbsiElo/rltAxHpSLgDB6ugD8 woX7knwhQqfTDzbVb8bxvGb61gOPywP1U/6OfIOZ8+0jC0236LD9YJ2hFc55O1ppyE6o DojA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=to:subject:message-id:date:from:mime-version:dkim-signature; bh=qAnV59kjB57rah4M90z8UXjuUAmGuKovITywH9KuMGo=; b=npaNa1hW/1zVN6dPMJQSKB2/fwyWxF9YMPFuTliCyeYTaK4tPLdmIf56HLBEVsZgVj 7nO8TLmcY76O8G9Xug+JyfsBAfpNiSo9dRt1R63jh/1Ue/lUJ2OCWm4OzgOUcrFheq49 ZAPaaJLXai06YPtzcWfwxkWh03DamrRKsi1VFG0OkXUsrIH+32mHZQNlnzGvbqWaHLjc OQIvCxS7I+/pT/En4Oa+s2EZgJBurQu6K5yy/Xz7ffo0QV0wkZpmue8/oO3mV6jubRwh EhZ384ehd88D/flSef2KAJgpfkiDFxZatKiBYGhYQIHUF9E44/hIFc0/hRspt+FR8kaa wKVw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=nezL+3h4; spf=pass (google.com: domain of joh.schrimpf@gmail.com designates 2607:f8b0:4864:20::b2f as permitted sender) smtp.mailfrom=joh.schrimpf@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com. [2607:f8b0:4864:20::b2f]) by gmr-mx.google.com with ESMTPS id o17si1583802pfu.5.2021.10.26.06.54.58 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Oct 2021 06:54:58 -0700 (PDT) Received-SPF: pass (google.com: domain of joh.schrimpf@gmail.com designates 2607:f8b0:4864:20::b2f as permitted sender) client-ip=2607:f8b0:4864:20::b2f; Received: by mail-yb1-xb2f.google.com with SMTP id 67so35110281yba.6 for ; Tue, 26 Oct 2021 06:54:58 -0700 (PDT) X-Received: by 2002:a25:3489:: with SMTP id b131mr12324651yba.359.1635256497962; Tue, 26 Oct 2021 06:54:57 -0700 (PDT) MIME-Version: 1.0 From: Johannes Schrimpf Date: Tue, 26 Oct 2021 15:54:47 +0200 Message-ID: Subject: [swupdate] Improvements to python client example To: swupdate@googlegroups.com X-Original-Sender: joh.schrimpf@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=nezL+3h4; spf=pass (google.com: domain of joh.schrimpf@gmail.com designates 2607:f8b0:4864:20::b2f as permitted sender) smtp.mailfrom=joh.schrimpf@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Hi, we are using the example python client in our test stations and have done some improvements on the way: - Start websocket client at the beginning of the update, so all messages are shown, not only the messages that are received after finishing the upload - Call requests.post in executor to not block the asyncio event loop - Use argparse to handle command line arguments - Handle timeout in both the upload task and the websocket task - Use flake8/black to lint and format the code - Change file name to swupdate_client.py, so it can be included from other python modules - Add support for logging, so external programs can register their own logger (in our case, we are logging directly to OpenHTF) - Handle json parse error in the version info message - Add Pipfile and README.md with documentation on how to install dependencies and run the program Best regards Johannes Schrimpf Blueye Robotics From ef6a26f0527013ff2aed9cfcec39cb1b7e0ae396 Mon Sep 17 00:00:00 2001 From: Johannes Schrimpf Date: Tue, 26 Oct 2021 15:21:31 +0200 Subject: [PATCH 1/3] Rename swupdate-client.py to swupdate_client.py Change file name to swupdate_client.py, so it can be included from other python modules Signed-off-by: Johannes Schrimpf --- examples/client/{swupdate-client.py => swupdate_client.py} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename examples/client/{swupdate-client.py => swupdate_client.py} (100%) diff --git a/examples/client/swupdate-client.py b/examples/client/swupdate_client.py similarity index 100% rename from examples/client/swupdate-client.py rename to examples/client/swupdate_client.py -- 2.30.1 (Apple Git-130) From 303c093b04d7d7b8ce6668c7b4a3c5165bd1b83a Mon Sep 17 00:00:00 2001 From: Johannes Schrimpf Date: Tue, 26 Oct 2021 15:22:26 +0200 Subject: [PATCH 2/3] Improve example client - Start websocket client at the beginning of the update, so all messages are shown - Call requests.post in executor to not block the asyncio event loop - Use argparse to handle command line arguments - Handle timeout in both the upload task and the websocket task - Use flake8/black to lint and format code - Add support for logging, so external programs can register their own logger - Handle json parse error in version info message Signed-off-by: Johannes Schrimpf --- examples/client/swupdate_client.py | 158 +++++++++++++++++++++-------- 1 file changed, 113 insertions(+), 45 deletions(-) diff --git a/examples/client/swupdate_client.py b/examples/client/swupdate_client.py index ae31ec9..fc0c1d4 100755 --- a/examples/client/swupdate_client.py +++ b/examples/client/swupdate_client.py @@ -6,68 +6,136 @@ import asyncio import json -import os import requests import websockets +import logging +import string +import argparse import sys class SWUpdater: - "" " Python helper class for SWUpdate " "" - - url_upload = 'http://{}:{}/upload' - url_status = 'ws://{}:{}/ws' - - def __init__ (self, path_image, host_name, port): - self.__image = path_image - self.__host_name = host_name - self.__port = port - - - async def wait_update_finished(self, timeout = 300): - print ("Wait update finished") - async def get_finish_messages (): - async with websockets.connect(self.url_status.format(self.__host_name, self.__port)) as websocket: + """Python helper class for SWUpdate""" + + url_upload = "http://{}:{}/upload" + url_status = "ws://{}:{}/ws" + + def __init__(self, path_image, host_name, port=8080, logger=None): + self._image = path_image + self._host_name = host_name + self._port = port + if logger is not None: + self._logger = logger + else: + logging.basicConfig(stream=sys.stdout, level=logging.INFO) + self._logger = logging.getLogger("swupdate") + + async def wait_update_finished(self): + self._logger.info("Waiting for messages on websocket connection") + try: + async with websockets.connect( + self.url_status.format(self._host_name, self._port) + ) as websocket: while True: - message = await websocket.recv() - data = json.loads(message) - - if data ["type"] != "message": + try: + message = await websocket.recv() + message = "".join( + filter(lambda x: x in set(string.printable), message) + ) + + except Exception as err: + self._logger.warning(err) continue - print (data["text"]) - if data ["text"] == "SWUPDATE successful !": - return + try: + data = json.loads(message) + except json.decoder.JSONDecodeError: + # As of 2021.04, the version info message contains invalid json + self._logger.warning(f"json parse error: {message}") + continue - await asyncio.wait_for(get_finish_messages(), timeout = timeout) + if data["type"] != "message": + continue - def update (self, timeout = 300): - print ("Start uploading image...") - print (self.url_upload.format(self.__host_name, self.__port)) + self._logger.info(data["text"]) + if "SWUPDATE successful" in data["text"]: + return True + if "Installation failed" in data["text"]: + return False + + except Exception as err: + self._logger.error(err) + return False + + def sync_upload(self, swu_file, timeout): + return requests.post( + self.url_upload.format(self._host_name, self._port), + files={"file": swu_file}, + timeout=timeout, + ) + + async def upload(self, timeout): + self._logger.info("Start uploading image...") try: - response = requests.post(self.url_upload.format(self.__host_name, self.__port), files = { 'file':open (self.__image, 'rb') }) + with open(self._image, "rb") as swu_file: + loop = asyncio.get_event_loop() + response = await loop.run_in_executor( + None, self.sync_upload, swu_file, timeout + ) if response.status_code != 200: - raise Exception ("Cannot upload software image: {}". format (response.status_code)) - - print ("Software image uploaded successfully. Wait for installation to be finished...\n") - asyncio.sleep(10) - asyncio.get_event_loop().run_until_complete(self.wait_update_finished(timeout = timeout)) - + self._logger.error( + "Cannot upload software image: {}".format(response.status_code) + ) + return False + + self._logger.info( + "Software image uploaded successfully." + "Wait for installation to be finished..." + ) + return True except ValueError: - print("No connection to host, exit") + self._logger.info("No connection to host, exit") + except FileNotFoundError: + self._logger.info("swu file not found") + except requests.exceptions.ConnectionError as e: + self._logger.info("Connection Error:\n%s" % str(e)) + return False + + async def start_tasks(self, timeout): + ws_task = asyncio.create_task(self.wait_update_finished()) + upload_task = asyncio.create_task(self.upload(timeout)) + + if not await upload_task: + self._logger.info("Cancelling websocket task") + ws_task.cancel() + return False + try: + result = await asyncio.wait_for(ws_task, timeout=timeout) + except asyncio.TimeoutError: + self._logger.info("timeout!") + return False -if __name__ == "__main__": - sys.path.append (os.getcwd ()) + return result - if len (sys.argv) == 3: - port = "8080" - elif len (sys.argv) == 4: - port = sys.argv[3] - else: - print ("Usage: swupdate.py [port]") - exit (1) + def update(self, timeout=300): + return asyncio.run(self.start_tasks(timeout)) - SWUpdater (sys.argv[1], sys.argv[2], port).update () +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("swu_file", help="Path to swu image") + parser.add_argument("host_name", help="Host name") + parser.add_argument("port", help="Port", type=int, default=8080, nargs="?") + parser.add_argument( + "--timeout", + help="Timeout for the whole swupdate process", + type=int, + default=300, + nargs="?", + ) + + args = parser.parse_args() + updater = SWUpdater(args.swu_file, args.host_name, args.port) + updater.update(timeout=args.timeout) -- 2.30.1 (Apple Git-130) From b0401b4d85d9f6ce6cbca91653f133872cb8cd72 Mon Sep 17 00:00:00 2001 From: Johannes Schrimpf Date: Tue, 26 Oct 2021 15:26:53 +0200 Subject: [PATCH 3/3] Add Pipfile and README.md Signed-off-by: Johannes Schrimpf --- examples/client/Pipfile | 13 +++++++++++ examples/client/README.md | 47 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 examples/client/Pipfile create mode 100644 examples/client/README.md diff --git a/examples/client/Pipfile b/examples/client/Pipfile new file mode 100644 index 0000000..a237601 --- /dev/null +++ b/examples/client/Pipfile @@ -0,0 +1,13 @@ +[[source]] +url = "https://pypi.org/simple" +verify_ssl = true +name = "pypi" + +[packages] +websockets = "*" +requests = "*" + +[dev-packages] + +[requires] +python_version = "3.9" diff --git a/examples/client/README.md b/examples/client/README.md new file mode 100644 index 0000000..b6e5d32 --- /dev/null +++ b/examples/client/README.md @@ -0,0 +1,47 @@ +## Install dependencies + +### apt +``` +sudo apt update +sudo apt install python3-websockets python3-requests +``` + +### pip +``` +pip install websockets requests +``` + +### pipenv +``` +pipenv install +``` + +## Usage + +### apt/pip +``` +./swupdate_client.py [port] +``` + +### pipenv +``` +pipenv run ./swupdate_client.py [port] +``` + + +## Development +### Import from another python program +``` +from swupdate_client import SWUpdater + +updater = SWUpdater("path-to-swu", "host-name") +if updater.update(): + print("Update successful!") +else: + print("Update failed!") +``` + +### Formatting +``` +black swupdate_client.py +``` \ No newline at end of file