From patchwork Fri Oct 18 10:18:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hosgor, Tolga (IOT DS EU TR MTS)" X-Patchwork-Id: 1179281 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::13a; helo=mail-lf1-x13a.google.com; envelope-from=swupdate+bncbdonz2w45qkrb75cu3wqkgqevbxty4i@googlegroups.com; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.b="Ad8b9Nqs"; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LhhNiUXh"; dkim-atps=neutral Received: from mail-lf1-x13a.google.com (mail-lf1-x13a.google.com [IPv6:2a00:1450:4864:20::13a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46vhnz6sH7z9sPT for ; Fri, 18 Oct 2019 21:18:46 +1100 (AEDT) Received: by mail-lf1-x13a.google.com with SMTP id m16sf1180924lfb.1 for ; Fri, 18 Oct 2019 03:18:46 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1571393919; cv=pass; d=google.com; s=arc-20160816; b=09MVUPgeOyMRjPcS9bYjiME6kmvCquaMnjqTBWnDZUdmLheKsYapivDOObxtzXoAq4 LX8G0ZbUkX35hVvi7nTouhcseMIZiHXtvuM+BfUsfmaYeERzoqm83xMvflQx84PtteHT 5FcBe2xEdnX8YnCiJCMV8DD7IldRSujeIokqCVfd4mK5YQxAL1WWDLw4e0svajjmeUbl U93+5wCeRRJYhqn7+3hV/QBFvc+KwZZyNI25FlDR0VFrR0lM0spR1aZBbP4Wwkmi5rDE tApCEVuJqyz3qB4DeuXgj0onDghYonR8z39IbN0FBjiY/bhRxMT7wgFZefyr+s353F2r ch2w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature:dkim-signature; bh=VuI5E1JfbO/Gjsik797f7z4xIcaCgx2ytdDYBO21Hno=; b=ykE8vrFbk62dRbSmtR4H8HH/TQTnPYmKfJFnm/LgbSC5izyh2N50g3KaEG2Mkxc5Yx FGgnhsvZUG2BXWVGRsgXz/XhZzPpC4hggSqZ2LJua6UrMoFqPHEDdwNmiJ+qj4EhadbV R2gU3+zOKKxoOO3v4srmMDOWp0/LgxzYifMApc3GPJqlTTi3fPns14UtD3m+ZE/hJpM4 eM+V8EPWRlATKi/MQPiRk7mJNCaUb51cBYud/cwsE5Em3YSC3YJa/ywpRIDOgVMi3FHT lclArdaCU+DBG/iSb3bK71M4m8XGvBWIZhHQFztkv8WnQoiYz51rK64115X4XJDdPo7R ZuAQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ch8QH0Tk; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::434 as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=VuI5E1JfbO/Gjsik797f7z4xIcaCgx2ytdDYBO21Hno=; b=Ad8b9Nqs8CUhjCJmLCGTdvgYJfAjClwzKRwGkwh0PbQ+eHeWpQUZ9XpFy7xu//kaLY WtLCaNj1zsC+IPUSxMiBwYlSXGNx+HDu+BfB2jvBKuzHHojFFY7abN6D1gZxI0J9R1nn wLHyRxueZd32V1AUIRbIvqL+rgFX+FiY57IujhVKqrxmowNh77tQlI1mcBgoksRPhDlB qxjOLwTFvlPrrUhoKNb88AGhRwesrP9cst5bvNAuXw6AHXUr+JJyGGPzHsRF9G/MBcVo YZY6Iy+RWughfSwHqgPpgi+QuT4jmyqEVo6eLPCM65dF2GvTAICqtc406j+JKp6HXMFU UoOA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=VuI5E1JfbO/Gjsik797f7z4xIcaCgx2ytdDYBO21Hno=; b=LhhNiUXhbLDEjS+cTZuNmd93PYSDQBxfCeALUZir34yIeu/ixGbq6wukZVfjzIbRLF oXX83vpIRvtGVor+KUi71Wvlya5BH6/1sUnJbzQfhdM4SLjsWitd/DyxWGAOArbJ5lgU 7M/do71792AWGX4bpm8btMmpH2CYK5RUa+HwmF6SnjZAWAAOLNnrDzB8bpaxSYX70pEy vKGxcpnzZdK1Y6rKs1BpoXMTLi2l99t9jNIUk5JgUtOC3Z4Lf4z1oZxu1BaiGs2WnDqO Jji23P0SHAv0CPuKByNLQW7SOKOemY4YSDywLeVCgv9UzqeTBM+pzKAeuuY3VjINOkYY lenQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=VuI5E1JfbO/Gjsik797f7z4xIcaCgx2ytdDYBO21Hno=; b=iA0pa1P/GxEBKimpqLELH3NfXpLM4gtnSwfKPNFpXta5Y4B+39edCANBZlwfUns8Zg l3SJGnUtq9KyqwNAh96PMtdqhqaex1kLUPAWB9KoVqB4BsAqPJVGlAuxm9PiAignw93Y 48JOcfGM9MUVJFmY7vDtmaZmPHcNO3mrQvXZs7PF/sJnfifvFhtzAwxzJg781TxgrteU 9JOuE9vaUVCIBaZ3qIGAHWJYtINJ291uc+ua2qtS/59tHRzLqPEO/kxqYiceuZ03GZ7K OKCAiLf2aBDANPCWxRR0CRHXDZT2Sfco8QCbrqPauHBMnMs63Hr7J6GC0F7w0cQzWYUp OkZw== Sender: swupdate@googlegroups.com X-Gm-Message-State: APjAAAUnbIJUcq9+y+gYxfAiZx38DHP/wZ7hqPQoofAL8vhH6EoOMM8I 2DV7zisKNfr/ETjWh5oCbPU= X-Google-Smtp-Source: APXvYqwoo9IkxeUt2TBa5GJyf8a9ZTPcK0GR6BA2z8a3j5egvcFQ/omVahEFkUC2ahRldPwE3P5M8Q== X-Received: by 2002:ac2:43a8:: with SMTP id t8mr5589134lfl.134.1571393919300; Fri, 18 Oct 2019 03:18:39 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a2e:95d4:: with SMTP id y20ls669063ljh.15.gmail; Fri, 18 Oct 2019 03:18:38 -0700 (PDT) X-Received: by 2002:a2e:331a:: with SMTP id d26mr5537949ljc.231.1571393918715; Fri, 18 Oct 2019 03:18:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571393918; cv=none; d=google.com; s=arc-20160816; b=v6KeUNJd8LwSgME33JiZASEek1AkHhsRAYRL7lbO9Hx1U3QqXp+epzSFSwhA++jR9x kkr2hPl6JpFjTPKhk3HBKs52mEKQFpx5fh0rj0N3b678yF/HrfVgL3jDPzbOuaaaBq1H wpBlflazYWOamTKEmdupcTZd83ni6aaXYCOwTD+ZIJl7f1BGTmZzZrp6yk4X6KZAw5d/ 3Bp8bMi/TL0aW9ZAAz/mYCbuP0eMYWll/UGr2Og1uQuQ9XO61QSqmsMiCtdwfHf9L2Zb en3kF+AjY120vZitE8ux+W6hemcJrRHZS5m7pf89jx49Y6T0e5nuFCdFNUSEm7YoLtFt RfDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:dkim-signature; bh=AKKKGP42dGH1BDmduToOIvgjZxCxi9UOfXZhPNizF+E=; b=xHxfK6qSRZqkolL81SXrJ0dB/wkL0nFBFAyMMSLtLc7VbwS9TMpaxpysxoPbbIaC97 dMHSL83r0rKVohv9H2kwlG6f0TyzbaCUpCbQ231Aisw/xFSfCwpBaY3CSI49SFAy0QdJ 9Al1yruNOSYj22RI6h9lOwrvmn57QPBEEQJbITDS2bjkWKYXJCTTcS3bGaTIbNcumaPZ atIMfhpqtFco0Gxh3Q3AsIl97lofHQoCwlxUS7/SDi7HARpO3nwmybp/YJiylEKCaxGU MAyIk3gyvTQgvf/oR71DSUVCcdSoBQRu1Kywo53zKiAhKWPlo3ispO0l8YC494hgIH0d MbDw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ch8QH0Tk; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::434 as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com. [2a00:1450:4864:20::434]) by gmr-mx.google.com with ESMTPS id h10si223778lfm.2.2019.10.18.03.18.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 18 Oct 2019 03:18:38 -0700 (PDT) Received-SPF: pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::434 as permitted sender) client-ip=2a00:1450:4864:20::434; Received: by mail-wr1-x434.google.com with SMTP id o18so5597238wrv.13 for ; Fri, 18 Oct 2019 03:18:38 -0700 (PDT) X-Received: by 2002:a5d:65cf:: with SMTP id e15mr61100wrw.391.1571393917805; Fri, 18 Oct 2019 03:18:37 -0700 (PDT) Received: from localhost ([185.46.212.166]) by smtp.gmail.com with ESMTPSA id r65sm5091411wmr.9.2019.10.18.03.18.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Oct 2019 03:18:37 -0700 (PDT) From: "Hosgor, Tolga (IOT DS EU TR MTS)" To: swupdate@googlegroups.com Subject: [swupdate] [PATCH] Add Zstd compression support Date: Fri, 18 Oct 2019 13:18:35 +0300 Message-Id: <20191018101835.2264070-1-tolga.hosgor@siemens.com> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 X-Original-Sender: tlghosgor@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ch8QH0Tk; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::434 as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , From: Hosgor, Tolga (IOT DS EU TR MTS) Build-time configuration to support Zstd decompression of raw compressed images. Signed-off-by: Hosgor, Tolga (IOT DS EU TR MTS) --- Kconfig | 19 ++++++++- Makefile.deps | 4 ++ Makefile.flags | 4 ++ core/cpio_utils.c | 99 +++++++++++++++++++++++++++++++++++++++++++---- 4 files changed, 116 insertions(+), 10 deletions(-) diff --git a/Kconfig b/Kconfig index 882ee1e..e9c535d 100644 --- a/Kconfig +++ b/Kconfig @@ -61,6 +61,10 @@ config HAVE_ZLIB bool option env="HAVE_ZLIB" +config HAVE_ZSTD + bool + option env="HAVE_ZSTD" + config HAVE_LIBSSL bool option env="HAVE_LIBSSL" @@ -441,10 +445,21 @@ source suricatta/Config.in source mongoose/Config.in +choice + prompt "Compression implementation to use" + default GUNZIP + help + Select compression implementation for decompressing images. + config GUNZIP - bool + bool "GUNZIP" depends on HAVE_ZLIB - default y + +config ZSTD + bool "ZSTD" + depends on HAVE_ZSTD + +endchoice source parser/Config.in source handlers/Config.in diff --git a/Makefile.deps b/Makefile.deps index 512d4b8..5e2bd2f 100644 --- a/Makefile.deps +++ b/Makefile.deps @@ -38,6 +38,10 @@ ifeq ($(HAVE_ZLIB),) export HAVE_ZLIB = y endif +ifeq ($(HAVE_ZSTD),) +export HAVE_ZSTD = y +endif + ifeq ($(HAVE_LIBUBOOTENV),) export HAVE_LIBUBOOTENV = y endif diff --git a/Makefile.flags b/Makefile.flags index 0edd48c..38db720 100644 --- a/Makefile.flags +++ b/Makefile.flags @@ -162,6 +162,10 @@ ifeq ($(CONFIG_GUNZIP),y) LDLIBS += z endif +ifeq ($(CONFIG_ZSTD),y) +LDLIBS += zstd +endif + ifeq ($(CONFIG_RDIFFHANDLER),y) LDLIBS += rsync endif diff --git a/core/cpio_utils.c b/core/cpio_utils.c index 1b6cb14..9332a4c 100644 --- a/core/cpio_utils.c +++ b/core/cpio_utils.c @@ -10,8 +10,10 @@ #include #include #include -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) #include +#elif defined(CONFIG_ZSTD) +#include #endif #include "generated/autoconf.h" @@ -233,7 +235,7 @@ static int decrypt_step(void *state, void *buffer, size_t size) return 0; } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) struct GunzipState { @@ -283,6 +285,55 @@ static int gunzip_step(void *state, void *buffer, size_t size) return outlen; } +#elif defined(CONFIG_ZSTD) + +struct ZstdState +{ + PipelineStep upstream_step; + void *upstream_state; + + ZSTD_DStream* dctx; + ZSTD_inBuffer input; + uint8_t input_buffer[BUFF_SIZE]; + bool eof; +}; + +static int zstd_step(void* state, void* buffer, size_t size) +{ + struct ZstdState *s = (struct ZstdState *)state; + size_t decompress_ret; + int ret; + ZSTD_outBuffer output = { buffer, size, 0 }; + + do { + if (s->input.pos == s->input.size) { + ret = s->upstream_step(s->upstream_state, s->input_buffer, sizeof s->input_buffer); + if (ret < 0) { + return ret; + } else if (ret == 0) { + s->eof = true; + } + s->input.size = ret; + s->input.pos = 0; + } + + do { + decompress_ret = ZSTD_decompressStream(s->dctx, &output, &s->input); + + if (ZSTD_isError(decompress_ret)) { + ERROR("ZSTD_decompressStream failed (returned %zu)", decompress_ret); + return -1; + } + + if (output.pos == output.size) { + break; + } + } while (s->input.pos < s->input.size); + } while (output.pos == 0 && !s->eof); + + return output.pos; +} + #endif int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsigned long long seek, @@ -314,7 +365,7 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi .outlen = 0, .eof = false }; -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) struct GunzipState gunzip_state = { .upstream_step = NULL, .upstream_state = NULL, .strm = { @@ -325,6 +376,13 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi .initialized = false, .eof = false }; +#elif defined(CONFIG_ZSTD) + struct ZstdState zstd_state = { + .upstream_step = NULL, .upstream_state = NULL, + .dctx = NULL, + .input = { NULL, 0, 0 }, + .eof = false + }; #endif PipelineStep step = NULL; @@ -356,7 +414,7 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi } if (compressed) { -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) /* * 16 + MAX_WBITS means that Zlib should expect and decode a * gzip header. @@ -367,8 +425,15 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi goto copyfile_exit; } gunzip_state.initialized = true; +#elif defined(CONFIG_ZSTD) + if ((zstd_state.dctx = ZSTD_createDStream()) == NULL) { + ERROR("ZSTD_createDStream failed"); + ret = -EFAULT; + goto copyfile_exit; + } + zstd_state.input.src = zstd_state.input_buffer; #else - TRACE("Request decompressing, but CONFIG_GUNZIP not set !"); + TRACE("Request decompressing, but CONFIG_GUNZIP or CONFIG_ZSTD not set !"); ret = -EINVAL; goto copyfile_exit; #endif @@ -384,7 +449,7 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi } } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) if (compressed) { if (encrypted) { decrypt_state.upstream_step = &input_step; @@ -398,6 +463,20 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi step = &gunzip_step; state = &gunzip_state; } else { +#elif defined(CONFIG_ZSTD) + if (compressed) { + if (encrypted) { + decrypt_state.upstream_step = &input_step; + decrypt_state.upstream_state = &input_state; + zstd_state.upstream_step = &decrypt_step; + zstd_state.upstream_state = &decrypt_state; + } else { + zstd_state.upstream_step = &input_step; + zstd_state.upstream_state = &input_state; + } + step = &zstd_step; + state = &zstd_state; + } else { #endif if (encrypted) { decrypt_state.upstream_step = &input_step; @@ -408,7 +487,7 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi step = &input_step; state = &input_state; } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) } #endif @@ -482,10 +561,14 @@ copyfile_exit: if (input_state.dgst) { swupdate_HASH_cleanup(input_state.dgst); } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) if (gunzip_state.initialized) { inflateEnd(&gunzip_state.strm); } +#elif defined(CONFIG_ZSTD) + if (zstd_state.dctx != NULL) { + ZSTD_freeDStream(zstd_state.dctx); + } #endif return ret;