From patchwork Tue Oct 22 21:11:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hosgor, Tolga (IOT DS EU TR MTS)" X-Patchwork-Id: 1181682 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::43c; helo=mail-wr1-x43c.google.com; envelope-from=swupdate+bncbdonz2w45qkrbdhbxxwqkgqelnsjkca@googlegroups.com; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.b="h36xOG1a"; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="giumA0yP"; dkim-atps=neutral Received: from mail-wr1-x43c.google.com (mail-wr1-x43c.google.com [IPv6:2a00:1450:4864:20::43c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46yR5Z6zHKz9sPZ for ; Wed, 23 Oct 2019 08:11:44 +1100 (AEDT) Received: by mail-wr1-x43c.google.com with SMTP id j14sf9480908wrm.6 for ; Tue, 22 Oct 2019 14:11:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1571778701; cv=pass; d=google.com; s=arc-20160816; b=R8cAXAs3jqSVbL7JhBkdTuoJF2oPowJ+PNwCCG4jvd44H8TkIR7Nirso1MeHRaD1/i Iqa0pBtKoILHiLgwhz4yhMf07lV5pIRFbAYscZMHg0Nnk0MnpWgeDohDmBBJyBKqBiFr LBENAqsAn1pF+CVRV8d4oD3tO+1uLzlBiIU+2ylCyBXauxoxK7bTVNPM/UEvT8ZyHaFG vrqUycpd+WDhuhyDcu+xtIkpHRBG+0S4C6eg4XEHDPbumyzNAoxEMcb55bmxrGNau87A 8Hs2HMAIYbesI3/BmgAodlXVV8jGmoQ5PagvFGNAmKrIWlNz9di6LjKIHOe/eD1NTh0N AeUQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature :dkim-signature; bh=FeXFt/ewiUyZqsHMRON2AdvBd7UHmk7OzxSQqgYNo6w=; b=rpjBicJohXeRxwnnl4VlFWQ/b/l05h7NgC/BUDokjSRCQHr/3gG7AwNs9cacqHLXnZ TCQZF1O4sUYVqU6YFFuHyiA3bNX6P+kgWsRCOtGY6NBJ8KXLxK3vUcZPv2Qbei3jGrRv GOEvMdYwDHuo+/aGEeHrolYD01Fq5ypEubSz7E6psN6j0X0xyGy6uaRyun/GlzzUJc+u jPtNp4P/A14w/6CIQbpnm5LH/vI3jTwoow+FmVJjwLrqVSNeYbMy++5CD9zvDctgmOOE lxLkMR3BVtE9IMyS2R48r0MibbREZmN768bhEYfD4rm7k1uUfVEtBjRleBVf/rHycFXa 1dtQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=repHxEJE; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=FeXFt/ewiUyZqsHMRON2AdvBd7UHmk7OzxSQqgYNo6w=; b=h36xOG1aWrkRnh1npp7Y/nRi7EIvbivBKv++7OiP9r5mVamQwmoVnOokMzVRx2XoFE V55caSlgrHUYfa9TKdUncF7BZ3zsToFUNlx7BIu1ebu/ENaQHFpYeQIwiGBAQTM9PRF5 zbmyqoIN4FwhmzMPbKCDcmSYgzz+QpqgpEsOCYwGV9SZsWHJ1JDPChXQgfFIFIb2lBVr 91twjwJJmVchiPnakjCOZ4vGZ5H7h9CTp44ltEsMi1hDIkdgx5t5Q6TKF5rtRkL1J21o BJe/grVXZ0QJx8yrHBV/ZXNFx8jT9jhs7jipU/VC0guBUOcMoWoFOsCFSWl2hXfNpQLy vKWw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=FeXFt/ewiUyZqsHMRON2AdvBd7UHmk7OzxSQqgYNo6w=; b=giumA0yPCSQep0NplcEJvC8xK9XCxznicuX8WgLBFOhi3eIUvJIeXE0Hn70JYcyyMn x7I8bj5pynuVpKxPXhYMC42zcm3BD4gtHijoQkKE5bNAxZ19C4phd4TcgH//aYfBUnX5 5DKPS+ED+6719vy/psRIIOfLyndQLSEw27GLHqDf7DxWJTt0g90M79OHepXCJRFZ8W9i GgXfIHO9wpl5OY8PHZWXyzEK/wYeqpi10A5attfa/g4yV4sQ9+CTC80GR1QWsF3il/gk ROZ9tGrjNcTt4Vj8ME8p2LiYl4vkjPsYVez4aQUmUOdkGyHvaj1qKIs1tsHwrYAlQG/7 bslA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=sender:x-gm-message-state:from:to:subject:date:message-id :in-reply-to:references:mime-version:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=FeXFt/ewiUyZqsHMRON2AdvBd7UHmk7OzxSQqgYNo6w=; b=RmAVAGHHnq7RdYR+QS99y7y/5vH5HnSWBor1VXXuVi1537u82IDe+BYxoOsQZU0ku9 Eg/ZOdF0y//Sw92lVlqbEo2dW0WSE24ew6tnZD2Y8GrHxLlCmSmNQND/mLW5mB3zSPVj 3tI38whk15qUzb1dWIjPV5A4tczsoXfkdgC4Wfzj7KIKURUQ1Sox00r5wqxsaf/XUCD7 SJOtUp2+GAuQyh+0zt0bR6Gi5Z+JWw13oeXeLVlgwuyptHwq92F26/w0t7e/iOIPqBvi yunP/Av/1OVbizlkuu/lNAOzXm/Kpjt2eEhDzNOtk/cwphvGvZ46L5Uvf18TsfYhmXGk KJfQ== Sender: swupdate@googlegroups.com X-Gm-Message-State: APjAAAXAesT410PQZog5/LOiK+gfxjNqRHijGXrGGK80LiN/lDlESJEZ DadM28QNYuzMmYScS7gEsPg= X-Google-Smtp-Source: APXvYqwwrR9F1vZtpi7DVbO4xmRV95BpEMOluoHBaj2PxOmnXhctKFo+BWGyuu9pVPrVnap+3ulQOw== X-Received: by 2002:adf:f3cb:: with SMTP id g11mr5287750wrp.120.1571778700987; Tue, 22 Oct 2019 14:11:40 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:adf:db07:: with SMTP id s7ls55775wri.6.gmail; Tue, 22 Oct 2019 14:11:40 -0700 (PDT) X-Received: by 2002:adf:e5cf:: with SMTP id a15mr5461197wrn.143.1571778700353; Tue, 22 Oct 2019 14:11:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571778700; cv=none; d=google.com; s=arc-20160816; b=wT1/pQGCu+VdPjTSUMt8Sjy6CulX0/XT3minaVjFYKZRe+ary3WNn5t0/TYE2ts4RD q3me1DrITh1EwuaeZToKQS45KuoTN3DDxtNgD8TdQJ96/pWlSXUJSRTp4L/Z+DTPun0E qrSABA7GZJrF3Wo+DipL97QaTAmVjyOI+f4BvCMXZVHsXxuG7LXrB3Iqe56ZSzvGvD4/ 3bZVehpmWNc9J9OX6a+LBIzDjHSQj6+3t5q+azbLIwO0e4+i7IFPpuXY12vGShXDJgF9 zYBayr7dcRU3g3ORl8Rp6r5lZzXvafyPO1WffpLSVgvtfTNH4YJQptIc1PzBk80kE+KZ mraw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:dkim-signature; bh=h2giLd9roUxZejb8RSmsEwgbx777gwjtKGDPkuM1lEM=; b=Gj3k9slWEGX9XRV6BWMUjNxE8c33elGv1km9shSars4O8FO5frN1Yt7Ee11wR5r3cd sq1XW4CpZayw/xAabt6SRSt17K8mLS5om/ITJtIu8FO5WxzWjyVgMmAENl/B88VaDrUo ALjuqcYA7gzaFg37ErKuWzQyclYzNaGZCG3v+crfoxyNXRDhZYLhibsc1ulbWk9sS9mc y5HG9btBxuMEIyqqOv6aOR7C0nP+E/bpSOI5hX9RJx6rkEINSE9Ndj0uVB4wGzI0FmpT 1IT+Si1iJvJE0yBmIs9zgdyr+x5kSbSxmbCkPUe7If4qga+/L6aAoUAcGT0kfK6XWftH Fyrw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=repHxEJE; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com. [2a00:1450:4864:20::42d]) by gmr-mx.google.com with ESMTPS id l23si743874wmg.2.2019.10.22.14.11.40 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Oct 2019 14:11:40 -0700 (PDT) Received-SPF: pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) client-ip=2a00:1450:4864:20::42d; Received: by mail-wr1-x42d.google.com with SMTP id r1so9842400wrs.9 for ; Tue, 22 Oct 2019 14:11:40 -0700 (PDT) X-Received: by 2002:adf:e747:: with SMTP id c7mr5226372wrn.384.1571778699230; Tue, 22 Oct 2019 14:11:39 -0700 (PDT) Received: from localhost ([185.46.212.110]) by smtp.gmail.com with ESMTPSA id q124sm34077754wma.5.2019.10.22.14.11.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2019 14:11:38 -0700 (PDT) From: "Hosgor, Tolga (IOT DS EU TR MTS)" To: swupdate@googlegroups.com Subject: [swupdate] [PATCH v5] Add Zstd compression support Date: Wed, 23 Oct 2019 00:11:36 +0300 Message-Id: <20191022211136.151818-1-tolga.hosgor@siemens.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191022205123.63974-1-tolga.hosgor@siemens.com> References: <20191022205123.63974-1-tolga.hosgor@siemens.com> MIME-Version: 1.0 X-Original-Sender: tlghosgor@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=repHxEJE; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , Zstd compression support can now be enabled with the CONFIG_ZSTD build configuration. This commit extends the 'compressed' field in the sw-description, allowing to specify 'compressed = "zstd"' or 'compressed = "zlib". The old method of specfying the boolean 'compressed = true' will default to using the 'zlib' format. Signed-off-by: Hosgor, Tolga (IOT DS EU TR MTS) --- The difference of this from v4 is that it does not introduce the value 'compression = "none"' and a documentation correction about unknown strings causing an error. Kconfig | 8 ++ Makefile.deps | 4 + Makefile.flags | 4 + core/cpio_utils.c | 158 ++++++++++++++++++++++++++-------- corelib/lua_interface.c | 30 ++++++- doc/source/roadmap.rst | 9 +- doc/source/sw-description.rst | 11 ++- doc/source/swupdate.rst | 2 +- include/swupdate.h | 7 ++ parser/parse_external.c | 19 +++- parser/parser.c | 16 +++- 11 files changed, 218 insertions(+), 50 deletions(-) diff --git a/Kconfig b/Kconfig index 70907c9..6f1e7ad 100644 --- a/Kconfig +++ b/Kconfig @@ -61,6 +61,10 @@ config HAVE_ZLIB bool option env="HAVE_ZLIB" +config HAVE_ZSTD + bool + option env="HAVE_ZSTD" + config HAVE_LIBSSL bool option env="HAVE_LIBSSL" @@ -434,5 +438,9 @@ config GUNZIP depends on HAVE_ZLIB default y +config ZSTD + bool "Zstd compression support" + depends on HAVE_ZSTD + source parser/Config.in source handlers/Config.in diff --git a/Makefile.deps b/Makefile.deps index 512d4b8..5e2bd2f 100644 --- a/Makefile.deps +++ b/Makefile.deps @@ -38,6 +38,10 @@ ifeq ($(HAVE_ZLIB),) export HAVE_ZLIB = y endif +ifeq ($(HAVE_ZSTD),) +export HAVE_ZSTD = y +endif + ifeq ($(HAVE_LIBUBOOTENV),) export HAVE_LIBUBOOTENV = y endif diff --git a/Makefile.flags b/Makefile.flags index b880d32..614d772 100644 --- a/Makefile.flags +++ b/Makefile.flags @@ -162,6 +162,10 @@ ifeq ($(CONFIG_GUNZIP),y) LDLIBS += z endif +ifeq ($(CONFIG_ZSTD),y) +LDLIBS += zstd +endif + ifeq ($(CONFIG_RDIFFHANDLER),y) LDLIBS += rsync endif diff --git a/core/cpio_utils.c b/core/cpio_utils.c index 1b6cb14..5910c69 100644 --- a/core/cpio_utils.c +++ b/core/cpio_utils.c @@ -13,6 +13,9 @@ #ifdef CONFIG_GUNZIP #include #endif +#ifdef CONFIG_ZSTD +#include +#endif #include "generated/autoconf.h" #include "cpiohdr.h" @@ -233,22 +236,29 @@ static int decrypt_step(void *state, void *buffer, size_t size) return 0; } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) +typedef int (*DecompressStep)(void *state, void *buffer, size_t size); -struct GunzipState -{ +struct DecompressState { PipelineStep upstream_step; void *upstream_state; + void *impl_state; + uint8_t input[BUFF_SIZE]; + bool eof; +}; +#endif +#ifdef CONFIG_GUNZIP + +struct GunzipState { z_stream strm; bool initialized; - uint8_t input[BUFF_SIZE]; - bool eof; }; static int gunzip_step(void *state, void *buffer, size_t size) { - struct GunzipState *s = (struct GunzipState *)state; + struct DecompressState *ds = (struct DecompressState *)state; + struct GunzipState *s = (struct GunzipState *)ds->impl_state; int ret; int outlen = 0; @@ -256,23 +266,23 @@ static int gunzip_step(void *state, void *buffer, size_t size) s->strm.avail_out = size; while (outlen == 0) { if (s->strm.avail_in == 0) { - ret = s->upstream_step(s->upstream_state, s->input, sizeof s->input); + ret = ds->upstream_step(ds->upstream_state, ds->input, sizeof ds->input); if (ret < 0) { return ret; } else if (ret == 0) { - s->eof = true; + ds->eof = true; } s->strm.avail_in = ret; - s->strm.next_in = s->input; + s->strm.next_in = ds->input; } - if (s->eof) { + if (ds->eof) { break; } ret = inflate(&s->strm, Z_NO_FLUSH); outlen = size - s->strm.avail_out; if (ret == Z_STREAM_END) { - s->eof = true; + ds->eof = true; break; } if (ret != Z_OK && ret != Z_BUF_ERROR) { @@ -285,6 +295,52 @@ static int gunzip_step(void *state, void *buffer, size_t size) #endif +#ifdef CONFIG_ZSTD + +struct ZstdState { + ZSTD_DStream* dctx; + ZSTD_inBuffer input_view; +}; + +static int zstd_step(void* state, void* buffer, size_t size) +{ + struct DecompressState *ds = (struct DecompressState *)state; + struct ZstdState *s = (struct ZstdState *)ds->impl_state; + size_t decompress_ret; + int ret; + ZSTD_outBuffer output = { buffer, size, 0 }; + + do { + if (s->input_view.pos == s->input_view.size) { + ret = ds->upstream_step(ds->upstream_state, ds->input, sizeof ds->input); + if (ret < 0) { + return ret; + } else if (ret == 0) { + ds->eof = true; + } + s->input_view.size = ret; + s->input_view.pos = 0; + } + + do { + decompress_ret = ZSTD_decompressStream(s->dctx, &output, &s->input_view); + + if (ZSTD_isError(decompress_ret)) { + ERROR("ZSTD_decompressStream failed (returned %zu)", decompress_ret); + return -1; + } + + if (output.pos == output.size) { + break; + } + } while (s->input_view.pos < s->input_view.size); + } while (output.pos == 0 && !ds->eof); + + return output.pos; +} + +#endif + int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsigned long long seek, int skip_file, int __attribute__ ((__unused__)) compressed, uint32_t *checksum, unsigned char *hash, int encrypted, writeimage callback) @@ -314,17 +370,29 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi .outlen = 0, .eof = false }; +#if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) + struct DecompressState decompress_state = { + .upstream_step = NULL, .upstream_state = NULL, + .impl_state = NULL + }; + + DecompressStep decompress_step = NULL; #ifdef CONFIG_GUNZIP struct GunzipState gunzip_state = { - .upstream_step = NULL, .upstream_state = NULL, .strm = { .zalloc = Z_NULL, .zfree = Z_NULL, .opaque = Z_NULL, .avail_in = 0, .next_in = Z_NULL, .avail_out = 0, .next_out = Z_NULL }, .initialized = false, - .eof = false }; +#endif +#ifdef CONFIG_ZSTD + struct ZstdState zstd_state = { + .dctx = NULL, + .input_view = { NULL, 0, 0 }, + }; +#endif #endif PipelineStep step = NULL; @@ -357,21 +425,38 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi if (compressed) { #ifdef CONFIG_GUNZIP - /* - * 16 + MAX_WBITS means that Zlib should expect and decode a - * gzip header. - */ - if (inflateInit2(&gunzip_state.strm, 16 + MAX_WBITS) != Z_OK) { - ERROR("inflateInit2 failed"); - ret = -EFAULT; + if (compressed == COMPRESSED_ZLIB || compressed == COMPRESSED_TRUE) { + /* + * 16 + MAX_WBITS means that Zlib should expect and decode a + * gzip header. + */ + if (inflateInit2(&gunzip_state.strm, 16 + MAX_WBITS) != Z_OK) { + ERROR("inflateInit2 failed"); + ret = -EFAULT; + goto copyfile_exit; + } + gunzip_state.initialized = true; + decompress_step = &gunzip_step; + decompress_state.impl_state = &gunzip_state; + } else +#endif +#ifdef CONFIG_ZSTD + if (compressed == COMPRESSED_ZSTD) { + if ((zstd_state.dctx = ZSTD_createDStream()) == NULL) { + ERROR("ZSTD_createDStream failed"); + ret = -EFAULT; + goto copyfile_exit; + } + zstd_state.input_view.src = decompress_state.input; + decompress_step = &zstd_step; + decompress_state.impl_state = &zstd_state; + } else +#endif + { + TRACE("Requested decompression method (%d) is not configured!", compressed); + ret = -EINVAL; goto copyfile_exit; } - gunzip_state.initialized = true; -#else - TRACE("Request decompressing, but CONFIG_GUNZIP not set !"); - ret = -EINVAL; - goto copyfile_exit; -#endif } if (seek) { @@ -384,19 +469,19 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi } } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) if (compressed) { if (encrypted) { decrypt_state.upstream_step = &input_step; decrypt_state.upstream_state = &input_state; - gunzip_state.upstream_step = &decrypt_step; - gunzip_state.upstream_state = &decrypt_state; + decompress_state.upstream_step = &decrypt_step; + decompress_state.upstream_state = &decrypt_state; } else { - gunzip_state.upstream_step = &input_step; - gunzip_state.upstream_state = &input_state; + decompress_state.upstream_step = &input_step; + decompress_state.upstream_state = &input_state; } - step = &gunzip_step; - state = &gunzip_state; + step = decompress_step; + state = &decompress_state; } else { #endif if (encrypted) { @@ -408,7 +493,7 @@ int copyfile(int fdin, void *out, unsigned int nbytes, unsigned long *offs, unsi step = &input_step; state = &input_state; } -#ifdef CONFIG_GUNZIP +#if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) } #endif @@ -487,6 +572,11 @@ copyfile_exit: inflateEnd(&gunzip_state.strm); } #endif +#ifdef CONFIG_ZSTD + if (zstd_state.dctx != NULL) { + ZSTD_freeDStream(zstd_state.dctx); + } +#endif return ret; } diff --git a/corelib/lua_interface.c b/corelib/lua_interface.c index 5a66db9..0c865f2 100644 --- a/corelib/lua_interface.c +++ b/corelib/lua_interface.c @@ -34,6 +34,12 @@ extern const char EMBEDDED_LUA_SRC_END[]; lua_settable(L, -3); \ } while (0) +#define LUA_PUSH_IMG_STRING_VALUE(img, attr, value) do { \ + lua_pushstring(L, attr); \ + lua_pushstring(L, value); \ + lua_settable(L, -3); \ +} while (0) + #define LUA_PUSH_IMG_BOOL(img, attr, field) do { \ lua_pushstring(L, attr); \ lua_pushboolean(L, img->field); \ @@ -241,6 +247,16 @@ static void lua_string_to_img(struct img_type *img, const char *key, const char offset[] = "offset"; char seek_str[MAX_SEEK_STRING_SIZE]; + if (!strcmp(key, "compressed")) { + if (!strcmp(value, "zlib")) { + img->compressed = COMPRESSED_ZLIB; + } else if (!strcmp(value, "zstd")) { + img->compressed = COMPRESSED_ZSTD; + } else { + ERROR("compressed argument: '%s' invalid, defaulting to 'false'", value); + img->compressed = false; + } + } if (!strcmp(key, "name")) { strncpy(img->id.name, value, sizeof(img->id.name)); @@ -317,7 +333,6 @@ static void lua_number_to_img(struct img_type *img, const char *key, img->size = (long long)val; if (!strcmp(key, "checksum")) img->checksum = (unsigned int)val; - } #ifdef CONFIG_HANDLER_IN_LUA @@ -461,7 +476,6 @@ static void update_table(lua_State* L, struct img_type *img) LUA_PUSH_IMG_STRING(img, "data", type_data); LUA_PUSH_IMG_STRING(img, "filesystem", filesystem); - LUA_PUSH_IMG_BOOL(img, "compressed", compressed); LUA_PUSH_IMG_BOOL(img, "installed_directly", install_directly); LUA_PUSH_IMG_BOOL(img, "install_if_different", id.install_if_different); LUA_PUSH_IMG_BOOL(img, "install_if_higher", id.install_if_higher); @@ -473,6 +487,18 @@ static void update_table(lua_State* L, struct img_type *img) LUA_PUSH_IMG_NUMBER(img, "size", size); LUA_PUSH_IMG_NUMBER(img, "checksum", checksum); + switch (img->compressed) { + case COMPRESSED_ZLIB: + LUA_PUSH_IMG_STRING_VALUE(img, "compressed", "zlib"); + break; + case COMPRESSED_ZSTD: + LUA_PUSH_IMG_STRING_VALUE(img, "compressed", "zstd"); + break; + default: + LUA_PUSH_IMG_BOOL(img, "compressed", compressed); + break; + } + lua_pushstring(L, "properties"); lua_newtable (L); LIST_FOREACH(property, &img->properties, next) { diff --git a/doc/source/roadmap.rst b/doc/source/roadmap.rst index bf0a25e..bb9d546 100644 --- a/doc/source/roadmap.rst +++ b/doc/source/roadmap.rst @@ -33,10 +33,11 @@ and not only UBI volumes, and add further features as restoring configuration da Support for further compressors =============================== -SWUpdate supports image compressed with zlib. This is a compromise between compression rate -and speed to decompress the single artifact. To reduce bandwidth or for big images, a stronger -compressor could help. Adding a new compressor must be careful done because it changes the -core of handling an image. +SWUpdate supports image compressed with following formats: zlib, zstd. This is +a compromise between compression rate and speed to decompress the single artifact. +To reduce bandwidth or for big images, a stronger compressor could help. +Adding a new compressor must be careful done because it changes the core of +handling an image. System Update ============= diff --git a/doc/source/sw-description.rst b/doc/source/sw-description.rst index 70efa30..e1d1709 100644 --- a/doc/source/sw-description.rst +++ b/doc/source/sw-description.rst @@ -1242,9 +1242,14 @@ There are 4 main sections inside sw-description: | | | scripts | regitsters itself. | | | | | Example: "ubivol", "raw", "rawfile", | +-------------+----------+------------+---------------------------------------+ - | compressed | bool | images | flag to indicate that "filename" is | - | | | files | zlib-compressed and must be | - | | | | decompressed before being installed | + | compressed | string | images | string to indicate the "filename" is | + | | or bool | files | compressed and must be decompressed | + | | | | before being installed. the value | + | | | | denotes the compression type. | + | | | | currently supported values are "zlib" | + | | | | and "zstd" . if the bool form is used | + | | | | , then the compression type is | + | | | | assumed to be "zlib". | +-------------+----------+------------+---------------------------------------+ | installed-\ | bool | images | flag to indicate that image is | | directly | | | streamed into the target without any | diff --git a/doc/source/swupdate.rst b/doc/source/swupdate.rst index 8cf911b..6c7a98e 100644 --- a/doc/source/swupdate.rst +++ b/doc/source/swupdate.rst @@ -56,7 +56,7 @@ General Overview is saved and restored after repartitioning with all data it contains, to maintain user's data. -- support for compressed images, using the zlib library. +- support for compressed images, using the zlib and zstd library. tarball (tgz file) are supported. - support for partitioned USB-pen or unpartitioned (mainly diff --git a/include/swupdate.h b/include/swupdate.h index b28504c..8cd1574 100644 --- a/include/swupdate.h +++ b/include/swupdate.h @@ -41,6 +41,13 @@ enum { INSTALL_FROM_STREAM }; +enum { + COMPRESSED_FALSE, + COMPRESSED_TRUE, + COMPRESSED_ZLIB, + COMPRESSED_ZSTD, +}; + struct sw_version { char name[SWUPDATE_GENERAL_STRING_SIZE]; char version[SWUPDATE_GENERAL_STRING_SIZE]; diff --git a/parser/parse_external.c b/parser/parse_external.c index 39b4300..ef6c74c 100644 --- a/parser/parse_external.c +++ b/parser/parse_external.c @@ -83,8 +83,19 @@ static void sw_append_stream(struct img_type *img, const char *key, ascii_to_hash(img->sha256, value); if (!strcmp(key, "encrypted")) img->is_encrypted = 1; - if (!strcmp(key, "compressed")) - img->compressed = 1; + if (!strcmp(key, "compressed")) { + if (value != NULL) { + if (!strcmp(value, "zlib")) { + img->compressed = COMPRESSED_ZLIB; + } else if (!strcmp(value, "zstd")) { + img->compressed = COMPRESSED_ZSTD; + } else { + img->compressed = COMPRESSED_TRUE; + } + } else { + img->compressed = COMPRESSED_TRUE; + } + } if (!strcmp(key, "installed-directly")) img->install_directly = 1; if (!strcmp(key, "install-if-different")) @@ -113,7 +124,7 @@ int parse_external(struct swupdate_cfg *software, const char *filename) if (ret) { LUAstackDump(L); ERROR("ERROR preparing Parser in Lua %d", ret); - + return 1; } @@ -153,7 +164,7 @@ int parse_external(struct swupdate_cfg *software, const char *filename) if (lua_type(L, -1) == LUA_TTABLE) { lua_pushnil(L); - image = (struct img_type *)calloc(1, sizeof(struct img_type)); + image = (struct img_type *)calloc(1, sizeof(struct img_type)); if (!image) { ERROR( "No memory: malloc failed"); return -ENOMEM; diff --git a/parser/parser.c b/parser/parser.c index 3cb3c97..67a56f7 100644 --- a/parser/parser.c +++ b/parser/parser.c @@ -139,7 +139,7 @@ static bool get_common_fields(parsertype p, void *cfg, struct swupdate_cfg *swcf ERROR("Missing version in configuration file"); return false; } - + GET_FIELD_STRING(p, setting, NULL, swcfg->version); TRACE("Version %s", swcfg->version); @@ -263,6 +263,7 @@ static int run_embscript(parsertype p, void *elem, struct img_type *img, static int parse_common_attributes(parsertype p, void *elem, struct img_type *image) { char seek_str[MAX_SEEK_STRING_SIZE]; + const char* compressed; /* * GET_FIELD_STRING does not touch the passed string if it is not @@ -290,7 +291,18 @@ static int parse_common_attributes(parsertype p, void *elem, struct img_type *im return -1; } - get_field(p, elem, "compressed", &image->compressed); + if ((compressed = get_field_string(p, elem, "compressed")) != NULL) { + if (!strcmp(compressed, "zlib")) { + image->compressed = COMPRESSED_ZLIB; + } else if (!strcmp(compressed, "zstd")) { + image->compressed = COMPRESSED_ZSTD; + } else { + ERROR("compressed argument: '%s' unknown", compressed); + return -1; + } + } else { + get_field(p, elem, "compressed", &image->compressed); + } get_field(p, elem, "installed-directly", &image->install_directly); get_field(p, elem, "preserve-attributes", &image->preserve_attributes); get_field(p, elem, "install-if-different", &image->id.install_if_different);