From patchwork Sun Nov 14 17:26:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Babic X-Patchwork-Id: 1554892 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20210112 header.b=Lm1i5ntW; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2a00:1450:4864:20::23f; helo=mail-lj1-x23f.google.com; envelope-from=swupdate+bncbcxploxj6ikrbf4oywgamgqecrlkhja@googlegroups.com; receiver=) Received: from mail-lj1-x23f.google.com (mail-lj1-x23f.google.com [IPv6:2a00:1450:4864:20::23f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HsfRM73q2z9sfG for ; Mon, 15 Nov 2021 04:27:58 +1100 (AEDT) Received: by mail-lj1-x23f.google.com with SMTP id q64-20020a2e2a43000000b00218c94eab9bsf4411234ljq.18 for ; Sun, 14 Nov 2021 09:27:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1636910871; cv=pass; d=google.com; s=arc-20160816; b=yxtI04iplkJZBuJD4bMBX24OT2KKkS//hP3iqXxHB1dSgepwusNvymXy5UhcFmUuu4 nMeBgn0eAR6wjwFUgbFlH/JpqrTAtU5jVzYryH7sBdDA8mcGtgGan6n0rzxEmM0pFtV0 J2Pfj3pF7uVpZMMlN+hBslan9oeaOfNAVyizwMOLKjuvZjVg1WIKuQEHQZZYMUA+Izoe 9DsDzPar41Fzjhqaocn56QfnOjprDqH/qLcaO+VSV2orSrP4sGcQHWLjmf5Pu9TLHtgd Bvt/L8MNp56b//5dX6AyZxk3/1IT8Nfh/TvEyKbZyxUbJDCCfwLM095s5Y4IrD7MtGKy VeJg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=P3d8844ggDXqBac50BJ+Yw9MUPufGcTkUOhSxcESXYc=; b=suJsoIPdkc6FlNyu/I1ljKHknHkGT1+6VOEOWYNnMPPGQN23FAhPKz4KGIkkiKB6xE NINciM3C7CxwQtV9SFV2Mh9tWuuRrE04ROHrf+qUgStT9vM2VC7QYbdZUdhoBcPCqv5W hQX5UCZ8o2cTva4sjlEOArozOmX12i0kSf5P0haNwXU0WFObLpMK/4wWwmeMkX/mitdR JmOj5RrkhH4ePM+slBKgJY3z/hWthqn4un2NDxoqYqEkGRP14CYF/am5yqZIjtksizVd kkP7pwk+PuEVwJYlzY/RN97ReiN5M8+GUhg1QtsuMpT5jvJIzhMuL9+O4jJS7U6TmpN9 vjaQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:mime-version :x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=P3d8844ggDXqBac50BJ+Yw9MUPufGcTkUOhSxcESXYc=; b=Lm1i5ntW5jBOprEy4+7bVQZvipgvtVIfn7w4QFIIWnTYVJ5zh5X2s4bdPQ2pubgf8a U+r0KVu/F5EJqDT1wDDlS4y+wh7ec+fQuVh2vRRoK1GS+5NFHN1ztzjob1cODQNiTJoF kXJkRt8wSo8P9aky0l8VOHJmIcDMsIOs1560xeKhSg/aVXu7omGVbk/psLXwpgLvAwu8 vkqpacV/iCmBR56k7hIRGs7AilkzyYE+6CuXDRo3K9VGYalvnJhmZOW63FXyZnvEnTz9 n56R6Pns5dRnpbM3cLvhb2sAme2/1Jtxl+Ldqr8l/dymHoMNmTWIyajslE6+qKkDrm6g t4nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=sender:x-gm-message-state:from:to:cc:subject:date:message-id :mime-version:x-original-sender:x-original-authentication-results :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=P3d8844ggDXqBac50BJ+Yw9MUPufGcTkUOhSxcESXYc=; b=oqCTpFhacDDIoygsE/cmrq966AGOrGnxeE3zuKSSfkdscaRZPtccuDDk+Sr3L118NB 4l3IVgiko0SF0Zao/5DhsgArZK+WoJGlX2us5niwJCGwb2ZkA0P8wrntk2hHOkErFi/u OtXbG1d35ecEDDLE/p/qEfBVNZBBJtGb4+VfFGuPqSS5tZ3hiTDLKsG+lAPvBCOUcEeQ r/c7PU3L4uqybXVm9OVg7KSFRoQ1Qs8xJ7B7sAxu7+F8iw/e491nqpu9ibyohD0WJSJi Grc0zUsZWDhvzjGoowBPi9UnneTywlTdjeJG0N8uMmLrTrX+ArzBLbsMuXwvfG7SgtyI ss7A== Sender: swupdate@googlegroups.com X-Gm-Message-State: AOAM530dEUqpwUY1J/R5QSCGWSS5wxByykBd1TuzPLaKk3IsL/CYnywh dfXSV0oRXisCkij1bc5/UQM= X-Google-Smtp-Source: ABdhPJwVjNdHQXX4xisgh+MvUvvzQb8CNczWLTOh4A/AbCI7FKCpXw5XtEnQq9ghiVjl4S6uQh6QMQ== X-Received: by 2002:a2e:3c13:: with SMTP id j19mr4111138lja.311.1636910871765; Sun, 14 Nov 2021 09:27:51 -0800 (PST) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a05:651c:b1f:: with SMTP id b31ls1834942ljr.0.gmail; Sun, 14 Nov 2021 09:27:50 -0800 (PST) X-Received: by 2002:a2e:9708:: with SMTP id r8mr33377606lji.36.1636910870838; Sun, 14 Nov 2021 09:27:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636910870; cv=none; d=google.com; s=arc-20160816; b=I5lxzE10pIWXtsujk6YcQB4xbM4b+zcAaaioEKGKX5dBrtTKO4J+xznoxCEx0is8cf 7W6aDJC/CqdiVgW7lvK+L/wFvdxue0gLMUj6jj/Ifh9sNLOxwEtl+yQG4/y0BaugBz/U INWZe3nx4+U/iD8y8K5pO/duB8EixmKgnnlEwFJJG2S9aRQO+YysK4Vgl3bl3NPjTkA9 RG53PZNSRHghvvURSNfEVcauc7+FINYAL6gZSVi+Hth+l1akiL+ey2qKg+8E+v2ZMUwM modF+rw+fzH5YwV0mBOwplKWKJnL8bSzoclE8joS7d2OPPu7fKOSPYP99QQIwDpwEdM8 u4Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=F2I2TK+aOOOMvAeLdBzSnYExWrc0Wlkd9b0mCXwbFlw=; b=OrGD32pxBwBUq7IVO71aVkBywMrEl+tckDk+CdK/upyDIhRdcbdrhqrADveDNBkdhW FppkP6nZpxKBMZjtDEcs57PsfLc1Vwj0QwgMmgxwFHekPOjH6ZRf8Q00+eHvJTkmLvUb XGS0XeQ1/Fw1tb11N4MrXaW0JWuzOWq/T0ZC5UeyXpXXtNTLl++acD5eoB5fRV6SjLwR ekGRapBvsWsjvVTxAYGZe4XhdWvi9SNbvEXeg29wSwhjkQCFQhhsIgdGsfzjyn1LP1G+ uoCj54R8wsqBocInizzM/IJxOcUztOwaq+tv3iiH0BDAK8CamhzIQ2+atrL+ZhiSKfP0 1MBA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.9]) by gmr-mx.google.com with ESMTPS id f25si359728lfj.4.2021.11.14.09.27.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Nov 2021 09:27:50 -0800 (PST) Received-SPF: neutral (google.com: 212.18.0.9 is neither permitted nor denied by domain of sbabic@denx.de) client-ip=212.18.0.9; Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4HsfRB0xtWz1qwdP; Sun, 14 Nov 2021 18:27:50 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4HsfRB0fYMz1qqkB; Sun, 14 Nov 2021 18:27:50 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id ky0sutGgoMCG; Sun, 14 Nov 2021 18:27:48 +0100 (CET) Received: from babic.homelinux.org (host-88-217-136-221.customer.m-online.net [88.217.136.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Sun, 14 Nov 2021 18:27:48 +0100 (CET) Received: from localhost (mail.babic.homelinux.org [127.0.0.1]) by babic.homelinux.org (Postfix) with ESMTP id 28B6545429A3; Sun, 14 Nov 2021 18:27:48 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at babic.homelinux.org Received: from babic.homelinux.org ([127.0.0.1]) by localhost (mail.babic.homelinux.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id psEarElzhmxK; Sun, 14 Nov 2021 18:27:44 +0100 (CET) Received: from paperino.fritz.box (paperino.fritz.box [192.168.178.48]) by babic.homelinux.org (Postfix) with ESMTP id B561B4540853; Sun, 14 Nov 2021 18:27:44 +0100 (CET) From: Stefano Babic To: swupdate@googlegroups.com Cc: Stefano Babic Subject: [swupdate] [PATCH V2 00/36] DELTA Update Date: Sun, 14 Nov 2021 18:26:57 +0100 Message-Id: <20211114172733.71602-1-sbabic@denx.de> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Original-Sender: sbabic@denx.de X-Original-Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by domain of sbabic@denx.de) smtp.mailfrom=sbabic@denx.de Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , This is a major feature and adds supports for binary delta update. This work is sponsored by Siemens CT (Munich) - many thanks to Siemens that made this development possible !!! The patchset is split into several parts: - Doc for delta update: this describes shortly the design, I plan to describe with more details later. - Add generic functions used later in delta handler, but that can be reused by other code. - rework channel and channel_curl to make it SWUpdate's unaware, that is the downloaded data can be forwarded to another process and not just to the installer. Add some more features, like downloading HTTP Range Requests. - Delta Handler, split into the downloader process and the delta handler itself. - Last patch is just an optimization, useful in case a filesystem is put into a partition much more larger. This allows to index just a part of the whole partition and to reduce memory consumption. This series fixes comments fron ML and rebased after requested PR for Zchunk was merged. See discussion at: https://github.com/zchunk/zchunk/pull/51 The serie is linked with current TOT of Zchunk project. Test: ----- I tested on eMMC / SD cards using the raw and rawfile as chained handler. But from design, any handler can be used to install the resulting artifact. Changes since V1: ----------------- - rebased on Zchunk TOT. Use own code to compute Byte Range request. - Check for file descriptor should be >= 0 instead of > 0 - Names of zchunk functions were changed (see PR for zchunk project) - fix coverity issues - check for OOM in multiparser library Stefano Babic (36): Doc for delta update Handlers: sort list of handlers in menuconfig Add utility function to detect filesystem util: add safe version for realloc() util: add function to convert string to lowercase Hide copyfile() implementation to add more input Introduce copybuffer to copy from memory Import small multipart library Fix warning in multipart code Check return from malloc in multiparser library Convert multiparser to SWUpdate codestyle channel_curl: statify entry points functions channel_curl: fix wrong usage of pointer in assert channel_curl: do not automatically add charset header curl: change signature of write data callback curl: add a noipc parameter channel_curl: pass channel_data to headers callback channel_curl: allow an external callback for headers channel_curl: add optional pointer for callbacks channel_curl: pass channel pointer to internal callbacks channel_curl: add the possibility to request a range channel_curl: do not check download size if range requested channel_curl: store HTTP return code before callbacks Be sure to initialize channel_data_t by users server_general: fix warnings delta: add process to download chunks tools: fix warning due to new SOURCE type example: userid and groupid in case of downloader Start chunks downloader if delta is enabled Introduce own version of zckRange from zchunk delta: add handler for delta update doc: add documentation for delta handler doc: drop delta update from roadmap delta: add the option to limit size for source all_handlers_defconfig: add delta and unique Updated roadmap Kconfig | 4 + Makefile.deps | 4 + Makefile.flags | 5 + configs/all_handlers_defconfig | 17 +- core/cpio_utils.c | 88 ++- core/swupdate.c | 12 + core/util.c | 24 + corelib/Makefile | 1 + corelib/channel_curl.c | 178 +++-- corelib/downloader.c | 5 +- corelib/multipart_parser.c | 318 ++++++++ doc/source/delta-update.rst | 223 ++++++ doc/source/handlers.rst | 86 +++ doc/source/index.rst | 1 + doc/source/roadmap.rst | 80 +- examples/configuration/swupdate.cfg | 6 + fs/diskformat.c | 28 +- handlers/Config.in | 268 ++++--- handlers/Makefile | 1 + handlers/delta_downloader.c | 217 ++++++ handlers/delta_handler.c | 1110 +++++++++++++++++++++++++++ handlers/delta_handler.h | 37 + handlers/zchunk_range.c | 187 +++++ handlers/zchunk_range.h | 41 + include/channel_curl.h | 8 +- include/delta_process.h | 10 + include/fs_interface.h | 1 + include/multipart_parser.h | 49 ++ include/swupdate_status.h | 3 +- include/util.h | 4 + suricatta/server_general.c | 8 +- suricatta/server_hawkbit.c | 28 +- tools/swupdate-ipc.c | 2 + tools/swupdate-progress.c | 3 + 34 files changed, 2785 insertions(+), 272 deletions(-) create mode 100644 corelib/multipart_parser.c create mode 100644 doc/source/delta-update.rst create mode 100644 handlers/delta_downloader.c create mode 100644 handlers/delta_handler.c create mode 100644 handlers/delta_handler.h create mode 100644 handlers/zchunk_range.c create mode 100644 handlers/zchunk_range.h create mode 100644 include/delta_process.h create mode 100644 include/multipart_parser.h