mbox series

[V2,00/36] DELTA Update

Message ID 20211114172733.71602-1-sbabic@denx.de
Headers show
Series DELTA Update | expand

Message

Stefano Babic Nov. 14, 2021, 5:26 p.m. UTC 
This is a major feature and adds supports for binary delta
update. This work is sponsored by Siemens CT (Munich) - many thanks
to Siemens that made this development possible !!!

The patchset is split into several parts:

- Doc for delta update: this describes shortly the design,
  I plan to describe with more details later.
- Add generic functions used later in delta handler,
  but that can be reused by other code.
- rework channel and channel_curl to make it SWUpdate's unaware,
  that is the downloaded data can be forwarded to another process
  and not just to the installer. Add some more features, like
  downloading HTTP Range Requests.
- Delta Handler, split into the downloader process and the delta
  handler itself.
- Last patch is just an optimization, useful in case a filesystem
  is put into a partition much more larger. This allows to index
  just a part of the whole partition and to reduce memory consumption.

This series fixes comments fron ML and rebased after requested PR for Zchunk was merged. See discussion at:

https://github.com/zchunk/zchunk/pull/51

The serie is linked with current TOT of Zchunk project.

Test:
-----

I tested on eMMC / SD cards using the raw and rawfile as chained handler.
But from design, any handler can be used to install the resulting artifact.

Changes since V1:
-----------------

- rebased on Zchunk TOT. Use own code to compute Byte Range request.
- Check for file descriptor should be >= 0 instead of > 0
- Names of zchunk functions were changed (see PR for zchunk project)
- fix coverity issues
- check for OOM in multiparser library


Stefano Babic (36):
  Doc for delta update
  Handlers: sort list of handlers in menuconfig
  Add utility function to detect filesystem
  util: add safe version for realloc()
  util: add function to convert string to lowercase
  Hide copyfile() implementation to add more input
  Introduce copybuffer to copy from memory
  Import small multipart library
  Fix warning in multipart code
  Check return from malloc in multiparser library
  Convert multiparser to SWUpdate codestyle
  channel_curl: statify entry points functions
  channel_curl: fix wrong usage of pointer in assert
  channel_curl: do not automatically add charset header
  curl: change signature of write data callback
  curl: add a noipc parameter
  channel_curl: pass channel_data to headers callback
  channel_curl: allow an external callback for headers
  channel_curl: add optional pointer for callbacks
  channel_curl: pass channel pointer to internal callbacks
  channel_curl: add the possibility to request a range
  channel_curl: do not check download size if range requested
  channel_curl: store HTTP return code before callbacks
  Be sure to initialize channel_data_t by users
  server_general: fix warnings
  delta: add process to download chunks
  tools: fix warning due to new SOURCE type
  example: userid and groupid in case of downloader
  Start chunks downloader if delta is enabled
  Introduce own version of zckRange from zchunk
  delta: add handler for delta update
  doc: add documentation for delta handler
  doc: drop delta update from roadmap
  delta: add the option to limit size for source
  all_handlers_defconfig: add delta and unique
  Updated roadmap

 Kconfig                             |    4 +
 Makefile.deps                       |    4 +
 Makefile.flags                      |    5 +
 configs/all_handlers_defconfig      |   17 +-
 core/cpio_utils.c                   |   88 ++-
 core/swupdate.c                     |   12 +
 core/util.c                         |   24 +
 corelib/Makefile                    |    1 +
 corelib/channel_curl.c              |  178 +++--
 corelib/downloader.c                |    5 +-
 corelib/multipart_parser.c          |  318 ++++++++
 doc/source/delta-update.rst         |  223 ++++++
 doc/source/handlers.rst             |   86 +++
 doc/source/index.rst                |    1 +
 doc/source/roadmap.rst              |   80 +-
 examples/configuration/swupdate.cfg |    6 +
 fs/diskformat.c                     |   28 +-
 handlers/Config.in                  |  268 ++++---
 handlers/Makefile                   |    1 +
 handlers/delta_downloader.c         |  217 ++++++
 handlers/delta_handler.c            | 1110 +++++++++++++++++++++++++++
 handlers/delta_handler.h            |   37 +
 handlers/zchunk_range.c             |  187 +++++
 handlers/zchunk_range.h             |   41 +
 include/channel_curl.h              |    8 +-
 include/delta_process.h             |   10 +
 include/fs_interface.h              |    1 +
 include/multipart_parser.h          |   49 ++
 include/swupdate_status.h           |    3 +-
 include/util.h                      |    4 +
 suricatta/server_general.c          |    8 +-
 suricatta/server_hawkbit.c          |   28 +-
 tools/swupdate-ipc.c                |    2 +
 tools/swupdate-progress.c           |    3 +
 34 files changed, 2785 insertions(+), 272 deletions(-)
 create mode 100644 corelib/multipart_parser.c
 create mode 100644 doc/source/delta-update.rst
 create mode 100644 handlers/delta_downloader.c
 create mode 100644 handlers/delta_handler.c
 create mode 100644 handlers/delta_handler.h
 create mode 100644 handlers/zchunk_range.c
 create mode 100644 handlers/zchunk_range.h
 create mode 100644 include/delta_process.h
 create mode 100644 include/multipart_parser.h