[v3,00/21] Multifd Migration Compression

[v3]
- rebased on top of upstream + previous multifd cancel series
- split multifd code into its own file (multifd.[ch])
- split zstd/zlib compression methods (multifd-zstd/zlib.c)
- use qemu module feauture to avoid ifdefs
  (my understanding is that zlib needs to be present, but
  we setup zstd only if it is not there or is disabled)
- multifd-method: none|zlib|zstd

  As far as I can see, there is no easy way to convince qapi that zstd
  option could/couldn't be there depending on compliation flags. I
  ended just checking in migrate_parameters_check() if it is enabled
  and giving an error message otherwise.

Questions:
- I am "reusing" the compress-level parameter for both zstd and zlib,
  but it poses a problem:
  * zlib values: 1-9 (default: 6?)
  * zstd values: 1-19 (default: 3)
So, what should I do:
  * create multifd-zstd-level and multifd-zlib-level (easier)
  * reuse compress-level, and change its maximum values depending on
    multifd-method
  * any other good option?

Please, review.

[v2] - rebase on top of previous arguments posted to the list -
introduces zlib compression - introduces zstd compression

Please help if you know anything about zstd/zlib compression.

This puts compression on top of multifd. Advantages about current
compression:

- We copy all pages in a single packet and then compress the whole
  thing.

- We reuse the compression stream for all the packets sent through the
  same channel.

- We can select nocomp/zlib/zstd levels of compression.

Please, review.

Juan Quintela (21):
  migration-test: Use g_free() instead of free()
  multifd: Make sure that we don't do any IO after an error
  qemu-file: Don't do IO after shutdown
  migration-test: Make sure that multifd and cancel works
  migration: Create migration_is_running()
  migration: Don't send data if we have stopped
  migration: Make multifd_save_setup() get an Error parameter
  migration: Make multifd_load_setup() get an Error parameter
  migration: Add multifd-compress parameter
  ram_addr: Split RAMBlock definition
  multifd: multifd_send_pages only needs the qemufile
  multifd: multifd_queue_page only needs the qemufile
  multifd: multifd_send_sync_main only needs the qemufile
  multifd: Use qemu_target_page_size()
  migration: Make checkpatch happy with comments
  migration: Add support for modules
  multifd: Split multifd code into its own file
  migration: Make no compression operations into its own structure
  migration: Add zlib compression multifd support
  configure: Enable test and libs for zstd
  migration: Add zstd compression multifd support

 MAINTAINERS                  |    1 +
 configure                    |   30 +
 hw/core/qdev-properties.c    |   13 +
 include/exec/ram_addr.h      |   40 +-
 include/exec/ramblock.h      |   64 ++
 include/hw/qdev-properties.h |    3 +
 include/qemu/module.h        |    2 +
 migration/Makefile.objs      |    3 +
 migration/migration.c        |   97 +++-
 migration/migration.h        |    4 +-
 migration/multifd-zlib.c     |  289 +++++++++
 migration/multifd-zstd.c     |  304 ++++++++++
 migration/multifd.c          | 1064 ++++++++++++++++++++++++++++++++++
 migration/multifd.h          |  185 ++++++
 migration/qemu-file.c        |   22 +-
 migration/ram.c              | 1006 +-------------------------------
 migration/ram.h              |    7 -
 migration/rdma.c             |    2 +-
 migration/savevm.c           |    4 +-
 monitor/hmp-cmds.c           |   13 +
 qapi/migration.json          |   30 +-
 tests/qtest/migration-test.c |  142 ++++-
 vl.c                         |    1 +
 23 files changed, 2266 insertions(+), 1060 deletions(-)
 create mode 100644 include/exec/ramblock.h
 create mode 100644 migration/multifd-zlib.c
 create mode 100644 migration/multifd-zstd.c
 create mode 100644 migration/multifd.c
 create mode 100644 migration/multifd.h

Juan Quintela <quintela@redhat.com> wrote:
> [v3]
> - rebased on top of upstream + previous multifd cancel series
> - split multifd code into its own file (multifd.[ch])
> - split zstd/zlib compression methods (multifd-zstd/zlib.c)
> - use qemu module feauture to avoid ifdefs
>   (my understanding is that zlib needs to be present, but
>   we setup zstd only if it is not there or is disabled)
> - multifd-method: none|zlib|zstd
>
>   As far as I can see, there is no easy way to convince qapi that zstd
>   option could/couldn't be there depending on compliation flags. I
>   ended just checking in migrate_parameters_check() if it is enabled
>   and giving an error message otherwise.
>
> Questions:
> - I am "reusing" the compress-level parameter for both zstd and zlib,
>   but it poses a problem:
>   * zlib values: 1-9 (default: 6?)
>   * zstd values: 1-19 (default: 3)
> So, what should I do:
>   * create multifd-zstd-level and multifd-zlib-level (easier)
>   * reuse compress-level, and change its maximum values depending on
>     multifd-method
>   * any other good option?
>
> Please, review.
>
> [v2] - rebase on top of previous arguments posted to the list -
> introduces zlib compression - introduces zstd compression
>
> Please help if you know anything about zstd/zlib compression.
>
> This puts compression on top of multifd. Advantages about current
> compression:
>
> - We copy all pages in a single packet and then compress the whole
>   thing.
>
> - We reuse the compression stream for all the packets sent through the
>   same channel.
>
> - We can select nocomp/zlib/zstd levels of compression.
>
> Please, review.
>
> Juan Quintela (21):
>   migration-test: Use g_free() instead of free()
>   multifd: Make sure that we don't do any IO after an error
>   qemu-file: Don't do IO after shutdown
>   migration-test: Make sure that multifd and cancel works
>   migration: Create migration_is_running()
>   migration: Don't send data if we have stopped

This patches are from my previous series.  I forgot to pass the -b
option to git-publish.  Please ignore them.

Sorry, Juan.

Juan Quintela <quintela@redhat.com> writes:

> [v3]
> - rebased on top of upstream + previous multifd cancel series
> - split multifd code into its own file (multifd.[ch])
> - split zstd/zlib compression methods (multifd-zstd/zlib.c)
> - use qemu module feauture to avoid ifdefs
>   (my understanding is that zlib needs to be present, but
>   we setup zstd only if it is not there or is disabled)
> - multifd-method: none|zlib|zstd
>
>   As far as I can see, there is no easy way to convince qapi that zstd
>   option could/couldn't be there depending on compliation flags. I
>   ended just checking in migrate_parameters_check() if it is enabled
>   and giving an error message otherwise.

Wild guess: this is about PATCH 21's

   diff --git a/qapi/migration.json b/qapi/migration.json
   index 1714ea51e3..65db85970e 100644
   --- a/qapi/migration.json
   +++ b/qapi/migration.json
   @@ -499,7 +499,7 @@
    #
    ##
    { 'enum': 'MultifdCompress',
   -  'data': [ 'none', 'zlib' ] }
   +  'data': [ 'none', 'zlib', 'zstd' ] }

    ##
    # @MigrationParameter:

where you want 'zstd' to be #ifdef CONFIG_ZSTD.  If not, please advise.
Else use something like

    { 'enum': 'MultifdCompress',
      'data': [ 'none', 'zlib',
                { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }

See docs/devel/qapi-code-gen.txt sections "Enumeration types" and
"Configuring the schema".

[...]