[PULL,v4,00/41] Migration cleanups, refactorings, stats, and more

Hi

v4: 20121017:
- fix linux-user compilation (Anthony found it)

Anthony, please pull.

Thanks, Juan.

The following changes since commit 6f4d6b09088ee161ff4be0e4db4e4c0962c79070:

  target-mips: Pass MIPSCPU to mips_vpe_sleep() (2012-10-17 01:32:11 +0200)

are available in the git repository at:

  http://repo.or.cz/r/qemu/quintela.git migration-next-20121017

for you to fetch changes up to aa723c23147e93fef8475bd80fd29e633378c34d:

  cpus: create qemu_in_vcpu_thread() (2012-10-17 18:35:00 +0200)


v3: 20121010:
- Add a reviwed-by: Paolo
- fix whitespace on comment by Eric

v2: (20121001)
- fix qemu_in_vcpu_thread: marcelo
- fix "fix migration sync on setup"
- addressed all marcelo comments

Please, review.


v1: (20120921)

This is the mergable part of migration thread work that I am doing.
What it does:
- cleanups left and right
- 2nd patch is a (fix migration sync) should be applied to stable IMHO
   Othrewise, we "could" be sending zero pages 2 times.
- It introduces the stats discused previously
   * expected_downtime
   * downtime
   * dirty_pages_rate
     All Eric & Luiz comments have been addressed
- buffered file has been basically unfolded.  It was used only once,
  with one user.  So we de-abstracted it.
  Notice that with migration patches, it makes no sense anymore
- migration bitmap

  Now the bitmap is 1bit per page.  Notice that this only affects to
  migration.  For the rest (aka TCG & VGA), we still use the old bitmap.

- remove qemu_file* functions that were not used or used only once
- last_error: we have made that almost all error paths return one error
  instead of zero, so we use much, much less the error on the qemu_file.
  Some of the last remmants are:
  * still there due to callbacks (can be removed when the thread is
    integrated)
  * are on the read path (I haven't touched a lot of those yet)
- qemu_cpu_is_vcpu()

  We used to test if we were _not_ an vcpu testing if we were the
  iothread.  With the migration thread that is not true anymore.  So
  create a function that does the right thing.

Please review.

Thanks, Juan.

Juan Quintela (40):
  buffered_file: g_realloc() can't fail
  fix migration sync
  migration: store end_time in a local variable
  migration: print total downtime for final phase of migration
  migration: rename expected_time to expected_downtime
  migration: export migrate_get_current()
  migration: print expected downtime in info migrate
  savevm: Factorize ram globals reset in its own function
  ram: introduce migration_bitmap_set_dirty()
  ram: Introduce migration_bitmap_test_and_reset_dirty()
  ram: Export last_ram_offset()
  ram: introduce migration_bitmap_sync()
  ram: create trace event for migration sync bitmap
  Separate migration bitmap
  migration: Add dirty_pages_rate to query migrate output
  buffered_file: rename opaque to migration_state
  buffered_file: opaque is MigrationState
  buffered_file: unfold migrate_fd_put_buffer
  buffered_file: unfold migrate_fd_put_ready
  buffered_file: unfold migrate_fd_wait_for_unfreeze
  buffered_file: unfold migrate_fd_close
  buffered_file: We can access directly to bandwidth_limit
  buffered_file: callers of buffered_flush() already check for errors
  buffered_file: make buffered_flush return the error code
  migration: make migrate_fd_wait_for_unfreeze() return errors
  savevm: unexport qemu_fflush
  virtio-net: use qemu_get_buffer() in a temp buffer
  savevm: Remove qemu_fseek()
  savevm: make qemu_fflush() return an error code
  savevm: unfold qemu_fclose_internal()
  savevm: unexport qemu_ftell()
  savevm: make qemu_fill_buffer() be consistent
  savevm: Only qemu_fflush() can generate errors
  buffered_file: buffered_put_buffer() don't need to set last_error
  block-migration: make flush_blks() return errors
  block-migration:  Switch meaning of return value
  block-migration: handle errors with the return codes correctly
  savevm: un-export qemu_file_set_error()
  savevm: make qemu_file_put_notify() return errors
  cpus: create qemu_in_vcpu_thread()

Paolo Bonzini (1):
  BufferedFile: append, then flush

 arch_init.c       | 141 +++++++++++++++++++++++++++++++++++++++++-------------
 block-migration.c |  56 +++++++++++-----------
 buffered_file.c   | 129 ++++++++++++++++++-------------------------------
 buffered_file.h   |  12 +----
 cpu-all.h         |   2 +-
 cpus.c            |   9 +++-
 exec-obsolete.h   |  10 ----
 exec.c            |   2 +-
 hmp.c             |  12 +++++
 hw/virtio-net.c   |   4 +-
 migration.c       |  46 +++++++++---------
 migration.h       |  10 ++++
 qapi-schema.json  |  20 ++++++--
 qemu-file.h       |   8 +---
 qmp-commands.hx   |   9 ++++
 savevm.c          | 117 +++++++++++++++++---------------------------
 trace-events      |   4 ++
 17 files changed, 319 insertions(+), 272 deletions(-)

Juan Quintela <quintela@redhat.com> writes:

> Hi
>
> v4: 20121017:
> - fix linux-user compilation (Anthony found it)
>
> Anthony, please pull.
>

Pulled. Thanks.

Regards,

Anthony Liguori

> Thanks, Juan.
>
> The following changes since commit 6f4d6b09088ee161ff4be0e4db4e4c0962c79070:
>
>   target-mips: Pass MIPSCPU to mips_vpe_sleep() (2012-10-17 01:32:11 +0200)
>
> are available in the git repository at:
>
>   http://repo.or.cz/r/qemu/quintela.git migration-next-20121017
>
> for you to fetch changes up to aa723c23147e93fef8475bd80fd29e633378c34d:
>
>   cpus: create qemu_in_vcpu_thread() (2012-10-17 18:35:00 +0200)
>
>
> v3: 20121010:
> - Add a reviwed-by: Paolo
> - fix whitespace on comment by Eric
>
> v2: (20121001)
> - fix qemu_in_vcpu_thread: marcelo
> - fix "fix migration sync on setup"
> - addressed all marcelo comments
>
> Please, review.
>
>
> v1: (20120921)
>
> This is the mergable part of migration thread work that I am doing.
> What it does:
> - cleanups left and right
> - 2nd patch is a (fix migration sync) should be applied to stable IMHO
>    Othrewise, we "could" be sending zero pages 2 times.
> - It introduces the stats discused previously
>    * expected_downtime
>    * downtime
>    * dirty_pages_rate
>      All Eric & Luiz comments have been addressed
> - buffered file has been basically unfolded.  It was used only once,
>   with one user.  So we de-abstracted it.
>   Notice that with migration patches, it makes no sense anymore
> - migration bitmap
>
>   Now the bitmap is 1bit per page.  Notice that this only affects to
>   migration.  For the rest (aka TCG & VGA), we still use the old bitmap.
>
> - remove qemu_file* functions that were not used or used only once
> - last_error: we have made that almost all error paths return one error
>   instead of zero, so we use much, much less the error on the qemu_file.
>   Some of the last remmants are:
>   * still there due to callbacks (can be removed when the thread is
>     integrated)
>   * are on the read path (I haven't touched a lot of those yet)
> - qemu_cpu_is_vcpu()
>
>   We used to test if we were _not_ an vcpu testing if we were the
>   iothread.  With the migration thread that is not true anymore.  So
>   create a function that does the right thing.
>
> Please review.
>
> Thanks, Juan.
>
> Juan Quintela (40):
>   buffered_file: g_realloc() can't fail
>   fix migration sync
>   migration: store end_time in a local variable
>   migration: print total downtime for final phase of migration
>   migration: rename expected_time to expected_downtime
>   migration: export migrate_get_current()
>   migration: print expected downtime in info migrate
>   savevm: Factorize ram globals reset in its own function
>   ram: introduce migration_bitmap_set_dirty()
>   ram: Introduce migration_bitmap_test_and_reset_dirty()
>   ram: Export last_ram_offset()
>   ram: introduce migration_bitmap_sync()
>   ram: create trace event for migration sync bitmap
>   Separate migration bitmap
>   migration: Add dirty_pages_rate to query migrate output
>   buffered_file: rename opaque to migration_state
>   buffered_file: opaque is MigrationState
>   buffered_file: unfold migrate_fd_put_buffer
>   buffered_file: unfold migrate_fd_put_ready
>   buffered_file: unfold migrate_fd_wait_for_unfreeze
>   buffered_file: unfold migrate_fd_close
>   buffered_file: We can access directly to bandwidth_limit
>   buffered_file: callers of buffered_flush() already check for errors
>   buffered_file: make buffered_flush return the error code
>   migration: make migrate_fd_wait_for_unfreeze() return errors
>   savevm: unexport qemu_fflush
>   virtio-net: use qemu_get_buffer() in a temp buffer
>   savevm: Remove qemu_fseek()
>   savevm: make qemu_fflush() return an error code
>   savevm: unfold qemu_fclose_internal()
>   savevm: unexport qemu_ftell()
>   savevm: make qemu_fill_buffer() be consistent
>   savevm: Only qemu_fflush() can generate errors
>   buffered_file: buffered_put_buffer() don't need to set last_error
>   block-migration: make flush_blks() return errors
>   block-migration:  Switch meaning of return value
>   block-migration: handle errors with the return codes correctly
>   savevm: un-export qemu_file_set_error()
>   savevm: make qemu_file_put_notify() return errors
>   cpus: create qemu_in_vcpu_thread()
>
> Paolo Bonzini (1):
>   BufferedFile: append, then flush
>
>  arch_init.c       | 141 +++++++++++++++++++++++++++++++++++++++++-------------
>  block-migration.c |  56 +++++++++++-----------
>  buffered_file.c   | 129 ++++++++++++++++++-------------------------------
>  buffered_file.h   |  12 +----
>  cpu-all.h         |   2 +-
>  cpus.c            |   9 +++-
>  exec-obsolete.h   |  10 ----
>  exec.c            |   2 +-
>  hmp.c             |  12 +++++
>  hw/virtio-net.c   |   4 +-
>  migration.c       |  46 +++++++++---------
>  migration.h       |  10 ++++
>  qapi-schema.json  |  20 ++++++--
>  qemu-file.h       |   8 +---
>  qmp-commands.hx   |   9 ++++
>  savevm.c          | 117 +++++++++++++++++---------------------------
>  trace-events      |   4 ++
>  17 files changed, 319 insertions(+), 272 deletions(-)
>
> -- 
> 1.7.11.7