[00/34] migration thread and queue

Hi

This is my queue for migration-thread and patches associated.  This
integrates review comments & code for Paolo.  This is the subset from
both approachs that we agreed with. rest of patches need more review
and are not here.

Migrating and idle guest with upstwream:

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 34251 milliseconds
downtime: 492 milliseconds
transferred ram: 762458 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3492606 pages
normal: 189762 pages
normal bytes: 759048 kbytes

with this series of patches.

(qemu) info migrate
capabilities: xbzrle: off
Migration status: completed
total time: 30712 milliseconds
downtime: 29 milliseconds
transferred ram: 738857 kbytes
remaining ram: 0 kbytes
total ram: 14688768 kbytes
duplicate: 3503423 pages
normal: 176671 pages
normal bytes: 706684 kbytes

Notice the big difference in downtime.  And that is also seen inside
the guest a program that just do an idle loop seeing how "long" it
takes to wait for 10ms.

with upstream:

[root@d1 ~]# ./timer
delay of 452 ms
delay of 114 ms
delay of 136 ms
delay of 135 ms
delay of 136 ms
delay of 131 ms
delay of 134 ms

with this series of patches, wait never takes 100ms, nothing is printed.

Please review.

Thanks, Juan.


The following changes since commit 914606d26e654d4c01bd5186f4d05e3fd445e219:

  Merge remote-tracking branch 'stefanha/trivial-patches' into staging (2012-12-18 15:41:43 -0600)

are available in the git repository at:


  git://repo.or.cz/qemu/quintela.git thread.next

for you to fetch changes up to 065bebbacf54dcad48aad42112417dbb44451499:

  migration: merge QEMUFileBuffered into MigrationState (2012-12-19 13:21:29 +0100)

----------------------------------------------------------------
Juan Quintela (25):
      migration: include qemu-file.h
      migration-fd: remove duplicate include
      buffered_file: Move from using a timer to use a thread
      migration: make qemu_fopen_ops_buffered() return void
      migration: move migration thread init code to migrate_fd_put_ready
      migration: make writes blocking
      migration: remove unfreeze logic
      migration: just lock migrate_fd_put_ready
      buffered_file: Unfold the trick to restart generating migration data
      buffered_file: don't flush on put buffer
      buffered_file: unfold buffered_append in buffered_put_buffer
      savevm: New save live migration method: pending
      migration: move buffered_file.c code into migration.c
      migration: add XFER_LIMIT_RATIO
      migration: move migration_fd_put_ready()
      migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
      migration: move migration notifier
      ram: rename last_block to last_seen_block
      ram: Add last_sent_block
      memory: introduce memory_region_test_and_clear_dirty
      ram: Use memory_region_test_and_clear_dirty
      ram: optimize migration bitmap walking
      ram: account the amount of transferred ram better
      ram: refactor ram_save_block() return value
      migration: merge QEMUFileBuffered into MigrationState

Paolo Bonzini (7):
      migration: fix migration_bitmap leak
      buffered_file: do not send more than s->bytes_xfer bytes per tick
      migration: remove double call to migrate_fd_close
      exec: change ramlist from MRU order to a 1-item cache
      exec: change RAM list to a TAILQ
      exec: sort the memory from biggest to smallest
      migration: fix qemu_get_fd for BufferedFile

Umesh Deshpande (2):
      add a version number to ram_list
      protect the ramlist with a separate mutex

 Makefile.objs           |   2 +-
 arch_init.c             | 245 +++++++++++++++---------------
 block-migration.c       |  49 ++----
 buffered_file.c         | 269 ---------------------------------
 buffered_file.h         |  22 ---
 cpu-all.h               |  15 +-
 dump.c                  |   8 +-
 exec.c                  | 128 +++++++++++-----
 memory.c                |  16 ++
 memory.h                |  16 ++
 memory_mapping.c        |   4 +-
 migration-exec.c        |   3 +-
 migration-fd.c          |   4 +-
 migration-tcp.c         |   3 +-
 migration-unix.c        |   3 +-
 migration.c             | 390 ++++++++++++++++++++++++++++++++++++------------
 migration.h             |  13 +-
 qemu-file.h             |   5 -
 savevm.c                |  24 ++-
 sysemu.h                |   1 +
 target-i386/arch_dump.c |   2 +-
 vmstate.h               |   1 +
 22 files changed, 599 insertions(+), 624 deletions(-)
 delete mode 100644 buffered_file.c
 delete mode 100644 buffered_file.h

Il 19/12/2012 13:33, Juan Quintela ha scritto:
> Hi
> 
> This is my queue for migration-thread and patches associated.  This
> integrates review comments & code for Paolo.  This is the subset from
> both approachs that we agreed with. rest of patches need more review
> and are not here.
> 
> Migrating and idle guest with upstwream:
> 
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 34251 milliseconds
> downtime: 492 milliseconds
> transferred ram: 762458 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3492606 pages
> normal: 189762 pages
> normal bytes: 759048 kbytes
> 
> with this series of patches.
> 
> (qemu) info migrate
> capabilities: xbzrle: off
> Migration status: completed
> total time: 30712 milliseconds
> downtime: 29 milliseconds
> transferred ram: 738857 kbytes
> remaining ram: 0 kbytes
> total ram: 14688768 kbytes
> duplicate: 3503423 pages
> normal: 176671 pages
> normal bytes: 706684 kbytes
> 
> Notice the big difference in downtime.  And that is also seen inside
> the guest a program that just do an idle loop seeing how "long" it
> takes to wait for 10ms.
> 
> with upstream:
> 
> [root@d1 ~]# ./timer
> delay of 452 ms
> delay of 114 ms
> delay of 136 ms
> delay of 135 ms
> delay of 136 ms
> delay of 131 ms
> delay of 134 ms
> 
> with this series of patches, wait never takes 100ms, nothing is printed.
> 
> Please review.
> 
> Thanks, Juan.
> 
> 
> The following changes since commit 914606d26e654d4c01bd5186f4d05e3fd445e219:
> 
>   Merge remote-tracking branch 'stefanha/trivial-patches' into staging (2012-12-18 15:41:43 -0600)
> 
> are available in the git repository at:
> 
> 
>   git://repo.or.cz/qemu/quintela.git thread.next
> 
> for you to fetch changes up to 065bebbacf54dcad48aad42112417dbb44451499:
> 
>   migration: merge QEMUFileBuffered into MigrationState (2012-12-19 13:21:29 +0100)
> 
> ----------------------------------------------------------------
> Juan Quintela (25):
>       migration: include qemu-file.h
>       migration-fd: remove duplicate include
>       buffered_file: Move from using a timer to use a thread
>       migration: make qemu_fopen_ops_buffered() return void
>       migration: move migration thread init code to migrate_fd_put_ready
>       migration: make writes blocking
>       migration: remove unfreeze logic
>       migration: just lock migrate_fd_put_ready
>       buffered_file: Unfold the trick to restart generating migration data
>       buffered_file: don't flush on put buffer
>       buffered_file: unfold buffered_append in buffered_put_buffer
>       savevm: New save live migration method: pending
>       migration: move buffered_file.c code into migration.c
>       migration: add XFER_LIMIT_RATIO
>       migration: move migration_fd_put_ready()
>       migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
>       migration: move migration notifier
>       ram: rename last_block to last_seen_block
>       ram: Add last_sent_block
>       memory: introduce memory_region_test_and_clear_dirty
>       ram: Use memory_region_test_and_clear_dirty
>       ram: optimize migration bitmap walking
>       ram: account the amount of transferred ram better
>       ram: refactor ram_save_block() return value
>       migration: merge QEMUFileBuffered into MigrationState
> 
> Paolo Bonzini (7):
>       migration: fix migration_bitmap leak
>       buffered_file: do not send more than s->bytes_xfer bytes per tick
>       migration: remove double call to migrate_fd_close
>       exec: change ramlist from MRU order to a 1-item cache
>       exec: change RAM list to a TAILQ
>       exec: sort the memory from biggest to smallest
>       migration: fix qemu_get_fd for BufferedFile
> 
> Umesh Deshpande (2):
>       add a version number to ram_list
>       protect the ramlist with a separate mutex
> 
>  Makefile.objs           |   2 +-
>  arch_init.c             | 245 +++++++++++++++---------------
>  block-migration.c       |  49 ++----
>  buffered_file.c         | 269 ---------------------------------
>  buffered_file.h         |  22 ---
>  cpu-all.h               |  15 +-
>  dump.c                  |   8 +-
>  exec.c                  | 128 +++++++++++-----
>  memory.c                |  16 ++
>  memory.h                |  16 ++
>  memory_mapping.c        |   4 +-
>  migration-exec.c        |   3 +-
>  migration-fd.c          |   4 +-
>  migration-tcp.c         |   3 +-
>  migration-unix.c        |   3 +-
>  migration.c             | 390 ++++++++++++++++++++++++++++++++++++------------
>  migration.h             |  13 +-
>  qemu-file.h             |   5 -
>  savevm.c                |  24 ++-
>  sysemu.h                |   1 +
>  target-i386/arch_dump.c |   2 +-
>  vmstate.h               |   1 +
>  22 files changed, 599 insertions(+), 624 deletions(-)
>  delete mode 100644 buffered_file.c
>  delete mode 100644 buffered_file.h
> 
> 

Yay.

Acked-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo