mbox

[PULL,00/26] Migration pull request

Message ID 1435747190-18017-1-git-send-email-quintela@redhat.com
State New
Headers show

Pull-request

git://github.com/juanquintela/qemu.git tags/migration/20150701

Message

Juan Quintela July 1, 2015, 10:39 a.m. UTC
Hi

This series includes:
- rdma fixes by Dave
- rdma memory fix by gonglei
- vmdescription for old machine types (dave)
- fix footers for power (dave)
- migration bitmap extensions (Li)
  just fixed the compilation issues for linux-users
- migration events (me)
- optional secttions (me)
- global  configuration (me)


Please, Apply.


The following changes since commit d2966f804d70a244f5dde395fc5d22a50ed3e74e:

  Merge remote-tracking branch 'remotes/vivier/tags/pull-m68k-20150629' into staging (2015-06-29 17:03:20 +0100)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20150701

for you to fetch changes up to a4fe58b0ea0d78f92461607f4f90be3384fa30e5:

  migration: Add migration events on target side (2015-07-01 12:35:05 +0200)

----------------------------------------------------------------
migration/next for 20150701

----------------------------------------------------------------
Dr. David Alan Gilbert (11):
      Only try and read a VMDescription if it should be there
      rdma typos
      Store block name in local blocks structure
      Translate offsets to destination address space
      Rework ram_control_load_hook to hook during block load
      Allow rdma_delete_block to work without the hash
      Rework ram block hash
      Sort destination RAMBlocks to be the same as the source
      Sanity check RDMA remote data
      Fail more cleanly in mismatched RAM cases
      Fix older machine type compatibility on power with section footers

Gonglei (1):
      rdma: fix memory leak

Juan Quintela (12):
      runstate: Add runstate store
      runstate: migration allows more transitions now
      migration: create new section to store global state
      global_state: Make section optional
      vmstate: Create optional sections
      migration: Add configuration section
      migration: Use cmpxchg correctly
      migration: ensure we start in NONE state
      migration: Use always helper to set state
      migration: No need to call trace_migrate_set_state()
      migration: create migration event
      migration: Add migration events on target side

Li Zhijian (2):
      migration: protect migration_bitmap
      migration: extend migration_bitmap

 docs/qmp/qmp-events.txt       |  14 ++
 exec.c                        |   5 +
 hw/i386/pc_piix.c             |   2 +
 hw/i386/pc_q35.c              |   2 +
 hw/ppc/spapr.c                |   3 +
 include/exec/exec-all.h       |   3 +
 include/migration/migration.h |   6 +-
 include/migration/qemu-file.h |  14 +-
 include/migration/vmstate.h   |   2 +
 include/sysemu/sysemu.h       |   1 +
 migration/migration.c         | 158 ++++++++++++++++++++---
 migration/qemu-file.c         |  16 ++-
 migration/ram.c               |  44 ++++++-
 migration/rdma.c              | 289 ++++++++++++++++++++++++++++++------------
 migration/savevm.c            | 104 +++++++++++++--
 migration/vmstate.c           |  11 ++
 qapi/event.json               |  12 ++
 trace-events                  |  16 ++-
 vl.c                          |  21 ++-
 19 files changed, 597 insertions(+), 126 deletions(-)

Comments

Peter Maydell July 2, 2015, 9:31 a.m. UTC | #1
On 1 July 2015 at 11:39, Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> This series includes:
> - rdma fixes by Dave
> - rdma memory fix by gonglei
> - vmdescription for old machine types (dave)
> - fix footers for power (dave)
> - migration bitmap extensions (Li)
>   just fixed the compilation issues for linux-users
> - migration events (me)
> - optional secttions (me)
> - global  configuration (me)
>
>
> Please, Apply.
>
>
> The following changes since commit d2966f804d70a244f5dde395fc5d22a50ed3e74e:
>
>   Merge remote-tracking branch 'remotes/vivier/tags/pull-m68k-20150629' into staging (2015-06-29 17:03:20 +0100)
>
> are available in the git repository at:
>
>   git://github.com/juanquintela/qemu.git tags/migration/20150701
>
> for you to fetch changes up to a4fe58b0ea0d78f92461607f4f90be3384fa30e5:
>
>   migration: Add migration events on target side (2015-07-01 12:35:05 +0200)
>
> ----------------------------------------------------------------
> migration/next for 20150701

On OSX at least every QEMU executable aborts immediately
with "qemu: qemu_mutex_lock: Invalid argument". Here's a backtrace:

#0  0x00007fff92c29286 in __pthread_kill ()
#1  0x00007fff8a4f342f in pthread_kill ()
#2  0x00007fff9240eb53 in abort ()
#3  0x00000001002b0915 in error_exit (err=<value temporarily
unavailable, due to optimizations>, msg=<value temporarily
unavailable, due to optimizations>) at
/Users/pm215/src/qemu/util/qemu-thread-posix.c:48
#4  0x00000001002b095d in qemu_mutex_lock (mutex=<value temporarily
unavailable, due to optimizations>) at
/Users/pm215/src/qemu/util/qemu-thread-posix.c:75
#5  0x0000000100050f20 in migration_bitmap_extend (old=0, new=32768)
at /Users/pm215/src/qemu/migration/ram.c:1068
#6  0x0000000100002a8d in ram_block_add [inlined] () at
/Users/pm215/src/qemu/exec.c:1407
#7  0x0000000100002a8d in qemu_ram_alloc_internal (size=<value
temporarily unavailable, due to optimizations>, max_size=<value
temporarily unavailable, due to optimizations>, resized=<value
temporarily unavailable, due to optimizations>, host=<value
temporarily unavailable, due to optimizations>, resizeable=false,
mr=0x100e97320, errp=<value temporarily unavailable, due to
optimizations>) at /Users/pm215/src/qemu/exec.c:1537
#8  0x0000000100002e71 in qemu_ram_alloc (size=1823, mr=0x100e97320,
errp=<value temporarily unavailable, due to optimizations>) at
/Users/pm215/src/qemu/exec.c:1554
#9  0x000000010004353f in memory_region_init_ram (mr=0x100e97320,
owner=<value temporarily unavailable, due to optimizations>,
name=<value temporarily unavailable, due to optimizations>,
size=134217728, errp=0x0) at /Users/pm215/src/qemu/memory.c:1216
#10 0x000000010003f10a in allocate_system_memory_nonnuma [inlined] ()
at /Users/pm215/src/qemu/numa.c:339
#11 0x000000010003f10a in memory_region_allocate_system_memory
(mr=0x100e97320, owner=0x6, name=0x0, ram_size=140734799798744) at
/Users/pm215/src/qemu/numa.c:352
#12 0x00000001000781cb in pc_memory_init (machine=0x102198630,
system_memory=0x102199690, below_4g_mem_size=134217728,
above_4g_mem_size=0, rom_memory=0x100e96de0,
ram_memory=0x7fff5fbfe4b8, guest_info=<value temporarily unavailable,
due to optimizations>) at /Users/pm215/src/qemu/hw/i386/pc.c:1254
#13 0x000000010007aae6 in pc_init1 (machine=0x102198630) at
/Users/pm215/src/qemu/hw/i386/pc_piix.c:182
#14 0x00000001000e3cf9 in realtime_init [inlined] () at
/Users/pm215/src/qemu/vl.c:4503
#15 0x00000001000e3cf9 in qemu_main (argc=<value temporarily
unavailable, due to optimizations>, argv=<value temporarily
unavailable, due to optimizations>, envp=0x0) at
/Users/pm215/src/qemu/vl.c:4505
#16 0x00000001002117be in -[QemuCocoaAppController
startEmulationWithArgc:argv:] (self=<value temporarily unavailable,
due to optimizations>, _cmd=<value temporarily unavailable, due to
optimizations>, argc=1823, argv=0x102199690) at
/Users/pm215/src/qemu/ui/cocoa.m:941

A breakpoint on ram_save_setup() is never hit, so it looks
like the problem is the mutex is being used before it is
initialized.

thanks
-- PMM
Wen Congyang July 2, 2015, 9:51 a.m. UTC | #2
On 07/02/2015 05:31 PM, Peter Maydell wrote:
> On 1 July 2015 at 11:39, Juan Quintela <quintela@redhat.com> wrote:
>> Hi
>>
>> This series includes:
>> - rdma fixes by Dave
>> - rdma memory fix by gonglei
>> - vmdescription for old machine types (dave)
>> - fix footers for power (dave)
>> - migration bitmap extensions (Li)
>>   just fixed the compilation issues for linux-users
>> - migration events (me)
>> - optional secttions (me)
>> - global  configuration (me)
>>
>>
>> Please, Apply.
>>
>>
>> The following changes since commit d2966f804d70a244f5dde395fc5d22a50ed3e74e:
>>
>>   Merge remote-tracking branch 'remotes/vivier/tags/pull-m68k-20150629' into staging (2015-06-29 17:03:20 +0100)
>>
>> are available in the git repository at:
>>
>>   git://github.com/juanquintela/qemu.git tags/migration/20150701
>>
>> for you to fetch changes up to a4fe58b0ea0d78f92461607f4f90be3384fa30e5:
>>
>>   migration: Add migration events on target side (2015-07-01 12:35:05 +0200)
>>
>> ----------------------------------------------------------------
>> migration/next for 20150701
> 
> On OSX at least every QEMU executable aborts immediately
> with "qemu: qemu_mutex_lock: Invalid argument". Here's a backtrace:
> 
> #0  0x00007fff92c29286 in __pthread_kill ()
> #1  0x00007fff8a4f342f in pthread_kill ()
> #2  0x00007fff9240eb53 in abort ()
> #3  0x00000001002b0915 in error_exit (err=<value temporarily
> unavailable, due to optimizations>, msg=<value temporarily
> unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/util/qemu-thread-posix.c:48
> #4  0x00000001002b095d in qemu_mutex_lock (mutex=<value temporarily
> unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/util/qemu-thread-posix.c:75
> #5  0x0000000100050f20 in migration_bitmap_extend (old=0, new=32768)
> at /Users/pm215/src/qemu/migration/ram.c:1068
> #6  0x0000000100002a8d in ram_block_add [inlined] () at
> /Users/pm215/src/qemu/exec.c:1407
> #7  0x0000000100002a8d in qemu_ram_alloc_internal (size=<value
> temporarily unavailable, due to optimizations>, max_size=<value
> temporarily unavailable, due to optimizations>, resized=<value
> temporarily unavailable, due to optimizations>, host=<value
> temporarily unavailable, due to optimizations>, resizeable=false,
> mr=0x100e97320, errp=<value temporarily unavailable, due to
> optimizations>) at /Users/pm215/src/qemu/exec.c:1537
> #8  0x0000000100002e71 in qemu_ram_alloc (size=1823, mr=0x100e97320,
> errp=<value temporarily unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/exec.c:1554
> #9  0x000000010004353f in memory_region_init_ram (mr=0x100e97320,
> owner=<value temporarily unavailable, due to optimizations>,
> name=<value temporarily unavailable, due to optimizations>,
> size=134217728, errp=0x0) at /Users/pm215/src/qemu/memory.c:1216
> #10 0x000000010003f10a in allocate_system_memory_nonnuma [inlined] ()
> at /Users/pm215/src/qemu/numa.c:339
> #11 0x000000010003f10a in memory_region_allocate_system_memory
> (mr=0x100e97320, owner=0x6, name=0x0, ram_size=140734799798744) at
> /Users/pm215/src/qemu/numa.c:352
> #12 0x00000001000781cb in pc_memory_init (machine=0x102198630,
> system_memory=0x102199690, below_4g_mem_size=134217728,
> above_4g_mem_size=0, rom_memory=0x100e96de0,
> ram_memory=0x7fff5fbfe4b8, guest_info=<value temporarily unavailable,
> due to optimizations>) at /Users/pm215/src/qemu/hw/i386/pc.c:1254
> #13 0x000000010007aae6 in pc_init1 (machine=0x102198630) at
> /Users/pm215/src/qemu/hw/i386/pc_piix.c:182
> #14 0x00000001000e3cf9 in realtime_init [inlined] () at
> /Users/pm215/src/qemu/vl.c:4503
> #15 0x00000001000e3cf9 in qemu_main (argc=<value temporarily
> unavailable, due to optimizations>, argv=<value temporarily
> unavailable, due to optimizations>, envp=0x0) at
> /Users/pm215/src/qemu/vl.c:4505
> #16 0x00000001002117be in -[QemuCocoaAppController
> startEmulationWithArgc:argv:] (self=<value temporarily unavailable,
> due to optimizations>, _cmd=<value temporarily unavailable, due to
> optimizations>, argc=1823, argv=0x102199690) at
> /Users/pm215/src/qemu/ui/cocoa.m:941
> 
> A breakpoint on ram_save_setup() is never hit, so it looks
> like the problem is the mutex is being used before it is
> initialized.

Yes. I don't know why our test doesn't trigger this problem.
Will fix it soon.

Thanks
Wen Congyang

> 
> thanks
> -- PMM
> 
> .
>
Juan Quintela July 2, 2015, 12:17 p.m. UTC | #3
Peter Maydell <peter.maydell@linaro.org> wrote:
> On 1 July 2015 at 11:39, Juan Quintela <quintela@redhat.com> wrote:
>> Hi
>>
>> This series includes:
>> - rdma fixes by Dave
>> - rdma memory fix by gonglei
>> - vmdescription for old machine types (dave)
>> - fix footers for power (dave)
>> - migration bitmap extensions (Li)
>>   just fixed the compilation issues for linux-users
>> - migration events (me)
>> - optional secttions (me)
>> - global  configuration (me)
>>
>>
>> Please, Apply.
>>
>>
>> The following changes since commit d2966f804d70a244f5dde395fc5d22a50ed3e74e:
>>
>>   Merge remote-tracking branch
>> 'remotes/vivier/tags/pull-m68k-20150629' into staging (2015-06-29
>> 17:03:20 +0100)
>>
>> are available in the git repository at:
>>
>>   git://github.com/juanquintela/qemu.git tags/migration/20150701
>>
>> for you to fetch changes up to a4fe58b0ea0d78f92461607f4f90be3384fa30e5:
>>
>>   migration: Add migration events on target side (2015-07-01 12:35:05 +0200)
>>
>> ----------------------------------------------------------------
>> migration/next for 20150701
>
> On OSX at least every QEMU executable aborts immediately
> with "qemu: qemu_mutex_lock: Invalid argument". Here's a backtrace:
>
> #0  0x00007fff92c29286 in __pthread_kill ()
> #1  0x00007fff8a4f342f in pthread_kill ()
> #2  0x00007fff9240eb53 in abort ()
> #3  0x00000001002b0915 in error_exit (err=<value temporarily
> unavailable, due to optimizations>, msg=<value temporarily
> unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/util/qemu-thread-posix.c:48
> #4  0x00000001002b095d in qemu_mutex_lock (mutex=<value temporarily
> unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/util/qemu-thread-posix.c:75
> #5  0x0000000100050f20 in migration_bitmap_extend (old=0, new=32768)
> at /Users/pm215/src/qemu/migration/ram.c:1068
> #6  0x0000000100002a8d in ram_block_add [inlined] () at
> /Users/pm215/src/qemu/exec.c:1407
> #7  0x0000000100002a8d in qemu_ram_alloc_internal (size=<value
> temporarily unavailable, due to optimizations>, max_size=<value
> temporarily unavailable, due to optimizations>, resized=<value
> temporarily unavailable, due to optimizations>, host=<value
> temporarily unavailable, due to optimizations>, resizeable=false,
> mr=0x100e97320, errp=<value temporarily unavailable, due to
> optimizations>) at /Users/pm215/src/qemu/exec.c:1537
> #8  0x0000000100002e71 in qemu_ram_alloc (size=1823, mr=0x100e97320,
> errp=<value temporarily unavailable, due to optimizations>) at
> /Users/pm215/src/qemu/exec.c:1554
> #9  0x000000010004353f in memory_region_init_ram (mr=0x100e97320,
> owner=<value temporarily unavailable, due to optimizations>,
> name=<value temporarily unavailable, due to optimizations>,
> size=134217728, errp=0x0) at /Users/pm215/src/qemu/memory.c:1216
> #10 0x000000010003f10a in allocate_system_memory_nonnuma [inlined] ()
> at /Users/pm215/src/qemu/numa.c:339
> #11 0x000000010003f10a in memory_region_allocate_system_memory
> (mr=0x100e97320, owner=0x6, name=0x0, ram_size=140734799798744) at
> /Users/pm215/src/qemu/numa.c:352
> #12 0x00000001000781cb in pc_memory_init (machine=0x102198630,
> system_memory=0x102199690, below_4g_mem_size=134217728,
> above_4g_mem_size=0, rom_memory=0x100e96de0,
> ram_memory=0x7fff5fbfe4b8, guest_info=<value temporarily unavailable,
> due to optimizations>) at /Users/pm215/src/qemu/hw/i386/pc.c:1254
> #13 0x000000010007aae6 in pc_init1 (machine=0x102198630) at
> /Users/pm215/src/qemu/hw/i386/pc_piix.c:182
> #14 0x00000001000e3cf9 in realtime_init [inlined] () at
> /Users/pm215/src/qemu/vl.c:4503
> #15 0x00000001000e3cf9 in qemu_main (argc=<value temporarily
> unavailable, due to optimizations>, argv=<value temporarily
> unavailable, due to optimizations>, envp=0x0) at
> /Users/pm215/src/qemu/vl.c:4505
> #16 0x00000001002117be in -[QemuCocoaAppController
> startEmulationWithArgc:argv:] (self=<value temporarily unavailable,
> due to optimizations>, _cmd=<value temporarily unavailable, due to
> optimizations>, argc=1823, argv=0x102199690) at
> /Users/pm215/src/qemu/ui/cocoa.m:941
>
> A breakpoint on ram_save_setup() is never hit, so it looks
> like the problem is the mutex is being used before it is
> initialized.
>
> thanks
> -- PMM

Why, o why, o why it didn't fail on linux, why.  I think I know where
the problem is.

Thanks for testing.
Peter Maydell July 2, 2015, 12:30 p.m. UTC | #4
On 2 July 2015 at 13:17, Juan Quintela <quintela@redhat.com> wrote:
> Why, o why, o why it didn't fail on linux, why.  I think I know where
> the problem is.

I think OSX defaults to being pickier about its mutexes,
whereas Linux defaults to "make lock/unlock as fast as
possible and assume the program is correct".

-- PMM
Markus Armbruster July 2, 2015, 12:55 p.m. UTC | #5
Peter Maydell <peter.maydell@linaro.org> writes:

> On 2 July 2015 at 13:17, Juan Quintela <quintela@redhat.com> wrote:
>> Why, o why, o why it didn't fail on linux, why.  I think I know where
>> the problem is.
>
> I think OSX defaults to being pickier about its mutexes,
> whereas Linux defaults to "make lock/unlock as fast as
> possible and assume the program is correct".

From pthread_mutexattr_settype(3p):

       PTHREAD_MUTEX_ERRORCHECK

              This type of mutex provides error checking. A thread
              attempting to relock this mutex without first unlocking it
              shall return with an error. A thread attempting to unlock
              a mutex which another thread has locked shall return with
              an error. A thread attempting to unlock an unlocked mutex
              shall return with an error.

Not sure it detects this particular problem.

Should we use it with --enable-debug or something?
Peter Maydell July 2, 2015, 12:58 p.m. UTC | #6
On 2 July 2015 at 13:55, Markus Armbruster <armbru@redhat.com> wrote:
> From pthread_mutexattr_settype(3p):
>
>        PTHREAD_MUTEX_ERRORCHECK
>
>               This type of mutex provides error checking. A thread
>               attempting to relock this mutex without first unlocking it
>               shall return with an error. A thread attempting to unlock
>               a mutex which another thread has locked shall return with
>               an error. A thread attempting to unlock an unlocked mutex
>               shall return with an error.
>
> Not sure it detects this particular problem.
>
> Should we use it with --enable-debug or something?

It breaks when fork is involved, unfortunately:

https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg06016.html

-- PMM