diff mbox

virtio-serial-pci very expensive during live migration

Message ID 536B9524.8070307@windriver.com
State New
Headers show

Commit Message

Chris Friesen May 8, 2014, 2:31 p.m. UTC 
On 05/08/2014 07:30 AM, Amit Shah wrote:
> On (Thu) 08 May 2014 [15:14:26], Paolo Bonzini wrote:
>> Il 08/05/2014 15:02, Amit Shah ha scritto:
>>>>> I tried the patch below.  Unfortunately it seems to cause qemu to crash.
>>> This doesn't remove the memory_region_transaction_begin() and
>>> _commit() from memory_region_add_eventfd(), which will still be
>>> called.
>>
>> That's correct, transactions can be nested.
>
> But each _commit() will add to the delay?

No, the _commit() call checks "memory_region_transaction_depth" and only 
does the address space scan when it hits zero.  So with nested 
transactions only the outer commit will actually do the address space scan.

The fact remains that qemu crashes when I apply the patch.  I also tried 
patching it as below in virtio_pci_vmstate_change().  That would allow 
the VM to boot, but it would crash when I tried to do a live migration.

Chris

Comments

Paolo Bonzini May 8, 2014, 2:34 p.m. UTC | #1 
Il 08/05/2014 16:31, Chris Friesen ha scritto:
>
>
> The fact remains that qemu crashes when I apply the patch.  I also tried
> patching it as below in virtio_pci_vmstate_change().  That would allow
> the VM to boot, but it would crash when I tried to do a live migration.

Can you give us your command line and a backtrace?

Paolo
Chris Friesen May 8, 2014, 3:57 p.m. UTC | #2 
On 05/08/2014 08:34 AM, Paolo Bonzini wrote:
> Il 08/05/2014 16:31, Chris Friesen ha scritto:
>>
>>
>> The fact remains that qemu crashes when I apply the patch.  I also tried
>> patching it as below in virtio_pci_vmstate_change().  That would allow
>> the VM to boot, but it would crash when I tried to do a live migration.
>
> Can you give us your command line and a backtrace?

For the backtrace, you mean with a core file?  Or is there a better way?

Chris
Paolo Bonzini May 8, 2014, 4:02 p.m. UTC | #3 
Il 08/05/2014 17:57, Chris Friesen ha scritto:
>>>
>>> The fact remains that qemu crashes when I apply the patch.  I also tried
>>> patching it as below in virtio_pci_vmstate_change().  That would allow
>>> the VM to boot, but it would crash when I tried to do a live migration.
>>
>> Can you give us your command line and a backtrace?
>
> For the backtrace, you mean with a core file?  Or is there a better way?

You can attach gdb to the qemu process, and run "thread apply all bt" 
when QEMU crashes.

Paolo
Chris Friesen May 8, 2014, 8:57 p.m. UTC | #4 
On 05/08/2014 10:02 AM, Paolo Bonzini wrote:
> Il 08/05/2014 17:57, Chris Friesen ha scritto:
>>>>
>>>> The fact remains that qemu crashes when I apply the patch.  I also
>>>> tried
>>>> patching it as below in virtio_pci_vmstate_change().  That would allow
>>>> the VM to boot, but it would crash when I tried to do a live migration.
>>>
>>> Can you give us your command line and a backtrace?
>>
>> For the backtrace, you mean with a core file?  Or is there a better way?
>
> You can attach gdb to the qemu process, and run "thread apply all bt"
> when QEMU crashes.


For the commandline, it's coming from OpenStack via libvirt, so it's 
fairly complicated:

/usr/bin/kvm -c 0x00000000000000000000000000000001 -n 4 
--proc-type=secondary --file-prefix=vs -- -enable-dpdk -name 
instance-00000007 -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m 512 
-mem-prealloc -mem-path /mnt/huge-2048kB/libvirt/qemu -smp 
2,sockets=2,cores=1,threads=1 -uuid b277d785-9c31-45be-91d3-15b7b1aca974 
-smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack 
Nova,version=2013.2.2,serial=00000000-0000-3848-3342-4e5037313536,uuid=b277d785-9c31-45be-91d3-15b7b1aca974 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000007.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive 
file=/etc/nova/instances/b277d785-9c31-45be-91d3-15b7b1aca974/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-chardev 
file,id=charserial0,path=/etc/nova/instances/b277d785-9c31-45be-91d3-15b7b1aca974/console.log 
-device isa-serial,chardev=charserial0,id=serial0 -chardev 
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 
-chardev 
socket,id=charchannel0,path=/var/lib/libvirt/qemu/cgcs.heartbeat.instance-00000007.sock,server,nowait 
-device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=cgcs.heartbeat 
-device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -device 
avp,port=4a05cedd-3868-449a-9728-899bbea10863,id=avp4a05cedd-38




As for the backtrace, I can attach gdb to the qemu process, but when I 
hit "c" to continue it gives an error:


Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libnss_files.so.2

warning: File "/lib64/libthread_db-1.0.so" auto-loading has been 
declined by your `auto-load safe-path' set to 
"$debugdir:$datadir/auto-load".

warning: Unable to find libthread_db matching inferior's thread library, 
thread debugging will not be available.
0x00007f56c6366d23 in select () from /lib64/libc.so.6
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint -1.
Error accessing memory address 0x2f7c50: Input/output error.


Chris
ChenLiang May 9, 2014, 1:44 a.m. UTC | #5 
On 2014/5/8 22:34, Paolo Bonzini wrote:

> Il 08/05/2014 16:31, Chris Friesen ha scritto:
>>
>>
>> The fact remains that qemu crashes when I apply the patch.  I also tried
>> patching it as below in virtio_pci_vmstate_change().  That would allow
>> the VM to boot, but it would crash when I tried to do a live migration.
> 
> Can you give us your command line and a backtrace?
> 
> Paolo
> 
> 
> 

Hi,
I have test the patch at the qemu.git, qemu crashed when vm is booting.

the backtrace is:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f6bf67f9700 (LWP 9740)]
0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
#1  0x00007f6bfacb4131 in abort () from /lib64/libc.so.6
#2  0x00007f6bfd51047c in kvm_io_ioeventfd_del (listener=
    0x7f6bfd9ffee0 <kvm_io_listener>, section=0x7f6bf67f87c0, match_data=true, data=
    0, e=0x7f6970000930) at /tmp/qemu/kvm-all.c:879
#3  0x00007f6bfd5163b5 in address_space_add_del_ioeventfds (as=
    0x7f6bfde3d6e0 <address_space_io>, fds_new=0x0, fds_new_nb=0, fds_old=
    0x7f6bfdfd8ce0, fds_old_nb=1) at /tmp/qemu/memory.c:628
#4  0x00007f6bfd51698e in address_space_update_ioeventfds (as=
    0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:687
#5  0x00007f6bfd517949 in address_space_update_topology (as=
    0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:780
#6  0x00007f6bfd517a68 in memory_region_transaction_commit ()
    at /tmp/qemu/memory.c:800
#7  0x00007f6bfd3e2942 in virtio_pci_stop_ioeventfd (proxy=0x7f6bfdfde080)
    at hw/virtio/virtio-pci.c:270
#8  0x00007f6bfd3e2aaa in virtio_ioport_write (opaque=0x7f6bfdfde080, addr=18, val=
    0) at hw/virtio/virtio-pci.c:309
#9  0x00007f6bfd3e2ff3 in virtio_pci_config_write (opaque=0x7f6bfdfde080, addr=18,
    val=0, size=1) at hw/virtio/virtio-pci.c:436
#10 0x00007f6bfd515368 in memory_region_write_accessor (mr=0x7f6bfdfde818, addr=18,
    value=0x7f6bf67f8b68, size=1, shift=0, mask=255) at /tmp/qemu/memory.c:441
#11 0x00007f6bfd5154a4 in access_with_adjusted_size (addr=18, value=0x7f6bf67f8b68,
    size=1, access_size_min=1, access_size_max=4, access=
    0x7f6bfd5152df <memory_region_write_accessor>, mr=0x7f6bfdfde818)
    at /tmp/qemu/memory.c:478
#12 0x00007f6bfd518305 in memory_region_dispatch_write (mr=0x7f6bfdfde818, addr=18,
    data=0, size=1) at /tmp/qemu/memory.c:985
#13 0x00007f6bfd51bb1b in io_mem_write (mr=0x7f6bfdfde818, addr=18, val=0, size=1)
    at /tmp/qemu/memory.c:1744
#14 0x00007f6bfd4a0c67 in address_space_rw (as=0x7f6bfde3d6e0 <address_space_io>,
    addr=49170, buf=0x7f6bfd189000 "", len=1, is_write=true)
    at /tmp/qemu/exec.c:2034
#15 0x00007f6bfd511e06 in kvm_handle_io (port=49170, data=0x7f6bfd189000, direction=
    1, size=1, count=1) at /tmp/qemu/kvm-all.c:1558
#16 0x00007f6bfd5123aa in kvm_cpu_exec (cpu=0x7f6bfdf54d50)
    at /tmp/qemu/kvm-all.c:1695
#17 0x00007f6bfd491db6 in qemu_kvm_cpu_thread_fn (arg=0x7f6bfdf54d50)
    at /tmp/qemu/cpus.c:873
#18 0x00007f6bfafff7f6 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f6bfad5b09d in clone () from /lib64/libc.so.6
#20 0x0000000000000000 in ?? ()


the commandline is:
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /tmp/qemu/x86_
64-softmmu/qemu-system-x86_64 -name cl_suse -S -machine pc-i440fx-1.5,accel=kvm,usb=o
ff -m 10240 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 5a09315c-d31
4-49a5-aa51-2168a71bf82d -no-user-config -nodefaults -chardev socket,id=charmonitor,p
ath=/var/lib/libvirt/qemu/cl_suse.monitor,server,nowait -mon chardev=charmonitor,id=m
onitor,mode=control -rtc base=utc -no-hpet -no-shutdown -device piix3-usb-uhci,id=usb
,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/sdb/cl/cl_sles11sp3.img,if=none,id=drive-vir
tio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,
addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charser
ial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -
vnc 0.0.0.0:0 -device cirrus-vga,id=video0,vgamem_mb=9,bus=pci.0,addr=0x2 -device vir
tio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
Chris Friesen May 9, 2014, 3:31 a.m. UTC | #6 
On 05/08/2014 07:44 PM, ChenLiang wrote:

> Hi,
> I have test the patch at the qemu.git, qemu crashed when vm is booting.
> 
> the backtrace is:
> 
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7f6bf67f9700 (LWP 9740)]
> 0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007f6bfacb2b55 in raise () from /lib64/libc.so.6
> #1  0x00007f6bfacb4131 in abort () from /lib64/libc.so.6
> #2  0x00007f6bfd51047c in kvm_io_ioeventfd_del (listener=
>      0x7f6bfd9ffee0 <kvm_io_listener>, section=0x7f6bf67f87c0, match_data=true, data=
>      0, e=0x7f6970000930) at /tmp/qemu/kvm-all.c:879
> #3  0x00007f6bfd5163b5 in address_space_add_del_ioeventfds (as=
>      0x7f6bfde3d6e0 <address_space_io>, fds_new=0x0, fds_new_nb=0, fds_old=
>      0x7f6bfdfd8ce0, fds_old_nb=1) at /tmp/qemu/memory.c:628
> #4  0x00007f6bfd51698e in address_space_update_ioeventfds (as=
>      0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:687
> #5  0x00007f6bfd517949 in address_space_update_topology (as=
>      0x7f6bfde3d6e0 <address_space_io>) at /tmp/qemu/memory.c:780
> #6  0x00007f6bfd517a68 in memory_region_transaction_commit ()
>      at /tmp/qemu/memory.c:800
> #7  0x00007f6bfd3e2942 in virtio_pci_stop_ioeventfd (proxy=0x7f6bfdfde080)
>      at hw/virtio/virtio-pci.c:270


Hah...I attached gdb to the source qemu this time and got basically the same thing.
This is with the stable-1.4 branch plus the patch, using "-machine accel=kvm".

#0  0x00007f4e08e1e9e9 in raise () from /lib64/libc.so.6
#1  0x00007f4e08e200f8 in abort () from /lib64/libc.so.6
#2  0x00007f4e0b30e8a4 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7f4e080f2c00, match_data=<optimized out>, data=0, e=<optimized out>) at /home/cfriesen/devel/qemu/kvm-all.c:804
#3  0x00007f4e0b311f60 in address_space_add_del_ioeventfds (fds_old_nb=64, fds_old=0x7f4df8004b40, fds_new_nb=0, fds_new=0x0, as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:603
#4  address_space_update_ioeventfds (as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:649
#5  address_space_update_topology (as=0x7f4e0bfb9fe0 <address_space_io>) at /home/cfriesen/devel/qemu/memory.c:730
#6  memory_region_transaction_commit () at /home/cfriesen/devel/qemu/memory.c:750
#7  0x00007f4e0b255cb5 in virtio_pci_stop_ioeventfd (proxy=0x7f4e0d3266a0) at hw/virtio-pci.c:259
#8  0x00007f4e0b3086d0 in virtio_vmstate_change (opaque=0x7f4e0d31c370, running=<optimized out>, state=<optimized out>) at /home/cfriesen/devel/qemu/hw/virtio.c:914
#9  0x00007f4e0b2bdf5b in vm_state_notify (running=running@entry=0, state=state@entry=RUN_STATE_FINISH_MIGRATE) at vl.c:1674
#10 0x00007f4e0b2c3f62 in do_vm_stop (state=RUN_STATE_FINISH_MIGRATE) at /home/cfriesen/devel/qemu/cpus.c:446
#11 vm_stop (state=state@entry=RUN_STATE_FINISH_MIGRATE) at /home/cfriesen/devel/qemu/cpus.c:1080
#12 0x00007f4e0b25e5c5 in buffered_file_thread (opaque=0x7f4e0b79a640 <current_migration.19392>) at migration.c:707
#13 0x00007f4e09cacc53 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f4e08ededbd in clone () from /lib64/libc.so.6

Chris
diff mbox

Patch

--- qemu-1.4.2.orig/hw/virtio-pci.c
+++ qemu-1.4.2/hw/virtio-pci.c
@@ -32,6 +32,7 @@ 
  #include "virtio-pci.h"
  #include "qemu/range.h"
  #include "virtio-bus.h"
+#include "exec/memory.h"

  /* from Linux's linux/virtio_pci.h */

@@ -854,7 +855,7 @@  static int virtio_pci_set_host_notifier(
  static void virtio_pci_vmstate_change(DeviceState *d, bool running)
  {
      VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
-
+    memory_region_transaction_begin();
      if (running) {
          /* Try to find out if the guest has bus master disabled, but is
             in ready state. Then we have a buggy guest OS. */
@@ -866,6 +867,7 @@  static void virtio_pci_vmstate_change(De
      } else {
          virtio_pci_stop_ioeventfd(proxy);
      }
+    memory_region_transaction_commit();
  }

  static const VirtIOBindings virtio_pci_bindings = {