SeaBios: Fix reset procedure reentrancy problem on qemu-kvm platform

On Mon, Dec 21, 2015 at 09:41:32AM +0000, Gonglei (Arei) wrote:
> When the gurb of OS is booting, then the softirq and C function send_disk_op()
> may use extra stack of SeaBIOS. If we inject a NMI, romlayout.S: irqentry_extrastack
> is invoked, and the extra stack will be used again. And the stack of first calling
> will be broken, so that the SeaBIOS stuck. 
> 
> You can easily reproduce the problem.
> 
> 1. start on guest
> 2. reset the guest
> 3. inject a NMI when the guest show the grub surface
> 4. then the guest stuck

Does the SeaBIOS patch below help?  I'm not familiar with how to
"inject a NMI" - can you describe the process in more detail?

-Kevin

> -----Original Message-----
> From: Kevin O'Connor [mailto:kevin@koconnor.net]
> Sent: Tuesday, December 22, 2015 2:47 AM
> To: Gonglei (Arei)
> Cc: Xulei (Stone); Paolo Bonzini; qemu-devel; seabios@seabios.org;
> Huangweidong (C); kvm@vger.kernel.org; Radim Krcmar
> Subject: Re: [Qemu-devel] [PATCH] SeaBios: Fix reset procedure reentrancy
> problem on qemu-kvm platform
> 
> On Mon, Dec 21, 2015 at 09:41:32AM +0000, Gonglei (Arei) wrote:
> > When the gurb of OS is booting, then the softirq and C function
> > send_disk_op() may use extra stack of SeaBIOS. If we inject a NMI,
> > romlayout.S: irqentry_extrastack is invoked, and the extra stack will
> > be used again. And the stack of first calling will be broken, so that the
> SeaBIOS stuck.
> >
> > You can easily reproduce the problem.
> >
> > 1. start on guest
> > 2. reset the guest
> > 3. inject a NMI when the guest show the grub surface 4. then the guest
> > stuck
> 
> Does the SeaBIOS patch below help?  

Sorry, it doesn't work. What's worse is we cannot stop SeaBIOS stuck by
Setting "CONFIG_ENTRY_EXTRASTACK=n" after applying this patch. 


> I'm not familiar with how to "inject a
> NMI" - can you describe the process in more detail?
> 

1. Qemu Command line:

#: /home/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 4096 -smp 8 -name suse -vnc 0.0.0.0:10 \
-device virtio-scsi-pci,id=scsi0 -drive file=/home/suse11_sp3_32_2,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,aio=native \
-device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
-chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios \
-monitor stdio -qmp unix:/tmp/qmp,server,nowait 

2. Inject a NMI by QMP:

#: /home/qemu/scripts/qmp # ./qmp-shell /tmp/qmp
Welcome to the QMP low-level shell!
Connected to QEMU 2.5.0

(QEMU) system_reset
{"return": {}}
(QEMU) inject-nmi  
{"return": {}}
(QEMU) inject-nmi
{"return": {}}


Regards,
-Gonglei

> -Kevin
> 
> 
> --- a/src/romlayout.S
> +++ b/src/romlayout.S
> @@ -548,7 +548,9 @@ entry_post:
>          ENTRY_INTO32 _cfunc32flat_handle_post   // Normal entry point
> 
>          ORG 0xe2c3
> -        IRQ_ENTRY 02
> +        .global entry_02
> +entry_02:
> +        ENTRY handle_02  // NMI handler does not switch onto extra
> +stack
> 
>          ORG 0xe3fe
>          .global entry_13_official

Hi, Kevin,
Can you tell how to reset/reboot this VM, if it goes to the handle_hwpic1()
on its booting procedure? I mean, usually, SeaBIOS would not go to 
handle_hwpic routine. But in my test case, SeaBIOS calls handle_hwpic when
KVM injects a #UD expcetion (not irq) and  SeaBIOS will loop to handle this
if KVM persistently injects exception.
 
Now, i just wish to reset/reboot this VM if it is fall into handle_hwpic. I
tried follwing patch and it seems not work. What can i do to force reset/reboot? 

@@ -7,10 +7,11 @@
  #include "biosvar.h" // SET_IVT
  #include "config.h" // CONFIG_*
  #include "output.h" // dprintf
  #include "pic.h" // pic_*
+ #include "hw/ps2port.h"
 
 u16
 pic_irqmask_read(void)
 {
     if (!CONFIG_HARDWARE_IRQ)
@@ -103,10 +104,11 @@ pic_isr2_read(void)
 void VISIBLE16
 handle_hwpic1(struct bregs *regs)
 {
     dprintf(DEBUG_ISR_hwpic1, "handle_hwpic1 irq=%x\n", pic_isr1_read());
     pic_eoi1();
+	 i8042_reboot();
 }

useful information:
kmod ftrace:
<...>-31509 [035] 154753.180077: kvm_exit: reason EXCEPTION_NMI rip 0x3 info 0 80000306
<...>-31509 [035] 154753.180077: kvm_emulate_insn: 0:3:f0 53 (real)
<...>-31509 [035] 154753.180077: kvm_inj_exception: #UD (0x0)
<...>-31509 [035] 154753.180077: kvm_entry: vcpu 0

bad SeaBIOS log:
[2015-12-17 12:37:30] In 32bit resume
[2015-12-17 12:37:30] =====Attempting a hard reboot====
[2015-12-17 12:37:30] SeaBIOS (version rel-1.8.1-0-g4adadbd-20151217_104405-linux-emBwNn)
[2015-12-17 12:37:30] No Xen hypervisor found.
[2015-12-17 12:37:30] Running on QEMU (i440fx)
[2015-12-17 12:37:30] Running on KVM
[2015-12-17 12:37:30] RamSize: 0x80000000 [cmos]
[2015-12-17 12:37:30] Relocating init from 0x000db230 to 0x7ffad360 (size 76768)
[2015-12-17 12:37:30] Found QEMU fw_cfg
[2015-12-17 12:37:30] RamBlock: addr 0x0000000000000000 len 0x0000000080000000 [e820]
[2015-12-17 12:37:30] Moving pm_base to 0x600
[2015-12-17 12:37:30] boot order:
[2015-12-17 12:37:30] 1: /pci@i0cf8/ide@1,1/drive@0/disk@0
[2015-12-17 12:37:30] 2: HALT
[2015-12-17 12:37:30] maininit
[2015-12-17 12:37:30] platform_hardware_setup
[2015-12-17 12:37:30] init pic
[2015-12-17 12:37:30] pic_setup
[2015-12-17 12:37:30] pic_reset
[2015-12-17 12:37:30] enable_hwirq
[2015-12-17 12:37:30] CPU Mhz=3304
[2015-12-17 12:37:30] enable_hwirq
[2015-12-17 12:37:30] enable_hwirq
[2015-12-17 12:37:30] === PCI bus & bridge init ===
[2015-12-17 12:37:30] PCI: pci_bios_init_bus_rec bus = 0x0
[2015-12-17 12:37:30] === PCI device probing ===
[2015-12-17 12:37:30] Found 6 PCI devices (max PCI bus is 00)
[2015-12-17 12:37:30] === PCI new allocation pass #1 ===
[2015-12-17 12:37:30] PCI: check devices
[2015-12-17 12:37:30] === PCI new allocation pass #2 ===
[2015-12-17 12:37:30] PCI: IO: c000 - c02f
[2015-12-17 12:37:30] PCI: 32: 0000000080000000 - 00000000fec00000
[2015-12-17 12:37:30] PCI: map device bdf=00:01.2  bar 4, addr 0000c000, size 00000020 [io]
[2015-12-17 12:37:30] PCI: map device bdf=00:01.1  bar 4, addr 0000c020, size 00000010 [io]
[2015-12-17 12:37:30] PCI: map device bdf=00:02.0  bar 6, addr febe0000, size 00010000 [mem]
[2015-12-17 12:37:30] PCI: map device bdf=00:02.0  bar 1, addr febf0000, size 00001000 [mem]
[2015-12-17 12:37:30] PCI: map device bdf=00:02.0  bar 0, addr fc000000, size 02000000 [prefmem]
[2015-12-17 12:37:30] PCI: init bdf=00:00.0 id=8086:1237
[2015-12-17 12:37:30] PCI: init bdf=00:01.0 id=8086:7000
[2015-12-17 12:37:30] PIIX3/PIIX4 init: elcr=00 0c
[2015-12-17 12:37:30] PCI: init bdf=00:01.1 id=8086:7010
[2015-12-17 12:37:30] PCI: init bdf=00:01.2 id=8086:7020
[2015-12-17 12:37:30] PCI: init bdf=00:01.3 id=8086:7113
[2015-12-17 12:37:30] Using pmtimer, ioport 0x608
[2015-12-17 12:37:30] PCI: init bdf=00:02.0 id=1013:00b8
[2015-12-17 12:37:30] PCI: Using 00:02.0 for primary VGA
[2015-12-17 12:37:30] handle_hshamanpnd:dl leae_p_sismcmp_p:i: d a=ap3               <<======= everytime stuck, AP setup log seems abnormal!
[2015-12-17 12:37:30] ièf[cf_^ifd_=f3
[2015-12-17 12:37:30] èf[f^f_f]fÃÍ^XË<90>Found 4 cpu(s) max supported 4 cpu(s)
[2015-12-17 12:37:30] Copying PIR from 0x7ffbea18 to 0x000f5700
[2015-12-17 12:37:30] Copying MPTABLE from 0x00006e30/7ffa42c0 to 0x000f55e0
[2015-12-17 12:37:30] Copying SMBIOS entry point from 0x00006e11 to 0x000f55c0
[2015-12-17 12:37:31] Scan for VGA option rom
[2015-12-17 12:37:31] Running option rom at c000:0003
[2015-12-17 12:37:31] Start SeaVGABIOS (version rel-1.8.1-0-g4adadbd-20150316_085902-nilsson.home.kraxel.org)
[2015-12-17 12:37:31] enter vga_post:
[2015-12-17 12:37:31]    a=00000010  b=0000ffff  c=00000000  d=0000ffff ds=0000 es=f000 ss=0000
[2015-12-17 12:37:31]   si=00000000 di=000057e0 bp=00000000 sp=00006dbe cs=f000 ip=d1fb  f=0000
[2015-12-17 12:37:31] cirrus init
[2015-12-17 12:37:31] cirrus init 2
[2015-12-17 12:37:31] Attempting to allocate VGA stack via pmm call to f000:d2a0   <<====== here stuck, loop handle PIC irq0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
[2015-12-17 12:37:35] handle_hwpic1 irq=0
... always hanle_hwpic1 irq=0, never ends anymore...


>> -----Original Message-----

>> From: Kevin O'Connor [mailto:kevin@koconnor.net]

>> Sent: Tuesday, December 22, 2015 2:47 AM

>> To: Gonglei (Arei)

>> Cc: Xulei (Stone); Paolo Bonzini; qemu-devel; seabios@seabios.org;

>> Huangweidong (C); kvm@vger.kernel.org; Radim Krcmar

>> Subject: Re: [Qemu-devel] [PATCH] SeaBios: Fix reset procedure reentrancy

>> problem on qemu-kvm platform

>>

>> On Mon, Dec 21, 2015 at 09:41:32AM +0000, Gonglei (Arei) wrote:

>> > When the gurb of OS is booting, then the softirq and C function

>> > send_disk_op() may use extra stack of SeaBIOS. If we inject a NMI,

>> > romlayout.S: irqentry_extrastack is invoked, and the extra stack will

>> > be used again. And the stack of first calling will be broken, so that the

>> SeaBIOS stuck.

>> >

>> > You can easily reproduce the problem.

>> >

>> > 1. start on guest

>> > 2. reset the guest

>> > 3. inject a NMI when the guest show the grub surface 4. then the guest

>> > stuck

>>

>> Does the SeaBIOS patch below help? 

>

>Sorry, it doesn't work. What's worse is we cannot stop SeaBIOS stuck by

>Setting "CONFIG_ENTRY_EXTRASTACK=n" after applying this patch.

>

>

>> I'm not familiar with how to "inject a

>> NMI" - can you describe the process in more detail?

>>

>

>1. Qemu Command line:

>

>#: /home/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 4096 -smp 8 -name suse -vnc 0.0.0.0:10 \

>-device virtio-scsi-pci,id=scsi0 -drive file=/home/suse11_sp3_32_2,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,aio=native \

>-device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \

>-chardev file,id=seabios,path=/home/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios \

>-monitor stdio -qmp unix:/tmp/qmp,server,nowait

>

>2. Inject a NMI by QMP:

>

>#: /home/qemu/scripts/qmp # ./qmp-shell /tmp/qmp

>Welcome to the QMP low-level shell!

>Connected to QEMU 2.5.0

>

>(QEMU) system_reset

>{"return": {}}

>(QEMU) inject-nmi 

>{"return": {}}

>(QEMU) inject-nmi

>{"return": {}}

>

>

>Regards,

>-Gonglei

>

>> -Kevin

>>

>>

>> --- a/src/romlayout.S

>> +++ b/src/romlayout.S

>> @@ -548,7 +548,9 @@ entry_post:

>>          ENTRY_INTO32 _cfunc32flat_handle_post   // Normal entry point

>>

>>          ORG 0xe2c3

>> -        IRQ_ENTRY 02

>> +        .global entry_02

>> +entry_02:

>> +        ENTRY handle_02  // NMI handler does not switch onto extra

>> +stack

>>

>>          ORG 0xe3fe

>>          .global entry_13_official

On Tue, Dec 22, 2015 at 03:15:26AM +0000, Xulei (Stone) wrote:
> Hi, Kevin,
> Can you tell how to reset/reboot this VM, if it goes to the handle_hwpic1()
> on its booting procedure? I mean, usually, SeaBIOS would not go to 
> handle_hwpic routine. But in my test case, SeaBIOS calls handle_hwpic when
> KVM injects a #UD expcetion (not irq) and  SeaBIOS will loop to handle this
> if KVM persistently injects exception.
>  
> Now, i just wish to reset/reboot this VM if it is fall into handle_hwpic. I
> tried follwing patch and it seems not work. What can i do to force reset/reboot? 

Call the reset() function.

-Kevin