Patchwork Re: [PATCH uq/master 2/2] MCE, unpoison memory address across reboot

login
register
mail settings
Submitter Huang Ying
Date Jan. 14, 2011, 1:51 a.m.
Message ID <1294969881.4596.80.camel@yhuang-dev>
Download mbox | patch
Permalink /patch/78858/
State New
Headers show

Comments

Huang Ying - Jan. 14, 2011, 1:51 a.m.
On Thu, 2011-01-13 at 17:01 +0800, Jan Kiszka wrote:
> Am 13.01.2011 09:34, Huang Ying wrote:
> > In Linux kernel HWPoison processing implementation, the virtual
> > address in processes mapping the error physical memory page is marked
> > as HWPoison.  So that, the further accessing to the virtual
> > address will kill corresponding processes with SIGBUS.
> > 
> > If the error physical memory page is used by a KVM guest, the SIGBUS
> > will be sent to QEMU, and QEMU will simulate a MCE to report that
> > memory error to the guest OS.  If the guest OS can not recover from
> > the error (for example, the page is accessed by kernel code), guest OS
> > will reboot the system.  But because the underlying host virtual
> > address backing the guest physical memory is still poisoned, if the
> > guest system accesses the corresponding guest physical memory even
> > after rebooting, the SIGBUS will still be sent to QEMU and MCE will be
> > simulated.  That is, guest system can not recover via rebooting.
> > 
> > In fact, across rebooting, the contents of guest physical memory page
> > need not to be kept.  We can allocate a new host physical page to
> > back the corresponding guest physical address.
> > 
> > This patch fixes this issue in QEMU via calling qemu_ram_remap() to
> > clear the corresponding page table entry, so that make it possible to
> > allocate a new page to recover the issue.
> > 
> > Signed-off-by: Huang Ying <ying.huang@intel.com>
> > ---
> >  kvm.h             |    2 ++
> >  target-i386/kvm.c |   39 +++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 41 insertions(+)
> > 
> > --- a/target-i386/kvm.c
> > +++ b/target-i386/kvm.c
> > @@ -580,6 +580,42 @@ static int kvm_get_supported_msrs(void)
> >      return ret;
> >  }
> >  
> > +struct HWPoisonPage;
> > +typedef struct HWPoisonPage HWPoisonPage;
> > +struct HWPoisonPage
> > +{
> > +    ram_addr_t ram_addr;
> > +    QLIST_ENTRY(HWPoisonPage) list;
> > +};
> > +
> > +static QLIST_HEAD(hwpoison_page_list, HWPoisonPage) hwpoison_page_list =
> > +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
> > +
> > +void kvm_unpoison_all(void *param)
> 
> Minor nit: This can be static now.

In uq/master, it can be make static.  But in kvm/master, kvm_arch_init
is not compiled because of conditional compiling, so we will get warning
and error for unused symbol.  Should we consider kvm/master in this
patch?

> > +{
> > +    HWPoisonPage *page, *next_page;
> > +
> > +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
> > +        QLIST_REMOVE(page, list);
> > +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
> > +        qemu_free(page);
> > +    }
> > +}
> > +
> > +static void kvm_hwpoison_page_add(ram_addr_t ram_addr)
> > +{
> > +    HWPoisonPage *page;
> > +
> > +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
> > +        if (page->ram_addr == ram_addr)
> > +            return;
> > +    }
> > +
> > +    page = qemu_malloc(sizeof(HWPoisonPage));
> > +    page->ram_addr = ram_addr;
> > +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
> > +}
> > +
> >  int kvm_arch_init(void)
> >  {
> >      uint64_t identity_base = 0xfffbc000;
> > @@ -632,6 +668,7 @@ int kvm_arch_init(void)
> >          fprintf(stderr, "e820_add_entry() table is full\n");
> >          return ret;
> >      }
> > +    qemu_register_reset(kvm_unpoison_all, NULL);
> >  
> >      return 0;
> >  }
> > @@ -1940,6 +1977,7 @@ int kvm_on_sigbus_vcpu(CPUState *env, in
> >                  hardware_memory_error();
> >              }
> >          }
> > +        kvm_hwpoison_page_add(ram_addr);
> >  
> >          if (code == BUS_MCEERR_AR) {
> >              /* Fake an Intel architectural Data Load SRAR UCR */
> > @@ -1984,6 +2022,7 @@ int kvm_on_sigbus(int code, void *addr)
> >                      "QEMU itself instead of guest system!: %p\n", addr);
> >              return 0;
> >          }
> > +        kvm_hwpoison_page_add(ram_addr);
> >          kvm_mce_inj_srao_memscrub2(first_cpu, paddr);
> >      } else
> >  #endif
> > --- a/kvm.h
> > +++ b/kvm.h
> > @@ -188,6 +188,8 @@ int kvm_physical_memory_addr_from_ram(ra
> >                                        target_phys_addr_t *phys_addr);
> >  #endif
> >  
> > +void kvm_unpoison_all(void *param);
> > +
> 
> To be removed if kvm_unpoison_all is static.
> 
> >  #endif
> >  int kvm_set_ioeventfd_mmio_long(int fd, uint32_t adr, uint32_t val, bool assign);
> >  
> > 
> 
> As indicated, I'm sitting on lots of fixes and refactorings of the MCE
> user space code. How do you test your patches? Any suggestions how to do
> this efficiently would be warmly welcome.

We use a self-made test script to test.  Repository is at:

git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git

The kvm test script is in kvm sub-directory.

The qemu patch attached is need by the test script.

Best Regards,
Huang Ying
Jan Kiszka - Jan. 14, 2011, 8:38 a.m.
Am 14.01.2011 02:51, Huang Ying wrote:
> On Thu, 2011-01-13 at 17:01 +0800, Jan Kiszka wrote:
>> Am 13.01.2011 09:34, Huang Ying wrote:
>>> In Linux kernel HWPoison processing implementation, the virtual
>>> address in processes mapping the error physical memory page is marked
>>> as HWPoison.  So that, the further accessing to the virtual
>>> address will kill corresponding processes with SIGBUS.
>>>
>>> If the error physical memory page is used by a KVM guest, the SIGBUS
>>> will be sent to QEMU, and QEMU will simulate a MCE to report that
>>> memory error to the guest OS.  If the guest OS can not recover from
>>> the error (for example, the page is accessed by kernel code), guest OS
>>> will reboot the system.  But because the underlying host virtual
>>> address backing the guest physical memory is still poisoned, if the
>>> guest system accesses the corresponding guest physical memory even
>>> after rebooting, the SIGBUS will still be sent to QEMU and MCE will be
>>> simulated.  That is, guest system can not recover via rebooting.
>>>
>>> In fact, across rebooting, the contents of guest physical memory page
>>> need not to be kept.  We can allocate a new host physical page to
>>> back the corresponding guest physical address.
>>>
>>> This patch fixes this issue in QEMU via calling qemu_ram_remap() to
>>> clear the corresponding page table entry, so that make it possible to
>>> allocate a new page to recover the issue.
>>>
>>> Signed-off-by: Huang Ying <ying.huang@intel.com>
>>> ---
>>>  kvm.h             |    2 ++
>>>  target-i386/kvm.c |   39 +++++++++++++++++++++++++++++++++++++++
>>>  2 files changed, 41 insertions(+)
>>>
>>> --- a/target-i386/kvm.c
>>> +++ b/target-i386/kvm.c
>>> @@ -580,6 +580,42 @@ static int kvm_get_supported_msrs(void)
>>>      return ret;
>>>  }
>>>  
>>> +struct HWPoisonPage;
>>> +typedef struct HWPoisonPage HWPoisonPage;
>>> +struct HWPoisonPage
>>> +{
>>> +    ram_addr_t ram_addr;
>>> +    QLIST_ENTRY(HWPoisonPage) list;
>>> +};
>>> +
>>> +static QLIST_HEAD(hwpoison_page_list, HWPoisonPage) hwpoison_page_list =
>>> +    QLIST_HEAD_INITIALIZER(hwpoison_page_list);
>>> +
>>> +void kvm_unpoison_all(void *param)
>>
>> Minor nit: This can be static now.
> 
> In uq/master, it can be make static.  But in kvm/master, kvm_arch_init
> is not compiled because of conditional compiling, so we will get warning
> and error for unused symbol.  Should we consider kvm/master in this
> patch?

qemu-kvm is very close to switching to upstream kvm_*init. As long as it
requires this service in its own modules, it will have to patch this
detail. It does this for other functions already.

> 
>>> +{
>>> +    HWPoisonPage *page, *next_page;
>>> +
>>> +    QLIST_FOREACH_SAFE(page, &hwpoison_page_list, list, next_page) {
>>> +        QLIST_REMOVE(page, list);
>>> +        qemu_ram_remap(page->ram_addr, TARGET_PAGE_SIZE);
>>> +        qemu_free(page);
>>> +    }
>>> +}
>>> +
>>> +static void kvm_hwpoison_page_add(ram_addr_t ram_addr)
>>> +{
>>> +    HWPoisonPage *page;
>>> +
>>> +    QLIST_FOREACH(page, &hwpoison_page_list, list) {
>>> +        if (page->ram_addr == ram_addr)
>>> +            return;
>>> +    }
>>> +
>>> +    page = qemu_malloc(sizeof(HWPoisonPage));
>>> +    page->ram_addr = ram_addr;
>>> +    QLIST_INSERT_HEAD(&hwpoison_page_list, page, list);
>>> +}
>>> +
>>>  int kvm_arch_init(void)
>>>  {
>>>      uint64_t identity_base = 0xfffbc000;
>>> @@ -632,6 +668,7 @@ int kvm_arch_init(void)
>>>          fprintf(stderr, "e820_add_entry() table is full\n");
>>>          return ret;
>>>      }
>>> +    qemu_register_reset(kvm_unpoison_all, NULL);
>>>  
>>>      return 0;
>>>  }
>>> @@ -1940,6 +1977,7 @@ int kvm_on_sigbus_vcpu(CPUState *env, in
>>>                  hardware_memory_error();
>>>              }
>>>          }
>>> +        kvm_hwpoison_page_add(ram_addr);
>>>  
>>>          if (code == BUS_MCEERR_AR) {
>>>              /* Fake an Intel architectural Data Load SRAR UCR */
>>> @@ -1984,6 +2022,7 @@ int kvm_on_sigbus(int code, void *addr)
>>>                      "QEMU itself instead of guest system!: %p\n", addr);
>>>              return 0;
>>>          }
>>> +        kvm_hwpoison_page_add(ram_addr);
>>>          kvm_mce_inj_srao_memscrub2(first_cpu, paddr);
>>>      } else
>>>  #endif
>>> --- a/kvm.h
>>> +++ b/kvm.h
>>> @@ -188,6 +188,8 @@ int kvm_physical_memory_addr_from_ram(ra
>>>                                        target_phys_addr_t *phys_addr);
>>>  #endif
>>>  
>>> +void kvm_unpoison_all(void *param);
>>> +
>>
>> To be removed if kvm_unpoison_all is static.
>>
>>>  #endif
>>>  int kvm_set_ioeventfd_mmio_long(int fd, uint32_t adr, uint32_t val, bool assign);
>>>  
>>>
>>
>> As indicated, I'm sitting on lots of fixes and refactorings of the MCE
>> user space code. How do you test your patches? Any suggestions how to do
>> this efficiently would be warmly welcome.
> 
> We use a self-made test script to test.  Repository is at:
> 
> git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
> 
> The kvm test script is in kvm sub-directory.
> 
> The qemu patch attached is need by the test script.
> 

Yeah, I already found this yesterday and started reading. I was just
searching for p2v in qemu, but now it's clear where it comes from. Will
have a look (if you want to preview my changes:
git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream).

I was almost about to use MADV_HWPOISON instead of the injection module.
Is there a way to recover the fake corruption afterward? I think that
would allow to move some of the test logic into qemu and avoid p2v which
- IIRC - was disliked upstream.

Also, is there a way to simulate corrected errors (BUS_MCEERR_AO)?

Thanks,
Jan
Huang Ying - Jan. 17, 2011, 2:08 a.m.
On Fri, 2011-01-14 at 16:38 +0800, Jan Kiszka wrote:
> Am 14.01.2011 02:51, Huang Ying wrote:
> > On Thu, 2011-01-13 at 17:01 +0800, Jan Kiszka wrote:
> >> Am 13.01.2011 09:34, Huang Ying wrote:
[snip]
> >>> +
> >>> +void kvm_unpoison_all(void *param)
> >>
> >> Minor nit: This can be static now.
> > 
> > In uq/master, it can be make static.  But in kvm/master, kvm_arch_init
> > is not compiled because of conditional compiling, so we will get warning
> > and error for unused symbol.  Should we consider kvm/master in this
> > patch?
> 
> qemu-kvm is very close to switching to upstream kvm_*init. As long as it
> requires this service in its own modules, it will have to patch this
> detail. It does this for other functions already.

OK.  I will change this.

[snip]
> >> As indicated, I'm sitting on lots of fixes and refactorings of the MCE
> >> user space code. How do you test your patches? Any suggestions how to do
> >> this efficiently would be warmly welcome.
> > 
> > We use a self-made test script to test.  Repository is at:
> > 
> > git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
> > 
> > The kvm test script is in kvm sub-directory.
> > 
> > The qemu patch attached is need by the test script.
> > 
> 
> Yeah, I already found this yesterday and started reading. I was just
> searching for p2v in qemu, but now it's clear where it comes from. Will
> have a look (if you want to preview my changes:
> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream).
> 
> I was almost about to use MADV_HWPOISON instead of the injection module.
> Is there a way to recover the fake corruption afterward? I think that
> would allow to move some of the test logic into qemu and avoid p2v which
> - IIRC - was disliked upstream.

I don't know how to fully recover from  MADV_HWPOISON.  You can recover
the virtual address space via qemu_ram_remap() introduced in 1/2 of this
patchset.  But you will lose one or several physical pages for each
testing.  I think that may be not a big issue for a testing machine.

Ccing Andi and Fengguang, they know more than me about MADV_HWPOISON.

> Also, is there a way to simulate corrected errors (BUS_MCEERR_AO)?

BUS_MCEERR_AO is recoverable uncorrected error instead of corrected
error.

The test script is for BUS_MCEERR_AO and BUS_MCEERR_AR.  To see the
effect of pure BUS_MCEERR_AO, just remove the memory accessing loop
(memset) in tools/simple_process/simple_process.c.

Best Regards,
Huang Ying
Jan Kiszka - Jan. 17, 2011, 9:48 a.m.
On 2011-01-17 03:08, Huang Ying wrote:
>>>> As indicated, I'm sitting on lots of fixes and refactorings of the MCE
>>>> user space code. How do you test your patches? Any suggestions how to do
>>>> this efficiently would be warmly welcome.
>>>
>>> We use a self-made test script to test.  Repository is at:
>>>
>>> git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
>>>
>>> The kvm test script is in kvm sub-directory.
>>>
>>> The qemu patch attached is need by the test script.
>>>
>>
>> Yeah, I already found this yesterday and started reading. I was just
>> searching for p2v in qemu, but now it's clear where it comes from. Will
>> have a look (if you want to preview my changes:
>> git://git.kiszka.org/qemu-kvm.git queues/kvm-upstream).
>>
>> I was almost about to use MADV_HWPOISON instead of the injection module.
>> Is there a way to recover the fake corruption afterward? I think that
>> would allow to move some of the test logic into qemu and avoid p2v which
>> - IIRC - was disliked upstream.
> 
> I don't know how to fully recover from  MADV_HWPOISON.  You can recover
> the virtual address space via qemu_ram_remap() introduced in 1/2 of this
> patchset.  But you will lose one or several physical pages for each
> testing.  I think that may be not a big issue for a testing machine.
> 
> Ccing Andi and Fengguang, they know more than me about MADV_HWPOISON.

"page-types -b hwpoison -x" does the trick of unpoisoning for me. It can
be found at linux/Documentation/vm/page-types.c. So it's quite easy to
set up and clean up a test case based on MADV_HWPOISON IMO. Not sure,
though, if that can simulate all of what you currently do via mce-inject.

> 
>> Also, is there a way to simulate corrected errors (BUS_MCEERR_AO)?
> 
> BUS_MCEERR_AO is recoverable uncorrected error instead of corrected
> error.
> 
> The test script is for BUS_MCEERR_AO and BUS_MCEERR_AR.  To see the
> effect of pure BUS_MCEERR_AO, just remove the memory accessing loop
> (memset) in tools/simple_process/simple_process.c.

Yeah, that question was based on lacking knowledge about the different
error types. Meanwhile, I was able to trigger BUS_MCEERR_AO via
MADV_HWPOISON - and also BUS_MCEERR_AR by accessing that page. However,
I did not succeed with using mce-inject so far, thus with mce-test. But
I need to check this again.

Jan

Patch

Author: Max Asbock <masbock@linux.vnet.ibm.com>
Subject: [PATCH -v3] Monitor command: x-gpa2hva, translate guest physical address to host virtual address

Add command x-gpa2hva to translate guest physical address to host
virtual address. Because gpa to hva translation is not consistent, so
this command is only used for debugging.

The x-gpa2hva command provides one step in a chain of translations from
guest virtual to guest physical to host virtual to host physical. Host
physical is then used to inject a machine check error. As a
consequence the HWPOISON code on the host and the MCE injection code
in qemu-kvm are exercised.

v3:

- Rename to x-gpa2hva
- Remove QMP support, because gpa2hva is not consistent

v2:

- Add QMP support

Signed-off-by: Max Asbock <masbock@linux.vnet.ibm.com>
Signed-off-by: Jiajia Zheng <jiajia.zheng@intel.com>
Signed-off-by: Huang Ying <ying.huang@intel.com>
---
 hmp-commands.hx |   15 +++++++++++++++
 monitor.c       |   22 ++++++++++++++++++++++
 2 files changed, 37 insertions(+)

--- a/monitor.c
+++ b/monitor.c
@@ -2708,6 +2708,28 @@  static void do_inject_mce(Monitor *mon,
 }
 #endif
 
+static void do_gpa2hva_print(Monitor *mon, const QObject *data)
+{
+    QInt *qint;
+
+    qint = qobject_to_qint(data);
+    monitor_printf(mon, "0x%lx\n", (unsigned long)qint->value);
+}
+
+static int do_gpa2hva(Monitor *mon, const QDict *qdict, QObject **ret_data)
+{
+    target_phys_addr_t paddr;
+    target_phys_addr_t size = TARGET_PAGE_SIZE;
+    void *vaddr;
+
+    paddr = qdict_get_int(qdict, "addr");
+    vaddr = cpu_physical_memory_map(paddr, &size, 0);
+    cpu_physical_memory_unmap(vaddr, size, 0, 0);
+    *ret_data = qobject_from_jsonf("%ld", (unsigned long)vaddr);
+
+    return 0;
+}
+
 static int do_getfd(Monitor *mon, const QDict *qdict, QObject **ret_data)
 {
     const char *fdname = qdict_get_str(qdict, "fdname");
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -311,6 +311,21 @@  Start gdbserver session (default @var{po
 ETEXI
 
     {
+        .name       = "x-gpa2hva",
+        .args_type  = "fmt:/,addr:l",
+        .params     = "/fmt addr",
+        .help       = "translate guest physical 'addr' to host virtual address, only for debugging",
+        .user_print = do_gpa2hva_print,
+        .mhandler.cmd_new = do_gpa2hva,
+    },
+
+STEXI
+@item x-gpa2hva @var{addr}
+@findex x-gpa2hva
+Translate guest physical @var{addr} to host virtual address, only for debugging.
+ETEXI
+
+    {
         .name       = "x",
         .args_type  = "fmt:/,addr:l",
         .params     = "/fmt addr",