From patchwork Thu Jun 27 06:45:58 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 254991 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 5339D2C0091 for ; Thu, 27 Jun 2013 16:53:57 +1000 (EST) Received: from localhost ([::1]:58324 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Us65b-0004ZI-8j for incoming@patchwork.ozlabs.org; Thu, 27 Jun 2013 02:53:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54237) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Us5ze-00034F-LL for qemu-devel@nongnu.org; Thu, 27 Jun 2013 02:47:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Us5zc-0007Kc-Ki for qemu-devel@nongnu.org; Thu, 27 Jun 2013 02:47:46 -0400 Received: from mail-ie0-f178.google.com ([209.85.223.178]:61955) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Us5zc-0007KH-CU for qemu-devel@nongnu.org; Thu, 27 Jun 2013 02:47:44 -0400 Received: by mail-ie0-f178.google.com with SMTP id u16so785580iet.23 for ; Wed, 26 Jun 2013 23:47:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references :x-gm-message-state; bh=/Ff11mP/vRfxpGMZRKKnBh+Qd7ms+GRPLeQMhe3Gvgk=; b=YGuzcfdGEvueMXqTDm4EedB20ydNaLOsOBZzcPsdb5wPG77dtO/AnXlTc+TxfAUxGp msB676R6ghB3Z0kLwRBwYoZ9APy6ACHGYXUpphQbXjOGgzC9L7zBtkAY9qt9ucryWUdW N2X26KxTSYYagQ9fa3So9PkOVETosdvLiKRBCXbscQ5uYILpUuuLLaK+jc/6TZ5yXt/t NE4J/IDdWCPKwZRvRyl7nPlSFFYK1ILc7IuiLvrT5vOuqqFQN2NDlJNLmyiPpCZHv/Rm i3nUU0cuxEehKnKLYQHJNV53qWCaF8HpX3t7M1sSExA2aTOYrMP1RXNxYtg/PXwcT045 MAHg== X-Received: by 10.50.1.20 with SMTP id 20mr1193935igi.56.1372315663553; Wed, 26 Jun 2013 23:47:43 -0700 (PDT) Received: from ka1.ozlabs.ibm.com (ibmaus65.lnk.telstra.net. [165.228.126.9]) by mx.google.com with ESMTPSA id x10sm11847910igl.3.2013.06.26.23.47.38 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 26 Jun 2013 23:47:42 -0700 (PDT) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Thu, 27 Jun 2013 16:45:58 +1000 Message-Id: <1372315560-5478-16-git-send-email-aik@ozlabs.ru> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1372315560-5478-1-git-send-email-aik@ozlabs.ru> References: <1372315560-5478-1-git-send-email-aik@ozlabs.ru> X-Gm-Message-State: ALoCoQkEKr0va1YNYrbd7urj/gyl2q7N+SFVQxoa6YxtAidVLXIhg4u6JN93Avc2eMdorBXYr9GE X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.85.223.178 Cc: Anthony Liguori , aik@ozlabs.ru, Alexander Graf , qemu-ppc@nongnu.org, Paolo Bonzini , Paul Mackerras , David Gibson Subject: [Qemu-devel] [PATCH 15/17] pseries: savevm support with KVM X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: David Gibson At present, the savevm / migration support for the pseries machine will not work when KVM is enabled. That's because KVM manages the guest's hash page table in the host kernel, so qemu has no visibility of it. This patch fixes this by using new kernel interfaces to extract and reinsert the guest's hash table during the migration process. Signed-off-by: David Gibson Signed-off-by: Alexey Kardashevskiy --- hw/ppc/spapr.c | 106 ++++++++++++++++++++++++++++++++++++++---------- include/hw/ppc/spapr.h | 1 + target-ppc/kvm.c | 69 +++++++++++++++++++++++++++++++ target-ppc/kvm_ppc.h | 22 ++++++++++ 4 files changed, 176 insertions(+), 22 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 211f434..9489edc 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -744,17 +744,27 @@ static int htab_save_setup(QEMUFile *f, void *opaque) { sPAPREnvironment *spapr = opaque; - spapr->htab_save_index = 0; - spapr->htab_first_pass = true; - /* "Iteration" header */ qemu_put_be32(f, spapr->htab_shift); + if (spapr->htab) { + spapr->htab_save_index = 0; + spapr->htab_first_pass = true; + } else { + assert(kvm_enabled()); + + spapr->htab_fd = kvmppc_get_htab_fd(false); + if (spapr->htab_fd < 0) { + fprintf(stderr, "Unable to open fd for reading hash table from KVM: %s\n", + strerror(errno)); + return -1; + } + } + + return 0; } -#define MAX_ITERATION_NS 5000000 /* 5 ms */ - static void htab_save_first_pass(QEMUFile *f, sPAPREnvironment *spapr, int64_t max_ns) { @@ -805,8 +815,8 @@ static void htab_save_first_pass(QEMUFile *f, sPAPREnvironment *spapr, spapr->htab_save_index = index; } -static bool htab_save_later_pass(QEMUFile *f, sPAPREnvironment *spapr, - int64_t max_ns) +static int htab_save_later_pass(QEMUFile *f, sPAPREnvironment *spapr, + int64_t max_ns) { bool final = max_ns < 0; int htabslots = HTAB_SIZE(spapr) / HASH_PTE_SIZE_64; @@ -879,21 +889,32 @@ static bool htab_save_later_pass(QEMUFile *f, sPAPREnvironment *spapr, spapr->htab_save_index = index; - return (examined >= htabslots) && (sent == 0); + return (examined >= htabslots) && (sent == 0) ? 1 : 0; } +#define MAX_ITERATION_NS 5000000 /* 5 ms */ +#define MAX_KVM_BUF_SIZE 2048 + static int htab_save_iterate(QEMUFile *f, void *opaque) { sPAPREnvironment *spapr = opaque; - bool nothingleft = false;; + int rc = 0; /* Iteration header */ qemu_put_be32(f, 0); - if (spapr->htab_first_pass) { + if (!spapr->htab) { + assert(kvm_enabled()); + + rc = kvmppc_save_htab(f, spapr->htab_fd, + MAX_KVM_BUF_SIZE, MAX_ITERATION_NS); + if (rc < 0) { + return rc; + } + } else if (spapr->htab_first_pass) { htab_save_first_pass(f, spapr, MAX_ITERATION_NS); } else { - nothingleft = htab_save_later_pass(f, spapr, MAX_ITERATION_NS); + rc = htab_save_later_pass(f, spapr, MAX_ITERATION_NS); } /* End marker */ @@ -901,7 +922,7 @@ static int htab_save_iterate(QEMUFile *f, void *opaque) qemu_put_be16(f, 0); qemu_put_be16(f, 0); - return nothingleft ? 1 : 0; + return rc; } static int htab_save_complete(QEMUFile *f, void *opaque) @@ -911,7 +932,20 @@ static int htab_save_complete(QEMUFile *f, void *opaque) /* Iteration header */ qemu_put_be32(f, 0); - htab_save_later_pass(f, spapr, -1); + if (!spapr->htab) { + int rc; + + assert(kvm_enabled()); + + rc = kvmppc_save_htab(f, spapr->htab_fd, MAX_KVM_BUF_SIZE, -1); + if (rc < 0) { + return rc; + } + close(spapr->htab_fd); + spapr->htab_fd = -1; + } else { + htab_save_later_pass(f, spapr, -1); + } /* End marker */ qemu_put_be32(f, 0); @@ -925,6 +959,7 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id) { sPAPREnvironment *spapr = opaque; uint32_t section_hdr; + int fd = -1; if (version_id < 1 || version_id > 1) { fprintf(stderr, "htab_load() bad version\n"); @@ -941,6 +976,16 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id) return 0; } + if (!spapr->htab) { + assert(kvm_enabled()); + + fd = kvmppc_get_htab_fd(true); + if (fd < 0) { + fprintf(stderr, "Unable to open fd to restore KVM hash table: %s\n", + strerror(errno)); + } + } + while (true) { uint32_t index; uint16_t n_valid, n_invalid; @@ -954,24 +999,41 @@ static int htab_load(QEMUFile *f, void *opaque, int version_id) break; } - if ((index + n_valid + n_invalid) >= + if ((index + n_valid + n_invalid) > (HTAB_SIZE(spapr) / HASH_PTE_SIZE_64)) { /* Bad index in stream */ fprintf(stderr, "htab_load() bad index %d (%hd+%hd entries) " - "in htab stream\n", index, n_valid, n_invalid); + "in htab stream (htab_shift=%d)\n", index, n_valid, n_invalid, + spapr->htab_shift); return -EINVAL; } - if (n_valid) { - qemu_get_buffer(f, HPTE(spapr->htab, index), - HASH_PTE_SIZE_64 * n_valid); - } - if (n_invalid) { - memset(HPTE(spapr->htab, index + n_valid), 0, - HASH_PTE_SIZE_64 * n_invalid); + if (spapr->htab) { + if (n_valid) { + qemu_get_buffer(f, HPTE(spapr->htab, index), + HASH_PTE_SIZE_64 * n_valid); + } + if (n_invalid) { + memset(HPTE(spapr->htab, index + n_valid), 0, + HASH_PTE_SIZE_64 * n_invalid); + } + } else { + int rc; + + assert(fd >= 0); + + rc = kvmppc_load_htab_chunk(f, fd, index, n_valid, n_invalid); + if (rc < 0) { + return rc; + } } } + if (!spapr->htab) { + assert(fd >= 0); + close(fd); + } + return 0; } diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 4cfe449..3da31f0 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -37,6 +37,7 @@ typedef struct sPAPREnvironment { /* Migration state */ int htab_save_index; bool htab_first_pass; + int htab_fd; } sPAPREnvironment; #define H_SUCCESS 0 diff --git a/target-ppc/kvm.c b/target-ppc/kvm.c index 33ddf63..ff85c19 100644 --- a/target-ppc/kvm.c +++ b/target-ppc/kvm.c @@ -65,6 +65,7 @@ static int cap_one_reg; static int cap_epr; static int cap_ppc_watchdog; static int cap_papr; +static int cap_htab_fd; /* XXX We have a race condition where we actually have a level triggered * interrupt, but the infrastructure can't expose that yet, so the guest @@ -101,6 +102,7 @@ int kvm_arch_init(KVMState *s) cap_ppc_watchdog = kvm_check_extension(s, KVM_CAP_PPC_BOOKE_WATCHDOG); /* Note: we don't set cap_papr here, because this capability is * only activated after this by kvmppc_set_papr() */ + cap_htab_fd = kvm_check_extension(s, KVM_CAP_PPC_HTAB_FD); if (!cap_interrupt_level) { fprintf(stderr, "KVM: Couldn't find level irq capability. Expect the " @@ -1802,6 +1804,73 @@ int kvmppc_define_rtas_token(uint32_t token, const char *function) return kvm_vm_ioctl(kvm_state, KVM_PPC_RTAS_DEFINE_TOKEN, &args); } +int kvmppc_get_htab_fd(bool write) +{ + struct kvm_get_htab_fd s = { + .flags = write ? KVM_GET_HTAB_WRITE : 0, + .start_index = 0, + }; + + if (!cap_htab_fd) { + fprintf(stderr, "KVM version doesn't support saving the hash table\n"); + return -1; + } + + return kvm_vm_ioctl(kvm_state, KVM_PPC_GET_HTAB_FD, &s); +} + +int kvmppc_save_htab(QEMUFile *f, int fd, size_t bufsize, int64_t max_ns) +{ + int64_t starttime = qemu_get_clock_ns(rt_clock); + uint8_t buf[bufsize]; + ssize_t rc; + + do { + rc = read(fd, buf, bufsize); + if (rc < 0) { + fprintf(stderr, "Error reading data from KVM HTAB fd: %s\n", + strerror(errno)); + return rc; + } else if (rc) { + /* Kernel already retuns data in BE format for the file */ + qemu_put_buffer(f, buf, rc); + } + } while ((rc != 0) + && ((max_ns < 0) + || ((qemu_get_clock_ns(rt_clock) - starttime) < max_ns))); + + return (rc == 0) ? 1 : 0; +} + +int kvmppc_load_htab_chunk(QEMUFile *f, int fd, uint32_t index, + uint16_t n_valid, uint16_t n_invalid) +{ + struct kvm_get_htab_header *buf; + size_t chunksize = sizeof(*buf) + n_valid*HASH_PTE_SIZE_64; + ssize_t rc; + + buf = alloca(chunksize); + /* This is KVM on ppc, so this is all big-endian */ + buf->index = index; + buf->n_valid = n_valid; + buf->n_invalid = n_invalid; + + qemu_get_buffer(f, (void *)(buf + 1), HASH_PTE_SIZE_64*n_valid); + + rc = write(fd, buf, chunksize); + if (rc < 0) { + fprintf(stderr, "Error writing KVM hash table: %s\n", + strerror(errno)); + return rc; + } + if (rc != chunksize) { + /* We should never get a short write on a single chunk */ + fprintf(stderr, "Short write, restoring KVM hash table\n"); + return -1; + } + return 0; +} + bool kvm_arch_stop_on_emulation_error(CPUState *cpu) { return true; diff --git a/target-ppc/kvm_ppc.h b/target-ppc/kvm_ppc.h index 21939a8..12564ef 100644 --- a/target-ppc/kvm_ppc.h +++ b/target-ppc/kvm_ppc.h @@ -39,6 +39,10 @@ uint64_t kvmppc_rma_size(uint64_t current_size, unsigned int hash_shift); int kvmppc_fixup_cpu(PowerPCCPU *cpu); bool kvmppc_has_cap_epr(void); int kvmppc_define_rtas_token(uint32_t token, const char *function); +int kvmppc_get_htab_fd(bool write); +int kvmppc_save_htab(QEMUFile *f, int fd, size_t bufsize, int64_t max_ns); +int kvmppc_load_htab_chunk(QEMUFile *f, int fd, uint32_t index, + uint16_t n_valid, uint16_t n_invalid); #else @@ -166,6 +170,24 @@ static inline int kvmppc_define_rtas_token(uint32_t token, { return -1; } + +static inline int kvmppc_get_htab_fd(bool write) +{ + return -1; +} + +static inline int kvmppc_save_htab(QEMUFile *f, int fd, size_t bufsize, + int64_t max_ns) +{ + abort(); +} + +static inline int kvmppc_load_htab_chunk(QEMUFile *f, int fd, uint32_t index, + uint16_t n_valid, uint16_t n_invalid) +{ + abort(); +} + #endif #ifndef CONFIG_KVM