diff mbox

[RFC,4/4] Relevant changes to enable KVM to TCG migration

Message ID 1393347170-28502-5-git-send-email-a.rigo@virtualopensystems.com
State New
Headers show

Commit Message

Alvise Rigo Feb. 25, 2014, 4:52 p.m. UTC 
CPUARMState:
* added adfsr cp register.
* added aifsr cp register.
These registers have been added because they are migrated by KVM. This prevents
the migration from failing when trying to copy their values.

ARMCPRegInfo:
* added a pointer to the parent that generated the register (if any).
* a flag to inform that we have already copied the value of the register:
  this prevents the register from being overwritten by the parent register
  value, which could be not set.

helper.c:
* added mechanism to track the cp register "parent".
* compare_cpreg_array(): compare the incoming cp registers with the
  cpreg_list keeping a list of the registers that do not succeeded the
  match.
* handle_cpreg_kvm2tcg_migration(): try to solve the mismatch of
  cp registers coming from KVM; without this additional step the
  migration would fail even if it's feasible.

Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
---
 target-arm/cpu.h     |  43 +++++++++++++
 target-arm/helper.c  | 167 +++++++++++++++++++++++++++++++++++++++++++++++++--
 target-arm/machine.c |  46 ++++++++++----
 3 files changed, 238 insertions(+), 18 deletions(-)

Comments

Peter Maydell Feb. 25, 2014, 6:25 p.m. UTC | #1 
On 25 February 2014 16:52, Alvise Rigo <a.rigo@virtualopensystems.com> wrote:
> CPUARMState:
> * added adfsr cp register.
> * added aifsr cp register.
> These registers have been added because they are migrated by KVM. This prevents
> the migration from failing when trying to copy their values.

This should be done in a separate patch.

> ARMCPRegInfo:
> * added a pointer to the parent that generated the register (if any).
> * a flag to inform that we have already copied the value of the register:
>   this prevents the register from being overwritten by the parent register
>   value, which could be not set.
>
> helper.c:
> * added mechanism to track the cp register "parent".
> * compare_cpreg_array(): compare the incoming cp registers with the
>   cpreg_list keeping a list of the registers that do not succeeded the
>   match.
> * handle_cpreg_kvm2tcg_migration(): try to solve the mismatch of
>   cp registers coming from KVM; without this additional step the
>   migration would fail even if it's feasible.
>
> Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
> ---
>  target-arm/cpu.h     |  43 +++++++++++++
>  target-arm/helper.c  | 167 +++++++++++++++++++++++++++++++++++++++++++++++++--
>  target-arm/machine.c |  46 ++++++++++----
>  3 files changed, 238 insertions(+), 18 deletions(-)
>
> diff --git a/target-arm/cpu.h b/target-arm/cpu.h
> index 3c8a2db..a97246d 100644
> --- a/target-arm/cpu.h
> +++ b/target-arm/cpu.h
> @@ -184,6 +184,8 @@ typedef struct CPUARMState {
>                          MPU write buffer control.  */
>          uint32_t c5_insn; /* Fault status registers.  */
>          uint32_t c5_data;
> +        uint32_t c5_adfsr;
> +        uint32_t c5_aifsr;
>          uint32_t c6_region[8]; /* MPU base/size registers.  */
>          uint32_t c6_insn; /* Fault address registers.  */
>          uint32_t c6_data;
> @@ -197,6 +199,7 @@ typedef struct CPUARMState {
>          uint32_t c9_pmxevtyper; /* perf monitor event type */
>          uint32_t c9_pmuserenr; /* perf monitor user enable */
>          uint32_t c9_pminten; /* perf monitor interrupt enables */
> +        uint32_t c9_l2ctlr; /* L2 Control Register */
>          uint32_t c12_vbar; /* vector base address register */
>          uint32_t c13_fcse; /* FCSE PID.  */
>          uint32_t c13_context; /* Context ID.  */
> @@ -867,6 +870,15 @@ struct ARMCPRegInfo {
>      uint8_t opc0;
>      uint8_t opc1;
>      uint8_t opc2;
> +    /* When migrating this flag tells if the register's value has already
> +     * been copied. This is used in some unlikely cases where KVM migrates
> +     * a register that in TCG is generated by a wildcarded or
> +     * ARM_CP_STATE_BOTH parent (hence a not migratable register);
> +     * in those cases, we will set the field "parent" to point to the
> +     * generating register. Most of the time this field can stay false.
> +     * */
> +    bool skip_cpreglist;

This reads to me like it's papering over a problem rather
than fixing it.

thanks
-- PMM
Alvise Rigo Feb. 26, 2014, 10:02 a.m. UTC | #2 
I agree that this is a sort of workaround, but it seems to me that a
proper solution is not possible without changing the ideas contemplated
now in the migration code.
Are we willing to accept some major changes in the code to embrace
this type of migration?

Thanks,
alvise


On Tue, Feb 25, 2014 at 7:25 PM, Peter Maydell <peter.maydell@linaro.org>wrote:

> On 25 February 2014 16:52, Alvise Rigo <a.rigo@virtualopensystems.com>
> wrote:
> > CPUARMState:
> > * added adfsr cp register.
> > * added aifsr cp register.
> > These registers have been added because they are migrated by KVM. This
> prevents
> > the migration from failing when trying to copy their values.
>
> This should be done in a separate patch.
>
> > ARMCPRegInfo:
> > * added a pointer to the parent that generated the register (if any).
> > * a flag to inform that we have already copied the value of the register:
> >   this prevents the register from being overwritten by the parent
> register
> >   value, which could be not set.
> >
> > helper.c:
> > * added mechanism to track the cp register "parent".
> > * compare_cpreg_array(): compare the incoming cp registers with the
> >   cpreg_list keeping a list of the registers that do not succeeded the
> >   match.
> > * handle_cpreg_kvm2tcg_migration(): try to solve the mismatch of
> >   cp registers coming from KVM; without this additional step the
> >   migration would fail even if it's feasible.
> >
> > Signed-off-by: Alvise Rigo <a.rigo@virtualopensystems.com>
> > ---
> >  target-arm/cpu.h     |  43 +++++++++++++
> >  target-arm/helper.c  | 167
> +++++++++++++++++++++++++++++++++++++++++++++++++--
> >  target-arm/machine.c |  46 ++++++++++----
> >  3 files changed, 238 insertions(+), 18 deletions(-)
> >
> > diff --git a/target-arm/cpu.h b/target-arm/cpu.h
> > index 3c8a2db..a97246d 100644
> > --- a/target-arm/cpu.h
> > +++ b/target-arm/cpu.h
> > @@ -184,6 +184,8 @@ typedef struct CPUARMState {
> >                          MPU write buffer control.  */
> >          uint32_t c5_insn; /* Fault status registers.  */
> >          uint32_t c5_data;
> > +        uint32_t c5_adfsr;
> > +        uint32_t c5_aifsr;
> >          uint32_t c6_region[8]; /* MPU base/size registers.  */
> >          uint32_t c6_insn; /* Fault address registers.  */
> >          uint32_t c6_data;
> > @@ -197,6 +199,7 @@ typedef struct CPUARMState {
> >          uint32_t c9_pmxevtyper; /* perf monitor event type */
> >          uint32_t c9_pmuserenr; /* perf monitor user enable */
> >          uint32_t c9_pminten; /* perf monitor interrupt enables */
> > +        uint32_t c9_l2ctlr; /* L2 Control Register */
> >          uint32_t c12_vbar; /* vector base address register */
> >          uint32_t c13_fcse; /* FCSE PID.  */
> >          uint32_t c13_context; /* Context ID.  */
> > @@ -867,6 +870,15 @@ struct ARMCPRegInfo {
> >      uint8_t opc0;
> >      uint8_t opc1;
> >      uint8_t opc2;
> > +    /* When migrating this flag tells if the register's value has
> already
> > +     * been copied. This is used in some unlikely cases where KVM
> migrates
> > +     * a register that in TCG is generated by a wildcarded or
> > +     * ARM_CP_STATE_BOTH parent (hence a not migratable register);
> > +     * in those cases, we will set the field "parent" to point to the
> > +     * generating register. Most of the time this field can stay false.
> > +     * */
> > +    bool skip_cpreglist;
>
> This reads to me like it's papering over a problem rather
> than fixing it.
>
> thanks
> -- PMM
>
Peter Maydell Feb. 26, 2014, 10:04 a.m. UTC | #3 
On 26 February 2014 10:02, alvise rigo <a.rigo@virtualopensystems.com> wrote:
> I agree that this is a sort of workaround, but it seems to me that a
> proper solution is not possible without changing the ideas contemplated
> now in the migration code.
> Are we willing to accept some major changes in the code to embrace
> this type of migration?

Well, the migration code as it stands is in theory supposed to cope
with KVM<->TCG migration. I would expect that we'd need to improve
a number of places where our TCG cpreg emulation is wrong.

thanks
-- PMM
Alvise Rigo Feb. 26, 2014, 10:27 a.m. UTC | #4 
Improvements of cpreg emulation apart, we will still need to tackle
with the great amount of registers not migrated because generated
by a wildcarded register; some of these registers are in fact migrated
by KVM. I think that the idea of keeping a reference
to the generator register (i.e. the only register which is migrated in a
wildcarded case) could be considered to handle the migration.

alvise


On Wed, Feb 26, 2014 at 11:04 AM, Peter Maydell <peter.maydell@linaro.org>wrote:

> On 26 February 2014 10:02, alvise rigo <a.rigo@virtualopensystems.com>
> wrote:
> > I agree that this is a sort of workaround, but it seems to me that a
> > proper solution is not possible without changing the ideas contemplated
> > now in the migration code.
> > Are we willing to accept some major changes in the code to embrace
> > this type of migration?
>
> Well, the migration code as it stands is in theory supposed to cope
> with KVM<->TCG migration. I would expect that we'd need to improve
> a number of places where our TCG cpreg emulation is wrong.
>
> thanks
> -- PMM
>
Peter Maydell Feb. 26, 2014, 10:33 a.m. UTC | #5 
On 26 February 2014 10:27, alvise rigo <a.rigo@virtualopensystems.com> wrote:
> Improvements of cpreg emulation apart, we will still need to tackle
> with the great amount of registers not migrated because generated
> by a wildcarded register; some of these registers are in fact migrated
> by KVM. I think that the idea of keeping a reference
> to the generator register (i.e. the only register which is migrated in a
> wildcarded case) could be considered to handle the migration.

These probably need to be fixed by being actually implemented in
TCG. If there's a register which we wildcard on TCG that means
we should usually fail the incoming migration, because the guest
might be actually using that register for something and we would
just be dropping the state in migration if we allowed the migration
to succeed.

thanks
-- PMM
Peter Maydell March 3, 2014, 9:39 p.m. UTC | #6 
On 26 February 2014 10:02, alvise rigo <a.rigo@virtualopensystems.com> wrote:
> I agree that this is a sort of workaround, but it seems to me that a
> proper solution is not possible without changing the ideas contemplated
> now in the migration code.
> Are we willing to accept some major changes in the code to embrace
> this type of migration?

Incidentally it occurred to me recently that the design we've gone for
for AArch64 support (canonical copy of system register info is the AArch64
view, AArch32 register encoding is a non-migratable view onto the same
underlying data) clashes rather with the idea of being able to migrate
AArch32 TCG<->KVM. (Migrating AArch32 TCG to/from a KVM which
is configured to run the VM in AArch32 on an AArch64 host doesn't run
into the same problems, because in that case what KVM exposes to
userspace will also be the AArch64 views of registers.)
That was accidental, not an intentional tradeoff, but I'm not entirely
sure how to reconcile things...

thanks
-- PMM
diff mbox

Patch

diff --git a/target-arm/cpu.h b/target-arm/cpu.h
index 3c8a2db..a97246d 100644
--- a/target-arm/cpu.h
+++ b/target-arm/cpu.h
@@ -184,6 +184,8 @@  typedef struct CPUARMState {
                         MPU write buffer control.  */
         uint32_t c5_insn; /* Fault status registers.  */
         uint32_t c5_data;
+        uint32_t c5_adfsr;
+        uint32_t c5_aifsr;
         uint32_t c6_region[8]; /* MPU base/size registers.  */
         uint32_t c6_insn; /* Fault address registers.  */
         uint32_t c6_data;
@@ -197,6 +199,7 @@  typedef struct CPUARMState {
         uint32_t c9_pmxevtyper; /* perf monitor event type */
         uint32_t c9_pmuserenr; /* perf monitor user enable */
         uint32_t c9_pminten; /* perf monitor interrupt enables */
+        uint32_t c9_l2ctlr; /* L2 Control Register */
         uint32_t c12_vbar; /* vector base address register */
         uint32_t c13_fcse; /* FCSE PID.  */
         uint32_t c13_context; /* Context ID.  */
@@ -867,6 +870,15 @@  struct ARMCPRegInfo {
     uint8_t opc0;
     uint8_t opc1;
     uint8_t opc2;
+    /* When migrating this flag tells if the register's value has already
+     * been copied. This is used in some unlikely cases where KVM migrates
+     * a register that in TCG is generated by a wildcarded or
+     * ARM_CP_STATE_BOTH parent (hence a not migratable register);
+     * in those cases, we will set the field "parent" to point to the
+     * generating register. Most of the time this field can stay false.
+     * */
+    bool skip_cpreglist;
+    ARMCPRegInfo *parent;
     /* Execution state in which this register is visible: ARM_CP_STATE_* */
     int state;
     /* Register type: ARM_CP_* bits/values */
@@ -995,8 +1007,39 @@  bool write_list_to_cpustate(ARMCPU *cpu);
  * Note that we do not stop early on failure -- we will attempt
  * reading all registers in the list.
  */
+
 bool write_cpustate_to_list(ARMCPU *cpu);
 
+typedef struct coproc_pair {
+    uint64_t id;
+    uint64_t val;
+} cp_pair;
+
+#ifndef KVM_REG_ARM_CORE
+#define KVM_REG_ARM_CORE                (0x0010 << CP_REG_ARM_COPROC_SHIFT)
+#endif
+#ifndef KVM_REG_ARM_DEMUX
+#define KVM_REG_ARM_DEMUX               (0x0011 << CP_REG_ARM_COPROC_SHIFT)
+#endif
+#ifndef KVM_REG_ARM_VFP
+#define KVM_REG_ARM_VFP                 (0x0012 << CP_REG_ARM_COPROC_SHIFT)
+#endif
+/* This method compares two arrays of coprocessor registers; in case of
+ * migration, the array b contains the incoming values. If a not NULL
+ * pointer to an empty list is provided, the method will fill it with
+ * pairs <id, value> of the registers that are in b but not in a.
+ * */
+int compare_cpreg_array(uint32_t len_a, uint64_t *idx_a, uint64_t *val_a,
+                        uint32_t len_b, uint64_t *idx_b, uint64_t *val_b,
+                        GList **mis_list);
+
+/* This method is supposed to take in input those registers which failed the
+ * copy in compare_cpreg_array and tries to handle them in such a way to allow
+ * migration from KVM to the TCG processor.
+ * It returns the number of registers not handled, -1 in case of error.
+ * */
+int handle_cpreg_kvm2tcg_migration(ARMCPU *cpu, GList *mis_list, uint32_t len);
+
 /* Does the core conform to the the "MicroController" profile. e.g. Cortex-M3.
    Note the M in older cores (eg. ARM7TDMI) stands for Multiply. These are
    conventional cores (ie. Application or Realtime profile).  */
diff --git a/target-arm/helper.c b/target-arm/helper.c
index 1b111b6..2abe169 100644
--- a/target-arm/helper.c
+++ b/target-arm/helper.c
@@ -197,7 +197,7 @@  bool write_list_to_cpustate(ARMCPU *cpu)
             ok = false;
             continue;
         }
-        if (ri->type & ARM_CP_NO_MIGRATE) {
+        if (ri->type & ARM_CP_NO_MIGRATE || (ri->skip_cpreglist)) {
             continue;
         }
         /* Write value and confirm it reads back as written
@@ -1213,6 +1213,10 @@  static const ARMCPRegInfo vmsa_cp_reginfo[] = {
     { .name = "DFAR", .cp = 15, .crn = 6, .crm = 0, .opc1 = 0, .opc2 = 0,
       .access = PL1_RW, .fieldoffset = offsetof(CPUARMState, cp15.c6_data),
       .resetvalue = 0, },
+    { .name = "ADFSR", .cp = 15, .crn = 5, .crm = 1, .opc1 = 0, .opc2 = 0,
+      .access = PL1_R, .fieldoffset = offsetof(CPUARMState, cp15.c5_adfsr) },
+    { .name = "AIFSR", .cp = 15, .crn = 5, .crm = 1, .opc1 = 0, .opc2 = 1,
+      .access = PL1_R, .fieldoffset = offsetof(CPUARMState, cp15.c5_aifsr) },
     REGINFO_SENTINEL
 };
 
@@ -1922,8 +1926,8 @@  CpuDefinitionInfoList *arch_query_cpu_definitions(Error **errp)
 }
 
 static void add_cpreg_to_hashtable(ARMCPU *cpu, const ARMCPRegInfo *r,
-                                   void *opaque, int state,
-                                   int crm, int opc1, int opc2)
+                                   void *opaque, int state, int crm, int opc1,
+                                   int opc2, ARMCPRegInfo **parent)
 {
     /* Private utility function for define_one_arm_cp_reg_with_opaque():
      * add a single reginfo struct to the hash table.
@@ -1931,6 +1935,12 @@  static void add_cpreg_to_hashtable(ARMCPU *cpu, const ARMCPRegInfo *r,
     uint32_t *key = g_new(uint32_t, 1);
     ARMCPRegInfo *r2 = g_memdup(r, sizeof(ARMCPRegInfo));
     int is64 = (r->type & ARM_CP_64BIT) ? 1 : 0;
+
+    /* these fields are used when migrating from TCG to KVM or the other way
+     * around*/
+    r2->skip_cpreglist = false;
+    r2->parent = NULL;
+
     if (r->state == ARM_CP_STATE_BOTH && state == ARM_CP_STATE_AA32) {
         /* The AArch32 view of a shared register sees the lower 32 bits
          * of a 64 bit backing field. It is not migratable as the AArch64
@@ -1945,6 +1955,9 @@  static void add_cpreg_to_hashtable(ARMCPU *cpu, const ARMCPRegInfo *r,
             r2->fieldoffset += sizeof(uint32_t);
         }
 #endif
+        if (*parent != NULL) {
+            r2->parent = *parent;
+        }
     }
     if (state == ARM_CP_STATE_AA64) {
         /* To allow abbreviation of ARMCPRegInfo
@@ -1979,6 +1992,21 @@  static void add_cpreg_to_hashtable(ARMCPU *cpu, const ARMCPRegInfo *r,
         ((r->opc1 == CP_ANY) && opc1 != 0) ||
         ((r->opc2 == CP_ANY) && opc2 != 0)) {
         r2->type |= ARM_CP_NO_MIGRATE;
+        if (*parent != NULL) {
+            r2->parent = *parent;
+        }
+    }
+
+    /* Check if this register is going to generate other "child"
+     * registers, in this case set the parent pointer.
+     * */
+    bool crm_cond   = (r->crm != CP_ANY)  ? true : (crm == 0);
+    bool opc1_cond  = (r->opc1 != CP_ANY) ? true : (opc1 == 0);
+    bool opc2_cond  = (r->opc2 != CP_ANY) ? true : (opc2 == 0);
+    bool state_cond = (state == ARM_CP_STATE_AA64 &&
+                      r->state == ARM_CP_STATE_BOTH);
+    if ((crm_cond && opc1_cond && opc2_cond) || state_cond) {
+        *parent = r2;
     }
 
     /* Overriding of an existing definition must be explicitly
@@ -2094,22 +2122,149 @@  void define_one_arm_cp_reg_with_opaque(ARMCPU *cpu,
     }
     /* Bad type field probably means missing sentinel at end of reg list */
     assert(cptype_valid(r->type));
+    ARMCPRegInfo *parent = NULL;
     for (crm = crmmin; crm <= crmmax; crm++) {
         for (opc1 = opc1min; opc1 <= opc1max; opc1++) {
             for (opc2 = opc2min; opc2 <= opc2max; opc2++) {
-                for (state = ARM_CP_STATE_AA32;
-                     state <= ARM_CP_STATE_AA64; state++) {
+                for (state = ARM_CP_STATE_AA64;
+                     state >= ARM_CP_STATE_AA32; state--) {
                     if (r->state != state && r->state != ARM_CP_STATE_BOTH) {
                         continue;
                     }
                     add_cpreg_to_hashtable(cpu, r, opaque, state,
-                                           crm, opc1, opc2);
+                                           crm, opc1, opc2, &parent);
                 }
             }
         }
     }
 }
 
+static void add_pair_to_list(uint64_t id, uint64_t val, GList **list)
+{
+    cp_pair *new = g_malloc(sizeof(cp_pair));
+    new->id = id;
+    new->val = val;
+    *list = g_list_prepend(*list, new);
+}
+
+static void set_skip_cpreglist(gpointer key, gpointer value, gpointer type)
+{
+    ARMCPRegInfo *reg = (ARMCPRegInfo *)value;
+
+    if ((type != NULL && reg->type & *(int *)type) || type == NULL) {
+        reg->skip_cpreglist = true;
+    }
+}
+
+static void unset_skip_cpreglist(gpointer key, gpointer value, gpointer type)
+{
+    ARMCPRegInfo *reg = (ARMCPRegInfo *)value;
+
+    if ((type != NULL && reg->type & *(int *)type) || type == NULL) {
+        reg->skip_cpreglist = false;
+    }
+}
+
+int compare_cpreg_array(uint32_t len_a, uint64_t *idx_a, uint64_t *val_a,
+                        uint32_t len_b, uint64_t *idx_b, uint64_t *val_b,
+                        GList **mis_list)
+{
+    int i = 0, j = 0, ret = 0;
+
+    while (i < len_a && j < len_b) {
+        if (idx_b[j] > idx_a[i]) {
+            i++;
+            continue;
+        }
+        if (idx_b[j] < idx_a[i]) {
+            if (mis_list != NULL) {
+                add_pair_to_list(idx_b[j], val_b[j], mis_list);
+            }
+            j++;
+
+            ret = -1;
+            continue;
+        }
+        val_a[i] = val_b[j];
+        j++;
+        i++;
+    }
+
+    if ((i == len_a) && (j != len_b)) {
+        if (mis_list != NULL) {
+            for (i = j; i < len_b; i++) {
+                add_pair_to_list(idx_b[i], val_b[i], mis_list);
+            }
+        }
+
+        ret = -1;
+    }
+
+    return ret;
+}
+
+int handle_cpreg_kvm2tcg_migration(ARMCPU *cpu, GList *mis_list, uint32_t len)
+{
+    GList *ptr_l = mis_list;
+
+    /* restore skip_cpreglist flag from last migration */
+    g_hash_table_foreach(cpu->cp_regs, unset_skip_cpreglist, NULL);
+
+    while (ptr_l != NULL) {
+        GList *n = ptr_l->next;
+        cp_pair *el = (cp_pair *)ptr_l->data;
+        const ARMCPRegInfo *ri;
+        /*
+         * KVM migrates all the CSSIDR cp registers while QEMU doesn't.
+         * They don't follow the usual nomenclature.
+         **/
+        switch (el->id & CP_REG_ARM_COPROC_MASK) {
+        case KVM_REG_ARM_CORE:
+        case KVM_REG_ARM_DEMUX:
+        case KVM_REG_ARM_VFP:
+            g_free(el);
+            mis_list = g_list_delete_link(mis_list, ptr_l);
+
+            break;
+        default:
+            /* probably its value has already been copied because
+             * the register is covered by a wilcarded case or it's
+             * a register not migrated by TCG. In any case, the
+             * register type has to be ARM_CP_NO_MIGRATE.
+             */
+            ri = get_arm_cp_reginfo(cpu->cp_regs, kvm_to_cpreg_id(el->id));
+            if (ri) {
+                if ((ri->type & ARM_CP_NO_MIGRATE) == 0) {
+                    /* error */
+                    return -1;
+                }
+                /* If it has a migratable parent, we raw write its value,
+                 * otherwise we just remove it from the list since not
+                 * migratable.
+                 * */
+                if (ri->parent != NULL && !((*(ri->parent)).type &
+                                            ARM_CP_NO_MIGRATE)) {
+                    write_raw_cp_reg(&cpu->env, ri, el->val);
+                    (*(ri->parent)).skip_cpreglist = true;
+                }
+
+                g_free(el);
+                mis_list = g_list_delete_link(mis_list, ptr_l);
+            } else {
+                return -1;
+            }
+        }
+        ptr_l = n;
+    }
+
+    /* set all ARM_CP_CONST as already migrated, this will prevent the migration
+     * from failing when TCG and KVM constant registers do not match */
+    int type = ARM_CP_CONST;
+    g_hash_table_foreach(cpu->cp_regs, set_skip_cpreglist, &type);
+
+    return g_list_length(mis_list);
+}
+
 void define_arm_cp_regs_with_opaque(ARMCPU *cpu,
                                     const ARMCPRegInfo *regs, void *opaque)
 {
diff --git a/target-arm/machine.c b/target-arm/machine.c
index 8f9e7d4..244ba41 100644
--- a/target-arm/machine.c
+++ b/target-arm/machine.c
@@ -155,11 +155,13 @@  static void cpu_pre_save(void *opaque)
     ARMCPU *cpu = opaque;
 
     if (kvm_enabled()) {
+        cpu->running_kvm = true;
         if (!write_kvmstate_to_list(cpu)) {
             /* This should never fail */
             abort();
         }
     } else {
+        cpu->running_kvm = false;
         if (!write_cpustate_to_list(cpu)) {
             /* This should never fail. */
             abort();
@@ -176,7 +178,6 @@  static void cpu_pre_save(void *opaque)
 static int cpu_post_load(void *opaque, int version_id)
 {
     ARMCPU *cpu = opaque;
-    int i, v;
 
     /* Update the values list from the incoming migration data.
      * Anything in the incoming data which we don't know about is
@@ -186,22 +187,42 @@  static int cpu_post_load(void *opaque, int version_id)
      * incoming migration index list so we can match the values array
      * entries with the right slots in our own values array.
      */
+    GList *missing_regs = NULL;
+    if (compare_cpreg_array(cpu->cpreg_array_len, cpu->cpreg_indexes,
+                cpu->cpreg_values, cpu->cpreg_vmstate_array_len,
+                cpu->cpreg_vmstate_indexes, cpu->cpreg_vmstate_values,
+                                                    &missing_regs)) {
+        /* three different cases:
+         *          1. KVM to TCG migration
+         *          2. TCG to KVM migration
+         *          3. KVM to KVM / TCG to TCG migration*/
+        if (cpu->running_kvm && !kvm_enabled()) {
+            /* case 1. */
+            if (handle_cpreg_kvm2tcg_migration(cpu, missing_regs,
+                                 g_list_length(missing_regs))) {
+                g_list_free(missing_regs);
+
+                return -1;
+            }
+            /* update the value for future migrations */
+            cpu->running_kvm = false;
+        } else if (!cpu->running_kvm && kvm_enabled()) {
+            /* case 2. TODO */
+            cpu->running_kvm = true;
+            g_list_free(missing_regs);
+
+            return -1;
+        } else {
+            /* case 3. - regular migration, an error occurred */
+            g_list_free(missing_regs);
 
-    for (i = 0, v = 0; i < cpu->cpreg_array_len
-             && v < cpu->cpreg_vmstate_array_len; i++) {
-        if (cpu->cpreg_vmstate_indexes[v] > cpu->cpreg_indexes[i]) {
-            /* register in our list but not incoming : skip it */
-            continue;
-        }
-        if (cpu->cpreg_vmstate_indexes[v] < cpu->cpreg_indexes[i]) {
-            /* register in their list but not ours: fail migration */
             return -1;
         }
-        /* matching register, copy the value over */
-        cpu->cpreg_values[i] = cpu->cpreg_vmstate_values[v];
-        v++;
     }
 
+    /* free the list */
+    g_list_free(missing_regs);
+
     if (kvm_enabled()) {
         if (!write_list_to_kvmstate(cpu)) {
             return -1;
@@ -238,6 +259,7 @@  const VMStateDescription vmstate_arm_cpu = {
             .offset = 0,
         },
         VMSTATE_UINT32(env.spsr, ARMCPU),
+        VMSTATE_BOOL(running_kvm, ARMCPU),
         VMSTATE_UINT32_ARRAY(env.banked_spsr, ARMCPU, 6),
         VMSTATE_UINT32_ARRAY(env.banked_r13, ARMCPU, 6),
         VMSTATE_UINT32_ARRAY(env.banked_r14, ARMCPU, 6),