diff mbox

RFC migration of zero pages

Message ID 5DEF77B3-5ECB-430D-9C87-EAE149BB0280@dlhnet.de
State New
Headers show

Commit Message

Peter Lieven Jan. 31, 2013, 11:53 a.m. UTC 
RFC patch is attached. Comments appreciated.
I have two concerns left:
a) what happens if a page turns from zero to non-zero in the first stage. Is
this page transferred in the same round or in the next?
b) what happens if live migration fails or is aborted and then again
a migration is started to the same target (if this is possible). Is the
memory at the target reinitialized?

Am 31.01.2013 um 10:37 schrieb Orit Wasserman <owasserm@redhat.com>:

> On 01/31/2013 11:25 AM, Peter Lieven wrote:
>> 
>> Am 31.01.2013 um 10:19 schrieb Orit Wasserman <owasserm@redhat.com>:
>> 
>>> On 01/31/2013 11:00 AM, Peter Lieven wrote:
>>>> 
>>>> Am 31.01.2013 um 09:59 schrieb Orit Wasserman <owasserm@redhat.com>:
>>>> 
>>>>> On 01/31/2013 10:37 AM, Peter Lieven wrote:
>>>>>> 
>>>>>> Am 31.01.2013 um 09:33 schrieb Orit Wasserman <owasserm@redhat.com>:
>>>>>> 
>>>>>>> On 01/31/2013 10:10 AM, Peter Lieven wrote:
>>>>>>>> 
>>>>>>>> Am 31.01.2013 um 08:47 schrieb Orit Wasserman <owasserm@redhat.com>:
>>>>>>>> 
>>>>>>>>> On 01/31/2013 08:57 AM, Peter Lieven wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I just came across an idea and would like to have feedback if it makes sence or not.
>>>>>>>>>> 
>>>>>>>>>> If a VM is started without preallocated memory all memory that has not been written to
>>>>>>>>>> reads as zeros, right?
>>>>>>>>> Hi,
>>>>>>>>> No the memory will be unmapped (we allocate on demand).
>>>>>>>> 
>>>>>>>> Yes, but those unmapped pages will read as zeroes if the guest accesses it?
>>>>>>> yes.
>>>>>>>> 
>>>>>>>>>> If a VM with a lot of unwritten memory is migrated or if the memory contains a lot
>>>>>>>>>> of zeroed out memory (e.g. Windows or Linux guest with page sanitization) all this memory
>>>>>>>>>> is allocated on the target during live migration. Especially with KSM this leads
>>>>>>>>>> to the problem that this memory is allocated and might be not available completely as
>>>>>>>>>> merging of the pages will happen async.
>>>>>>>>>> 
>>>>>>>>>> Wouldn't it make sense to not send zero pages in the first round where the complete
>>>>>>>>>> ram is sent (if it is detectable that we are in this stage)?
>>>>>>>>> We send one byte per zero page at the moment (see is_dup_page) we can further optimizing it
>>>>>>>>> by not sending it.
>>>>>>>>> I have to point out that this is a very idle guest and we need to work on a loaded guest 
>>>>>>>>> which is the more hard problem in migration.
>>>>>>>> 
>>>>>>>> I was not talking about saving one byte (+ 8 bytes for header), my concern was that we memset all (dup) pages
>>>>>>>> including the special case of a zero dup page on the migration target. This allocates the memory or does it not?
>>>>>>>> 
>>>>>>> 
>>>>>>>> If my above assumption that the guest reads unmapped memory as zeroes is right, this mapping
>>>>>>>> is not necessary in the case of a zero dup page.
>>>>>>>> 
>>>>>>>> We just have to make sure that we are still in the very first round when deciding not to sent
>>>>>>>> a zero page, because otherwise it could be a page that has become zero during migration and
>>>>>>>> this of course has to be transferred.
>>>>>>> 
>>>>>>> OK, so if we won't send the pages than it won't be allocate in the dst and it can improve both 
>>>>>>> memory usage and reduce cpu consumption on it.
>>>>>>> That can be good for over commit scenario.
>>>>>> 
>>>>>> Yes. On the Source host those zero pages have likely all been merged by KSM already, but on the destination
>>>>>> they are allocated and initially consume real memory. This can be a problem if a lot of incoming migrations happen
>>>>>> at the same time.
>>>>> 
>>>>> That can be very effective.
>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Also I notice that the bottle neck in migrating unmapped pages is the detection of those pages
>>>>>>>>> because we map the pages in order to check them, for a large guest this is very expensive as mapping a page
>>>>>>>>> results in a page fault in the host.
>>>>>>>>> So what will be very helpful is actually locating those pages without mapping them
>>>>>>>>> which looks very complicated.
>>>>>>>> 
>>>>>>>> This would be a nice improvement, but as you said a guest will sooner or later allocate
>>>>>>>> all memory if it is not totally idle. However, bigger parts of this memory might have been reset to zeroes.
>>>>>>>> This happens on page deallocation in a Windows Guest by default and can also be enforced in LInux
>>>>>>>> with page sanitization.
>>>>>>> 
>>>>>>> true, but it those cases we will want to zero the page in the dst as this is done for security reasons.
>>>>>> 
>>>>>> if i migrate it to a destination where initially all memory is unmapped not migrating the zero page turns it
>>>>>> into an unmapped page (which reads a zero?). where is the security problem? its like rethinning on a storage.
>>>>>> Or do I understand something wrong here? Is the actual mapping information migrated?
>>>>> 
>>>>> I was referring to pages that had some data and were migrated, so when the guest OS zeros them we need to zero them
>>>>> also in destination because the data is also there.
>>>> 
>>>> Ok, so can we with the current implementation effectively decide if a page is transferred for the first time?
>>> 
>>> In the old code (before 1.3 or 1.2  we add a separate function for the first full transfer but now we don't.
>>> So I guess you will need to implement it, it shouldn't be too complicated.
>>> I would add a flag to the existing code.
>>>> 
>>>> Do we always migrate the complete memory once and then iterate over dirty pages? I have to check the code
>>>> that searches for dirty pages to confirm that.
>>> We set all the bitmap as dirty in the beginning of migration so in the first iteration all pages will be sent.
>>> The code is in arch_init.c, look at ram_save_setup and ram_save_iterate.
>> 
>> I will have a look and sent a RFC patch once I have tested it.
> Great!
diff mbox

Patch

diff --git a/arch_init.c b/arch_init.c
index dada6de..33f3b12 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -426,6 +426,8 @@  static void migration_bitmap_sync(void)
  *           0 means no dirty pages
  */
 
+static uint64_t complete_rounds;
+
 static int ram_save_block(QEMUFile *f, bool last_stage)
 {
     RAMBlock *block = last_seen_block;
@@ -451,6 +453,10 @@  static int ram_save_block(QEMUFile *f, bool last_stage)
             if (!block) {
                 block = QTAILQ_FIRST(&ram_list.blocks);
                 complete_round = true;
+                if (!complete_rounds) {
+                    error_report("ram_save_block: finished bulk ram migration");
+                }
+                complete_rounds++;
             }
         } else {
             uint8_t *p;
@@ -463,10 +469,17 @@  static int ram_save_block(QEMUFile *f, bool last_stage)
             bytes_sent = -1;
             if (is_dup_page(p)) {
                 acct_info.dup_pages++;
-                bytes_sent = save_block_hdr(f, block, offset, cont,
+                /* we can skip transferring zero pages in the first round because
+                   memory is unmapped (reads as zero) at the target anyway or initialized
+                   to zero in case of mem-prealloc. */
+                if (complete_rounds || *p) {
+                    bytes_sent = save_block_hdr(f, block, offset, cont,
                                             RAM_SAVE_FLAG_COMPRESS);
-                qemu_put_byte(f, *p);
-                bytes_sent += 1;
+                    qemu_put_byte(f, *p);
+                    bytes_sent += 1;
+                } else {
+                    bytes_sent = 1;
+                }
             } else if (migrate_use_xbzrle()) {
                 current_addr = block->offset + offset;
                 bytes_sent = save_xbzrle_page(f, p, current_addr, block,
@@ -569,6 +582,7 @@  static int ram_save_setup(QEMUFile *f, void *opaque)
 
     qemu_mutex_lock_ramlist();
     bytes_transferred = 0;
+    complete_rounds = 0;
     reset_ram_globals();
 
     if (migrate_use_xbzrle()) {