[RFC,v2,01/12] mc: add documentation for micro-checkpointing

From: "Michael R. Hines" <mrhines@us.ibm.com>

Wiki: http://wiki.qemu.org/Features/MicroCheckpointing
Github: git@github.com:hinesmr/qemu.git, 'mc' branch

NOTE: This is a direct copy of the QEMU wiki page for the convenience
of the review process. Since this series very much in flux, instead of
maintaing two copies of documentation in two different formats, this
documentation will be properly formatted in the future when the review
process has completed.

Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
---
 docs/mc.txt | 222 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 222 insertions(+)
 create mode 100644 docs/mc.txt

+6. We make every attempt to change as little of the existing migration call path as possible.

* mrhines@linux.vnet.ibm.com (mrhines@linux.vnet.ibm.com) wrote:
> From: "Michael R. Hines" <mrhines@us.ibm.com>
> 
> Wiki: http://wiki.qemu.org/Features/MicroCheckpointing
> Github: git@github.com:hinesmr/qemu.git, 'mc' branch
> 
> NOTE: This is a direct copy of the QEMU wiki page for the convenience
> of the review process. Since this series very much in flux, instead of
> maintaing two copies of documentation in two different formats, this
> documentation will be properly formatted in the future when the review
> process has completed.

It seems to be picking up some truncations as well.

> +The Micro-Checkpointing Process
> +Basic Algorithm
> +Micro-Checkpoints (MC) work against the existing live migration path in QEMU, and can effectively be understood as a "live migration that never ends". As such, iteration rounds happen at the granularity of 10s of milliseconds and perform the following steps:
> +
> +1. After N milliseconds, stop the VM.
> +3. Generate a MC by invoking the live migration software path to identify and copy dirty memory into a local staging area inside QEMU.
> +4. Resume the VM immediately so that it can make forward progress.
> +5. Transmit the checkpoint to the destination.
> +6. Repeat
> +Upon failure, load the contents of the last MC at the destination back into memory and run the VM normally.

Later you talk about the memory allocation and how you grow the memory as needed
to fit the checkpoint, have you tried going the other way and triggering the
checkpoints sooner if they're taking too much memory?

> +1. MC over TCP/IP: Once the socket connection breaks, we assume
> failure. This happens very early in the loss of the latest MC not only
> because a very large amount of bytes is typically being sequenced in a
> TCP stream but perhaps also because of the timeout in acknowledgement
> of the receipt of a commit message by the destination.
> +
> +2. MC over RDMA: Since Infiniband does not provide any underlying
> timeout mechanisms, this implementation enhances QEMU's RDMA migration
> protocol to include a simple keep-alive. Upon the loss of multiple
> keep-alive messages, the sender is deemed to have failed.
> +
> +In both cases, either due to a failed TCP socket connection or lost RDMA keep-alive group, both the sender or the receiver can be deemed to have failed.
> +
> +If the sender is deemed to have failed, the destination takes over immediately using the contents of the last checkpoint.
> +
> +If the destination is deemed to be lost, we perform the same action
> as a live migration: resume the sender normally and wait for management
> software to make a policy decision about whether or not to re-protect
> the VM, which may involve a third-party to identify a new destination
>host again to use as a backup for the VM.

In this world what is making the decision about whether the sender/destination
should win - how do you avoid a split brain situation where both
VMs are running but the only thing that failed is the comms between them?
Is there any guarantee that you'll have received knowledge of the comms
failure before you pull the plug out and enable the corked packets to be
sent on the sender side?

<snip>

> +RDMA is used for two different reasons:
> +
> +1. Checkpoint generation (RDMA-based memcpy):
> +2. Checkpoint transmission
> +Checkpoint generation must be done while the VM is paused. In the
> worst case, the size of the checkpoint can be equal in size to the amount
> of memory in total use by the VM. In order to resume VM execution as
> fast as possible, the checkpoint is copied consistently locally into
> a staging area before transmission. A standard memcpy() of potentially
> such a large amount of memory not only gets no use out of the CPU cache
> but also potentially clogs up the CPU pipeline which would otherwise
> be useful by other neighbor VMs on the same physical node that could be
> scheduled for execution. To minimize the effect on neighbor VMs, we use
> RDMA to perform a "local" memcpy(), bypassing the host processor. On
> more recent processors, a 'beefy' enough memory bus architecture can
> move memory just as fast (sometimes faster) as a pure-software CPU-only
> optimized memcpy() from libc. However, on older computers, this feature
> only gives you the benefit of lower CPU-utilization at the expense of

Isn't there a generic kernel DMA ABI for doing this type of thing (I
think there was at one point, people have suggested things like using
graphics cards to do it but I don't know if it ever happened).
The other question is, do you always need to copy - what about something
like COWing the pages?

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

On 02/18/2014 08:45 PM, Dr. David Alan Gilbert wrote:
>> +The Micro-Checkpointing Process
>> +Basic Algorithm
>> +Micro-Checkpoints (MC) work against the existing live migration path in QEMU, and can effectively be understood as a "live migration that never ends". As such, iteration rounds happen at the granularity of 10s of milliseconds and perform the following steps:
>> +
>> +1. After N milliseconds, stop the VM.
>> +3. Generate a MC by invoking the live migration software path to identify and copy dirty memory into a local staging area inside QEMU.
>> +4. Resume the VM immediately so that it can make forward progress.
>> +5. Transmit the checkpoint to the destination.
>> +6. Repeat
>> +Upon failure, load the contents of the last MC at the destination back into memory and run the VM normally.
> Later you talk about the memory allocation and how you grow the memory as needed
> to fit the checkpoint, have you tried going the other way and triggering the
> checkpoints sooner if they're taking too much memory?

There is a 'knob' in this patch called "mc-set-delay" which was designed
to solve exactly that problem. It allows policy or management software
to make an independent decision about what the frequency of the
checkpoints should be.

I wasn't comfortable implementing policy directly inside the patch as
that seemed less likely to get accepted by the community sooner.

>> +1. MC over TCP/IP: Once the socket connection breaks, we assume
>> failure. This happens very early in the loss of the latest MC not only
>> because a very large amount of bytes is typically being sequenced in a
>> TCP stream but perhaps also because of the timeout in acknowledgement
>> of the receipt of a commit message by the destination.
>> +
>> +2. MC over RDMA: Since Infiniband does not provide any underlying
>> timeout mechanisms, this implementation enhances QEMU's RDMA migration
>> protocol to include a simple keep-alive. Upon the loss of multiple
>> keep-alive messages, the sender is deemed to have failed.
>> +
>> +In both cases, either due to a failed TCP socket connection or lost RDMA keep-alive group, both the sender or the receiver can be deemed to have failed.
>> +
>> +If the sender is deemed to have failed, the destination takes over immediately using the contents of the last checkpoint.
>> +
>> +If the destination is deemed to be lost, we perform the same action
>> as a live migration: resume the sender normally and wait for management
>> software to make a policy decision about whether or not to re-protect
>> the VM, which may involve a third-party to identify a new destination
>> host again to use as a backup for the VM.
> In this world what is making the decision about whether the sender/destination
> should win - how do you avoid a split brain situation where both
> VMs are running but the only thing that failed is the comms between them?
> Is there any guarantee that you'll have received knowledge of the comms
> failure before you pull the plug out and enable the corked packets to be
> sent on the sender side?

Good question in general - I'll add it to the FAQ. The patch implements
a basic 'transaction' mechanism in coordination with an outbound I/O
buffer (documented further down). With these two things in
places, split-brain is not possible because the destination is not running.
We don't allow the destination to resume execution until a committed
transaction has been acknowledged by the destination and only until
then do we allow any outbound network traffic to be release to the
outside world.

> <snip>
>
>> +RDMA is used for two different reasons:
>> +
>> +1. Checkpoint generation (RDMA-based memcpy):
>> +2. Checkpoint transmission
>> +Checkpoint generation must be done while the VM is paused. In the
>> worst case, the size of the checkpoint can be equal in size to the amount
>> of memory in total use by the VM. In order to resume VM execution as
>> fast as possible, the checkpoint is copied consistently locally into
>> a staging area before transmission. A standard memcpy() of potentially
>> such a large amount of memory not only gets no use out of the CPU cache
>> but also potentially clogs up the CPU pipeline which would otherwise
>> be useful by other neighbor VMs on the same physical node that could be
>> scheduled for execution. To minimize the effect on neighbor VMs, we use
>> RDMA to perform a "local" memcpy(), bypassing the host processor. On
>> more recent processors, a 'beefy' enough memory bus architecture can
>> move memory just as fast (sometimes faster) as a pure-software CPU-only
>> optimized memcpy() from libc. However, on older computers, this feature
>> only gives you the benefit of lower CPU-utilization at the expense of
> Isn't there a generic kernel DMA ABI for doing this type of thing (I
> think there was at one point, people have suggested things like using
> graphics cards to do it but I don't know if it ever happened).
> The other question is, do you always need to copy - what about something
> like COWing the pages?

Excellent question! Responding in two parts:

1) The kernel ABI 'vmsplice' is what I think you're referring to. Correct
      me if I'm wrong, but vmsplice was actually designed to avoid copies
      entirely between two userspace programs to be able to move memory
      more efficiently - whereas a fault tolerant system actually *needs*
      copy to be made.

2) Using COW: Actually, I think that's an excellent idea. I've bounced that
      around with my colleagues, but we simply didn't have the manpower
      to implement it and benchmark it. There was also some concern about
      performance: Would the writable working set of the guest be so 
active/busy
      that COW would not get you much benefit? I think it's worth a try.
      Patches welcome =)

- Michael

* Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
> On 02/18/2014 08:45 PM, Dr. David Alan Gilbert wrote:
> >>+The Micro-Checkpointing Process
> >>+Basic Algorithm
> >>+Micro-Checkpoints (MC) work against the existing live migration path in QEMU, and can effectively be understood as a "live migration that never ends". As such, iteration rounds happen at the granularity of 10s of milliseconds and perform the following steps:
> >>+
> >>+1. After N milliseconds, stop the VM.
> >>+3. Generate a MC by invoking the live migration software path to identify and copy dirty memory into a local staging area inside QEMU.
> >>+4. Resume the VM immediately so that it can make forward progress.
> >>+5. Transmit the checkpoint to the destination.
> >>+6. Repeat
> >>+Upon failure, load the contents of the last MC at the destination back into memory and run the VM normally.
> >Later you talk about the memory allocation and how you grow the memory as needed
> >to fit the checkpoint, have you tried going the other way and triggering the
> >checkpoints sooner if they're taking too much memory?
> 
> There is a 'knob' in this patch called "mc-set-delay" which was designed
> to solve exactly that problem. It allows policy or management software
> to make an independent decision about what the frequency of the
> checkpoints should be.
> 
> I wasn't comfortable implementing policy directly inside the patch as
> that seemed less likely to get accepted by the community sooner.

I was just wondering if a separate 'max buffer size' knob would allow
you to more reasonably bound memory without setting policy; I don't think
people like having potentially x2 memory.

> >>+1. MC over TCP/IP: Once the socket connection breaks, we assume
> >>failure. This happens very early in the loss of the latest MC not only
> >>because a very large amount of bytes is typically being sequenced in a
> >>TCP stream but perhaps also because of the timeout in acknowledgement
> >>of the receipt of a commit message by the destination.
> >>+
> >>+2. MC over RDMA: Since Infiniband does not provide any underlying
> >>timeout mechanisms, this implementation enhances QEMU's RDMA migration
> >>protocol to include a simple keep-alive. Upon the loss of multiple
> >>keep-alive messages, the sender is deemed to have failed.
> >>+
> >>+In both cases, either due to a failed TCP socket connection or lost RDMA keep-alive group, both the sender or the receiver can be deemed to have failed.
> >>+
> >>+If the sender is deemed to have failed, the destination takes over immediately using the contents of the last checkpoint.
> >>+
> >>+If the destination is deemed to be lost, we perform the same action
> >>as a live migration: resume the sender normally and wait for management
> >>software to make a policy decision about whether or not to re-protect
> >>the VM, which may involve a third-party to identify a new destination
> >>host again to use as a backup for the VM.
> >In this world what is making the decision about whether the sender/destination
> >should win - how do you avoid a split brain situation where both
> >VMs are running but the only thing that failed is the comms between them?
> >Is there any guarantee that you'll have received knowledge of the comms
> >failure before you pull the plug out and enable the corked packets to be
> >sent on the sender side?
> 
> Good question in general - I'll add it to the FAQ. The patch implements
> a basic 'transaction' mechanism in coordination with an outbound I/O
> buffer (documented further down). With these two things in
> places, split-brain is not possible because the destination is not running.
> We don't allow the destination to resume execution until a committed
> transaction has been acknowledged by the destination and only until
> then do we allow any outbound network traffic to be release to the
> outside world.

Yeh I see the IO buffer, what I've not figured out is how:
  1) MC over TCP/IP gets an acknowledge on the source to know when
     it can unplug it's buffer.
  2) Lets say the MC connection fails, so that ack never arrives,
     the source must assume the destination has failed and release it's
     packets and carry on.
     The destination must assume the source has failed and take over.

     Now they're both running - and that's bad and it's standard
     split brain.
  3) If we're relying on TCP/IP timeout that's quite long.

> >>+RDMA is used for two different reasons:
> >>+
> >>+1. Checkpoint generation (RDMA-based memcpy):
> >>+2. Checkpoint transmission
> >>+Checkpoint generation must be done while the VM is paused. In the
> >>worst case, the size of the checkpoint can be equal in size to the amount
> >>of memory in total use by the VM. In order to resume VM execution as
> >>fast as possible, the checkpoint is copied consistently locally into
> >>a staging area before transmission. A standard memcpy() of potentially
> >>such a large amount of memory not only gets no use out of the CPU cache
> >>but also potentially clogs up the CPU pipeline which would otherwise
> >>be useful by other neighbor VMs on the same physical node that could be
> >>scheduled for execution. To minimize the effect on neighbor VMs, we use
> >>RDMA to perform a "local" memcpy(), bypassing the host processor. On
> >>more recent processors, a 'beefy' enough memory bus architecture can
> >>move memory just as fast (sometimes faster) as a pure-software CPU-only
> >>optimized memcpy() from libc. However, on older computers, this feature
> >>only gives you the benefit of lower CPU-utilization at the expense of
> >Isn't there a generic kernel DMA ABI for doing this type of thing (I
> >think there was at one point, people have suggested things like using
> >graphics cards to do it but I don't know if it ever happened).
> >The other question is, do you always need to copy - what about something
> >like COWing the pages?
> 
> Excellent question! Responding in two parts:
> 
> 1) The kernel ABI 'vmsplice' is what I think you're referring to. Correct
>      me if I'm wrong, but vmsplice was actually designed to avoid copies
>      entirely between two userspace programs to be able to move memory
>      more efficiently - whereas a fault tolerant system actually *needs*
>      copy to be made.

No, I wasn't thinking of vmsplice; I just have vague memories of suggestions
of the use of Intel's I/OAT, graphics cards, etc for doing things like page
zeroing and DMAing data around; I can see there is a dmaengine API in the
kernel, I haven't found where if anywhere that is available to userspace.

> 2) Using COW: Actually, I think that's an excellent idea. I've bounced that
>      around with my colleagues, but we simply didn't have the manpower
>      to implement it and benchmark it. There was also some concern about
>      performance: Would the writable working set of the guest be so
> active/busy
>      that COW would not get you much benefit? I think it's worth a try.
>      Patches welcome =)

It's possible that might be doable with some of the same tricks I'm
looking at for post-copy, I'll see what I can do.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
>
> I was just wondering if a separate 'max buffer size' knob would allow
> you to more reasonably bound memory without setting policy; I don't think
> people like having potentially x2 memory.

Note: Checkpoint memory is not monotonic in this patchset (which
is unique to this implementation). Only if the guest actually dirties
100% of it's memory between one checkpoint to the next will
the host experience 2x memory usage for a short period of time.

The patch has a 'slab' mechanism built in to it which implements
a water-mark style policy that throws away unused portions of
the 2x checkpoint memory if later checkpoints are much smaller
(which is likely to be the case if the writable working set size changes).

However, to answer your question: Such a knob could be achieved, but
the same could be achieved simply by tuning the checkpoint frequency
itself. Memory usage would thus be a function of the checkpoint frequency.

If the guest application was maniacal, banging away at all the memory,
there's very little that can be done in the first place, but if the 
guest application
was mildly busy, you don't want to throw away your ability to be fault
tolerant - you would just need more frequent checkpoints to keep up with
the dirty rate.

Once the application died down - the water-mark policy would kick in
and start freeing checkpoint memory. (Note: this policy happens on
both sides in the patchset because the patch has to be fully compatible
with RDMA memory pinning).

What is *not* exposed, however, is the watermark knobs themselves,
I definitely think that needs to be exposed - that would also get you a 
similar
control to 'max buffer size' - you could place a time limit on the
slab list in the patch or something like that.......


>>
>> Good question in general - I'll add it to the FAQ. The patch implements
>> a basic 'transaction' mechanism in coordination with an outbound I/O
>> buffer (documented further down). With these two things in
>> places, split-brain is not possible because the destination is not running.
>> We don't allow the destination to resume execution until a committed
>> transaction has been acknowledged by the destination and only until
>> then do we allow any outbound network traffic to be release to the
>> outside world.
> Yeh I see the IO buffer, what I've not figured out is how:
>    1) MC over TCP/IP gets an acknowledge on the source to know when
>       it can unplug it's buffer.

Only partially correct (See the steps on the wiki). There are two I/O
buffers at any given time which protect against a split-brain scenario:
One buffer for the current checkpoint that is being generated (running VM)
and one buffer for the checkpoint that is being committed in a transaction.

>    2) Lets say the MC connection fails, so that ack never arrives,
>       the source must assume the destination has failed and release it's
>       packets and carry on.

Only the packets for Buffer A are released for the current committed
checkpoint after a completed transaction. The packets for Buffer B
(the current running VM) are still being held up until the next 
transaction starts.
Later once the transaction completes and A is released, B becomes the
new A and a new buffer is installed to become the new Buffer B for
the current running VM.


>       The destination must assume the source has failed and take over.

The destination must also receive an ACK. The ack goes both ways.

Once the source and destination both acknowledge a completed
transation does the source VM resume execution - and even then
it's packets are still being buffered until the next transaction starts.
(That's why it's important to checkpoint as frequently as possible).


>    3) If we're relying on TCP/IP timeout that's quite long.
>

Actually, my experience is been that TCP seems to have more than
one kind of timeout - if receiver is not responding *at all* - it seems that
TCP has a dedicated timer for that. The socket API immediately
sends back an error code and the patchset closes the conneciton
on the destination and recovers.

> No, I wasn't thinking of vmsplice; I just have vague memories of suggestions
> of the use of Intel's I/OAT, graphics cards, etc for doing things like page
> zeroing and DMAing data around; I can see there is a dmaengine API in the
> kernel, I haven't found where if anywhere that is available to userspace.
>
>> 2) Using COW: Actually, I think that's an excellent idea. I've bounced that
>>       around with my colleagues, but we simply didn't have the manpower
>>       to implement it and benchmark it. There was also some concern about
>>       performance: Would the writable working set of the guest be so
>> active/busy
>>       that COW would not get you much benefit? I think it's worth a try.
>>       Patches welcome =)
> It's possible that might be doable with some of the same tricks I'm
> looking at for post-copy, I'll see what I can do.

That's great news - I'm very interested to see how this applies
to post-copy and any kind patches.

- Michael

* Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
> On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
> >
> >I was just wondering if a separate 'max buffer size' knob would allow
> >you to more reasonably bound memory without setting policy; I don't think
> >people like having potentially x2 memory.
> 
> Note: Checkpoint memory is not monotonic in this patchset (which
> is unique to this implementation). Only if the guest actually dirties
> 100% of it's memory between one checkpoint to the next will
> the host experience 2x memory usage for a short period of time.

Right, but that doesn't really help - if someone comes along and says
'How much memory do I need to be able to run an mc system?' the only
safe answer is 2x, otherwise we're adding a reason why the previously
stable guest might OOM.

> The patch has a 'slab' mechanism built in to it which implements
> a water-mark style policy that throws away unused portions of
> the 2x checkpoint memory if later checkpoints are much smaller
> (which is likely to be the case if the writable working set size changes).
> 
> However, to answer your question: Such a knob could be achieved, but
> the same could be achieved simply by tuning the checkpoint frequency
> itself. Memory usage would thus be a function of the checkpoint frequency.

> If the guest application was maniacal, banging away at all the memory,
> there's very little that can be done in the first place, but if the
> guest application
> was mildly busy, you don't want to throw away your ability to be fault
> tolerant - you would just need more frequent checkpoints to keep up with
> the dirty rate.

I'm not convinced; I can tune my checkpoint frequency until normal operation
makes a reasonable trade off between mc frequency and RAM usage,
but that doesn't prevent it running away when a garbage collect or some
other thing suddenly dirties a load of ram in one particular checkpoint.
Some management tool that watches ram usage etc can also help tune
it, but in the end it can't stop it taking loads of RAM.

> Once the application died down - the water-mark policy would kick in
> and start freeing checkpoint memory. (Note: this policy happens on
> both sides in the patchset because the patch has to be fully compatible
> with RDMA memory pinning).
> 
> What is *not* exposed, however, is the watermark knobs themselves,
> I definitely think that needs to be exposed - that would also get
> you a similar
> control to 'max buffer size' - you could place a time limit on the
> slab list in the patch or something like that.......
> 
> 
> >>
> >>Good question in general - I'll add it to the FAQ. The patch implements
> >>a basic 'transaction' mechanism in coordination with an outbound I/O
> >>buffer (documented further down). With these two things in
> >>places, split-brain is not possible because the destination is not running.
> >>We don't allow the destination to resume execution until a committed
> >>transaction has been acknowledged by the destination and only until
> >>then do we allow any outbound network traffic to be release to the
> >>outside world.
> >Yeh I see the IO buffer, what I've not figured out is how:
> >   1) MC over TCP/IP gets an acknowledge on the source to know when
> >      it can unplug it's buffer.
> 
> Only partially correct (See the steps on the wiki). There are two I/O
> buffers at any given time which protect against a split-brain scenario:
> One buffer for the current checkpoint that is being generated (running VM)
> and one buffer for the checkpoint that is being committed in a transaction.
> 
> >   2) Lets say the MC connection fails, so that ack never arrives,
> >      the source must assume the destination has failed and release it's
> >      packets and carry on.
> 
> Only the packets for Buffer A are released for the current committed
> checkpoint after a completed transaction. The packets for Buffer B
> (the current running VM) are still being held up until the next
> transaction starts.
> Later once the transaction completes and A is released, B becomes the
> new A and a new buffer is installed to become the new Buffer B for
> the current running VM.
> 
> 
> >      The destination must assume the source has failed and take over.
> 
> The destination must also receive an ACK. The ack goes both ways.
> 
> Once the source and destination both acknowledge a completed
> transation does the source VM resume execution - and even then
> it's packets are still being buffered until the next transaction starts.
> (That's why it's important to checkpoint as frequently as possible).

I think I understand normal operation - my question here is about failure;
what happens when neither side gets any ACKs.

> >   3) If we're relying on TCP/IP timeout that's quite long.
> >
> 
> Actually, my experience is been that TCP seems to have more than
> one kind of timeout - if receiver is not responding *at all* - it seems that
> TCP has a dedicated timer for that. The socket API immediately
> sends back an error code and the patchset closes the conneciton
> on the destination and recovers.

How did you test that?
My experience is that if a host knows that it has no route to the destination
(e.g. it has no route to try, that matches the destination, because someone
took the network interface away) you immediately get a 'no route to host',
however if an intermediate link disappears then it takes a while to time out.

> >No, I wasn't thinking of vmsplice; I just have vague memories of suggestions
> >of the use of Intel's I/OAT, graphics cards, etc for doing things like page
> >zeroing and DMAing data around; I can see there is a dmaengine API in the
> >kernel, I haven't found where if anywhere that is available to userspace.
> >
> >>2) Using COW: Actually, I think that's an excellent idea. I've bounced that
> >>      around with my colleagues, but we simply didn't have the manpower
> >>      to implement it and benchmark it. There was also some concern about
> >>      performance: Would the writable working set of the guest be so
> >>active/busy
> >>      that COW would not get you much benefit? I think it's worth a try.
> >>      Patches welcome =)
> >It's possible that might be doable with some of the same tricks I'm
> >looking at for post-copy, I'll see what I can do.
> 
> That's great news - I'm very interested to see how this applies
> to post-copy and any kind patches.
> 
> - Michael
> 

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Dr. David Alan Gilbert wrote:
> * Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
>    
>> On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
>>      
>>> I was just wondering if a separate 'max buffer size' knob would allow
>>> you to more reasonably bound memory without setting policy; I don't think
>>> people like having potentially x2 memory.
>>>        
>> Note: Checkpoint memory is not monotonic in this patchset (which
>> is unique to this implementation). Only if the guest actually dirties
>> 100% of it's memory between one checkpoint to the next will
>> the host experience 2x memory usage for a short period of time.
>>      
> Right, but that doesn't really help - if someone comes along and says
> 'How much memory do I need to be able to run an mc system?' the only
> safe answer is 2x, otherwise we're adding a reason why the previously
> stable guest might OOM.
>
>    

so we may have to involve some disk operations
to handle memory exhaustion.

Thanks!

>> The patch has a 'slab' mechanism built in to it which implements
>> a water-mark style policy that throws away unused portions of
>> the 2x checkpoint memory if later checkpoints are much smaller
>> (which is likely to be the case if the writable working set size changes).
>>
>> However, to answer your question: Such a knob could be achieved, but
>> the same could be achieved simply by tuning the checkpoint frequency
>> itself. Memory usage would thus be a function of the checkpoint frequency.
>>      
>    
>> If the guest application was maniacal, banging away at all the memory,
>> there's very little that can be done in the first place, but if the
>> guest application
>> was mildly busy, you don't want to throw away your ability to be fault
>> tolerant - you would just need more frequent checkpoints to keep up with
>> the dirty rate.
>>      
> I'm not convinced; I can tune my checkpoint frequency until normal operation
> makes a reasonable trade off between mc frequency and RAM usage,
> but that doesn't prevent it running away when a garbage collect or some
> other thing suddenly dirties a load of ram in one particular checkpoint.
> Some management tool that watches ram usage etc can also help tune
> it, but in the end it can't stop it taking loads of RAM.
>
>    
>> Once the application died down - the water-mark policy would kick in
>> and start freeing checkpoint memory. (Note: this policy happens on
>> both sides in the patchset because the patch has to be fully compatible
>> with RDMA memory pinning).
>>
>> What is *not* exposed, however, is the watermark knobs themselves,
>> I definitely think that needs to be exposed - that would also get
>> you a similar
>> control to 'max buffer size' - you could place a time limit on the
>> slab list in the patch or something like that.......
>>
>>
>>      
>>>> Good question in general - I'll add it to the FAQ. The patch implements
>>>> a basic 'transaction' mechanism in coordination with an outbound I/O
>>>> buffer (documented further down). With these two things in
>>>> places, split-brain is not possible because the destination is not running.
>>>> We don't allow the destination to resume execution until a committed
>>>> transaction has been acknowledged by the destination and only until
>>>> then do we allow any outbound network traffic to be release to the
>>>> outside world.
>>>>          
>>> Yeh I see the IO buffer, what I've not figured out is how:
>>>    1) MC over TCP/IP gets an acknowledge on the source to know when
>>>       it can unplug it's buffer.
>>>        
>> Only partially correct (See the steps on the wiki). There are two I/O
>> buffers at any given time which protect against a split-brain scenario:
>> One buffer for the current checkpoint that is being generated (running VM)
>> and one buffer for the checkpoint that is being committed in a transaction.
>>
>>      
>>>    2) Lets say the MC connection fails, so that ack never arrives,
>>>       the source must assume the destination has failed and release it's
>>>       packets and carry on.
>>>        
>> Only the packets for Buffer A are released for the current committed
>> checkpoint after a completed transaction. The packets for Buffer B
>> (the current running VM) are still being held up until the next
>> transaction starts.
>> Later once the transaction completes and A is released, B becomes the
>> new A and a new buffer is installed to become the new Buffer B for
>> the current running VM.
>>
>>
>>      
>>>       The destination must assume the source has failed and take over.
>>>        
>> The destination must also receive an ACK. The ack goes both ways.
>>
>> Once the source and destination both acknowledge a completed
>> transation does the source VM resume execution - and even then
>> it's packets are still being buffered until the next transaction starts.
>> (That's why it's important to checkpoint as frequently as possible).
>>      
> I think I understand normal operation - my question here is about failure;
> what happens when neither side gets any ACKs.
>
>    
>>>    3) If we're relying on TCP/IP timeout that's quite long.
>>>
>>>        
>> Actually, my experience is been that TCP seems to have more than
>> one kind of timeout - if receiver is not responding *at all* - it seems that
>> TCP has a dedicated timer for that. The socket API immediately
>> sends back an error code and the patchset closes the conneciton
>> on the destination and recovers.
>>      
> How did you test that?
> My experience is that if a host knows that it has no route to the destination
> (e.g. it has no route to try, that matches the destination, because someone
> took the network interface away) you immediately get a 'no route to host',
> however if an intermediate link disappears then it takes a while to time out.
>
>    
>>> No, I wasn't thinking of vmsplice; I just have vague memories of suggestions
>>> of the use of Intel's I/OAT, graphics cards, etc for doing things like page
>>> zeroing and DMAing data around; I can see there is a dmaengine API in the
>>> kernel, I haven't found where if anywhere that is available to userspace.
>>>
>>>        
>>>> 2) Using COW: Actually, I think that's an excellent idea. I've bounced that
>>>>       around with my colleagues, but we simply didn't have the manpower
>>>>       to implement it and benchmark it. There was also some concern about
>>>>       performance: Would the writable working set of the guest be so
>>>> active/busy
>>>>       that COW would not get you much benefit? I think it's worth a try.
>>>>       Patches welcome =)
>>>>          
>>> It's possible that might be doable with some of the same tricks I'm
>>> looking at for post-copy, I'll see what I can do.
>>>        
>> That's great news - I'm very interested to see how this applies
>> to post-copy and any kind patches.
>>
>> - Michael
>>
>>      
> Dave
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
>

On 02/20/2014 06:09 PM, Dr. David Alan Gilbert wrote:
> * Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
>> On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
>>> I was just wondering if a separate 'max buffer size' knob would allow
>>> you to more reasonably bound memory without setting policy; I don't think
>>> people like having potentially x2 memory.
>> Note: Checkpoint memory is not monotonic in this patchset (which
>> is unique to this implementation). Only if the guest actually dirties
>> 100% of it's memory between one checkpoint to the next will
>> the host experience 2x memory usage for a short period of time.
> Right, but that doesn't really help - if someone comes along and says
> 'How much memory do I need to be able to run an mc system?' the only
> safe answer is 2x, otherwise we're adding a reason why the previously
> stable guest might OOM.

Yes, exactly. Running MC is expensive and will probably always be
more or less to some degree. Saving memory and having 100%
fault tolerance are (at times) sometimes mutually exclusive.
Expectations have to be managed here.

The bottom line is: if you put a *hard* constraint on memory usage,
what will happen to the guest when that garbage collection you mentioned
shows up later and runs for several minutes? How about an hour?
Are we just going to block the guest from being allowed to start a
checkpoint until the memory usage goes down just for the sake of avoiding
the 2x memory usage? If you block the guest from being checkpointed,
then what happens if there is a failure during that extended period?
We will have saved memory at the expense of availability.

The customer that is expecting 100% fault tolerance and the provider
who is supporting it need to have an understanding that fault tolerance
is not free and that constraining memory usage will adversely affect
the VM's ability to be protected.

Do I understand your expectations correctly? Is fault tolerance
something you're willing to sacrifice?

>> The patch has a 'slab' mechanism built in to it which implements
>> a water-mark style policy that throws away unused portions of
>> the 2x checkpoint memory if later checkpoints are much smaller
>> (which is likely to be the case if the writable working set size changes).
>>
>> However, to answer your question: Such a knob could be achieved, but
>> the same could be achieved simply by tuning the checkpoint frequency
>> itself. Memory usage would thus be a function of the checkpoint frequency.
>> If the guest application was maniacal, banging away at all the memory,
>> there's very little that can be done in the first place, but if the
>> guest application
>> was mildly busy, you don't want to throw away your ability to be fault
>> tolerant - you would just need more frequent checkpoints to keep up with
>> the dirty rate.
> I'm not convinced; I can tune my checkpoint frequency until normal operation
> makes a reasonable trade off between mc frequency and RAM usage,
> but that doesn't prevent it running away when a garbage collect or some
> other thing suddenly dirties a load of ram in one particular checkpoint.
> Some management tool that watches ram usage etc can also help tune
> it, but in the end it can't stop it taking loads of RAM.

That's correct. See above comment....

>
>> Once the application died down - the water-mark policy would kick in
>> and start freeing checkpoint memory. (Note: this policy happens on
>> both sides in the patchset because the patch has to be fully compatible
>> with RDMA memory pinning).
>>
>> What is *not* exposed, however, is the watermark knobs themselves,
>> I definitely think that needs to be exposed - that would also get
>> you a similar
>> control to 'max buffer size' - you could place a time limit on the
>> slab list in the patch or something like that.......
>>
>>
>>>> Good question in general - I'll add it to the FAQ. The patch implements
>>>> a basic 'transaction' mechanism in coordination with an outbound I/O
>>>> buffer (documented further down). With these two things in
>>>> places, split-brain is not possible because the destination is not running.
>>>> We don't allow the destination to resume execution until a committed
>>>> transaction has been acknowledged by the destination and only until
>>>> then do we allow any outbound network traffic to be release to the
>>>> outside world.
>>> Yeh I see the IO buffer, what I've not figured out is how:
>>>    1) MC over TCP/IP gets an acknowledge on the source to know when
>>>       it can unplug it's buffer.
>> Only partially correct (See the steps on the wiki). There are two I/O
>> buffers at any given time which protect against a split-brain scenario:
>> One buffer for the current checkpoint that is being generated (running VM)
>> and one buffer for the checkpoint that is being committed in a transaction.
>>
>>>    2) Lets say the MC connection fails, so that ack never arrives,
>>>       the source must assume the destination has failed and release it's
>>>       packets and carry on.
>> Only the packets for Buffer A are released for the current committed
>> checkpoint after a completed transaction. The packets for Buffer B
>> (the current running VM) are still being held up until the next
>> transaction starts.
>> Later once the transaction completes and A is released, B becomes the
>> new A and a new buffer is installed to become the new Buffer B for
>> the current running VM.
>>
>>
>>>       The destination must assume the source has failed and take over.
>> The destination must also receive an ACK. The ack goes both ways.
>>
>> Once the source and destination both acknowledge a completed
>> transation does the source VM resume execution - and even then
>> it's packets are still being buffered until the next transaction starts.
>> (That's why it's important to checkpoint as frequently as possible).
> I think I understand normal operation - my question here is about failure;
> what happens when neither side gets any ACKs.

Well, that's simple: If there is a failure of the source, the destination
will simply revert to the previous checkpoint using the same mode
of operation. The lost ACKs that you're curious about only
apply to the checkpoint that is in progress. Just because a
checkpoint is in progress does not mean that the previous checkpoint
is thrown away - it is already loaded into the destination's memory
and ready to be activated.

>
>>>    3) If we're relying on TCP/IP timeout that's quite long.
>>>
>> Actually, my experience is been that TCP seems to have more than
>> one kind of timeout - if receiver is not responding *at all* - it seems that
>> TCP has a dedicated timer for that. The socket API immediately
>> sends back an error code and the patchset closes the conneciton
>> on the destination and recovers.
> How did you test that?
> My experience is that if a host knows that it has no route to the destination
> (e.g. it has no route to try, that matches the destination, because someone
> took the network interface away) you immediately get a 'no route to host',
> however if an intermediate link disappears then it takes a while to time out.

We have a script architecture (not on github) which runs MC in a tight
loop hundreds of times and kills the source QEMU and timestamps how 
quickly the
destination QEMU loses the TCP socket connection receives an error code
from the kernel - every single time, the destination resumes nearly 
instantaneously.
I've not empirically seen a case where the socket just hangs or doesn't
change state.

I'm not very familiar with the internal linux TCP/IP stack 
implementation itself,
but I have not had a problem with the dependability of the linux socket
not being able to shutdown the socket as soon as possible.

The RDMA implementation uses a manual keepalive mechanism that
I had to write from scratch - but I never ported this to the TCP 
implementation
simply because the failures always worked fine without it.

- Michael

On 02/20/2014 07:14 PM, Li Guang wrote:
> Dr. David Alan Gilbert wrote:
>> * Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
>>> On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
>>>> I was just wondering if a separate 'max buffer size' knob would allow
>>>> you to more reasonably bound memory without setting policy; I don't 
>>>> think
>>>> people like having potentially x2 memory.
>>> Note: Checkpoint memory is not monotonic in this patchset (which
>>> is unique to this implementation). Only if the guest actually dirties
>>> 100% of it's memory between one checkpoint to the next will
>>> the host experience 2x memory usage for a short period of time.
>> Right, but that doesn't really help - if someone comes along and says
>> 'How much memory do I need to be able to run an mc system?' the only
>> safe answer is 2x, otherwise we're adding a reason why the previously
>> stable guest might OOM.
>>
>
> so we may have to involve some disk operations
> to handle memory exhaustion.
>
> Thanks!

Like a cgroups memory limit, for example?

- Michael

* Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
> On 02/20/2014 06:09 PM, Dr. David Alan Gilbert wrote:
> >* Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
> >>On 02/19/2014 07:27 PM, Dr. David Alan Gilbert wrote:
> >>>I was just wondering if a separate 'max buffer size' knob would allow
> >>>you to more reasonably bound memory without setting policy; I don't think
> >>>people like having potentially x2 memory.
> >>Note: Checkpoint memory is not monotonic in this patchset (which
> >>is unique to this implementation). Only if the guest actually dirties
> >>100% of it's memory between one checkpoint to the next will
> >>the host experience 2x memory usage for a short period of time.
> >Right, but that doesn't really help - if someone comes along and says
> >'How much memory do I need to be able to run an mc system?' the only
> >safe answer is 2x, otherwise we're adding a reason why the previously
> >stable guest might OOM.
> 
> Yes, exactly. Running MC is expensive and will probably always be
> more or less to some degree. Saving memory and having 100%
> fault tolerance are (at times) sometimes mutually exclusive.
> Expectations have to be managed here.

I'm happy to use more memory to get FT, all I'm trying to do is see
if it's possible to put a lower bound than 2x on it while still maintaining
full FT, at the expense of performance in the case where it uses
a lot of memory.

> The bottom line is: if you put a *hard* constraint on memory usage,
> what will happen to the guest when that garbage collection you mentioned
> shows up later and runs for several minutes? How about an hour?
> Are we just going to block the guest from being allowed to start a
> checkpoint until the memory usage goes down just for the sake of avoiding
> the 2x memory usage?

Yes, or move to the next checkpoint sooner than the N milliseconds when
we see the buffer is getting full.

> If you block the guest from being checkpointed,
> then what happens if there is a failure during that extended period?
> We will have saved memory at the expense of availability.

If the active machine fails during this time then the secondary carries
on from it's last good snapshot in the knowledge that the active
never finished the new snapshot and so never uncorked it's previous packets.

If the secondary machine fails during this time then tha active drops
it's nascent snapshot and carries on.

However, what you have made me realise is that I don't have an answer
for the memory usage on the secondary; while the primary can pause
it's guest until the secondary ack's the checkpoint, the secondary has
to rely on the primary not to send it huge checkpoints.

> The customer that is expecting 100% fault tolerance and the provider
> who is supporting it need to have an understanding that fault tolerance
> is not free and that constraining memory usage will adversely affect
> the VM's ability to be protected.
> 
> Do I understand your expectations correctly? Is fault tolerance
> something you're willing to sacrifice?

As above, no I'm willing to sacrifice performance but not fault tolerance.
(It is entirely possible that others would want the other trade off, i.e.
some minimum performance is worse than useless, so if we can't maintain
that performance then dropping FT leaves us in a more-working position).

> >>The patch has a 'slab' mechanism built in to it which implements
> >>a water-mark style policy that throws away unused portions of
> >>the 2x checkpoint memory if later checkpoints are much smaller
> >>(which is likely to be the case if the writable working set size changes).
> >>
> >>However, to answer your question: Such a knob could be achieved, but
> >>the same could be achieved simply by tuning the checkpoint frequency
> >>itself. Memory usage would thus be a function of the checkpoint frequency.
> >>If the guest application was maniacal, banging away at all the memory,
> >>there's very little that can be done in the first place, but if the
> >>guest application
> >>was mildly busy, you don't want to throw away your ability to be fault
> >>tolerant - you would just need more frequent checkpoints to keep up with
> >>the dirty rate.
> >I'm not convinced; I can tune my checkpoint frequency until normal operation
> >makes a reasonable trade off between mc frequency and RAM usage,
> >but that doesn't prevent it running away when a garbage collect or some
> >other thing suddenly dirties a load of ram in one particular checkpoint.
> >Some management tool that watches ram usage etc can also help tune
> >it, but in the end it can't stop it taking loads of RAM.
> 
> That's correct. See above comment....
> 
> >
> >>Once the application died down - the water-mark policy would kick in
> >>and start freeing checkpoint memory. (Note: this policy happens on
> >>both sides in the patchset because the patch has to be fully compatible
> >>with RDMA memory pinning).
> >>
> >>What is *not* exposed, however, is the watermark knobs themselves,
> >>I definitely think that needs to be exposed - that would also get
> >>you a similar
> >>control to 'max buffer size' - you could place a time limit on the
> >>slab list in the patch or something like that.......
> >>
> >>
> >>>>Good question in general - I'll add it to the FAQ. The patch implements
> >>>>a basic 'transaction' mechanism in coordination with an outbound I/O
> >>>>buffer (documented further down). With these two things in
> >>>>places, split-brain is not possible because the destination is not running.
> >>>>We don't allow the destination to resume execution until a committed
> >>>>transaction has been acknowledged by the destination and only until
> >>>>then do we allow any outbound network traffic to be release to the
> >>>>outside world.
> >>>Yeh I see the IO buffer, what I've not figured out is how:
> >>>   1) MC over TCP/IP gets an acknowledge on the source to know when
> >>>      it can unplug it's buffer.
> >>Only partially correct (See the steps on the wiki). There are two I/O
> >>buffers at any given time which protect against a split-brain scenario:
> >>One buffer for the current checkpoint that is being generated (running VM)
> >>and one buffer for the checkpoint that is being committed in a transaction.
> >>
> >>>   2) Lets say the MC connection fails, so that ack never arrives,
> >>>      the source must assume the destination has failed and release it's
> >>>      packets and carry on.
> >>Only the packets for Buffer A are released for the current committed
> >>checkpoint after a completed transaction. The packets for Buffer B
> >>(the current running VM) are still being held up until the next
> >>transaction starts.
> >>Later once the transaction completes and A is released, B becomes the
> >>new A and a new buffer is installed to become the new Buffer B for
> >>the current running VM.
> >>
> >>
> >>>      The destination must assume the source has failed and take over.
> >>The destination must also receive an ACK. The ack goes both ways.
> >>
> >>Once the source and destination both acknowledge a completed
> >>transation does the source VM resume execution - and even then
> >>it's packets are still being buffered until the next transaction starts.
> >>(That's why it's important to checkpoint as frequently as possible).
> >I think I understand normal operation - my question here is about failure;
> >what happens when neither side gets any ACKs.
> 
> Well, that's simple: If there is a failure of the source, the destination
> will simply revert to the previous checkpoint using the same mode
> of operation. The lost ACKs that you're curious about only
> apply to the checkpoint that is in progress. Just because a
> checkpoint is in progress does not mean that the previous checkpoint
> is thrown away - it is already loaded into the destination's memory
> and ready to be activated.

I still don't see why, if the link between them fails, the destination
doesn't fall back it it's previous checkpoint, AND the source carries
on running - I don't see how they can differentiate which of them has failed.

> >>>   3) If we're relying on TCP/IP timeout that's quite long.
> >>>
> >>Actually, my experience is been that TCP seems to have more than
> >>one kind of timeout - if receiver is not responding *at all* - it seems that
> >>TCP has a dedicated timer for that. The socket API immediately
> >>sends back an error code and the patchset closes the conneciton
> >>on the destination and recovers.
> >How did you test that?
> >My experience is that if a host knows that it has no route to the destination
> >(e.g. it has no route to try, that matches the destination, because someone
> >took the network interface away) you immediately get a 'no route to host',
> >however if an intermediate link disappears then it takes a while to time out.
> 
> We have a script architecture (not on github) which runs MC in a tight
> loop hundreds of times and kills the source QEMU and timestamps how
> quickly the
> destination QEMU loses the TCP socket connection receives an error code
> from the kernel - every single time, the destination resumes nearly
> instantaneously.
> I've not empirically seen a case where the socket just hangs or doesn't
> change state.
> 
> I'm not very familiar with the internal linux TCP/IP stack
> implementation itself,
> but I have not had a problem with the dependability of the linux socket
> not being able to shutdown the socket as soon as possible.

OK, that only covers a very small range of normal failures.
When you kill the destination QEMU the host OS knows that QEMU is dead
and sends a packet back closing the socket, hence the source knows
the destination is dead very quickly.
If:
   a) The destination machine was to lose power or hang
   b) Or a network link fail  (other than the one attached to the source
      possibly)

the source would have to do a full TCP timeout.

To test a,b I'd use an iptables rule somewhere to cause the packets to
be dropped (not rejected).  Stopping the qemu in gdb might be good enough.

> The RDMA implementation uses a manual keepalive mechanism that
> I had to write from scratch - but I never ported this to the TCP
> implementation
> simply because the failures always worked fine without it.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

On 02/21/2014 12:32 AM, Dr. David Alan Gilbert wrote:
>
> I'm happy to use more memory to get FT, all I'm trying to do is see
> if it's possible to put a lower bound than 2x on it while still maintaining
> full FT, at the expense of performance in the case where it uses
> a lot of memory.
>
>> The bottom line is: if you put a *hard* constraint on memory usage,
>> what will happen to the guest when that garbage collection you mentioned
>> shows up later and runs for several minutes? How about an hour?
>> Are we just going to block the guest from being allowed to start a
>> checkpoint until the memory usage goes down just for the sake of avoiding
>> the 2x memory usage?
> Yes, or move to the next checkpoint sooner than the N milliseconds when
> we see the buffer is getting full.

OK, I see there is definitely some common ground there: So to be
more specific, what we really need is two things: (I've learned that
the reviewers are very cautious about adding to much policy into
QEMU itself, but let's iron this out anyway:)

1. First, we need to throttle down the guest (QEMU can already do this
     using the recently introduced "auto-converge" feature). This means
     that the guest is still making forward progress, albeit slow progress.

2. Then we would need some kind of policy, or better yet, a trigger that
     does something to the effect of "we're about to use a whole lot of
     checkpoint memory soon - can we afford this much memory usage".
     Such a trigger would be conditional on the current policy of the
     administrator or management software: We would either have a QMP
     command that with a boolean flag that says "Yes" or "No", it's
     tolerable or not to use that much memory in the next checkpoint.

     If the answer is "Yes", then nothing changes.
     If the answer is "No", then we should either:
        a) throttle down the guest
        b) Adjust the checkpoint frequency
        c) Or pause it altogether while we migrate some other VMs off the
            host such that we can complete the next checkpoint in its 
entirety.

It's not clear to me how much of this (or any) of this control loop should
be in QEMU or in the management software, but I would definitely agree
that a minimum of at least the ability to detect the situation and remedy
the situation should be in QEMU. I'm not entirely convince that the
ability to *decide* to remedy the situation should be in QEMU, though.


>
>> If you block the guest from being checkpointed,
>> then what happens if there is a failure during that extended period?
>> We will have saved memory at the expense of availability.
> If the active machine fails during this time then the secondary carries
> on from it's last good snapshot in the knowledge that the active
> never finished the new snapshot and so never uncorked it's previous packets.
>
> If the secondary machine fails during this time then tha active drops
> it's nascent snapshot and carries on.

Yes, that makes sense. Where would that policy go, though,
continuing the above concern?

> However, what you have made me realise is that I don't have an answer
> for the memory usage on the secondary; while the primary can pause
> it's guest until the secondary ack's the checkpoint, the secondary has
> to rely on the primary not to send it huge checkpoints.

Good question: There's a lot of work ideas out there in the academic
community to compress the secondary, or push the secondary to
a flash-based device, or de-duplicate the secondary. I'm sure any
of them would put a dent in the problem, but I'm not seeing a smoking
gun solution that would absolutely save all that memory completely.

(Personally, I don't believe in swap. I wouldn't even consider swap
or any kind of traditional disk-based remedy to be a viable solution).

>> The customer that is expecting 100% fault tolerance and the provider
>> who is supporting it need to have an understanding that fault tolerance
>> is not free and that constraining memory usage will adversely affect
>> the VM's ability to be protected.
>>
>> Do I understand your expectations correctly? Is fault tolerance
>> something you're willing to sacrifice?
> As above, no I'm willing to sacrifice performance but not fault tolerance.
> (It is entirely possible that others would want the other trade off, i.e.
> some minimum performance is worse than useless, so if we can't maintain
> that performance then dropping FT leaves us in a more-working position).
>

Agreed - I think a "proactive" failover in this case would solve the 
problem.
If we observed that availability/fault tolerance was going to be at
risk soon (which is relatively easy to detect) - we could just *force*
a failover to the secondary host and restart the protection from
scratch.


>>
>> Well, that's simple: If there is a failure of the source, the destination
>> will simply revert to the previous checkpoint using the same mode
>> of operation. The lost ACKs that you're curious about only
>> apply to the checkpoint that is in progress. Just because a
>> checkpoint is in progress does not mean that the previous checkpoint
>> is thrown away - it is already loaded into the destination's memory
>> and ready to be activated.
> I still don't see why, if the link between them fails, the destination
> doesn't fall back it it's previous checkpoint, AND the source carries
> on running - I don't see how they can differentiate which of them has failed.

I think you're forgetting that the source I/O is buffered - it doesn't
matter that the source VM is still running. As long as it's output is
buffered - it cannot have any non-fault-tolerant affect on the outside
world.

In the future, if a technician access the machine or the network
is restored, the management software can terminate the stale
source virtual machine.

>> We have a script architecture (not on github) which runs MC in a tight
>> loop hundreds of times and kills the source QEMU and timestamps how
>> quickly the
>> destination QEMU loses the TCP socket connection receives an error code
>> from the kernel - every single time, the destination resumes nearly
>> instantaneously.
>> I've not empirically seen a case where the socket just hangs or doesn't
>> change state.
>>
>> I'm not very familiar with the internal linux TCP/IP stack
>> implementation itself,
>> but I have not had a problem with the dependability of the linux socket
>> not being able to shutdown the socket as soon as possible.
> OK, that only covers a very small range of normal failures.
> When you kill the destination QEMU the host OS knows that QEMU is dead
> and sends a packet back closing the socket, hence the source knows
> the destination is dead very quickly.
> If:
>     a) The destination machine was to lose power or hang
>     b) Or a network link fail  (other than the one attached to the source
>        possibly)
>
> the source would have to do a full TCP timeout.
>
> To test a,b I'd use an iptables rule somewhere to cause the packets to
> be dropped (not rejected).  Stopping the qemu in gdb might be good enough.

Very good idea - I'll add that to the "todo" list of things to do
in my test infrastructure. It may indeed turn out be necessary
to add a formal keepalive between the source and destination.

- Michael

* Michael R. Hines (mrhines@linux.vnet.ibm.com) wrote:
> On 02/21/2014 12:32 AM, Dr. David Alan Gilbert wrote:
> >
> >I'm happy to use more memory to get FT, all I'm trying to do is see
> >if it's possible to put a lower bound than 2x on it while still maintaining
> >full FT, at the expense of performance in the case where it uses
> >a lot of memory.
> >
> >>The bottom line is: if you put a *hard* constraint on memory usage,
> >>what will happen to the guest when that garbage collection you mentioned
> >>shows up later and runs for several minutes? How about an hour?
> >>Are we just going to block the guest from being allowed to start a
> >>checkpoint until the memory usage goes down just for the sake of avoiding
> >>the 2x memory usage?
> >Yes, or move to the next checkpoint sooner than the N milliseconds when
> >we see the buffer is getting full.
> 
> OK, I see there is definitely some common ground there: So to be
> more specific, what we really need is two things: (I've learned that
> the reviewers are very cautious about adding to much policy into
> QEMU itself, but let's iron this out anyway:)
> 
> 1. First, we need to throttle down the guest (QEMU can already do this
>     using the recently introduced "auto-converge" feature). This means
>     that the guest is still making forward progress, albeit slow progress.
> 
> 2. Then we would need some kind of policy, or better yet, a trigger that
>     does something to the effect of "we're about to use a whole lot of
>     checkpoint memory soon - can we afford this much memory usage".
>     Such a trigger would be conditional on the current policy of the
>     administrator or management software: We would either have a QMP
>     command that with a boolean flag that says "Yes" or "No", it's
>     tolerable or not to use that much memory in the next checkpoint.
> 
>     If the answer is "Yes", then nothing changes.
>     If the answer is "No", then we should either:
>        a) throttle down the guest
>        b) Adjust the checkpoint frequency
>        c) Or pause it altogether while we migrate some other VMs off the
>            host such that we can complete the next checkpoint in its
> entirety.

Yes I think so, although what I was thinking was mainly (b) possibly
to the point of not starting the next checkpoint.

> It's not clear to me how much of this (or any) of this control loop should
> be in QEMU or in the management software, but I would definitely agree
> that a minimum of at least the ability to detect the situation and remedy
> the situation should be in QEMU. I'm not entirely convince that the
> ability to *decide* to remedy the situation should be in QEMU, though.

The management software access is low frequency, high latency; it should
be setting general parameters (max memory allowed, desired checkpoint
frequency etc) but I don't see that we can use it to do anything on
a sooner than a few second basis; so yes it can monitor things and
tweek the knobs if it sees the host as a whole is getting tight on RAM
etc - but we can't rely on it to throw in the breaks if this guest
suddenly decides to take bucket loads of RAM; something has to react
quickly in relation to previously set limits.

> >>If you block the guest from being checkpointed,
> >>then what happens if there is a failure during that extended period?
> >>We will have saved memory at the expense of availability.
> >If the active machine fails during this time then the secondary carries
> >on from it's last good snapshot in the knowledge that the active
> >never finished the new snapshot and so never uncorked it's previous packets.
> >
> >If the secondary machine fails during this time then tha active drops
> >it's nascent snapshot and carries on.
> 
> Yes, that makes sense. Where would that policy go, though,
> continuing the above concern?

I think there has to be some input from the management layer for failover,
because (as per my split-brain concerns) something has to make the decision
about which of the source/destination is to take over, and I don't
believe individual instances have that information.

> >However, what you have made me realise is that I don't have an answer
> >for the memory usage on the secondary; while the primary can pause
> >it's guest until the secondary ack's the checkpoint, the secondary has
> >to rely on the primary not to send it huge checkpoints.
> 
> Good question: There's a lot of work ideas out there in the academic
> community to compress the secondary, or push the secondary to
> a flash-based device, or de-duplicate the secondary. I'm sure any
> of them would put a dent in the problem, but I'm not seeing a smoking
> gun solution that would absolutely save all that memory completely.

Ah, I was thinking that flash would be a good solution for secondary;
it would be a nice demo.

> (Personally, I don't believe in swap. I wouldn't even consider swap
> or any kind of traditional disk-based remedy to be a viable solution).

Well it certainly exists - I've seen it!
Swap works well in limited circumstances; but as soon as you've got
multiple VMs fighting over something with 10s of ms latency you're doomed.

> >>The customer that is expecting 100% fault tolerance and the provider
> >>who is supporting it need to have an understanding that fault tolerance
> >>is not free and that constraining memory usage will adversely affect
> >>the VM's ability to be protected.
> >>
> >>Do I understand your expectations correctly? Is fault tolerance
> >>something you're willing to sacrifice?
> >As above, no I'm willing to sacrifice performance but not fault tolerance.
> >(It is entirely possible that others would want the other trade off, i.e.
> >some minimum performance is worse than useless, so if we can't maintain
> >that performance then dropping FT leaves us in a more-working position).
> >
> 
> Agreed - I think a "proactive" failover in this case would solve the
> problem.
> If we observed that availability/fault tolerance was going to be at
> risk soon (which is relatively easy to detect) - we could just *force*
> a failover to the secondary host and restart the protection from
> scratch.
> 
> 
> >>
> >>Well, that's simple: If there is a failure of the source, the destination
> >>will simply revert to the previous checkpoint using the same mode
> >>of operation. The lost ACKs that you're curious about only
> >>apply to the checkpoint that is in progress. Just because a
> >>checkpoint is in progress does not mean that the previous checkpoint
> >>is thrown away - it is already loaded into the destination's memory
> >>and ready to be activated.
> >I still don't see why, if the link between them fails, the destination
> >doesn't fall back it it's previous checkpoint, AND the source carries
> >on running - I don't see how they can differentiate which of them has failed.
> 
> I think you're forgetting that the source I/O is buffered - it doesn't
> matter that the source VM is still running. As long as it's output is
> buffered - it cannot have any non-fault-tolerant affect on the outside
> world.
> 
> In the future, if a technician access the machine or the network
> is restored, the management software can terminate the stale
> source virtual machine.

I think going with my comment above; I'm working on the basis it's just
as likely for the destination to fail as it is for the source to fail,
and a destination failure shouldn't kill the source; and in the case
of a destination failure the source is going to have to let it's buffered
I/Os start going again.

> >>We have a script architecture (not on github) which runs MC in a tight
> >>loop hundreds of times and kills the source QEMU and timestamps how
> >>quickly the
> >>destination QEMU loses the TCP socket connection receives an error code
> >>from the kernel - every single time, the destination resumes nearly
> >>instantaneously.
> >>I've not empirically seen a case where the socket just hangs or doesn't
> >>change state.
> >>
> >>I'm not very familiar with the internal linux TCP/IP stack
> >>implementation itself,
> >>but I have not had a problem with the dependability of the linux socket
> >>not being able to shutdown the socket as soon as possible.
> >OK, that only covers a very small range of normal failures.
> >When you kill the destination QEMU the host OS knows that QEMU is dead
> >and sends a packet back closing the socket, hence the source knows
> >the destination is dead very quickly.
> >If:
> >    a) The destination machine was to lose power or hang
> >    b) Or a network link fail  (other than the one attached to the source
> >       possibly)
> >
> >the source would have to do a full TCP timeout.
> >
> >To test a,b I'd use an iptables rule somewhere to cause the packets to
> >be dropped (not rejected).  Stopping the qemu in gdb might be good enough.
> 
> Very good idea - I'll add that to the "todo" list of things to do
> in my test infrastructure. It may indeed turn out be necessary
> to add a formal keepalive between the source and destination.
> 
> - Michael

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

On 02/21/2014 05:44 PM, Dr. David Alan Gilbert wrote:
>> It's not clear to me how much of this (or any) of this control loop should
>> be in QEMU or in the management software, but I would definitely agree
>> that a minimum of at least the ability to detect the situation and remedy
>> the situation should be in QEMU. I'm not entirely convince that the
>> ability to *decide* to remedy the situation should be in QEMU, though.
> The management software access is low frequency, high latency; it should
> be setting general parameters (max memory allowed, desired checkpoint
> frequency etc) but I don't see that we can use it to do anything on
> a sooner than a few second basis; so yes it can monitor things and
> tweek the knobs if it sees the host as a whole is getting tight on RAM
> etc - but we can't rely on it to throw in the breaks if this guest
> suddenly decides to take bucket loads of RAM; something has to react
> quickly in relation to previously set limits.

I agree - the boolean flag I mentioned previously would do just
that: setting the flag (or state, perhaps instead of boolean),
would indicate to QEMU to make a particular type of sacrifice:

A flag of "0" might mean "Throttle the guest in an emergency"
A flag of "1" might mean "Throttling is not acceptable, just let the 
guest use the extra memory"
A flag of "2" might mean "Neither one is acceptable, fail now and inform 
the management software to restart somewhere else".

Or something to that effect........

>>>> If you block the guest from being checkpointed,
>>>> then what happens if there is a failure during that extended period?
>>>> We will have saved memory at the expense of availability.
>>> If the active machine fails during this time then the secondary carries
>>> on from it's last good snapshot in the knowledge that the active
>>> never finished the new snapshot and so never uncorked it's previous packets.
>>>
>>> If the secondary machine fails during this time then tha active drops
>>> it's nascent snapshot and carries on.
>> Yes, that makes sense. Where would that policy go, though,
>> continuing the above concern?
> I think there has to be some input from the management layer for failover,
> because (as per my split-brain concerns) something has to make the decision
> about which of the source/destination is to take over, and I don't
> believe individual instances have that information.

Agreed - so the "ability" (as hinted on above) should be in QEMU,
but the decision to recover from the situation probably should not
be, where "recover" is defined as the VM is back in a fully running,
fully fault-tolerant protected state (potentially where the source VM
is on a different machine than it was before).

>
>>>> Well, that's simple: If there is a failure of the source, the destination
>>>> will simply revert to the previous checkpoint using the same mode
>>>> of operation. The lost ACKs that you're curious about only
>>>> apply to the checkpoint that is in progress. Just because a
>>>> checkpoint is in progress does not mean that the previous checkpoint
>>>> is thrown away - it is already loaded into the destination's memory
>>>> and ready to be activated.
>>> I still don't see why, if the link between them fails, the destination
>>> doesn't fall back it it's previous checkpoint, AND the source carries
>>> on running - I don't see how they can differentiate which of them has failed.
>> I think you're forgetting that the source I/O is buffered - it doesn't
>> matter that the source VM is still running. As long as it's output is
>> buffered - it cannot have any non-fault-tolerant affect on the outside
>> world.
>>
>> In the future, if a technician access the machine or the network
>> is restored, the management software can terminate the stale
>> source virtual machine.
> I think going with my comment above; I'm working on the basis it's just
> as likely for the destination to fail as it is for the source to fail,
> and a destination failure shouldn't kill the source; and in the case
> of a destination failure the source is going to have to let it's buffered
> I/Os start going again.

Yes, that's correct, but only after management software knows about
the failure. If we're on a tightly-coupled fast lan, there's no reason
to believe that libvirt, for example, would be so slow that we cannot
wait a few extra (10s of?) milliseconds after destination failure to
choose a new destination and restart the previous checkpoint.

But if management *is* too slow, which is not unlikely, then I think
we should just tell the source to Migrate entirely and get out of that
environment.

Either way - this isn't something QEMU itself necessarily needs to
worry about - it just needs to know not to explode if the destination
fails and wait for instructions on what to do next.......

Alternatively, if the administrator "prefers" restarting the fault-tolerance
instead of Migration, we could have a QMP command that specifies
a "backup" destination (or even a "duplicate" destination) that QEMU
would automatically know about in the case of destination failure.

But, I wouldn't implement something like that until at least a first version
was accepted by the community.

- Michael