| Message ID | 1452676712-24239-6-git-send-email-xiecl.fnst@cn.fujitsu.com |
|---|---|
| State | New |
| Headers | show |
On 01/13/2016 02:18 AM, Changlong Xie wrote: > From: Wen Congyang <wency@cn.fujitsu.com> > > Signed-off-by: Wen Congyang <wency@cn.fujitsu.com> > Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com> > Signed-off-by: Gonglei <arei.gonglei@huawei.com> > Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com> > --- > docs/block-replication.txt | 229 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 229 insertions(+) > create mode 100644 docs/block-replication.txt > > diff --git a/docs/block-replication.txt b/docs/block-replication.txt > new file mode 100644 > index 0000000..d1a231e > --- /dev/null > +++ b/docs/block-replication.txt > @@ -0,0 +1,229 @@ > +Block replication > +---------------------------------------- > +Copyright Fujitsu, Corp. 2015 > +Copyright (c) 2015 Intel Corporation > +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD. Do you want to claim 2016 for any of this? > + > +This work is licensed under the terms of the GNU GPL, version 2 or later. > +See the COPYING file in the top-level directory. > + > +Block replication is used for continuous checkpoints. It is designed > +for COLO (COarse-grain LOck-stepping) where the Secondary VM is running. > +It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario, > +where the Secondary VM is not running. > + > +This document gives an overview of block replication's design. > + > +== Background == > +High availability solutions such as micro checkpoint and COLO will do > +consecutive checkpoints. The VM state of Primary VM and Secondary VM is s/of Primary/of the Primary/ > +identical right after a VM checkpoint, but becomes different as the VM > +executes till the next checkpoint. To support disk contents checkpoint, > +the modified disk contents in the Secondary VM must be buffered, and are > +only dropped at next checkpoint time. To reduce the network transportation > +effort at the time of checkpoint, the disk modification operations of s/at the time of/during a vmstate/ s/operations of/operations of the/ > +Primary disk are asynchronously forwarded to the Secondary node. > + > +== Workflow == > +== Architecture == > + > +6) The drive-backup job(sync=none) is run to allow hidden-disk to buffer Space before ( in English description. > +any state that would otherwise be lost by the speculative write-through > +of the NBD server into the secondary disk. So before block replication, > +the primary disk and secondary disk should contain the same data. > + > +== Failure Handling == > +== Usage == > +Primary: > + -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ > + children.0.file.filename=1.raw,\ > + children.0.driver=raw > + > + Run qmp command in primary qemu: > + { 'execute': 'human-monitor-command', > + 'arguments': { > + 'command-line': 'drive_add buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none' Eww. We shouldn't ever have to pack a command line as a single QMP string that needs reparsing. Instead, you should pass the information as a nested QMP dictionary, something like: 'arguments': { 'remote-command': { 'command': 'drive_add', 'name': 'buddy', 'driver': 'replication', 'mode': 'primary', 'file': { 'driver': 'nbd', 'host': 'xxxx', ... } } } > + } > + } > + { 'execute': 'x-blockdev-change', > + 'arguments': { > + 'parent': 'colo1', > + 'node': 'nbd_client1' > + } > + } > + Note: > + 1. There should be only one NBD Client for each primary disk. > + 2. host is the secondary physical machine's hostname or IP > + 3. Each disk must have its own export name. > + 4. It is all a single argument to -drive and you should ignore the > + leading whitespace. > + 5. The qmp command line must be run after running qmp command line in > + secondary qemu. > + > +Secondary: > + -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \ > + -drive if=xxx,driver=replication,mode=secondary,\ > + file.file.filename=active_disk.qcow2,\ > + file.driver=qcow2,\ > + file.backing.file.filename=hidden_disk.qcow2,\ > + file.backing.driver=qcow2,\ > + file.backing.backing=colo1 > + > + Then run qmp command in secondary qemu: > + { 'execute': 'nbd-server-start', > + 'arguments': { > + 'addr': { > + 'type': 'inet', > + 'data': { > + 'host': 'xxx', > + 'port': 'xxx' > + } > + } > + } > + } > + { 'execute': 'nbd-server-add', > + 'arguments': { > + 'device': 'colo1', > + 'writable': true > + } > + } > + > + Note: > + 1. The export name in secondary QEMU command line is the secondary > + disk's id. > + 2. The export name for the same disk must be the same > + 3. The qmp command nbd-server-start and nbd-server-add must be run > + before running the qmp command migrate on primary QEMU > + 4. Active disk, hidden disk and nbd target's length should be the > + same. > + 5. It is better to put active disk and hidden disk in ramdisk. > + 6. It is all a single argument to -drive, and you should ignore > + the leading whitespace. > + > +After Failover: > +Primary: > + The secondary host is down, so we should run the following qmp command > + to remove the nbd child from the quorum: > + { 'execute': 'x-blockdev-change', > + 'arguments': { > + 'parent': 'colo1', > + 'child': 'children.1' > + } > + } > + Note: there is no qmp command to remove the blockdev now > + > +Secondary: > + The primary host is down, so we should do the following thing: > + { 'execute': 'nbd-server-stop' } > + > +TODO: > +1. Continuous block replication > +2. Shared disk >
On 02/02/2016 11:20 PM, Eric Blake wrote: > On 01/13/2016 02:18 AM, Changlong Xie wrote: >> From: Wen Congyang <wency@cn.fujitsu.com> >> >> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com> >> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com> >> Signed-off-by: Gonglei <arei.gonglei@huawei.com> >> Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com> >> --- >> docs/block-replication.txt | 229 +++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 229 insertions(+) >> create mode 100644 docs/block-replication.txt >> >> diff --git a/docs/block-replication.txt b/docs/block-replication.txt >> new file mode 100644 >> index 0000000..d1a231e >> --- /dev/null >> +++ b/docs/block-replication.txt >> @@ -0,0 +1,229 @@ >> +Block replication >> +---------------------------------------- >> +Copyright Fujitsu, Corp. 2015 >> +Copyright (c) 2015 Intel Corporation >> +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD. > > Do you want to claim 2016 for any of this? Will update in next version. > >> + >> +This work is licensed under the terms of the GNU GPL, version 2 or later. >> +See the COPYING file in the top-level directory. >> + >> +Block replication is used for continuous checkpoints. It is designed >> +for COLO (COarse-grain LOck-stepping) where the Secondary VM is running. >> +It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario, >> +where the Secondary VM is not running. >> + >> +This document gives an overview of block replication's design. >> + >> +== Background == >> +High availability solutions such as micro checkpoint and COLO will do >> +consecutive checkpoints. The VM state of Primary VM and Secondary VM is > > s/of Primary/of the Primary/ OK > >> +identical right after a VM checkpoint, but becomes different as the VM >> +executes till the next checkpoint. To support disk contents checkpoint, >> +the modified disk contents in the Secondary VM must be buffered, and are >> +only dropped at next checkpoint time. To reduce the network transportation >> +effort at the time of checkpoint, the disk modification operations of > > s/at the time of/during a vmstate/ > s/operations of/operations of the/ OK > >> +Primary disk are asynchronously forwarded to the Secondary node. >> + >> +== Workflow == > >> +== Architecture == > >> + >> +6) The drive-backup job(sync=none) is run to allow hidden-disk to buffer > > Space before ( in English description. OK > >> +any state that would otherwise be lost by the speculative write-through >> +of the NBD server into the secondary disk. So before block replication, >> +the primary disk and secondary disk should contain the same data. >> + >> +== Failure Handling == > >> +== Usage == >> +Primary: >> + -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ >> + children.0.file.filename=1.raw,\ >> + children.0.driver=raw >> + >> + Run qmp command in primary qemu: >> + { 'execute': 'human-monitor-command', >> + 'arguments': { >> + 'command-line': 'drive_add buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none' > > Eww. We shouldn't ever have to pack a command line as . single QMP > string that needs reparsing. Instead, you should pass the information > as a nested QMP dictionary, something like: > > 'arguments': { > 'remote-command': { 'command': 'drive_add', 'name': 'buddy', > 'driver': 'replication', 'mode': 'primary', > 'file': { 'driver': 'nbd', 'host': 'xxxx', Hi Eric What is 'remote-command' here? I just tried below commands, but got some errors. {'execute': 'human-monitor-command', 'arguments': { 'command-line': { 'command': 'drive_add', 'name': 'buddy', 'driver': 'replication', 'mode': 'primary', 'if': 'none', 'node-name': 'primary_nbd_node', 'file': { 'driver': 'nbd', 'host': '192.168.3.2', 'port': '8889', 'export': 'colo-disk', } } } } {"error": {"class": "GenericError", "desc": "Invalid JSON syntax"}} 'blockdev-add' doesn't support 'nbd'. So we use 'drive_add' here, and it's a hmp command. If i'm right, there seems just one way to execute hmp commands in QMP: ================================================================ EQMP { .name = "human-monitor-command", .args_type = "command-line:s,cpu-index:i?", .mhandler.cmd_new = qmp_marshal_human_monitor_command, }, SQMP human-monitor-command --------------------- Execute a Human Monitor command. Arguments: - command-line: the command name and its arguments, just like the Human Monitor's shell (json-string) - cpu-index: select the CPU number to be used by commands which access CPU data, like 'info registers'. The Monitor selects CPU 0 if this argument is not provided (json-int, optional) Example: -> { "execute": "human-monitor-command", "arguments": { "command-line": "info kvm" } } <- { "return": "kvm support: enabled\r\n" } ================================================================== Thanks -Xie > ... } } } > >> + } >> + } >> + { 'execute': 'x-blockdev-change', >> + 'arguments': { >> + 'parent': 'colo1', >> + 'node': 'nbd_client1' >> + } >> + } >> + Note: >> + 1. There should be only one NBD Client for each primary disk. >> + 2. host is the secondary physical machine's hostname or IP >> + 3. Each disk must have its own export name. >> + 4. It is all a single argument to -drive and you should ignore the >> + leading whitespace. >> + 5. The qmp command line must be run after running qmp command line in >> + secondary qemu. >> + >> +Secondary: >> + -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \ >> + -drive if=xxx,driver=replication,mode=secondary,\ >> + file.file.filename=active_disk.qcow2,\ >> + file.driver=qcow2,\ >> + file.backing.file.filename=hidden_disk.qcow2,\ >> + file.backing.driver=qcow2,\ >> + file.backing.backing=colo1 >> + >> + Then run qmp command in secondary qemu: >> + { 'execute': 'nbd-server-start', >> + 'arguments': { >> + 'addr': { >> + 'type': 'inet', >> + 'data': { >> + 'host': 'xxx', >> + 'port': 'xxx' >> + } >> + } >> + } >> + } >> + { 'execute': 'nbd-server-add', >> + 'arguments': { >> + 'device': 'colo1', >> + 'writable': true >> + } >> + } >> + >> + Note: >> + 1. The export name in secondary QEMU command line is the secondary >> + disk's id. >> + 2. The export name for the same disk must be the same >> + 3. The qmp command nbd-server-start and nbd-server-add must be run >> + before running the qmp command migrate on primary QEMU >> + 4. Active disk, hidden disk and nbd target's length should be the >> + same. >> + 5. It is better to put active disk and hidden disk in ramdisk. >> + 6. It is all a single argument to -drive, and you should ignore >> + the leading whitespace. >> + >> +After Failover: >> +Primary: >> + The secondary host is down, so we should run the following qmp command >> + to remove the nbd child from the quorum: >> + { 'execute': 'x-blockdev-change', >> + 'arguments': { >> + 'parent': 'colo1', >> + 'child': 'children.1' >> + } >> + } >> + Note: there is no qmp command to remove the blockdev now >> + >> +Secondary: >> + The primary host is down, so we should do the following thing: >> + { 'execute': 'nbd-server-stop' } >> + >> +TODO: >> +1. Continuous block replication >> +2. Shared disk >> >
On 02/02/2016 08:18 PM, Changlong Xie wrote: > On 02/02/2016 11:20 PM, Eric Blake wrote: >> On 01/13/2016 02:18 AM, Changlong Xie wrote: >>> From: Wen Congyang <wency@cn.fujitsu.com> >>> >>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com> >>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com> >>> Signed-off-by: Gonglei <arei.gonglei@huawei.com> >>> Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com> >>> --- >>> docs/block-replication.txt | 229 >>> +== Usage == >>> +Primary: >>> + -drive >>> if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ >>> + children.0.file.filename=1.raw,\ >>> + children.0.driver=raw >>> + >>> + Run qmp command in primary qemu: >>> + { 'execute': 'human-monitor-command', >>> + 'arguments': { >>> + 'command-line': 'drive_add buddy >>> driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none' >>> >> >> Eww. We shouldn't ever have to pack a command line as . single QMP >> string that needs reparsing. Instead, you should pass the information >> as a nested QMP dictionary, something like: >> >> 'arguments': { >> 'remote-command': { 'command': 'drive_add', 'name': 'buddy', >> 'driver': 'replication', 'mode': 'primary', >> 'file': { 'driver': 'nbd', 'host': 'xxxx', > > Hi Eric > > What is 'remote-command' here? I just tried below commands, but got some > errors. Oh, I totally missed that this was using the existing 'human-monitor-command' to send an HMP command, instead of trying to send a formal QMP command. I thought you were documenting a new QMP command. Still, it would be nice to do this command using strict QMP (that would be via 'blockdev-add') rather than via HMP (an all-in-one text command crammed into the single 'command-line' argument). > > 'blockdev-add' doesn't support 'nbd'. So we use 'drive_add' here, and > it's a hmp command. If i'm right, there seems just one way to execute > hmp commands in QMP: For 2.6, we _really_ ought to get blockdev-add working for everything. We're running short on time, though :(
On 02/03/2016 11:35 AM, Eric Blake wrote: > On 02/02/2016 08:18 PM, Changlong Xie wrote: >> On 02/02/2016 11:20 PM, Eric Blake wrote: >>> On 01/13/2016 02:18 AM, Changlong Xie wrote: >>>> From: Wen Congyang <wency@cn.fujitsu.com> >>>> >>>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com> >>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com> >>>> Signed-off-by: Gonglei <arei.gonglei@huawei.com> >>>> Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com> >>>> --- >>>> docs/block-replication.txt | 229 > >>>> +== Usage == >>>> +Primary: >>>> + -drive >>>> if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ >>>> + children.0.file.filename=1.raw,\ >>>> + children.0.driver=raw >>>> + >>>> + Run qmp command in primary qemu: >>>> + { 'execute': 'human-monitor-command', >>>> + 'arguments': { >>>> + 'command-line': 'drive_add buddy >>>> driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none' >>>> >>> >>> Eww. We shouldn't ever have to pack a command line as . single QMP >>> string that needs reparsing. Instead, you should pass the information >>> as a nested QMP dictionary, something like: >>> >>> 'arguments': { >>> 'remote-command': { 'command': 'drive_add', 'name': 'buddy', >>> 'driver': 'replication', 'mode': 'primary', >>> 'file': { 'driver': 'nbd', 'host': 'xxxx', >> >> Hi Eric >> >> What is 'remote-command' here? I just tried below commands, but got some >> errors. > > Oh, I totally missed that this was using the existing > 'human-monitor-command' to send an HMP command, instead of trying to > send a formal QMP command. I thought you were documenting a new QMP > command. > > Still, it would be nice to do this command using strict QMP (that would > be via 'blockdev-add') rather than via HMP (an all-in-one text command > crammed into the single 'command-line' argument). > >> >> 'blockdev-add' doesn't support 'nbd'. So we use 'drive_add' here, and >> it's a hmp command. If i'm right, there seems just one way to execute >> hmp commands in QMP: > > For 2.6, we _really_ ought to get blockdev-add working for everything. > We're running short on time, though :( > If the qmp command 'blockdev-add' supports nbd in qemu-2.6, we will update this document when it is suppoted. Thanks Wen Congyang
diff --git a/docs/block-replication.txt b/docs/block-replication.txt new file mode 100644 index 0000000..d1a231e --- /dev/null +++ b/docs/block-replication.txt @@ -0,0 +1,229 @@ +Block replication +---------------------------------------- +Copyright Fujitsu, Corp. 2015 +Copyright (c) 2015 Intel Corporation +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD. + +This work is licensed under the terms of the GNU GPL, version 2 or later. +See the COPYING file in the top-level directory. + +Block replication is used for continuous checkpoints. It is designed +for COLO (COarse-grain LOck-stepping) where the Secondary VM is running. +It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario, +where the Secondary VM is not running. + +This document gives an overview of block replication's design. + +== Background == +High availability solutions such as micro checkpoint and COLO will do +consecutive checkpoints. The VM state of Primary VM and Secondary VM is +identical right after a VM checkpoint, but becomes different as the VM +executes till the next checkpoint. To support disk contents checkpoint, +the modified disk contents in the Secondary VM must be buffered, and are +only dropped at next checkpoint time. To reduce the network transportation +effort at the time of checkpoint, the disk modification operations of +Primary disk are asynchronously forwarded to the Secondary node. + +== Workflow == +The following is the image of block replication workflow: + + +----------------------+ +------------------------+ + |Primary Write Requests| |Secondary Write Requests| + +----------------------+ +------------------------+ + | | + | (4) + | V + | /-------------\ + | Copy and Forward | | + |---------(1)----------+ | Disk Buffer | + | | | | + | (3) \-------------/ + | speculative ^ + | write through (2) + | | | + V V | + +--------------+ +----------------+ + | Primary Disk | | Secondary Disk | + +--------------+ +----------------+ + + 1) Primary write requests will be copied and forwarded to Secondary + QEMU. + 2) Before Primary write requests are written to Secondary disk, the + original sector content will be read from Secondary disk and + buffered in the Disk buffer, but it will not overwrite the existing + sector content (it could be from either "Secondary Write Requests" or + previous COW of "Primary Write Requests") in the Disk buffer. + 3) Primary write requests will be written to Secondary disk. + 4) Secondary write requests will be buffered in the Disk buffer and it + will overwrite the existing sector content in the buffer. + +== Architecture == +We are going to implement block replication from many basic +blocks that are already in QEMU. + + virtio-blk || + ^ || .---------- + | || | Secondary + 1 Quorum || '---------- + / \ || + / \ || + Primary 2 filter + disk ^ virtio-blk + | ^ + 3 NBD -------> 3 NBD | + client || server 2 filter + || ^ ^ +--------. || | | +Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4 +--------' || | backing ^ backing + || | | + || | | + || '-------------------------' + || drive-backup sync=none 6 + +1) The disk on the primary is represented by a block device with two +children, providing replication between a primary disk and the host that +runs the secondary VM. The read pattern for quorum can be extended to +make the primary always read from the local disk instead of going through +NBD. + +2) The new block filter (the name is replication) will control the block +replication. + +3) The secondary disk receives writes from the primary VM through QEMU's +embedded NBD server (speculative write-through). + +4) The disk on the secondary is represented by a custom block device +(called active-disk). It should start as an empty disk, and the format +should support bdrv_make_empty() and backing file. + +5) The hidden-disk is created automatically. It buffers the original content +that is modified by the primary VM. It should also start as an empty disk, +and the driver supports bdrv_make_empty() and backing file. + +6) The drive-backup job(sync=none) is run to allow hidden-disk to buffer +any state that would otherwise be lost by the speculative write-through +of the NBD server into the secondary disk. So before block replication, +the primary disk and secondary disk should contain the same data. + +== Failure Handling == +There are 7 internal errors when block replication is running: +1. I/O error on primary disk +2. Forwarding primary write requests failed +3. Backup failed +4. I/O error on secondary disk +5. I/O error on active disk +6. Making active disk or hidden disk empty failed +7. Doing failover failed +In case 1 and 5, we just report the error to the disk layer. In case 2, 3, +4 and 6, we just report block replication's error to FT/HA manager (which +decides when to do a new checkpoint, when to do failover). +In case 7, if active commit failed, we use replication failover failed state +in Secondary's write operation (what decides which target to write). + +== New block driver interface == +We add three block driver interfaces to control block replication: +a. block_replication_start_all() + Start block replication, called in migration/checkpoint thread. + We must call block_replication_start_all() in secondary QEMU before + calling block_replication_start_all() in primary QEMU. The caller + must hold the I/O mutex lock if it is in migration/checkpoint + thread. +b. block_replication_do_checkpoint_all() + This interface is called after all VM state is transferred to + Secondary QEMU. The Disk buffer will be dropped in this interface. + The caller must hold the I/O mutex lock if it is in migration/checkpoint + thread. +c. block_replication_stop_all() + It is called on failover. We will flush the Disk buffer into + Secondary Disk and stop block replication. The vm should be stopped + before calling it if you use this API to shutdown the guest, or other + things except failover. The caller must hold the I/O mutex lock if it is + in migration/checkpoint thread. + +== Usage == +Primary: + -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ + children.0.file.filename=1.raw,\ + children.0.driver=raw + + Run qmp command in primary qemu: + { 'execute': 'human-monitor-command', + 'arguments': { + 'command-line': 'drive_add buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none' + } + } + { 'execute': 'x-blockdev-change', + 'arguments': { + 'parent': 'colo1', + 'node': 'nbd_client1' + } + } + Note: + 1. There should be only one NBD Client for each primary disk. + 2. host is the secondary physical machine's hostname or IP + 3. Each disk must have its own export name. + 4. It is all a single argument to -drive and you should ignore the + leading whitespace. + 5. The qmp command line must be run after running qmp command line in + secondary qemu. + +Secondary: + -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \ + -drive if=xxx,driver=replication,mode=secondary,\ + file.file.filename=active_disk.qcow2,\ + file.driver=qcow2,\ + file.backing.file.filename=hidden_disk.qcow2,\ + file.backing.driver=qcow2,\ + file.backing.backing=colo1 + + Then run qmp command in secondary qemu: + { 'execute': 'nbd-server-start', + 'arguments': { + 'addr': { + 'type': 'inet', + 'data': { + 'host': 'xxx', + 'port': 'xxx' + } + } + } + } + { 'execute': 'nbd-server-add', + 'arguments': { + 'device': 'colo1', + 'writable': true + } + } + + Note: + 1. The export name in secondary QEMU command line is the secondary + disk's id. + 2. The export name for the same disk must be the same + 3. The qmp command nbd-server-start and nbd-server-add must be run + before running the qmp command migrate on primary QEMU + 4. Active disk, hidden disk and nbd target's length should be the + same. + 5. It is better to put active disk and hidden disk in ramdisk. + 6. It is all a single argument to -drive, and you should ignore + the leading whitespace. + +After Failover: +Primary: + The secondary host is down, so we should run the following qmp command + to remove the nbd child from the quorum: + { 'execute': 'x-blockdev-change', + 'arguments': { + 'parent': 'colo1', + 'child': 'children.1' + } + } + Note: there is no qmp command to remove the blockdev now + +Secondary: + The primary host is down, so we should do the following thing: + { 'execute': 'nbd-server-stop' } + +TODO: +1. Continuous block replication +2. Shared disk