diff mbox

[v13,00/10] Block replication for continuous checkpoints

Message ID 56B3253B.3030300@cn.fujitsu.com
State New
Headers show

Commit Message

Changlong Xie Feb. 4, 2016, 10:17 a.m. UTC 
On 02/04/2016 05:07 PM, Dr. David Alan Gilbert wrote:
> * Changlong Xie (xiecl.fnst@cn.fujitsu.com) wrote:
>> On 02/01/2016 09:18 AM, Wen Congyang wrote:
>>> On 01/29/2016 06:47 PM, Dr. David Alan Gilbert wrote:
>>>> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>>>>> On 01/29/2016 06:07 PM, Dr. David Alan Gilbert wrote:
>>>>>> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>>>>>>> On 01/27/2016 07:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>>> Hi,
>>>>>>>>    I've got a block error if I kill the secondary.
>>>>>>>>
>>>>>>>> Start both primary & secondary
>>>>>>>> kill -9 secondary qemu
>>>>>>>> x_colo_lost_heartbeat on primary
>>>>>>>>
>>>>>>>> The guest sees a block error and the ext4 root switches to read-only.
>>>>>>>>
>>>>>>>> I gdb'd the primary with a breakpoint on quorum_report_bad; see
>>>>>>>> backtrace below.
>>>>>>>> (This is based on colo-v2.4-periodic-mode of the framework
>>>>>>>> code with the block and network proxy merged in; so it could be my
>>>>>>>> merging but I don't think so ?)
>>>>>>>>
>>>>>>>>
>>>>>>>> (gdb) where
>>>>>>>> #0  quorum_report_bad (node_name=0x7f2946a0892c "node0", ret=-5, acb=0x7f2946cb3910, acb=0x7f2946cb3910)
>>>>>>>>      at /root/colo/jan-2016/qemu/block/quorum.c:222
>>>>>>>> #1  0x00007f2943b23058 in quorum_aio_cb (opaque=<optimized out>, ret=<optimized out>)
>>>>>>>>      at /root/colo/jan-2016/qemu/block/quorum.c:315
>>>>>>>> #2  0x00007f2943b311be in bdrv_co_complete (acb=0x7f2946cb3f60) at /root/colo/jan-2016/qemu/block/io.c:2122
>>>>>>>> #3  0x00007f2943ae777d in aio_bh_call (bh=<optimized out>) at /root/colo/jan-2016/qemu/async.c:64
>>>>>>>> #4  aio_bh_poll (ctx=ctx@entry=0x7f2945b771d0) at /root/colo/jan-2016/qemu/async.c:92
>>>>>>>> #5  0x00007f2943af5090 in aio_dispatch (ctx=0x7f2945b771d0) at /root/colo/jan-2016/qemu/aio-posix.c:305
>>>>>>>> #6  0x00007f2943ae756e in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>,
>>>>>>>>      user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
>>>>>>>> #7  0x00007f293b84a79a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
>>>>>>>> #8  0x00007f2943af3a00 in glib_pollfds_poll () at /root/colo/jan-2016/qemu/main-loop.c:211
>>>>>>>> #9  os_host_main_loop_wait (timeout=<optimized out>) at /root/colo/jan-2016/qemu/main-loop.c:256
>>>>>>>> #10 main_loop_wait (nonblocking=<optimized out>) at /root/colo/jan-2016/qemu/main-loop.c:504
>>>>>>>> #11 0x00007f29438529ee in main_loop () at /root/colo/jan-2016/qemu/vl.c:1945
>>>>>>>> #12 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /root/colo/jan-2016/qemu/vl.c:4707
>>>>>>>>
>>>>>>>> (gdb) p s->num_children
>>>>>>>> $1 = 2
>>>>>>>> (gdb) p acb->success_count
>>>>>>>> $2 = 0
>>>>>>>> (gdb) p acb->is_read
>>>>>>>> $5 = false
>>>>>>>
>>>>>>> Sorry for the late reply.
>>>>>>
>>>>>> No problem.
>>>>>>
>>>>>>> What it the value of acb->count?
>>>>>>
>>>>>> (gdb) p acb->count
>>>>>> $1 = 1
>>>>>
>>>>> Note, the count is 1, not 2. Writing to children.0 is in flight. If writing to children.0 successes,
>>>>> the guest doesn't know this error.
>>>>>>> If secondary host is down, you should remove quorum's children.1. Otherwise, you will get
>>>>>>> I/O error event.
>>>>>>
>>>>>> Is that safe?  If the secondary fails, do you always have time to issue the command to
>>>>>> remove the children.1  before the guest sees the error?
>>>>>
>>>>> We will write to two children, and expect that writing to children.0 will success. If so,
>>>>> the guest doesn't know this error. You just get the I/O error event.
>>>>
>>>> I think children.0 is the disk, and that should be OK - so only the children.1/replication should
>>>> be failing - so in that case why do I see the error?
>>>
>>> I don't know, and I will check the codes.
>>>
>>>> The 'node0' in the backtrace above is the name of the replication, so it does look like the error
>>>> is coming from the replication.
>>>
>>> No, the backtrace is just report an I/O error events to the management application.
>>>
>>>>
>>>>>> Anyway, I tried removing children.1 but it segfaults now, I guess the replication is unhappy:
>>>>>>
>>>>>> (qemu) x_block_change colo-disk0 -d children.1
>>>>>> (qemu) x_colo_lost_heartbeat
>>>>>
>>>>> Hmm, you should not remove the child before failover. I will check it how to avoid it in the codes.
>>>>
>>>>   But you said 'If secondary host is down, you should remove quorum's children.1' - is that not
>>>> what you meant?
>>>
>>> Yes, you should excute 'x_colo_lost_heartbeat' fist, and then excute 'x_block_change ... -d ...'.
>>>
>> Hi david
>
> Hi Xie,
>    Thanks for the response.
>
>> It seems we missed 'drive_del' command, and will document it in next
>> version. Here is the right commands order:
>>
>> { "execute": "x-colo-lost-heartbeat" }
>> { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk',
>> 'child': 'children.1'}}
>> { 'execute': 'human-monitor-command', 'arguments': {'command-line':
>> 'drive_del xxxxx'}}
>
> OK,  however, you should fix the seg fault if you don't issue the drive_del;
> qemu should never crash.
> (Also I still get the IO error in the guest if I do the x-colo-lost-heartbeat).
>

Here is a quick fix, i just tested for several times. It work well to me.

     bugfix

     Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>



> Dave
>
>> Thanks
>> 	-Xie
>>>>
>>>>>> 12973 Segmentation fault      (core dumped) ./try/x86_64-softmmu/qemu-system-x86_64 -enable-kvm $console_param -S -boot c -m 4080 -smp 4 -machine pc-i440fx-2.5,accel=kvm -name debug-threads=on -trace events=trace-file -device virtio-rng-pci $block_param $net_param
>>>>>>
>>>>>> #0  0x00007f0a398a864c in bdrv_stop_replication (bs=0x7f0a3b0a8430, failover=true, errp=0x7fff6a5c3420)
>>>>>>      at /root/colo/jan-2016/qemu/block.c:4426
>>>>>>
>>>>>> (gdb) p drv
>>>>>> $1 = (BlockDriver *) 0x5d2a
>>>>>>
>>>>>>    it looks like the whole of bs is bogus.
>>>>>>
>>>>>> #1  0x00007f0a398d87f6 in quorum_stop_replication (bs=<optimized out>, failover=<optimized out>,
>>>>>>      errp=<optimized out>) at /root/colo/jan-2016/qemu/block/quorum.c:1213
>>>>>>
>>>>>> (gdb) p s->replication_index
>>>>>> $3 = 1
>>>>>>
>>>>>> I guess quorum_del_child needs to stop replication before it removes the child?
>>>>>
>>>>> Yes, but in the newest version, quorum doesn't know the block replication, and I think
>>>>> we shoud add an reference to the bs when starting block replication.
>>>>
>>>> Do you have a new version ready to test?  I'm interested to try it (and also interested
>>>> to try the latest version of the colo-proxy)
>>>
>>> I think we can post the newest version this week.
>>>
>>> Thanks
>>> Wen Congyang
>>>
>>>>
>>>> Dave
>>>>
>>>>> Thanks
>>>>> Wen Congyang
>>>>>
>>>>>> (although it would have to be careful not to block on the dead nbd).
>>>>>>
>>>>>> #2  0x00007f0a398a8901 in bdrv_stop_replication_all (failover=failover@entry=true, errp=errp@entry=0x7fff6a5c3478)
>>>>>>      at /root/colo/jan-2016/qemu/block.c:4504
>>>>>> #3  0x00007f0a3984b0af in primary_vm_do_failover () at /root/colo/jan-2016/qemu/migration/colo.c:144
>>>>>> #4  colo_do_failover (s=<optimized out>) at /root/colo/jan-2016/qemu/migration/colo.c:162
>>>>>> #5  0x00007f0a3989d7fd in aio_bh_call (bh=<optimized out>) at /root/colo/jan-2016/qemu/async.c:64
>>>>>> #6  aio_bh_poll (ctx=ctx@entry=0x7f0a3a6c21d0) at /root/colo/jan-2016/qemu/async.c:92
>>>>>> #7  0x00007f0a398ab110 in aio_dispatch (ctx=0x7f0a3a6c21d0) at /root/colo/jan-2016/qemu/aio-posix.c:305
>>>>>> #8  0x00007f0a3989d5ee in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>,
>>>>>>      user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
>>>>>> #9  0x00007f0a3160079a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
>>>>>> #10 0x00007f0a398a9a80 in glib_pollfds_poll () at /root/colo/jan-2016/qemu/main-loop.c:211
>>>>>> #11 os_host_main_loop_wait (timeout=<optimized out>) at /root/colo/jan-2016/qemu/main-loop.c:256
>>>>>> #12 main_loop_wait (nonblocking=<optimized out>) at /root/colo/jan-2016/qemu/main-loop.c:504
>>>>>> #13 0x00007f0a396089ee in main_loop () at /root/colo/jan-2016/qemu/vl.c:1945
>>>>>> #14 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /root/colo/jan-2016/qemu/vl.c:4707
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>>> Thanks
>>>>>>> Wen Congyang
>>>>>>>
>>>>>>>>
>>>>>>>> (qemu) info block
>>>>>>>> colo-disk0 (#block080): json:{"children": [{"driver": "raw", "file": {"driver": "file", "filename": "/root/colo/bugzilla.raw"}}, {"driver": "replication", "mode": "primary", "file": {"port": "8889", "host": "ibpair", "driver": "nbd", "export": "colo-disk0"}}], "driver": "quorum", "blkverify": false, "rewrite-corrupted": false, "vote-threshold": 1} (quorum)
>>>>>>>>      Cache mode:       writeback, direct
>>>>>>>>
>>>>>>>> Dave
>>>>>>>>
>>>>>>>> * Changlong Xie (xiecl.fnst@cn.fujitsu.com) wrote:
>>>>>>>>> Block replication is a very important feature which is used for
>>>>>>>>> continuous checkpoints(for example: COLO).
>>>>>>>>>
>>>>>>>>> You can get the detailed information about block replication from here:
>>>>>>>>> http://wiki.qemu.org/Features/BlockReplication
>>>>>>>>>
>>>>>>>>> Usage:
>>>>>>>>> Please refer to docs/block-replication.txt
>>>>>>>>>
>>>>>>>>> This patch series is based on the following patch series:
>>>>>>>>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg04570.html
>>>>>>>>>
>>>>>>>>> You can get the patch here:
>>>>>>>>> https://github.com/Pating/qemu/tree/changlox/block-replication-v13
>>>>>>>>>
>>>>>>>>> You can get the patch with framework here:
>>>>>>>>> https://github.com/Pating/qemu/tree/changlox/colo_framework_v12
>>>>>>>>>
>>>>>>>>> TODO:
>>>>>>>>> 1. Continuous block replication. It will be started after basic functions
>>>>>>>>>     are accepted.
>>>>>>>>>
>>>>>>>>> Changs Log:
>>>>>>>>> V13:
>>>>>>>>> 1. Rebase to the newest codes
>>>>>>>>> 2. Remove redundant marcos and semicolon in replication.c
>>>>>>>>> 3. Fix typos in block-replication.txt
>>>>>>>>> V12:
>>>>>>>>> 1. Rebase to the newest codes
>>>>>>>>> 2. Use backing reference to replcace 'allow-write-backing-file'
>>>>>>>>> V11:
>>>>>>>>> 1. Reopen the backing file when starting blcok replication if it is not
>>>>>>>>>     opened in R/W mode
>>>>>>>>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>>>>>>>>     when opening backing file
>>>>>>>>> 3. Block the top BDS so there is only one block job for the top BDS and
>>>>>>>>>     its backing chain.
>>>>>>>>> V10:
>>>>>>>>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>>>>>>>>     reference.
>>>>>>>>> 2. Address the comments from Eric Blake
>>>>>>>>> V9:
>>>>>>>>> 1. Update the error messages
>>>>>>>>> 2. Rebase to the newest qemu
>>>>>>>>> 3. Split child add/delete support. These patches are sent in another patchset.
>>>>>>>>> V8:
>>>>>>>>> 1. Address Alberto Garcia's comments
>>>>>>>>> V7:
>>>>>>>>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>>>>>>>>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
>>>>>>>>> V6:
>>>>>>>>> 1. Rebase to the newest qemu.
>>>>>>>>> V5:
>>>>>>>>> 1. Address the comments from Gong Lei
>>>>>>>>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>>>>>>>>     if there are too many I/O requests.
>>>>>>>>> V4:
>>>>>>>>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>>>>>>>>> V3:
>>>>>>>>> 1: use error_setg() instead of error_set()
>>>>>>>>> 2. Add a new block job API
>>>>>>>>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>>>>>>>>> 4. Add a testcase to test new hbitmap API
>>>>>>>>> V2:
>>>>>>>>> 1. Redesign the secondary qemu(use image-fleecing)
>>>>>>>>> 2. Use Error objects to return error message
>>>>>>>>> 3. Address the comments from Max Reitz and Eric Blake
>>>>>>>>>
>>>>>>>>> Wen Congyang (10):
>>>>>>>>>    unblock backup operations in backing file
>>>>>>>>>    Store parent BDS in BdrvChild
>>>>>>>>>    Backup: clear all bitmap when doing block checkpoint
>>>>>>>>>    Allow creating backup jobs when opening BDS
>>>>>>>>>    docs: block replication's description
>>>>>>>>>    Add new block driver interfaces to control block replication
>>>>>>>>>    quorum: implement block driver interfaces for block replication
>>>>>>>>>    Implement new driver for block replication
>>>>>>>>>    support replication driver in blockdev-add
>>>>>>>>>    Add a new API to start/stop replication, do checkpoint to all BDSes
>>>>>>>>>
>>>>>>>>>   block.c                    | 145 ++++++++++++
>>>>>>>>>   block/Makefile.objs        |   3 +-
>>>>>>>>>   block/backup.c             |  14 ++
>>>>>>>>>   block/quorum.c             |  78 +++++++
>>>>>>>>>   block/replication.c        | 545 +++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>   blockjob.c                 |  11 +
>>>>>>>>>   docs/block-replication.txt | 227 +++++++++++++++++++
>>>>>>>>>   include/block/block.h      |   9 +
>>>>>>>>>   include/block/block_int.h  |  15 ++
>>>>>>>>>   include/block/blockjob.h   |  12 +
>>>>>>>>>   qapi/block-core.json       |  33 ++-
>>>>>>>>>   11 files changed, 1089 insertions(+), 3 deletions(-)
>>>>>>>>>   create mode 100644 block/replication.c
>>>>>>>>>   create mode 100644 docs/block-replication.txt
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> 1.9.3
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>>>>
>>>>>>>>
>>>>>>>> .
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>>
>>>>>>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>
>>>>
>>>> .
>>>>
>>>
>>> .
>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> .
>
diff mbox

Patch

diff --git a/block/quorum.c b/block/quorum.c
index e5a7e4f..f4f1d28 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -458,6 +458,11 @@  static QuorumVoteVersion 
*quorum_get_vote_winner(QuorumVotes *votes)
          if (candidate->vote_count > max) {
              max = candidate->vote_count;
              winner = candidate;
+            continue;
+        }
+        if (candidate->vote_count == max &&
+                    candidate->value.l > winner->value.l) {
+            winner = candidate;
          }
      }