From patchwork Wed Jul 24 07:56:31 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: MORITA Kazutaka X-Patchwork-Id: 261334 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 62ECF2C0092 for ; Wed, 24 Jul 2013 18:07:30 +1000 (EST) Received: from localhost ([::1]:60910 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V1u6a-0005aS-6p for incoming@patchwork.ozlabs.org; Wed, 24 Jul 2013 04:07:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36078) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V1ty6-0000Wd-7y for qemu-devel@nongnu.org; Wed, 24 Jul 2013 03:58:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1V1ty4-0000fb-4o for qemu-devel@nongnu.org; Wed, 24 Jul 2013 03:58:42 -0400 Received: from sh.osrg.net ([192.16.179.4]:38797) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V1ty3-0000fF-Ga for qemu-devel@nongnu.org; Wed, 24 Jul 2013 03:58:40 -0400 Received: from fs.osrg.net (localns.osrg.net [10.0.0.11]) by sh.osrg.net (8.14.4/8.14.4/OSRG-NET) with ESMTP id r6O7wQDq014980; Wed, 24 Jul 2013 16:58:27 +0900 Received: from localhost (unknown [10.32.32.72]) by fs.osrg.net (Postfix) with ESMTP id 37040F94E; Wed, 24 Jul 2013 16:58:25 +0900 (JST) From: MORITA Kazutaka To: Kevin Wolf , Stefan Hajnoczi , Paolo Bonzini , qemu-devel@nongnu.org Date: Wed, 24 Jul 2013 16:56:31 +0900 Message-Id: <1374652593-7242-8-git-send-email-morita.kazutaka@lab.ntt.co.jp> X-Mailer: git-send-email 1.8.1.3.566.gaa39828 In-Reply-To: <1374652593-7242-1-git-send-email-morita.kazutaka@lab.ntt.co.jp> References: <1374652593-7242-1-git-send-email-morita.kazutaka@lab.ntt.co.jp> X-Dispatcher: imput version 20100215(IM150) Lines: 186 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.3.9 (sh.osrg.net [192.16.179.4]); Wed, 24 Jul 2013 16:58:27 +0900 (JST) X-Virus-Scanned: clamav-milter 0.97.8 at sh X-Virus-Status: Clean X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 192.16.179.4 Cc: sheepdog@lists.wpkg.org Subject: [Qemu-devel] [PATCH v2 7/9] sheepdog: try to reconnect to sheepdog after network error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This introduces a failed request queue and links all the inflight requests to the list after network error happens. After QEMU reconnects to the sheepdog server successfully, the sheepdog block driver will retry all the requests in the failed queue. Signed-off-by: MORITA Kazutaka --- block/sheepdog.c | 72 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/block/sheepdog.c b/block/sheepdog.c index cde887b..303354e 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -318,8 +318,11 @@ typedef struct BDRVSheepdogState { Coroutine *co_recv; uint32_t aioreq_seq_num; + + /* Every aio request must be linked to either of these queues. */ QLIST_HEAD(inflight_aio_head, AIOReq) inflight_aio_head; QLIST_HEAD(pending_aio_head, AIOReq) pending_aio_head; + QLIST_HEAD(failed_aio_head, AIOReq) failed_aio_head; } BDRVSheepdogState; static const char * sd_strerror(int err) @@ -613,6 +616,8 @@ static int coroutine_fn add_aio_request(BDRVSheepdogState *s, AIOReq *aio_req, enum AIOCBState aiocb_type); static int coroutine_fn resend_aioreq(BDRVSheepdogState *s, AIOReq *aio_req); static int reload_inode(BDRVSheepdogState *s, uint32_t snapid, const char *tag); +static int get_sheep_fd(BDRVSheepdogState *s); +static void co_write_request(void *opaque); static AIOReq *find_pending_req(BDRVSheepdogState *s, uint64_t oid) { @@ -654,6 +659,44 @@ static void coroutine_fn send_pending_req(BDRVSheepdogState *s, uint64_t oid) } } +static coroutine_fn void reconnect_to_sdog(void *opaque) +{ + BDRVSheepdogState *s = opaque; + AIOReq *aio_req, *next; + + qemu_aio_set_fd_handler(s->fd, NULL, NULL, NULL, NULL); + close(s->fd); + s->fd = -1; + + /* Wait for outstanding write requests to be completed. */ + while (s->co_send != NULL) { + co_write_request(opaque); + } + + /* Move all the inflight requests to the failed queue. */ + QLIST_FOREACH_SAFE(aio_req, &s->inflight_aio_head, aio_siblings, next) { + QLIST_REMOVE(aio_req, aio_siblings); + QLIST_INSERT_HEAD(&s->failed_aio_head, aio_req, aio_siblings); + } + + /* Try to reconnect the sheepdog server every one second. */ + while (s->fd < 0) { + s->fd = get_sheep_fd(s); + if (s->fd < 0) { + dprintf("Wait for connection to be established\n"); + co_aio_sleep_ns(1000000000ULL); + } + }; + + /* Resend all the failed aio requests. */ + while (!QLIST_EMPTY(&s->failed_aio_head)) { + aio_req = QLIST_FIRST(&s->failed_aio_head); + QLIST_REMOVE(aio_req, aio_siblings); + QLIST_INSERT_HEAD(&s->inflight_aio_head, aio_req, aio_siblings); + resend_aioreq(s, aio_req); + } +} + /* * Receive responses of the I/O requests. * @@ -670,15 +713,11 @@ static void coroutine_fn aio_read_response(void *opaque) SheepdogAIOCB *acb; uint64_t idx; - if (QLIST_EMPTY(&s->inflight_aio_head)) { - goto out; - } - /* read a header */ ret = qemu_co_recv(fd, &rsp, sizeof(rsp)); if (ret < sizeof(rsp)) { error_report("failed to get the header, %s", strerror(errno)); - goto out; + goto err; } /* find the right aio_req from the inflight aio list */ @@ -689,7 +728,7 @@ static void coroutine_fn aio_read_response(void *opaque) } if (!aio_req) { error_report("cannot find aio_req %x", rsp.id); - goto out; + goto err; } acb = aio_req->aiocb; @@ -729,7 +768,7 @@ static void coroutine_fn aio_read_response(void *opaque) aio_req->iov_offset, rsp.data_length); if (ret < rsp.data_length) { error_report("failed to get the data, %s", strerror(errno)); - goto out; + goto err; } break; case AIOCB_FLUSH_CACHE: @@ -763,10 +802,9 @@ static void coroutine_fn aio_read_response(void *opaque) if (s->inode.vdi_id == oid_to_vid(aio_req->oid)) { ret = reload_inode(s, 0, ""); if (ret < 0) { - goto out; + goto err; } } - if (is_data_obj(aio_req->oid)) { aio_req->oid = vid_to_data_oid(s->inode.vdi_id, data_oid_to_idx(aio_req->oid)); @@ -794,6 +832,10 @@ static void coroutine_fn aio_read_response(void *opaque) } out: s->co_recv = NULL; + return; +err: + s->co_recv = NULL; + reconnect_to_sdog(opaque); } static void co_read_response(void *opaque) @@ -819,7 +861,8 @@ static int aio_flush_request(void *opaque) BDRVSheepdogState *s = opaque; return !QLIST_EMPTY(&s->inflight_aio_head) || - !QLIST_EMPTY(&s->pending_aio_head); + !QLIST_EMPTY(&s->pending_aio_head) || + !QLIST_EMPTY(&s->failed_aio_head); } /* @@ -1094,23 +1137,21 @@ static int coroutine_fn add_aio_request(BDRVSheepdogState *s, AIOReq *aio_req, /* send a header */ ret = qemu_co_send(s->fd, &hdr, sizeof(hdr)); if (ret < sizeof(hdr)) { - qemu_co_mutex_unlock(&s->lock); error_report("failed to send a req, %s", strerror(errno)); - return -errno; + goto out; } if (wlen) { ret = qemu_co_sendv(s->fd, iov, niov, aio_req->iov_offset, wlen); if (ret < wlen) { - qemu_co_mutex_unlock(&s->lock); error_report("failed to send a data, %s", strerror(errno)); - return -errno; } } - +out: socket_set_cork(s->fd, 0); qemu_aio_set_fd_handler(s->fd, co_read_response, NULL, aio_flush_request, s); + s->co_send = NULL; qemu_co_mutex_unlock(&s->lock); return 0; @@ -1300,6 +1341,7 @@ static int sd_open(BlockDriverState *bs, QDict *options, int flags) QLIST_INIT(&s->inflight_aio_head); QLIST_INIT(&s->pending_aio_head); + QLIST_INIT(&s->failed_aio_head); s->fd = -1; memset(vdi, 0, sizeof(vdi));