From patchwork Tue Nov 29 11:32:27 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 700459 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3tShQk1df9z9vFP for ; Tue, 29 Nov 2016 22:39:10 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="K5FjVE+e"; dkim-atps=neutral Received: from localhost ([::1]:36222 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cBgkd-0005Y4-Iq for incoming@patchwork.ozlabs.org; Tue, 29 Nov 2016 06:39:07 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34093) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cBgfL-00011f-9V for qemu-devel@nongnu.org; Tue, 29 Nov 2016 06:33:43 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cBgfG-0003Fl-Jw for qemu-devel@nongnu.org; Tue, 29 Nov 2016 06:33:39 -0500 Received: from mail-wm0-f67.google.com ([74.125.82.67]:35818) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cBgfG-0003Ez-9T for qemu-devel@nongnu.org; Tue, 29 Nov 2016 06:33:34 -0500 Received: by mail-wm0-f67.google.com with SMTP id a20so23844671wme.2 for ; Tue, 29 Nov 2016 03:33:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=F1lV1JDcLsV6EjZhrlC04xLIhLNJPZPE+99XFtYaYqM=; b=K5FjVE+e5Hn0LsVT/tj2cLDbUR9/fEyNiPZZzf9WjhkTp+5Pij01F4+Fd5mbKICpfX O06o87nl3lGEymNfdBCX2rOCt2SuSKyHlmZP0n7GuXI5gucXFFZ1/IbCvaA2jqj7UU8E nqBZPDSVijCgIp23F5Fw2eEAS4cP8wjCRTCFjKzAYxr7QN3wzQocqHSUcEHUlRftLuC6 Q+fYPrbn5ViNhM9ERFTHas81pJ7NnaRtzzhhADbUbwRjbdoDjRr9C1Pr4B2K2jIz0OUj NlGixmhqlGCztrIAUjQv8q2wwQOgCztTS+cRuQrqvc16AAfWm0x3xLdz5rahJCFAiDJ2 ue9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=F1lV1JDcLsV6EjZhrlC04xLIhLNJPZPE+99XFtYaYqM=; b=Tj+ct3dxLVWR9p1f2+iGgXkZRS4eZN60rZy5mzf//aFJtRf23gpWfRWYkOekPk7SKL 9v9bIxhSGeCVHTWF6JHDo70TYPkNZUurxhbFyjFBCytQP8Jy83RVRXEqLRlJeIrnBHoh buFqYYbkcbwd28e6BKQZuiyp5p3/oNx5CDH34vn7yuAAYLNofH+zxPJJLH2wLgdHmnNo p2coBhjldrPk32RpI2sWRA820LG9QpYNJwImDIkkz9EetoexDAh8A2Srvf+h5TpcbKz+ LWqGXFV/NmpQAkbMHFIBtNGYg7/NTccpIAIB5Db7kED6O/Sl3c2bK7zgdhOzod5RsAoo Zcpw== X-Gm-Message-State: AKaTC03nn7Rp/cwKXX8RCtASSrBUe/EKEn5yYwFOISjL/rjCKmXW/A8of1y4E0notM9ugQ== X-Received: by 10.28.10.147 with SMTP id 141mr22974175wmk.65.1480419152809; Tue, 29 Nov 2016 03:32:32 -0800 (PST) Received: from donizetti.lan (94-39-142-128.adsl-ull.clienti.tiscali.it. [94.39.142.128]) by smtp.gmail.com with ESMTPSA id vr9sm67301750wjc.35.2016.11.29.03.32.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 Nov 2016 03:32:32 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 29 Nov 2016 12:32:27 +0100 Message-Id: <20161129113230.32621-2-pbonzini@redhat.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20161129113230.32621-1-pbonzini@redhat.com> References: <20161129113230.32621-1-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 74.125.82.67 Subject: [Qemu-devel] [PATCH 2/5] sheepdog: reorganize coroutine flow X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Delimit co_recv's lifetime clearly in aio_read_response. Do a simple qemu_coroutine_enter in aio_read_response, letting sd_co_writev call sd_write_done. Handle nr_pending in the same way in sd_co_rw_vector, sd_write_done and sd_co_flush_to_disk. Remove sd_co_rw_vector's return value; just leave with no pending requests. --- block/sheepdog.c | 115 ++++++++++++++++++++----------------------------------- 1 file changed, 41 insertions(+), 74 deletions(-) diff --git a/block/sheepdog.c b/block/sheepdog.c index 0b30524..84b6645 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -345,8 +345,6 @@ struct SheepdogAIOCB { enum AIOCBState aiocb_type; Coroutine *coroutine; - void (*aio_done_func)(SheepdogAIOCB *); - int nr_pending; uint32_t min_affect_data_idx; @@ -449,14 +447,13 @@ static const char * sd_strerror(int err) * * 1. In sd_co_rw_vector, we send the I/O requests to the server and * link the requests to the inflight_list in the - * BDRVSheepdogState. The function exits without waiting for + * BDRVSheepdogState. The function yields while waiting for * receiving the response. * * 2. We receive the response in aio_read_response, the fd handler to - * the sheepdog connection. If metadata update is needed, we send - * the write request to the vdi object in sd_write_done, the write - * completion function. We switch back to sd_co_readv/writev after - * all the requests belonging to the AIOCB are finished. + * the sheepdog connection. We switch back to sd_co_readv/sd_writev + * after all the requests belonging to the AIOCB are finished. If + * needed, sd_co_writev will send another requests for the vdi object. */ static inline AIOReq *alloc_aio_req(BDRVSheepdogState *s, SheepdogAIOCB *acb, @@ -491,12 +488,6 @@ static inline void free_aio_req(BDRVSheepdogState *s, AIOReq *aio_req) acb->nr_pending--; } -static void coroutine_fn sd_finish_aiocb(SheepdogAIOCB *acb) -{ - qemu_coroutine_enter(acb->coroutine); - qemu_aio_unref(acb); -} - static const AIOCBInfo sd_aiocb_info = { .aiocb_size = sizeof(SheepdogAIOCB), }; @@ -517,7 +508,6 @@ static SheepdogAIOCB *sd_aio_setup(BlockDriverState *bs, QEMUIOVector *qiov, acb->sector_num = sector_num; acb->nb_sectors = nb_sectors; - acb->aio_done_func = NULL; acb->coroutine = qemu_coroutine_self(); acb->ret = 0; acb->nr_pending = 0; @@ -788,9 +778,6 @@ static void coroutine_fn aio_read_response(void *opaque) switch (acb->aiocb_type) { case AIOCB_WRITE_UDATA: - /* this coroutine context is no longer suitable for co_recv - * because we may send data to update vdi objects */ - s->co_recv = NULL; if (!is_data_obj(aio_req->oid)) { break; } @@ -838,6 +825,11 @@ static void coroutine_fn aio_read_response(void *opaque) } } + /* No more data for this aio_req (reload_inode below uses its own file + * descriptor handler which doesn't use co_recv). + */ + s->co_recv = NULL; + switch (rsp.result) { case SD_RES_SUCCESS: break; @@ -855,7 +847,7 @@ static void coroutine_fn aio_read_response(void *opaque) aio_req->oid = vid_to_vdi_oid(s->inode.vdi_id); } resend_aioreq(s, aio_req); - goto out; + return; default: acb->ret = -EIO; error_report("%s", sd_strerror(rsp.result)); @@ -868,13 +860,10 @@ static void coroutine_fn aio_read_response(void *opaque) * We've finished all requests which belong to the AIOCB, so * we can switch back to sd_co_readv/writev now. */ - acb->aio_done_func(acb); + qemu_coroutine_enter(acb->coroutine); } -out: - s->co_recv = NULL; - return; + err: - s->co_recv = NULL; reconnect_to_sdog(opaque); } @@ -1973,7 +1962,6 @@ static int sd_truncate(BlockDriverState *bs, int64_t offset) /* * This function is called after writing data objects. If we need to * update metadata, this sends a write request to the vdi object. - * Otherwise, this switches back to sd_co_readv/writev. */ static void coroutine_fn sd_write_done(SheepdogAIOCB *acb) { @@ -1986,6 +1974,7 @@ static void coroutine_fn sd_write_done(SheepdogAIOCB *acb) mx = acb->max_dirty_data_idx; if (mn <= mx) { /* we need to update the vdi object. */ + ++acb->nr_pending; offset = sizeof(s->inode) - sizeof(s->inode.data_vdi_id) + mn * sizeof(s->inode.data_vdi_id[0]); data_len = (mx - mn + 1) * sizeof(s->inode.data_vdi_id[0]); @@ -1999,13 +1988,10 @@ static void coroutine_fn sd_write_done(SheepdogAIOCB *acb) data_len, offset, 0, false, 0, offset); QLIST_INSERT_HEAD(&s->inflight_aio_head, aio_req, aio_siblings); add_aio_request(s, aio_req, &iov, 1, AIOCB_WRITE_UDATA); - - acb->aio_done_func = sd_finish_aiocb; - acb->aiocb_type = AIOCB_WRITE_UDATA; - return; + if (--acb->nr_pending) { + qemu_coroutine_yield(); + } } - - sd_finish_aiocb(acb); } /* Delete current working VDI on the snapshot chain */ @@ -2117,7 +2103,7 @@ out: * Returns 1 when we need to wait a response, 0 when there is no sent * request and -errno in error cases. */ -static int coroutine_fn sd_co_rw_vector(void *p) +static void coroutine_fn sd_co_rw_vector(void *p) { SheepdogAIOCB *acb = p; int ret = 0; @@ -2138,7 +2124,7 @@ static int coroutine_fn sd_co_rw_vector(void *p) ret = sd_create_branch(s); if (ret) { acb->ret = -EIO; - goto out; + return; } } @@ -2212,11 +2198,9 @@ static int coroutine_fn sd_co_rw_vector(void *p) idx++; done += len; } -out: - if (!--acb->nr_pending) { - return acb->ret; + if (--acb->nr_pending) { + qemu_coroutine_yield(); } - return 1; } static bool check_overlapping_aiocb(BDRVSheepdogState *s, SheepdogAIOCB *aiocb) @@ -2249,7 +2233,6 @@ static coroutine_fn int sd_co_writev(BlockDriverState *bs, int64_t sector_num, } acb = sd_aio_setup(bs, qiov, sector_num, nb_sectors); - acb->aio_done_func = sd_write_done; acb->aiocb_type = AIOCB_WRITE_UDATA; retry: @@ -2258,20 +2241,14 @@ retry: goto retry; } - ret = sd_co_rw_vector(acb); - if (ret <= 0) { - QLIST_REMOVE(acb, aiocb_siblings); - qemu_co_queue_restart_all(&s->overlapping_queue); - qemu_aio_unref(acb); - return ret; - } - - qemu_coroutine_yield(); + sd_co_rw_vector(acb); + sd_write_done(acb); QLIST_REMOVE(acb, aiocb_siblings); qemu_co_queue_restart_all(&s->overlapping_queue); - - return acb->ret; + ret = acb->ret; + qemu_aio_unref(acb); + return ret; } static coroutine_fn int sd_co_readv(BlockDriverState *bs, int64_t sector_num, @@ -2283,7 +2260,6 @@ static coroutine_fn int sd_co_readv(BlockDriverState *bs, int64_t sector_num, acb = sd_aio_setup(bs, qiov, sector_num, nb_sectors); acb->aiocb_type = AIOCB_READ_UDATA; - acb->aio_done_func = sd_finish_aiocb; retry: if (check_overlapping_aiocb(s, acb)) { @@ -2291,25 +2267,20 @@ retry: goto retry; } - ret = sd_co_rw_vector(acb); - if (ret <= 0) { - QLIST_REMOVE(acb, aiocb_siblings); - qemu_co_queue_restart_all(&s->overlapping_queue); - qemu_aio_unref(acb); - return ret; - } - - qemu_coroutine_yield(); + sd_co_rw_vector(acb); QLIST_REMOVE(acb, aiocb_siblings); qemu_co_queue_restart_all(&s->overlapping_queue); - return acb->ret; + ret = acb->ret; + qemu_aio_unref(acb); + return ret; } static int coroutine_fn sd_co_flush_to_disk(BlockDriverState *bs) { BDRVSheepdogState *s = bs->opaque; SheepdogAIOCB *acb; + int ret; AIOReq *aio_req; if (s->cache_flags != SD_FLAG_CMD_CACHE) { @@ -2318,15 +2289,19 @@ static int coroutine_fn sd_co_flush_to_disk(BlockDriverState *bs) acb = sd_aio_setup(bs, NULL, 0, 0); acb->aiocb_type = AIOCB_FLUSH_CACHE; - acb->aio_done_func = sd_finish_aiocb; + acb->nr_pending++; aio_req = alloc_aio_req(s, acb, vid_to_vdi_oid(s->inode.vdi_id), 0, 0, 0, false, 0, 0); QLIST_INSERT_HEAD(&s->inflight_aio_head, aio_req, aio_siblings); add_aio_request(s, aio_req, NULL, 0, acb->aiocb_type); - qemu_coroutine_yield(); - return acb->ret; + if (--acb->nr_pending) { + qemu_coroutine_yield(); + } + ret = acb->ret; + qemu_aio_unref(acb); + return ret; } static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) @@ -2783,7 +2758,6 @@ static coroutine_fn int sd_co_pdiscard(BlockDriverState *bs, int64_t offset, acb = sd_aio_setup(bs, &discard_iov, offset >> BDRV_SECTOR_BITS, count >> BDRV_SECTOR_BITS); acb->aiocb_type = AIOCB_DISCARD_OBJ; - acb->aio_done_func = sd_finish_aiocb; retry: if (check_overlapping_aiocb(s, acb)) { @@ -2791,20 +2765,13 @@ retry: goto retry; } - ret = sd_co_rw_vector(acb); - if (ret <= 0) { - QLIST_REMOVE(acb, aiocb_siblings); - qemu_co_queue_restart_all(&s->overlapping_queue); - qemu_aio_unref(acb); - return ret; - } - - qemu_coroutine_yield(); + sd_co_rw_vector(acb); QLIST_REMOVE(acb, aiocb_siblings); qemu_co_queue_restart_all(&s->overlapping_queue); - - return acb->ret; + ret = acb->ret; + qemu_aio_unref(acb); + return ret; } static coroutine_fn int64_t