From patchwork Mon Oct 7 12:09:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1172745 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46n0Cv565kz9sCJ for ; Mon, 7 Oct 2019 23:29:31 +1100 (AEDT) Received: from localhost ([::1]:44038 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iHS8n-00022Y-EX for incoming@patchwork.ozlabs.org; Mon, 07 Oct 2019 08:29:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48519) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iHRrw-0003Sp-Gg for qemu-devel@nongnu.org; Mon, 07 Oct 2019 08:12:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iHRrt-0005O8-WB for qemu-devel@nongnu.org; Mon, 07 Oct 2019 08:12:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57484) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iHRrq-0005MZ-1J; Mon, 07 Oct 2019 08:11:58 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3D7EA4E926; Mon, 7 Oct 2019 12:11:57 +0000 (UTC) Received: from localhost (unknown [10.36.118.98]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1E9410013A1; Mon, 7 Oct 2019 12:11:55 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Subject: [PATCH 16/16] block/io_uring: adds fd registration Date: Mon, 7 Oct 2019 13:09:37 +0100 Message-Id: <20191007120937.5862-17-stefanha@redhat.com> In-Reply-To: <20191007120937.5862-1-stefanha@redhat.com> References: <20191007120937.5862-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Mon, 07 Oct 2019 12:11:57 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: oleksandr@redhat.com, Kevin Wolf , qemu-block@nongnu.org, Julia Suvorova , Julia Suvorova , Markus Armbruster , Max Reitz , Stefan Hajnoczi , Paolo Bonzini , Fam Zheng , Aarushi Mehta Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Aarushi Mehta File descriptor registration eliminates the need to get file descriptors for each request on the host kernel side. The host kernel keeps references to a set of files that were registered by the application. These files can be quickly accessed by index instead of a traditional file descriptor number. Signed-off-by: Aarushi Mehta Signed-off-by: Stefan Hajnoczi --- v10: * Add io_uring_register_files() return value to trace_luring_fd_register() * Fix indentation in luring_fd_unregister() * Set s->fd_reg.fd_array to NULL after g_free() to avoid dangling pointers * Simplify fd registration code * Add luring_fd_unregister() and call it from file-posix.c to prevent fd leaks * Add trace_luring_fd_unregister() trace event --- include/block/raw-aio.h | 1 + block/file-posix.c | 18 +++++ block/io_uring.c | 150 +++++++++++++++++++++++++++++++++++++++- block/trace-events | 2 + 4 files changed, 170 insertions(+), 1 deletion(-) diff --git a/include/block/raw-aio.h b/include/block/raw-aio.h index 251b10d273..5fe6481bc4 100644 --- a/include/block/raw-aio.h +++ b/include/block/raw-aio.h @@ -68,6 +68,7 @@ void luring_detach_aio_context(LuringState *s, AioContext *old_context); void luring_attach_aio_context(LuringState *s, AioContext *new_context); void luring_io_plug(BlockDriverState *bs, LuringState *s); void luring_io_unplug(BlockDriverState *bs, LuringState *s); +void luring_fd_unregister(LuringState *s, int fd); #endif #ifdef _WIN32 diff --git a/block/file-posix.c b/block/file-posix.c index 264c44fc3f..604b5427c3 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -2010,6 +2010,20 @@ static int raw_co_flush_to_disk(BlockDriverState *bs) return raw_thread_pool_submit(bs, handle_aiocb_flush, &acb); } +static void raw_aio_detach_aio_context(BlockDriverState *bs) +{ +#ifdef CONFIG_LINUX_IO_URING + BDRVRawState *s = bs->opaque; + + if (s->fd >= 0 && s->use_linux_io_uring) { + LuringState *luring; + + luring = aio_get_linux_io_uring(bdrv_get_aio_context(bs)); + luring_fd_unregister(luring, s->fd); + } +#endif +} + static void raw_aio_attach_aio_context(BlockDriverState *bs, AioContext *new_context) { @@ -2994,6 +3008,7 @@ BlockDriver bdrv_file = { .bdrv_refresh_limits = raw_refresh_limits, .bdrv_io_plug = raw_aio_plug, .bdrv_io_unplug = raw_aio_unplug, + .bdrv_detach_aio_context = raw_aio_detach_aio_context, .bdrv_attach_aio_context = raw_aio_attach_aio_context, .bdrv_co_truncate = raw_co_truncate, @@ -3470,6 +3485,7 @@ static BlockDriver bdrv_host_device = { .bdrv_refresh_limits = raw_refresh_limits, .bdrv_io_plug = raw_aio_plug, .bdrv_io_unplug = raw_aio_unplug, + .bdrv_detach_aio_context = raw_aio_detach_aio_context, .bdrv_attach_aio_context = raw_aio_attach_aio_context, .bdrv_co_truncate = raw_co_truncate, @@ -3594,6 +3610,7 @@ static BlockDriver bdrv_host_cdrom = { .bdrv_refresh_limits = raw_refresh_limits, .bdrv_io_plug = raw_aio_plug, .bdrv_io_unplug = raw_aio_unplug, + .bdrv_detach_aio_context = raw_aio_detach_aio_context, .bdrv_attach_aio_context = raw_aio_attach_aio_context, .bdrv_co_truncate = raw_co_truncate, @@ -3726,6 +3743,7 @@ static BlockDriver bdrv_host_cdrom = { .bdrv_refresh_limits = raw_refresh_limits, .bdrv_io_plug = raw_aio_plug, .bdrv_io_unplug = raw_aio_unplug, + .bdrv_detach_aio_context = raw_aio_detach_aio_context, .bdrv_attach_aio_context = raw_aio_attach_aio_context, .bdrv_co_truncate = raw_co_truncate, diff --git a/block/io_uring.c b/block/io_uring.c index 56892fd1ab..4d6e50ce19 100644 --- a/block/io_uring.c +++ b/block/io_uring.c @@ -46,10 +46,16 @@ typedef struct LuringQueue { QSIMPLEQ_HEAD(, LuringAIOCB) submit_queue; } LuringQueue; +typedef struct LuringFd { + int *fd_array; + GHashTable *fd_lookup; +} LuringFd; + typedef struct LuringState { AioContext *aio_context; struct io_uring ring; + LuringFd fd_reg; /* io queue for submit at batch. Protected by AioContext lock. */ LuringQueue io_q; @@ -298,6 +304,136 @@ static void ioq_init(LuringQueue *io_q) io_q->blocked = false; } +/** + * luring_fd_unregister: + * + * Remove a file descriptor from the registered fds list. This is a slow + * operation because all registered fds are refreshed, but this function should + * not be called often anyway. + * + * Only call this function while there are no requests in flight. + */ +void luring_fd_unregister(LuringState *s, int fd) +{ + LuringFd *fd_reg = &s->fd_reg; + void *value; + int idx; + int ret; + int nr; + int i; + + if (!g_hash_table_lookup_extended(fd_reg->fd_lookup, GINT_TO_POINTER(fd), + NULL, &value)) { + return; + } + + ret = io_uring_unregister_files(&s->ring); + if (ret < 0) { + return; + } + + idx = GPOINTER_TO_INT(value); + nr = g_hash_table_size(fd_reg->fd_lookup) - 1; /* minus this fd */ + + trace_luring_fd_unregister(s, fd, idx); + + memmove(&fd_reg->fd_array[idx], &fd_reg->fd_array[idx + 1], nr - idx); + + /* Rebuild hash table */ + g_hash_table_remove_all(fd_reg->fd_lookup); + for (i = 0; i < nr; i++) { + g_hash_table_insert(fd_reg->fd_lookup, + GINT_TO_POINTER(fd_reg->fd_array[i]), + GINT_TO_POINTER(i)); + } + + io_uring_register_files(&s->ring, fd_reg->fd_array, nr); +} + +/** + * luring_fd_register: + * + * Register file descriptors, see luring_fd_lookup + */ +static int luring_fd_register(struct io_uring *ring, LuringFd *fd_reg, int fd) +{ + int ret, nr; + GHashTable *lookup = fd_reg->fd_lookup; + nr = g_hash_table_size(lookup); + + /* If adding new, API requires older registrations to be removed */ + if (nr) { + /* + * See linux b19062a56726, register needs the ring mutex, any + * submission in progress will complete before unregistering begins + * and new ones will have to wait. + */ + ret = io_uring_unregister_files(ring); + if (ret < 0) { + return ret; + } + } + + fd_reg->fd_array = g_realloc_n(fd_reg->fd_array, nr + 1, sizeof(int)); + fd_reg->fd_array[nr] = fd; + + g_hash_table_insert(lookup, GINT_TO_POINTER(fd), GINT_TO_POINTER(nr)); + ret = io_uring_register_files(ring, fd_reg->fd_array, nr + 1); + trace_luring_fd_register(fd, nr, ret); + if (ret < 0) { + /* Leave fd_array[] alone, fd will be overwritten next time anyway */ + g_hash_table_remove(lookup, GINT_TO_POINTER(fd)); + return ret; + } + return nr; +} + +/** + * luring_fd_init: + * + * Initialize file descriptors + */ +static void luring_fd_init(LuringState *s) +{ + s->fd_reg.fd_lookup = g_hash_table_new(g_direct_hash, g_direct_equal); +} + +/** + * luring_fd_cleanup: + * + * Unregisters file descriptors, TODO: error handling + */ +static void luring_fd_cleanup(LuringState *s) +{ + io_uring_unregister_files(&s->ring); + g_hash_table_unref(s->fd_reg.fd_lookup); + g_free(s->fd_reg.fd_array); + s->fd_reg.fd_array = NULL; +} + +/** + * luring_fd_lookup: + * + * Used to lookup fd index in registered array at submission time + * If the lookup table has not been created or the fd is not in the table, + * the fd is registered. + * + * If registration errors, this function returns -errno. + * + * Unregistering is done in luring_detach_aio_context(). + */ +static int luring_fd_lookup(LuringState *s, int fd) +{ + void *index; + + if (g_hash_table_lookup_extended(s->fd_reg.fd_lookup, GINT_TO_POINTER(fd), + NULL, &index)) { + return GPOINTER_TO_INT(index); + } + + return luring_fd_register(&s->ring, &s->fd_reg, fd); +} + void luring_io_plug(BlockDriverState *bs, LuringState *s) { trace_luring_io_plug(s); @@ -329,9 +465,14 @@ void luring_io_unplug(BlockDriverState *bs, LuringState *s) static int luring_do_submit(int fd, LuringAIOCB *luringcb, LuringState *s, uint64_t offset, int type) { - int ret; + int ret, fd_index; struct io_uring_sqe *sqes = &luringcb->sqeq; + fd_index = luring_fd_lookup(s, fd); + if (fd_index >= 0) { + fd = fd_index; + } + switch (type) { case QEMU_AIO_WRITE: io_uring_prep_writev(sqes, fd, luringcb->qiov->iov, @@ -349,7 +490,11 @@ static int luring_do_submit(int fd, LuringAIOCB *luringcb, LuringState *s, __func__, type); abort(); } + io_uring_sqe_set_data(sqes, luringcb); + if (fd_index >= 0) { + io_uring_sqe_set_flags(sqes, IOSQE_FIXED_FILE); + } QSIMPLEQ_INSERT_TAIL(&s->io_q.submit_queue, luringcb, next); s->io_q.in_queue++; @@ -375,6 +520,7 @@ int coroutine_fn luring_co_submit(BlockDriverState *bs, LuringState *s, int fd, .qiov = qiov, .is_read = (type == QEMU_AIO_READ), }; + trace_luring_co_submit(bs, s, &luringcb, fd, offset, qiov ? qiov->size : 0, type); ret = luring_do_submit(fd, &luringcb, s, offset, type); @@ -391,6 +537,7 @@ int coroutine_fn luring_co_submit(BlockDriverState *bs, LuringState *s, int fd, void luring_detach_aio_context(LuringState *s, AioContext *old_context) { + luring_fd_cleanup(s); aio_set_fd_handler(old_context, s->ring.ring_fd, false, NULL, NULL, NULL, s); qemu_bh_delete(s->completion_bh); @@ -403,6 +550,7 @@ void luring_attach_aio_context(LuringState *s, AioContext *new_context) s->completion_bh = aio_bh_new(new_context, qemu_luring_completion_bh, s); aio_set_fd_handler(s->aio_context, s->ring.ring_fd, false, qemu_luring_completion_cb, NULL, qemu_luring_poll_cb, s); + luring_fd_init(s); } LuringState *luring_init(Error **errp) diff --git a/block/trace-events b/block/trace-events index 2e3a9851b6..6a19cc00b7 100644 --- a/block/trace-events +++ b/block/trace-events @@ -72,6 +72,8 @@ luring_co_submit(void *bs, void *s, void *luringcb, int fd, uint64_t offset, siz luring_process_completion(void *s, void *aiocb, int ret) "LuringState %p luringcb %p ret %d" luring_io_uring_submit(void *s, int ret) "LuringState %p ret %d" luring_resubmit_short_read(void *s, void *luringcb, int nread) "LuringState %p luringcb %p nread %d" +luring_fd_register(int fd, int index, int ret) "fd %d index %d ret %d" +luring_fd_unregister(void *s, int fd, int index) "LuringState %p fd %d index %d" # qcow2.c qcow2_writev_start_req(void *co, int64_t offset, int bytes) "co %p offset 0x%" PRIx64 " bytes %d"